Telnet BBSing double-spacing issue

[Divarin]

First off my apologies if this is the wrong place I wasn't quite sure where to ask this. I'm certain if anyone knows the answer to this it would be someone on this forum even though it's not about a specific computer or a specific piece of software.

I am working on making a terminal emulator for the C128, I have the CBM/PETSCII stuff going pretty well and now am working on ASCII/ANSI support.
What I am noticing is that some boards I connect to give me double newlines. It's not a newline followed by a line feed (char 10) I'm already filtering out 10's (line feeds).
I added some debugging features that let me see the decimal value of each byte so I can confirm I am actually getting two 13's (newlines) every time the board should be sending just one.

I also see the same behavior when I try to connect to those boards with certain systems such as ProTerm on my Apple //c. However if I connect with say SyncTerm (PC) or DesTerm, StrikeTerm, Novaterm, CCGMS etc... then it seems fine (just one newline and the end of each line).

My only guess is maybe there was some terminal negotiation going on during connection that my terminal (and ProTerm) wasn't playing along with and somehow the BBS decided I needed two newlines for each line (for ... reasons?!)

Any clue what is going on here?

 

[rlauzon]

What are you using as the connection to the BBS? If it's a Wifi modem, that would be a good place to look to see if there's some sort of translation setting.

 

[modem7]

... I am actually getting two 13's (newlines) ...

In ASCII, I think calling the carriage return (CR) byte a "newline" is misleading, because many systems will treat CR literally, which is: take the carriage/cursor back to the far left position on the same line (i.e. no linefeed [LF] included).

Accordingly, many ASCII terminals/software cater for systems that either, send a CR+LF at the end of each line, or only a CR at the end of each line (i.e. implied LF).
For example, the TeleVideo 910 ASCII terminal:

... I am actually getting two 13's (newlines) ...

As for possible purpose of {CR+CR}, 'double line spacing is required' is all that I can come up with: Sending system sending {CR+CR}, expecting that the receiving system will perform two LF's, one for each CR ?

Other than that, I see no purpose in sending {CR+CR}.

I added some debugging features that let me see the decimal value of each byte so I can confirm I am actually getting two 13's (newlines) every time the board should be sending just one.

Debugging code at the sender or receiver? If the latter, I think that you need to be careful about assumptions if there is something between the receiving UART and your 'debugging feature/s'. Ideally, a snooping device monitoring the serial cable. That could even be an oscilloscope, connected to the applicable data wire on the serial cable

 

[rlauzon]

A duplex issue?

If you have half-duplex set, the local computer echos what you type, and if the remote system also does so, you end up with duplicates.
But that would duplicate everything, not just carriage returns.

In Linux, the "newline" is just a carriage return.
But under systems like MS-DOS, a "newline" is a carriage return + line feed.

I wonder if that is part of the problem.

 

[cjs]

What I am noticing is that some boards I connect to give me double newlines. It's not a newline followed by a line feed (char 10) I'm already filtering out 10's (line feeds).
I added some debugging features that let me see the decimal value of each byte so I can confirm I am actually getting two 13's (newlines) every time the board should be sending just one.

As others have mentioned, first, it's best to use correct terminology. In the following, I use a dollar sign in front of a number to indicate it's hexadecimal (Motorola style).

In ASCII (and character sets/encodings that include ASCII, such as UTF-8), you are dealing with the following characters:

1. 10 or $0A, "line feed" or LF, which generally indicates that the cursor or print position should move down to the next line, without moving horizontally.
2. 13 or $0D, "carriage return" or CR, which generally indicates that the cursor or print position should move to the left-hand column, without moving it vertically.

A "newline" is not an ASCII character but a concept, usually referring to what separates lines within files and other text streams within an operating system. This is not necessarily the same thing that the OS's terminal driver will send to a terminal. For example, in CP/M and MS-DOS a CR-LF combination is considered to be a newline (and that is what's sent to a terminal when you TYPE a file), but in Unix a newline is considered to be a single LF character; the terminal driver will usually convert this to CR-LF when displaying text on a terminal.

In Linux, the "newline" is just a carriage return.

No, under Linux, just as any other Unix, a newline is an LF, not a CR. (Unix does sometimes use bare CRs, usually to go back and print over previous text so you can do things like underlining on printing terminals.) In the old MacOS, though, a newline was a CR. (I think MacOS X works like most Unixes: it's LF. But I'm not sure, and maybe it depends on things.)

 

[cjs]

What I am noticing is that some boards I connect to give me double newlines. It's not a newline followed by a line feed (char 10) I'm already filtering out 10's (line feeds).
I added some debugging features that let me see the decimal value of each byte so I can confirm I am actually getting two 13's (newlines) every time the board should be sending just one.

No, you're not getting double "newlines"; you're getting double carriage returns (CRs). I am not sure why it might be doing this (perhaps to add a little delay for slow systems to catch up), but you should handle the second CR the way you handled the first CR: move the cursor to the start of the current line. (The second one is clearly a no-op.)

 

[Divarin]

Okay double CRs then. I'm not getting double anything else.
I'm also not getting double CRs when not connected to a BBS and some BBSs I don't get double CRs at all.
Even on boards where I am getting double CRs sometimes I'm not, for example one BBS would give me double CRs everywhere except the menu prompt where I get just one CR per line.

How am I connecting? Multiple ways:
* VICE emulator using TCPSER
* C128 using Link232 cartridge
* Apple //c using a WiFiModem

The physical connection doesn't seem to matter, for example I can use another commodore based terminal emulator on the C128 or through VICE and there's no double CRs. I can't speak for ProTERM on the Apple //c but as for my own (work in progress) term program it's obviously something I'm doing or failing to do in software, I just can't figure out what yet.

As for the debugging info I'm not injecting any kind of snooping device I just mean that in my code I have a feature where I can review the last few bytes that came in and view their values in decimal format instead of printing the character.

 

[eswan]

Maybe a carriage return at the end of terminal width triggering a line-wrap carriage return?

 

[cjs]

Okay double CRs then. I'm not getting double anything else.
I'm also not getting double CRs when not connected to a BBS and some BBSs I don't get double CRs at all.
Even on boards where I am getting double CRs sometimes I'm not, for example one BBS would give me double CRs everywhere except the menu prompt where I get just one CR per line.

But what exactly is the issue here? Is it that your software isn't handling this properly? If so, make it handle CR properly: move to the start of the current line when you get a CR.

If not, you're not asking about your software but the particular BBS software you're connecting to, in which case you should specify what that is and give us a link to the source code, and perhaps from that we can find out why it's doing that. (Not that it makes any difference to the end user, who sees the same thing whether she gets one CR or a hundred.)

 

[mbbrutman]

I've written a terminal emulator/Telnet program before. Some things to think about:

• Are you really using the Telnet protocol, or are you using the term "Telnet" to refer to sending serial data over a TCP socket? There is a big difference. When writing a proper Telnet client that connects to a Telnet server you have to implement the Telnet protocol, and that includes negotiating character-at-time mode vs. line mode, 7 bit vs 8 bit, local echo vs. remote echo, etc. All of that happens up front at the start of the connection. A lot of people don't realize this and they are just talking about sending serial characters over a TCP/IP connection, like what the WiFi modems tend to do.
• In the full Telnet protocol a newline is always a "CR/LF" pair, regardless of your client or the host system. Both systems have to adjust and send the CR/LF pair.
• You should not see the effect of two CR characters in the row, as they both cause the cursor to go to the same exact place.

 

[Divarin]

I've written a terminal emulator/Telnet program before. Some things to think about:

• Are you really using the Telnet protocol, or are you using the term "Telnet" to refer to sending serial data over a TCP socket? There is a big difference. When writing a proper Telnet client that connects to a Telnet server you have to implement the Telnet protocol, and that includes negotiating character-at-time mode vs. line mode, 7 bit vs 8 bit, local echo vs. remote echo, etc. All of that happens up front at the start of the connection. A lot of people don't realize this and they are just talking about sending serial characters over a TCP/IP connection, like what the WiFi modems tend to do.
• In the full Telnet protocol a newline is always a "CR/LF" pair, regardless of your client or the host system. Both systems have to adjust and send the CR/LF pair.
• You should not see the effect of two CR characters in the row, as they both cause the cursor to go to the same exact place.

I think this is what I'm looking for, yes currently my term program is just sending serial data over a TCP socket, I would like it to properly implement the Telnet protocol, I just need some guidance on getting there.

I've started looking at the telnet protocol here: https://datatracker.ietf.org/doc/html/rfc854

"the "Interpret as Command" (IAC) escape character followed by the code
for the command. The commands dealing with option negotiation are
three byte sequences, the third byte being the code for the option
referenced. "

Here is a small chuck of bytes that I picked up while connecting to one of the boards that's giving me the double-CR:

51, 55, 59, 52, 48, 109, 32, 47, 79, 13, 13, 10, 255, 0, 13 109, 32, 47, 79, 13, 13, 10, 255, 0, 13, 10, 78, 79, 32, 67, 65, 82, 82, 73, 69, 82, 13, 10, 97, 47, 255, 1, 13, 10, 67, 79, 78, 78, 69,

255 is the IAC so I'm looking for what the command is after that. In the RFC the command codes start at 240 and go up from there. I'm not seeing any such bytes so I'm not sure how to interpret what I'm getting.

 

[Chuck(G)]

13, 13, 10, 255,

Carriage-return carriage-return, linefeed, rubout.
Clearly, the two CRs and the rubout were there to give time for the (mechanical) terminal to complete the action. Should result in single-spacing, unless your terminal interprets a carriage-return as having an implicit linefeed. The rubout is non-printing and is there to insert a delay in the data stream.

The usual for a TTY was CR LF RO RO.

 

[cjs]

51, 55, 59, 52, 48, 109, 32, 47, 79, 13, 13, 10, 255, 0, 13 109, 32, 47, 79, 13, 13, 10, 255, 0, 13, 10, 78, 79, 32, 67, 65, 82, 82, 73, 69, 82, 13, 10, 97, 47, 255, 1, 13, 10, 67, 79, 78, 78, 69,

255 is the IAC so I'm looking for what the command is after that.

Carriage-return carriage-return, linefeed, rubout.

Except that 255 is not a rubout (DEL), that's 127 ($7F). $FF is a non-ASCII character (usually, depending on your encoding).

And everything I'm aware of would use NUL, not DEL, for padding because DEL might, well, delete things. :-)

 

[cjs]

How am I connecting? Multiple ways:
* VICE emulator using TCPSER
* C128 using Link232 cartridge
* Apple //c using a WiFiModem

I'd missed the part in the title where you mentioned "Telnet," or had mistakenly assumed you were not referencing the specific protocol name. So now what you need to do is explain clearly where which protocols are being used. Which software is speaking Telnet, which software is speaking straight bytestream, and where is the conversion (if any) happening between the two?

The physical connection doesn't seem to matter, for example I can use another commodore based terminal emulator on the C128 or through VICE and there's no double CRs.

How exactly did you determine this? What were you measuring and where, both for your program and the other program?

 

[cjs]

Oh, and again, it would really help if you would describe exactly the problem you're trying to solve here. As I said in an earlier message, two CRs in a row is not a problem for standard terminal programs because the second one is a no-op. You talk later about the telnet protocol; is that the only thing you're trying to deal with, and the CR-CR sequence not a problem for you after all?

 

[furball1985]

just about every modern BBS is using a Telnet BBS. this is what gives the double 255 characters.
using a wifi modem they have a setting for telnet mode.
you have to use the command "ATNET1" this will collapse the extra 255 characters.

 

[Chuck(G)]

Except that 255 is not a rubout (DEL), that's 127 ($7F). $FF is a non-ASCII character (usually, depending on your encoding).

And everything I'm aware of would use NUL, not DEL, for padding because DEL might, well, delete things. :-)

Rubout when transmitted is a non-printing character to a teletype. It was used extensively as a padding character in transmission (eg TOPS). Parity is irrelevant in this usage.

RO is only alternately known as DEL because of its application in paper tape. When editing same, the only way to delete a character was to punch all holes over the character, which resulted in nonprinting and effective character deletion.

We called it RUBOUT back in the day. When you were punching a paper tape, you co... | Hacker News

news.ycombinator.com

 

[cjs]

Rubout when transmitted is a non-printing character to a teletype. It was used extensively as a padding character in transmission (eg TOPS). Parity is irrelevant in this usage.

Parity is irrelevant in any usage of the TELNET protocol because parity is neither provided nor available. The channel is always an 8-bit clear connection, with the exception of the IAC escape character (usually $FF). The default code set "is seven-bit USASCII in an eight-bit field," though this can of course be changed by agreement between the endpoints.

Thus, if anything's sending a DEL, it would be sending $7F, not $FF.

RO is only alternately known as DEL because of its application in paper tape. When editing same, the only way to delete a character was to punch all holes over the character, which resulted in nonprinting and effective character deletion.

"Rubout" is not a character name; like "newline" it's a conceptual command that may be implemented in different ways. DEL was not any sort of "alternate" name for the character; it was the one and only official name. From ANSI X3.4-1977, American National Standard Code for Information Interchange:



It's interesting that some systems used it for padding; I'd always seen NUL used for this purpose. (Possibly because many of the systems I used were designed to work with certain terminals that took over DEL to indicate a rubout, rather than backspace.)

And interestingly, the ASCII standard itself suggested the Unix idea of using LF as a "newline" character, which I did not know:

 

[Chuck(G)]

Crikey -- 1977 standards? That's kind of late, isn't it?

"Rubout" is not a character name; like "newline" it's a conceptual command that may be implemented in different ways. DEL was not any sort of "alternate" name for the character; it was the one and only official name. From ANSI X3.4-1977, American National Standard Code for Information Interchange:

It's a key on a TTY:

 

[cjs]

Crikey -- 1977 standards? That's kind of late, isn't it?

I am very surprised that you were not aware that that standards regularly get re-issued with clarifications and minor changes, and even more surprised that you think ASCII wasn't standardised until 1977. X3.4 had revisions issued in 1963, 1967, 1968, 1977, 1986, 1992, 1997, 2002, 2007, 2012, and 2017. But that doesn't make it a 2017 standard, of course. DEL has had the same position, name and meaning in ASCII from the very start in 1963 (and no characters in the code set have changed since 1967).

That I happened to quote a 1977 version of the standard was just because that was conveniently available. You should feel free to find the 1963 version and compare, if you think what I quoted will say anything substantially different about DEL, such as calling it "rubout," but it won't.

It's a key on a TTY:

So? Many terminals have used a name other than the standard DEL for a key that sends that character, and many others have had keys labeled DEL or similar that do not send that character. The VT510, for example, has a "Delete" key that in PC mode sends DEL, but in VT mode sends a CSI 3 ~ (^[[3~) sequence.