Support for terminals with magic cookie glitch

[doshea]

Can anyone suggest software - and maybe particular older versions of software, or just older Linux distributions or even other OSes - that work with terminals that "use magic cookies" or "have the magic cookie glitch" (I'm not sure which is the correct terminology), and perhaps popular terminals that had that glitch I could try emulating?

Details of what I'm trying to do, and a little explanation of what the magic cookie glitch is for anyone who doesn't know:

I've got an IBM 3476 green screen twinax terminal hooked up to a Linux machine using one of inmbolmie's converters. Currently the software running on the Linux side presents the terminal to Linux applications as a VT52 with no support for attributes. The terminal actually supports some attributes: reverse video, intensity, underline, blink, column separators and no display, although not all combinations of those. I'd like to update the software so that I can get some highlighting on the screen, but the problem is that the way you change attributes with this terminal is by storing a byte in the terminal's memory buffer which actually takes up a character position on screen, which in the *nix world seems to generally be referred to as a "magic cookie".

From some research, I've learned that some versions of termcap, terminfo, ncurses, etc. support indicating that a terminal leaves behind these blank spaces, and that software using them needs to have special handling for that, or that perhaps some older ncurses versions could in some cases do some things automatically for you. It looks like the magic cookie support has been broken since ncurses 5.6, which added this to its TO-DO file:

+ Magic cookie support (for nonzero xmc values) does not work, since the logic
  does not take into account refresh.  Also, the initial optimize does not
  adjust the current location when a cookie is emitted.

GNU Emacs version 22 added this to its NEWS.22 file:

*** Support for `magic cookie' standout modes has been removed.

Emacs still works on terminals that require magic cookies in order to
use standout mode, but they can no longer display mode-lines in
inverse-video.

Given the hint as to where they removed it, I tried with version 21.4a with ncurses support disabled at build time; I couldn't get it to actually turn on any attributes but it did seem to turn off attributes at the start of every line, which is a start! I suppose they probably removed it because it was already broken, and I should have tried a much older version, but 21.4a is almost 19 years old so what I probably really need to do is just try some Linux distributions from the '90s or something.

 

[Svenska]

Modern software is often hard-coded to VT100 / ANSI-style terminals and never tested on anything else. My VT52 emulation connected to a modern Linux spits out random ANSI sequences every once in a while, and some tools are simply unusable.

The best way forward is to use something like screen or tmux, which presents a standard ANSI terminal to the application and deals with your specific terminal weirdness behind the scenes. ANSI has become the one standard today, so just bite the bullet and convert in- and outbound traffic to match.

 

[stefan-lokeldarn]

part of reason is probably that it isn't easy describing a terminal like a 3476 in terminfo, and so ncurses is gonna have a bit problem, and programmers gets trouble differing between broken terminfo and broken code in edge cases like a block-mode terminal.

 

[Chuck(G)]

The "attribute takes up a space" was very common on inexpensive terminals of the 80s. See, for example, those using the Intel 8275 CRTC. While the scheme allows one to cut the memory requirements (i.e. you don't need a separate attribute for each character position), making the attribute invisible, while possible creates bookkeeping nightmares in the firmware.

Been there, done that.

 

[doshea]

Modern software is often hard-coded to VT100 / ANSI-style terminals and never tested on anything else. My VT52 emulation connected to a modern Linux spits out random ANSI sequences every once in a while, and some tools are simply unusable.

Yeah, inmbolmie said the same thing to me

The best way forward is to use something like screen or tmux, which presents a standard ANSI terminal to the application and deals with your specific terminal weirdness behind the scenes. ANSI has become the one standard today, so just bite the bullet and convert in- and outbound traffic to match.

Thanks, that's an interesting suggestion.

inmbolmie suggested something similar - the code that presents a PTY on the Linux side should just implement a subset of VT100/ANSI, since that's what so much code assumes.

The problem is that I think there is very little you can do to make the 3476 behave like VT100/ANSI in terms of attributes. If the application thinks it can apply an attribute to some text without the location of the text and the length of the line changing, but they do change, it's going to make a real mess. Say there are 79 characters with no attributes applied on a line and the application decides to apply some attribute to a word, now there are 2 extra "characters" (or at least bytes) on the line, so it's wrapped to the next line, but the application doesn't know that, so it may redraw the next line over the top, etc.

I suspect then that screen and tmux probably just don't use attributes for terminals that have the magic cookie glitch, although I'll try to check sometime.

One thing I could possibly do - which I think may be a bit like Emacs did - is expose a terminal that is only 79 characters wide, reserving the first position on every line on the real terminal for an attribute byte. Then whenever an attribute is applied to the first character on a line, it could be applied to the entire screen row. That would mean Emacs's mode line should be highlighted properly, and also lines in diffs/patches. I might want to tell Emacs to not highlight the region (selection) so that I don't get confused when what is highlighted ends up being every line which has at least one character as part of the selection.

Perhaps a nicer heuristic would be to check what attribute combination is applied to the majority of the characters on the row and apply that to the whole row, but it would be a lot more work to actually do those checks every time there is a screen update, and move the cursor back to where the attribute byte is to update it.

I imagine though that I could get nice results if I modified Lynx for example. Currently I have no idea where the links even are on a page until I cycle through them, so at a minimum it would be nice if Lynx could be configured to surround them with some configurable characters to highlight them for users whose terminals have no attributes. It ought to be able to handle this reasonably well by just flowing the paragraph as if the link text was 2 characters longer. Then if those two characters actually changed attributes, I could get links underlined, and highlight the active link.

 

[doshea]

The "attribute takes up a space" was very common on inexpensive terminals of the 80s. See, for example, those using the Intel 8275 CRTC.

Interesting, thanks!

While the scheme allows one to cut the memory requirements (i.e. you don't need a separate attribute for each character position), making the attribute invisible, while possible creates bookkeeping nightmares in the firmware.

Yeah, this terminal only has 2000 bytes of user-accessible memory, and a separate memory buffer for attributes was a feature of the next model up Yes, that sounds like it would be a nightmare!

 

[Svenska]

If the application thinks it can apply an attribute to some text without the location of the text and the length of the line changing, but they do change, it's going to make a real mess.

This is a limitation in the application logic and applies to any such terminal. The easiest fix is to not expose attributes to curses-based applications so that they will at least not mess up the screen. Not much else you can do.

I suspect then that screen and tmux probably just don't use attributes for terminals that have the magic cookie glitch, although I'll try to check sometime.

They probably don't support such terminals at all, but I was more thinking of the general idea than these two in particular. Changing applications is not sustainable in my opinion, a translating program is probably the best way forward: Take the incoming ANSI stream, parse and render it into a memory buffer on the Linux system, then try to replicate it on your terminal as best as possible.

One idea is to expose a 78-character wide terminal and shift the whole screen one character to the right, allowing a single full-line attribute. Another idea is to check whether the character before an attribute change is a blank and move the attribute change there (this would work for Lynx, because most links have a space before them). You could also check if there are enough blanks at the end of a line to render all attributes, and selectively drop them if lines get full. This could be made optional with a key combination as well (some things would work well for Lynx, but not for Midnight Commander - and vice versa).

 

[Chuck(G)]

From the scum ring etched into my memory, I recall that the 8275 permitted a maximum of 16 invisible attribute bytes per line. Apparently, the chip has a fifo that can absorb that much without screwing up the video timing.