Writing Assemblers... What should a good assembler do?

[cjs]

NBN is only an internet Data service. And currently the only way for millions of Australians to have a home non-cellular telephone service with a handset in Australia too. Unless you steal a payphone ( which, interestingly, still exist ) and drag the wires into your home. I still have a normal handset connected to mine. Mainly I've had one for years simply so my MIL could call my wife at local call rates.

Unless Wikipedia is massively incorrect, you have it backwards. It says that NBN provides no Internet connectivity at all; you get that from "Retail service providers," who are also customers of NBN: "NBN Co wholesales Layer 2 network access to retail service providers (RSPs), who retail Internet access and other services."

This is how it works in Japan, too, and I think you may have no idea how lucky you are to have this. Internet access in the U.S. has been severely crippled because some facilities owners have an effective monopoly on "last mile" service, and force you to buy Internet access from them as well.f

 

[cj7hawk]

Unless Wikipedia is massively incorrect, you have it backwards. It says that NBN provides no Internet connectivity at all; you get that from "Retail service providers," who are also customers of NBN: "NBN Co wholesales Layer 2 network access to retail service providers (RSPs), who retail Internet access and other services."

This is how it works in Japan, too, and I think you may have no idea how lucky you are to have this. Internet access in the U.S. has been severely crippled because some facilities owners have an effective monopoly on "last mile" service, and force you to buy Internet access from them as well.f

Well, a technicality, yes, NBN is a last mile carrier and not an ISP and the don't provide Internet connectivity directly. They have 120 or so points of presence where they aggregate all of the links to houses and they sell this back to ISPs. ISPs can run pseudowire and similar services, but at the same time, they don't provide point to point services like a normal last mile provider, so you connect into the routers of an ISP... And typically from there, you connect to the Internet. So for all intents and purposes, if you connect to the NBN, you're connected to the Internet. And you get a modern VDSL router that has some POTS connectors. It gets configured to connect a POTS telephone to an internet based SIP server.

Australia was a little different. We had a higher number of competing ISPs than anywhere else in the world. And they drove newer and faster services that continually pushed and drove technology to be faster and faster. I imagine that Telstra ( our incumbent ) didn't like that and there was some severe criminality going on there.... And Telstra was being difficult about letting ISPs install new stuff.

So they just created a new monster to replace the old one, and that's the NBN. It's OK at best. At worst it actually inhibits a whole lot of technology and they are just like a company - if it's not profitable ( and let's face it, they aren't very good at that... Just ex Telstra people etc ) then they don't do it, and so market failure still hurts people, especially disproportionally Australian Aboriginal people and people in rural and remote locations, which is most of Australia.

I have mixed feelings about them, but generally I feel their existence is a bad thing. And pretty much everything in Australia is IP, and more so, it's *internet* IP. All Internet monopolies are bad. NBN is a monopoly and even if it doesn't supply Internet, it's an Internet monopoly. If you go to their website, they let you order internet only, from one of their partners. You can check out their website for ordering services.

I've had mediocre services for years, and typical speeds here are around 30 kbps at best. I get slightly more than that out of luck. I used to run an ISP out of my house many years ago, but that's a while different story. I just put my order in today for fiber. The NBN has been dragging it's heels to avoid connecting people.

Also they get criticized a lot. I have a 100x40 Mbps service. I get less than half that on a good day. Maybe 2/3s and have a CVC (shared maximum speed ) of around 1.5 Mbps. I've only just ordered their fiber service today, because they shafted people like us during the rollout, while only providing high quality high speed internet to wealthy suburbs.

Elon Musk is quite the Internet Hero in Australia with Starlink...

I really hope Japan has better internet than that. Heck, even if the Yakuza ran it, it would still probably be better than the NBN and probably more honest and customer focused. I have lots of memories of just how bad Australian telco's were. People died and had their lives destroyed too, and much of that was deliberate intentional and criminally willful in nature. It's something I've never forgotten even if I was barely affected personally.

 

[cjs]

Well, a technicality, yes, NBN is a last mile carrier and not an ISP and the don't provide Internet connectivity directly.

That's not a technicality, that's the entire point. Communications networks have been moving first to digital TDM and then to packet switching for decades. That you happen to have a device that speaks the same layer 2 protocol as your telco is irrelevant to Internet access as that's a very different thing. (And consumers having devices that do this started with ISDN in the'90s, before the switch to IP for telephony. That never took off in most consumer applications because the telcos were rather slow to get it out, though it was widely used in business.)

ISPs can run pseudowire and similar services, but at the same time, they don't provide point to point services like a normal last mile provider....

"Normal" last mile providers haven't provided point-to-point services for years, and the whole point, again, is that you don't want them providing anything but connectivity to other providers (Internet or otherwise) at POPs, because that's what introduces a monopoly on services, and leaves you in the same place the US is now: poor and expensive Internet connectivity.

So for all intents and purposes, if you connect to the NBN, you're connected to the Internet. And you get a modern VDSL router that has some POTS connectors. It gets configured to connect a POTS telephone to an internet based SIP server.

Given your previous confusion, I'm dubious about whether or not this is true. (And I'm assuming that you did not deliberately make the 'i' in your second use of 'internet' above lower case; if you're distinguishing between the Internet and internetworks, best to make that clear since in common use 'internet' has come to mean 'the Internet.')

I have mixed feelings about them, but generally I feel their existence is a bad thing. And pretty much everything in Australia is IP, and more so, it's *internet* IP. All Internet monopolies are bad. NBN is a monopoly and even if it doesn't supply Internet, it's an Internet monopoly. If you go to their website, they let you order internet only, from one of their partners. You can check out their website for ordering services.

Well, of course. Those "partners" are actually their customers, and they can let connect only to someone else who connects to them. But this is a vastly different situation from the U.S. where, for example, in many areas you can get local fibre only from Comcast, and you can get Internet only from Comcast.

I've had mediocre services for years, and typical speeds here are around 30 kbps at best.

This, again, I have difficulty believing. You're saying that your service is slower than a 1990s dial-up modem? So slow that you simply can't do any video (YouTube, Netflix, etc.) at all?

Also they get criticized a lot. I have a 100x40 Mbps service. I get less than half that on a good day. Maybe 2/3s and have a CVC (shared maximum speed ) of around 1.5 Mbps. I've only just ordered their fiber service today...

Ah, it seems you do get considerably more bandwidth. I can't tell from all these numbers you're throwing around what it actually is; you've now listed three rates that are three different orders of magnitude....

 

[cj7hawk]

That's not a technicality, that's the entire point. Communications networks have been moving first to digital TDM and then to packet switching for decades. That you happen to have a device that speaks the same layer 2 protocol as your telco is irrelevant to Internet access as that's a very different thing. (And consumers having devices that do this started with ISDN in the'90s, before the switch to IP for telephony. That never took off in most consumer applications because the telcos were rather slow to get it out, though it was widely used in business.)

Yes, but the fact it's all "Internet" connected in the end was the point - You can't have a phone service anymore that isn't cellular or Internet connected. They don't exist outside of public phone booths and even those might be IP now. Well, they might exist as a possibility for large corporatations but not for most retail customers.

"Normal" last mile providers haven't provided point-to-point services for years, and the whole point, again, is that you don't want them providing anything but connectivity to other providers (Internet or otherwise) at POPs, because that's what introduces a monopoly on services, and leaves you in the same place the US is now: poor and expensive Internet connectivity.

It's a nice idea in theory. A bit like communism. In practice it doesn't work as well as it should, and I have acceptable speeds, compared to some places. Though I also have a better idea of what we could have had instead than most here.

Given your previous confusion, I'm dubious about whether or not this is true. (And I'm assuming that you did not deliberately make the 'i' in your second use of 'internet' above lower case; if you're distinguishing between the Internet and internetworks, best to make that clear since in common use 'internet' has come to mean 'the Internet.')

Well, I am a master of "weasel words" as you put it, and usually capitalise Internet, but I use Internet and internet interchangeable. Both have exactly the same meaning. When I'm talking private internet, I'll usually refer to them as RFC1918. If I'm not talking IP, I use terms like PLLs. I'm ex-telco. Anyway, it's an observation and it's only as reliable as what I've been exposed to. But you wanted to know where the Internet was mandatory, and I'd say Australia, as a country, meets that standard.

Interestingly, this also means we have an effective underclass of people who can't afford Internet, who are very digital-literate, living off of free wifi hotspots. That's a very real problem the country is working to address. Fortunately they have little trouble getting hold of old phones and the like to access those services.

This, again, I have difficulty believing. You're saying that your service is slower than a 1990s dial-up modem? So slow that you simply can't do any video (YouTube, Netflix, etc.) at all?

Typo.... Meant Mbps as you noted later... Though 1.5 Mbps is like a T1 ( Australia had E1s which were 2 Mbps )

Yes, it's way better than when I first connected via a modem... And better than most of the services I had personally.

 

[cjs]

Yes, but the fact it's all "Internet" connected in the end was the point....

Well, then your "point" is dead wrong. IP is not "the Internet," and there are vast systems across the world that use IP that do not have Internet connectivity. (And for very good reasons.)

It's a nice idea in theory. A bit like communism. In practice it doesn't work as well as it should, and I have acceptable speeds, compared to some places. Though I also have a better idea of what we could have had instead than most here.

It's not at all like communism in that the idea has and does work very well in practice. Your particular service may not, but if you're suggesting you switch to the U.S. system, where in most areas you get one broadband ISP, take it or leave it, I guarantee you that things will be much worse. If you're proposing something else, well, make it known. But I can tell you right now that if you think that separate last-mile providers is a bad thing, you are utterly and completely wrong. Your problems lie elsewhere.

Well, I am a master of "weasel words" as you put it, and usually capitalise Internet, but I use Internet and internet interchangeable. Both have exactly the same meaning.

Technically, no. As terms of art in IP networking, the "Internet" is one particular "internet." (Everybody who's read Stevens understands this. If you don't know who Stevens is, well, you're not particularly versed in the technicalities of IP networking, then.)

That said, I recognise that most people who are not network engineers use "Internet" and "internet" interchangeably to mean the former, so no problem, go ahead and keep using it that way. I was merely making sure you were doing what most other people do, rather than deliberately using "internet" (with a lower-case "i") as a term of art. (Which term of art I suggest people no longer use, BTW.)

When I'm talking private internet, I'll usually refer to them as RFC1918.

Well, that's just confusing since many (probably most) RFC 1918 networks are connected to the Internet (through the magic of NAT), including the very system I'm using right now (and probably yours, too). Not to mention that there are also plenty of non-RFC 1918 systems that are not Internet-connected. Just say "private network."

As a somewhat simplified test, if you can make an SSH connection to github.com, you're connected to the Internet.

If I'm not talking IP, I use terms like PLLs. I'm ex-telco.

Well, that's a) beside the point, since plenty of telephony has been running over IP for ages, and b) there are other protocols besides that. Just say "layer 2" if that's what you mean. Or at least use a term that's listed on Wikipedia.

 

[cj7hawk]

Well, then your "point" is dead wrong. IP is not "the Internet," and there are vast systems across the world that use IP that do not have Internet connectivity. (And for very good reasons.)

Public IP. Just to be really clear, I'm suggesting public internet-connected real-world reachable (sans firewall considerations) IP... Not just a private IP network.

There is no private IP network that most people in Australia can connect to just for phone calls, nor am I aware of anyone supplying such a service without requiring an actual internet service.

It's not at all like communism in that the idea has and does work very well in practice. Your particular service may not, but if you're suggesting you switch to the U.S. system, where in most areas you get one broadband ISP, take it or leave it, I guarantee you that things will be much worse. If you're proposing something else, well, make it known. But I can tell you right now that if you think that separate last-mile providers is a bad thing, you are utterly and completely wrong. Your problems lie elsewhere.

I'm not comparing this to the US. I'm comparing it to how things used to be in Australia, which was arguably better, and very few people were locked into a single provider. Services were declared so that Telstra had to provide access to competitors. Of course, dirty play ensued such that Telstra often found ways to lock them out, but the wholesale cost was still required, so in the end, you got the ISP of your choice, and Telstra carried the signal. And the ISPs constantly upgraded their services where they could, which put Telstra under pressure to also compete. It was a good example of how the free market is supposed to work.

I am arguing how it was working *before* the NBN was reasonable. IMHO, the new system introduced enforced market failure. The old system still let people compete.

So I'm comparing Australia before and after NBN, not comparing Australia to the US model.

Well, that's just confusing since many (probably most) RFC 1918 networks are connected to the Internet (through the magic of NAT), including the very system I'm using right now (and probably yours, too). Not to mention that there are also plenty of non-RFC 1918 systems that are not Internet-connected. Just say "private network."

I just use it for private internets, and to be fair, where it is implemented like that, most people tend to follow RFC1918 in practice in Australia. Sometimes they make mistakes, but generally it was an error and not intentional.

And while most phone systems use RFC 1918, they still require an Internet to run over. You can't use private internets end-to-end.

As a somewhat simplified test, if you can make an SSH connection to github.com, you're connected to the Internet.

That's not a great test. It would infer that if you had a firewall rule to prevent that, you're not connected to the Internet.

I am inferring that, were there not firewall rules to prevent it, most of these phones could talk to the Internet.

In practice also, it's amazing how many of these IP Phones connecting to SIP servers use 8.8.8.8 for the DNS server, so I would think that also counts -

Likewise, even if Github were contactable by the phone, that wouldn't necessarily infer that the phone requires an internet connection.

I think its reasonable in this case to state that if connection to the Internet is a requirement ( ie, the service will not work without this ) then it's a requirement, whether private internets are involved or not.

Well, that's a) beside the point, since plenty of telephony has been running over IP for ages, and b) there are other protocols besides that. Just say "layer 2" if that's what you mean. Or at least use a term that's listed on Wikipedia.

No, as PLLs are often Layer 3 also. Especially in Australia. So are Layer 1 PLLs, especially where Government was concerned. We're a bit different like that. Also, it's really common to use SIP to TRUNK converters here to maintain old services, as well as E1 over IP. Less common now, but was a common transition technology.

 

[cjs]

There is no private IP network that most people in Australia can connect to just for phone calls....

I should hope not! The whole point of a private network is that the public can't connect to it! IP telephony is used extensively here in Japan, and it's all on private networks that customers do not connect to directly. There are twenty or thirty "Level I" telephony providers that talk to each other mostly via private IP, and anybody who wants to make a phone call goes through one of them.

That's not a great test. It would infer that if you had a firewall rule to prevent that, you're not connected to the Internet.

Yes, that's exactly why I said, "somewhat simplified." I'm still unclear as to whether you understand the difference between the Internet and private IP networks.

I am inferring that, were there not firewall rules to prevent it, most of these phones could talk to the Internet.

Is that how you design your telephony systems in Australia? Interesting. A pretty dumb way to do it, but interesting.

I think its reasonable in this case to state that if connection to the Internet is a requirement ( ie, the service will not work without this ) then it's a requirement, whether private internets are involved or not.

Sure. But I've yet to see a PSTN that just allows any random phone to connect via SIP over the Internet, much less requires it.

 

[cj7hawk]

Is that how you design your telephony systems in Australia? Interesting. A pretty dumb way to do it, but interesting.

I wouldn't say it was "designed" that way - it just happened organically after they got rid of home phone lines without anything planned to replace them outside of use of mobile services.

And routers with POTS ports and SIP capabilities were available.

Sure. But I've yet to see a PSTN that just allows any random phone to connect via SIP over the Internet, much less requires it.

Not sure what you mean by random phone. That's how most Internet SIP systems work - and often there is no firewalling to present the phones from seeing the Internet. SIP supports name/password combinations for access so maybe that's what you mean. SIP providers then connect by trunk (SIP or otherwise) into larger telcos who sell them the line.

You could connect into the Australian network and have an Australian phone number. All you would need to do is pay money and they will let you in. Telemarketters and Scammers do this all the time.

 

[cj7hawk]

Version 1.0 of the assembler was completed today, with all of the expected functions. Thanks to everyone who provided suggestions in this thread. I didn't include all fo them, but I did learn a lot from it.

I'm still writing the instruction manual, the OS macro file is yet to be started and some other elements still require testing, and I haven't written the post-failure toolkit for addressing assembler failure analysis, but generally it looks like it's all working as hoped, if a little slow.

 

[whartung]

Great news @cj7hawk ! That's quite an accomplishment. That's just a lot of assembly to write.

Can you talk to what kind of rhythms you fell into?

By that I mean, I'm sure that as you started getting into it, you found lots of patterns of code, constantly repeated.

How you did loops, how you did math, how you passed parameters, etc.

Folks think that assembly programming is minutiae on top of minutiae, but, save for perhaps some carefully coded routines, I don't think that's so.

That you can think at a "C" level, and just "hand compile" on the fly, not sweating every cycle or carry flag.

So, just curious if you have any insight to what you have discovered during your writing of the assembler.

 

[cj7hawk]

Rythms... Well, one was the main routine just calls a series of tests and each test works out if it's relevant and exits otherwise.

If I get to the bottom of the routine, it's an error, so I crash out and display an unknown error (eg, not an opcode, macro or anything else that was expected ).

Like this;
CALL MAIN:

MAIN:
CALL TEST_JP
CALL TEST_INC
CALL TEST_SOMETHINGELSE
CALL TEST_ORG
CALL TEST_MACRO
JP ERROR_OUT.


Which might seem strange, but it means I can just exit the test routine if it wasn't correct, and let something else try. I could have used a table for this, and it probably would have been better in some ways, but I'll save that for another time.

Each routine did it's check and acted on it. If it acted, then it exited with

POP HL
RET

So that it would go back to the original call rather than the MAIN routine.

Also, the maths was simple. Once something has been read into the buffer and it's not understood, the final possibility is that it's a formula or label. ( Everything like Macro's are just complex labels ). - So I just keep calling EVAL to allow the value to be determined. If it finds an operator, it repeats and calculates in INFIX IMMEDIATE format, so the maths is really simple and limited to 16 bit integer arithmetic. Some suggested that accepting decimal or FP numbers is useful, but unless you have a standard format for those functions, they aren't that useful to a programmer working in assembler, so I left them out.

The label table is a linked list, so it's very simple. In the long run that and some other decisions prevented me from using things like hash tables, but it also allowed me to switch on and off groups of labels at a time which creates a very simple way to support local labels.

I push the file sytem onto the system stack when an include in encountered, and some suggestions like Macros and Includes I never thought of at first... Then other suggestions such as making it possible to add arguments complicated things a little, but generally a page or less of assemble was required for most functions. I think includes were the most complex.

Also, adding switches to the command line and HEX output was nice, but it was always intended to write COM binaries in a single pass, while handling anything up to an operating system in complexity.

Mostly it was about writing small routines that I could call. The parser, the matching code, the function evaluator etc. Once each piece was made, I could concentrate on dealing with the instructions.

What else did I learn?

That there's a fine line between an OS assembler and a Cross Assembler. Cross Assemblers are way more powerful and some features don't have a place in a small OS based assembler that is native to it.

Labels need to be LONG and you don't need 8 and 16 bit labels. It's really easy to work with numbers and just use what you need. I got rid of all of the "DEFINE" type stuff that clutters assemblers also - mostly it's just DB - because in 8 bit, everything is a byte. Text, Numbers, etc. Strings are a series of 8 bit numbers separated by commas, well, they can be. The only other ones I used were DW for a 16 bit number, which is for 2 bytes, and BLOCK for a large block of the same 8 bit number. But mostly, DB is enough for anything I needed.

I supported Binary and Hex, and some people asked about octal, but Binary is there, so most good assembler writers should be able to read and write octal in binary.

Also, you can put spaces between numbers - eg, %0101010101010101 is the same as %0 101 010 101 010 101 so spaces make hex and binary easier to read.

I made dots invisible so .ORG an ORG are the same. That's just because I do .ORG out of habit, but it's not necessary. So dots can be used by the programmer for whatever they want.

What did I learn aside? Well, it's a lot easier to write an assembler on a cross assembler, and having a cross assemble and a native OS assembler that take the same source files is quite useful.

Though the z80 assembler ended up much more powerful than the cross assembler that I used to write it.

But mostly, I learned there are no guides to writing something like an assembler. They just don't exist. You can read other people's works and learn from them, but no one writes a "How to write a good assembler" guide, so I stumbled across everything I learnt from the experience.

A bit like finding furniture in the dark. You do it with your shins.