If you look in the source code you will see the assembly code uses the invoke pseudo op to call the pre-declared routines. This matches the calling convention of the routine so you don't have to manage the parameter passing details by hand.
To figure out why your dll isn't being loaded, a few well placed printfs in the loaddll routine ought to identify what it doesn't like. Hexdump would also be your friend here. It doesn't look to be a particularly complex parsing of the dll to get it to load
Yes, that's what I had been doing: adding printf statements and comparing the outputs with a hex viewer. But it was still unclear why exactly it failed to load.
I recommend keeping the assembly shim to ensure that the driver is structured exactly as required. It's much easier to make an assembler emit exactly what you want, compared to a C compiler.
That's what I ended up doing. And the good news: it worked! In fact, I got it working with both DJGPP and the Open Watcom (v2) compilers.
However, I ran into a limitation: I didn't have access to the standard C library (stdlib) that came with those compilers. To build a properly loadable AIL/32 driver DLL, I need to use either Watcom's wlink or the JWLink fork by Japheth a.ka. Baron-von-Riedesel. The reason being that the object files need to be linked into an OS/2 LX DLL, since that's the binary format that the DLL loader of the AIL/32 driver framework expects. Both those linkers need to be explicitly told where to find the C library file that implements stdio.h for use in DOS executables. Apparently, the standard C libraries that both the DJPP and Watcom compilers provide for building DOS projects seem to require a main() function. In addition to that, the C library of DJGPP also appears to require some symbols that are missing in my project. The C code I'm using in the project is just library code, to be called from the assembly shim, and therefore shouldn't need a main() entrypoint. And even if I provide a dummy main() function so that it at least works with Watcom, it once again results in an DLL that the DLL loader can't load. (That's with Watcom. With DJGPP it still wouldn't build at all, because it was missing other symbols that I somehow couldn't provide in my C code.)
Not being able to use the standard provided C libraries of these compilers will complicate things a bit, since I'll have to provide any libraries and such myself. Perhaps I can use something like Picolibc to offer C library functionality in my project instead?
Also, if anybody knows how I can use DJGPP's linker to link the object files into that LX DLL format, preferably in a way that would still allow me to use the standard C library that comes with DJGPP, please let me know.
I'm making progress step by step.
Again, thank you all for your help so far!
