• Please review our updated Terms and Rules here

Improving the HiTech C compiler

Laci53

Member
Joined
Apr 27, 2022
Messages
12
Hi all,

I published on GitHub ( https://github.com/Laci1953/RC2014-CPM/tree/main/HiTech C compiler optimization ) an enhanced set of tools, replacing the original HiTech C toolchain.

The main goal of this project was to obtain an enhanced HiTech C compiler toolchain, to be used in two environments:

1 : generic, for all Z80 computers running CP/M 2.2
2 : specific, only for RC2014's provided with 128 KB RAM (SC108, SC114, SC118 or Phillip Stevens memory module) running CP/M 2.2

For the second (specific) environment, the basic idea was to use the upper 64KB RAM to extend the RAM space available for the HiTech tools.
This was done by intercepting (part of) the malloc/free calls used by a tool, by using some extra assembler code, to return pointers not from the "usual" lower 64KB RAM, but from the upper 64 KB RAM.
Practically, a new "memory allocator" was written, to handle the upper 64KB RAM space.
Of course, all the references to those pointers, in the C sources, must be handled by special routines, to move bytes from/to the upper 64KB RAM to the lower 64KB RAM.

Obviously, this means some extra code is executed, compared to the "original" HiTech tools, resulting in a longer execution time.

However, this loss of speed ( 10 to 15% ) is largely compensated by the big advantage obtained: to be able to compile larger C source files.

The results:

For the generic environment (all Z80 computers running CP/M 2.2) some limited improvements were obtained, mainly for P1 (4KB more RAM available)

For RC2014's provided with 128 KB RAM (SC108, SC114, SC118 or Phillip Stevens memory module) and 64MB CF,
some significant improvements were obtained, listed below:

- P1 (4KB more RAM available)
- CGEN (using the upper 64KB RAM for some of the malloc's)
- OPTIM (using the upper 64KB RAM for some of the malloc's)

The most significant improvement for this second environment (128KB RAM computers), is that OPTIM can handle now files that cannot be compiled on any available Z80 computer, including here also the emulators/simulators (Z80SIM, ZXCC).

For example, I was able to compile some of the CGEN files ONLY on my SC108 based RC2014.
Z80SIM and ZXCC failed to run OPTIM on those files ("no room"), while the enhanced OPTIM, executed on my SC108 based RC2014, finished the job without any problem.

Also, do not forget my Z80AS, a Z80 assembler compatible with the HiTech's ZAS ( see https://github.com/Laci1953/Z80AS ).

Z80AS must be added to this new toolset, to eliminate all the worries related to the size of files to be assembled.

As a conclusion, I consider this new HiTech tools as an improvement over the "classic" HiTech tools.

Mainly, the RC2014's provided with 128 KB RAM can now take full advantage of having more memory space, while working with the HiTech C tools.

Ladislau
 

whartung

Veteran Member
Joined
Apr 23, 2020
Messages
729
I raise my hand in ignorance, but was the source of HiTech C released to enable you to hack it like this, or were you patching binaries to pull this off?
 

alank2

Veteran Member
Joined
Aug 3, 2016
Messages
2,229
Location
USA
It seems like I recall another project where someone was improving the HiTech compiler.
 

Laci53

Member
Joined
Apr 27, 2022
Messages
12
Hi,

This work was based on the decompiled HiTech tools found at https://github.com/nikitinprior

Thanks to the excellent work of Andrey Nikitin and Mark Ogden, I was able to have a very solid foundation to build on.

Only for P1 (decompilation still missing...) I was constrained to work directly on the assembler sources that I obtained by disassembling the original P1.COM

Ladislau
 

alank2

Veteran Member
Joined
Aug 3, 2016
Messages
2,229
Location
USA
Looks pretty sweet; good work. Now...someday will it be able to compile programs that can access and use that 128K, some sort of memory model!
 

alank2

Veteran Member
Joined
Aug 3, 2016
Messages
2,229
Location
USA
That is why I was wondering if it would be compatible with the 128K improvements you've made. You mention a few SC models, but not 131.
 

Laci53

Member
Joined
Apr 27, 2022
Messages
12
True, the software that enables allocating 128KB RAM works only on RC2014 provided with one of the following boards: SC108, SC114, SC118, SC150, SC152 or the Phillip Stevens memory module.
 

Laci53

Member
Joined
Apr 27, 2022
Messages
12
I just published on GitHub ( https://github.com/Laci1953/HiTech-C-compiler-enhanced ) the enhanced version of HiTech C compiler, including now support not only for 128KB RAM Z80 systems but also for Z80 systems provided with 512KB RAM memory boards.

Hardware requirements
-------------------------------

The improved HiTech C compiler works on any Z80-based system provided with one of the following RAM modules:

- 128 KB RAM (SC108, SC114, SC118, SC150, SC152 or the Phillip Stevens’s memory module)

- 512 KB RAM (512 KB RAM + 512 KB ROM Spencer Owen’s module)

Software requirements
------------------------------

For the 128KB version, a customised CP/M is needed (one with a smaller BIOS).

Why is this a mandatory constraint?
Because it is vital for the C compiler to access the upper 64KB RAM in order to “gain” more available memory where to store its various data structures used by its routines.
But, in order to access the upper 64KB RAM, a small piece of code must be stored in memory as high as possible, in both the lower and the upper 64KB RAM, at exactly the same address.
This “shadow” code will be responsible to move bytes between these two 64KB RAM partitions.
Unfortunately, the “official” BIOS delivered with the RC2014 is too big, “eating” all the available RAM space until the end of the physical 64K memory space.
Thus, no space is left for the “shadow” code.
The only possible solution was to build a custom, smaller BIOS.

You can find the customised CP/M’s here: https://github.com/Laci1953/HiTech-C-compiler-enhanced/tree/main/PutSys - the folder contains all the possible CP/M variants according to the needed serial type (SIO, KIO, ACIA) and CF size (64MB or 128MB).

For the 512KB version, no CP/M customization is needed.

One last remark: the 64MB CF’s CP/M is preferable because it allows a larger TPA ( 2 and half KB larger compared to the 128MB CF version, who “eats” more buffer space needed by the extra disk drives ... ) ;

Design details
-------------------

The customised HiTech C compiler components (P1, CGEN, OPTIM) use now two kinds of memory allocation routines:
- the classical “malloc”
- the new memory allocation routine, allowing access to the extra RAM provided by the hadware

For the 128KB version, a custom memory allocator was built (assembler). This made possible accessing an extra 64KB RAM area, compared to the “classic” TPA area.

For the 512KB version, another memory allocator was built (assembler), allowing access to at least extra 256 KB RAM.
For the 512KB version also a code overlay support module was built (assembler). The code overlays are stored on disk files, being read at start time and stored on specific 16KB RAM partitions; when called, the requested code overlay is loaded from its 16KB RAM partition to a fixed 16KB RAM internal buffer and executed there. This makes the overlay mechanism very efficient.

Thus, the 512KB RAM space 16KB partitions are used to store allocated memory and code overlays.

The 512KB version uses the following memory map model:

0000H – 4000H Base partition (code, data, bss + first memory heap area)
4000H – 8000H Overlay partition (code, data)
8000H – C000H Dynamic memory buffer (loaded with a 16KB RAM memory partition)
C000H – FFFFH Secondary memory heap area + stack + BDOS, BIOS

Also, for the 512KB version, because it was important to gain as much free RAM as possible in TPA, a mechanism for “hiding” and “restoring” the HiTech $EXEC batch processor was implemented, gaining an extra 1 KB of memory.

For both 128KB and 512KB versions, code was kept small also by using a special psect, named “top”, placed after the BSS psect; this “top” psect contains code that is executed only once, at initialization time, and then can be “dropped”, the “top” psect practically is being overwritten by subsequent malloc’s.

Practical results
---------------------

Both the 128KB and 512KB HiTech C compiler custom versions outperform the old 64KB version, allowing to compile quite large C source files.

See the TESTS folder for examples of large C source files that can now be compiled, but will fail with the old HiTech C compiler.

-----------------------------------------------------
regards,
Ladislau
 
Top