Strange disk problem Kaypro selfbuilt bios for harddisk

[gertk]

Hello,

Having added an ISA MFM controller to my Kaypro 4-83 and trying to create a bios for it, I am running into the following problem:
Note: Harddrive is C, floppydrives are still A and B (maybe a bit unconventional for CP/M)

When the current drive is A: I can succesfully do a dir C: but when the current drive is C: I can not get the directory of A with dir A:

The result is this:


I lowered the CCP, BDOS and Bios to D400/EA00 to get space for my own routines for testing, later on I will add them to the rom.
For now I divert the diverse bios disk functions for A or B to the original rom routines and for the harddisk to my own code.


Copying with PIP from floppy to harddisk is also not working, neither A to C nor C to A, it does create a xxxx.$$$ file but it is empty

When PIP is executed on the C drive itself (copying to C) it is working OK, same for using PIP on A to A

I managed to install some files on the C drive with the terminal program ST which seems to have no problems writing to C (even when started from A)

Assembling and such works fine on the C drive alone.

Any ideas where to look ?

 

[durgadas311]

Can't tell for sure how your ROM differs from the original Kaypro ROM BIOS, but Kaypro used a chunk of high memory for buffers and variables for the ROM. If you are using any of the original Kaypro ROM code, you'll need to avoid touching any of that RAM in your routines. Or at the very least you must use it in a compatible way.

 

[gertk]

Can't tell for sure how your ROM differs from the original Kaypro ROM BIOS, but Kaypro used a chunk of high memory for buffers and variables for the ROM. If you are using any of the original Kaypro ROM code, you'll need to avoid touching any of that RAM in your routines. Or at the very least you must use it in a compatible way.

As far as I can tell from the 81-149C rom source (which seems mostly the same as my 81-232 rom) and the KBIOS.MAC source I used as reference, the range from $FC00-$FFFF is used by the rom/bdos/original bios.
My code sits at $EA00 (with the CCP at $D400) and goes up to $F0A9 so it should not conflict but never say never..

Strange thing is that if I try to do a DIR A: from C: it never touches the floppy, I can see the harddisk light flash a few times and then it prints the garbage directory so it is not reading the correct drive.
I do not know how CP/M handles the indirection given by prepending the path with A:, does it do a seldsk or does it use some buffer for that and if so can I force a reload if necessary?

If typing A: (return)
and then doing a DIR it works fine
same for C: (return)
DIR

This is my seldsk function at the moment:

; select drive, number in C
seldsk:
	mov	a,c
	cpi	2		; drive C ?
	jz	hd0seldsk
	cpi	3		; drive D ?
	jz	hd1seldsk

	mvi	l,rom+0FH	; select disk drive
	jmp	callrom   ; dispatch to oroginal rom code

hd0seldsk:
   	lxi 	h,hd0dpbase ; set to my harddisk disk parameter header
	 sta	lastdrive  ; set current drive number (at $0004)
	ret

hd1seldsk;
    lxi h,0    ; not installed
	ret

; disk Parameter header for disk C
hd0dpbase:
    	dw  	0000h, 0000h
	dw 	0000h, 0000h
	dw 	dirbf, dpblk
	dw 	chk00, all00
	db	00h
; disk parameter header for disk D
hd1dpbase:
    	dw 	0000h, 0000h
	dw  	0000h, 0000h
	dw 	dirbf, dpblk
	dw 	chk01, all01
	db	00h
;
dpblk: ;disk parameter block for hard disk
  	dw  68 	;sectors per track
	db 	7 	;block shift factor
	db 	127	;block mask
	db 	7 	;null mask
	dw 	663	;disk size-1
	dw 	63 	;directory max
	db 	240	;alloc 0
	db 	0 	;alloc 1
	dw 	0 	;check size
	dw 	8 	;track offset

 

[durgadas311]

I'll spend some more thought on it, but first idea I had was that maybe the Kaypro ROM (since you are using it directly) does not realize that you had selected a different disk and so it was taking an optimization that it wasn't entitled to. I'm not sure what that might be, though. Does your code use the same 'dirbf' as the ROM BIOS (floppy)?

Another possibility is that 'all00' is not the right size (too small), and some memory corruption results.

While this is not going to cause any fatal problems, I do notice that your "alloc 0/1" bytes (DPB.ALV0) are reserving 4 blocks for the directory, but you are using (DPB.DRM) only 64 directory entries (which is very small for a harddisk). You reserved 4 blocks, and have specified 16K block size, so that is a total of 2048 directory entries reserved (possible, in the space reserved). The number in the DPB.DRM word defines how much work the BDOS does when scanning the directory, and the DPB.ALV0 bytes just reserve the space. Kaypro floppies reserve extra space for the system (CCP or BIOS, I forget which) and so have a similar anomaly - just much smaller discrepancy. CP/M 2.2 generally does not perform well with huge directories, so you need to find a balance between the size of the drive (DPB.DSM) and what is practical for CP/M 2.2. Your drive size is 664 blocks (less 4 reserved), and you have a very large block size, so most files are going to fit in a single block. In order to be able to use all the drive space, you would need 660 directory entries. Of course, if a number of files will be larger than 16K then you could reduce that number.

 

[gertk]

I'll spend some more thought on it, but first idea I had was that maybe the Kaypro ROM (since you are using it directly) does not realize that you had selected a different disk and so it was taking an optimization that it wasn't entitled to. I'm not sure what that might be, though. Does your code use the same 'dirbf' as the ROM BIOS (floppy)?

Was thinking along that line too, maybe the rom code takes a shortcut in calculating the DPH and since it normally only does only 2 drives and now finds a drive number of 3 in lastdrive, it gets the wrong disk parameters.
My (seperate) 'dirbuf' points to the end of my code, just beyond the sector buffer I have.

Another possibility is that 'all00' is not the right size (too small), and some memory corruption results.
While this is not going to cause any fatal problems, I do notice that your "alloc 0/1" bytes (DPB.ALV0) are reserving 4 blocks for the directory, but you are using (DPB.DRM) only 64 directory entries (which is very small for a harddisk). You reserved 4 blocks, and have specified 16K block size, so that is a total of 2048 directory entries reserved (possible, in the space reserved). The number in the DPB.DRM word defines how much work the BDOS does when scanning the directory, and the DPB.ALV0 bytes just reserve the space.

Have been fiddling with those numbers and since it is not a really big harddrive (only 10 MByte) I came up with these. Since CP/M seems to have no notion of 'heads' and to keep numbers in manageable quantities I made the following scheme:

316 cylinders
17 sectors (of 512 bytes) per track
4 heads

By incorporating the head number into the track number as bottom two bits (and since that was already a 16 bit number), head movement would be optimized for every 4 cpm tracks.
Same thing for the sector number, by shifting the sector number two bits up, I can use the bottom two bits as index for which of the 128 bytes cpm wants to read/write out of the 512 byte physical sectors.
That way the sector number is still within 8 bits and easier to code with. My bios read/write sector routines simply receive a track and a sector number. My blocking/deblocking routine is always reading/writing a whole 512 byte sector for every 128 byte cpm sector aka 'not cached' and so not optimal but since the speed is more than adequate I do not mind (yet).

Kaypro floppies reserve extra space for the system (CCP or BIOS, I forget which) and so have a similar anomaly - just much smaller discrepancy.

Yes I did find out that Kaypro floppies have the CCP/BDOS/BIOS sectors on track 0 and 1, and in a strange layout: on track 0 it uses cpm sectors 1 to 39 but then it advances to sector 16 on track 1 for the rest.
Sectors 0-15 on track 1 are for the directory. Oh, and sector 0 on track 0 is used to store some info on where to put the CCP/BDOS/BIOS in memory (so it is not a boot sector as such).
The strange layout got me puzzled as all the CP/M books just say: load in the first two tracks and you're done. So I wrote my 'getsys' program to copy the system to the first track(s) of the harddrive and just got rubbish (since the directory data got in the middle of the code..)

CP/M 2.2 generally does not perform well with huge directories, so you need to find a balance between the size of the drive (DPB.DSM) and what is practical for CP/M 2.2. Your drive size is 664 blocks (less 4 reserved), and you have a very large block size, so most files are going to fit in a single block. In order to be able to use all the drive space, you would need 660 directory entries. Of course, if a number of files will be larger than 16K then you could reduce that number.

Yes maybe I need to recalculate the DPB for the harddrive, for now it seems to work. Also CP/M does not really know how to 'format' the drive. I ended up just writing a bunch of $E5 on the directory sectors (in fact: on a large part of the disk) and started writing some files downloaded with the communication program (ST.COM) and so far it looks ok. STAT gave me about 9000k on the empty drive and now with some 35 files on it, it gives 8576k free space. Considering I wrote the blocking/deblocking code in one stretch and so far the CP/M programs I stored onto the drive seem to behave I am quite happy with that..
On the original Kaypro disks I have there is a beautiful basic (compiler) called S-Basic and it came with a pretty large demo program called XAMN.BAS, it displays the disk information it finds from the DPB(s). It compiled without a hitch on the harddisk btw.

For disk 2 (my harddisk) it comes up with this):

When selecting disk 0 or 1 it also comes up with sensible data:

 

[durgadas311]

Just realized that you have a CP/M 2.2 drive with a capacity of 10624K. That actually won't work, CP/M 2.2 can only address a drive of 8192K maximum. I *think* that only becomes a problem when you start allocating blocks past 8192K, but you should not be running that large of a drive. The BDOS is not able to do the math, and it will likely start overwriting the beginning of the drive. The maximum number of (16K) blocks should be 512, or DPB.DSM = 511.

 

[gertk]

Just realized that you have a CP/M 2.2 drive with a capacity of 10624K. That actually won't work, CP/M 2.2 can only address a drive of 8192K maximum. I *think* that only becomes a problem when you start allocating blocks past 8192K, but you should not be running that large of a drive. The BDOS is not able to do the math, and it will likely start overwriting the beginning of the drive. The maximum number of (16K) blocks should be 512, or DPB.DSM = 511.

Ah, a good point, although BDOS seems to get the math right with the free space .. I read that the file size limit is 8192k
But it is easily fixed by adjusting the DPB. The rest of the space I can create a seperate DPB for and make it drive D:

 

[durgadas311]

...

Have been fiddling with those numbers and since it is not a really big harddrive (only 10 MByte) I came up with these. Since CP/M seems to have no notion of 'heads' and to keep numbers in manageable quantities I made the following scheme:

316 cylinders
17 sectors (of 512 bytes) per track
4 heads

By incorporating the head number into the track number as bottom two bits (and since that was already a 16 bit number), head movement would be optimized for every 4 cpm tracks.
Same thing for the sector number, by shifting the sector number two bits up, I can use the bottom two bits as index for which of the 128 bytes cpm wants to read/write out of the 512 byte physical sectors.
That way the sector number is still within 8 bits and easier to code with. My bios read/write sector routines simply receive a track and a sector number. My blocking/deblocking routine is always reading/writing a whole 512 byte sector for every 128 byte cpm sector aka 'not cached' and so not optimal but since the speed is more than adequate I do not mind (yet).


...

Not sure if this relates to the problem you are seeing, but I just noticed another math problem. You tell CP/M that you have 68 sectors per track (128B records). But, the drive has only 17 sectors per track (as you say you are not using the full 512B only the first 128B). So, what you describe for the sector arithmetic would apply to a BIOS where you were actually doing deblocking, NOT the case where you only use the first 128B of each physical sector. something is wrong there.

 

[gertk]

Not sure if this relates to the problem you are seeing, but I just noticed another math problem. You tell CP/M that you have 68 sectors per track (128B records). But, the drive has only 17 sectors per track (as you say you are not using the full 512B only the first 128B). So, what you describe for the sector arithmetic would apply to a BIOS where you were actually doing deblocking, NOT the case where you only use the first 128B of each physical sector. something is wrong there.

No, my own blocking/deblocking code takes care of that, If the sector number is 0 the first 128 bytes of the 512 byte sector are read, if sector number is 1, bytes 128 to 255 are read, when 2 bytes 256 to 383 are read and when 3 bytes 384 to 512 are read. That works fine. When writing it takes a bit of overhead as I need to read the whole 512 byte sector in, overwrite the appropriate part and write it back. Not optimal but works

 

[gertk]

Just did a quick test: XAMN.BAS can scan for bad sectors and asks for start and end track

I entered 1023 as starting track and 1024 as end track and it scans track 1023 (harddrive light flashing) and when it hits track 1024 the program still runs but the harddrive light stays dark.
So I definitely need to adjust the DPB

 

[durgadas311]

Just did a quick test: XAMN.BAS can scan for bad sectors and asks for start and end track

I entered 1023 as starting track and 1024 as end track and it scans track 1023 (harddrive light flashing) and when it hits track 1024 the program still runs but the harddrive light stays dark.
So I definitely need to adjust the DPB

Does XAMN.BAS access the track/sector via BIOS, or does it somehow create a fake/temp file and access via BDOS? Only the BDOS (file access) has the 8192K limit. Track and sector access via BIOS should not care, as long as you are using the full 16-bit values for track. It seems suspicious that you run into a problem at exactly tracks 0x03ff/0x0400, as that would be the point where the cylinder number (after extracting head number) overflows 8-bits.

 

[gertk]

Does XAMN.BAS access the track/sector via BIOS, or does it somehow create a fake/temp file and access via BDOS? Only the BDOS (file access) has the 8192K limit. Track and sector access via BIOS should not care, as long as you are using the full 16-bit values for track. It seems suspicious that you run into a problem at exactly tracks 0x03ff/0x0400, as that would be the point where the cylinder number (after extracting head number) overflows 8-bits.

XAMN.BAS uses BIOS calls

This is my 'hdmath' code for calculating the head, sector and cylinder number from the numbers given by BDOS, on first glance I do not see something amiss here.

;
; calculate head, sector and cylinder number
; from the given cpm sector (stored in hdsecno)
; and track number (in HL)
;
hdmath:
	lxi	h,0		    ; calculate sector offset
	lxi	d,128		; cpm sector size
	lda	hdsecno		; get cpm sector number
	ani	03h		    ; which sector offset ?
	cpi	00h		    ; start at 0
	jz	hdmath1     ; if so we're done
	dad	d           ; else  add offset
	cpi	01h		    ; and start at 128
	jz	hdmath1     ; done
	dad	d           ; else add offset
	cpi	02h		    ; and start at 256
	jz	hdmath1     ; done
	dad	d		    ; else add offset and start at 384
hdmath1:
	shld	hdsecoff	; store in sector offset

    ; calculate cylinder number
	lhld	hdtrackno	; get 16 bit track number
	mov	a,l		; get lower byte of tracknumber and keep bits 0 and 1
	ani	03h		; make head number
	mov	e,a		; save in E
	; shift the hdtrackno two bits to the right
	; (HL still contains the cpm track number)
	mov	a,h
	ora	a	; clear carry
	rar		; rotate right through carry
	mov	h,a ; save the result
	mov	a,l ; get lower byte
	rar     ; rotate right
	mov	l,a ; store back
	; and again
	mov	a,h ; get high byte
	ora	a	; clear carry
	rar		; rotate right through carry
	mov	h,a ; save the result
	mov	a,l ; get low byte
	rar     ; rotate right
	mov	l,a ; store back

	; shift the sector number down by two bits (divide by 4)
    ; as each 512 byte sector can contain four 128 byte cpm sectors
	lda	hdsecno ; get sector number
	ora	a       ; clear carry
	rar         ; rotate right
	ora	a       ; clear carry again
	rar         ; and rotate once more
	mov	d,a     ; store in D
	; HL now contains the physical cylinder number (tracknumber divided by 4=the number of heads)
	; E contains the head number (0-3)
	; D contains the 512 byte physical sector number (0-16)
	ret

 

[durgadas311]

I also don't see anything wrong with the track/head/sector math. From you description of the symptoms, it sounds like when the track reaches 0x0400 (or greater) that the HDD is simply not selected/accessed. Maybe something is going wrong in the case where H is non-zero (cylinder > 00FF, track > 03FF).

But, that's probably not the cause of the problem you originally saw? Just doing "DIR" you are not going beyond track 03FF?

 

[gertk]

I also don't see anything wrong with the track/head/sector math. From you description of the symptoms, it sounds like when the track reaches 0x0400 (or greater) that the HDD is simply not selected/accessed. Maybe something is going wrong in the case where H is non-zero (cylinder > 00FF, track > 03FF).

But, that's probably not the cause of the problem you originally saw? Just doing "DIR" you are not going beyond track 03FF?

Ha, the 1023 track (well actually 255 track) problem was a nice one and caused by inconsistent information in the controller datasheet..
Found the solution in a piece of Linux kernel source.. (from a long time ago...)

For setting the desired cylinder you need to write two bytes to the controller, the high byte needs to be shifted up so the sector number fits in the lower bits.
According to multiple datasheets the high byte of the cylinder number needs to go in bits 7,6 and 5 and the remaining bits are for the sector number.
In the kernel source they use only bits 7 and 6 which also seems more in accordance to the 1023 cylinder limit of the controller.

	; send high byte of cylinder (H)
	; reduced to three bits in bit 5,6 and 7 (not correct ? only two bits ?)
	; and put sector number over it
	call waitforsr
	mov a,h
	rrc
	rrc
;	rrc
;	ani 0e0h
    ani 0c0h
	ora d
	out WDData

Now that is fixed it I also changed the DPH/DPB to split the drive in two 'partitions'.

hd0dpblk:
    ; disk parameter block for (hard)disk C
    ; 4k blocks on 4 x 70 physical cylinders with 17 sectors a 512 bytes
    ; total byte size 2437120 bytes
    ; 2437120 / 128 =  19040 cpm sectors
    ; 2437120 / 4096 = 595 blocks of 4 k
  	dw  68 	    ;sectors per track
	db 	5 	    ;block shift factor
	db 	31	    ;block mask
	db 	7 	    ;null mask
	dw 	594	    ;disk size-1    (594+1)*4096 bytes
	dw 	255 	;directory max
	db 	0c0h	;alloc 0    (two blocks of 4096 bytes need to store (255+1) x 32 bytes entries)
	db 	0 	    ;alloc 1
	dw 	0 	    ;check size
	dw 	8 	    ;track offset

hd1dpblk:
    ; disk parameter block for (hard)disk D
    ; 16k blocks on 4 x 233 physical cylinders with 17 sectors a 512 bytes
    ; total byte size 8112128 bytes
    ; 8112128 / 128 =  63376 cpm sectors
    ; 8112128 / 16384 = 495 blocks of 16k
  	dw  68 	    ;sectors per track
	db 	7 	    ;block shift factor
	db 	127	    ;block mask
	db 	7 	    ;null mask
	dw 	494	    ;disk size-1
	dw 	1023	;directory max
	db 	0c0h	;alloc 0    (two blocks of 16384 bytes needed to store (1023+1) x 32 bytes entries)
	db 	0 	    ;alloc 1
	dw 	0 	    ;check size
	dw 	288     ;track offset

BTW: Copying files with PIP works fine between C and D, still not to (or from) A to C (or D)
Will go through the rom source to see if I can find the cause.

 

[gertk]

Quick update:

copying seems to work with PIP but the directory gets messed up after some copy actions.
Writing on the C or D drive from within the comm program (ST.COM) works flawless though.

 

[durgadas311]

My guess is still that there is some conflict or interference between the ROM BIOS and your add-on BIOS. Unfortunately, my simulator does not support the earliest Kaypro hardware (II, 4/83) (that's on my "someday todo list") or else we could run your software there and try to diagnose it.

Are you saying that the directory (on disk) gets corrupted after/during PIP between floppy and HDD? A dump of the corrupted directory might give some clue as to what is causing it. Also, when running the "DIR" test, does it always show the same bogus directory pattern: are those file names filled with apostrophes? And there are always (exactly) 12 of those bogus files shown?

 

[gertk]

The directories got corrupted even when copying from C to D or vice versa, I tried to re-use the original bios buffer space but also ran into trouble.
Right now I am thinking that my 'perfect' harddisk is probably not perfect..
I will write a testprogram for that to rule that out.

My benchmark stresstest is compiling the XAMN.BAS program and it compiles fine on drive D but not on drive C, it errors out with a strange REF-DEV LIB error.
Same files (for as far as I can tell, I have no tool to proof) different partitions, different results.

Tried renaming the sbasic files and copying 'fresh' copies from the working D drive but the error stays the same, so it could also be some weird stack or other memory problem.

P.S. do my new disk parameter blocks make a bit of sense ?

 

[gertk]

Update:
Checked the harddisk by reading each sector, writing the full sector with $AA, read it again, compare the data and signal if error, then write the full sector with $55, reading, comparing and finally writing the original data back. No errors except for the very last track 305, head 3 sector 9, then it goes in semi endless retry seeks. That sound I never have heard during normal CP/M use.
Since that track is not a part of any active partition I can say the harddrive itself is fine. So back to debugging my code

 

[durgadas311]

I did get a Kaypro 4/83 simulator running (but not with an ISA MFM HDD). One thing I noticed is that the graphic displayed for the NUL (00H) character is what I was mistaking for an apostrophe (27H). So, it means that the "bogus" directory listing you get is full of NULs. That may help with what is going on. From the CCP perspective, it must be getting "success" back from the search calls, but the dma buffer is filled with NULs. That would imply that the BIOS is returning NULs, or possibly the buffer being used (0080H) has NULs (garbage) in it and the BIOS is failing to copy the disk data to the current DMA buffer. I have not studied the normal Kaypro BIOS for over 30 years, but could it be that your BIOS changes have affected the final "deblocking" copy done for the floppy? I suspect the ROM BIOS read routine for the floppy cannot copy the data to the user's DMA buffer, as it is likely to be "underneath" the ROM. So, the ROM probably reads the physical sector into high memory and leaves it to the CP/M BIOS to do the final copy after the ROM returns. This requires the two to be in perfect sync, right down to knowing the exact buffer address as well as knowing which partial sector to copy.

This doesn't explain why only 12 directory entries are printed, as I'd expect that to keep happening for the total number configured for the floppy disk. But, perhaps there is more randomness somewhere and the buffer contents changes.

 

[gertk]

Some progress:

The directory corruption (and compile problems of S-Basic) is fixed: my Extent Mask (other CP/M book named it 'Null Mask') was wrong on the C partition.
On the D drive the compilation went ok but on the C Drive it failed with the most exotic errors like: I'm Lost...

My next step would have been to make the block size and such the same as the D Drive and try again and then I saw the Null Mask/Extent Mask value being the same for both partitions and thought: that can't be right ? So back to my new favourite CP/M book: Programmers CP/M Handbook by Andy Johnson-Laird.

There this value was called 'extent mask' and with more detailed explanation on how to choose the value. (I had a value of 7, which was fine for the 16k block D drive but not for the 4k blocks of C). Changed the value into '1' and wiped the directory of C.
PIP-ed the files from D to C and retried the compilation of XAMN.BAS on the C drive and it worked!

Now for the floppy drive problems:

The original rom routines do quite a lot of trickery by buffering sectors and only write/read when absolutely necessary (for example if you change drives)
Since the original rom routines/bios have no support for more than 2 drives it gets confused when you forcibly set the 'last drive' number to 2 or 3 (which is totally acceptable for BDOS).

The way the bios signals a disk change is quite deeply buried in the code. Since I only have the annotated source of the Kaypro II rom it is like trying to find a needle in a haystack. The voodoo for the floppies is quite extensive, including support for single density and double density, single sided and double sided, 128 byte sectors and 512 byte sectors.

Modifying the original bios is daunting as so much is hardcoded for just two (floppy) drives.

I bet my chances are higher in just adding the floppy routines from scratch in my own bios, it might not end up as being efficient but hey, if it works..
With the harddisk up and running the floppy drive will only be used occasionally so if it does not copy as fast as the original code, so be it.