• Please review our updated Terms and Rules here

Roland D-20 decoding the mysterious floppy format.

I have it working --- see screenshot. Some notes:
  • I am not convinced the filename is right. There's clearly a machine-readable filename in the directory and a human-readable one in the file header, but I am unsure whether the 333 is part of the filename or represents something else.
  • The maximum file length appears to be 48kB. This seems a bit small. It'd be nice to find a way to put big files on the disk and see what happens.
  • The code can easily get confused by bad sectors leading to a corrupted directory.
  • Decoding the contents of the files is a different problem to decoding the file system!
You should be able to get binaries for OSX and Windows from here: https://github.com/davidgiven/fluxengine/actions/runs/5673495069 (look at the bottom of the page under 'Artifacts'). PTAL?

1690403985619.png
 
Fantastic news, and also, amazingly quick response!
Thanks a bunch for that.
Regarding the strings "Initial Patch", it's as I feared, SRAM corruption.
It most likely means that the SRAM for that area was reset by the CPU to its default "empty/initial" SysEx preset.
It should contain the preset Patches (SysEx), but my unit struggles to hold it's memory (ToDo list)
The structure should then be identical for every 32 or 64 presets.

I do have at least one floppy that is very full, but I don't know which (forgot to label...).
Your e-mail should very soon have received a re-dump of the 10 floppies I'm trying to archive, at least one should be very full (Also one or two may actually be FAT12).
I still fear that quarter stepping is required to read these disks properly, as i see lots of issues on the older disk (still readable by the D20 itself).
Again, the solution could be to interface the native D20 drive as an Apple II drive by converting the /STEP /DIR to Phase A-D, perhaps by an arduino.
 
hjalfi said:
I have it working --- see screenshot. Some notes:
  • I am not convinced the filename is right. There's clearly a machine-readable filename in the directory and a human-readable one in the file header, but I am unsure whether the 333 is part of the filename or represents something else.
  • The maximum file length appears to be 48kB. This seems a bit small. It'd be nice to find a way to put big files on the disk and see what happens.
  • The code can easily get confused by bad sectors leading to a corrupted directory.
  • Decoding the contents of the files is a different problem to decoding the file system!
You should be able to get binaries for OSX and Windows from here: https://github.com/davidgiven/fluxengine/actions/runs/5673495069 (look at the bottom of the page under 'Artifacts'). PTAL?

View attachment 1261393
Click to expand...
Again, Amazing!
Is there a way for me to test this?

You should soon have all 10 floppies re-dumped
 
I got to download the build! Had to log in first.
I got one image to decode (D20_2.flux), and I can see that the file names should read "AMBIENT" and "ANALOG_PAD" (ANALOG PAD in real life), which means that 222 and 333 should indicate the file name extension, as in 333 is most likely a sound patch, and 222 is a sequence, leaving hypothetically 111 as a drum pattern.
By logic, the file names are stored as 10 characters, padded by "_" which gives logic to "AMBIENT" being padded by three under scores
All by memory, so I may be wrong, but I'm fairly certain.
If you're writing a guide on this in the future, there are many names for this, but the User manual should have a good direction on what these files should be called.

Regarding the maximum file length as 48kB, this may be connected to the Factory Patches file size: 44kB.
Not a fluke by my logic.

Also, regarding the actual dumping of these floppies, is there a way to filter out leftover MFM / FAT12?
I think that is going to be an issue if there is no native D20 floppy drive support.
 
Last edited:
The client doesn't change the dump format at all --- if you're using the GUI you can just load an existing .flux file and reread it with 'Read Disk' or 'Browse Disk'.

Disk 1: about 50% bad sectors.
Disk 2: almost (but not quite) clean.
Disk 3: the first half is fine, the second garbage. However, I can see that this contains a file bigger than 48kB, which means I can tell that this uses a CP/M-style extent system to handle big files.
Disk 4: garbage.
Disk 5: garbage.
Disk 6: about 10% bad sectors.
Disk 7: garbage, but in a different way --- about 90% of the sectors seem to be missing.
Disk 8: as above.
Disk 9: a 720kB MFM disk with no filesystem at all on it! At least, as far as I can tell; the flux file only contains data for one side.
Disk 10: about 5% bad sectors, but the directory seems to be corrupt --- not bad, just containing apparently invalid data:

Code: 
0001D400   00 52 4F 4C  41 4E 44 2D  47 43 52 44  4F 53 4E 00  .ROLAND-GCRDOSN.
0001D410   00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  ................
0001D420   00 45 4D 50  54 59 5F 5F  5F 5F 5F 32  32 32 04 00  .EMPTY_____222..
0001D430   01 02 03 04  05 06 07 00  00 00 00 00  00 00 00 00  ................
0001D440   00 54 45 52  52 41 5F 46  55 45 4C 32  32 32 07 00  .TERRA_FUEL222..
0001D450   08 09 0A 0B  0C 0D 0E 0F  10 11 00 00  00 00 00 00  ................
0001D460   E5 54 45 52  52 41 5F 46  55 45 4C 32  32 32 07 00  .TERRA_FUEL222..
0001D470   08 09 0A 0B  0C 0D 0E 0F  10 11 00 00  00 00 00 00  ................
0001D480   E5 20 20 20  20 20 20 20  20 20 20 20  20 20 00 00  .             ..
0001D490   00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  ................
0001D4A0   E5 20 20 20  20 20 20 20  20 20 20 20  20 20 54 45  .             TE
0001D4B0 52 52 41 20 46 55 45 4C 00 00 00 00 00 00 14 05  RRA FUEL........
0001D4C0   12 12 01 00  06 15 05 0C  20 20 20 20  20 20 00 00  ........      ..
0001D4D0   00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  ................
0001D4E0   E5 20 20 20  20 20 20 20  20 20 20 20  20 20 00 00  .             ..
0001D4F0   00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  ................

That "TERRA FUEL" (with a space) seems misplaced? You can see it overwriting a bit of the next directory entry...

I've added big file support; thanks! (It uses byte 15 of the directory entry as an extent counter.) Redumping with the new drive should produce better results. But I'm afraid that to get good reads you'll need to somehow use a quarterstep drive, or risk wrecking the alignment on a normal drive.

Updated binary should be here (once it builds): https://github.com/davidgiven/fluxengine/actions/runs/5674336385
 
BTW, re extensions: I think you're right. AMBIENT___222 and ANALOG_PAD333 have completely different structures, with the 222 file having a bigger file header with a lot of extra data in it. I'll need to think about how to map filenames. It's kind out of FluxEngine's scope to read the display name out of the file header; it should be limiting itself to the filesystem structure only. (It may be possible that if you create a file on the D20, and then change the name, it only updates the name in the file header and not the actual name of the file in the directory.)

Not sure what you mean by MFM/FAT12 disks --- the rolandd20 decoder will just ignore any MFM data as the sector headers don't match...
 
Again, amazing work!
With the new version I can see that the directory listings are starting to appear in almost all the dumps, although not all files readable :D
I can even see a .111 file in some of the dumps for the first time.
I'm now doubting what is what regarding .111, .222 and .333.
Needs more comparison, but my D20 is still waiting for a drive belt (still working with a glued belt, but do not want to reassemble with a hack job).
Don't worry about it though as I understand that is not what FluxEngine is about.
A post-dump tool should be able to do that in the future, hopefully combining hooking directly up to the D20 native drive through some interface.

In what stage do you consider the file system encoder to be in by now?
Are we in a state were we can create an empty formatted image, write to a floppy then try a load and save ritual?
Would you consider the files extracted from the image to be "100%" correctly decoded?
As in, the image may be 50% bad, but all relevant data can be extracted unaffected by the bad sectors (if lucky).
Also, if the CRC is bad, you won't be able to save the file, or pad the file in the corrupted sections as an option.
Should all the files be visible in the directory listings?
I do see sign of corruption in one (bad) image.

The "TERRA FUEL" section may stem from how files are saved.
Say, i created "TERRA FUEL" (TERRA_FUEL), saved it, made a revision and called it "TERRA 2" (also TERRA_FUEL, but saved as TERRA FUEL).
I can see both files in the directory listing of the sixth image if it is that image you are referring to.
Just a wild theory though. Don't trust me on this, but am fairly certain that this is what actually happened.

I do know that one of the floppies has many files on it (more than 5), but it may have been one of the bad dumps.
Do you see a theoretical maximum files in the file system?

By MFM/FAT12, I was thinking of something that may not work in real life.
The floppies were formatted originally as FAT12, and then formatted in the D20 with bad alignment.
That probably means a lot of noise from partially overlapping tracks.
This is probably where quarter stepping comes in, as the floppies work in the D20.
Was hoping for the possibility of a "filter" in software or hardware, but likely impossible.

I may have asked before, but how difficult would it be to adapt the "apple2_drive" function in this context with the native D20 drive?
 
I have actually just added write support, so go ahead!. You can format disks and do basic file management. Assuming the build works, it'll be here:

github.com

Fix issue where trying to rename files by clicking on the tree wasn't… · davidgiven/fluxengine@1927cc7

PSOC5 floppy disk imaging interface. Contribute to davidgiven/fluxengine development by creating an account on GitHub.
github.com github.com

To format a disk: fire up the GUI, select the drive and the rolandd20 profile, and press the 'format disk' button. You'll be presented with the file browser. You can add files here. Once done, press the 'commit changes' button and the disk will be written and formatted in a single pass.

Note that disks written in a normal PC drive will have the correct alignment --- meaning they should be readable! So if you can format a disk on the PC drive, then copy files onto it with the D20, the D20 should follow the same alignment as the PC drive, meaning you should be able to read them. I don't know if the D20 has the ability to copy files from one disk to another, but that's an easy way to recovering individual files. Could you try this and let me know what happens?

There are some remaining unknowns. There's one byte in the directory entry I don't understand; I've seen it set to 4 for a 222 file but 0 for a 333 file. Some kind of file attribute, maybe? There's no way of knowing without trying it on a real machine and seeing. The biggest issue preventing me from calling this done is the entire unknown first half of the disk. I'd like to see an image of a disk that the D20 thinks is almost full.

Re MFM disks: unfortunately noise can't be filtered out, or at least, not by me. Reading two tracks at once produces complete garbage. It sounds like it should be possible to remove the noise by reading the neighbouring track, but floppy disk drives are opinionated about the data they read and will drop pulses if they're too close together, rendering the results useless. You might be able to do it with special hardware, but a quarter-stepping drive is more feasible.

Re the Apple II drive: you'd need to figure out the D20 drive pinout. The Apple II drive just exposes four pins for the four stepper motor phases, but who knows what the D20 drive does...
 
Fantastic!

If you see back at these post:
note some important corrections at post:
https://forum.vcfed.org/index.php?t...mysterious-floppy-format.1243226/post-1319758 )
Pinout at post https://forum.vcfed.org/index.php?t...mysterious-floppy-format.1243226/post-1318657

I'll repeat the corrections here to avoid confusion:
Native D20 drive during read

WD (write data, we don't want to) 5V
WCC (write current, we don't want to) 0V
WG (write or read? we want to read, also the default state when idle): 5V
MTR (motor on) 0V
FRD (pulse train output)

you see that I may have exposed the relevant pins and what state they should be in, including how the stepper phases behave.
The stepper and control is internally powered by the floppy drive, so only logic is used to control it externally.
I need to verify that the stepper phases are equal to an Apple II drive as in A=1, B=2 etc, as seen in the screen shots (the logs have much more data).
This I can possibly verify at eevblog or somewhere else, unless someone here steps in (pun?)

If the "apple2_drive" option "Bit bangs" and quarter steps until a valid GCR signal is found, we may have a winner!
I may have observed the D20 drive quarter stepping during reading further into the disk, which may indicate that quarter stepping up or down during a partially bad track may be necessary.
 
Ah, I completely forgot you posted that! That looks very plausible for wiring up to something which is expecting an Apple II drive. You'd need to experiment as to how the phases map, but it should be doable. Unfortunately I don't actually have an Apple II drive or one of the adapters, so I can't help there --- all that code was contributed, but it should be well supported by the Greaseweazle and I at least have reports that the FluxEngine client integration works.

To enable this, add `apple2_drive` to the read line:

Code: 
./fluxengine read rolandd20 apple2_drive ...

If you're using the GUI, press the 'custom configuration' button and add `apple2_drive` on a line of its own. This won't actually read overlapping tracks yet; there may be custom code required there, but you should be able to use `--drive.head_bias=` to adjust the overall position of the read (in quarter-step tracks). There will probably be some fiddling required; this hasn't been tested much...
 
I cannot for the life of me find any details on the adapter, how it's wired and other necessary details.
My google-fu is fairly good, but I may have missed something.
Do you have any details / links whatsoever?
Seems like I may have to consider loosening those two stepper screws after all.

Is there any way of making a good flux dump out of two bad?
 
On second thought, I do have some old Apple 20pin 3.5" (Have easy access to an SE/30, more in external storage) floppy drives around, and it looks like they may be able to quarter step as they also use phase instead of /STEP /DIR
Could potentially be one of the solutions.
 
I also still believe that an Arduino Mega and a fork of this project could easily handle the native D20 drive to dump a clean image, probably without flux, but still a clean image.
Should also provide potentially an "easy" solution for other owners of a D20 to dump their floppies.
Pinout and handling of the signals should be very identical.
github.com

GitHub - eedede/treckr: Recover your old Apple II formatted DOS 3.3 disks to USB using Arduino

Recover your old Apple II formatted DOS 3.3 disks to USB using Arduino - eedede/treckr
github.com github.com

Any suggestions of a forum where such coding help could be provided?
 
Mac floppy drives are special --- they can spin at variable speeds and there's extra pins for controlling this. I've never worked with one and have no idea what's involved in driving one. (Neither the Greaseweazle nor the FluxEngine need variable speed control to access these disks.) I have not heard that they do quarter steps. Apple II drives do quarter steps, but AIUI they are all 40 track drives, and the D20 is an 80 track disk.

The trekr project does look like it'll work, but to decode the images you'll need to somehow get it to save flux images as something which FluxEngine can read. Alternatively, it'd be fairly straightforward to add direct support for the trekr hardware to the FluxEngine client; the protocol looks pretty simple.

Edit: actually, another look at trekr is less encouraging. It looks like the hardware doesn't communicate raw flux data to the PC but instead does some processing that's specific to Apple formats. The Apple II format uses a clock period of 4µs and Brother uses 3.83µs, which may be close enough to work, but also may not be. See https://github.com/eedede/treckr/blob/master/treckr/treckr_assembly.ino, although I think the comment on line 19 is wrong (I think it says 64µs instead of 4µs). You might need to adjust the timer interval.
 
Last edited:
Thanks for having a look!
What I was hoping for with treckr is filesystem imaging with an arduino.
Flux level would be asking too much of it I believe.
Sounds out of reach, or am I onto something?
 
I took the chance and rotated that stepper ever so slightly counterclockwise.
There I immediately got perfect readings😍
All while following carefully on the oscilloscope to see realtime signal levels.
Some floppies also needed realtime adjustments during the dump.
The worst part is how easy it actually was, although scary.
Smart as I rarely am, I made a scratched line on the edge of the stepper and chassis before doing anything, so re-aligning shouldn't be too far off.
Should be 99% fine for reading FAT12 floppies again, but writing I doubt I'd trust anything important with.

hjalfi, you should by now have all the 10 dumps in your mail.
All but one floppy should be 100% clean dumps according to the logs.
As mentioned in the mail, there seems to be a file system issue with "Invalid filesystem: sector nnn is out of bounds"
 
Excellent! I'm encouraged to know how straightforward the recalibration was.

The invalid filesystem error is because the track numbers do wrap round, but oddly. Luckily, the way that each file starts with its own filename means I can correlate directory entries with files.

Disk #10's SPACEMAMBO.111 claims to start in block 51, and its file header is at 0x13800, which is track 26.
Disk #5's ELICTIKRMX.222 claims to start in block 56, and its file header is at 0x0fc00, which is track 21.
Disk #4's TUBULAR_Q_.222 claims to start in block 63, and its file header is at 0x0a800, which is track 14.

There are 39 tracks in the top half of a disk, and above block 38 blocks are allocated in reverse, from the directory block at track 39 down. So:

SPACEMAMBO.111: 38-(51-39)=26, which matches.
ELICTIKRMX.22: 38-(56-39)=21, which matches.
TUBULAR_Q_.222: 38-(63-39)=14, which matches.

This took way longer to figure out than it should, primarily because I found another bug where if extents where in the directory in the wrong order then FluxEngine would read the blocks in the wrong order. So I thought SPACEMAMBO.111 started at block 63 which threw off all the analysis. I've fixed it.

I've verified that a bunch of files I read contain what look like correct headers, so I'm relatively sure that the files are intact. Unfortunately as they're all huge binary blobs, who knows. It'll take someone to figure out the format to know for sure.

Binaries should go here, eventually: https://github.com/davidgiven/fluxengine/actions/runs/5719261889

I think that's the bulk of the filesystem support done, so I'll write it up and push it to head. There are still the other magic bytes in the directory to figure out but we won't know that until trying to write disks and seeing what a real D20 does with them.
 
BTW: if your D20 is currently working, could you try writing a disk and seeing if it works? Also, I'd really like a flux file from a blank D20 disk formatted on the machine, to add to my test corpus and help prevent accidental regressions in the future. Thanks!

Also, would you like a credit in the documentation, and if so, what as?
 
Now, that’s a head scratcher!
Amazing that you figured it out.
Realy hope that bad sector in image 3 was not in use.
I thought about dumping the empty formated floppy, but forgot it.
Will do!
Will also try to format the other way arround. Won’t be tomorrow but comming weeks should.

Do you have any sugestions for stress testing the file system?
Many small files?
Files deletion?
I still believe a .$$$ is a deleted file.

Don’t mind a credit, but lets be realistic here. It is your tool and your knowledge.
I’ll let you decide what and how.
I have a correction for your documentation btw: it cannot store samples (audio) as stated. Only patches (SysEx), drum patterns (midi) and sequence (midi)

Speaking of, the binary blobs may be relative to Roland’s other dedicated midi sequencers at the time.
It wouldn’t be economical of them to reinvent the wheel just for the D20.
I’ll do some research that way in the coming weeks.
 
You must log in or register to reply here.
Back
Top