This is unfortunate and frustrating, because I've solved about half each of your "want a 6809 emulator that runs on a PC" and "want a better 6809 monitor" problems, but it's all done in Python, and so therefore useless to you. :-)
More seriously, I've been dealing with similar problems for a while, so what I have right now is:
1. Frameworks for CPU simulation and unit testing assembly code, in my
t8dev project. Currently the CPU simulation supports 8080, 6800 and 6502, but much of this work (including probably many of the 6800 opcode implementations) can be re-used for 6809.
2. A reasonably decent yet small (under 2K ROM, under 64 bytes RAM) monitor,
tmon, ported to 8080, with a fairly full set of unit tests that are fairly easily portable to implementations for other CPUs. (Some tests, e.g., examining memory, can be extracted to a more generic level and used just as they are: typing an examine command and making sure the correct data are displayed is not at all CPU-specific. Other tests, such as ensuring the registers are saved and reloaded correctly, are system specific but can be parametrized so that most of the test code can be generic and just the register definitions need to be specified separately for each CPU. I already have done this for other software; one example is the first four tests of a 6800 "qdigit" routine
here that just directly import generic tests from
here.
Writing the 6809 simulator and porting tmon to 6809 has been on my long-term to-do list for a while now (I have several 6809 machines which also have lousy monitors.) So I don't mind moving that up to a "current priority" project if there's someone around who wants to help with it.
If you want to have a play with tmon and see how the testing framework works, you can clone my
8bdev repo on to a Linux machine and type
./Test -E .build/obj/exe/nec/pc8001/exprom1.bin. You may have to go a couple of rounds of "get informed what additional system packages you need to install" (nothing particularly weird, and all available on virtually any Linux distro) and re-run the
Test command, but eventually you should see a NEC PC-8001 emulator pop up with that extension ROM added; you can type
CMD TMON and you'll be in the monitor ready to play. (That command will change to just
MON at some point.)
If you're interested in working on this, drop me a PM and we'll sort out the next steps.