I had a similar problem, Linux has kernel command line option, mem. I had a Laptop that took a 512MB memory module and BIOS recognized 512MB. But I only could use 256MB, once I accessed 256MB+1Byte it crashed.
So I added in bootloader to kernel command line "mem=256M". This works quite well. A lot of people seem to have problems with RAM, when they add such a function...
On that OS topic:
I suggest minix, it is still under development, X11 along with a lot of other stuff people are used to. It isn't that different from Linux or BSD, when viewing at userland. Programming related stuff and kernel differs a lot. It is POSIX compilant and has gcc, therefore a lot of programs are working on minix.
maybe haiku would also would work, but this needs some more power I think.
Both provide a live CD, but on a pentium I wouldn't excpect it will run smoothly from CD.