There are some details which could hinder the OpenVMS installation on a PWS 500au. Could be easier to assist if knowing what happens when you try to install OpenVMS.
The graphics adapter is not necessary for the installation. It can be done completely using the serial console (COM1 port).