Just found this over at xtravirt
A built in alternative to Memtest86 for ESX3
Instead of Memtest86 you also have the option of running 'ramcheck' which is a background memory tester built into ESX3. To start it, log into the ESX Service Console as root (or su / sudo) and run:
service ramcheck start
Check the link for more details....