Though Unity works as expected (awesome!), there still seems to be some other bugs with VirtualBox integration. A big one for me is that seamless mode doesn't work when running 11.04.
You can still run Unity on any VM (or lower powered machine) by using Unity 2D. It doesn't ship with 11.04 (it will in 11.10, I believe) but you can get it like this:
It's in 11.04, you can just install the "unity-2d" package; you should only use the dailies if you're looking for trunk builds that might not be stable
It's classic or aught.