To see that this is correct, just consider what happens when you use the exact same hardware with a 64-bit OS or with PAE. You get all 4 GB of your RAM. If you were right, this would mean that that RAM couldn't be used for your ROM and video cards and whatever, so why don't those stop working? Because they were only using address space in the first place, not RAM.
Put another way: if you get 3.5 GB of memory displayed as usable to a 32-bit OS, and on the exact same hardware you get 4 GB of memory displayed as usable to a 64-bit OS, and everything else works just as well on the latter OS – then that 0.5 GB of RAM is wasted on the 32-bit OS, however you slice it.