There are two issues here.
The first is that USB only works when there's an SD card in the machine. This fact wasn't documented, and I only discovered it (firstly by experimentation, and then by chance) in an off-the-cuff remark on one of the GP2X forums.
The second issue is that it's not possible to transfer data to the internal NAND memory. This was not my understanding of the system when I original bought it, and because I hadn't kept up with the pre-release hype and news in the forums, I hadn't learnt of the change in direction. In light of this information, it now makes sense for the USB to not function without it. However, there are no error messages indicating you should have an SD card. This kind of omission is true of so much software: either the error message doesn't exist (because the programmer just caught the exceptions and ignored them because it's too much work to re-throw or produce an error code), or is too cryptic.
Once I had an SD card in the back, the USB icons started appearing! Ok - so I still have the USB2 incompatibilities under Windows XP (so you have to disable USB2 functionality from the device manager) but presumably it'll work when I remember my cable.
Note that you still have to explicitly go into the USB menu before the connection happens, which is different to every other USB memory device I know, which caused some confusion and frustration.
Of course, the bigger issue here is that the "manual" that comes with the product does little more that tell you which button is the "A button" (hint - it's the one with 'A' on it!) but needs to include such information ahead of time. Fortunately, there is a community written GP2X manual written by GuyFawkes (presumably not his real name) which does help a bit.
And also, I found a second protective film over the screen, so it no longer looks scratched.
Thanks go to both the Reverend Simon Rumble, and Jurgen Molna for independently supplying with me the aforementioned SD memory cards to help me get my GP2X off the runway for the first time.