Woohoo! Got Debian's XFree86 4.0.1 packaged for potato. It's faster with my Trident 3D Imàge 975 than XFree86 3.3.6, because it uses more hardware acceleration (in fact, this was the main reason I tried to find it).
Of course, it couldn't be a painless experience.
For configuration, dexter didn't work (it died reproducibly when I was configuring it) and xf86cfg didn't work (I can't remember the reason why it didn't work). So, I was "forced" to use X's internal autoprobing. But then, my old, 2-button, serial mouse couldn't be used with 3 button emulation. It took me a lot of time to figure out how to do that (since the documentation -- both man pages and extra documents -- don't mention it). After some painful hours, I've discovered the magical line `Option "Emulate3Buttons"'. It then worked.
But then, there were remaining problems: some dock WindowMaker utilities were removed when I installed the new X (due to dependendy problems). I had to recompile them. Then, rxvt was strange. Recompile time again.
Now, everything seems fine. Another thing to throw on that "extras" CD.