Even a lean distro like Xubuntu has a bunch of fonts you’ll never use. Which ones you won’t need depends on who you are and where you live, but for my purposes the Asian and Arabic fonts just clutter up the font selection menus.
They’re all in /usr/share/fonts/truetype/, neatly stashed in separate directories. Figure out which ones you don’t need (perhaps browsing with Thunar will help), then:
cd /usr/share/fonts/truetype sudo rm -rf arphic kochi thai
And so forth and so on, until you’re left with the ones you’ll actually use.
I also have a bunch of fonts I actually paid money for. While this is a single-user box, it’s easy enough to install ’em in the default system locations (rather than a user directory), with a pair of directories to organize them:
cd /usr/local/share/fonts sudo mkdir truetype sudo mkdir type1
Then copy the store-bought fonts into the appropriate locations. After doing this a few times, I organized the original fonts under a few directories, so this need not degenerate into a search-and-copy operation every time.
Worst case: when you’re done, just copy those two directories back to your font dump and copy them back for the next installation.
The final step is updating all the font caches, which point to directories & fonts that no longer exist:
sudo fc-cache -f -v
You may need the -r option to forcibly delete existing caches and re-scan from scratch.
More details there.
The details differ, often wildly, among distros, so perhaps this manual command-line fiddling will grossly confuse whatever automated GUI routines you’re supposed to use.