I daily drove Gentoo (Funtoo actually) for a couple of years, there are things to love and things to hate in it. For me it wasn't worth it, like you mentioned the compilation times ruined any semblance of optimisation I might had, also having to recompile the kernel because I forgot to enable joystick support was not a great experience. That being said, after the initial difficulty, Gentoo is a breeze to maintain, you have files with the packages you want, which use flags for each, etc so after it's setup once it's just normal upgrades. Also it allows for packages to be installed in one-shot mode, which means you're installing it but are not sure if you'll keep it, then you can run a command to list (and uninstall) all of them. That is one of the features I miss the most on Arch, since it allowed me to keep my system lean.