Aren’t they inherently less secure than a TOTP code?

They can be, depending on the types of threats you expect to face. If physical theft is an expected threat, then a hardware token runs the risk of being stolen and abused. For example, your attackers might just buy off cops to rob you and take your stuff. Having the physical device locked with a PIN/Passcode can mitigate this threat somewhat. But, that just becomes another password the attackers need to figure out.

On the other side of the coin, TOTP applications have started offering Cloud Backup options for accounts. What this demonstrates is that it's possible to move those accounts between devices remotely. A hacked device means those codes may be exfiltrated to an attackers device and you will be none the wiser. Good security hygiene and device hardening can help mitigate these issues. But, it also means you need to a lot of trust in a lot of third parties. Also, you need to be unimportant enough for an attacker to not burn a 0-day on.

Ultimately, security is all about trade-offs. If you worry about physical security and don't expect to face a threat which might compromise your phone, then a TOTP app might be a better option. If you are more worried about a hacked device being used to leak credentials, then a physical token may be a better choice. Each way you go has some ability to mitigate the risks. PIN for a physical token and device hardening for TOTP. But, neither is a silver bullet.

And, if your threat model includes someone willing and able to engage in rubber hose cryptanalysis, then you're probably fucked anyway.

I’ve heard that in the US, the 5th amendment protects you from being forced to divulge a password, but they can physically place your finger on the finger print scanner.

Ya, it's a weird space that you cannot be legally forced to divulge a password, except in cases where the content of the drive is a "foregone conclusion" (as defined by the US Supreme Court). But, they can absolutely collect biometric markers (including forcing a fingerprint scan).

In theory that might be true, but you're forgetting that your phone can be hacked, remotely even.

There are some hardware keys that have PINs, like OnlyKey, but you're always going to have trade-offs (OnlyKey has some as well). You are correct that someone could steal the key and know your password, but the likelihood that someone could do both is likely low for most people, and if it's high for you, you're probably a state actor or big cybercriminal engaging in some crazy shit anyway.

The core tenets of good security are something you know and something you have. For example, my work uses an RFID badge and a personal code to get into the building. Neither works alone. If somebody steals my badge, they can't get into the building without my access code. If somebody steals a database of access codes, they can't get in without the badge that pairs with the code.

A TOTP is something you have, sure, but it's only secure if someone is unable to extract the secret key (or the code at runtime, which is possible via screen-reading techniques); as long as it exists digitally, secrecy is not guaranteed, and the attack surface is larger—for any device connected to the internet—to have that key stolen.

A physical key is harder to steal over the internet, and an attacker would have to know you have one in the first place to even go about beginning to search for it (I don't think a court could force you to divulge that you have one or where it is, since you have a right to remain silent, but I'm NAL).

Ultimately, there is no panacea of security, or else everybody would use that one thing. In the end, it comes down to your threat model and the kinds of attacks that are likely to be thrown at you.