you’re not usually directly accessing/working on the hardware

I mean, you are. Sure, there's a layer of abstraction when doing tasks that require the intervention of the kernel, but you are still dealing with cpu registers and stuff like that. Merely by writing in assembly you are making your software less portable because you are writing for a specific ISA that only a certain family of processors can read, and talking with the kernel through an API or ABI that is specific to the kernel (standards like Posix mitigate the latter part somewhat, but some systems (windows) aren't Posix compilant).