The primary problem is using agile all the time instead of when it is actually intended to be used: short term work that needs to be done quickly by a small team that are all on the same page already.

I think you got it entirely backwards.

The whole point of Agile is being able to avoid the "big design up front" approach that kills so many projects, and instead go through multiple design and implementation rounds to adapt your work to the end goal based on what lessons you're picking up along the way.

The whole point is improving the ability to deliver within long term projects. Hence the need to iterate and to adapt. None of these issues pose a challenge in short term work.