Some religions, for example Mormonism, picked up a taboo of coffee and tea alongside other abolitionist sentiments popular at the time, with similar reasoning to alcohol - caffeine is a psychoactive drug, so it's not "pure." Mormonism was founded when early puritanical and evangelical Christian movements grew rapidly in the US, and you'll see a similar taboo among other Christian sects originating in this time period.

Yoga probably has the same taboo in American Christianity as Dungeons and Dragons once did during the satanic scare era, and that taboo is likely only due to the link to spiritualist practices in yoga. To the brainwashed evangelical, since it's not Christianity, it's Satan.

Edit: a few words added for clarity

You're supposed to "be IN the world, not OF the world."

It means that while you technically live here, among other people, you're supposed to hold yourself apart from whatever those people are doing. It leads to a whole "that's not meant for me" mindset where you avoid learning new things because that might make you too knowledgeable and worldly. Like, you could befriend non-Mormons, but it really should be for the purposes of converting them, not to get to know them or have a diverse social circle. You shouldn't let any of THEM influence YOU in any way. Mormons are weird.