Some religions, for example Mormonism, picked up a taboo of coffee and tea alongside other abolitionist sentiments popular at the time, with similar reasoning to alcohol - caffeine is a psychoactive drug, so it's not "pure." Mormonism was founded when early puritanical and evangelical Christian movements grew rapidly in the US, and you'll see a similar taboo among other Christian sects originating in this time period.

Yoga probably has the same taboo in American Christianity as Dungeons and Dragons once did during the satanic scare era, and that taboo is likely only due to the link to spiritualist practices in yoga. To the brainwashed evangelical, since it's not Christianity, it's Satan.

Edit: a few words added for clarity