Sounds gross to me. The human body is dirty. All that oil and dead skin gets caked into the jeans like any other piece of clothing. That then starts to decompose and attract bacteria and other microbes in addition to holding on to your body odor. Those microorganisms can't survive on denim alone, but that skin oil and flakes are plenty to keep them alive for a while.

Unwashed clothes are a common cause of acne and other skin conditions because even if you wash your skin, the oil and bacteria just rub right back off of the clothing into your pores.

I'm not saying you have to wash every piece of clothing, every time you wear it. But be logical. If you wear it with clothing in between, like a jacket or hoodie over a T-shirt, it doesn't collect as much gunk. If you wear it while exercising, it probably gets more gunk. If it's made of a material designed to repel water and oil, it probably doesn't need to be washed as much. But plain denim is absorbent and not directly a hostile environment to the microorganisms.