But what if they aren't happy to chat on their coffee breaks either?
What you're suggesting is basically just hitting the snooze button. "I'm sorry but I'm just the type of person who don't do small talk." in a polite but firm manner have worked wonders for me before.
Truly a sign of absolute madness.