Mostly it's not a real thing. Most people who speak multiple languages don't get worse at one of them. Europe is full of multilingual people who speak like native speakers in their first language, and then near native in a bunch of other ones. There are times when you can't tell what someone's first language is.

On the other hand, it can happen a bit if you never use your native language, especially in the first few minutes of trying to switch. One famous example of that is Arnold Schwarzenegger who has spent so long in the US speaking English exclusively that his German has a slight American accent (though mostly it's a heavy Styrian Austrian accent). Also, it is true that you can temporarily forget words, even in your native language. It's not so much that you're forgetting the word, just that your brain insists the object is a "pepino" (the Spanish word for cucumber) and for a moment you can't say "ok, yeah, I know it's pepino in Spanish, but what is it in English?"

But, "code switching" is a real thing. If an American moves to Australia, it's really helpful to adopt some Australian pronunciations just to get by. If you ask for a "wah-der" in Australia, they'll have to think for a second, if you ask for a "whoa-tah" they get what you want right away. Someone who speaks English fluently but is always among Brazilians who only speak a bit of English might use a subset of English to talk to them, use local phrasing that isn't proper English but is Portuguese idioms or expressions moved into English word for word, and use Brazilian pronunciations so they're understood more easily, even if those are mispronunciations under most English accents. But, it would be surprising if they couldn't revert to normal English in an English-only setting.