Have you heard of social engineering and phishing? I consider those to be analogous to uploading new rules for ChatGPT, but since humans are still smarter, phishing and social engineering seems more advanced.

We can create rules and a human can understand if they are breaking them or not...

So I take it you are not a lawyer, nor any sort of compliance specialist?

They aren't thinking about it and deciding it's the right thing to do.

That's almost certainly true; and I'm not trying to insinuate that AI is anywhere near true human-level intelligence yet. But it's certainly got some surprisingly similar behaviors.