Where their creativity lies at the moment seems to be a controlled mixing of previous things. Which in some areas works for the definition of creativity, such as with artistic images or some literature. Less so with things that require precision to work, such as analysis or programming. The difference from LLMs and humans in using past works to bring new things to life is that a human is actually (usually) thinking throughout the process on what adds and subtracts. Right now the human feedback on the results is still important. I can't think of any example where we've yet successfully unleashed LLMs into the world confident enough of their output to not filter it. It's still only a tool of generation, albeit a very complex one.

What's troubling throughout the whole explosion of LLMs is how safety of the potentials is still an afterthought, or a "we'll figure it out" mentality. Not a great look for AGI research. I want to say if LLMs had been a door to AGI we would have been in serious trouble, but I'm not even sure I can say it hasn't sparked something, as an AGI that gains awareness fast enough sure isn't going to reveal itself if it has even a small idea of what humans are like. And LLMs were trained on apparently the whole internet, so...