In my field we use AI a lot for classification task that would need too much manual labour otherwise. We train the model for a specific task which will stay the same for a long time. We use statistical models to validate the results, and they are correct. The model will always be used for this task and we are happy we can use it. We actually do not care how the model did it, we are only interested in the result. Using XAI (explainable AI) we can actually can get pretty far in answering the question how the model works "mathematically" if we wanted. But of course we cannot infer causality from it.

There are so many tasks that can be solved by AI that are not critical, I see no reason why we should not use it in science until we found the real math behind the processes. The topics that are covered by media have to be discussed critically, the whole debate around GPT and the huge models big tech is building, but there is so much use of AI in other uncritical fields the public is just not aware of.

AI will never be smarter than the collective human race. It is our role to train its shortcomings so we can evolve ourselves. It's a cycle of training and learning