That is not a silly point at all. You confuse understanding the mechanics of it with having a theory of why it works. For the physics simulation, physics provides us with the theories which give us the equations underlying the physics simulation. For neural networks, why is next word prediction giving us AI that can do math? We don't really know!
There is no "theory" of tomorrow's weather. We understand the math of every single individual equation of an LLM (for example), just like we understand every single equation in the physics simulation. It's the entirety of the system that we don't understand (and hence can't predict) in our heads.
From a biology perspective - we have a good understanding of the carbon atom and how forces influence it. We really don't know much about a cell.
