Earth’s spin axis slowly moves relative to the crust over time. A 120-year-long record of this polar motion from astronomical and more modern geodetic measurements displays interannual and multidecadal fluctuations of 20 to 40 milliarcseconds (mas) superimposed on a secular trend of about 3 mas/year. Earth’s polar motion is thought to be driven by various surface and interior processes, but how these processes operate and interact to produce the observed signal remains enigmatic. Here we show that predictions made by an ensemble of physics-informed neural networks trained on measurements to capture geophysical processes can explain the main features of the observed polar motion. We find that glacial isostatic adjustment and mantle convection primarily account for the secular trend. Mass redistribution on the Earth’s surface – for example ice melting and global changes in water storage – yields a relatively weak trend, but explains about 90% of the interannual and multidecadal variations. We also find that core processes contribute to both the secular trend and fluctuations in polar motion, either due to variations in torque at the core-mantle boundary or dynamical feedback of the core in response to surface mass changes. Our findings provide constraints on core-mantle interactions, for which observations are rare, and global ice-mass balance over the past century, and suggest feedbacks operating between climate-related surface processes and core dynamics.