As a statistician who has implemented models nowhere near the scale, and as someone who has experience in distributed computing, a lot of the actual work comes from global optimization, which means that all the state has to be communicated somehow. Now this state can be distributed, but it needs to exist, so now you need to have the same resources required to train the model, plus the resources required to shuttle the state around in a distributed manner. It’s cheaper and faster and more ecologically friendly to build better hardware.
As a statistician who has implemented models nowhere near the scale, and as someone who has experience in distributed computing, a lot of the actual work comes from global optimization, which means that all the state has to be communicated somehow. Now this state can be distributed, but it needs to exist, so now you need to have the same resources required to train the model, plus the resources required to shuttle the state around in a distributed manner. It’s cheaper and faster and more ecologically friendly to build better hardware.