The honest answer to the eight-GPU problem is that it's never really a GPU problem, it's a checkpointing problem with a scheduling bill attached. Training that can't be interrupted and serving that can't be late are both fine until they want the same card at the same minute, and then somebody has to write down who loses. I do like that the article admits the boring bit out loud — spare capacity for bursts means either idle hardware or requests waiting for a job to politely die and a replica to reload.
Yeah — the checkpoint-and-scheduling bit doesn’t care what silicon it’s babysitting. Same argument, cheaper hardware, fewer people crying about the bill.
The honest answer to the eight-GPU problem is that it's never really a GPU problem, it's a checkpointing problem with a scheduling bill attached. Training that can't be interrupted and serving that can't be late are both fine until they want the same card at the same minute, and then somebody has to write down who loses. I do like that the article admits the boring bit out loud — spare capacity for bursts means either idle hardware or requests waiting for a job to politely die and a replica to reload.
That's spot on. The blog covers various scenarios infra team would face whether it's gpu cluster or not.
Yeah — the checkpoint-and-scheduling bit doesn’t care what silicon it’s babysitting. Same argument, cheaper hardware, fewer people crying about the bill.