Diagnosing a network that won't train

Work this checklist in order rather than guessing at hyperparameters -- a genuine bug and a bad learning rate can look identical from the outside, and the ordered checklist is what tells them apart cheaply. Start with the single most informative test available: can the network overfit one tiny batch, perfectly, with regularization turned off? If it can't, the problem is structural (a shape mismatch, a frozen gradient, a label bug), not a matter of tuning, and no amount of learning-rate searching will fix it. Only once that test passes does it make sense to debug symptoms directly -- a loss that won't decrease at all, a loss that explodes to NaN, or an accuracy that's stuck despite the loss moving -- each with its own distinct set of usual causes.