Every activation function, compared

Default to ReLU for a hidden layer -- it's cheap, it doesn't saturate for positive inputs, and it's the safest first choice for a new architecture. Switch to leaky ReLU or ELU if training reports dead units (a large fraction of activations stuck at exactly zero, contributing no gradient). Reach for GELU or SiLU in a Transformer or any architecture where a smooth, non-monotonic activation has been shown to outperform ReLU empirically -- both are the default in modern large-scale models. Sigmoid and tanh are rarely a hidden-layer default today; keep sigmoid for a final binary-classification output and tanh where a zero-centered, bounded activation is specifically wanted (a gate inside an LSTM cell, for instance). Whatever the choice, the question that should follow it immediately is: does this function saturate, and if so, where?