Apple SHARP

SHARP optimizes large-scale model training by utilizing a Smooth Hamiltonian Ascent approach to find flatter minima and improve generalization.

Apple researchers developed SHARP (Smooth Hamiltonian Ascent for Resilient Protocol) to tackle the sharpness-aware minimization challenge in deep learning. By leveraging a Hamiltonian dynamics framework, the optimizer efficiently navigates loss landscapes to locate flatter minima, which directly correlates to better test-time performance. In benchmarks against standard SGD and Adam, SHARP demonstrates superior robustness across ImageNet and various Transformer architectures while maintaining computational efficiency (reducing the overhead typically associated with second-order optimization methods).

https://machinelearning.apple.com/research/sharp-smooth-hamiltonian-ascent