Grokking (machine learning)

Edit links
Source: Wikipedia, the free encyclopedia.
Not to be confused with Grok (chatbot).
The blue loss curves represent early memorization of the training set (overfitting), and the red curves show the late generalization, with the learning of a modular addition algorithm that works with unseen inputs.[1]

In

interpolation threshold, after many iterations of seemingly little progress, as opposed to the usual process where generalization occurs slowly and progressively once the interpolation threshold has been reached.[2][3][4]

Transition

Grokking was introduced in January 2022 by OpenAI researchers investigating how neural network perform calculations. It is derived from the word grok coined by Robert Heinlein in his novel Stranger in a Strange Land.[1]

Grokking can be understood as a phase transition during the training process.[5] While grokking has been thought of as largely a phenomenon of relatively shallow models, grokking has been observed in deep neural networks and non-neural models and is the subject of active research.[6][7][8][9]

One potential explanation is that the

weight decay (a component of the loss function that penalizes higher values of the neural network parameters, also called regularization) slightly favors the general solution that involves lower weight values, but that is also harder to find. According to Neel Nanda, the process of learning the general solution may be gradual, even though the transition to the general solution occurs more suddenly later.[1]

Recent theories[10][11] have hypothesized that grokking occurs when neural networks transition from a "lazy training"[12] regime where the weights do not deviate far from initialization, to a "rich" regime where weights abruptly begin to move in task-relevant directions. Follow-up empirical and theoretical work[13] has accumulated evidence in support of this perspective, and it offers a unifying view of earlier work as the transition from lazy to rich training dynamics is known to arise from properties of adaptive optimizers,[14] weight decay,[15] initial parameter weight norm,[8] and more.

See also

  • Deep double descent

References

  1. ^ a b c Ananthaswamy, Anil (2024-04-12). "How Do Machines 'Grok' Data?". Quanta Magazine. Retrieved 2025-01-21.
  2. ^ Pearce, Adam; Ghandeharioun, Asma; Hussein, Nada; Thain, Nithum; Wattenberg, Martin; Dixon, Lucas. "Do Machine Learning Models Memorize or Generalize?". pair.withgoogle.com. Retrieved 2024-06-04.
  3. arXiv:2201.02177 [cs.LG
    ].
  4. arXiv:2310.19470 [cs.LG
    ].
  5. arXiv:2205.10343
    .
  6. arXiv:2405.19454 [cs.LG
    ].
  7. arXiv:2310.17247 [cs.LG
    ].
  8. ^
    arXiv:2210.01117
    .
  9. ISBN 978-1-7281-8671-9
    .
  10. doi:10.48550/arXiv.2310.06110
    , arXiv:2310.06110, retrieved 2025-02-17
  11. doi:10.48550/arXiv.2311.18817
    , arXiv:2311.18817, retrieved 2025-02-17
  12. doi:10.48550/arXiv.1812.07956
    , arXiv:1812.07956, retrieved 2025-02-17
  13. doi:10.48550/arXiv.2407.12332
    , arXiv:2407.12332, retrieved 2025-02-17
  14. doi:10.48550/arXiv.2206.04817
    , arXiv:2206.04817, retrieved 2025-02-17
  15. doi:10.48550/arXiv.2309.02390
    , arXiv:2309.02390, retrieved 2025-02-17
Retrieved from "https://en.wikipedia.org/w/index.php?title=Grokking_(machine_learning)&oldid=1283890657"