Efficient and Interpretable AI Models Through Sparse Nonlinearity

ML-driven sequence models are everywhere, underpinning chatbots like ChatGPT and Claude, AI weather forecasts, the 2024 Nobel Prize-winning work in physics (Hopfield networks) and chemistry (protein structure prediction), financial forecasting, and much more.

Sequence model learn patterns in ordered data by identifying relationships between elements across time or position. Different architectures solve this differently: RNNs process sequentially with explicit memory, while Transformers use attention to relate all positions simultaneously.

The range of sequence modeling tasks, and corresponding architectures, is quite broad. A many-to-one task might classify a movie review as positive or negative after reading all the words. One-to-many tasks generate sequences from a single input, like forecasting the weather for the rest of the week based on the current weather, while many-to-many tasks handle tasks like language translation, where both input and output are sequences.

Image by the author.

Given the scope of applications of sequence models in science and beyond, understanding and optimizing these models is one of the most important research directions in modern ML.

A crucial lens for analyzing ML models in general is via the lense of linearity versus nonlinearity. In my…