Foundation models, now powering most of the exciting applications in deep learning, are almost universally
based on the Transformer architecture and its core attention module. Many subquadratic-time architectures
such as linear attention, gated convolution and recurrent models, and structured state space models (SSMs)
have been developed to address Transformers’ computational inefficiency on long sequences, but they have not
performed as well as attention on important modalities such as language. We identify that a key weakness of
such models is their inability to perform content-based reasoning, and make several improvements. First, simply
letting the SSM parameters be functions of the input addresses their weakness with discrete modalities, allowing
the model to selectively propagate or forget information along the sequence length dimension depending on
the current token. Second, even though this change prevents the use of efficient convolutions, we design a
hardware-aware parallel algorithm in recurrent mode. We integrate these selective SSMs into a simplified
end-to-end neural network architecture without attention or even MLP blocks (Mamba). Mamba enjoys fast
inference (5× higher throughput than Transformers) and linear scaling in sequence length, and its performance
improves on real data up to million-length sequences. As a general sequence model backbone, Mamba achieves
state-of-the-art performance across several modalities such as language, audio, and genomics. On language
modeling, our Mamba-3B model outperforms Transformers of the same size and matches Transformers twice
its size, both in pretraining and downstream evaluation.
Tu as des subsides pour la LAMAL. Pour vivre à 1300 tu es obligé de les utiliser.
Je confirme 1300 c'est obligatoirement une colocation ou en couple et tu ne vas jamais au restaurant.
Mais à 4000 tu n'as plus toutes ces contraintes. Quand j'ai touché 3600 CHF pour la première fois j'avais vraiment l'impression d'être riche. Je pouvais aller au restaurant tous les jours, partir à l'autre bout de l'Europe sur un coup de tête etc...