Intro
Transformers have become the dominant architecture in modern artificial intelligence, powering large language models, generative AI systems, computer vision applications and increasingly sophisticated multimodal models. Their self-attention mechanism has proved remarkably effective at modelling relationships between elements in a sequence, enabling the rapid development of systems capable of processing text, images, audio, video and other forms of data. However, the computational cost of attention, particularly as context windows become longer, has encouraged researchers to investigate deep learning architectures beyond Transformers.
A growing group of alternatives is now attracting attention across AI research, including state space models such as Mamba, Mamba-2, recurrent architectures such as xLSTM and RWKV, convolutional approaches such as Hyena and StripedHyena, hybrid architectures such as MambaVision and Griffin, and newer memory-oriented approaches such as Titans. These architectures do not necessarily represent replacements for Transformers. Instead, they explore different ways of processing long sequences, reducing memory requirements, improving inference efficiency and introducing alternative inductive biases. For data scientists, understanding these emerging deep learning architectures is increasingly important as AI research moves towards models that balance performance, computational efficiency and specialised applications.
Lets Dive In
Why Researchers Are Looking Beyond Transformers
The Transformer architecture transformed deep learning because self-attention provides an effective mechanism for modelling relationships between tokens or other input elements. Rather than processing a sequence strictly from beginning to end, attention allows a model to establish relationships between different parts of the input.
This has been particularly valuable for natural language processing. Transformers can identify relationships between words separated by large distances, making them effective for translation, summarisation, question answering and language generation. The same general principles have subsequently been adapted to computer vision, speech recognition, biological sequence modelling and multimodal AI.
However, standard self-attention has an important computational limitation. The attention operation generally has quadratic complexity with respect to sequence length. As the number of tokens increases, the computational and memory requirements can grow rapidly. This becomes increasingly important when models process lengthy documents, source code, genomic sequences, video or other high-volume sequential data.
This does not mean that Transformers are becoming obsolete. Instead, researchers are exploring whether different architectures can provide better efficiency for particular workloads. Some alternatives aim to reduce computational complexity, while others introduce persistent memory, recurrent processing or specialised mechanisms for long-range dependencies.
The result is a more diverse deep learning architecture landscape in which Transformers remain important but are increasingly being evaluated alongside alternative sequence models.
State Space Models and the Rise of Mamba
State space models, commonly abbreviated as SSMs, are among the most important developments in the search for Transformer alternatives.
The basic idea behind a state space model is to maintain a representation of previous information in a continuously updated internal state. Instead of calculating direct attention relationships between every pair of tokens, the model updates and propagates information through this state.
Earlier SSM approaches demonstrated the potential of this concept, but models such as Mamba significantly increased interest in the approach.
Mamba, introduced by Albert Gu and Tri Dao, uses selective state spaces in which the parameters controlling information propagation depend on the input. This allows the model to selectively retain or forget information rather than treating every element of a sequence identically. The original research reported linear scaling with sequence length and significantly higher inference throughput than comparable Transformers in its evaluations.
The significance of Mamba extends beyond raw benchmark performance. Its architecture demonstrates that sequence models do not necessarily need conventional self-attention to achieve strong results across multiple domains.
The researchers reported strong results across language, audio and genomics, with Mamba models demonstrating competitive performance while offering advantages in long-sequence processing. This makes the architecture particularly interesting for data science applications involving large sequential datasets.
Mamba-2 and State Space Duality
Mamba-2 takes the concept further by connecting state space models and attention through what researchers describe as state space duality.
The ICML 2024 research showed that Transformers and certain state space models have deeper mathematical relationships than might initially appear. The authors used this framework to develop Mamba-2, whose core layer was reported to be between two and eight times faster than the original Mamba selective state-space layer while remaining competitive with Transformers on language modelling tasks.
This is particularly important because it challenges the idea that attention and recurrent or state-space processing are completely separate architectural philosophies.
Instead, research increasingly explores combinations and mathematical connections between them. The practical consequence is that future AI architectures may not be defined simply as either “Transformer” or “non-Transformer”. Hybrid approaches can combine attention with state-space processing, recurrent memory or other sequence-processing mechanisms.
For data scientists, this means model selection is likely to become more task-specific. A Transformer may remain appropriate when precise token-to-token relationships are critical, while a state space architecture may become attractive when processing extremely long sequences under constrained computational budgets.
xLSTM: Bringing Recurrent Networks Back
Long before Transformers became dominant, recurrent neural networks and Long Short-Term Memory networks were widely used for sequence modelling.
LSTMs introduced mechanisms for retaining important information over time while controlling what should be forgotten. Their sequential nature, however, made them difficult to scale efficiently for large modern workloads, and Transformers subsequently became the dominant approach.
The xLSTM architecture revisits recurrent processing using an expanded LSTM design intended to address some of the limitations of traditional recurrent neural networks.
The architecture has attracted attention because it offers a different route towards efficient sequence modelling. The 2025 xLSTM 7B research introduced a seven-billion-parameter recurrent language model and evaluated its inference characteristics against similarly sized alternatives. The researchers reported comparable downstream performance alongside faster inference and greater efficiency than the Llama- and Mamba-based models used in their evaluation.
xLSTM is particularly interesting because it demonstrates that recurrent architectures have not necessarily reached the end of their development path.
Modern hardware and software optimisation can change the practical economics of architectures that were previously considered inefficient. For applications requiring substantial inference workloads, constant-memory behaviour and linear scaling with sequence length could become significant advantages.
Potential applications include language modelling, edge AI, streaming data and systems where continuous sequential processing is more important than maintaining access to every previous token.
RWKV and Linear Recurrent Processing
RWKV is another architecture exploring the space between Transformers and recurrent neural networks.
The name refers to Receptance Weighted Key Value, reflecting mechanisms that retain some conceptual similarities to attention while using recurrent processing. This allows RWKV to be trained in parallel while supporting recurrent-style inference.
The architecture is particularly relevant to long-sequence applications because its recurrent formulation can reduce the computational requirements associated with conventional self-attention.
The architectural philosophy behind RWKV illustrates an important direction in AI research: rather than completely abandoning ideas from Transformers, alternative models often preserve useful concepts while changing how information flows through the network.
This makes RWKV relevant to applications where inference efficiency, memory consumption and sequential processing are priorities. It can also be attractive for experimentation because researchers can investigate Transformer-like representations without relying on conventional quadratic attention.
However, performance comparisons need to be interpreted carefully. Results vary according to model size, dataset, training budget, sequence length and benchmark. A model that performs particularly well on long-context language modelling may not necessarily outperform a Transformer across general-purpose tasks.
Hyena and Long Convolutional Architectures
Another important direction involves replacing attention with long convolutions.
The Hyena architecture was proposed as a subquadratic alternative to attention using implicitly parameterised long convolutions combined with data-controlled gating. The research targeted one of the central limitations of Transformers: the increasing cost of processing long sequences.
Hyena is particularly interesting because it approaches sequence modelling from a signal-processing perspective.
Convolutions have long been important in computer vision and signal processing, but traditional convolutional operations are not naturally suited to extremely long-range relationships. Hyena attempts to address this limitation using long filters and gating mechanisms.
The research reported strong results on long-sequence language tasks and demonstrated significant efficiency improvements at longer context lengths. These findings helped establish long convolution as another potential direction for deep learning architectures beyond Transformers.
For data science, Hyena-like architectures could have applications beyond language. Long sequential datasets appear in genomics, financial time series, sensor monitoring, audio processing and scientific computing.
The major attraction is the possibility of processing very long sequences without paying the full computational cost associated with standard attention.
StripedHyena and Hybrid Sequence Models
StripedHyena builds on the Hyena research programme by combining attention and gated convolutions.
This hybrid approach is important because it reflects the direction in which much current architecture research is moving. Rather than trying to identify a single operator that completely replaces attention, researchers are experimenting with combinations that assign different computational responsibilities to different mechanisms.
Together AI reported that StripedHyena-7B achieved competitive results with similarly sized open-source Transformer models while offering advantages in long-context training and inference. Its published evaluations reported progressively larger training speed improvements as sequence length increased, alongside smaller autoregressive caches than comparable Transformer configurations.
StripedHyena has also demonstrated potential outside conventional language modelling. The Evo biological foundation model used a StripedHyena-based architecture to model DNA, RNA and proteins at extremely long sequence lengths. Evo was designed to operate at nucleotide-level resolution across biological sequences extending to hundreds of thousands of tokens.
This is a useful example of why architecture diversity matters to data science. An architecture that is interesting because of its language modelling performance can become even more valuable when it addresses a completely different data problem.
Griffin and Recurrent Hybrid Architectures
Google’s Griffin architecture provides another example of combining recurrent processing with attention.
Griffin combines gated linear recurrences with local sliding-window attention. Google subsequently used the architecture in RecurrentGemma, demonstrating how recurrent mechanisms can be integrated into modern generative AI systems.
The architecture reflects an important compromise. Local attention can provide detailed interactions within a recent portion of the sequence, while recurrent processing maintains information across longer spans.
The approach can therefore reduce some of the computational requirements associated with global attention.
However, this type of architecture also illustrates the trade-offs involved. A fixed-size recurrent state can reduce performance on certain retrieval tasks compared with architectures capable of directly accessing the full context.
This is a critical consideration when comparing Transformers with alternative architectures. Efficiency improvements often involve some form of information compression, and compression can introduce weaknesses in exact retrieval or long-range reasoning.
Titans and Neural Long-Term Memory
One of the more recent developments is the Titans architecture, presented at NeurIPS 2025.
Titans explores the concept of neural long-term memory that can learn to memorise information during inference. Rather than relying exclusively on a fixed context window, Titans introduces a memory module designed to retain historical information and work alongside attention.
The architecture is based on the idea that attention can act as a form of short-term memory while a learned neural memory provides longer-term storage.
This distinction addresses a major limitation of both traditional recurrent models and standard attention. Recurrent models can process long sequences efficiently but compress information into a limited state. Attention can access a much larger context directly but becomes expensive as the context grows.
Titans attempts to combine these approaches.
The researchers reported experiments in language modelling, common-sense reasoning and time-series tasks, while also demonstrating performance on needle-in-a-haystack evaluations with context windows exceeding two million tokens.
Because Titans is still an emerging research architecture, it should not be treated as a mature replacement for Transformers. Its importance lies primarily in the new design space it opens around test-time learning and adaptive memory.
For data scientists, this could eventually have applications in long-running agents, personalisation, scientific analysis, time-series modelling and systems that need to maintain information over extended interactions.
MambaVision and Beyond-Transformer Computer Vision
Transformer alternatives are also becoming increasingly relevant to computer vision.
MambaVision, presented at CVPR 2025, combines Mamba-style state space processing with Transformer components in a hybrid vision backbone. The architecture was specifically designed to address visual feature modelling while retaining the efficiency characteristics associated with state space models.
This is significant because vision Transformers have become widely used in image classification, object detection and other computer vision tasks.
However, visual data can contain substantial spatial information, and researchers are investigating whether attention is always the most efficient mechanism for modelling those relationships.
Hybrid architectures such as MambaVision provide a potential middle ground. Instead of assuming that one architectural mechanism should handle every visual relationship, different components can specialise in local, global and sequential information.
This could become particularly valuable in video understanding, medical imaging, autonomous systems and other workloads where the amount of visual information can become extremely large.
Comparing Emerging Architectures with Transformers
The most important difference between Transformers and many emerging architectures is how information is represented and retrieved.
Transformers use attention to create direct relationships between elements of the input. This provides powerful content-based retrieval but creates substantial computational demands as sequence length increases.
State space models such as Mamba instead maintain an evolving state. Their computational structure can scale more efficiently with sequence length, but information must be compressed into that state.
Recurrent architectures such as xLSTM and RWKV similarly emphasise sequential state management. Their potential advantages include lower memory requirements and efficient inference, but their compressed representations can make certain retrieval problems more difficult.
Hyena and StripedHyena use long convolutions and gating mechanisms to capture relationships over large sequences. These architectures can offer significant efficiency improvements on long-context workloads, although their performance depends strongly on the task and implementation.
Titans introduces another possibility by adding a learned long-term memory mechanism. Rather than choosing between full attention and fixed recurrent state, it attempts to create an adaptive memory system.
The comparison therefore involves more than simply asking which architecture produces the highest benchmark score.
Data scientists need to consider training cost, inference latency, memory consumption, sequence length, hardware compatibility, model quality, implementation maturity and the characteristics of the underlying dataset.
Performance: Why Benchmark Results Need Context
Comparing emerging deep learning architectures is particularly difficult because benchmark results can vary considerably.
A model may outperform a Transformer on long-context inference but perform less strongly on short-context reasoning. Another may require fewer computational resources but achieve lower accuracy. A third may provide excellent results for genomics but have limited evidence in general-purpose language modelling.
This means statements such as “Mamba is faster than Transformers” require qualification. Mamba research reported major efficiency improvements under particular configurations, while other studies have found task-specific trade-offs. Similarly, xLSTM 7B reported faster inference against selected Llama- and Mamba-based models, but this does not establish universal superiority across all model families and workloads.
The same principle applies to Titans, Hyena and RWKV.
For data science teams, the appropriate evaluation process should therefore reproduce representative workloads rather than relying entirely on published headline figures.
Sequence length, batch size, GPU type, quantisation, model size and implementation quality can all influence performance.
Potential Applications Beyond Language Models
One of the strongest reasons to investigate alternative architectures is that many real-world datasets are sequential without being linguistic.
Financial time series, sensor data, weather measurements, medical signals, genomic sequences and industrial telemetry can contain extremely long sequences.
A Transformer may be effective for some of these workloads, but its computational characteristics can become challenging as sequence lengths grow.
State space models are particularly interesting for time-series forecasting because their recurrent state can naturally represent evolving temporal information. Research has identified applications for SSMs across time-series analysis, recommendation systems, genomics, medical applications, speech, audio and video.
Genomics is another particularly promising area. DNA sequences can contain millions of positions, making efficient long-range modelling extremely important. Models such as Evo demonstrate how non-Transformer architectures can be applied to biological foundation models operating at genomic scale.
Audio and speech represent another opportunity. Audio streams are naturally sequential, and applications such as transcription, classification, generation and real-time processing can benefit from architectures capable of handling long sequences efficiently.
The Importance of Hybrid Architectures
The emerging architecture landscape suggests that the future may not be about choosing between Transformers and alternatives.
Instead, hybrid models may become increasingly important.
MambaVision already demonstrates this approach in computer vision. Griffin combines recurrent processing and local attention. StripedHyena combines attention with gated convolutions. Titans combines neural memory with attention.
These architectures recognise that different mechanisms are good at different things.
Attention is highly effective for direct content-based retrieval. Recurrence provides efficient sequential processing. State space models provide compact representations of long histories. Convolutions can efficiently capture structured local and long-range patterns. External or learned memory can provide persistent information storage.
Combining these capabilities can therefore produce architectures that are more flexible than attempting to replace attention completely.
This also changes the role of the data scientist. Rather than treating architecture as a fixed choice determined by the latest popular model, practitioners increasingly need to understand the characteristics of their data and select architectures accordingly.
What Data Scientists Should Learn in 2026
For data scientists, emerging deep learning architectures represent an opportunity to develop skills beyond conventional Transformer-based AI development.
Understanding attention remains essential because Transformers continue to dominate many production AI systems. However, knowledge of state space models, recurrent neural networks, sequence modelling, convolutional architectures and efficient inference is becoming increasingly valuable.
Python remains central to experimentation, while frameworks such as PyTorch and JAX provide the foundations for implementing and evaluating emerging architectures.
Data scientists should also develop stronger skills in model benchmarking. Comparing architectures requires more than measuring accuracy. Memory consumption, inference throughput, latency, training cost and energy consumption can be equally important.
Long-context evaluation is particularly valuable because it exposes architectural differences that may not appear in conventional benchmarks.
The ability to reproduce research papers, modify architectures and conduct controlled experiments can therefore become an important skill for professionals working in advanced AI research.
Recommended Online Courses to Build Deep Learning Skills in 2026
As alternative deep learning architectures continue to emerge, developing a strong foundation in neural networks, PyTorch, recurrent models, model optimisation and Transformer mechanisms can help data scientists understand how newer approaches such as Mamba, xLSTM, RWKV and Hyena differ from established architectures. The following courses provide practical and highly rated routes into those underlying skills.
Deep Learning A-Z [2026]: DL, AI in Python & AWS + LLM Prize — Udemy
Platform: Udemy
Level: Beginner to Intermediate
Focus: Deep learning, neural networks, computer vision, time series, Python and model development
This bestseller provides broad practical coverage of deep learning using Python, including artificial neural networks, supervised and unsupervised deep learning, computer vision, time-series analysis, fraud detection and recommender systems. Its coverage of different neural network applications makes it a useful foundation for understanding why alternative architectures may be appropriate for different datasets and workloads.
Course Link: Deep Learning A-Z [2026] — Udemy
A Deep Understanding of Deep Learning — Udemy
Platform: Udemy
Level: Beginner
Focus: Deep learning theory, neural network architectures, PyTorch, optimisation and model performance
This bestseller is particularly useful for learners who want a deeper theoretical and practical understanding of neural network architecture. It covers the mathematical foundations of deep learning, feedforward and convolutional networks, PyTorch, gradient descent, fine-tuning, autoencoders, transfer learning and model optimisation. These foundations provide useful preparation for understanding why newer architectures modify or replace conventional components of neural networks.
Course Link: A Deep Understanding of Deep Learning — Udemy
Deep Learning Specialization — Coursera
Platform: Coursera / DeepLearning.AI
Level: Intermediate
Focus: Neural networks, CNNs, RNNs, sequence models, optimisation and deep learning applications
The Deep Learning Specialization provides a broader academic and practical foundation covering neural networks, convolutional networks, recurrent neural networks, optimisation, model evaluation and modern AI applications. Its coverage of recurrent networks is particularly relevant to an article examining architectures such as xLSTM and RWKV, while its broader treatment of deep learning provides useful context for comparing alternative architectures with Transformers.
Course Link: Deep Learning Specialization — Coursera
The Future of Deep Learning Architecture
The rapid development of Mamba, xLSTM, RWKV, Hyena, StripedHyena, Griffin, MambaVision and Titans suggests that deep learning architecture research is entering a period of greater diversity.
Transformers remain extremely powerful, and the emergence of alternatives does not indicate that attention-based models are disappearing. Instead, researchers are investigating how different computational mechanisms can solve specific weaknesses associated with scaling, long contexts, memory and inference.
State space models currently represent one of the most mature alternatives, with research demonstrating competitive performance and efficiency across several domains. Mamba-2’s connection between state space models and attention also suggests that the boundary between these categories may become increasingly blurred.
Other approaches remain earlier in their development. Titans introduces an intriguing concept of test-time memory, while newer recurrent and convolutional architectures continue to explore alternative methods of representing long-range dependencies.
The likely outcome is not a single architecture replacing Transformers. Instead, AI systems may increasingly use different architectures for different workloads.
Final Thoughts
Emerging deep learning architectures beyond Transformers are expanding the possibilities for efficient AI and sequence modelling. Mamba and Mamba-2 demonstrate the potential of state space models, xLSTM and RWKV revisit recurrent processing, Hyena and StripedHyena explore long convolutions, Griffin and MambaVision demonstrate hybrid approaches, while Titans introduces neural long-term memory. Research across these architectures shows that there are multiple ways to process sequential information without relying exclusively on conventional self-attention.
For data scientists, the most useful lesson is that architecture selection should increasingly be driven by the characteristics of the problem rather than by popularity alone. Transformers remain highly capable, but alternative architectures can provide advantages for long-context processing, streaming data, genomics, time series, audio, vision and resource-constrained inference. As AI research continues to explore combinations of attention, recurrence, state-space processing, convolution and learned memory, understanding these emerging architectures will become an increasingly valuable skill for professionals working at the leading edge of data science and artificial intelligence.
