Transformer Architectures
Encoder-Only (BERT-style)
Attention: Bidirectional (sees all tokens) Use cases: Classification, NER, embeddings
Examples: BERT, RoBERTa, ALBERT
Decoder-Only (GPT-style)
Attention: Causal/unidirectional (only sees past) Use cases: Text generation, chat, reasoning
Examples: GPT-4, Claude, LLaMA, Gemini
Encoder-Decoder (T5-style)
Encoder: Processes input bidirectionally Decoder: Generates output causally Use cases: Translation, summarization
Examples: T5, BART, mT5
Comparison
Architecture | Bidirectional | Generation | Best For |
|--------------|---------------|------------|----------|
Encoder-only | ✓ | ✗ | Understanding |
Decoder-only | ✗ | ✓ | Generation |
Enc-Dec | ✓ (enc) | ✓ (dec) | Seq2Seq |
Modern Trend
Decoder-only models dominate: