AhaBulbStuff Everyone Should Know
AhaBulb: Stuff Everyone Should Know · ahabulb.com/ai-bert-bidirectional

How did BERT's pretraining differ from a left-to-right GPT-style model?

Start playing free

  1. BERT reads text right to left only
  2. BERT uses context from both left and right
  3. BERT has no Transformer layers

🔒 The answer is for AhaBulb members. Members see the answer, why it's right and the official sources for every card of Stuff Every AI Expert Should Know, plus timed practice exams.

The first 50 cards of every test are free to play.

Play the free cards See membership

Like this card?
Play the game

References

  1. Devlin et al., BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (arXiv 1810.04805) arxiv.org/abs/1810.04805 Checked 2026-10-10.

Think this answer is wrong? Tap "Challenge this answer" on the card in the game and tell us why.

Test yourself in the game

Artificial IntelligenceWhich 2017 paper introduced the Transformer architecture?Artificial IntelligenceWhich two sequence mechanisms did the original Transformer dispense with?Artificial IntelligenceWhat does transfer learning reuse from a previously trained model?

That's stuff everyone should know.