AhaBulbStuff Everyone Should Know
AhaBulb: Stuff Everyone Should Know · ahabulb.com/ai-gsm1k-fresh-test

Researchers wrote GSM1k, fresh math problems like GSM8K. What happened to some models' scores?

Start playing free

  1. They dropped, by up to 8 percent
  2. They rose sharply
  3. They stayed exactly the same for all

🔒 The answer is for AhaBulb members. Members see the answer, why it's right and the official sources for every card of Stuff Every AI Expert Should Know, plus timed practice exams.

The first 50 cards of every test are free to play.

Play the free cards See membership

Like this card?
Play the game

References

  1. Zhang et al., A Careful Examination of Large Language Model Performance on Grade School Arithmetic arxiv.org/abs/2405.00332 Checked 2026-10-11.
  2. Zhang et al., A Careful Examination of Large Language Model Performance on Grade School Arithmetic arxiv.org/abs/2405.00332 Checked 2026-10-11.

Think this answer is wrong? Tap "Challenge this answer" on the card in the game and tell us why.

Test yourself in the game

Artificial IntelligenceWhat is benchmark contamination for a language model?Artificial IntelligenceThe MMLU benchmark tests language models across how many subjects, from math to law?Artificial IntelligenceWhich data split is saved for the final check after tuning on the validation set?

That's stuff everyone should know.