Metrics, Leaderboards and Baselines

Year Tasks Metric 1 Metric 2 Metric 3 Metric 4 Leaderboard Baseline
2013 MCTest Accuracy N/A N/A N/A
2015 CNN/Daily Mail Accuracy N/A N/A N/A N/A
2015 CuratedTREC Exact Match N/A N/A N/A
2015 WikiQA Precision Recall F1 N/A N/A
2016 WikiMovies Accuracy N/A N/A N/A N/A
2016 SQuAD1.1 F1 Exact Match N/A N/A
2016 Who-did-What Accuracy N/A N/A N/A
2016 MS MARCO Rouge-L BLEU-1 N/A N/A
2016 NewsQA Exact Match F1 N/A N/A N/A N/A
2016 LAMBADA Accuracy N/A N/A N/A N/A
2016 WikiReading Mean F1 N/A N/A N/A N/A
2016 Facebook CBT Accuracy on Named Entities Accuracy on Common Nouns Accuracy on Verbs Accuracy on Prepositions N/A
2016 BookTest Accuracy N/A N/A N/A N/A N/A
2016 Google MC-AFP Accuracy N/A N/A N/A N/A N/A
2016 MovieQA Accuracy of Video Clips Accuracy of Plots and Subtitles N/A
2017 TriviaQA(Web) Exact Match F1 Verified-EM Verified-F1
2017 TriviaQA (Wiki) Exact Match F1 Verified-EM Verified-F1
2017 RACE Accuracy N/A N/A N/A
2017 Quasar-S Accuracy N/A N/A N/A N/A
2017 Quasar-T Exact Match F1 N/A N/A N/A
2017 SearchQA F1 score (for n-gram)(Single word or Multi-word) Unigram(Single word) Accuracy N/A N/A N/A N/A
2017 NarrativeQA ROUGE-L BLEU-1 BLEU-4 Meteor N/A
2017 SciQ Accuracy N/A N/A N/A N/A N/A
2017 Qangaroo(MedHop) Accuracy N/A N/A N/A N/A
2017 Qangaroo(WikiHop) Accuracy N/A N/A N/A N/A
2017 TQA Accuracy of All Accuracy of Diagram N/A N/A N/A
2017 COMICS Accuracy of Text Cloze Accuracy of Visual Cloze Accuracy of Coherence N/A N/A
2018 QuAC F1 HEQ-Q HEQ-D N/A
2018 CoQA F1 overall F1 out-of-domain overall F1 in-domain overall N/A
2018 SQuAD2.0 F1 EM N/A N/A
2018 HotpotQA(Distractor) EM of Answer F1 of Answer EM of Supportings F1 of Supportings
2018 HotpotQA(Fullwiki) EM of Answer F1 of Answer EM of Supportings F1 of Supportings
2018 DuoRC-Self Accuracy F1 N/A N/A
2018 DuoRC-Paraphrase Accuracy F1 N/A N/A
2018 CLOTH Accuracy N/A N/A N/A
2018 ReCoRD Exact Match F1 N/A N/A N/A
2018 CliCR Exact Match F1 BLEU-2 BLEU-4 N/A
2018 ReviewQA Accuracy N/A N/A N/A N/A N/A
2018 ARC-Challenge Set Accuracy N/A N/A N/A
2018 ARC-Easy Set Accuracy N/A N/A N/A
2018 OpenBookQA Accuracy N/A N/A N/A N/A
2018 SciTail Accuracy N/A N/A N/A N/A
2018 MultiRC F1m Exact Match N/A N/A
2018 RecipeQA Accuracy of Textual Cloze Accuracy of Visual Cloze Accuracy of Visual Ordering Accuracy of Visual Coherence N/A
2018 PaperQA-T(Park etc 2018) Accuracy N/A N/A N/A N/A N/A
2018 PaperQA-L(Park etc 2018) Accuracy N/A N/A N/A N/A N/A
2018 PaperQA(Hong etc 2018) F1 N/A N/A N/A N/A N/A
2018 MCScript Accuracy N/A N/A N/A N/A
2018 ProPara Accuracy N/A N/A N/A N/A
2019 Natrual Question(Short) Precision Recall F1 N/A
2019 Natrual Question(Long) Precision Recall N/A N/A
2019 DREAM Accuracy N/A N/A N/A
2019 ShARC Micro Accuracy Macro Accuracy BLEU-1 BLEU-4 N/A
2019 CommonSenseQA Accuracy N/A N/A N/A
2019 DROP EM F1 N/A N/A