Machine Reading Comprehension Tasks, Metrics, and Datasets

build

MRC Tasks

In order to have a better understanding of MRC tasks, we analyzed the existing classification method of tasks and pointed out the limitations of this method. After analyzing 57 MRC tasks, we propose a more precise classification method of MRC tasks which has 4 different attributes of MRC task and each of them could be divided into several types, and the statistical data of 57 MRC tasks are also given in this section.

中国科学院

cloud_queue

MRC Datasets

The recent success of neural reading comprehension is driven by both large-scale datasets and neural models. In this section, we analyzed the attributes of different machine reading comprehension benchmark datasets, including: the size of the dataset, the generation method of datasets, the source of corpus, the type of context, the availability of leaderboards and baselines, reasoning skills and citations of related papers.

中国科学院大学

check_box

MRC Metrics

In this section, we introduce the computation methods of MRC evaluation metrics. Typical evaluation metrics of MRC datasets are: Accuracy, Exact Match, F1 score, ROUGE, BLEU, HEQ and Meteor. Many datasets use more than one evaluation metric. Moreover, some datasets adopt detailed evaluation metrics according to their own characteristics,such as Accuracy on Named Entities, F1 of Supportings, etc.

中国科学院成都计算机应用研究所