カンファレンス (国際) Overview of the NTCIR-13 Short Text Conversation Task

Lifeng Shang(Huawei),Tetsuya Sakai(Waseda University),Hang Li(Toutiao),Ryuichiro Higashinaka(NTT Media Intelligence Laboratories),Yusuke Miyao(National Institute of Informatics),Yuki Arase(Osaka University),and Masako Nomoto

The 13th NTCIR Conference (NTCIR-13)


We give an overview of the NII Testbeds and Community for Information access Research (NTCIR)-13 Short Text Conversation (STC) task, which was a core task of NTCIR-13. At NTCIR-12, STC was taken as an IR problem by maintaining a large repository of post-comment pairs then finding a clever method of reusing these existing comments to respond to new posts. At NTCIR-13, besides the retrieval-based method, we focused on a new method called generation-based method to generate ``new'' comments. The generation-based method has gained a great deal of attention in recent years, even though there the problem still remains of whether the retrieval-based method should be wholly replaced with or combined with the generation-based method for the STC task. By organizing this task at NTCIR-13, we provided a transparent platform to compare the two aforementioned methods by conducting comprehensive evaluations. For the Chinese subtask, there were a total of 34 registrations, and 22 teams finally submitted 120 runs. For the Japanese subtask, there were a total of 9 registrations, and 5 teams submitted 15 runs. In this paper, we review the task definition, evaluation measures, test collections, and evaluation results of all teams.

Paper : Overview of the NTCIR-13 Short Text Conversation Task (外部サイト)