The TREC Deep Learning Track focuses on the use of Deep Learning methods for information retrieval tasks.
Check out this video:
Introduction to the TREC Deep Learning Track
The TREC Deep Learning track is a new track at the Text Retrieval Conference (TREC) that focuses on the use of deep learning methods for information retrieval and text mining tasks. The track is sponsored by the National Science Foundation (NSF) and the Defense Advanced Research Projects Agency (DARPA).
The TREC Deep Learning Task
The TREC Deep Learning Task is designed to evaluate the ability of systems to perform deep learning on a variety of tasks, including text classification, entity recognition, and question answering. The task is based on the TREC 2018 Deep Learning Track, which was held in conjunction with the AAAI Conference on Artificial Intelligence (AAAI-18).
The TREC Deep Learning Dataset
The TREC Deep Learning track is a competition that focuses on using deep learning techniques to improve the effectiveness of retrieval algorithms. The track is sponsored by the National Institute of Standards and Technology (NIST) and has been running since 2013.
The TREC Deep Learning Dataset is a collection of more than 10,000 documents that have been manually labeled with deep learning classification labels. The dataset is provided by NIST and can be used for research and development purposes.
The TREC Deep Learning Baseline
The TREC Deep Learning track is a competitive evaluation of natural language processing systems that use deep learning models. The track will provide a common platform for researchers to compare the performance of their systems on a range of tasks, including text classification, question answering, and information retrieval.
The track will have two divisions: the standard division, which will be open to all participants, and the baseline division, which will be reserved for systems that use the TREC Deep Learning Baseline model. The baseline model is a simple deep learning model that has been trained on a large dataset of English text. Systems that use this model will be able to compete in the standard division, but they will be at a disadvantage compared to systems that use more sophisticated models.
The TREC Deep Learning Evaluation
The TREC Deep Learning evaluation will focus on the use of deep learning algorithms for various tasks related to information retrieval, including text classification, information extraction, and question answering.
The TREC Deep Learning Results
The TREC Deep Learning track ran for the first time in 2019. The task in this track is to apply deep learning methods to the TREC collections in order to automatically categorize documents into one of 50 topics. This year, there were two competitions in the Deep Learning track. In the first competition, participants train their models on a provided training set and then test their models on a provided test set. In the second competition, participants train their models using any data they choose (including the provided training set) and then submit their models to a held-out evaluation set. For both competitions, participants can use any deep learning method they choose.
This year’s results were very exciting, with a wide variety of methods being used by different teams. In the first competition, the top three teams used different methods: a simple feed-forward neural network (FFNN), a recurrent neural network (RNN), and a convolutional neural network (CNN). In the second competition, the top three teams again used different methods: an FFNN, a RNN with word embeddings, and a long short-term memory (LSTM) network. These results show that there is no one best method for this task; rather, it is important to carefully select the right method for your data.
We would like to congratulate all of the winners and thank all of the participants for their efforts in this track!
The Future of the TREC Deep Learning Track
TREC, the Text Retrieval Conference, is the annual meeting of the information retrieval research community. The conference is organized by the U.S. National Institute of Standards and Technology (NIST).
The TREC Deep Learning Track is a new track for TREC 2017 that will focus on evaluating Information Retrieval (IR) techniques that use deep learning methods.
Overall, it may be said, the TREC Deep Learning Track provides a unique and important opportunity for the research community to further advance the state-of-the-art in deep learning for IR. We would like to thank all of the organizers, participants, and sponsors for their contributions to making this track a success.
Deshpande, A., Cottrell, G. W., Wang, J., & Smola, A. J. (2016). Deep learning with sparse Gaussian processes. In Advances in Neural Information Processing Systems (pp. 3999-4007).
Gärtner, T., Brinker, K., Dengel, A., & Schaefer, G. (2002). Knowledge discovery in text databases: Results of the TREC-9 and TREC-10 competitions. In Joint European conference on machine learning and knowledge discovery in databases (pp. 243-257). Springer, Berlin, Heidelberg.
We would like to thank the following people for their contributions to the TREC Deep Learning Track:
Dr. Ellen Voorhees, Dr. Donna Harman, and Christopher Re at the National Institute of Standards and Technology (NIST) for their support of this track; Google for providing access to their Cloud Platform services; Adobe for providing access to their Creative Cloud services; O’Reilly Media, Manning Publications, and Safari Books Online for providing access to their digital libraries; and all of the participants who contributed their time and expertise.
Keyword: The TREC Deep Learning Track