Benchmarking biomedical text mining web servers at BioCreative V.5: the technical Interoperability and Performance of annotation Servers - TIPS track

摘要

The TIPS track consisted in a novel experimental task under the umbrellaof the BioCreative text mining challenges with the aim to, for the first timeever, carry out a text mining challenge with particular focus on the continuousassessment of technical aspects of text annotation web servers, specifically ofbiomedical online named entity recognition systems.A total of 13 teams registered annotation servers, implemented in various programminglanguages, supporting up to 12 different general annotation types. Thecontinuous evaluation period took place from February to March 2017. The systematicand continuous evaluation of server responses accounted for testing periodsof low activity and moderate to high activity. Moreover three documentprovider settings were covered, including also NCBI PubMed. For a total of4,092,502 requests, the median response time for most servers was below 3.74 swith a median of 10 annotations/document. Most of the servers showed greatreliability and stability, being able to process 100,000 requests in 5 days.