Google group for participants and prospective participants

Participants and prospective participants in task are invited to join the DDI Extraction Google Group:

http://groups.google.com/group/DDIExtraction2011


Evaluation Methodology

Each team participating in this task will initially have access only to the training data. Later, the teams will have access to unlabeled testing data (that is, there will be drug labels, but no DDI labels). The teams will enter their algorithms' guesses for the labels for the testing data. When DDIExtraction2011 is over, the labels for the testing data will be released to the public.

Algorithms are not allowed to skip examples, that is, they must always provide a prediction for each pair of drugs.

An algorithm's score for a given relation will be the F score, the harmonic mean of precision and recall. Algorithms will be ranked according to their F scores.

Participants are allowed to submit a maximum of 5 runs. Each run can include different sources of information and use different techniques. For example, some algorithms may not need the information provided by the MetaMap tool.

For each run, a submission file (run) must be an txt file that includes all pairs of drugs and its prediction. Algorithms identify all possible DDI candidate pairs appearing in a single sentence of text (it should be remembered that only DDIs occurring within the text unit of a sentence are to be considered for this task). When the DDI candidate pair is predicted to be a non-authentic DDI, participant algorithms should generate the label "0". Conversely, when the DDI candidate pair is predicted to be an authentic DDI, participant algorithms should generate the label "1".

A more detailed description of the format can be found here

Please name your submission file with your team name followed by the number of the run. For example, the UC3M team may submit the following files: UC3M_1.txt, UC3M_2.TXT, UC3M_3.TXT, UC3M_4.TXT and UC3M_5.TXT.

Download

An Example of a test dataset in Unified format can be found here.

An Example of a test dataset in MMTx format can be found here.

An example of a submission file for the test dataset in Unified format can be found here.

An example of a submission file for the test dataset in MMTx format can be found here.


Universidad Carlos III de Madrid - Computer Science Department - LABDA http://labda.inf.uc3m.es

Valid XHTML 1.0 Strict - ¡CSS Válido! - Icono de conformidad con el Nivel Triple-A, 
				de las Directrices de Accesibilidad para el 
				Contenido Web 1.0 del W3C-WAI