Submission Tutorial

This tutorial will walk you through the whole submission process. By following it step by step, you will get your results on the hidden test set, and, if you wish, have your results listed on the public leaderboard.

Note: If the CodaLab worksheet cannot work well or appears to be dead, you can directly email your code and checkpoints to me (cst_hanxu13@163.com).

Evaluating on the dev set

Step 0: Create a CodaLab account

1. Sign up on the website of CodaLab and complete your own information.

2. If you already have an account, please sign in.

   

Step 1: Create a CodaLab worksheet

1. Click My Dashboard.

   

2. Click New Worksheet to create your own worksheet.

   

   

Step 2: Add the dev set and scripts

1. In the web interface terminal on the top of the page, type in the following command:

   Add Train/val data:

   cl add bundle 0x22dc32baa68f43a1ace0dd3d5921386a .

   Add a script to sample N-way K-shot dataset from a data division:

   cl add bundle 0x2a02b4a8119b4220ad37520d7ee3b5e8 .

   Add the code for evaluating final results:

   cl add bundle 0x1f88373e182e42f5a7a91d3ed0d02124 .

   

   

Step 3: Upload your own models and parameters

1. Click the Upload button in the top-right corner. to upload your own models and parameters. Then, your models and parameters will appear in the contents of the worksheet as several bundles, e.g., we upload our own demo models and parameters as follows,

   

Step 4: Run evaluation script on the dev set

1. Write and upload the evaluation script, e.g., we upload our demo evaluation script as follows,

   

   The code of the script is

   python sample_io.py data/val.json 100 5 5 12345 input > input.json
   python sample_io.py data/val.json 100 5 5 12345 output > output.json
   python test_demo.py input.json > predict.json
   python evaluate.py predict.json output.json

   Note that: More details about data, evaluation scripts and demo examples can be found in our worksheet.

2. In the web interface terminal on the top of the page, type in the following command:

   Sample instances from the dev set and evaluate the models:

   cl run 'data:0x22dc32' 'sample_io.py:0x2a02b4' 'evaluate.py:0x1f8837' 'checkpoint:0x9f3d8f' 'fewshot_re_kit:0x70219b' '_processed_data:0xb636c7' 'models:0xbe1716' 'test_demo.py:0x8fa335' 'evaluate.sh:0xda5e83' 'bash evaluate.sh'  --request-docker-image codalab/default-cpu

   Note that: for each file, its uuid should be detailed, e.g., 'data:0x22dc32'.

3. Get the results:

   

Submitting a test set evaluation request

To preserve the integrity of test results, we do not release the test set to the public. Instead, we require you to upload your model onto CodaLab so that we can run it on the test set for you. Before evaluating your models on the test set, make sure you have completed evaluation of development set as detailed above. Thus we will assume you are already familiar with uploading files and running commands on CodaLab. Note that, we request you to use our provided data sampling file sample_io.py (FewRel 1.0 and FewRel 2.0 DA) / sample_io_nota.py (FewRel 2.0 NOTA) and evaluation file evaluate.py in this worksheet.

Step 0: Upload your own models and make sure they can run on the dev set

Step 1: Submit an evaluation request

1. You have tested your scripts on the dev set.
2. All your codes / checkpoints are set to appropriate permission so we can access.
3. You use the following files:
   1. sample_io.py:0x2a02b4 or sample_io_nota.py:0x0e4467
   2. evaluate.py:0x1f8837
   3. data:0xbac6f7
   Note that data:0xbac6f7 is private so only we have the access.
4. Data sampling requirements:
   1. For FewRel 1.0: use "python sample_io.py data/test_wiki.json 10000 N K 12345 input" to sample data, where N K = 5 1 / 5 5 / 10 1 / 10 5
   2. For FewRel 2.0 DA: use "python sample_io.py data/test_pubmed.json 10000 N K 12345 input" to sample data, where N K = 5 1 / 5 5 / 10 1 / 10 5
   3. For FewRel 2.0 NOTA: use "python sample_io_nota.py data/test_wiki.json 10000 N K NOTA 12345 input" to sample data, where N K NOTA = 5 1 0.15 / 5 5 0.15 / 5 1 0.5 / 5 5 0.5

cl run 'data:0xbac6f7' 'sample_io.py:0x2a02b4' 'evaluate.py:0x1f8837' 'checkpoint:0x9f3d8f' 'fewshot_re_kit:0x70219b' '_processed_data:0xb636c7'  'models:0xbe1716' 'test_demo.py:0x8fa335' 'evaluate.sh:0xda5e83' 'bash evaluate.sh' --request-docker-image codalab/default-gpu --request-gpus 1 --request-disk 2g --request-time 2d

Should you have any questions, please send an email to fewrel@googlegroups.com.