Submission Tutorial

This tutorial will walk you through the whole submission process. By following it step by step, you will get your results on the hidden test set, and, if you wish, have your results listed on the public leaderboard.

Note: If the CodaLab worksheet cannot work well or appears to be dead, you can directly email your code and checkpoints to me (

Evaluating on the dev set

Step 0: Create a CodaLab account

  1. Sign up on the website of CodaLab and complete your own information.

  2. If you already have an account, please sign in.

Step 1: Create a CodaLab worksheet

  1. Click My Dashboard.

  2. Click New Worksheet to create your own worksheet.

Step 2: Add the dev set and scripts

  1. In the web interface terminal on the top of the page, type in the following command:

    Add Train/val data:

    cl add bundle 0x22dc32baa68f43a1ace0dd3d5921386a .

    Add a script to sample N-way K-shot dataset from a data division:

    cl add bundle 0x2a02b4a8119b4220ad37520d7ee3b5e8 .

    Add the code for evaluating final results:

    cl add bundle 0x1f88373e182e42f5a7a91d3ed0d02124 .

Step 3: Upload your own models and parameters

  1. Click the Upload button in the top-right corner. to upload your own models and parameters. Then, your models and parameters will appear in the contents of the worksheet as several bundles, e.g., we upload our own demo models and parameters as follows,

Step 4: Run evaluation script on the dev set

  1. Write and upload the evaluation script, e.g., we upload our demo evaluation script as follows,

    The code of the script is

    python data/val.json 100 5 5 12345 input > input.json
    python data/val.json 100 5 5 12345 output > output.json
    python input.json > predict.json
    python predict.json output.json

    Note that: More details about data, evaluation scripts and demo examples can be found in our worksheet.

  2. In the web interface terminal on the top of the page, type in the following command:

    Sample instances from the dev set and evaluate the models:

    cl run 'data:0x22dc32' '' '' 'checkpoint:0x9f3d8f' 'fewshot_re_kit:0x70219b' '_processed_data:0xb636c7' 'models:0xbe1716' '' '' 'bash'  --request-docker-image codalab/default-cpu

    Note that: for each file, its uuid should be detailed, e.g., 'data:0x22dc32'.

  3. Get the results:

Submitting a test set evaluation request

To preserve the integrity of test results, we do not release the test set to the public. Instead, we require you to upload your model onto CodaLab so that we can run it on the test set for you. Before evaluating your models on the test set, make sure you have completed evaluation of development set as detailed above. Thus we will assume you are already familiar with uploading files and running commands on CodaLab. Note that, we request you to use our provided data sampling file (FewRel 1.0 and FewRel 2.0 DA) / (FewRel 2.0 NOTA) and evaluation file in this worksheet.

Step 0: Upload your own models and make sure they can run on the dev set

Step 1: Submit an evaluation request

Make sure you fulfill all the following requirements before submitting:
  1. You have tested your scripts on the dev set.
  2. All your codes / checkpoints are set to appropriate permission so we can access.
  3. You use the following files:
    1. or
    3. data:0xbac6f7
    Note that data:0xbac6f7 is private so only we have the access.
  4. Data sampling requirements:
    1. For FewRel 1.0: use "python data/test_wiki.json 10000 N K 12345 input" to sample data, where N K = 5 1 / 5 5 / 10 1 / 10 5
    2. For FewRel 2.0 DA: use "python data/test_pubmed.json 10000 N K 12345 input" to sample data, where N K = 5 1 / 5 5 / 10 1 / 10 5
    3. For FewRel 2.0 NOTA: use "python data/test_wiki.json 10000 N K NOTA 12345 input" to sample data, where N K NOTA = 5 1 0.15 / 5 5 0.15 / 5 1 0.5 / 5 5 0.5
If all requirements are met, please fill out the submission form and we will evaluate your model within one week. If you want us to evaluate on multiple models or challenges, please submit multiple forms. We will require you to provide codalab script (starting with "cl run"). An example for the script is:
cl run 'data:0xbac6f7' '' '' 'checkpoint:0x9f3d8f' 'fewshot_re_kit:0x70219b' '_processed_data:0xb636c7'  'models:0xbe1716' '' '' 'bash' --request-docker-image codalab/default-gpu --request-gpus 1 --request-disk 2g --request-time 2d

Should you have any questions, please send an email to