Not logged in. Login | Signup

ImageNet Large Scale Visual Recognition Challenge (ILSVRC)

Date: June 2, 2015

Dear ILSVRC community,

This is a follow up to the announcement on May 19, 2015 with some more details and the status of the test server.

During the period of November 28th, 2014 to May 13th, 2015, there were at least 30 accounts used by a team from Baidu to submit to the test server at least 200 times, far exceeding the specified limit of two submissions per week. This includes short periods of very high usage, for example with more than 40 submissions over 5 days from March 15th, 2015 to March 19th, 2015. Figure A below shows submissions from ImageNet accounts known to be associated with the team in question. Figure B shows a comparison to the activity from all other accounts.

Figure A Figure B

The results obtained during this period are reported in a recent arXiv paper. Because of the violation of the regulations of the test server, these results may not be directly comparable to results obtained and reported by other teams. To make this clear, by exploiting the ability to test many slightly different solutions on the test server it is possible to 1) select the best out of a set of very similar solutions based on test performance and achieve a small but potentially significant advantage and 2) choose methods for further research and development based directly on the test data instead of using only the training and validation data for such choices.

We noticed these irregularities on May 14th, 2015. The authors of the paper were notified on May 17th, 2015. On May 22nd, 2015, upon further discussion and consultation with senior advisors in our research community, we informed the authors that

  1. Their results obtained from the ImageNet test server are not directly comparable to results obtained by others, and,
  2. There is also concern about any new submission from Baidu that builds on top of these results. This includes submissions to all of the ILSVRC challenge tasks (classification, localization, and detection) as parts of the datasets are shared. We therefore requested that they refrain from submitting to the evaluation server or the challenge for the next 12 months.

We are in communication with the team involved in this incident. They have asked us to forward the message included further below to the community. Note that in order to stay neutral and independent, we have not and cannot work with any team to interpret or review results.

The test server is now back online. We look forward to continued progress in visual recognition!

The ILSVRC organizers

Message from the team in question:

Dear ILSVRC community,

Recently the ILSVRC organizers contacted the Heterogeneous Computing team to inform us that we exceeded the allowable number of weekly submissions to the ImageNet servers (~ 200 submissions during the lifespan of our project).

We apologize for this mistake and are continuing to review the results. We have added a note to our research paper, Deep Image: Scaling up Image Recognition, and will continue to provide relevant updates as we learn more.

We are staunch supporters of fairness and transparency in the ImageNet Challenge and are committed to the integrity of the scientific process.

Ren Wu – Baidu Heterogeneous Computing Team