The Activity in Extended Videos Prize Challenge (ActEV-PC) seeks to encourage the development of robust automatic activity detection algorithms for an extended video. ActEV-PC is operated by NIST and sponsored by IARPA. Challenge participants will develop activity detection and temporal localization algorithms for 18 activities that are to be found in extended videos. These videos contain significant spans without any activities and intervals with potentially multiple concurrent activities. Activities must be detected and localized temporally. ActEV-PC will contain two phases – an open leaderboard evaluation (Phase 1) and an independent ActEV prize evaluation (Phase 2).
Open leaderboard evaluation (Phase 1)
For open leaderboard evaluation, the challenge participants should run their software on their systems and configurations and submit the system output defined by the Evaluation Plan to the NIST Scoring Server. This phase will serve as a qualifying stage where the top 8 participants will proceed to phase 2.
Independent ActEV prize evaluation (Phase 2)
For independent ActEV prize evaluation, the challenge participants should submit their runnable system to NIST using the forthcoming Evaluation Container Submission Instructions. NIST will evaluate system performance on sequestered data using NIST hardware–details coming soon.
The developers of the most accurate activity detection algorithms will be eligible to win cash prizes from a total prize purse of $50,000. Prizes will be distributed based on the following criteria: Prizes distributed will be:
- First Place Prize – $25,000
- Second Place Prize – $15,000
- Third Place Prize – $10,000
To receive the full award payment, it is required that each winning team send at least one representative to a workshop organized for the challenge results and present the winning team’s results at the workshop. The workshop may be planned in conjunction with an existing conference or may be held as a standalone workshop. This workshop is tentatively planned for the ActivityNet workshop at CVPR’19. See the complete eligibility requirements in the Evaluation Plan.
For this evaluation, an activity is defined to be “one or more people performing a specified movement or interacting with an object or group of objects”. The activities covered in this challenge are defined in the Evaluation Plan.
How to Participate
To take part in the ActEV-PC evaluation you need to register on the actev.nist.gov website and then acknowledge that you have read and accepted the terms and conditions from the ActEV-PC Evaluation Plan and the VIRAT data license PDF, by clicking on the two checkboxes to agree.
We will then send you information on how to download data. Using a valid registered account, you will be able to upload your JSON format results (see website for Evaluation Plan) to the ActEV scoring server and challenge participate in the ActEV-PC Leaderboard evaluation. The challenge will then proceed with the selection of 8 participants, who will deliver their activity detection algorithm to NIST in the CLI based distribution. Each challenge participant’s algorithm will then be run by NIST on a sequestered video dataset.
The main scoring metrics will be based on activity detection and temporal localization. Standard evaluation measures will be used, incorporating the probability of missed detection and rate of false alarm. A complete description of the metrics is provided in the Evaluation Plan.
The scoring software is available at https://github.com/usnistgov/ActEV_Scorer.
The main scoring metrics will be based on Activity detection, temporal localization and temporal localization. We will use standard evaluation measures (e.g., probability of missed detection, rate of false alarm) (see: Evaluation Plan).
ActEV NIST team ([email protected] )