Evaluation Guidelines for Stage II (Simulation Stage) of the
ARTPARK Robotics Challenge
The simulation stage is meant to ensure that the software stack and other elements of the final solution are ready and working. With some minimal changes such as tuning of the algorithms based on some real-world data, this part of the challenge should be reusable. Therefore, it is worth investing some time in choosing the right simulation software, so that it will be compatible with the final (physical) solution that you will be building.
What you submit needs to be a simulation and not an animation. It should be an integrated simulation where the perception, control, and navigation software is integrated with the simulation environment. You are supposed to create a realistic environment using the simulation software so that it is consistent with the description given in the problem statement. The robot, along with any other attachments such as manipulators, is expected to navigate around the environment autonomously. The robot needs to have (simulated) sensors attached to it, and it should process the sensor data (including camera data) for navigation, decision making, and control. During the final presentation, you may assume that any mapping or tuning of the algorithms has already been done. Any machine learning or computer vision algorithms should be designed and tuned using your own synthetic and real data.
The trash items consist of small paper cups (120-150 ml) and beverage cans (approximately 250 ml). You can assume that both are made of light but inflexible material. The markings on the floor should resemble lines drawn with standard colored markers used for writing on whiteboards.
You may assume indoor lighting conditions of 800 lumen, with lights mounted on the ceiling.
The sanitizer spray should be modelled so that when the sanitizer is sprayed on a surface,the part of the surface on which the spray falls changes colour. This way, we can check which part of the surface has been sprayed. Similarly, the markings should be modelled so that when the robot attempts to clear a marking by using a brush or other means, only the part of the marking that is touched by the brush or other cleaning mechanism should disappear.
At the beginning of each run, it should be possible to change the location of the dustbin, the number (up to a maximum of 4) and locations of the markings on the floor, and the number (up to a maximum of 5) and locations of the trash items, which consist of a random combination of paper cups and cans. At the beginning of each run, it should also be possible to choose a starting point outside of the washroom.
The scoring system
Note: The scoring system described here applies only to the simulation stage. It will be different for the final competition.
There is a time limit of 30 minutes to run the simulation. Each submission will receive points based on how many tasks are completed within this time and how completely each task is completed. There is no limit on the number of attempts that the robot may make within the 30 minutes. In addition to task completion, some general and specific considerations will also be used to award points, as described below:
1. General: (10 points) How well does the environment match the environment in the problem description? Is the washroom structured according to what is mentioned in the problem description? Are all the objects in the specified size range? Is the trash modeled well?
2. General: (10 points) How close is the simulation to a real-world setting? Do the perception, navigation, and control aspects make any assumptions that will fail in a real-world setting?
3. Specific: (10 points) Is the spray modelled in a way that it is realistic? The part of the floor on which the spray falls is clearly highlighted in some way?
4. Specific: (10 points) Is the marking modelled so that it reflects lines drawn by a marker? Is the clearing (disappearance) of the marking consistent with the movement of the cleaning mechanism?
5. Task 1: (10 points) Was the robot successfully able to detect and navigate through the door and enter the washroom without hitting the edges of the door or any walls?
6. Task 2: (10 points) How many of the (up to 5) trash items did the simulation pick up correctly?
7. Task 3: (10 points) How many of the trash items did it drop correctly?
8. Task 4: (10 points) What percentage of the counter and washbasin surface was sprayed with sanitizer?
9. Task 5: (10 points) What percentage of the counter was cleared of the sanitizer?
10. Task 6: (10 points) What percentage of the markings were removed properly?
11. Task 7: (10 points) Did the robot exit the washroom correctly and return to where it started (within some range?)?
12. Task 8: (10 points - bonus) What percentage of the wash basin was cleared of the sanitizer?
13. Task 9: (10 points - bonus) Keep a soap dispenser and a glass on the counter in a random location and successfully complete Task 5.
What needs to be submitted
We will create private repositories for each team within the ARTPARK Github Organization. The submission will be a Github repository along with a README and dockerfile, which the evaluation team can use to run the simulation on their own machine. If the simulation is based on open source software such as Gazebo, Ignition, Isaac, Carla, Webots, Unity, Airsim, etc. the technical evaluation committee will be able to run the simulation and test it. However, if the simulation is based on licensed/proprietary software, then the committee would need remote access to the simulation. The team would have to discuss the logistics of this setup with the evaluation committee ahead of time.