The talk will review the testing practices that were designed and implemented during the development of the software infrastructure used for the Virtual Robotics Challenge. The scope of the techniques applied goes from automated testing of VRC software pieces (Gazebo simulator, DRCSim - DRC specific ROS wrappers and materials and the Cloudsim web provisioning tool) to the manual testing plan. Interesting points to explore in the talk:

• The root of all testing done: our continuous testing integration (CI) setup. Jenkins, and how GPU testing is done on it for different platforms.
• The importance of moving continuous testing integration to run as soon as possible in production machines (real challenge machines).
• How different cloud providers were tested and when and how the problems appeared. Experience here is nice to know since problems were not trivial.
• How the “release often, release early” philosophy was applied during the development cycle to keep users in the testing loop. How nightly builds played a fundamental role in this.
• How the ideal testing does not exists and how can real production tests with real users be the best of your test suites. How was that scheduled in VRC.
• How exceptional cases, like identifying an exploit hours before the competition, were managed to not affect the testing done before.

The goal would be to provide the audience with feedback and conclusions about software testing and testing decisions in a real big open source robotics software event from first hand.