MP0: Inception

In this MP, you will work with a complete software pipeline for braking a car when a stop sign is sensed. You will learn about ROS, Docker, Righthook, VMs, and simple image processing.
Please find the source code at the Gitlab repo.

This MP is designed to get you all setup with the computing platform and software tools we will use for this class. This MP will not be graded, but it is very important that you finish by Jan 23rd, before MP1 is released.


This course will make heavy use of the Righthook simulator in addition to various ROS and Python libraries for image processing, control, and machine learning. As you will see, Righthook allows us to simulate interesting scenarios with multiple vehicles and pedestrians in a photorealistic world. The vehicles can have different sensors like cameras, LIDARs, etc., and the simulation is guaranteed to be deterministic. We are tremendously grateful to the folks at Righthook Inc. for letting us use this powerful tool (for free) for this class!

All of the above requires serious computing horsepower. We have setup several GPU-enabled virtual machines with Righthook for you. This tutorial will walk you through the different pieces you need to get started.


FastX is a simple X Windows terminal emulator. It allows you to connect over the Internet and work on remote Linux desktop at LAN speed. In this class, you will be running Righthook on remote VMs from your local computer using FastX.

  1. Install FastX from the webstore.
  2. Opening FastX, you will see an interface like this: 
  3. Click the ‘+’ sign on the top-right hand side. Choose ssh option and you see a window like this. Fill up the blanks with the information shown (use your own netid) and click Save. 
  4. Then click on the newly created entry on the FastX window and enter your password. This should give you access to the remote Linux machine with Righthook over FastX (Viola!).


Please sign up at this Google Sheet for VM distribution purpose. It will be changed later when new VMs are ready. If you already have a team of two, please name your team and put the team name on the google sheet. So that we can assign the same VM for both of you. Since our “official” Virtual Machines are not ready yet. We will give you access to two old testing VMs so that you can run the simulation earlier.

In order to balance the load, for people whose netid starting with letter “a” to “m” (including “m”), please use:

For people whose netid starting with letter “n” to “z”, please use:

You should be able to log in using your netid.

The API-Key is the same for everyone:


Setting up Right-Hook and putting it all together

Complete the following steps

  • In the terminal window type the following
nvidia-docker run --name NetID --env="DISPLAY" -e "KEY=API_KEY" -e 
-v /tmp/.X11-unix:/tmp/.X11-unix:rw -v 
 --ulimit nofile=65535:65535 rh_sim/minimaps:c000140725e017ab00810eea6ab55e1cc9310182

This function will essentially create docker container in with the name being your NetId. So, the next time you want to re-start this container just enter the following command.

nvidia-docker start -a NetId
  • Start a new terminal and clone this repository
git clone
  • change directory to this repository
cd mp_release
  • run ros set configuration files
  • build the ros repository
  • To start the simulation Right-hook server
  • To start our own ROS implementation
source devel/setup.bash
roslaunch mp0 run_mp.launch
  • To kill the simulation
  • Kill the docker container after you are done with your code
docker kill NetId


As far as grading is concerned MP0 will not be gradedWe also would not be expecting any submissions.

Having said, this we do expect students to play around with the ROS-code to get themselves acquainted with ROS and its interfacing with Right-Hook. To understand the ROS-code given in MP0 we strongly suggest students to go through ROS-basics webpage. We have also provided a copy of the ROS-basics code in you mp0 files as


To get yourselves acquainted with ROS, try the following:

  1. Figure out what the file is doing in you mp0-release repository.
  2. Figure out what the callback() function in the file is actually doing. Specifically figure out the reason behind the line 35 in the ROS-code.
  3. How does the Stop Sign detector actually work? Try to play around with the threshold values in (line 81) and relevant braking values in (line 90) for your algorithm to perform
  4. Suppose we want to write a ROS-node which essentially draws a bounding box in the Image around which Stop Sign has been detected and publishes it on the topic ‘\stop_sign_boundingbox ‘. Try to figure out how would you go about writing such a ROS-node. If this too easy for you or you feel too ambitious in the beginning of the semester try writing out this node on your own. It will surely help you in the future MPs