Using jupyter notebook on keeling

Updated 4 August 2016

I usually develop code for python using the jupyter Notebook (formerly iPython notebook), which is a browser-based python interactive environment.  I am assuming you already have anaconda python installed following the directions here.

I created a notebooks subdirectory in my $HOME/python directory,

mkdir -p $HOME/python/notebooks

and launch jupyter from there to save the files in a convenient, organized location.  To start jupyter notebook, do the following:

I will describe two modes of working with jupyter notebook.

Local machine. If you’re working on your local machine (mac/linux), and want the notebook to launch in your local web browser, simply type the following in a terminal window:

jupyter notebook

this will launch a web browser window/tab.

 

Running a remote iPython session from a remote computer on your local machine. If you have installed anaconda python on your remote computer (e.g. Keeling in DAS), you have the option to run jupyter notebook remotely on keeling (using its computing resources) while displaying and interacting with jupyter notebook on your local machine’s browser.

First, you need to launch jupyter on keeling, with the option to not launch a browser.  You’re also going to set up a port for jupyter to connect to (a number between 7000 and 8000).  This number is reserved for your session, and if another keeling user or process uses the same port, then it could get confused.  If you have problems, try a different port number.

On keeling, start the jupyter notebook server, where the port number is your port number:

jupyter notebook –port=xxxx –no-browser

Now, on your local machine, set up an ssh tunnel to keeling using this port you run (with your netid).  If you are using OS X or linux, use a terminal window.  If you are using windows, follow directions here using the source and destination ports described below.

ssh -N -f -L xxxx:localhost:xxxx netid@keeling.earth.illinois.edu

This will set up an ssh tunnel from port xxxx (which is accessible from your web browser) to the jupyter server on keeling.  You will need to kill this ssh session (with Control-C) when you are finished, as you can’t connect again to this port without killing it explicitly.  If you are having trouble killing it, do the following in another window, typing in the process number of the hanging ssh session (be careful to kill the correct job, otherwise results my vary):

[snesbitt@chuva:~/Downloads]$ ps -u snesbitt -f | grep ssh
snesbitt   850   251   0 31Jan14 ??         0:01.20 /usr/bin/ssh-agent -l
snesbitt 11900  4163   0 Fri03PM ttys000    0:00.07 ssh -Y keeling.earth.illinois.edu
snesbitt 12848 12839   0  6:47PM ttys001    0:00.01 ssh -N -L localhost:8080:localhost:7000 snesbitt@keeling.earth.illinois.edu
snesbitt 12559  4456   0  5:43PM ttys003    0:00.12 ssh -Y keeling.earth.illinois.edu
snesbitt 12868 12862   0  6:49PM ttys004    0:00.00 grep ssh
[snesbitt@chuva:~/Downloads]$ kill -9 12848

Note that in this case, process 12848 (second column) was the ssh port command I used for forwarding.

Now, point a web browser to http://localhost:xxxx.  You should see an iPython notebook that displays locally, but executes code on keeling, and has access to keeling file systems.  Note that it will not have access to your local file systems.

Important notes:

  • Closing your web browser does not close your ssh tunnel or end your python session on keeling.  Remember to close each of these when you are done with them to save resources on keeling.
  • You should be able to reconnect to a keeling jupyter session if your local computer goes to sleep, or disconnects from the network.  You may need to re-establish your ssh tunnel, and kill it locally (either with Control-C or using the kill command) as described above if it doesn’t respond.

Here are some screen shots of jupyter notebook from keeling displaying on my OS X laptop (from home).  I can execute code as if I were connected on keeling.  Note that some things (like complicated graphics or lots of text output in my iPython notebook) may significantly tax my network connnection between my local computer and keeling, but this can be typically avoided.

Opening up an iPython notebook:

Advanced topics you can see me about

•ipyparallel

•executing interactive code on keeling compute nodes (see my class lecture on this topic for more information: jupyter notebook on a high-performance computing cluster)

Leave a Reply