previous episode

Reproducible Research with Interactive Jupyter Dashboards

next episode

Jupyter dashboards

Overview

Teaching: 25 min
Exercises: 25 min
Questions

  • What is a jupyter notebook?

  • What is a jupyter dashboard?

  • How to create Jupyter dashboards on your laptop?

  • Why Jupyter dashboards make our research more reproducible?

Objectives

  • Learn about jupyter notebook and its dashboard extension

  • Start a new jupyter dashboard on your laptop

  • Understand jupyter dashboards as a tool to make reproducible research

What is a jupyter notebook?

Jupyter notebook, formerly known as IPython notebook (or Interactive Python), is a flexible and powerful open source research tool that can help you keep a narrative of your coding process. The name Jupyter is an acronym of the three core languages it was designed for: JUlia, PYThon, and R. Project Jupyter supports interactive data science and scientific computing across more than 40 programming languages.

https://jupyter.org/

Jupyter as Coding Diary

You can think of the notebook as a lab or field diary that keeps a detailed record of the steps you take as you develop scripts and programming workflows. Just as you would with a field notebook, it is important to develop good note-taking habits. This workshop is designed to impart a set of skills, tools, and best practices you can implement in your own research to enhance reprodubility, which will make modifications, collaboration, and publishing easier.

Jupyter Architecture

Jupyter is comprised of several components, some of which the user doesn’t directly interact with, but should at least be aware of. On the front-end, the user will work with the:

  1. Web Application: Browser-based tool for interactive development of notebook documents
  2. Notebook Document: A representation of all content visible in the web application, including inputs and outputs of the computations, explanatory text, mathematics, images, and rich media representations of objects. These documents are internally JSON files and are saved with the .ipynb extension. Since JSON is a plain text format, they can be version-controlled and shared with colleagues.

Jupyter also has some back-end processes, including the:

  1. Kernel: A separate process responsible for running user code. For the purposes of this workshop, we will be working on Python kernels, although Jupyter is capable of interfacing with other programming languages as well.
  2. Notebook Server: Communicates with kernel and routes the Python programming language to the web browser.

Jupyter Ecosystem

There has been considerable development by both Project Jupyter and external collaborators that have yielded a multitude of options for Jupyter users. This diagram gives a sample of some of the possibilities.

Launch Jupyter notebook App

Make sure you installed Anaconda python 3.6 and Anaconda navigator along with all the additional python packages we will be using for this lesson. For more information read and follow the setup instructions for this lesson.

Start jupyter notebook

We want to start a new jupyter notebook using our newly created conda environment. To select “jupyter_dashboards_workshop” conda environment:

Jupyter starts in your default browser at the address http://localhost:8888/tree and shows the tree (directories) from the root directory (here is root directory is to be understood from the installation location of Anaconda). It may vary from one laptop to another but you should not worry about it.

When clicking on “New” (right hand side) as shown on the Figure below:

You should be able to select:

For now, the most important is to be able to start a new jupyter notebook. We will now look into jupyter dashboards and we will learn more about jupyter notebook/dashboard later in this lesson.

Default browser

You may need to change the default browser for starting your jupyter notebook. All the examples in this lesson have been tested with Google chrome

What is a Jupyter Dashboard?

A jupyter dashboard is very much like a jupyter notebook but thanks to the dashboards layout extension you can arrange your notebook outputs (text, plots, widgets, etc.) in a grid - or report-like layouts. The layouts are saved in your notebook document so the layouts can be shared too.

Don’t worry if you do not understand all the details now (for instance if you don’t know what are widgets, what we mean by grid layouts). For now our goal is to check if the dashboards layout extension works properly on your laptop.

In your newly created jupyter notebook, you should have the dashboard view extension:

Why are Jupyter Notebooks/dashboards a powerful tool for conducting reproducible research?

We have chosen in this lesson to use Jupyter dashboards and the Python language to demonstrate how reproducibility of research work can be improved.

The main advantages of both Jupyter notebook/dashboard and python are:

Jupyter Notebooks/dashboards are great because they facilitate:

Knowing your way around jupyter dashboard

The Jupyter Notebook is a web application that allows you to create and share documents that contain live code, equations, visualizations and explanatory text. The following links may be useful in gaining a basic understanding of Jupyter and how to work in Jupyter notebook:

  1. http://jupyter.org/about.html: general information about Jupyter
  2. https://jupyter-notebook-beginner-guide.readthedocs.io/en/latest/execute.html: starting guide
  3. http://nbviewer.jupyter.org/github/jupyter/notebook/blob/master/docs/source/examples/Notebook/Notebook%20Basics.ipynb: notebook basics

If this is your first time using Jupyter, start with taking a tour of the user interface:

Here are some other helpful tips and shortcuts:

Cell Types

Cells are designated as Code, Markdown, or Raw NBConvert (Notebook Conversion). To switch from one type to another, click on Cell -> Cell Type.

Throughout our tutorials, we will mainly be working in code cells to learn how to use Python. However, each of the tutorials include Markdown cells to provide introductions and learning objectives for each tutorial, background information, and instructions throughout. We encourage you to open these markdown cells to become familiar with the syntax, and add markdown notes as you see fit. Learning to use Markdown and Notebook Conversion cells will become a useful skill for Python scripts and notebooks you will use in your future research. Throughout this data institute, you will learn to use the full set of Jupyter Notebook tools for conducting reproducible (and publishable) research.

Code

Code cells are where we will write and execute our Python scripts and functions.

Markdown

Markdown cells are mainly used to neatly document the code that is written in Jupyter notebooks. While this course does not focus on Markdown, we encourage you to explore the Markdown capabilities, and add Markdown cells to your notebooks as you see fit. Some useful resources for learning Markdown Syntax include:

  1. Markdown Cheatsheet: http://nestacms.com/docs/creating-content/markdown-cheat-sheet
  2. https://ipython.org/ipython-doc/3/notebook/notebook.html#markdown-cells
  3. Example Markdown Notebook: http://nbviewer.jupyter.org/github/ipython/ipython/blob/1.x/examples/notebooks/Part%204%20-%20Markdown%20Cells.ipynb

NBConvert - Notebook Conversion

The NBConvert tool allows you to convert a Jupyter .ipynb notebook document file into another format. The nbconvert documentation contains a complete description of this tool’s capabilities: https://media.readthedocs.org/pdf/nbconvert/latest/nbconvert.pdf. The description below, copied from the document, gives a brief overview of the tool:

Using nbconvert enables:

Overall, notebook conversion and the nbconvert tool give scientists and researchers the flexibility to deliver information in a timely way across different formats.

Primarily, the nbconvert tool allows you to convert a Jupyter .ipynb notebook document file into another static format including HTML, LaTeX, PDF, Markdown, reStructuredText, and more. The nbconvert tool can also add productivity to your workflow when used to execute notebooks programmatically.

If used as a Python library (import nbconvert), nbconvert adds notebook conversion within a project. For example, nbconvert is used to implement the “Download as” feature within the Jupyter Notebook web application. When used as a command line tool (invoked as jupyter nbconvert …), users can conveniently convert just one or a batch of notebook files to another format.

Use nbconvert to export your Jupyter dashboard/notebook

  • Create a new text cell in your notebook
  • Download it as HTML i.e. Select File –> Download as HTML (.html)
  • View the newly created HTML file with your browser

Solution

Be aware that by default you can’t save your notebook in all the listed formats. For instance to download your notebook as a LaTeX file, you need to install latex on your laptop. To unlock nbconvert’s full capabilities, nbconvert requires Pandoc and TeX (specifically, XeLaTeX). These must be installed separately.

Jupyter Notebook tips, tricks, and shortcuts

This link has some useful tips that may come in handy in your research: https://www.dataquest.io/blog/jupyter-notebook-tips-tricks-shortcuts/

Display variables with %whos

Since the Jupyter notebook doesn’t have a separate window to display the variables that have been generated, we can use the commands %who, %who_ls, or %whos to show a complete list of imported modules, functions, variables, and other outputs (eg. figures). The command %who lists the variable names in row format, %who_ls lists the variable names in column format, and %whos is the most comprehensive, displaying variable name, type, and data/info. Remember this tool throughout the following tutorials if you need to check whether a module or function was already imported, a variable created, etc.

# Display variables & imported modules
%whos

Get familiar with Jupyter notebooks

To familiarise participants with the notebook environment, build up a simple notebook from scratch demonstrating the following operations:

See Navigating jupyter notebook instructions for instructors for reference.

Documentation and references

Curious to learn more? Refer to these Jupyter documents:

You are not alone! There are many resources for Jupyter Notebooks and Python:

Project Jupyter

Python

Key Points

  • Jupyter dashboards

previous episode

next episode