From Doodles to Dashboards: Notebooks as a Cloud Platform

Peter Parente (@parente), Developer in IBM Cloud Emerging Technology

Hello!

I'm a member of the IBM Cloud Emerging Technology team
We innovate, incubate, and integrate disruptive tech
I'm here to talk about notebooks and why they're on our radar

Old problem, new in-roads

We are drowning in data but thirsting for knowledge.

Diagram of the Berkeley Data Analytics Stack

e.g., Berkeley Data Analytics Stack from AMPLab

✓ Open analytics stacks
✓ Cost-effective cloud
✓ Data science training
UX to bring it all together?

Notebooks

Screenshot of a notebook for exploring the Lorenz system of differential equations

Text
+
Code
+
Visualizations
+
Widgets
+
Compute
+
Data

Project Jupyter (née IPython)

"Open source, interactive data science and scientific computing"
Notebook authoring and related tools
Large, growing community and ecosystem
Open formats, protocols, APIs

Screenshot of the Jupyter community visualization from the Project Jupyter homepage

Screenshot of the Jupyter nbviewer (notebook viewer) site

Screenshot of a Jupyter notebook modeling the number of notebook files on GitHub

Why such interest?

"As the [on-demand] storage/compute functions become embedded, we’re seeing platforms emerging everywhere and all of the time. Drones are a new platform. Satellites are a new platform. Rockets are a new platform. Daily Fantasy Sports are a new *platform*. Med-tech is a new platform. Fin-tech is a new platform. The wired human being is a new platform."

Eric Norlin, Defrag 2015: The Frontier of Platforms

Notebooks are a new platform.

For what, exactly?

Consider this scenario

My team of data scientists, developers, and evangelists wants to increase attendance at IBM meetups. To get started, we first need to dabble with the data available to us. We then need to document and iterate on our best ideas for reaching new attendees and measuring our impact. When we've agreed on a first pass, we need to make our insights accessible for everyday use. As we improve our approach over time we need to make driving meetup attendance part of our culture.

Doodle

We first need to dabble with the data

Screenshot of a rough-cut meetup notebook

Document

We then need to document and iterate on our best ideas

Dashboard

We need to make our insights accessible

Discussion

We need to make driving meetup attendance part of our culture

Screenshot of a Slack bot that responds to meetup queries by supported by functions in a notebook

Evolution

Notebooks are a platform

for ideation, experimentation, evaluation, and collaboration
for controlling powerful data analytics stacks
for prototyping and delivering just-good-enough data products
for evolving computational narratives
for facilitating the creation of knowledge from data

To learn more

github.com/ibm-et/defrag2015 - Slides, videos, notebooks from this talk
jupyter.org - Project Jupyter Homepage
blog.ibmjstart.net - IBM Cloud Emerging Tech Blog
spark.tc - Spark Technology Center
bluemix.net/catalog/apache-spark/ - Bluemix Data and Analytics
datascientistworkbench.com - IBM Data Scientist Workbench