Practical pydata tips¶
- Conda/anaconda is the easiest way to manage python dependencies.
- Jupyter notebooks are great for interactive data analysis, reproducibility.
- Jupyter notebooks shift-tab feature is the best way to explore APIs through docstrings and experimentation.
- This presentation was made from a Jupyter Notebook, converted to reveal.js.
- Pandas pd.read_csv() is probably the most useful thing for a data scientist.
- HackerRank is a great way to learn Python and discover "coding accents"-- large variety of how people code the same thing.
- Bokeh is a cool thing for making interactive D3 plots from Python.
- Tensorflow has a Python API.
Conda/anaconda is the easiest way to manage python dependencies.¶
In [4]:
! wget https://repo.continuum.io/miniconda/Miniconda3-latest-MacOSX-x86_64.sh
In [6]:
! ls -1 Miniconda*
In [ ]:
! conda update conda
In [8]:
! conda install numpy scipy pandas matplotlib jupyter seaborn bokeh mkl
In [6]:
! conda list
Jupyter notebooks are great for interactive data analysis, reproducibility.¶
The function is $y = \frac{\sin^2(x)}{x^2}$.
In [20]:
import matplotlib.pyplot as plt
import numpy as np
%matplotlib inline
x = np.linspace(-2.0*np.pi, 2.0*np.pi, 10000)
y = np.sinc(x)**2
plt.plot(x, np.log10(y))
plt.ylim(-3, 0)
Out[20]:
Jupyter notebooks shift-tab feature is the best way to explore APIs through docstrings and experimentation.¶
In [22]:
import pandas as pd
In [ ]:
pd.read_csv( #hit shift-tab and see what happens!
In [12]:
Image('../figs/shift_tab_example.png')
Out[12]:
This presentation was made from a Jupyter Notebook, and converted to reveal.js.¶
In [25]:
! jupyter-nbconvert BJpy_01-01_First_trial.ipynb --to slides --post serve
Pandas pd.read_csv() is probably the most useful thing for a data scientist.¶
In [27]:
Image(filename='../figs/shift_tab_example.png')
Out[27]:
HackerRank is a great way to learn Python and discover "coding accents"-- the variety of how people code the same problem in the same language (Python).¶
In [29]:
Image('../figs/hackerrank_python.png', width=300)
Out[29]:
Find the Second Largest Number: https://www.hackerrank.com/challenges/find-second-maximum-number-in-a-list
In [30]:
Image('../figs/hackerrank_problem_setup.png')
Out[30]:
In [32]:
# Python 2!
n = int(raw_input())
uniq_vals = list(set(map(int, raw_input().split())))
uniq_vals.sort()
print uniq_vals[-2]
In [34]:
# Python 3
n = int(input())
uniq_vals = list(set(map(int, input().split())))
uniq_vals.sort()
print(uniq_vals[-2])
In [ ]:
# From user alexander_zhou
def findSM(l):
f, s = l[0], l[0]
for i in range(len(l)):
if l[i] > f:
s, f = f, l[i]
elif l[i] < f:
if f == s:
s = l[i]
elif l[i] > s:
s = l[i]
return s
n = int(input())
l = input().split()
for i in range(n):
l[i] = int(l[i])
print(findSM(l))
In [ ]:
# From user richmond
import random
import sys
i = 1
for line in sys.stdin:
if i == 1:
if 2>int(line) and int(line)>100:
break
else:
r = line
s = []
sss = r.split(' ')
for ss in sss:
s.append(int(ss))
__s = -100
for i in s:
if i > __s:
__s = i
_s = -100
for i in s:
if i != __s:
if i > _s:
_s = i
print _s
break
i += 1
In [ ]:
# From user dragonfury
a = int(raw_input())
b = [int(x) for x in raw_input().split(' ')]
m=-101
for i in range(0,a):
if m<b[i]:
m=b[i]
for i in range(0,a):
if b[i] == m:
b[i]=-101
m=-101
for i in range(0,a):
if m<b[i]:
m=b[i]
print(m)
In [ ]:
x=input()
a=raw_input()
a=a.split(' ')
a=map(int,a)
a=set(a)
a=list(a)
a.sort()
print a[len(a)-2]
In [ ]:
# From user Matiel
N = int(input())
liste = input().split()
i = 0
for nombre in liste:
liste[i] = int(nombre)
i += 1
maximum = max(liste)
preMax = min(liste)
for nombre in liste:
if (nombre < maximum and nombre > preMax):
preMax = nombre
print(preMax)
In [ ]:
# from user Kabashka
import subprocess
import os
import sys
sizeOfData = int(sys.stdin.readline())
a = map(int,sys.stdin.readline().replace('\n','').split(' '))
lastMax = a[0]
max = a[0]
for i in a:
if i > max:
lastMax=max
max = i
elif i==max:
continue
elif i > lastMax:
lastMax = i
elif lastMax==max:
lastMax = i
sys.stdout.write(str(lastMax))
Bokeh makes interactive D3 plots from Python matplotlib code.¶
In [38]:
import numpy as np
from bokeh.plotting import figure, show, output_file
from bokeh.io import output_notebook
output_notebook()
N = 4000
x = np.random.random(size=N) * 100
y = np.random.random(size=N) * 100
radii = np.random.random(size=N) * 1.5
colors = [
"#%02x%02x%02x" % (int(r), int(g), 150) for r, g in zip(50+2*x, 30+2*y)
]
TOOLS="crosshair,pan,wheel_zoom,box_zoom,undo,redo,reset,tap,save,box_select,poly_select,lasso_select"
p = figure(tools=TOOLS)
p.scatter(x, y, radius=radii,
fill_color=colors, fill_alpha=0.6,
line_color=None)
output_file("color_scatter.html", title="color_scatter.py example")
show(p) # open a browser
Out[38]:
Tensorflow has a Python API.¶
Fizz Buzz in Tensorflow http://joelgrus.com/2016/05/23/fizz-buzz-in-tensorflow/¶
interviewer: OK, so are you familiar with "fizz buzz"?
me: ...
interviewer: Is that a yes or a no?
me: It's more of a "I can't believe you're asking me that."
interviewer: OK, so I need you to print the numbers from 1 to 100, except that if the number is divisible by 3 print "fizz", if it's divisible by 5 print "buzz", and if it's divisible by 15 print "fizzbuzz".
me: I'm familiar with it.
interviewer: Great, we find that candidates who can't get this right don't do well here.
me: ...
interviewer: Here's a marker and an eraser.
interviewer: Do you need help getting started?
me: No, no, I'm good. So let's start with some standard imports:
In [ ]:
import numpy as np
import tensorflow as tf
interviewer: Um, you understand the problem is fizzbuzz, right?
me: Do I ever. So, now let's talk models. I'm thinking a simple multi-layer-perceptron with one hidden layer.
interviewer: OK, that's probably enough.
me: That's enough setup, you're exactly right. Now we need to generate some training data. It would be cheating to use the numbers 1 to 100 in our training data, so let's train it on all the remaining numbers up to 1024:
In [ ]:
def fizz_buzz_encode(i):
if i % 15 == 0: return np.array([0, 0, 0, 1])
elif i % 5 == 0: return np.array([0, 0, 1, 0])
elif i % 3 == 0: return np.array([0, 1, 0, 0])
else: return np.array([1, 0, 0, 0])
def fizz_buzz(i, prediction):
return [str(i), "fizz", "buzz", "fizzbuzz"][prediction]
def model(X, w_h, w_o):
h = tf.nn.relu(tf.matmul(X, w_h))
return tf.matmul(h, w_o)
ipython
In [185]: output
Out[185]:
array(['1', '2', 'fizz', '4', 'buzz', 'fizz', '7', '8', 'fizz', 'buzz',
'11', 'fizz', '13', '14', 'fizzbuzz', '16', '17', 'fizz', '19',
'buzz', '21', '22', '23', 'fizz', 'buzz', '26', 'fizz', '28', '29',
'fizzbuzz', '31', 'fizz', 'fizz', '34', 'buzz', 'fizz', '37', '38',
'fizz', 'buzz', '41', '42', '43', '44', 'fizzbuzz', '46', '47',
'fizz', '49', 'buzz', 'fizz', '52', 'fizz', 'fizz', 'buzz', '56',
'fizz', '58', '59', 'fizzbuzz', '61', '62', 'fizz', '64', 'buzz',
'fizz', '67', '68', '69', 'buzz', '71', 'fizz', '73', '74',
'fizzbuzz', '76', '77', 'fizz', '79', 'buzz', '81', '82', '83',
'84', 'buzz', '86', '87', '88', '89', 'fizzbuzz', '91', '92', '93',
'94', 'buzz', 'fizz', '97', '98', 'fizz', 'fizz'],
dtype='<U8')