I had the great pleasure of taking part in Crunch 2019
in Budapest, Hungary
(Data Conference. October 16-18, 2019).
Below you will find impressions from the conference, and links for further reading.
The Driver (aka an application’s driver process) is responsible for converting a user application into smaller execution units called tasks and then schedule them to run on the executors.
# Setting up a simple linear regression model, for the
# California housing problem, in sklearn:
from sklearn.datasets.california_housing import fetch_california_housing
from sklearn.model_selection import train_test_split
import warnings
warnings.filterwarnings("ignore")
import numpy as np
np.random.seed(0)
cal_housing = fetch_california_housing()
# split 80/20 train-test
X_train, X_test, y_train, y_test = train_test_split(cal_housing.data,
cal_housing.target,
test_size=0.2,
random_state=1)
Which was followed by training (of the linear regression model), and using
the model to make predictions.
from tensorflow.keras.layers import Dense
from tensorflow.keras.models import Sequential
import tensorflow as tf
tf.set_random_seed(42) # For reproducibility
tf.logging.set_verbosity(tf.logging.ERROR)
np.random.seed(42)
model = Sequential()
# Input layer
model.add(Dense(20, input_dim=8, activation='relu'))
# Automatically infers the input_dim based on the layer before it
model.add(Dense(20, activation='relu'))
# Output layer
model.add(Dense(1, activation='linear'))
Including setting up a loss function in Keras.
from tensorflow.keras import metrics
from tensorflow.keras import losses
loss = "mse" # Or loss = losses.mse
metrics = ["mae", "mse"] # Or metrics = [metrics.mae, metrics.mse]
model.compile(optimizer="sgd", loss=loss, metrics=metrics)
model.fit(X_train, y_train, epochs=10)
Works better with Adam optimizer:
from tensorflow.keras import optimizers
model = build_model()
optimizer = optimizers.Adam(lr=0.001)
model.compile(optimizer=optimizer, loss=loss, metrics=metrics)
history = model.fit(X_train, y_train, epochs=20)
Throughout the day we tried out different (neural net) models.
Notebooks have rapidly grown in popularity among data scientists to become the de facto standard for quick prototyping and exploratory analysis.According to Ufford:
Data powers Netflix. It permeates our thoughts, informs our decisions, and challenges our assumptions. It fuels experimentation and innovation at unprecedented scale.Indeed, Netflix is clearly a BigData place...
At Netflix, they must handle 3 trillion (10^12) daily user events involving 12 petabytes of data [1].
Netflix spent $12 billions on content in 2018. Analysts expect that to grow to $15 Billion this year [2].All Netflix users are unique, and are given different suggestions, including setting up different images for the same products, based on recorded ''events'' for the user.
The output of LIME is a list of explanations, reflecting the contribution of each feature to the prediction of a data sample.
The company’s experimental hiring tool used artificial intelligence to give job candidates scores ranging from one to five stars — much like shoppers rate products on Amazon...Sounded awesome, here you would :
Literally have an engine that you could give 100 resumes, and it would spit out the top five, which Amazon would then hire.But, but... human recruiters would never trust a resume alone. And when you screen resumes, it makes sense to give unusual candidates a chance. Diversity of backgrounds and different point of views is valuable in a team.
1 in 38 adult Americans (2.2 million) is under some form of correctional supervision.Again: Not good, and certainly not good, if the tool itself is a ''blackbox'' technology, where you can't ask the model how it came to its conclusion...
Under immense pressure to reduce prison numbers without risking a rise in crime, courtrooms across the US have turned to automated tools in attempts to shuffle defendants through the legal system as efficiently and safely as possible.
Police departments use predictive algorithms to strategize about where to send their ranks.
But the most controversial tool by far comes after police have made an arrest.
Say hello to criminal risk assessment algorithms!
You may have already spotted the problem. Modern-day risk assessment tools are often driven by algorithms trained on historical crime data. As a result, the algorithm could amplify and perpetuate embedded biases and generate even more bias-tainted data to feed a vicious cycle [3].
Tell a story with dataOr, at the very least, share our data stories more effectively.
(Curated dashboards help guide data analysis, so your data can tell the story you want it to tell).
People hear statistics, but they feel stories.Stories are memorable:
Jennifer Aaker, a marketing professor at Stanford’s Graduate School of Business, had each of her students give a 1-minute pitch.Rather cleverly, Dykes reminded us that, some, people seem to think that crafting a story around their data seem like an unnecessary, time-consuming effort.
Only one in 10 students used a story within his or her pitch, while the others stuck to more traditional pitch elements, such as facts and figures. The professor then asked the class to write down everything they remembered about each pitch:
5 percent of students cited a statistic, but a whopping 63 percent remembered the story [4].
In fact, neuroscientists have confirmed that decisions are often based on emotion, not logic.In his book Effective Data Storytelling, he drives home the message:
E.g. Antonio Damasio has described patients, who had brain damage in an area that helped to process emotions (prefrontal cortex), that struggled to make (even) basic decisions when choosing between alternatives.
Apparently, their basic decision-making skills were significantly impaired by the lack of emotional judgment [5].
See also Review: ''The Feeling of What Happens'' (d).
Stories can influence decisions and drive change. Most other books focus only on data visualization, while neglecting the powerful narrative and psychological aspects of telling stories with data.Still, the stories have to based on (our) data. After all, a non-fact based story is basicly a forgery.
Here it is shown how the three central elements of data storytelling ―data, narrative, and visuals - can be used for maximum effectiveness
(For more, see Twitter, userprofile AnalyticsHero).
You will be the same person in five years as you are today, except for the people you meet and the books you read.Certainly, this Crunch presentation was a beautiful story.
Shirshanka is the architect for LinkedIn’s Data Analytics Infrastructure team. He was among the original authors of a variety of open and closed source projects built at LinkedIn, including Databus, Espresso, and Apache Helix. His current focus at LinkedIn includes all things Hadoop, high-performance distributed OLAP engines, large-scale data... [6].At Linkedin the mission is to map the ''human graph'' (people, jobs, schools, companies) into the digital world. A huge task!
Apache Kafka allows users to subscribe and publish data to any number of systems or real-time applications. E.g. it is used for managing passenger and driver matching at Uber. At LinkedIn it is used for providing real-time analytics and numerous real-time services across the LinkedIn backend. Etc. For more, see here.
A chart must be understandable in under half the time it is on the screen.Which is tricky, as it is likely that ''Your boss has a big ego, so he is running late by 15 minutes. Meaning that you only have 5 minutes for your entire presentation''. So, better keep it (your presentation) relatively simple.
From self-driving cars to smart surveillance cams, society is slowly learning to trust AI over human eyes. But although our new machine vision systems are tireless and ever-vigilant, they’re far from infallible.
By making only subtle changes to the input of a convolutional neural network, the output of the network can be swayed to output a completely different result. The first attacks did this by changing pixel values of an input image slightly to fool a classifier to output the wrong class.
An adversarial patch that is successfully able to hide persons from a person detector [7].
If you print off one of the students specially designed patches and hang it around your neck, from the AIs (YOLOv2) point of view, you may as well have slipped under an invisibility cloak [8].
Many very cool technologies where demo'ed at the conference. Here a YOLOv2-like image
detection system.
The Széchenyi Chain Bridge opened in 1849.
