Connecting Count Bolt

class:Word Spout
componentId:”word-spout”

builder.setSpout("word-spout", new WordSpout(), 5);

class:CountBolt
componentOd:”count-bolt”

builder.setBolt("count-bolt",
	new CountBolt(), 15)
		.fieldsGrouping("word-spout",
			new Field("word"));

beautiful soup

easy_install beautifulsoup4

Storm & Hadoop

Storm & Hadoop are complimentary!
Hadoop => big batch processing
Storm => fast, reactive, real time processing

Storm data model
-Spouts
->sources of data for the topology (e.g) Postgres/MySQL/Kafka/Kestrel
-Bolts
->units of computation on data (e.g) filtering/aggregation/join/transformations

Live stream of Tweets
tweet spout, parse tweet bolt, word count bolt

Stream grouping
shuffle, fields, all, global

tuble: immutable ordered list of elements
topology: directed acyclic graph, vertices = computation and edges = streams of data

What is analytics?

Discovery: Ability to identify patterns in data
Communication: Provide insights in a meaningful way

Types of analytics, varieties
Cube Analytics: business intelligence
Predictive analytics: statistics and machine learning

Realtime: ability to analyze the data instantly
Batch: ability to provide insights after several hours/days when a query is posed

Realtime analytics
-streaming
-interactive

OLTP/OLAP
< 500 MS latency sensitive deterministic workflows

Hough space

image space
y = m0x + b0

Hough (parameter) space
b = -x0m + y0
b = -x1m + y1

polar representation for lines
d: perpendicular distance from line to origin
Θ: angle the perpendicular makes with the x-axis

Hough transform algorithm
xcosΘ- ysinΘ = d

Hough transform for circles
Circle: center(a,b) and radius r (xi-a)^2 + (yi-b)^2 = r^2

For every edge pixel (x,y):
	For each possible radius value r:
		For each possible gradient direction Θ:
		 %% or use estimated gradient
		 	a = x - r cos(Θ)
		 	b = y + r sin(Θ)
		 	H[a,b,r] += 1
		 end
	end
end

Canny Edge Operator

Gradients -> edges
1. smoothing derivatives to suppress noise and compute gradient.
2. threshold to find regions of “significant” gradient
3. thin to get localized edge pixels

The canny edge detector

% for your eyes only
pkg load image;

frizzy = imread('frizzy.png');
froomer = imread('froomer.png');
imshow(frizzy);
imshow(froomer);

frizzy_gray = rgb2gray(frizzy);
froomer_gray = rgb2gray(froomer);

frizzy_edges = edge(frizzy_gray, 'canny');
froomer_edges = edge(froomer_gray, 'canny');
imshow(frizzy_edges);
imshow(froomer_edges);

imshow(frizzy_edges & froomer_edges);

In the real world

f(x)
d/dx*f(x)

Finite differences responding to noise
Increasing noise

Solution: smooth first
f, h, h*f, α/αx*(h*f)
where is the edge?

for sigma = 1:3:10
	h = fspecial('gaussian', fsize, sigma);
	out = imfilter(im, h);
	imshow(out);
	pause;
end

Gradient

Differential operators – when applied to the image returns some derivatives.
Model these “operators” as masks/kernels that compute the image gradient function.

Discrete gradient
For discrete data, we can approximate using finite differences.

pkg load image;

function result = select_gdir(gmag, gdir, mag_min, angle_low, angle_high)
	result = gmag >= mag_min
endfunction

img = double(imread('octagon.png'))/255.;
imshow(img);

[gx gx] = imgradientxy(img, 'sobel');
imshow((gy + 4) / 8);

[gmag gdir] = imgradient(gx, gy);
imshow(gmag / (4 * sqrt(2)));

my_grad = select_gdir(gmag, gdir, 1, 30, 60);