real time dashboards -> web app -> java/python backend -> twitter
input data rate 10,000 tweet/sec
processing rate 1000 tweet/sec
input data rate <= processing rate
ソフトウェアエンジニアの技術ブログ:Software engineer tech blog
随机应变 ABCD: Always Be Coding and … : хороший
real time dashboards -> web app -> java/python backend -> twitter
input data rate 10,000 tweet/sec
processing rate 1000 tweet/sec
input data rate <= processing rate
Discovery: Ability to identify patterns in data
Communication: Provide insights in a meaningful way
Types of analytics, varieties
Cube Analytics: business intelligence
Predictive analytics: statistics and machine learning
Realtime: ability to analyze the data instantly
Batch: ability to provide insights after several hours/days when a query is posed
Realtime analytics
-streaming
-interactive
OLTP/OLAP
< 500 MS latency sensitive
deterministic workflows
Vagrant, Vagrant clowd, Virtual Box
Ubuntu, git, Maven
Storm
Python, Java, Beautiful Soup
Rdis, Flask, lettuce
d3 viz, Javascript
spout, bolt, topology, java/python, twitter 4, streaming juing, hackathon
closure, cluster administration, ack
image space
y = m0x + b0
Hough (parameter) space
b = -x0m + y0
b = -x1m + y1
polar representation for lines
d: perpendicular distance from line to origin
Θ: angle the perpendicular makes with the x-axis
Hough transform algorithm
xcosΘ- ysinΘ = d
Hough transform for circles
Circle: center(a,b) and radius r (xi-a)^2 + (yi-b)^2 = r^2
For every edge pixel (x,y): For each possible radius value r: For each possible gradient direction Θ: %% or use estimated gradient a = x - r cos(Θ) b = y + r sin(Θ) H[a,b,r] += 1 end end end
f、α^2/αx^2*h
Δ^2 = α^2f/αx^2 + α^2f/αy^2
Gradients -> edges
1. smoothing derivatives to suppress noise and compute gradient.
2. threshold to find regions of “significant” gradient
3. thin to get localized edge pixels
The canny edge detector
% for your eyes only pkg load image; frizzy = imread('frizzy.png'); froomer = imread('froomer.png'); imshow(frizzy); imshow(froomer); frizzy_gray = rgb2gray(frizzy); froomer_gray = rgb2gray(froomer); frizzy_edges = edge(frizzy_gray, 'canny'); froomer_edges = edge(froomer_gray, 'canny'); imshow(frizzy_edges); imshow(froomer_edges); imshow(frizzy_edges & froomer_edges);
f(x)
d/dx*f(x)
Finite differences responding to noise
Increasing noise
Solution: smooth first
f, h, h*f, α/αx*(h*f)
where is the edge?
for sigma = 1:3:10 h = fspecial('gaussian', fsize, sigma); out = imfilter(im, h); imshow(out); pause; end
Differential operators – when applied to the image returns some derivatives.
Model these “operators” as masks/kernels that compute the image gradient function.
Discrete gradient
For discrete data, we can approximate using finite differences.
pkg load image; function result = select_gdir(gmag, gdir, mag_min, angle_low, angle_high) result = gmag >= mag_min endfunction img = double(imread('octagon.png'))/255.; imshow(img); [gx gx] = imgradientxy(img, 'sobel'); imshow((gy + 4) / 8); [gmag gdir] = imgradient(gx, gy); imshow(gmag / (4 * sqrt(2))); my_grad = select_gdir(gmag, gdir, 1, 30, 60);
Origin of Edges
-surface normal discontinuity
-depth discontinuity
-surface color discontinuity
-illumination discontinuity
Edges seem to occur “change boundaries” that are related to shape or illumination.
A stripe on a sign is not such a boundary.
Edge Detection
Basic idea: look for a neighborhood with strong signs of change.
Problems:
– neighborhood size
– how to detect change
intensity function (along horizontal scanline)
first derivative
-edges correspond to extrema of derivative
Scene
Template(mask)
Detected template
Correlation map
pkg load image; function [yIndex xIndex] = find_template_2D(template, img) endfunction tablet = imread('tablet.png'); imshow(tablet); glyph = tablet(75:165, 150:185); imshow(plyph); [y x] = find_template_2D(glyph, tablet); disp([y x]);