ソフトウェアエンジニアの技術ブログ：Software engineer tech blog

Objectives
1. Rays to pixels
2. A camera without optics
3. Lens in the camera system
4. The lens equation

Context of computational photography
3D scene, illumination, optics, sensor, processing, display, user

Rays -> pixel
optics, sensor, image processing, computer graphics, computer vision

Scene via a 2D array of pixels
rays are fundamental primitives
illumination follows a path from the source to the scene
computation can control the parameters of the optics, sensor and illumination

ViewFinder
Focus or Zoom Ring
Shutter Release
Frontal Glass Lens
Diaphragm
Photographic film
Focal Plane Shutters

Pinhole photograph
-heoretically
no distortion straight lines remain straight
infinite depth of field: everything in focus(but there is optical blurring)

1. smoothing suppress noise
2. Compute gradient
3. Apply Edge “enhancement”
4. Edge localization
1. Edges vs. noise
5. Threshold, Thin

Canny Edge Detector
1. filter image with derivative of Gaussian
2. Find magnitude and orientation of gradient
dx, dy, mag of gradient, theta

1. compute edges
2. derivatives using kernels and neighborhood operations
3. three methods for computing edges using kernels
4. image noise can complicate the computation of gradient
5. the canny edge detector

Derivatives as a local product
δF(x,y)/δx = F(x+1,y) – F(x,y)
=[-1 1]*[F(x,y), F(x+1,y)]^T

Desired: An “operator” that effectively compute discrete derivative values with cross-correlation(i.e. using finite differences*)

Hx, Hy

Average of “left” and “right” derivatives

Impact of Noise on Gradients
It gets harder to detect an edge when there is significant noise in the signal.
original signal no noise
original signal + 0.1 x noise
original signal + 0.25 x noise
BWB, increse noise

Convolution and Gradients
recall, convolution is G = h * F
derivative of a convolution δG/δx = δ/δx(h*F)
If D is a kernel to compute derivatives and H is the Kernel for smoothing…

Features
– parts of an image that encode it in a compact form

Edges
edges in an image
surface normal, depth, surface color, illumination
information theory view: edges encode change, therefore edges efficiently encode an image

Edges appear as ridges in the 3D height map of an image

Edge Detection
Basic Idea
Look for a neighborhood with strong signs of change
Derivatives of F(x,y) to get Edges

Need an operation that when applied to an image returns its derivatives
model these “operatiors” as mask/kernel
then “threshold” this gradient function to select edge pixels

Gradient of an image = measure of change in Image function.
ΔF = [δF/δx,δF/δy]
measure of change in image function (F) in x and y
Gradient direction is
Θ = tan^-1[δF/δx,δF/δy]

1.cross-correlation
2.convolution
3.difference between cross-correlation and convolution
4.properties of these methods!

A mathematical representation for smoothing
F[3,3]
F[i,j]

In signal processing, cross-correlation is a measure of similarity of two waveforms as a function of a time-lag applied to one of them.
Also known as a sliding dot product or sliding inner-product

Cross-Correlation method
G[i,j] = kΣu=-k*kΣv=-k*h[u,v]F[i+u,j+v]

Box Filter
Gaussian Filter
size:21×21
Values: Gaussian or Normal distribution
σ1, 3, 6, 9
F[i,j], h[i,j], G[i,j]
G[i,j] = kΣu=-k*kΣv=-k*h[u,v]F[i-u,j-v]
Denoted by G = h * F
Flip filter in both dimensions
bottom to top
right to left
Then apply cross-correlation

Linear and shift invariants
behaves the same everywhere
the value of the output depends on the pattern in the image neighborhood, not the position of the neighborhood

identity: unit umpulse
E = […0,0,1,0,0…]
F * E = F
true of cross-correlation

Separable
if the filter is separable, convolve all rows, then convolve

0, 0, 0
0, 1, 0
0, 0, 0
x 2 =
1/9, 1/9, 1/9
1/9, 1/9, 1/9
1/9, 1/9, 1/9

1.Smooth an image over a neighborhood of pixels
2.Median filtering as a special non-linear filtering and smoothing approach

Determine the average of neighboring values
1. moving average [1 1 1 1 1] x 1/5
2. weighted moving average [1 4 6 4 1] x 1/16

smoothing process over an image using averages

some observation
-small image “rubbed” over bigger image
kernel h(i,j)
3×3 area around each original pixel used (i.e., neighborhood size k=1)
window size is 2k+1, therefore our wind is 3×3
G[i,j] = 1/(2k+1)^2 ΣΣF[i+u,j+v]

Special Case: Median Filtering
F[i,j]
median and average is changed value
Apply median filter to all 3*3 regions over the whole image
Median filtering: nonlinear operation often used in image processing

reduces noise, but
preserves edges(sharp lines!)
main idea: use median of all pixel in area, instead of mean

Median Filtering for noise removal

-Divide (brightens photos)
-Addition (too many whites)
-Subtract (too many blacks)
-Difference (subtract with scaling)
-Darken fblend(a,b)=min(a,b) for RGB
-Lighten fblend(a,b)=max(a,b) for RGB

multiply fblend(a,b)=ab
darker
Screen fblend(a,b)=1-(1-a)(1-b)
brighter

Dodge and burn
-dodge and burn change the lightness of the pictures
-dodging lightens an image, while burning darkens it
-dodge builds on screen mode
-burn build on multiply mode
-there are numerous variations of each!

Darken

-a real life example of point arithmetic
-variety of blending modes build on concept of point processes

a bit about the setup
the final output is a pixel blend
-> screen capture + camera

Blending Pixels
-blend two pixels from two images: fblend(a,b)
For example
-average: fblend(a,b)=(a+b)/2 : lighter color
-normal: fblend(a,b)=b

-order of color channels
-image compression information
-matadata about pictures
(Exchangeable image file format EXIF, etc.)

point-process computations
add/subtract images
a-blending & its applications
image histograms

i(x,y)

How to obtain discrete values?
– sample the two-dimensional (2D) space on a regular grid
– quantize each sample (round to “nearest integer”)
Result: matrix of integer values (range, e.g. 0-255)

Point-Process: Pixel/Point Arithmetic
subtract value, if it is negative, it would be Black
image 1 – image 2
binary(image 1 – image 2)

CD+AE+LD
-> whole are become over 255

alpha-blending
.34 CD + .34 AE + .34 LD
transparency(conversely, opacity)
usually represented as α
α　varies from 0 to 1.0
RGB -> αRGB 0-1, 0-255

PImage img;

void setup(){
	size(345, 567);
	// The image file must be in the data  folder of the current sketch
	// to load successfully
	img = loadImage("buzz3.jpg"); // Load the image into the program
}

void draw(){
	// display the image at its actual size at point(0,0)
	image(img, 0, 0);
	// Displays the image at point(0, height/2) at half of its size
	// image(img, 0, height/2, img.width/2, img.height/2);
}

openCV
here you can download
http://opencv.org/releases.html
http://opencv.jp/

in python-openCV

import cv2
img = cv2.imread('input.png')
cv2.imwrite('output.png', img)

img = imread('input.png')
imwrite('output.png', img)

imshow(img)
imageinfo('input.png')
help images