Category: Machine Learning

bss +ica -pca
directional +ica -pca
faces
natural sene ->ica <-edge documents->topics

information theory
x1, x2, x3 -> learner -> y

010101
A 0, B 110, C 111, D 10

Joint Entropy
H(x,y)= -ΣP(x,y)logP(x,y)

Conditional Entropy
Hcy(x) = -Σp(x,y)logP(y|x)

H(y|x)
I(x,y)=H(y)-H(x|y)

D(p||g) = sp(x)log p(x)/q(x)

Independent components anarysis

-pca correlation, maximizing rariance
-ica independence
x1 x2 x3 -> y1 y2 .. yi

I(yi:yj)=0
I(y・x)=1

Feature transformation

-information retrieval
features? words
->polysemous:car
->synonomy:car

data mining, LISP

principal components anarysis
-maximizes variance

global algorithm
-best reconstruction
min errors moving from N -> m dimensions

E[Zij] = P(x=xi|μ-μj)/kΣi=1 P(x=xi|μ=μj)
Mj = ΣiE[Zij]xi/ΣiE[Zij]
Expectation(define z from μ), Maximization(define μ from z)

Clustering properties Pd<-clustering scheme ・Richness ・Scale-variance ・Consistency

k-means clustering
-pick k centers(at random)
-each center “claims” its closest points
-recompute the centers by overaging the clustered points
-repeat until convergence

P+(x): Partition / cluster of object x
c+ : set of all points in cluster i = {x s.t. P(x)=i}
center+i = ΣyeCi y / |Ci|

P+(x) = argmin||x-center+-1||2

K-means as optimization
configurations -center, P
scores – E(P,center) = Σx||centerp(x)-x||22
neighborhood-p,center= {(p’, center)}U {(P, center’)}

Properties of k-means clustering
-each iteration polynomial o(k n)
-finite(exponential) iterations o(kn)
-error decreases(if ties broken consistently)[with one exception]
-can get stuck!

-consider each object a cluster (n objects)
-define intercluster distance as the distance between the closest two points in the two clusters
-merge two closest clusters
-repeat n-k times to make n clusters

-mimic does well with structure
-representing PΘ Vo
-local optima
-probability theory
-time complexity

Clustering & EM
unsupervised learning
– supervised learning use labeled training data to generalize labels to new instances
– unsupervised learning make sense out of unlabeled data

Basic clustering problem
given: set of objects X
inter-object distances D(x,y) = D(y,x) x, yeX
output: partition pd(x) = pd(y)
if x &y in same cluster

Dkl(P||P^π)= Σp[lgp – lg^p]
= -h(p)+Σh(x1π(xi))
Jπ=Σh(xi|π(x))
J’ = ΣI(xi:π(xi))

Generate samples from p(x)Θ
set Θt+1 to rfth percentile
retain only those samples s.b. f(x)>= Θt
estimate p(x)Θbth
repeat

Population of individuals
Mutation – local search
Cross over – population holds information
Generations – iterations of improvement

GA Skeleton
P0 = initial population of size K
repeat until converged
compute fitness of all exP+
select “most fit” individuals (top half, weighted prab)
pair up individuals, replacing “least fit” individuals
vra crossover/mutiation

Randomized Optimization: mimic
– only points, no structure
– unclear probability distribution

minic: estimating distinations
p(x) = p(x1|x2…xn)p(x2|x3…xn)…p(xn)
x=[x1, x2, x3…xn]

x = {1,…, 100}
f(x) = (x mod 6)2 mod 7 – sin(x)
x = 11

X = R
f(x) = -x4 + 1000×3 – 20×2 + 4x -6

optimization approaches
– generate & test: small input space, complex function
– calculus: function has derivative
– Newtons method
what if assumptions don’t hold?

big input space, complex function, no derivative (or hard to find)
possibly many local oftimg

Randomize optimization

Hill climbing
Guess xe x
repeat
let n = argmax f(n)
Un N(x)
if f(x) > f(x):x = n
else: stop

Thinking of a 5-bit sequence
f(x) = #correct bit
x: 5-bit sequences