ソフトウェアエンジニアの技術ブログ：Software engineer tech blog

spow workers assign mappers
assign reducers

Issues to be handled by the run time
Master data structures
– location of files created by completed mapper
– score board of mapper/reducer assignment
– fault tolerance
* start new instances if no timely resonse
* completion message from redundant stragglers
– locality management
– task granlarity
– backup tasks

Content Delivery Networks
Napster started for music sharing

content hash, node-id where content stored
content-hash = 149 => <149, 80>

Name space {key-space, node-space}
content -> sha-1 -> unique 160-bit key, unique 160-bit node id
Objective:
-> node id such that –
APIs
– put key, getkey

CDN – an overlay network
routing table at A, name, node id, next

multimedia API
middleware
commodity OS

Parallel program
-pthreads: API for parallel programs
Distributed programs
-sockets: API for distributed programs
Conventional Distributed program
client – unix(socket) – server e.g. NFS

Novel multimedia Apps
sensor-based: sensors, distributed
sense -> prioritize -> process -> actuate
computationally intensive, real time

large-scale situation awareness

PTS Programming Model
Channel ch1 =
lookup(“video channel”)
while(1){
// get data
response γ =
ch1.get()
// process data
…
// produce output
ch2.put(item,)
}

RioVista
– performance-conscious design of persistent memory
Quicksilver
– making recovery a first class citizen in os design

LRVM Revisited
begin-xact, nomal program writes, end-xact
cpu, undo <- memory, cpu memory, memory data reda CPU, application memory, File cache mmap => normal application memory secomes rersistent

Crash Recovery
Treat like abort
– recover old image from undo log -> servives crashes since it is in RIO file cache
Crash during crash recovery?
– idempotency of recovery -> no problem

Server design
-persistent metadata m1, m2, ..mm
-normal data structures + code
=>
create external data segments to back persistent data structures
-apps manage their persistence needs

Designer’s choice to use single or multiple data segment

RVM Primitives
Initialization
– initialize(options)
– map(region, options)
– unmap(region)

Body of server code
– begin_xact (tid, restore_mode)
– set_range(tid, addr, size)
– end_xact(tid, connect, t_mode)
– abort_xact(tid)

GC to reduce log space
– flush()
– truncate()

bigin_xact(tid,mode);
	set range(tid, base_addr, #bytes);
	write meta data m1
	write meta data m2
end_xact(tid, mode);

No action by LRVM
LRVM create redo log in memory

Redo log
-AV changes of different region between begin and end xact

Commit
Redo log, window of vulnerability

Log truncation
apply to data seg, read redo log

Client: memory, log seq, log fragments parity, LAN, storage services

Stripe Group, stripe group for x, y, z, L, M, N
-subset server into stripe group
-parallel client activities
-increased availability
-efficient log cleaning

Cache coherence
-single writer, multiple reader
-fileblock unit of coherence
write- request
-receive token
-manager revokes

Log cleaning
log segment evolution, writes to file blocks
– may belong to different files

Unix File System
{filename, offset} -> i-node -> data blocks on disk

Client node action
filename -> mmap -> metadata manager
Manager node actions
filename -> File Dir -> i-number -> i-map -> i-node a dir -> stripe group map -> storage server -> index node of logseg id -> stripe group map -> storage servers -> data blocks form -> storage severs

LRVM
– persistent memory layer in support of system service
Riovista
– performance conscious design of persistent – memory
Quicksilver
– making recovery a first clacks citizen in os design

Persistence
– need of os subsystems
– make virtual memory persistent
– subsystem designers 4 is performant
– use persistent logs to record changes to VM
VM -> replace VM

Write(x)
DSM + OS coop x -> twin, original writable
Write project after release Twin, X’ diff run length

Non-page-based DSM
shared r/w -> dsm software -> <-data library-based -annotate shared variables -coherence achons inserted at point of access API calls lang runtime structured DSM -APU for struct -coherence action on app calls DSM and speedup LAN - DSM library - p -> mem

First NFS
-SUN, 1985

Client, LAN, Servers(student), file cache

DFS
no central server
-each file distributed across several nodes
DFS implemented across all disks in the network

Preliminaries: Striping a file to multiple disks
-increase I/O bandwidth by striping to parallel disks
-Failure protection by ECC

Log Structured File System
– Buffer changes to multiple files in one contiguous log segment data structure
– log segment to disk once a file up or periodically solves small write program
mods to x, mods to y, log segment written the disk

Eebra File System(uc-berkeley)
-combines LFS + RAID
use commodity hardware
-stripe by segment on multiple nodes’ disks in software

xFS – a DFS
-log based striping, cooperative caching, dynamic manageent of data, subsetting storage servers, distributed log cleaning

Traditional NFS with centralized server(s)
– memory content
* metadata, file cache, client caching divectory