Ch4-Hadoop MapReduce

-put 等同于 -copyFromLocal
someone@anynode:hadoop$ bin/hadoop dfs –put /home/someone/interestingFile.txt
/user/yourUserName/
Put上传整个目录 someone@anynode:hadoop$ bin/hadoop dfs –put source-directory destination
-test -[ezd] path
Returns 1 if path exists; has zero length; or is a directory, or 0 otherwise.
Prints information about path. format is a string which accepts file size in blocks (%b), filename (%n), block size (%o), replication (%r), and modification date (%y, %Y). Shows the last 1KB of file on stdout.
Multiple FsImage and EditLog Checkpoint
HDFS设计要点
命名空间 副本选择 Rack Awareness 安全模式 刚启动的时候，等待每一个DataNode报告情况 退出安全模式的时候才进行副本复制操作 NameNode有自己的 FsImage和EditLog，前者有自己的文件系统
HDFS文件系统操作命令
-setrep [-R] [-w] rep path Sets the target replication factor for files identified by path to rep. (The actual replication factor will move toward the target over time)
HDFS基本实现构架
MapReduce App App. MapReduce Framework Abstract FileSystem
HDFS (ClietnProtocol,DataNodeProtocol) (NameNode,DataNode,DFSClient)
IPC(Hadoop RPC) TCP Socket JMX
建立用户自己的目录，用户目录在/user中，需要建立
someone@anynode:hadoop$ bin/hadoop dfs -mkdir /user someone@anynode:hadoop$ bin/hadoop dfs -mkdir /user/someone
用-put命令在Linux文件系统与HDFS之间复制数据文件
Retrieves all files that match the path src in HDFS, and copies them to a single, merged file in the local file system identified by localDest.
-cat filename -copyToLocal [-crc] src localDest -moveToLocal [-crc] src localDest
-cp src dest
Copies the file or directory identified by src to dest, within HDFS.
HDFS文件系统操作命令
-rm path Removes the file or empty directory identified by path.
鸣谢：本课程得到Google公司(北京） 中国大学合作部精品课程计划资助
Ch.4. Hadoop MapReduce基本构架
1.Hadoop 分布式文件系统HDFS
2.Hadoop MapReduce的基本工作原理
3.分布式结构化数据表HBase
HDFS的基本特征
模仿Google GFS设计实现 存储极大数目的信息（terabytes or petabytes），将数据保存到
-touchz path
Creates a file at path containing the current time as a timestamp. Fails if a file already exists at path, unless the file is already size 0.
HDFS文件系统操作命令
someone@anynode:hadoop$ bin/hadoop dfs -ls someone@anynode:hadoop$ someone@anynode:hadoop$ bin/hadoop dfs -ls / Found 2 items drwxr-xr-x - hadoop supergroup 0 2008-09-20 19:40 /hadoop drwxr-xr-x - hadoop supergroup 0 2008-09-20 20:08 /tmp
HDFS文件系统操作命令
-get [-crc] src localDest Copies the file or directory in HDFS identified by src to the local file system path identified by localDest.
-getmerge src localDest [addnl]
Displays the contents of filename on stdout. Identical to -get Works like -get, but deletes the HDFS copy on success.
-mkdir path
Creates a directory named path in HDFS. Creates any parent directories in path that are missing (e.g., like mkdir -p in Linux).
HDFS文件系统操作命令
Command: bin/hadoop dfs -put foo bar Assuming: Outcome: No file/directory named Uploads local file foo to a file /user/$USER/bar exists in HDFS named /user/$USER/bar Uploads local file foo to a file named /user/$USER/bar/foo Uploads local file foo to a file named /user/$USER/somedir/somefile, creating the missing directory No change in HDFS, and an error is returned to the user.
HDFS数据分布设计
多副本数据块形式存储，按照块的方式随机选择存储节点 默认副本数目是3
HDFS数据分布设计
HDFS可靠性与出错恢复
DataNode节点的检测 心跳：NameNode 不断检测DataNode是否有效 若失效，则寻找新的节点替代，将失效节点数据重新分布 集群负载均衡 数据一致性: 校验和checksum 主节点元数据失效
状态，后者是还没有更新的记录
HDFS的安装和启动
下载hadoop-0.20.1.tar.gz（或者最新版本0.21） tar zxvf hadoop-0.20.1.tar.gz，解压后Hadoop系统包括HDFS和所
有配置文件都在指定的文件目录中 在Linux下进行必要的系统配置 设置与Hadoop相关的Java运行环境变量 启动Java虚拟机 启动Hadoop，则Hadoop和HDFS文件系统开始运行
减少元数据的量块的方式随机选择存储节点，
默认副本数目是3
HDFS基本构架
对等于GFS Master
应用程序 HDFS客户端
文件名或数据块号
HDFS NameNode
数据块号，数据块位置
对等于GFS ChunkServer DataNode 数据 DataNode 数据 DataNode 数据
-copyFromLocal localSrc dest -moveFromLocal localSrc dest
Copies the file or directory from the local file system identified by localSrc to dest within the HDFS. Identical to -put Copies the file or directory from the local file system identified by localSrc to dest within HDFS, then deletes the local copy on success.
/user/$USER/bar is already a file in HDFS
HDFS文件系统操作命令
-ls path Lists the contents of the directory specified by path, showing the names, permissions, owner, size and modification date for each entry.
-lsr path
-du path -dus path -mv src dest
Behaves like -ls, but recursively displays entries in all subdirectories of path.