- 浏览: 139910 次
- 性别:
- 来自: 深圳
文章分类
- 全部博客 (43)
- web服务器 (1)
- Linux (6)
- MySQL (3)
- xen (1)
- SpringSide (2)
- Windows (1)
- WebService (1)
- Hadoop (12)
- hbase (0)
- sqoop (1)
- Java (1)
- SQL Server 存储过程 xml (0)
- hive mysql (1)
- Eclipse Hadoop 源码 编译 (1)
- Perl (2)
- Shell (1)
- Nutch (1)
- NFS (0)
- CHM (0)
- SVN (1)
- eclipse (1)
- NekoHTML (0)
- MapReduce (0)
- hive (2)
- spring hibernate (0)
- jsp (0)
- CYGWIN (0)
- maven (0)
- selenium server (0)
- CentOS (1)
- hibernate (1)
- spring mvc (0)
- Mahout (0)
- openvpn (0)
- vpn (0)
- kvm (0)
- git (1)
- CPU (1)
- thrift avro (0)
最新评论
-
rmn190:
不错, 多谢分享!
hdfs mount为linux本地文件系统 -
melburg:
请教一下,hadoop 1.0.3版本中,为什么无法启动bac ...
Hadoop SecondaryNameNode备份及恢复
1、同步各个服务器时间
yum install ntp
ntpdate ntp.fudan.edu.cn
hdfs-site.xml配置
如果没有配置这一项,hadoop默认是0.0.0.0:50090,如果你的namenode和secondarynamenode配置在同一台服务器上,是没有问题的。如果分开部署没有指定该值,则hadoop会在namenode本机上找,就会出错
<property>
<name>dfs.secondary.http.address</name>
<value>snn0001:50090</value>
</property>
<property>
<name>dfs.http.address</name>
<value>nn0001:50070</value>
</property>
1、edits和fsimage
NameNode会把用户对FileSystem的操作保存在日志文件edits中
每次NameNode重启时,首先从镜像文件fsimage中读取HDFS数据,并把日志文件合并到fsimage中。
2、checkpoint
有两个参数控制SecondaryNameNode checkpoint
fs.checkpoint.period表示两次checkpoint的时间间隔,默认为3600s
fs.checkpoint.size规定edits文件最大值,超过该值即checkpoint,默认64M
可以通过NameNode的start-dfs.sh启动SecondaryNameNode
也可以通过./hadoop secondarynamenode -checkpoint或者./hadoop secondarynamenode -checkpoint force
3、恢复数据:
配置一台和NameNode一样的服务器
创建dfs.name.dir文件夹,注意:该文件夹不能包含合法的fsimage,否则会执行失败。因为NameNode会检查fs.checkpoint.dir目录下镜像的一致性,但是不会做任何改动。
注意:可以使用nfs备份dfs.name.dir和${hadoop.tmp.dir}/dfs/namesecondary
新建目录/hadoop/dfs/namenode和/hadoop/dfs/secondarynamenode
执行命令:./hadoop namenode -importCheckpoint,NameNode会读取checkpoint文件,保存到dfs.name.dir
出现以下错误:
12/01/24 00:02:56 WARN mortbay.log: /getimage: java.io.IOException: GetImage failed. java.net.ConnectException: Connection refused
这是因为没有在hdfs-site.xml配置dfs.secondary.http.address,上面已经配置了
下面的错误是由于数据块完整率没有达到hadoop规定的0.9990要求,所以namenode处于安全状态
把dfs.repliation值设置为2,重新格式化namenode,并上传数据,再次执行./hadoop namenode -importCheckpoint
在0.21.0中,可以通过Checkpoint Node和Backup Node做checkpoint
以下是官网对SecondaryNameNode的说明
The NameNode stores modifications to the file system as a log appended to a native file system file (edits). When a NameNode starts up, it reads HDFS state from an image file (fsimage) and then applies edits from the edits log file. It then writes new HDFS state to the fsimage and starts normal operation with an empty edits file. Since NameNode merges fsimage and edits files only during start up, the edits log file could get very large over time on a busy cluster. Another side effect of a larger edits file is that next restart of NameNode takes longer.
The secondary NameNode merges the fsimage and the edits log files periodically and keeps edits log size within a limit. It is usually run on a different machine than the primary NameNode since its memory requirements are on the same order as the primary NameNode. The secondary NameNode is started by bin/start-dfs.sh on the nodes specified in conf/masters file.
The start of the checkpoint process on the secondary NameNode is controlled by two configuration parameters.
- fs.checkpoint.period, set to 1 hour by default, specifies the maximum delay between two consecutive checkpoints, and
- fs.checkpoint.size, set to 64MB by default, defines the size of the edits log file that forces an urgent checkpoint even if the maximum checkpoint delay is not reached.
The secondary NameNode stores the latest checkpoint in a directory which is structured the same way as the primary NameNode's directory. So that the check pointed image is always ready to be read by the primary NameNode if necessary.
The latest checkpoint can be imported to the primary NameNode if all other copies of the image and the edits files are lost. In order to do that one should:
- Create an empty directory specified in the dfs.name.dir configuration variable;
- Specify the location of the checkpoint directory in the configuration variable fs.checkpoint.dir;
- and start the NameNode with -importCheckpoint option.
The NameNode will upload the checkpoint from the fs.checkpoint.dir directory and then save it to the NameNode directory(s) set in dfs.name.dir. The NameNode will fail if a legal image is contained in dfs.name.dir. The NameNode verifies that the image in fs.checkpoint.dir is consistent, but does not modify it in any way.
评论
发表评论
-
升级hadoop
2013-10-09 10:21 1406在没有更换前先备份数据 [hadoop@Hadoop-1 ... -
hadoop的Avatar机制
2013-08-21 15:45 0http://www.wangyuxiong.com/arch ... -
hive
2013-07-31 14:12 0hive行号 select row_number() o ... -
Hadoop RACK ID Awareness Configuration
2013-01-25 17:21 0The configuration includes ... -
window hadoop
2012-12-28 14:31 0http://hayesdavis.net/2008/06/1 ... -
hadoop读写流程
2012-07-27 15:20 0客户端通过调用FileSystem ... -
hadoop三个配置文件的参数含义说明
2012-07-14 13:03 02 常用的端口配置 2.1 HDFS端 ... -
编译hadoop 1.0.3 eclipse plugin jar包
2012-09-13 10:32 2329环境:Win 7 32bit 1、修改hadoop- ... -
编译hadoop 1.0.3 eclipse plugin jar包
2012-07-07 23:21 3928环境:Win 7 32bit 1、修改hadoop-1. ... -
hadoop fsck使用
2012-05-08 15:05 0首先,执行fsck命令行的客户端必须在hdfs-site.xm ... -
hive使用
2012-05-03 17:33 0[root@cnn001 hive-0.8.1]# bin/h ... -
AvatarNode
2012-04-24 13:28 0http://blog.csdn.net/rzhzhz/art ... -
hdfs mount为linux本地文件系统
2012-03-21 00:08 44951、软件下载 hdfs-webdav.war http:/ ... -
扩展hadoop hdfs,实现WebDav协议,将hdfs mount为linux本地文件系统
2012-03-15 16:18 1518本文引自:http://badqiu.iteye.com/bl ... -
Hadoop MapReduce统计指定目录下所有文本文件中数字总和
2012-03-06 16:16 0package org.apache.hadoop.exa ... -
Hadoop Zookeeper HBase集群
2012-02-16 16:19 9167注:Hadoop环境搭建请参考上一篇文档。 环境: ... -
Hadoop NameNode NFS备份及恢复
2012-02-16 14:28 1484准备任意一台Linux服务器 [root@localhost ... -
Hadoop 0.20.205.0安装配置
2012-02-15 15:55 1241环境: 10.0.30.235 NameNode ... -
hadoop mapred-default.xml配置文件
2012-02-15 13:25 4167name value description ... -
hadoop hdfs-default.xml配置文件
2012-02-15 13:05 4122name value description d ...
相关推荐
基于CommVault的Hadoop大数据平台备份恢复方案.pptx
snapshot为hadoop2.1.0时新增加的功能。 主要为防止用户误删数据,和数据备份的作用 快照的对象为HDFS目录,前提是该目录是可以允许设置快照的(SnapShotable)
Windows下hadoop3.1.0 64位hadoop.dll和...start "Apache Hadoop Distribution" hadoop secondarynamenode Taskkill /FI "WINDOWTITLE eq Apache Hadoop Distribution - hadoop secondarynamenode"。 hadoop winutils。
Hadoop集群配置文件备份
《Hadoop系统搭建及项目实践》课程标准.pdf《Hadoop系统搭建及项目实践》课程标准.pdf《Hadoop系统搭建及项目实践》课程标准.pdf《Hadoop系统搭建及项目实践》课程标准.pdf《Hadoop系统搭建及项目实践》课程标准.pdf...
《Hadoop系统搭建及项目实践》课件05Hadoop IO操作.pdf《Hadoop系统搭建及项目实践》课件05Hadoop IO操作.pdf《Hadoop系统搭建及项目实践》课件05Hadoop IO操作.pdf《Hadoop系统搭建及项目实践》课件05Hadoop IO操作...
《Hadoop系统搭建及项目实践》课件02Hadoop安装与配置管理.pdf《Hadoop系统搭建及项目实践》课件02Hadoop安装与配置管理.pdf《Hadoop系统搭建及项目实践》课件02Hadoop安装与配置管理.pdf《Hadoop系统搭建及项目实践...
基于Hadoop的研究及性能分析.pdf基于Hadoop的研究及性能分析.pdf基于Hadoop的研究及性能分析.pdf基于Hadoop的研究及性能分析.pdf基于Hadoop的研究及性能分析.pdf基于Hadoop的研究及性能分析.pdf基于Hadoop的研究及...
《Hadoop系统搭建及项目实践》课件10Hadoop 与RDMS数据迁移工具Sqoop.pdf《Hadoop系统搭建及项目实践》课件10Hadoop 与RDMS数据迁移工具Sqoop.pdf《Hadoop系统搭建及项目实践》课件10Hadoop 与RDMS数据迁移工具Sqoop...
《Hadoop系统搭建及项目实践》课件04MapReduce技术.pdf《Hadoop系统搭建及项目实践》课件04MapReduce技术.pdf《Hadoop系统搭建及项目实践》课件04MapReduce技术.pdf《Hadoop系统搭建及项目实践》课件04MapReduce技术...
Cloudera Hadoop 5&Hadoop高阶管理及调优课程,完整版,提供课件代码资料下载。 内容简介 本教程针对有一定Hadoop基础的学员,深入讲解如下方面的内容: 1、Hadoop2.0高阶运维,包括Hadoop节点增加删除、HDFS和...
整理的一些hadoop面试题及答案,希望对你有帮助 整理的一些hadoop面试题及答案,希望对你有帮助
在windows环境下开发hadoop时,需要配置HADOOP_HOME环境变量,变量值D:\hadoop-common-2.7.3-bin-master,并在Path追加%HADOOP_HOME%\bin,有可能出现如下错误: org.apache.hadoop.io.nativeio.NativeIO$Windows....
安装hadoop的时候或者使用的时候,会出现hadoop常见问题及解决方法
Hadoop及源码 Hadoop及源码 Hadoop及源码 Hadoop及源码
Hadoop的背景及应用现状分析:为了更为经济的从高频率获取的、大容量的、不同结构和类型 的数据中获取价值,而设计的新一代架构和技术。
Hadoop产品介绍及典型案例,大数据的一些案例,
在网上搜集的以及本人自己总结的hadoop集群常见问题及解决办法,融合了网上常常搜到的一些文档以及个人自己的经验。