博客
关于我
强烈建议你试试无所不能的chatGPT,快点击我
Hadoop快照介绍
阅读量:4351 次
发布时间:2019-06-07

本文共 3971 字,大约阅读时间需要 13 分钟。

快照使用场景:

  1. Protection against user errors(防止用户误操作):  Admin sets up a process to take read-only (RO) snapshots periodically in a rolling manner so that there are always x number of RO snapshots on HDFS. If a user accidentally deletes a file, the file can be restored from the latest RO snapshot that contains the file.
  2. Backup(备份): Admin wants backup the entire file system, a subtree in the file system or just a file. Depending on the requirements, admin takes a read-only (henceforth referred to as RO) snapshot and uses this snapshot as the starting point of a full backup. Incremental backups are then taken by doing a diff between two snapshots.
  3. Experimental/Test setups(进行试验或测试):  A user wants to test an application against the main dataset. Normally, without doing a full copy of the dataset, this is a very risky proposition because the test setup can corrupt/overwrite production data. Admin creates a read-write (henceforth referred to as RW) snapshot of the production dataset and assigns the RW snapshot to the user to be used for experiment. Changes done to the RW snapshot will not be reflected on the production dataset.
  4. Disaster Recovery(灾难恢复):  RO Snapshots can be used to create a consistent point in time image for replication and this can be copied over to remote site for Disaster Recovery.

快照类别:

  1. Read-only (RO) snapshots: These are immutable copies of underlying elements of the file system.
  2. Read-write (RW) snapshots: RW snaps can be modified by a user.

快照的两种实现:

Option #1(namenode与datanode都做快照): Both datanodes and namenode are aware of the snapshots and save state internally about the snapshots. Datanode is aware of the fact that some of the blocks are for the snapshot files.

Option #2(仅namenode做快照): Only namenode is aware of the snapshot. Datanode is not aware of the fact that some of the blocks are owned by snapshots of the original file.

快照操作:

A key requirement is to ensure that it is very easy to create and delete snapshots. Snapshot creation and deletion is an admin-only capability. To create a snapshot, one specifies a  snapshot name, a path to the root of the subtree whose snapshot is to be taken, and whether or not the snapshot is read-only or a read-write. Deleting snapshot requires just a snapshot name. A command to list all the snaps in the filesystem will be provided.(需保证操作简捷,一键完成)

快照使用:

Examples:  Consider a directory structure of /a/b/c/foo.txt. Admin has created a snapshot hdfs1 at /a/b. To access data related to snapshot hdfs1, some examples of the commands would be:

hadoop dfs -ls /a/b/.snapshot_hdfs1/c/foo.txt 

快照说明:

Snapshots are a very useful feature to have in a mature filesystem. This is a work in progress and we have a functional prototype implemented. The first version of this feature will support RO snapshots only. The support for RW snapshots will be added in the subsequent releases. There are several features that can be incorporated into snapshots, such as time to live for snapshots with auto deletion, schedule based creation of snapshots, marking specific directories as snapshot-worthy, quota based restriction on space used by RW snapshots and delegation of authority for creating/deleting snapshots at specific locations to users etc.(目前只有RO模式,RW模式后续开发)

HBASE使用快照代替EXP/CP的好处

Aside from the better consistency guarantees that a snapshot can provide compared to a Copy\/Export Job, the main difference between Exporting a Snapshot and Copying\/Exporting a table is that ExportSnapshot operates at HDFS level. This means that Master and Region Servers are not involved in this operations. Consequently, no unnecessary caches for data are created and there is no triggering of additional GC pauses due to the number of objects created during the scan process. The performance impact on the HBase cluster stems from the extra network and disk workload experienced by the DataNodes.(主要是操作级别的问题,快照不需要在Master与Region服务器上进行,不需要进行垃圾回收等操作)

转载于:https://www.cnblogs.com/jarlean/archive/2013/05/04/3059667.html

你可能感兴趣的文章
关于"无法解析的外部符号"问题的解决
查看>>
【JavaScript】【译】编写高性能JavaScript
查看>>
【随笔】入行必读:互联网行业薪酬等级
查看>>
Android使用开源框架加载图片
查看>>
CLR是怎么加载到内存的?
查看>>
fckeditor
查看>>
backbone.js
查看>>
python类的特殊成员变量
查看>>
sublime text3最新版本注册码(build 3143)
查看>>
linux使用技巧
查看>>
必背公式及常数
查看>>
公平博弈 【2006】二2取石子游戏
查看>>
利用CSS、JavaScript及Ajax实现图片预加载的三大方法
查看>>
大陆手机号正则
查看>>
Navicat连接Oracle11g 错误的解决办法
查看>>
MAVEN 新建Servlet类 找不到 javax.servlet.annotation.WebServlet
查看>>
EntityManager的merge()方法
查看>>
Spring中线程池的应用
查看>>
前端登录jq图形验证码
查看>>
软件设计
查看>>