Given the size of data, there can be several approaches here:
1. Moving the boxes
Not possible, as I suppose the data must be needed for 24x7 analytics.
2. Mirroring the data.
This is a good solution. However, if you have data being written/removed
continuously (if a part of live system), there are chances of losing some
of the data during mirroring happens, unless
a) You block writes/updates during that time (if you do so, that would be
as good as unplugging and moving the machine around), or,
b) Keep a track of what was modified since you started the mirroring
process.
I would recommend you to go with 2b) because it minimizes downtime. Here is
how I think you can do it, by using some of the tools provided by Hadoop
itself.
a) You can use some fast distributed copying tool to copy large chunks of
data. Before you kick-off with this, you can create a utility that tracks
the modification of data made to your live system while copying is going on
in the background. The utility will log the modifications into an audit
trail.
b) Once you're done copying the files, allow the new data store
replication to catch up by reading the real-time modifications that were
made, from your utility's log file. Once sync'ed up you can begin with the
minimal downtime by switching off the JobTracker in live cluster so that
new files are not created.
c) As soon as you reach the last chunk of copying, change the DNS entries
so that the hostnames referenced by the Hadoop jobs points to the new
location.
d) Turn on the JobTracker for the new cluster.
e) Enjoy a drink with the money you saved by not using other paid third
party solutions and pat your back! ;)
The key of the above solution is to make data copying of step a) as fast as
possible. Lesser the time, lesser the contents in audit trail, lesser the
overall downtime.
You can develop some in house solution for this, or use DistCp, provided by
Hadoop that uses copies over the data using Map/Reduce.
On Sat, Aug 4, 2012 at 3:27 AM, Michael Segel michael_se...@hotmail.comwrote:
Sorry at 1PB of disk... compression isn't going to really help a whole
heck of a lot. Your networking bandwidth will be your bottleneck.
So lets look at the problem.
How much down time can you afford?
What does your hardware look like?
How much space do you have in your current data center?
You have 1PB of data. OK, what does the access pattern look like?
There are a couple of ways to slice and dice this. How many trucks do you
have?
On Aug 3, 2012, at 4:24 PM, Harit Himanshu harit.subscripti...@gmail.com
wrote:
Moving 1 PB of data would take loads of time,
- check if this new data center provides something similar to
http://aws.amazon.com/importexport/
- Consider multi part uploading of data
- consider compressing the data
On Aug 3, 2012, at 2:19 PM, Patai Sangbutsarakum wrote:
thanks for response.
Physical move is not a choice in this case. Purely looking for copying
data and how to catch up with the update of a file while it is being
migrated.
On Fri, Aug 3, 2012 at 12:40 PM, Chen He airb...@gmail.com wrote:
sometimes, physically moving hard drives helps. :)
On Aug 3, 2012 1:50 PM, Patai Sangbutsarakum
silvianhad...@gmail.com
wrote:
Hi Hadoopers,
We have a plan to migrate Hadoop cluster to a different datacenter
where we can triple the size of the cluster.
Currently, our 0.20.2 cluster have around 1PB of data. We use only
Java/Pig.
I would like to get some input how we gonna handle with transferring
1PB of data to a new site, and also keep up with
new files that thrown into cluster all the time.
Happy friday !!
P
