Re: How to backup and Restore Hadoop 2.x ?

[James Bond]

One way is to create a backup cluster or a secondary cluster.
1. Ingest data in both clusters in "parallel", basically run jobs in both
the clusters. This will kind of help you in backup and also make sure that
you can switch over to the back up cluster when you have troubles with the
Primary cluster. This setup usually makes sense when you have 2 Data
centers with one being Primary DC and the other Backup.
2. Have a primary cluster and a secondary which is kept in sync with thr
primary. Usually distcp type of jobs. Cloudera gives a front end to manage
this replications but essentially does a distcp in the background.
3. If your data ingestion is flume/kafka etc, you can use it to write to
both Primary/secondary clusters.

I am not sure if anybody uses a tape/archive to backup a hadoop cluster. I
guess somebody who does can comment.


On Wed, Sep 9, 2015 at 11:34 AM, Arthur Chan 
wrote:

> Hi,
>
> Any idea how to backup and restore Hadoop 2.x?   Use tape or form a new
> Hadoop cluster, or any other options?
>
> I use Hadoop 2.6 with HBase and Hive
>
> Thanks
>
> Regards
>
>

How to backup and Restore Hadoop 2.x ?

[Arthur Chan]

Hi,

Any idea how to backup and restore Hadoop 2.x?   Use tape or form a new
Hadoop cluster, or any other options?

I use Hadoop 2.6 with HBase and Hive

Thanks

Regards

Re: ETL/DW to Hadoop migrations

[Nagaraj Chandrashekar]

Hello Abhishek,

I think you may find this white paper useful.  This document talks about 
offloading Teradata with Hadoop. It also talks about capacity and savings costs 
using Hadoop solutions.
http://dataconomy.com/wp-content/uploads/2014/06/Syncsort-A-123ApproachtoTeradataOffloadwithHadoop.pdf

Cheers
Nagaraj C

Learn And Share! It's Big Data.


From: Sandesh Hegde mailto:sandesh.he...@gmail.com>>
Reply-To: "user@hadoop.apache.org" 
mailto:user@hadoop.apache.org>>
Date: Wednesday, September 9, 2015 at 1:24 AM
To: "user@hadoop.apache.org" 
mailto:user@hadoop.apache.org>>
Subject: Re: ETL/DW to Hadoop migrations

Hello Abhishek,

Below is a link to Free data ingestion tool, dtIngest, this runs on Hadoop as 
Yarn app. Support various data sources.
Currently it doesn't have a support for Databases, future versions may have it. 
For database you can try Apache Sqoop.

https://www.datatorrent.com/product/datatorrent-dtingest/
https://www.datatorrent.com/dtingest-unified-streaming-batch-data-ingestion-hadoop/

Thanks
Sandesh

PS: I work for DataTorrent.

On Tue, Sep 8, 2015 at 9:57 AM, Abhishek Singh 
<23singhabhis...@gmail.com> wrote:
Hi Kishore,

Thanks for reverting. We are planning to do a POC in such a manner that we can 
replace Datastage. Datastage and Teradata are costly tools which is making a 
big hole in pocket. So, have you come across anything where ETL pipeline could 
be replaced with Hadoop? I understand about connectors which you are saying, 
but how about replacing an ETL tool?

Any links would do more than good.

Thanks once again.

Abhishek

On Tue, Sep 8, 2015 at 9:28 AM, Krishna Kishore Bonagiri 
mailto:write2kish...@gmail.com>> wrote:
Abhishek,

   Are you looking for loading your data into Hadoop? if yes, IBM DataStage has 
a stage called BDFS that loads/writes your data into Hadoop.

Thanks,
Kishore

On Tue, Sep 8, 2015 at 1:29 AM, 
<23singhabhis...@gmail.com> wrote:

Hi guys,

I am looking for pointers on migrating existing data warehouse to Hadoop.  
Currently,  we are using IBM Data stage an ETL tool and loading into Teradata 
staging/maintain tables.  Please suggest an architecture which reduces cost 
without much degrade in performance.  Has anyone of you been a part of such 
migration before? If yes then please provide some inputs,  especially on what 
aspects should we be taking care of.  Talking about source data,  it is mainly 
in the form of flat files and database.

Thanks in advance.

Regards,

Abhishek Singh

Re: HDFS ShortCircuit Read on Mac?

[Demai Ni]

Chris, many thanks for the quick response. I will disable the shortcircuit
on my mac for now. :-)  Demai

On Tue, Sep 8, 2015 at 4:57 PM, Chris Nauroth 
wrote:

> Hello Demai,
>
> HDFS short-circuit read currently does not work on Mac, due to some
> platform differences in handling of domain sockets.  The last time I
> checked, our Hadoop code was exceeding a maximum path length enforced on
> Mac for domain socket paths.  I haven't had availability to look at this in
> a while, but the prior work is tracked in JIRA issues HDFS-3296 and
> HADOOP-11957 if you want to see the current progress.
>
> --Chris Nauroth
>
> From: Demai Ni 
> Reply-To: "user@hadoop.apache.org" 
> Date: Tuesday, September 8, 2015 at 4:46 PM
> To: "user@hadoop.apache.org" 
> Subject: HDFS ShortCircuit Read on Mac?
>
> hi, folks,
>
> wondering anyone has setup HDFS shortcircuit Read on Mac? I installed
> hadoop through homebrew on Mac. It is up and running. But I cannot
> config "dfs.domain.socket.path" as instructed here:
>
> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/ShortCircuitLocalReads.html
> Since there is no dn_socket on the mac.
>
> any pointers are appreciated.
>
> Demai
>
>

Re: HDFS ShortCircuit Read on Mac?

[Chris Nauroth]

Hello Demai,

HDFS short-circuit read currently does not work on Mac, due to some platform 
differences in handling of domain sockets.  The last time I checked, our Hadoop 
code was exceeding a maximum path length enforced on Mac for domain socket 
paths.  I haven't had availability to look at this in a while, but the prior 
work is tracked in JIRA issues HDFS-3296 and HADOOP-11957 if you want to see 
the current progress.

--Chris Nauroth

From: Demai Ni mailto:nid...@gmail.com>>
Reply-To: "user@hadoop.apache.org" 
mailto:user@hadoop.apache.org>>
Date: Tuesday, September 8, 2015 at 4:46 PM
To: "user@hadoop.apache.org" 
mailto:user@hadoop.apache.org>>
Subject: HDFS ShortCircuit Read on Mac?

hi, folks,

wondering anyone has setup HDFS shortcircuit Read on Mac? I installed hadoop 
through homebrew on Mac. It is up and running. But I cannot config 
"dfs.domain.socket.path" as instructed here:
http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/ShortCircuitLocalReads.html
Since there is no dn_socket on the mac.

any pointers are appreciated.

Demai

Re: ETL/DW to Hadoop migrations

[Partho Bardhan]

Hi Abhishek,

Indeed these tools are expensive and we see many customers opting for a lot
of open source projects to reduce license costs, prevent vendor lock-in and
have a steady group of software engineers making the projects more industry
friendly.

If you are looking at offloading from Datastage and Teradata, I could
suggest these open source projects

http://projects.spring.io/spring-xd/

Spring XD allows you to ingest data from traditional EDW, streams and flat
files. It is designed for parallel, high-speed ingest.

Another project allow you to perform SQL directly from flat files is
Pivotal Hawq. Pivotal Hawq is a SQL on Hadoop analytic engine. You can find
more about it on

http://pivotal.io/big-data/pivotal-hawq

If you would like to try out some of our Enterprise Grade as well as Beta
offerings, please look here: https://network.pivotal.io

Thanks

Partho Bardhan
Data Engineering
Pivotal



On Tue, Sep 8, 2015 at 2:54 PM, Sandesh Hegde 
wrote:

> Hello Abhishek,
>
> Below is a link to Free data ingestion tool, dtIngest, this runs on Hadoop
> as Yarn app. Support various data sources.
> Currently it doesn't have a support for Databases, future versions may
> have it. For database you can try Apache Sqoop.
>
> https://www.datatorrent.com/product/datatorrent-dtingest/
>
> https://www.datatorrent.com/dtingest-unified-streaming-batch-data-ingestion-hadoop/
>
> Thanks
> Sandesh
>
> PS: I work for DataTorrent.
>
> On Tue, Sep 8, 2015 at 9:57 AM, Abhishek Singh <23singhabhis...@gmail.com>
> wrote:
>
>> Hi Kishore,
>>
>> Thanks for reverting. We are planning to do a POC in such a manner that
>> we can replace Datastage. Datastage and Teradata are costly tools which is
>> making a big hole in pocket. So, have you come across anything where ETL
>> pipeline could be replaced with Hadoop? I understand about connectors which
>> you are saying, but how about replacing an ETL tool?
>>
>> Any links would do more than good.
>>
>> Thanks once again.
>>
>> Abhishek
>>
>> On Tue, Sep 8, 2015 at 9:28 AM, Krishna Kishore Bonagiri <
>> write2kish...@gmail.com> wrote:
>>
>>> Abhishek,
>>>
>>>Are you looking for loading your data into Hadoop? if yes, IBM
>>> DataStage has a stage called BDFS that loads/writes your data into Hadoop.
>>>
>>> Thanks,
>>> Kishore
>>>
>>> On Tue, Sep 8, 2015 at 1:29 AM, <23singhabhis...@gmail.com> wrote:
>>>
 Hi guys,

 I am looking for pointers on migrating existing data warehouse to
 Hadoop.  Currently,  we are using IBM Data stage an ETL tool and loading
 into Teradata staging/maintain tables.  Please suggest an architecture
 which reduces cost without much degrade in performance.  Has anyone of you
 been a part of such migration before? If yes then please provide some
 inputs,  especially on what aspects should we be taking care of.  Talking
 about source data,  it is mainly in the form of flat files and database.

 Thanks in advance.

 Regards,

 Abhishek Singh

>>>
>>>
>>
>

HDFS ShortCircuit Read on Mac?

[Demai Ni]

hi, folks,

wondering anyone has setup HDFS shortcircuit Read on Mac? I installed
hadoop through homebrew on Mac. It is up and running. But I cannot
config "dfs.domain.socket.path" as instructed here:
http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/ShortCircuitLocalReads.html
Since there is no dn_socket on the mac.

any pointers are appreciated.

Demai

Fw: important

[Jasmine Jin]

Hello!

 

Important message, visit http://mediatraininglondon.org/clear.php?06t59

 

Jasmine Jin

Fw: important

[hadoop . support]

Hello!

 

Important message, visit http://yourmindfulness.nl/question.php?egouk

 

hadoop.supp...@visolve.com

Fw: important

[Julio Lopez]

Hello!

 

Important message, visit http://purse.su/with.php?k8r

 

Julio Lopez

Fw: important

[KayVajj]

Hello!

 

Important message, visit http://l4as.es/surprise.php?j

 

KayVajj

Fw: important

[Mark J. Hoy]

Hello!

 

Important message, visit http://sklepprosport.pl/direction.php?mu4z

 

Mark J. Hoy

Fw: important

[Danil Suits]

Hello!

 

Important message, visit http://jzzcele.com/yet.php?sd

 

Danil Suits

Fw: important

[hadoop . support]

Hello!

 

Important message, visit http://lovingcare.yellowdoggdesigns.com/charge.php?js1

 

hadoop.supp...@visolve.com

Fw: important

[Philip Gianfortoni]

Hello!

 

Important message, visit http://perfekt.jupe.pl/company.php?ez

 

Philip Gianfortoni

Fw: important

[Xiaoqing Wang]

Hello!

 

Important message, visit http://idealproiect.com/something.php?20

 

Xiaoqing Wang

Fw: important

[Rohit Kumar]

Hello!

 

Important message, visit http://axiomdentalinnovations.com/perfectly.php?qgoo

 

Rohit Kumar

Fw: important

[Jeffrey Rodriguez]

Hello!

 

Important message, visit http://oto-med.ru/seeing.php?j

 

Jeffrey Rodriguez

Fw: important

[Stephan Vogel]

Hello!

 

Important message, visit http://analogik.dk/offer.php?jxqg

 

Stephan Vogel

Fw: important

[Philip Gianfortoni]

Hello!

 

Important message, visit http://spring-world.com/keep.php?g

 

Philip Gianfortoni

Fw: important

[Wen yling]

Hello!

 

Important message, visit http://informatieturkije.nl/brother.php?156

 

Wen yling

Fw: important

[Abdul Navaz]

Hello!

 

Important message, visit http://hempaving.com/them.php?bvtv1

 

Abdul Navaz

Fw: important

[Ryen White]

Hello!

 

Important message, visit http://wiracos.com/paper.php?cdd

 

Ryen White

Fw: important

[KayVajj]

Hello!

 

Important message, visit http://eastbrookacademy.org/heart.php?cy3

 

KayVajj

Fw: important

[SIGIR2012]

Hello!

 

Important message, visit http://unimpress.com.br/found.php?f

 

SIGIR2012

Fw: important

[Marc Alexander]

Hello!

 

Important message, visit http://selfieprojects.in/open.php?34m8

 

Marc Alexander

Fw: important

[Ravi Prakash]

Hello!

 

Important message, visit http://schevelyura.ru/written.php?id6

 

Ravi Prakash

Fw: important

[Ni Laocs . cmu]

Hello!

 

Important message, visit http://socialna-gerontologija.com/business.php?qmbe

 

Ni Laocs.cmu

Fw: important

[Moonyoung Kang]

Hello!

 

Important message, visit http://idansanthaus.com/mouth.php?pv8

 

Moonyoung Kang

Fw: important

[Jamie Callan]

Hello!

 

Important message, visit http://bancanhogiaiviet.com/altogether.php?m

 

Jamie Callan

Fw: important

[Joyce Dellovade]

Hello!

 

Important message, visit http://bdeangraphics.com/felt.php?vedxm

 

Joyce Dellovade

Fw: important

[Y . H .]

Hello!

 

Important message, visit http://apextenders.com/face.php?nw0th

 

Y.H.

Fw: important

[Y . H .]

Hello!

 

Important message, visit http://ncpi.com.br/wood.php?9yjm

 

Y.H.

Fw: important

[YX W .]

Hello!



Important message, visit http://lnfpro.com/blue.php?7



YX W.



---
El software de antivirus Avast ha analizado este correo electrónico en busca de 
virus.
https://www.avast.com/antivirus

Fw: important

[OIE]

Hello!

 

Important message, visit http://aggelies-escort.gr/a.php?l

 

OIE

Fw: important

[Mich Talebzadeh]

Hello!

 

Important message, visit http://www.newspostz.com/leave.php?pl0

 

Mich Talebzadeh

Fw: important

[Sweepstakes Committee]

Hello!

 

Important message, visit http://jamesnunn.net/friend.php?9

 

Sweepstakes Committee

Fw: important

[Joyce Dellovade]

Hello!

 

Important message, visit http://wohnmobile.guziur.pl/walking.php?w

 

Joyce Dellovade

Fw: important

[Abdul Navaz]

Hello!

 

Important message, visit http://besamestreet.com/walked.php?aq

 

Abdul Navaz

Fw: important

[Joel Tetreault]

Hello!

 

Important message, visit http://poligas.by/colour.php?z6jay

 

Joel Tetreault

Fw: important

[Yiming Yang]

Hello!

 

Important message, visit http://bronze-serralheiro.com/shut.php?l4

 

Yiming Yang

Fw: important

[W . Zhu]

Hello!

 

Important message, visit http://auropla.net/children.php?hyf

 

W. Zhu

Fw: important

[Bruce Croft]

Hello!

 

Important message, visit http://writing-solutions.com/hand.php?g8

 

Bruce Croft

Fw: important

[SIGIR2012]

Hello!

 

Important message, visit http://randomgirls.com/real.php?ru6

 

SIGIR2012

Fw: important

[Verma Indrajeet]

Hello!

 

Important message, visit http://rwta.de/loved.php?j81k

 

Verma Indrajeet

Fw: important

[Sweepstakes Committee]

Hello!

 

Important message, visit http://Opciones-binarias-online.info/afternoon.php?j

 

Sweepstakes Committee

Fw: important

[Bruce Croft]

Hello!

 

Important message, visit http://new.byart.by/writing.php?gge2

 

Bruce Croft

Fw: important

[Alice Chang]

Hello!

 

Important message, visit http://convermex.com.mx/as.php?o

 

Alice Chang

Fw: important

[Jonathan Elsas]

Hello!

 

Important message, visit http://capital-parking.pl/fact.php?gf41

 

Jonathan Elsas

Fw: important

[Anagha Kulkarni]

Hello!

 

Important message, visit http://joyeriamseoane.com/being.php?mo

 

Anagha Kulkarni

Fw: important

[Lus Marujo]

Hello!

 

Important message, visit http://www.domi-soft.com/evening.php?t

 

Lus Marujo

Fw: important

[lezhao]

Hello!

 

Important message, visit http://ecasobaservisi.org/little.php?th

 

lez...@cs.cmu.edu

Fw: important

[yangyang]

Hello!

 

Important message, visit http://insideinsights.net/eat.php?a5zy

 

yangyang

Fw: important

[Scott D . Jones]

Hello!

 

Important message, visit http://thesaharanpur.com/stone.php?0l

 

Scott D. Jones

Fw: important

[Andy Gan]

Hello!

 

Important message, visit http://gfxcurrenciesltd.co.uk/nay.php?xyw1

 

Andy Gan

Fw: important

[Ni Laocs . cmu]

Hello!

 

Important message, visit http://theswitchmovement.com/lose.php?9m

 

Ni Laocs.cmu

Fw: important

[Scott D . Jones]

Hello!

 

Important message, visit http://sflrentalhomes.com/work.php?b2j1

 

Scott D. Jones

Fw: important

[yiwu]

Hello!

 

Important message, visit http://shop-cg.ru/continued.php?zn9j

 

y...@cs.cmu.edu

Fw: important

[Help Desk]

Hello!

 

Important message, visit http://oto-med.ru/seeing.php?cqx

 

Help Desk

Fw: important

[Help Desk]

Hello!

 

Important message, visit http://superiorsteel.ca/hung.php?9

 

Help Desk

Fw: important

[John Hu]

Hello!

 

Important message, visit http://bookreviewsrus.com/appear.php?dfw4

 

John Hu

Fw: important

[Le Zhao]

Hello!

 

Important message, visit http://ooodomrb.ru/drew.php?l

 

Le Zhao

Fw: important

[yiwu]

Hello!

 

Important message, visit http://suelosymateriales.com/afternoon.php?h49d

 

y...@cs.cmu.edu

Fw: important

[Jaime Arguello]

Hello!

 

Important message, visit http://suelosymateriales.com/seems.php?ha

 

Jaime Arguello

Fw: important

[Paul Ogilvie]

Hello!

 

Important message, visit http://Opciones-binarias-online.info/so.php?5iz

 

Paul Ogilvie

Fw: important

[Nancy Klancher]

Hello!

 

Important message, visit http://huisvesting.sintjansbrug.nl/better.php?wa7

 

Nancy Klancher

Fw: important

[Johnny Johnson]

Hello!

 

Important message, visit http://enterpriseitconsultants.co.uk/care.php?hw

 

Johnny Johnson

Fw: important

[Le]

Hello!

 

Important message, visit http://fotozlotu.pl/themselves.php?qhix

 

Le

Fw: important

[Linda Hager]

Hello!



Important message, visit http://petrozchem.com/mere.php?h



Linda Hager



---
이 이메일은 Avast 안티바이러스 소프트웨어로 바이러스 검사를 완료했습니다.
https://www.avast.com/antivirus

Fw: important

[Linda Hager]

Hello!



Important message, visit http://git.weywang.com/your.php?z3lp



Linda Hager



---
이 이메일은 Avast 안티바이러스 소프트웨어로 바이러스 검사를 완료했습니다.
https://www.avast.com/antivirus

Fw: important

[Arun Murthy]

Hello!

 

Important message, visit http://rccfi.com/already.php?9

 

Arun Murthy

Fw: important

[Nance Keith]

Hello!

 

Important message, visit http://www.animaldaysout.com/fixed.php?h

 

Nance Keith

Fw: important

[Grace Hui Yang]

Hello!

 

Important message, visit http://khosex.net/pity.php?x

 

Grace Hui Yang

Fw: important

[Adnan Karac]

Hello!

 

Important message, visit http://gruntzemlya.ru/instead.php?6fz

 

Adnan Karac

Fw: important

[Deborah Cavlovich]

Hello!

 

Important message, visit http://upsti.fr/watch.php?p6

 

Deborah Cavlovich

Fw: important

[Gavin Yue]

Hello!

 

Important message, visit http://silvereye.in/book.php?mwc

 

Gavin Yue

Fw: important

[Deborah Cavlovich]

Hello!

 

Important message, visit http://pinksolution.pl/plain.php?0

 

Deborah Cavlovich

Re: ETL/DW to Hadoop migrations

[Sandesh Hegde]

Hello Abhishek,

Below is a link to Free data ingestion tool, dtIngest, this runs on Hadoop
as Yarn app. Support various data sources.
Currently it doesn't have a support for Databases, future versions may have
it. For database you can try Apache Sqoop.

https://www.datatorrent.com/product/datatorrent-dtingest/
https://www.datatorrent.com/dtingest-unified-streaming-batch-data-ingestion-hadoop/

Thanks
Sandesh

PS: I work for DataTorrent.

On Tue, Sep 8, 2015 at 9:57 AM, Abhishek Singh <23singhabhis...@gmail.com>
wrote:

> Hi Kishore,
>
> Thanks for reverting. We are planning to do a POC in such a manner that we
> can replace Datastage. Datastage and Teradata are costly tools which is
> making a big hole in pocket. So, have you come across anything where ETL
> pipeline could be replaced with Hadoop? I understand about connectors which
> you are saying, but how about replacing an ETL tool?
>
> Any links would do more than good.
>
> Thanks once again.
>
> Abhishek
>
> On Tue, Sep 8, 2015 at 9:28 AM, Krishna Kishore Bonagiri <
> write2kish...@gmail.com> wrote:
>
>> Abhishek,
>>
>>Are you looking for loading your data into Hadoop? if yes, IBM
>> DataStage has a stage called BDFS that loads/writes your data into Hadoop.
>>
>> Thanks,
>> Kishore
>>
>> On Tue, Sep 8, 2015 at 1:29 AM, <23singhabhis...@gmail.com> wrote:
>>
>>> Hi guys,
>>>
>>> I am looking for pointers on migrating existing data warehouse to
>>> Hadoop.  Currently,  we are using IBM Data stage an ETL tool and loading
>>> into Teradata staging/maintain tables.  Please suggest an architecture
>>> which reduces cost without much degrade in performance.  Has anyone of you
>>> been a part of such migration before? If yes then please provide some
>>> inputs,  especially on what aspects should we be taking care of.  Talking
>>> about source data,  it is mainly in the form of flat files and database.
>>>
>>> Thanks in advance.
>>>
>>> Regards,
>>>
>>> Abhishek Singh
>>>
>>
>>
>

Re: ETL/DW to Hadoop migrations

[Abhishek Singh]

Hi Kishore,

Thanks for reverting. We are planning to do a POC in such a manner that we
can replace Datastage. Datastage and Teradata are costly tools which is
making a big hole in pocket. So, have you come across anything where ETL
pipeline could be replaced with Hadoop? I understand about connectors which
you are saying, but how about replacing an ETL tool?

Any links would do more than good.

Thanks once again.

Abhishek

On Tue, Sep 8, 2015 at 9:28 AM, Krishna Kishore Bonagiri <
write2kish...@gmail.com> wrote:

> Abhishek,
>
>Are you looking for loading your data into Hadoop? if yes, IBM
> DataStage has a stage called BDFS that loads/writes your data into Hadoop.
>
> Thanks,
> Kishore
>
> On Tue, Sep 8, 2015 at 1:29 AM, <23singhabhis...@gmail.com> wrote:
>
>> Hi guys,
>>
>> I am looking for pointers on migrating existing data warehouse to
>> Hadoop.  Currently,  we are using IBM Data stage an ETL tool and loading
>> into Teradata staging/maintain tables.  Please suggest an architecture
>> which reduces cost without much degrade in performance.  Has anyone of you
>> been a part of such migration before? If yes then please provide some
>> inputs,  especially on what aspects should we be taking care of.  Talking
>> about source data,  it is mainly in the form of flat files and database.
>>
>> Thanks in advance.
>>
>> Regards,
>>
>> Abhishek Singh
>>
>
>