Re: Python Hadoop Example
https://www.michael-noll.com/tutorials/writing-an-hadoop-mapreduce-program-in-python/ On Sun, Jun 16, 2019, 9:18 AM Mike IT Expert wrote: > Please let me know where I can find a good/simple example of mapreduce > Python code running on Hadoop. Like tutorial or sth. > > Thank you > > >
Re: hadoop mapreduce job rest api
Take a look at webhcat api On Dec 24, 2015 12:50 AM, "ram kumar" wrote: > Hi, > > I want to submit a mapreduce job using rest api, > and get the status of the job every n interval. > Is there a way to do it? > > Thanks >
Re: Start/stop scripts - particularly start-dfs.sh - in Hortonworks Data Platform 2.3.X
Look in /usr/hdp/2.3 On Oct 23, 2015 10:07 AM, "Stephen Boesch" wrote: > > We are setting up automated deployments on a headless system: so using the > GUI is not an option here. When we search for those scripts under HDP > they are not found: > > $ pwd > /usr/hdp/current > > Which scripts exist in HDP ? > > [stack@s1-639016 current]$ find -L . -name \*.sh > ... > > There are ZERO start/stop sh scripts.. > > In particular I am interested in the *start-dfs.sh* script that starts > the namenode(s) , journalnode, and datanodes. > >
Re: Active Namenode keeps crashing
Check whether connectivity between servers is stable. Error says it can't
reach one node. Also check that time is synched between nodes.
On Aug 9, 2015 9:31 PM, "Suresh V" wrote:
> In our HA setup, the active namenode keeps crashing once a week or so. The
> cluster is quite idle without many jobs running and not much user activity.
>
> Below is logs from journal nodes. Can someone help us with this please?
>
>
> 2015-08-04 13:00:20,054 INFO server.Journal
> (Journal.java:updateLastPromisedEpoch(315)) - Updating lastPromisedEpoch
> from 9 to 10 for client /172.26.44.133
>
> 2015-08-04 13:00:20,175 INFO server.Journal
> (Journal.java:scanStorageForLatestEdits(188)) - Scanning storage
> FileJournalManager(root=/hadoop/hdfs/journal/HDPPROD)
>
> 2015-08-04 13:00:20,220 INFO server.Journal
> (Journal.java:scanStorageForLatestEdits(194)) - Latest log is
> EditLogFile(file=/hadoop/hdfs/journal/HDPPROD/current/edits_inprogress_0523903,first=0523903,last=0523925,inProgress=true,hasCorruptHeader=false)
>
> 2015-08-04 13:00:20,891 INFO server.Journal
> (Journal.java:getSegmentInfo(687)) - getSegmentInfo(523903):
> EditLogFile(file=/hadoop/hdfs/journal/HDPPROD/current/edits_inprogress_0523903,first=0523903,last=0523925,inProgress=true,hasCorruptHeader=false)
> -> startTxId: 523903 endTxId: 523925 isInProgress: true
>
> 2015-08-04 13:00:20,891 INFO server.Journal
> (Journal.java:prepareRecovery(731)) - Prepared recovery for segment 523903:
> segmentState { startTxId: 523903 endTxId: 523925 isInProgress: true }
> lastWriterEpoch: 9 lastCommittedTxId: 523924
>
> 2015-08-04 13:00:20,956 INFO server.Journal
> (Journal.java:getSegmentInfo(687)) - getSegmentInfo(523903):
> EditLogFile(file=/hadoop/hdfs/journal/HDPPROD/current/edits_inprogress_0523903,first=0523903,last=0523925,inProgress=true,hasCorruptHeader=false)
> -> startTxId: 523903 endTxId: 523925 isInProgress: true
>
> 2015-08-04 13:00:20,956 INFO server.Journal
> (Journal.java:acceptRecovery(817)) - Skipping download of log startTxId:
> 523903 endTxId: 523925 isInProgress: true: already have up-to-date logs
>
> 2015-08-04 13:00:20,989 INFO server.Journal
> (Journal.java:acceptRecovery(850)) - Accepted recovery for segment 523903:
> segmentState { startTxId: 523903 endTxId: 523925 isInProgress: true }
> acceptedInEpoch: 10
>
> 2015-08-04 13:00:21,791 INFO server.Journal
> (Journal.java:finalizeLogSegment(584)) - Validating log segment
> /hadoop/hdfs/journal/HDPPROD/current/edits_inprogress_0523903
> about to be finalized
>
> 2015-08-04 13:00:21,805 INFO namenode.FileJournalManager
> (FileJournalManager.java:finalizeLogSegment(133)) - Finalizing edits file
> /hadoop/hdfs/journal/HDPPROD/current/edits_inprogress_0523903
> ->
> /hadoop/hdfs/journal/HDPPROD/current/edits_0523903-0523925
>
> 2015-08-04 13:00:22,257 INFO server.Journal
> (Journal.java:startLogSegment(532)) - Updating lastWriterEpoch from 9 to 10
> for client /172.26.44.133
>
> 2015-08-04 13:00:23,699 INFO ipc.Server (Server.java:run(2060)) - IPC
> Server handler 4 on 8485, call
> org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocol.journal from
> 172.26.44.135:43678 Call#304302 Retry#0
>
> java.io.IOException: IPC's epoch 9 is less than the last promised epoch 10
>
> at
> org.apache.hadoop.hdfs.qjournal.server.Journal.checkRequest(Journal.java:414)
>
> at
> org.apache.hadoop.hdfs.qjournal.server.Journal.checkWriteRequest(Journal.java:442)
>
> at
> org.apache.hadoop.hdfs.qjournal.server.Journal.journal(Journal.java:342)
>
> at
> org.apache.hadoop.hdfs.qjournal.server.JournalNodeRpcServer.journal(JournalNodeRpcServer.java:148)
>
> at
> org.apache.hadoop.hdfs.qjournal.protocolPB.QJournalProtocolServerSideTranslatorPB.journal(QJournalProtocolServerSideTranslatorPB.java:158)
>
> at
> org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocolProtos$QJournalProtocolService$2.callBlockingMethod(QJournalProtocolProtos.java:25421)
>
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
>
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)
>
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)
>
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:415)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)
>
> 2015-08-06 19:13:14,012 INFO httpclient.HttpMethodDirector
> (HttpMethodDirector.java:executeWithRetry(439)) - I/O exception
> (org.apache.commons.httpclient.NoHttpResponseException) caught when
> processing request: The server a
Re: Backup of individual component of Hadoop ecosystem
There are several approaches. I would check hdfs trash folder of the user deleting a file. Expiration of items in trash is controlled by fs.trash.interval property on core-site.xml. Artem Ervits On Feb 26, 2015 1:31 PM, "Krish Donald" wrote: > Hi, > > As per my understanding we don't take backup of Hadoop cluster as the size > is very large generally . > > However in case if somebody has dropped a table by mistake then how should > we recover the data ? > > How to take backup of Hadoop ecosystem individual component. > > Thanks > Krish >
Re: Running MapReduce jobs in batch mode on different data sets
Take a look at Apache Oozie Artem Ervits On Feb 21, 2015 6:35 AM, "tesm...@gmail.com" wrote: > Hi, > > Is it possible to run jobs on Hadoop in batch mode? > > I have 5 different datasets in HDFS and need to run the same MapReduce > application on these datasets sets one after the other. > > Right now I am doing it manually How can I automate this? > > How can I save the log of each execution in text files for later > processing? > > Regards, > Tariq >
Re: Home for Apache Big Data Solutions?
I believe Apache Bigtop is what you're looking for. Artem Ervits On Feb 9, 2015 8:15 AM, "Jean-Baptiste Onofré" wrote: > Hi Amir, > > thanks for the update. > > Please, let me know if you need some help on the proposal and to "qualify" > your ideas. > > Regards > JB > > On 02/09/2015 02:05 PM, MrAsanjar . wrote: > >> Hi Chris, >> thanks for the information, will get on it ... >> >> Hi JB >> Glad that you are familiar with Juju, however my personal goal is not to >> promote any tool but >> to take the next step, which is to build a community for apache big data >> solutions. >> >> >>do you already have a kind of proposal/description of your projects ? >> working on it :) I got the idea while flying back from South Africa on >> Saturday. During my trip I noticed most of the communities spending >> their precious resources on solution plumbing, without much of emphasis >> on solution best practices due to the lack of expertise. By the time >> Big Data solution framework becomes operational, funding has diminished >> enough to limit solution activity (i.e data analytic payload >> development). I am sure we could find >> similar scenarios with other institutions and SMB (small and >> medium-size businesses) anywhere. >> In the nutshell my goals are as follow: >> 1) Make Big Data solutions available to everyone >> 2) Encapsulate the best practices >> 3) All Orchestration tools are welcomed - Some solutions could have >> hybrid tooling model >> 4) Enforce automated testing and quality control. >> 5) Share analytic payloads (i.e mapreduce apps, storm topology, Pig >> scripts,...) >> >> >> >>Is it like distribution, or tooling ? >> Good question, I envision to have a distribution model as it has >> dependency on Apache hadoop projects distributions. >> >> >>What's the current license ? >> Charms/Bundles are moving to Apache 2.0 license, target data 2/27. >> >> Regards >> Amir Sanjar >> Big Data Solution Lead >> Canonical >> >> On Sun, Feb 8, 2015 at 10:46 AM, Mattmann, Chris A (3980) >> mailto:chris.a.mattm...@jpl.nasa.gov>> >> wrote: >> >> Dear Amir, >> >> Thank you for your interest in contributing these projects >> to the ASF! Sincerely appreciate it. >> >> My suggestion would be to look into the Apache Incubator, >> which is the home for incoming projects at the ASF. The >> TL;DR answer is: >> >> 1. You’ll need to create a proposal for each project >> that you would like to bring in using: >> http://incubator.apache.org/guides/proposal.html >> >> >> 2. You should put your proposal up on a public wiki >> for each project: >> http://wiki.apache.org/incubator/ >> create a new page e.g., YourProjectProposal, which would in >> turn become http://wiki.apache.org/incubator/YouProjectProposal >> You will need to request permissions to add the page on the >> wiki >> >> 3. Recruit at least 3 IPMC/ASF members to mentor your project: >> http://people.apache.org/committers-by-project.html#incubator-pmc >> >> http://people.apache.org/committers-by-project.html#member >> >> >> 4. Submit your proposal for consideration at the Incubator >> 5. Enjoy! >> >> Cheers and good luck. >> >> Cheers, >> Chris >> >> ++ >> Chris Mattmann, Ph.D. >> Chief Architect >> Instrument Software and Science Data Systems Section (398) >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >> Office: 168-519, Mailstop: 168-527 >> Email: chris.a.mattm...@nasa.gov <mailto:chris.a.mattm...@nasa.gov> >> WWW: http://sunset.usc.edu/~mattmann/ >> ++ >> Adjunct Associate Professor, Computer Science Department >> University of Southern California, Los Angeles, CA 90089 USA >> ++ >> >> >> >> >> >> >> -Original Message- >> From: "MrAsanjar ." mailto:afsan...@gmail.com>> >> Reply-To: "user@hadoop.apache.org <mailto:user@hadoop.apache.org>" >> mailto:user@hadoop.apache.org>> >> Date: Sunday, February 8, 2015 at 8:36 AM >> To: "
Re: Adding datanodes to Hadoop cluster - Will data redistribute?
Look at hdfs balancer Artem Ervits On Feb 6, 2015 5:54 PM, "Manoj Venkatesh" wrote: > Dear Hadoop experts, > > I have a Hadoop cluster of 8 nodes, 6 were added during cluster creation > and 2 additional nodes were added later to increase disk and CPU capacity. > What i see is that processing is shared amongst all the nodes whereas the > storage is reaching capacity on the original 6 nodes whereas the newly > added machines have relatively large amount of storage still unoccupied. > > I was wondering if there is an automated or any way of redistributing data > so that all the nodes are equally utilized. I have checked for the > configuration parameter - *dfs.datanode.fsdataset.volume.choosing.policy* > have options 'Round Robin' or 'Available Space', are there any other > configurations which need to be reviewed. > > Thanks, > Manoj >
Re: tools.DistCp: Invalid arguments
Another good option is hftp. Artem Ervits On Feb 3, 2015 6:42 AM, "xeonmailinglist" wrote: > I have found the problem. I started to use `webhdfs` and everything is ok. > > > On 03-02-2015 10:40, xeonmailinglist wrote: > > What do you mean by no path is given? Even if I launch this command, I > get the same error…. What path should I put here? > > $ hadoop distcp hdfs://hadoop-coc-1:50070/input1 > hdfs://hadoop-coc-2:50070/input1 > > Thanks, > > On 02-02-2015 19:59, Alexander Alten-Lorenz wrote: > > Have a closer look: > >hdfs://hadoop-coc-2:50070/ > > > No Path is given. > > > On 02 Feb 2015, at 20:52, xeonmailinglist > wrote: > > Hi, > > I am trying to copy data using distcp but I get this error. Both hadoop > runtime are working properly. Why is this happening? > > > vagrant@hadoop-coc-1:~/Programs/hadoop$ hadoop distcp > hdfs://hadoop-coc-1:50070/input1 hdfs://hadoop-coc-2:50070/ > 15/02/02 19:46:37 ERROR tools.DistCp: Invalid arguments: > java.io.IOException: Failed on local exception: > com.google.protobuf.InvalidProtocolBufferException: Protocol message > end-group tag did not match expected tag.; Host Details : local host is: > "hadoop-coc-1/127.0.1.1"; destination host is: "hadoop-coc-2":50070; > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772) > at org.apache.hadoop.ipc.Client.call(Client.java:1472) > at org.apache.hadoop.ipc.Client.call(Client.java:1399) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) > at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) > at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source) > at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1988) > at > org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1118) > at > org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1114) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1114) > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1400) > at org.apache.hadoop.tools.DistCp.setTargetPathExists(DistCp.java:188) > at org.apache.hadoop.tools.DistCp.run(DistCp.java:111) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.hadoop.tools.DistCp.main(DistCp.java:401) > Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol > message end-group tag did not match expected tag. > at > com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:94) > at > com.google.protobuf.CodedInputStream.checkLastTagWas(CodedInputStream.java:124) > at > com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:202) > at > com.google.protobuf.AbstractParser.parsePartialDelimitedFrom(AbstractParser.java:241) > at > com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:253) > at > com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:259) > at > com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:49) > at > org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.parseDelimitedFrom(RpcHeaderProtos.java:3167) > at > org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1072) > at org.apache.hadoop.ipc.Client$Connection.run(Client.java:966) > Invalid arguments: Failed on local exception: > com.google.protobuf.InvalidProtocolBufferException: Protocol message > end-group tag did not match expected tag.; Host Details : local host is: > "hadoop-coc-1/127.0.1.1"; destination host is: "hadoop-coc-2":50070; > usage: distcp OPTIONS [source_path...] > > Thanks, > > > > > > >
Re: Copy data between clusters during the job execution.
take a look at oozie, once first job completes you can distcp to another server. Artem Ervits On Feb 2, 2015 5:46 AM, "Daniel Haviv" wrote: > It should run after your job finishes. > You can create the flow using a simple bash script > > Daniel > > On 2 בפבר׳ 2015, at 12:31, xeonmailinglist > wrote: > > But can I use discp inside my job, or I need to program something that > executes distcp after executing my job? > > > On 02-02-2015 10:20, Daniel Haviv wrote: > > an use distcp > > Daniel > > On 2 בפבר׳ 2015, at 11:12, > > >
Re: Any working VM of Apache Hadoop ?
+1 for Bigtop Artem Ervits On Jan 18, 2015 4:10 PM, "Jay Vyas" wrote: > Also BigTop has a very flexible vagrant infrastructure: > > > https://github.com/apache/bigtop/tree/master/bigtop-deploy/vm/vagrant-puppet > > On Jan 18, 2015, at 3:37 PM, Andre Kelpe wrote: > > Try our vagrant setup: > https://github.com/Cascading/vagrant-cascading-hadoop-cluster > > - André > > On Sat, Jan 17, 2015 at 10:07 PM, Krish Donald > wrote: > >> Hi, >> >> I am looking for working VM of Apache Hadoop. >> Not looking for cloudera or Horton works VMs. >> If anybody has it and if they can share that would be great . >> >> Thanks >> Krish >> > > > > -- > André Kelpe > an...@concurrentinc.com > http://concurrentinc.com > >
Re: ExportSnapshot webhdfs problems
Does it work with a small table? I prefer to use hftp instead of webhdfs. From: Brian Jeltema [mailto:brian.jelt...@digitalenvoy.net] Sent: Friday, October 03, 2014 11:01 AM To: user@hadoop.apache.org Subject: ExportSnapshot webhdfs problems I posted this on users@hbase, but got not response, so I thought I’d try here: I’m trying to use ExportSnapshot to copy a snapshot from a Hadoop 1 to a Hadoop 2 cluster using the webhdfs protocol. I’ve done this successfully before, though there are always mapper failures and retries in the job log. However, I’m not having success with a rather large table due to an excessive number of failures. The exceptions in the job log are always: 14/09/29 20:28:11 INFO mapred.JobClient: Task Id : attempt_201409241055_0024_m_05_1, Status : FAILED org.apache.hadoop.ipc.RemoteException at org.apache.hadoop.hdfs.web.JsonUtil.toRemoteException(JsonUtil.java:114) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:290) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$500(WebHdfsFileSystem.java:98) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$2.close(WebHdfsFileSystem.java:653) at org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.copyFile(ExportSnapshot.java:204) at org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.map(ExportSnapshot.java:146) … … … So I presume the real exception is taking place on the target system. However, examining the namenode logs and a handful of the datanode logs has not revealed any exceptions that correlate with those in the job log. Is there some other log I should be looking at? I reduced the number of mappers to 6 and the target cluster has 10 datanodes, so it’s hard to believe its a capacity problem. Thanks Brian This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you.
Re: Using Ambari to deploy Apache hadoop
For offline you need to set up local mirror, there's a guide in the manual. As far as bits, its an Apache release not Hortonworks, same as if you were to install from BigTop. Artem Ervits Data Analyst New York Presbyterian Hospital - Original Message - From: Siddharth Tiwari [mailto:siddharth.tiw...@live.com] Sent: Friday, October 18, 2013 05:13 AM To: user@hadoop.apache.org Subject: Using Ambari to deploy Apache hadoop Hi team, Is it possible to deploy hadoop from Apache via Ambari ? Also is there a link for full offline installation ? We do not have access to outside world and we want to use Ambari for reploying Hadoop ( not hortonworks release though ) Sent from my iPhone This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you.
Re: Hadoop Metrics Issue in ganglia.
Check firewall and /etc/hosts also make sure hosts lines up with result of hostname -f command. Both hostname -f and hosts entries should have fqdn names. I use ambari to install my cluster, including ganglia metrics and I had identical issue. Once I corrected that it started working. Artem Ervits Data Analyst New York Presbyterian Hospital - Original Message - From: Chris Embree [mailto:cemb...@gmail.com] Sent: Wednesday, September 11, 2013 02:40 PM To: user@hadoop.apache.org Subject: Re: Hadoop Metrics Issue in ganglia. Did you try ganglia forums/lists? On 9/11/13, orahad bigdata wrote: > Hi All, > > Can somebody help me please? > > Thanks > On 9/11/13, orahad bigdata wrote: >> Hi All, >> >> I'm facing an issue while showing Hadoop metrics in ganglia, Though I >> have installed ganglia on my master/slaves nodes and I'm able to see >> all the default metrics on ganglia UI from all the nodes but I'm not >> able to see Hadoop metrics in metrics section. >> >> versions:- >> Hadoop 1.1.1 >> ganglia 3.0.7 >> >> hadoop-metrics2.properties - >> >> dfs.class=org.apache.hadoop.metrics.ganglia.GangliaContext31 >> dfs.period=10 >> dfs.servers=master:8649 >> mapred.class=org.apache.hadoop.metrics.ganglia.GangliaContext31 >> mapred.period=10 >> mapred.servers=master:8649 >> >> please guide. >> >> Thanks >> > This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you.
protect from accidental deletes
Hello all, I'd like to know what users are doing to protect themselves from accidental deletes of files and directories in HDFS? Any suggestions are appreciated. Thanks.
Hadoop 1.1.2 is it beta or stable?
Hello all, I'd like to know whether version 1.1.2 is the new stable release. The download page stills says it's beta. Thank you. Artem Ervits New York Presbyterian Hospital This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you. This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you.
Re: How to install Oozie 3.3.1 on Hadoop 1.1.1
It may not necessarily be correct but I built the project using 0.20.200 and am running it without problems "so far" on 1.0.3 hadoop. Artem Ervits Data Analyst New York Presbyterian Hospital From: Jagat Singh [mailto:jagatsi...@gmail.com] Sent: Friday, February 15, 2013 09:42 PM To: user@hadoop.apache.org Subject: Re: How to install Oozie 3.3.1 on Hadoop 1.1.1 Hi, I can see that in pom.xml the supported hadoop version is 1.0.1 You can try to build your self with version you want to see if it works. Also try to ask your question on oozie mailing list. Regards, Jagat Singh On Sat, Feb 16, 2013 at 12:45 PM, Hemanth Yamijala mailto:yhema...@thoughtworks.com>> wrote: Hi, It may be useful to post this question on the oozie user mailing list. There are likely to be more expert users there. u...@oozie.apache.org<mailto:u...@oozie.apache.org> Thanks Hemanth On Friday, February 15, 2013, anand verma wrote: Hi, I am struggling for many days to install Oozie 3.3.1 on Hadoop 1.1.1. Oozie documentation is very poorly written I am not able to figure it out. While installing I got an error saying it doesn't support Hadoop v1.1.1. Please help me out. -- Regards Ananda Prakash Verma This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you. This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you.
Dell Hardware
Hello all, Does anyone run Hadoop on Dell R720 model of servers? Dell site lists C2100 model of servers as best fit for Hadoop workloads. What does community recommend? Artem Ervits New York Presbyterian Hospital This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you. This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you.
Re: JobCache directory cleanup
As soon as job completes, your jobcache should be cleared. Check your mapred-site.xml for mapred.local.dir setting and make sure job cleanup step is successful in web UI. Setting your job's intermediate output setting to true will keep the jobcache folder smaller. Artem Ervits Data Analyst New York Presbyterian Hospital From: Hemanth Yamijala [mailto:yhema...@thoughtworks.com] Sent: Thursday, January 10, 2013 07:37 AM To: user@hadoop.apache.org Subject: Re: JobCache directory cleanup Hi, On Thu, Jan 10, 2013 at 5:17 PM, Ivan Tretyakov mailto:itretya...@griddynamics.com>> wrote: Thanks for replies! Hemanth, I could see following exception in TaskTracker log: https://issues.apache.org/jira/browse/MAPREDUCE-5 But I'm not sure if it is related to this issue. > Now, when a job completes, the directories under the jobCache must get > automatically cleaned up. However it doesn't look like this is happening in > your case. So, If I've no running jobs, jobcache directory should be empty. Is it correct? That is correct. I just verified it with my Hadoop 1.0.2 version Thanks Hemanth On Thu, Jan 10, 2013 at 8:18 AM, Hemanth Yamijala mailto:yhema...@thoughtworks.com>> wrote: Hi, The directory name you have provided is /data?/mapred/local/taskTracker/persona/jobcache/. This directory is used by the TaskTracker (slave) daemons to localize job files when the tasks are run on the slaves. Hence, I don't think this is related to the parameter "mapreduce.jobtracker.retiredjobs.cache.size", which is a parameter related to the jobtracker process. Now, when a job completes, the directories under the jobCache must get automatically cleaned up. However it doesn't look like this is happening in your case. Could you please look at the logs of the tasktracker machine where this has gotten filled up to see if there are any errors that could give clues ? Also, since this is a CDH release, it could be a problem specific to that - and maybe reaching out on the CDH mailing lists will also help Thanks hemanth On Wed, Jan 9, 2013 at 8:11 PM, Ivan Tretyakov mailto:itretya...@griddynamics.com>> wrote: Hello! I've found that jobcache directory became very large on our cluster, e.g.: # du -sh /data?/mapred/local/taskTracker/user/jobcache 465G/data1/mapred/local/taskTracker/user/jobcache 464G/data2/mapred/local/taskTracker/user/jobcache 454G/data3/mapred/local/taskTracker/user/jobcache And it stores information for about 100 jobs: # ls -1 /data?/mapred/local/taskTracker/persona/jobcache/ | sort | uniq | wc -l I've found that there is following parameter: mapreduce.jobtracker.retiredjobs.cache.size 1000 The number of retired job status to keep in the cache. So, if I got it right it intended to control job cache size by limiting number of jobs to store cache for. Also, I've seen that some hadoop users uses cron approach to cleanup jobcache: http://grokbase.com/t/hadoop/common-user/102ax9bze1/cleaning-jobcache-manually (http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201002.mbox/%3c99484d561002100143s4404df98qead8f2cf687a7...@mail.gmail.com%3E) Are there other approaches to control jobcache size? What is more correct way to do it? Thanks in advance! P.S. We are using CDH 4.1.1. -- Best Regards Ivan Tretyakov Deployment Engineer Grid Dynamics +7 812 640 38 76 Skype: ivan.tretyakov www.griddynamics.com<http://www.griddynamics.com> itretya...@griddynamics.com<mailto:itretya...@griddynamics.com> -- Best Regards Ivan Tretyakov Deployment Engineer Grid Dynamics +7 812 640 38 76 Skype: ivan.tretyakov www.griddynamics.com<http://www.griddynamics.com> itretya...@griddynamics.com<mailto:itretya...@griddynamics.com> This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you. This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you.
Re: JobCache directory cleanup
Just ran into similar problem. If you compress intermediate data, it will kEep jobcache folder manageable. Artem Ervits Data Analyst New York Presbyterian Hospital From: Ivan Tretyakov [mailto:itretya...@griddynamics.com] Sent: Wednesday, January 09, 2013 10:22 AM To: user@hadoop.apache.org Subject: Re: JobCache directory cleanup Thanks a lot Alexander! What is mapreduce.jobtracker.retiredjobs.cache.size for? Does cron approach safe for hadoop? Is that only way at the moment? On Wed, Jan 9, 2013 at 6:50 PM, Alexander Alten-Lorenz mailto:wget.n...@gmail.com>> wrote: Hi, Per default (and not configurable) the logs will be persist for 30 days. This will be configurable in future (https://issues.apache.org/jira/browse/MAPREDUCE-4643). - Alex On Jan 9, 2013, at 3:41 PM, Ivan Tretyakov mailto:itretya...@griddynamics.com>> wrote: > Hello! > > I've found that jobcache directory became very large on our cluster, e.g.: > > # du -sh /data?/mapred/local/taskTracker/user/jobcache > 465G/data1/mapred/local/taskTracker/user/jobcache > 464G/data2/mapred/local/taskTracker/user/jobcache > 454G/data3/mapred/local/taskTracker/user/jobcache > > And it stores information for about 100 jobs: > > # ls -1 /data?/mapred/local/taskTracker/persona/jobcache/ | sort | uniq | > wc -l > > I've found that there is following parameter: > > > mapreduce.jobtracker.retiredjobs.cache.size > 1000 > The number of retired job status to keep in the cache. > > > > So, if I got it right it intended to control job cache size by limiting > number of jobs to store cache for. > > Also, I've seen that some hadoop users uses cron approach to cleanup > jobcache: > http://grokbase.com/t/hadoop/common-user/102ax9bze1/cleaning-jobcache-manually > ( > http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201002.mbox/%3c99484d561002100143s4404df98qead8f2cf687a7...@mail.gmail.com%3E > ) > > Are there other approaches to control jobcache size? > What is more correct way to do it? > > Thanks in advance! > > P.S. We are using CDH 4.1.1. > > -- > Best Regards > Ivan Tretyakov > > Deployment Engineer > Grid Dynamics > +7 812 640 38 76 > Skype: ivan.tretyakov > www.griddynamics.com<http://www.griddynamics.com> > itretya...@griddynamics.com<mailto:itretya...@griddynamics.com> -- Alexander Alten-Lorenz http://mapredit.blogspot.com German Hadoop LinkedIn Group: http://goo.gl/N8pCF -- Best Regards Ivan Tretyakov Deployment Engineer Grid Dynamics +7 812 640 38 76 Skype: ivan.tretyakov www.griddynamics.com<http://www.griddynamics.com> itretya...@griddynamics.com<mailto:itretya...@griddynamics.com> This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you. This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you.
RE: Hadoop throughput question
John, the two programs below, one is from the Definitive Guide chapter 4 with
slight mods and the other is in-house but similar to Hadoop in Action chap 3.
package sequencefileprocessor;
// cc SequenceFileReadDemo Reading a SequenceFile
import java.io.IOException;
import java.net.URI;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IOUtils;
import org.apache.hadoop.io.SequenceFile;
import org.apache.hadoop.io.Writable;
import org.apache.hadoop.util.ReflectionUtils;
// vv SequenceFileReadDemo
public class SequenceFileProcessor
{
public static void main(String[] args) throws IOException
{
Configuration conf = new Configuration();
conf.set("mapred.map.child.java.opts", "-Xmx256m");
conf.set("mapred.reduce.child.java.opts", "-Xmx256m");
//conf.set("io.file.buffer.size", "65536"); //10mb/sec improvement,
jumped from 26mb/sec to 36mb/sec
conf.set("io.file.buffer.size", "131072"); // 15mb/sec improvement,
jumped from 26mb/sec to 39mb/sec
FileSystem fs = null;
Path path = null;
int total_count = 0;
int count = 0;
long start = System.currentTimeMillis();
for (String uri : args)
{
fs = FileSystem.get(URI.create(uri), conf);
path = new Path(uri);
SequenceFile.Reader reader = null;
try
{
reader = new SequenceFile.Reader(fs, path, conf);
Writable key = (Writable)
ReflectionUtils.newInstance(reader.getKeyClass(), conf);
Writable value = (Writable)
ReflectionUtils.newInstance(reader.getValueClass(), conf);
long position = reader.getPosition();
while (reader.next(key, value))
{
String syncSeen = reader.syncSeen() ? "*" : "";
//System.out.printf("[%s%s]\t%s\t%s\n", position, syncSeen,
key, value);
position = reader.getPosition(); // beginning of next record
count += 1;
//System.out.println("count is: " + count);
if((count % 100) == 0)
System.out.println("processed " + count + " records");
}
}
finally
{
IOUtils.closeStream(reader);
}
}
total_count += count;
System.out.println("Total count: " + total_count);
System.out.println("Elapsed time: " + ((System.currentTimeMillis() -
start) / 1000) + " seconds");
}
}
// ^^ SequenceFileReadDemo
/*
* To change this template, choose Tools | Templates
* and open the template in the editor.
*/
package hdfsspeedtest;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.util.Date;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.fs.FileStatus;
/ code is */
public class HDFSSpeedTest {
public static void main(String[] args) throws Exception {
System.out.println(new Date().toString());
Path pt = new Path(args[0]);
try {
// Use this for reading the data.
byte[] buffer = new byte[32*1024*1024];
Configuration conf = new Configuration();
//conf.set(null, null);
FileSystem fs = FileSystem.get(conf);
FileStatus[] inputFiles = fs.listStatus(pt);
long total = 0;
for(int i=0; imailto:john.lil...@redpoint.net]
Sent: Thursday, January 03, 2013 9:04 PM
To: user@hadoop.apache.org
Subject: RE: Hadoop throughput question
Perhaps if Artem posted the presumably-simple code we could get other users to
benchmark other 4-node systems and compare.
--John Lilley
Artem Ervits mailto:are9...@nyp.org>> wrote:
Setting the property to 64k made the throughput jump to 36mb/sec, 39mb for 128k.
Thank you for the tip.
From: Michael Katzenellenbogen
[mailto:mich...@cloudera.com]<mailto:[mailto:mich...@cloudera.com]>
Sent: Thursday, January 03, 2013 7:28 PM
To: user@hadoop.apache.org<mailto:user@hadoop.apache.org>
Subject: Re: Hadoop throughput question
What is the value of the io.file.buffer.size property? Try tuning it up to 64k
or 128k and see if this improves performance when reading SequenceFiles.
-Michael
On Jan 3, 2013, at 7:00 PM, Artem Ervits
mailto:are9...@nyp.org>> wrote:
I will follow up on that certainly, thank you for the information.
So further investigation showed that counting SequenceFile records takes about
26mb/sec. If I simply read bytes on the same cluster and th
RE: Hadoop throughput question
Setting the property to 64k made the throughput jump to 36mb/sec, 39mb for 128k. Thank you for the tip. From: Michael Katzenellenbogen [mailto:mich...@cloudera.com] Sent: Thursday, January 03, 2013 7:28 PM To: user@hadoop.apache.org Subject: Re: Hadoop throughput question What is the value of the io.file.buffer.size property? Try tuning it up to 64k or 128k and see if this improves performance when reading SequenceFiles. -Michael On Jan 3, 2013, at 7:00 PM, Artem Ervits mailto:are9...@nyp.org>> wrote: I will follow up on that certainly, thank you for the information. So further investigation showed that counting SequenceFile records takes about 26mb/sec. If I simply read bytes on the same cluster and the same file, the speed is 70mb/sec. Is there a configuration for optimizing SequenceFile processing? Thank you. From: John Lilley [mailto:john.lil...@redpoint.net] Sent: Thursday, January 03, 2013 6:09 PM To: user@hadoop.apache.org<mailto:user@hadoop.apache.org> Subject: RE: Hadoop throughput question Unless the Hadoop processing and the OneFS storage are co-located, MapReduce can't schedule tasks so as to take advantage of data locality. You would basically be doing a distributed computation against a separate NAS, so throughput would be limited by the performance properties of the Insilon NAS and the network switch architecture. Still, 26MB/sec in aggregate is far worse than what I'd expect Insilon to deliver, even over a single 1GB connection. john From: Artem Ervits [mailto:are9...@nyp.org]<mailto:[mailto:are9...@nyp.org]> Sent: Thursday, January 03, 2013 4:02 PM To: user@hadoop.apache.org<mailto:user@hadoop.apache.org> Subject: RE: Hadoop throughput question Hadoop is using OneFS, not HDFS in our configuration. Isilon NAS and the Hadoop nodes are in the same datacenter but as far as rack locations, I cannot tell. From: John Lilley [mailto:john.lil...@redpoint.net] Sent: Thursday, January 03, 2013 5:15 PM To: user@hadoop.apache.org<mailto:user@hadoop.apache.org> Subject: RE: Hadoop throughput question Let's suppose you are doing a read-intensive job like, for example, counting records. This is will be disk bandwidth limited. On a 4-node cluster with 2 local SATA on each node you should easily read 400MB/sec in aggregate. When you are running the Hadoop cluster, is the Hadoop processing co-located with the Ilsilon nodes? Is Hadoop configured to use OneFS or HDFS? John From: Artem Ervits [mailto:are9...@nyp.org]<mailto:[mailto:are9...@nyp.org]> Sent: Thursday, January 03, 2013 3:00 PM To: user@hadoop.apache.org<mailto:user@hadoop.apache.org> Subject: Hadoop throughput question Hello all, I'd like to pick the community brain on average throughput speeds for a moderately specced 4-node Hadoop cluster with 1GigE networking. Is it reasonable to expect constant average speeds of 150-200mb/sec on such setup? Forgive me if the question is loaded but we're Hadoop cluster with HDFS served via EMC Isilon storage. We're getting about 30mb/sec with our machines and we do not see a difference in job speed between 2 node cluster and 4 node cluster. Thank you. This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you. This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you. This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you. -
RE: Hadoop throughput question
I will follow up on that certainly, thank you for the information. So further investigation showed that counting SequenceFile records takes about 26mb/sec. If I simply read bytes on the same cluster and the same file, the speed is 70mb/sec. Is there a configuration for optimizing SequenceFile processing? Thank you. From: John Lilley [mailto:john.lil...@redpoint.net] Sent: Thursday, January 03, 2013 6:09 PM To: user@hadoop.apache.org Subject: RE: Hadoop throughput question Unless the Hadoop processing and the OneFS storage are co-located, MapReduce can't schedule tasks so as to take advantage of data locality. You would basically be doing a distributed computation against a separate NAS, so throughput would be limited by the performance properties of the Insilon NAS and the network switch architecture. Still, 26MB/sec in aggregate is far worse than what I'd expect Insilon to deliver, even over a single 1GB connection. john From: Artem Ervits [mailto:are9...@nyp.org]<mailto:[mailto:are9...@nyp.org]> Sent: Thursday, January 03, 2013 4:02 PM To: user@hadoop.apache.org<mailto:user@hadoop.apache.org> Subject: RE: Hadoop throughput question Hadoop is using OneFS, not HDFS in our configuration. Isilon NAS and the Hadoop nodes are in the same datacenter but as far as rack locations, I cannot tell. From: John Lilley [mailto:john.lil...@redpoint.net] Sent: Thursday, January 03, 2013 5:15 PM To: user@hadoop.apache.org<mailto:user@hadoop.apache.org> Subject: RE: Hadoop throughput question Let's suppose you are doing a read-intensive job like, for example, counting records. This is will be disk bandwidth limited. On a 4-node cluster with 2 local SATA on each node you should easily read 400MB/sec in aggregate. When you are running the Hadoop cluster, is the Hadoop processing co-located with the Ilsilon nodes? Is Hadoop configured to use OneFS or HDFS? John From: Artem Ervits [mailto:are9...@nyp.org]<mailto:[mailto:are9...@nyp.org]> Sent: Thursday, January 03, 2013 3:00 PM To: user@hadoop.apache.org<mailto:user@hadoop.apache.org> Subject: Hadoop throughput question Hello all, I'd like to pick the community brain on average throughput speeds for a moderately specced 4-node Hadoop cluster with 1GigE networking. Is it reasonable to expect constant average speeds of 150-200mb/sec on such setup? Forgive me if the question is loaded but we're Hadoop cluster with HDFS served via EMC Isilon storage. We're getting about 30mb/sec with our machines and we do not see a difference in job speed between 2 node cluster and 4 node cluster. Thank you. This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you. This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you. This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you. This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you.
RE: Hadoop throughput question
Hadoop is using OneFS, not HDFS in our configuration. Isilon NAS and the Hadoop nodes are in the same datacenter but as far as rack locations, I cannot tell. From: John Lilley [mailto:john.lil...@redpoint.net] Sent: Thursday, January 03, 2013 5:15 PM To: user@hadoop.apache.org Subject: RE: Hadoop throughput question Let's suppose you are doing a read-intensive job like, for example, counting records. This is will be disk bandwidth limited. On a 4-node cluster with 2 local SATA on each node you should easily read 400MB/sec in aggregate. When you are running the Hadoop cluster, is the Hadoop processing co-located with the Ilsilon nodes? Is Hadoop configured to use OneFS or HDFS? John From: Artem Ervits [mailto:are9...@nyp.org]<mailto:[mailto:are9...@nyp.org]> Sent: Thursday, January 03, 2013 3:00 PM To: user@hadoop.apache.org<mailto:user@hadoop.apache.org> Subject: Hadoop throughput question Hello all, I'd like to pick the community brain on average throughput speeds for a moderately specced 4-node Hadoop cluster with 1GigE networking. Is it reasonable to expect constant average speeds of 150-200mb/sec on such setup? Forgive me if the question is loaded but we're Hadoop cluster with HDFS served via EMC Isilon storage. We're getting about 30mb/sec with our machines and we do not see a difference in job speed between 2 node cluster and 4 node cluster. Thank you. This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you. This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you.
RE: Hadoop throughput question
I have a 4.5gb file with records in SequenceFile format. If I use SequenceFile.Reader class to count the records in this file, which amount to 5.5million records, it takes 176seconds, or roughly 26mb/sec. Thank you. From: Michael Katzenellenbogen [mailto:mich...@cloudera.com] Sent: Thursday, January 03, 2013 5:08 PM To: user@hadoop.apache.org Subject: Re: Hadoop throughput question Loaded question indeed. How are you measuring that 30mb/s? Is that per machine / NIC? HDFS throughout? Some other metric? -Michael On Jan 3, 2013, at 5:01 PM, Artem Ervits mailto:are9...@nyp.org>> wrote: Hello all, I'd like to pick the community brain on average throughput speeds for a moderately specced 4-node Hadoop cluster with 1GigE networking. Is it reasonable to expect constant average speeds of 150-200mb/sec on such setup? Forgive me if the question is loaded but we're Hadoop cluster with HDFS served via EMC Isilon storage. We're getting about 30mb/sec with our machines and we do not see a difference in job speed between 2 node cluster and 4 node cluster. Thank you. This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you. This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you. This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you. This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you.
Hadoop throughput question
Hello all, I'd like to pick the community brain on average throughput speeds for a moderately specced 4-node Hadoop cluster with 1GigE networking. Is it reasonable to expect constant average speeds of 150-200mb/sec on such setup? Forgive me if the question is loaded but we're Hadoop cluster with HDFS served via EMC Isilon storage. We're getting about 30mb/sec with our machines and we do not see a difference in job speed between 2 node cluster and 4 node cluster. Thank you. This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you. This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you.
RE: Hadoop on Isilon problem
Yup, that was it. I confused this tmp with another tmp we created before.
Thank you.
-Original Message-
From: Harsh J [mailto:ha...@cloudera.com]
Sent: Wednesday, October 17, 2012 3:36 PM
To: user@hadoop.apache.org
Subject: Re: Hadoop on Isilon problem
Hi Artem,
I don't think this is a Isilon issue as much as its a /tmp directory permission
issue. Ensure that /tmp has proper world+rw permissions, OR if thats not
doable, set the JVM system property of -Djava.io.tmpdir to a local user
writable directory and also re-point config hadoop.tmp.dir to such a thing
thats non /tmp but writable by the user (/home/${user.name}/.tmp maybe).
Of course, easiest is just to fix the /tmp to be drwxrwxrwt,
On Thu, Oct 18, 2012 at 12:48 AM, Artem Ervits wrote:
> Anyone using Hadoop running on Isilon NAS? I am trying to submit a job
> with a user other than the one running Hadoop and I'm getting the
> following
> error:
>
>
>
> Exception in thread "main" java.io.IOException: Permission denied
>
> at java.io.UnixFileSystem.createFileExclusively(Native Method)
>
> at java.io.File.checkAndCreate(File.java:1717)
>
> at java.io.File.createTempFile0(File.java:1738)
>
> at java.io.File.createTempFile(File.java:1815)
>
> at org.apache.hadoop.util.RunJar.main(RunJar.java:115)
>
>
>
>
>
> Any ideas?
>
>
>
>
>
> Artem Ervits
>
> Data Analyst
>
> New York Presbyterian Hospital
>
>
>
>
>
> This electronic message is intended to be for the use only of the
> named recipient, and may contain information that is confidential or
> privileged.
> If you are not the intended recipient, you are hereby notified that
> any disclosure, copying, distribution or use of the contents of this
> message is strictly prohibited. If you have received this message in
> error or are not the named recipient, please notify us immediately by
> contacting the sender at the electronic mail address noted above, and
> delete and destroy all copies of this message. Thank you.
>
>
>
> This electronic message is intended to be for the use only of the
> named recipient, and may contain information that is confidential or
> privileged.
> If you are not the intended recipient, you are hereby notified that
> any disclosure, copying, distribution or use of the contents of this
> message is strictly prohibited. If you have received this message in
> error or are not the named recipient, please notify us immediately by
> contacting the sender at the electronic mail address noted above, and
> delete and destroy all copies of this message. Thank you.
>
>
--
Harsh J
This electronic message is intended to be for the use only of the named
recipient, and may contain information that is confidential or privileged. If
you are not the intended recipient, you are hereby notified that any
disclosure, copying, distribution or use of the contents of this message is
strictly prohibited. If you have received this message in error or are not the
named recipient, please notify us immediately by contacting the sender at the
electronic mail address noted above, and delete and destroy all copies of this
message. Thank you.
This electronic message is intended to be for the use only of the named
recipient, and may contain information that is confidential or privileged. If
you are not the intended recipient, you are hereby notified that any
disclosure, copying, distribution or use of the contents of this message is
strictly prohibited. If you have received this message in error or are not the
named recipient, please notify us immediately by contacting the sender at the
electronic mail address noted above, and delete and destroy all copies of this
message. Thank you.
Re: Hadoop on Isilon problem
With Isilon, there is no need for hdfs-site configuration file. Isilon takes care of replication, although you can certainly add hadoop replication. The biggest plus is the scalability for storage layer. We keep a lot of our data in Isilon and importing into hdfs will result in two locations of massive amounts of data. With this solution, we can read and write in one location. Performance wise, I cannot answer yet, so far our standard cluster runs jobs 3x faster than Isilon but we have many factors why it is so and results are inconclusive. Artem Ervits Data Analyst New York Presbyterian Hospital From: Rita [mailto:rmorgan...@gmail.com] Sent: Wednesday, October 17, 2012 08:23 PM To: user@hadoop.apache.org Subject: Re: Hadoop on Isilon problem out of curiosity, what does running HDFS give you when running thru an Isilon cluster? On Wed, Oct 17, 2012 at 3:59 PM, Mohit Anchlia mailto:mohitanch...@gmail.com>> wrote: Look at the directory permissions? On Wed, Oct 17, 2012 at 12:18 PM, Artem Ervits mailto:are9...@nyp.org>> wrote: Anyone using Hadoop running on Isilon NAS? I am trying to submit a job with a user other than the one running Hadoop and I’m getting the following error: Exception in thread "main" java.io.IOException: Permission denied at java.io.UnixFileSystem.createFileExclusively(Native Method) at java.io.File.checkAndCreate(File.java:1717) at java.io.File.createTempFile0(File.java:1738) at java.io.File.createTempFile(File.java:1815) at org.apache.hadoop.util.RunJar.main(RunJar.java:115) Any ideas? Artem Ervits Data Analyst New York Presbyterian Hospital This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you. This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you. -- --- Get your facts first, then you can distort them as you please.-- This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you. This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you.
Re: Hadoop on Isilon problem
I will try that first thing in the morning, thank you very much!
Artem Ervits
Data Analyst
New York Presbyterian Hospital
- Original Message -
From: Harsh J [mailto:ha...@cloudera.com]
Sent: Wednesday, October 17, 2012 03:36 PM
To: user@hadoop.apache.org
Subject: Re: Hadoop on Isilon problem
Hi Artem,
I don't think this is a Isilon issue as much as its a /tmp directory
permission issue. Ensure that /tmp has proper world+rw permissions, OR
if thats not doable, set the JVM system property of -Djava.io.tmpdir
to a local user writable directory and also re-point config
hadoop.tmp.dir to such a thing thats non /tmp but writable by the user
(/home/${user.name}/.tmp maybe).
Of course, easiest is just to fix the /tmp to be drwxrwxrwt,
On Thu, Oct 18, 2012 at 12:48 AM, Artem Ervits wrote:
> Anyone using Hadoop running on Isilon NAS? I am trying to submit a job with
> a user other than the one running Hadoop and I’m getting the following
> error:
>
>
>
> Exception in thread "main" java.io.IOException: Permission denied
>
> at java.io.UnixFileSystem.createFileExclusively(Native Method)
>
> at java.io.File.checkAndCreate(File.java:1717)
>
> at java.io.File.createTempFile0(File.java:1738)
>
> at java.io.File.createTempFile(File.java:1815)
>
> at org.apache.hadoop.util.RunJar.main(RunJar.java:115)
>
>
>
>
>
> Any ideas?
>
>
>
>
>
> Artem Ervits
>
> Data Analyst
>
> New York Presbyterian Hospital
>
>
>
>
>
> This electronic message is intended to be for the use only of the named
> recipient, and may contain information that is confidential or privileged.
> If you are not the intended recipient, you are hereby notified that any
> disclosure, copying, distribution or use of the contents of this message is
> strictly prohibited. If you have received this message in error or are not
> the named recipient, please notify us immediately by contacting the sender
> at the electronic mail address noted above, and delete and destroy all
> copies of this message. Thank you.
>
>
>
> This electronic message is intended to be for the use only of the named
> recipient, and may contain information that is confidential or privileged.
> If you are not the intended recipient, you are hereby notified that any
> disclosure, copying, distribution or use of the contents of this message is
> strictly prohibited. If you have received this message in error or are not
> the named recipient, please notify us immediately by contacting the sender
> at the electronic mail address noted above, and delete and destroy all
> copies of this message. Thank you.
>
>
--
Harsh J
This electronic message is intended to be for the use only of the named
recipient, and may contain information that is confidential or privileged. If
you are not the intended recipient, you are hereby notified that any
disclosure, copying, distribution or use of the contents of this message is
strictly prohibited. If you have received this message in error or are not the
named recipient, please notify us immediately by contacting the sender at the
electronic mail address noted above, and delete and destroy all copies of this
message. Thank you.
This electronic message is intended to be for the use only of the named
recipient, and may contain information that is confidential or privileged. If
you are not the intended recipient, you are hereby notified that any
disclosure, copying, distribution or use of the contents of this message is
strictly prohibited. If you have received this message in error or are not the
named recipient, please notify us immediately by contacting the sender at the
electronic mail address noted above, and delete and destroy all copies of this
message. Thank you.
Hadoop on Isilon problem
Anyone using Hadoop running on Isilon NAS? I am trying to submit a job with a user other than the one running Hadoop and I'm getting the following error: Exception in thread "main" java.io.IOException: Permission denied at java.io.UnixFileSystem.createFileExclusively(Native Method) at java.io.File.checkAndCreate(File.java:1717) at java.io.File.createTempFile0(File.java:1738) at java.io.File.createTempFile(File.java:1815) at org.apache.hadoop.util.RunJar.main(RunJar.java:115) Any ideas? Artem Ervits Data Analyst New York Presbyterian Hospital This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you. This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you.
RE: Hadoop recovery test
Harsh, That was actually what it was. I was messing with HBase install and edited the /etc/hosts file to add the 127.0.0.1 address. Once I removed the entry, datanodes were able to see the namenode. I was also able to successfully test the recovery. Hadoop fsck -blocks reports a healthy filesystem now. Thank you very much. -Original Message- From: Harsh J [mailto:ha...@cloudera.com] Sent: Tuesday, September 18, 2012 11:37 PM To: user@hadoop.apache.org Subject: Re: Hadoop recovery test Artem, If you check the logs of the other DNs, do you see issues with connectivity to NameNode? Basic questions, but need to ask to be sure: have you checked if the firewalls are down or properly configured? Are you sure that your hostname of the master machine resolves not to the loopback address but to the external interface provided IP? On Tue, Sep 18, 2012 at 10:29 PM, Artem Ervits wrote: > I didn't realize that I didn't edit core-site and mapred-site on all machines > to point to the new namenode. Although that didn't make a difference, I still > see only one datanode which Is also the namenode: > > Datanodes available: 1 (1 total, 0 dead) > > Name: 127.0.0.1:50010 > Decommission Status : Normal > Configured Capacity: 105425190912 (98.18 GB) DFS Used: 1058557952 > (1009.52 MB) Non DFS Used: 200396800 (191.11 MB) DFS Remaining: > 104166236160(97.01 GB) DFS Used%: 1% DFS Remaining%: 98.81% Last > contact: Tue Sep 18 12:58:07 EDT 2012 > > The other strange thing is that it points to local 127.0.0.1 rather than > namenode's IP. > > -Original Message- > From: Artem Ervits [mailto:are9...@nyp.org] > Sent: Tuesday, September 18, 2012 9:57 AM > To: user@hadoop.apache.org > Cc: James Brown > Subject: RE: Hadoop recovery test > > No it only sees itself. It doesn't see the rest of the nodes. > > -Original Message- > From: James Brown [mailto:jb...@syndicate.net] > Sent: Monday, September 17, 2012 5:49 PM > To: user@hadoop.apache.org > Subject: Re: Hadoop recovery test > > Does the new NameNode server see all of the DataNodes? > > On 9/17/2012 2:38 PM, Artem Ervits wrote: >> Hello all, >> >> I am testing the Hadoop recovery as per >> http://wiki.apache.org/hadoop/NameNode document. But instead of using >> an NFS share, I am copying to another directory. Then when I shut >> down the cluster, I scp that directory to another server and start >> Hadoop cluster using that machine as the namenode. I see in the log >> that some blocks are corrupt and/or missing. Do I have to wait for >> replication to recover all blocks or am I doing something else >> altogether? I am using Hadoop 1.0.3. Can someone point me to a more >> detailed document than the wiki in case I'm doing something wrong. >> >> p.s. if I restart the cluster using the original namenode, filesystem >> reports as healthy. >> >> Thank you. >> >> . >> >> /hdfs/hadoop/tmp/mapred/system/jobtracker.info: CORRUPT block >> blk_9043419219670949307 >> >> /hdfs/hadoop/tmp/mapred/system/jobtracker.info: MISSING 1 blocks of >> total size 4 B... >> >> /user/hduser/teragen/_logs/history/job_201209120941_0002_1347458152167_hduser_TeraGen: >> Under replicated blk_-976282286234272458_1079. Target Replicas is 3 >> but found 1 replica(s). >> >> . >> >> /user/hduser/teragen/_logs/history/job_201209120941_0002_conf.xml: >> Under replicated blk_137658109390447967_1075. Target Replicas is 3 >> but found 1 replica(s). >> >> . >> >> /user/hduser/teragen/_partition.lst: Under replicated >> blk_-3005280481530403302_1080. Target Replicas is 3 but found 1 replica(s). >> >> . >> >> /user/hduser/teragen/part-0: Under replicated >> blk_-7008813028808832816_1077. Target Replicas is 3 but found 1 replica(s). >> >> . >> >> /user/hduser/teragen/part-1: Under replicated >> blk_-5256967771026054061_1078. Target Replicas is 3 but found 1 replica(s). >> >> .. >> >> /user/hduser/teragen-out/_logs/history/job_201209120941_0003_1347458249920_hduser_TeraSort: >> Under replicated blk_1137779303840586677_1089. Target Replicas is 3 >> but found 1 replica(s). >> >> . >> >> /user/hduser/teragen-out/_logs/history/job_201209120941_0003_conf.xml: >> Under replicated blk_7701720691642589882_1086. Target Replicas is 3 >> but found 1 replica(s). >> >> . >> >> /user/hduser/teragen-out/part-0: CORRUPT block >> blk_8059469267617478950 >> >> /user/hduser/teragen-out/part-0: MISSING 1 blocks of total size >> 100 B..
RE: Hadoop recovery test
I didn't realize that I didn't edit core-site and mapred-site on all machines to point to the new namenode. Although that didn't make a difference, I still see only one datanode which Is also the namenode: Datanodes available: 1 (1 total, 0 dead) Name: 127.0.0.1:50010 Decommission Status : Normal Configured Capacity: 105425190912 (98.18 GB) DFS Used: 1058557952 (1009.52 MB) Non DFS Used: 200396800 (191.11 MB) DFS Remaining: 104166236160(97.01 GB) DFS Used%: 1% DFS Remaining%: 98.81% Last contact: Tue Sep 18 12:58:07 EDT 2012 The other strange thing is that it points to local 127.0.0.1 rather than namenode's IP. -Original Message- From: Artem Ervits [mailto:are9...@nyp.org] Sent: Tuesday, September 18, 2012 9:57 AM To: user@hadoop.apache.org Cc: James Brown Subject: RE: Hadoop recovery test No it only sees itself. It doesn't see the rest of the nodes. -Original Message- From: James Brown [mailto:jb...@syndicate.net] Sent: Monday, September 17, 2012 5:49 PM To: user@hadoop.apache.org Subject: Re: Hadoop recovery test Does the new NameNode server see all of the DataNodes? On 9/17/2012 2:38 PM, Artem Ervits wrote: > Hello all, > > I am testing the Hadoop recovery as per > http://wiki.apache.org/hadoop/NameNode document. But instead of using > an NFS share, I am copying to another directory. Then when I shut down > the cluster, I scp that directory to another server and start Hadoop > cluster using that machine as the namenode. I see in the log that some > blocks are corrupt and/or missing. Do I have to wait for replication > to recover all blocks or am I doing something else altogether? I am > using Hadoop 1.0.3. Can someone point me to a more detailed document > than the wiki in case I'm doing something wrong. > > p.s. if I restart the cluster using the original namenode, filesystem > reports as healthy. > > Thank you. > > . > > /hdfs/hadoop/tmp/mapred/system/jobtracker.info: CORRUPT block > blk_9043419219670949307 > > /hdfs/hadoop/tmp/mapred/system/jobtracker.info: MISSING 1 blocks of > total size 4 B... > > /user/hduser/teragen/_logs/history/job_201209120941_0002_1347458152167_hduser_TeraGen: > Under replicated blk_-976282286234272458_1079. Target Replicas is 3 > but found 1 replica(s). > > . > > /user/hduser/teragen/_logs/history/job_201209120941_0002_conf.xml: > Under replicated blk_137658109390447967_1075. Target Replicas is 3 but > found 1 replica(s). > > . > > /user/hduser/teragen/_partition.lst: Under replicated > blk_-3005280481530403302_1080. Target Replicas is 3 but found 1 replica(s). > > . > > /user/hduser/teragen/part-0: Under replicated > blk_-7008813028808832816_1077. Target Replicas is 3 but found 1 replica(s). > > . > > /user/hduser/teragen/part-1: Under replicated > blk_-5256967771026054061_1078. Target Replicas is 3 but found 1 replica(s). > > .. > > /user/hduser/teragen-out/_logs/history/job_201209120941_0003_1347458249920_hduser_TeraSort: > Under replicated blk_1137779303840586677_1089. Target Replicas is 3 > but found 1 replica(s). > > . > > /user/hduser/teragen-out/_logs/history/job_201209120941_0003_conf.xml: > Under replicated blk_7701720691642589882_1086. Target Replicas is 3 > but found 1 replica(s). > > . > > /user/hduser/teragen-out/part-0: CORRUPT block > blk_8059469267617478950 > > /user/hduser/teragen-out/part-0: MISSING 1 blocks of total size > 100 B... > > /user/hduser/teragen-validate/_logs/history/job_201209120941_0004_1347458495941_hduser_TeraValidate: > Under replicated blk_5680565744062298575_1098. Target Replicas is 3 > but found 1 replica(s). > > . > > /user/hduser/teragen-validate/_logs/history/job_201209120941_0004_conf.xml: > Under replicated blk_1566253937037013126_1095. Target Replicas is 3 > but found 1 replica(s). > > .Status: CORRUPT > > Total size:1050720258 B > > Total dirs:39 > > Total files: 32 > > Total blocks (validated): 42 (avg. block size 25017149 B) > > > >CORRUPT FILES:2 > >MISSING BLOCKS: 2 > >MISSING SIZE: 104 B > >CORRUPT BLOCKS: 2 > > > > Minimally replicated blocks: 40 (95.2381 %) > > Over-replicated blocks:0 (0.0 %) > > Under-replicated blocks: 40 (95.2381 %) > > Mis-replicated blocks: 0 (0.0 %) > > Default replication factor:3 > > Average block replication: 0.95238096 > > Corrupt blocks:2 > > Missing replicas: 80 (200.0 %) > > Number of data-nodes: 1 > > Number of racks:
RE: Hadoop recovery test
Thanks Harsh, I'm aware of the implications of copying periodically. This is just a test until I get an NFS share to play with. Do you just let Hadoop write to two directories where one is an NFS share or is there another way? -Original Message- From: Harsh J [mailto:ha...@cloudera.com] Sent: Monday, September 17, 2012 10:44 PM To: user@hadoop.apache.org Subject: Re: Hadoop recovery test Hi Artem, You are running 1 DN in this cluster from what I see, and hence you can ignore the reports that go: Under replicated blk_7701720691642589882_1086. Target Replicas is 3 but found 1 replica(s). The two truly missing blocks are: /hdfs/hadoop/tmp/mapred/system/jobtracker.info: MISSING 1 blocks /user/hduser/teragen-out/part-0: MISSING 1 blocks Which may be cause of those being written at the time of your copy of the fsimage and edits (thats a wrong way to go about it, btw - you should configure for redundant writes such that you also sustain failures, not copy it periodically - thats not a consistent way to keep a backup, and you can rather go for dfsadmin methods to fetchImage instead). Does that sound likely? On Tue, Sep 18, 2012 at 3:08 AM, Artem Ervits wrote: > Hello all, > > > > I am testing the Hadoop recovery as per > http://wiki.apache.org/hadoop/NameNode document. But instead of using > an NFS share, I am copying to another directory. Then when I shut down > the cluster, I scp that directory to another server and start Hadoop > cluster using that machine as the namenode. I see in the log that some > blocks are corrupt and/or missing. Do I have to wait for replication > to recover all blocks or am I doing something else altogether? I am > using Hadoop 1.0.3. Can someone point me to a more detailed document > than the wiki in case I'm doing something wrong. > > > > p.s. if I restart the cluster using the original namenode, filesystem > reports as healthy. > > > > Thank you. > > > > . > > /hdfs/hadoop/tmp/mapred/system/jobtracker.info: CORRUPT block > blk_9043419219670949307 > > > > /hdfs/hadoop/tmp/mapred/system/jobtracker.info: MISSING 1 blocks of > total size 4 B... > > /user/hduser/teragen/_logs/history/job_201209120941_0002_1347458152167_hduser_TeraGen: > Under replicated blk_-976282286234272458_1079. Target Replicas is 3 > but found 1 replica(s). > > . > > /user/hduser/teragen/_logs/history/job_201209120941_0002_conf.xml: > Under replicated blk_137658109390447967_1075. Target Replicas is 3 but > found 1 replica(s). > > . > > /user/hduser/teragen/_partition.lst: Under replicated > blk_-3005280481530403302_1080. Target Replicas is 3 but found 1 replica(s). > > . > > /user/hduser/teragen/part-0: Under replicated > blk_-7008813028808832816_1077. Target Replicas is 3 but found 1 replica(s). > > . > > /user/hduser/teragen/part-1: Under replicated > blk_-5256967771026054061_1078. Target Replicas is 3 but found 1 replica(s). > > .. > > /user/hduser/teragen-out/_logs/history/job_201209120941_0003_1347458249920_hduser_TeraSort: > Under replicated blk_1137779303840586677_1089. Target Replicas is 3 > but found 1 replica(s). > > . > > /user/hduser/teragen-out/_logs/history/job_201209120941_0003_conf.xml: > Under replicated blk_7701720691642589882_1086. Target Replicas is 3 > but found 1 replica(s). > > . > > /user/hduser/teragen-out/part-0: CORRUPT block > blk_8059469267617478950 > > > > /user/hduser/teragen-out/part-0: MISSING 1 blocks of total size > 100 B... > > /user/hduser/teragen-validate/_logs/history/job_201209120941_0004_1347458495941_hduser_TeraValidate: > Under replicated blk_5680565744062298575_1098. Target Replicas is 3 > but found 1 replica(s). > > . > > /user/hduser/teragen-validate/_logs/history/job_201209120941_0004_conf.xml: > Under replicated blk_1566253937037013126_1095. Target Replicas is 3 > but found 1 replica(s). > > .Status: CORRUPT > > Total size:1050720258 B > > Total dirs:39 > > Total files: 32 > > Total blocks (validated): 42 (avg. block size 25017149 B) > > > > CORRUPT FILES:2 > > MISSING BLOCKS: 2 > > MISSING SIZE: 104 B > > CORRUPT BLOCKS: 2 > > > > Minimally replicated blocks: 40 (95.2381 %) > > Over-replicated blocks:0 (0.0 %) > > Under-replicated blocks: 40 (95.2381 %) > > Mis-replicated blocks: 0 (0.0 %) > > Default replication factor:3 > > Average block replication: 0.95238096 > > Corrupt blocks:2 > > Missing replicas: 80 (200.0 %) &g
RE: Hadoop recovery test
I tried copying two ways. Once while Hadoop was running and second time when I shut down the original cluster. I used scp -r command, is there a better option, rsync? I also tried scp with -rp4 switch but still can't get the folders to look identical Original node: total 20 drwxr-xr-x. 16 hduser hadoop 4096 Sep 17 13:31 .. drwxrwx---. 2 hduser hadoop 4096 Sep 17 13:41 image drwxrwx---. 2 hduser hadoop 4096 Sep 17 16:39 previous.checkpoint drwxrwx---. 2 hduser hadoop 4096 Sep 18 09:57 current drwxrwx---. 5 hduser hadoop 4096 Sep 18 09:59 . new namenode: total 20 drwxrwx---. 2 hduser hadoop 4096 Sep 17 13:41 image drwxrwx---. 2 hduser hadoop 4096 Sep 17 16:39 previous.checkpoint drwxrwx---. 2 hduser hadoop 4096 Sep 18 09:57 current drwxrwx---. 5 hduser hadoop 4096 Sep 18 09:59 . drwxr-xr-x. 17 hduser hadoop 4096 Sep 18 10:03 .. From: Robert Molina [mailto:rmol...@hortonworks.com] Sent: Monday, September 17, 2012 5:55 PM To: user@hadoop.apache.org Subject: Re: Hadoop recovery test Hi Artem, At what point do you do the copy, was namenode still running? Does the copy of the edits file and fsimage file match up with the original (i.e filesize)? -Robert On Mon, Sep 17, 2012 at 2:38 PM, Artem Ervits mailto:are9...@nyp.org>> wrote: Hello all, I am testing the Hadoop recovery as per http://wiki.apache.org/hadoop/NameNode document. But instead of using an NFS share, I am copying to another directory. Then when I shut down the cluster, I scp that directory to another server and start Hadoop cluster using that machine as the namenode. I see in the log that some blocks are corrupt and/or missing. Do I have to wait for replication to recover all blocks or am I doing something else altogether? I am using Hadoop 1.0.3. Can someone point me to a more detailed document than the wiki in case I'm doing something wrong. p.s. if I restart the cluster using the original namenode, filesystem reports as healthy. Thank you. . /hdfs/hadoop/tmp/mapred/system/jobtracker.info<http://jobtracker.info>: CORRUPT block blk_9043419219670949307 /hdfs/hadoop/tmp/mapred/system/jobtracker.info<http://jobtracker.info>: MISSING 1 blocks of total size 4 B... /user/hduser/teragen/_logs/history/job_201209120941_0002_1347458152167_hduser_TeraGen: Under replicated blk_-976282286234272458_1079. Target Replicas is 3 but found 1 replica(s). . /user/hduser/teragen/_logs/history/job_201209120941_0002_conf.xml: Under replicated blk_137658109390447967_1075. Target Replicas is 3 but found 1 replica(s). . /user/hduser/teragen/_partition.lst: Under replicated blk_-3005280481530403302_1080. Target Replicas is 3 but found 1 replica(s). . /user/hduser/teragen/part-0: Under replicated blk_-7008813028808832816_1077. Target Replicas is 3 but found 1 replica(s). . /user/hduser/teragen/part-1: Under replicated blk_-5256967771026054061_1078. Target Replicas is 3 but found 1 replica(s). .. /user/hduser/teragen-out/_logs/history/job_201209120941_0003_1347458249920_hduser_TeraSort: Under replicated blk_1137779303840586677_1089. Target Replicas is 3 but found 1 replica(s). . /user/hduser/teragen-out/_logs/history/job_201209120941_0003_conf.xml: Under replicated blk_7701720691642589882_1086. Target Replicas is 3 but found 1 replica(s). . /user/hduser/teragen-out/part-0: CORRUPT block blk_8059469267617478950 /user/hduser/teragen-out/part-0: MISSING 1 blocks of total size 100 B... /user/hduser/teragen-validate/_logs/history/job_201209120941_0004_1347458495941_hduser_TeraValidate: Under replicated blk_5680565744062298575_1098. Target Replicas is 3 but found 1 replica(s). . /user/hduser/teragen-validate/_logs/history/job_201209120941_0004_conf.xml: Under replicated blk_1566253937037013126_1095. Target Replicas is 3 but found 1 replica(s). .Status: CORRUPT Total size:1050720258 B Total dirs:39 Total files: 32 Total blocks (validated): 42 (avg. block size 25017149 B) CORRUPT FILES:2 MISSING BLOCKS: 2 MISSING SIZE: 104 B CORRUPT BLOCKS: 2 Minimally replicated blocks: 40 (95.2381 %) Over-replicated blocks:0 (0.0 %) Under-replicated blocks: 40 (95.2381 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor:3 Average block replication: 0.95238096 Corrupt blocks:2 Missing replicas: 80 (200.0 %) Number of data-nodes: 1 Number of racks: 1 FSCK ended at Mon Sep 17 17:29:08 EDT 2012 in 21 milliseconds The filesystem under path '/' is CORRUPT Artem Ervits Data Analyst New York Presbyterian Hospital This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any discl
RE: Hadoop recovery test
No it only sees itself. It doesn't see the rest of the nodes. -Original Message- From: James Brown [mailto:jb...@syndicate.net] Sent: Monday, September 17, 2012 5:49 PM To: user@hadoop.apache.org Subject: Re: Hadoop recovery test Does the new NameNode server see all of the DataNodes? On 9/17/2012 2:38 PM, Artem Ervits wrote: > Hello all, > > I am testing the Hadoop recovery as per > http://wiki.apache.org/hadoop/NameNode document. But instead of using > an NFS share, I am copying to another directory. Then when I shut down > the cluster, I scp that directory to another server and start Hadoop > cluster using that machine as the namenode. I see in the log that some > blocks are corrupt and/or missing. Do I have to wait for replication > to recover all blocks or am I doing something else altogether? I am > using Hadoop 1.0.3. Can someone point me to a more detailed document > than the wiki in case I'm doing something wrong. > > p.s. if I restart the cluster using the original namenode, filesystem > reports as healthy. > > Thank you. > > . > > /hdfs/hadoop/tmp/mapred/system/jobtracker.info: CORRUPT block > blk_9043419219670949307 > > /hdfs/hadoop/tmp/mapred/system/jobtracker.info: MISSING 1 blocks of > total size 4 B... > > /user/hduser/teragen/_logs/history/job_201209120941_0002_1347458152167_hduser_TeraGen: > Under replicated blk_-976282286234272458_1079. Target Replicas is 3 > but found 1 replica(s). > > . > > /user/hduser/teragen/_logs/history/job_201209120941_0002_conf.xml: > Under replicated blk_137658109390447967_1075. Target Replicas is 3 but > found 1 replica(s). > > . > > /user/hduser/teragen/_partition.lst: Under replicated > blk_-3005280481530403302_1080. Target Replicas is 3 but found 1 replica(s). > > . > > /user/hduser/teragen/part-0: Under replicated > blk_-7008813028808832816_1077. Target Replicas is 3 but found 1 replica(s). > > . > > /user/hduser/teragen/part-1: Under replicated > blk_-5256967771026054061_1078. Target Replicas is 3 but found 1 replica(s). > > .. > > /user/hduser/teragen-out/_logs/history/job_201209120941_0003_1347458249920_hduser_TeraSort: > Under replicated blk_1137779303840586677_1089. Target Replicas is 3 > but found 1 replica(s). > > . > > /user/hduser/teragen-out/_logs/history/job_201209120941_0003_conf.xml: > Under replicated blk_7701720691642589882_1086. Target Replicas is 3 > but found 1 replica(s). > > . > > /user/hduser/teragen-out/part-0: CORRUPT block > blk_8059469267617478950 > > /user/hduser/teragen-out/part-0: MISSING 1 blocks of total size > 100 B... > > /user/hduser/teragen-validate/_logs/history/job_201209120941_0004_1347458495941_hduser_TeraValidate: > Under replicated blk_5680565744062298575_1098. Target Replicas is 3 > but found 1 replica(s). > > . > > /user/hduser/teragen-validate/_logs/history/job_201209120941_0004_conf.xml: > Under replicated blk_1566253937037013126_1095. Target Replicas is 3 > but found 1 replica(s). > > .Status: CORRUPT > > Total size:1050720258 B > > Total dirs:39 > > Total files: 32 > > Total blocks (validated): 42 (avg. block size 25017149 B) > > > >CORRUPT FILES:2 > >MISSING BLOCKS: 2 > >MISSING SIZE: 104 B > >CORRUPT BLOCKS: 2 > > > > Minimally replicated blocks: 40 (95.2381 %) > > Over-replicated blocks:0 (0.0 %) > > Under-replicated blocks: 40 (95.2381 %) > > Mis-replicated blocks: 0 (0.0 %) > > Default replication factor:3 > > Average block replication: 0.95238096 > > Corrupt blocks:2 > > Missing replicas: 80 (200.0 %) > > Number of data-nodes: 1 > > Number of racks: 1 > > FSCK ended at Mon Sep 17 17:29:08 EDT 2012 in 21 milliseconds > > The filesystem under path '/' is CORRUPT > > Artem Ervits > > Data Analyst > > New York Presbyterian Hospital > > > -- > -- This electronic message is intended to be for the use only of the > named recipient, and may contain information that is confidential or > privileged. If you are not the intended recipient, you are hereby > notified that any disclosure, copying, distribution or use of the > contents of this message is strictly prohibited. If you have received > this message in error or are not the named recipient, please notify us > immediately by contacting the sender at the electronic mail address >
Hadoop recovery test
Hello all, I am testing the Hadoop recovery as per http://wiki.apache.org/hadoop/NameNode document. But instead of using an NFS share, I am copying to another directory. Then when I shut down the cluster, I scp that directory to another server and start Hadoop cluster using that machine as the namenode. I see in the log that some blocks are corrupt and/or missing. Do I have to wait for replication to recover all blocks or am I doing something else altogether? I am using Hadoop 1.0.3. Can someone point me to a more detailed document than the wiki in case I'm doing something wrong. p.s. if I restart the cluster using the original namenode, filesystem reports as healthy. Thank you. . /hdfs/hadoop/tmp/mapred/system/jobtracker.info: CORRUPT block blk_9043419219670949307 /hdfs/hadoop/tmp/mapred/system/jobtracker.info: MISSING 1 blocks of total size 4 B... /user/hduser/teragen/_logs/history/job_201209120941_0002_1347458152167_hduser_TeraGen: Under replicated blk_-976282286234272458_1079. Target Replicas is 3 but found 1 replica(s). . /user/hduser/teragen/_logs/history/job_201209120941_0002_conf.xml: Under replicated blk_137658109390447967_1075. Target Replicas is 3 but found 1 replica(s). . /user/hduser/teragen/_partition.lst: Under replicated blk_-3005280481530403302_1080. Target Replicas is 3 but found 1 replica(s). . /user/hduser/teragen/part-0: Under replicated blk_-7008813028808832816_1077. Target Replicas is 3 but found 1 replica(s). . /user/hduser/teragen/part-1: Under replicated blk_-5256967771026054061_1078. Target Replicas is 3 but found 1 replica(s). .. /user/hduser/teragen-out/_logs/history/job_201209120941_0003_1347458249920_hduser_TeraSort: Under replicated blk_1137779303840586677_1089. Target Replicas is 3 but found 1 replica(s). . /user/hduser/teragen-out/_logs/history/job_201209120941_0003_conf.xml: Under replicated blk_7701720691642589882_1086. Target Replicas is 3 but found 1 replica(s). . /user/hduser/teragen-out/part-0: CORRUPT block blk_8059469267617478950 /user/hduser/teragen-out/part-0: MISSING 1 blocks of total size 100 B... /user/hduser/teragen-validate/_logs/history/job_201209120941_0004_1347458495941_hduser_TeraValidate: Under replicated blk_5680565744062298575_1098. Target Replicas is 3 but found 1 replica(s). . /user/hduser/teragen-validate/_logs/history/job_201209120941_0004_conf.xml: Under replicated blk_1566253937037013126_1095. Target Replicas is 3 but found 1 replica(s). .Status: CORRUPT Total size:1050720258 B Total dirs:39 Total files: 32 Total blocks (validated): 42 (avg. block size 25017149 B) CORRUPT FILES:2 MISSING BLOCKS: 2 MISSING SIZE: 104 B CORRUPT BLOCKS: 2 Minimally replicated blocks: 40 (95.2381 %) Over-replicated blocks:0 (0.0 %) Under-replicated blocks: 40 (95.2381 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor:3 Average block replication: 0.95238096 Corrupt blocks:2 Missing replicas: 80 (200.0 %) Number of data-nodes: 1 Number of racks: 1 FSCK ended at Mon Sep 17 17:29:08 EDT 2012 in 21 milliseconds The filesystem under path '/' is CORRUPT Artem Ervits Data Analyst New York Presbyterian Hospital This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you. This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you. This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named re
Re: Hadoop and MainFrame integration
Can you read the data off backup tapes and dump it to flat files? Artem Ervits Data Analyst New York Presbyterian Hospital From: Marcos Ortiz [mailto:mlor...@uci.cu] Sent: Tuesday, August 28, 2012 06:51 PM To: user@hadoop.apache.org Cc: Siddharth Tiwari Subject: Re: Hadoop and MainFrame integration The problem with it, is that Hadoop depends on top of HDFS to storage in blocks of 64/128 MB of size (or the size that you determine, 64 MB is the de-facto size), and then make the calculations. So, you need to move all your data to a HDFS cluster to use data in MapReduce jobs if you want to make the calculations with Hadoop. Best wishes El 28/08/2012 12:24, Siddharth Tiwari escribió: Hi Users. We have flat files on mainframes with around a billion records. We need to sort them and then use them with different jobs on mainframe for report generation. I was wondering was there any way I could integrate the mainframe with hadoop do the sorting and keep the file on the sever itself ( I do not want to ftp the file to a hadoop cluster and then ftp back the sorted file to Mainframe as it would waste MIPS and nullify the advantage ). This way I could save on MIPS and ultimately improve profitability. Thank you in advance ** Cheers !!! Siddharth Tiwari Have a refreshing day !!! "Every duty is holy, and devotion to duty is the highest form of worship of God.� "Maybe other people will try to limit me but I don't limit myself" [http://universidad.uci.cu/email.gif] <http://www.uci.cu/> [http://universidad.uci.cu/email.gif] <http://www.uci.cu/> This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you. This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you. This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you.
Re: hadoop download path missing
I also tried to download hadoop last night and all mirrors I'd tried returned error. At the end I gave up and searched my work pc for a download file. Artem Ervits Data Analyst New York Presbyterian Hospital - Original Message - From: Harsh J [mailto:ha...@cloudera.com] Sent: Saturday, August 25, 2012 10:11 AM To: user@hadoop.apache.org Subject: Re: hadoop download path missing Hi Steven, Please file an INFRA issue with a list of URLs that are inaccessible. I tried a few and found them to work at the moment. It may have been a transient issue? On Fri, Aug 24, 2012 at 9:10 PM, Steven Willis wrote: > Hi Sonal, > > > > Yes, that URL is working, but the vast majority of the mirrors listed at: > http://www.apache.org/dyn/closer.cgi/hadoop/core/ which is what > http://hadoop.apache.org/common/releases.html#Download points users to are > returning 404s. In fact I don’t even see techartifact on the list of > mirrors. > > > > -Steve Willis > > > > From: Sonal Goyal [mailto:sonalgoy...@gmail.com] > Sent: Friday, August 24, 2012 11:35 AM > To: user@hadoop.apache.org > Subject: Re: hadoop download path missing > > > > I just tried and could go to > http://apache.techartifact.com/mirror/hadoop/common/hadoop-2.0.1-alpha/ > > > > Is this still happening for you? > > > Best Regards, > Sonal > Crux: Reporting for HBase > Nube Technologies > > > > > > > On Fri, Aug 24, 2012 at 8:59 PM, Steven Willis wrote: > > All the links at: http://www.apache.org/dyn/closer.cgi/hadoop/common/ are > returning 404s, even the backup site at: > http://www.us.apache.org/dist/hadoop/common/. However, the eu site: > http://www.eu.apache.org/dist/hadoop/common/ does work. > > -Steven Willis > > -- Harsh J This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you. This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you. This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you.
Hadoop web interface authentication
Hello all, I added the following setting hadoop.http.authentication.simple.anonymous.allowed. Now when I try to login to Web UI, I get 401 error unless I specify user.name=username. This is exactly what I want but I noticed that I can pass any user name and it will work as long as it is not null. What else am I missing from the core-site file to make it work? Also, is there a way I can add a configuration so that a password would be required to get access? I am aware of Kerberos authentication method but I have it configured with Active directory and until I find the correct location of the keytab, I can't use it. Thanks all. Artem Ervits Data Analyst New York Presbyterian Hospital This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you. This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you. This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you.
Hadoop web interface authentication
Hello all, I added the following setting hadoop.http.authentication.simple.anonymous.allowed. Now when I try to login to Web UI, I get 401 error unless I specify user.name=username. This is exactly what I want but I noticed that I can pass any user name and it will work as long as it is not null. What else am I missing from the core-site file to make it work? Also, is there a way I can add a configuration so that a password would be required to get access? I am aware of Kerberos authentication method but I have it configured with Active directory and until I find the correct location of the keytab, I can't use it. Thanks all. Artem Ervits Data Analyst New York Presbyterian Hospital This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you. This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you. This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you.
RE: Setting up HTTP authentication
This was a version dependency issue. The class is not in 0.20.203.0. From: Artem Ervits [mailto:are9...@nyp.org] Sent: Wednesday, August 08, 2012 2:34 PM To: user@hadoop.apache.org Subject: Setting up HTTP authentication Hello all, I followed the 1.0.3 docs to setup http simple authentication. I'd like to pass a username to open the web interface. My new settings for core-site.xml are below: hadoop.http.filter.initializers org.apache.hadoop.http.lib.StaticUserWebFilter,org.apache.hadoop.security.AuthenticationFilterInitializer HTTP Authentication document in hadoop tar file hadoop.http.authentication.type simple authentication type for web UI hadoop.http.authentication.token.validity 36000 how long authentication token is valid before it needs to be renewed hadoop.http.authentication.signature.secret test signature secret for signing authentication tokens hadoop.http.authentication.cookie.domain domain to use for the http cookie that stores authentication token hadoop.http.authentication.simple.anonymous.allowed false anonymous web UI requests enabled or disabled Unfortunately, I don't have a 1.0.3 hadoop cluster handy and am testing this on 0.20.203.0. When I start the cluster, jobtracker doesn't start. Looking at the logs, I see the following: tasktracker log: 2012-08-08 13:40:26,178 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already exists! and 2012-08-08 13:40:26,445 ERROR org.apache.hadoop.mapred.TaskTracker: Can not start task tracker because java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.hadoop.security.AuthenticationFilterInitializer Namenode log shows: 2012-08-08 14:18:17,839 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: master/10.0.0.101:54310. Already tried 9 time(s). 2012-08-08 14:18:17,839 INFO org.apache.hadoop.ipc.RPC: Server at master/10.0.0.101:54310 not available yet, Z... 2012-08-08 14:18:19,841 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: master/10.0.0.101:54310. Already tried 0 time(s). My question is, is the org.apache.hadoop.security.AuthenticationFilterInitiazlier only available in 1.* releases or the problem is somewhere else? Thank you. Artem Ervits Data Analyst New York Presbyterian Hospital This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you. This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you. This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you. Confidential Information subject to NYP's (and its affiliates') information management and security policies (http://infonet.nyp.org/QA/HospitalManual). This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you. This electronic message is intended to be for the use only of the named recipient, and may co
Setting up HTTP authentication
Hello all, I followed the 1.0.3 docs to setup http simple authentication. I'd like to pass a username to open the web interface. My new settings for core-site.xml are below: hadoop.http.filter.initializers org.apache.hadoop.http.lib.StaticUserWebFilter,org.apache.hadoop.security.AuthenticationFilterInitializer HTTP Authentication document in hadoop tar file hadoop.http.authentication.type simple authentication type for web UI hadoop.http.authentication.token.validity 36000 how long authentication token is valid before it needs to be renewed hadoop.http.authentication.signature.secret test signature secret for signing authentication tokens hadoop.http.authentication.cookie.domain domain to use for the http cookie that stores authentication token hadoop.http.authentication.simple.anonymous.allowed false anonymous web UI requests enabled or disabled Unfortunately, I don't have a 1.0.3 hadoop cluster handy and am testing this on 0.20.203.0. When I start the cluster, jobtracker doesn't start. Looking at the logs, I see the following: tasktracker log: 2012-08-08 13:40:26,178 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already exists! and 2012-08-08 13:40:26,445 ERROR org.apache.hadoop.mapred.TaskTracker: Can not start task tracker because java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.hadoop.security.AuthenticationFilterInitializer Namenode log shows: 2012-08-08 14:18:17,839 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: master/10.0.0.101:54310. Already tried 9 time(s). 2012-08-08 14:18:17,839 INFO org.apache.hadoop.ipc.RPC: Server at master/10.0.0.101:54310 not available yet, Z... 2012-08-08 14:18:19,841 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: master/10.0.0.101:54310. Already tried 0 time(s). My question is, is the org.apache.hadoop.security.AuthenticationFilterInitiazlier only available in 1.* releases or the problem is somewhere else? Thank you. Artem Ervits Data Analyst New York Presbyterian Hospital This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you. This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you. This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you.
