Re: Need info on mapred.child.java.opts, mapred.map.child.java.opts and mapred.reduce.child.java.opts
Its easier to understand if you know the history. First there was just "mapred.child.java.opts", which controlled java options for both Map and Reduce tasks (i.e. all tasks). Then there came a need for task-specific java opts, so the project introduced mapred.map.child.java.opts and mapred.reduce.child.java.opts, while keeping around mapred.child.java.opts. Hence, if mapred.map.child.java.opts is present, it is preferred over the mapred.child.java.opts, likewise for mapred.reduce.child.java.opts vs. mapred.child.java.opts. If neither of the specifics is present, we look for mapred.child.java.opts. P.s. Please do not cross-post to multiple email lists; it is a bad practice and potentially spawns two different diverging conversation threads on the same topic. This question is apt-enough for user@hadoop.apache.org alone as it is not CDH specific, so I've moved cdh-u...@cloudera.org to bcc. On Fri, Mar 8, 2013 at 3:41 PM, Gaurav Dasgupta wrote: > Hi, > > While I was reading about the important Hadoop configuration properties, I > came across a state of doubt regarding the Java heap space properties for > the child tasks. According to my understanding, mapred.child.java.opts is > the overall heap size allocated to any task (map or reduce). Then when we > are setting mapred.map.child.java.opts and mapred.reduce.child.java.opts > separately, are they overriding the mapred.child.java.opts? > > For example, if I have the following configuration: > mapred.child.java.opts = -Xmx1g > > mapred.map.child.java.opts = -Xmx2g > > mapred.reduce.child.java.opts = -Xmx512m > > > Then how exactly the memory allocation is getting distributed between map > and reduce? My mapper gets more than the overall heap space as specified or > it is restricted to 1g? > Can some one help me understand this concept? Also, what are the other heap > space related properties which we can use with the above and how? > > Thanks, > Gaurav -- Harsh J
Re: OutOfMemory during Plain Java MapReduce
-
>> // The Custom Writable
>> // Needed to implement a own toString Method bring the output into the
>> right format. Maybe i can to this also with a own OutputFormat class.
>> //---
>> public class UserSetWritable implements Writable {
>> private final Set userIds = new HashSet();
>>
>> public void add(final String userId) {
>> this.userIds.add(userId);
>> }
>>
>> @Override
>> public void write(final DataOutput out) throws IOException {
>> out.writeInt(this.userIds.size());
>> for (final String userId : this.userIds) {
>> out.writeUTF(userId);
>> }
>> }
>>
>> @Override
>> public void readFields(final DataInput in) throws IOException {
>> final int size = in.readInt();
>> for (int i = 0; i < size; i++) {
>> final String readUTF = in.readUTF();
>> this.userIds.add(readUTF);
>> }
>> }
>>
>> @Override
>> public String toString() {
>> String result = "";
>> for (final String userId : this.userIds) {
>> result += userId + "\t";
>> }
>>
>> result += this.userIds.size();
>> return result;
>> }
>> }
>>
>> As Outputformat I used the default TextOutputFormat.
>>
>> A potential problem could be, that a reduce is going to write files >600MB
>> and our mapred.child.java.opts is set to ~380MB.
>> I digged deeper into the TextOutputFormat and saw, that the
>> HdfsDataOutputStream is not implementing .flush().
>> And .flush is also not used in TextOutputFormat. This means, that the
>> whole file is kept in RAM and then persisted at the end of processing, or?
>> And of course, this leads into the exception.
>>
>> With PIG I am able to query the same Data. Even with one reducer only.
>> But I have a bet to make it faster with plain MapReduce :)
>>
>> Could you help me how to debug this and maybe point me into the right
>> direction?
>>
>> Best Regards,
>> Christian.
>
>
--
Harsh J
Re: Find current version & cluster info of hadoop
Something like "hadoop version" and "hdfs dfsadmin -report" is what you're looking for? On Fri, Mar 8, 2013 at 11:11 AM, Sai Sai wrote: > Just wondering if there r any commands in Hadoop which would give us the > current version that we > r using and any command which will give us the info of cluster setup of H we > r working on. > Thanks > Sai -- Harsh J
Re: Job driver and 3rd party jars
To be precise, did you use -libjar or -libjars? The latter is the right option. On Fri, Mar 8, 2013 at 12:18 AM, Barak Yaish wrote: > Hi, > > I'm able to run M/R jobs where the mapper and reducer required to use 3rd > party jars. I'm registering those jars in -libjar while invoking the hadoop > jar command. I'm facing a strange problem, though, when the job driver > itself ( extends Configured implements Tool ) required to run such code ( > for example notify some remote service upon start and end). Is there a way > to configure classpath when submitting jobs using hadoop jar? Seems like > -libjar doesn't work for this case... > > Exception in thread "main" java.lang.NoClassDefFoundError: > com/me/context/DefaultContext > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632) > at java.lang.ClassLoader.defineClass(ClassLoader.java:616) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:283) > at java.net.URLClassLoader.access$000(URLClassLoader.java:58) > at java.net.URLClassLoader$1.run(URLClassLoader.java:197) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > at java.lang.ClassLoader.loadClass(ClassLoader.java:307) > at java.lang.ClassLoader.loadClass(ClassLoader.java:248) > at > com.peer39.bigdata.mr.pnm.PnmDataCruncher.run(PnmDataCruncher.java:50) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) > at com.me.mr.pnm.PnmMR.main(PnmDataCruncher.java:261) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.util.RunJar.main(RunJar.java:156) > Caused by: java.lang.ClassNotFoundException: com.me.context.DefaultContext > at java.net.URLClassLoader$1.run(URLClassLoader.java:202) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > at java.lang.ClassLoader.loadClass(ClassLoader.java:307) > at java.lang.ClassLoader.loadClass(ClassLoader.java:248) -- Harsh J
Re: mapred.max.tracker.failures
It is a per-job config which controls the automatic job-level blacklist: If, for a single job, a specific tracker has failed 4 (or X) total tasks, then as prevent scheduling anymore of the job's tasks to that tracker (but we don't eliminate more than 25% of the available trackers this way, as for a bad logic job causing failures, that'd make the job simply hang). On Thu, Mar 7, 2013 at 11:21 AM, Mohit Anchlia wrote: > I am wondering what the correct behaviour is of this parameter? If it's set > to 4 does it mean job should fail if a job has more than 4 failures? -- Harsh J
Re: Best practices for adding services to Hadoop cluster?
The only thing wrong would be what is said for the DB-talking jobs as well: Distributed mappers talking to a single point of service can bring it down. On Thu, Mar 7, 2013 at 10:59 AM, Mark Kerzner wrote: > Okay, > > then there is nothing wrong with the mapper directly talking to the server, > and failing the map task if the service does not work out. > > Thank you, > Mark > > > On Wed, Mar 6, 2013 at 11:21 PM, Harsh J wrote: >> >> Can the mapper not directly talk to whatever application server the >> Windows server runs? Is the work needed to be done in the map step >> (i.e. per record)? If not, you can perhaps also consider the SSH >> action of Oozie (although I've never tried it with a Windows machine) >> under a workflow. >> >> On Thu, Mar 7, 2013 at 8:16 AM, Mark Kerzner >> wrote: >> > Hi, >> > >> > my Hadop cluster needs help: some tasks have to be done by a Windows >> > server >> > with specialized closed-source software. How do I add them to the mix? >> > For >> > example, I can run Tomcat, and the mapper would be calling a servlet >> > there. >> > Is there anything better, which would be closer to the fault-tolerant >> > architecture of Hadoop itself? >> > >> > Thank you, >> > Mark >> >> >> >> -- >> Harsh J > > -- Harsh J
Re: Best practices for adding services to Hadoop cluster?
Can the mapper not directly talk to whatever application server the Windows server runs? Is the work needed to be done in the map step (i.e. per record)? If not, you can perhaps also consider the SSH action of Oozie (although I've never tried it with a Windows machine) under a workflow. On Thu, Mar 7, 2013 at 8:16 AM, Mark Kerzner wrote: > Hi, > > my Hadop cluster needs help: some tasks have to be done by a Windows server > with specialized closed-source software. How do I add them to the mix? For > example, I can run Tomcat, and the mapper would be calling a servlet there. > Is there anything better, which would be closer to the fault-tolerant > architecture of Hadoop itself? > > Thank you, > Mark -- Harsh J
Re: HDFS network traffic
Yes, the simple copy is a client operation. Client reads bytes from source and writes to the destination, thereby being in control of failures, etc.. However, if you want your cluster to do the copy (and if the copy is a big set), consider using the DistCp (distributed-copy) MR job to do it. On Thu, Mar 7, 2013 at 9:51 AM, Bill Q wrote: > Hi All, > I am working on converting a sequence file to mapfile and just discovered > something I wasn't aware of. > > For example, suppose I am working on a 2-node cluster, one > master/namenode/datanode, one slave/datanode. If I do hadoop dfs -cp > /data/file1 /data/file2 (a 1G file) from the master, and monitor the NIC of > both nodes, I saw that the master node send the entire file of 1G traffic to > the slave. This surprised me. Does this mean all the traffic has to go > through the client node that runs the command (in this case, the master) > when I do hadoop dfs -cp? > > Many thanks. > > > Bill -- Harsh J
Re: dfs.datanode.du.reserved
Hey John, Ideas, comments and patches are welcome on https://issues.apache.org/jira/browse/HDFS-1564 for achieving this! On Wed, Mar 6, 2013 at 9:56 PM, John Meza wrote: > Thanks for the reply. This sounds like it has potential, but also seems to > be a rather duct-tape type of work around. It would be nice if there was a > mod to dfs.datanode.du.reserved that worked within Hadoop, so that would > imply that hadoop was a little more certain to adhere to it. > > I understand that dfs.datanode.du.reserved defines reservd storage on each > volume. I would like to give each volume a different reserved value. > > An example: > > dfs.datanode.du.reserved >///hstore1/dfs/dn:161061273600,///hstore2/dfs/dn:53687091200 > > Or something similiar. > thanks > John > -- > Date: Wed, 6 Mar 2013 10:25:17 +0100 > Subject: Re: dfs.datanode.du.reserved > From: decho...@gmail.com > To: user@hadoop.apache.org > > > Not that I know. If so you should be able to identify each volume and as > of now this isn't the case. > BUT it can be done without Hadoop knowing about it, at the OS level, using > different partitions/mounts for datanode and jobtracker stuff. That should > solve your problem. > > Regards > > Bertrand > > On Mon, Mar 4, 2013 at 10:26 PM, John Meza wrote: > > I'm probably not being clear. > this seems to describe it: dfs.datanode.du.reserved configured > per-volume. > https://issues.apache.org/jira/browse/HDFS-1564 > > thanks > John > -- > From: outlaw...@gmail.com > Date: Mon, 4 Mar 2013 15:37:36 -0500 > Subject: Re: dfs.datanode.du.reserved > To: user@hadoop.apache.org > > > Possible to reserve 0 from various testing I have done yet that could > cause the obvious side effect of achieving zero disk space:) Have only > tested in development environment, however. Yet there are various tuning > white papers and other benchmarks where the very same has been tested. > > Thanks. > > On Mon, Mar 4, 2013 at 2:00 PM, John Meza wrote: > > the parameter: dfs.datanode.du.reserved is used to reserve disk space PER > datanode. Is it possible to reserve a different amount of disk space per > DISK? > > thanks > John > > > > > -- > Ellis R. Miller > 937.829.2380 > > <http://my.wisestamp.com/link?u=2hxhdfd4p76bkhcm&site=www.wisestamp.com/email-install> > > Mundo Nulla Fides > > > > <http://my.wisestamp.com/link?u=gfbmwhzrwxzcrjqx&site=www.wisestamp.com/email-install> > > > > > > -- Harsh J
Re: Hadoop Jobtracker API
The Java API of JobClient class lets you query all jobs and provides some task-level info as a public API. In YARN (2.x onwards), the MRv2's AM publishes a REST API that lets you query it (the RM lets you get a list of such AMs as well, as a first step). This sounds more like what you need. A REST API similar to what was added in YARN for MRv2 was also added to the 1.x's JobTracker recently via https://issues.apache.org/jira/browse/MAPREDUCE-4837, appearing in 1.2.0 release onwards. On Thu, Mar 7, 2013 at 12:13 AM, Kyle B wrote: > Hello, > > I was wondering if the Hadoop job tracker had an API, such as a web service > or xml feed? I'm trying to track Hadoop jobs as they progress. Right now, > I'm parsing the HTML of the "Running Jobs" section at > http://hadoop:50030/jobtracker.jsp, but this is definitely not desired if > there is a better way. Is there a simple web service for the data on > jobtracker.php & jobdetails.jsp? > > Has anyone run into this problem of trying to track Hadoop progress from a > remote machine programatically? > > Any help is appreciated, > > -Kyle -- Harsh J
Re: location of log files for mapreduce job run through eclipse
If you've not changed any configs, look under /tmp/hadoop-${user.name}/ perhaps.
On Thu, Mar 7, 2013 at 3:19 AM, Sayan Kole wrote:
> Hi,
> I cannot find the log files for the wordcount job: job_local_0001 when I
> run it through eclipse. I am getting the standard output on screen showing
> me the progress of the maps/reduces, but when I check my log directory
> inside the hadoop folder, I do not see any log files for this job nor does
> the jobtracker log file have any information for this job.
>
> Sayan Kole
--
Harsh J
Re: preferential distribution of tasks to tasktracker by jobtracker based on specific criteria (cpu frequency)
MRv1: Both the CapacityScheduler and FairScheduler support some form of resource aware scheduling (Capacity supports memory provisions while Fair provides an interface you can plug into to influence its decisions). You can also implement a custom scheduler - as TaskTrackers do send their resource details (which includes CPU). The class you need to derive for this is https://github.com/apache/hadoop-common/blob/branch-1/src/mapred/org/apache/hadoop/mapred/TaskScheduler.java, and the assignTasks(…)'s TaskTracker object (a repr. of the TT for the JT) lets you get the TaskTrackerStatus object which has the relevant information you seek: https://github.com/apache/hadoop-common/blob/branch-1/src/mapred/org/apache/hadoop/mapred/TaskTrackerStatus.java Note though that in MRv2, which runs on YARN and is present in the current 2.x releases, the JobTracker and the TaskTrackers are both gone and the scheduler is now a part of the Resource Manager as a generic component under YARN. Just something to keep in mind if you're continuing to base your work (research or otherwise) on an older, maintenance release. Your MRv1 implementation may become irrelevant or cease to work after the upgrade in future. YARN: The JIRA https://issues.apache.org/jira/browse/YARN-2 added native CPU core based scheduling, which should also help your need right out of the box perhaps. On Thu, Mar 7, 2013 at 2:17 AM, Sayan Kole wrote: > Hi, >I want the jobtracker to prioritize the assignment of tasks to certain > tasktrackers. > eg: If a tasktracker meets certain criteria better than other ones, I want > to assign task to that tasktracker first (ideally I want the jobtracker to > sort tasktrackers based on certain criteria (eg cpu frequency) and then > assign tasks to tasktracker based on that sorted assignment. > > Could you point me to the files(eg Jobtracker.java) and routines to > modify or a guideline to a proper way of implementing it. > > Thanks, > Sayan Kole. > -- Harsh J
Re: FileStatus.getPath
The FileStatus is a container of metadata for a specific path, and hence carries the Path object the rest of the details are for. What do you exactly mean by "has no defined contract"? If you want a qualified path (for a specific FS), then doing "path.makeQualified(…)" is always the right way. On Tue, Mar 5, 2013 at 11:31 PM, Jay Vyas wrote: > Hi it appears that: > > http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileStatus.html > > getPath() > > has no defined contract. > > Why does FileStatus have a "getPAth" method? Would it be the equivalent > effect to > simply make a path qualified using the FileSystem object? > > i.e. path.makeQualified(FileSystem.get()) ? > > -- > Jay Vyas > http://jayunit100.blogspot.com -- Harsh J
Re: basic question about rack awareness and computation migration
Your concern is correct: If your input is a list of files, rather than the files themselves, then the tasks would not be data-local - since the task input would just be the list of files, and the files' data may reside on any node/rack of the cluster. However, your job will still run as the HDFS reads do remote reads transparently without developer intervention and all will still work as you've written it to. If a block is found local to the DN, it is read locally as well - all of this is automatic. Are your input lists big (for each compressed output)? And is the list arbitrary or a defined list per goal? On Tue, Mar 5, 2013 at 5:19 PM, Julian Bui wrote: > Hi hadoop users, > > I'm trying to find out if computation migration is something the developer > needs to worry about or if it's supposed to be hidden. > > I would like to use hadoop to take in a list of image paths in the hdfs and > then have each task compress these large, raw images into something much > smaller - say jpeg files. > > Input: list of paths > Output: compressed jpeg > > Since I don't really need a reduce task (I'm more using hadoop for its > reliability and orchestration aspects), my mapper ought to just take the > list of image paths and then work on them. As I understand it, each image > will likely be on multiple data nodes. > > My question is how will each mapper task "migrate the computation" to the > data nodes? I recall reading that the namenode is supposed to deal with > this. Is it hidden from the developer? Or as the developer, do I need to > discover where the data lies and then migrate the task to that node? Since > my input is just a list of paths, it seems like the namenode couldn't really > do this for me. > > Another question: Where can I find out more about this? I've looked up > "rack awareness" and "computation migration" but haven't really found much > code relating to either one - leading me to believe I'm not supposed to have > to write code to deal with this. > > Anyway, could someone please help me out or set me straight on this? > > Thanks, > -Julian -- Harsh J
Re: mapper combiner and partitioner for particular dataset
The MultipleInputs class only supports mapper configuration per dataset. It does not let you specify a partitioner and combiner as well. You will need a custom written "high level" partitioner and combiner that can create multiple instances of sub-partitioners/combiners and use the most likely one based on their input's characteristics (such as instance type, some tag, config., etc.). On Sun, Mar 3, 2013 at 4:07 PM, Vikas Jadhav wrote: > > > > > Hello > > 1) I have multiple types of datasets as input to my hadoop job > > i want write my own inputformat (Exa. MyTableInputformat) > and how to specify mapper partitioner combiner per dataset manner > I know MultiFileInputFormat class but if i want to asscoite combiner and > partitioner class > it wont help. it only sets mapper class for per dataset manner. > > 2) Also i am looking MapTask.java file from source code > > just want to know where does mapper partitioner and combiner classes are > set for particular filesplit > while executing job > > Thank You > > -- > * > * > * > > Thanx and Regards* > * Vikas Jadhav* > > > > -- > * > * > * > > Thanx and Regards* > * Vikas Jadhav* > -- Harsh J
Re: Urgent Requirement: How to copy File from One cluster to another cluster using java client(throught java Program)
Samir, It was pointed out by another member in your earlier post but here it is again. The error returned is sorta clear enough: org.apache.hadoop.security.AccessControlException: Permission denied: user=hadoop, access=WRITE, inode="/user/dasmohap/samir_tmp":dasmohap:dasmohap:drwxr-xr-x 1. Your program that is trying to write a file is running as a user called "hadoop". 2. The target location/cluster you are writing to, under a path /user/dasmohap/samir_tmp, is owned by the user "dasmohap" there. 3. The target path has the permission 755, preventing writes by any other user except its owner. 4. Your program, as mentioned in (1), tries to write as user "hadoop", and fails due to lack of allowing permission on the target. If you need the program to write there, you have to give it permissions to write first. You need to make /user/dasmohap/samir_tmp more globally accessible for writes, with perhaps a 775 (if you can and know how to setup groups), or 777 (if you want a quick fix and aren't concerned in any way about permissions and security). Hope this helps. On Sat, Mar 2, 2013 at 1:13 AM, samir das mohapatra wrote: > Hi All, > Any one had gone through scenario to copy file from one cluster to > another cluster using java application program (Using Hadoop FileSystem > API) . > > I have done some thing using java application but within the same > cluster it is working file while I am copying the file from one cluster to > another cluster I am getting the error. > > File not found org.apache.hadoop.security. > AccessControlException: Permission denied: user=hadoop, access=WRITE, > inode="/user/dasmohap/samir_tmp":dasmohap:dasmohap:drwxr-xr-x > at > org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:205) > at > org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:186) > at > org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:135) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:4547) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:4518) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1755) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:1690) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1669) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:409) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:205) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:44068) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687) > > > Regards, > samir. > > -- > > > > -- Harsh J
Re: NameNode low on available disk space
Actually, I did: http://search-hadoop.com/m/qvn7EhQQHm1. Perhaps you aren't subscribed to the lists and wouldn't have received anything sent back. On Fri, Mar 1, 2013 at 1:21 PM, Mohit Vadhera wrote: > thanks Harsh but you didn't answer on this before, I will try to move old > directory "name" to new location and restart services . Hope it will not > loose any data . > > old Location > > $ sudo ls -l /var/lib/hadoop-hdfs/cache/hdfs/dfs/ > total 12 > drwx--. 3 hdfs hdfs 4096 Dec 19 02:37 data > drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 11:28 name > drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 11:28 namesecondary > > > New location > > $ sudo ls -l /mnt/san1/hdfs/hdfs/dfs/ > total 8 > drwx--. 3 hdfs hdfs 4096 Feb 28 11:28 data > drwxr-xr-x 2 hdfs hdfs 4096 Feb 28 11:28 namesecondary > > > Thanks, > > > On Fri, Mar 1, 2013 at 12:14 PM, Harsh J wrote: > >> I believe I already responded to another one of your multiple threads >> on this problem, that you have to move contents of the old dir to the >> new dir and not format. Thats how one changes the dfs.name.dir >> properly. >> >> On Fri, Mar 1, 2013 at 12:11 PM, Mohit Vadhera >> wrote: >> > Hi, >> > >> > May i know the answer plz ? >> > >> > Thanks, >> > >> > >> > On Thu, Feb 28, 2013 at 4:28 PM, Mohit Vadhera >> > wrote: >> >> >> >> Even i created the file /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock >> >> and set permission . when i restart hadoop services. It removes and I >> find >> >> below logs. >> >> >> >> Do I need to format the NN? >> >> Below is the command to format the NN ? >> >> Any kind of loss while formatting ? >> >> Is there any way to avoid formatting and change the cache path ? >> >> >> >> 2013-02-28 05:57:50,902 INFO >> org.apache.hadoop.hdfs.server.common.Storage: >> >> Lock on /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by >> nodename >> >> 81...@opera-mast1.ny.os.local >> >> 2013-02-28 05:57:50,904 INFO >> >> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode >> metrics >> >> system... >> >> 2013-02-28 05:57:50,904 INFO >> >> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics >> system >> >> stopped. >> >> 2013-02-28 05:57:50,904 INFO >> >> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics >> system >> >> shutdown complete. >> >> 2013-02-28 05:57:50,905 FATAL >> >> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode >> join >> >> java.io.IOException: NameNode is not formatted. >> >> >> >> Command to format the NN. >> >> >> >> sudo -u hdfs hdfs namenode -format >> >> >> >> Thanks, >> >> >> >> >> >> On Thu, Feb 28, 2013 at 3:47 PM, Mohit Vadhera >> >> wrote: >> >>> >> >>> After creating the directory and setting permission I tried to restart >> >>> the services and i get error >> "/mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock >> >>> acquired by nodename 7...@opera-mast1.ny.os.local" and services are >> not >> >>> being started. >> >>> >> >>> Need to check few logs from below logs. >> >>> === >> >>> 2013-02-28 05:06:24,905 WARN >> org.apache.hadoop.hdfs.server.common.Util: >> >>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI >> in >> >>> configuration files. Please update hdfs configuration. >> >>> 2013-02-28 05:06:24,905 WARN >> org.apache.hadoop.hdfs.server.common.Util: >> >>> Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI >> in >> >>> configuration files. Please update hdfs configuration. >> >>> 2013-02-28 05:06:24,906 WARN >> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image >> storage >> >>> directory (dfs.namenode.name.dir) configured. Beware of dataloss due >> to lack >> >>> of redundant sto >> >>> rage directories! >> >>> 2013-02-28 05:06:24,906 WARN >> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one >> namespace >> >>> edits storage directory (dfs.namenode.edits.dir
Re: NameNode low on available disk space
;>> dundant storage directories! >>> 2013-02-28 05:06:25,618 INFO org.apache.hadoop.util.HostsFileReader: >>> Refreshing hosts (include/exclude) list >>> 2013-02-28 05:06:25,623 INFO >>> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: >>> dfs.block.invalidate.limit=1000 >>> 2013-02-28 05:06:26,015 INFO >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: >>> dfs.block.access.token.enable=false >>> 2013-02-28 05:06:26,015 INFO >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: >>> defaultReplication = 1 >>> 2013-02-28 05:06:26,015 INFO >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplication >>> = 512 >>> 2013-02-28 05:06:26,015 INFO >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: minReplication >>> = 1 >>> 2013-02-28 05:06:26,015 INFO >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: >>> maxReplicationStreams = 2 >>> 2013-02-28 05:06:26,016 INFO >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: >>> shouldCheckForEnoughRacks = false >>> 2013-02-28 05:06:26,016 INFO >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: >>> replicationRecheckInterval = 3000 >>> 2013-02-28 05:06:26,016 INFO >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: >>> encryptDataTransfer= false >>> 2013-02-28 05:06:26,022 INFO >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner = >>> hdfs (auth:SIMPLE) >>> 2013-02-28 05:06:26,022 INFO >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup = >>> hadmin >>> 2013-02-28 05:06:26,022 INFO >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled = >>> true >>> 2013-02-28 05:06:26,023 INFO >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false >>> 2013-02-28 05:06:26,026 INFO >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true >>> 2013-02-28 05:06:26,359 INFO >>> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occuring >>> more than 10 times >>> 2013-02-28 05:06:26,361 INFO >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: >>> dfs.namenode.safemode.threshold-pct = 0.999128746033 >>> 2013-02-28 05:06:26,361 INFO >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: >>> dfs.namenode.safemode.min.datanodes = 0 >>> 2013-02-28 05:06:26,361 INFO >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: >>> dfs.namenode.safemode.extension = 0 >>> 2013-02-28 05:06:26,378 INFO >>> org.apache.hadoop.hdfs.server.common.Storage: Lock on >>> /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by nodename >>> 7...@opera-mast1.ny.os.local >>> 2013-02-28 05:06:26,381 INFO >>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode metrics >>> system... >>> 2013-02-28 05:06:26,381 INFO >>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system >>> stopped. >>> 2013-02-28 05:06:26,381 INFO >>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system >>> shutdown complete. >>> 2013-02-28 05:06:26,382 FATAL >>> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join >>> java.io.IOException: NameNode is not formatted. >>> at >>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:211) >>> at >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534) >>> at >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424) >>> at >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386) >>> at >>> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398) >>> at >>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432) >>> at >>> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:608) >>> at >>> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:589) >>> at >>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140) >>> at >>> org.apache.hadoop.hdfs.server.namen
Re: TotalOrdering in Python Streaming?
Currently, the partitioner supplied for streaming jobs needs to be a java class implementing its features. You can still reuse the TotalOrderPartitioner by passing it as -partitioner perhaps? On Fri, Mar 1, 2013 at 3:32 AM, Periya.Data wrote: > Hi, >I am trying to see if I can implement total ordering with Python > Streaming. I do not want to use 1 reducer to accomplish my ordering. I came > to know that there is a nice TotalOrderPartitioner in Java that is used in > regular sort and terasort. Is there something like that in Python which I > can use? > > Thanks, > PD > > -- > > > -- Harsh J
Re: How to make a MapReduce job with no input?
The default # of map tasks is set to 2 (via mapred.map.tasks from mapred-default.xml) - which explains your 2-map run for even one line of text. For running with no inputs, take a look at Sleep Job's EmptySplits technique on trunk: http://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/SleepJob.java?view=markup (~line 70) On Fri, Mar 1, 2013 at 2:46 AM, Mike Spreitzer wrote: > I am using the mapred API of Hadoop 1.0. I want to make a job that does not > really depend on any input (the job conf supplies all the info needed in > Mapper). What is a good way to do this? > > What I have done so far is write a job in which MyMapper.configure(..) reads > all the real input from the JobConf, and MyMapper.map(..) ignores the given > key and value, writing the output implied by the JobConf. I set the > InputFormat to TextInputFormat and the input paths to be a list of one > filename; the named file contains one line of text (the word "one"), > terminated by a newline. When I run this job (on Linux, hadoop-1.0.0), I > find it has two map tasks --- one reads the first two bytes of my non-input > file, and other reads the last two bytes of my non-input file! How can I > make a job with just one map task? > > Thanks, > Mike -- Harsh J
Re: hdfs.h C API
Hi!,
Assuming you have a "hadoop" command available (i.e. the
$HADOOP_HOME/bin/hadoop script), try to do:
export CLASSPATH=`hadoop classpath`
And then try to run your ./a.out.
Does this help?
Also, user questions may go to user@hadoop.apache.org. The
hdfs-...@hadoop.apache.org is for Apache HDFS project
developer/development discussions alone. I've moved your thread to the
proper place.
On Thu, Feb 28, 2013 at 4:59 PM, Philip Herron wrote:
> Hey all
>
> I am trying to use the c api to access hdfs:
>
> #include
> #include
>
> #include
> #include
>
> #include
>
> int main (int argc, char ** argv)
> {
> hdfsFS fs = hdfsConnect ("default", 0);
> assert (fs);
>
> const char * wp = "test";
> hdfsFile wf = hdfsOpenFile (fs, wp, O_WRONLY | O_CREAT,
> 0, 0, 0);
> if (!wf)
> {
> fprintf (stderr, "Failed to open %s for writing!\n", wp);
> exit (-1);
> }
>
> const char * buffer = "Hello, World!";
> tSize num_written_bytes = hdfsWrite (fs, wf, (void *) buffer,
>strlen (buffer) + 1);
> assert (num_written_bytes);
> if (hdfsFlush (fs, wf))
> {
> fprintf (stderr, "Failed to 'flush' %s\n", wp);
> exit (-1);
> }
> hdfsCloseFile(fs, wf);
>
> return 0;
> }
>
> --
>
> gcc t.c -lhdfs -lpthread -L/usr/java/default/lib/amd64/server -ljvm -Wall
>
> But i am getting:
>
> Environment variable CLASSPATH not set!
> getJNIEnv: getGlobalJNIEnv failed
> a.out: t.c:12: main: Assertion `fs' failed.
> Aborted
>
> Not sure what i need to do now to get this example working.
>
> --Phil
--
Harsh J
Re: NameNode low on available disk space
ver.namenode.FSImage.recoverTransitionRead(FSImage.java:201)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:608)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:589)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
> 2013-02-28 02:08:48,908 INFO org.apache.hadoop.util.ExitUtil: Exiting with
> status 1
> 2013-02-28 02:08:48,913 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
> /
> SHUTDOWN_MSG: Shutting down NameNode at OPERA-MAST1.ny.os.local/192.168.1.3
>
>
> On Thu, Feb 28, 2013 at 1:27 PM, Mohit Vadhera
> wrote:
>>
>> Hi Guys,
>>
>> I have space on other partition. Can I change the path for cache files on
>> other partition ? I have below properties . Can it resolve the issue ? If i
>> change the path to other directories and restart services I get the below
>> error while starting the service namenode. I didn't find anything in logs so
>> far. Can you please suggest something ?
>>
>>
>> hadoop.tmp.dir
>> /var/lib/hadoop-hdfs/cache/${user.name}
>>
>>
>> dfs.namenode.name.dir
>> /var/lib/hadoop-hdfs/cache/${user.name}/dfs/name
>>
>>
>> dfs.namenode.checkpoint.dir
>>
>> /var/lib/hadoop-hdfs/cache/${user.name}/dfs/namesecondary
>>
>>
>>
>>
>> Service namenode is failing
>>
>> # for service in /etc/init.d/hadoop-hdfs-* ; do sudo $service status; done
>> Hadoop datanode is running [ OK ]
>> Hadoop namenode is dead and pid file exists[FAILED]
>> Hadoop secondarynamenode is running[ OK ]
>>
>> Thanks,
>>
>>
>>
>> On Wed, Jan 23, 2013 at 11:15 PM, Mohit Vadhera
>> wrote:
>>>
>>>
>>> On Wed, Jan 23, 2013 at 10:41 PM, Harsh J wrote:
>>>>
>>>> http://NNHOST:50070/conf
>>>
>>>
>>>
>>> Harsh, I changed the value as said & restarted service NN. For verifying
>>> i checked the http link that you gave and i saw the property their but on
>>> http://NNHOST:50070 i noticed warning( WARNING : There are 4 missing
>>> blocks. Please check the logs or run fsck in order to identify the missing
>>> blocks.) when i clicked on this link i can see file names . Do I need to
>>> reboot the machine to run fsck on root fs/ or is there hadoop command fsck
>>> that i can run on the running hadoop ?
>>>
>>> Thanks,
>>>
>>
>
--
Harsh J
Re: namenode is failing
If you move the NN dir, you also have to move its older location's
contents there before starting the NN.
On Thu, Feb 28, 2013 at 1:28 PM, Mohit Vadhera
wrote:
> Hi Guys,
>
> Namenode switches into safemode when it has low disk space on the root fs /
> i have to manually run a command to leave it
> I have space on other partition. Can I change the path for cache files on
> other partition ? I have below properties . Can it resolve the issue ? If i
> change the path to other directories and restart services I get the below
> error while starting the service namenode. I didn't find anything in logs so
> far. Can you please suggest something ?
>
>
> hadoop.tmp.dir
> /var/lib/hadoop-hdfs/cache/${user.name}
>
>
> dfs.namenode.name.dir
> /var/lib/hadoop-hdfs/cache/${user.name}/dfs/name
>
>
> dfs.namenode.checkpoint.dir
>
> /var/lib/hadoop-hdfs/cache/${user.name}/dfs/namesecondary
>
>
>
>
> Service namenode is failing
>
> # for service in /etc/init.d/hadoop-hdfs-* ; do sudo $service status; done
> Hadoop datanode is running [ OK ]
> Hadoop namenode is dead and pid file exists[FAILED]
> Hadoop secondarynamenode is running[ OK ]
>
> Thanks,
>
--
Harsh J
Re: where reduce is copying?
The latter (from other machines, inbound to where the reduce is running, onto the reduce's local disk, via mapred.local.dir). The reduce will, obviously, copy outputs from all maps that may have produced data for its assigned partition ID. On Thu, Feb 28, 2013 at 12:27 PM, Patai Sangbutsarakum wrote: > Good evening Hadoopers! > > at the jobtracker page, click on a job, and click at running reduce > task, I am going to see > > task_201302271736_0638_r_00 reduce > copy (136 of 261 at 0.44 MB/s) > > I am really curious where is the data is being copy. > if i clicked at the task, it will show a host that is running the task > attempt. > > question is "reduce > copy" is referring data copy outbound from host > that is running task attempt, or > referring to data is being copy from other machines inbound to this > host (that's running task attempt) > > and in both cases how do i know what machines that host is copy data from/to? > > Regards, > Patai -- Harsh J
Re: How to find Replication factor for one perticular folder in HDFS
Its "hdfs getconf", not "hdfs -getconf". The first sub-command is not an option arg, generally, when using the hadoop/hdfs/yarn/mapred scripts. On Wed, Feb 27, 2013 at 3:40 PM, Dhanasekaran Anbalagan wrote: > HI YouPeng Yang , > > Hi already configured dfs.replication factor=2 > >>> 1. To get the key from configuration : > /bin/hdfs -getconf -conKey dfs.replication > > hdfs@dvcliftonhera227:~$ hdfs -getconf -conKey dfs.replication > Unrecognized option: -getconf > Could not create the Java virtual machine. > > Please guide me. > > -Dhanasekaran > > > Did I learn something today? If not, I wasted it. > > > On Mon, Feb 25, 2013 at 7:31 PM, YouPeng Yang > wrote: >> >> Hi Dhanasekaran Anbalagan >> >> 1. To get the key from configuration : >> /bin/hdfs -getconf -conKey dfs.replication >> >> >>2.Maybe you can add the attribute >>true to your dfs.replication : >> >> >> dfs.replication >>2 >>true >> >> >> >> regards. >> >> >> >> 2013/2/26 Nitin Pawar >>> >>> see if the link below helps you >>> >>> >>> http://www.michael-noll.com/blog/2011/10/20/understanding-hdfs-quotas-and-hadoop-fs-and-fsck-tools/ >>> >>> >>> On Mon, Feb 25, 2013 at 10:36 PM, Dhanasekaran Anbalagan >>> wrote: >>>> >>>> Hi Guys, >>>> >>>> How to query particular folder witch replication factor configured. In >>>> my cluster some folder in HDFS configured 2 and some of them configured as >>>> three. How to query. >>>> >>>> please guide me >>>> >>>> -Dhanasekaran >>>> >>>> Did I learn something today? If not, I wasted it. >>> >>> >>> >>> >>> -- >>> Nitin Pawar >> >> > -- Harsh J
Re: WordPairCount Mapreduce question.
d1.compareTo(o.word1);
>> if (diff == 0) {
>> diff = word2.compareTo(o.word2);
>> }
>> return diff;
>> }
>>
>> @Override
>> public int hashCode() {
>> return word1.hashCode() + 31 * word2.hashCode();
>> }
>>
>>
>> public String getWord1() {
>> return word1;
>> }
>>
>> public void setWord1(String word1) {
>> this.word1 = word1;
>> }
>>
>> public String getWord2() {
>> return word2;
>> }
>>
>> public void setWord2(String word2) {
>> this.word2 = word2;
>> }
>>
>> @Override
>> public void readFields(DataInput in) throws IOException {
>> word1 = in.readUTF();
>> word2 = in.readUTF();
>> }
>>
>> @Override
>> public void write(DataOutput out) throws IOException {
>> out.writeUTF(word1);
>> out.writeUTF(word2);
>> }
>>
>>
>> @Override
>> public String toString() {
>> return "[word1=" + word1 + ", word2=" + word2 + "]";
>> }
>>
>> }
>>
>> **
>>
>> Any help will be really appreciated.
>> Thanks
>> Sai
>>
>>
>>
>>
>
--
Harsh J
Re: MapReduce job over HBase-0.94.3 fails
Hi Bhushan, Please email the user@hadoop.apache.org lists for any Apache Hadoop user-related questions (instead of sending it to me directly this way). You can subscribe by following instructions at http://hadoop.apache.org/mailing_lists.html. I've added the list in my reply here. On Mon, Feb 25, 2013 at 1:09 PM, wrote: > I am using Hadoop-1.0.3 and HBase-0.94.3. I am able to run MapReduce job over > Hadoop-1.0.3. But I got following error while running MapReduce job over > HBase-0.94.3. > Exception in thread "main" java.lang.NullPointerException > at org.apache.hadoop.net.DNS.reverseDns(DNS.java:72) > at > org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.reverseDNS(TableInputFormatBase.java:218) > at > org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.getSplits(TableInputFormatBase.java:183) > at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:962) > at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:979) > at org.apache.hadoop.mapred.JobClient.access$600(JobClient.java:174) > at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:897) > at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:850) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:416) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121) > at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:850) > at org.apache.hadoop.mapreduce.Job.submit(Job.java:500) > at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:530) > at org.apache.hadoop.hbase.mapreduce.CopyTable.main(CopyTable.java:237) > > > Is their any versioning issue for HBase? > > How to resolve it? > > Thanks in advance. > -- Harsh J
Re: map reduce and sync
> >>>>> Lucas >>>>> >>>>> >>>>> >>>>> On Sat, Feb 23, 2013 at 4:37 AM, Hemanth Yamijala >>>>> wrote: >>>>>> >>>>>> Could you please clarify, are you opening the file in your mapper code >>>>>> and reading from there ? >>>>>> >>>>>> Thanks >>>>>> Hemanth >>>>>> >>>>>> On Friday, February 22, 2013, Lucas Bernardi wrote: >>>>>>> >>>>>>> Hello there, I'm trying to use hadoop map reduce to process an open >>>>>>> file. The writing process, writes a line to the file and syncs the file >>>>>>> to >>>>>>> readers. >>>>>>> (org.apache.hadoop.fs.FSDataOutputStream.sync()). >>>>>>> >>>>>>> If I try to read the file from another process, it works fine, at >>>>>>> least using >>>>>>> org.apache.hadoop.fs.FSDataInputStream. >>>>>>> >>>>>>> hadoop -fs -tail also works just fine >>>>>>> >>>>>>> But it looks like map reduce doesn't read any data. I tried using the >>>>>>> word count example, same thing, it is like if the file were empty for >>>>>>> the >>>>>>> map reduce framework. >>>>>>> >>>>>>> I'm using hadoop 1.0.3. and pig 0.10.0 >>>>>>> >>>>>>> I need some help around this. >>>>>>> >>>>>>> Thanks! >>>>>>> >>>>>>> Lucas >>>>> >>>>> >>>> >>> >> > -- Harsh J
Re: childerror
Given your other thread post, I'd say you may have an inconsistency in the JDK deployed on the cluster. We generally recommend using the Oracle JDK6 (for 1.0.4). More details on which version to pick can be found at http://wiki.apache.org/hadoop/HadoopJavaVersions. In any case, the JDK installation across the cluster has to be consistent (you can check via a package tool such as yum, or via "java -version" commands on each node). On Mon, Feb 25, 2013 at 7:16 AM, Jean-Marc Spaggiari wrote: > Hi Fatih, > > Have you looked in the logs files? Anything there? > > JM > > > 2013/2/24 Fatih Haltas >> >> I am always getting the Child Error, I googled but I could not solve the >> problem, did anyone encounter with same problem before? >> >> >> [hadoop@ADUAE042-LAP-V conf]$ hadoop jar >> /home/hadoop/project/hadoop-1.0.4/hadoop-examples-1.0.4.jar >> aggregatewordcount /home/hadoop/project/hadoop-data/NetFlow test1614.out >> Warning: $HADOOP_HOME is deprecated. >> >> 13/02/24 15:53:15 INFO mapred.FileInputFormat: Total input paths to >> process : 1 >> 13/02/24 15:53:15 INFO mapred.JobClient: Running job: >> job_201301141457_0048 >> 13/02/24 15:53:16 INFO mapred.JobClient: map 0% reduce 0% >> 13/02/24 15:53:23 INFO mapred.JobClient: Task Id : >> attempt_201301141457_0048_m_02_0, Status : FAILED >> java.lang.Throwable: Child Error >> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271) >> Caused by: java.io.IOException: Task process exit with nonzero status of >> 1. >> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258) >> > -- Harsh J
Re: Slow MR time and high network utilization with all local data
Hi Robert, How are you measuring the network usage? Note that unless short circuit reading is on, data reads are done over a local socket as well, and may appear in network traffic observing tools too (but do not mean they are over the network). On Mon, Feb 25, 2013 at 2:35 AM, Robert Dyer wrote: > I have a small 6 node dev cluster. I use a 1GB SequenceFile as input to a > MapReduce job, using a custom split size of 10MB (to increase the number of > maps). Each map call will read random entries out of a shared MapFile (that > is around 50GB). > > I set replication to 6 on both of these files, so all of the data should be > local for each map task. I verified via fsck that no blocks are > under-replicated. > > Despite this, for some reason the MR job maxes out the network and takes an > extremely long time. What could be causing this? > > Note that the total number of map outputs for this job is around 400 and the > reducer just passes the values through, so there shouldn't be much network > utilized by the output. > > As an experiment, I switched from the SeqFile input to an HBase table and > now see almost no network used. I also tried leaving the SeqFile as input > and switched the MapFile to an HBase table and see about 30% network used > (which makes sense, as now that 50GB data isn't always local). > > What is going on here? How can I debug to see what data is being > transferred over the network? -- Harsh J
Re: Errors come out when starting Balancer
Hi, A few points: - The property "fs.default.name" has been deprecated in favor of the newer name "fs.defaultFS". - When using nameservice IDs, you shouldn't rely on the RPC address. That is, your fs.defaultFS should instead simply be "hdfs://ns1", and the config will load the right RPC that is bound to that NS automatically from the hdfs-site.xml. On Mon, Feb 25, 2013 at 10:35 AM, YouPeng Yang wrote: > Hi . > Sorry to bother you againt. > > I need to close this question by myself. > Here is my core-site.xml: > ** > > >fs.default.name >hdfs://Hadoop01:8020 > > > ** > my hdfs-site.xml: > * > > >dfs.nameservices >ns1,ns2 > > >dfs.namenode.rpc-address.ns1 > Hadoop01:8040 > > Other common configuration ... > > ** > > When start the balancer,the script using the fs.default.name of > core-site.xml by defulat. > according to hdfs-site.xml,the port 8020 does no start.So it goes fo fail > when i start the balancer usig the port 8020 of core-site.xml. > > So the ports of fs.default.name and > dfs.namenode.rpc-address. shoud be consistent. > > > Thanks > > Regards > > > > > > 2013/2/24 YouPeng Yang >> >> Hi ALL >> >> I start the Balancer on the HDFS Federation,however it comes an error: >> java.net.ConnectException: Call From Hadoop01/10.167.14.221 to >> Hadoop01:8020 failed on connection exception: java.net.ConnectException: >> Connection refused; For more details see: >> http://wiki.apache.org/hadoop/ConnectionRefused. Exiting ... >> Balancing took 11.163 seconds >> >> How Can i solve this. >> >> By the way,I have done what explained on the >> http://wiki.apache.org/hadoop/ConnectionRefused when i build my basic and >> original HDFS cluster.So i think the above error maybe because of another >> reasons. >> >> thanks >> >> Regards. >> > -- Harsh J
Re: One NameNode keeps Rolling Edit Log on HDFS Federation
Could you send us the log messages (with timestamps) which you think is behind the behavior you see? The new edit log format is different than the one in 1.x, and uses several smaller edit log files. This new format is described in PDFs attached at https://issues.apache.org/jira/browse/HDFS-1073, and the logs may mean that the NN just opened a new file and closed the older transactions file. On Mon, Feb 25, 2013 at 10:16 AM, YouPeng Yang wrote: > Hi Azuryy,Harsh > >It is strange that the NameNode rolling edits log rolls edits logs all > the times,even i do nothing on the NameNode vi client. > > Any futher help will be appreciated. > > > regard. > > > > > > > 2013/2/25 YouPeng Yang >> >> Hi Azuryy >> >>Yes.that is what is. >>Thank you for your reply. I am making effort to make clear about >> federation and HA. >> >> Regards. >> >> >> >> >> 2013/2/25 Azuryy Yu >>> >>> I think you mixed federation with HA. am I right? >>> >>> If another name node hasn't changes, then It doesn't do any edit log >>> rolling. federated NNs don't keep concurrency( I think you want say keep >>> sync-able?) >>> >>> >>> On Sun, Feb 24, 2013 at 11:09 PM, YouPeng Yang >>> wrote: >>>> >>>> Hi All >>>> >>>> I'm testing the HDFS Federation.I have 2 namenodes in my cluster. >>>> I find that it is Rolling Edit Log continuously on one >>>> namenode,however the other one changes nothing. >>>> >>>> My question : >>>> 1. Is it the right situation. >>>> 2. I have thought that the two namenodes shoud keep concurrency. >>>> why they get differences. >>>> 3. is there any advanced docs about HDFS Federation. >>>> >>>> thanks. >>>> >>>> Regards. >>>> >>>> >>>> >>> >>> >> > -- Harsh J
Re: One NameNode keeps Rolling Edit Log on HDFS Federation
Mike, I don't see how SPOF comes into the picture when HA is already present in the releases that also carry Federation and each NN (federated or non) can be assigned further Standby-NN roles. At this point we can stop using the word "SPOF" completely for HDFS. Would do great good for avoiding further FUD around this :) The federated NameNodes are namespace-divided; and typically I think users may use it to divide applications into using different namespaces each, while using the same set of compute and data nodes. So if HA isn't deployed, an application may be lost but not "portion of the cluster", since DNs will remain unaffected and so would the other namespaces. On Sun, Feb 24, 2013 at 11:23 PM, Michel Segel wrote: > I think part of the confusion stems from the fact that federation of name > nodes only splits the very large cluster in to smaller portions of the same > cluster. If you lose a federated name node, you only lose a portion of the > cluster not the whole thing. So now instead of one SPOF, you have two SPOFs. > > The advantage of Federation is that you reduce the amount of memory required > for the NN. > > > Sent from a remote device. Please excuse any typos... > > Mike Segel > > On Feb 24, 2013, at 10:48 AM, Harsh J wrote: > >> Hi, >> >> Federated namenodes are independent of one another (except that they >> both get reports from all the/common DNs in the cluster). It is >> natural to see one roll its edit logs based on its own rate of >> metadata growth, as compared to the other. Their edits, image, etc. >> everything is independent - they also do not know about/talk to each >> other. >> >> Given the above, I'd say (1) is true and (2) is invalid, and (3) can >> be found in PDFs attached to >> https://issues.apache.org/jira/browse/HDFS-1052. >> >> On Sun, Feb 24, 2013 at 8:39 PM, YouPeng Yang >> wrote: >>> Hi All >>> >>> I'm testing the HDFS Federation.I have 2 namenodes in my cluster. >>> I find that it is Rolling Edit Log continuously on one namenode,however >>> the other one changes nothing. >>> >>> My question : >>> 1. Is it the right situation. >>> 2. I have thought that the two namenodes shoud keep concurrency. >>> why they get differences. >>> 3. is there any advanced docs about HDFS Federation. >>> >>> thanks. >>> >>> Regards. >> >> >> >> -- >> Harsh J >> -- Harsh J
Re: One NameNode keeps Rolling Edit Log on HDFS Federation
Hi, Federated namenodes are independent of one another (except that they both get reports from all the/common DNs in the cluster). It is natural to see one roll its edit logs based on its own rate of metadata growth, as compared to the other. Their edits, image, etc. everything is independent - they also do not know about/talk to each other. Given the above, I'd say (1) is true and (2) is invalid, and (3) can be found in PDFs attached to https://issues.apache.org/jira/browse/HDFS-1052. On Sun, Feb 24, 2013 at 8:39 PM, YouPeng Yang wrote: > Hi All > > I'm testing the HDFS Federation.I have 2 namenodes in my cluster. > I find that it is Rolling Edit Log continuously on one namenode,however > the other one changes nothing. > > My question : > 1. Is it the right situation. > 2. I have thought that the two namenodes shoud keep concurrency. > why they get differences. > 3. is there any advanced docs about HDFS Federation. > > thanks. > > Regards. > > > -- Harsh J
Re: Rookie to Hadoop
Hi Ganesh, First off, this is the wrong list to ask user questions. The right list is user@hadoop.apache.org, as is instructed on http://hadoop.apache.org/mailing_lists.html. If any text there is unclear and lead you sending your request here, let us know. Please do not send help requests to general@, going forward :) Second - we have a tutorial to setup and run a cluster at http://hadoop.apache.org/docs/stable/single_node_setup.html and more (follow sidebar links). Have you followed that? On Fri, Feb 22, 2013 at 1:25 PM, Ganesh Hariharan wrote: > Hi Team, > > Please guide me through as how do I test Hadoop with minimal configuration, > what are the prerequisites to do that, and results to be seen. > > Regards, > GHH -- Harsh J
Re: MapReduce processing with extra (possibly non-serializable) configuration
How do you imagine sending "data" of any kind (be it in object form, etc.) over the network to other nodes, without implementing or relying on a serialization for it? Are you looking for "easy" Java ways such as the distributed cache from Hazelcast, etc., where this may be taken care for you automatically in some way? :) On Fri, Feb 22, 2013 at 2:40 AM, Public Network Services wrote: > Hi... > > I am trying to put an existing file processing application into Hadoop and > need to find the best way of propagating some extra configuration per split, > in the form of complex and proprietary custom Java objects. > > The general idea is > > A custom InputFormat splits the input data > The same InputFormat prepares the appropriate configuration for each split > Hadoop processes each split in MapReduce, using the split itself and the > corresponding configuration > > The problem is that these configuration objects contain a lot of properties > and references to other complex objects, and so on, therefore it will take a > lot of work to cover all the possible combinations and make the whole thing > serializable (if it can be done in the first place). > > Most probably this is the only way forward, but if anyone has ever dealt > with this problem, please suggest the best approach to follow. > > Thanks! > -- Harsh J
Re: How to test Hadoop MapReduce under another File System NOT HDFS
Hi Ling, On Thu, Feb 21, 2013 at 6:42 PM, Ling Kun wrote: > Dear all, > I am currently look into some other filesystem implementation, like > lustre, gluster, or other NAS with POSIX support, and trying to replace HDFS > with it. > > I have implement a filesystem class( AFS) which will provide interface > to Hadoop MapReduce, like the one of RawLocalFileSystem, and examples like > wordcount, terasort works well. > >However, I am not sure whether my implementation is correct for all the > MapReduce applications that Hadoop MapReduce+Hadoop HDFS can run. > >My question is : > 1. How Hadoop community do MapReduce regression test for any update of > Hadoop HDFS and Hadoop MapReduce We have several unit tests under HDFS sources (you can view it in the sources) which catch quite a bit of regressions, if not all, and for performance differences we manually test with real-life workloads, aside of generic stressing tests such as teragen/sort. The Apache Bigtop project also has integration level tests for the Apache Hadoop ecosystem, which adds onto the stack. > 2. Beside MapReduce wordcount and Terasort examples, are there any missing > filesystem interface support for MapReduce application. Since the FileSystem > has POSIX support, the hsync have also supported. Not sure what you mean here. The major bit of HDFS that MR makes good use of is not the FS impl but the ability of HDFS to expose its locations of replicas to MR for it to schedule its tasks better. This is done by implementing the http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileSystem.html#getFileBlockLocations(org.apache.hadoop.fs.FileStatus,%20long,%20long) form of APIs in HDFS. The blocks feature of HDFS increases processing parallelism potentials, and the locality APIs help leverage that. > 3. According to my test, the performance is worse than the HDFS+MapReduce. > Any suggestion or hint on the performance analysis? ( Without MapReduce, the > performance of the filesystem is better than HDFS and also local > filesystem). > 3.1 the following are the same for the performance comparation: > 3.1.1 architecture: 4 node for MR, and another different 4 nodes for > HDFS/AFS > 3.1.2 application: the input size , the number of mapper and reducers are > the same. Am guessing most likely your trouble is in exposing proper information to MR for scheduling. HDFS gives 3 locations per block, for example, so the MR scheduler has a small collection to choose from. You can devise other tests but what you've thought of above is a good, simple one. -- Harsh J
Re: How to add another file system in Hadoop
What Hemanth points to (fs.TYPE.impl, i.e. fs.cdmi.impl being set to the classname of the custom FS impl.) is correct for the 1.x releases. In 2.x and ahead, the class for a URI is auto-discovered from the classpath (a 'service'). So as long as your jar is present on the user's runtime, the FS class implementing cdmi:// will get loaded and work without requiring a config injection globally like 1.x did. On Thu, Feb 21, 2013 at 6:19 PM, Hemanth Yamijala wrote: > I may be guessing here a bit. Basically a filesystem is identified by the > protocol part of the URI of a file - so a file on the S3 filesystem will > have a URI like s3://... If you look at the core-default.xml file in Hadoop > source, you will see configuration keys like fs..impl and the value > is a class that implements that type of filesystem. So, you'd basically have > to define a cdmi:// type protocol and define a fs.cdmi.impl key in your > Hadoop configuration. > > Thanks > hemanth > > > On Thu, Feb 21, 2013 at 4:18 PM, Agarwal, Nikhil > wrote: >> >> Hi, >> >> >> >> I am planning to add a file system called CDMI under org.apache.hadoop.fs >> in Hadoop, something similar to KFS or S3 which are already there under >> org.apache.hadoop.fs. I wanted to ask that say, I write my file system for >> CDMI and add the package under fs but then how do I tell the core-site.xml >> or other configuration files to use CDMI file system. Where all do I need to >> make changes to enable CDMI file system become a part of Hadoop ? >> >> >> >> Thanks a lot in advance. >> >> >> >> Regards, >> >> Nikhil > > -- Harsh J
Re: ISSUE :Hadoop with HANA using sqoop
The error is truncated, check the actual failed task's logs for complete info: Caused by: com.sap… what? Seems more like a SAP side fault than a Hadoop side one and you should ask on their forums with the stacktrace posted. On Thu, Feb 21, 2013 at 11:58 AM, samir das mohapatra wrote: > Hi All > Can you plese tell me why I am getting error while loading data from > SAP HANA to Hadoop HDFS using sqoop (4.1.2). > > Error Log: > > java.io.IOException: SQLException in nextKeyValue > at > org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:265) > at > org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:458) > at > org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:76) > at > org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:85) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:139) > at > org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:182) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:645) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325) > at org.apache.hadoop.mapred.Child$4.run(Child.java:268) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:416) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332) > at org.apache.hadoop.mapred.Child.main(Child.java:262) > Caused by: com.sap > > Regards, > samir. > > > > -- > > > -- Harsh J
Re: About Hadoop Deb file
Try the debs from the Apache Bigtop project 0.3 release, its a bit of
an older 1.x release but the debs would work well:
http://archive.apache.org/dist/incubator/bigtop/bigtop-0.3.0-incubating/repos/
On Thu, Feb 21, 2013 at 10:26 AM, Mayur Patil wrote:
> Hello,
>
>I am using Ubuntu 12.04 Desktop.
>
>I had downloaded hadoop-1.1.1-1.deb file and check with md5 check-sum it
> says verified and OK.
>
>But when I try to install on Ubuntu it gives warning
>
>
>> Package is of bad quality.
>>
>> This could cause serious problems on your computer
>>
>> Lintian check results for /media/abc/hadoop_1.1.1-1_i386.deb:
>> Use of uninitialized value $ENV{"HOME"} in concatenation (.) or string at
>> /usr/bin/lintian line 108.
>> E: hadoop: arch-independent-package-contains-binary-or-object
>> usr/bin/task-controller
>> E: hadoop: arch-independent-package-contains-binary-or-object
>> usr/lib/libhadoop.a
>> E: hadoop: arch-independent-package-contains-binary-or-object
>> usr/lib/libhadoop.so
>> E: hadoop: arch-independent-package-contains-binary-or-object
>> usr/lib/libhadoop.so.1
>> E: hadoop: arch-independent-package-contains-binary-or-object
>> usr/lib/libhadoop.so.1.0.0
>> E: hadoop: arch-independent-package-contains-binary-or-object
>> usr/lib/libhadooppipes.a
>> E: hadoop: arch-independent-package-contains-binary-or-object
>> usr/lib/libhadooputils.a
>> E: hadoop: arch-independent-package-contains-binary-or-object
>> usr/lib/libhdfs.a
>> E: hadoop: arch-independent-package-contains-binary-or-object
>> usr/lib/libhdfs.so
>> E: hadoop: arch-independent-package-contains-binary-or-object
>> usr/lib/libhdfs.so.0
>> E: hadoop: arch-independent-package-contains-binary-or-object
>> usr/lib/libhdfs.so.0.0.0
>> E: hadoop: arch-independent-package-contains-binary-or-object
>> usr/libexec/jsvc.i386
>
>
> What's gone wrong ??
>
> Thanks !!
> --
> Cheers,
> Mayur.
--
Harsh J
Re: Topology script frequency
NN refers to the rack topology script/class when a new node joins (i.e. it doesn't have the node's IP already in cache), when it starts up, and (I think) also when you issue -refreshNodes. The ideal way to "add" a node to the rack right now is to first update the rack config at the NN, then boot the DN up. On Thu, Feb 21, 2013 at 1:47 AM, Chris Embree wrote: > I've checked all of the documentation, books and google searches I can think > of > > I have a working topology script. I have dynamic IP's. I have an automated > process to update the rack data when a datanode changes IP. > > What I don't have is any clue as to when the NN reads this script. If I > execute it right now with 10.10.10.10 it'll return /dc1/rack3 When I run > hadoop balancer I see /default/rack/10.10.10.10:50010 > I really don't want to have to restart the NN for IP changes to be noticed. > > Any brilliant insights? > > Thanks > Chris -- Harsh J
Re: copy chunk of hadoop output
No problem JM, I was confused as well. AFAIK, there's no shell utility that can let you specify an offset # of bytes to start off with (similar to skip in dd?), but that can be done from the FS API. On Thu, Feb 21, 2013 at 1:14 AM, Jean-Marc Spaggiari wrote: > Hi Harsh, > > My bad. > > I read the example quickly and I don't know why I tought you used tail > and not head. > > head will work perfectly. But tail will not since it will need to read > the entier file. My comment was for tail, not for head, and therefore > not application to the example you gave. > > > hadoop fs -cat 100-byte-dfs-file | tail -c 5 > 5-byte-local-file > > Will have to download the entire file. > > Is there a way to "jump" into a certain position in a file and "cat" from > there? > > JM > > 2013/2/20, Harsh J : >> Hi JM, >> >> I am not sure how "dangerous" it is, since we're using a pipe here, >> and as you yourself note, it will only last as long as the last bytes >> have been got and then terminate. >> >> The -cat process will terminate because the >> process we're piping to will terminate first after it reaches its goal >> of -c ; so certainly the "-cat" program will not fetch the >> whole file down but it may fetch a few bytes extra over communication >> due to use of read buffers (the extra data won't be put into the target >> file, and get discarded). >> >> We can try it out and observe the "clienttrace" logged >> at the DN at the end of the -cat's read. Here's an example: >> >> I wrote a 1.6~ MB file into a file called "foo.jar", see "bytes" >> below, its ~1.58 MB: >> >> 2013-02-20 23:55:19,777 INFO >> org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: >> /127.0.0.1:58785, dest: /127.0.0.1:50010, bytes: 1658314, op: >> HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_915204057_1, offset: 0, >> srvID: DS-1092147940-192.168.2.1-50010-1349279636946, blockid: >> BP-1461691939-192.168.2.1-1349279623549:blk_2568668834545125596_73870, >> duration: 192289000 >> >> I ran the command "hadoop fs -cat foo.jar | head -c 5 > foo.xml" to >> store first 5 bytes onto a local file: >> >> Asserting that post command we get 5 bytes: >> ➜ ~ wc -c foo.xml >>5 foo.xml >> >> Asserting that DN didn't IO-read the whole file, see the read op below >> and its "bytes" parameter, its only about 193 KB, not the whole block >> of 1.58 MB we wrote earlier: >> >> 2013-02-21 00:01:32,437 INFO >> org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: >> /127.0.0.1:50010, dest: /127.0.0.1:58802, bytes: 198144, op: >> HDFS_READ, cliID: DFSClient_NONMAPREDUCE_-1698829178_1, offset: 0, >> srvID: DS-1092147940-192.168.2.1-50010-1349279636946, blockid: >> BP-1461691939-192.168.2.1-1349279623549:blk_2568668834545125596_73870, >> duration: 19207000 >> >> I don't see how this is anymore dangerous than doing a >> -copyToLocal/-get, which retrieves the whole file anyway? >> >> On Wed, Feb 20, 2013 at 9:25 PM, Jean-Marc Spaggiari >> wrote: >>> But be careful. >>> >>> hadoop fs -cat will retrieve the entire file and last only when it >>> will have retrieve the last bytes you are looking for. >>> >>> If your file is many GB big, it will take a lot of time for this >>> command to complete and will put some pressure on your network. >>> >>> JM >>> >>> 2013/2/19, jamal sasha : >>>> Awesome thanks :) >>>> >>>> >>>> On Tue, Feb 19, 2013 at 2:14 PM, Harsh J wrote: >>>> >>>>> You can instead use 'fs -cat' and the 'head' coreutil, as one example: >>>>> >>>>> hadoop fs -cat 100-byte-dfs-file | head -c 5 > 5-byte-local-file >>>>> >>>>> On Wed, Feb 20, 2013 at 3:38 AM, jamal sasha >>>>> wrote: >>>>> > Hi, >>>>> > I was wondering in the following command: >>>>> > >>>>> > bin/hadoop dfs -copyToLocal hdfspath localpath >>>>> > can we have specify to copy not full but like xMB's of file to local >>>>> drive? >>>>> > >>>>> > Is something like this possible >>>>> > Thanks >>>>> > Jamal >>>>> >>>>> >>>>> >>>>> -- >>>>> Harsh J >>>>> >>>> >> >> >> >> -- >> Harsh J >> -- Harsh J
Re: copy chunk of hadoop output
Hi JM, I am not sure how "dangerous" it is, since we're using a pipe here, and as you yourself note, it will only last as long as the last bytes have been got and then terminate. The -cat process will terminate because the process we're piping to will terminate first after it reaches its goal of -c ; so certainly the "-cat" program will not fetch the whole file down but it may fetch a few bytes extra over communication due to use of read buffers (the extra data won't be put into the target file, and get discarded). We can try it out and observe the "clienttrace" logged at the DN at the end of the -cat's read. Here's an example: I wrote a 1.6~ MB file into a file called "foo.jar", see "bytes" below, its ~1.58 MB: 2013-02-20 23:55:19,777 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /127.0.0.1:58785, dest: /127.0.0.1:50010, bytes: 1658314, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_915204057_1, offset: 0, srvID: DS-1092147940-192.168.2.1-50010-1349279636946, blockid: BP-1461691939-192.168.2.1-1349279623549:blk_2568668834545125596_73870, duration: 192289000 I ran the command "hadoop fs -cat foo.jar | head -c 5 > foo.xml" to store first 5 bytes onto a local file: Asserting that post command we get 5 bytes: ➜ ~ wc -c foo.xml 5 foo.xml Asserting that DN didn't IO-read the whole file, see the read op below and its "bytes" parameter, its only about 193 KB, not the whole block of 1.58 MB we wrote earlier: 2013-02-21 00:01:32,437 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /127.0.0.1:50010, dest: /127.0.0.1:58802, bytes: 198144, op: HDFS_READ, cliID: DFSClient_NONMAPREDUCE_-1698829178_1, offset: 0, srvID: DS-1092147940-192.168.2.1-50010-1349279636946, blockid: BP-1461691939-192.168.2.1-1349279623549:blk_2568668834545125596_73870, duration: 19207000 I don't see how this is anymore dangerous than doing a -copyToLocal/-get, which retrieves the whole file anyway? On Wed, Feb 20, 2013 at 9:25 PM, Jean-Marc Spaggiari wrote: > But be careful. > > hadoop fs -cat will retrieve the entire file and last only when it > will have retrieve the last bytes you are looking for. > > If your file is many GB big, it will take a lot of time for this > command to complete and will put some pressure on your network. > > JM > > 2013/2/19, jamal sasha : >> Awesome thanks :) >> >> >> On Tue, Feb 19, 2013 at 2:14 PM, Harsh J wrote: >> >>> You can instead use 'fs -cat' and the 'head' coreutil, as one example: >>> >>> hadoop fs -cat 100-byte-dfs-file | head -c 5 > 5-byte-local-file >>> >>> On Wed, Feb 20, 2013 at 3:38 AM, jamal sasha >>> wrote: >>> > Hi, >>> > I was wondering in the following command: >>> > >>> > bin/hadoop dfs -copyToLocal hdfspath localpath >>> > can we have specify to copy not full but like xMB's of file to local >>> drive? >>> > >>> > Is something like this possible >>> > Thanks >>> > Jamal >>> >>> >>> >>> -- >>> Harsh J >>> >> -- Harsh J
Re: copy chunk of hadoop output
You can instead use 'fs -cat' and the 'head' coreutil, as one example: hadoop fs -cat 100-byte-dfs-file | head -c 5 > 5-byte-local-file On Wed, Feb 20, 2013 at 3:38 AM, jamal sasha wrote: > Hi, > I was wondering in the following command: > > bin/hadoop dfs -copyToLocal hdfspath localpath > can we have specify to copy not full but like xMB's of file to local drive? > > Is something like this possible > Thanks > Jamal -- Harsh J
Re: webapps/ CLASSPATH err
TAL
> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
> java.io.FileNotFoundException: webapps/hdfs not found in CLASSPATH
> at
> org.apache.hadoop.http.HttpServer.getWebAppsPath(HttpServer.java:560)
> at org.apache.hadoop.http.HttpServer.(HttpServer.java:247)
> at org.apache.hadoop.http.HttpServer.(HttpServer.java:171)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer$1.(NameNodeHttpServer.java:89)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:87)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:547)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.startCommonServices(NameNode.java:480)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:443)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:608)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:589)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
> 2013-02-19 19:15:20,447 INFO org.apache.hadoop.util.ExitUtil: Exiting with
> status 1
> 2013-02-19 19:15:20,474 INFO org.apache.hadoop.hdfs.server.namenode.NameNode:
> SHUTDOWN_MSG:
> /
> SHUTDOWN_MSG: Shutting down NameNode at ip-13-0-177-11/127.0.0.1
> /
>
> This is particularly confusing because, while the hadoop-2.0.0-mr1-cdh4.1.3/
> dir does have a webapps/ dir, there is no "hdfs" file or dir in that
> webapps/. It contains job/, static/, and task/.
>
> If I start over from a freshly formatted namenode and take a slightly
> different approach -- if I try to start the datanode immediately after
> starting the namenode -- once again it fails, and in a very similar way.
> This time the command to start the datanode has two effects: the namenode log
> still can't find webapps/hdfs, just as shown above, and also, there is now a
> datanode log file, and it likewise can't find webapps/datanode
> ("java.io.FileNotFoundException: webapps/datanode not found in CLASSPATH") so
> I get two very similar errors at once, one on the namenode and one on the
> datanode.
>
> This webapps/ dir business makes no sense since the files (or directories)
> the logs claim to be looking for inside webapps/ ("hdfs" and "datanode")
> don't exist!
>
> Thoughts?
>
>
> Keith Wiley kwi...@keithwiley.com keithwiley.com
> music.keithwiley.com
>
> "It's a fine line between meticulous and obsessive-compulsive and a slippery
> rope between obsessive-compulsive and debilitatingly slow."
>-- Keith Wiley
>
>
--
Harsh J
Re: Trouble in running MapReduce application
Oops. I just noticed Hemanth has been answering on a dupe thread as well. Lets drop this thread and carry on there :) On Tue, Feb 19, 2013 at 11:14 PM, Harsh J wrote: > Hi, > > The new error usually happens if you compile using Java 7 and try to > run via Java 6 (for example). That is, an incompatibility in the > runtimes for the binary artifact produced. > > On Tue, Feb 19, 2013 at 10:09 PM, Fatih Haltas wrote: >> Thank you very much Harsh, >> >> Now, as I promised earlier I am much obliged to you. >> >> But, now I solved that problem by just changing the directories then again >> creating a jar file of org. but I am getting this error: >> >> 1.) What I got >> -- >> [hadoop@ADUAE042-LAP-V flowclasses_18_02]$ hadoop jar flow19028pm.jar >> org.myorg.MapReduce /home/hadoop/project/hadoop-data/NetFlow 19_02.out >> Warning: $HADOOP_HOME is deprecated. >> >> Exception in thread "main" java.lang.UnsupportedClassVersionError: >> org/myorg/MapReduce : Unsupported major.minor version 51.0 >> at java.lang.ClassLoader.defineClass1(Native Method) >> at java.lang.ClassLoader.defineClass(ClassLoader.java:634) >> at >> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) >> at java.net.URLClassLoader.defineClass(URLClassLoader.java:277) >> at java.net.URLClassLoader.access$000(URLClassLoader.java:73) >> at java.net.URLClassLoader$1.run(URLClassLoader.java:212) >> at java.security.AccessController.doPrivileged(Native Method) >> at java.net.URLClassLoader.findClass(URLClassLoader.java:205) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:321) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:266) >> at java.lang.Class.forName0(Native Method) >> at java.lang.Class.forName(Class.java:266) >> at org.apache.hadoop.util.RunJar.main(RunJar.java:149) >> >> 2.) How I create my jar >> - >> [hadoop@ADUAE042-LAP-V flowclasses_18_02]$ jar cvf flow19028pm.jar org >> added manifest >> adding: org/(in = 0) (out= 0)(stored 0%) >> adding: org/myorg/(in = 0) (out= 0)(stored 0%) >> adding: org/myorg/MapReduce$FlowPortReducer.class(in = 1661) (out= >> 690)(deflated 58%) >> adding: org/myorg/MapReduce.class(in = 1587) (out= 903)(deflated 43%) >> adding: org/myorg/MapReduce$FlowPortMapper.class(in = 1874) (out= >> 823)(deflated 56%) >> >> 3.) Content of my jar file >> --- >> [hadoop@ADUAE042-LAP-V flowclasses_18_02]$ jar tf flow19028pm.jar >> META-INF/ >> META-INF/MANIFEST.MF >> org/ >> org/myorg/ >> org/myorg/MapReduce$FlowPortReducer.class >> org/myorg/MapReduce.class >> org/myorg/MapReduce$FlowPortMapper.class >> - >> >> >> Thank you very much. >> >> >> On Tue, Feb 19, 2013 at 8:20 PM, Harsh J wrote: >>> >>> Your point (4) explains the problem. The jar packed structure should >>> look like the below, and not how it is presently (one extra top level >>> dir is present): >>> >>> META-INF/ >>> META-INF/MANIFEST.MF >>> org/ >>> org/myorg/ >>> org/myorg/WordCount.class >>> org/myorg/WordCount$TokenizerMapper.class >>> org/myorg/WordCount$IntSumReducer.class >>> >>> On Tue, Feb 19, 2013 at 9:29 PM, Fatih Haltas >>> wrote: >>> > Hi everyone, >>> > >>> > I know this is the common mistake to not specify the class adress while >>> > trying to run a jar, however, >>> > although I specified, I am still getting the ClassNotFound exception. >>> > >>> > What may be the reason for it? I have been struggling for this problem >>> > more >>> > than a 2 days. >>> > I just wrote different MapReduce application for some anlaysis. I got >>> > this >>> > problem. >>> > >>> > To check, is there something wrong with my system, i tried to run >>> > WordCount >>> > example. >>> > When I just run hadoop-examples wordcount, it is working fine. >>> > >>> > But when I add just "package org.myorg;" command at the beginning, it >>
Re: Trouble in running MapReduce application
Hi, The new error usually happens if you compile using Java 7 and try to run via Java 6 (for example). That is, an incompatibility in the runtimes for the binary artifact produced. On Tue, Feb 19, 2013 at 10:09 PM, Fatih Haltas wrote: > Thank you very much Harsh, > > Now, as I promised earlier I am much obliged to you. > > But, now I solved that problem by just changing the directories then again > creating a jar file of org. but I am getting this error: > > 1.) What I got > -- > [hadoop@ADUAE042-LAP-V flowclasses_18_02]$ hadoop jar flow19028pm.jar > org.myorg.MapReduce /home/hadoop/project/hadoop-data/NetFlow 19_02.out > Warning: $HADOOP_HOME is deprecated. > > Exception in thread "main" java.lang.UnsupportedClassVersionError: > org/myorg/MapReduce : Unsupported major.minor version 51.0 > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:634) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:277) > at java.net.URLClassLoader.access$000(URLClassLoader.java:73) > at java.net.URLClassLoader$1.run(URLClassLoader.java:212) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:205) > at java.lang.ClassLoader.loadClass(ClassLoader.java:321) > at java.lang.ClassLoader.loadClass(ClassLoader.java:266) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:266) > at org.apache.hadoop.util.RunJar.main(RunJar.java:149) > > 2.) How I create my jar > - > [hadoop@ADUAE042-LAP-V flowclasses_18_02]$ jar cvf flow19028pm.jar org > added manifest > adding: org/(in = 0) (out= 0)(stored 0%) > adding: org/myorg/(in = 0) (out= 0)(stored 0%) > adding: org/myorg/MapReduce$FlowPortReducer.class(in = 1661) (out= > 690)(deflated 58%) > adding: org/myorg/MapReduce.class(in = 1587) (out= 903)(deflated 43%) > adding: org/myorg/MapReduce$FlowPortMapper.class(in = 1874) (out= > 823)(deflated 56%) > > 3.) Content of my jar file > --- > [hadoop@ADUAE042-LAP-V flowclasses_18_02]$ jar tf flow19028pm.jar > META-INF/ > META-INF/MANIFEST.MF > org/ > org/myorg/ > org/myorg/MapReduce$FlowPortReducer.class > org/myorg/MapReduce.class > org/myorg/MapReduce$FlowPortMapper.class > - > > > Thank you very much. > > > On Tue, Feb 19, 2013 at 8:20 PM, Harsh J wrote: >> >> Your point (4) explains the problem. The jar packed structure should >> look like the below, and not how it is presently (one extra top level >> dir is present): >> >> META-INF/ >> META-INF/MANIFEST.MF >> org/ >> org/myorg/ >> org/myorg/WordCount.class >> org/myorg/WordCount$TokenizerMapper.class >> org/myorg/WordCount$IntSumReducer.class >> >> On Tue, Feb 19, 2013 at 9:29 PM, Fatih Haltas >> wrote: >> > Hi everyone, >> > >> > I know this is the common mistake to not specify the class adress while >> > trying to run a jar, however, >> > although I specified, I am still getting the ClassNotFound exception. >> > >> > What may be the reason for it? I have been struggling for this problem >> > more >> > than a 2 days. >> > I just wrote different MapReduce application for some anlaysis. I got >> > this >> > problem. >> > >> > To check, is there something wrong with my system, i tried to run >> > WordCount >> > example. >> > When I just run hadoop-examples wordcount, it is working fine. >> > >> > But when I add just "package org.myorg;" command at the beginning, it >> > doesnot work. >> > >> > Here is what I have done so far >> > >> > * >> > 1. I just copied wordcount code from the apaches own examples source >> > code >> > and I just changed package decleration as "package org.myorg;" >> > >> > ** >> > 2. Then I tried to run that command: >> > >> > **
Re: Namenode formatting problem
To simplify my previous post, your IPs for the master/slave/etc. in /etc/hosts file should match the ones reported by "ifconfig" always. In proper deployments, IP is static. If IP is dynamic, we'll need to think of some different ways. On Tue, Feb 19, 2013 at 9:53 PM, Harsh J wrote: > Hey Keith, > > I'm guessing whatever "ip-13-0-177-110" is resolving to (ping to > check), is not what is your local IP on that machine (or rather, it > isn't the machine you intended to start it on)? > > Not sure if EC2 grants static IPs, but otherwise a change in the > assigned IP (checkable via ifconfig) would probably explain the > "Cannot assign" error received when we tried a bind() syscall. > > On Tue, Feb 19, 2013 at 4:30 AM, Keith Wiley wrote: >> This is Hadoop 2.0. Formatting the namenode produces no errors in the >> shell, but the log shows this: >> >> 2013-02-18 22:19:46,961 FATAL >> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join >> java.net.BindException: Problem binding to [ip-13-0-177-110:9212] >> java.net.BindException: Cannot assign requested address; For more details >> see: http://wiki.apache.org/hadoop/BindException >> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:710) >> at org.apache.hadoop.ipc.Server.bind(Server.java:356) >> at org.apache.hadoop.ipc.Server$Listener.(Server.java:454) >> at org.apache.hadoop.ipc.Server.(Server.java:1833) >> at org.apache.hadoop.ipc.RPC$Server.(RPC.java:866) >> at >> org.apache.hadoop.ipc.ProtobufRpcEngine$Server.(ProtobufRpcEngine.java:375) >> at >> org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:350) >> at org.apache.hadoop.ipc.RPC.getServer(RPC.java:695) >> at org.apache.hadoop.ipc.RPC.getServer(RPC.java:684) >> at >> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.(NameNodeRpcServer.java:238) >> at >> org.apache.hadoop.hdfs.server.namenode.NameNode.createRpcServer(NameNode.java:452) >> at >> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:434) >> at >> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:608) >> at >> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:589) >> at >> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140) >> at >> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204) >> 2013-02-18 22:19:46,988 INFO org.apache.hadoop.util.ExitUtil: Exiting with >> status 1 >> 2013-02-18 22:19:46,990 INFO >> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG: >> / >> SHUTDOWN_MSG: Shutting down NameNode at ip-13-0-177-11/127.0.0.1 >> / >> >> No java processes begin (although I wouldn't expect formatting the namenode >> to start any processes, only starting the namenode or datanode should do >> that), and "hadoop fs -ls /" gives me this: >> >> ls: Call From [CLIENT_HOST]/127.0.0.1 to [MASTER_HOST]:9000 failed on >> connection exception: java.net.ConnectException: Connection refused; For >> more details see: http://wiki.apache.org/hadoop/ConnectionRefused >> >> My /etc/hosts looks like this: >> 127.0.0.1 localhost localhost.localdomain CLIENT_HOST >> MASTER_IP MASTER_HOST master >> SLAVE_IP SLAVE_HOST slave01 >> >> This is on EC2. All of the nodes are in the same security group and the >> security group has full inbound access. I can ssh between all three >> machines (client/master/slave) without a password ala authorized_keys. I >> can ping the master node from the client machine (although I don't know how >> to ping a specific port, such as the hdfs port (9000)). Telnet doesn't >> behave on EC2 which makes port testing a little difficult. >> >> Any ideas? >> >> >> Keith Wiley kwi...@keithwiley.com keithwiley.com >> music.keithwiley.com >> >> "The easy confidence with which I know another man's religion is folly >> teaches >> me to suspect that my own is also." >>-- Mark Twain >> >> > > > > -- > Harsh J -- Harsh J
Re: Namenode formatting problem
Hey Keith, I'm guessing whatever "ip-13-0-177-110" is resolving to (ping to check), is not what is your local IP on that machine (or rather, it isn't the machine you intended to start it on)? Not sure if EC2 grants static IPs, but otherwise a change in the assigned IP (checkable via ifconfig) would probably explain the "Cannot assign" error received when we tried a bind() syscall. On Tue, Feb 19, 2013 at 4:30 AM, Keith Wiley wrote: > This is Hadoop 2.0. Formatting the namenode produces no errors in the shell, > but the log shows this: > > 2013-02-18 22:19:46,961 FATAL > org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join > java.net.BindException: Problem binding to [ip-13-0-177-110:9212] > java.net.BindException: Cannot assign requested address; For more details > see: http://wiki.apache.org/hadoop/BindException > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:710) > at org.apache.hadoop.ipc.Server.bind(Server.java:356) > at org.apache.hadoop.ipc.Server$Listener.(Server.java:454) > at org.apache.hadoop.ipc.Server.(Server.java:1833) > at org.apache.hadoop.ipc.RPC$Server.(RPC.java:866) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server.(ProtobufRpcEngine.java:375) > at > org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:350) > at org.apache.hadoop.ipc.RPC.getServer(RPC.java:695) > at org.apache.hadoop.ipc.RPC.getServer(RPC.java:684) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.(NameNodeRpcServer.java:238) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.createRpcServer(NameNode.java:452) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:434) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:608) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:589) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204) > 2013-02-18 22:19:46,988 INFO org.apache.hadoop.util.ExitUtil: Exiting with > status 1 > 2013-02-18 22:19:46,990 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: > SHUTDOWN_MSG: > / > SHUTDOWN_MSG: Shutting down NameNode at ip-13-0-177-11/127.0.0.1 > / > > No java processes begin (although I wouldn't expect formatting the namenode > to start any processes, only starting the namenode or datanode should do > that), and "hadoop fs -ls /" gives me this: > > ls: Call From [CLIENT_HOST]/127.0.0.1 to [MASTER_HOST]:9000 failed on > connection exception: java.net.ConnectException: Connection refused; For more > details see: http://wiki.apache.org/hadoop/ConnectionRefused > > My /etc/hosts looks like this: > 127.0.0.1 localhost localhost.localdomain CLIENT_HOST > MASTER_IP MASTER_HOST master > SLAVE_IP SLAVE_HOST slave01 > > This is on EC2. All of the nodes are in the same security group and the > security group has full inbound access. I can ssh between all three machines > (client/master/slave) without a password ala authorized_keys. I can ping the > master node from the client machine (although I don't know how to ping a > specific port, such as the hdfs port (9000)). Telnet doesn't behave on EC2 > which makes port testing a little difficult. > > Any ideas? > > > Keith Wiley kwi...@keithwiley.com keithwiley.com > music.keithwiley.com > > "The easy confidence with which I know another man's religion is folly teaches > me to suspect that my own is also." >-- Mark Twain > > -- Harsh J
Re: Trouble in running MapReduce application
Your point (4) explains the problem. The jar packed structure should look like the below, and not how it is presently (one extra top level dir is present): META-INF/ META-INF/MANIFEST.MF org/ org/myorg/ org/myorg/WordCount.class org/myorg/WordCount$TokenizerMapper.class org/myorg/WordCount$IntSumReducer.class On Tue, Feb 19, 2013 at 9:29 PM, Fatih Haltas wrote: > Hi everyone, > > I know this is the common mistake to not specify the class adress while > trying to run a jar, however, > although I specified, I am still getting the ClassNotFound exception. > > What may be the reason for it? I have been struggling for this problem more > than a 2 days. > I just wrote different MapReduce application for some anlaysis. I got this > problem. > > To check, is there something wrong with my system, i tried to run WordCount > example. > When I just run hadoop-examples wordcount, it is working fine. > > But when I add just "package org.myorg;" command at the beginning, it > doesnot work. > > Here is what I have done so far > * > 1. I just copied wordcount code from the apaches own examples source code > and I just changed package decleration as "package org.myorg;" > ** > 2. Then I tried to run that command: > * > "hadoop jar wordcount_19_02.jar org.myorg.WordCount > /home/hadoop/project/hadoop-data/NetFlow 19_02_wordcount.output" > * > 3. I got following error: > ** > [hadoop@ADUAE042-LAP-V project]$ hadoop jar wordcount_19_02.jar > org.myorg.WordCount /home/hadoop/project/hadoop-data/NetFlow > 19_02_wordcount.output > Warning: $HADOOP_HOME is deprecated. > > Exception in thread "main" java.lang.ClassNotFoundException: > org.myorg.WordCount > at java.net.URLClassLoader$1.run(URLClassLoader.java:217) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:205) > at java.lang.ClassLoader.loadClass(ClassLoader.java:321) > at java.lang.ClassLoader.loadClass(ClassLoader.java:266) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:266) > at org.apache.hadoop.util.RunJar.main(RunJar.java:149) > > ** > 4. This is the content of my .jar file: > > [hadoop@ADUAE042-LAP-V project]$ jar tf wordcount_19_02.jar > META-INF/ > META-INF/MANIFEST.MF > wordcount_classes/ > wordcount_classes/org/ > wordcount_classes/org/myorg/ > wordcount_classes/org/myorg/WordCount.class > wordcount_classes/org/myorg/WordCount$TokenizerMapper.class > wordcount_classes/org/myorg/WordCount$IntSumReducer.class > ** > 5. This is the 'ls' output of my working directory: > ** > [hadoop@ADUAE042-LAP-V project]$ ls > flowclasses_18_02 flowclasses_18_02.jar hadoop-1.0.4 hadoop-1.0.4.tar.gz > hadoop-data MapReduce.java sample wordcount_19_02.jar wordcount_classes > WordCountClasses WordCount.java > * > So as you see, package decleration is fine but I am really helpless, I > googled but they are all saying samething you should specify the package > hierarchy of your main class. I did know it already I am specifying but > doesn't work. > > I would be much obliged to anyone helped me > > Regards, -- Harsh J
Re: Piping output of hadoop command
Hi, The command you're looking for is not -copyToLocal (it doesn't really emit the file, which you seem to need here), but rather a simple -cat: Something like the below would make your command work: $ hadoop fs -cat FILE_IN_HDFS | ssh REMOTE_HOST "dd of=TARGET_FILE" On Mon, Feb 18, 2013 at 10:46 PM, Julian Wissmann wrote: > Hi, > > we're running a Hadoop cluster with hbase for the purpose of > evaluating it as database for a research project and we've more or > less decided to go with it. > So now I'm exploring backup mechanisms and have decided to experiment > with hadoops export functionality for that. > > What I am trying to achieve is getting data out of hbase and into hdfs > via hadoop export and then copy it out of hdfs onto a backup system. > However while copying data out of hdfs to the backup machine I am > experiencing problems. > > What I am trying to do is the following: > > hadoop fs -copyToLocal FILE_IN_HDFS | ssh REMOTE_HOST "dd of=TARGET_FILE" > > It creates a file on the remote host, however this file is 0kb in > size; instead of copying any data over there, the file just lands in > my home folder. > > The command output looks like this: hadoop fs -copyToLocal > FILE_IN_HDFS | ssh REMOTE_HOST "dd of=FILE_ON REMOTE_HOST" > 0+0 Datensätze ein > 0+0 Datensätze aus > 0 Bytes (0 B) kopiert, 1,10011 s, 0,0 kB/s > > I cannot think of any reason, why this command would behave in this > way. Is this some Java-ism that I'm missing here (like not correctly > treating stdout), or am I actually doing it wrong? > > The Hadoop Version is 2.0.0-cdh4.1.2 > > Regards > > Julian -- Harsh J
Re: Namenode failures
Hi Robert, Are you by any chance adding files carrying unusual encoding? If its possible, can we be sent a bundle of the corrupted log set (all of the dfs.name.dir contents) to inspect what seems to be causing the corruption? The only identified (but rarely occurring) bug around this part in 1.0.4 would be https://issues.apache.org/jira/browse/HDFS-4423. The other major corruption bug I know of is already fixed in your version, being https://issues.apache.org/jira/browse/HDFS-3652 specifically. We've not had this report from other users so having a reproduced file set (data not required) would be most helpful. If you have logs leading to the shutdown and crash as well, that'd be good to have too. P.s. How exactly are you shutting down the NN each time? A kill -9 or a regular SIGTERM shutdown? On Mon, Feb 18, 2013 at 4:31 AM, Robert Dyer wrote: > On Sun, Feb 17, 2013 at 4:41 PM, Mohammad Tariq wrote: >> >> You can make use of offine image viewer to diagnose >> the fsimage file. > > > Is this not included in the 1.0.x branch? All of the documentation I find > for it says to run 'bin/hdfs oev' but I do not have a 'bin/hdfs'. > >> >> Warm Regards, >> Tariq >> https://mtariq.jux.com/ >> cloudfront.blogspot.com >> >> >> On Mon, Feb 18, 2013 at 3:31 AM, Robert Dyer wrote: >>> >>> It just happened again. This was after a fresh format of HDFS/HBase and >>> I am attempting to re-import the (backed up) data. >>> >>> http://pastebin.com/3fsWCNQY >>> >>> So now if I restart the namenode, I will lose data from the past 3 hours. >>> >>> What is causing this? How can I avoid it in the future? Is there an >>> easy way to monitor (other than a script grep'ing the logs) the checkpoints >>> to see when this happens? >>> >>> >>> On Sat, Feb 16, 2013 at 2:39 PM, Robert Dyer wrote: >>>> >>>> Forgot to mention: Hadoop 1.0.4 >>>> >>>> >>>> On Sat, Feb 16, 2013 at 2:38 PM, Robert Dyer wrote: >>>>> >>>>> I am at a bit of wits end here. Every single time I restart the >>>>> namenode, I get this crash: >>>>> >>>>> 2013-02-16 14:32:42,616 INFO >>>>> org.apache.hadoop.hdfs.server.common.Storage: Image file of size 168058 >>>>> loaded in 0 seconds. >>>>> 2013-02-16 14:32:42,618 ERROR >>>>> org.apache.hadoop.hdfs.server.namenode.NameNode: >>>>> java.lang.NullPointerException >>>>> at >>>>> org.apache.hadoop.hdfs.server.namenode.FSDirectory.addChild(FSDirectory.java:1099) >>>>> at >>>>> org.apache.hadoop.hdfs.server.namenode.FSDirectory.addChild(FSDirectory.java:) >>>>> at >>>>> org.apache.hadoop.hdfs.server.namenode.FSDirectory.addNode(FSDirectory.java:1014) >>>>> at >>>>> org.apache.hadoop.hdfs.server.namenode.FSDirectory.unprotectedAddFile(FSDirectory.java:208) >>>>> at >>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog.loadFSEdits(FSEditLog.java:631) >>>>> at >>>>> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSEdits(FSImage.java:1021) >>>>> at >>>>> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:839) >>>>> at >>>>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:377) >>>>> at >>>>> org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:100) >>>>> at >>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:388) >>>>> at >>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.(FSNamesystem.java:362) >>>>> at >>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:276) >>>>> at >>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:496) >>>>> at >>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1279) >>>>> at >>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1288) >>>>> >>>>> I am following best practices here, as far as I know. I have the >>>>> namenode writing into 3 directories (2 local, 1 NFS). All 3 of these dirs >>>>> have the exact same files in them. >>>>> >>>>> I also run a secondary checkpoint node. This one appears to have >>>>> started failing a week ago. So checkpoints were *not* being done since >>>>> then. Thus I can get the NN up and running, but with a week old data! >>>>> >>>>> What is going on here? Why does my NN data *always* wind up causing >>>>> this exception over time? Is there some easy way to get notified when the >>>>> checkpointing starts to fail? >>>> >>>> >>>> >>>> >>>> -- >>>> >>>> Robert Dyer >>>> rd...@iastate.edu >>> >>> >>> >>> >>> -- >>> >>> Robert Dyer >>> rd...@iastate.edu >> >> > > > > -- > > Robert Dyer > rd...@iastate.edu -- Harsh J
Re: executing hadoop commands from python?
Instead of 'scraping' this way, consider using a library such as Pydoop (http://pydoop.sourceforge.net) which provides pythonic ways and APIs to interact with Hadoop components. There are also other libraries covered at http://blog.cloudera.com/blog/2013/01/a-guide-to-python-frameworks-for-hadoop/ for example. On Sun, Feb 17, 2013 at 4:17 AM, jamal sasha wrote: > Hi, > > This might be more of a python centric question but was wondering if > anyone has tried it out... > > I am trying to run few hadoop commands from python program... > > For example if from command line, you do: > > bin/hadoop dfs -ls /hdfs/query/path > > it returns all the files in the hdfs query path.. > So very similar to unix > > > Now I am trying to basically do this from python.. and do some manipulation > from it. > > exec_str = "path/to/hadoop/bin/hadoop dfs -ls " + query_path > os.system(exec_str) > > Now, I am trying to grab this output to do some manipulation in it. > For example.. count number of files? > I looked into subprocess module but then... these are not native shell > commands. hence not sure whether i can apply those concepts > How to solve this? > > Thanks > > -- Harsh J
Re: QJM deployment
Hi Azuryy, Thanks for your feedback on the docs! I've filed https://issues.apache.org/jira/browse/HDFS-4508 on your behalf to address them. Feel free to file JIRA with documentation complaints with change patches to have them improved yourself :) On Sun, Feb 17, 2013 at 2:25 PM, Azuryy Yu wrote: > Hi, > > who can kindly make 2.0.3-alpha QJM deployment document more better, I > cannot understand it successfully, thanks. > > such as : > 1) by running the command "hdfs namenode -bootstrapStandby" on the > unformatted NameNode > > - - it should be start formatted namenode firstly. > 2) If you are converting a non-HA NameNode to be HA, you should run the > command "hdfs -initializeSharedEdits", which will initialize the > JournalNodes with the edits data from the local NameNode edits directories. > > -- run this command on which node? and does that this command exists? > > > QJM "Deployment details" section is bad > > -- Harsh J
Re: building from subversion repository
Hi George, The error below is your issue: > [ERROR] Could not find goal 'protoc' in plugin > org.apache.hadoop:hadoop-maven-plugins:3.0.0-SNAPSHOT among available goals > -> [Help 1] To build trunk, a protocol buffers (protobuf) compiler installation of version 2.4 at least is required, cause we have that as a dependency. This is mentioned on http://wiki.apache.org/hadoop/HowToContribute, http://wiki.apache.org/hadoop/QwertyManiac/BuildingHadoopTrunk and also the SVN trunk's BUILDING.txt (look for "proto"). Once installed in your OS, the following command and output can be seen to work, and your build should continue successfully: ➜ ~ protoc --version libprotoc 2.4.1 On Sun, Feb 17, 2013 at 5:43 PM, George R Goffe wrote: > Hi, > > I'm trying to build hadoop from a current check out of the repository and am > receiving the following messages. Can someone enlighten me as to what I'm > doing wrong please? > > Thanks, > > George... > > > > [INFO] BUILD FAILURE > [INFO] > > [INFO] Total time: 1:55.493s > [INFO] Finished at: Sun Feb 17 03:58:15 PST 2013 > [INFO] Final Memory: 31M/332M > [INFO] > > [ERROR] Could not find goal 'protoc' in plugin > org.apache.hadoop:hadoop-maven-plugins:3.0.0-SNAPSHOT among available goals > -> [Help 1] > [ERROR] > [ERROR] To see the full stack trace of the errors, re-run Maven with the -e > switch. > [ERROR] Re-run Maven using the -X switch to enable full debug logging. > [ERROR] > [ERROR] For more information about the errors and possible solutions, please > read the following articles: > [ERROR] [Help 1] > http://cwiki.apache.org/confluence/display/MAVEN/MojoNotFoundException > -- Harsh J
Re: why my test result on dfs short circuit read is slower?
If you want HBase to leverage the shortcircuit, the DN config "dfs.block.local-path-access.user" should be set to the user running HBase (i.e. hbase, for example), and the hbase-site.xml should have "dfs.client.read.shortcircuit" defined in all its RegionServers. Doing this wrong could result in performance penalty and some warn-logging, as local reads will be attempted but will begin to fail. On Sat, Feb 16, 2013 at 8:40 AM, Liu, Raymond wrote: > Hi > > I tried to use short circuit read to improve my hbase cluster MR scan > performance. > > I have the following setting in hdfs-site.xml > > dfs.client.read.shortcircuit set to true > dfs.block.local-path-access.user set to MR job runner. > > The cluster is 1+4 node and each data node have 16cpu/4HDD, with all > hbase table major compact thus all data is local. > I have hoped that the short circuit read will improve the performance. > > While the test result is that with short circuit read enabled, the > performance actually dropped 10-15%. Say scan a 50G table cost around 100s > instead of 90s. > > My hadoop version is 1.1.1, any idea on this? Thx! > > Best Regards, > Raymond Liu > > -- Harsh J
Re: Hadoop 2.0.3 namenode issue
I don't see a crash log in your snippets. Mind pastebinning the NN crash log up somewhere? Did both NNs go down? In any case, the log below is due to a client attempting to connect with an older HDFS library. This would log such warns (and also indicate the client IP/attempt port as you notice), but in no case will it cause an NN crash. > 2013-02-13 10:20:02,550 WARN ipc.Server - Incorrect header or version > mismatch from10.232.29.4:49938 got version 4 expected version 7 On Fri, Feb 15, 2013 at 11:58 PM, Dheeren Bebortha wrote: > > > HI, > In one of our test clusters that has Namenode HA using QJM+ YARN + HBase > 0.94, namenode came down with following logs. I am trying to root cause the > issue. Any help is appreciated. > = > 2013-02-13 10:18:27,521 INFO hdfs.StateChange - BLOCK* > NameSystem.fsync: file > /hbase/.logs/datanode-X.sfdomain.com,60020,1360091866476/datanode-X.sfdomain.com%2C60020%2C1360091866476.1360750706694 > for DFSClient_hb_rs_datanode-X.sfdomain.com,60020,1360091866476_470800334_38 > 2013-02-13 10:20:01,861 WARN ipc.Server - Incorrect header or version > mismatch from 10.232.29.4:49933 got version 4 expected version 7 > 2013-02-13 10:20:01,884 WARN ipc.Server - Incorrect header or version > mismatch from 10.232.29.4:49935 got version 4 expected version 7 > 2013-02-13 10:20:02,550 WARN ipc.Server - Incorrect header or version > mismatch from 10.232.29.4:49938 got version 4 expected version 7 > 2013-02-13 10:20:08,210 INFO namenode.FSNamesystem - Roll Edit Log from > 10.232.29.14 > = > = > == > 2013-02-13 12:14:32,879 INFO namenode.FileJournalManager - Finalizing edits > file /data/hdfs/current/edits_inprogress_0065699 -> > /data/hdfs/current/edits_0065699-0065700 > 2013-02-13 12:14:32,879 INFO namenode.FSEditLog - Starting log segment at > 65701 > 2013-02-13 12:15:02,507 INFO namenode.NameNode - FSCK started by sfdc > (auth:SIMPLE) from /10.232.29.4 for path / at Wed Feb 13 12:15:02 > GMT+00:00 2013 > 2013-02-13 12:15:02,663 WARN ipc.Server - Incorrect header or version > mismatch from 10.232.29.4:40025 got version 4 expected version 7 > 2013-02-13 12:15:02,663 WARN ipc.Server - Incorrect header or version > mismatch from 10.232.29.4:40027 got version 4 expected version 7 > 2013-02-13 12:15:03,391 WARN ipc.Server - Incorrect header or version > mismatch from 10.232.29.4:40031 got version 4 expected version 7 > 2013-02-13 12:16:33,181 INFO namenode.FSNamesystem - Roll Edit Log from > 10.232.29.14 > == > == -- Harsh J
Re: Java submit job to remote server
;
>> If that line is disabled, then the job is completed. However, in reviewing
>> the hadoop server administration page
>> (http://localhost:50030/jobtracker.jsp) I don't see the job as processed by
>> the server. Instead, I wonder if my Java code is simply running the
>> necessary mapper Java code, bypassing the locally installed server.
>>
>> Thanks in advance.
>>
>> Alex
>>
>> public class OfflineDataTool extends Configured implements Tool {
>>
>> public int run(final String[] args) throws Exception {
>> final Configuration conf = getConf();
>> //conf.set("mapred.job.tracker", "localhost:9001");
>>
>> final Job job = new Job(conf);
>> job.setJarByClass(getClass());
>> job.setJobName(getClass().getName());
>>
>> job.setMapperClass(OfflineDataMapper.class);
>>
>> job.setInputFormatClass(TextInputFormat.class);
>>
>> job.setMapOutputKeyClass(Text.class);
>> job.setMapOutputValueClass(Text.class);
>>
>> job.setOutputKeyClass(Text.class);
>> job.setOutputValueClass(Text.class);
>>
>> FileInputFormat.addInputPath(job, new
>> org.apache.hadoop.fs.Path(args[0]));
>>
>> final org.apache.hadoop.fs.Path output = new org.a
>
>
--
Harsh J
Re: .deflate trouble
Am pretty sure we do not maintain AMIs here at Apache Hadoop at least. Perhaps those are from some earlier effort by AMZN itself? There is this page that may interest you though?: http://wiki.apache.org/hadoop/AmazonEC2. Sorry for the short responses - I'm not well versed with these things. Someone else can probably assist you further. On Fri, Feb 15, 2013 at 5:05 AM, Keith Wiley wrote: > > I'll look into the job.xml issue, thanks for the suggestion. In the > meantime, who is in charge of maintaining the "official" AWS Hadoop AMIs? > The following are the contents of the hadoop-images/ S3 bucket. As you can > see, it tops out at 19: > > IMAGE ami-65987c0chadoop-images/hadoop-0.17.1-i386.manifest.xml > 914733919441available public i386machine aki-a71cf9ce > ari-a51cf9ccinstance-store paravirtual xen > IMAGE ami-4b987c22hadoop-images/hadoop-0.17.1-x86_64.manifest.xml > 914733919441available public x86_64 machine aki-b51cf9dc > ari-b31cf9dainstance-store paravirtual xen > IMAGE ami-b0fe1ad9hadoop-images/hadoop-0.18.0-i386.manifest.xml > 914733919441available public i386machine aki-a71cf9ce > ari-a51cf9ccinstance-store paravirtual xen > IMAGE ami-90fe1af9hadoop-images/hadoop-0.18.0-x86_64.manifest.xml > 914733919441available public x86_64 machine aki-b51cf9dc > ari-b31cf9dainstance-store paravirtual xen > IMAGE ami-ea36d283hadoop-images/hadoop-0.18.1-i386.manifest.xml > 914733919441available public i386machine aki-a71cf9ce > ari-a51cf9ccinstance-store paravirtual xen > IMAGE ami-fe37d397hadoop-images/hadoop-0.18.1-x86_64.manifest.xml > 914733919441available public x86_64 machine aki-b51cf9dc > ari-b31cf9dainstance-store paravirtual xen > IMAGE ami-fa6a8e93hadoop-images/hadoop-0.19.0-i386.manifest.xml > 914733919441available public i386machine aki-a71cf9ce > ari-a51cf9ccinstance-store paravirtual xen > IMAGE ami-cd6a8ea4hadoop-images/hadoop-0.19.0-x86_64.manifest.xml > 914733919441available public x86_64 machine aki-b51cf9dc > ari-b31cf9dainstance-store paravirtual xen > IMAGE ami-15e80f7chadoop-images/hadoop-base-20090210-i386.manifest.xml > 914733919441available public i386machine > aki-a71cf9ceari-a51cf9ccinstance-store paravirtual xen > IMAGE ami-1ee80f77 > hadoop-images/hadoop-base-20090210-x86_64.manifest.xml 914733919441 > available public x86_64 machine aki-b51cf9dcari-b31cf9da > instance-store paravirtual xen > > > On Feb 14, 2013, at 15:02 , Harsh J wrote: > > > 0.19 is really old and thats probably why the Text utility (fs -text) > > doesn't support automatic decompression based on extensions (or > > specifically, of .deflate). > > > > Did the job.xml of the job that produced this output also carry > > mapred.output.compress=false in it? The file should be viewable on the > > JT UI page for the job. Unless explicitly turned out, even 0.19 > > wouldn't have enabled compression on its own. > > > > On Fri, Feb 15, 2013 at 3:50 AM, Keith Wiley wrote: > >> I just got hadoop running on EC2 (0.19 just because that's the AMI the > >> scripts seemed to go for). The PI example worked and I believe the > >> wordcount example worked too. However, the output file is in .deflate > >> format. "hadoop fs -text" fails to decompress the file -- it produces the > >> same binary output as "hadoop fs -cat", which I find counterintuitive; > >> isn't -text specifically supposed to handle this situation? > >> > >> I copied the file to local and tried manually decompressing it with gunzip > >> and lzop (by appending appropriate suffixes), but both tools failed to > >> recognize the file. To add to the confusion, I see this in the default > >> configuration offered by the EC2 scripts: > >> > >> mapred.output.compress > >> false > >> Should the job outputs be compressed? > >> > >> > >> ...so I don't understand why the output was compressed in the first place. > >> > >> At this point, I'm kind of stuck. The output shouldn't be compressed to > >> begin with, and all attempts to
Re: .deflate trouble
0.19 is really old and thats probably why the Text utility (fs -text) doesn't support automatic decompression based on extensions (or specifically, of .deflate). Did the job.xml of the job that produced this output also carry mapred.output.compress=false in it? The file should be viewable on the JT UI page for the job. Unless explicitly turned out, even 0.19 wouldn't have enabled compression on its own. On Fri, Feb 15, 2013 at 3:50 AM, Keith Wiley wrote: > I just got hadoop running on EC2 (0.19 just because that's the AMI the > scripts seemed to go for). The PI example worked and I believe the wordcount > example worked too. However, the output file is in .deflate format. "hadoop > fs -text" fails to decompress the file -- it produces the same binary output > as "hadoop fs -cat", which I find counterintuitive; isn't -text specifically > supposed to handle this situation? > > I copied the file to local and tried manually decompressing it with gunzip > and lzop (by appending appropriate suffixes), but both tools failed to > recognize the file. To add to the confusion, I see this in the default > configuration offered by the EC2 scripts: > > mapred.output.compress > false > Should the job outputs be compressed? > > > ...so I don't understand why the output was compressed in the first place. > > At this point, I'm kind of stuck. The output shouldn't be compressed to > begin with, and all attempts to decompress it have failed. > > Any ideas? > > Thanks. > > > Keith Wiley kwi...@keithwiley.com keithwiley.com > music.keithwiley.com > > "And what if we picked the wrong religion? Every week, we're just making God > madder and madder!" >-- Homer Simpson > > -- Harsh J
Re: Documentation Link is broken on
We're back to serving docs now I think. Thanks Doug! On Thu, Feb 14, 2013 at 1:07 AM, Harsh J wrote: > I see some change work being done, and docs should appear again soon > enough: http://svn.apache.org/viewvc/hadoop/common/site/common/publish/docs/. > I think we're moving the docs to use a different publishing system - I > notice Doug is on this, but I haven't more info. I'll check back in an > hour if it is still not published live until then. > > On Thu, Feb 14, 2013 at 1:01 AM, Harsh J wrote: >> Looking at the host publish page, I notice docs/ has gone empty, is >> that normal? I don't see a relevant recent commit to have made such a >> change though. >> >> :/www/hadoop.apache.org/docs$ ls -l >> total 0 >> :/www/hadoop.apache.org/docs$ svn log | head >> >> r1382982 | cutting | 2012-09-10 16:58:57 + (Mon, 10 Sep 2012) | 1 line >> >> HADOOP-8662. Consolidate former subproject websites into a single site. >> >> r1134994 | todd | 2011-06-12 22:00:51 + (Sun, 12 Jun 2011) | 2 lines >> >> HADOOP-7106. Reorganize SVN layout to combine HDFS, Common, and MR in >> a single tree (project unsplit) >> >> == >> >> The most recent change I notice made was on 7th, the release push of >> 0.23.6 but this problem happened today and that release's diff has no >> indications of any deletes either, so it is not that. An INFRA issue? >> >> On Thu, Feb 14, 2013 at 12:43 AM, Mayank Bansal wrote: >>> HI Guys, >>> >>> All the documentation links are broken on apache. >>> http://hadoop.apache.org/docs/r0.20.2/ >>> >>> Does anybody know how to fix this? >>> >>> Thanks, >>> Mayank >> >> >> >> -- >> Harsh J > > > > -- > Harsh J -- Harsh J
Re: Documentation Link is broken on
I see some change work being done, and docs should appear again soon enough: http://svn.apache.org/viewvc/hadoop/common/site/common/publish/docs/. I think we're moving the docs to use a different publishing system - I notice Doug is on this, but I haven't more info. I'll check back in an hour if it is still not published live until then. On Thu, Feb 14, 2013 at 1:01 AM, Harsh J wrote: > Looking at the host publish page, I notice docs/ has gone empty, is > that normal? I don't see a relevant recent commit to have made such a > change though. > > :/www/hadoop.apache.org/docs$ ls -l > total 0 > :/www/hadoop.apache.org/docs$ svn log | head > > r1382982 | cutting | 2012-09-10 16:58:57 + (Mon, 10 Sep 2012) | 1 line > > HADOOP-8662. Consolidate former subproject websites into a single site. > > r1134994 | todd | 2011-06-12 22:00:51 + (Sun, 12 Jun 2011) | 2 lines > > HADOOP-7106. Reorganize SVN layout to combine HDFS, Common, and MR in > a single tree (project unsplit) > > == > > The most recent change I notice made was on 7th, the release push of > 0.23.6 but this problem happened today and that release's diff has no > indications of any deletes either, so it is not that. An INFRA issue? > > On Thu, Feb 14, 2013 at 12:43 AM, Mayank Bansal wrote: >> HI Guys, >> >> All the documentation links are broken on apache. >> http://hadoop.apache.org/docs/r0.20.2/ >> >> Does anybody know how to fix this? >> >> Thanks, >> Mayank > > > > -- > Harsh J -- Harsh J
Re: Documentation Link is broken on
Looking at the host publish page, I notice docs/ has gone empty, is that normal? I don't see a relevant recent commit to have made such a change though. :/www/hadoop.apache.org/docs$ ls -l total 0 :/www/hadoop.apache.org/docs$ svn log | head r1382982 | cutting | 2012-09-10 16:58:57 + (Mon, 10 Sep 2012) | 1 line HADOOP-8662. Consolidate former subproject websites into a single site. r1134994 | todd | 2011-06-12 22:00:51 + (Sun, 12 Jun 2011) | 2 lines HADOOP-7106. Reorganize SVN layout to combine HDFS, Common, and MR in a single tree (project unsplit) == The most recent change I notice made was on 7th, the release push of 0.23.6 but this problem happened today and that release's diff has no indications of any deletes either, so it is not that. An INFRA issue? On Thu, Feb 14, 2013 at 12:43 AM, Mayank Bansal wrote: > HI Guys, > > All the documentation links are broken on apache. > http://hadoop.apache.org/docs/r0.20.2/ > > Does anybody know how to fix this? > > Thanks, > Mayank -- Harsh J
Re: mapreduce.map.env
I was incorrect here: MR2 does support this; I failed to look for the
right constant reference and there were two.
On Wed, Feb 13, 2013 at 11:32 PM, Harsh J wrote:
> What version are you specifically asking about?
>
> The MR2 (2.x) does not have this anymore in use (regression? not sure
> what the replacement is, will have to figure out), but 1.x has
> mapred.map/reduce.child.env which you can use to the same effect.
>
> On Wed, Feb 13, 2013 at 11:05 PM, Saptarshi Guha
> wrote:
>> Hello,
>>
>> Is this still valid? Can i set
>>
>> mapreduce.{map|reduce}.env
>>
>> When i google i get links
>>
>> http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0CDIQFjAA&url=http%3A%2F%2Fhadoop.apache.org%2Fdocs%2Fcurrent%2Fhadoop-mapreduce-client%2Fhadoop-mapreduce-client-core%2Fmapred-default.xml&ei=oc4bUa-GDom4igK974DwDw&usg=AFQjCNEnb_-tEiCRsOndsyuGLqtWOJ8OPw&sig2=JVUiA78CsXC7aIdo1X09Lw&bvm=bv.42261806,d.cGE&cad=rja
>>
>> and
>>
>> http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=2&ved=0CDkQFjAB&url=http%3A%2F%2Fhadoop.apache.org%2Fdocs%2Fmapreduce%2Fr0.21.0%2Fapi%2Fconstant-values.html&ei=oc4bUa-GDom4igK974DwDw&usg=AFQjCNFmbVNRmo70tLRRH-m9iwZUhnWDJQ&sig2=7r4odDgIBIpN92JYDfmjew&bvm=bv.42261806,d.cGE&cad=rja
>>
>>
>> none of which work.
>
>
>
> --
> Harsh J
--
Harsh J
Re: mapreduce.map.env
What version are you specifically asking about?
The MR2 (2.x) does not have this anymore in use (regression? not sure
what the replacement is, will have to figure out), but 1.x has
mapred.map/reduce.child.env which you can use to the same effect.
On Wed, Feb 13, 2013 at 11:05 PM, Saptarshi Guha
wrote:
> Hello,
>
> Is this still valid? Can i set
>
> mapreduce.{map|reduce}.env
>
> When i google i get links
>
> http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0CDIQFjAA&url=http%3A%2F%2Fhadoop.apache.org%2Fdocs%2Fcurrent%2Fhadoop-mapreduce-client%2Fhadoop-mapreduce-client-core%2Fmapred-default.xml&ei=oc4bUa-GDom4igK974DwDw&usg=AFQjCNEnb_-tEiCRsOndsyuGLqtWOJ8OPw&sig2=JVUiA78CsXC7aIdo1X09Lw&bvm=bv.42261806,d.cGE&cad=rja
>
> and
>
> http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=2&ved=0CDkQFjAB&url=http%3A%2F%2Fhadoop.apache.org%2Fdocs%2Fmapreduce%2Fr0.21.0%2Fapi%2Fconstant-values.html&ei=oc4bUa-GDom4igK974DwDw&usg=AFQjCNFmbVNRmo70tLRRH-m9iwZUhnWDJQ&sig2=7r4odDgIBIpN92JYDfmjew&bvm=bv.42261806,d.cGE&cad=rja
>
>
> none of which work.
--
Harsh J
Re: What resources are used by idle NameNode and JobTracker tasks
The NN has a constant memory use and the CPU usage should also be patterned in idle mode. The JT usually retires jobs over the days, and the memory and CPU usage should both go down over time in idle mode, especially if you indicate a week of idleness. There is however the DN block scanner that may trigger itself and begin consuming a bit of an IO in verifying block data integrity (to re-populate rot/etc. corrupted blocks on disks), which would probably be the only usage registered on an idle cluster with no users talking to it. What exact 'resources' was the complaint about? Having more idea on that would help us trace or answer better. On Mon, Feb 11, 2013 at 11:37 PM, Steve Lewis wrote: > I have Hadoop running on 8 nodes in a 50 node cluster. The NameNode and > Jobtrackers have been running for months while jobs were run occasionally - > usually no more than a dozen a week and more typically none. Recently the > cluster manager accused Hadoop of consuming a large number of resources in > the cluster - this occurred during a week an which I was running no jobs. > While I stopped Hadoop the trackers have been running for many months > stably. > Is there any reason why either of these jobs should suddenly and without > jobs being run increase their consumption of resources in a serious way?? > > > -- > Steven M. Lewis PhD > 4221 105th Ave NE > Kirkland, WA 98033 > 206-384-1340 (cell) > Skype lordjoe_com > -- Harsh J
Re: configure mapreduce to work with pem files.
Hi, Nodes in Hadoop do not communicate using SSH. See http://wiki.apache.org/hadoop/FAQ#Does_Hadoop_require_SSH.3F On Wed, Feb 13, 2013 at 5:16 PM, Pedro Sá da Costa wrote: > I'm trying to configure ssh for the Hadoop mapreduce, but my nodes only > communicate with each others using RSA keys in pem format. > > (It doesn't work) > ssh user@host > Permission denied (publickey). > > (It works) > ssh -i ~/key.pem user@host > > The nodes in mapreduce communicate using ssh. How I configure the ssh, or > the mapreduce to work with the pem file. > > > -- > Best regards, > P -- Harsh J
Re: Anyway to load certain Key/Value pair fast?
My reply to your questions is inline. On Wed, Feb 13, 2013 at 10:59 AM, Harsh J wrote: > Please do not use the general@ lists for any user-oriented questions. > Please redirect them to user@hadoop.apache.org lists, which is where > the user community and questions lie. > > I've moved your post there and have added you on CC in case you > haven't subscribed there. Please reply back only to the user@ > addresses. The general@ list is for Apache Hadoop project-level > management and release oriented discussions alone. > > On Wed, Feb 13, 2013 at 10:54 AM, William Kang wrote: >> Hi All, >> I am trying to figure out a good solution for such a scenario as following. >> >> 1. I have a 2T file (let's call it A), filled by key/value pairs, >> which is stored in the HDFS with the default 64M block size. In A, >> each key is less than 1K and each value is about 20M. >> >> 2. Occasionally, I will run analysis by using a different type of data >> (usually less than 10G, and let's call it B) and do look-up table >> alike operations by using the values in A. B resides in HDFS as well. >> >> 3. This analysis would require loading only a small number of values >> from A (usually less than 1000 of them) into the memory for fast >> look-up against the data in B. The way B finds the few values in A is >> by looking up for the key in A. About 1000 such rows would equal a memory expense of near 20 GB, given the value size of A you've noted above. The solution may need to be considered with this in mind, if the whole lookup table is to be eventually generated into the memory and never discarded until the end of processing. >> Is there an efficient way to do this? Since HBase may be too much for your simple needs, have you instead considered using MapFiles, which allow fast key lookups at a file level over HDFS/MR? You can have these files either highly replicated (if their size is large), or distributed via the distributed cache in the lookup jobs (if they are infrequently used and small sized), and be able to use the MapFile reader API to perform lookups of keys and read values only when you want them. >> I was thinking if I could identify the locality of the block that >> contains the few values, I might be able to push the B into the few >> nodes that contains the few values in A? Since I only need to do this >> occasionally, maintaining a distributed database such as HBase cant be >> justified. I agree that HBase may not be wholly suited to be run just for this purpose (unless A's also gonna be scaling over time). Maintaining value -> locality mapping would need to be done by you. FS APIs provide locality info calls, and your files may be key-partitioned enough to identify each one's range, and you can combine the knowledge of these two to do something along these lines. Using HBase may also turn out to be "easier", but thats upto you. You can also choose to tear it down (i.e. the services) when not needed, btw. >> Many thanks. >> >> >> Cao > > > > -- > Harsh J -- Harsh J
Re: Anyway to load certain Key/Value pair fast?
Please do not use the general@ lists for any user-oriented questions. Please redirect them to user@hadoop.apache.org lists, which is where the user community and questions lie. I've moved your post there and have added you on CC in case you haven't subscribed there. Please reply back only to the user@ addresses. The general@ list is for Apache Hadoop project-level management and release oriented discussions alone. On Wed, Feb 13, 2013 at 10:54 AM, William Kang wrote: > Hi All, > I am trying to figure out a good solution for such a scenario as following. > > 1. I have a 2T file (let's call it A), filled by key/value pairs, > which is stored in the HDFS with the default 64M block size. In A, > each key is less than 1K and each value is about 20M. > > 2. Occasionally, I will run analysis by using a different type of data > (usually less than 10G, and let's call it B) and do look-up table > alike operations by using the values in A. B resides in HDFS as well. > > 3. This analysis would require loading only a small number of values > from A (usually less than 1000 of them) into the memory for fast > look-up against the data in B. The way B finds the few values in A is > by looking up for the key in A. > > Is there an efficient way to do this? > > I was thinking if I could identify the locality of the block that > contains the few values, I might be able to push the B into the few > nodes that contains the few values in A? Since I only need to do this > occasionally, maintaining a distributed database such as HBase cant be > justified. > > Many thanks. > > > Cao -- Harsh J
Re: max number of map/reduce per node
Hi,
My reply inline.
On Mon, Feb 11, 2013 at 5:15 PM, Oleg Ruchovets wrote:
> Hi
>I found that my job runs with such parameters:
> mapred.tasktracker.map.tasks.maximum4
> mapred.tasktracker.reduce.tasks.maximum2
>
>I try to change these parameters from my java code
>
> Properties properties = new Properties();
> properties.put("mapred.tasktracker.map.tasks.maximum" , "8");
> properties.put("mapred.tasktracker.reduce.tasks.maximum" , "4");
These properties are a per-tasktracker configuration, not applicable
or read from clients.
Also, if you're tweaking client-end properties, using the Java
Properties class is not the right way to go about it. See
Configuration API:
http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/conf/Configuration.html
> But executing the job I didn't get updated values of these parameters , it
> remains:
>
> mapred.tasktracker.map.tasks.maximum 4
> mapred.tasktracker.reduce.tasks.maximum 2
>
>
> Should I change the parameters on hadoop XML configuration files?
Yes, as these are per *tasktracker* properties, not client ones.
> Please advice.
>
>
>
>
>
>
--
Harsh J
Re: How can I limit reducers to one-per-node?
The suggestion to add a combiner is to help reduce the shuffle load (and perhaps, reduce # of reducers needed?), but it doesn't affect scheduling of a set number of reduce tasks nor does a scheduler care currently if you add that step in or not. On Mon, Feb 11, 2013 at 7:59 AM, David Parks wrote: > I guess the FairScheduler is doing multiple assignments per heartbeat, hence > the behavior of multiple reduce tasks per node even when they should > otherwise be full distributed. > > > > Adding a combiner will change this behavior? Could you explain more? > > > > Thanks! > > David > > > > > > From: Michael Segel [mailto:michael_se...@hotmail.com] > Sent: Monday, February 11, 2013 8:30 AM > > > To: user@hadoop.apache.org > Subject: Re: How can I limit reducers to one-per-node? > > > > Adding a combiner step first then reduce? > > > > > > On Feb 8, 2013, at 11:18 PM, Harsh J wrote: > > > > Hey David, > > There's no readily available way to do this today (you may be > interested in MAPREDUCE-199 though) but if your Job scheduler's not > doing multiple-assignments on reduce tasks, then only one is assigned > per TT heartbeat, which gives you almost what you're looking for: 1 > reduce task per node, round-robin'd (roughly). > > On Sat, Feb 9, 2013 at 9:24 AM, David Parks wrote: > > I have a cluster of boxes with 3 reducers per node. I want to limit a > particular job to only run 1 reducer per node. > > > > This job is network IO bound, gathering images from a set of webservers. > > > > My job has certain parameters set to meet “web politeness” standards (e.g. > limit connects and connection frequency). > > > > If this job runs from multiple reducers on the same node, those per-host > limits will be violated. Also, this is a shared environment and I don’t > want long running network bound jobs uselessly taking up all reduce slots. > > > > > -- > Harsh J > > > > Michael Segel | (m) 312.755.9623 > > Segel and Associates > > -- Harsh J
Re: HOW TO WORK WITH HADOOP SOURCE CODE
Hi, I have some notes at http://wiki.apache.org/hadoop/QwertyManiac/BuildingHadoopTrunk that may be of help to you. You'll need branch-1 or a release tag (see all tags here http://svn.apache.org/repos/asf/hadoop/common/tags/) checked out for 1.x based sources. Alternatively, if you use a Maven project, you can ask it to download sources of dependencies too, which lets you browse sources as you develop as well. Also, do read http://www.hoax-slayer.com/do-not-use-all-capitals.html :) On Mon, Feb 11, 2013 at 9:07 AM, Dibyendu Karmakar wrote: > Hi, > I am trying to view HADOOP SOURCE CODE. I am using HADOOP 1.0.3. > In HADOOP distribution, only jar files are there. > Give me some instruction to view source code... please > > I have seen "contribute to hadoop" page ( > wiki.apache.org/hadoop/HowToContribute ) where something about git is > written. I am not getting what it is. And while trying to download, I have > found many links with different names ( git.apache.org ). Don't know what to > do. > PLEASE HELP. > THANK YOU. -- Harsh J
Re: How can I limit reducers to one-per-node?
Hey David, There's no readily available way to do this today (you may be interested in MAPREDUCE-199 though) but if your Job scheduler's not doing multiple-assignments on reduce tasks, then only one is assigned per TT heartbeat, which gives you almost what you're looking for: 1 reduce task per node, round-robin'd (roughly). On Sat, Feb 9, 2013 at 9:24 AM, David Parks wrote: > I have a cluster of boxes with 3 reducers per node. I want to limit a > particular job to only run 1 reducer per node. > > > > This job is network IO bound, gathering images from a set of webservers. > > > > My job has certain parameters set to meet “web politeness” standards (e.g. > limit connects and connection frequency). > > > > If this job runs from multiple reducers on the same node, those per-host > limits will be violated. Also, this is a shared environment and I don’t > want long running network bound jobs uselessly taking up all reduce slots. -- Harsh J
Re: How MapReduce selects data blocks for processing user request
Hi Mehal, > I am confused over how MapReduce tasks select data blocks for processing user > requests ? I suggest reading chapter 6 of Tom White's Hadoop: The Definitive Guide, titled "How MapReduce Works". It explains almost everything you need to know in very clear language, and should help you generally if you get this or other such good books. > As data block replication replicates single data block over multiple > datanodes, during job processing how uniquely data blocks are selected for > processing user requests ? The first point to clear up is that MapReduce is not hard-tied to HDFS. It generates splits on any FS and the splits are unique, based on your given input path. Each split therefore relates to one task and the task's input goal is hence defined at submit-time itself. Each split is further defined by its path, start offset into the file and length after offset to be processed - "uniquely" defining itself. > How does it guarantees that no same block gets chosen twice or thrice for > different mapper task. See above - each "block" (or a "split" in MR terms), is defined by its start-offset and length. No two splits generated for a single file would be the same, as we generate it that way - and hence there won't be such a case you're worried about. On Sat, Feb 9, 2013 at 6:10 AM, Mehal Patel wrote: > Hello All, > > I am confused over how MapReduce tasks select data blocks for processing > user requests ? > > As data block replication replicates single data block over multiple > datanodes, during job processing how uniquely > data blocks are selected for processing user requests ? How does it > guarantees that no same block gets chosen twice or thrice > for different mapper task. > > > Thank you > > -Mehal -- Harsh J
Re: How to installl FUSE with Hadoop 1.0.x?
I recall having mentioned it before, but why not use the Apache Bigtop RPM/DEB build scripts to generate all these binaries in good package form for yourself? It also builds up the fuse extensions and has clearer dependencies/build-process. Check out their 0.3.x branches for 1.x specific build scripts via http://bigtop.apache.org Otherwise, the path you're already following (i.e. ant native builds) is the correct one and you should be able to get a successful result if you have all the necessary dependencies. On Fri, Feb 8, 2013 at 10:00 PM, Jean-Marc Spaggiari wrote: > Hi, > > I'm wondering what's the best way to install FUSE with Hadoop 1.0.3? > > I'm trying to follow all the steps described here: > http://wiki.apache.org/hadoop/MountableHDFS but it's failing on each > one, taking hours to fix it and move to the next one. > > So I think I'm following the wrong path. There should be an easier > solution to install that. > > I tried the fuse-j-hadoopfs option too, but the JAR is no more > available when we try to build. > > Does anyone have already setup FUSE with Hadoop 1.0.x? > > Thanks, > > JM -- Harsh J
Re: Secondary Sort example error
The JIRA https://issues.apache.org/jira/browse/MAPREDUCE-2584 should help such cases, if what I speculated above is indeed the case. On Fri, Feb 8, 2013 at 12:16 AM, Harsh J wrote: > Thanks, I managed to correlate proper line numbers. > > Are you using some form of custom serialization in your job code? That > is, are your keys non-Writable types and are of some other type? The > specific NPE is arising from the SerializationFactory not being able > to find a serializer for your Map-Output key class. You may want to > look into that direction, or share your code for the list to spot it > instead. > > On Fri, Feb 8, 2013 at 12:11 AM, Ravi Chandran > wrote: >> hi, >> >> it is Hadoop 2.0.0-cdh4.1.1. the whole output is given below: >> >> Hadoop 2.0.0-cdh4.1.1 >> Subversion >> file:///data/1/jenkins/workspace/generic-package-centos32-6/topdir/BUILD/hadoop-2.0.0-cdh4.1.1/src/hadoop-common-project/hadoop-common >> -r 581959ba23e4af85afd8db98b7687662fe9c5f20 >> >> >> >> On Fri, Feb 8, 2013 at 12:04 AM, Harsh J wrote: >>> >>> Hey Ravi, >>> >>> What version of Hadoop is this exactly? (Type and send output of >>> "hadoop version" if unsure) >>> >>> On Thu, Feb 7, 2013 at 11:55 PM, Ravi Chandran >>> wrote: >>> > Hi, >>> > >>> > I am trying to do a name sorting using secondary sort. I have a working >>> > example, which I am taking as a reference. But I am getting a null >>> > pointer >>> > error in the MapTask class. I am not able to locate the reason. as the >>> > logic >>> > to create the Custom Object from a given file has been tested through a >>> > java >>> > class.. >>> > I am getting this error: >>> > >>> > 13/02/07 12:23:42 WARN snappy.LoadSnappy: Snappy native library is >>> > available >>> > 13/02/07 12:23:42 INFO snappy.LoadSnappy: Snappy native library loaded >>> > 13/02/07 12:23:42 INFO mapred.FileInputFormat: Total input paths to >>> > process >>> > : 1 >>> > 13/02/07 12:23:43 INFO mapred.JobClient: Running job: >>> > job_201301301056_0014 >>> > 13/02/07 12:23:44 INFO mapred.JobClient: map 0% reduce 0% >>> > 13/02/07 12:23:56 INFO mapred.JobClient: Task Id : >>> > attempt_201301301056_0014_m_00_0, Status : FAILED >>> > java.lang.NullPointerException >>> > at >>> > >>> > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.(MapTask.java:814) >>> > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:385) >>> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:327) >>> > at org.apache.hadoop.mapred.Child$4.run(Child.java:268) >>> > at java.security.AccessController.doPrivileged(Native Method) >>> > at javax.security.auth.Subject.doAs(Subject.java:396) >>> > at >>> > >>> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332) >>> > at org.apache.hadoop.mapred.Child.main(Child.java:262) >>> > 13/02/07 12:23:57 INFO mapred.JobClient: Task Id : >>> > attempt_201301301056_0014_m_01_0, Status : FAILED >>> > java.lang.NullPointerException >>> > at >>> > >>> > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.(MapTask.java:814) >>> > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:385) >>> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:327) >>> > at org.apache.hadoop.mapred.Child$4.run(Child.java:268) >>> > at java.security.AccessController.doPrivileged(Native Method) >>> > at javax.security.auth.Subject.doAs(Subject.java:396) >>> > at >>> > >>> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332) >>> > at org.apache.hadoop.mapred.Child.main(Child.java:262) >>> > >>> > I am giving the Mapper code below: >>> > >>> > import java.io.IOException; >>> > import org.apache.hadoop.io.LongWritable; >>> > import org.apache.hadoop.io.Text; >>> > import org.apache.hadoop.mapred.MapReduceBase; >>> > import org.apache.hadoop.mapred.Mapper; >>> > import org.apache.hadoop.mapred.OutputCollector; >>> > import org.apache.hadoop.mapred.Reporter; >>> > import org.apache.log4j.Logger; >>> > import com.pom.Name; >>> > >>> > public class StubMapper extend
Re: Secondary Sort example error
Thanks, I managed to correlate proper line numbers.
Are you using some form of custom serialization in your job code? That
is, are your keys non-Writable types and are of some other type? The
specific NPE is arising from the SerializationFactory not being able
to find a serializer for your Map-Output key class. You may want to
look into that direction, or share your code for the list to spot it
instead.
On Fri, Feb 8, 2013 at 12:11 AM, Ravi Chandran
wrote:
> hi,
>
> it is Hadoop 2.0.0-cdh4.1.1. the whole output is given below:
>
> Hadoop 2.0.0-cdh4.1.1
> Subversion
> file:///data/1/jenkins/workspace/generic-package-centos32-6/topdir/BUILD/hadoop-2.0.0-cdh4.1.1/src/hadoop-common-project/hadoop-common
> -r 581959ba23e4af85afd8db98b7687662fe9c5f20
>
>
>
> On Fri, Feb 8, 2013 at 12:04 AM, Harsh J wrote:
>>
>> Hey Ravi,
>>
>> What version of Hadoop is this exactly? (Type and send output of
>> "hadoop version" if unsure)
>>
>> On Thu, Feb 7, 2013 at 11:55 PM, Ravi Chandran
>> wrote:
>> > Hi,
>> >
>> > I am trying to do a name sorting using secondary sort. I have a working
>> > example, which I am taking as a reference. But I am getting a null
>> > pointer
>> > error in the MapTask class. I am not able to locate the reason. as the
>> > logic
>> > to create the Custom Object from a given file has been tested through a
>> > java
>> > class..
>> > I am getting this error:
>> >
>> > 13/02/07 12:23:42 WARN snappy.LoadSnappy: Snappy native library is
>> > available
>> > 13/02/07 12:23:42 INFO snappy.LoadSnappy: Snappy native library loaded
>> > 13/02/07 12:23:42 INFO mapred.FileInputFormat: Total input paths to
>> > process
>> > : 1
>> > 13/02/07 12:23:43 INFO mapred.JobClient: Running job:
>> > job_201301301056_0014
>> > 13/02/07 12:23:44 INFO mapred.JobClient: map 0% reduce 0%
>> > 13/02/07 12:23:56 INFO mapred.JobClient: Task Id :
>> > attempt_201301301056_0014_m_00_0, Status : FAILED
>> > java.lang.NullPointerException
>> > at
>> >
>> > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.(MapTask.java:814)
>> > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:385)
>> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:327)
>> > at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
>> > at java.security.AccessController.doPrivileged(Native Method)
>> > at javax.security.auth.Subject.doAs(Subject.java:396)
>> > at
>> >
>> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
>> > at org.apache.hadoop.mapred.Child.main(Child.java:262)
>> > 13/02/07 12:23:57 INFO mapred.JobClient: Task Id :
>> > attempt_201301301056_0014_m_01_0, Status : FAILED
>> > java.lang.NullPointerException
>> > at
>> >
>> > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.(MapTask.java:814)
>> > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:385)
>> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:327)
>> > at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
>> > at java.security.AccessController.doPrivileged(Native Method)
>> > at javax.security.auth.Subject.doAs(Subject.java:396)
>> > at
>> >
>> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
>> > at org.apache.hadoop.mapred.Child.main(Child.java:262)
>> >
>> > I am giving the Mapper code below:
>> >
>> > import java.io.IOException;
>> > import org.apache.hadoop.io.LongWritable;
>> > import org.apache.hadoop.io.Text;
>> > import org.apache.hadoop.mapred.MapReduceBase;
>> > import org.apache.hadoop.mapred.Mapper;
>> > import org.apache.hadoop.mapred.OutputCollector;
>> > import org.apache.hadoop.mapred.Reporter;
>> > import org.apache.log4j.Logger;
>> > import com.pom.Name;
>> >
>> > public class StubMapper extends MapReduceBase implements
>> > Mapper {
>> >
>> > private static Logger logger =
>> > Logger.getLogger(StubMapper.class.getName());
>> >
>> > StringBuffer readLine = new StringBuffer();
>> > private final Name name = new Name();
>> > @Override
>> > public void map(LongWritable key, Text value,
>> > OutputCollector output, Reporter reporter)
>> > throws IOException {
>> > String line = value.toString();
>> > String[] pac
Re: Secondary Sort example error
Hey Ravi,
What version of Hadoop is this exactly? (Type and send output of
"hadoop version" if unsure)
On Thu, Feb 7, 2013 at 11:55 PM, Ravi Chandran
wrote:
> Hi,
>
> I am trying to do a name sorting using secondary sort. I have a working
> example, which I am taking as a reference. But I am getting a null pointer
> error in the MapTask class. I am not able to locate the reason. as the logic
> to create the Custom Object from a given file has been tested through a java
> class..
> I am getting this error:
>
> 13/02/07 12:23:42 WARN snappy.LoadSnappy: Snappy native library is available
> 13/02/07 12:23:42 INFO snappy.LoadSnappy: Snappy native library loaded
> 13/02/07 12:23:42 INFO mapred.FileInputFormat: Total input paths to process
> : 1
> 13/02/07 12:23:43 INFO mapred.JobClient: Running job: job_201301301056_0014
> 13/02/07 12:23:44 INFO mapred.JobClient: map 0% reduce 0%
> 13/02/07 12:23:56 INFO mapred.JobClient: Task Id :
> attempt_201301301056_0014_m_00_0, Status : FAILED
> java.lang.NullPointerException
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.(MapTask.java:814)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:385)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:327)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
> at org.apache.hadoop.mapred.Child.main(Child.java:262)
> 13/02/07 12:23:57 INFO mapred.JobClient: Task Id :
> attempt_201301301056_0014_m_01_0, Status : FAILED
> java.lang.NullPointerException
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.(MapTask.java:814)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:385)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:327)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
> at org.apache.hadoop.mapred.Child.main(Child.java:262)
>
> I am giving the Mapper code below:
>
> import java.io.IOException;
> import org.apache.hadoop.io.LongWritable;
> import org.apache.hadoop.io.Text;
> import org.apache.hadoop.mapred.MapReduceBase;
> import org.apache.hadoop.mapred.Mapper;
> import org.apache.hadoop.mapred.OutputCollector;
> import org.apache.hadoop.mapred.Reporter;
> import org.apache.log4j.Logger;
> import com.pom.Name;
>
> public class StubMapper extends MapReduceBase implements
> Mapper {
>
> private static Logger logger =
> Logger.getLogger(StubMapper.class.getName());
>
> StringBuffer readLine = new StringBuffer();
> private final Name name = new Name();
> @Override
> public void map(LongWritable key, Text value,
> OutputCollector output, Reporter reporter)
> throws IOException {
> String line = value.toString();
> String[] packer = null;
>
> packer = line.split(" ");
>
> // create the object
> if(packer.length>2)
> {
> // take everything except last name
> for (int i = 0; i < packer.length-1; i++) {
>readLine.append(packer[i]+" ");
> }
>
> name.setfName(readLine.toString());
> name.setlName(packer[packer.length-1]);
>
> //clear the variable
> readLine.delete(0, readLine.length());
> }
> else if(packer.length>0)
> {
> name.setfName(packer[0]);
> name.setlName(packer[1]);
>}
>
> output.collect(name, new Text(name.getlName()));
>
> }
> }
>
> I am not able to figure out the possible cause..
>
> --
> Thanks & Regards
> Ravi
--
Harsh J
Re: why does OldCombinerRunner pass Reporter.NULL to the combiner instead of the real reporter?
I agree its a bug if there is a discrepancy between the APIs (we are supposed to be supporting both for the time being). Please do file a JIRA with a patch - there shouldn't be any harm in re-passing the reporter object within the combiner. On Thu, Feb 7, 2013 at 7:10 PM, Jim Donofrio wrote: > Is there a good reason why the OldCombinerRunner passes Reporter.NULL to the > combiner instead of the actual TaskReporter? The NewCombinerRunner does use > the TaskReporter when creating the context. If this is a bug I will submit a > JIRA with a patch -- Harsh J
Re: Creating files through the hadoop streaming interface
The raw streaming interface has much issues of this manner. The python open(…, 'w') calls won't open files on HDFS, further. Perhaps, since you wish to use Python for its various advantages, check out this detailed comparison guide of various Python-based Hadoop frameworks (including the raw streaming we offer as part of Apache Hadoop) at http://blog.cloudera.com/blog/2013/01/a-guide-to-python-frameworks-for-hadoop/ by Uri? Many of these provide python extensions to HDFS/etc., letting you do much more than plain streaming. On Thu, Feb 7, 2013 at 6:43 AM, Julian Bui wrote: > Hi hadoop users, > > I am trying to use the streaming interface to use my python script mapper to > create some files but am running into difficulties actually creating files > on the hdfs. > > I have a python script mapper with no reducers. Currently, it doesn't even > read the input and instead reads in the env variable for the output dir > (outdir = os.environ['mapred_output_dir']) and attempts to create an empty > file at that location. However, that appears to fail with the [vague] error > message appended to this email. > > I am using the streaming interface because the python file examples seem so > much cleaner and abstract a lot of the details away for me but if I instead > need to use the java bindings (and create a mapper and reducer class) then > please let me know. I'm still learning hadoop. As I understand it, I > should be able to create files in hadoop but perhaps there is limited > ability while using the streaming i/o interface. > > Further questions: If my mapper absolutely must send my output to stdout, is > there a way to rename the file after it has been created? > > Please help. > > Thanks, > -Julian > > Python mapper code: > outdir = os.environ['mapred_output_dir'] > f = open(outdir + "/testfile.txt", "wb") > f.close() > > > 13/02/06 17:07:55 INFO streaming.StreamJob: map 100% reduce 100% > 13/02/06 17:07:55 INFO streaming.StreamJob: To kill this job, run: > 13/02/06 17:07:55 INFO streaming.StreamJob: > /opt/hadoop/libexec/../bin/hadoop job > -Dmapred.job.tracker=gcn-13-88.ibnet0:54311 -kill job_201302061706_0001 > 13/02/06 17:07:55 INFO streaming.StreamJob: Tracking URL: > http://gcn-13-88.ibnet0:50030/jobdetails.jsp?jobid=job_201302061706_0001 > 13/02/06 17:07:55 ERROR streaming.StreamJob: Job not successful. Error: # of > failed Map Tasks exceeded allowed limit. FailedCount: 1. LastFailedTask: > task_201302061706_0001_m_00 > 13/02/06 17:07:55 INFO streaming.StreamJob: killJob... > Streaming Command Failed! > -- Harsh J
Re: How to find HDFS folders file count.
You have posted the "fs -count" command which is also the answer to your question - does it not help? On Wed, Feb 6, 2013 at 3:56 PM, Dhanasekaran Anbalagan wrote: > Hi Guys, > > We have done moving local file to HDFS > hadoop fs -copyFromLocal > > we have verified some of the file missing in the HDFS, We want validate > source to destination. > > We have already have source files count > > How to find HDFS folder file count please guide me. > > hadoop.apache.org/docs/r0.20.2/hdfs_shell.html#count > hadoop fs -count -q > > Did I learn something today? If not, I wasted it. -- Harsh J
Re: Specific HDFS tasks where is passwordless SSH is necessary
It isn't the NN that does the SSH technically, its the scripts we ship for an easier start/stop: http://wiki.apache.org/hadoop/FAQ#Does_Hadoop_require_SSH.3F So wherever you launch the script, the SSH may happen from that point. On Wed, Feb 6, 2013 at 4:36 AM, Jay Vyas wrote: > When setting up passwordless ssh on a cluster, its clear that the namenode > needs to be able to ssh into task trackers to start/stop nodes and restart > the cluster. > > What else is passwordless SSH used for? Do TaskTrackers/DataNodes ever SSH > into each other horizontally ? Or is SSH only used for one-way nn to tt > operations? > > -- > Jay Vyas > http://jayunit100.blogspot.com -- Harsh J
Re: how to set default replication factor for specific directories
Replication factor is a client-sent property for files (doesn't apply to 'directories'), so there's currently no way to do this outside of controlling the client/client-config of those that write to that directory (for example, HBase can have a separate replication config, for all writes to /hbase), or by periodically setting the factor recursively on the directory (as your sample command does). On Mon, Feb 4, 2013 at 12:14 PM, Austin Chungath wrote: > Hi, > > Is there any way for setting the default replication factor for specific > directories? > The default replication factor is 3 for the cluster. I don't want to change > this global default, but I want specific directories to have different > replication factor. > I can use the following command to set the replication factor of files after > they are created, but how to make it default for that directory? > hadoop dfs -setrep -R -w 1 /user > > Thanks, > Austin -- Harsh J
Re: Apache Development Snapshot Repository does not work
Are you trying to build a specific submodule here, i.e, under what checked out directory are you running the "mvn install" under? I'd recommend running it under the root of the checkout (the most parent pom.xml) first before trying to build a specific sub-module. On Fri, Feb 1, 2013 at 10:23 PM, YouPeng Yang wrote: > Hi >I have got the latest source from Git. > when I perform mvn install -DskipTests. > it was stuck when it needed to download some dependences from > https://repository.apache.org/content/repositories/snapshots. > > does it need to set the Maven proxy setttings .I have setup that acordding > to the maven guide,it does not work.Maybe the proxy site was change. > could anybody give me available proxy settings. > > > regards. > YouPeng Yang -- Harsh J
Re: How to find Blacklisted Nodes via cli.
There isn't a concept of 'blacklist' in HDFS, it exists only in MapReduce (MR1) and YARN. Dead Nodes (and those that have been excluded) are reported via "hdfs dfsadmin -report" as indicated earlier. On Thu, Jan 31, 2013 at 7:28 PM, Dhanasekaran Anbalagan wrote: > Hi Hemanth, > > Thanks for spending your valuable time for me. > > This mapred job -list-blacklisted-trackers for find jobtracker level but I > want hdfs level I didn't find any command. [HDFS Blacklisted nodes] Please > guide me. And also to clear blacklisted nodes via command line > > -Dhanasekaran. > > Did I learn something today? If not, I wasted it. > > > On Wed, Jan 30, 2013 at 11:25 PM, Hemanth Yamijala > wrote: >> >> Hi, >> >> Part answer: you can get the blacklisted tasktrackers using the command >> line: >> >> mapred job -list-blacklisted-trackers. >> >> Also, I think that a blacklisted tasktracker becomes 'unblacklisted' if it >> works fine after some time. Though I am not very sure about this. >> >> Thanks >> hemanth >> >> >> On Wed, Jan 30, 2013 at 9:35 PM, Dhanasekaran Anbalagan >> wrote: >>> >>> Hi Guys, >>> >>> How to find Blacklisted Nodes via, command line. I want to see job >>> Tracker Blacklisted Nodes and hdfs Blacklisted Nodes. >>> >>> and also how to clear blacklisted nodes to clear start. The only option >>> to restart the service. some other way clear the Blacklisted Nodes. >>> >>> please guide me. >>> >>> -Dhanasekaran. >>> >>> Did I learn something today? If not, I wasted it. >> >> > -- Harsh J
Re: Data migration from one cluster to other running diff. versions
DistCp is the fastest option, letting you copy data in parallel. For incompatible RPC versions between different HDFS clusters, the HFTP solution can work (documented on the DistCp manual). On Wed, Jan 30, 2013 at 10:13 PM, Siddharth Tiwari wrote: > Hi Team, > > What is the best way to migrate data residing on one cluster to another > cluster ? > Are there better methods available than distcp ? > What if both the clusters running different RPC protocol versions ? > > > ** > Cheers !!! > Siddharth Tiwari > Have a refreshing day !!! > "Every duty is holy, and devotion to duty is the highest form of worship of > God.” > "Maybe other people will try to limit me but I don't limit myself" -- Harsh J
Re: what will happen when HDFS restarts but with some dead nodes
Yes, if there are missing blocks (i.e. all replicas lost), and the
block availability threshold is set to its default of 0.999f (99.9%
availability required), then NN will not come out of safemode
automatically. You can control this behavior by configuring
dfs.namenode.safemode.threshold.
On Wed, Jan 30, 2013 at 10:06 PM, Chen He wrote:
> Hi Harsh
>
> I have a question. How namenode gets out of safemode in condition of data
> blocks lost, only administrator? Accordin to my experiences, the NN (0.21)
> stayed in safemode about several days before I manually turn safemode off.
> There were 2 blocks lost.
>
> Chen
>
>
> On Wed, Jan 30, 2013 at 10:27 AM, Harsh J wrote:
>>
>> NN does recalculate new replication work to do due to unavailable
>> replicas ("under-replication") when it starts and receives all block
>> reports, but executes this only after out of safemode. When in
>> safemode, across the HDFS services, no mutations are allowed.
>>
>> On Wed, Jan 30, 2013 at 8:34 AM, Nan Zhu wrote:
>> > Hi, all
>> >
>> > I'm wondering if HDFS is stopped, and some of the machines of the
>> > cluster
>> > are moved, some of the block replication are definitely lost for moving
>> > machines
>> >
>> > when I restart the system, will the namenode recalculate the data
>> > distribution?
>> >
>> > Best,
>> >
>> > --
>> > Nan Zhu
>> > School of Computer Science,
>> > McGill University
>> >
>> >
>>
>>
>>
>> --
>> Harsh J
>
>
--
Harsh J
Re: what will happen when HDFS restarts but with some dead nodes
NN does recalculate new replication work to do due to unavailable
replicas ("under-replication") when it starts and receives all block
reports, but executes this only after out of safemode. When in
safemode, across the HDFS services, no mutations are allowed.
On Wed, Jan 30, 2013 at 8:34 AM, Nan Zhu wrote:
> Hi, all
>
> I'm wondering if HDFS is stopped, and some of the machines of the cluster
> are moved, some of the block replication are definitely lost for moving
> machines
>
> when I restart the system, will the namenode recalculate the data
> distribution?
>
> Best,
>
> --
> Nan Zhu
> School of Computer Science,
> McGill University
>
>
--
Harsh J
Re: Oozie workflow error - renewing token issue
) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > > at java.lang.reflect.Method.invoke(Method.java:597) > > at > org.apache.hadoop.ipc.WritableRpcEngine$Server$WritableRpcInvoker.call(WritableRpcEngine.java:474) > > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898) > > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693) > > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689) > > at java.security.AccessController.doPrivileged(Native Method) > > at javax.security.auth.Subject.doAs(Subject.java:396) > > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332) > > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687) > > > > 2013-01-29 17:11:29,212 WARN org.apache.hadoop.security.token.Token: Cannot > find class for token kind MAPREDUCE_DELEGATION_TOKEN > > 2013-01-29 17:11:29,212 ERROR > org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal: Exception > renewing tokenKind: MAPREDUCE_DELEGATION_TOKEN, Service: 10.204.12.62:8021, > Ident: 00 04 68 64 66 73 08 6d 72 20 74 6f 6b 65 6e 05 6f 6f 7a 69 65 8a 01 > 3c 88 94 58 67 8a 01 3c ac a0 dc 67 4c 08. Not rescheduled > > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): > Client mapred tries to renew a token with renewer specified as mr token > > at > org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.renewToken(AbstractDelegationTokenSecretManager.java:274) > > at > org.apache.hadoop.mapred.JobTracker.renewDelegationToken(JobTracker.java:3738) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > > at java.lang.reflect.Method.invoke(Method.java:597) > > at > org.apache.hadoop.ipc.WritableRpcEngine$Server$WritableRpcInvoker.call(WritableRpcEngine.java:474) > > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898) > > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693) > > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689) > > at java.security.AccessController.doPrivileged(Native Method) > > at javax.security.auth.Subject.doAs(Subject.java:396) > > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332) > > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687) > > > > at org.apache.hadoop.ipc.Client.call(Client.java:1160) > > at > org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:225) > > at org.apache.hadoop.mapred.$Proxy12.renewDelegationToken(Unknown > Source) > > at > org.apache.hadoop.mapred.JobClient$Renewer.renew(JobClient.java:541) > > at org.apache.hadoop.security.token.Token.renew(Token.java:372) > > at > org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$RenewalTimerTask$1.run(DelegationTokenRenewal.java:218) > > at > org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$RenewalTimerTask$1.run(DelegationTokenRenewal.java:214) > > at java.security.AccessController.doPrivileged(Native Method) > > at javax.security.auth.Subject.doAs(Subject.java:396) > > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332) > > at > org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$RenewalTimerTask.run(DelegationTokenRenewal.java:213) > > at java.util.TimerThread.mainLoop(Timer.java:512) > > at java.util.TimerThread.run(Timer.java:462) > > > > ~Corbett Martin > > Software Architect > > AbsoluteAR Accounts Receivable Services - An NHIN Solution > > > > > > This message and its contents (to include attachments) are the property of > National Health Systems, Inc. and may contain confidential and proprietary > information. This email and any files transmitted with it are intended > solely for the use of the individual or entity to whom they are addressed. > You are hereby notified that any unauthorized disclosure, copying, or > distribution of this message, or the taking of any unauthorized action based > on information contained herein is strictly prohibited. Unauthorized use of > information contained herein may subject you to civil and criminal > prosecution and penalties. If you are not the intended recipient, you should > delete this message immediately and notify the sender immediately by > telephone or by replying to this transmission. -- Harsh J
Re: Tricks to upgrading Sequence Files?
This is a pretty interesting question, but unfortunately there isn't an inbuilt way in SequenceFiles itself to handle this. However, your key/value classes can be made to handle versioning perhaps - detecting if what they've read is of an older time and decoding it appropriately (while handling newer encoding separately, in the normal fashion). This would be much better than going down the classloader hack paths I think? On Tue, Jan 29, 2013 at 1:11 PM, David Parks wrote: > Anyone have any good tricks for upgrading a sequence file. > > > > We maintain a sequence file like a flat file DB and the primary object in > there changed in recent development. > > > > It’s trivial to write a job to read in the sequence file, update the object, > and write it back out in the new format. > > > > But since sequence files read and write the key/value class I would either > need to rename the model object with a version number, or change the header > of each sequence file. > > > > Just wondering if there are any nice tricks to this. -- Harsh J
Re: Multiple reduce task retries running at same time
Hi Ben, Take a look at the Speculative Execution feature of MR which should answer your question. See its section under 'Fault Tolerance' here: http://developer.yahoo.com/hadoop/tutorial/module4.html#tolerence. On Tue, Jan 29, 2013 at 1:08 PM, Ben Kim wrote: > Attached a screenshot showing the retries > > > On Tue, Jan 29, 2013 at 4:35 PM, Ben Kim wrote: >> >> Hi! >> >> I have come across the situation where i found a single reducer task >> executing with multiple retries simultaneously. >> Which is potent for slowing down the whole reduce process for large data >> sets. >> >> Is this pretty normal to yall for hadoop 1.0.3? >> >> -- >> >> Benjamin Kim >> benkimkimben at gmail > > > > > -- > > Benjamin Kim > benkimkimben at gmail -- Harsh J
Re: How to learn Mapreduce
I'd recommend discovering a need first - attempt to solve a problem and then write a program around it. Hadoop is data-driven, so its rather hard if you try to learn it without working with any data. A book may interest you as well: http://wiki.apache.org/hadoop/Books On Tue, Jan 29, 2013 at 10:17 AM, abdul wajeed wrote: > Hi Sir/Mam, > I am very new to Hadoop Technology so any one can help me > how to write simple MapReduce program except wordcount program > I would like to write our own program in mapreduce ...As well as in pig > Latin > so please any one can help me ... > >Thanks > > Regards > Abdul Wajeed -- Harsh J
Re: number of mapper tasks
Hi again, (Inline) On Mon, Jan 28, 2013 at 10:01 PM, Marcelo Elias Del Valle wrote: > Hello Harsh, > > First of all, thanks for the answer! > > > 2013/1/28 Harsh J >> >> So depending on your implementation of the job here, you may or may >> not see it act in effect. Hope this helps. > > > Is there anything I can do in my job, my code or in my inputFormat so that > hadoop would choose to run more mappers? My text file and 10 million lines > and each mapper task process 1 line at a time, very fastly. I would like to > have 40 threads in parallel or even more processing those lines. This seems CPU-oriented. You probably want the NLineInputFormat? See http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/lib/NLineInputFormat.html. This should let you spawn more maps as we, based on your N factor. >> >> > When I run my job with just 1 instance, I see it only creates 1 >> > mapper. >> > When I run my job with 5 instances (1 master and 4 cores), I can see >> > only 2 >> > mapper slots are used and 6 stay open. >> >> Perhaps the job itself launched with 2 total map tasks? You can check >> this on the JobTracker UI or whatever EMR offers as a job viewer. > > > I am trying to figure this out. Here is what I have from EMR: > http://mvalle.com/downloads/hadoop_monitor.png > I will try to get their support to understand this, but I didn't understand > what you said about the job being launched with 2 total map tasks... if I > have 8 slots, shouldn't all of them be filled always? Not really - "Slots" are capacities, rather than split factors themselves. You can have N slots always available, but your job has to supply as many map tasks (based on its input/needs/etc.) to use them up. >> >> >> This is a typical waiting reduce task log, what are you asking here >> specifically? > > > I have no reduce tasks. My map does the job without putting anything in the > output. Is it happening because reduce tasks receive nothing as input? Unless your job sets the number of reducers to 0 manually, 1 default reducer is always run that waits to see if it has any outputs from maps. If it does not receive any outputs after maps have all completed, it dies out with behavior equivalent to a NOP. Hope this helps! -- Harsh J
Re: reg max map task config
There's probably different variables (esp. since you run HBase, calculated resources) involved for this, but you can generally find some reading material on this in books and http://wiki.apache.org/hadoop/HowManyMapsAndReduces which should help you come to a good value. Pro-tip I usually get from Todd: Consistently monitor and tune, for it is harmless to restart a slave to set in new values. On Mon, Jan 28, 2013 at 6:04 PM, Jean-Marc Spaggiari wrote: > Hi Harsh, > > Is there a recommanded way to configure this value? I think I read > something like 0.7 x cores but I'm not 100% sure... > > JM > > 2013/1/28, Manoj Babu : >> Thank you Harsh. >> >> Cheers! >> Manoj. >> >> >> On Mon, Jan 28, 2013 at 10:50 AM, Harsh J wrote: >> >>> Hi, >>> >>> As noted on >>> http://hadoop.apache.org/docs/stable/cluster_setup.html#Configuring+the+Hadoop+Daemons >>> , >>> and by its own name, the config 'mapred.tasktracker.map.tasks.maximum' >>> applies to a TaskTracker daemon and therefore needs to be applied to >>> its local mapred-site.xml and has to be restarted to take new values >>> into effect. >>> >>> On Mon, Jan 28, 2013 at 10:36 AM, Manoj Babu wrote: >>> > Hi All, >>> > >>> > I am trying to override the value of >>> mapred.tasktracker.map.tasks.maximum in >>> > mapred-site.xml through my job configuration object will it reflect in >>> the >>> > corresponding task trackers? >>> > or do i have to change it in mapred-site.xml following a restart for >>> > the >>> > changes to apply? >>> > >>> > Thanks in advance. >>> > >>> > Cheers! >>> > Manoj. >>> >>> >>> >>> -- >>> Harsh J >>> >> -- Harsh J
Re: warnings from log4j -- where to look to resolve?
Hi, Does your code instantiate and work with a self logger object? If so, is it a log4j instance or a commons logging one? On Mon, Jan 28, 2013 at 4:06 PM, Calvin wrote: > Hi, > > Each time I run a job, I get thousands of these warnings. I've looked at > conf/log4j.properties and nothing out of the ordinary (a diff with the > default log4j.properties shows no lines of consequence changed). > > Has anyone else come across this problem? Where else would I look to find > potentially conflicting settings for log4j? > > Thanks. > > --- > > attempt_201301161411_0244_m_54_0: log4j:WARN Not allowed to write to a > closed appender. > attempt_201301161411_0244_m_54_0: log4j:WARN Not allowed to write to a > closed appender. > attempt_201301161411_0244_m_54_0: log4j:WARN Not allowed to write to a > closed appender. -- Harsh J
Re: hadoop imbalanced topology
Hi Jameson,
Inline reply below for a specific point of yours ("scheduler deadlock"):
On Mon, Jan 28, 2013 at 8:59 PM, Jameson Li wrote:
> Hi,
>
> Is it right that hadoop Network Topology design treat an imbalanced topology
> as an invalid topology?
>
> This is mentioned here
> https://issues.apache.org/jira/browse/MAPREDUCE-4095(TestJobInProgress#testLocality
> uses a bogus topology.)
> said:"(https://issues.apache.org/jira/secure/attachment/12345251/Rack_aware_HDFS_proposal.pdf)
> which does not support imbalanced topology. "
>
> But when I re-read the desigh pdf, there just draw a balanced topology but
> really not mentioned it will not support imbalanced topology.
>
> Also I found this JIRA:
> https://issues.apache.org/jira/browse/HADOOP-8159(NetworkTopology: getLeaf
> should check for invalid topologies).
>
> Just no why for my question, because of the design(Right?):
> Why below is a invalid topology(except imbalanced), for hadoop rack
> awareness:
> serverA /default-rack##default rack
> serverB /dc1/rack2##actually the node has a pre "dc1"
> Should all of the hadoop datanode have the same level number? If not, there
> will be some issues for namenode and jt(like scheduler deadlock)?
This was a problem in earlier releases, where an imbalanced or
improper config for topology levels could lead to a jarring halt in
the JobTracker's schedulers. This has been fixed in 1.1.0 already via
https://issues.apache.org/jira/browse/MAPREDUCE-1740.
--
Harsh J
Re: number of mapper tasks
I'm unfamiliar with EMR myself (perhaps the question fits EMR's own boards) but here's my take anyway: On Mon, Jan 28, 2013 at 9:24 PM, Marcelo Elias Del Valle wrote: > Hello, > > I am using hadoop with TextInputFormat, a mapper and no reducers. I am > running my jobs at Amazon EMR. When I run my job, I set both following > options: > -s,mapred.tasktracker.map.tasks.maximum=10 > -jobconf,mapred.map.tasks=10 The first property you've given, refers to a single tasktracker's maximum concurrency. This means, if you have 4 TaskTrackers, with this property at each of them, then you have 40 total concurrent map slots available in all - perhaps more than you intended to configure? Again, this may be an EMR specific and I may be wrong, since I haven't seen anyone pass this via CLI before and it is generally to be configured at a service level. The second property is more to do with your problem. MR typically decides the number of map tasks it requires for a job, based on the input size. In the stable API (the org.apache.hadoop.mapred one), the mapred.map.tasks can be passed in the way you seem to be passing above, for an input format to take it as a 'hint' to decide number of map splits to enforce out of the input, no matter if it isn't large enough to necessitate that many maps. However, the new API code accepts no such config-based hints (and such logic changes need to be done in the programs' own code). So depending on your implementation of the job here, you may or may not see it act in effect. Hope this helps. > When I run my job with just 1 instance, I see it only creates 1 mapper. > When I run my job with 5 instances (1 master and 4 cores), I can see only 2 > mapper slots are used and 6 stay open. Perhaps the job itself launched with 2 total map tasks? You can check this on the JobTracker UI or whatever EMR offers as a job viewer. > I am trying to figure why I am not being able to run more mappers in > parallel. When I see the logs, I find some messages like these: > > INFO org.apache.hadoop.mapred.ReduceTask (main): > attempt_201301281437_0001_r_03_0 Scheduled 0 outputs (0 slow hosts and0 > dup hosts) > org.apache.hadoop.mapred.ReduceTask (main): > attempt_201301281437_0001_r_03_0 Need another 1 map output(s) where 0 is > already in progress This is a typical waiting reduce task log, what are you asking here specifically? -- Harsh J
Re: what does the yarn.nodemanager.aux-services stand for
Hi, The function of yarn.nodemanager.aux-services is to run any custom Service [1] implementations from the user, in the YARN NodeManager's runtime. The "aux" stands for auxiliary. One such example of where this comes useful is the ShuffleHandler [2] service, which helps MR do its shuffle work, without needing the NM itself to provide a shuffle service unnecessarily (in case MR is not required). Therefore, a ShuffleHandler is necessary to be configured if you need to run MR2 apps (jobs), but not otherwise. A clear difference, in case you're confused between MR2 and YARN, can be found at [3]. Do feel free to ask any further questions you have. [1] - http://svn.apache.org/viewvc/hadoop/common/branches/branch-2/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/service/Service.java?view=markup [2] - http://svn.apache.org/viewvc/hadoop/common/branches/branch-2/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java?view=markup [3] - http://www.cloudera.com/blog/2012/10/mr2-and-yarn-briefly-explained/ On Mon, Jan 28, 2013 at 6:43 PM, YouPeng Yang wrote: > Hi All >I am wondering what the yarn.nodemanager.aux-services stands for 。 > there even is not description about it in the yarn-default.xml。So i‘ve just > ignored it when i tried the example,as a result ,it failed。 > I have do some google，however，everyone just setup it in the > yarn-site.xml,but no one tells the description about it。 > please,helps will be appreciated . > > . > -- Harsh J
Re: what's this version 'version = 1.0.4-SNAPSHOT' ?
Hi, This is a minor annoyance, cause we ship the release tarball with the build files still carrying -SNAPSHOT suffix. Your version is still 1.0.4 after build and isn't 'upgraded' as you've pointed out. You can probably pass a -Dversion=1.0.4 to get around this as well. On Mon, Jan 28, 2013 at 3:59 PM, Xibin Liu wrote: > I downloaded the stable version of hadoop, which is a tar.gz file, when I > compile libhdfs with 'ant compile-contrib -Dlibhdfs=1 -Dfusedfs=1' the > version upgraded automatically, but I don't want it upgrade automatically, > is there any way to disable this? > > > 2013/1/28 Nitin Pawar >> >> can you give me which location you checked out the codebase from ? >> >> >> >> >> On Mon, Jan 28, 2013 at 12:31 PM, Xibin Liu >> wrote: >>> >>> Could you please give me more detail about change the dependency version >>> in pom? I don't really know what's pom and how to configure it, thanks! >>> >>> >>> 2013/1/28 Nitin Pawar >>>> >>>> any snapshot version is dev version >>>> >>>> you can change the dependency version to just 1.0.4 in the pom and it >>>> will just build 1.0.4 >>>> >>>> >>>> On Mon, Jan 28, 2013 at 12:06 PM, Xibin Liu >>>> wrote: >>>>> >>>>> hi, >>>>> After I compiled libhdfs, the version of hadoop become 1.0.4-SNAPSHOT, >>>>> is this a stable version or a dev version? Can I use this version on >>>>> production? If this version is not for production, how can I configure to >>>>> generate a stable version? >>>>> >>>>> -- >>>>> Thanks & Regards >>>>> Xibin Liu >>>>> >>>> >>>> >>>> >>>> -- >>>> Nitin Pawar >>> >>> >>> >>> >>> -- >>> Thanks & Regards >>> Xibin Liu >>> >> >> >> >> -- >> Nitin Pawar > > > > > -- > Thanks & Regards > Xibin Liu > -- Harsh J
Re: reg max map task config
Hi, As noted on http://hadoop.apache.org/docs/stable/cluster_setup.html#Configuring+the+Hadoop+Daemons, and by its own name, the config 'mapred.tasktracker.map.tasks.maximum' applies to a TaskTracker daemon and therefore needs to be applied to its local mapred-site.xml and has to be restarted to take new values into effect. On Mon, Jan 28, 2013 at 10:36 AM, Manoj Babu wrote: > Hi All, > > I am trying to override the value of mapred.tasktracker.map.tasks.maximum in > mapred-site.xml through my job configuration object will it reflect in the > corresponding task trackers? > or do i have to change it in mapred-site.xml following a restart for the > changes to apply? > > Thanks in advance. > > Cheers! > Manoj. -- Harsh J
