Re: How to get JobID in code?
Ok, will try that. Thanks! ..meghana On 6 September 2011 20:25, John Conwell j...@iamjohn.me wrote: You can override the configure method in you mapper or reducer, and call JobConf.get(xpatterns.hadoop.content.job.id) This will return you the UUID for the job id On Mon, Sep 5, 2011 at 10:02 PM, Meghana meghana.mara...@germinait.com wrote: Hi, We configure a org.apache.hadoop.mapreduce.Job, and then call job.submit(). The job starts running on the cluster. At this point, we need to log/store the job id of the submitted job. But calling job.getJobID() returns null, for our instance of the job! Is there another way to do this? We recently shifted to using 0.20.203. job.getJobID() used to work in 0.19 Thanks, Meghana -- Thanks, John C
How to get JobID in code?
Hi, We configure a org.apache.hadoop.mapreduce.Job, and then call job.submit(). The job starts running on the cluster. At this point, we need to log/store the job id of the submitted job. But calling job.getJobID() returns null, for our instance of the job! Is there another way to do this? We recently shifted to using 0.20.203. job.getJobID() used to work in 0.19 Thanks, Meghana
Reader/Writer problem in HDFS
Hi,
We have a job where the map tasks are given the path to an output folder.
Each map task writes a single file to that folder. There is no reduce phase.
There is another thread, which constantly looks for new files in the output
folder. If found, it persists the contents to index, and deletes the file.
We use this code in the map task:
try {
OutputStream oStream = fileSystem.create(path);
IOUtils.write(xyz, oStream);
} finally {
IOUtils.closeQuietly(oStream);
}
The problem: Some times the reader thread sees tries to read a file which
is not yet fully written to HDFS (or the checksum is not written yet, etc),
and throws an error. Is it possible to write an HDFS file in such a way that
it won't be visible until it is fully written?
We use Hadoop 0.20.203.
Thanks,
Meghana
Re: Reader/Writer problem in HDFS
Thanks Laxman! That would definitely help things. :)
Is there a better FileSystem/other method call to create a file in one go
(i.e. atomic i guess?), without having to call create() and then write to
the stream?
..meghana
On 28 July 2011 16:12, Laxman lakshman...@huawei.com wrote:
One approach can be use some .tmp extension while writing. Once the write
is completed rename back to original file name. Also, reducer has to filter
out .tmp files.
This will ensure reducer will not pickup the partial files.
We do have the similar scenario where the a/m approach resolved the issue.
-Original Message-
From: Meghana [mailto:meghana.mara...@germinait.com]
Sent: Thursday, July 28, 2011 1:38 PM
To: common-user; hdfs-u...@hadoop.apache.org
Subject: Reader/Writer problem in HDFS
Hi,
We have a job where the map tasks are given the path to an output folder.
Each map task writes a single file to that folder. There is no reduce
phase.
There is another thread, which constantly looks for new files in the output
folder. If found, it persists the contents to index, and deletes the file.
We use this code in the map task:
try {
OutputStream oStream = fileSystem.create(path);
IOUtils.write(xyz, oStream);
} finally {
IOUtils.closeQuietly(oStream);
}
The problem: Some times the reader thread sees tries to read a file which
is not yet fully written to HDFS (or the checksum is not written yet, etc),
and throws an error. Is it possible to write an HDFS file in such a way
that
it won't be visible until it is fully written?
We use Hadoop 0.20.203.
Thanks,
Meghana
Too many open files
Hi, We have a small cluster of 7 machines, of which 4 are also tasktrackers, running Hadoop 0.19 (yes, we should move to 0.20...). There is no Reduce phase; instead each Map task writes output to an HDFS file. On completion of a job, there is a listener that reads every such file writes the results to a Solr index. We observed a very strange scenario this weekend: the master complains about too many open files every 24 hours or so, and stops working, though the HDFS cluster itself remains up. We get two errors: 11/07/10 05:16:57 ERROR our.Class: java.io.FileNotFoundException: /home/user/project/conf/hadoop-default.xml (Too many open files) : org.apache.hadoop.conf.Configuration : 1043 11/07/09 05:01:20 ERROR namenode.LeaseManager: In org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor java.util.ConcurrentModificationException at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1100) at java.util.TreeMap$KeyIterator.next(TreeMap.java:1154) at org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:355) at java.lang.Thread.run(Thread.java:619) When i checked /proc/sys/fs/file-nr it showed about 37500 or so. This issue has never cropped up before... Please help! :) Thanks, Meghana
Jobtracker throws GC overhead limit exceeded
Hi, We are using Hadoop 0.19 on a small cluster of 7 machines (7 datanodes, 4 task trackers), and we typically have 3-4 jobs running at a time. We have been facing the following error on the Jobtracker: java.io.IOException: java.lang.OutOfMemoryError: GC overhead limit exceeded It seems to be thrown by RunningJob.killJob() and usually occurs after a day or so of starting up the cluster. In the Jobtracker's output file: Exception in thread initJobs java.lang.OutOfMemoryError: GC overhead limit exceeded at java.lang.String.substring(String.java:1939) at java.lang.String.substring(String.java:1904) at org.apache.hadoop.fs.Path.getName(Path.java:188) at org.apache.hadoop.fs.ChecksumFileSystem.isChecksumFile(ChecksumFileSystem.java:70) at org.apache.hadoop.fs.ChecksumFileSystem$1.accept(ChecksumFileSystem.java:442) at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:726) at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:748) at org.apache.hadoop.fs.ChecksumFileSystem.listStatus(ChecksumFileSystem.java:457) at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:723) at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:748) at org.apache.hadoop.mapred.JobHistory$JobInfo.getJobHistoryFileName(JobHistory.java:660) at org.apache.hadoop.mapred.JobHistory$JobInfo.finalizeRecovery(JobHistory.java:746) at org.apache.hadoop.mapred.JobTracker.finalizeJob(JobTracker.java:1532) at org.apache.hadoop.mapred.JobInProgress.garbageCollect(JobInProgress.java:2232) at org.apache.hadoop.mapred.JobInProgress.terminateJob(JobInProgress.java:1938) at org.apache.hadoop.mapred.JobInProgress.terminate(JobInProgress.java:1953) at org.apache.hadoop.mapred.JobInProgress.fail(JobInProgress.java:2012) at org.apache.hadoop.mapred.EagerTaskInitializationListener$JobInitThread.run(EagerTaskInitializationListener.java:62) at java.lang.Thread.run(Thread.java:619) Please help! Thanks, Meghana
Re: Jobtracker throws GC overhead limit exceeded
Hey Arun, The JT heapsize (Xmx) is 512m. Will try increasing it, thanks! Yes, migrating to 0.20 is definitely on my to-do list, but some urgent issues have taken priority for now :( Thanks, ..meghana On 5 July 2011 12:25, Arun C Murthy a...@hortonworks.com wrote: Meghana, What is the heapsize for your JT? Try increasing that. Also, we've fixed a huge number of issues in the JT (and Hadoop overall) since 0.19. Can you upgrade to 0.20.203, the latest stable release? thanks, Arun Sent from my iPhone On Jul 4, 2011, at 11:10 PM, Meghana meghana.mara...@germinait.com wrote: Hi, We are using Hadoop 0.19 on a small cluster of 7 machines (7 datanodes, 4 task trackers), and we typically have 3-4 jobs running at a time. We have been facing the following error on the Jobtracker: java.io.IOException: java.lang.OutOfMemoryError: GC overhead limit exceeded It seems to be thrown by RunningJob.killJob() and usually occurs after a day or so of starting up the cluster. In the Jobtracker's output file: Exception in thread initJobs java.lang.OutOfMemoryError: GC overhead limit exceeded at java.lang.String.substring(String.java:1939) at java.lang.String.substring(String.java:1904) at org.apache.hadoop.fs.Path.getName(Path.java:188) at org.apache.hadoop.fs.ChecksumFileSystem.isChecksumFile(ChecksumFileSystem.java:70) at org.apache.hadoop.fs.ChecksumFileSystem$1.accept(ChecksumFileSystem.java:442) at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:726) at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:748) at org.apache.hadoop.fs.ChecksumFileSystem.listStatus(ChecksumFileSystem.java:457) at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:723) at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:748) at org.apache.hadoop.mapred.JobHistory$JobInfo.getJobHistoryFileName(JobHistory.java:660) at org.apache.hadoop.mapred.JobHistory$JobInfo.finalizeRecovery(JobHistory.java:746) at org.apache.hadoop.mapred.JobTracker.finalizeJob(JobTracker.java:1532) at org.apache.hadoop.mapred.JobInProgress.garbageCollect(JobInProgress.java:2232) at org.apache.hadoop.mapred.JobInProgress.terminateJob(JobInProgress.java:1938) at org.apache.hadoop.mapred.JobInProgress.terminate(JobInProgress.java:1953) at org.apache.hadoop.mapred.JobInProgress.fail(JobInProgress.java:2012) at org.apache.hadoop.mapred.EagerTaskInitializationListener$JobInitThread.run(EagerTaskInitializationListener.java:62) at java.lang.Thread.run(Thread.java:619) Please help! Thanks, Meghana
