To make sure I understand you correctly, you need a globally unique
one up counter for each output record?
If you had an upper bound on the number of records a single reducer
could output and you can afford to have gaps, you could just use the
task id and multiply that by the max number of
it.
But what are counters for? They seem to be exactly that.
Mark
On Fri, May 20, 2011 at 12:01 PM, Joey Echeverria j...@cloudera.com wrote:
To make sure I understand you correctly, you need a globally unique
one up counter for each output record?
If you had an upper bound on the number
Are you storing the data in sequence files?
-Joey
On Fri, May 20, 2011 at 10:33 AM, W.P. McNeill bill...@gmail.com wrote:
The keys are Text and the values are large custom data structures serialized
with Avro.
I also have counters for the job that generates these files that gives me
this
Why do you need to move the script from $HADOOP_HOME/bin?
Can't you just symlink it or write a script which runs the original?
-Joey
On May 19, 2011, at 4:15, Gabriele Kahlout gabri...@mysimpatico.com wrote:
I'm still having the following problem, any suggestions?
I'm trying to modify the
, May 19, 2011 at 3:33 PM, Joey Echeverria j...@cloudera.com
wrote:
Why do you need to move the script from $HADOOP_HOME/bin?
Can't you just symlink it or write a script which runs the original?
-Joey
On May 19, 2011, at 4:15, Gabriele Kahlout gabri...@mysimpatico.com
wrote
1041718
Compiled by hammer on Mon Dec 6 17:38:16 CET 2010
On Thu, May 19, 2011 at 4:55 PM, Joey Echeverria j...@cloudera.com
wrote:
What version of hadoop is installed?
-Joey
On May 19, 2011 7:49 AM, Gabriele Kahlout gabri...@mysimpatico.com
wrote:
I said i don't have write access
hdfs.
On Thu, May 19, 2011 at 5:02 PM, Joey Echeverria j...@cloudera.com wrote:
Why do you need the hdfs script? Typically 0.20.x is used with just the
hadoop script.
-Joey
On May 19, 2011 8:00 AM, Gabriele Kahlout gabri...@mysimpatico.com
wrote:
$ hadoop version
Hadoop 0.20.3-SNAPSHOT
? No changes to hdfs-config.sh?
What about all the other stuff in the hdfs?
For example the script calls hdfs dfs , like that won't it crash?
elif [ $COMMAND = dfs ] ; then
CLASS=org.apache.hadoop.fs.FsShell
On Thu, May 19, 2011 at 5:26 PM, Joey Echeverria j...@cloudera.com wrote:
I would just
Filing a bug is a great idea. InputSampler is in the MapReduce hadoop
sub-project which has it's own Jira project:
https://issues.apache.org/jira/browse/MAPREDUCE
-Joey
On Thu, May 19, 2011 at 9:28 AM, W.P. McNeill bill...@gmail.com wrote:
Should I file a bug then? Do I do that
The one advantage you would get with a large number of reducers is
that the scheduler will be able to give open reduce slots to other
jobs without having to be preemptive.
This will reduce the risk of you losing a reducer 3 hours into a 4 hour run.
-Joey
On Wed, May 18, 2011 at 3:08 PM, James
That sounds like a bug to me.
I think the easiest way would be to modify InputSampler to handle non Text keys.
-Joey
On Wed, May 18, 2011 at 4:24 PM, W.P. McNeill bill...@gmail.com wrote:
I want to do a total sort on some data whose key type is Writable but not
Text. I wrote an
The sequence file writer definitely does it serially as you can only
ever write to the end of a file in Hadoop.
Doing copyFromLocal could write multiple files in parallel (I'm not
sure if it does or not), but a single file would be written serially.
-Joey
On Tue, May 17, 2011 at 5:44 PM, Mapred
Which version of hadoop are you running?
Are you running on linux?
-Joey
On Thu, May 12, 2011 at 1:39 PM, Adi adi.pan...@gmail.com wrote:
For one long running job we are noticing that the mapper jvms do not exit
even after the mapper is done. Any suggestions on why this could be
happening.
Hadoop 0.21.0 with some patches.
Hadoop 0.21.0 doesn't get much use, so I'm not sure how much help I can be.
2011-05-12 13:52:04,147 WARN org.apache.hadoop.mapreduce.util.ProcessTree:
Error executing shell command
org.apache.hadoop.util.Shell$ExitCodeException: kill -12545: No such process
You can control the number of reducers by calling
job.setNumReduceTasks() before you launch it.
-Joey
On Thu, May 12, 2011 at 6:33 PM, Jun Young Kim juneng...@gmail.com wrote:
yes. that is a general solution to control counts of output files.
however, if you need to control counts of outputs
Your delcaration of the Map class needs to include the input and
output types, e.g.:
public static class Map extends MapReduceBase implements
MapperLongWritable, Text, Text, LongWritable {
...
}
-Joey
On Mon, Apr 25, 2011 at 4:38 AM, praveenesh kumar praveen...@gmail.com wrote:
Hi,
I am
Did you try calling fs.setConf(configuration)?
On Apr 22, 2011 9:09 PM, W.P. McNeill bill...@gmail.com wrote:
101 - 117 of 117 matches
Mail list logo
