Re: Is Hadoop's TooRunner thread-safe?
JIRA, test, patch and review? I am sure the community would welcome it. And
if you don't, well, it is unlikely to be appear soon into hadoop trunk.
Bertrand
On Fri, Mar 21, 2014 at 12:49 AM, Something Something
mailinglist...@gmail.com wrote:
Confirmed that ToolRunner is NOT thread-safe:
*Original code (which runs into problems):*
public static int run(Configuration conf, Tool tool, String[] args)
throws Exception{
if(conf == null) {
conf = new Configuration();
}
GenericOptionsParser parser = new GenericOptionsParser(conf, args);
//set the configuration back, so that Tool can configure itself
tool.setConf(conf);
//get the args w/o generic hadoop args
String[] toolArgs = parser.getRemainingArgs();
return tool.run(toolArgs);
}
*New code (which works):*
public static int run(Configuration conf, Tool tool, String[] args)
throws Exception{
if(conf == null) {
conf = new Configuration();
}
GenericOptionsParser parser = getParser(conf, args);
tool.setConf(conf);
//get the args w/o generic hadoop args
String[] toolArgs = parser.getRemainingArgs();
return tool.run(toolArgs);
}
private static *synchronized *GenericOptionsParser
getParser(Configuration conf, String[] args) throws Exception {
return new GenericOptionsParser(conf, args);
}
On Wed, Mar 19, 2014 at 10:15 AM, Something Something
mailinglist...@gmail.com wrote:
I would like to trigger a few Hadoop jobs simultaneously. I've created
a pool of threads using Executors.newFixedThreadPool. Idea is that if
the pool size is 2, my code will trigger 2 Hadoop jobs at the same exact
time using 'ToolRunner.run'. In my testing, I noticed that these 2
threads keep stepping on each other.
When I looked under the hood, I noticed that ToolRunner creates
GenericOptionsParser which in turn calls a static method
'buildGeneralOptions'. This method uses 'OptionBuilder.withArgName'
which uses an instance variable called, 'argName'. This doesn't look
thread safe to me and I believe is the root cause of issues I am running
into.
Any thoughts?
Re: Is Hadoop's TooRunner thread-safe?
I will be happy to follow all these steps if someone confirms that this is
the best way to handle it. Seems harmless to me, but just wondering.
Thanks.
On Fri, Mar 21, 2014 at 1:26 AM, Bertrand Dechoux decho...@gmail.comwrote:
JIRA, test, patch and review? I am sure the community would welcome it.
And if you don't, well, it is unlikely to be appear soon into hadoop trunk.
Bertrand
On Fri, Mar 21, 2014 at 12:49 AM, Something Something
mailinglist...@gmail.com wrote:
Confirmed that ToolRunner is NOT thread-safe:
*Original code (which runs into problems):*
public static int run(Configuration conf, Tool tool, String[] args)
throws Exception{
if(conf == null) {
conf = new Configuration();
}
GenericOptionsParser parser = new GenericOptionsParser(conf, args);
//set the configuration back, so that Tool can configure itself
tool.setConf(conf);
//get the args w/o generic hadoop args
String[] toolArgs = parser.getRemainingArgs();
return tool.run(toolArgs);
}
*New code (which works):*
public static int run(Configuration conf, Tool tool, String[] args)
throws Exception{
if(conf == null) {
conf = new Configuration();
}
GenericOptionsParser parser = getParser(conf, args);
tool.setConf(conf);
//get the args w/o generic hadoop args
String[] toolArgs = parser.getRemainingArgs();
return tool.run(toolArgs);
}
private static *synchronized *GenericOptionsParser
getParser(Configuration conf, String[] args) throws Exception {
return new GenericOptionsParser(conf, args);
}
On Wed, Mar 19, 2014 at 10:15 AM, Something Something
mailinglist...@gmail.com wrote:
I would like to trigger a few Hadoop jobs simultaneously. I've created
a pool of threads using Executors.newFixedThreadPool. Idea is that if
the pool size is 2, my code will trigger 2 Hadoop jobs at the same exact
time using 'ToolRunner.run'. In my testing, I noticed that these 2
threads keep stepping on each other.
When I looked under the hood, I noticed that ToolRunner creates
GenericOptionsParser which in turn calls a static method
'buildGeneralOptions'. This method uses 'OptionBuilder.withArgName'
which uses an instance variable called, 'argName'. This doesn't look
thread safe to me and I believe is the root cause of issues I am running
into.
Any thoughts?
Re: Is Hadoop's TooRunner thread-safe?
Yes, this is the best way to go.
Sent from my iPhone5s
On 2014年3月22日, at 3:03, Something Something mailinglist...@gmail.com wrote:
I will be happy to follow all these steps if someone confirms that this is
the best way to handle it. Seems harmless to me, but just wondering. Thanks.
On Fri, Mar 21, 2014 at 1:26 AM, Bertrand Dechoux decho...@gmail.com wrote:
JIRA, test, patch and review? I am sure the community would welcome it. And
if you don't, well, it is unlikely to be appear soon into hadoop trunk.
Bertrand
On Fri, Mar 21, 2014 at 12:49 AM, Something Something
mailinglist...@gmail.com wrote:
Confirmed that ToolRunner is NOT thread-safe:
Original code (which runs into problems):
public static int run(Configuration conf, Tool tool, String[] args)
throws Exception{
if(conf == null) {
conf = new Configuration();
}
GenericOptionsParser parser = new GenericOptionsParser(conf, args);
//set the configuration back, so that Tool can configure itself
tool.setConf(conf);
//get the args w/o generic hadoop args
String[] toolArgs = parser.getRemainingArgs();
return tool.run(toolArgs);
}
New code (which works):
public static int run(Configuration conf, Tool tool, String[] args)
throws Exception{
if(conf == null) {
conf = new Configuration();
}
GenericOptionsParser parser = getParser(conf, args);
tool.setConf(conf);
//get the args w/o generic hadoop args
String[] toolArgs = parser.getRemainingArgs();
return tool.run(toolArgs);
}
private static synchronized GenericOptionsParser
getParser(Configuration conf, String[] args) throws Exception {
return new GenericOptionsParser(conf, args);
}
On Wed, Mar 19, 2014 at 10:15 AM, Something Something
mailinglist...@gmail.com wrote:
I would like to trigger a few Hadoop jobs simultaneously. I’ve created a
pool of threads using Executors.newFixedThreadPool. Idea is that if the
pool size is 2, my code will trigger 2 Hadoop jobs at the same exact time
using ‘ToolRunner.run’. In my testing, I noticed that these 2 threads
keep stepping on each other.
When I looked under the hood, I noticed that ToolRunner creates
GenericOptionsParser which in turn calls a static method
‘buildGeneralOptions’. This method uses ‘OptionBuilder.withArgName’ which
uses an instance variable called, ‘argName’. This doesn’t look thread
safe to me and I believe is the root cause of issues I am running into.
Any thoughts?
Re: Is Hadoop's TooRunner thread-safe?
Confirmed that ToolRunner is NOT thread-safe:
*Original code (which runs into problems):*
public static int run(Configuration conf, Tool tool, String[] args)
throws Exception{
if(conf == null) {
conf = new Configuration();
}
GenericOptionsParser parser = new GenericOptionsParser(conf, args);
//set the configuration back, so that Tool can configure itself
tool.setConf(conf);
//get the args w/o generic hadoop args
String[] toolArgs = parser.getRemainingArgs();
return tool.run(toolArgs);
}
*New code (which works):*
public static int run(Configuration conf, Tool tool, String[] args)
throws Exception{
if(conf == null) {
conf = new Configuration();
}
GenericOptionsParser parser = getParser(conf, args);
tool.setConf(conf);
//get the args w/o generic hadoop args
String[] toolArgs = parser.getRemainingArgs();
return tool.run(toolArgs);
}
private static *synchronized *GenericOptionsParser
getParser(Configuration conf, String[] args) throws Exception {
return new GenericOptionsParser(conf, args);
}
On Wed, Mar 19, 2014 at 10:15 AM, Something Something
mailinglist...@gmail.com wrote:
I would like to trigger a few Hadoop jobs simultaneously. I've created a
pool of threads using Executors.newFixedThreadPool. Idea is that if the
pool size is 2, my code will trigger 2 Hadoop jobs at the same exact time
using 'ToolRunner.run'. In my testing, I noticed that these 2 threads
keep stepping on each other.
When I looked under the hood, I noticed that ToolRunner creates
GenericOptionsParser which in turn calls a static method
'buildGeneralOptions'. This method uses 'OptionBuilder.withArgName'
which uses an instance variable called, 'argName'. This doesn't look
thread safe to me and I believe is the root cause of issues I am running
into.
Any thoughts?
Re: Is Hadoop's TooRunner thread-safe?
Any thoughts on this? Confirm or Deny it's an issue.. may be? On Mon, Mar 17, 2014 at 11:43 AM, Something Something mailinglist...@gmail.com wrote: I would like to trigger a few Hadoop jobs simultaneously. I've created a pool of threads using Executors.newFixedThreadPool. Idea is that if the pool size is 2, my code will trigger 2 Hadoop jobs at the same exact time using 'ToolRunner.run'. In my testing, I noticed that these 2 threads keep stepping on each other. When I looked under the hood, I noticed that ToolRunner creates GenericOptionsParser which in turn calls a static method 'buildGeneralOptions'. This method uses 'OptionBuilder.withArgName' which uses an instance variable called, 'argName'. This doesn't look thread safe to me and I believe is the root cause of issues I am running into. Any thoughts?
Is Hadoop's TooRunner thread-safe?
I would like to trigger a few Hadoop jobs simultaneously. I've created a pool of threads using Executors.newFixedThreadPool. Idea is that if the pool size is 2, my code will trigger 2 Hadoop jobs at the same exact time using 'ToolRunner.run'. In my testing, I noticed that these 2 threads keep stepping on each other. When I looked under the hood, I noticed that ToolRunner creates GenericOptionsParser which in turn calls a static method 'buildGeneralOptions'. This method uses 'OptionBuilder.withArgName' which uses an instance variable called, 'argName'. This doesn't look thread safe to me and I believe is the root cause of issues I am running into. Any thoughts?
