Re: Command-line jobConf options in 0.18.3
Tom White wrote: Actually, the space is needed, to be interpreted as a Hadoop option by ToolRunner. Without the space it sets a Java system property, which Hadoop will not automatically pick up. I don't think space is required. Something like -Dfs.default.name=host:port works. I don't see ToolRunner setting any java properties. Ian, try putting the options after the classname and see if that helps. Otherwise, it would be useful to see a snippet of the program code. right. the options that go to the class should appear after the class name. Note that it is not necessary to use ToolRunner (which I don't find very convenient in many cases). You can use GenericOptionsParser directly. an example : https://issues.apache.org/jira/browse/HADOOP-5961 Raghu. Thanks, Tom On Thu, Jun 4, 2009 at 8:23 PM, Vasyl Keretsman wrote: Perhaps, there should not be the "space" between -D and your option ? -Dprise.collopts= Vasyl 2009/6/4 Ian Soboroff : bin/hadoop jar -files collopts -D prise.collopts=collopts p3l-3.5.jar gov.nist.nlpir.prise.mapred.MapReduceIndexer input output The 'prise.collopts' option doesn't appear in the JobConf. Ian Aaron Kimball writes: Can you give an example of the exact arguments you're sending on the command line? - Aaron On Wed, Jun 3, 2009 at 5:46 PM, Ian Soboroff wrote: If after I call getConf to get the conf object, I manually add the key/ value pair, it's there when I need it. So it feels like ToolRunner isn't parsing my args for some reason. Ian On Jun 3, 2009, at 8:45 PM, Ian Soboroff wrote: Yes, and I get the JobConf via 'JobConf job = new JobConf(conf, the.class)'. The conf is the Configuration object that comes from getConf. Pretty much copied from the WordCount example (which this program used to be a long while back...) thanks, Ian On Jun 3, 2009, at 7:09 PM, Aaron Kimball wrote: Are you running your program via ToolRunner.run()? How do you instantiate the JobConf object? - Aaron On Wed, Jun 3, 2009 at 10:19 AM, Ian Soboroff < ian.sobor...@nist.gov> wrote: I'm backporting some code I wrote for 0.19.1 to 0.18.3 (long story), and I'm finding that when I run a job and try to pass options with -D on the command line, that the option values aren't showing up in my JobConf. I logged all the key/value pairs in the JobConf, and the option I passed through with -D isn't there. This worked in 0.19.1... did something change with command-line options from 18 to 19? Thanks, Ian
Re: Command-line jobConf options in 0.18.3
Actually, the space is needed, to be interpreted as a Hadoop option by ToolRunner. Without the space it sets a Java system property, which Hadoop will not automatically pick up. Ian, try putting the options after the classname and see if that helps. Otherwise, it would be useful to see a snippet of the program code. Thanks, Tom On Thu, Jun 4, 2009 at 8:23 PM, Vasyl Keretsman wrote: > Perhaps, there should not be the "space" between -D and your option ? > > -Dprise.collopts= > > Vasyl > > > > 2009/6/4 Ian Soboroff : >> >> bin/hadoop jar -files collopts -D prise.collopts=collopts p3l-3.5.jar >> gov.nist.nlpir.prise.mapred.MapReduceIndexer input output >> >> The 'prise.collopts' option doesn't appear in the JobConf. >> >> Ian >> >> Aaron Kimball writes: >> >>> Can you give an example of the exact arguments you're sending on the command >>> line? >>> - Aaron >>> >>> On Wed, Jun 3, 2009 at 5:46 PM, Ian Soboroff wrote: >>> >>> If after I call getConf to get the conf object, I manually add the key/ >>> value pair, it's there when I need it. So it feels like ToolRunner >>> isn't >>> parsing my args for some reason. >>> >>> Ian >>> >>> On Jun 3, 2009, at 8:45 PM, Ian Soboroff wrote: >>> >>> Yes, and I get the JobConf via 'JobConf job = new JobConf(conf, >>> the.class)'. The conf is the Configuration object that comes from >>> getConf. Pretty much copied from the WordCount example (which this >>> program used to be a long while back...) >>> >>> thanks, >>> Ian >>> >>> On Jun 3, 2009, at 7:09 PM, Aaron Kimball wrote: >>> >>> Are you running your program via ToolRunner.run()? How do you >>> instantiate the JobConf object? >>> - Aaron >>> >>> On Wed, Jun 3, 2009 at 10:19 AM, Ian Soboroff < >>> ian.sobor...@nist.gov> wrote: >>> I'm backporting some code I wrote for 0.19.1 to 0.18.3 (long >>> story), and I'm finding that when I run a job and try to pass >>> options with -D on the command line, that the option values >>> aren't >>> showing up in my JobConf. I logged all the key/value pairs in >>> the >>> JobConf, and the option I passed through with -D isn't there. >>> >>> This worked in 0.19.1... did something change with command-line >>> options from 18 to 19? >>> >>> Thanks, >>> Ian >> >
Re: Command-line jobConf options in 0.18.3
Perhaps, there should not be the "space" between -D and your option ? -Dprise.collopts= Vasyl 2009/6/4 Ian Soboroff : > > bin/hadoop jar -files collopts -D prise.collopts=collopts p3l-3.5.jar > gov.nist.nlpir.prise.mapred.MapReduceIndexer input output > > The 'prise.collopts' option doesn't appear in the JobConf. > > Ian > > Aaron Kimball writes: > >> Can you give an example of the exact arguments you're sending on the command >> line? >> - Aaron >> >> On Wed, Jun 3, 2009 at 5:46 PM, Ian Soboroff wrote: >> >> If after I call getConf to get the conf object, I manually add the key/ >> value pair, it's there when I need it. So it feels like ToolRunner isn't >> parsing my args for some reason. >> >> Ian >> >> On Jun 3, 2009, at 8:45 PM, Ian Soboroff wrote: >> >> Yes, and I get the JobConf via 'JobConf job = new JobConf(conf, >> the.class)'. The conf is the Configuration object that comes from >> getConf. Pretty much copied from the WordCount example (which this >> program used to be a long while back...) >> >> thanks, >> Ian >> >> On Jun 3, 2009, at 7:09 PM, Aaron Kimball wrote: >> >> Are you running your program via ToolRunner.run()? How do you >> instantiate the JobConf object? >> - Aaron >> >> On Wed, Jun 3, 2009 at 10:19 AM, Ian Soboroff < >> ian.sobor...@nist.gov> wrote: >> I'm backporting some code I wrote for 0.19.1 to 0.18.3 (long >> story), and I'm finding that when I run a job and try to pass >> options with -D on the command line, that the option values >> aren't >> showing up in my JobConf. I logged all the key/value pairs in >> the >> JobConf, and the option I passed through with -D isn't there. >> >> This worked in 0.19.1... did something change with command-line >> options from 18 to 19? >> >> Thanks, >> Ian >
Re: Command-line jobConf options in 0.18.3
bin/hadoop jar -files collopts -D prise.collopts=collopts p3l-3.5.jar gov.nist.nlpir.prise.mapred.MapReduceIndexer input output The 'prise.collopts' option doesn't appear in the JobConf. Ian Aaron Kimball writes: > Can you give an example of the exact arguments you're sending on the command > line? > - Aaron > > On Wed, Jun 3, 2009 at 5:46 PM, Ian Soboroff wrote: > > If after I call getConf to get the conf object, I manually add the key/ > value pair, it's there when I need it. So it feels like ToolRunner isn't > parsing my args for some reason. > > Ian > > On Jun 3, 2009, at 8:45 PM, Ian Soboroff wrote: > > Yes, and I get the JobConf via 'JobConf job = new JobConf(conf, > the.class)'. The conf is the Configuration object that comes from > getConf. Pretty much copied from the WordCount example (which this > program used to be a long while back...) > > thanks, > Ian > > On Jun 3, 2009, at 7:09 PM, Aaron Kimball wrote: > > Are you running your program via ToolRunner.run()? How do you > instantiate the JobConf object? > - Aaron > > On Wed, Jun 3, 2009 at 10:19 AM, Ian Soboroff < > ian.sobor...@nist.gov> wrote: > I'm backporting some code I wrote for 0.19.1 to 0.18.3 (long > story), and I'm finding that when I run a job and try to pass > options with -D on the command line, that the option values aren't > showing up in my JobConf. I logged all the key/value pairs in the > JobConf, and the option I passed through with -D isn't there. > > This worked in 0.19.1... did something change with command-line > options from 18 to 19? > > Thanks, > Ian
Re: Command-line jobConf options in 0.18.3
Can you give an example of the exact arguments you're sending on the command line? - Aaron On Wed, Jun 3, 2009 at 5:46 PM, Ian Soboroff wrote: > If after I call getConf to get the conf object, I manually add the > key/value pair, it's there when I need it. So it feels like ToolRunner > isn't parsing my args for some reason. > > Ian > > > On Jun 3, 2009, at 8:45 PM, Ian Soboroff wrote: > > Yes, and I get the JobConf via 'JobConf job = new JobConf(conf, >> the.class)'. The conf is the Configuration object that comes from getConf. >> Pretty much copied from the WordCount example (which this program used to >> be a long while back...) >> >> thanks, >> Ian >> >> On Jun 3, 2009, at 7:09 PM, Aaron Kimball wrote: >> >> Are you running your program via ToolRunner.run()? How do you instantiate >>> the JobConf object? >>> - Aaron >>> >>> On Wed, Jun 3, 2009 at 10:19 AM, Ian Soboroff >>> wrote: >>> I'm backporting some code I wrote for 0.19.1 to 0.18.3 (long story), and >>> I'm finding that when I run a job and try to pass options with -D on the >>> command line, that the option values aren't showing up in my JobConf. I >>> logged all the key/value pairs in the JobConf, and the option I passed >>> through with -D isn't there. >>> >>> This worked in 0.19.1... did something change with command-line options >>> from 18 to 19? >>> >>> Thanks, >>> Ian >>> >>> >>> >> >
Re: Command-line jobConf options in 0.18.3
If after I call getConf to get the conf object, I manually add the key/ value pair, it's there when I need it. So it feels like ToolRunner isn't parsing my args for some reason. Ian On Jun 3, 2009, at 8:45 PM, Ian Soboroff wrote: Yes, and I get the JobConf via 'JobConf job = new JobConf(conf, the.class)'. The conf is the Configuration object that comes from getConf. Pretty much copied from the WordCount example (which this program used to be a long while back...) thanks, Ian On Jun 3, 2009, at 7:09 PM, Aaron Kimball wrote: Are you running your program via ToolRunner.run()? How do you instantiate the JobConf object? - Aaron On Wed, Jun 3, 2009 at 10:19 AM, Ian Soboroff wrote: I'm backporting some code I wrote for 0.19.1 to 0.18.3 (long story), and I'm finding that when I run a job and try to pass options with -D on the command line, that the option values aren't showing up in my JobConf. I logged all the key/value pairs in the JobConf, and the option I passed through with -D isn't there. This worked in 0.19.1... did something change with command-line options from 18 to 19? Thanks, Ian
Re: Command-line jobConf options in 0.18.3
Yes, and I get the JobConf via 'JobConf job = new JobConf(conf, the.class)'. The conf is the Configuration object that comes from getConf. Pretty much copied from the WordCount example (which this program used to be a long while back...) thanks, Ian On Jun 3, 2009, at 7:09 PM, Aaron Kimball wrote: Are you running your program via ToolRunner.run()? How do you instantiate the JobConf object? - Aaron On Wed, Jun 3, 2009 at 10:19 AM, Ian Soboroff wrote: I'm backporting some code I wrote for 0.19.1 to 0.18.3 (long story), and I'm finding that when I run a job and try to pass options with - D on the command line, that the option values aren't showing up in my JobConf. I logged all the key/value pairs in the JobConf, and the option I passed through with -D isn't there. This worked in 0.19.1... did something change with command-line options from 18 to 19? Thanks, Ian
Re: Command-line jobConf options in 0.18.3
Are you running your program via ToolRunner.run()? How do you instantiate the JobConf object? - Aaron On Wed, Jun 3, 2009 at 10:19 AM, Ian Soboroff wrote: > I'm backporting some code I wrote for 0.19.1 to 0.18.3 (long story), and > I'm finding that when I run a job and try to pass options with -D on the > command line, that the option values aren't showing up in my JobConf. I > logged all the key/value pairs in the JobConf, and the option I passed > through with -D isn't there. > > This worked in 0.19.1... did something change with command-line options > from 18 to 19? > > Thanks, > Ian > >
Command-line jobConf options in 0.18.3
I'm backporting some code I wrote for 0.19.1 to 0.18.3 (long story), and I'm finding that when I run a job and try to pass options with -D on the command line, that the option values aren't showing up in my JobConf. I logged all the key/value pairs in the JobConf, and the option I passed through with -D isn't there. This worked in 0.19.1... did something change with command-line options from 18 to 19? Thanks, Ian
