Namit - I have tried hive-trunk as of this afternoon and hive release 814942 ( revision with CombineHiveInputFormat commit ) .
Also - there are no logs that get generated on the tasktrackers for the hadoop job that fails. The only log that is generated on the jobtracker is the jobconf. Thanks -Matt On Thu, Oct 8, 2009 at 1:26 AM, Namit Jain <[email protected]> wrote: > Hi Matt, > > Sorry for the late reply. > > hive> set > hive.input.format=org.apache.hadoop.hive.ql.io.CombineHiveInputFormat; > > I tried it running on hadoop 20 and it ran fine for me. > > Which hive release are you using ? > > Also, you got a runtime error – can you see the stderr logs on the tracker > ? > > Thanks, > -namit > > > > On 10/1/09 5:01 PM, "Matt Pestritto" <[email protected]> wrote: > > Namit - > Any idea on how to resolve ? > Thanks > > On Thu, Oct 1, 2009 at 10:52 AM, Matt Pestritto <[email protected]> > wrote: > > > There were errors in the hive.log > > > > 2009-10-01 10:40:53,631 ERROR DataNucleus.Plugin > > (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires > > "org.eclipse.core.resources" but it cannot be resolved. > > 2009-10-01 10:40:53,631 ERROR DataNucleus.Plugin > > (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires > > "org.eclipse.core.resources" but it cannot be resolved. > > 2009-10-01 10:40:53,633 ERROR DataNucleus.Plugin > > (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires > > "org.eclipse.core.runtime" but it cannot be resolved. > > 2009-10-01 10:40:53,633 ERROR DataNucleus.Plugin > > (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires > > "org.eclipse.core.runtime" but it cannot be resolved. > > 2009-10-01 10:40:53,634 ERROR DataNucleus.Plugin > > (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires > > "org.eclipse.text" but it cannot be resolved. > > 2009-10-01 10:40:53,634 ERROR DataNucleus.Plugin > > (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires > > "org.eclipse.text" but it cannot be resolved. > > 2009-10-01 10:40:57,143 WARN mapred.JobClient > > (JobClient.java:configureCommandLineOptions(539)) - Use > GenericOptionsParser > > for parsing the arguments. Applications should implement Tool for the > same. > > 2009-10-01 10:40:58,609 ERROR exec.ExecDriver > > (SessionState.java:printError(248)) - Ended Job = job_200909301537_0068 > with > > errors > > 2009-10-01 10:40:58,622 ERROR ql.Driver > (SessionState.java:printError(248)) > > - FAILED: Execution Error, return code 2 from > > org.apache.hadoop.hive.ql.exec.ExecDriver > > > > > > > > On Wed, Sep 30, 2009 at 5:26 PM, Namit Jain <[email protected]> wrote: > > > >> What you are doing seems OK ? > >> Can you get the stack trace from /tmp/<username>/hive.log ? > >> > >> > >> > >> > >> > >> -----Original Message----- > >> From: Matt Pestritto [mailto:[email protected]] <[email protected]]> > >> Sent: Wednesday, September 30, 2009 6:51 AM > >> To: [email protected]; [email protected] > >> Subject: Fwd: Hive-74 > >> > >> Including hive-user in case someone has any experience with this.. > >> Thanks > >> -Matt > >> > >> ---------- Forwarded message ---------- > >> From: Matt Pestritto <[email protected]> > >> Date: Tue, Sep 29, 2009 at 5:26 PM > >> Subject: Hive-74 > >> To: [email protected] > >> > >> > >> Hi- > >> > >> I'm having a problem using CombineHiveInputSplit. I believe this was > >> patched in http://issues.apache.org/jira/browse/HIVE-74 > >> > >> I'm currently running hadoop 20.1 using hive trunk. > >> > >> hive-default.xml has the following property: > >> <property> > >> <name>hive.input.format</name> > >> <value></value> > >> <description>The default input format, if it is not specified, the > system > >> assigns it. It is set to HiveInputFormat for hadoop versions 17, 18 and > >> 19, > >> whereas it is set to CombinedHiveInputFormat for hadoop 20. The user can > >> always overwrite it - if there is a bug in CombinedHiveInputFormat, it > can > >> always be manually set to HiveInputFormat. </description> > >> </property> > >> > >> I added the following to hive-site.xml: ( Notice, the description in > >> hive-default.xml has CombinedHiveInputFormat which does not work for me > - > >> the property value seems to be Combine(-d) ) > >> <property> > >> <name>hive.input.format</name> > >> <value>org.apache.hadoop.hive.ql.io.CombineHiveInputFormat</value> > >> <description>The default input format, if it is not specified, the > system > >> assigns it. It is set to HiveInputFormat for hadoop versions 17, 18 and > >> 19, > >> whereas it is set to CombinedHiveInputFormat for hadoop 20. The user can > >> always overwrite it - if there is a bug in CombinedHiveInputFormat, it > can > >> always be manually set to HiveInputFormat. </description> > >> </property> > >> > >> When I launch a job the cli exits immediately: > >> hive> select count(1) from my_table; > >> Total MapReduce jobs = 1 > >> Number of reduce tasks determined at compile time: 1 > >> In order to change the average load for a reducer (in bytes): > >> set hive.exec.reducers.bytes.per.reducer=<number> > >> In order to limit the maximum number of reducers: > >> set hive.exec.reducers.max=<number> > >> In order to set a constant number of reducers: > >> set mapred.reduce.tasks=<number> > >> FAILED: Execution Error, return code 2 from > >> org.apache.hadoop.hive.ql.exec.ExecDriver > >> hive> exit ; > >> > >> If I set the property value to > >> org.apache.hadoop.hive.ql.io.HiveInputFormat, > >> the job runs fine. > >> > >> Suggestions ? Is there something that I am missing ? > >> > >> Thanks > >> -Matt > >> > > > > > >
