RE: MapJoin error: .hashtable file not found
Thanks Ashutosh. I will give it a try. -Vivek From: Ashutosh Chauhan [hashut...@apache.org] Sent: 28 November 2012 12:43 To: user@hive.apache.org Subject: Re: MapJoin error: .hashtable file not found Hi Vivek, I will encourage you to try out latest trunk or hive-0.10 branch. Lots of work has gone in since 0.9 to make Hive work better on Windows. Ashutosh On Tue, Nov 27, 2012 at 11:08 PM, Vivek Mishra mailto:vivek.mis...@impetus.co.in>> wrote: Hi, I am using 0.9.0 hive with Windows 2008 server. I did try debugging into code, but no luck to understand: " Can somebody exaplain how exactly such ".hastable" files get created? What could be the reason they are not created though gz files are there? " -Vivek From: Mark Grover [grover.markgro...@gmail.com<mailto:grover.markgro...@gmail.com>] Sent: 28 November 2012 11:41 To: user@hive.apache.org<mailto:user@hive.apache.org> Subject: Re: MapJoin error: .hashtable file not found Vivek, What version of Hive are you using? And, on what OS? Mark On Tue, Nov 27, 2012 at 9:34 PM, Vivek Mishra mailto:vivek.mis...@impetus.co.in><mailto:vivek.mis...@impetus.co.in<mailto:vivek.mis...@impetus.co.in>>> wrote: Any pointers? Any help? Can somebody exaplain how exactly such ".hastable" files get created? What could be the reason they are not created though gz files are there -Vivek From: Vivek Mishra Sent: 24 November 2012 11:52 To: user@hive.apache.org<mailto:user@hive.apache.org><mailto:user@hive.apache.org<mailto:user@hive.apache.org>> Subject: MapJoin error: .hashtable file not found Hi, I am trying to run a MapJoin query and somehow getting below error. I can see that file is not there in specified directory. Did some debugging but no luck as well. Here is the error: nOperator:Load back 1 hashtable file from tmp file uri:c:/hadoop/hdfs/mapred/local/taskTracker/distcache/-6923915657555089159_-359650783_821098197/localhost/tmp/hive-vivek.mishra/hive_2012-11-24_11-12-30_853_4076782611151037010/-mr-10004/HashTable-Stage-5/Stage-5.tar.gz/MapJoin-mapfile10--.hashtable 2012-11-24 11:13:45,389 ERROR org.apache.hadoop.hive.ql.exec.MapJoinOperator: Load Distributed Cache Error 2012-11-24 11:13:45,390 FATAL ExecMapper: org.apache.hadoop.hive.ql.metadata.HiveException: c:\hadoop\hdfs\mapred\local\taskTracker\distcache\-6923915657555089159_-359650783_821098197\localhost\tmp\hive-vivek.mishra\hive_2012-11-24_11-12-30_853_4076782611151037010\-mr-10004\HashTable-Stage-5\Stage-5.tar.gz\MapJoin-mapfile10--.hashtable (The system cannot find the file specified) at org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:198) at org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:212) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1321) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1325) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1325) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1325) at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:495) at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:143) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372) at org.apache.hadoop.mapred.Child$4.run(Child.java:271) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135) at org.apache.hadoop.mapred.Child.main(Child.java:265) Also, i have verified there is no such OOM issue as well. Any idea? -Vivek Neustar VP and Impetus CEO to present on ‘Innovative information services powered by Cloud and Big Data technologies’at Cloud Expo - Santa Clara, Nov 6th. http://www.impetus.com/events#2. Check out Impetus contribution to build Luminar - a new business unit at Entravision. http://lf1.me/MS/ NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference. Neustar VP and
RE: MapJoin error: .hashtable file not found
Thanks. This is what i understood. I am using set auto.convert.join=true. As i want hive to determine smaller table to fit in memory. I did look into MapJoinOperator and MapJoinProcessor.java , but could not find any pointer to my problem. -Vivek From: Navis류승우 [navis@nexr.com] Sent: 28 November 2012 12:44 To: user@hive.apache.org Subject: Re: MapJoin error: .hashtable file not found hi, The hashtable file you mentioned is pre-staged file for MapJoin. If you remove MAPJOIN hint and set auto.convert.join=false, it will be disabled 2012/11/28 Vivek Mishra mailto:vivek.mis...@impetus.co.in>> Hi, I am using 0.9.0 hive with Windows 2008 server. I did try debugging into code, but no luck to understand: " Can somebody exaplain how exactly such ".hastable" files get created? What could be the reason they are not created though gz files are there? " -Vivek From: Mark Grover [grover.markgro...@gmail.com<mailto:grover.markgro...@gmail.com>] Sent: 28 November 2012 11:41 To: user@hive.apache.org<mailto:user@hive.apache.org> Subject: Re: MapJoin error: .hashtable file not found Vivek, What version of Hive are you using? And, on what OS? Mark On Tue, Nov 27, 2012 at 9:34 PM, Vivek Mishra mailto:vivek.mis...@impetus.co.in><mailto:vivek.mis...@impetus.co.in<mailto:vivek.mis...@impetus.co.in>>> wrote: Any pointers? Any help? Can somebody exaplain how exactly such ".hastable" files get created? What could be the reason they are not created though gz files are there -Vivek From: Vivek Mishra Sent: 24 November 2012 11:52 To: user@hive.apache.org<mailto:user@hive.apache.org><mailto:user@hive.apache.org<mailto:user@hive.apache.org>> Subject: MapJoin error: .hashtable file not found Hi, I am trying to run a MapJoin query and somehow getting below error. I can see that file is not there in specified directory. Did some debugging but no luck as well. Here is the error: nOperator:Load back 1 hashtable file from tmp file uri:c:/hadoop/hdfs/mapred/local/taskTracker/distcache/-6923915657555089159_-359650783_821098197/localhost/tmp/hive-vivek.mishra/hive_2012-11-24_11-12-30_853_4076782611151037010/-mr-10004/HashTable-Stage-5/Stage-5.tar.gz/MapJoin-mapfile10--.hashtable 2012-11-24 11:13:45,389 ERROR org.apache.hadoop.hive.ql.exec.MapJoinOperator: Load Distributed Cache Error 2012-11-24 11:13:45,390 FATAL ExecMapper: org.apache.hadoop.hive.ql.metadata.HiveException: c:\hadoop\hdfs\mapred\local\taskTracker\distcache\-6923915657555089159_-359650783_821098197\localhost\tmp\hive-vivek.mishra\hive_2012-11-24_11-12-30_853_4076782611151037010\-mr-10004\HashTable-Stage-5\Stage-5.tar.gz\MapJoin-mapfile10--.hashtable (The system cannot find the file specified) at org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:198) at org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:212) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1321) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1325) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1325) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1325) at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:495) at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:143) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372) at org.apache.hadoop.mapred.Child$4.run(Child.java:271) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135) at org.apache.hadoop.mapred.Child.main(Child.java:265) Also, i have verified there is no such OOM issue as well. Any idea? -Vivek Neustar VP and Impetus CEO to present on ‘Innovative information services powered by Cloud and Big Data technologies’at Cloud Expo - Santa Clara, Nov 6th. http://www.impetus.com/events#2. Check out Impetus contribution to build Luminar - a new business unit at Entravision. http://lf1.me/MS/ NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this co
Re: MapJoin error: .hashtable file not found
hi, The hashtable file you mentioned is pre-staged file for MapJoin. If you remove MAPJOIN hint and set auto.convert.join=false, it will be disabled 2012/11/28 Vivek Mishra > Hi, > > I am using 0.9.0 hive with Windows 2008 server. > > I did try debugging into code, but no luck to understand: > > " > Can somebody exaplain how exactly such ".hastable" files get created? What > could be the reason they are not created though gz files are there? > " > -Vivek > > From: Mark Grover [grover.markgro...@gmail.com] > Sent: 28 November 2012 11:41 > To: user@hive.apache.org > Subject: Re: MapJoin error: .hashtable file not found > > Vivek, > What version of Hive are you using? And, on what OS? > > Mark > > On Tue, Nov 27, 2012 at 9:34 PM, Vivek Mishra <mailto:vivek.mis...@impetus.co.in>> wrote: > Any pointers? Any help? > > Can somebody exaplain how exactly such ".hastable" files get created? What > could be the reason they are not created though gz files are there > > -Vivek > > From: Vivek Mishra > Sent: 24 November 2012 11:52 > To: user@hive.apache.org<mailto:user@hive.apache.org> > Subject: MapJoin error: .hashtable file not found > > Hi, > > I am trying to run a MapJoin query and somehow getting below error. I can > see that file is not there in specified directory. Did some debugging but > no luck as well. > > Here is the error: > > nOperator:Load back 1 hashtable file from tmp file > uri:c:/hadoop/hdfs/mapred/local/taskTracker/distcache/-6923915657555089159_-359650783_821098197/localhost/tmp/hive-vivek.mishra/hive_2012-11-24_11-12-30_853_4076782611151037010/-mr-10004/HashTable-Stage-5/Stage-5.tar.gz/MapJoin-mapfile10--.hashtable > 2012-11-24 11:13:45,389 ERROR > org.apache.hadoop.hive.ql.exec.MapJoinOperator: Load Distributed Cache Error > 2012-11-24 11:13:45,390 FATAL ExecMapper: > org.apache.hadoop.hive.ql.metadata.HiveException: > c:\hadoop\hdfs\mapred\local\taskTracker\distcache\-6923915657555089159_-359650783_821098197\localhost\tmp\hive-vivek.mishra\hive_2012-11-24_11-12-30_853_4076782611151037010\-mr-10004\HashTable-Stage-5\Stage-5.tar.gz\MapJoin-mapfile10--.hashtable > (The system cannot find the file specified) > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:198) >at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:212) > at > org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1321) > at > org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1325) >at > org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1325) > at > org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1325) > at > org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:495) > at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:143) >at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436) >at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372) > at org.apache.hadoop.mapred.Child$4.run(Child.java:271) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) >at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135) >at org.apache.hadoop.mapred.Child.main(Child.java:265) > > > Also, i have verified there is no such OOM issue as well. > > > Any idea? > > > -Vivek > > > > Neustar VP and Impetus CEO to present on ‘Innovative information services > powered by Cloud and Big Data technologies’at Cloud Expo - Santa Clara, Nov > 6th. http://www.impetus.com/events#2. > > Check out Impetus contribution to build Luminar - a new business unit at > Entravision. http://lf1.me/MS/ > > > NOTE: This message may contain information that is confidential, > proprietary, privileged or otherwise protected by law. The message is > intended solely for the named addressee. If received in error, please > destroy and notify the sender. Any use of this email is prohibited when > received in error. Impetus does not represent, warrant and/or guarantee, > that the integrity of this communication has been maintained nor that the > communication is free of errors, virus, interception or interference. > > > > Neustar VP and Impetus CEO to present on ‘Innovative information services > po
Re: MapJoin error: .hashtable file not found
Hi Vivek, I will encourage you to try out latest trunk or hive-0.10 branch. Lots of work has gone in since 0.9 to make Hive work better on Windows. Ashutosh On Tue, Nov 27, 2012 at 11:08 PM, Vivek Mishra wrote: > Hi, > > I am using 0.9.0 hive with Windows 2008 server. > > I did try debugging into code, but no luck to understand: > > " > Can somebody exaplain how exactly such ".hastable" files get created? What > could be the reason they are not created though gz files are there? > " > -Vivek > > From: Mark Grover [grover.markgro...@gmail.com] > Sent: 28 November 2012 11:41 > To: user@hive.apache.org > Subject: Re: MapJoin error: .hashtable file not found > > Vivek, > What version of Hive are you using? And, on what OS? > > Mark > > On Tue, Nov 27, 2012 at 9:34 PM, Vivek Mishra <mailto:vivek.mis...@impetus.co.in>> wrote: > Any pointers? Any help? > > Can somebody exaplain how exactly such ".hastable" files get created? What > could be the reason they are not created though gz files are there > > -Vivek > > From: Vivek Mishra > Sent: 24 November 2012 11:52 > To: user@hive.apache.org<mailto:user@hive.apache.org> > Subject: MapJoin error: .hashtable file not found > > Hi, > > I am trying to run a MapJoin query and somehow getting below error. I can > see that file is not there in specified directory. Did some debugging but > no luck as well. > > Here is the error: > > nOperator:Load back 1 hashtable file from tmp file > uri:c:/hadoop/hdfs/mapred/local/taskTracker/distcache/-6923915657555089159_-359650783_821098197/localhost/tmp/hive-vivek.mishra/hive_2012-11-24_11-12-30_853_4076782611151037010/-mr-10004/HashTable-Stage-5/Stage-5.tar.gz/MapJoin-mapfile10--.hashtable > 2012-11-24 11:13:45,389 ERROR > org.apache.hadoop.hive.ql.exec.MapJoinOperator: Load Distributed Cache Error > 2012-11-24 11:13:45,390 FATAL ExecMapper: > org.apache.hadoop.hive.ql.metadata.HiveException: > c:\hadoop\hdfs\mapred\local\taskTracker\distcache\-6923915657555089159_-359650783_821098197\localhost\tmp\hive-vivek.mishra\hive_2012-11-24_11-12-30_853_4076782611151037010\-mr-10004\HashTable-Stage-5\Stage-5.tar.gz\MapJoin-mapfile10--.hashtable > (The system cannot find the file specified) > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:198) >at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:212) > at > org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1321) > at > org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1325) >at > org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1325) > at > org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1325) > at > org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:495) > at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:143) >at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436) >at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372) > at org.apache.hadoop.mapred.Child$4.run(Child.java:271) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) >at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135) >at org.apache.hadoop.mapred.Child.main(Child.java:265) > > > Also, i have verified there is no such OOM issue as well. > > > Any idea? > > > -Vivek > > > > Neustar VP and Impetus CEO to present on ‘Innovative information services > powered by Cloud and Big Data technologies’at Cloud Expo - Santa Clara, Nov > 6th. http://www.impetus.com/events#2. > > Check out Impetus contribution to build Luminar - a new business unit at > Entravision. http://lf1.me/MS/ > > > NOTE: This message may contain information that is confidential, > proprietary, privileged or otherwise protected by law. The message is > intended solely for the named addressee. If received in error, please > destroy and notify the sender. Any use of this email is prohibited when > received in error. Impetus does not represent, warrant and/or guarantee, > that the integrity of this communication has been maintained nor that the > communication is free of errors, virus, interception or interference. > > > > Neustar VP and Impetus CEO to present on ‘Innova
RE: MapJoin error: .hashtable file not found
Hi, I am using 0.9.0 hive with Windows 2008 server. I did try debugging into code, but no luck to understand: " Can somebody exaplain how exactly such ".hastable" files get created? What could be the reason they are not created though gz files are there? " -Vivek From: Mark Grover [grover.markgro...@gmail.com] Sent: 28 November 2012 11:41 To: user@hive.apache.org Subject: Re: MapJoin error: .hashtable file not found Vivek, What version of Hive are you using? And, on what OS? Mark On Tue, Nov 27, 2012 at 9:34 PM, Vivek Mishra mailto:vivek.mis...@impetus.co.in>> wrote: Any pointers? Any help? Can somebody exaplain how exactly such ".hastable" files get created? What could be the reason they are not created though gz files are there -Vivek From: Vivek Mishra Sent: 24 November 2012 11:52 To: user@hive.apache.org<mailto:user@hive.apache.org> Subject: MapJoin error: .hashtable file not found Hi, I am trying to run a MapJoin query and somehow getting below error. I can see that file is not there in specified directory. Did some debugging but no luck as well. Here is the error: nOperator:Load back 1 hashtable file from tmp file uri:c:/hadoop/hdfs/mapred/local/taskTracker/distcache/-6923915657555089159_-359650783_821098197/localhost/tmp/hive-vivek.mishra/hive_2012-11-24_11-12-30_853_4076782611151037010/-mr-10004/HashTable-Stage-5/Stage-5.tar.gz/MapJoin-mapfile10--.hashtable 2012-11-24 11:13:45,389 ERROR org.apache.hadoop.hive.ql.exec.MapJoinOperator: Load Distributed Cache Error 2012-11-24 11:13:45,390 FATAL ExecMapper: org.apache.hadoop.hive.ql.metadata.HiveException: c:\hadoop\hdfs\mapred\local\taskTracker\distcache\-6923915657555089159_-359650783_821098197\localhost\tmp\hive-vivek.mishra\hive_2012-11-24_11-12-30_853_4076782611151037010\-mr-10004\HashTable-Stage-5\Stage-5.tar.gz\MapJoin-mapfile10--.hashtable (The system cannot find the file specified) at org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:198) at org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:212) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1321) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1325) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1325) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1325) at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:495) at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:143) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372) at org.apache.hadoop.mapred.Child$4.run(Child.java:271) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135) at org.apache.hadoop.mapred.Child.main(Child.java:265) Also, i have verified there is no such OOM issue as well. Any idea? -Vivek Neustar VP and Impetus CEO to present on ‘Innovative information services powered by Cloud and Big Data technologies’at Cloud Expo - Santa Clara, Nov 6th. http://www.impetus.com/events#2. Check out Impetus contribution to build Luminar - a new business unit at Entravision. http://lf1.me/MS/ NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference. Neustar VP and Impetus CEO to present on ‘Innovative information services powered by Cloud and Big Data technologies’at Cloud Expo - Santa Clara, Nov 6th. http://www.impetus.com/events#2. Check out Impetus contribution to build Luminar - a new business unit at Entravision. http://lf1.me/MS/ NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication
Re: MapJoin error: .hashtable file not found
Vivek, What version of Hive are you using? And, on what OS? Mark On Tue, Nov 27, 2012 at 9:34 PM, Vivek Mishra wrote: > Any pointers? Any help? > > Can somebody exaplain how exactly such ".hastable" files get created? What > could be the reason they are not created though gz files are there > > -Vivek > > From: Vivek Mishra > Sent: 24 November 2012 11:52 > To: user@hive.apache.org > Subject: MapJoin error: .hashtable file not found > > Hi, > > I am trying to run a MapJoin query and somehow getting below error. I can > see that file is not there in specified directory. Did some debugging but > no luck as well. > > Here is the error: > > nOperator:Load back 1 hashtable file from tmp file > uri:c:/hadoop/hdfs/mapred/local/taskTracker/distcache/-6923915657555089159_-359650783_821098197/localhost/tmp/hive-vivek.mishra/hive_2012-11-24_11-12-30_853_4076782611151037010/-mr-10004/HashTable-Stage-5/Stage-5.tar.gz/MapJoin-mapfile10--.hashtable > 2012-11-24 11:13:45,389 ERROR > org.apache.hadoop.hive.ql.exec.MapJoinOperator: Load Distributed Cache Error > 2012-11-24 11:13:45,390 FATAL ExecMapper: > org.apache.hadoop.hive.ql.metadata.HiveException: > c:\hadoop\hdfs\mapred\local\taskTracker\distcache\-6923915657555089159_-359650783_821098197\localhost\tmp\hive-vivek.mishra\hive_2012-11-24_11-12-30_853_4076782611151037010\-mr-10004\HashTable-Stage-5\Stage-5.tar.gz\MapJoin-mapfile10--.hashtable > (The system cannot find the file specified) > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:198) >at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:212) > at > org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1321) > at > org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1325) >at > org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1325) > at > org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1325) > at > org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:495) > at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:143) >at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436) >at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372) > at org.apache.hadoop.mapred.Child$4.run(Child.java:271) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) >at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135) >at org.apache.hadoop.mapred.Child.main(Child.java:265) > > > Also, i have verified there is no such OOM issue as well. > > > Any idea? > > > -Vivek > > > > Neustar VP and Impetus CEO to present on ‘Innovative information services > powered by Cloud and Big Data technologies’at Cloud Expo - Santa Clara, Nov > 6th. http://www.impetus.com/events#2. > > Check out Impetus contribution to build Luminar - a new business unit at > Entravision. http://lf1.me/MS/ > > > NOTE: This message may contain information that is confidential, > proprietary, privileged or otherwise protected by law. The message is > intended solely for the named addressee. If received in error, please > destroy and notify the sender. Any use of this email is prohibited when > received in error. Impetus does not represent, warrant and/or guarantee, > that the integrity of this communication has been maintained nor that the > communication is free of errors, virus, interception or interference. > > > > Neustar VP and Impetus CEO to present on ‘Innovative information services > powered by Cloud and Big Data technologies’at Cloud Expo - Santa Clara, Nov > 6th. http://www.impetus.com/events#2. > > Check out Impetus contribution to build Luminar - a new business unit at > Entravision. http://lf1.me/MS/ > > > NOTE: This message may contain information that is confidential, > proprietary, privileged or otherwise protected by law. The message is > intended solely for the named addressee. If received in error, please > destroy and notify the sender. Any use of this email is prohibited when > received in error. Impetus does not represent, warrant and/or guarantee, > that the integrity of this communication has been maintained nor that the > communication is free of errors, virus, interception or interference. >
RE: MapJoin error: .hashtable file not found
Any pointers? Any help? Can somebody exaplain how exactly such ".hastable" files get created? What could be the reason they are not created though gz files are there -Vivek From: Vivek Mishra Sent: 24 November 2012 11:52 To: user@hive.apache.org Subject: MapJoin error: .hashtable file not found Hi, I am trying to run a MapJoin query and somehow getting below error. I can see that file is not there in specified directory. Did some debugging but no luck as well. Here is the error: nOperator:Load back 1 hashtable file from tmp file uri:c:/hadoop/hdfs/mapred/local/taskTracker/distcache/-6923915657555089159_-359650783_821098197/localhost/tmp/hive-vivek.mishra/hive_2012-11-24_11-12-30_853_4076782611151037010/-mr-10004/HashTable-Stage-5/Stage-5.tar.gz/MapJoin-mapfile10--.hashtable 2012-11-24 11:13:45,389 ERROR org.apache.hadoop.hive.ql.exec.MapJoinOperator: Load Distributed Cache Error 2012-11-24 11:13:45,390 FATAL ExecMapper: org.apache.hadoop.hive.ql.metadata.HiveException: c:\hadoop\hdfs\mapred\local\taskTracker\distcache\-6923915657555089159_-359650783_821098197\localhost\tmp\hive-vivek.mishra\hive_2012-11-24_11-12-30_853_4076782611151037010\-mr-10004\HashTable-Stage-5\Stage-5.tar.gz\MapJoin-mapfile10--.hashtable (The system cannot find the file specified) at org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:198) at org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:212) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1321) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1325) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1325) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1325) at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:495) at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:143) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372) at org.apache.hadoop.mapred.Child$4.run(Child.java:271) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135) at org.apache.hadoop.mapred.Child.main(Child.java:265) Also, i have verified there is no such OOM issue as well. Any idea? -Vivek Neustar VP and Impetus CEO to present on ‘Innovative information services powered by Cloud and Big Data technologies’at Cloud Expo - Santa Clara, Nov 6th. http://www.impetus.com/events#2. Check out Impetus contribution to build Luminar - a new business unit at Entravision. http://lf1.me/MS/ NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference. Neustar VP and Impetus CEO to present on ‘Innovative information services powered by Cloud and Big Data technologies’at Cloud Expo - Santa Clara, Nov 6th. http://www.impetus.com/events#2. Check out Impetus contribution to build Luminar - a new business unit at Entravision. http://lf1.me/MS/ NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
