Re: Error ERROR 2088: Fetch failed. Couldn't retrieve result happened during HCatLoader() then DUMP
Hi Lulynn,
I've opened PIG-4238 and HIVE-8484.
--Lorand
On 16/10/14 04:55, lulynn_2008 wrote:
Hi Lorand,
Have you opened JIRA for Hive side?
Thanks
At 2014-10-15 15:53:41, Lorand Bendig lben...@gmail.com wrote:
Hi Rohini and Lulynn,
The exception occurs when DUMP is executed:
X = load 'junit_unparted_basic' using
org.apache.hive.hcatalog.pig.HCatLoader();
dump X
I made some tests on Hive 0.14 / Pig 0.14 and found a bug.
When HCatLoader#setLocation(String location, Job job) is called on the
Hive side, it
can't set the outputschema because
HCatUtil#checkJobContextIfRunningFromBackend(job)
always returns false. This is because in fetch mode we don't have a
mapred.task.id.
A null outputschema will raise an exception when
HCatBaseLoader#getNext() calls
PigHCatUtil#transformToTuple(hr, outputSchema);
I can think of two solutions:
1. At Hive side:
If a job is fetchable then we set
PigImplConstants.CONVERTED_TO_FETCH in the PigContext,
which we could check in
HCatUtil#checkJobContextIfRunningFromBackend(job) .
2. At Pig side:
Set a dummy mapred.task.id when fetch is initialized so that
HCatUtil#checkJobContextIfRunningFromBackend(job) can return true.
I'd go with the first one.
Thanks,
Lorand
On 15/10/14 06:55, Rohini Palaniswamy wrote:
Lorand,
Isn't fetch optimization supposed to be only for DUMP and not STORE ?
-Rohini
On Tue, Oct 14, 2014 at 6:47 PM, lulynn_2008 lulynn_2...@163.com wrote:
Hi Lorand,
The query run fine is I disable fetch. Thanks for your help. Could you
tell why we need to disable fetch?
BTW, I was using pig-0.13.0 and hive-0.13.1.
At 2014-10-14 21:03:13, Lorand Bendig lben...@gmail.com wrote:
Hi,
Which Pig and Hive versions do you use? Does the query run fine if you
disable fetch (set opt.fetch false) ?
Thanks,
Lorand
On 14/10/14 10:50, lulynn_2008 wrote:
Hi All,
I was running HCatStore and HCatLoader in pig grunt. But encounter
ERROR 2088: Fetch failed. Couldn't retrieve result. Please help give a
glance and give your suggestions. Thanks.
Test case:
1. Create table in hive:
create table junit_unparted_basic(a int, b string) stored as RCFILE
tblproperties('hcat.isd'='org.apache.hive.hcatalog.rcfile.RCFileInputDriver','hcat.osd'='org.apache.hive.hcatalog.rcfile.RCFileOutputDriver');
2. copy basic.input.data file into hdfs, here is the content in file:
1S1S
1S2S
1S3S
2S1S
2S2S
2S3S
3S1S
3S2S
3S3S
3. run Pig: pig -useHCatalog
4. grunt A = load 'basic.input.data' as (a:int, b:chararray);
5. grunt store A into 'junit_unparted_basic' using
org.apache.hive.hcatalog.pig.HCatStorer();
6. X = load 'junit_unparted_basic' using
org.apache.hive.hcatalog.pig.HCatLoader();
7. grunt dump X
Error Log:
Pig Stack Trace
---
ERROR 2088: Fetch failed. Couldn't retrieve result
org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable
to open iterator for alias X
at org.apache.pig.PigServer.openIterator(PigServer.java:912)
at
org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:752)
at
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:372)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:228)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:203)
at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:66)
at org.apache.pig.Main.run(Main.java:542)
at org.apache.pig.Main.main(Main.java:156)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:94)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:619)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: org.apache.pig.PigException: ERROR 1002: Unable to store
alias X
at org.apache.pig.PigServer.storeEx(PigServer.java:1015)
at org.apache.pig.PigServer.store(PigServer.java:974)
at org.apache.pig.PigServer.openIterator(PigServer.java:887)
... 12 more
Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR
2088: Fetch failed. Couldn't retrieve result
at
org.apache.pig.backend.hadoop.executionengine.fetch.FetchLauncher.runPipeline(FetchLauncher.java:180)
at
org.apache.pig.backend.hadoop.executionengine.fetch.FetchLauncher.launchPig(FetchLauncher.java:81)
at
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:275)
at org.apache.pig.PigServer.launchPlan(PigServer.java:1367)
at
org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1352)
at
Re: Error ERROR 2088: Fetch failed. Couldn't retrieve result happened during HCatLoader() then DUMP
Hi Rohini and Lulynn,
The exception occurs when DUMP is executed:
X = load 'junit_unparted_basic' using
org.apache.hive.hcatalog.pig.HCatLoader();
dump X
I made some tests on Hive 0.14 / Pig 0.14 and found a bug.
When HCatLoader#setLocation(String location, Job job) is called on the
Hive side, it
can't set the outputschema because
HCatUtil#checkJobContextIfRunningFromBackend(job)
always returns false. This is because in fetch mode we don't have a
mapred.task.id.
A null outputschema will raise an exception when
HCatBaseLoader#getNext() calls
PigHCatUtil#transformToTuple(hr, outputSchema);
I can think of two solutions:
1. At Hive side:
If a job is fetchable then we set
PigImplConstants.CONVERTED_TO_FETCH in the PigContext,
which we could check in
HCatUtil#checkJobContextIfRunningFromBackend(job) .
2. At Pig side:
Set a dummy mapred.task.id when fetch is initialized so that
HCatUtil#checkJobContextIfRunningFromBackend(job) can return true.
I'd go with the first one.
Thanks,
Lorand
On 15/10/14 06:55, Rohini Palaniswamy wrote:
Lorand,
Isn't fetch optimization supposed to be only for DUMP and not STORE ?
-Rohini
On Tue, Oct 14, 2014 at 6:47 PM, lulynn_2008 lulynn_2...@163.com wrote:
Hi Lorand,
The query run fine is I disable fetch. Thanks for your help. Could you
tell why we need to disable fetch?
BTW, I was using pig-0.13.0 and hive-0.13.1.
At 2014-10-14 21:03:13, Lorand Bendig lben...@gmail.com wrote:
Hi,
Which Pig and Hive versions do you use? Does the query run fine if you
disable fetch (set opt.fetch false) ?
Thanks,
Lorand
On 14/10/14 10:50, lulynn_2008 wrote:
Hi All,
I was running HCatStore and HCatLoader in pig grunt. But encounter
ERROR 2088: Fetch failed. Couldn't retrieve result. Please help give a
glance and give your suggestions. Thanks.
Test case:
1. Create table in hive:
create table junit_unparted_basic(a int, b string) stored as RCFILE
tblproperties('hcat.isd'='org.apache.hive.hcatalog.rcfile.RCFileInputDriver','hcat.osd'='org.apache.hive.hcatalog.rcfile.RCFileOutputDriver');
2. copy basic.input.data file into hdfs, here is the content in file:
1S1S
1S2S
1S3S
2S1S
2S2S
2S3S
3S1S
3S2S
3S3S
3. run Pig: pig -useHCatalog
4. grunt A = load 'basic.input.data' as (a:int, b:chararray);
5. grunt store A into 'junit_unparted_basic' using
org.apache.hive.hcatalog.pig.HCatStorer();
6. X = load 'junit_unparted_basic' using
org.apache.hive.hcatalog.pig.HCatLoader();
7. grunt dump X
Error Log:
Pig Stack Trace
---
ERROR 2088: Fetch failed. Couldn't retrieve result
org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable
to open iterator for alias X
at org.apache.pig.PigServer.openIterator(PigServer.java:912)
at
org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:752)
at
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:372)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:228)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:203)
at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:66)
at org.apache.pig.Main.run(Main.java:542)
at org.apache.pig.Main.main(Main.java:156)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:94)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:619)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: org.apache.pig.PigException: ERROR 1002: Unable to store
alias X
at org.apache.pig.PigServer.storeEx(PigServer.java:1015)
at org.apache.pig.PigServer.store(PigServer.java:974)
at org.apache.pig.PigServer.openIterator(PigServer.java:887)
... 12 more
Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR
2088: Fetch failed. Couldn't retrieve result
at
org.apache.pig.backend.hadoop.executionengine.fetch.FetchLauncher.runPipeline(FetchLauncher.java:180)
at
org.apache.pig.backend.hadoop.executionengine.fetch.FetchLauncher.launchPig(FetchLauncher.java:81)
at
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:275)
at org.apache.pig.PigServer.launchPlan(PigServer.java:1367)
at
org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1352)
at org.apache.pig.PigServer.storeEx(PigServer.java:1011)
... 14 more
Re: Error ERROR 2088: Fetch failed. Couldn't retrieve result happened during HCatLoader() then DUMP
Hi,
Which Pig and Hive versions do you use? Does the query run fine if you
disable fetch (set opt.fetch false) ?
Thanks,
Lorand
On 14/10/14 10:50, lulynn_2008 wrote:
Hi All,
I was running HCatStore and HCatLoader in pig grunt. But encounter ERROR 2088:
Fetch failed. Couldn't retrieve result. Please help give a glance and give your
suggestions. Thanks.
Test case:
1. Create table in hive:
create table junit_unparted_basic(a int, b string) stored as RCFILE
tblproperties('hcat.isd'='org.apache.hive.hcatalog.rcfile.RCFileInputDriver','hcat.osd'='org.apache.hive.hcatalog.rcfile.RCFileOutputDriver');
2. copy basic.input.data file into hdfs, here is the content in file:
1S1S
1S2S
1S3S
2S1S
2S2S
2S3S
3S1S
3S2S
3S3S
3. run Pig: pig -useHCatalog
4. grunt A = load 'basic.input.data' as (a:int, b:chararray);
5. grunt store A into 'junit_unparted_basic' using
org.apache.hive.hcatalog.pig.HCatStorer();
6. X = load 'junit_unparted_basic' using
org.apache.hive.hcatalog.pig.HCatLoader();
7. grunt dump X
Error Log:
Pig Stack Trace
---
ERROR 2088: Fetch failed. Couldn't retrieve result
org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open
iterator for alias X
at org.apache.pig.PigServer.openIterator(PigServer.java:912)
at
org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:752)
at
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:372)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:228)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:203)
at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:66)
at org.apache.pig.Main.run(Main.java:542)
at org.apache.pig.Main.main(Main.java:156)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:94)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:619)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: org.apache.pig.PigException: ERROR 1002: Unable to store alias X
at org.apache.pig.PigServer.storeEx(PigServer.java:1015)
at org.apache.pig.PigServer.store(PigServer.java:974)
at org.apache.pig.PigServer.openIterator(PigServer.java:887)
... 12 more
Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 2088:
Fetch failed. Couldn't retrieve result
at
org.apache.pig.backend.hadoop.executionengine.fetch.FetchLauncher.runPipeline(FetchLauncher.java:180)
at
org.apache.pig.backend.hadoop.executionengine.fetch.FetchLauncher.launchPig(FetchLauncher.java:81)
at
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:275)
at org.apache.pig.PigServer.launchPlan(PigServer.java:1367)
at
org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1352)
at org.apache.pig.PigServer.storeEx(PigServer.java:1011)
... 14 more
Re: Re: Error ERROR 2088: Fetch failed. Couldn't retrieve result happened during HCatLoader() then DUMP
Lorand,
Isn't fetch optimization supposed to be only for DUMP and not STORE ?
-Rohini
On Tue, Oct 14, 2014 at 6:47 PM, lulynn_2008 lulynn_2...@163.com wrote:
Hi Lorand,
The query run fine is I disable fetch. Thanks for your help. Could you
tell why we need to disable fetch?
BTW, I was using pig-0.13.0 and hive-0.13.1.
At 2014-10-14 21:03:13, Lorand Bendig lben...@gmail.com wrote:
Hi,
Which Pig and Hive versions do you use? Does the query run fine if you
disable fetch (set opt.fetch false) ?
Thanks,
Lorand
On 14/10/14 10:50, lulynn_2008 wrote:
Hi All,
I was running HCatStore and HCatLoader in pig grunt. But encounter
ERROR 2088: Fetch failed. Couldn't retrieve result. Please help give a
glance and give your suggestions. Thanks.
Test case:
1. Create table in hive:
create table junit_unparted_basic(a int, b string) stored as RCFILE
tblproperties('hcat.isd'='org.apache.hive.hcatalog.rcfile.RCFileInputDriver','hcat.osd'='org.apache.hive.hcatalog.rcfile.RCFileOutputDriver');
2. copy basic.input.data file into hdfs, here is the content in file:
1S1S
1S2S
1S3S
2S1S
2S2S
2S3S
3S1S
3S2S
3S3S
3. run Pig: pig -useHCatalog
4. grunt A = load 'basic.input.data' as (a:int, b:chararray);
5. grunt store A into 'junit_unparted_basic' using
org.apache.hive.hcatalog.pig.HCatStorer();
6. X = load 'junit_unparted_basic' using
org.apache.hive.hcatalog.pig.HCatLoader();
7. grunt dump X
Error Log:
Pig Stack Trace
---
ERROR 2088: Fetch failed. Couldn't retrieve result
org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable
to open iterator for alias X
at org.apache.pig.PigServer.openIterator(PigServer.java:912)
at
org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:752)
at
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:372)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:228)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:203)
at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:66)
at org.apache.pig.Main.run(Main.java:542)
at org.apache.pig.Main.main(Main.java:156)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:94)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:619)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: org.apache.pig.PigException: ERROR 1002: Unable to store
alias X
at org.apache.pig.PigServer.storeEx(PigServer.java:1015)
at org.apache.pig.PigServer.store(PigServer.java:974)
at org.apache.pig.PigServer.openIterator(PigServer.java:887)
... 12 more
Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR
2088: Fetch failed. Couldn't retrieve result
at
org.apache.pig.backend.hadoop.executionengine.fetch.FetchLauncher.runPipeline(FetchLauncher.java:180)
at
org.apache.pig.backend.hadoop.executionengine.fetch.FetchLauncher.launchPig(FetchLauncher.java:81)
at
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:275)
at org.apache.pig.PigServer.launchPlan(PigServer.java:1367)
at
org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1352)
at org.apache.pig.PigServer.storeEx(PigServer.java:1011)
... 14 more
