Thanks for the prompt reply. As you mentioned optimization is in its
developing stage, does it mean optimization framework is not complete or
only rules are in developing stage? In addition to that, I would really
appreciate if you could give a rough idea when the patch will be available
and what functionality will it contain?
Actually, I had attached seven files in my previous mail to reproduce the
bug including the error log but as you couldn't find them I am inlining all
the attachments :
*
My patch:* (To enable the optimization)
Index: src/org/apache/pig/PigServer.java
===================================================================
--- src/org/apache/pig/PigServer.java (revision 951297)
+++ src/org/apache/pig/PigServer.java (working copy)
@@ -179,6 +179,11 @@
aggregateWarning =
"true".equalsIgnoreCase(pigContext.getProperties().getProperty("aggregate.warning"));
isMultiQuery =
"true".equalsIgnoreCase(pigContext.getProperties().getProperty("opt.multiquery","true"));
+
getPigContext().getProperties().setProperty("pig.usenewlogicalplan",
"true");
+ log.info(
+ "---------> pig.usenewlogicalplan set to " +
+
getPigContext().getProperties().getProperty("pig.usenewlogicalplan",
"false") +
+ " in PigServer" );
if (connect) {
pigContext.connect();
*Script 1: *
A = load '/home/pig/exfile1' USING PigStorage(' ') as (x:int,y:int);
B = Group A by x;
dump B;
* Script 2:*
A = load '/home/pig/exfile1' USING PigStorage(',') as (a1:int,a2:int);
B = load '/home/pig/exfile1' USING PigStorage(',') as (b1:int,b2:int);
C = JOIN A by a1, B by b1;
dump C;
*exfile1:*
1,5
Please let me know if you have any further questions.
Thanks,
Swati
On Sun, Jun 27, 2010 at 9:32 PM, Daniel Dai <dai...@gmail.com> wrote:
> Swati,
> New logical plan is half way done so it is not surprising to see exceptions
> at current stage. We are actively developing it and will deliver patch
> shortly. Meanwhile, please attach the problematic scripts (I didn't see it
> in your mail) so we can make sure those exceptions are addressed.
>
> Thanks,
> Daniel
>
>
> From: Swati Jain
> Sent: Sunday, June 27, 2010 7:07 PM
> To: pig-dev@hadoop.apache.org
> Subject: Bug in new logical optimizer framework?
>
>
> Folks,
>
> Posting on the dev since this is regarding the new logical plan
> optimization framework which is not enabled yet. I was interested in playing
> around with the new optimization framework and try adding some simple rules
> to it.
>
> I have attached two simple programs which do not work when the new logical
> optimization framework is enabled (they work when it is disabled). My
> changes to enable the new optimizer are pretty straightforward and the diff
> on branch-0.7 are attached (I just set the appropriate property to true). I
> have attached two very simple scripts both of which raise an exception (in
> local mode of execution) "java.io.IOException: Type mismatch in key from
> map: expected org.apache.pig.impl.io.NullableIntWritable, recieved
> org.apache.pig.impl.io.NullableBytesWritable" if there is atleast 1 row to
> be output. The error goes away if I replace "DUMP" with "EXPLAIN"
> (presumably because the bug manifests during plan execution). It would be
> great if someone could throw some light on this issue or give pointers on
> workarounds or ways to fix this. I have not filed a JIRA for the above,
> please let me know if I should.
>
> Also, it would be great to get some guidance on the state of the new
> optimizer wrt testing (I do understand it is not GA ready since it is
> disabled by default) and whether it is too early to start playing around
> with adding new rules.
>
> Thanks
> Swati
>
