Thanks for the pointers regarding 1).
Any ideas on 2) - namely why only the deferenced schema is available and how to
get a hold of the actual user declaration?
Cheers and Merry Christmas!
On 24/12/2013 1:05 AM, Cheolsoo Park wrote:
As for #1, pushdownProject() is called only if it's applicable-
https://github.com/apache/pig/blob/trunk/src/org/apache/pig/newplan/optimizer/PlanOptimizer.java#L108
Set a breakpoint in ColumnMapKeyPrune.java and see whether check() returns
true or false-
https://github.com/apache/pig/blob/trunk/src/org/apache/pig/newplan/logical/rules/ColumnMapKeyPrune.java#L85
It probably returns false in your case, and that's why your
pushProjection() is never called.
On Thu, Dec 19, 2013 at 6:09 AM, Costin Leau <[email protected]> wrote:
Forgot to specify the aforementioned thread [1]
[1] http://www.mail-archive.com/[email protected]/msg06285.html
On 19/12/2013 4:08 PM, Costin Leau wrote:
Hi,
I'm trying to get a hold of the schema specified for a loader through
'AS' using Apache Pig 0.12 :
A = LOAD 'pig/tupleartists' USING MyStorage() AS (name: chararray, links:
(url:chararray, picture:chararray));
B = FOREACH A GENERATE name, links.url;
DUMP B;
1.
My loader implements LoadPushDown#pushProjection() which does not seem to
be called at all (tried breakpoints,
System.out - nothing). The API docs and this thread [1] suggest it should
be call yet in my tests (using a local
PigServer) this does not happen. Am I missing something?
2.
As an alternative, I'm loading the POStore objects (from pig.map.store
and pig.reduce.store) but the schema that I'm
getting is incorrect, namely:
"(name: chararray, url: charray)" without any mention of the "links"
field. Is there any way to recreate/retrieve the
actual schema defined by the user or at least determine which fields are
nested ("links.url") as oppose to the top level
ones ("name")?
Thanks,
--
Costin
--
Costin
