To clarify,
it fails for 0.12.1 . I did not try pig versions before that.
Sorry for any confusion.
Regards,
Debabrata Pani
On Sat, Nov 21, 2015 at 9:55 AM, Debabrata Pani <android.p...@gmail.com>
wrote:
> This seems to be a problem with apache pig upto 0.12.1
>
> I tried the script with pig 0.13 and it does not throw any errors
>
> The script evaluated:
>
> a = load 'test1.txt' using PigStorage(',') as
> (A:chararray,B:chararray,C:chararray);
> b = group a by (A,B);
> c = foreach b {
> asdf = filter $1 by 1==1;
> generate COUNT_STAR($1) as TARGET:int;
> };
>
> DESCRIBE c;
> store c into 'output' USING PigStorage(',');
> d = limit c 10;
> describe d;
> e = foreach d generate TARGET;
> DESCRIBE e;
> store e into 'output-e' USING PigStorage(','); -- store command necessary for
> explain to work
>
>
>
> and instead of running the entire program, I just tried to do an "explain"
> of the script .
> The explain command :
>
> pig -e 'explain -script testpigschema.pig'
>
> EXPLAIN fails in pig upto 0.12.1
> But goes through for 0.13.0
>
> Unfortunately, this does not really solve the problem for you except
> hinting that this may be a bug in apache pig ?
>
> Regards,
> Debabrata Pani
>
> On Mon, Nov 16, 2015 at 9:24 AM, Arvind S <arvind18...@gmail.com> wrote:
>
>> does not seem to be a issue in pig 0.15 .. (tested in local mode only as
>> of
>> now)
>>
>> a = load '/tmp/test/test.txt' using PigStorage(',') as
>> (A:chararray,B:chararray,C:chararray);
>> b = group a by (A,B);
>> c = foreach b {
>> asdf = filter $1 by (1==1);
>> generate COUNT_STAR($1) as TARGET;
>> };
>> d = limit c 10;
>> e = foreach d generate TARGET;
>> dump e;
>>
>> end output ...
>> (1)
>>
>>
>> *Cheers !!*
>> Arvind
>>
>> On Sat, Nov 14, 2015 at 12:18 AM, Christopher Maier <
>> christopher.ma...@gm.com> wrote:
>>
>> > Hi,
>> >
>> > I haven't received a response on this, has anyone had a chance to
>> > reproduce the error?
>> >
>> > Thanks,
>> > Kit
>> >
>> > From: Christopher Maier
>> > Sent: Tuesday, October 20, 2015 4:02 PM
>> > To: 'user@pig.apache.org' <user@pig.apache.org>
>> > Subject: Schema changes based on subquery
>> >
>> > Hi,
>> >
>> > I am getting the wrong counts from Pig for a certain query. I have
>> > simplified the query to what's below, which shows as a failure instead
>> of a
>> > wrong count.
>> >
>> > Why does the first line of the subquery cause the output schema to
>> revert
>> > to be the same as the input schema? This line should not have any
>> impact on
>> > the output.
>> >
>> > (I've removed some of the extra logging output.)
>> >
>> > pig -version
>> > Apache Pig version 0.12.0 (rexported)
>> > compiled Oct 26 2014, 23:43:04
>> >
>> > Query
>> > grunt> a = load 'test1.txt' using PigStorage(',') as
>> > (A:chararray,B:chararray,C:chararray);
>> > grunt> b = group a by (A,B);
>> > grunt> c = foreach b {
>> > >> asdf = filter $1 by (1==1);
>> > >> generate COUNT_STAR($1) as TARGET;
>> > >> };
>> > grunt> d = limit c 10;
>> >
>> > Values
>> > grunt> dump a;
>> > (a,b,c)
>> > grunt> dump b;
>> > ((a,b),{(a,b,c)})
>> > grunt> dump c;
>> > (1)
>> > grunt> dump d;
>> > (1)
>> >
>> > Schema 'describe' at each step looks good
>> > grunt> describe a;
>> > a: {A: chararray,B: chararray,C: chararray}
>> > grunt> describe b;
>> > b: {group: (A: chararray,B: chararray),a: {(A: chararray,B: chararray,C:
>> > chararray)}}
>> > grunt> describe c;
>> > c: {TARGET: long}
>> > grunt> describe d;
>> > d: {TARGET: long}
>> >
>> > Attempted next step fails
>> > grunt> e = foreach d generate TARGET;
>> > <line 8, column 23> Invalid field projection. Projected field [TARGET]
>> > does not exist in schema: A:chararray,B:chararray,C:chararray.
>> >
>> > Progress of real schema through query
>> > grunt> z = foreach a generate FAKE;
>> > <line 8, column 23> Invalid field projection. Projected field [FAKE]
>> does
>> > not exist in schema: A:chararray,B:chararray,C:chararray.
>> > grunt> z = foreach b generate FAKE;
>> > <line 8, column 23> Invalid field projection. Projected field [FAKE]
>> does
>> > not exist in schema:
>> >
>> group:tuple(A:chararray,B:chararray),a:bag{:tuple(A:chararray,B:chararray,C:chararray)}.
>> > grunt> z = foreach c generate FAKE;
>> > <line 8, column 23> Invalid field projection. Projected field [FAKE]
>> does
>> > not exist in schema: TARGET:long.
>> > grunt> z = foreach d generate FAKE;
>> > <line 8, column 23> Invalid field projection. Projected field [FAKE]
>> does
>> > not exist in schema: A:chararray,B:chararray,C:chararray.
>> >
>> > Alternate query shows no error
>> > grunt> c = foreach b {
>> > >> generate COUNT_STAR($1) as TARGET;
>> > >> };
>> > grunt> d = limit c 10;
>> > grunt> e = foreach d generate TARGET;
>> > grunt> dump e;
>> > (1)
>> >
>> > Thanks,
>> > Kit Maier
>> >
>> >
>> >
>> > Nothing in this message is intended to constitute an electronic
>> signature
>> > unless a specific statement to the contrary is included in this message.
>> >
>> > Confidentiality Note: This message is intended only for the person or
>> > entity to which it is addressed. It may contain confidential and/or
>> > privileged material. Any review, transmission, dissemination or other
>> use,
>> > or taking of any action in reliance upon this message by persons or
>> > entities other than the intended recipient is prohibited and may be
>> > unlawful. If you received this message in error, please contact the
>> sender
>> > and delete it from your computer.
>> >
>>
>
>
