[jira] [Commented] (PIG-3608) ClassCastException when looking up a value from AvroMapWrapper using a Utf8 key
[
https://issues.apache.org/jira/browse/PIG-3608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13840940#comment-13840940
]
Rohini Palaniswamy commented on PIG-3608:
-
With the testcase in PIG-3420, the key in AvroMapper was utf8 (i.e
isUtf8key=true), but the pig map had the key passed as String. I don't know in
what case you are getting ClassCastException. The full stack trace and pig
script would help to know what is happening.
> ClassCastException when looking up a value from AvroMapWrapper using a Utf8
> key
> ---
>
> Key: PIG-3608
> URL: https://issues.apache.org/jira/browse/PIG-3608
> Project: Pig
> Issue Type: Bug
> Components: impl
>Affects Versions: 0.12.0
>Reporter: Richard Ding
>Assignee: Richard Ding
>Priority: Minor
> Attachments: PIG-3608.patch
>
>
> One got the following exception:
> {code}
> java.lang.ClassCastException: org.apache.avro.util.Utf8 incompatible with
> java.lang.String
> at
> org.apache.pig.impl.util.avro.AvroMapWrapper.get(AvroMapWrapper.java:80)
> {code}
> This is related to the change by PIG-3420.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (PIG-3608) ClassCastException when looking up a value from AvroMapWrapper using a Utf8 key
[
https://issues.apache.org/jira/browse/PIG-3608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13840791#comment-13840791
]
Richard Ding commented on PIG-3608:
---
Actually I have a question: should it be
{code}
if (isUtf8key) {
v = innerMap.get(key);
} else {
v = innerMap.get(new Utf8((String) key));
}
{code}
since isUft8key == true means the key is already Utf8?
> ClassCastException when looking up a value from AvroMapWrapper using a Utf8
> key
> ---
>
> Key: PIG-3608
> URL: https://issues.apache.org/jira/browse/PIG-3608
> Project: Pig
> Issue Type: Bug
> Components: impl
>Affects Versions: 0.12.0
>Reporter: Richard Ding
>Assignee: Richard Ding
>Priority: Minor
> Attachments: PIG-3608.patch
>
>
> One got the following exception:
> {code}
> java.lang.ClassCastException: org.apache.avro.util.Utf8 incompatible with
> java.lang.String
> at
> org.apache.pig.impl.util.avro.AvroMapWrapper.get(AvroMapWrapper.java:80)
> {code}
> This is related to the change by PIG-3420.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (PIG-3612) Storing schema does not work cross cluster with PigStorage and JsonStorage
[ https://issues.apache.org/jira/browse/PIG-3612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohini Palaniswamy updated PIG-3612: Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to trunk (0.13). Thanks for the review Aniket. > Storing schema does not work cross cluster with PigStorage and JsonStorage > -- > > Key: PIG-3612 > URL: https://issues.apache.org/jira/browse/PIG-3612 > Project: Pig > Issue Type: Bug >Reporter: Rohini Palaniswamy >Assignee: Rohini Palaniswamy > Fix For: 0.13.0 > > Attachments: PIG-3612-1.patch > > > java.lang.IllegalArgumentException: Wrong FS: > hdfs://cluster2-nn:8020/tmp/output/.pig_schema, > expected: hdfs://cluster1-nn1 > at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:582) > at > org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:155) > at > org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:734) > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1272) > at > org.apache.pig.backend.hadoop.datastorage.HDataStorage.isContainer(HDataStorage.java:200) > at > org.apache.pig.backend.hadoop.datastorage.HDataStorage.asElement(HDataStorage.java:128) > at > org.apache.pig.backend.hadoop.datastorage.HDataStorage.asElement(HDataStorage.java:144) > at org.apache.pig.builtin.JsonMetadata.storeSchema(JsonMetadata.java:294) > at org.apache.pig.builtin.JsonStorage.storeSchema(JsonStorage.java:274) > at -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] Subscription: PIG patch available
Issue Subscription Filter: PIG patch available (10 issues) Subscriber: pigdaily Key Summary PIG-3612Storing schema does not work cross cluster with PigStorage and JsonStorage https://issues.apache.org/jira/browse/PIG-3612 PIG-3609ClassCastException when calling compareTo method on AvroBagWrapper https://issues.apache.org/jira/browse/PIG-3609 PIG-3608ClassCastException when looking up a value from AvroMapWrapper using a Utf8 key https://issues.apache.org/jira/browse/PIG-3608 PIG-3592Should not try to create success file for non-fs schemes like hbase https://issues.apache.org/jira/browse/PIG-3592 PIG-3587add functionality for rolling over dates https://issues.apache.org/jira/browse/PIG-3587 PIG-3573Provide StoreFunc and LoadFunc for Accumulo https://issues.apache.org/jira/browse/PIG-3573 PIG-3572Fix all unit test for during build pig with Hadoop 2.X on Windows. https://issues.apache.org/jira/browse/PIG-3572 PIG-3453Implement a Storm backend to Pig https://issues.apache.org/jira/browse/PIG-3453 PIG-3441Allow Pig to use default resources from Configuration objects https://issues.apache.org/jira/browse/PIG-3441 PIG-3347Store invocation brings side effect https://issues.apache.org/jira/browse/PIG-3347 You may edit this subscription at: https://issues.apache.org/jira/secure/FilterSubscription!default.jspa?subId=13225&filterId=12322384
[jira] [Updated] (PIG-3609) ClassCastException when calling compareTo method on AvroBagWrapper
[
https://issues.apache.org/jira/browse/PIG-3609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Richard Ding updated PIG-3609:
--
Attachment: PIG-3609.patch
Attaching a patch.
> ClassCastException when calling compareTo method on AvroBagWrapper
> ---
>
> Key: PIG-3609
> URL: https://issues.apache.org/jira/browse/PIG-3609
> Project: Pig
> Issue Type: Bug
> Components: impl
>Affects Versions: 0.12.0
>Reporter: Richard Ding
>Priority: Minor
> Attachments: PIG-3609.patch
>
>
> One got the following exception when calling compareTo method on
> AvroBagWrapper with an AvroBagWrapper object:
> {code}
> java.lang.ClassCastException: org.apache.pig.impl.util.avro.AvroBagWrapper
> incompatible with java.util.Collection
> at org.apache.avro.generic.GenericData.compare(GenericData.java:786)
> at org.apache.avro.generic.GenericData.compare(GenericData.java:760)
> at
> org.apache.pig.impl.util.avro.AvroBagWrapper.compareTo(AvroBagWrapper.java:78)
> {code}
> Looking at the code, it compares objects with different types:
> {code}
> return GenericData.get().compare(theArray, o, theArray.getSchema());
> {code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (PIG-3609) ClassCastException when calling compareTo method on AvroBagWrapper
[
https://issues.apache.org/jira/browse/PIG-3609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Richard Ding updated PIG-3609:
--
Status: Patch Available (was: Open)
> ClassCastException when calling compareTo method on AvroBagWrapper
> ---
>
> Key: PIG-3609
> URL: https://issues.apache.org/jira/browse/PIG-3609
> Project: Pig
> Issue Type: Bug
> Components: impl
>Affects Versions: 0.12.0
>Reporter: Richard Ding
>Assignee: Richard Ding
>Priority: Minor
> Attachments: PIG-3609.patch
>
>
> One got the following exception when calling compareTo method on
> AvroBagWrapper with an AvroBagWrapper object:
> {code}
> java.lang.ClassCastException: org.apache.pig.impl.util.avro.AvroBagWrapper
> incompatible with java.util.Collection
> at org.apache.avro.generic.GenericData.compare(GenericData.java:786)
> at org.apache.avro.generic.GenericData.compare(GenericData.java:760)
> at
> org.apache.pig.impl.util.avro.AvroBagWrapper.compareTo(AvroBagWrapper.java:78)
> {code}
> Looking at the code, it compares objects with different types:
> {code}
> return GenericData.get().compare(theArray, o, theArray.getSchema());
> {code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Assigned] (PIG-3609) ClassCastException when calling compareTo method on AvroBagWrapper
[
https://issues.apache.org/jira/browse/PIG-3609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Richard Ding reassigned PIG-3609:
-
Assignee: Richard Ding
> ClassCastException when calling compareTo method on AvroBagWrapper
> ---
>
> Key: PIG-3609
> URL: https://issues.apache.org/jira/browse/PIG-3609
> Project: Pig
> Issue Type: Bug
> Components: impl
>Affects Versions: 0.12.0
>Reporter: Richard Ding
>Assignee: Richard Ding
>Priority: Minor
> Attachments: PIG-3609.patch
>
>
> One got the following exception when calling compareTo method on
> AvroBagWrapper with an AvroBagWrapper object:
> {code}
> java.lang.ClassCastException: org.apache.pig.impl.util.avro.AvroBagWrapper
> incompatible with java.util.Collection
> at org.apache.avro.generic.GenericData.compare(GenericData.java:786)
> at org.apache.avro.generic.GenericData.compare(GenericData.java:760)
> at
> org.apache.pig.impl.util.avro.AvroBagWrapper.compareTo(AvroBagWrapper.java:78)
> {code}
> Looking at the code, it compares objects with different types:
> {code}
> return GenericData.get().compare(theArray, o, theArray.getSchema());
> {code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (PIG-3607) PigRecordReader should report progress for each inputsplit processed
[ https://issues.apache.org/jira/browse/PIG-3607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohini Palaniswamy updated PIG-3607: Resolution: Fixed Fix Version/s: 0.13.0 Status: Resolved (was: Patch Available) Committed to trunk (0.13). Thanks Aniket for the review. > PigRecordReader should report progress for each inputsplit processed > > > Key: PIG-3607 > URL: https://issues.apache.org/jira/browse/PIG-3607 > Project: Pig > Issue Type: Bug >Affects Versions: 0.11.1 >Reporter: Rohini Palaniswamy >Assignee: Rohini Palaniswamy > Fix For: 0.13.0 > > Attachments: PIG-3607-1.patch > > > Currently progress() is called only when records are processed. In a case > where there were lot of empty input files, the task timed out and was killed > because no progress was reported. Too many empty input files are bad, but we > still don't want to fail. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (PIG-2629) Wrong Usage of Scalar which is null causes high namenode operation
[
https://issues.apache.org/jira/browse/PIG-2629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohini Palaniswamy updated PIG-2629:
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Committed to branch-0.12 (0.12.1) and trunk (0.13). Thanks Cheolsoo for the
review.
> Wrong Usage of Scalar which is null causes high namenode operation
> ---
>
> Key: PIG-2629
> URL: https://issues.apache.org/jira/browse/PIG-2629
> Project: Pig
> Issue Type: Bug
>Affects Versions: 0.8.1, 0.9.2, 0.10.0
>Reporter: Anitha Raju
>Assignee: Rohini Palaniswamy
> Fix For: 0.12.1, 0.13.0
>
> Attachments: PIG-2629-1.patch, PIG-2629-2.patch
>
>
> Hi,
> Script
> {code}
> A = LOAD 'test3.txt' AS (from:chararray);
> B = LOAD 'test2.txt'AS (source:chararray,to:chararray);
> C = FILTER A BY (from == 'temp' );
> D = FILTER B BY (source MATCHES '.*xyz*.');
> E = JOIN C by (from) left outer,D by (to);
> F = FILTER E BY (D.to IS NULL);
> dump F;
> {code}
> Inputs
> {code}
> $ cat test2.txt
> temptemp
> temptemp
> temptemp
> temptemp
> temptemp
> tepmtepm
> $ cat test3.txt |head
> temp
> temp
> temp
> temp
> temp
> temp
> tepm
> temp
> temp
> temp
> {code}
> Here I have by mistake called 'to' using 'D.to' instead of 'D::to'. The D
> relation gives null output.
> First Map Reduce job computes D which give null results.
> The MapPlan of 2nd job
> {code}
> Union[tuple] - scope-56
> |
> |---E: Local Rearrange[tuple]{chararray}(false) - scope-36
> | | |
> | | Project[chararray][0] - scope-37
> | |
> | |---C: Filter[bag] - scope-26
> | | |
> | | Equal To[boolean] - scope-29
> | | |
> | | |---Project[chararray][0] - scope-27
> | | |
> | | |---Constant(temp) - scope-28
> | |
> | |---A: New For Each(false)[bag] - scope-25
> | | |
> | | Cast[chararray] - scope-23
> | | |
> | | |---Project[bytearray][0] - scope-22
> | |
> | |---F: Filter[bag] - scope-17
> | | |
> | | POIsNull[boolean] - scope-21
> | | |
> | |
> |---POUserFunc(org.apache.pig.impl.builtin.ReadScalars)[chararray] - scope-20
> | | |
> | | |---Constant(1) - scope-18
> | | |
> | |
> |---Constant(hdfs://nn-nn1/tmp/temp-1607149525/tmp281350188) - scope-19
> | |
> | |---A:
> Load(hdfs://nn-nn1/user/anithar/test3.txt:org.apache.pig.builtin.PigStorage)
> - scope-0
> |
> |---E: Local Rearrange[tuple]{chararray}(false) - scope-38
> | |
> | Project[chararray][1] - scope-39
> |
>
> |---Load(hdfs://nn-nn1/tmp/temp-1607149525/tmp-458164144:org.apache.pig.impl.io.TFileStorage)
> - scope-53
> {code}
> Here at F , the file /tmp/temp-1607149525/tmp281350188 which is the output of
> the 1st Mapreduce Job is repeatedly read.
> If the input to F was non empty, since I am calling the scalar wrongly, it
> would have failed with the expected error message 'Scalar has more than 1 row
> in the output'.
> But since its null, it returns in ReadScalars before the exception is thrown
> and gives these in the task logs repeatedly
> {code}
> 2012-04-03 11:46:58,824 INFO
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat: Total input paths to
> process : 1
> 2012-04-03 11:46:58,824 INFO
> org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil: Total input
> paths to process : 1
> 2012-04-03 11:46:58,827 WARN org.apache.pig.impl.builtin.ReadScalars: No
> scalar field to read, returning null
>
>
> {code}
> That is its reading the '/tmp/temp-1607149525/tmp281350188' file again and
> again which was causing high namenode operation.
> The cost of one small mistake had ended up causing heavy namenode operations.
> Regards,
> Anitha
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (PIG-2629) Wrong Usage of Scalar which is null causes high namenode operation
[
https://issues.apache.org/jira/browse/PIG-2629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohini Palaniswamy updated PIG-2629:
Attachment: PIG-2629-2.patch
Addressed Cheolsoo's comment.
> Wrong Usage of Scalar which is null causes high namenode operation
> ---
>
> Key: PIG-2629
> URL: https://issues.apache.org/jira/browse/PIG-2629
> Project: Pig
> Issue Type: Bug
>Affects Versions: 0.8.1, 0.9.2, 0.10.0
>Reporter: Anitha Raju
>Assignee: Rohini Palaniswamy
> Fix For: 0.12.1, 0.13.0
>
> Attachments: PIG-2629-1.patch, PIG-2629-2.patch
>
>
> Hi,
> Script
> {code}
> A = LOAD 'test3.txt' AS (from:chararray);
> B = LOAD 'test2.txt'AS (source:chararray,to:chararray);
> C = FILTER A BY (from == 'temp' );
> D = FILTER B BY (source MATCHES '.*xyz*.');
> E = JOIN C by (from) left outer,D by (to);
> F = FILTER E BY (D.to IS NULL);
> dump F;
> {code}
> Inputs
> {code}
> $ cat test2.txt
> temptemp
> temptemp
> temptemp
> temptemp
> temptemp
> tepmtepm
> $ cat test3.txt |head
> temp
> temp
> temp
> temp
> temp
> temp
> tepm
> temp
> temp
> temp
> {code}
> Here I have by mistake called 'to' using 'D.to' instead of 'D::to'. The D
> relation gives null output.
> First Map Reduce job computes D which give null results.
> The MapPlan of 2nd job
> {code}
> Union[tuple] - scope-56
> |
> |---E: Local Rearrange[tuple]{chararray}(false) - scope-36
> | | |
> | | Project[chararray][0] - scope-37
> | |
> | |---C: Filter[bag] - scope-26
> | | |
> | | Equal To[boolean] - scope-29
> | | |
> | | |---Project[chararray][0] - scope-27
> | | |
> | | |---Constant(temp) - scope-28
> | |
> | |---A: New For Each(false)[bag] - scope-25
> | | |
> | | Cast[chararray] - scope-23
> | | |
> | | |---Project[bytearray][0] - scope-22
> | |
> | |---F: Filter[bag] - scope-17
> | | |
> | | POIsNull[boolean] - scope-21
> | | |
> | |
> |---POUserFunc(org.apache.pig.impl.builtin.ReadScalars)[chararray] - scope-20
> | | |
> | | |---Constant(1) - scope-18
> | | |
> | |
> |---Constant(hdfs://nn-nn1/tmp/temp-1607149525/tmp281350188) - scope-19
> | |
> | |---A:
> Load(hdfs://nn-nn1/user/anithar/test3.txt:org.apache.pig.builtin.PigStorage)
> - scope-0
> |
> |---E: Local Rearrange[tuple]{chararray}(false) - scope-38
> | |
> | Project[chararray][1] - scope-39
> |
>
> |---Load(hdfs://nn-nn1/tmp/temp-1607149525/tmp-458164144:org.apache.pig.impl.io.TFileStorage)
> - scope-53
> {code}
> Here at F , the file /tmp/temp-1607149525/tmp281350188 which is the output of
> the 1st Mapreduce Job is repeatedly read.
> If the input to F was non empty, since I am calling the scalar wrongly, it
> would have failed with the expected error message 'Scalar has more than 1 row
> in the output'.
> But since its null, it returns in ReadScalars before the exception is thrown
> and gives these in the task logs repeatedly
> {code}
> 2012-04-03 11:46:58,824 INFO
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat: Total input paths to
> process : 1
> 2012-04-03 11:46:58,824 INFO
> org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil: Total input
> paths to process : 1
> 2012-04-03 11:46:58,827 WARN org.apache.pig.impl.builtin.ReadScalars: No
> scalar field to read, returning null
>
>
> {code}
> That is its reading the '/tmp/temp-1607149525/tmp281350188' file again and
> again which was causing high namenode operation.
> The cost of one small mistake had ended up causing heavy namenode operations.
> Regards,
> Anitha
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (PIG-3608) ClassCastException when looking up a value from AvroMapWrapper using a Utf8 key
[
https://issues.apache.org/jira/browse/PIG-3608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13840737#comment-13840737
]
Rohini Palaniswamy commented on PIG-3608:
-
Code looks good. Can we just have a unit testcase added or modified to have
this case so that some other change does not break it?
> ClassCastException when looking up a value from AvroMapWrapper using a Utf8
> key
> ---
>
> Key: PIG-3608
> URL: https://issues.apache.org/jira/browse/PIG-3608
> Project: Pig
> Issue Type: Bug
> Components: impl
>Affects Versions: 0.12.0
>Reporter: Richard Ding
>Assignee: Richard Ding
>Priority: Minor
> Attachments: PIG-3608.patch
>
>
> One got the following exception:
> {code}
> java.lang.ClassCastException: org.apache.avro.util.Utf8 incompatible with
> java.lang.String
> at
> org.apache.pig.impl.util.avro.AvroMapWrapper.get(AvroMapWrapper.java:80)
> {code}
> This is related to the change by PIG-3420.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (PIG-3608) ClassCastException when looking up a value from AvroMapWrapper using a Utf8 key
[
https://issues.apache.org/jira/browse/PIG-3608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Richard Ding updated PIG-3608:
--
Status: Patch Available (was: Open)
> ClassCastException when looking up a value from AvroMapWrapper using a Utf8
> key
> ---
>
> Key: PIG-3608
> URL: https://issues.apache.org/jira/browse/PIG-3608
> Project: Pig
> Issue Type: Bug
> Components: impl
>Affects Versions: 0.12.0
>Reporter: Richard Ding
>Assignee: Richard Ding
>Priority: Minor
> Attachments: PIG-3608.patch
>
>
> One got the following exception:
> {code}
> java.lang.ClassCastException: org.apache.avro.util.Utf8 incompatible with
> java.lang.String
> at
> org.apache.pig.impl.util.avro.AvroMapWrapper.get(AvroMapWrapper.java:80)
> {code}
> This is related to the change by PIG-3420.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Assigned] (PIG-3608) ClassCastException when looking up a value from AvroMapWrapper using a Utf8 key
[
https://issues.apache.org/jira/browse/PIG-3608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Richard Ding reassigned PIG-3608:
-
Assignee: Richard Ding
> ClassCastException when looking up a value from AvroMapWrapper using a Utf8
> key
> ---
>
> Key: PIG-3608
> URL: https://issues.apache.org/jira/browse/PIG-3608
> Project: Pig
> Issue Type: Bug
> Components: impl
>Affects Versions: 0.12.0
>Reporter: Richard Ding
>Assignee: Richard Ding
>Priority: Minor
> Attachments: PIG-3608.patch
>
>
> One got the following exception:
> {code}
> java.lang.ClassCastException: org.apache.avro.util.Utf8 incompatible with
> java.lang.String
> at
> org.apache.pig.impl.util.avro.AvroMapWrapper.get(AvroMapWrapper.java:80)
> {code}
> This is related to the change by PIG-3420.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (PIG-3608) ClassCastException when looking up a value from AvroMapWrapper using a Utf8 key
[
https://issues.apache.org/jira/browse/PIG-3608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Richard Ding updated PIG-3608:
--
Attachment: PIG-3608.patch
Attach a simple patch.
> ClassCastException when looking up a value from AvroMapWrapper using a Utf8
> key
> ---
>
> Key: PIG-3608
> URL: https://issues.apache.org/jira/browse/PIG-3608
> Project: Pig
> Issue Type: Bug
> Components: impl
>Affects Versions: 0.12.0
>Reporter: Richard Ding
>Priority: Minor
> Attachments: PIG-3608.patch
>
>
> One got the following exception:
> {code}
> java.lang.ClassCastException: org.apache.avro.util.Utf8 incompatible with
> java.lang.String
> at
> org.apache.pig.impl.util.avro.AvroMapWrapper.get(AvroMapWrapper.java:80)
> {code}
> This is related to the change by PIG-3420.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (PIG-3607) PigRecordReader should report progress for each inputsplit processed
[ https://issues.apache.org/jira/browse/PIG-3607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13840708#comment-13840708 ] Aniket Mokashi commented on PIG-3607: - +1 > PigRecordReader should report progress for each inputsplit processed > > > Key: PIG-3607 > URL: https://issues.apache.org/jira/browse/PIG-3607 > Project: Pig > Issue Type: Bug >Affects Versions: 0.11.1 >Reporter: Rohini Palaniswamy >Assignee: Rohini Palaniswamy > Attachments: PIG-3607-1.patch > > > Currently progress() is called only when records are processed. In a case > where there were lot of empty input files, the task timed out and was killed > because no progress was reported. Too many empty input files are bad, but we > still don't want to fail. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (PIG-3612) Storing schema does not work cross cluster with PigStorage and JsonStorage
[ https://issues.apache.org/jira/browse/PIG-3612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13840691#comment-13840691 ] Aniket Mokashi commented on PIG-3612: - +1 > Storing schema does not work cross cluster with PigStorage and JsonStorage > -- > > Key: PIG-3612 > URL: https://issues.apache.org/jira/browse/PIG-3612 > Project: Pig > Issue Type: Bug >Reporter: Rohini Palaniswamy >Assignee: Rohini Palaniswamy > Fix For: 0.13.0 > > Attachments: PIG-3612-1.patch > > > java.lang.IllegalArgumentException: Wrong FS: > hdfs://cluster2-nn:8020/tmp/output/.pig_schema, > expected: hdfs://cluster1-nn1 > at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:582) > at > org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:155) > at > org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:734) > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1272) > at > org.apache.pig.backend.hadoop.datastorage.HDataStorage.isContainer(HDataStorage.java:200) > at > org.apache.pig.backend.hadoop.datastorage.HDataStorage.asElement(HDataStorage.java:128) > at > org.apache.pig.backend.hadoop.datastorage.HDataStorage.asElement(HDataStorage.java:144) > at org.apache.pig.builtin.JsonMetadata.storeSchema(JsonMetadata.java:294) > at org.apache.pig.builtin.JsonStorage.storeSchema(JsonStorage.java:274) > at -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (PIG-3612) Storing schema does not work cross cluster with PigStorage and JsonStorage
[ https://issues.apache.org/jira/browse/PIG-3612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohini Palaniswamy updated PIG-3612: Status: Patch Available (was: Open) > Storing schema does not work cross cluster with PigStorage and JsonStorage > -- > > Key: PIG-3612 > URL: https://issues.apache.org/jira/browse/PIG-3612 > Project: Pig > Issue Type: Bug >Reporter: Rohini Palaniswamy >Assignee: Rohini Palaniswamy > Fix For: 0.13.0 > > Attachments: PIG-3612-1.patch > > > java.lang.IllegalArgumentException: Wrong FS: > hdfs://cluster2-nn:8020/tmp/output/.pig_schema, > expected: hdfs://cluster1-nn1 > at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:582) > at > org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:155) > at > org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:734) > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1272) > at > org.apache.pig.backend.hadoop.datastorage.HDataStorage.isContainer(HDataStorage.java:200) > at > org.apache.pig.backend.hadoop.datastorage.HDataStorage.asElement(HDataStorage.java:128) > at > org.apache.pig.backend.hadoop.datastorage.HDataStorage.asElement(HDataStorage.java:144) > at org.apache.pig.builtin.JsonMetadata.storeSchema(JsonMetadata.java:294) > at org.apache.pig.builtin.JsonStorage.storeSchema(JsonStorage.java:274) > at -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (PIG-3607) PigRecordReader should report progress for each inputsplit processed
[ https://issues.apache.org/jira/browse/PIG-3607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohini Palaniswamy updated PIG-3607: Status: Patch Available (was: Open) > PigRecordReader should report progress for each inputsplit processed > > > Key: PIG-3607 > URL: https://issues.apache.org/jira/browse/PIG-3607 > Project: Pig > Issue Type: Bug >Affects Versions: 0.11.1 >Reporter: Rohini Palaniswamy >Assignee: Rohini Palaniswamy > Attachments: PIG-3607-1.patch > > > Currently progress() is called only when records are processed. In a case > where there were lot of empty input files, the task timed out and was killed > because no progress was reported. Too many empty input files are bad, but we > still don't want to fail. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (PIG-3607) PigRecordReader should report progress for each inputsplit processed
[ https://issues.apache.org/jira/browse/PIG-3607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohini Palaniswamy updated PIG-3607: Attachment: PIG-3607-1.patch > PigRecordReader should report progress for each inputsplit processed > > > Key: PIG-3607 > URL: https://issues.apache.org/jira/browse/PIG-3607 > Project: Pig > Issue Type: Bug >Affects Versions: 0.11.1 >Reporter: Rohini Palaniswamy >Assignee: Rohini Palaniswamy > Attachments: PIG-3607-1.patch > > > Currently progress() is called only when records are processed. In a case > where there were lot of empty input files, the task timed out and was killed > because no progress was reported. Too many empty input files are bad, but we > still don't want to fail. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (PIG-3612) Storing schema does not work cross cluster with PigStorage and JsonStorage
[ https://issues.apache.org/jira/browse/PIG-3612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohini Palaniswamy updated PIG-3612: Attachment: PIG-3612-1.patch Just a minor constructor change to get the correct HDataStorage based on URI. Reading from schema file, was already getting correct HDataStorage based on URI. Only store had issue. > Storing schema does not work cross cluster with PigStorage and JsonStorage > -- > > Key: PIG-3612 > URL: https://issues.apache.org/jira/browse/PIG-3612 > Project: Pig > Issue Type: Bug >Reporter: Rohini Palaniswamy >Assignee: Rohini Palaniswamy > Fix For: 0.13.0 > > Attachments: PIG-3612-1.patch > > > java.lang.IllegalArgumentException: Wrong FS: > hdfs://cluster2-nn:8020/tmp/output/.pig_schema, > expected: hdfs://cluster1-nn1 > at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:582) > at > org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:155) > at > org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:734) > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1272) > at > org.apache.pig.backend.hadoop.datastorage.HDataStorage.isContainer(HDataStorage.java:200) > at > org.apache.pig.backend.hadoop.datastorage.HDataStorage.asElement(HDataStorage.java:128) > at > org.apache.pig.backend.hadoop.datastorage.HDataStorage.asElement(HDataStorage.java:144) > at org.apache.pig.builtin.JsonMetadata.storeSchema(JsonMetadata.java:294) > at org.apache.pig.builtin.JsonStorage.storeSchema(JsonStorage.java:274) > at -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (PIG-3612) Storing schema does not work cross cluster with PigStorage and JsonStorage
Rohini Palaniswamy created PIG-3612: --- Summary: Storing schema does not work cross cluster with PigStorage and JsonStorage Key: PIG-3612 URL: https://issues.apache.org/jira/browse/PIG-3612 Project: Pig Issue Type: Bug Reporter: Rohini Palaniswamy Assignee: Rohini Palaniswamy Fix For: 0.13.0 java.lang.IllegalArgumentException: Wrong FS: hdfs://cluster2-nn:8020/tmp/output/.pig_schema, expected: hdfs://cluster1-nn1 at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:582) at org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:155) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:734) at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1272) at org.apache.pig.backend.hadoop.datastorage.HDataStorage.isContainer(HDataStorage.java:200) at org.apache.pig.backend.hadoop.datastorage.HDataStorage.asElement(HDataStorage.java:128) at org.apache.pig.backend.hadoop.datastorage.HDataStorage.asElement(HDataStorage.java:144) at org.apache.pig.builtin.JsonMetadata.storeSchema(JsonMetadata.java:294) at org.apache.pig.builtin.JsonStorage.storeSchema(JsonStorage.java:274) at -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Resolved] (PIG-3403) JsonLoader does not support null value in float field
[ https://issues.apache.org/jira/browse/PIG-3403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohini Palaniswamy resolved PIG-3403. - Resolution: Duplicate > JsonLoader does not support null value in float field > - > > Key: PIG-3403 > URL: https://issues.apache.org/jira/browse/PIG-3403 > Project: Pig > Issue Type: Bug > Components: internal-udfs >Affects Versions: 0.10.0, 0.11, 0.10.1, 0.11.1 >Reporter: Choon Hui Teo > > Sanity check for null value before parsing is available for all supported > DataTypes except DataType.FLOAT. Please see lines #197 and #198 in > http://svn.apache.org/viewvc/pig/trunk/src/org/apache/pig/builtin/JsonLoader.java?revision=1482145&view=markup. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Resolved] (PIG-2594) JsonLoader/JsonStorage does not work with boolean
[
https://issues.apache.org/jira/browse/PIG-2594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohini Palaniswamy resolved PIG-2594.
-
Resolution: Duplicate
Actually this is fixed by PIG-3454. Closing as duplicate
> JsonLoader/JsonStorage does not work with boolean
> -
>
> Key: PIG-2594
> URL: https://issues.apache.org/jira/browse/PIG-2594
> Project: Pig
> Issue Type: Bug
> Components: impl
>Affects Versions: 0.10.0
>Reporter: Daniel Dai
> Attachments: PIG-2594-2.patch, PIG-2594.patch, PIG-2594.patch
>
>
> The following script fail:
> {code}
> A = LOAD 'allscalar10k.json' using JsonLoader();
> store B into 'output';
> {code}
> Exception:
> java.io.IOException: Unknown type in input schema: 5
> at org.apache.pig.builtin.JsonLoader.readField(JsonLoader.java:292)
> at org.apache.pig.builtin.JsonLoader.getNext(JsonLoader.java:157)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader.nextKeyValue(PigRecordReader.java:187)
> at
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:532)
> at
> org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
--
This message was sent by Atlassian JIRA
(v6.1#6144)
