[jira] [Commented] (PARQUET-1711) [parquet-protobuf] stack overflow when work with well known json type
[ https://issues.apache.org/jira/browse/PARQUET-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17542604#comment-17542604 ] J Y commented on PARQUET-1711: -- i believe google's [Struct|https://github.com/protocolbuffers/protobuf/blob/main/src/google/protobuf/struct.proto] also runs afoul here, too. any new progress here? > [parquet-protobuf] stack overflow when work with well known json type > - > > Key: PARQUET-1711 > URL: https://issues.apache.org/jira/browse/PARQUET-1711 > Project: Parquet > Issue Type: Bug >Affects Versions: 1.10.1 >Reporter: Lawrence He >Priority: Major > > Writing following protobuf message as parquet file is not possible: > {code:java} > syntax = "proto3"; > import "google/protobuf/struct.proto"; > package test; > option java_outer_classname = "CustomMessage"; > message TestMessage { > map data = 1; > } {code} > Protobuf introduced "well known json type" such like > [ListValue|https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#listvalue] > to work around json schema conversion. > However writing above messages traps parquet writer into an infinite loop due > to the "general type" support in protobuf. Current implementation will keep > referencing 6 possible types defined in protobuf (null, bool, number, string, > struct, list) and entering infinite loop when referencing "struct". > {code:java} > java.lang.StackOverflowErrorjava.lang.StackOverflowError at > java.base/java.util.Arrays$ArrayItr.(Arrays.java:4418) at > java.base/java.util.Arrays$ArrayList.iterator(Arrays.java:4410) at > java.base/java.util.Collections$UnmodifiableCollection$1.(Collections.java:1044) > at > java.base/java.util.Collections$UnmodifiableCollection.iterator(Collections.java:1043) > at > org.apache.parquet.proto.ProtoSchemaConverter.convertFields(ProtoSchemaConverter.java:64) > at > org.apache.parquet.proto.ProtoSchemaConverter.addField(ProtoSchemaConverter.java:96) > at > org.apache.parquet.proto.ProtoSchemaConverter.convertFields(ProtoSchemaConverter.java:66) > at > org.apache.parquet.proto.ProtoSchemaConverter.addField(ProtoSchemaConverter.java:96) > at > org.apache.parquet.proto.ProtoSchemaConverter.convertFields(ProtoSchemaConverter.java:66) > at > org.apache.parquet.proto.ProtoSchemaConverter.addField(ProtoSchemaConverter.java:96) > at > org.apache.parquet.proto.ProtoSchemaConverter.convertFields(ProtoSchemaConverter.java:66) > at > org.apache.parquet.proto.ProtoSchemaConverter.addField(ProtoSchemaConverter.java:96) > at > org.apache.parquet.proto.ProtoSchemaConverter.convertFields(ProtoSchemaConverter.java:66) > at > org.apache.parquet.proto.ProtoSchemaConverter.addField(ProtoSchemaConverter.java:96) > {code} -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (PARQUET-1711) [parquet-protobuf] stack overflow when work with well known json type
[ https://issues.apache.org/jira/browse/PARQUET-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17552396#comment-17552396 ] J Y commented on PARQUET-1711: -- i'd be ok of that approach: a proto option annotation to limit the recursion limit, then failing over to treat it as proto bytes. if the recursion limit is omitted/missing, then just treat the recursive definition as bytes after the first occurrence. forgive me if this is a naive question, but what's the difficulty in supporting "typing" properly to handle recursive nesting? > [parquet-protobuf] stack overflow when work with well known json type > - > > Key: PARQUET-1711 > URL: https://issues.apache.org/jira/browse/PARQUET-1711 > Project: Parquet > Issue Type: Bug >Affects Versions: 1.10.1 >Reporter: Lawrence He >Priority: Major > > Writing following protobuf message as parquet file is not possible: > {code:java} > syntax = "proto3"; > import "google/protobuf/struct.proto"; > package test; > option java_outer_classname = "CustomMessage"; > message TestMessage { > map data = 1; > } {code} > Protobuf introduced "well known json type" such like > [ListValue|https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#listvalue] > to work around json schema conversion. > However writing above messages traps parquet writer into an infinite loop due > to the "general type" support in protobuf. Current implementation will keep > referencing 6 possible types defined in protobuf (null, bool, number, string, > struct, list) and entering infinite loop when referencing "struct". > {code:java} > java.lang.StackOverflowErrorjava.lang.StackOverflowError at > java.base/java.util.Arrays$ArrayItr.(Arrays.java:4418) at > java.base/java.util.Arrays$ArrayList.iterator(Arrays.java:4410) at > java.base/java.util.Collections$UnmodifiableCollection$1.(Collections.java:1044) > at > java.base/java.util.Collections$UnmodifiableCollection.iterator(Collections.java:1043) > at > org.apache.parquet.proto.ProtoSchemaConverter.convertFields(ProtoSchemaConverter.java:64) > at > org.apache.parquet.proto.ProtoSchemaConverter.addField(ProtoSchemaConverter.java:96) > at > org.apache.parquet.proto.ProtoSchemaConverter.convertFields(ProtoSchemaConverter.java:66) > at > org.apache.parquet.proto.ProtoSchemaConverter.addField(ProtoSchemaConverter.java:96) > at > org.apache.parquet.proto.ProtoSchemaConverter.convertFields(ProtoSchemaConverter.java:66) > at > org.apache.parquet.proto.ProtoSchemaConverter.addField(ProtoSchemaConverter.java:96) > at > org.apache.parquet.proto.ProtoSchemaConverter.convertFields(ProtoSchemaConverter.java:66) > at > org.apache.parquet.proto.ProtoSchemaConverter.addField(ProtoSchemaConverter.java:96) > at > org.apache.parquet.proto.ProtoSchemaConverter.convertFields(ProtoSchemaConverter.java:66) > at > org.apache.parquet.proto.ProtoSchemaConverter.addField(ProtoSchemaConverter.java:96) > {code} -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Comment Edited] (PARQUET-1711) [parquet-protobuf] stack overflow when work with well known json type
[ https://issues.apache.org/jira/browse/PARQUET-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17552396#comment-17552396 ] J Y edited comment on PARQUET-1711 at 6/9/22 7:16 PM: -- i'd be ok of that approach: a proto option annotation to limit the recursion limit, then failing over to treat it as proto bytes. if the recursion limit is omitted/missing, then just treat the recursive definition as bytes after the first occurrence. forgive me if this is a naive question, but what's the difficulty in supporting "typing" properly to handle recursive nesting? PARQUET-129 is very much related/the same issue... was (Author: jinyius): i'd be ok of that approach: a proto option annotation to limit the recursion limit, then failing over to treat it as proto bytes. if the recursion limit is omitted/missing, then just treat the recursive definition as bytes after the first occurrence. forgive me if this is a naive question, but what's the difficulty in supporting "typing" properly to handle recursive nesting? > [parquet-protobuf] stack overflow when work with well known json type > - > > Key: PARQUET-1711 > URL: https://issues.apache.org/jira/browse/PARQUET-1711 > Project: Parquet > Issue Type: Bug >Affects Versions: 1.10.1 >Reporter: Lawrence He >Priority: Major > > Writing following protobuf message as parquet file is not possible: > {code:java} > syntax = "proto3"; > import "google/protobuf/struct.proto"; > package test; > option java_outer_classname = "CustomMessage"; > message TestMessage { > map data = 1; > } {code} > Protobuf introduced "well known json type" such like > [ListValue|https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#listvalue] > to work around json schema conversion. > However writing above messages traps parquet writer into an infinite loop due > to the "general type" support in protobuf. Current implementation will keep > referencing 6 possible types defined in protobuf (null, bool, number, string, > struct, list) and entering infinite loop when referencing "struct". > {code:java} > java.lang.StackOverflowErrorjava.lang.StackOverflowError at > java.base/java.util.Arrays$ArrayItr.(Arrays.java:4418) at > java.base/java.util.Arrays$ArrayList.iterator(Arrays.java:4410) at > java.base/java.util.Collections$UnmodifiableCollection$1.(Collections.java:1044) > at > java.base/java.util.Collections$UnmodifiableCollection.iterator(Collections.java:1043) > at > org.apache.parquet.proto.ProtoSchemaConverter.convertFields(ProtoSchemaConverter.java:64) > at > org.apache.parquet.proto.ProtoSchemaConverter.addField(ProtoSchemaConverter.java:96) > at > org.apache.parquet.proto.ProtoSchemaConverter.convertFields(ProtoSchemaConverter.java:66) > at > org.apache.parquet.proto.ProtoSchemaConverter.addField(ProtoSchemaConverter.java:96) > at > org.apache.parquet.proto.ProtoSchemaConverter.convertFields(ProtoSchemaConverter.java:66) > at > org.apache.parquet.proto.ProtoSchemaConverter.addField(ProtoSchemaConverter.java:96) > at > org.apache.parquet.proto.ProtoSchemaConverter.convertFields(ProtoSchemaConverter.java:66) > at > org.apache.parquet.proto.ProtoSchemaConverter.addField(ProtoSchemaConverter.java:96) > at > org.apache.parquet.proto.ProtoSchemaConverter.convertFields(ProtoSchemaConverter.java:66) > at > org.apache.parquet.proto.ProtoSchemaConverter.addField(ProtoSchemaConverter.java:96) > {code} -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (PARQUET-2180) make the default behavior for proto writing not-backwards compatible
J Y created PARQUET-2180: Summary: make the default behavior for proto writing not-backwards compatible Key: PARQUET-2180 URL: https://issues.apache.org/jira/browse/PARQUET-2180 Project: Parquet Issue Type: Improvement Components: parquet-protobuf Reporter: J Y https://issues.apache.org/jira/browse/PARQUET-968 introduced supporting maps and lists in a spec compliant way. however, to not break existing libraries, a flag was introduced and defaulted the write behavior to NOT use the specs compliant writes. it's been over 5 years, and people should be really off of it. so much so, that trying to use the new parquet-cli tool to read parquet files generated by flink using doesn't work b/c it's hard coded to never allow the old style. the deprecated parquet-tools reads these files fine b/c it's the older style. i started coding up a workaround in flink-parquet and parquet-cli, but stopped. we really should just move on at this point, imho. protobufs often have repeated primitives and maps now, so it just makes sense to move on at this point. we should keep the flag around and let people override it back to being backwards compatible though. i have the code written and can submit a PR if you'd like. i'm not an expert in parquet though, so i'm unclear as to the deep downstream ramifications of this change, so i would love to get feedback in this area. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (PARQUET-2180) make the default behavior for proto writing not-backwards compatible
[ https://issues.apache.org/jira/browse/PARQUET-2180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] J Y updated PARQUET-2180: - Description: https://issues.apache.org/jira/browse/PARQUET-968 introduced supporting maps and lists in a spec compliant way. however, to not break existing libraries, a flag was introduced and defaulted the write behavior to NOT use the specs compliant writes. it's been over 5 years, and people should be really off of it. so much so, that trying to use the new parquet-cli tool to read parquet files generated by flink doesn't work b/c it's hard coded to never allow the old style. the deprecated parquet-tools reads these files fine b/c it's the older style. i started coding up a workaround in flink-parquet and parquet-cli, but stopped. we really should just move on at this point, imho. protobufs often have repeated primitives and maps now, so it just makes sense to move on at this point. we should keep the flag around and let people override it back to being backwards compatible though. i have the code written and can submit a PR if you'd like. i'm not an expert in parquet though, so i'm unclear as to the deep downstream ramifications of this change, so i would love to get feedback in this area. was: https://issues.apache.org/jira/browse/PARQUET-968 introduced supporting maps and lists in a spec compliant way. however, to not break existing libraries, a flag was introduced and defaulted the write behavior to NOT use the specs compliant writes. it's been over 5 years, and people should be really off of it. so much so, that trying to use the new parquet-cli tool to read parquet files generated by flink using doesn't work b/c it's hard coded to never allow the old style. the deprecated parquet-tools reads these files fine b/c it's the older style. i started coding up a workaround in flink-parquet and parquet-cli, but stopped. we really should just move on at this point, imho. protobufs often have repeated primitives and maps now, so it just makes sense to move on at this point. we should keep the flag around and let people override it back to being backwards compatible though. i have the code written and can submit a PR if you'd like. i'm not an expert in parquet though, so i'm unclear as to the deep downstream ramifications of this change, so i would love to get feedback in this area. > make the default behavior for proto writing not-backwards compatible > > > Key: PARQUET-2180 > URL: https://issues.apache.org/jira/browse/PARQUET-2180 > Project: Parquet > Issue Type: Improvement > Components: parquet-protobuf >Reporter: J Y >Priority: Minor > > https://issues.apache.org/jira/browse/PARQUET-968 introduced supporting maps > and lists in a spec compliant way. however, to not break existing libraries, > a flag was introduced and defaulted the write behavior to NOT use the specs > compliant writes. > it's been over 5 years, and people should be really off of it. so much so, > that trying to use the new parquet-cli tool to read parquet files generated > by flink doesn't work b/c it's hard coded to never allow the old style. the > deprecated parquet-tools reads these files fine b/c it's the older style. > i started coding up a workaround in flink-parquet and parquet-cli, but > stopped. we really should just move on at this point, imho. protobufs often > have repeated primitives and maps now, so it just makes sense to move on at > this point. we should keep the flag around and let people override it back > to being backwards compatible though. > i have the code written and can submit a PR if you'd like. > i'm not an expert in parquet though, so i'm unclear as to the deep downstream > ramifications of this change, so i would love to get feedback in this area. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (PARQUET-2180) make the default behavior for proto writing not-backwards compatible
[ https://issues.apache.org/jira/browse/PARQUET-2180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] J Y updated PARQUET-2180: - Description: https://issues.apache.org/jira/browse/PARQUET-968 introduced supporting maps and lists in a spec compliant way. however, to not break existing libraries, a flag was introduced and defaulted the write behavior to NOT use the specs compliant writes. it's been over 5 years, and people should be really off of it. so much so, that trying to use the new parquet-cli tool to read parquet files generated by flink doesn't work b/c it's hard coded to never allow the old style. the deprecated parquet-tools reads these files fine b/c it's the older style. i started coding up a workaround in flink-parquet and parquet-cli, but stopped. we really should just move on at this point, imho. protobufs often have repeated primitives and maps, so it's more pressing to get proper specs compliant support for it now. we should keep the flag around and let people override it back to being backwards compatible though. i have the code written and can submit a PR if you'd like. i'm not an expert in parquet though, so i'm unclear as to the deep downstream ramifications of this change, so i would love to get feedback in this area. was: https://issues.apache.org/jira/browse/PARQUET-968 introduced supporting maps and lists in a spec compliant way. however, to not break existing libraries, a flag was introduced and defaulted the write behavior to NOT use the specs compliant writes. it's been over 5 years, and people should be really off of it. so much so, that trying to use the new parquet-cli tool to read parquet files generated by flink doesn't work b/c it's hard coded to never allow the old style. the deprecated parquet-tools reads these files fine b/c it's the older style. i started coding up a workaround in flink-parquet and parquet-cli, but stopped. we really should just move on at this point, imho. protobufs often have repeated primitives and maps now, so it just makes sense to move on at this point. we should keep the flag around and let people override it back to being backwards compatible though. i have the code written and can submit a PR if you'd like. i'm not an expert in parquet though, so i'm unclear as to the deep downstream ramifications of this change, so i would love to get feedback in this area. > make the default behavior for proto writing not-backwards compatible > > > Key: PARQUET-2180 > URL: https://issues.apache.org/jira/browse/PARQUET-2180 > Project: Parquet > Issue Type: Improvement > Components: parquet-protobuf >Reporter: J Y >Priority: Minor > > https://issues.apache.org/jira/browse/PARQUET-968 introduced supporting maps > and lists in a spec compliant way. however, to not break existing libraries, > a flag was introduced and defaulted the write behavior to NOT use the specs > compliant writes. > it's been over 5 years, and people should be really off of it. so much so, > that trying to use the new parquet-cli tool to read parquet files generated > by flink doesn't work b/c it's hard coded to never allow the old style. the > deprecated parquet-tools reads these files fine b/c it's the older style. > i started coding up a workaround in flink-parquet and parquet-cli, but > stopped. we really should just move on at this point, imho. protobufs often > have repeated primitives and maps, so it's more pressing to get proper specs > compliant support for it now. we should keep the flag around and let people > override it back to being backwards compatible though. > i have the code written and can submit a PR if you'd like. > i'm not an expert in parquet though, so i'm unclear as to the deep downstream > ramifications of this change, so i would love to get feedback in this area. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (PARQUET-2181) parquet-cli fails at supporting parquet-protobuf generated schemas that have repeated primitives in them
J Y created PARQUET-2181: Summary: parquet-cli fails at supporting parquet-protobuf generated schemas that have repeated primitives in them Key: PARQUET-2181 URL: https://issues.apache.org/jira/browse/PARQUET-2181 Project: Parquet Issue Type: Bug Components: parquet-cli Reporter: J Y i generated a parquet file using a protobuf with this proto definition: {quote}message IndexPath { // Index of item in path. repeated int32 index = 1; } message SomeEvent { // truncated/obfuscated wrapper optional IndexPath client_position = 1; } {quote} this gets translated to the following parquet schema using the new compliant schema for lists: {quote}message SomeEvent { optional group client_position = 24 { optional group index (LIST) = 1 { repeated group list { required int32 element; } } } }{quote} this causes parquet-cli cat to barf on a file containing these events: {quote}java.lang.RuntimeException: Failed on record 0 at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:86) at org.apache.parquet.cli.Main.run(Main.java:157) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.parquet.cli.Main.main(Main.java:187) Caused by: java.lang.ClassCastException: required int32 element is not a group at org.apache.parquet.schema.Type.asGroupType(Type.java:248) at org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:284) at org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:228) at org.apache.parquet.avro.AvroRecordConverter.access$100(AvroRecordConverter.java:74) at org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter$ElementConverter.(AvroRecordConverter.java:539) at org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter.(AvroRecordConverter.java:489) at org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:293) at org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:137) at org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:284) at org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:137) at org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:91) at org.apache.parquet.avro.AvroRecordMaterializer.(AvroRecordMaterializer.java:33) at org.apache.parquet.avro.AvroReadSupport.prepareForRead(AvroReadSupport.java:142) at org.apache.parquet.hadoop.InternalParquetRecordReader.initialize(InternalParquetRecordReader.java:190) at org.apache.parquet.hadoop.ParquetReader.initReader(ParquetReader.java:166) at org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:135) at org.apache.parquet.cli.BaseCommand$1$1.advance(BaseCommand.java:363) at org.apache.parquet.cli.BaseCommand$1$1.(BaseCommand.java:344) at org.apache.parquet.cli.BaseCommand$1.iterator(BaseCommand.java:342) at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:73) ... 3 more{quote} using the old parquet-tools binary to cat this file works fine. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (PARQUET-2181) parquet-cli fails at supporting parquet-protobuf generated schemas that have repeated primitives in them
[ https://issues.apache.org/jira/browse/PARQUET-2181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] J Y updated PARQUET-2181: - Description: i generated a parquet file using a protobuf with this proto definition: {quote}message IndexPath { // Index of item in path. repeated int32 index = 1; } message SomeEvent { // truncated/obfuscated wrapper optional IndexPath client_position = 1; } {quote} this gets translated to the following parquet schema using the new compliant schema for lists: {quote}message SomeEvent { optional group client_position = 24 { optional group index (LIST) = 1 { repeated group list {{}} required int32 element; } } } }{quote} this causes parquet-cli cat to barf on a file containing these events: {quote}java.lang.RuntimeException: Failed on record 0 at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:86) at org.apache.parquet.cli.Main.run(Main.java:157) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.parquet.cli.Main.main(Main.java:187) Caused by: java.lang.ClassCastException: required int32 element is not a group at org.apache.parquet.schema.Type.asGroupType(Type.java:248) at org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:284) at org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:228) at org.apache.parquet.avro.AvroRecordConverter.access$100(AvroRecordConverter.java:74) at org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter$ElementConverter.(AvroRecordConverter.java:539) at org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter.(AvroRecordConverter.java:489) at org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:293) at org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:137) at org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:284) at org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:137) at org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:91) at org.apache.parquet.avro.AvroRecordMaterializer.(AvroRecordMaterializer.java:33) at org.apache.parquet.avro.AvroReadSupport.prepareForRead(AvroReadSupport.java:142) at org.apache.parquet.hadoop.InternalParquetRecordReader.initialize(InternalParquetRecordReader.java:190) at org.apache.parquet.hadoop.ParquetReader.initReader(ParquetReader.java:166) at org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:135) at org.apache.parquet.cli.BaseCommand$1$1.advance(BaseCommand.java:363) at org.apache.parquet.cli.BaseCommand$1$1.(BaseCommand.java:344) at org.apache.parquet.cli.BaseCommand$1.iterator(BaseCommand.java:342) at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:73) ... 3 more{quote} using the old parquet-tools binary to cat this file works fine. was: i generated a parquet file using a protobuf with this proto definition: {quote}message IndexPath { // Index of item in path. repeated int32 index = 1; } message SomeEvent { // truncated/obfuscated wrapper optional IndexPath client_position = 1; } {quote} this gets translated to the following parquet schema using the new compliant schema for lists: {quote}message SomeEvent { optional group client_position = 24 { optional group index (LIST) = 1 { repeated group list { required int32 element; } } } }{quote} this causes parquet-cli cat to barf on a file containing these events: {quote}java.lang.RuntimeException: Failed on record 0 at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:86) at org.apache.parquet.cli.Main.run(Main.java:157) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.parquet.cli.Main.main(Main.java:187) Caused by: java.lang.ClassCastException: required int32 element is not a group at org.apache.parquet.schema.Type.asGroupType(Type.java:248) at org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:284) at org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:228) at org.apache.parquet.avro.AvroRecordConverter.access$100(AvroRecordConverter.java:74) at org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter$ElementConverter.(AvroRecordConverter.java:539) at org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter.(AvroRecordConverter.java:489) at org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:293) at org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:137) at org.apach
[jira] [Updated] (PARQUET-2181) parquet-cli fails at supporting parquet-protobuf generated schemas that have repeated primitives in them
[ https://issues.apache.org/jira/browse/PARQUET-2181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] J Y updated PARQUET-2181: - Description: i generated a parquet file using a protobuf with this proto definition: {quote}message IndexPath { // Index of item in path. repeated int32 index = 1; } message SomeEvent { // truncated/obfuscated wrapper optional IndexPath client_position = 1; } {quote} this gets translated to the following parquet schema using the new compliant schema for lists: {quote}message SomeEvent { optional group client_position = 24 { optional group index (LIST) = 1 { repeated group list { required int32 element; } } } }{quote} this causes parquet-cli cat to barf on a file containing these events: {quote}java.lang.RuntimeException: Failed on record 0 at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:86) at org.apache.parquet.cli.Main.run(Main.java:157) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.parquet.cli.Main.main(Main.java:187) Caused by: java.lang.ClassCastException: required int32 element is not a group at org.apache.parquet.schema.Type.asGroupType(Type.java:248) at org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:284) at org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:228) at org.apache.parquet.avro.AvroRecordConverter.access$100(AvroRecordConverter.java:74) at org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter$ElementConverter.(AvroRecordConverter.java:539) at org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter.(AvroRecordConverter.java:489) at org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:293) at org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:137) at org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:284) at org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:137) at org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:91) at org.apache.parquet.avro.AvroRecordMaterializer.(AvroRecordMaterializer.java:33) at org.apache.parquet.avro.AvroReadSupport.prepareForRead(AvroReadSupport.java:142) at org.apache.parquet.hadoop.InternalParquetRecordReader.initialize(InternalParquetRecordReader.java:190) at org.apache.parquet.hadoop.ParquetReader.initReader(ParquetReader.java:166) at org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:135) at org.apache.parquet.cli.BaseCommand$1$1.advance(BaseCommand.java:363) at org.apache.parquet.cli.BaseCommand$1$1.(BaseCommand.java:344) at org.apache.parquet.cli.BaseCommand$1.iterator(BaseCommand.java:342) at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:73) ... 3 more{quote} using the old parquet-tools binary to cat this file works fine. was: i generated a parquet file using a protobuf with this proto definition: {quote}message IndexPath { // Index of item in path. repeated int32 index = 1; } message SomeEvent { // truncated/obfuscated wrapper optional IndexPath client_position = 1; } {quote} this gets translated to the following parquet schema using the new compliant schema for lists: {quote}message SomeEvent { optional group client_position = 24 { optional group index (LIST) = 1 { repeated group list } required int32 element; } } } }{quote} this causes parquet-cli cat to barf on a file containing these events: {quote}java.lang.RuntimeException: Failed on record 0 at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:86) at org.apache.parquet.cli.Main.run(Main.java:157) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.parquet.cli.Main.main(Main.java:187) Caused by: java.lang.ClassCastException: required int32 element is not a group at org.apache.parquet.schema.Type.asGroupType(Type.java:248) at org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:284) at org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:228) at org.apache.parquet.avro.AvroRecordConverter.access$100(AvroRecordConverter.java:74) at org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter$ElementConverter.(AvroRecordConverter.java:539) at org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter.(AvroRecordConverter.java:489) at org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:293) at org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:137) at org.apache.p
[jira] [Updated] (PARQUET-2181) parquet-cli fails at supporting parquet-protobuf generated schemas that have repeated primitives in them
[ https://issues.apache.org/jira/browse/PARQUET-2181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] J Y updated PARQUET-2181: - Description: i generated a parquet file using a protobuf with this proto definition: {quote}message IndexPath { // Index of item in path. repeated int32 index = 1; } message SomeEvent { // truncated/obfuscated wrapper optional IndexPath client_position = 1; } {quote} this gets translated to the following parquet schema using the new compliant schema for lists: {quote}message SomeEvent { optional group client_position = 24 { optional group index (LIST) = 1 { repeated group list } required int32 element; } } } }{quote} this causes parquet-cli cat to barf on a file containing these events: {quote}java.lang.RuntimeException: Failed on record 0 at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:86) at org.apache.parquet.cli.Main.run(Main.java:157) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.parquet.cli.Main.main(Main.java:187) Caused by: java.lang.ClassCastException: required int32 element is not a group at org.apache.parquet.schema.Type.asGroupType(Type.java:248) at org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:284) at org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:228) at org.apache.parquet.avro.AvroRecordConverter.access$100(AvroRecordConverter.java:74) at org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter$ElementConverter.(AvroRecordConverter.java:539) at org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter.(AvroRecordConverter.java:489) at org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:293) at org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:137) at org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:284) at org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:137) at org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:91) at org.apache.parquet.avro.AvroRecordMaterializer.(AvroRecordMaterializer.java:33) at org.apache.parquet.avro.AvroReadSupport.prepareForRead(AvroReadSupport.java:142) at org.apache.parquet.hadoop.InternalParquetRecordReader.initialize(InternalParquetRecordReader.java:190) at org.apache.parquet.hadoop.ParquetReader.initReader(ParquetReader.java:166) at org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:135) at org.apache.parquet.cli.BaseCommand$1$1.advance(BaseCommand.java:363) at org.apache.parquet.cli.BaseCommand$1$1.(BaseCommand.java:344) at org.apache.parquet.cli.BaseCommand$1.iterator(BaseCommand.java:342) at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:73) ... 3 more{quote} using the old parquet-tools binary to cat this file works fine. was: i generated a parquet file using a protobuf with this proto definition: {quote}message IndexPath { // Index of item in path. repeated int32 index = 1; } message SomeEvent { // truncated/obfuscated wrapper optional IndexPath client_position = 1; } {quote} this gets translated to the following parquet schema using the new compliant schema for lists: {quote}message SomeEvent { optional group client_position = 24 { optional group index (LIST) = 1 { repeated group list { required int32 element; } } } }{quote} this causes parquet-cli cat to barf on a file containing these events: {quote}java.lang.RuntimeException: Failed on record 0 at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:86) at org.apache.parquet.cli.Main.run(Main.java:157) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.parquet.cli.Main.main(Main.java:187) Caused by: java.lang.ClassCastException: required int32 element is not a group at org.apache.parquet.schema.Type.asGroupType(Type.java:248) at org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:284) at org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:228) at org.apache.parquet.avro.AvroRecordConverter.access$100(AvroRecordConverter.java:74) at org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter$ElementConverter.(AvroRecordConverter.java:539) at org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter.(AvroRecordConverter.java:489) at org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:293) at org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:137) at org.apache.p
[jira] [Updated] (PARQUET-2181) parquet-cli fails at supporting parquet-protobuf generated schemas that have repeated primitives in them
[ https://issues.apache.org/jira/browse/PARQUET-2181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] J Y updated PARQUET-2181: - Description: i generated a parquet file using a protobuf with this proto definition: {quote}message IndexPath { // Index of item in path. repeated int32 index = 1; } message SomeEvent { // truncated/obfuscated wrapper optional IndexPath client_position = 1; } {quote} this gets translated to the following parquet schema using the new compliant schema for lists: {quote}message SomeEvent { optional group client_position = 24 { optional group index (LIST) = 1 { repeated group list { required int32 element; } } } }{quote} this causes parquet-cli cat to barf on a file containing these events: {quote}java.lang.RuntimeException: Failed on record 0 at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:86) at org.apache.parquet.cli.Main.run(Main.java:157) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.parquet.cli.Main.main(Main.java:187) Caused by: java.lang.ClassCastException: required int32 element is not a group at org.apache.parquet.schema.Type.asGroupType(Type.java:248) at org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:284) at org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:228) at org.apache.parquet.avro.AvroRecordConverter.access$100(AvroRecordConverter.java:74) at org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter$ElementConverter.(AvroRecordConverter.java:539) at org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter.(AvroRecordConverter.java:489) at org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:293) at org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:137) at org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:284) at org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:137) at org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:91) at org.apache.parquet.avro.AvroRecordMaterializer.(AvroRecordMaterializer.java:33) at org.apache.parquet.avro.AvroReadSupport.prepareForRead(AvroReadSupport.java:142) at org.apache.parquet.hadoop.InternalParquetRecordReader.initialize(InternalParquetRecordReader.java:190) at org.apache.parquet.hadoop.ParquetReader.initReader(ParquetReader.java:166) at org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:135) at org.apache.parquet.cli.BaseCommand$1$1.advance(BaseCommand.java:363) at org.apache.parquet.cli.BaseCommand$1$1.(BaseCommand.java:344) at org.apache.parquet.cli.BaseCommand$1.iterator(BaseCommand.java:342) at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:73) ... 3 more{quote} using the old parquet-tools binary to cat this file works fine. was: i generated a parquet file using a protobuf with this proto definition: {quote}message IndexPath { // Index of item in path. repeated int32 index = 1; } message SomeEvent { // truncated/obfuscated wrapper optional IndexPath client_position = 1; } {quote} this gets translated to the following parquet schema using the new compliant schema for lists: {quote}message SomeEvent { optional group client_position = 24 { optional group index (LIST) = 1 { repeated group list {{}} required int32 element; } } } }{quote} this causes parquet-cli cat to barf on a file containing these events: {quote}java.lang.RuntimeException: Failed on record 0 at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:86) at org.apache.parquet.cli.Main.run(Main.java:157) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.parquet.cli.Main.main(Main.java:187) Caused by: java.lang.ClassCastException: required int32 element is not a group at org.apache.parquet.schema.Type.asGroupType(Type.java:248) at org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:284) at org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:228) at org.apache.parquet.avro.AvroRecordConverter.access$100(AvroRecordConverter.java:74) at org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter$ElementConverter.(AvroRecordConverter.java:539) at org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter.(AvroRecordConverter.java:489) at org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:293) at org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:137) at org.apach
[jira] [Updated] (PARQUET-2181) parquet-cli fails at supporting parquet-protobuf generated schemas that have repeated primitives in them
[ https://issues.apache.org/jira/browse/PARQUET-2181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] J Y updated PARQUET-2181: - Description: i generated a parquet file using a protobuf with this proto definition: {quote}message IndexPath { // Index of item in path. repeated int32 index = 1; } message SomeEvent { // truncated/obfuscated wrapper optional IndexPath client_position = 1; } {quote} this gets translated to the following parquet schema using the new compliant schema for lists: {quote}message SomeEvent { optional group client_position = 24 { optional group index (LIST) = 1 { repeated group list { required int32 element; } } } }{quote} this causes parquet-cli cat to barf on a file containing these events: {quote}java.lang.RuntimeException: Failed on record 0 at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:86) at org.apache.parquet.cli.Main.run(Main.java:157) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.parquet.cli.Main.main(Main.java:187) Caused by: java.lang.ClassCastException: required int32 element is not a group at org.apache.parquet.schema.Type.asGroupType(Type.java:248) at org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:284) at org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:228) at org.apache.parquet.avro.AvroRecordConverter.access$100(AvroRecordConverter.java:74) at org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter$ElementConverter.(AvroRecordConverter.java:539) at org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter.(AvroRecordConverter.java:489) at org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:293) at org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:137) at org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:284) at org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:137) at org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:91) at org.apache.parquet.avro.AvroRecordMaterializer.(AvroRecordMaterializer.java:33) at org.apache.parquet.avro.AvroReadSupport.prepareForRead(AvroReadSupport.java:142) at org.apache.parquet.hadoop.InternalParquetRecordReader.initialize(InternalParquetRecordReader.java:190) at org.apache.parquet.hadoop.ParquetReader.initReader(ParquetReader.java:166) at org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:135) at org.apache.parquet.cli.BaseCommand$1$1.advance(BaseCommand.java:363) at org.apache.parquet.cli.BaseCommand$1$1.(BaseCommand.java:344) at org.apache.parquet.cli.BaseCommand$1.iterator(BaseCommand.java:342) at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:73) ... 3 more {quote} using the old parquet-tools binary to cat this file works fine. was: i generated a parquet file using a protobuf with this proto definition: {quote}message IndexPath { // Index of item in path. repeated int32 index = 1; } message SomeEvent { // truncated/obfuscated wrapper optional IndexPath client_position = 1; } {quote} this gets translated to the following parquet schema using the new compliant schema for lists: {quote}message SomeEvent { optional group client_position = 24 { optional group index (LIST) = 1 { repeated group list { required int32 element; } } } }{quote} this causes parquet-cli cat to barf on a file containing these events: {quote}java.lang.RuntimeException: Failed on record 0 at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:86) at org.apache.parquet.cli.Main.run(Main.java:157) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.parquet.cli.Main.main(Main.java:187) Caused by: java.lang.ClassCastException: required int32 element is not a group at org.apache.parquet.schema.Type.asGroupType(Type.java:248) at org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:284) at org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:228) at org.apache.parquet.avro.AvroRecordConverter.access$100(AvroRecordConverter.java:74) at org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter$ElementConverter.(AvroRecordConverter.java:539) at org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter.(AvroRecordConverter.java:489) at org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:293) at org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:137) at org.apache.parquet.
[jira] [Updated] (PARQUET-2181) parquet-cli fails at supporting parquet-protobuf generated schemas that have repeated primitives in them
[ https://issues.apache.org/jira/browse/PARQUET-2181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] J Y updated PARQUET-2181: - Description: i generated a parquet file using a protobuf with this proto definition: {code:java} message IndexPath { // Index of item in path. repeated int32 index = 1; } message SomeEvent { // truncated/obfuscated wrapper optional IndexPath client_position = 1; } {code} this gets translated to the following parquet schema using the new compliant schema for lists: {code:java} message SomeEvent { optional group client_position = 24 { optional group index (LIST) = 1 { repeated group list { required int32 element; } } } } {code} this causes parquet-cli cat to barf on a file containing these events: {quote}java.lang.RuntimeException: Failed on record 0 at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:86) at org.apache.parquet.cli.Main.run(Main.java:157) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.parquet.cli.Main.main(Main.java:187) Caused by: java.lang.ClassCastException: required int32 element is not a group at org.apache.parquet.schema.Type.asGroupType(Type.java:248) at org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:284) at org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:228) at org.apache.parquet.avro.AvroRecordConverter.access$100(AvroRecordConverter.java:74) at org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter$ElementConverter.(AvroRecordConverter.java:539) at org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter.(AvroRecordConverter.java:489) at org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:293) at org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:137) at org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:284) at org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:137) at org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:91) at org.apache.parquet.avro.AvroRecordMaterializer.(AvroRecordMaterializer.java:33) at org.apache.parquet.avro.AvroReadSupport.prepareForRead(AvroReadSupport.java:142) at org.apache.parquet.hadoop.InternalParquetRecordReader.initialize(InternalParquetRecordReader.java:190) at org.apache.parquet.hadoop.ParquetReader.initReader(ParquetReader.java:166) at org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:135) at org.apache.parquet.cli.BaseCommand$1$1.advance(BaseCommand.java:363) at org.apache.parquet.cli.BaseCommand$1$1.(BaseCommand.java:344) at org.apache.parquet.cli.BaseCommand$1.iterator(BaseCommand.java:342) at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:73) ... 3 more {quote} using the old parquet-tools binary to cat this file works fine. was: i generated a parquet file using a protobuf with this proto definition: {code:java} message IndexPath { // Index of item in path. repeated int32 index = 1; } message SomeEvent { // truncated/obfuscated wrapper optional IndexPath client_position = 1; } {code} this gets translated to the following parquet schema using the new compliant schema for lists: {code:java} message SomeEvent { optional group client_position = 24 { optional group index (LIST) = 1 { repeated group list { required int32 element; } } } } {code} this causes parquet-cli cat to barf on a file containing these events: {quote}java.lang.RuntimeException: Failed on record 0 at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:86) at org.apache.parquet.cli.Main.run(Main.java:157) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.parquet.cli.Main.main(Main.java:187) Caused by: java.lang.ClassCastException: required int32 element is not a group at org.apache.parquet.schema.Type.asGroupType(Type.java:248) at org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:284) at org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:228) at org.apache.parquet.avro.AvroRecordConverter.access$100(AvroRecordConverter.java:74) at org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter$ElementConverter.(AvroRecordConverter.java:539) at org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter.(AvroRecordConverter.java:489) at org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:293) at org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:137) at
[jira] [Updated] (PARQUET-2181) parquet-cli fails at supporting parquet-protobuf generated schemas that have repeated primitives in them
[ https://issues.apache.org/jira/browse/PARQUET-2181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] J Y updated PARQUET-2181: - Description: i generated a parquet file using a protobuf with this proto definition: {code:java} message IndexPath { // Index of item in path. repeated int32 index = 1; } message SomeEvent { // truncated/obfuscated wrapper optional IndexPath client_position = 1; } {code} this gets translated to the following parquet schema using the new compliant schema for lists: {code:java} message SomeEvent { optional group client_position = 24 { optional group index (LIST) = 1 { repeated group list { required int32 element; } } } } {code} this causes parquet-cli cat to barf on a file containing these events: {quote}java.lang.RuntimeException: Failed on record 0 at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:86) at org.apache.parquet.cli.Main.run(Main.java:157) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.parquet.cli.Main.main(Main.java:187) Caused by: java.lang.ClassCastException: required int32 element is not a group at org.apache.parquet.schema.Type.asGroupType(Type.java:248) at org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:284) at org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:228) at org.apache.parquet.avro.AvroRecordConverter.access$100(AvroRecordConverter.java:74) at org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter$ElementConverter.(AvroRecordConverter.java:539) at org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter.(AvroRecordConverter.java:489) at org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:293) at org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:137) at org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:284) at org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:137) at org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:91) at org.apache.parquet.avro.AvroRecordMaterializer.(AvroRecordMaterializer.java:33) at org.apache.parquet.avro.AvroReadSupport.prepareForRead(AvroReadSupport.java:142) at org.apache.parquet.hadoop.InternalParquetRecordReader.initialize(InternalParquetRecordReader.java:190) at org.apache.parquet.hadoop.ParquetReader.initReader(ParquetReader.java:166) at org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:135) at org.apache.parquet.cli.BaseCommand$1$1.advance(BaseCommand.java:363) at org.apache.parquet.cli.BaseCommand$1$1.(BaseCommand.java:344) at org.apache.parquet.cli.BaseCommand$1.iterator(BaseCommand.java:342) at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:73) ... 3 more {quote} using the old parquet-tools binary to cat this file works fine. was: i generated a parquet file using a protobuf with this proto definition: {quote}message IndexPath { // Index of item in path. repeated int32 index = 1; } message SomeEvent { // truncated/obfuscated wrapper optional IndexPath client_position = 1; } {quote} this gets translated to the following parquet schema using the new compliant schema for lists: {quote}message SomeEvent { optional group client_position = 24 { optional group index (LIST) = 1 { repeated group list { required int32 element; } } } }{quote} this causes parquet-cli cat to barf on a file containing these events: {quote}java.lang.RuntimeException: Failed on record 0 at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:86) at org.apache.parquet.cli.Main.run(Main.java:157) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.parquet.cli.Main.main(Main.java:187) Caused by: java.lang.ClassCastException: required int32 element is not a group at org.apache.parquet.schema.Type.asGroupType(Type.java:248) at org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:284) at org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:228) at org.apache.parquet.avro.AvroRecordConverter.access$100(AvroRecordConverter.java:74) at org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter$ElementConverter.(AvroRecordConverter.java:539) at org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter.(AvroRecordConverter.java:489) at org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:293) at org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:137) at org.apache.pa
[jira] [Updated] (PARQUET-2181) parquet-cli fails at supporting parquet-protobuf generated schemas that have repeated primitives in them
[ https://issues.apache.org/jira/browse/PARQUET-2181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] J Y updated PARQUET-2181: - Description: i generated a parquet file using a protobuf with this proto definition: {code:java} message IndexPath { // Index of item in path. repeated int32 index = 1; } message SomeEvent { // truncated/obfuscated wrapper optional IndexPath client_position = 1; } {code} this gets translated to the following parquet schema using the new compliant schema for lists: {code:java} message SomeEvent { optional group client_position = 1 { optional group index (LIST) = 1 { repeated group list { required int32 element; } } } } {code} this causes parquet-cli cat to barf on a file containing these events: {quote}java.lang.RuntimeException: Failed on record 0 at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:86) at org.apache.parquet.cli.Main.run(Main.java:157) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.parquet.cli.Main.main(Main.java:187) Caused by: java.lang.ClassCastException: required int32 element is not a group at org.apache.parquet.schema.Type.asGroupType(Type.java:248) at org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:284) at org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:228) at org.apache.parquet.avro.AvroRecordConverter.access$100(AvroRecordConverter.java:74) at org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter$ElementConverter.(AvroRecordConverter.java:539) at org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter.(AvroRecordConverter.java:489) at org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:293) at org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:137) at org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:284) at org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:137) at org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:91) at org.apache.parquet.avro.AvroRecordMaterializer.(AvroRecordMaterializer.java:33) at org.apache.parquet.avro.AvroReadSupport.prepareForRead(AvroReadSupport.java:142) at org.apache.parquet.hadoop.InternalParquetRecordReader.initialize(InternalParquetRecordReader.java:190) at org.apache.parquet.hadoop.ParquetReader.initReader(ParquetReader.java:166) at org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:135) at org.apache.parquet.cli.BaseCommand$1$1.advance(BaseCommand.java:363) at org.apache.parquet.cli.BaseCommand$1$1.(BaseCommand.java:344) at org.apache.parquet.cli.BaseCommand$1.iterator(BaseCommand.java:342) at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:73) ... 3 more {quote} using the old parquet-tools binary to cat this file works fine. was: i generated a parquet file using a protobuf with this proto definition: {code:java} message IndexPath { // Index of item in path. repeated int32 index = 1; } message SomeEvent { // truncated/obfuscated wrapper optional IndexPath client_position = 1; } {code} this gets translated to the following parquet schema using the new compliant schema for lists: {code:java} message SomeEvent { optional group client_position = 24 { optional group index (LIST) = 1 { repeated group list { required int32 element; } } } } {code} this causes parquet-cli cat to barf on a file containing these events: {quote}java.lang.RuntimeException: Failed on record 0 at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:86) at org.apache.parquet.cli.Main.run(Main.java:157) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.parquet.cli.Main.main(Main.java:187) Caused by: java.lang.ClassCastException: required int32 element is not a group at org.apache.parquet.schema.Type.asGroupType(Type.java:248) at org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:284) at org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:228) at org.apache.parquet.avro.AvroRecordConverter.access$100(AvroRecordConverter.java:74) at org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter$ElementConverter.(AvroRecordConverter.java:539) at org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter.(AvroRecordConverter.java:489) at org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:293) at org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:137) a
[jira] [Updated] (PARQUET-2181) parquet-cli fails at supporting parquet-protobuf generated schemas that have repeated primitives in them
[ https://issues.apache.org/jira/browse/PARQUET-2181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] J Y updated PARQUET-2181: - Attachment: samples.tgz > parquet-cli fails at supporting parquet-protobuf generated schemas that have > repeated primitives in them > > > Key: PARQUET-2181 > URL: https://issues.apache.org/jira/browse/PARQUET-2181 > Project: Parquet > Issue Type: Bug > Components: parquet-cli >Reporter: J Y >Priority: Major > Attachments: samples.tgz > > > i generated a parquet file using a protobuf with this proto definition: > {code:java} > message IndexPath { > // Index of item in path. > repeated int32 index = 1; > } > message SomeEvent { > // truncated/obfuscated wrapper > optional IndexPath client_position = 1; > } > {code} > this gets translated to the following parquet schema using the new compliant > schema for lists: > {code:java} > message SomeEvent { > optional group client_position = 1 { > optional group index (LIST) = 1 { > repeated group list { > required int32 element; > } > } > } > } > {code} > this causes parquet-cli cat to barf on a file containing these events: > {quote}java.lang.RuntimeException: Failed on record 0 > at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:86) > at org.apache.parquet.cli.Main.run(Main.java:157) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) > at org.apache.parquet.cli.Main.main(Main.java:187) > Caused by: java.lang.ClassCastException: required int32 element is not a group > at org.apache.parquet.schema.Type.asGroupType(Type.java:248) > at > org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:284) > at > org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:228) > at > org.apache.parquet.avro.AvroRecordConverter.access$100(AvroRecordConverter.java:74) > at > org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter$ElementConverter.(AvroRecordConverter.java:539) > at > org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter.(AvroRecordConverter.java:489) > at > org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:293) > at > org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:137) > at > org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:284) > at > org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:137) > at > org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:91) > at > org.apache.parquet.avro.AvroRecordMaterializer.(AvroRecordMaterializer.java:33) > at > org.apache.parquet.avro.AvroReadSupport.prepareForRead(AvroReadSupport.java:142) > at > org.apache.parquet.hadoop.InternalParquetRecordReader.initialize(InternalParquetRecordReader.java:190) > at > org.apache.parquet.hadoop.ParquetReader.initReader(ParquetReader.java:166) > at > org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:135) > at > org.apache.parquet.cli.BaseCommand$1$1.advance(BaseCommand.java:363) > at org.apache.parquet.cli.BaseCommand$1$1.(BaseCommand.java:344) > at org.apache.parquet.cli.BaseCommand$1.iterator(BaseCommand.java:342) > at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:73) > ... 3 more > {quote} > using the old parquet-tools binary to cat this file works fine. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (PARQUET-2181) parquet-cli fails at supporting parquet-protobuf generated schemas that have repeated primitives in them
[ https://issues.apache.org/jira/browse/PARQUET-2181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17598196#comment-17598196 ] J Y commented on PARQUET-2181: -- i've attached some parquet files that all read fine using parquet-tools (both the deprecated version from parquet-mr and the [one written in python|https://github.com/ktrueda/parquet-tools]) *but do not read at all using parquet-cli*. parquet-cli's meta command works fine. it turns out, there's other stack traces when trying to use parquet-cli to read these files. in addition to the repeated primitive issue highlighted originally, there's 2 other issues like the following exhibited in these files: {quote}--- ./raw/delivery-log/dt=2022-08-10/hour=04/part-02a95a0e-bd21-4476-9d0f-d1896687b12a-0 Argument error: Map key type must be binary (UTF8): required int32 key --- ./raw/user/dt=2022-08-10/hour=04/part-8cac1d0c-fb7f-4a9a-b77e-b3dd59f89333-0 Unknown error java.lang.RuntimeException: Failed on record 0 at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:86) at org.apache.parquet.cli.Main.run(Main.java:157) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.parquet.cli.Main.main(Main.java:187) Caused by: org.apache.parquet.io.InvalidRecordException: Parquet/Avro schema mismatch: Avro field 'null_value' not found at org.apache.parquet.avro.AvroRecordConverter.getAvroField(AvroRecordConverter.java:221) {quote} is using AvroReadSupport and AvroRecrodConverter the right way to go for protobufs? it looks like the parquet-tools that was deprecated in 1.12.3+ doesn't use the parquet-avro approach to reading (it uses [its own SimpleReadSupport approach|https://github.com/apache/parquet-mr/tree/apache-parquet-1.12.2/parquet-tools-deprecated/src/main/java/org/apache/parquet/tools/read]), which makes sense to me given the underlying schema and data written in parquet-protobuf generated files are not avro... should we move parquet-cli back to SimpleReadSupport instead of relying on what appears to be a broken AvroReadSupport when dealing with proto generated files? > parquet-cli fails at supporting parquet-protobuf generated schemas that have > repeated primitives in them > > > Key: PARQUET-2181 > URL: https://issues.apache.org/jira/browse/PARQUET-2181 > Project: Parquet > Issue Type: Bug > Components: parquet-cli >Reporter: J Y >Priority: Major > Attachments: samples.tgz > > > i generated a parquet file using a protobuf with this proto definition: > {code:java} > message IndexPath { > // Index of item in path. > repeated int32 index = 1; > } > message SomeEvent { > // truncated/obfuscated wrapper > optional IndexPath client_position = 1; > } > {code} > this gets translated to the following parquet schema using the new compliant > schema for lists: > {code:java} > message SomeEvent { > optional group client_position = 1 { > optional group index (LIST) = 1 { > repeated group list { > required int32 element; > } > } > } > } > {code} > this causes parquet-cli cat to barf on a file containing these events: > {quote}java.lang.RuntimeException: Failed on record 0 > at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:86) > at org.apache.parquet.cli.Main.run(Main.java:157) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) > at org.apache.parquet.cli.Main.main(Main.java:187) > Caused by: java.lang.ClassCastException: required int32 element is not a group > at org.apache.parquet.schema.Type.asGroupType(Type.java:248) > at > org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:284) > at > org.
[jira] [Updated] (PARQUET-2181) parquet-cli fails at supporting parquet-protobuf generated schemas that have repeated primitives in them
[ https://issues.apache.org/jira/browse/PARQUET-2181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] J Y updated PARQUET-2181: - Priority: Critical (was: Major) > parquet-cli fails at supporting parquet-protobuf generated schemas that have > repeated primitives in them > > > Key: PARQUET-2181 > URL: https://issues.apache.org/jira/browse/PARQUET-2181 > Project: Parquet > Issue Type: Bug > Components: parquet-cli >Reporter: J Y >Priority: Critical > Attachments: samples.tgz > > > i generated a parquet file using a protobuf with this proto definition: > {code:java} > message IndexPath { > // Index of item in path. > repeated int32 index = 1; > } > message SomeEvent { > // truncated/obfuscated wrapper > optional IndexPath client_position = 1; > } > {code} > this gets translated to the following parquet schema using the new compliant > schema for lists: > {code:java} > message SomeEvent { > optional group client_position = 1 { > optional group index (LIST) = 1 { > repeated group list { > required int32 element; > } > } > } > } > {code} > this causes parquet-cli cat to barf on a file containing these events: > {quote}java.lang.RuntimeException: Failed on record 0 > at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:86) > at org.apache.parquet.cli.Main.run(Main.java:157) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) > at org.apache.parquet.cli.Main.main(Main.java:187) > Caused by: java.lang.ClassCastException: required int32 element is not a group > at org.apache.parquet.schema.Type.asGroupType(Type.java:248) > at > org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:284) > at > org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:228) > at > org.apache.parquet.avro.AvroRecordConverter.access$100(AvroRecordConverter.java:74) > at > org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter$ElementConverter.(AvroRecordConverter.java:539) > at > org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter.(AvroRecordConverter.java:489) > at > org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:293) > at > org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:137) > at > org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:284) > at > org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:137) > at > org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:91) > at > org.apache.parquet.avro.AvroRecordMaterializer.(AvroRecordMaterializer.java:33) > at > org.apache.parquet.avro.AvroReadSupport.prepareForRead(AvroReadSupport.java:142) > at > org.apache.parquet.hadoop.InternalParquetRecordReader.initialize(InternalParquetRecordReader.java:190) > at > org.apache.parquet.hadoop.ParquetReader.initReader(ParquetReader.java:166) > at > org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:135) > at > org.apache.parquet.cli.BaseCommand$1$1.advance(BaseCommand.java:363) > at org.apache.parquet.cli.BaseCommand$1$1.(BaseCommand.java:344) > at org.apache.parquet.cli.BaseCommand$1.iterator(BaseCommand.java:342) > at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:73) > ... 3 more > {quote} > using the old parquet-tools binary to cat this file works fine. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (PARQUET-2181) parquet-cli fails at supporting parquet-protobuf generated files
[ https://issues.apache.org/jira/browse/PARQUET-2181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] J Y updated PARQUET-2181: - Summary: parquet-cli fails at supporting parquet-protobuf generated files (was: parquet-cli fails at supporting parquet-protobuf generated schemas that have repeated primitives in them) > parquet-cli fails at supporting parquet-protobuf generated files > > > Key: PARQUET-2181 > URL: https://issues.apache.org/jira/browse/PARQUET-2181 > Project: Parquet > Issue Type: Bug > Components: parquet-cli >Reporter: J Y >Priority: Critical > Attachments: samples.tgz > > > i generated a parquet file using a protobuf with this proto definition: > {code:java} > message IndexPath { > // Index of item in path. > repeated int32 index = 1; > } > message SomeEvent { > // truncated/obfuscated wrapper > optional IndexPath client_position = 1; > } > {code} > this gets translated to the following parquet schema using the new compliant > schema for lists: > {code:java} > message SomeEvent { > optional group client_position = 1 { > optional group index (LIST) = 1 { > repeated group list { > required int32 element; > } > } > } > } > {code} > this causes parquet-cli cat to barf on a file containing these events: > {quote}java.lang.RuntimeException: Failed on record 0 > at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:86) > at org.apache.parquet.cli.Main.run(Main.java:157) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) > at org.apache.parquet.cli.Main.main(Main.java:187) > Caused by: java.lang.ClassCastException: required int32 element is not a group > at org.apache.parquet.schema.Type.asGroupType(Type.java:248) > at > org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:284) > at > org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:228) > at > org.apache.parquet.avro.AvroRecordConverter.access$100(AvroRecordConverter.java:74) > at > org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter$ElementConverter.(AvroRecordConverter.java:539) > at > org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter.(AvroRecordConverter.java:489) > at > org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:293) > at > org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:137) > at > org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:284) > at > org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:137) > at > org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:91) > at > org.apache.parquet.avro.AvroRecordMaterializer.(AvroRecordMaterializer.java:33) > at > org.apache.parquet.avro.AvroReadSupport.prepareForRead(AvroReadSupport.java:142) > at > org.apache.parquet.hadoop.InternalParquetRecordReader.initialize(InternalParquetRecordReader.java:190) > at > org.apache.parquet.hadoop.ParquetReader.initReader(ParquetReader.java:166) > at > org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:135) > at > org.apache.parquet.cli.BaseCommand$1$1.advance(BaseCommand.java:363) > at org.apache.parquet.cli.BaseCommand$1$1.(BaseCommand.java:344) > at org.apache.parquet.cli.BaseCommand$1.iterator(BaseCommand.java:342) > at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:73) > ... 3 more > {quote} > using the old parquet-tools binary to cat this file works fine. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (PARQUET-2181) parquet-cli fails at supporting parquet-protobuf generated files
[ https://issues.apache.org/jira/browse/PARQUET-2181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17598508#comment-17598508 ] J Y commented on PARQUET-2181: -- the more i think about this, the more i believe using avro reading as the basis for parquet reading is broken. for example, {{Argument error: Map key type must be binary (UTF8): required int32 key}} is due to avro requiring all map keys to be strings. parquet and protos do not have this limitation. avro as the schema definition doesn't seem expressive enough to easily interoperate with these other formats. > parquet-cli fails at supporting parquet-protobuf generated files > > > Key: PARQUET-2181 > URL: https://issues.apache.org/jira/browse/PARQUET-2181 > Project: Parquet > Issue Type: Bug > Components: parquet-cli >Reporter: J Y >Priority: Critical > Attachments: samples.tgz > > > i generated a parquet file using a protobuf with this proto definition: > {code:java} > message IndexPath { > // Index of item in path. > repeated int32 index = 1; > } > message SomeEvent { > // truncated/obfuscated wrapper > optional IndexPath client_position = 1; > } > {code} > this gets translated to the following parquet schema using the new compliant > schema for lists: > {code:java} > message SomeEvent { > optional group client_position = 1 { > optional group index (LIST) = 1 { > repeated group list { > required int32 element; > } > } > } > } > {code} > this causes parquet-cli cat to barf on a file containing these events: > {quote}java.lang.RuntimeException: Failed on record 0 > at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:86) > at org.apache.parquet.cli.Main.run(Main.java:157) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) > at org.apache.parquet.cli.Main.main(Main.java:187) > Caused by: java.lang.ClassCastException: required int32 element is not a group > at org.apache.parquet.schema.Type.asGroupType(Type.java:248) > at > org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:284) > at > org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:228) > at > org.apache.parquet.avro.AvroRecordConverter.access$100(AvroRecordConverter.java:74) > at > org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter$ElementConverter.(AvroRecordConverter.java:539) > at > org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter.(AvroRecordConverter.java:489) > at > org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:293) > at > org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:137) > at > org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:284) > at > org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:137) > at > org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:91) > at > org.apache.parquet.avro.AvroRecordMaterializer.(AvroRecordMaterializer.java:33) > at > org.apache.parquet.avro.AvroReadSupport.prepareForRead(AvroReadSupport.java:142) > at > org.apache.parquet.hadoop.InternalParquetRecordReader.initialize(InternalParquetRecordReader.java:190) > at > org.apache.parquet.hadoop.ParquetReader.initReader(ParquetReader.java:166) > at > org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:135) > at > org.apache.parquet.cli.BaseCommand$1$1.advance(BaseCommand.java:363) > at org.apache.parquet.cli.BaseCommand$1$1.(BaseCommand.java:344) > at org.apache.parquet.cli.BaseCommand$1.iterator(BaseCommand.java:342) > at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:73) > ... 3 more > {quote} > using the old parquet-tools binary to cat this file works fine. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (PARQUET-2181) parquet-cli fails at supporting parquet-protobuf generated files
[ https://issues.apache.org/jira/browse/PARQUET-2181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] J Y updated PARQUET-2181: - Attachment: sample-depth-1.tgz > parquet-cli fails at supporting parquet-protobuf generated files > > > Key: PARQUET-2181 > URL: https://issues.apache.org/jira/browse/PARQUET-2181 > Project: Parquet > Issue Type: Bug > Components: parquet-cli >Reporter: J Y >Priority: Critical > Attachments: sample-depth-1.tgz, samples.tgz > > > i generated a parquet file using a protobuf with this proto definition: > {code:java} > message IndexPath { > // Index of item in path. > repeated int32 index = 1; > } > message SomeEvent { > // truncated/obfuscated wrapper > optional IndexPath client_position = 1; > } > {code} > this gets translated to the following parquet schema using the new compliant > schema for lists: > {code:java} > message SomeEvent { > optional group client_position = 1 { > optional group index (LIST) = 1 { > repeated group list { > required int32 element; > } > } > } > } > {code} > this causes parquet-cli cat to barf on a file containing these events: > {quote}java.lang.RuntimeException: Failed on record 0 > at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:86) > at org.apache.parquet.cli.Main.run(Main.java:157) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) > at org.apache.parquet.cli.Main.main(Main.java:187) > Caused by: java.lang.ClassCastException: required int32 element is not a group > at org.apache.parquet.schema.Type.asGroupType(Type.java:248) > at > org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:284) > at > org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:228) > at > org.apache.parquet.avro.AvroRecordConverter.access$100(AvroRecordConverter.java:74) > at > org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter$ElementConverter.(AvroRecordConverter.java:539) > at > org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter.(AvroRecordConverter.java:489) > at > org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:293) > at > org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:137) > at > org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:284) > at > org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:137) > at > org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:91) > at > org.apache.parquet.avro.AvroRecordMaterializer.(AvroRecordMaterializer.java:33) > at > org.apache.parquet.avro.AvroReadSupport.prepareForRead(AvroReadSupport.java:142) > at > org.apache.parquet.hadoop.InternalParquetRecordReader.initialize(InternalParquetRecordReader.java:190) > at > org.apache.parquet.hadoop.ParquetReader.initReader(ParquetReader.java:166) > at > org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:135) > at > org.apache.parquet.cli.BaseCommand$1$1.advance(BaseCommand.java:363) > at org.apache.parquet.cli.BaseCommand$1$1.(BaseCommand.java:344) > at org.apache.parquet.cli.BaseCommand$1.iterator(BaseCommand.java:342) > at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:73) > ... 3 more > {quote} > using the old parquet-tools binary to cat this file works fine. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (PARQUET-2181) parquet-cli fails at supporting parquet-protobuf generated files
[ https://issues.apache.org/jira/browse/PARQUET-2181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17599721#comment-17599721 ] J Y commented on PARQUET-2181: -- so here's an interesting update. one of my colleagues who is more familiar with this area suggested that the schema itself is probably too large. i regenerated this data using a more limited recursion depth, and the parquet-cli tool worked fine. i'm attaching the set of files that work ([^sample-depth-1.tgz]). so it seems the core issue here is that there's a hard limit somewhere on the schema sizes... > parquet-cli fails at supporting parquet-protobuf generated files > > > Key: PARQUET-2181 > URL: https://issues.apache.org/jira/browse/PARQUET-2181 > Project: Parquet > Issue Type: Bug > Components: parquet-cli >Reporter: J Y >Priority: Critical > Attachments: sample-depth-1.tgz, samples.tgz > > > i generated a parquet file using a protobuf with this proto definition: > {code:java} > message IndexPath { > // Index of item in path. > repeated int32 index = 1; > } > message SomeEvent { > // truncated/obfuscated wrapper > optional IndexPath client_position = 1; > } > {code} > this gets translated to the following parquet schema using the new compliant > schema for lists: > {code:java} > message SomeEvent { > optional group client_position = 1 { > optional group index (LIST) = 1 { > repeated group list { > required int32 element; > } > } > } > } > {code} > this causes parquet-cli cat to barf on a file containing these events: > {quote}java.lang.RuntimeException: Failed on record 0 > at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:86) > at org.apache.parquet.cli.Main.run(Main.java:157) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) > at org.apache.parquet.cli.Main.main(Main.java:187) > Caused by: java.lang.ClassCastException: required int32 element is not a group > at org.apache.parquet.schema.Type.asGroupType(Type.java:248) > at > org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:284) > at > org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:228) > at > org.apache.parquet.avro.AvroRecordConverter.access$100(AvroRecordConverter.java:74) > at > org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter$ElementConverter.(AvroRecordConverter.java:539) > at > org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter.(AvroRecordConverter.java:489) > at > org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:293) > at > org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:137) > at > org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:284) > at > org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:137) > at > org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:91) > at > org.apache.parquet.avro.AvroRecordMaterializer.(AvroRecordMaterializer.java:33) > at > org.apache.parquet.avro.AvroReadSupport.prepareForRead(AvroReadSupport.java:142) > at > org.apache.parquet.hadoop.InternalParquetRecordReader.initialize(InternalParquetRecordReader.java:190) > at > org.apache.parquet.hadoop.ParquetReader.initReader(ParquetReader.java:166) > at > org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:135) > at > org.apache.parquet.cli.BaseCommand$1$1.advance(BaseCommand.java:363) > at org.apache.parquet.cli.BaseCommand$1$1.(BaseCommand.java:344) > at org.apache.parquet.cli.BaseCommand$1.iterator(BaseCommand.java:342) > at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:73) > ... 3 more > {quote} > using the old parquet-tools binary to cat this file works fine. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] (PARQUET-2181) parquet-cli fails at supporting parquet-protobuf generated files
[ https://issues.apache.org/jira/browse/PARQUET-2181 ] J Y deleted comment on PARQUET-2181: -- was (Author: jinyius): so here's an interesting update. one of my colleagues who is more familiar with this area suggested that the schema itself is probably too large. i regenerated this data using a more limited recursion depth, and the parquet-cli tool worked fine. i'm attaching the set of files that work ([^sample-depth-1.tgz]). so it seems the core issue here is that there's a hard limit somewhere on the schema sizes... > parquet-cli fails at supporting parquet-protobuf generated files > > > Key: PARQUET-2181 > URL: https://issues.apache.org/jira/browse/PARQUET-2181 > Project: Parquet > Issue Type: Bug > Components: parquet-cli >Reporter: J Y >Priority: Critical > Attachments: sample-depth-1.tgz, samples.tgz > > > i generated a parquet file using a protobuf with this proto definition: > {code:java} > message IndexPath { > // Index of item in path. > repeated int32 index = 1; > } > message SomeEvent { > // truncated/obfuscated wrapper > optional IndexPath client_position = 1; > } > {code} > this gets translated to the following parquet schema using the new compliant > schema for lists: > {code:java} > message SomeEvent { > optional group client_position = 1 { > optional group index (LIST) = 1 { > repeated group list { > required int32 element; > } > } > } > } > {code} > this causes parquet-cli cat to barf on a file containing these events: > {quote}java.lang.RuntimeException: Failed on record 0 > at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:86) > at org.apache.parquet.cli.Main.run(Main.java:157) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) > at org.apache.parquet.cli.Main.main(Main.java:187) > Caused by: java.lang.ClassCastException: required int32 element is not a group > at org.apache.parquet.schema.Type.asGroupType(Type.java:248) > at > org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:284) > at > org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:228) > at > org.apache.parquet.avro.AvroRecordConverter.access$100(AvroRecordConverter.java:74) > at > org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter$ElementConverter.(AvroRecordConverter.java:539) > at > org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter.(AvroRecordConverter.java:489) > at > org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:293) > at > org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:137) > at > org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:284) > at > org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:137) > at > org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:91) > at > org.apache.parquet.avro.AvroRecordMaterializer.(AvroRecordMaterializer.java:33) > at > org.apache.parquet.avro.AvroReadSupport.prepareForRead(AvroReadSupport.java:142) > at > org.apache.parquet.hadoop.InternalParquetRecordReader.initialize(InternalParquetRecordReader.java:190) > at > org.apache.parquet.hadoop.ParquetReader.initReader(ParquetReader.java:166) > at > org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:135) > at > org.apache.parquet.cli.BaseCommand$1$1.advance(BaseCommand.java:363) > at org.apache.parquet.cli.BaseCommand$1$1.(BaseCommand.java:344) > at org.apache.parquet.cli.BaseCommand$1.iterator(BaseCommand.java:342) > at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:73) > ... 3 more > {quote} > using the old parquet-tools binary to cat this file works fine. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (PARQUET-2181) parquet-cli fails at supporting parquet-protobuf generated files
[ https://issues.apache.org/jira/browse/PARQUET-2181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] J Y updated PARQUET-2181: - Attachment: (was: sample-depth-1.tgz) > parquet-cli fails at supporting parquet-protobuf generated files > > > Key: PARQUET-2181 > URL: https://issues.apache.org/jira/browse/PARQUET-2181 > Project: Parquet > Issue Type: Bug > Components: parquet-cli >Reporter: J Y >Priority: Critical > Attachments: samples.tgz > > > i generated a parquet file using a protobuf with this proto definition: > {code:java} > message IndexPath { > // Index of item in path. > repeated int32 index = 1; > } > message SomeEvent { > // truncated/obfuscated wrapper > optional IndexPath client_position = 1; > } > {code} > this gets translated to the following parquet schema using the new compliant > schema for lists: > {code:java} > message SomeEvent { > optional group client_position = 1 { > optional group index (LIST) = 1 { > repeated group list { > required int32 element; > } > } > } > } > {code} > this causes parquet-cli cat to barf on a file containing these events: > {quote}java.lang.RuntimeException: Failed on record 0 > at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:86) > at org.apache.parquet.cli.Main.run(Main.java:157) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) > at org.apache.parquet.cli.Main.main(Main.java:187) > Caused by: java.lang.ClassCastException: required int32 element is not a group > at org.apache.parquet.schema.Type.asGroupType(Type.java:248) > at > org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:284) > at > org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:228) > at > org.apache.parquet.avro.AvroRecordConverter.access$100(AvroRecordConverter.java:74) > at > org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter$ElementConverter.(AvroRecordConverter.java:539) > at > org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter.(AvroRecordConverter.java:489) > at > org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:293) > at > org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:137) > at > org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:284) > at > org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:137) > at > org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:91) > at > org.apache.parquet.avro.AvroRecordMaterializer.(AvroRecordMaterializer.java:33) > at > org.apache.parquet.avro.AvroReadSupport.prepareForRead(AvroReadSupport.java:142) > at > org.apache.parquet.hadoop.InternalParquetRecordReader.initialize(InternalParquetRecordReader.java:190) > at > org.apache.parquet.hadoop.ParquetReader.initReader(ParquetReader.java:166) > at > org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:135) > at > org.apache.parquet.cli.BaseCommand$1$1.advance(BaseCommand.java:363) > at org.apache.parquet.cli.BaseCommand$1$1.(BaseCommand.java:344) > at org.apache.parquet.cli.BaseCommand$1.iterator(BaseCommand.java:342) > at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:73) > ... 3 more > {quote} > using the old parquet-tools binary to cat this file works fine. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (PARQUET-2181) parquet-cli fails at supporting parquet-protobuf generated files
[ https://issues.apache.org/jira/browse/PARQUET-2181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17599874#comment-17599874 ] J Y commented on PARQUET-2181: -- here's the same data with a more limited recursion depth to keep the schema a more more manageable: [^sample-depth-1.tgz] > parquet-cli fails at supporting parquet-protobuf generated files > > > Key: PARQUET-2181 > URL: https://issues.apache.org/jira/browse/PARQUET-2181 > Project: Parquet > Issue Type: Bug > Components: parquet-cli >Reporter: J Y >Priority: Critical > Attachments: sample-depth-1.tgz, samples.tgz > > > i generated a parquet file using a protobuf with this proto definition: > {code:java} > message IndexPath { > // Index of item in path. > repeated int32 index = 1; > } > message SomeEvent { > // truncated/obfuscated wrapper > optional IndexPath client_position = 1; > } > {code} > this gets translated to the following parquet schema using the new compliant > schema for lists: > {code:java} > message SomeEvent { > optional group client_position = 1 { > optional group index (LIST) = 1 { > repeated group list { > required int32 element; > } > } > } > } > {code} > this causes parquet-cli cat to barf on a file containing these events: > {quote}java.lang.RuntimeException: Failed on record 0 > at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:86) > at org.apache.parquet.cli.Main.run(Main.java:157) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) > at org.apache.parquet.cli.Main.main(Main.java:187) > Caused by: java.lang.ClassCastException: required int32 element is not a group > at org.apache.parquet.schema.Type.asGroupType(Type.java:248) > at > org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:284) > at > org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:228) > at > org.apache.parquet.avro.AvroRecordConverter.access$100(AvroRecordConverter.java:74) > at > org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter$ElementConverter.(AvroRecordConverter.java:539) > at > org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter.(AvroRecordConverter.java:489) > at > org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:293) > at > org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:137) > at > org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:284) > at > org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:137) > at > org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:91) > at > org.apache.parquet.avro.AvroRecordMaterializer.(AvroRecordMaterializer.java:33) > at > org.apache.parquet.avro.AvroReadSupport.prepareForRead(AvroReadSupport.java:142) > at > org.apache.parquet.hadoop.InternalParquetRecordReader.initialize(InternalParquetRecordReader.java:190) > at > org.apache.parquet.hadoop.ParquetReader.initReader(ParquetReader.java:166) > at > org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:135) > at > org.apache.parquet.cli.BaseCommand$1$1.advance(BaseCommand.java:363) > at org.apache.parquet.cli.BaseCommand$1$1.(BaseCommand.java:344) > at org.apache.parquet.cli.BaseCommand$1.iterator(BaseCommand.java:342) > at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:73) > ... 3 more > {quote} > using the old parquet-tools binary to cat this file works fine. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (PARQUET-2181) parquet-cli fails at supporting parquet-protobuf generated files
[ https://issues.apache.org/jira/browse/PARQUET-2181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] J Y updated PARQUET-2181: - Attachment: sample-depth-1.tgz > parquet-cli fails at supporting parquet-protobuf generated files > > > Key: PARQUET-2181 > URL: https://issues.apache.org/jira/browse/PARQUET-2181 > Project: Parquet > Issue Type: Bug > Components: parquet-cli >Reporter: J Y >Priority: Critical > Attachments: sample-depth-1.tgz, samples.tgz > > > i generated a parquet file using a protobuf with this proto definition: > {code:java} > message IndexPath { > // Index of item in path. > repeated int32 index = 1; > } > message SomeEvent { > // truncated/obfuscated wrapper > optional IndexPath client_position = 1; > } > {code} > this gets translated to the following parquet schema using the new compliant > schema for lists: > {code:java} > message SomeEvent { > optional group client_position = 1 { > optional group index (LIST) = 1 { > repeated group list { > required int32 element; > } > } > } > } > {code} > this causes parquet-cli cat to barf on a file containing these events: > {quote}java.lang.RuntimeException: Failed on record 0 > at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:86) > at org.apache.parquet.cli.Main.run(Main.java:157) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) > at org.apache.parquet.cli.Main.main(Main.java:187) > Caused by: java.lang.ClassCastException: required int32 element is not a group > at org.apache.parquet.schema.Type.asGroupType(Type.java:248) > at > org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:284) > at > org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:228) > at > org.apache.parquet.avro.AvroRecordConverter.access$100(AvroRecordConverter.java:74) > at > org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter$ElementConverter.(AvroRecordConverter.java:539) > at > org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter.(AvroRecordConverter.java:489) > at > org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:293) > at > org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:137) > at > org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:284) > at > org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:137) > at > org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:91) > at > org.apache.parquet.avro.AvroRecordMaterializer.(AvroRecordMaterializer.java:33) > at > org.apache.parquet.avro.AvroReadSupport.prepareForRead(AvroReadSupport.java:142) > at > org.apache.parquet.hadoop.InternalParquetRecordReader.initialize(InternalParquetRecordReader.java:190) > at > org.apache.parquet.hadoop.ParquetReader.initReader(ParquetReader.java:166) > at > org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:135) > at > org.apache.parquet.cli.BaseCommand$1$1.advance(BaseCommand.java:363) > at org.apache.parquet.cli.BaseCommand$1$1.(BaseCommand.java:344) > at org.apache.parquet.cli.BaseCommand$1.iterator(BaseCommand.java:342) > at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:73) > ... 3 more > {quote} > using the old parquet-tools binary to cat this file works fine. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (PARQUET-2181) parquet-cli fails at supporting parquet-protobuf generated files
[ https://issues.apache.org/jira/browse/PARQUET-2181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17601491#comment-17601491 ] J Y commented on PARQUET-2181: -- [~theosib-amazon], i think they're similar in that there's schema issues w/ the avro conversion. i believe the root cause is that using avro internally to read parquet files loses expressiveness, so you get avro schema validation or mismatched schema issues as a consequence. specs-wise, you have to work around the limitations of avro's dsl to truly capture the parquet schema properly. i believe people typically don't hit this since the typical open source pattern is to start w/ an avro schema as the basis. for people who aren't, you'll have problems. > parquet-cli fails at supporting parquet-protobuf generated files > > > Key: PARQUET-2181 > URL: https://issues.apache.org/jira/browse/PARQUET-2181 > Project: Parquet > Issue Type: Bug > Components: parquet-cli >Reporter: J Y >Priority: Critical > Attachments: sample-depth-1.tgz, samples.tgz > > > i generated a parquet file using a protobuf with this proto definition: > {code:java} > message IndexPath { > // Index of item in path. > repeated int32 index = 1; > } > message SomeEvent { > // truncated/obfuscated wrapper > optional IndexPath client_position = 1; > } > {code} > this gets translated to the following parquet schema using the new compliant > schema for lists: > {code:java} > message SomeEvent { > optional group client_position = 1 { > optional group index (LIST) = 1 { > repeated group list { > required int32 element; > } > } > } > } > {code} > this causes parquet-cli cat to barf on a file containing these events: > {quote}java.lang.RuntimeException: Failed on record 0 > at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:86) > at org.apache.parquet.cli.Main.run(Main.java:157) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) > at org.apache.parquet.cli.Main.main(Main.java:187) > Caused by: java.lang.ClassCastException: required int32 element is not a group > at org.apache.parquet.schema.Type.asGroupType(Type.java:248) > at > org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:284) > at > org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:228) > at > org.apache.parquet.avro.AvroRecordConverter.access$100(AvroRecordConverter.java:74) > at > org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter$ElementConverter.(AvroRecordConverter.java:539) > at > org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter.(AvroRecordConverter.java:489) > at > org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:293) > at > org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:137) > at > org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:284) > at > org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:137) > at > org.apache.parquet.avro.AvroRecordConverter.(AvroRecordConverter.java:91) > at > org.apache.parquet.avro.AvroRecordMaterializer.(AvroRecordMaterializer.java:33) > at > org.apache.parquet.avro.AvroReadSupport.prepareForRead(AvroReadSupport.java:142) > at > org.apache.parquet.hadoop.InternalParquetRecordReader.initialize(InternalParquetRecordReader.java:190) > at > org.apache.parquet.hadoop.ParquetReader.initReader(ParquetReader.java:166) > at > org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:135) > at > org.apache.parquet.cli.BaseCommand$1$1.advance(BaseCommand.java:363) > at org.apache.parquet.cli.BaseCommand$1$1.(BaseCommand.java:344) > at org.apache.parquet.cli.BaseCommand$1.iterator(BaseCommand.java:342) > at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:73) > ... 3 more > {quote} > using the old parquet-tools binary to cat this file works fine. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (PARQUET-1711) [parquet-protobuf] stack overflow when work with well known json type
[ https://issues.apache.org/jira/browse/PARQUET-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17608085#comment-17608085 ] J Y commented on PARQUET-1711: -- ping.. the pr i wrote is the byte encoding version that [~emkornfi...@gmail.com] suggested in https://issues.apache.org/jira/browse/PARQUET-1711?focusedCommentId=17543672&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17543672 > [parquet-protobuf] stack overflow when work with well known json type > - > > Key: PARQUET-1711 > URL: https://issues.apache.org/jira/browse/PARQUET-1711 > Project: Parquet > Issue Type: Bug >Affects Versions: 1.10.1 >Reporter: Lawrence He >Priority: Major > > Writing following protobuf message as parquet file is not possible: > {code:java} > syntax = "proto3"; > import "google/protobuf/struct.proto"; > package test; > option java_outer_classname = "CustomMessage"; > message TestMessage { > map data = 1; > } {code} > Protobuf introduced "well known json type" such like > [ListValue|https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#listvalue] > to work around json schema conversion. > However writing above messages traps parquet writer into an infinite loop due > to the "general type" support in protobuf. Current implementation will keep > referencing 6 possible types defined in protobuf (null, bool, number, string, > struct, list) and entering infinite loop when referencing "struct". > {code:java} > java.lang.StackOverflowErrorjava.lang.StackOverflowError at > java.base/java.util.Arrays$ArrayItr.(Arrays.java:4418) at > java.base/java.util.Arrays$ArrayList.iterator(Arrays.java:4410) at > java.base/java.util.Collections$UnmodifiableCollection$1.(Collections.java:1044) > at > java.base/java.util.Collections$UnmodifiableCollection.iterator(Collections.java:1043) > at > org.apache.parquet.proto.ProtoSchemaConverter.convertFields(ProtoSchemaConverter.java:64) > at > org.apache.parquet.proto.ProtoSchemaConverter.addField(ProtoSchemaConverter.java:96) > at > org.apache.parquet.proto.ProtoSchemaConverter.convertFields(ProtoSchemaConverter.java:66) > at > org.apache.parquet.proto.ProtoSchemaConverter.addField(ProtoSchemaConverter.java:96) > at > org.apache.parquet.proto.ProtoSchemaConverter.convertFields(ProtoSchemaConverter.java:66) > at > org.apache.parquet.proto.ProtoSchemaConverter.addField(ProtoSchemaConverter.java:96) > at > org.apache.parquet.proto.ProtoSchemaConverter.convertFields(ProtoSchemaConverter.java:66) > at > org.apache.parquet.proto.ProtoSchemaConverter.addField(ProtoSchemaConverter.java:96) > at > org.apache.parquet.proto.ProtoSchemaConverter.convertFields(ProtoSchemaConverter.java:66) > at > org.apache.parquet.proto.ProtoSchemaConverter.addField(ProtoSchemaConverter.java:96) > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (PARQUET-1711) [parquet-protobuf] stack overflow when work with well known json type
[ https://issues.apache.org/jira/browse/PARQUET-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17638109#comment-17638109 ] J Y commented on PARQUET-1711: -- this should be resolved after #995 now. > [parquet-protobuf] stack overflow when work with well known json type > - > > Key: PARQUET-1711 > URL: https://issues.apache.org/jira/browse/PARQUET-1711 > Project: Parquet > Issue Type: Bug >Affects Versions: 1.10.1 >Reporter: Lawrence He >Priority: Major > > Writing following protobuf message as parquet file is not possible: > {code:java} > syntax = "proto3"; > import "google/protobuf/struct.proto"; > package test; > option java_outer_classname = "CustomMessage"; > message TestMessage { > map data = 1; > } {code} > Protobuf introduced "well known json type" such like > [ListValue|https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#listvalue] > to work around json schema conversion. > However writing above messages traps parquet writer into an infinite loop due > to the "general type" support in protobuf. Current implementation will keep > referencing 6 possible types defined in protobuf (null, bool, number, string, > struct, list) and entering infinite loop when referencing "struct". > {code:java} > java.lang.StackOverflowErrorjava.lang.StackOverflowError at > java.base/java.util.Arrays$ArrayItr.(Arrays.java:4418) at > java.base/java.util.Arrays$ArrayList.iterator(Arrays.java:4410) at > java.base/java.util.Collections$UnmodifiableCollection$1.(Collections.java:1044) > at > java.base/java.util.Collections$UnmodifiableCollection.iterator(Collections.java:1043) > at > org.apache.parquet.proto.ProtoSchemaConverter.convertFields(ProtoSchemaConverter.java:64) > at > org.apache.parquet.proto.ProtoSchemaConverter.addField(ProtoSchemaConverter.java:96) > at > org.apache.parquet.proto.ProtoSchemaConverter.convertFields(ProtoSchemaConverter.java:66) > at > org.apache.parquet.proto.ProtoSchemaConverter.addField(ProtoSchemaConverter.java:96) > at > org.apache.parquet.proto.ProtoSchemaConverter.convertFields(ProtoSchemaConverter.java:66) > at > org.apache.parquet.proto.ProtoSchemaConverter.addField(ProtoSchemaConverter.java:96) > at > org.apache.parquet.proto.ProtoSchemaConverter.convertFields(ProtoSchemaConverter.java:66) > at > org.apache.parquet.proto.ProtoSchemaConverter.addField(ProtoSchemaConverter.java:96) > at > org.apache.parquet.proto.ProtoSchemaConverter.convertFields(ProtoSchemaConverter.java:66) > at > org.apache.parquet.proto.ProtoSchemaConverter.addField(ProtoSchemaConverter.java:96) > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010)