Re: ArrayWritableGroupConverter
Anyone know anything about this? We have our errors stored in this data structure, so it's hard to debug when we can't analyze our errors :\ Shameless bump. On Thu, Aug 14, 2014 at 10:43 AM, Raymond Lau wrote: > Original Thrift schema: > > struct teststruct { > 1: optional string field1; > 2: optional string field2; > 3: optional string field3; > } > > struct mainstruct { > 1: optional list teststructs; > } > > This parquet file schema was generated: > > message ParquetSchema { > optional group teststructs { > repeated group teststruct_tuple { > optional binary field1; > optional binary field2; > optional binary field3; > } > } > } > > When i try to run queries involving this 'teststructs' column, I get this > error: > > Failed with exception java.io.IOException:java.lang.RuntimeException: > Invalid parquet hive schema: repeated group teststruct_tuple { > optional binary field1; > optional binary field2; > optional binary field3; > } > > > > On Thu, Aug 14, 2014 at 8:35 AM, Brock Noland wrote: > >> Hi, >> >> Can you share your parquet schema? >> >> Brock >> >> >> On Tue, Aug 12, 2014 at 5:06 PM, Raymond Lau >> wrote: >> >>> Hello. (First off, sorry if I accidentally posted to the wrong mailing >>> list before - dev - and you are getting this again) >>> >>> Regarding the ArrayWritableGroupConverter class: I was just wondering >>> how come the field count has to be either 1 or 2? I'm trying to read a >>> column where the amount is fields is 3 and I'm getting an invalid parquet >>> hive schema (in hive 0.12) error when I try to do so. It looks like it >>> links back to here. >>> >>> *https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ArrayWritableGroupConverter.java >>> <https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ArrayWritableGroupConverter.java>* >>> >>> >>> Thanks, >>> -Raymond >>> >> >> > > > -- > *Raymond Lau* > Software Engineer - Intern | > r...@ooyala.com | (925) 395-3806 > -- *Raymond Lau* Software Engineer - Intern | r...@ooyala.com | (925) 395-3806
Re: ArrayWritableGroupConverter
Original Thrift schema: struct teststruct { 1: optional string field1; 2: optional string field2; 3: optional string field3; } struct mainstruct { 1: optional list teststructs; } This parquet file schema was generated: message ParquetSchema { optional group teststructs { repeated group teststruct_tuple { optional binary field1; optional binary field2; optional binary field3; } } } When i try to run queries involving this 'teststructs' column, I get this error: Failed with exception java.io.IOException:java.lang.RuntimeException: Invalid parquet hive schema: repeated group teststruct_tuple { optional binary field1; optional binary field2; optional binary field3; } On Thu, Aug 14, 2014 at 8:35 AM, Brock Noland wrote: > Hi, > > Can you share your parquet schema? > > Brock > > > On Tue, Aug 12, 2014 at 5:06 PM, Raymond Lau > wrote: > >> Hello. (First off, sorry if I accidentally posted to the wrong mailing >> list before - dev - and you are getting this again) >> >> Regarding the ArrayWritableGroupConverter class: I was just wondering how >> come the field count has to be either 1 or 2? I'm trying to read a column >> where the amount is fields is 3 and I'm getting an invalid parquet hive >> schema (in hive 0.12) error when I try to do so. It looks like it links >> back to here. >> >> *https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ArrayWritableGroupConverter.java >> <https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ArrayWritableGroupConverter.java>* >> >> >> Thanks, >> -Raymond >> > > -- *Raymond Lau* Software Engineer - Intern | r...@ooyala.com | (925) 395-3806
Re: ArrayWritableGroupConverter
Hi, Can you share your parquet schema? Brock On Tue, Aug 12, 2014 at 5:06 PM, Raymond Lau wrote: > Hello. (First off, sorry if I accidentally posted to the wrong mailing > list before - dev - and you are getting this again) > > Regarding the ArrayWritableGroupConverter class: I was just wondering how > come the field count has to be either 1 or 2? I'm trying to read a column > where the amount is fields is 3 and I'm getting an invalid parquet hive > schema (in hive 0.12) error when I try to do so. It looks like it links > back to here. > > *https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ArrayWritableGroupConverter.java > <https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ArrayWritableGroupConverter.java>* > > > Thanks, > -Raymond >
ArrayWritableGroupConverter
Hello. (First off, sorry if I accidentally posted to the wrong mailing list before - dev - and you are getting this again) Regarding the ArrayWritableGroupConverter class: I was just wondering how come the field count has to be either 1 or 2? I'm trying to read a column where the amount is fields is 3 and I'm getting an invalid parquet hive schema (in hive 0.12) error when I try to do so. It looks like it links back to here. *https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ArrayWritableGroupConverter.java <https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ArrayWritableGroupConverter.java>* Thanks, -Raymond