One Mapper runs in ACID tables
Hi Team, I have created ACID tables in Hive , while executing queries found a very unusual behavior Only one mapper runs if we have only delta files and in order to optimize it we have to run major compactions to merge to base files Want to confirm why we have implemented it in this way and if any correction in understanding is required please update Thanks, Ashish Singh
Bucketing issue with Hive
Hi Friends, I am facing an issue with Buckets in Hive , Need some inputs / guidance We initially created one table and whenever query (simple aggregation query on one table) was executed multiple mappers were launched. Later on considering optimization perspective, we altered table to include bucketing on one column. Since then queries on this table are executing with single mapper and hence significant delay in production queries execution. As mentioned in hive documentation, altering table to include bucketing doesn't impact previous data, but this behavior was not expected. Table is ORC formatted and with Snappy Compression. Disk size of table is approx. 3.5 GB with 70 million records Thanks, Ashish Singh
Re: Review Request 28372: HIVE-8950: Add support in ParquetHiveSerde to create table schema from a parquet file
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28372/ --- (Updated April 30, 2015, 12:13 a.m.) Review request for hive. Changes --- Addressed Ryan's review comments. Summary (updated) - HIVE-8950: Add support in ParquetHiveSerde to create table schema from a parquet file Bugs: HIVE-8950 https://issues.apache.org/jira/browse/HIVE-8950 Repository: hive-git Description --- HIVE-8950: Add support in ParquetHiveSerde to create table schema from a parquet file Diffs (updated) - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 65ec1b94f081379d8c6abf0503b6c50305a35ef4 hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java 50e022da57575cc3020de34a0d34510f224a37de metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 3f267ff0eb20560c36a19b74353f9d6749c8b333 metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java b89b07a7460e456918f7cb8054f8f2ff1a752033 ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetSchemaReader.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetToHiveSchemaConverter.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java 7fd5e9612d4e3c9bf3b816bc48dbdbe59fb8a5a8 ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 39c48b52665e4de024cf2abc99e0637549867fa3 ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestParquetToHiveSchemaConverter.java PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_optional_elements_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_required_elements_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_single_field_struct_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema_ext.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_avro_array_of_primitives_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_decimal_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q PRE-CREATION ql/src/test/results/clientpositive/create_view_partitioned.q.out ebf9a6bc4f2321d7f539b7a445b3f279e3285b8a ql/src/test/results/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_optional_elements_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_required_elements_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_single_field_struct_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema_ext.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_avro_array_of_primitives_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_decimal_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q.out PRE-CREATION Diff: https://reviews.apache.org/r/28372/diff/ Testing --- Tested by adding appropriate qTests. Thanks, Ashish Singh
Re: Review Request 28372: Address review comments
> On April 28, 2015, 9:30 p.m., Ryan Blue wrote: > > Looking good to me. I found a few things to fix, and I'm also wondering > > where the qtest outputs are. Thanks for the review Ryan! qtest outputs are in *.q.out files. - Ashish --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28372/#review81884 --- On April 17, 2015, 3:01 a.m., Ashish Singh wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/28372/ > --- > > (Updated April 17, 2015, 3:01 a.m.) > > > Review request for hive. > > > Bugs: HIVE-8950 > https://issues.apache.org/jira/browse/HIVE-8950 > > > Repository: hive-git > > > Description > --- > > HIVE-8950: Add support in ParquetHiveSerde to create table schema from a > parquet file > > > Diffs > - > > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java > e138800e6dadd6fe76345f21eb76c906165c438d > > hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java > 53d913435cd43c96f044cf8668461fc686817ef4 > metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java > 3f267ff0eb20560c36a19b74353f9d6749c8b333 > metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java > 907ad2b8d7bae9b13eb4d9605ff2a3fe60c03ee8 > > ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetSchemaReader.java > PRE-CREATION > > ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetToHiveSchemaConverter.java > PRE-CREATION > > ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java > 7fd5e9612d4e3c9bf3b816bc48dbdbe59fb8a5a8 > ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java > a029f10c116fd46070b2d41790043f0a7001390f > > ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestParquetToHiveSchemaConverter.java > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_array_of_optional_elements_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_array_of_required_elements_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_array_of_single_field_struct_gen_schema.q > PRE-CREATION > ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema_ext.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_avro_array_of_primitives_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q > PRE-CREATION > ql/src/test/queries/clientpositive/parquet_decimal_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q > PRE-CREATION > ql/src/test/results/clientpositive/create_view_partitioned.q.out > ebf9a6bc4f2321d7f539b7a445b3f279e3285b8a > > ql/src/test/results/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/parquet_array_of_optional_elements_gen_schema.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/parquet_array_of_required_elements_gen_schema.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/parquet_array_of_single_field_struct_gen_schema.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema_ext.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/parquet_avro_array_of_primitives_gen_schema.q.out >
Re: Review Request 28372: Address review comments
> On Dec. 13, 2014, 1:20 a.m., Ryan Blue wrote: > > ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java, > > line 138 > > <https://reviews.apache.org/r/28372/diff/8/?file=790874#file790874line138> > > > > I'd rather not use parquet.file at all. Is there a compelling reason to > > keep it now that this can find files in the table location? > > Ashish Singh wrote: > parquet.file or a similar table property will be required to create a > parquet backed managed table. Hive expects atleast one column in the table, > which can either come from user specified hive schema or from table's serde. Removed parquet.file and restricted the automatic schema generation to external tables poiniting to dir with parquet data. - Ashish --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28372/#review65004 ------- On April 17, 2015, 3:01 a.m., Ashish Singh wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/28372/ > --- > > (Updated April 17, 2015, 3:01 a.m.) > > > Review request for hive. > > > Bugs: HIVE-8950 > https://issues.apache.org/jira/browse/HIVE-8950 > > > Repository: hive-git > > > Description > --- > > HIVE-8950: Add support in ParquetHiveSerde to create table schema from a > parquet file > > > Diffs > - > > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java > e138800e6dadd6fe76345f21eb76c906165c438d > > hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java > 53d913435cd43c96f044cf8668461fc686817ef4 > metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java > 3f267ff0eb20560c36a19b74353f9d6749c8b333 > metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java > 907ad2b8d7bae9b13eb4d9605ff2a3fe60c03ee8 > > ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetSchemaReader.java > PRE-CREATION > > ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetToHiveSchemaConverter.java > PRE-CREATION > > ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java > 7fd5e9612d4e3c9bf3b816bc48dbdbe59fb8a5a8 > ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java > a029f10c116fd46070b2d41790043f0a7001390f > > ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestParquetToHiveSchemaConverter.java > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_array_of_optional_elements_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_array_of_required_elements_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_array_of_single_field_struct_gen_schema.q > PRE-CREATION > ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema_ext.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_avro_array_of_primitives_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q > PRE-CREATION > ql/src/test/queries/clientpositive/parquet_decimal_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q > PRE-CREATION > ql/src/test/results/clientpositive/create_view_partitioned.q.out > ebf9a6bc4f2321d7f539b7a445b3f279e3285b8a > > ql/src/test/results/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/parquet_array_of_optional_elements_gen_schema.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/parquet_array_of_required_elements_gen_schema.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/parquet_array_of_single_field_struct_gen_schema.q.out
Re: Review Request 28372: Address review comments
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28372/ --- (Updated April 17, 2015, 3:01 a.m.) Review request for hive. Changes --- Removed parquet.file table property, restricting the capability of automatic schema generation to external tables. Summary (updated) - Address review comments Bugs: HIVE-8950 https://issues.apache.org/jira/browse/HIVE-8950 Repository: hive-git Description --- HIVE-8950: Add support in ParquetHiveSerde to create table schema from a parquet file Diffs (updated) - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e138800e6dadd6fe76345f21eb76c906165c438d hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java 53d913435cd43c96f044cf8668461fc686817ef4 metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 3f267ff0eb20560c36a19b74353f9d6749c8b333 metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 907ad2b8d7bae9b13eb4d9605ff2a3fe60c03ee8 ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetSchemaReader.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetToHiveSchemaConverter.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java 7fd5e9612d4e3c9bf3b816bc48dbdbe59fb8a5a8 ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java a029f10c116fd46070b2d41790043f0a7001390f ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestParquetToHiveSchemaConverter.java PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_optional_elements_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_required_elements_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_single_field_struct_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema_ext.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_avro_array_of_primitives_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_decimal_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q PRE-CREATION ql/src/test/results/clientpositive/create_view_partitioned.q.out ebf9a6bc4f2321d7f539b7a445b3f279e3285b8a ql/src/test/results/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_optional_elements_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_required_elements_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_single_field_struct_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema_ext.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_avro_array_of_primitives_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_decimal_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q.out PRE-CREATION Diff: https://reviews.apache.org/r/28372/diff/ Testing --- Tested by adding appropriate qTests. Thanks, Ashish Singh
Re: Review Request 28372: HIVE-8950: Add support in ParquetHiveSerde to create table schema from a parquet file
> On Dec. 16, 2014, 12:36 a.m., Brock Noland wrote: > > ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetToHiveSchemaConverter.java, > > line 163 > > <https://reviews.apache.org/r/28372/diff/8/?file=790873#file790873line163> > > > > Now we are treating MAP_KEY_VALUE as a string. Is that correct?? My bad. Fixed! - Ashish --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28372/#review65157 --- On Dec. 24, 2014, 4:19 a.m., Ashish Singh wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/28372/ > --- > > (Updated Dec. 24, 2014, 4:19 a.m.) > > > Review request for hive. > > > Bugs: HIVE-8950 > https://issues.apache.org/jira/browse/HIVE-8950 > > > Repository: hive-git > > > Description > --- > > HIVE-8950: Add support in ParquetHiveSerde to create table schema from a > parquet file > > > Diffs > - > > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java > fafd78e63e9b41c9fdb0e017b567dc719d151784 > > hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java > 32186391e7e4cfc9b4d06d7376663e82ec08d9e6 > metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java > b137fcb86e27ac91ed3c733b4d8788228d379a09 > metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java > 2db2658fbc57fba01c892c9213baef6c498e659b > > ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetSchemaReader.java > PRE-CREATION > > ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetToHiveSchemaConverter.java > PRE-CREATION > > ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java > 4effe736fcf9d3715f03eed9885c299a7aa040dd > ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java > cd3d349e8bd8785d6cadaf9ed8fa7598f223774a > > ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestParquetToHiveSchemaConverter.java > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_array_of_optional_elements_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_array_of_required_elements_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_array_of_single_field_struct_gen_schema.q > PRE-CREATION > ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema_ext.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_avro_array_of_primitives_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q > PRE-CREATION > ql/src/test/queries/clientpositive/parquet_decimal_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q > PRE-CREATION > ql/src/test/results/clientpositive/create_view_partitioned.q.out > ebf9a6bc4f2321d7f539b7a445b3f279e3285b8a > > ql/src/test/results/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/parquet_array_of_optional_elements_gen_schema.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/parquet_array_of_required_elements_gen_schema.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/parquet_array_of_single_field_struct_gen_schema.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema_ext.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/parquet_array_of_unannotated_primitives_gen_schema
Re: Review Request 28372: HIVE-8950: Add support in ParquetHiveSerde to create table schema from a parquet file
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28372/ --- (Updated Dec. 24, 2014, 4:19 a.m.) Review request for hive. Changes --- Add unit tests Bugs: HIVE-8950 https://issues.apache.org/jira/browse/HIVE-8950 Repository: hive-git Description --- HIVE-8950: Add support in ParquetHiveSerde to create table schema from a parquet file Diffs (updated) - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java fafd78e63e9b41c9fdb0e017b567dc719d151784 hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java 32186391e7e4cfc9b4d06d7376663e82ec08d9e6 metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java b137fcb86e27ac91ed3c733b4d8788228d379a09 metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 2db2658fbc57fba01c892c9213baef6c498e659b ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetSchemaReader.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetToHiveSchemaConverter.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java 4effe736fcf9d3715f03eed9885c299a7aa040dd ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java cd3d349e8bd8785d6cadaf9ed8fa7598f223774a ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestParquetToHiveSchemaConverter.java PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_optional_elements_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_required_elements_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_single_field_struct_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema_ext.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_avro_array_of_primitives_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_decimal_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q PRE-CREATION ql/src/test/results/clientpositive/create_view_partitioned.q.out ebf9a6bc4f2321d7f539b7a445b3f279e3285b8a ql/src/test/results/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_optional_elements_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_required_elements_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_single_field_struct_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema_ext.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_avro_array_of_primitives_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_decimal_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q.out PRE-CREATION Diff: https://reviews.apache.org/r/28372/diff/ Testing --- Tested by adding appropriate qTests. Thanks, Ashish Singh
Re: Review Request 28372: HIVE-8950: Add support in ParquetHiveSerde to create table schema from a parquet file
> On Dec. 13, 2014, 1:20 a.m., Ryan Blue wrote: > > I don't know why I didn't catch this earlier, but I'd also like to see a > > unit test for the new schema converter. Added unit tests. > On Dec. 13, 2014, 1:20 a.m., Ryan Blue wrote: > > ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java, > > line 141 > > <https://reviews.apache.org/r/28372/diff/8/?file=790874#file790874line141> > > > > Why does this use a location property instead of the SD location? tbl is just a Properties object, not actual Table object that has access to table's SD location. > On Dec. 13, 2014, 1:20 a.m., Ryan Blue wrote: > > ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java, line 692 > > <https://reviews.apache.org/r/28372/diff/8/?file=790875#file790875line692> > > > > This appears to store the location in the storage descriptor, so I > > think that the javadoc is no longer correct. Removed > On Dec. 13, 2014, 1:20 a.m., Ryan Blue wrote: > > ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java, > > line 138 > > <https://reviews.apache.org/r/28372/diff/8/?file=790874#file790874line138> > > > > I'd rather not use parquet.file at all. Is there a compelling reason to > > keep it now that this can find files in the table location? parquet.file or a similar table property will be required to create a parquet backed managed table. Hive expects atleast one column in the table, which can either come from user specified hive schema or from table's serde. > On Dec. 13, 2014, 1:20 a.m., Ryan Blue wrote: > > metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java, > > line 2294 > > <https://reviews.apache.org/r/28372/diff/8/?file=790870#file790870line2294> > > > > It isn't obvious why this is done or why deepCopy is used. Could you > > add a comment? Done. > On Dec. 13, 2014, 1:20 a.m., Ryan Blue wrote: > > ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetToHiveSchemaConverter.java, > > line 96 > > <https://reviews.apache.org/r/28372/diff/8/?file=790873#file790873line96> > > > > I'm not sure we want to assume that an int96 is a timestamp. That's all > > it's used for today, but maybe we should make it an opt-in so that users > > must set the hive type to timestamp explicitly. That way, we keep from > > assuming that int96 will always be a timestamp or building behavior we > > might have to break later. User can make hive assume parquet's int96 to map to hive's timestamp by setting "parquet.int96.is.timestamp" in table properties. - Ashish --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28372/#review65004 --- On Dec. 13, 2014, 12:03 a.m., Ashish Singh wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/28372/ > --- > > (Updated Dec. 13, 2014, 12:03 a.m.) > > > Review request for hive. > > > Bugs: HIVE-8950 > https://issues.apache.org/jira/browse/HIVE-8950 > > > Repository: hive-git > > > Description > --- > > HIVE-8950: Add support in ParquetHiveSerde to create table schema from a > parquet file > > > Diffs > - > > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java > fafd78e63e9b41c9fdb0e017b567dc719d151784 > > hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java > 32186391e7e4cfc9b4d06d7376663e82ec08d9e6 > metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java > b137fcb86e27ac91ed3c733b4d8788228d379a09 > metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java > 2db2658fbc57fba01c892c9213baef6c498e659b > > ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetSchemaReader.java > PRE-CREATION > > ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetToHiveSchemaConverter.java > PRE-CREATION > > ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java > 4effe736fcf9d3715f03eed9885c299a7aa040dd > ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java > cd3d349e8bd8785d6cadaf9ed8fa7598f223774a > > ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestParquetToHiveSchemaConverter.java > PRE-CREATION > > ql/src/test/queries/clientpositive/par
Re: Review Request 28372: HIVE-8950: Add support in ParquetHiveSerde to create table schema from a parquet file
> On Dec. 11, 2014, 2:41 a.m., Brock Noland wrote: > > This looks great!! I have a few comments below.. Thanks for the review. Addressed your comments. > On Dec. 11, 2014, 2:41 a.m., Brock Noland wrote: > > ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetToHiveSchemaConverter.java, > > line 96 > > <https://reviews.apache.org/r/28372/diff/7/?file=788512#file788512line96> > > > > Isn't this a timestamp? It is :). > On Dec. 11, 2014, 2:41 a.m., Brock Noland wrote: > > ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java, > > line 106 > > <https://reviews.apache.org/r/28372/diff/7/?file=788513#file788513line106> > > > > constants should be final I guess you meant static - Ashish --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28372/#review64671 ------- On Dec. 13, 2014, 12:03 a.m., Ashish Singh wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/28372/ > --- > > (Updated Dec. 13, 2014, 12:03 a.m.) > > > Review request for hive. > > > Bugs: HIVE-8950 > https://issues.apache.org/jira/browse/HIVE-8950 > > > Repository: hive-git > > > Description > --- > > HIVE-8950: Add support in ParquetHiveSerde to create table schema from a > parquet file > > > Diffs > - > > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java > fafd78e63e9b41c9fdb0e017b567dc719d151784 > > hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java > 32186391e7e4cfc9b4d06d7376663e82ec08d9e6 > metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java > b137fcb86e27ac91ed3c733b4d8788228d379a09 > metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java > 2db2658fbc57fba01c892c9213baef6c498e659b > > ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetSchemaReader.java > PRE-CREATION > > ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetToHiveSchemaConverter.java > PRE-CREATION > > ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java > 4effe736fcf9d3715f03eed9885c299a7aa040dd > ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java > cd3d349e8bd8785d6cadaf9ed8fa7598f223774a > > ql/src/test/queries/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_array_of_optional_elements_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_array_of_required_elements_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_array_of_single_field_struct_gen_schema.q > PRE-CREATION > ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema_ext.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_avro_array_of_primitives_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q > PRE-CREATION > ql/src/test/queries/clientpositive/parquet_decimal_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q > PRE-CREATION > ql/src/test/results/clientpositive/create_view_partitioned.q.out > ebf9a6bc4f2321d7f539b7a445b3f279e3285b8a > > ql/src/test/results/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/parquet_array_of_optional_elements_gen_schema.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/parquet_array_of_required_elements_gen_schema.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/parquet_array_of_single_field_struct_gen_schema.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema.q.out > PRE-CREATION > > ql/src/test/results/client
Re: Review Request 28372: HIVE-8950: Add support in ParquetHiveSerde to create table schema from a parquet file
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28372/ --- (Updated Dec. 13, 2014, 12:03 a.m.) Review request for hive. Changes --- Addressed review comments. Bugs: HIVE-8950 https://issues.apache.org/jira/browse/HIVE-8950 Repository: hive-git Description --- HIVE-8950: Add support in ParquetHiveSerde to create table schema from a parquet file Diffs (updated) - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java fafd78e63e9b41c9fdb0e017b567dc719d151784 hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java 32186391e7e4cfc9b4d06d7376663e82ec08d9e6 metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java b137fcb86e27ac91ed3c733b4d8788228d379a09 metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 2db2658fbc57fba01c892c9213baef6c498e659b ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetSchemaReader.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetToHiveSchemaConverter.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java 4effe736fcf9d3715f03eed9885c299a7aa040dd ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java cd3d349e8bd8785d6cadaf9ed8fa7598f223774a ql/src/test/queries/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_optional_elements_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_required_elements_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_single_field_struct_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema_ext.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_avro_array_of_primitives_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_decimal_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q PRE-CREATION ql/src/test/results/clientpositive/create_view_partitioned.q.out ebf9a6bc4f2321d7f539b7a445b3f279e3285b8a ql/src/test/results/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_optional_elements_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_required_elements_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_single_field_struct_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema_ext.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_avro_array_of_primitives_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_decimal_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q.out PRE-CREATION Diff: https://reviews.apache.org/r/28372/diff/ Testing --- Tested by adding appropriate qTests. Thanks, Ashish Singh
Re: Review Request 28372: HIVE-8950: Add support in ParquetHiveSerde to create table schema from a parquet file
> On Dec. 9, 2014, 11:47 p.m., Ryan Blue wrote: > > ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java, > > line 144 > > <https://reviews.apache.org/r/28372/diff/5/?file=785269#file785269line144> > > > > There's a comment in the Avro SerDe that it can't throw an Exception > > because the SerDe is loaded when trying to change table properties. So an > > invalid table with a SerDe that throws an Exception can't be changed > > through the API. This is why Avro returns an invalid schema that is named > > something like "error_error_error_error_error_error_error". Here, maybe we > > can just return an empty schema? It was my bad. Planned on doing it, but slipped out of my mind. Thanks for pointing this out. - Ashish --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28372/#review64469 ------- On Dec. 10, 2014, 8:34 p.m., Ashish Singh wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/28372/ > --- > > (Updated Dec. 10, 2014, 8:34 p.m.) > > > Review request for hive. > > > Bugs: HIVE-8950 > https://issues.apache.org/jira/browse/HIVE-8950 > > > Repository: hive-git > > > Description > --- > > HIVE-8950: Add support in ParquetHiveSerde to create table schema from a > parquet file > > > Diffs > - > > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java > fafd78e63e9b41c9fdb0e017b567dc719d151784 > > hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java > 32186391e7e4cfc9b4d06d7376663e82ec08d9e6 > metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java > b137fcb86e27ac91ed3c733b4d8788228d379a09 > metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java > 2db2658fbc57fba01c892c9213baef6c498e659b > > ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetSchemaReader.java > PRE-CREATION > > ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetToHiveSchemaConverter.java > PRE-CREATION > > ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java > 4effe736fcf9d3715f03eed9885c299a7aa040dd > ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java > cd3d349e8bd8785d6cadaf9ed8fa7598f223774a > > ql/src/test/queries/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_array_of_optional_elements_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_array_of_required_elements_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_array_of_single_field_struct_gen_schema.q > PRE-CREATION > ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema_ext.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_avro_array_of_primitives_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q > PRE-CREATION > ql/src/test/queries/clientpositive/parquet_decimal_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q > PRE-CREATION > ql/src/test/results/clientpositive/create_view_partitioned.q.out > ebf9a6bc4f2321d7f539b7a445b3f279e3285b8a > > ql/src/test/results/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/parquet_array_of_optional_elements_gen_schema.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/parquet_array_of_required_elements_gen_schema.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/parquet_array_of_single_field_struct_gen_schema.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema.q.out > PRE-CREATION > > ql/src/t
Re: Review Request 28372: HIVE-8950: Add support in ParquetHiveSerde to create table schema from a parquet file
> On Dec. 9, 2014, 11:45 p.m., Ryan Blue wrote: > > ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetToHiveSchemaConverter.java, > > line 111 > > <https://reviews.apache.org/r/28372/diff/5/?file=785268#file785268line111> > > > > I think the check for DECIMAL needs to be outside of this check. > > DECIMAL can be used to annotate int32, int64, fixed, and bytes. Catching it > > before this means we don't have to have this check in each convertX method. > > > > Also note that we validate the metadata as we read it from the file, so > > we can rely on the fact that if a field is annotated with DECIMAL, its type > > is compatible with DECIMAL. Good point! - Ashish --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28372/#review64467 ------- On Dec. 10, 2014, 8:34 p.m., Ashish Singh wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/28372/ > --- > > (Updated Dec. 10, 2014, 8:34 p.m.) > > > Review request for hive. > > > Bugs: HIVE-8950 > https://issues.apache.org/jira/browse/HIVE-8950 > > > Repository: hive-git > > > Description > --- > > HIVE-8950: Add support in ParquetHiveSerde to create table schema from a > parquet file > > > Diffs > - > > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java > fafd78e63e9b41c9fdb0e017b567dc719d151784 > > hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java > 32186391e7e4cfc9b4d06d7376663e82ec08d9e6 > metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java > b137fcb86e27ac91ed3c733b4d8788228d379a09 > metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java > 2db2658fbc57fba01c892c9213baef6c498e659b > > ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetSchemaReader.java > PRE-CREATION > > ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetToHiveSchemaConverter.java > PRE-CREATION > > ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java > 4effe736fcf9d3715f03eed9885c299a7aa040dd > ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java > cd3d349e8bd8785d6cadaf9ed8fa7598f223774a > > ql/src/test/queries/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_array_of_optional_elements_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_array_of_required_elements_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_array_of_single_field_struct_gen_schema.q > PRE-CREATION > ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema_ext.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_avro_array_of_primitives_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q > PRE-CREATION > ql/src/test/queries/clientpositive/parquet_decimal_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q > PRE-CREATION > ql/src/test/results/clientpositive/create_view_partitioned.q.out > ebf9a6bc4f2321d7f539b7a445b3f279e3285b8a > > ql/src/test/results/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/parquet_array_of_optional_elements_gen_schema.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/parquet_array_of_required_elements_gen_schema.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/parquet_array_of_single_field_struct_gen_schema.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema_ext.q.out
Re: Review Request 28372: HIVE-8950: Add support in ParquetHiveSerde to create table schema from a parquet file
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28372/ --- (Updated Dec. 10, 2014, 8:34 p.m.) Review request for hive. Changes --- Address review comments. Bugs: HIVE-8950 https://issues.apache.org/jira/browse/HIVE-8950 Repository: hive-git Description --- HIVE-8950: Add support in ParquetHiveSerde to create table schema from a parquet file Diffs (updated) - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java fafd78e63e9b41c9fdb0e017b567dc719d151784 hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java 32186391e7e4cfc9b4d06d7376663e82ec08d9e6 metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java b137fcb86e27ac91ed3c733b4d8788228d379a09 metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 2db2658fbc57fba01c892c9213baef6c498e659b ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetSchemaReader.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetToHiveSchemaConverter.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java 4effe736fcf9d3715f03eed9885c299a7aa040dd ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java cd3d349e8bd8785d6cadaf9ed8fa7598f223774a ql/src/test/queries/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_optional_elements_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_required_elements_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_single_field_struct_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema_ext.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_avro_array_of_primitives_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_decimal_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q PRE-CREATION ql/src/test/results/clientpositive/create_view_partitioned.q.out ebf9a6bc4f2321d7f539b7a445b3f279e3285b8a ql/src/test/results/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_optional_elements_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_required_elements_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_single_field_struct_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema_ext.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_avro_array_of_primitives_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_decimal_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q.out PRE-CREATION Diff: https://reviews.apache.org/r/28372/diff/ Testing --- Tested by adding appropriate qTests. Thanks, Ashish Singh
Re: Review Request 28372: HIVE-8950: Add support in ParquetHiveSerde to create table schema from a parquet file
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28372/ --- (Updated Dec. 10, 2014, 12:47 a.m.) Review request for hive. Changes --- Move to earlier version to pass table location to serde. Bugs: HIVE-8950 https://issues.apache.org/jira/browse/HIVE-8950 Repository: hive-git Description --- HIVE-8950: Add support in ParquetHiveSerde to create table schema from a parquet file Diffs (updated) - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java fafd78e63e9b41c9fdb0e017b567dc719d151784 hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java 32186391e7e4cfc9b4d06d7376663e82ec08d9e6 metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java b137fcb86e27ac91ed3c733b4d8788228d379a09 metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 2db2658fbc57fba01c892c9213baef6c498e659b ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetSchemaReader.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetToHiveSchemaConverter.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java 4effe736fcf9d3715f03eed9885c299a7aa040dd ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java cd3d349e8bd8785d6cadaf9ed8fa7598f223774a ql/src/test/queries/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_optional_elements_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_required_elements_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_single_field_struct_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema_ext.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_avro_array_of_primitives_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_decimal_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q PRE-CREATION ql/src/test/results/clientpositive/create_view_partitioned.q.out ebf9a6bc4f2321d7f539b7a445b3f279e3285b8a ql/src/test/results/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_optional_elements_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_required_elements_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_single_field_struct_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema_ext.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_avro_array_of_primitives_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_decimal_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q.out PRE-CREATION Diff: https://reviews.apache.org/r/28372/diff/ Testing --- Tested by adding appropriate qTests. Thanks, Ashish Singh
Re: Review Request 28372: HIVE-8950: Add support in ParquetHiveSerde to create table schema from a parquet file
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28372/ --- (Updated Dec. 8, 2014, 5:50 a.m.) Review request for hive. Changes --- Avoid changing hundreds of qtests' expected output by saving table location in table properties rather than serdeinfo. Bugs: HIVE-8950 https://issues.apache.org/jira/browse/HIVE-8950 Repository: hive-git Description --- HIVE-8950: Add support in ParquetHiveSerde to create table schema from a parquet file Diffs (updated) - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java fafd78e63e9b41c9fdb0e017b567dc719d151784 metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 2db2658fbc57fba01c892c9213baef6c498e659b ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetSchemaReader.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetToHiveSchemaConverter.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java 4effe736fcf9d3715f03eed9885c299a7aa040dd ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java cd3d349e8bd8785d6cadaf9ed8fa7598f223774a ql/src/test/queries/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_optional_elements_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_required_elements_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_single_field_struct_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema_ext.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_avro_array_of_primitives_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_decimal_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_optional_elements_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_required_elements_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_single_field_struct_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema_ext.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_avro_array_of_primitives_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_decimal_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q.out PRE-CREATION Diff: https://reviews.apache.org/r/28372/diff/ Testing --- Tested by adding appropriate qTests. Thanks, Ashish Singh
Re: Review Request 28372: HIVE-8950: Add support in ParquetHiveSerde to create table schema from a parquet file
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28372/ --- (Updated Dec. 8, 2014, 1:06 a.m.) Review request for hive. Changes --- Fix test failures. Bugs: HIVE-8950 https://issues.apache.org/jira/browse/HIVE-8950 Repository: hive-git Description --- HIVE-8950: Add support in ParquetHiveSerde to create table schema from a parquet file Diffs (updated) - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java fafd78e63e9b41c9fdb0e017b567dc719d151784 metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 2db2658fbc57fba01c892c9213baef6c498e659b ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetSchemaReader.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetToHiveSchemaConverter.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java 4effe736fcf9d3715f03eed9885c299a7aa040dd ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java cd3d349e8bd8785d6cadaf9ed8fa7598f223774a ql/src/test/queries/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_optional_elements_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_required_elements_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_single_field_struct_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema_ext.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_avro_array_of_primitives_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_decimal_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q PRE-CREATION ql/src/test/results/clientnegative/alter_view_as_select_with_partition.q.out 447dc3aeba921a3c56ba587a52cd5624599b4499 ql/src/test/results/clientpositive/create_view_partitioned.q.out ebf9a6bc4f2321d7f539b7a445b3f279e3285b8a ql/src/test/results/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_optional_elements_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_required_elements_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_single_field_struct_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema_ext.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_avro_array_of_primitives_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_decimal_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q.out PRE-CREATION Diff: https://reviews.apache.org/r/28372/diff/ Testing --- Tested by adding appropriate qTests. Thanks, Ashish Singh
Re: Review Request 28372: HIVE-8950: Add support in ParquetHiveSerde to create table schema from a parquet file
> On Dec. 1, 2014, 8:13 p.m., Ryan Blue wrote: > > Ryan, thanks for the review. Made appropriate changes based on your comments. Kindly take a look. - Ashish --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28372/#review63424 --- On Dec. 7, 2014, 1:40 a.m., Ashish Singh wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/28372/ > --- > > (Updated Dec. 7, 2014, 1:40 a.m.) > > > Review request for hive. > > > Bugs: HIVE-8950 > https://issues.apache.org/jira/browse/HIVE-8950 > > > Repository: hive-git > > > Description > --- > > HIVE-8950: Add support in ParquetHiveSerde to create table schema from a > parquet file > > > Diffs > - > > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java > fafd78e63e9b41c9fdb0e017b567dc719d151784 > metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java > 2db2658fbc57fba01c892c9213baef6c498e659b > > ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetSchemaReader.java > PRE-CREATION > > ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetToHiveSchemaConverter.java > PRE-CREATION > > ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java > 4effe736fcf9d3715f03eed9885c299a7aa040dd > ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java > cd3d349e8bd8785d6cadaf9ed8fa7598f223774a > > ql/src/test/queries/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_array_of_optional_elements_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_array_of_required_elements_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_array_of_single_field_struct_gen_schema.q > PRE-CREATION > ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema_ext.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_avro_array_of_primitives_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q > PRE-CREATION > ql/src/test/queries/clientpositive/parquet_decimal_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q > PRE-CREATION > > ql/src/test/results/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/parquet_array_of_optional_elements_gen_schema.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/parquet_array_of_required_elements_gen_schema.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/parquet_array_of_single_field_struct_gen_schema.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema_ext.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/parquet_avro_array_of_primitives_gen_schema.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q.out > PRE-CREATION > ql/src/test/results/clientpositive/parquet_decimal_gen_schema.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q.out > PRE-CREATION > > Diff: https://reviews.apache.org/r/28372/diff/ > > > Testing > --- > > Tested by adding appropriate qTests. > > > Thanks, > > Ashish Singh > >
Re: Review Request 28372: HIVE-8950: Add support in ParquetHiveSerde to create table schema from a parquet file
> On Dec. 1, 2014, 8:13 p.m., Ryan Blue wrote: > > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java, line 762 > > <https://reviews.apache.org/r/28372/diff/2/?file=777120#file777120line762> > > > > What does this change do? Serdes in this list must have one or more columns in the create table statement. This change removes this check for ParquetHiveSerDe. > On Dec. 1, 2014, 8:13 p.m., Ryan Blue wrote: > > ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ColInfoFromParquetFile.java, > > line 36 > > <https://reviews.apache.org/r/28372/diff/2/?file=777121#file777121line36> > > > > I don't think ColInfoFromParquetFile is a great name for this class. > > What you've implemented here is a schema converter, like > > HiveSchemaConverter but from MessageType to TypeInfo rather than the other > > way around. I think this should either go in HiveSchemaConverter or a new > > class, like ParquetToHiveSchemaConverter. > > > > This update would also help clarify the methods exposed. I think that > > the primary method this should expose is: > > StructType convert(GroupType parquetSchema) > > > > File reading should be done elsewhere if it is necessary. > > > > Also, I think this should return a StructType instead of a Pair with > > Strings. That would be a lot cleaner and would avoid the custom type string > > building immediately followed by parsing those type strings. Made appropriate changes. > On Dec. 1, 2014, 8:13 p.m., Ryan Blue wrote: > > ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java, > > line 107 > > <https://reviews.apache.org/r/28372/diff/2/?file=777122#file777122line107> > > > > Is it possible to avoid setting a file property? There are lots of > > cases where a file in the dataset would be removed so this is a brittle > > method of configuring the table. Ideally, we would check for the > > LIST_COLUMN_TYPES property and if we don't find it, conver the schema of > > the first file that we find. Below are the different scenarios possible and how they will be handled. 1. Managed table with parquet.file specified: 1.a. While creating the table ParquetHiveSerDe will use that file to create hive schema. 1.b. If at a later point of time, say while select data from table, file specified by parquet.file is deleted, then ParquetHiveSerDe will look for a file in table dir and will try to create hive schema from that file's parquet metadata. 1.c. If parquet.file is deleted and table dir is empty, there is no way hive schema can be created. So, an informative exception will be thrown, "Either provide schema for table or point to parquet file using parquet.file in tblproperties or make sure that table has atleast one parquet file with required metadata". Note that if a table reaches this state, it can be fixed either by altering table and pointing parquet.file to a valid file or by copying a parquet file to table dir. This is valid for all cases. 2. Managed table without parquet.file specified: 2.a. While creating the table ParquetHiveSerDe will use any file in table dir to create hive schema. If table dir is empty, it will throw an informative exception. 3. External table with parquet.file specified: Same as (1). 4. External table without parquet.file specified and empty table dir: Same as (1.c) 5. External table without parquet.file specified and a parquet file present in table dir: Same as (2) - Ashish ------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28372/#review63424 --- On Nov. 27, 2014, 1:08 a.m., Ashish Singh wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/28372/ > --- > > (Updated Nov. 27, 2014, 1:08 a.m.) > > > Review request for hive. > > > Bugs: HIVE-8950 > https://issues.apache.org/jira/browse/HIVE-8950 > > > Repository: hive-git > > > Description > --- > > HIVE-8950: Add support in ParquetHiveSerde to create table schema from a > parquet file > > > Diffs > - > > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java > fafd78e63e9b41c9fdb0e017b567dc719d151784 > metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java > 2db2658fbc57fba01c892c9213baef6c498e659b > > ql/src/java/org/apache/hadoop/hive/ql/io/parquet/
Re: Review Request 28372: HIVE-8950: Add support in ParquetHiveSerde to create table schema from a parquet file
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28372/ --- (Updated Dec. 7, 2014, 1:40 a.m.) Review request for hive. Changes --- Addressed review comments. Bugs: HIVE-8950 https://issues.apache.org/jira/browse/HIVE-8950 Repository: hive-git Description --- HIVE-8950: Add support in ParquetHiveSerde to create table schema from a parquet file Diffs (updated) - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java fafd78e63e9b41c9fdb0e017b567dc719d151784 metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 2db2658fbc57fba01c892c9213baef6c498e659b ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetSchemaReader.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetToHiveSchemaConverter.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java 4effe736fcf9d3715f03eed9885c299a7aa040dd ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java cd3d349e8bd8785d6cadaf9ed8fa7598f223774a ql/src/test/queries/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_optional_elements_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_required_elements_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_single_field_struct_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema_ext.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_avro_array_of_primitives_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_decimal_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_optional_elements_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_required_elements_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_single_field_struct_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema_ext.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_avro_array_of_primitives_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_decimal_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q.out PRE-CREATION Diff: https://reviews.apache.org/r/28372/diff/ Testing --- Tested by adding appropriate qTests. Thanks, Ashish Singh
Re: Review Request 28372: HIVE-8950: Add support in ParquetHiveSerde to create table schema from a parquet file
> On Nov. 25, 2014, 12:37 a.m., Sergio Pena wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ColInfoFromParquetFile.java,
> > line 35
> > <https://reviews.apache.org/r/28372/diff/1/?file=773791#file773791line35>
> >
> > This class will need more work in order to detect unannotated types as
> > specified in the following tickets:
> > https://issues.apache.org/jira/browse/HIVE-8909
> >
> > https://github.com/rdblue/incubator-parquet-format/blob/PARQUET-113-add-list-and-map-spec/LogicalTypes.md
> >
> > I was going to add more comments, but I then noticed that this code
> > will look as little similar (for loops, and converters) to the HIVE-8909
> > patch. Of course, the HIVE-8909 patch returns converter objects, and this
> > returns the column names & types. So, I was thinking if we can make use of
> > the DataWritableRecordConverter.java class to get the correct converters,
> > and then translate the converters to column names & types.
> >
> > This is what I found while debugging valid parquet files:
> > Each example has 3 blocks:
> > - parquet file schema
> > - hive columne names & types
> > - converters returned by DataWritableRecordConverter
> >
> > Could we use the converter objects and translate them to names?
> >
> > message SingleFieldGroupInList {
> > optional group single_element_groups (LIST) {
> > repeated group single_element_group {
> > required int64 count;
> > }
> > }
> > }
> >
> > single_element_groups ARRAY
> >
> > hivestructconverter
> > hivecollectionconverter:elementconverterarray<>
> > EINT64_CONVERTER
> > bigint
> >
> >
> >
> >
> > message HiveRequiredGroupInList {
> > optional group locations (LIST) {
> > repeated group bag {
> > required group element {
> > required double latitude;
> > required double longitude;
> > }
> > }
> > }
> > }
> >
> > locations ARRAY>
> >
> > hivestructconverter
> > hivecollectionconverter:elementconverterarray<>
> > hivestructconverter
> > struct<>
> > DOUBLE_CONVERTER
> > double
> > DOUBLE_CONVERTER
> > double
> >
> >
> >
> >
> >
> > message UnannotatedListOfPrimitives {
> > repeated int32 list_of_ints;
> > }
> >
> > list_of_ints ARRAY
> >
> > hivestructconverter
> > RepeatedPrimitiveConverter
> > array<>
> > EINT32_CONVERTER
> > int
> >
> >
> >
Sergio, thanks for the review and valuable comments!
DataWritableRecordConverter expects hive schema. So, not sure if that can be
used to create hive schema. I have updated the patch to take care of rules
listed on
"https://github.com/rdblue/incubator-parquet-format/blob/PARQUET-113-add-list-and-map-spec/LogicalTypes.md";.
Kindly take a look.
- Ashish
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28372/#review62911
---
On Nov. 27, 2014, 1:08 a.m., Ashish Singh wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/28372/
> ---
Re: Review Request 28372: HIVE-8950: Add support in ParquetHiveSerde to create table schema from a parquet file
> On Nov. 24, 2014, 10:44 p.m., Sergio Pena wrote: > > ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java, > > line 107 > > <https://reviews.apache.org/r/28372/diff/1/?file=773792#file773792line107> > > > > Avro uses avro.schema.url to specify the file that will be used for the > > schema. What if we use the same Avro synatix to specify the parquet schema? > > > > Something like: parquet.schema.url The reason I did not call it "parquet.schema.url", as unlike avro schema file which contains only avro schema, parquet file will contain data as well. It might be misleading for users as there is no concept of parquet schema file. However, advantage of using "parquet.schema.url" will be that usage becomes uniform between avro and parquet. Let me know what you think based on the discussion above. - Ashish --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28372/#review62889 ------- On Nov. 27, 2014, 1:08 a.m., Ashish Singh wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/28372/ > --- > > (Updated Nov. 27, 2014, 1:08 a.m.) > > > Review request for hive. > > > Bugs: HIVE-8950 > https://issues.apache.org/jira/browse/HIVE-8950 > > > Repository: hive-git > > > Description > --- > > HIVE-8950: Add support in ParquetHiveSerde to create table schema from a > parquet file > > > Diffs > - > > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java > fafd78e63e9b41c9fdb0e017b567dc719d151784 > > ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ColInfoFromParquetFile.java > PRE-CREATION > > ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java > 4effe736fcf9d3715f03eed9885c299a7aa040dd > > ql/src/test/queries/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_array_of_optional_elements_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_array_of_required_elements_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_array_of_single_field_struct_gen_schema.q > PRE-CREATION > ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_avro_array_of_primitives_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q > PRE-CREATION > ql/src/test/queries/clientpositive/parquet_decimal_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q > PRE-CREATION > > ql/src/test/queries/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q > PRE-CREATION > > ql/src/test/results/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/parquet_array_of_optional_elements_gen_schema.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/parquet_array_of_required_elements_gen_schema.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/parquet_array_of_single_field_struct_gen_schema.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/parquet_avro_array_of_primitives_gen_schema.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q.out > PRE-CREATION > ql/src/test/results/clientpositive/parquet_decimal_gen_schema.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q.out > PRE-CREATION > > Diff: https://reviews.apache.org/r/28372/diff/ > > > Testing > --- > > Tested by adding appropriate qTests. > > > Thanks, > > Ashish Singh > >
Re: Review Request 28372: HIVE-8950: Add support in ParquetHiveSerde to create table schema from a parquet file
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28372/ --- (Updated Nov. 27, 2014, 1:08 a.m.) Review request for hive. Changes --- Add more qtests. Bugs: HIVE-8950 https://issues.apache.org/jira/browse/HIVE-8950 Repository: hive-git Description --- HIVE-8950: Add support in ParquetHiveSerde to create table schema from a parquet file Diffs (updated) - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java fafd78e63e9b41c9fdb0e017b567dc719d151784 ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ColInfoFromParquetFile.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java 4effe736fcf9d3715f03eed9885c299a7aa040dd ql/src/test/queries/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_optional_elements_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_required_elements_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_single_field_struct_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_avro_array_of_primitives_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_decimal_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_optional_elements_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_required_elements_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_single_field_struct_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_avro_array_of_primitives_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_decimal_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q.out PRE-CREATION Diff: https://reviews.apache.org/r/28372/diff/ Testing --- Tested by adding appropriate qTests. Thanks, Ashish Singh
Review Request 28372: HIVE-8950: Add support in ParquetHiveSerde to create table schema from a parquet file
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28372/ --- Review request for hive. Bugs: HIVE-8950 https://issues.apache.org/jira/browse/HIVE-8950 Repository: hive-git Description --- HIVE-8950: Add support in ParquetHiveSerde to create table schema from a parquet file Diffs - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java fafd78e63e9b41c9fdb0e017b567dc719d151784 data/files/data.parq PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ColInfoFromParquetFile.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java 4effe736fcf9d3715f03eed9885c299a7aa040dd ql/src/test/queries/clientpositive/parquet_create_gen_schema.q PRE-CREATION ql/src/test/queries/clientpositive/parquet_create_gen_schema1.q PRE-CREATION ql/src/test/results/clientpositive/parquet_create_gen_schema.q.out PRE-CREATION ql/src/test/results/clientpositive/parquet_create_gen_schema1.q.out PRE-CREATION Diff: https://reviews.apache.org/r/28372/diff/ Testing --- Tested by adding appropriate qTests. Thanks, Ashish Singh
Re: [ANNOUNCE] New Hive PMC Member - Alan Gates
Congrats Alan! On Mon, Oct 27, 2014 at 6:11 PM, Vikram Dixit K wrote: > Congrats Alan! > > On Mon, Oct 27, 2014 at 6:00 PM, Szehon Ho wrote: > > > Congratulations, Alan! > > > > On Mon, Oct 27, 2014 at 4:17 PM, Hari Subramaniyan < > > hsubramani...@hortonworks.com> wrote: > > > > > Congrats Alan! > > > > > > Thanks > > > Hari > > > > > > On Mon, Oct 27, 2014 at 4:08 PM, Vaibhav Gumashta < > > > vgumas...@hortonworks.com > > > > wrote: > > > > > > > Congratulations Alan! > > > > > > > > On Mon, Oct 27, 2014 at 3:54 PM, Chris Drome > > > wrote: > > > > > > > >> Congratulations Alan! > > > >> > > > >> From: Carl Steinbach > > > >> Reply-To: "u...@hive.apache.org" > > > >> Date: Monday, October 27, 2014 at 3:38 PM > > > >> To: "dev@hive.apache.org" , " > > u...@hive.apache.org" > > > < > > > >> u...@hive.apache.org>, "ga...@apache.org" > > > >> Subject: [ANNOUNCE] New Hive PMC Member - Alan Gates > > > >> > > > >> I am pleased to announce that Alan Gates has been elected to the > > Hive > > > >> Project Management Committee. Please join me in congratulating Alan! > > > >> > > > >> Thanks. > > > >> > > > >> - Carl > > > >> > > > > > > > > > > > > CONFIDENTIALITY NOTICE > > > > NOTICE: This message is intended for the use of the individual or > > entity > > > > to which it is addressed and may contain information that is > > > confidential, > > > > privileged and exempt from disclosure under applicable law. If the > > reader > > > > of this message is not the intended recipient, you are hereby > notified > > > that > > > > any printing, copying, dissemination, distribution, disclosure or > > > > forwarding of this communication is strictly prohibited. If you have > > > > received this communication in error, please contact the sender > > > immediately > > > > and delete it from your system. Thank You. > > > > > > > > > > -- > > > CONFIDENTIALITY NOTICE > > > NOTICE: This message is intended for the use of the individual or > entity > > to > > > which it is addressed and may contain information that is confidential, > > > privileged and exempt from disclosure under applicable law. If the > reader > > > of this message is not the intended recipient, you are hereby notified > > that > > > any printing, copying, dissemination, distribution, disclosure or > > > forwarding of this communication is strictly prohibited. If you have > > > received this communication in error, please contact the sender > > immediately > > > and delete it from your system. Thank You. > > > > > > > > > -- > Nothing better than when appreciated for hard work. > -Mark > -- Regards, Ashish
Re: Review Request 24853: HIVE-7783: QTests throw exception during cleanup
> On Aug. 19, 2014, 5:37 p.m., Venki Korukanti wrote: > > itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java, line 629 > > <https://reviews.apache.org/r/24853/diff/1/?file=664330#file664330line629> > > > > Isn't this fixed by HIVE-7684? Are you still seeing errors on trunk? Ahh.. I had older versoin of trunk. Good to know, this is already fixed. Thanks. - Ashish --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24853/#review50985 --- On Aug. 19, 2014, 5:29 p.m., Ashish Singh wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/24853/ > --- > > (Updated Aug. 19, 2014, 5:29 p.m.) > > > Review request for hive. > > > Bugs: HIVE-7783 > https://issues.apache.org/jira/browse/HIVE-7783 > > > Repository: hive-git > > > Description > --- > > HIVE-7783: QTests throw exception during cleanup > > > Diffs > - > > itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java > af4a3e575ca5aea2746a5b51862ff178b59e403d > > Diff: https://reviews.apache.org/r/24853/diff/ > > > Testing > --- > > Ran a couple of qTests locally. > > > Thanks, > > Ashish Singh > >
Re: Review Request 24853: HIVE-7783: QTests throw exception during cleanup
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24853/ --- (Updated Aug. 19, 2014, 5:29 p.m.) Review request for hive. Changes --- Link Hive JIRA. Bugs: HIVE-7783 https://issues.apache.org/jira/browse/HIVE-7783 Repository: hive-git Description --- HIVE-7783: QTests throw exception during cleanup Diffs - itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java af4a3e575ca5aea2746a5b51862ff178b59e403d Diff: https://reviews.apache.org/r/24853/diff/ Testing --- Ran a couple of qTests locally. Thanks, Ashish Singh
Review Request 24853: HIVE-7783: QTests throw exception during cleanup
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24853/ --- Review request for hive. Repository: hive-git Description --- HIVE-7783: QTests throw exception during cleanup Diffs - itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java af4a3e575ca5aea2746a5b51862ff178b59e403d Diff: https://reviews.apache.org/r/24853/diff/ Testing --- Ran a couple of qTests locally. Thanks, Ashish Singh
Re: Review Request 24085: HIVE-7446: Add support to ALTER TABLE .. ADD COLUMN to Avro backed tables
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24085/ --- (Updated Aug. 8, 2014, midnight) Review request for hive. Changes --- Add test to make sure special case found in HIVE-6835 is handled. Bugs: HIVE-7446 https://issues.apache.org/jira/browse/HIVE-7446 Repository: hive-git Description --- HIVE-7446: Add support to ALTER TABLE .. ADD COLUMN to Avro backed tables Diffs (updated) - ql/src/test/queries/clientpositive/avro_add_column.q PRE-CREATION ql/src/test/queries/clientpositive/avro_add_column2.q PRE-CREATION ql/src/test/queries/clientpositive/avro_add_column3.q PRE-CREATION ql/src/test/results/clientpositive/avro_add_column.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_add_column2.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_add_column3.q.out PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/avro/TypeInfoToSchema.java 915f01679183904d0d93b9b8a88dc1a64ac2af78 serde/src/test/org/apache/hadoop/hive/serde2/avro/TestTypeInfoToSchema.java 722bdf9f8452fe8632db7d9167182310e467281d serde/src/test/resources/avro-nested-struct.avsc 785af83cd01fe91626741b3d7659d8f515854774 serde/src/test/resources/avro-struct.avsc 313c74f6140615d2737ef1a49a2777656f35f4e3 Diff: https://reviews.apache.org/r/24085/diff/ Testing --- qTests Thanks, Ashish Singh
Re: Review Request 24085: HIVE-7446: Add support to ALTER TABLE .. ADD COLUMN to Avro backed tables
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24085/ --- (Updated Aug. 7, 2014, 10:09 p.m.) Review request for hive. Changes --- Add default values to avro schema elements. Bugs: HIVE-7446 https://issues.apache.org/jira/browse/HIVE-7446 Repository: hive-git Description --- HIVE-7446: Add support to ALTER TABLE .. ADD COLUMN to Avro backed tables Diffs (updated) - ql/src/test/queries/clientpositive/avro_add_column.q PRE-CREATION ql/src/test/queries/clientpositive/avro_add_column2.q PRE-CREATION ql/src/test/results/clientpositive/avro_add_column.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_add_column2.q.out PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/avro/TypeInfoToSchema.java 915f01679183904d0d93b9b8a88dc1a64ac2af78 serde/src/test/org/apache/hadoop/hive/serde2/avro/TestTypeInfoToSchema.java 722bdf9f8452fe8632db7d9167182310e467281d serde/src/test/resources/avro-nested-struct.avsc 785af83cd01fe91626741b3d7659d8f515854774 serde/src/test/resources/avro-struct.avsc 313c74f6140615d2737ef1a49a2777656f35f4e3 Diff: https://reviews.apache.org/r/24085/diff/ Testing --- qTests Thanks, Ashish Singh
Re: Review Request 23953: HIVE-7519: Refactor QTestUtil to remove its duplication with QFileClient for qtest setup and teardown
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23953/ --- (Updated Aug. 7, 2014, 3:39 a.m.) Review request for hive. Changes --- Fix test errors. Bugs: HIVE-7519 https://issues.apache.org/jira/browse/HIVE-7519 Repository: hive-git Description --- HIVE-7519: Refactor QTestUtil to remove its duplication with QFileClient for qtest setup and teardown Diffs (updated) - ant/src/org/apache/hadoop/hive/ant/QTestGenTask.java 33f227fe6eb0ea6df936775f02e4339ed496f6ad data/conf/hive-site.xml 128e08f55b8dfd1c473a3af622faa68058d6a05b data/conf/tez/hive-site.xml 0c99bb6914bd26de26cef77cf29cf37f070098dc data/scripts/q_test_cleanup.sql 31bd7205d85916ea352f715f2fd1462efc788208 data/scripts/q_test_init.sql 12afdf391132e3fdd219aaa581e1f2e210d6dee2 hbase-handler/src/test/templates/TestHBaseCliDriver.vm 01d596aa6591ddccff016436c7f31324b3896d00 hbase-handler/src/test/templates/TestHBaseNegativeCliDriver.vm 45c73389cb26d0d461080cc146c5d74aee199c4e itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestLocationQueries.java 9edd7f30ff91bf7e01a2f52699192994fe0829f5 itests/qtest/pom.xml 249956fc170c0cef2b8f98454fa952c498b9e29e itests/util/src/main/java/org/apache/hadoop/hive/hbase/HBaseQTestUtil.java 96a0de2829c2ec065b7835b12c4932d1278f9a84 itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java ccfb58fd5acd722e9d9dc83b0df3cdd5d6e8a7c3 ql/src/test/templates/TestCliDriver.vm 4776c75c16329c7d3f6f1a032eef192d553cc3cc ql/src/test/templates/TestCompareCliDriver.vm f6f43b847fdd4039328632ef70d841fce9006d6d ql/src/test/templates/TestNegativeCliDriver.vm 991d5ac1b2fde66dbe60b39c853916577449b1a4 ql/src/test/templates/TestParse.vm c476536940dc3a48000bf4e60e0b551ec7904d63 ql/src/test/templates/TestParseNegative.vm f62f17e4df5c1439d3787fc5c361804121bfcaf1 Diff: https://reviews.apache.org/r/23953/diff/ Testing --- qTests. Thanks, Ashish Singh
Re: Review Request 23953: HIVE-7519: Refactor QTestUtil to remove its duplication with QFileClient for qtest setup and teardown
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23953/ --- (Updated Aug. 4, 2014, 8:21 p.m.) Review request for hive. Changes --- Print headers only for TestParse tests. Bugs: HIVE-7519 https://issues.apache.org/jira/browse/HIVE-7519 Repository: hive-git Description --- HIVE-7519: Refactor QTestUtil to remove its duplication with QFileClient for qtest setup and teardown Diffs (updated) - ant/src/org/apache/hadoop/hive/ant/QTestGenTask.java 33f227fe6eb0ea6df936775f02e4339ed496f6ad data/conf/hive-site.xml 128e08f55b8dfd1c473a3af622faa68058d6a05b data/conf/tez/hive-site.xml 0c99bb6914bd26de26cef77cf29cf37f070098dc data/scripts/q_test_cleanup.sql 31bd7205d85916ea352f715f2fd1462efc788208 data/scripts/q_test_init.sql 12afdf391132e3fdd219aaa581e1f2e210d6dee2 hbase-handler/src/test/templates/TestHBaseCliDriver.vm 01d596aa6591ddccff016436c7f31324b3896d00 hbase-handler/src/test/templates/TestHBaseNegativeCliDriver.vm 45c73389cb26d0d461080cc146c5d74aee199c4e itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestLocationQueries.java 9edd7f30ff91bf7e01a2f52699192994fe0829f5 itests/qtest/pom.xml 249956fc170c0cef2b8f98454fa952c498b9e29e itests/util/src/main/java/org/apache/hadoop/hive/hbase/HBaseQTestUtil.java 96a0de2829c2ec065b7835b12c4932d1278f9a84 itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java ccfb58fd5acd722e9d9dc83b0df3cdd5d6e8a7c3 ql/src/test/templates/TestCliDriver.vm 4776c75c16329c7d3f6f1a032eef192d553cc3cc ql/src/test/templates/TestCompareCliDriver.vm f6f43b847fdd4039328632ef70d841fce9006d6d ql/src/test/templates/TestNegativeCliDriver.vm 991d5ac1b2fde66dbe60b39c853916577449b1a4 ql/src/test/templates/TestParse.vm c476536940dc3a48000bf4e60e0b551ec7904d63 ql/src/test/templates/TestParseNegative.vm f62f17e4df5c1439d3787fc5c361804121bfcaf1 Diff: https://reviews.apache.org/r/23953/diff/ Testing --- qTests. Thanks, Ashish Singh
Re: Review Request 24086: HIVE-7348: Beeline could not parse ; separated queries provided with -e option
> On Aug. 1, 2014, 8:32 a.m., Lars Francke wrote:
> > I'm not sure about this patch.
> >
> > Something like this will now be completely garbled:
> >
> > create table ${D_TBL} (d string);select * from ${D_TBL} where d = ";";
> > show tables;
> >
> > Even something like this won't work anymore:
> >
> > select * from ${D_TBL} where d = ";"
> >
> > So I'm afraid I'm on -1 on this patch as it breaks existing behavior. The
> > only way to do this properly is to parse the statement fully and split
> > where needed.
Lars that is a good point. However, hive cli also does not support queries with
";" as part of query with -e.
I do agree that we should have a better way to handle ";" and support multiple
queries at the same time with -e. I will try to see if there is something that
can be formulated with reasonable effort. Feel free to suggest any solution
that you have in mind.
- Ashish
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24086/#review49327
---
On July 31, 2014, 8:05 a.m., Ashish Singh wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/24086/
> ---
>
> (Updated July 31, 2014, 8:05 a.m.)
>
>
> Review request for hive.
>
>
> Bugs: HIVE-7348
> https://issues.apache.org/jira/browse/HIVE-7348
>
>
> Repository: hive-git
>
>
> Description
> ---
>
> HIVE-7348: Beeline could not parse ; separated queries provided with -e option
>
>
> Diffs
> -
>
> beeline/src/java/org/apache/hive/beeline/BeeLine.java
> 10fd2e2daac78ca43d45c74fcbad6b720a8d28ad
>
> itests/hive-unit/src/test/java/org/apache/hive/beeline/TestBeeLineWithArgs.java
> bd97aff5959fd9040fc0f0a1f6b782f2aa6f
>
> Diff: https://reviews.apache.org/r/24086/diff/
>
>
> Testing
> ---
>
> Tested manually.
>
>
> Thanks,
>
> Ashish Singh
>
>
Re: Review Request 24081: HIVE-7432: Remove deprecated Avro's Schema.parse usages
> On Aug. 2, 2014, 12:09 a.m., Brock Noland wrote: > > serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerdeUtils.java, line > > 212 > > <https://reviews.apache.org/r/24081/diff/2/?file=646682#file646682line212> > > > > I don't think this line will log the stack trace. We should instead > > change the line above to LOG.info(msg + e.getMessage(), e); > > > > However, I think there is a larger issue. In the case of an exception, > > this method will return null which I don't see the callers expecting. If > > the exception is a true error condition we should consider throwing the > > IOException. You might have to wrap it in a RuntimeException. Brock, thanks for reviewing! Updated the patch based on your comment. - Ashish --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24081/#review49410 ------- On Aug. 2, 2014, 1:45 a.m., Ashish Singh wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/24081/ > --- > > (Updated Aug. 2, 2014, 1:45 a.m.) > > > Review request for hive. > > > Bugs: HIVE-7432 > https://issues.apache.org/jira/browse/HIVE-7432 > > > Repository: hive-git > > > Description > --- > > HIVE-7432: Remove deprecated Avro's Schema.parse usages > > > Diffs > - > > ql/src/java/org/apache/hadoop/hive/ql/io/avro/AvroGenericRecordReader.java > 60b43888b957fe315720c4ee5562b9b67a07d0e2 > > serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroGenericRecordWritable.java > b55474331736ecbdeb5958dad9342e132642d889 > serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerdeUtils.java > 8c5cf3e87078fd87d0dc9b41d9545486d76903f3 > > serde/src/java/org/apache/hadoop/hive/serde2/avro/SchemaResolutionProblem.java > 3dceb6384000e255e87df832f6189c80c636531b > serde/src/java/org/apache/hadoop/hive/serde2/avro/TypeInfoToSchema.java > 915f01679183904d0d93b9b8a88dc1a64ac2af78 > serde/src/test/org/apache/hadoop/hive/serde2/avro/TestAvroDeserializer.java > 198bd24dcb1c2552fd45b919ecb39ef7a29ed321 > > serde/src/test/org/apache/hadoop/hive/serde2/avro/TestAvroObjectInspectorGenerator.java > 76c1940fb05a0c8c6b74d570d6d788829e17de01 > serde/src/test/org/apache/hadoop/hive/serde2/avro/TestAvroSerde.java > 072225dcc80bfdb84f0a31f67693616393c264df > serde/src/test/org/apache/hadoop/hive/serde2/avro/TestAvroSerdeUtils.java > 67d557082eec88eefdde76cb1fead6d51f7784a4 > serde/src/test/org/apache/hadoop/hive/serde2/avro/TestAvroSerializer.java > f8161da44312c2ad9b4dd2bab2aa242692a42d5a > > serde/src/test/org/apache/hadoop/hive/serde2/avro/TestGenericAvroRecordWritable.java > cf3b16ce65d07c0a714530ef4a26adef7188ea2e > serde/src/test/org/apache/hadoop/hive/serde2/avro/TestSchemaReEncoder.java > 8dd61097433ab0c2b1c3e326978bf06337f815e6 > > serde/src/test/org/apache/hadoop/hive/serde2/avro/TestThatEvolvedSchemasActAsWeWant.java > 4b8cc98bfc75ea01f25944d7833f00da1b6911f0 > > Diff: https://reviews.apache.org/r/24081/diff/ > > > Testing > --- > > qTests > > > Thanks, > > Ashish Singh > >
Re: Review Request 24081: HIVE-7432: Remove deprecated Avro's Schema.parse usages
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24081/ --- (Updated Aug. 2, 2014, 1:45 a.m.) Review request for hive. Changes --- Address review comment. Bugs: HIVE-7432 https://issues.apache.org/jira/browse/HIVE-7432 Repository: hive-git Description --- HIVE-7432: Remove deprecated Avro's Schema.parse usages Diffs (updated) - ql/src/java/org/apache/hadoop/hive/ql/io/avro/AvroGenericRecordReader.java 60b43888b957fe315720c4ee5562b9b67a07d0e2 serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroGenericRecordWritable.java b55474331736ecbdeb5958dad9342e132642d889 serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerdeUtils.java 8c5cf3e87078fd87d0dc9b41d9545486d76903f3 serde/src/java/org/apache/hadoop/hive/serde2/avro/SchemaResolutionProblem.java 3dceb6384000e255e87df832f6189c80c636531b serde/src/java/org/apache/hadoop/hive/serde2/avro/TypeInfoToSchema.java 915f01679183904d0d93b9b8a88dc1a64ac2af78 serde/src/test/org/apache/hadoop/hive/serde2/avro/TestAvroDeserializer.java 198bd24dcb1c2552fd45b919ecb39ef7a29ed321 serde/src/test/org/apache/hadoop/hive/serde2/avro/TestAvroObjectInspectorGenerator.java 76c1940fb05a0c8c6b74d570d6d788829e17de01 serde/src/test/org/apache/hadoop/hive/serde2/avro/TestAvroSerde.java 072225dcc80bfdb84f0a31f67693616393c264df serde/src/test/org/apache/hadoop/hive/serde2/avro/TestAvroSerdeUtils.java 67d557082eec88eefdde76cb1fead6d51f7784a4 serde/src/test/org/apache/hadoop/hive/serde2/avro/TestAvroSerializer.java f8161da44312c2ad9b4dd2bab2aa242692a42d5a serde/src/test/org/apache/hadoop/hive/serde2/avro/TestGenericAvroRecordWritable.java cf3b16ce65d07c0a714530ef4a26adef7188ea2e serde/src/test/org/apache/hadoop/hive/serde2/avro/TestSchemaReEncoder.java 8dd61097433ab0c2b1c3e326978bf06337f815e6 serde/src/test/org/apache/hadoop/hive/serde2/avro/TestThatEvolvedSchemasActAsWeWant.java 4b8cc98bfc75ea01f25944d7833f00da1b6911f0 Diff: https://reviews.apache.org/r/24081/diff/ Testing --- qTests Thanks, Ashish Singh
Re: Review Request 23953: HIVE-7519: Refactor QTestUtil to remove its duplication with QFileClient for qtest setup and teardown
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23953/ --- (Updated Aug. 1, 2014, 11:46 p.m.) Review request for hive. Changes --- Rebase and fix test failures. Bugs: HIVE-7519 https://issues.apache.org/jira/browse/HIVE-7519 Repository: hive-git Description --- HIVE-7519: Refactor QTestUtil to remove its duplication with QFileClient for qtest setup and teardown Diffs (updated) - ant/src/org/apache/hadoop/hive/ant/QTestGenTask.java 33f227fe6eb0ea6df936775f02e4339ed496f6ad data/conf/hive-site.xml 128e08f55b8dfd1c473a3af622faa68058d6a05b data/conf/tez/hive-site.xml 0c99bb6914bd26de26cef77cf29cf37f070098dc data/scripts/q_test_cleanup.sql 31bd7205d85916ea352f715f2fd1462efc788208 data/scripts/q_test_init.sql 12afdf391132e3fdd219aaa581e1f2e210d6dee2 hbase-handler/src/test/templates/TestHBaseCliDriver.vm 01d596aa6591ddccff016436c7f31324b3896d00 hbase-handler/src/test/templates/TestHBaseNegativeCliDriver.vm 45c73389cb26d0d461080cc146c5d74aee199c4e itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestLocationQueries.java 9edd7f30ff91bf7e01a2f52699192994fe0829f5 itests/qtest/pom.xml 249956fc170c0cef2b8f98454fa952c498b9e29e itests/util/src/main/java/org/apache/hadoop/hive/hbase/HBaseQTestUtil.java 96a0de2829c2ec065b7835b12c4932d1278f9a84 itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java ccfb58fd5acd722e9d9dc83b0df3cdd5d6e8a7c3 ql/src/test/templates/TestCliDriver.vm 4776c75c16329c7d3f6f1a032eef192d553cc3cc ql/src/test/templates/TestCompareCliDriver.vm f6f43b847fdd4039328632ef70d841fce9006d6d ql/src/test/templates/TestNegativeCliDriver.vm 991d5ac1b2fde66dbe60b39c853916577449b1a4 ql/src/test/templates/TestParse.vm c476536940dc3a48000bf4e60e0b551ec7904d63 ql/src/test/templates/TestParseNegative.vm f62f17e4df5c1439d3787fc5c361804121bfcaf1 Diff: https://reviews.apache.org/r/23953/diff/ Testing --- qTests. Thanks, Ashish Singh
Re: Review Request 24086: HIVE-7348: Beeline could not parse ; separated queries provided with -e option
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24086/ --- (Updated July 31, 2014, 8:05 a.m.) Review request for hive. Changes --- Added unit test. Bugs: HIVE-7348 https://issues.apache.org/jira/browse/HIVE-7348 Repository: hive-git Description --- HIVE-7348: Beeline could not parse ; separated queries provided with -e option Diffs (updated) - beeline/src/java/org/apache/hive/beeline/BeeLine.java 10fd2e2daac78ca43d45c74fcbad6b720a8d28ad itests/hive-unit/src/test/java/org/apache/hive/beeline/TestBeeLineWithArgs.java bd97aff5959fd9040fc0f0a1f6b782f2aa6f Diff: https://reviews.apache.org/r/24086/diff/ Testing --- Tested manually. Thanks, Ashish Singh
Re: Review Request 24081: HIVE-7432: Remove deprecated Avro's Schema.parse usages
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24081/ --- (Updated July 31, 2014, 3:49 a.m.) Review request for hive. Changes --- Parser.parse maintains state and can not be reused. Add a util method to take care of creating avro schema from string, file or inputstream. Bugs: HIVE-7432 https://issues.apache.org/jira/browse/HIVE-7432 Repository: hive-git Description --- HIVE-7432: Remove deprecated Avro's Schema.parse usages Diffs (updated) - ql/src/java/org/apache/hadoop/hive/ql/io/avro/AvroGenericRecordReader.java 60b43888b957fe315720c4ee5562b9b67a07d0e2 serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroGenericRecordWritable.java b55474331736ecbdeb5958dad9342e132642d889 serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerdeUtils.java 8c5cf3e87078fd87d0dc9b41d9545486d76903f3 serde/src/java/org/apache/hadoop/hive/serde2/avro/SchemaResolutionProblem.java 3dceb6384000e255e87df832f6189c80c636531b serde/src/java/org/apache/hadoop/hive/serde2/avro/TypeInfoToSchema.java 915f01679183904d0d93b9b8a88dc1a64ac2af78 serde/src/test/org/apache/hadoop/hive/serde2/avro/TestAvroDeserializer.java 198bd24dcb1c2552fd45b919ecb39ef7a29ed321 serde/src/test/org/apache/hadoop/hive/serde2/avro/TestAvroObjectInspectorGenerator.java 76c1940fb05a0c8c6b74d570d6d788829e17de01 serde/src/test/org/apache/hadoop/hive/serde2/avro/TestAvroSerde.java 072225dcc80bfdb84f0a31f67693616393c264df serde/src/test/org/apache/hadoop/hive/serde2/avro/TestAvroSerdeUtils.java 67d557082eec88eefdde76cb1fead6d51f7784a4 serde/src/test/org/apache/hadoop/hive/serde2/avro/TestAvroSerializer.java f8161da44312c2ad9b4dd2bab2aa242692a42d5a serde/src/test/org/apache/hadoop/hive/serde2/avro/TestGenericAvroRecordWritable.java cf3b16ce65d07c0a714530ef4a26adef7188ea2e serde/src/test/org/apache/hadoop/hive/serde2/avro/TestSchemaReEncoder.java 8dd61097433ab0c2b1c3e326978bf06337f815e6 serde/src/test/org/apache/hadoop/hive/serde2/avro/TestThatEvolvedSchemasActAsWeWant.java 4b8cc98bfc75ea01f25944d7833f00da1b6911f0 Diff: https://reviews.apache.org/r/24081/diff/ Testing --- qTests Thanks, Ashish Singh
Re: Review Request 24086: HIVE-7348: Beeline could not parse ; separated queries provided with -e option
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24086/ --- (Updated July 30, 2014, 11:48 p.m.) Review request for hive. Changes --- Move changes to only effect -e path. Bugs: HIVE-7348 https://issues.apache.org/jira/browse/HIVE-7348 Repository: hive-git Description --- HIVE-7348: Beeline could not parse ; separated queries provided with -e option Diffs (updated) - beeline/src/java/org/apache/hive/beeline/BeeLine.java 10fd2e2daac78ca43d45c74fcbad6b720a8d28ad Diff: https://reviews.apache.org/r/24086/diff/ Testing --- Tested manually. Thanks, Ashish Singh
Review Request 24086: HIVE-7348: Beeline could not parse ; separated queries provided with -e option
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24086/ --- Review request for hive. Bugs: HIVE-7348 https://issues.apache.org/jira/browse/HIVE-7348 Repository: hive-git Description --- HIVE-7348: Beeline could not parse ; separated queries provided with -e option Diffs - beeline/src/java/org/apache/hive/beeline/BeeLine.java 10fd2e2daac78ca43d45c74fcbad6b720a8d28ad Diff: https://reviews.apache.org/r/24086/diff/ Testing --- Tested manually. Thanks, Ashish Singh
Review Request 24085: HIVE-7446: Add support to ALTER TABLE .. ADD COLUMN to Avro backed tables
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24085/ --- Review request for hive. Bugs: HIVE-7446 https://issues.apache.org/jira/browse/HIVE-7446 Repository: hive-git Description --- HIVE-7446: Add support to ALTER TABLE .. ADD COLUMN to Avro backed tables Diffs - ql/src/test/queries/clientpositive/avro_add_column.q PRE-CREATION ql/src/test/queries/clientpositive/avro_add_column2.q PRE-CREATION ql/src/test/results/clientpositive/avro_add_column.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_add_column2.q.out PRE-CREATION Diff: https://reviews.apache.org/r/24085/diff/ Testing --- qTests Thanks, Ashish Singh
Review Request 24081: HIVE-7432: Remove deprecated Avro's Schema.parse usages
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24081/ --- Review request for hive. Bugs: HIVE-7432 https://issues.apache.org/jira/browse/HIVE-7432 Repository: hive-git Description --- HIVE-7432: Remove deprecated Avro's Schema.parse usages Diffs - ql/src/java/org/apache/hadoop/hive/ql/io/avro/AvroGenericRecordReader.java 60b43888b957fe315720c4ee5562b9b67a07d0e2 serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroGenericRecordWritable.java b55474331736ecbdeb5958dad9342e132642d889 serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerdeUtils.java 8c5cf3e87078fd87d0dc9b41d9545486d76903f3 serde/src/java/org/apache/hadoop/hive/serde2/avro/SchemaResolutionProblem.java 3dceb6384000e255e87df832f6189c80c636531b serde/src/test/org/apache/hadoop/hive/serde2/avro/TestAvroDeserializer.java 198bd24dcb1c2552fd45b919ecb39ef7a29ed321 serde/src/test/org/apache/hadoop/hive/serde2/avro/TestAvroObjectInspectorGenerator.java 76c1940fb05a0c8c6b74d570d6d788829e17de01 serde/src/test/org/apache/hadoop/hive/serde2/avro/TestAvroSerde.java 072225dcc80bfdb84f0a31f67693616393c264df serde/src/test/org/apache/hadoop/hive/serde2/avro/TestAvroSerdeUtils.java 67d557082eec88eefdde76cb1fead6d51f7784a4 serde/src/test/org/apache/hadoop/hive/serde2/avro/TestAvroSerializer.java f8161da44312c2ad9b4dd2bab2aa242692a42d5a serde/src/test/org/apache/hadoop/hive/serde2/avro/TestGenericAvroRecordWritable.java cf3b16ce65d07c0a714530ef4a26adef7188ea2e serde/src/test/org/apache/hadoop/hive/serde2/avro/TestSchemaReEncoder.java 8dd61097433ab0c2b1c3e326978bf06337f815e6 serde/src/test/org/apache/hadoop/hive/serde2/avro/TestThatEvolvedSchemasActAsWeWant.java 4b8cc98bfc75ea01f25944d7833f00da1b6911f0 Diff: https://reviews.apache.org/r/24081/diff/ Testing --- qTests Thanks, Ashish Singh
Re: Review Request 23953: HIVE-7519: Refactor QTestUtil to remove its duplication with QFileClient for qtest setup and teardown
> On July 29, 2014, 12:47 a.m., Szehon Ho wrote: > > itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java, line 147 > > <https://reviews.apache.org/r/23953/diff/3/?file=643975#file643975line147> > > > > Can we pass it in via the pom, like the other variables? I think it > > would be easier to find/change there if we ever refactor, in addition to > > being more consistent with the other vars. Good point. Updated patch. - Ashish --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23953/#review48954 ------- On July 29, 2014, 11:46 p.m., Ashish Singh wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/23953/ > --- > > (Updated July 29, 2014, 11:46 p.m.) > > > Review request for hive. > > > Bugs: HIVE-7519 > https://issues.apache.org/jira/browse/HIVE-7519 > > > Repository: hive-git > > > Description > --- > > HIVE-7519: Refactor QTestUtil to remove its duplication with QFileClient for > qtest setup and teardown > > > Diffs > - > > ant/src/org/apache/hadoop/hive/ant/QTestGenTask.java > 33f227fe6eb0ea6df936775f02e4339ed496f6ad > data/conf/hive-site.xml fe8080addcadac4d52868866457dd038ea8d3d91 > data/conf/tez/hive-site.xml 0c99bb6914bd26de26cef77cf29cf37f070098dc > data/scripts/q_test_cleanup.sql 31bd7205d85916ea352f715f2fd1462efc788208 > data/scripts/q_test_init.sql 12afdf391132e3fdd219aaa581e1f2e210d6dee2 > hbase-handler/src/test/templates/TestHBaseCliDriver.vm > 01d596aa6591ddccff016436c7f31324b3896d00 > hbase-handler/src/test/templates/TestHBaseNegativeCliDriver.vm > 45c73389cb26d0d461080cc146c5d74aee199c4e > > itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestLocationQueries.java > 9edd7f30ff91bf7e01a2f52699192994fe0829f5 > itests/qtest/pom.xml 249956fc170c0cef2b8f98454fa952c498b9e29e > itests/util/src/main/java/org/apache/hadoop/hive/hbase/HBaseQTestUtil.java > 96a0de2829c2ec065b7835b12c4932d1278f9a84 > itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java > 2fefa067791bd74412c0b4efb697dc0d8bb03cd7 > ql/src/test/templates/TestCliDriver.vm > 4776c75c16329c7d3f6f1a032eef192d553cc3cc > ql/src/test/templates/TestCompareCliDriver.vm > f6f43b847fdd4039328632ef70d841fce9006d6d > ql/src/test/templates/TestNegativeCliDriver.vm > 991d5ac1b2fde66dbe60b39c853916577449b1a4 > ql/src/test/templates/TestParse.vm c476536940dc3a48000bf4e60e0b551ec7904d63 > ql/src/test/templates/TestParseNegative.vm > f62f17e4df5c1439d3787fc5c361804121bfcaf1 > > Diff: https://reviews.apache.org/r/23953/diff/ > > > Testing > --- > > qTests. > > > Thanks, > > Ashish Singh > >
Re: Review Request 23953: HIVE-7519: Refactor QTestUtil to remove its duplication with QFileClient for qtest setup and teardown
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23953/ --- (Updated July 29, 2014, 11:46 p.m.) Review request for hive. Changes --- Address review comment and test failures. Bugs: HIVE-7519 https://issues.apache.org/jira/browse/HIVE-7519 Repository: hive-git Description --- HIVE-7519: Refactor QTestUtil to remove its duplication with QFileClient for qtest setup and teardown Diffs (updated) - ant/src/org/apache/hadoop/hive/ant/QTestGenTask.java 33f227fe6eb0ea6df936775f02e4339ed496f6ad data/conf/hive-site.xml fe8080addcadac4d52868866457dd038ea8d3d91 data/conf/tez/hive-site.xml 0c99bb6914bd26de26cef77cf29cf37f070098dc data/scripts/q_test_cleanup.sql 31bd7205d85916ea352f715f2fd1462efc788208 data/scripts/q_test_init.sql 12afdf391132e3fdd219aaa581e1f2e210d6dee2 hbase-handler/src/test/templates/TestHBaseCliDriver.vm 01d596aa6591ddccff016436c7f31324b3896d00 hbase-handler/src/test/templates/TestHBaseNegativeCliDriver.vm 45c73389cb26d0d461080cc146c5d74aee199c4e itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestLocationQueries.java 9edd7f30ff91bf7e01a2f52699192994fe0829f5 itests/qtest/pom.xml 249956fc170c0cef2b8f98454fa952c498b9e29e itests/util/src/main/java/org/apache/hadoop/hive/hbase/HBaseQTestUtil.java 96a0de2829c2ec065b7835b12c4932d1278f9a84 itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 2fefa067791bd74412c0b4efb697dc0d8bb03cd7 ql/src/test/templates/TestCliDriver.vm 4776c75c16329c7d3f6f1a032eef192d553cc3cc ql/src/test/templates/TestCompareCliDriver.vm f6f43b847fdd4039328632ef70d841fce9006d6d ql/src/test/templates/TestNegativeCliDriver.vm 991d5ac1b2fde66dbe60b39c853916577449b1a4 ql/src/test/templates/TestParse.vm c476536940dc3a48000bf4e60e0b551ec7904d63 ql/src/test/templates/TestParseNegative.vm f62f17e4df5c1439d3787fc5c361804121bfcaf1 Diff: https://reviews.apache.org/r/23953/diff/ Testing --- qTests. Thanks, Ashish Singh
Re: Review Request 23953: HIVE-7519: Refactor QTestUtil to remove its duplication with QFileClient for qtest setup and teardown
> On July 26, 2014, 12:34 a.m., Szehon Ho wrote: > > data/scripts/q_test_init.sql, line 8 > > <https://reviews.apache.org/r/23953/diff/1/?file=642522#file642522line8> > > > > Is there a lot of work to implement using hiveconf variable like > > BeelineDriver did? If not, can we consider it? Turns out we can do it. Thanks for telling that variable substitutions are possible with hive cli as well. Updated patch. - Ashish --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23953/#review48791 --- On July 28, 2014, 11:16 p.m., Ashish Singh wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/23953/ > --- > > (Updated July 28, 2014, 11:16 p.m.) > > > Review request for hive. > > > Bugs: HIVE-7519 > https://issues.apache.org/jira/browse/HIVE-7519 > > > Repository: hive-git > > > Description > --- > > HIVE-7519: Refactor QTestUtil to remove its duplication with QFileClient for > qtest setup and teardown > > > Diffs > - > > data/scripts/q_test_cleanup.sql 31bd7205d85916ea352f715f2fd1462efc788208 > data/scripts/q_test_init.sql 12afdf391132e3fdd219aaa581e1f2e210d6dee2 > itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java > 2fefa067791bd74412c0b4efb697dc0d8bb03cd7 > > Diff: https://reviews.apache.org/r/23953/diff/ > > > Testing > --- > > qTests. > > > Thanks, > > Ashish Singh > >
Re: Review Request 23953: HIVE-7519: Refactor QTestUtil to remove its duplication with QFileClient for qtest setup and teardown
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23953/ --- (Updated July 28, 2014, 11:16 p.m.) Review request for hive. Changes --- Remove hard coded path to data files and use parameter passed to QTestUtils. Bugs: HIVE-7519 https://issues.apache.org/jira/browse/HIVE-7519 Repository: hive-git Description --- HIVE-7519: Refactor QTestUtil to remove its duplication with QFileClient for qtest setup and teardown Diffs (updated) - data/scripts/q_test_cleanup.sql 31bd7205d85916ea352f715f2fd1462efc788208 data/scripts/q_test_init.sql 12afdf391132e3fdd219aaa581e1f2e210d6dee2 itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 2fefa067791bd74412c0b4efb697dc0d8bb03cd7 Diff: https://reviews.apache.org/r/23953/diff/ Testing --- qTests. Thanks, Ashish Singh
Re: Review Request 23953: HIVE-7519: Refactor QTestUtil to remove its duplication with QFileClient for qtest setup and teardown
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23953/ --- (Updated July 28, 2014, 8:53 p.m.) Review request for hive. Changes --- Address review comments. Bugs: HIVE-7519 https://issues.apache.org/jira/browse/HIVE-7519 Repository: hive-git Description --- HIVE-7519: Refactor QTestUtil to remove its duplication with QFileClient for qtest setup and teardown Diffs (updated) - data/scripts/q_test_cleanup.sql 31bd7205d85916ea352f715f2fd1462efc788208 data/scripts/q_test_init.sql 12afdf391132e3fdd219aaa581e1f2e210d6dee2 itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 2fefa067791bd74412c0b4efb697dc0d8bb03cd7 Diff: https://reviews.apache.org/r/23953/diff/ Testing --- qTests. Thanks, Ashish Singh
Re: Review Request 23953: HIVE-7519: Refactor QTestUtil to remove its duplication with QFileClient for qtest setup and teardown
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23953/#review48802 --- data/scripts/q_test_init.sql <https://reviews.apache.org/r/23953/#comment85547> Even I wanted to have this, but unfortunately hive cli does not preprocess hiveconf variables. So, at this time I do not see an easy way to achieve this. One way would be to parse each statement and replace this variable with value obtained from hive conf. But, I do not consider that to be a good solution. Let me know if you think otherwise. itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java <https://reviews.apache.org/r/23953/#comment85675> This is now obtained from system properties, however, I have set the default value as what we have right now. Let me know if this is not what you were suggesting. itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java <https://reviews.apache.org/r/23953/#comment85674> Done. itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java <https://reviews.apache.org/r/23953/#comment85677> ClientSessionState tells hive from where to get its input and where to log output and error messages. Redirecting the output and error to stdout, redirects the logs from the init scripts to go into surefire report. Let me know if this doesn't answer your question. - Ashish Singh On July 25, 2014, 10:49 p.m., Ashish Singh wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/23953/ > --- > > (Updated July 25, 2014, 10:49 p.m.) > > > Review request for hive. > > > Bugs: HIVE-7519 > https://issues.apache.org/jira/browse/HIVE-7519 > > > Repository: hive-git > > > Description > --- > > HIVE-7519: Refactor QTestUtil to remove its duplication with QFileClient for > qtest setup and teardown > > > Diffs > - > > data/scripts/q_test_cleanup.sql 31bd7205d85916ea352f715f2fd1462efc788208 > data/scripts/q_test_init.sql 12afdf391132e3fdd219aaa581e1f2e210d6dee2 > itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java > 2fefa067791bd74412c0b4efb697dc0d8bb03cd7 > > Diff: https://reviews.apache.org/r/23953/diff/ > > > Testing > --- > > qTests. > > > Thanks, > > Ashish Singh > >
Review Request 23953: HIVE-7519: Refactor QTestUtil to remove its duplication with QFileClient for qtest setup and teardown
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23953/ --- Review request for hive. Bugs: HIVE-7519 https://issues.apache.org/jira/browse/HIVE-7519 Repository: hive-git Description --- HIVE-7519: Refactor QTestUtil to remove its duplication with QFileClient for qtest setup and teardown Diffs - data/scripts/q_test_cleanup.sql 31bd7205d85916ea352f715f2fd1462efc788208 data/scripts/q_test_init.sql 12afdf391132e3fdd219aaa581e1f2e210d6dee2 itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 2fefa067791bd74412c0b4efb697dc0d8bb03cd7 Diff: https://reviews.apache.org/r/23953/diff/ Testing --- qTests. Thanks, Ashish Singh
Re: Review Request 23387: HIVE-6806: Native avro support
> On July 23, 2014, 4:29 a.m., Brock Noland wrote: > > Looks great! Can you fix my nits and attach the patch to jira? Done. - Ashish --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23387/#review48474 --- On July 23, 2014, 6:38 p.m., Ashish Singh wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/23387/ > --- > > (Updated July 23, 2014, 6:38 p.m.) > > > Review request for hive. > > > Bugs: HIVE-6806 > https://issues.apache.org/jira/browse/HIVE-6806 > > > Repository: hive-git > > > Description > --- > > HIVE-6806: Native avro support > > > Diffs > - > > ql/src/java/org/apache/hadoop/hive/ql/io/AvroStorageFormatDescriptor.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/io/IOConstants.java > 1bae0a8fee04049f90b16d813ff4c96707b349c8 > > ql/src/main/resources/META-INF/services/org.apache.hadoop.hive.ql.io.StorageFormatDescriptor > a23ff115512da5fe3167835a88d582c427585b8e > ql/src/test/org/apache/hadoop/hive/ql/io/TestStorageFormatDescriptor.java > d53ebc65174d66bfeee25fd2891c69c78f9137ee > ql/src/test/queries/clientpositive/avro_compression_enabled_native.q > PRE-CREATION > ql/src/test/queries/clientpositive/avro_decimal_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_joins_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_partitioned_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_schema_evolution_native.q > PRE-CREATION > ql/src/test/results/clientpositive/avro_compression_enabled_native.q.out > PRE-CREATION > ql/src/test/results/clientpositive/avro_decimal_native.q.out PRE-CREATION > ql/src/test/results/clientpositive/avro_joins_native.q.out PRE-CREATION > ql/src/test/results/clientpositive/avro_native.q.out PRE-CREATION > ql/src/test/results/clientpositive/avro_partitioned_native.q.out > PRE-CREATION > ql/src/test/results/clientpositive/avro_schema_evolution_native.q.out > PRE-CREATION > serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java > 0db12437406170686a21b6055d83156fe5d6a55f > serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerDe.java > 1fe31e0034f8988d03a0c51a90904bb93e7cb157 > serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerdeUtils.java > 4564e75d9bfc73f8e10f160e2535f1a08b90ff79 > serde/src/java/org/apache/hadoop/hive/serde2/avro/TypeInfoToSchema.java > PRE-CREATION > serde/src/test/org/apache/hadoop/hive/serde2/avro/TestTypeInfoToSchema.java > PRE-CREATION > serde/src/test/resources/avro-nested-struct.avsc PRE-CREATION > serde/src/test/resources/avro-struct.avsc PRE-CREATION > > Diff: https://reviews.apache.org/r/23387/diff/ > > > Testing > --- > > Added qTests and unit tests > > > Thanks, > > Ashish Singh > >
Re: Review Request 23387: HIVE-6806: Native avro support
> On July 23, 2014, 7:26 a.m., Lars Francke wrote: > > serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerDe.java, line 89 > > <https://reviews.apache.org/r/23387/diff/9-11/?file=634614#file634614line89> > > > > At this point columnNameProperty can't be empty because it'll have been > > checked by the previous if statement already. So this can be > > simplified/unwrapped to just the else branch Great catch! - Ashish --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23387/#review48481 ------- On July 23, 2014, 6:38 p.m., Ashish Singh wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/23387/ > --- > > (Updated July 23, 2014, 6:38 p.m.) > > > Review request for hive. > > > Bugs: HIVE-6806 > https://issues.apache.org/jira/browse/HIVE-6806 > > > Repository: hive-git > > > Description > --- > > HIVE-6806: Native avro support > > > Diffs > - > > ql/src/java/org/apache/hadoop/hive/ql/io/AvroStorageFormatDescriptor.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/io/IOConstants.java > 1bae0a8fee04049f90b16d813ff4c96707b349c8 > > ql/src/main/resources/META-INF/services/org.apache.hadoop.hive.ql.io.StorageFormatDescriptor > a23ff115512da5fe3167835a88d582c427585b8e > ql/src/test/org/apache/hadoop/hive/ql/io/TestStorageFormatDescriptor.java > d53ebc65174d66bfeee25fd2891c69c78f9137ee > ql/src/test/queries/clientpositive/avro_compression_enabled_native.q > PRE-CREATION > ql/src/test/queries/clientpositive/avro_decimal_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_joins_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_partitioned_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_schema_evolution_native.q > PRE-CREATION > ql/src/test/results/clientpositive/avro_compression_enabled_native.q.out > PRE-CREATION > ql/src/test/results/clientpositive/avro_decimal_native.q.out PRE-CREATION > ql/src/test/results/clientpositive/avro_joins_native.q.out PRE-CREATION > ql/src/test/results/clientpositive/avro_native.q.out PRE-CREATION > ql/src/test/results/clientpositive/avro_partitioned_native.q.out > PRE-CREATION > ql/src/test/results/clientpositive/avro_schema_evolution_native.q.out > PRE-CREATION > serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java > 0db12437406170686a21b6055d83156fe5d6a55f > serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerDe.java > 1fe31e0034f8988d03a0c51a90904bb93e7cb157 > serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerdeUtils.java > 4564e75d9bfc73f8e10f160e2535f1a08b90ff79 > serde/src/java/org/apache/hadoop/hive/serde2/avro/TypeInfoToSchema.java > PRE-CREATION > serde/src/test/org/apache/hadoop/hive/serde2/avro/TestTypeInfoToSchema.java > PRE-CREATION > serde/src/test/resources/avro-nested-struct.avsc PRE-CREATION > serde/src/test/resources/avro-struct.avsc PRE-CREATION > > Diff: https://reviews.apache.org/r/23387/diff/ > > > Testing > --- > > Added qTests and unit tests > > > Thanks, > > Ashish Singh > >
Re: Review Request 23387: HIVE-6806: Native avro support
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23387/ --- (Updated July 23, 2014, 6:38 p.m.) Review request for hive. Changes --- Addressed final review comments. Bugs: HIVE-6806 https://issues.apache.org/jira/browse/HIVE-6806 Repository: hive-git Description --- HIVE-6806: Native avro support Diffs (updated) - ql/src/java/org/apache/hadoop/hive/ql/io/AvroStorageFormatDescriptor.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/io/IOConstants.java 1bae0a8fee04049f90b16d813ff4c96707b349c8 ql/src/main/resources/META-INF/services/org.apache.hadoop.hive.ql.io.StorageFormatDescriptor a23ff115512da5fe3167835a88d582c427585b8e ql/src/test/org/apache/hadoop/hive/ql/io/TestStorageFormatDescriptor.java d53ebc65174d66bfeee25fd2891c69c78f9137ee ql/src/test/queries/clientpositive/avro_compression_enabled_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_decimal_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_joins_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_partitioned_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_schema_evolution_native.q PRE-CREATION ql/src/test/results/clientpositive/avro_compression_enabled_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_decimal_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_joins_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_partitioned_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_schema_evolution_native.q.out PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java 0db12437406170686a21b6055d83156fe5d6a55f serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerDe.java 1fe31e0034f8988d03a0c51a90904bb93e7cb157 serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerdeUtils.java 4564e75d9bfc73f8e10f160e2535f1a08b90ff79 serde/src/java/org/apache/hadoop/hive/serde2/avro/TypeInfoToSchema.java PRE-CREATION serde/src/test/org/apache/hadoop/hive/serde2/avro/TestTypeInfoToSchema.java PRE-CREATION serde/src/test/resources/avro-nested-struct.avsc PRE-CREATION serde/src/test/resources/avro-struct.avsc PRE-CREATION Diff: https://reviews.apache.org/r/23387/diff/ Testing --- Added qTests and unit tests Thanks, Ashish Singh
Re: Review Request 23387: HIVE-6806: Native avro support
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23387/#review48541 --- - Ashish Singh On July 23, 2014, 6:38 p.m., Ashish Singh wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/23387/ > --- > > (Updated July 23, 2014, 6:38 p.m.) > > > Review request for hive. > > > Bugs: HIVE-6806 > https://issues.apache.org/jira/browse/HIVE-6806 > > > Repository: hive-git > > > Description > --- > > HIVE-6806: Native avro support > > > Diffs > - > > ql/src/java/org/apache/hadoop/hive/ql/io/AvroStorageFormatDescriptor.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/io/IOConstants.java > 1bae0a8fee04049f90b16d813ff4c96707b349c8 > > ql/src/main/resources/META-INF/services/org.apache.hadoop.hive.ql.io.StorageFormatDescriptor > a23ff115512da5fe3167835a88d582c427585b8e > ql/src/test/org/apache/hadoop/hive/ql/io/TestStorageFormatDescriptor.java > d53ebc65174d66bfeee25fd2891c69c78f9137ee > ql/src/test/queries/clientpositive/avro_compression_enabled_native.q > PRE-CREATION > ql/src/test/queries/clientpositive/avro_decimal_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_joins_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_partitioned_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_schema_evolution_native.q > PRE-CREATION > ql/src/test/results/clientpositive/avro_compression_enabled_native.q.out > PRE-CREATION > ql/src/test/results/clientpositive/avro_decimal_native.q.out PRE-CREATION > ql/src/test/results/clientpositive/avro_joins_native.q.out PRE-CREATION > ql/src/test/results/clientpositive/avro_native.q.out PRE-CREATION > ql/src/test/results/clientpositive/avro_partitioned_native.q.out > PRE-CREATION > ql/src/test/results/clientpositive/avro_schema_evolution_native.q.out > PRE-CREATION > serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java > 0db12437406170686a21b6055d83156fe5d6a55f > serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerDe.java > 1fe31e0034f8988d03a0c51a90904bb93e7cb157 > serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerdeUtils.java > 4564e75d9bfc73f8e10f160e2535f1a08b90ff79 > serde/src/java/org/apache/hadoop/hive/serde2/avro/TypeInfoToSchema.java > PRE-CREATION > serde/src/test/org/apache/hadoop/hive/serde2/avro/TestTypeInfoToSchema.java > PRE-CREATION > serde/src/test/resources/avro-nested-struct.avsc PRE-CREATION > serde/src/test/resources/avro-struct.avsc PRE-CREATION > > Diff: https://reviews.apache.org/r/23387/diff/ > > > Testing > --- > > Added qTests and unit tests > > > Thanks, > > Ashish Singh > >
Re: Review Request 23387: HIVE-6806: Native avro support
> On July 19, 2014, 12:43 a.m., David Chen wrote:
> > serde/src/test/org/apache/hadoop/hive/serde2/avro/TestTypeInfoToSchema.java,
> > line 294
> > <https://reviews.apache.org/r/23387/diff/9/?file=634616#file634616line294>
> >
> > It would improve maintainability to keep the test schemas in separate
> > .avsc files under serde/src/test/resources rather than inline in the file.
> > You can use Guava's Resources class to get the file and construct the
> > schema. For example:
> >
> > Schema expectedSchema = new Schema.Parser().parse(
> > Resources.getResource("record1.avsc").openStream());
>
> Ashish Singh wrote:
> David, as the tests have a lot common in their schema I am using a method
> to generate the common schema part and each test only provides a part of
> schema that is specific to the test. This made my tests have much less LOC.
> If I create a .avsc file for each test it will much more cumbersome for both
> maintaining and adding new tests.
>
> David Chen wrote:
> Hi Ashish, sorry I was a bit unclear. The tests for the individual data
> types are fine. I thought that moving just the two large schemas into their
> own files may make them easier to maintain since keeping them inline requires
> a large number of escape characters.
Davis, its done now.
- Ashish
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23387/#review48168
---
On July 22, 2014, 5:13 p.m., Ashish Singh wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/23387/
> ---
>
> (Updated July 22, 2014, 5:13 p.m.)
>
>
> Review request for hive.
>
>
> Bugs: HIVE-6806
> https://issues.apache.org/jira/browse/HIVE-6806
>
>
> Repository: hive-git
>
>
> Description
> ---
>
> HIVE-6806: Native avro support
>
>
> Diffs
> -
>
> ql/src/java/org/apache/hadoop/hive/ql/io/AvroStorageFormatDescriptor.java
> PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/IOConstants.java
> 1bae0a8fee04049f90b16d813ff4c96707b349c8
>
> ql/src/main/resources/META-INF/services/org.apache.hadoop.hive.ql.io.StorageFormatDescriptor
> a23ff115512da5fe3167835a88d582c427585b8e
> ql/src/test/org/apache/hadoop/hive/ql/io/TestStorageFormatDescriptor.java
> d53ebc65174d66bfeee25fd2891c69c78f9137ee
> ql/src/test/queries/clientpositive/avro_compression_enabled_native.q
> PRE-CREATION
> ql/src/test/queries/clientpositive/avro_decimal_native.q PRE-CREATION
> ql/src/test/queries/clientpositive/avro_joins_native.q PRE-CREATION
> ql/src/test/queries/clientpositive/avro_native.q PRE-CREATION
> ql/src/test/queries/clientpositive/avro_partitioned_native.q PRE-CREATION
> ql/src/test/queries/clientpositive/avro_schema_evolution_native.q
> PRE-CREATION
> ql/src/test/results/clientpositive/avro_compression_enabled_native.q.out
> PRE-CREATION
> ql/src/test/results/clientpositive/avro_decimal_native.q.out PRE-CREATION
> ql/src/test/results/clientpositive/avro_joins_native.q.out PRE-CREATION
> ql/src/test/results/clientpositive/avro_native.q.out PRE-CREATION
> ql/src/test/results/clientpositive/avro_partitioned_native.q.out
> PRE-CREATION
> ql/src/test/results/clientpositive/avro_schema_evolution_native.q.out
> PRE-CREATION
> serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java
> 0db12437406170686a21b6055d83156fe5d6a55f
> serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerDe.java
> 1fe31e0034f8988d03a0c51a90904bb93e7cb157
> serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerdeUtils.java
> 4564e75d9bfc73f8e10f160e2535f1a08b90ff79
> serde/src/java/org/apache/hadoop/hive/serde2/avro/TypeInfoToSchema.java
> PRE-CREATION
> serde/src/test/org/apache/hadoop/hive/serde2/avro/TestTypeInfoToSchema.java
> PRE-CREATION
> serde/src/test/resources/avro-nested-struct.avsc PRE-CREATION
> serde/src/test/resources/avro-struct.avsc PRE-CREATION
>
> Diff: https://reviews.apache.org/r/23387/diff/
>
>
> Testing
> ---
>
> Added qTests and unit tests
>
>
> Thanks,
>
> Ashish Singh
>
>
Re: Review Request 23387: HIVE-6806: Native avro support
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23387/ --- (Updated July 22, 2014, 5:13 p.m.) Review request for hive. Changes --- Move avro schema in unit tests to separate .avsc files. Bugs: HIVE-6806 https://issues.apache.org/jira/browse/HIVE-6806 Repository: hive-git Description --- HIVE-6806: Native avro support Diffs (updated) - ql/src/java/org/apache/hadoop/hive/ql/io/AvroStorageFormatDescriptor.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/io/IOConstants.java 1bae0a8fee04049f90b16d813ff4c96707b349c8 ql/src/main/resources/META-INF/services/org.apache.hadoop.hive.ql.io.StorageFormatDescriptor a23ff115512da5fe3167835a88d582c427585b8e ql/src/test/org/apache/hadoop/hive/ql/io/TestStorageFormatDescriptor.java d53ebc65174d66bfeee25fd2891c69c78f9137ee ql/src/test/queries/clientpositive/avro_compression_enabled_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_decimal_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_joins_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_partitioned_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_schema_evolution_native.q PRE-CREATION ql/src/test/results/clientpositive/avro_compression_enabled_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_decimal_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_joins_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_partitioned_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_schema_evolution_native.q.out PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java 0db12437406170686a21b6055d83156fe5d6a55f serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerDe.java 1fe31e0034f8988d03a0c51a90904bb93e7cb157 serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerdeUtils.java 4564e75d9bfc73f8e10f160e2535f1a08b90ff79 serde/src/java/org/apache/hadoop/hive/serde2/avro/TypeInfoToSchema.java PRE-CREATION serde/src/test/org/apache/hadoop/hive/serde2/avro/TestTypeInfoToSchema.java PRE-CREATION serde/src/test/resources/avro-nested-struct.avsc PRE-CREATION serde/src/test/resources/avro-struct.avsc PRE-CREATION Diff: https://reviews.apache.org/r/23387/diff/ Testing --- Added qTests and unit tests Thanks, Ashish Singh
Re: Review Request 23387: HIVE-6806: Native avro support
> On July 18, 2014, 1:57 p.m., Tom White wrote: > > serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerDe.java, line 112 > > <https://reviews.apache.org/r/23387/diff/9/?file=634614#file634614line112> > > > > Is it possible to default name to table name, namespace to database > > name, and doc to table comment? > > Ashish Singh wrote: > I was planning to do this, but slipped off my mind. Thanks for pointing > this out. I don't think it is possible to retrieve database name inside > serde. Addressed name and doc. > > Tom White wrote: > Thanks for fixing this. There's no test that "name" and "comment" are > picked up from the table definition - perhaps you could add one, or at least > confirm it manually. I couldn't see where in Hive they get set... > > Otherwise, +1 from me - thanks for addressing all my comments. This is a > great feature to add. > > Ashish Singh wrote: > Tom, actually its tested by all the unit tests now. Look at the diffs in > https://reviews.apache.org/r/23387/diff/8-9/#index_header. > > Tom White wrote: > Unless I am missing something, the unit tests in TestTypeInfoToSchema > don't test this since they hardcode the table name to "avrotest" and the > table comment to "This is to test hive-avro". > > Perhaps this is tested indirectly through the ql tests since Avro schema > resolution rules mean that a record schema's name must match for both the > reader and writer. However, this isn't true for comments (Avro schema doc), > and it would be good to confirm that inserting data into an Avro-backed Hive > table creates Avro files with the expected top-level name and comment. Tom, my bad. I thought we are talking about having hive typeinfo in doc for corresponding avro schema. I did verify that top-level name and comment are being created as expected before posting the patch here. I log created avro schema in AvroSerde.java and that came handy to verify this. - Ashish --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23387/#review48120 --- On July 19, 2014, 5:11 a.m., Ashish Singh wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/23387/ > --- > > (Updated July 19, 2014, 5:11 a.m.) > > > Review request for hive. > > > Bugs: HIVE-6806 > https://issues.apache.org/jira/browse/HIVE-6806 > > > Repository: hive-git > > > Description > --- > > HIVE-6806: Native avro support > > > Diffs > - > > ql/src/java/org/apache/hadoop/hive/ql/io/AvroStorageFormatDescriptor.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/io/IOConstants.java > 1bae0a8fee04049f90b16d813ff4c96707b349c8 > > ql/src/main/resources/META-INF/services/org.apache.hadoop.hive.ql.io.StorageFormatDescriptor > a23ff115512da5fe3167835a88d582c427585b8e > ql/src/test/org/apache/hadoop/hive/ql/io/TestStorageFormatDescriptor.java > d53ebc65174d66bfeee25fd2891c69c78f9137ee > ql/src/test/queries/clientpositive/avro_compression_enabled_native.q > PRE-CREATION > ql/src/test/queries/clientpositive/avro_decimal_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_joins_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_partitioned_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_schema_evolution_native.q > PRE-CREATION > ql/src/test/results/clientpositive/avro_compression_enabled_native.q.out > PRE-CREATION > ql/src/test/results/clientpositive/avro_decimal_native.q.out PRE-CREATION > ql/src/test/results/clientpositive/avro_joins_native.q.out PRE-CREATION > ql/src/test/results/clientpositive/avro_native.q.out PRE-CREATION > ql/src/test/results/clientpositive/avro_partitioned_native.q.out > PRE-CREATION > ql/src/test/results/clientpositive/avro_schema_evolution_native.q.out > PRE-CREATION > serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java > 0db12437406170686a21b6055d83156fe5d6a55f > serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerDe.java > 1fe31e0034f8988d03a0c51a90904bb93e7cb157 > serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerdeUtils.java > 4564e75d9bfc73f8e10f160e2535f1a08b90ff79 > serde/src/java/org/apache/hadoop/hive/serde2/avro/TypeInfoToSchema.java > PRE-CREATION > serde/src/test/org/apache/hadoop/hive/serde2/avro/TestTypeInfoToSchema.java > PRE-CREATION > > Diff: https://reviews.apache.org/r/23387/diff/ > > > Testing > --- > > Added qTests and unit tests > > > Thanks, > > Ashish Singh > >
Re: Review Request 23387: HIVE-6806: Native avro support
> On July 18, 2014, 1:57 p.m., Tom White wrote: > > serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerDe.java, line 112 > > <https://reviews.apache.org/r/23387/diff/9/?file=634614#file634614line112> > > > > Is it possible to default name to table name, namespace to database > > name, and doc to table comment? > > Ashish Singh wrote: > I was planning to do this, but slipped off my mind. Thanks for pointing > this out. I don't think it is possible to retrieve database name inside > serde. Addressed name and doc. > > Tom White wrote: > Thanks for fixing this. There's no test that "name" and "comment" are > picked up from the table definition - perhaps you could add one, or at least > confirm it manually. I couldn't see where in Hive they get set... > > Otherwise, +1 from me - thanks for addressing all my comments. This is a > great feature to add. Tom, actually its tested by all the unit tests now. Look at the diffs in https://reviews.apache.org/r/23387/diff/8-9/#index_header. - Ashish --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23387/#review48120 --- On July 19, 2014, 5:11 a.m., Ashish Singh wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/23387/ > --- > > (Updated July 19, 2014, 5:11 a.m.) > > > Review request for hive. > > > Bugs: HIVE-6806 > https://issues.apache.org/jira/browse/HIVE-6806 > > > Repository: hive-git > > > Description > --- > > HIVE-6806: Native avro support > > > Diffs > - > > ql/src/java/org/apache/hadoop/hive/ql/io/AvroStorageFormatDescriptor.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/io/IOConstants.java > 1bae0a8fee04049f90b16d813ff4c96707b349c8 > > ql/src/main/resources/META-INF/services/org.apache.hadoop.hive.ql.io.StorageFormatDescriptor > a23ff115512da5fe3167835a88d582c427585b8e > ql/src/test/org/apache/hadoop/hive/ql/io/TestStorageFormatDescriptor.java > d53ebc65174d66bfeee25fd2891c69c78f9137ee > ql/src/test/queries/clientpositive/avro_compression_enabled_native.q > PRE-CREATION > ql/src/test/queries/clientpositive/avro_decimal_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_joins_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_partitioned_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_schema_evolution_native.q > PRE-CREATION > ql/src/test/results/clientpositive/avro_compression_enabled_native.q.out > PRE-CREATION > ql/src/test/results/clientpositive/avro_decimal_native.q.out PRE-CREATION > ql/src/test/results/clientpositive/avro_joins_native.q.out PRE-CREATION > ql/src/test/results/clientpositive/avro_native.q.out PRE-CREATION > ql/src/test/results/clientpositive/avro_partitioned_native.q.out > PRE-CREATION > ql/src/test/results/clientpositive/avro_schema_evolution_native.q.out > PRE-CREATION > serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java > 0db12437406170686a21b6055d83156fe5d6a55f > serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerDe.java > 1fe31e0034f8988d03a0c51a90904bb93e7cb157 > serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerdeUtils.java > 4564e75d9bfc73f8e10f160e2535f1a08b90ff79 > serde/src/java/org/apache/hadoop/hive/serde2/avro/TypeInfoToSchema.java > PRE-CREATION > serde/src/test/org/apache/hadoop/hive/serde2/avro/TestTypeInfoToSchema.java > PRE-CREATION > > Diff: https://reviews.apache.org/r/23387/diff/ > > > Testing > --- > > Added qTests and unit tests > > > Thanks, > > Ashish Singh > >
Re: Review Request 23387: HIVE-6806: Native avro support
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23387/ --- (Updated July 19, 2014, 5:11 a.m.) Review request for hive. Bugs: HIVE-6806 https://issues.apache.org/jira/browse/HIVE-6806 Repository: hive-git Description --- HIVE-6806: Native avro support Diffs (updated) - ql/src/java/org/apache/hadoop/hive/ql/io/AvroStorageFormatDescriptor.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/io/IOConstants.java 1bae0a8fee04049f90b16d813ff4c96707b349c8 ql/src/main/resources/META-INF/services/org.apache.hadoop.hive.ql.io.StorageFormatDescriptor a23ff115512da5fe3167835a88d582c427585b8e ql/src/test/org/apache/hadoop/hive/ql/io/TestStorageFormatDescriptor.java d53ebc65174d66bfeee25fd2891c69c78f9137ee ql/src/test/queries/clientpositive/avro_compression_enabled_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_decimal_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_joins_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_partitioned_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_schema_evolution_native.q PRE-CREATION ql/src/test/results/clientpositive/avro_compression_enabled_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_decimal_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_joins_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_partitioned_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_schema_evolution_native.q.out PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java 0db12437406170686a21b6055d83156fe5d6a55f serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerDe.java 1fe31e0034f8988d03a0c51a90904bb93e7cb157 serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerdeUtils.java 4564e75d9bfc73f8e10f160e2535f1a08b90ff79 serde/src/java/org/apache/hadoop/hive/serde2/avro/TypeInfoToSchema.java PRE-CREATION serde/src/test/org/apache/hadoop/hive/serde2/avro/TestTypeInfoToSchema.java PRE-CREATION Diff: https://reviews.apache.org/r/23387/diff/ Testing --- Added qTests and unit tests Thanks, Ashish Singh
Re: Review Request 23387: HIVE-6806: Native avro support
> On July 19, 2014, 12:43 a.m., David Chen wrote:
> > serde/src/test/org/apache/hadoop/hive/serde2/avro/TestTypeInfoToSchema.java,
> > line 294
> > <https://reviews.apache.org/r/23387/diff/9/?file=634616#file634616line294>
> >
> > It would improve maintainability to keep the test schemas in separate
> > .avsc files under serde/src/test/resources rather than inline in the file.
> > You can use Guava's Resources class to get the file and construct the
> > schema. For example:
> >
> > Schema expectedSchema = new Schema.Parser().parse(
> > Resources.getResource("record1.avsc").openStream());
David, as the tests have a lot common in their schema I am using a method to
generate the common schema part and each test only provides a part of schema
that is specific to the test. This made my tests have much less LOC. If I
create a .avsc file for each test it will much more cumbersome for both
maintaining and adding new tests.
- Ashish
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23387/#review48168
-------
On July 17, 2014, 9:10 p.m., Ashish Singh wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/23387/
> ---
>
> (Updated July 17, 2014, 9:10 p.m.)
>
>
> Review request for hive.
>
>
> Bugs: HIVE-6806
> https://issues.apache.org/jira/browse/HIVE-6806
>
>
> Repository: hive-git
>
>
> Description
> ---
>
> HIVE-6806: Native avro support
>
>
> Diffs
> -
>
> ql/src/java/org/apache/hadoop/hive/ql/io/AvroStorageFormatDescriptor.java
> PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/IOConstants.java
> 1bae0a8fee04049f90b16d813ff4c96707b349c8
>
> ql/src/main/resources/META-INF/services/org.apache.hadoop.hive.ql.io.StorageFormatDescriptor
> a23ff115512da5fe3167835a88d582c427585b8e
> ql/src/test/org/apache/hadoop/hive/ql/io/TestStorageFormatDescriptor.java
> d53ebc65174d66bfeee25fd2891c69c78f9137ee
> ql/src/test/queries/clientpositive/avro_compression_enabled_native.q
> PRE-CREATION
> ql/src/test/queries/clientpositive/avro_decimal_native.q PRE-CREATION
> ql/src/test/queries/clientpositive/avro_joins_native.q PRE-CREATION
> ql/src/test/queries/clientpositive/avro_native.q PRE-CREATION
> ql/src/test/queries/clientpositive/avro_partitioned_native.q PRE-CREATION
> ql/src/test/queries/clientpositive/avro_schema_evolution_native.q
> PRE-CREATION
> ql/src/test/results/clientpositive/avro_compression_enabled_native.q.out
> PRE-CREATION
> ql/src/test/results/clientpositive/avro_decimal_native.q.out PRE-CREATION
> ql/src/test/results/clientpositive/avro_joins_native.q.out PRE-CREATION
> ql/src/test/results/clientpositive/avro_native.q.out PRE-CREATION
> ql/src/test/results/clientpositive/avro_partitioned_native.q.out
> PRE-CREATION
> ql/src/test/results/clientpositive/avro_schema_evolution_native.q.out
> PRE-CREATION
> serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java
> 0db12437406170686a21b6055d83156fe5d6a55f
> serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerDe.java
> 1fe31e0034f8988d03a0c51a90904bb93e7cb157
> serde/src/java/org/apache/hadoop/hive/serde2/avro/TypeInfoToSchema.java
> PRE-CREATION
> serde/src/test/org/apache/hadoop/hive/serde2/avro/TestTypeInfoToSchema.java
> PRE-CREATION
>
> Diff: https://reviews.apache.org/r/23387/diff/
>
>
> Testing
> ---
>
> Added qTests and unit tests
>
>
> Thanks,
>
> Ashish Singh
>
>
Re: Review Request 23387: HIVE-6806: Native avro support
> On July 18, 2014, 11:42 p.m., Lars Francke wrote: > > serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java, > > line 269 > > <https://reviews.apache.org/r/23387/diff/9/?file=634613#file634613line269> > > > > "final" is not used elsewhere in this file for local variables and I > > think it's true for most of the rest of Hive's code. That does not mean using final for local variables is wrong. Convention over correctness? Unless there is a strong reason for not declaring variables, not supposed to be modified, as final, I would argue using final is correct. > On July 18, 2014, 11:42 p.m., Lars Francke wrote: > > serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java, > > line 272 > > <https://reviews.apache.org/r/23387/diff/9/?file=634613#file634613line272> > > > > You're accessing fileSchema in the previous line so it's guaranteed to > > not be null here. Not sure if that's a bug and the check needs to happen a > > line earlier? If not this ternary can be removed. Good catch, this got introduced while refactoring a dual ternary statement. > On July 18, 2014, 11:42 p.m., Lars Francke wrote: > > serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerDe.java, line 76 > > <https://reviews.apache.org/r/23387/diff/9/?file=634614#file634614line76> > > > > "final" not consistent with Hive codebase Same as above. > On July 18, 2014, 11:42 p.m., Lars Francke wrote: > > serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerDe.java, line 82 > > <https://reviews.apache.org/r/23387/diff/9/?file=634614#file634614line82> > > > > .length() == 0 > > > > can be replaced with > > > > .isEmpty() Done. > On July 18, 2014, 11:42 p.m., Lars Francke wrote: > > serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerDe.java, line 87 > > <https://reviews.apache.org/r/23387/diff/9/?file=634614#file634614line87> > > > > .isEmpty() > > > > and two more times Done. On July 18, 2014, 11:42 p.m., Ashish Singh wrote: > > Only minor comments. Patch looks good. Thanks for this and I'm looking > > forward to have it included. Thanks for the review. - Ashish --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23387/#review48167 --- On July 17, 2014, 9:10 p.m., Ashish Singh wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/23387/ > --- > > (Updated July 17, 2014, 9:10 p.m.) > > > Review request for hive. > > > Bugs: HIVE-6806 > https://issues.apache.org/jira/browse/HIVE-6806 > > > Repository: hive-git > > > Description > --- > > HIVE-6806: Native avro support > > > Diffs > - > > ql/src/java/org/apache/hadoop/hive/ql/io/AvroStorageFormatDescriptor.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/io/IOConstants.java > 1bae0a8fee04049f90b16d813ff4c96707b349c8 > > ql/src/main/resources/META-INF/services/org.apache.hadoop.hive.ql.io.StorageFormatDescriptor > a23ff115512da5fe3167835a88d582c427585b8e > ql/src/test/org/apache/hadoop/hive/ql/io/TestStorageFormatDescriptor.java > d53ebc65174d66bfeee25fd2891c69c78f9137ee > ql/src/test/queries/clientpositive/avro_compression_enabled_native.q > PRE-CREATION > ql/src/test/queries/clientpositive/avro_decimal_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_joins_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_partitioned_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_schema_evolution_native.q > PRE-CREATION > ql/src/test/results/clientpositive/avro_compression_enabled_native.q.out > PRE-CREATION > ql/src/test/results/clientpositive/avro_decimal_native.q.out PRE-CREATION > ql/src/test/results/clientpositive/avro_joins_native.q.out PRE-CREATION > ql/src/test/results/clientpositive/avro_native.q.out PRE-CREATION > ql/src/test/results/clientpositive/avro_partitioned_native.q.out > PRE-CREATION > ql/src/test/results/clientpositive/avro_schema_evolution_native.q.out > PRE-CREATION > serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java > 0db12437406170686a21b6055d83156fe5d6a55f > serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerDe.java > 1fe31e0034f8988d03a0c51a90904bb93e7cb157 > serde/src/java/org/apache/hadoop/hive/serde2/avro/TypeInfoToSchema.java > PRE-CREATION > serde/src/test/org/apache/hadoop/hive/serde2/avro/TestTypeInfoToSchema.java > PRE-CREATION > > Diff: https://reviews.apache.org/r/23387/diff/ > > > Testing > --- > > Added qTests and unit tests > > > Thanks, > > Ashish Singh > >
Re: Review Request 23387: HIVE-6806: Native avro support
> On July 18, 2014, 1:57 p.m., Tom White wrote: > > ql/src/test/queries/clientpositive/avro_schema_evolution_native.q, line 25 > > <https://reviews.apache.org/r/23387/diff/9/?file=634606#file634606line25> > > > > It would be nice to support ALTER TABLE t ADD COLUMN ..., but that is > > an enhancement - file another JIRA for that? Created HIVE-7446. > On July 18, 2014, 1:57 p.m., Tom White wrote: > > serde/src/java/org/apache/hadoop/hive/serde2/avro/TypeInfoToSchema.java, > > line 238 > > <https://reviews.apache.org/r/23387/diff/9/?file=634615#file634615line238> > > > > This should add to the start of the list. Also, needs a test for this > > case. This is actually unreachable code. removeDuplicateNullSchemas makes sure that null is at the beginning of the list. And there will never be a case that a union will be reach here without having a null. Added a couple of test cases to be sure. > On July 18, 2014, 1:57 p.m., Tom White wrote: > > serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerDe.java, line 112 > > <https://reviews.apache.org/r/23387/diff/9/?file=634614#file634614line112> > > > > Is it possible to default name to table name, namespace to database > > name, and doc to table comment? I was planning to do this, but slipped off my mind. Thanks for pointing this out. I don't think it is possible to retrieve database name inside serde. Addressed name and doc. - Ashish --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23387/#review48120 --- On July 17, 2014, 9:10 p.m., Ashish Singh wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/23387/ > --- > > (Updated July 17, 2014, 9:10 p.m.) > > > Review request for hive. > > > Bugs: HIVE-6806 > https://issues.apache.org/jira/browse/HIVE-6806 > > > Repository: hive-git > > > Description > --- > > HIVE-6806: Native avro support > > > Diffs > - > > ql/src/java/org/apache/hadoop/hive/ql/io/AvroStorageFormatDescriptor.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/io/IOConstants.java > 1bae0a8fee04049f90b16d813ff4c96707b349c8 > > ql/src/main/resources/META-INF/services/org.apache.hadoop.hive.ql.io.StorageFormatDescriptor > a23ff115512da5fe3167835a88d582c427585b8e > ql/src/test/org/apache/hadoop/hive/ql/io/TestStorageFormatDescriptor.java > d53ebc65174d66bfeee25fd2891c69c78f9137ee > ql/src/test/queries/clientpositive/avro_compression_enabled_native.q > PRE-CREATION > ql/src/test/queries/clientpositive/avro_decimal_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_joins_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_partitioned_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_schema_evolution_native.q > PRE-CREATION > ql/src/test/results/clientpositive/avro_compression_enabled_native.q.out > PRE-CREATION > ql/src/test/results/clientpositive/avro_decimal_native.q.out PRE-CREATION > ql/src/test/results/clientpositive/avro_joins_native.q.out PRE-CREATION > ql/src/test/results/clientpositive/avro_native.q.out PRE-CREATION > ql/src/test/results/clientpositive/avro_partitioned_native.q.out > PRE-CREATION > ql/src/test/results/clientpositive/avro_schema_evolution_native.q.out > PRE-CREATION > serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java > 0db12437406170686a21b6055d83156fe5d6a55f > serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerDe.java > 1fe31e0034f8988d03a0c51a90904bb93e7cb157 > serde/src/java/org/apache/hadoop/hive/serde2/avro/TypeInfoToSchema.java > PRE-CREATION > serde/src/test/org/apache/hadoop/hive/serde2/avro/TestTypeInfoToSchema.java > PRE-CREATION > > Diff: https://reviews.apache.org/r/23387/diff/ > > > Testing > --- > > Added qTests and unit tests > > > Thanks, > > Ashish Singh > >
Re: Review Request 23387: HIVE-6806: Native avro support
> On July 17, 2014, 1:49 p.m., Tom White wrote: > > serde/src/java/org/apache/hadoop/hive/serde2/avro/TypeInfoToSchema.java, > > line 229 > > <https://reviews.apache.org/r/23387/diff/8/?file=634160#file634160line229> > > > > It would be simpler to make sure that NULL is included (and is the > > first branch in the union) in the createAvroUnion() method, and just fall > > through here. > > Ashish Singh wrote: > I do not think this will be something good or feasible without > redesigning many parts without any obvious gain. createAvroUnion() only > creates a schema for union, based on union typeinfo passed to it. If I hack > it to add null to all unions, I will still need to handle union here > differently as union of unions is not possible. > > Tom White wrote: > That's OK. What you have now is much better, although I found a small bug > where the null isn't being added to the start of the list (see other comment). OK. Dropping this issue. - Ashish --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23387/#review48004 ------- On July 17, 2014, 9:10 p.m., Ashish Singh wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/23387/ > --- > > (Updated July 17, 2014, 9:10 p.m.) > > > Review request for hive. > > > Bugs: HIVE-6806 > https://issues.apache.org/jira/browse/HIVE-6806 > > > Repository: hive-git > > > Description > --- > > HIVE-6806: Native avro support > > > Diffs > - > > ql/src/java/org/apache/hadoop/hive/ql/io/AvroStorageFormatDescriptor.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/io/IOConstants.java > 1bae0a8fee04049f90b16d813ff4c96707b349c8 > > ql/src/main/resources/META-INF/services/org.apache.hadoop.hive.ql.io.StorageFormatDescriptor > a23ff115512da5fe3167835a88d582c427585b8e > ql/src/test/org/apache/hadoop/hive/ql/io/TestStorageFormatDescriptor.java > d53ebc65174d66bfeee25fd2891c69c78f9137ee > ql/src/test/queries/clientpositive/avro_compression_enabled_native.q > PRE-CREATION > ql/src/test/queries/clientpositive/avro_decimal_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_joins_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_partitioned_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_schema_evolution_native.q > PRE-CREATION > ql/src/test/results/clientpositive/avro_compression_enabled_native.q.out > PRE-CREATION > ql/src/test/results/clientpositive/avro_decimal_native.q.out PRE-CREATION > ql/src/test/results/clientpositive/avro_joins_native.q.out PRE-CREATION > ql/src/test/results/clientpositive/avro_native.q.out PRE-CREATION > ql/src/test/results/clientpositive/avro_partitioned_native.q.out > PRE-CREATION > ql/src/test/results/clientpositive/avro_schema_evolution_native.q.out > PRE-CREATION > serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java > 0db12437406170686a21b6055d83156fe5d6a55f > serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerDe.java > 1fe31e0034f8988d03a0c51a90904bb93e7cb157 > serde/src/java/org/apache/hadoop/hive/serde2/avro/TypeInfoToSchema.java > PRE-CREATION > serde/src/test/org/apache/hadoop/hive/serde2/avro/TestTypeInfoToSchema.java > PRE-CREATION > > Diff: https://reviews.apache.org/r/23387/diff/ > > > Testing > --- > > Added qTests and unit tests > > > Thanks, > > Ashish Singh > >
Re: Review Request 23387: HIVE-6806: Native avro support
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23387/#review48137 --- serde/src/java/org/apache/hadoop/hive/serde2/avro/TypeInfoToSchema.java <https://reviews.apache.org/r/23387/#comment84434> Good catch. - Ashish Singh On July 17, 2014, 9:10 p.m., Ashish Singh wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/23387/ > --- > > (Updated July 17, 2014, 9:10 p.m.) > > > Review request for hive. > > > Bugs: HIVE-6806 > https://issues.apache.org/jira/browse/HIVE-6806 > > > Repository: hive-git > > > Description > --- > > HIVE-6806: Native avro support > > > Diffs > - > > ql/src/java/org/apache/hadoop/hive/ql/io/AvroStorageFormatDescriptor.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/io/IOConstants.java > 1bae0a8fee04049f90b16d813ff4c96707b349c8 > > ql/src/main/resources/META-INF/services/org.apache.hadoop.hive.ql.io.StorageFormatDescriptor > a23ff115512da5fe3167835a88d582c427585b8e > ql/src/test/org/apache/hadoop/hive/ql/io/TestStorageFormatDescriptor.java > d53ebc65174d66bfeee25fd2891c69c78f9137ee > ql/src/test/queries/clientpositive/avro_compression_enabled_native.q > PRE-CREATION > ql/src/test/queries/clientpositive/avro_decimal_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_joins_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_partitioned_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_schema_evolution_native.q > PRE-CREATION > ql/src/test/results/clientpositive/avro_compression_enabled_native.q.out > PRE-CREATION > ql/src/test/results/clientpositive/avro_decimal_native.q.out PRE-CREATION > ql/src/test/results/clientpositive/avro_joins_native.q.out PRE-CREATION > ql/src/test/results/clientpositive/avro_native.q.out PRE-CREATION > ql/src/test/results/clientpositive/avro_partitioned_native.q.out > PRE-CREATION > ql/src/test/results/clientpositive/avro_schema_evolution_native.q.out > PRE-CREATION > serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java > 0db12437406170686a21b6055d83156fe5d6a55f > serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerDe.java > 1fe31e0034f8988d03a0c51a90904bb93e7cb157 > serde/src/java/org/apache/hadoop/hive/serde2/avro/TypeInfoToSchema.java > PRE-CREATION > serde/src/test/org/apache/hadoop/hive/serde2/avro/TestTypeInfoToSchema.java > PRE-CREATION > > Diff: https://reviews.apache.org/r/23387/diff/ > > > Testing > --- > > Added qTests and unit tests > > > Thanks, > > Ashish Singh > >
Re: Review Request 23387: HIVE-6806: Native avro support
> On July 17, 2014, 5:33 a.m., David Chen wrote: > > ql/src/java/org/apache/hadoop/hive/ql/io/IOConstants.java, line 37 > > <https://reviews.apache.org/r/23387/diff/8/?file=634143#file634143line37> > > > > Is using AVROFILE rather than AVRO a common use case? If not, should we > > be allowing both? > > Ashish Singh wrote: > David, I initially did not add AVROFILE. Brock suggested to be consistent > with other storage formats, its good to have it. I do not see any harm in > having it. > > David Chen wrote: > I see. That's fine with me. I was just wondering. Ok, then I will leave it there. Thanks for the review. - Ashish --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23387/#review47984 ------- On July 17, 2014, 9:10 p.m., Ashish Singh wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/23387/ > --- > > (Updated July 17, 2014, 9:10 p.m.) > > > Review request for hive. > > > Bugs: HIVE-6806 > https://issues.apache.org/jira/browse/HIVE-6806 > > > Repository: hive-git > > > Description > --- > > HIVE-6806: Native avro support > > > Diffs > - > > ql/src/java/org/apache/hadoop/hive/ql/io/AvroStorageFormatDescriptor.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/io/IOConstants.java > 1bae0a8fee04049f90b16d813ff4c96707b349c8 > > ql/src/main/resources/META-INF/services/org.apache.hadoop.hive.ql.io.StorageFormatDescriptor > a23ff115512da5fe3167835a88d582c427585b8e > ql/src/test/org/apache/hadoop/hive/ql/io/TestStorageFormatDescriptor.java > d53ebc65174d66bfeee25fd2891c69c78f9137ee > ql/src/test/queries/clientpositive/avro_compression_enabled_native.q > PRE-CREATION > ql/src/test/queries/clientpositive/avro_decimal_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_joins_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_partitioned_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_schema_evolution_native.q > PRE-CREATION > ql/src/test/results/clientpositive/avro_compression_enabled_native.q.out > PRE-CREATION > ql/src/test/results/clientpositive/avro_decimal_native.q.out PRE-CREATION > ql/src/test/results/clientpositive/avro_joins_native.q.out PRE-CREATION > ql/src/test/results/clientpositive/avro_native.q.out PRE-CREATION > ql/src/test/results/clientpositive/avro_partitioned_native.q.out > PRE-CREATION > ql/src/test/results/clientpositive/avro_schema_evolution_native.q.out > PRE-CREATION > serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java > 0db12437406170686a21b6055d83156fe5d6a55f > serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerDe.java > 1fe31e0034f8988d03a0c51a90904bb93e7cb157 > serde/src/java/org/apache/hadoop/hive/serde2/avro/TypeInfoToSchema.java > PRE-CREATION > serde/src/test/org/apache/hadoop/hive/serde2/avro/TestTypeInfoToSchema.java > PRE-CREATION > > Diff: https://reviews.apache.org/r/23387/diff/ > > > Testing > --- > > Added qTests and unit tests > > > Thanks, > > Ashish Singh > >
Re: Review Request 23387: HIVE-6806: Native avro support
> On July 17, 2014, 5:33 a.m., David Chen wrote: > > ql/src/java/org/apache/hadoop/hive/ql/io/IOConstants.java, line 37 > > <https://reviews.apache.org/r/23387/diff/8/?file=634143#file634143line37> > > > > Is using AVROFILE rather than AVRO a common use case? If not, should we > > be allowing both? David, I initially did not add AVROFILE. Brock suggested to be consistent with other storage formats, its good to have it. I do not see any harm in having it. - Ashish --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23387/#review47984 --- On July 17, 2014, 9:10 p.m., Ashish Singh wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/23387/ > --- > > (Updated July 17, 2014, 9:10 p.m.) > > > Review request for hive. > > > Bugs: HIVE-6806 > https://issues.apache.org/jira/browse/HIVE-6806 > > > Repository: hive-git > > > Description > --- > > HIVE-6806: Native avro support > > > Diffs > - > > ql/src/java/org/apache/hadoop/hive/ql/io/AvroStorageFormatDescriptor.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/io/IOConstants.java > 1bae0a8fee04049f90b16d813ff4c96707b349c8 > > ql/src/main/resources/META-INF/services/org.apache.hadoop.hive.ql.io.StorageFormatDescriptor > a23ff115512da5fe3167835a88d582c427585b8e > ql/src/test/org/apache/hadoop/hive/ql/io/TestStorageFormatDescriptor.java > d53ebc65174d66bfeee25fd2891c69c78f9137ee > ql/src/test/queries/clientpositive/avro_compression_enabled_native.q > PRE-CREATION > ql/src/test/queries/clientpositive/avro_decimal_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_joins_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_partitioned_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_schema_evolution_native.q > PRE-CREATION > ql/src/test/results/clientpositive/avro_compression_enabled_native.q.out > PRE-CREATION > ql/src/test/results/clientpositive/avro_decimal_native.q.out PRE-CREATION > ql/src/test/results/clientpositive/avro_joins_native.q.out PRE-CREATION > ql/src/test/results/clientpositive/avro_native.q.out PRE-CREATION > ql/src/test/results/clientpositive/avro_partitioned_native.q.out > PRE-CREATION > ql/src/test/results/clientpositive/avro_schema_evolution_native.q.out > PRE-CREATION > serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java > 0db12437406170686a21b6055d83156fe5d6a55f > serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerDe.java > 1fe31e0034f8988d03a0c51a90904bb93e7cb157 > serde/src/java/org/apache/hadoop/hive/serde2/avro/TypeInfoToSchema.java > PRE-CREATION > serde/src/test/org/apache/hadoop/hive/serde2/avro/TestTypeInfoToSchema.java > PRE-CREATION > > Diff: https://reviews.apache.org/r/23387/diff/ > > > Testing > --- > > Added qTests and unit tests > > > Thanks, > > Ashish Singh > >
Re: Review Request 23387: HIVE-6806: Native avro support
> On July 17, 2014, 1:49 p.m., Tom White wrote: > > Ashish, thanks for addressing my feedback. Here's a bit more. Thanks again for the review. > On July 17, 2014, 1:49 p.m., Tom White wrote: > > serde/src/java/org/apache/hadoop/hive/serde2/avro/TypeInfoToSchema.java, > > line 229 > > <https://reviews.apache.org/r/23387/diff/8/?file=634160#file634160line229> > > > > It would be simpler to make sure that NULL is included (and is the > > first branch in the union) in the createAvroUnion() method, and just fall > > through here. I do not think this will be something good or feasible without redesigning many parts without any obvious gain. createAvroUnion() only creates a schema for union, based on union typeinfo passed to it. If I hack it to add null to all unions, I will still need to handle union here differently as union of unions is not possible. - Ashish --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23387/#review48004 ------- On July 17, 2014, 2:50 a.m., Ashish Singh wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/23387/ > --- > > (Updated July 17, 2014, 2:50 a.m.) > > > Review request for hive. > > > Bugs: HIVE-6806 > https://issues.apache.org/jira/browse/HIVE-6806 > > > Repository: hive-git > > > Description > --- > > HIVE-6806: Native avro support > > > Diffs > - > > ql/src/java/org/apache/hadoop/hive/ql/io/AvroStorageFormatDescriptor.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/io/IOConstants.java > 1bae0a8fee04049f90b16d813ff4c96707b349c8 > > ql/src/main/resources/META-INF/services/org.apache.hadoop.hive.ql.io.StorageFormatDescriptor > a23ff115512da5fe3167835a88d582c427585b8e > ql/src/test/org/apache/hadoop/hive/ql/io/TestStorageFormatDescriptor.java > d53ebc65174d66bfeee25fd2891c69c78f9137ee > ql/src/test/queries/clientpositive/avro_compression_enabled_native.q > PRE-CREATION > ql/src/test/queries/clientpositive/avro_decimal_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_joins_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_partitioned_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_schema_evolution_native.q > PRE-CREATION > ql/src/test/results/clientpositive/avro_compression_enabled_native.q.out > PRE-CREATION > ql/src/test/results/clientpositive/avro_decimal_native.q.out PRE-CREATION > ql/src/test/results/clientpositive/avro_joins_native.q.out PRE-CREATION > ql/src/test/results/clientpositive/avro_native.q.out PRE-CREATION > ql/src/test/results/clientpositive/avro_partitioned_native.q.out > PRE-CREATION > ql/src/test/results/clientpositive/avro_schema_evolution_native.q.out > PRE-CREATION > serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java > 0db12437406170686a21b6055d83156fe5d6a55f > serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerDe.java > 1fe31e0034f8988d03a0c51a90904bb93e7cb157 > serde/src/java/org/apache/hadoop/hive/serde2/avro/TypeInfoToSchema.java > PRE-CREATION > serde/src/test/org/apache/hadoop/hive/serde2/avro/TestTypeInfoToSchema.java > PRE-CREATION > > Diff: https://reviews.apache.org/r/23387/diff/ > > > Testing > --- > > Added qTests and unit tests > > > Thanks, > > Ashish Singh > >
Re: Review Request 23387: HIVE-6806: Native avro support
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23387/ --- (Updated July 17, 2014, 9:10 p.m.) Review request for hive. Changes --- Addressed more review comments. Bugs: HIVE-6806 https://issues.apache.org/jira/browse/HIVE-6806 Repository: hive-git Description --- HIVE-6806: Native avro support Diffs (updated) - ql/src/java/org/apache/hadoop/hive/ql/io/AvroStorageFormatDescriptor.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/io/IOConstants.java 1bae0a8fee04049f90b16d813ff4c96707b349c8 ql/src/main/resources/META-INF/services/org.apache.hadoop.hive.ql.io.StorageFormatDescriptor a23ff115512da5fe3167835a88d582c427585b8e ql/src/test/org/apache/hadoop/hive/ql/io/TestStorageFormatDescriptor.java d53ebc65174d66bfeee25fd2891c69c78f9137ee ql/src/test/queries/clientpositive/avro_compression_enabled_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_decimal_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_joins_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_partitioned_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_schema_evolution_native.q PRE-CREATION ql/src/test/results/clientpositive/avro_compression_enabled_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_decimal_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_joins_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_partitioned_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_schema_evolution_native.q.out PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java 0db12437406170686a21b6055d83156fe5d6a55f serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerDe.java 1fe31e0034f8988d03a0c51a90904bb93e7cb157 serde/src/java/org/apache/hadoop/hive/serde2/avro/TypeInfoToSchema.java PRE-CREATION serde/src/test/org/apache/hadoop/hive/serde2/avro/TestTypeInfoToSchema.java PRE-CREATION Diff: https://reviews.apache.org/r/23387/diff/ Testing --- Added qTests and unit tests Thanks, Ashish Singh
Re: Review Request 23387: HIVE-6806: Native avro support
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23387/#review47979
---
ql/src/java/org/apache/hadoop/hive/ql/io/IOConstants.java
<https://reviews.apache.org/r/23387/#comment84224>
Done.
serde/src/java/org/apache/hadoop/hive/serde2/avro/TypeInfoToSchema.java
<https://reviews.apache.org/r/23387/#comment84225>
Done.
- Ashish Singh
On July 17, 2014, 2:50 a.m., Ashish Singh wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/23387/
> ---
>
> (Updated July 17, 2014, 2:50 a.m.)
>
>
> Review request for hive.
>
>
> Bugs: HIVE-6806
> https://issues.apache.org/jira/browse/HIVE-6806
>
>
> Repository: hive-git
>
>
> Description
> ---
>
> HIVE-6806: Native avro support
>
>
> Diffs
> -
>
> ql/src/java/org/apache/hadoop/hive/ql/io/AvroStorageFormatDescriptor.java
> PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/IOConstants.java
> 1bae0a8fee04049f90b16d813ff4c96707b349c8
>
> ql/src/main/resources/META-INF/services/org.apache.hadoop.hive.ql.io.StorageFormatDescriptor
> a23ff115512da5fe3167835a88d582c427585b8e
> ql/src/test/org/apache/hadoop/hive/ql/io/TestStorageFormatDescriptor.java
> d53ebc65174d66bfeee25fd2891c69c78f9137ee
> ql/src/test/queries/clientpositive/avro_compression_enabled_native.q
> PRE-CREATION
> ql/src/test/queries/clientpositive/avro_decimal_native.q PRE-CREATION
> ql/src/test/queries/clientpositive/avro_joins_native.q PRE-CREATION
> ql/src/test/queries/clientpositive/avro_native.q PRE-CREATION
> ql/src/test/queries/clientpositive/avro_partitioned_native.q PRE-CREATION
> ql/src/test/queries/clientpositive/avro_schema_evolution_native.q
> PRE-CREATION
> ql/src/test/results/clientpositive/avro_compression_enabled_native.q.out
> PRE-CREATION
> ql/src/test/results/clientpositive/avro_decimal_native.q.out PRE-CREATION
> ql/src/test/results/clientpositive/avro_joins_native.q.out PRE-CREATION
> ql/src/test/results/clientpositive/avro_native.q.out PRE-CREATION
> ql/src/test/results/clientpositive/avro_partitioned_native.q.out
> PRE-CREATION
> ql/src/test/results/clientpositive/avro_schema_evolution_native.q.out
> PRE-CREATION
> serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java
> 0db12437406170686a21b6055d83156fe5d6a55f
> serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerDe.java
> 1fe31e0034f8988d03a0c51a90904bb93e7cb157
> serde/src/java/org/apache/hadoop/hive/serde2/avro/TypeInfoToSchema.java
> PRE-CREATION
> serde/src/test/org/apache/hadoop/hive/serde2/avro/TestTypeInfoToSchema.java
> PRE-CREATION
>
> Diff: https://reviews.apache.org/r/23387/diff/
>
>
> Testing
> ---
>
> Added qTests and unit tests
>
>
> Thanks,
>
> Ashish Singh
>
>
Re: Review Request 23387: HIVE-6806: Native avro support
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23387/ --- (Updated July 17, 2014, 2:50 a.m.) Review request for hive. Changes --- Addressed review comments. Bugs: HIVE-6806 https://issues.apache.org/jira/browse/HIVE-6806 Repository: hive-git Description --- HIVE-6806: Native avro support Diffs (updated) - ql/src/java/org/apache/hadoop/hive/ql/io/AvroStorageFormatDescriptor.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/io/IOConstants.java 1bae0a8fee04049f90b16d813ff4c96707b349c8 ql/src/main/resources/META-INF/services/org.apache.hadoop.hive.ql.io.StorageFormatDescriptor a23ff115512da5fe3167835a88d582c427585b8e ql/src/test/org/apache/hadoop/hive/ql/io/TestStorageFormatDescriptor.java d53ebc65174d66bfeee25fd2891c69c78f9137ee ql/src/test/queries/clientpositive/avro_compression_enabled_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_decimal_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_joins_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_partitioned_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_schema_evolution_native.q PRE-CREATION ql/src/test/results/clientpositive/avro_compression_enabled_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_decimal_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_joins_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_partitioned_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_schema_evolution_native.q.out PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java 0db12437406170686a21b6055d83156fe5d6a55f serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerDe.java 1fe31e0034f8988d03a0c51a90904bb93e7cb157 serde/src/java/org/apache/hadoop/hive/serde2/avro/TypeInfoToSchema.java PRE-CREATION serde/src/test/org/apache/hadoop/hive/serde2/avro/TestTypeInfoToSchema.java PRE-CREATION Diff: https://reviews.apache.org/r/23387/diff/ Testing --- Added qTests and unit tests Thanks, Ashish Singh
Re: Review Request 23387: HIVE-6806: Native avro support
> On July 16, 2014, 4:45 p.m., Tom White wrote:
> > serde/src/java/org/apache/hadoop/hive/serde2/avro/TypeInfoToSchema.java,
> > line 97
> > <https://reviews.apache.org/r/23387/diff/7/?file=633382#file633382line97>
> >
> > Are optional fields supported properly? All schemas should probably be
> > wrapped in an Avro null union to allow values to be null:
> >
> > private static Schema optional(Schema schema) {
> > return
> > Schema.createUnion(Arrays.asList(Schema.create(Schema.Type.NULL), schema));
> > }
Tom, could you elaborate on "Are optional fields supported properly?". I am in
process of addressing your review comments, which are good. Thanks for the
review.
- Ashish
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23387/#review47894
-------
On July 16, 2014, 3:35 a.m., Ashish Singh wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/23387/
> ---
>
> (Updated July 16, 2014, 3:35 a.m.)
>
>
> Review request for hive.
>
>
> Bugs: HIVE-6806
> https://issues.apache.org/jira/browse/HIVE-6806
>
>
> Repository: hive-git
>
>
> Description
> ---
>
> HIVE-6806: Native avro support
>
>
> Diffs
> -
>
> ql/src/java/org/apache/hadoop/hive/ql/io/AvroStorageFormatDescriptor.java
> PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/IOConstants.java
> 1bae0a8fee04049f90b16d813ff4c96707b349c8
>
> ql/src/main/resources/META-INF/services/org.apache.hadoop.hive.ql.io.StorageFormatDescriptor
> a23ff115512da5fe3167835a88d582c427585b8e
> ql/src/test/org/apache/hadoop/hive/ql/io/TestStorageFormatDescriptor.java
> d53ebc65174d66bfeee25fd2891c69c78f9137ee
> ql/src/test/queries/clientpositive/avro_compression_enabled_native.q
> PRE-CREATION
> ql/src/test/queries/clientpositive/avro_decimal_native.q PRE-CREATION
> ql/src/test/queries/clientpositive/avro_joins_native.q PRE-CREATION
> ql/src/test/queries/clientpositive/avro_native.q PRE-CREATION
> ql/src/test/queries/clientpositive/avro_partitioned_native.q PRE-CREATION
> ql/src/test/queries/clientpositive/avro_schema_evolution_native.q
> PRE-CREATION
> ql/src/test/results/clientpositive/avro_compression_enabled_native.q.out
> PRE-CREATION
> ql/src/test/results/clientpositive/avro_decimal_native.q.out PRE-CREATION
> ql/src/test/results/clientpositive/avro_joins_native.q.out PRE-CREATION
> ql/src/test/results/clientpositive/avro_native.q.out PRE-CREATION
> ql/src/test/results/clientpositive/avro_partitioned_native.q.out
> PRE-CREATION
> ql/src/test/results/clientpositive/avro_schema_evolution_native.q.out
> PRE-CREATION
> serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerDe.java
> 1fe31e0034f8988d03a0c51a90904bb93e7cb157
> serde/src/java/org/apache/hadoop/hive/serde2/avro/TypeInfoToSchema.java
> PRE-CREATION
> serde/src/test/org/apache/hadoop/hive/serde2/avro/TestTypeInfoToSchema.java
> PRE-CREATION
>
> Diff: https://reviews.apache.org/r/23387/diff/
>
>
> Testing
> ---
>
> Added qTests and unit tests
>
>
> Thanks,
>
> Ashish Singh
>
>
Re: Review Request 23387: HIVE-6806: Native avro support
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23387/#review47854 --- serde/src/test/org/apache/hadoop/hive/serde2/avro/TestTypeInfoToSchema.java <https://reviews.apache.org/r/23387/#comment84087> Done. - Ashish Singh On July 16, 2014, 3:35 a.m., Ashish Singh wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/23387/ > --- > > (Updated July 16, 2014, 3:35 a.m.) > > > Review request for hive. > > > Bugs: HIVE-6806 > https://issues.apache.org/jira/browse/HIVE-6806 > > > Repository: hive-git > > > Description > --- > > HIVE-6806: Native avro support > > > Diffs > - > > ql/src/java/org/apache/hadoop/hive/ql/io/AvroStorageFormatDescriptor.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/io/IOConstants.java > 1bae0a8fee04049f90b16d813ff4c96707b349c8 > > ql/src/main/resources/META-INF/services/org.apache.hadoop.hive.ql.io.StorageFormatDescriptor > a23ff115512da5fe3167835a88d582c427585b8e > ql/src/test/org/apache/hadoop/hive/ql/io/TestStorageFormatDescriptor.java > d53ebc65174d66bfeee25fd2891c69c78f9137ee > ql/src/test/queries/clientpositive/avro_compression_enabled_native.q > PRE-CREATION > ql/src/test/queries/clientpositive/avro_decimal_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_joins_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_partitioned_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_schema_evolution_native.q > PRE-CREATION > ql/src/test/results/clientpositive/avro_compression_enabled_native.q.out > PRE-CREATION > ql/src/test/results/clientpositive/avro_decimal_native.q.out PRE-CREATION > ql/src/test/results/clientpositive/avro_joins_native.q.out PRE-CREATION > ql/src/test/results/clientpositive/avro_native.q.out PRE-CREATION > ql/src/test/results/clientpositive/avro_partitioned_native.q.out > PRE-CREATION > ql/src/test/results/clientpositive/avro_schema_evolution_native.q.out > PRE-CREATION > serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerDe.java > 1fe31e0034f8988d03a0c51a90904bb93e7cb157 > serde/src/java/org/apache/hadoop/hive/serde2/avro/TypeInfoToSchema.java > PRE-CREATION > serde/src/test/org/apache/hadoop/hive/serde2/avro/TestTypeInfoToSchema.java > PRE-CREATION > > Diff: https://reviews.apache.org/r/23387/diff/ > > > Testing > --- > > Added qTests and unit tests > > > Thanks, > > Ashish Singh > >
Re: Review Request 23387: HIVE-6806: Native avro support
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23387/ --- (Updated July 16, 2014, 3:35 a.m.) Review request for hive. Changes --- Fixed some more spacing issue. Bugs: HIVE-6806 https://issues.apache.org/jira/browse/HIVE-6806 Repository: hive-git Description --- HIVE-6806: Native avro support Diffs (updated) - ql/src/java/org/apache/hadoop/hive/ql/io/AvroStorageFormatDescriptor.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/io/IOConstants.java 1bae0a8fee04049f90b16d813ff4c96707b349c8 ql/src/main/resources/META-INF/services/org.apache.hadoop.hive.ql.io.StorageFormatDescriptor a23ff115512da5fe3167835a88d582c427585b8e ql/src/test/org/apache/hadoop/hive/ql/io/TestStorageFormatDescriptor.java d53ebc65174d66bfeee25fd2891c69c78f9137ee ql/src/test/queries/clientpositive/avro_compression_enabled_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_decimal_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_joins_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_partitioned_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_schema_evolution_native.q PRE-CREATION ql/src/test/results/clientpositive/avro_compression_enabled_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_decimal_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_joins_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_partitioned_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_schema_evolution_native.q.out PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerDe.java 1fe31e0034f8988d03a0c51a90904bb93e7cb157 serde/src/java/org/apache/hadoop/hive/serde2/avro/TypeInfoToSchema.java PRE-CREATION serde/src/test/org/apache/hadoop/hive/serde2/avro/TestTypeInfoToSchema.java PRE-CREATION Diff: https://reviews.apache.org/r/23387/diff/ Testing --- Added qTests and unit tests Thanks, Ashish Singh
Re: Review Request 23387: HIVE-6806: Native avro support
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23387/#review47852 --- serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerDe.java <https://reviews.apache.org/r/23387/#comment84079> Done. serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerDe.java <https://reviews.apache.org/r/23387/#comment84080> Done. serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerDe.java <https://reviews.apache.org/r/23387/#comment84081> Done. serde/src/java/org/apache/hadoop/hive/serde2/avro/TypeInfoToSchema.java <https://reviews.apache.org/r/23387/#comment84082> Changed it to convert, as TypeInfoToSchema.convert says it all. serde/src/java/org/apache/hadoop/hive/serde2/avro/TypeInfoToSchema.java <https://reviews.apache.org/r/23387/#comment84083> Done. serde/src/java/org/apache/hadoop/hive/serde2/avro/TypeInfoToSchema.java <https://reviews.apache.org/r/23387/#comment84084> Done. serde/src/java/org/apache/hadoop/hive/serde2/avro/TypeInfoToSchema.java <https://reviews.apache.org/r/23387/#comment84085> Done. - Ashish Singh On July 16, 2014, 3:28 a.m., Ashish Singh wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/23387/ > --- > > (Updated July 16, 2014, 3:28 a.m.) > > > Review request for hive. > > > Bugs: HIVE-6806 > https://issues.apache.org/jira/browse/HIVE-6806 > > > Repository: hive-git > > > Description > --- > > HIVE-6806: Native avro support > > > Diffs > - > > ql/src/java/org/apache/hadoop/hive/ql/io/AvroStorageFormatDescriptor.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/io/IOConstants.java > 1bae0a8fee04049f90b16d813ff4c96707b349c8 > > ql/src/main/resources/META-INF/services/org.apache.hadoop.hive.ql.io.StorageFormatDescriptor > a23ff115512da5fe3167835a88d582c427585b8e > ql/src/test/org/apache/hadoop/hive/ql/io/TestStorageFormatDescriptor.java > d53ebc65174d66bfeee25fd2891c69c78f9137ee > ql/src/test/queries/clientpositive/avro_compression_enabled_native.q > PRE-CREATION > ql/src/test/queries/clientpositive/avro_decimal_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_joins_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_partitioned_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_schema_evolution_native.q > PRE-CREATION > ql/src/test/results/clientpositive/avro_compression_enabled_native.q.out > PRE-CREATION > ql/src/test/results/clientpositive/avro_decimal_native.q.out PRE-CREATION > ql/src/test/results/clientpositive/avro_joins_native.q.out PRE-CREATION > ql/src/test/results/clientpositive/avro_native.q.out PRE-CREATION > ql/src/test/results/clientpositive/avro_partitioned_native.q.out > PRE-CREATION > ql/src/test/results/clientpositive/avro_schema_evolution_native.q.out > PRE-CREATION > serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerDe.java > 1fe31e0034f8988d03a0c51a90904bb93e7cb157 > serde/src/java/org/apache/hadoop/hive/serde2/avro/TypeInfoToSchema.java > PRE-CREATION > serde/src/test/org/apache/hadoop/hive/serde2/avro/TestTypeInfoToSchema.java > PRE-CREATION > > Diff: https://reviews.apache.org/r/23387/diff/ > > > Testing > --- > > Added qTests and unit tests > > > Thanks, > > Ashish Singh > >
Re: Review Request 23387: HIVE-6806: Native avro support
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23387/ --- (Updated July 16, 2014, 3:28 a.m.) Review request for hive. Changes --- Addressed review comments. Bugs: HIVE-6806 https://issues.apache.org/jira/browse/HIVE-6806 Repository: hive-git Description --- HIVE-6806: Native avro support Diffs (updated) - ql/src/java/org/apache/hadoop/hive/ql/io/AvroStorageFormatDescriptor.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/io/IOConstants.java 1bae0a8fee04049f90b16d813ff4c96707b349c8 ql/src/main/resources/META-INF/services/org.apache.hadoop.hive.ql.io.StorageFormatDescriptor a23ff115512da5fe3167835a88d582c427585b8e ql/src/test/org/apache/hadoop/hive/ql/io/TestStorageFormatDescriptor.java d53ebc65174d66bfeee25fd2891c69c78f9137ee ql/src/test/queries/clientpositive/avro_compression_enabled_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_decimal_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_joins_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_partitioned_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_schema_evolution_native.q PRE-CREATION ql/src/test/results/clientpositive/avro_compression_enabled_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_decimal_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_joins_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_partitioned_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_schema_evolution_native.q.out PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerDe.java 1fe31e0034f8988d03a0c51a90904bb93e7cb157 serde/src/java/org/apache/hadoop/hive/serde2/avro/TypeInfoToSchema.java PRE-CREATION serde/src/test/org/apache/hadoop/hive/serde2/avro/TestTypeInfoToSchema.java PRE-CREATION Diff: https://reviews.apache.org/r/23387/diff/ Testing --- Added qTests and unit tests Thanks, Ashish Singh
Re: Review Request 23387: HIVE-6806: Native avro support
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23387/ --- (Updated July 16, 2014, 3:06 a.m.) Review request for hive. Changes --- Add qtest for avro schema evolution. Bugs: HIVE-6806 https://issues.apache.org/jira/browse/HIVE-6806 Repository: hive-git Description (updated) --- HIVE-6806: Native avro support Diffs (updated) - ql/src/java/org/apache/hadoop/hive/ql/io/AvroStorageFormatDescriptor.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/io/IOConstants.java 1bae0a8fee04049f90b16d813ff4c96707b349c8 ql/src/main/resources/META-INF/services/org.apache.hadoop.hive.ql.io.StorageFormatDescriptor a23ff115512da5fe3167835a88d582c427585b8e ql/src/test/org/apache/hadoop/hive/ql/io/TestStorageFormatDescriptor.java d53ebc65174d66bfeee25fd2891c69c78f9137ee ql/src/test/queries/clientpositive/avro_compression_enabled_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_decimal_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_joins_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_partitioned_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_schema_evolution_native.q PRE-CREATION ql/src/test/results/clientpositive/avro_compression_enabled_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_decimal_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_joins_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_partitioned_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_schema_evolution_native.q.out PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerDe.java 1fe31e0034f8988d03a0c51a90904bb93e7cb157 serde/src/java/org/apache/hadoop/hive/serde2/avro/TypeInfoToSchema.java PRE-CREATION serde/src/test/org/apache/hadoop/hive/serde2/avro/TestTypeInfoToSchema.java PRE-CREATION Diff: https://reviews.apache.org/r/23387/diff/ Testing --- Added qTests and unit tests Thanks, Ashish Singh
Re: Review Request 23253: HIVE-7340: Beeline fails to read a query with comments correctly
> On July 9, 2014, 12:39 a.m., Deepesh Khandelwal wrote: > > According to the sqlline doc on which beeline is based, it only mentions > > "Lines beginning with "#" are interpreted as comments and ignored". > > Interpreting inline "#" as comments will restrict us from writing queries > > which have "#" appearing in the query body. > > Ashish Singh wrote: > Deepesh, I agree with you on '#', but we should still let '--' identify > inline comments. SQL92 also supports inline comments with '--'. Let me know > if you think otherwise. > > Deepesh Khandelwal wrote: > Yes, my concern was only for the inline '#', I am fine with supporting > the following comment variants: > - Inline '--' > - Lines beginning with '--' > - Lines beginning with '#' Addressed. - Ashish --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23253/#review47481 --- On July 14, 2014, 10:29 p.m., Ashish Singh wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/23253/ > --- > > (Updated July 14, 2014, 10:29 p.m.) > > > Review request for hive. > > > Bugs: HIVE-7340 > https://issues.apache.org/jira/browse/HIVE-7340 > > > Repository: hive-git > > > Description > --- > > HIVE-7340: Beeline fails to read a query with comments correctly > > > Diffs > - > > beeline/src/java/org/apache/hive/beeline/Commands.java > 88a94d76a3750dcde31ff47913bf28b827b3b212 > > itests/hive-unit/src/test/java/org/apache/hive/beeline/TestBeeLineWithArgs.java > 140c1bccedb9ef3c81e89026db44ce4b59150ef4 > > Diff: https://reviews.apache.org/r/23253/diff/ > > > Testing > --- > > Added unit tests. > > > Thanks, > > Ashish Singh > >
Re: Review Request 23253: HIVE-7340: Beeline fails to read a query with comments correctly
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23253/ --- (Updated July 14, 2014, 10:29 p.m.) Review request for hive. Changes --- Addressed review comments Bugs: HIVE-7340 https://issues.apache.org/jira/browse/HIVE-7340 Repository: hive-git Description --- HIVE-7340: Beeline fails to read a query with comments correctly Diffs (updated) - beeline/src/java/org/apache/hive/beeline/Commands.java 88a94d76a3750dcde31ff47913bf28b827b3b212 itests/hive-unit/src/test/java/org/apache/hive/beeline/TestBeeLineWithArgs.java 140c1bccedb9ef3c81e89026db44ce4b59150ef4 Diff: https://reviews.apache.org/r/23253/diff/ Testing --- Added unit tests. Thanks, Ashish Singh
Re: Review Request 23387: HIVE-6806: Native avro support
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23387/ --- (Updated July 14, 2014, 10:05 p.m.) Review request for hive. Changes --- Reverting to original summary. rbt post tool changes it to last commit message. Bugs: HIVE-6806 https://issues.apache.org/jira/browse/HIVE-6806 Repository: hive-git Description --- HIVE-6806: Native Avro support in Hive Diffs - ql/src/java/org/apache/hadoop/hive/ql/io/AvroStorageFormatDescriptor.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/io/IOConstants.java 1bae0a8fee04049f90b16d813ff4c96707b349c8 ql/src/main/resources/META-INF/services/org.apache.hadoop.hive.ql.io.StorageFormatDescriptor a23ff115512da5fe3167835a88d582c427585b8e ql/src/test/org/apache/hadoop/hive/ql/io/TestStorageFormatDescriptor.java d53ebc65174d66bfeee25fd2891c69c78f9137ee ql/src/test/queries/clientpositive/avro_compression_enabled_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_decimal_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_joins_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_partitioned_native.q PRE-CREATION ql/src/test/results/clientpositive/avro_compression_enabled_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_decimal_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_joins_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_partitioned_native.q.out PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerDe.java 1fe31e0034f8988d03a0c51a90904bb93e7cb157 serde/src/java/org/apache/hadoop/hive/serde2/avro/TypeInfoToSchema.java PRE-CREATION serde/src/test/org/apache/hadoop/hive/serde2/avro/TestTypeInfoToSchema.java PRE-CREATION Diff: https://reviews.apache.org/r/23387/diff/ Testing --- Added qTests and unit tests Thanks, Ashish Singh
Re: Review Request 23387: HIVE-6806: Native avro support
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23387/ --- (Updated July 14, 2014, 10:02 p.m.) Review request for hive. Changes --- Reverting the description to original description. rbt post tool changes it to last commit message. Bugs: HIVE-6806 https://issues.apache.org/jira/browse/HIVE-6806 Repository: hive-git Description (updated) --- HIVE-6806: Native Avro support in Hive Diffs - ql/src/java/org/apache/hadoop/hive/ql/io/AvroStorageFormatDescriptor.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/io/IOConstants.java 1bae0a8fee04049f90b16d813ff4c96707b349c8 ql/src/main/resources/META-INF/services/org.apache.hadoop.hive.ql.io.StorageFormatDescriptor a23ff115512da5fe3167835a88d582c427585b8e ql/src/test/org/apache/hadoop/hive/ql/io/TestStorageFormatDescriptor.java d53ebc65174d66bfeee25fd2891c69c78f9137ee ql/src/test/queries/clientpositive/avro_compression_enabled_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_decimal_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_joins_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_partitioned_native.q PRE-CREATION ql/src/test/results/clientpositive/avro_compression_enabled_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_decimal_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_joins_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_partitioned_native.q.out PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerDe.java 1fe31e0034f8988d03a0c51a90904bb93e7cb157 serde/src/java/org/apache/hadoop/hive/serde2/avro/TypeInfoToSchema.java PRE-CREATION serde/src/test/org/apache/hadoop/hive/serde2/avro/TestTypeInfoToSchema.java PRE-CREATION Diff: https://reviews.apache.org/r/23387/diff/ Testing --- Added qTests and unit tests Thanks, Ashish Singh
Re: Review Request 23387: HIVE-6806: Native avro support
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23387/ --- (Updated July 14, 2014, 9:57 p.m.) Review request for hive. Changes --- Rebased Summary (updated) - HIVE-6806: Native avro support Bugs: HIVE-6806 https://issues.apache.org/jira/browse/HIVE-6806 Repository: hive-git Description (updated) --- HIVE-6806: Native avro support Diffs (updated) - ql/src/java/org/apache/hadoop/hive/ql/io/AvroStorageFormatDescriptor.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/io/IOConstants.java 1bae0a8fee04049f90b16d813ff4c96707b349c8 ql/src/main/resources/META-INF/services/org.apache.hadoop.hive.ql.io.StorageFormatDescriptor a23ff115512da5fe3167835a88d582c427585b8e ql/src/test/org/apache/hadoop/hive/ql/io/TestStorageFormatDescriptor.java d53ebc65174d66bfeee25fd2891c69c78f9137ee ql/src/test/queries/clientpositive/avro_compression_enabled_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_decimal_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_joins_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_partitioned_native.q PRE-CREATION ql/src/test/results/clientpositive/avro_compression_enabled_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_decimal_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_joins_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_partitioned_native.q.out PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerDe.java 1fe31e0034f8988d03a0c51a90904bb93e7cb157 serde/src/java/org/apache/hadoop/hive/serde2/avro/TypeInfoToSchema.java PRE-CREATION serde/src/test/org/apache/hadoop/hive/serde2/avro/TestTypeInfoToSchema.java PRE-CREATION Diff: https://reviews.apache.org/r/23387/diff/ Testing --- Added qTests and unit tests Thanks, Ashish Singh
Re: Review Request 23253: HIVE-7340: Beeline fails to read a query with comments correctly
> On July 9, 2014, 12:39 a.m., Deepesh Khandelwal wrote: > > According to the sqlline doc on which beeline is based, it only mentions > > "Lines beginning with "#" are interpreted as comments and ignored". > > Interpreting inline "#" as comments will restrict us from writing queries > > which have "#" appearing in the query body. Deepesh, I agree with you on '#', but we should still let '--' identify inline comments. SQL92 also supports inline comments with '--'. Let me know if you think otherwise. - Ashish --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23253/#review47481 --- On July 4, 2014, 1 a.m., Ashish Singh wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/23253/ > --- > > (Updated July 4, 2014, 1 a.m.) > > > Review request for hive. > > > Bugs: HIVE-7340 > https://issues.apache.org/jira/browse/HIVE-7340 > > > Repository: hive-git > > > Description > --- > > HIVE-7340: Beeline fails to read a query with comments correctly > > > Diffs > - > > beeline/src/java/org/apache/hive/beeline/Commands.java > 88a94d76a3750dcde31ff47913bf28b827b3b212 > > itests/hive-unit/src/test/java/org/apache/hive/beeline/TestBeeLineWithArgs.java > 140c1bccedb9ef3c81e89026db44ce4b59150ef4 > > Diff: https://reviews.apache.org/r/23253/diff/ > > > Testing > --- > > Added unit tests. > > > Thanks, > > Ashish Singh > >
Re: Review Request 23256: HIVE-7345: Beeline changes its prompt to reflect successful database connection even after failing to connect
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23256/ --- (Updated July 14, 2014, 1:15 a.m.) Review request for hive. Changes --- Address review comment from JIRA. Bugs: HIVE-7345 https://issues.apache.org/jira/browse/HIVE-7345 Repository: hive-git Description --- HIVE-7345: Beeline changes its prompt to reflect successful database connection even after failing to connect Diffs (updated) - beeline/src/java/org/apache/hive/beeline/BeeLine.java 2f3350e79f6168b11c13c6b4f84128c9255e0383 beeline/src/java/org/apache/hive/beeline/DatabaseConnection.java 00b49afb72531a4c15d0239ba08b04faa229d262 Diff: https://reviews.apache.org/r/23256/diff/ Testing --- NA Thanks, Ashish Singh
Re: Review Request 23387: HIVE-6806: Native Avro support in Hive
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23387/ --- (Updated July 12, 2014, 2:18 a.m.) Review request for hive. Changes --- Add missing .q.out file and have no schema provided be dealt as it used to be before this patch Bugs: HIVE-6806 https://issues.apache.org/jira/browse/HIVE-6806 Repository: hive-git Description --- HIVE-6806: Native Avro support in Hive Diffs (updated) - ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 75394f3bc4f2285a7ced97ea90788d5bbff6b563 ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 6cd1f39df3d419651755c35aab7cfc06833b16a4 ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 412a046488eaea42a6416c7cbd514715d37e249f ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g f934ac4e3b736eed1b3060fa516124c67f9a2f87 ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g 9c001c1495b423c19f3fa710c74f1bb1e24a08f4 ql/src/test/queries/clientpositive/avro_compression_enabled_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_decimal_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_joins_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_partitioned_native.q PRE-CREATION ql/src/test/results/clientpositive/avro_compression_enabled_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_decimal_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_joins_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_partitioned_native.q.out PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerDe.java 1fe31e0034f8988d03a0c51a90904bb93e7cb157 serde/src/java/org/apache/hadoop/hive/serde2/avro/TypeInfoToSchema.java PRE-CREATION serde/src/test/org/apache/hadoop/hive/serde2/avro/TestTypeInfoToSchema.java PRE-CREATION Diff: https://reviews.apache.org/r/23387/diff/ Testing --- Added qTests and unit tests Thanks, Ashish Singh
Re: Review Request 23387: HIVE-6806: Native Avro support in Hive
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23387/ --- (Updated July 11, 2014, 9:16 p.m.) Review request for hive. Changes --- RBT tools seems to mess up the summary. Restoring it back to original summary. Summary (updated) - HIVE-6806: Native Avro support in Hive Bugs: HIVE-6806 https://issues.apache.org/jira/browse/HIVE-6806 Repository: hive-git Description --- HIVE-6806: Native Avro support in Hive Diffs - ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 75394f3bc4f2285a7ced97ea90788d5bbff6b563 ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 6cd1f39df3d419651755c35aab7cfc06833b16a4 ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 412a046488eaea42a6416c7cbd514715d37e249f ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g f934ac4e3b736eed1b3060fa516124c67f9a2f87 ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g 9c001c1495b423c19f3fa710c74f1bb1e24a08f4 ql/src/test/queries/clientpositive/avro_compression_enabled_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_decimal_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_joins_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_partitioned_native.q PRE-CREATION ql/src/test/results/clientpositive/avro_compression_enabled_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_decimal_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_joins_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_partitioned_native.q.out PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerDe.java 1fe31e0034f8988d03a0c51a90904bb93e7cb157 serde/src/java/org/apache/hadoop/hive/serde2/avro/TypeInfoToSchema.java PRE-CREATION serde/src/test/org/apache/hadoop/hive/serde2/avro/TestTypeInfoToSchema.java PRE-CREATION Diff: https://reviews.apache.org/r/23387/diff/ Testing --- Added qTests and unit tests Thanks, Ashish Singh
Re: Review Request 23387: Addressed review comments
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23387/ --- (Updated July 11, 2014, 9:07 p.m.) Review request for hive. Changes --- Addressed review comments. Summary (updated) - Addressed review comments Bugs: HIVE-6806 https://issues.apache.org/jira/browse/HIVE-6806 Repository: hive-git Description --- HIVE-6806: Native Avro support in Hive Diffs (updated) - ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 75394f3bc4f2285a7ced97ea90788d5bbff6b563 ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 6cd1f39df3d419651755c35aab7cfc06833b16a4 ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 412a046488eaea42a6416c7cbd514715d37e249f ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g f934ac4e3b736eed1b3060fa516124c67f9a2f87 ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g 9c001c1495b423c19f3fa710c74f1bb1e24a08f4 ql/src/test/queries/clientpositive/avro_compression_enabled_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_decimal_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_joins_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_partitioned_native.q PRE-CREATION ql/src/test/results/clientpositive/avro_compression_enabled_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_decimal_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_joins_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_partitioned_native.q.out PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerDe.java 1fe31e0034f8988d03a0c51a90904bb93e7cb157 serde/src/java/org/apache/hadoop/hive/serde2/avro/TypeInfoToSchema.java PRE-CREATION serde/src/test/org/apache/hadoop/hive/serde2/avro/TestTypeInfoToSchema.java PRE-CREATION Diff: https://reviews.apache.org/r/23387/diff/ Testing (updated) --- Added qTests and unit tests Thanks, Ashish Singh
Re: Review Request 23387: HIVE-6806: Native Avro support in Hive
> On July 10, 2014, 8:13 p.m., David Chen wrote: > > serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSchemaGenerator.java, > > line 34 > > <https://reviews.apache.org/r/23387/diff/1/?file=627564#file627564line34> > > > > It might be better to call this TypeInfoToSchema to make it consistent > > with SchemaToTypeInfo, which converts from Avro Schema to Hive TypeInfo. > > Ashish Singh wrote: > That sounds reasonable, but the functionality of this class is not just > to convert typeinfo to schema. It also uses column names. > > From functionality point of view, AvroSchemaGenerator still sounds better > than TypeInfoToSchema. TypeInfoToSchema would have been suitable had the > class been converting one typeInfo to avro schema. Let me know if you > disagree. Just realized that SchemaToTypeInfo also generates all typeinfos from schema. That makes your comment even more reasonable. Will do. - Ashish --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23387/#review47609 ------- On July 10, 2014, 4:50 a.m., Ashish Singh wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/23387/ > --- > > (Updated July 10, 2014, 4:50 a.m.) > > > Review request for hive. > > > Bugs: HIVE-6806 > https://issues.apache.org/jira/browse/HIVE-6806 > > > Repository: hive-git > > > Description > --- > > HIVE-6806: Native Avro support in Hive > > > Diffs > - > > ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java > 75394f3bc4f2285a7ced97ea90788d5bbff6b563 > ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java > 6cd1f39df3d419651755c35aab7cfc06833b16a4 > ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g > 412a046488eaea42a6416c7cbd514715d37e249f > ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g > f934ac4e3b736eed1b3060fa516124c67f9a2f87 > ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g > 9c001c1495b423c19f3fa710c74f1bb1e24a08f4 > ql/src/test/queries/clientpositive/avro_compression_enabled_native.q > PRE-CREATION > ql/src/test/queries/clientpositive/avro_decimal_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_joins_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_partitioned_native.q PRE-CREATION > ql/src/test/results/clientpositive/avro_compression_enabled_native.q.out > PRE-CREATION > ql/src/test/results/clientpositive/avro_decimal_native.q.out PRE-CREATION > ql/src/test/results/clientpositive/avro_joins_native.q.out PRE-CREATION > ql/src/test/results/clientpositive/avro_partitioned_native.q.out > PRE-CREATION > serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSchemaGenerator.java > PRE-CREATION > serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerDe.java > 1fe31e0034f8988d03a0c51a90904bb93e7cb157 > > Diff: https://reviews.apache.org/r/23387/diff/ > > > Testing > --- > > Added qTests > > > Thanks, > > Ashish Singh > >
Re: Review Request 23387: HIVE-6806: Native Avro support in Hive
> On July 10, 2014, 8:13 p.m., David Chen wrote: > > serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSchemaGenerator.java, > > line 34 > > <https://reviews.apache.org/r/23387/diff/1/?file=627564#file627564line34> > > > > It might be better to call this TypeInfoToSchema to make it consistent > > with SchemaToTypeInfo, which converts from Avro Schema to Hive TypeInfo. That sounds reasonable, but the functionality of this class is not just to convert typeinfo to schema. It also uses column names. >From functionality point of view, AvroSchemaGenerator still sounds better than >TypeInfoToSchema. TypeInfoToSchema would have been suitable had the class been >converting one typeInfo to avro schema. Let me know if you disagree. - Ashish --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23387/#review47609 ------- On July 10, 2014, 4:50 a.m., Ashish Singh wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/23387/ > --- > > (Updated July 10, 2014, 4:50 a.m.) > > > Review request for hive. > > > Bugs: HIVE-6806 > https://issues.apache.org/jira/browse/HIVE-6806 > > > Repository: hive-git > > > Description > --- > > HIVE-6806: Native Avro support in Hive > > > Diffs > - > > ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java > 75394f3bc4f2285a7ced97ea90788d5bbff6b563 > ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java > 6cd1f39df3d419651755c35aab7cfc06833b16a4 > ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g > 412a046488eaea42a6416c7cbd514715d37e249f > ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g > f934ac4e3b736eed1b3060fa516124c67f9a2f87 > ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g > 9c001c1495b423c19f3fa710c74f1bb1e24a08f4 > ql/src/test/queries/clientpositive/avro_compression_enabled_native.q > PRE-CREATION > ql/src/test/queries/clientpositive/avro_decimal_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_joins_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_partitioned_native.q PRE-CREATION > ql/src/test/results/clientpositive/avro_compression_enabled_native.q.out > PRE-CREATION > ql/src/test/results/clientpositive/avro_decimal_native.q.out PRE-CREATION > ql/src/test/results/clientpositive/avro_joins_native.q.out PRE-CREATION > ql/src/test/results/clientpositive/avro_partitioned_native.q.out > PRE-CREATION > serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSchemaGenerator.java > PRE-CREATION > serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerDe.java > 1fe31e0034f8988d03a0c51a90904bb93e7cb157 > > Diff: https://reviews.apache.org/r/23387/diff/ > > > Testing > --- > > Added qTests > > > Thanks, > > Ashish Singh > >
Re: Review Request 23387: HIVE-6806: Native Avro support in Hive
> On July 10, 2014, 8:28 p.m., Brock Noland wrote: > > ql/src/test/queries/clientpositive/avro_decimal_native.q, line 13 > > <https://reviews.apache.org/r/23387/diff/1/?file=627556#file627556line13> > > > > I think we can remove COMMENT 'from deserializer' as well. > > > > Thx!! Yes, it is redundant and we will be better without it. Will do. - Ashish --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23387/#review47612 ------- On July 10, 2014, 4:50 a.m., Ashish Singh wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/23387/ > --- > > (Updated July 10, 2014, 4:50 a.m.) > > > Review request for hive. > > > Bugs: HIVE-6806 > https://issues.apache.org/jira/browse/HIVE-6806 > > > Repository: hive-git > > > Description > --- > > HIVE-6806: Native Avro support in Hive > > > Diffs > - > > ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java > 75394f3bc4f2285a7ced97ea90788d5bbff6b563 > ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java > 6cd1f39df3d419651755c35aab7cfc06833b16a4 > ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g > 412a046488eaea42a6416c7cbd514715d37e249f > ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g > f934ac4e3b736eed1b3060fa516124c67f9a2f87 > ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g > 9c001c1495b423c19f3fa710c74f1bb1e24a08f4 > ql/src/test/queries/clientpositive/avro_compression_enabled_native.q > PRE-CREATION > ql/src/test/queries/clientpositive/avro_decimal_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_joins_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_partitioned_native.q PRE-CREATION > ql/src/test/results/clientpositive/avro_compression_enabled_native.q.out > PRE-CREATION > ql/src/test/results/clientpositive/avro_decimal_native.q.out PRE-CREATION > ql/src/test/results/clientpositive/avro_joins_native.q.out PRE-CREATION > ql/src/test/results/clientpositive/avro_partitioned_native.q.out > PRE-CREATION > serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSchemaGenerator.java > PRE-CREATION > serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerDe.java > 1fe31e0034f8988d03a0c51a90904bb93e7cb157 > > Diff: https://reviews.apache.org/r/23387/diff/ > > > Testing > --- > > Added qTests > > > Thanks, > > Ashish Singh > >
Re: Review Request 23387: HIVE-6806: Native Avro support in Hive
> On July 10, 2014, 3:01 p.m., Brock Noland wrote: > > I *love* this patch! Thank you so much. Thanks! > On July 10, 2014, 3:01 p.m., Brock Noland wrote: > > serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSchemaGenerator.java, > > line 34 > > <https://reviews.apache.org/r/23387/diff/1/?file=627564#file627564line34> > > > > Can we add some unit tests for this class? Will do. > On July 10, 2014, 3:01 p.m., Brock Noland wrote: > > serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSchemaGenerator.java, > > line 110 > > <https://reviews.apache.org/r/23387/diff/1/?file=627564#file627564line110> > > > > Two thoughts: > > > > 1) Char/varchar support? > > 2) By defaulting to null won't any new types end up with null if this > > code is not updated? I think instead we should throw an exception for > > unknown types. 1. Char/varchar is not yet supported by avro. It is planned though. 2. Good point. Will do. - Ashish --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23387/#review47568 --- On July 10, 2014, 4:50 a.m., Ashish Singh wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/23387/ > --- > > (Updated July 10, 2014, 4:50 a.m.) > > > Review request for hive. > > > Bugs: HIVE-6806 > https://issues.apache.org/jira/browse/HIVE-6806 > > > Repository: hive-git > > > Description > --- > > HIVE-6806: Native Avro support in Hive > > > Diffs > - > > ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java > 75394f3bc4f2285a7ced97ea90788d5bbff6b563 > ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java > 6cd1f39df3d419651755c35aab7cfc06833b16a4 > ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g > 412a046488eaea42a6416c7cbd514715d37e249f > ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g > f934ac4e3b736eed1b3060fa516124c67f9a2f87 > ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g > 9c001c1495b423c19f3fa710c74f1bb1e24a08f4 > ql/src/test/queries/clientpositive/avro_compression_enabled_native.q > PRE-CREATION > ql/src/test/queries/clientpositive/avro_decimal_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_joins_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_native.q PRE-CREATION > ql/src/test/queries/clientpositive/avro_partitioned_native.q PRE-CREATION > ql/src/test/results/clientpositive/avro_compression_enabled_native.q.out > PRE-CREATION > ql/src/test/results/clientpositive/avro_decimal_native.q.out PRE-CREATION > ql/src/test/results/clientpositive/avro_joins_native.q.out PRE-CREATION > ql/src/test/results/clientpositive/avro_partitioned_native.q.out > PRE-CREATION > serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSchemaGenerator.java > PRE-CREATION > serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerDe.java > 1fe31e0034f8988d03a0c51a90904bb93e7cb157 > > Diff: https://reviews.apache.org/r/23387/diff/ > > > Testing > --- > > Added qTests > > > Thanks, > > Ashish Singh > >
Re: Review Request 23387: HIVE-6806: Native Avro support in Hive
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23387/ --- (Updated July 10, 2014, 4:50 a.m.) Review request for hive. Changes --- Link hive jira. Bugs: HIVE-6806 https://issues.apache.org/jira/browse/HIVE-6806 Repository: hive-git Description --- HIVE-6806: Native Avro support in Hive Diffs - ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 75394f3bc4f2285a7ced97ea90788d5bbff6b563 ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 6cd1f39df3d419651755c35aab7cfc06833b16a4 ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 412a046488eaea42a6416c7cbd514715d37e249f ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g f934ac4e3b736eed1b3060fa516124c67f9a2f87 ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g 9c001c1495b423c19f3fa710c74f1bb1e24a08f4 ql/src/test/queries/clientpositive/avro_compression_enabled_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_decimal_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_joins_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_partitioned_native.q PRE-CREATION ql/src/test/results/clientpositive/avro_compression_enabled_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_decimal_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_joins_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_partitioned_native.q.out PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSchemaGenerator.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerDe.java 1fe31e0034f8988d03a0c51a90904bb93e7cb157 Diff: https://reviews.apache.org/r/23387/diff/ Testing --- Added qTests Thanks, Ashish Singh
Review Request 23387: HIVE-6806: Native Avro support in Hive
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23387/ --- Review request for hive. Repository: hive-git Description --- HIVE-6806: Native Avro support in Hive Diffs - ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 75394f3bc4f2285a7ced97ea90788d5bbff6b563 ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 6cd1f39df3d419651755c35aab7cfc06833b16a4 ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 412a046488eaea42a6416c7cbd514715d37e249f ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g f934ac4e3b736eed1b3060fa516124c67f9a2f87 ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g 9c001c1495b423c19f3fa710c74f1bb1e24a08f4 ql/src/test/queries/clientpositive/avro_compression_enabled_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_decimal_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_joins_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_native.q PRE-CREATION ql/src/test/queries/clientpositive/avro_partitioned_native.q PRE-CREATION ql/src/test/results/clientpositive/avro_compression_enabled_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_decimal_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_joins_native.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_partitioned_native.q.out PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSchemaGenerator.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerDe.java 1fe31e0034f8988d03a0c51a90904bb93e7cb157 Diff: https://reviews.apache.org/r/23387/diff/ Testing --- Added qTests Thanks, Ashish Singh
Re: Review Request 23253: HIVE-7340: Beeline fails to read a query with comments correctly
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23253/ --- (Updated July 4, 2014, 1 a.m.) Review request for hive. Changes --- Attaching bug id. Bugs: HIVE-7340 https://issues.apache.org/jira/browse/HIVE-7340 Repository: hive-git Description --- HIVE-7340: Beeline fails to read a query with comments correctly Diffs - beeline/src/java/org/apache/hive/beeline/Commands.java 88a94d76a3750dcde31ff47913bf28b827b3b212 itests/hive-unit/src/test/java/org/apache/hive/beeline/TestBeeLineWithArgs.java 140c1bccedb9ef3c81e89026db44ce4b59150ef4 Diff: https://reviews.apache.org/r/23253/diff/ Testing --- Added unit tests. Thanks, Ashish Singh
Review Request 23256: HIVE-7345: Beeline changes its prompt to reflect successful database connection even after failing to connect
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23256/ --- Review request for hive. Bugs: HIVE-7345 https://issues.apache.org/jira/browse/HIVE-7345 Repository: hive-git Description --- HIVE-7345: Beeline changes its prompt to reflect successful database connection even after failing to connect Diffs - beeline/src/java/org/apache/hive/beeline/BeeLine.java 2f3350e79f6168b11c13c6b4f84128c9255e0383 beeline/src/java/org/apache/hive/beeline/DatabaseConnection.java 00b49afb72531a4c15d0239ba08b04faa229d262 Diff: https://reviews.apache.org/r/23256/diff/ Testing --- NA Thanks, Ashish Singh
Review Request 23253: HIVE-7340: Beeline fails to read a query with comments correctly
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23253/ --- Review request for hive. Repository: hive-git Description --- HIVE-7340: Beeline fails to read a query with comments correctly Diffs - beeline/src/java/org/apache/hive/beeline/Commands.java 88a94d76a3750dcde31ff47913bf28b827b3b212 itests/hive-unit/src/test/java/org/apache/hive/beeline/TestBeeLineWithArgs.java 140c1bccedb9ef3c81e89026db44ce4b59150ef4 Diff: https://reviews.apache.org/r/23253/diff/ Testing --- Added unit tests. Thanks, Ashish Singh
Re: Review Request 22772: HIVE-6637: UDF in_file() doesn't take CHAR or VARCHAR as input
> On June 27, 2014, 5:33 p.m., Xuefu Zhang wrote: > > data/files/in_file.dat, line 1 > > <https://reviews.apache.org/r/22772/diff/6/?file=618531#file618531line1> > > > > It's fine. I just wanted to point out that reusing existing file/table > > is preferred. We don't want to create an one-line for each test we added. Agreed Xuefu. - Ashish --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/22772/#review46879 ------- On June 26, 2014, 5:12 p.m., Ashish Singh wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/22772/ > --- > > (Updated June 26, 2014, 5:12 p.m.) > > > Review request for hive. > > > Bugs: HIVE-6637 > https://issues.apache.org/jira/browse/HIVE-6637 > > > Repository: hive-git > > > Description > --- > > HIVE-6637: UDF in_file() doesn't take CHAR or VARCHAR as input > > > Diffs > - > > data/files/in_file.dat PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFInFile.java > ea52537d0b85191f0b633a29aa3f7ddb556c288d > ql/src/test/queries/clientpositive/udf_in_file.q > 9d9efe8e23d6e73429ee5cd2c8470359ba2b3498 > ql/src/test/results/clientpositive/udf_in_file.q.out > b63143760d80f3f6a8ba0a23c0d87e8bb86fce66 > > Diff: https://reviews.apache.org/r/22772/diff/ > > > Testing > --- > > Tested with qtest. > > > Thanks, > > Ashish Singh > >
Re: Review Request 22772: HIVE-6637: UDF in_file() doesn't take CHAR or VARCHAR as input
> On June 26, 2014, 6:10 p.m., Xuefu Zhang wrote: > > data/files/in_file.dat, line 1 > > <https://reviews.apache.org/r/22772/diff/6/?file=618531#file618531line1> > > > > Could we reuse existing file instead of creating new ones? I could not find an existing data file with the data I need. test2.dat has the data but then to use it I will have to have complex DDL and load statements leading to multiple MR stages. Let me know if I am missing something here. - Ashish --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/22772/#review46768 --- On June 26, 2014, 5:12 p.m., Ashish Singh wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/22772/ > --- > > (Updated June 26, 2014, 5:12 p.m.) > > > Review request for hive. > > > Bugs: HIVE-6637 > https://issues.apache.org/jira/browse/HIVE-6637 > > > Repository: hive-git > > > Description > --- > > HIVE-6637: UDF in_file() doesn't take CHAR or VARCHAR as input > > > Diffs > - > > data/files/in_file.dat PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFInFile.java > ea52537d0b85191f0b633a29aa3f7ddb556c288d > ql/src/test/queries/clientpositive/udf_in_file.q > 9d9efe8e23d6e73429ee5cd2c8470359ba2b3498 > ql/src/test/results/clientpositive/udf_in_file.q.out > b63143760d80f3f6a8ba0a23c0d87e8bb86fce66 > > Diff: https://reviews.apache.org/r/22772/diff/ > > > Testing > --- > > Tested with qtest. > > > Thanks, > > Ashish Singh > >
Re: Review Request 22772: HIVE-6637: UDF in_file() doesn't take CHAR or VARCHAR as input
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/22772/ --- (Updated June 26, 2014, 5:12 p.m.) Review request for hive. Changes --- Add missing data file. Bugs: HIVE-6637 https://issues.apache.org/jira/browse/HIVE-6637 Repository: hive-git Description --- HIVE-6637: UDF in_file() doesn't take CHAR or VARCHAR as input Diffs (updated) - data/files/in_file.dat PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFInFile.java ea52537d0b85191f0b633a29aa3f7ddb556c288d ql/src/test/queries/clientpositive/udf_in_file.q 9d9efe8e23d6e73429ee5cd2c8470359ba2b3498 ql/src/test/results/clientpositive/udf_in_file.q.out b63143760d80f3f6a8ba0a23c0d87e8bb86fce66 Diff: https://reviews.apache.org/r/22772/diff/ Testing --- Tested with qtest. Thanks, Ashish Singh
Re: [ANNOUNCE] New Hive Committers - Gopal Vijayaraghavan and Szehon Ho
Congrats Szehon and Gopal! On Sun, Jun 22, 2014 at 9:04 PM, Rajesh Balamohan < rajesh.balamo...@gmail.com> wrote: > Congratulations Gopal and Szehon > > > > On Mon, Jun 23, 2014 at 9:12 AM, Carl Steinbach wrote: > > > The Apache Hive PMC has voted to make Gopal Vijayaraghavan and Szehon Ho > > committers on the Apache Hive Project. > > > > Please join me in congratulating Gopal and Szehon! > > > > Thanks. > > > > - Carl > > > > > > -- > ~Rajesh.B > -- Regards, Ashish
Re: Review Request 22772: HIVE-6637: UDF in_file() doesn't take CHAR or VARCHAR as input
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/22772/ --- (Updated June 21, 2014, 1:27 a.m.) Review request for hive. Changes --- Remove redundant text from desc. Bugs: HIVE-6637 https://issues.apache.org/jira/browse/HIVE-6637 Repository: hive-git Description (updated) --- HIVE-6637: UDF in_file() doesn't take CHAR or VARCHAR as input Diffs - ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFInFile.java ea52537d0b85191f0b633a29aa3f7ddb556c288d ql/src/test/queries/clientpositive/udf_in_file.q 9d9efe8e23d6e73429ee5cd2c8470359ba2b3498 ql/src/test/results/clientpositive/udf_in_file.q.out b63143760d80f3f6a8ba0a23c0d87e8bb86fce66 Diff: https://reviews.apache.org/r/22772/diff/ Testing --- Tested with qtest. Thanks, Ashish Singh
Re: Review Request 22772: HIVE-6637: UDF in_file() doesn't take CHAR or VARCHAR as input
> On June 20, 2014, 11:52 p.m., Jason Dere wrote: > > ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFInFile.java, > > line 84 > > <https://reviews.apache.org/r/22772/diff/4/?file=614380#file614380line84> > > > > Same thing here about using getPrimitiveCategory() rather than > > class/class name. Sorry I missed this on the previous review. That was my bad. Thanks for catching it. - Ashish --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/22772/#review46347 ------- On June 21, 2014, 1:26 a.m., Ashish Singh wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/22772/ > --- > > (Updated June 21, 2014, 1:26 a.m.) > > > Review request for hive. > > > Bugs: HIVE-6637 > https://issues.apache.org/jira/browse/HIVE-6637 > > > Repository: hive-git > > > Description > --- > > HIVE-6637: UDF in_file() doesn't take CHAR or VARCHAR as input > > > HIVE-6637: UDF in_file() doesn't take CHAR or VARCHAR as input > > > HIVE-6637: UDF in_file() doesn't take CHAR or VARCHAR as input > > > HIVE-6637: UDF in_file() doesn't take CHAR or VARCHAR as input > > > HIVE-6637: UDF in_file() doesn't take CHAR or VARCHAR as input > > > Add unit tests for Udf in_file > > > Diffs > - > > ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFInFile.java > ea52537d0b85191f0b633a29aa3f7ddb556c288d > ql/src/test/queries/clientpositive/udf_in_file.q > 9d9efe8e23d6e73429ee5cd2c8470359ba2b3498 > ql/src/test/results/clientpositive/udf_in_file.q.out > b63143760d80f3f6a8ba0a23c0d87e8bb86fce66 > > Diff: https://reviews.apache.org/r/22772/diff/ > > > Testing > --- > > Tested with qtest. > > > Thanks, > > Ashish Singh > >
