[jira] [Commented] (HIVE-16250) Update grammar in documentation
[ https://issues.apache.org/jira/browse/HIVE-16250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931582#comment-15931582 ] Lefty Leverenz commented on HIVE-16250: --- Done. Thanks for the edits, [~deborahjhunt]. > Update grammar in documentation > --- > > Key: HIVE-16250 > URL: https://issues.apache.org/jira/browse/HIVE-16250 > Project: Hive > Issue Type: Improvement >Reporter: Deborah Hunt >Assignee: Lefty Leverenz > > Update text recommended: Hive is widely applied as a solution to numerous > distinct problem types in the domain of big data. Quite clearly it is often > used for the ad hoc querying of large datasets. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HIVE-16250) Update grammar in documentation
[ https://issues.apache.org/jira/browse/HIVE-16250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lefty Leverenz reassigned HIVE-16250: - Assignee: Lefty Leverenz (was: anybudy) > Update grammar in documentation > --- > > Key: HIVE-16250 > URL: https://issues.apache.org/jira/browse/HIVE-16250 > Project: Hive > Issue Type: Improvement >Reporter: Deborah Hunt >Assignee: Lefty Leverenz > > Update text recommended: Hive is widely applied as a solution to numerous > distinct problem types in the domain of big data. Quite clearly it is often > used for the ad hoc querying of large datasets. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16250) Update grammar in documentation
[ https://issues.apache.org/jira/browse/HIVE-16250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931579#comment-15931579 ] Lefty Leverenz commented on HIVE-16250: --- This update applies to the first sentences in Unit Testing Hive SQL: "Hive is widely applied as solution to numerous distinct problem types in the domain of big data. Quite clearly it often used for the ad hoc querying of large datasets." * [Unit Testing Hive SQL | https://cwiki.apache.org/confluence/display/Hive/Unit+Testing+Hive+SQL] +1 > Update grammar in documentation > --- > > Key: HIVE-16250 > URL: https://issues.apache.org/jira/browse/HIVE-16250 > Project: Hive > Issue Type: Improvement >Reporter: Deborah Hunt >Assignee: anybudy > > Update text recommended: Hive is widely applied as a solution to numerous > distinct problem types in the domain of big data. Quite clearly it is often > used for the ad hoc querying of large datasets. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16234) Add support for quarter in trunc udf
[
https://issues.apache.org/jira/browse/HIVE-16234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931576#comment-15931576
]
Lefty Leverenz commented on HIVE-16234:
---
[~deepesh], you can wait until 2.2.0 is released but there's a good chance the
doc will be forgotten by then, so people usually document things ahead of the
release.
If you search for "2.2.0" in the UDF doc, you'll find four examples.
> Add support for quarter in trunc udf
>
>
> Key: HIVE-16234
> URL: https://issues.apache.org/jira/browse/HIVE-16234
> Project: Hive
> Issue Type: Improvement
> Components: UDF
>Reporter: Deepesh Khandelwal
>Assignee: Deepesh Khandelwal
> Labels: TODOC2.2
> Fix For: 2.2.0
>
> Attachments: HIVE-16234.1.patch
>
>
> Hive has a Date function trunc(string date, string format) that returns date
> truncated to the unit specified by the format. Supported formats:
> MONTH/MON/MM, YEAR//YY.
> Goal here is to extend support to QUARTER/Q.
> Example:
> SELECT trunc('2017-03-15', 'Q');
> '2017-01-01'
> SELECT trunc('2017-12-31', 'Q');
> '2017-10-01'
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-12274) Increase width of columns used for general configuration in the metastore.
[
https://issues.apache.org/jira/browse/HIVE-12274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Naveen Gangam updated HIVE-12274:
-
Status: Patch Available (was: Open)
> Increase width of columns used for general configuration in the metastore.
> --
>
> Key: HIVE-12274
> URL: https://issues.apache.org/jira/browse/HIVE-12274
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Affects Versions: 2.0.0
>Reporter: Elliot West
>Assignee: Naveen Gangam
> Labels: metastore
> Attachments: HIVE-12274.2.patch, HIVE-12274.3.patch,
> HIVE-12274.4.patch, HIVE-12274.5.patch, HIVE-12274.example.ddl.hql,
> HIVE-12274.patch
>
>
> h2. Overview
> This issue is very similar in principle to HIVE-1364. We are hitting a limit
> when processing JSON data that has a large nested schema. The struct
> definition is truncated when inserted into the metastore database column
> {{COLUMNS_V2.YPE_NAME}} as it is greater than 4000 characters in length.
> Given that the purpose of these columns is to hold very loosely defined
> configuration values it seems rather limiting to impose such a relatively low
> length bound. One can imagine that valid use cases will arise where
> reasonable parameter/property values exceed the current limit.
> h2. Context
> These limitations were in by the [patch
> attributed|https://github.com/apache/hive/commit/c21a526b0a752df2a51d20a2729cc8493c228799]
> to HIVE-1364 which mentions the _"max length on Oracle 9i/10g/11g"_ as the
> reason. However, nowadays the limit can be increased because:
> * Oracle DB's {{varchar2}} supports 32767 bytes now, by setting the
> configuration parameter {{MAX_STRING_SIZE}} to {{EXTENDED}}.
> ([source|http://docs.oracle.com/database/121/SQLRF/sql_elements001.htm#SQLRF55623])
> * Postgres supports a max of 1GB for {{character}} datatype.
> ([source|http://www.postgresql.org/docs/8.3/static/datatype-character.html])
> * MySQL can support upto 65535 bytes for the entire row. So long as the
> {{PARAM_KEY}} value + {{PARAM_VALUE}} is less than 65535, we should be good.
> ([source|http://dev.mysql.com/doc/refman/5.0/en/char.html])
> * SQL Server's {{varchar}} max length is 8000 and can go beyond using
> "varchar(max)" with the same limitation as MySQL being 65535 bytes for the
> entire row. ([source|http://dev.mysql.com/doc/refman/5.0/en/char.html])
> * Derby's {{varchar}} can be upto 32672 bytes.
> ([source|https://db.apache.org/derby/docs/10.7/ref/rrefsqlj41207.html])
> h2. Proposal
> Can these columns not use CLOB-like types as for example as used by
> {{TBLS.VIEW_EXPANDED_TEXT}}? It would seem that suitable type equivalents
> exist for all targeted database platforms:
> * MySQL: {{mediumtext}}
> * Postgres: {{text}}
> * Oracle: {{CLOB}}
> * Derby: {{LONG VARCHAR}}
> I'd suggest that the candidates for type change are:
> * {{COLUMNS_V2.TYPE_NAME}}
> * {{TABLE_PARAMS.PARAM_VALUE}}
> * {{SERDE_PARAMS.PARAM_VALUE}}
> * {{SD_PARAMS.PARAM_VALUE}}
> After updating the maximum length the metastore database needs to be
> configured and restarted with the new settings. Altering {{MAX_STRING_SIZE}}
> will update database objects and possibly invalidate them, as follows:
> * Tables with virtual columns will be updated with new data type metadata for
> virtual columns of {{VARCHAR2(4000)}}, 4000-byte {{NVARCHAR2}}, or
> {{RAW(2000)}} type.
> * Functional indexes will become unusable if a change to their associated
> virtual columns causes the index key to exceed index key length limits.
> Attempts to rebuild such indexes will fail with {{ORA-01450: maximum key
> length exceeded}}.
> * Views will be invalidated if they contain {{VARCHAR2(4000)}}, 4000-byte
> {{NVARCHAR2}}, or {{RAW(2000)}} typed expression columns.
> * Materialized views will be updated with new metadata {{VARCHAR2(4000)}},
> 4000-byte {{NVARCHAR2}}, and {{RAW(2000)}} typed expression columns
> * So the limitation could be raised to 32672 bytes, with the caveat that
> MySQL and SQL Server limit the row length to 65535 bytes, so that should also
> be validated to provide consistency.
> Finally, will this limitation persist in the work resulting from HIVE-9452?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-12274) Increase width of columns used for general configuration in the metastore.
[
https://issues.apache.org/jira/browse/HIVE-12274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Naveen Gangam updated HIVE-12274:
-
Attachment: HIVE-12274.5.patch
Fixed the test failures.
> Increase width of columns used for general configuration in the metastore.
> --
>
> Key: HIVE-12274
> URL: https://issues.apache.org/jira/browse/HIVE-12274
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Affects Versions: 2.0.0
>Reporter: Elliot West
>Assignee: Naveen Gangam
> Labels: metastore
> Attachments: HIVE-12274.2.patch, HIVE-12274.3.patch,
> HIVE-12274.4.patch, HIVE-12274.5.patch, HIVE-12274.example.ddl.hql,
> HIVE-12274.patch
>
>
> h2. Overview
> This issue is very similar in principle to HIVE-1364. We are hitting a limit
> when processing JSON data that has a large nested schema. The struct
> definition is truncated when inserted into the metastore database column
> {{COLUMNS_V2.YPE_NAME}} as it is greater than 4000 characters in length.
> Given that the purpose of these columns is to hold very loosely defined
> configuration values it seems rather limiting to impose such a relatively low
> length bound. One can imagine that valid use cases will arise where
> reasonable parameter/property values exceed the current limit.
> h2. Context
> These limitations were in by the [patch
> attributed|https://github.com/apache/hive/commit/c21a526b0a752df2a51d20a2729cc8493c228799]
> to HIVE-1364 which mentions the _"max length on Oracle 9i/10g/11g"_ as the
> reason. However, nowadays the limit can be increased because:
> * Oracle DB's {{varchar2}} supports 32767 bytes now, by setting the
> configuration parameter {{MAX_STRING_SIZE}} to {{EXTENDED}}.
> ([source|http://docs.oracle.com/database/121/SQLRF/sql_elements001.htm#SQLRF55623])
> * Postgres supports a max of 1GB for {{character}} datatype.
> ([source|http://www.postgresql.org/docs/8.3/static/datatype-character.html])
> * MySQL can support upto 65535 bytes for the entire row. So long as the
> {{PARAM_KEY}} value + {{PARAM_VALUE}} is less than 65535, we should be good.
> ([source|http://dev.mysql.com/doc/refman/5.0/en/char.html])
> * SQL Server's {{varchar}} max length is 8000 and can go beyond using
> "varchar(max)" with the same limitation as MySQL being 65535 bytes for the
> entire row. ([source|http://dev.mysql.com/doc/refman/5.0/en/char.html])
> * Derby's {{varchar}} can be upto 32672 bytes.
> ([source|https://db.apache.org/derby/docs/10.7/ref/rrefsqlj41207.html])
> h2. Proposal
> Can these columns not use CLOB-like types as for example as used by
> {{TBLS.VIEW_EXPANDED_TEXT}}? It would seem that suitable type equivalents
> exist for all targeted database platforms:
> * MySQL: {{mediumtext}}
> * Postgres: {{text}}
> * Oracle: {{CLOB}}
> * Derby: {{LONG VARCHAR}}
> I'd suggest that the candidates for type change are:
> * {{COLUMNS_V2.TYPE_NAME}}
> * {{TABLE_PARAMS.PARAM_VALUE}}
> * {{SERDE_PARAMS.PARAM_VALUE}}
> * {{SD_PARAMS.PARAM_VALUE}}
> After updating the maximum length the metastore database needs to be
> configured and restarted with the new settings. Altering {{MAX_STRING_SIZE}}
> will update database objects and possibly invalidate them, as follows:
> * Tables with virtual columns will be updated with new data type metadata for
> virtual columns of {{VARCHAR2(4000)}}, 4000-byte {{NVARCHAR2}}, or
> {{RAW(2000)}} type.
> * Functional indexes will become unusable if a change to their associated
> virtual columns causes the index key to exceed index key length limits.
> Attempts to rebuild such indexes will fail with {{ORA-01450: maximum key
> length exceeded}}.
> * Views will be invalidated if they contain {{VARCHAR2(4000)}}, 4000-byte
> {{NVARCHAR2}}, or {{RAW(2000)}} typed expression columns.
> * Materialized views will be updated with new metadata {{VARCHAR2(4000)}},
> 4000-byte {{NVARCHAR2}}, and {{RAW(2000)}} typed expression columns
> * So the limitation could be raised to 32672 bytes, with the caveat that
> MySQL and SQL Server limit the row length to 65535 bytes, so that should also
> be validated to provide consistency.
> Finally, will this limitation persist in the work resulting from HIVE-9452?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-12274) Increase width of columns used for general configuration in the metastore.
[
https://issues.apache.org/jira/browse/HIVE-12274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Naveen Gangam updated HIVE-12274:
-
Status: Open (was: Patch Available)
> Increase width of columns used for general configuration in the metastore.
> --
>
> Key: HIVE-12274
> URL: https://issues.apache.org/jira/browse/HIVE-12274
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Affects Versions: 2.0.0
>Reporter: Elliot West
>Assignee: Naveen Gangam
> Labels: metastore
> Attachments: HIVE-12274.2.patch, HIVE-12274.3.patch,
> HIVE-12274.4.patch, HIVE-12274.example.ddl.hql, HIVE-12274.patch
>
>
> h2. Overview
> This issue is very similar in principle to HIVE-1364. We are hitting a limit
> when processing JSON data that has a large nested schema. The struct
> definition is truncated when inserted into the metastore database column
> {{COLUMNS_V2.YPE_NAME}} as it is greater than 4000 characters in length.
> Given that the purpose of these columns is to hold very loosely defined
> configuration values it seems rather limiting to impose such a relatively low
> length bound. One can imagine that valid use cases will arise where
> reasonable parameter/property values exceed the current limit.
> h2. Context
> These limitations were in by the [patch
> attributed|https://github.com/apache/hive/commit/c21a526b0a752df2a51d20a2729cc8493c228799]
> to HIVE-1364 which mentions the _"max length on Oracle 9i/10g/11g"_ as the
> reason. However, nowadays the limit can be increased because:
> * Oracle DB's {{varchar2}} supports 32767 bytes now, by setting the
> configuration parameter {{MAX_STRING_SIZE}} to {{EXTENDED}}.
> ([source|http://docs.oracle.com/database/121/SQLRF/sql_elements001.htm#SQLRF55623])
> * Postgres supports a max of 1GB for {{character}} datatype.
> ([source|http://www.postgresql.org/docs/8.3/static/datatype-character.html])
> * MySQL can support upto 65535 bytes for the entire row. So long as the
> {{PARAM_KEY}} value + {{PARAM_VALUE}} is less than 65535, we should be good.
> ([source|http://dev.mysql.com/doc/refman/5.0/en/char.html])
> * SQL Server's {{varchar}} max length is 8000 and can go beyond using
> "varchar(max)" with the same limitation as MySQL being 65535 bytes for the
> entire row. ([source|http://dev.mysql.com/doc/refman/5.0/en/char.html])
> * Derby's {{varchar}} can be upto 32672 bytes.
> ([source|https://db.apache.org/derby/docs/10.7/ref/rrefsqlj41207.html])
> h2. Proposal
> Can these columns not use CLOB-like types as for example as used by
> {{TBLS.VIEW_EXPANDED_TEXT}}? It would seem that suitable type equivalents
> exist for all targeted database platforms:
> * MySQL: {{mediumtext}}
> * Postgres: {{text}}
> * Oracle: {{CLOB}}
> * Derby: {{LONG VARCHAR}}
> I'd suggest that the candidates for type change are:
> * {{COLUMNS_V2.TYPE_NAME}}
> * {{TABLE_PARAMS.PARAM_VALUE}}
> * {{SERDE_PARAMS.PARAM_VALUE}}
> * {{SD_PARAMS.PARAM_VALUE}}
> After updating the maximum length the metastore database needs to be
> configured and restarted with the new settings. Altering {{MAX_STRING_SIZE}}
> will update database objects and possibly invalidate them, as follows:
> * Tables with virtual columns will be updated with new data type metadata for
> virtual columns of {{VARCHAR2(4000)}}, 4000-byte {{NVARCHAR2}}, or
> {{RAW(2000)}} type.
> * Functional indexes will become unusable if a change to their associated
> virtual columns causes the index key to exceed index key length limits.
> Attempts to rebuild such indexes will fail with {{ORA-01450: maximum key
> length exceeded}}.
> * Views will be invalidated if they contain {{VARCHAR2(4000)}}, 4000-byte
> {{NVARCHAR2}}, or {{RAW(2000)}} typed expression columns.
> * Materialized views will be updated with new metadata {{VARCHAR2(4000)}},
> 4000-byte {{NVARCHAR2}}, and {{RAW(2000)}} typed expression columns
> * So the limitation could be raised to 32672 bytes, with the caveat that
> MySQL and SQL Server limit the row length to 65535 bytes, so that should also
> be validated to provide consistency.
> Finally, will this limitation persist in the work resulting from HIVE-9452?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15126) Branch-1.2: Fix TestCliDriver.join_merge_multi_expressions.q
[ https://issues.apache.org/jira/browse/HIVE-15126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931565#comment-15931565 ] Daniel Dai commented on HIVE-15126: --- +1 > Branch-1.2: Fix TestCliDriver.join_merge_multi_expressions.q > > > Key: HIVE-15126 > URL: https://issues.apache.org/jira/browse/HIVE-15126 > Project: Hive > Issue Type: Sub-task > Components: Test >Affects Versions: 1.2.1 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Blocker > Attachments: HIVE-15082.1-branch-1.2.patch, HIVE-15126.1.patch > > > The .out file for join_merge_multi_expressions.q needs to be updated. Current > one does not show Select operator in the plan that explain returns. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16251) Vectorization: new octet_length function (HIVE-15979) get NPE
[
https://issues.apache.org/jira/browse/HIVE-16251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931555#comment-15931555
]
Matt McCline commented on HIVE-16251:
-
The tests didn't use ORC for input files and did not vectorized. I discovered
it because was turning on Text Vectorization by default in HIVE-15784. I'll
hold off my commit until this issue is resolved.
> Vectorization: new octet_length function (HIVE-15979) get NPE
> -
>
> Key: HIVE-16251
> URL: https://issues.apache.org/jira/browse/HIVE-16251
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Teddy Choi
>Priority: Critical
>
> Stack trace:
> {code}
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.OctetLength.evaluate(OctetLength.java:53)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:125)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.closeOp(VectorMapOperator.java:900)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:189)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
> ~[hadoop-mapreduce-client-core-2.7.2.jar:?]
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
> ~[hadoop-mapreduce-client-core-2.7.2.jar:?]
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> ~[hadoop-mapreduce-client-core-2.7.2.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
> ~[hadoop-mapreduce-client-common-2.7.2.jar:?]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[?:1.8.0_91]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[?:1.8.0_91]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[?:1.8.0_91]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> ~[?:1.8.0_91]
> at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_91]
> {code}
> I think it is the use of transient:
> {code}
> private transient int colNum;
> private transient int outputColumn;
> {code}
> Transient columns lose their value when serialized from plan to execution.
> On execution both are 0. Not sure why tests didn't fail.
> CC: [~ashutoshc]
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (HIVE-16251) Vectorization: new octet_length function (HIVE-15979) get NPE
[
https://issues.apache.org/jira/browse/HIVE-16251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline reassigned HIVE-16251:
---
> Vectorization: new octet_length function (HIVE-15979) get NPE
> -
>
> Key: HIVE-16251
> URL: https://issues.apache.org/jira/browse/HIVE-16251
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Teddy Choi
>Priority: Critical
>
> Stack trace:
> {code}
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.OctetLength.evaluate(OctetLength.java:53)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:125)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.closeOp(VectorMapOperator.java:900)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:189)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
> ~[hadoop-mapreduce-client-core-2.7.2.jar:?]
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
> ~[hadoop-mapreduce-client-core-2.7.2.jar:?]
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> ~[hadoop-mapreduce-client-core-2.7.2.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
> ~[hadoop-mapreduce-client-common-2.7.2.jar:?]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[?:1.8.0_91]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[?:1.8.0_91]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[?:1.8.0_91]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> ~[?:1.8.0_91]
> at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_91]
> {code}
> I think it is the use of transient:
> {code}
> private transient int colNum;
> private transient int outputColumn;
> {code}
> Transient columns lose their value when serialized from plan to execution.
> On execution both are 0. Not sure why tests didn't fail.
> CC: [~ashutoshc]
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16184) Remove use of optionals in union; declare TProgressUpdateResp before its usage
[ https://issues.apache.org/jira/browse/HIVE-16184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931512#comment-15931512 ] Ian Macalinao commented on HIVE-16184: -- I see no problem with these Thrift changes. We should not be using optionals in unions anyway. > Remove use of optionals in union; declare TProgressUpdateResp before its usage > -- > > Key: HIVE-16184 > URL: https://issues.apache.org/jira/browse/HIVE-16184 > Project: Hive > Issue Type: Bug > Components: Thrift API >Reporter: Ian Macalinao >Assignee: Ian Macalinao >Priority: Minor > Attachments: HIVE-16184.patch > > > A Scala Thrift generator (https://github.com/twitter/scrooge) does not like > the Thrift file. I have removed some bad Thrift practices (i.e. optionals in > union, dependent types declared after their usage) and replaced them with > things that work in Scrooge. > See https://github.com/apache/hive/pull/157 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15007) Hive 1.2.2 release planning
[ https://issues.apache.org/jira/browse/HIVE-15007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931370#comment-15931370 ] Vaibhav Gumashta commented on HIVE-15007: - Run on branch-1.1 failed during setup. I tried branch-1 as well. Failures on branch-1 run: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby11 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_drop_table_removes_partition_dirs org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_mapjoin org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_semijoin5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_table org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_string_decimal org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_temp_table_gb1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynpart_sort_optimization_acid org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_fast_stats org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynpart_sort_optimization org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_windowing org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby10 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_drop_database_removes_partition_dirs org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_index_bitmap_auto org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_groupby2 org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx_cbo_2 org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_right_side_join org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_stats_only_null org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynamic_partition_pruning org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_dynamic_partition_pruning org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_smb_empty org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_groupby2 org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_union_fast_stats org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_drop_partition org.apache.hive.minikdc.TestMiniHiveKdc.testLogin org.apache.hive.minikdc.TestHiveAuthFactory.testStartTokenManagerForMemoryTokenStore org.apache.hive.minikdc.TestHiveAuthFactory.testStartTokenManagerForDBTokenStore org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testSplitGenFailure > Hive 1.2.2 release planning > --- > > Key: HIVE-15007 > URL: https://issues.apache.org/jira/browse/HIVE-15007 > Project: Hive > Issue Type: Task >Affects Versions: 1.2.1 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > Attachments: HIVE-15007-branch-1.1.patch, > HIVE-15007-branch-1.2.patch, HIVE-15007-branch-1.2.patch, > HIVE-15007-branch-1.2.patch, HIVE-15007-branch-1.2.patch, > HIVE-15007-branch-1.2.patch, HIVE-15007-branch-1.2.patch, > HIVE-15007.branch-1.2.patch, HIVE-15007-branch-1.patch > > > Discussed with [~spena] about triggering unit test runs for 1.2.2 release and > creating a patch which will trigger precommits looks like a good way. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15007) Hive 1.2.2 release planning
[
https://issues.apache.org/jira/browse/HIVE-15007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931365#comment-15931365
]
Hive QA commented on HIVE-15007:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12859432/HIVE-15007-branch-1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 161 failed/errored test(s), 8083 tests
executed
*Failed tests:*
{noformat}
TestAcidOnTez - did not produce a TEST-*.xml file (likely timed out)
(batchId=376)
TestAdminUser - did not produce a TEST-*.xml file (likely timed out)
(batchId=358)
TestAuthorizationPreEventListener - did not produce a TEST-*.xml file (likely
timed out) (batchId=391)
TestAuthzApiEmbedAuthorizerInEmbed - did not produce a TEST-*.xml file (likely
timed out) (batchId=368)
TestAuthzApiEmbedAuthorizerInRemote - did not produce a TEST-*.xml file (likely
timed out) (batchId=374)
TestBeeLineWithArgs - did not produce a TEST-*.xml file (likely timed out)
(batchId=398)
TestCLIAuthzSessionContext - did not produce a TEST-*.xml file (likely timed
out) (batchId=416)
TestClearDanglingScratchDir - did not produce a TEST-*.xml file (likely timed
out) (batchId=383)
TestClientSideAuthorizationProvider - did not produce a TEST-*.xml file (likely
timed out) (batchId=390)
TestCompactor - did not produce a TEST-*.xml file (likely timed out)
(batchId=379)
TestCreateUdfEntities - did not produce a TEST-*.xml file (likely timed out)
(batchId=378)
TestCustomAuthentication - did not produce a TEST-*.xml file (likely timed out)
(batchId=399)
TestDBTokenStore - did not produce a TEST-*.xml file (likely timed out)
(batchId=342)
TestDDLWithRemoteMetastoreSecondNamenode - did not produce a TEST-*.xml file
(likely timed out) (batchId=377)
TestDynamicSerDe - did not produce a TEST-*.xml file (likely timed out)
(batchId=345)
TestEmbeddedHiveMetaStore - did not produce a TEST-*.xml file (likely timed
out) (batchId=355)
TestEmbeddedThriftBinaryCLIService - did not produce a TEST-*.xml file (likely
timed out) (batchId=402)
TestFilterHooks - did not produce a TEST-*.xml file (likely timed out)
(batchId=350)
TestFolderPermissions - did not produce a TEST-*.xml file (likely timed out)
(batchId=385)
TestHS2AuthzContext - did not produce a TEST-*.xml file (likely timed out)
(batchId=419)
TestHS2AuthzSessionContext - did not produce a TEST-*.xml file (likely timed
out) (batchId=420)
TestHS2ClearDanglingScratchDir - did not produce a TEST-*.xml file (likely
timed out) (batchId=406)
TestHS2ImpersonationWithRemoteMS - did not produce a TEST-*.xml file (likely
timed out) (batchId=407)
TestHiveAuthorizerCheckInvocation - did not produce a TEST-*.xml file (likely
timed out) (batchId=394)
TestHiveAuthorizerShowFilters - did not produce a TEST-*.xml file (likely timed
out) (batchId=393)
TestHiveHistory - did not produce a TEST-*.xml file (likely timed out)
(batchId=396)
TestHiveMetaStoreTxns - did not produce a TEST-*.xml file (likely timed out)
(batchId=370)
TestHiveMetaStoreWithEnvironmentContext - did not produce a TEST-*.xml file
(likely timed out) (batchId=360)
TestHiveMetaTool - did not produce a TEST-*.xml file (likely timed out)
(batchId=373)
TestHiveServer2 - did not produce a TEST-*.xml file (likely timed out)
(batchId=422)
TestHiveServer2SessionTimeout - did not produce a TEST-*.xml file (likely timed
out) (batchId=423)
TestHiveSessionImpl - did not produce a TEST-*.xml file (likely timed out)
(batchId=403)
TestHs2Hooks - did not produce a TEST-*.xml file (likely timed out)
(batchId=375)
TestHs2HooksWithMiniKdc - did not produce a TEST-*.xml file (likely timed out)
(batchId=451)
TestJdbcDriver2 - did not produce a TEST-*.xml file (likely timed out)
(batchId=410)
TestJdbcMetadataApiAuth - did not produce a TEST-*.xml file (likely timed out)
(batchId=421)
TestJdbcWithLocalClusterSpark - did not produce a TEST-*.xml file (likely timed
out) (batchId=415)
TestJdbcWithMiniHS2 - did not produce a TEST-*.xml file (likely timed out)
(batchId=412)
TestJdbcWithMiniKdc - did not produce a TEST-*.xml file (likely timed out)
(batchId=448)
TestJdbcWithMiniKdcCookie - did not produce a TEST-*.xml file (likely timed
out) (batchId=447)
TestJdbcWithMiniKdcSQLAuthBinary - did not produce a TEST-*.xml file (likely
timed out) (batchId=445)
TestJdbcWithMiniKdcSQLAuthHttp - did not produce a TEST-*.xml file (likely
timed out) (batchId=450)
TestJdbcWithMiniMr - did not produce a TEST-*.xml file (likely timed out)
(batchId=411)
TestJdbcWithSQLAuthUDFBlacklist - did not produce a TEST-*.xml file (likely
timed out) (batchId=417)
TestJdbcWithSQLAuthorization - did not produce a TEST-*.xml file (likely timed
out) (batchId=418)
TestLocationQueries - did not produce a TEST-*.xml file (likely timed out)
(batchId=382)
TestMTQueries - did not produce a TEST-*.xml file (likely timed out)
[jira] [Commented] (HIVE-16184) Remove use of optionals in union; declare TProgressUpdateResp before its usage
[ https://issues.apache.org/jira/browse/HIVE-16184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931363#comment-15931363 ] Edward Capriolo commented on HIVE-16184: We should not be modifying things for compatibility with scrooge https://github.com/twitter/scrooge https://github.com/twitter/scrooge/issues/263 This is a port and we should not be forcing ourselves to support it. > Remove use of optionals in union; declare TProgressUpdateResp before its usage > -- > > Key: HIVE-16184 > URL: https://issues.apache.org/jira/browse/HIVE-16184 > Project: Hive > Issue Type: Bug > Components: Thrift API >Reporter: Ian Macalinao >Assignee: Ian Macalinao >Priority: Minor > Attachments: HIVE-16184.patch > > > A Scala Thrift generator (https://github.com/twitter/scrooge) does not like > the Thrift file. I have removed some bad Thrift practices (i.e. optionals in > union, dependent types declared after their usage) and replaced them with > things that work in Scrooge. > See https://github.com/apache/hive/pull/157 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-13282) GroupBy and select operator encounter ArrayIndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HIVE-13282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vaibhav Gumashta updated HIVE-13282:
Target Version/s: 1.3.0, 2.2.0 (was: 1.2.2, 2.2.0)
> GroupBy and select operator encounter ArrayIndexOutOfBoundsException
>
>
> Key: HIVE-13282
> URL: https://issues.apache.org/jira/browse/HIVE-13282
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 1.2.1, 2.0.0, 2.1.0
>Reporter: Vikram Dixit K
>Assignee: Matt McCline
>Priority: Blocker
> Attachments: HIVE-13282.01.patch, smb_fail_issue.patch,
> smb_groupby.q, smb_groupby.q.out
>
>
> The group by and select operators run into the ArrayIndexOutOfBoundsException
> when they incorrectly initialize themselves with tag 0 but the incoming tag
> id is different.
> {code}
> select count(*) from
> (select rt1.id from
> (select t1.key as id, t1.value as od from tab t1 group by key, value) rt1) vt1
> join
> (select rt2.id from
> (select t2.key as id, t2.value as od from tab_part t2 group by key, value)
> rt2) vt2
> where vt1.id=vt2.id;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-13282) GroupBy and select operator encounter ArrayIndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HIVE-13282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931353#comment-15931353
]
Vaibhav Gumashta commented on HIVE-13282:
-
Removing target 1.2.2. [~mmccline] feel free to add it back if you feel this is
a must have in 1.2.2
> GroupBy and select operator encounter ArrayIndexOutOfBoundsException
>
>
> Key: HIVE-13282
> URL: https://issues.apache.org/jira/browse/HIVE-13282
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 1.2.1, 2.0.0, 2.1.0
>Reporter: Vikram Dixit K
>Assignee: Matt McCline
>Priority: Blocker
> Attachments: HIVE-13282.01.patch, smb_fail_issue.patch,
> smb_groupby.q, smb_groupby.q.out
>
>
> The group by and select operators run into the ArrayIndexOutOfBoundsException
> when they incorrectly initialize themselves with tag 0 but the incoming tag
> id is different.
> {code}
> select count(*) from
> (select rt1.id from
> (select t1.key as id, t1.value as od from tab t1 group by key, value) rt1) vt1
> join
> (select rt2.id from
> (select t2.key as id, t2.value as od from tab_part t2 group by key, value)
> rt2) vt2
> where vt1.id=vt2.id;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15126) Branch-1.2: Fix TestCliDriver.join_merge_multi_expressions.q
[ https://issues.apache.org/jira/browse/HIVE-15126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931352#comment-15931352 ] Vaibhav Gumashta commented on HIVE-15126: - Tests look good: https://builds.apache.org/job/PreCommit-HIVE-Build/4226/#showFailuresLink: org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchEmptyCommit org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_from_utc_timestamp org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_merge_multi_expressions org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_to_utc_timestamp org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_exchgpartition2lel org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_auto_smb_mapjoin_14 org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_smb_empty org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_bucket_map_join_tez1 org.apache.hive.minikdc.TestMiniHiveKdc.testLogin org.apache.hive.minikdc.TestHiveAuthFactory.testStartTokenManagerForMemoryTokenStore org.apache.hive.minikdc.TestHiveAuthFactory.testStartTokenManagerForDBTokenStore [~daijy] can you please take a look at the patch? > Branch-1.2: Fix TestCliDriver.join_merge_multi_expressions.q > > > Key: HIVE-15126 > URL: https://issues.apache.org/jira/browse/HIVE-15126 > Project: Hive > Issue Type: Sub-task > Components: Test >Affects Versions: 1.2.1 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Blocker > Attachments: HIVE-15082.1-branch-1.2.patch, HIVE-15126.1.patch > > > The .out file for join_merge_multi_expressions.q needs to be updated. Current > one does not show Select operator in the plan that explain returns. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (HIVE-15082) Hive-1.2 cannot read data from complex data types with TIMESTAMP column, stored in Parquet
[
https://issues.apache.org/jira/browse/HIVE-15082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931347#comment-15931347
]
Vaibhav Gumashta edited comment on HIVE-15082 at 3/18/17 7:24 PM:
--
No extra failures with this patch:
https://builds.apache.org/job/PreCommit-HIVE-Build/4224/#showFailuresLink, when
compared with HIVE-15007:
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_from_utc_timestamp
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_merge_multi_expressions
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_to_utc_timestamp
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_exchgpartition2lel
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_auto_smb_mapjoin_14
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_smb_empty
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_bucket_map_join_tez1
org.apache.hive.minikdc.TestMiniHiveKdc.testLogin
org.apache.hive.minikdc.TestHiveAuthFactory.testStartTokenManagerForMemoryTokenStore
org.apache.hive.minikdc.TestHiveAuthFactory.testStartTokenManagerForDBTokenStore
This however needs a review as well. [~osayankin] Do you know if this is a bug
in master as well? [~csun] would you like to take a look?
I'll keep the target as 1.2.2 for now if we're able to resolve it in time.
was (Author: vgumashta):
No extra failures with this patch:
https://builds.apache.org/job/PreCommit-HIVE-Build/4224/#showFailuresLink, when
compared with HIVE-15007:
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_from_utc_timestamp
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_merge_multi_expressions
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_to_utc_timestamp
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_exchgpartition2lel
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_auto_smb_mapjoin_14
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_smb_empty
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_bucket_map_join_tez1
org.apache.hive.minikdc.TestMiniHiveKdc.testLogin
org.apache.hive.minikdc.TestHiveAuthFactory.testStartTokenManagerForMemoryTokenStore
org.apache.hive.minikdc.TestHiveAuthFactory.testStartTokenManagerForDBTokenStore
I'll commit this to 1.2 shortly.
> Hive-1.2 cannot read data from complex data types with TIMESTAMP column,
> stored in Parquet
> --
>
> Key: HIVE-15082
> URL: https://issues.apache.org/jira/browse/HIVE-15082
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Blocker
> Attachments: HIVE-15082.1-branch-1.2.patch,
> HIVE-15082.1-branch-1.2.patch, HIVE-15082-branch-1.2.patch,
> HIVE-15082-branch-1.patch
>
>
> *STEP 1. Create test data*
> {code:sql}
> select * from dual;
> {code}
> *EXPECTED RESULT:*
> {noformat}
> Pretty_UnIQUe_StrinG
> {noformat}
> {code:sql}
> create table test_parquet1(login timestamp) stored as parquet;
> insert overwrite table test_parquet1 select from_unixtime(unix_timestamp())
> from dual;
> select * from test_parquet1 limit 1;
> {code}
> *EXPECTED RESULT:*
> No exceptions. Current timestamp as result.
> {noformat}
> 2016-10-27 10:58:19
> {noformat}
> *STEP 2. Store timestamp in array in parquet file*
> {code:sql}
> create table test_parquet2(x array) stored as parquet;
> insert overwrite table test_parquet2 select array(login) from test_parquet1;
> select * from test_parquet2;
> {code}
> *EXPECTED RESULT:*
> No exceptions. Current timestamp in brackets as result.
> {noformat}
> ["2016-10-27 10:58:19"]
> {noformat}
> *ACTUAL RESULT:*
> {noformat}
> ERROR [main]: CliDriver (SessionState.java:printError(963)) - Failed with
> exception java.io.IOException:parquet.io.ParquetDecodingException: Can not
> read value at 0 in block -1 in file
> hdfs:///user/hive/warehouse/test_parquet2/00_0
> java.io.IOException: parquet.io.ParquetDecodingException: Can not read value
> at 0 in block -1 in file hdfs:///user/hive/warehouse/test_parquet2/00_0
> {noformat}
> *ROOT-CAUSE:*
> Incorrect initialization of {{metadata}} {{HashMap}} causes that it has
> {{null}} value in enumeration
> {{org.apache.hadoop.hive.ql.io.parquet.convert.ETypeConverter}} when
> executing following line:
> {code:java}
> boolean skipConversion =
>
[jira] [Commented] (HIVE-15082) Hive-1.2 cannot read data from complex data types with TIMESTAMP column, stored in Parquet
[
https://issues.apache.org/jira/browse/HIVE-15082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931347#comment-15931347
]
Vaibhav Gumashta commented on HIVE-15082:
-
No extra failures with this patch:
https://builds.apache.org/job/PreCommit-HIVE-Build/4224/#showFailuresLink, when
compared with HIVE-15007:
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_from_utc_timestamp
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_merge_multi_expressions
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_to_utc_timestamp
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_exchgpartition2lel
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_auto_smb_mapjoin_14
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_smb_empty
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_bucket_map_join_tez1
org.apache.hive.minikdc.TestMiniHiveKdc.testLogin
org.apache.hive.minikdc.TestHiveAuthFactory.testStartTokenManagerForMemoryTokenStore
org.apache.hive.minikdc.TestHiveAuthFactory.testStartTokenManagerForDBTokenStore
I'll commit this to 1.2 shortly.
> Hive-1.2 cannot read data from complex data types with TIMESTAMP column,
> stored in Parquet
> --
>
> Key: HIVE-15082
> URL: https://issues.apache.org/jira/browse/HIVE-15082
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Blocker
> Attachments: HIVE-15082.1-branch-1.2.patch,
> HIVE-15082.1-branch-1.2.patch, HIVE-15082-branch-1.2.patch,
> HIVE-15082-branch-1.patch
>
>
> *STEP 1. Create test data*
> {code:sql}
> select * from dual;
> {code}
> *EXPECTED RESULT:*
> {noformat}
> Pretty_UnIQUe_StrinG
> {noformat}
> {code:sql}
> create table test_parquet1(login timestamp) stored as parquet;
> insert overwrite table test_parquet1 select from_unixtime(unix_timestamp())
> from dual;
> select * from test_parquet1 limit 1;
> {code}
> *EXPECTED RESULT:*
> No exceptions. Current timestamp as result.
> {noformat}
> 2016-10-27 10:58:19
> {noformat}
> *STEP 2. Store timestamp in array in parquet file*
> {code:sql}
> create table test_parquet2(x array) stored as parquet;
> insert overwrite table test_parquet2 select array(login) from test_parquet1;
> select * from test_parquet2;
> {code}
> *EXPECTED RESULT:*
> No exceptions. Current timestamp in brackets as result.
> {noformat}
> ["2016-10-27 10:58:19"]
> {noformat}
> *ACTUAL RESULT:*
> {noformat}
> ERROR [main]: CliDriver (SessionState.java:printError(963)) - Failed with
> exception java.io.IOException:parquet.io.ParquetDecodingException: Can not
> read value at 0 in block -1 in file
> hdfs:///user/hive/warehouse/test_parquet2/00_0
> java.io.IOException: parquet.io.ParquetDecodingException: Can not read value
> at 0 in block -1 in file hdfs:///user/hive/warehouse/test_parquet2/00_0
> {noformat}
> *ROOT-CAUSE:*
> Incorrect initialization of {{metadata}} {{HashMap}} causes that it has
> {{null}} value in enumeration
> {{org.apache.hadoop.hive.ql.io.parquet.convert.ETypeConverter}} when
> executing following line:
> {code:java}
> boolean skipConversion =
> Boolean.valueOf(metadata.get(HiveConf.ConfVars.HIVE_PARQUET_TIMESTAMP_SKIP_CONVERSION.varname));
> {code}
> in element {{ETIMESTAMP_CONVERTER}}.
> JVM throws NPE and parquet library can not read data from file and throws
> {noformat}
> java.io.IOException:parquet.io.ParquetDecodingException: Can not read value
> at 0 in block -1 in file hdfs:///user/hive/warehouse/test_parquet2/00_0
> {noformat}
> for its turn.
> *SOLUTION:*
> Perform initialization in separate method to skip overriding it with {{null}}
> value in block of code
> {code:java}
> if (parent != null) {
> setMetadata(parent.getMetadata());
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-15007) Hive 1.2.2 release planning
[ https://issues.apache.org/jira/browse/HIVE-15007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-15007: Attachment: HIVE-15007-branch-1.patch > Hive 1.2.2 release planning > --- > > Key: HIVE-15007 > URL: https://issues.apache.org/jira/browse/HIVE-15007 > Project: Hive > Issue Type: Task >Affects Versions: 1.2.1 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > Attachments: HIVE-15007-branch-1.1.patch, > HIVE-15007-branch-1.2.patch, HIVE-15007-branch-1.2.patch, > HIVE-15007-branch-1.2.patch, HIVE-15007-branch-1.2.patch, > HIVE-15007-branch-1.2.patch, HIVE-15007-branch-1.2.patch, > HIVE-15007.branch-1.2.patch, HIVE-15007-branch-1.patch > > > Discussed with [~spena] about triggering unit test runs for 1.2.2 release and > creating a patch which will trigger precommits looks like a good way. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
