[jira] [Resolved] (HIVE-2270) Hive should not call methods from org.apache.hadoop.mapred.JobTracker
[ https://issues.apache.org/jira/browse/HIVE-2270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu resolved HIVE-2270. --- Resolution: Not A Problem Assignee: (was: Amareshwari Sriramadasu) > Hive should not call methods from org.apache.hadoop.mapred.JobTracker > - > > Key: HIVE-2270 > URL: https://issues.apache.org/jira/browse/HIVE-2270 > Project: Hive > Issue Type: Bug >Reporter: Amareshwari Sriramadasu > > Currently Hive calls getAddress() and getState() api from > org.apache.hadoop.mapred.JobTracker. These should be replaced with > client-side ones if any, since the class is private(annotated) to hadoop. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13095) Support view column authorization
[ https://issues.apache.org/jira/browse/HIVE-13095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lefty Leverenz updated HIVE-13095: -- Labels: TODOC2.1 (was: ) > Support view column authorization > - > > Key: HIVE-13095 > URL: https://issues.apache.org/jira/browse/HIVE-13095 > Project: Hive > Issue Type: New Feature >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Labels: TODOC2.1 > Fix For: 2.1.0 > > Attachments: HIVE-13095.01.patch, HIVE-13095.02.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13095) Support view column authorization
[
https://issues.apache.org/jira/browse/HIVE-13095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15175263#comment-15175263
]
Lefty Leverenz commented on HIVE-13095:
---
Okay, I'll give this bugture a TODOC2.1 label.
Does it only apply to the original ("default") authorization mode?
* [Hive Default Authorization |
https://cwiki.apache.org/confluence/display/Hive/Hive+Default+Authorization+-+Legacy+Mode]
> Support view column authorization
> -
>
> Key: HIVE-13095
> URL: https://issues.apache.org/jira/browse/HIVE-13095
> Project: Hive
> Issue Type: New Feature
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Labels: TODOC2.1
> Fix For: 2.1.0
>
> Attachments: HIVE-13095.01.patch, HIVE-13095.02.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10165) Improve hive-hcatalog-streaming extensibility and support updates and deletes.
[
https://issues.apache.org/jira/browse/HIVE-10165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lefty Leverenz updated HIVE-10165:
--
Labels: streaming_api (was: TODOC2.0 streaming_api)
> Improve hive-hcatalog-streaming extensibility and support updates and deletes.
> --
>
> Key: HIVE-10165
> URL: https://issues.apache.org/jira/browse/HIVE-10165
> Project: Hive
> Issue Type: Improvement
> Components: HCatalog
>Affects Versions: 1.2.0
>Reporter: Elliot West
>Assignee: Elliot West
> Labels: streaming_api
> Fix For: 2.0.0
>
> Attachments: HIVE-10165.0.patch, HIVE-10165.10.patch,
> HIVE-10165.4.patch, HIVE-10165.5.patch, HIVE-10165.6.patch,
> HIVE-10165.7.patch, HIVE-10165.9.patch, mutate-system-overview.png
>
>
> h3. Overview
> I'd like to extend the
> [hive-hcatalog-streaming|https://cwiki.apache.org/confluence/display/Hive/Streaming+Data+Ingest]
> API so that it also supports the writing of record updates and deletes in
> addition to the already supported inserts.
> h3. Motivation
> We have many Hadoop processes outside of Hive that merge changed facts into
> existing datasets. Traditionally we achieve this by: reading in a
> ground-truth dataset and a modified dataset, grouping by a key, sorting by a
> sequence and then applying a function to determine inserted, updated, and
> deleted rows. However, in our current scheme we must rewrite all partitions
> that may potentially contain changes. In practice the number of mutated
> records is very small when compared with the records contained in a
> partition. This approach results in a number of operational issues:
> * Excessive amount of write activity required for small data changes.
> * Downstream applications cannot robustly read these datasets while they are
> being updated.
> * Due to scale of the updates (hundreds or partitions) the scope for
> contention is high.
> I believe we can address this problem by instead writing only the changed
> records to a Hive transactional table. This should drastically reduce the
> amount of data that we need to write and also provide a means for managing
> concurrent access to the data. Our existing merge processes can read and
> retain each record's {{ROW_ID}}/{{RecordIdentifier}} and pass this through to
> an updated form of the hive-hcatalog-streaming API which will then have the
> required data to perform an update or insert in a transactional manner.
> h3. Benefits
> * Enables the creation of large-scale dataset merge processes
> * Opens up Hive transactional functionality in an accessible manner to
> processes that operate outside of Hive.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10165) Improve hive-hcatalog-streaming extensibility and support updates and deletes.
[
https://issues.apache.org/jira/browse/HIVE-10165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15175345#comment-15175345
]
Lefty Leverenz commented on HIVE-10165:
---
Okay, I made those changes and now I'll optimistically remove the TODOC2.0
label. (But feel free to tinker with my edits.)
* [edits to HCatalog Streaming Mutation API |
https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=61337025&selectedPageVersions=5&selectedPageVersions=6]
> Improve hive-hcatalog-streaming extensibility and support updates and deletes.
> --
>
> Key: HIVE-10165
> URL: https://issues.apache.org/jira/browse/HIVE-10165
> Project: Hive
> Issue Type: Improvement
> Components: HCatalog
>Affects Versions: 1.2.0
>Reporter: Elliot West
>Assignee: Elliot West
> Labels: streaming_api
> Fix For: 2.0.0
>
> Attachments: HIVE-10165.0.patch, HIVE-10165.10.patch,
> HIVE-10165.4.patch, HIVE-10165.5.patch, HIVE-10165.6.patch,
> HIVE-10165.7.patch, HIVE-10165.9.patch, mutate-system-overview.png
>
>
> h3. Overview
> I'd like to extend the
> [hive-hcatalog-streaming|https://cwiki.apache.org/confluence/display/Hive/Streaming+Data+Ingest]
> API so that it also supports the writing of record updates and deletes in
> addition to the already supported inserts.
> h3. Motivation
> We have many Hadoop processes outside of Hive that merge changed facts into
> existing datasets. Traditionally we achieve this by: reading in a
> ground-truth dataset and a modified dataset, grouping by a key, sorting by a
> sequence and then applying a function to determine inserted, updated, and
> deleted rows. However, in our current scheme we must rewrite all partitions
> that may potentially contain changes. In practice the number of mutated
> records is very small when compared with the records contained in a
> partition. This approach results in a number of operational issues:
> * Excessive amount of write activity required for small data changes.
> * Downstream applications cannot robustly read these datasets while they are
> being updated.
> * Due to scale of the updates (hundreds or partitions) the scope for
> contention is high.
> I believe we can address this problem by instead writing only the changed
> records to a Hive transactional table. This should drastically reduce the
> amount of data that we need to write and also provide a means for managing
> concurrent access to the data. Our existing merge processes can read and
> retain each record's {{ROW_ID}}/{{RecordIdentifier}} and pass this through to
> an updated form of the hive-hcatalog-streaming API which will then have the
> required data to perform an update or insert in a transactional manner.
> h3. Benefits
> * Enables the creation of large-scale dataset merge processes
> * Opens up Hive transactional functionality in an accessible manner to
> processes that operate outside of Hive.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Work started] (HIVE-13189) Consider using Joda DateTimeFormatter instead of SimpleDateFormat in GenericUDFDateAdd
[
https://issues.apache.org/jira/browse/HIVE-13189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-13189 started by varun a kumar.
> Consider using Joda DateTimeFormatter instead of SimpleDateFormat in
> GenericUDFDateAdd
> --
>
> Key: HIVE-13189
> URL: https://issues.apache.org/jira/browse/HIVE-13189
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Rajesh Balamohan
>Assignee: varun a kumar
>
> Quite an amount was spent by tasks in trying to parse date string in
> GenericUDFDateAdd.
> {noformat}
> java.lang.Thread.State: RUNNABLE
> at java.text.DecimalFormat.subparse(DecimalFormat.java:1467)
> at java.text.DecimalFormat.parse(DecimalFormat.java:1268)
> at java.text.SimpleDateFormat.subParse(SimpleDateFormat.java:2088)
> at java.text.SimpleDateFormat.parse(SimpleDateFormat.java:1455)
> at java.text.DateFormat.parse(DateFormat.java:355)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFDateAdd.evaluate(GenericUDFDateAdd.java:172)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:186)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator$DeferredExprObject.get(ExprNodeGenericFuncEvaluator.java:87)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPGreaterThan.evaluate(GenericUDFOPGreaterThan.java:80)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:186)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:65)
> at
> org.apache.hadoop.hive.ql.exec.FilterOperator.process(FilterOperator.java:108)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:838)
> at
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator.internalForward(CommonJoinOperator.java:644)
> {noformat}
> Joda DateTimeFormatter can be considered for better performance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-13189) Consider using Joda DateTimeFormatter instead of SimpleDateFormat in GenericUDFDateAdd
[
https://issues.apache.org/jira/browse/HIVE-13189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
varun a kumar reassigned HIVE-13189:
Assignee: varun a kumar
> Consider using Joda DateTimeFormatter instead of SimpleDateFormat in
> GenericUDFDateAdd
> --
>
> Key: HIVE-13189
> URL: https://issues.apache.org/jira/browse/HIVE-13189
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Rajesh Balamohan
>Assignee: varun a kumar
>
> Quite an amount was spent by tasks in trying to parse date string in
> GenericUDFDateAdd.
> {noformat}
> java.lang.Thread.State: RUNNABLE
> at java.text.DecimalFormat.subparse(DecimalFormat.java:1467)
> at java.text.DecimalFormat.parse(DecimalFormat.java:1268)
> at java.text.SimpleDateFormat.subParse(SimpleDateFormat.java:2088)
> at java.text.SimpleDateFormat.parse(SimpleDateFormat.java:1455)
> at java.text.DateFormat.parse(DateFormat.java:355)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFDateAdd.evaluate(GenericUDFDateAdd.java:172)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:186)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator$DeferredExprObject.get(ExprNodeGenericFuncEvaluator.java:87)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPGreaterThan.evaluate(GenericUDFOPGreaterThan.java:80)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:186)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:65)
> at
> org.apache.hadoop.hive.ql.exec.FilterOperator.process(FilterOperator.java:108)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:838)
> at
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator.internalForward(CommonJoinOperator.java:644)
> {noformat}
> Joda DateTimeFormatter can be considered for better performance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10165) Improve hive-hcatalog-streaming extensibility and support updates and deletes.
[
https://issues.apache.org/jira/browse/HIVE-10165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15175410#comment-15175410
]
Lefty Leverenz commented on HIVE-10165:
---
[~teabot], do you still intend to update the Locking and Transactions docs? (I
see some changes you made for concatenate, but that's all so far.)
See comment on 9 July 2015:
{quote}
... the current locking documentation does not cover update/delete scenarios or
semi-shared locks. I'll volunteer to update these pages once I have a clearer
understanding of how these lock types apply to these operations and partitions:
* https://cwiki.apache.org/confluence/display/Hive/Locking
* https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions
{quote}
> Improve hive-hcatalog-streaming extensibility and support updates and deletes.
> --
>
> Key: HIVE-10165
> URL: https://issues.apache.org/jira/browse/HIVE-10165
> Project: Hive
> Issue Type: Improvement
> Components: HCatalog
>Affects Versions: 1.2.0
>Reporter: Elliot West
>Assignee: Elliot West
> Labels: streaming_api
> Fix For: 2.0.0
>
> Attachments: HIVE-10165.0.patch, HIVE-10165.10.patch,
> HIVE-10165.4.patch, HIVE-10165.5.patch, HIVE-10165.6.patch,
> HIVE-10165.7.patch, HIVE-10165.9.patch, mutate-system-overview.png
>
>
> h3. Overview
> I'd like to extend the
> [hive-hcatalog-streaming|https://cwiki.apache.org/confluence/display/Hive/Streaming+Data+Ingest]
> API so that it also supports the writing of record updates and deletes in
> addition to the already supported inserts.
> h3. Motivation
> We have many Hadoop processes outside of Hive that merge changed facts into
> existing datasets. Traditionally we achieve this by: reading in a
> ground-truth dataset and a modified dataset, grouping by a key, sorting by a
> sequence and then applying a function to determine inserted, updated, and
> deleted rows. However, in our current scheme we must rewrite all partitions
> that may potentially contain changes. In practice the number of mutated
> records is very small when compared with the records contained in a
> partition. This approach results in a number of operational issues:
> * Excessive amount of write activity required for small data changes.
> * Downstream applications cannot robustly read these datasets while they are
> being updated.
> * Due to scale of the updates (hundreds or partitions) the scope for
> contention is high.
> I believe we can address this problem by instead writing only the changed
> records to a Hive transactional table. This should drastically reduce the
> amount of data that we need to write and also provide a means for managing
> concurrent access to the data. Our existing merge processes can read and
> retain each record's {{ROW_ID}}/{{RecordIdentifier}} and pass this through to
> an updated form of the hive-hcatalog-streaming API which will then have the
> required data to perform an update or insert in a transactional manner.
> h3. Benefits
> * Enables the creation of large-scale dataset merge processes
> * Opens up Hive transactional functionality in an accessible manner to
> processes that operate outside of Hive.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12994) Implement support for NULLS FIRST/NULLS LAST
[
https://issues.apache.org/jira/browse/HIVE-12994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15175415#comment-15175415
]
Hive QA commented on HIVE-12994:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12790722/HIVE-12994.12.patch
{color:green}SUCCESS:{color} +1 due to 9 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 12 failed/errored test(s), 9767 tests
executed
*Failed tests:*
{noformat}
TestMiniTezCliDriver-explainuser_4.q-mapreduce1.q-vectorization_14.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-groupby3_map.q-sample2.q-auto_join14.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-groupby_map_ppr_multi_distinct.q-table_access_keys_stats.q-groupby4_noskew.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join_rc.q-insert1.q-vectorized_rcfile_columnar.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-ppd_join4.q-join9.q-ppd_join3.q-and-12-more - did not
produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_join_with_different_encryption_keys
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_order_null
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_char_simple
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_transform_ppr2
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7142/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7142/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7142/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 12 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12790722 - PreCommit-HIVE-TRUNK-Build
> Implement support for NULLS FIRST/NULLS LAST
>
>
> Key: HIVE-12994
> URL: https://issues.apache.org/jira/browse/HIVE-12994
> Project: Hive
> Issue Type: New Feature
> Components: CBO, Parser, Serializers/Deserializers
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-12994.01.patch, HIVE-12994.02.patch,
> HIVE-12994.03.patch, HIVE-12994.04.patch, HIVE-12994.05.patch,
> HIVE-12994.06.patch, HIVE-12994.06.patch, HIVE-12994.07.patch,
> HIVE-12994.08.patch, HIVE-12994.09.patch, HIVE-12994.10.patch,
> HIVE-12994.11.patch, HIVE-12994.12.patch, HIVE-12994.patch
>
>
> From SQL:2003, the NULLS FIRST and NULLS LAST options can be used to
> determine whether nulls appear before or after non-null data values when the
> ORDER BY clause is used.
> SQL standard does not specify the behavior by default. Currently in Hive,
> null values sort as if lower than any non-null value; that is, NULLS FIRST is
> the default for ASC order, and NULLS LAST for DESC order.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12994) Implement support for NULLS FIRST/NULLS LAST
[ https://issues.apache.org/jira/browse/HIVE-12994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-12994: --- Resolution: Fixed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) Regenerated remaining q files and pushed to master. Thanks for reviewing [~ashutoshc]! > Implement support for NULLS FIRST/NULLS LAST > > > Key: HIVE-12994 > URL: https://issues.apache.org/jira/browse/HIVE-12994 > Project: Hive > Issue Type: New Feature > Components: CBO, Parser, Serializers/Deserializers >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Fix For: 2.1.0 > > Attachments: HIVE-12994.01.patch, HIVE-12994.02.patch, > HIVE-12994.03.patch, HIVE-12994.04.patch, HIVE-12994.05.patch, > HIVE-12994.06.patch, HIVE-12994.06.patch, HIVE-12994.07.patch, > HIVE-12994.08.patch, HIVE-12994.09.patch, HIVE-12994.10.patch, > HIVE-12994.11.patch, HIVE-12994.12.patch, HIVE-12994.patch > > > From SQL:2003, the NULLS FIRST and NULLS LAST options can be used to > determine whether nulls appear before or after non-null data values when the > ORDER BY clause is used. > SQL standard does not specify the behavior by default. Currently in Hive, > null values sort as if lower than any non-null value; that is, NULLS FIRST is > the default for ASC order, and NULLS LAST for DESC order. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13069) Enable cartesian product merging
[ https://issues.apache.org/jira/browse/HIVE-13069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13069: --- Attachment: HIVE-13069.01.patch > Enable cartesian product merging > > > Key: HIVE-13069 > URL: https://issues.apache.org/jira/browse/HIVE-13069 > Project: Hive > Issue Type: Improvement > Components: Parser >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13069.01.patch, HIVE-13069.01.patch, > HIVE-13069.patch > > > Currently we can merge 2-way joins into n-way joins when the joins are > executed over the same column. > In turn, CBO might produce plans containing cartesian products if the join > columns are constant values; after HIVE-12543 went in, this is rather common, > as those constant columns are correctly pruned. However, currently we do not > merge a cartesian product with two inputs into a cartesian product with > multiple inputs, which could result in performance loss. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-4570) More information to user on GetOperationStatus in Hive Server2 when query is still executing
[ https://issues.apache.org/jira/browse/HIVE-4570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay Goyal updated HIVE-4570: --- Status: Open (was: Patch Available) > More information to user on GetOperationStatus in Hive Server2 when query is > still executing > > > Key: HIVE-4570 > URL: https://issues.apache.org/jira/browse/HIVE-4570 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Amareshwari Sriramadasu >Assignee: Akshay Goyal > Attachments: HIVE-4570.01.patch, HIVE-4570.02.patch, > HIVE-4570.03.patch, HIVE-4570.04.patch > > > Currently in Hive Server2, when the query is still executing only the status > is set as STILL_EXECUTING. > This issue is to give more information to the user such as progress and > running job handles, if possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HIVE-4570) More information to user on GetOperationStatus in Hive Server2 when query is still executing
[ https://issues.apache.org/jira/browse/HIVE-4570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-4570 started by Akshay Goyal. -- > More information to user on GetOperationStatus in Hive Server2 when query is > still executing > > > Key: HIVE-4570 > URL: https://issues.apache.org/jira/browse/HIVE-4570 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Amareshwari Sriramadasu >Assignee: Akshay Goyal > Attachments: HIVE-4570.01.patch, HIVE-4570.02.patch, > HIVE-4570.03.patch, HIVE-4570.04.patch > > > Currently in Hive Server2, when the query is still executing only the status > is set as STILL_EXECUTING. > This issue is to give more information to the user such as progress and > running job handles, if possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-4570) More information to user on GetOperationStatus in Hive Server2 when query is still executing
[ https://issues.apache.org/jira/browse/HIVE-4570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay Goyal updated HIVE-4570: --- Attachment: HIVE-4570.04.patch > More information to user on GetOperationStatus in Hive Server2 when query is > still executing > > > Key: HIVE-4570 > URL: https://issues.apache.org/jira/browse/HIVE-4570 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Amareshwari Sriramadasu >Assignee: Akshay Goyal > Attachments: HIVE-4570.01.patch, HIVE-4570.02.patch, > HIVE-4570.03.patch, HIVE-4570.04.patch > > > Currently in Hive Server2, when the query is still executing only the status > is set as STILL_EXECUTING. > This issue is to give more information to the user such as progress and > running job handles, if possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-4570) More information to user on GetOperationStatus in Hive Server2 when query is still executing
[ https://issues.apache.org/jira/browse/HIVE-4570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay Goyal updated HIVE-4570: --- Status: Patch Available (was: In Progress) > More information to user on GetOperationStatus in Hive Server2 when query is > still executing > > > Key: HIVE-4570 > URL: https://issues.apache.org/jira/browse/HIVE-4570 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Amareshwari Sriramadasu >Assignee: Akshay Goyal > Attachments: HIVE-4570.01.patch, HIVE-4570.02.patch, > HIVE-4570.03.patch, HIVE-4570.04.patch > > > Currently in Hive Server2, when the query is still executing only the status > is set as STILL_EXECUTING. > This issue is to give more information to the user such as progress and > running job handles, if possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12749) Constant propagate returns string values in incorrect format
[
https://issues.apache.org/jira/browse/HIVE-12749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-12749:
---
Resolution: Fixed
Fix Version/s: 2.1.0
Status: Resolved (was: Patch Available)
Pushed to master, branch-2. Thanks [~AleKsey Vovchenko]!
I have logged HIVE-13197 to create the constprog2.q and constprog_partitioner.q
tests with compatible types.
> Constant propagate returns string values in incorrect format
>
>
> Key: HIVE-12749
> URL: https://issues.apache.org/jira/browse/HIVE-12749
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.0.0, 1.2.0
>Reporter: Oleksiy Sayankin
>Assignee: Aleksey Vovchenko
> Fix For: 2.1.0, 2.0.1
>
> Attachments: HIVE-12749.1.patch, HIVE-12749.2.patch,
> HIVE-12749.3.patch, HIVE-12749.4.patch, HIVE-12749.5.patch,
> HIVE-12749.6.patch, HIVE-12749.7.patch, HIVE-12749.8.patch
>
>
> h2. STEP 1. Create and upload test data
> Execute in command line:
> {noformat}
> nano stest.data
> {noformat}
> Add to file:
> {noformat}
> 000126,000777
> 000126,000778
> 000126,000779
> 000474,000888
> 000468,000889
> 000272,000880
> {noformat}
> {noformat}
> hadoop fs -put stest.data /
> {noformat}
> {noformat}
> hive> create table stest(x STRING, y STRING) ROW FORMAT DELIMITED FIELDS
> TERMINATED BY ',';
> hive> LOAD DATA INPATH '/stest.data' OVERWRITE INTO TABLE stest;
> {noformat}
> h2. STEP 2. Execute test query (with cast for x)
> {noformat}
> select x from stest where cast(x as int) = 126;
> {noformat}
> EXPECTED RESULT:
> {noformat}
> 000126
> 000126
> 000126
> {noformat}
> ACTUAL RESULT:
> {noformat}
> 126
> 126
> 126
> {noformat}
> h2. STEP 3. Execute test query (no cast for x)
> {noformat}
> hive> select x from stest where x = 126;
> {noformat}
> EXPECTED RESULT:
> {noformat}
> 000126
> 000126
> 000126
> {noformat}
> ACTUAL RESULT:
> {noformat}
> 126
> 126
> 126
> {noformat}
> In steps #2, #3 I expected '000126' because the origin type of x is STRING in
> stest table.
> Note, setting hive.optimize.constant.propagation=false fixes the issue.
> {noformat}
> hive> set hive.optimize.constant.propagation=false;
> hive> select x from stest where x = 126;
> OK
> 000126
> 000126
> 000126
> {noformat}
> Related to HIVE-11104, HIVE-8555
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13145) Separate the output path of metrics file of HS2 and HMS
[
https://issues.apache.org/jira/browse/HIVE-13145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15175524#comment-15175524
]
Shinichi Yamashita commented on HIVE-13145:
---
For the concern to embedded mode, it can be settled by change of the property
name (e.g. hive.remote-metastore.metrics.file.location) and the addition of the
document.
However, I think that the user is difficult to treat by this patch. So, I will
change this patch.
> Separate the output path of metrics file of HS2 and HMS
> ---
>
> Key: HIVE-13145
> URL: https://issues.apache.org/jira/browse/HIVE-13145
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2, Metastore
>Reporter: Shinichi Yamashita
>Assignee: Shinichi Yamashita
> Attachments: HIVE-13145.1.patch, HIVE-13145.2.patch
>
>
> The output path of metrics file of HS2 and HMS can define by
> {{hive.service.metrics.file.location}} property at present.
> When it starts HS2 and HMS by the same server, both metrics is written in the
> same file. And when confirming this file, it is difficult to judge which
> metrics it is.
> Therefore the output path of metrics file of HS2 and HMS is separated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13179) Allow custom HiveConf to be passed to Authentication Providers
[ https://issues.apache.org/jira/browse/HIVE-13179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15175556#comment-15175556 ] Amareshwari Sriramadasu commented on HIVE-13179: [~prongs], Can you check test failures TestHiveMetaStorePartitionSpecs and org.apache.hive.jdbc.TestSSL? Seems first time failures on the build. > Allow custom HiveConf to be passed to Authentication Providers > -- > > Key: HIVE-13179 > URL: https://issues.apache.org/jira/browse/HIVE-13179 > Project: Hive > Issue Type: Improvement >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Attachments: HIVE-13179.1.patch, HIVE-13179.patch, HIVE-13179.patch > > > Right now if I want to create an ldap auth provider, I have to create a > hive-site.xml, set endpoints and other relevant properties there, then > instantiate `LdapAuthenticationProviderImpl`, since inside the constructor, a > new HiveConf is constructed. > A better and more reusable design would be to ask for the conf in the > constructor itself. That will allow an external user to create a HiveConf, > set all relevant properties and instantiate `LdapAuthenticationProviderImpl` > with that conf. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13175) Disallow making external tables transactional
[
https://issues.apache.org/jira/browse/HIVE-13175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15175601#comment-15175601
]
Hive QA commented on HIVE-13175:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12790764/HIVE-13175.2.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 9764 tests executed
*Failed tests:*
{noformat}
TestMiniTezCliDriver-vector_distinct_2.q-vector_interval_2.q-load_dyn_part2.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-groupby3_map.q-sample2.q-auto_join14.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-groupby_map_ppr_multi_distinct.q-table_access_keys_stats.q-groupby4_noskew.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join_rc.q-insert1.q-vectorized_rcfile_columnar.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-ppd_join4.q-join9.q-ppd_join3.q-and-12-more - did not
produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7143/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7143/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7143/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12790764 - PreCommit-HIVE-TRUNK-Build
> Disallow making external tables transactional
> -
>
> Key: HIVE-13175
> URL: https://issues.apache.org/jira/browse/HIVE-13175
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.0.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-13175.1.patch, HIVE-13175.2.patch
>
>
> The fact that compactor rewrites contents of ACID tables is in conflict with
> what is expected of external tables.
> Conversely, end user can write to External table which certainly not what is
> expected of ACID table.
> So we should explicitly disallow making an external table ACID.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13178) Enhance ORC Schema Evolution to handle more standard data type conversions
[ https://issues.apache.org/jira/browse/HIVE-13178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-13178: Status: In Progress (was: Patch Available) > Enhance ORC Schema Evolution to handle more standard data type conversions > -- > > Key: HIVE-13178 > URL: https://issues.apache.org/jira/browse/HIVE-13178 > Project: Hive > Issue Type: Bug > Components: Hive, ORC >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-13178.01.patch > > > Currently, SHORT -> INT -> BIGINT is supported. > Handle ORC data type conversions permitted by Implicit conversion allowed by > TypeIntoUtils.implicitConvertible method. >* STRING_GROUP -> DOUBLE >* STRING_GROUP -> DECIMAL >* DATE_GROUP -> STRING >* NUMERIC_GROUP -> STRING >* STRING_GROUP -> STRING_GROUP >* >* // Upward from "lower" type to "higher" numeric type: >* BYTE -> SHORT -> INT -> BIGINT -> FLOAT -> DOUBLE -> DECIMAL -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13129) CliService leaks HMS connection
[
https://issues.apache.org/jira/browse/HIVE-13129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-13129:
Attachment: HIVE-13129.2.patch
Talked to Chaoyu offline. He is planning to use the connection for delegation
token.
So patch-2 moves the closure to the stop() method. The resources would be
released eventually when the HS2 closes, but seems still a good idea to close
them immediately when it's no use especially for the case of multiple HS2
instances.
> CliService leaks HMS connection
> ---
>
> Key: HIVE-13129
> URL: https://issues.apache.org/jira/browse/HIVE-13129
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Affects Versions: 2.1.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-13129.2.patch, HIVE-13129.patch
>
>
> HIVE-12790 fixes the HMS connection leaking. But seems there is one more
> connection from CLIService.
> The init() function in CLIService will get info from DB but we never close
> the HMS connection for this service main thread.
> {noformat}
> // creates connection to HMS and thus *must* occur after kerberos login
> above
> try {
> applyAuthorizationConfigPolicy(hiveConf);
> } catch (Exception e) {
> throw new RuntimeException("Error applying authorization policy on hive
> configuration: "
> + e.getMessage(), e);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13178) Enhance ORC Schema Evolution to handle more standard data type conversions
[ https://issues.apache.org/jira/browse/HIVE-13178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-13178: Attachment: HIVE-13178.02.patch > Enhance ORC Schema Evolution to handle more standard data type conversions > -- > > Key: HIVE-13178 > URL: https://issues.apache.org/jira/browse/HIVE-13178 > Project: Hive > Issue Type: Bug > Components: Hive, ORC >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-13178.01.patch, HIVE-13178.02.patch > > > Currently, SHORT -> INT -> BIGINT is supported. > Handle ORC data type conversions permitted by Implicit conversion allowed by > TypeIntoUtils.implicitConvertible method. >* STRING_GROUP -> DOUBLE >* STRING_GROUP -> DECIMAL >* DATE_GROUP -> STRING >* NUMERIC_GROUP -> STRING >* STRING_GROUP -> STRING_GROUP >* >* // Upward from "lower" type to "higher" numeric type: >* BYTE -> SHORT -> INT -> BIGINT -> FLOAT -> DOUBLE -> DECIMAL -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13178) Enhance ORC Schema Evolution to handle more standard data type conversions
[ https://issues.apache.org/jira/browse/HIVE-13178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-13178: Status: Patch Available (was: In Progress) > Enhance ORC Schema Evolution to handle more standard data type conversions > -- > > Key: HIVE-13178 > URL: https://issues.apache.org/jira/browse/HIVE-13178 > Project: Hive > Issue Type: Bug > Components: Hive, ORC >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-13178.01.patch, HIVE-13178.02.patch > > > Currently, SHORT -> INT -> BIGINT is supported. > Handle ORC data type conversions permitted by Implicit conversion allowed by > TypeIntoUtils.implicitConvertible method. >* STRING_GROUP -> DOUBLE >* STRING_GROUP -> DECIMAL >* DATE_GROUP -> STRING >* NUMERIC_GROUP -> STRING >* STRING_GROUP -> STRING_GROUP >* >* // Upward from "lower" type to "higher" numeric type: >* BYTE -> SHORT -> INT -> BIGINT -> FLOAT -> DOUBLE -> DECIMAL -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13112) Expose Lineage information in case of CTAS
[ https://issues.apache.org/jira/browse/HIVE-13112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Butani updated HIVE-13112: - Status: Patch Available (was: Open) > Expose Lineage information in case of CTAS > -- > > Key: HIVE-13112 > URL: https://issues.apache.org/jira/browse/HIVE-13112 > Project: Hive > Issue Type: Bug >Reporter: Harish Butani >Assignee: Harish Butani > Attachments: HIVE-13112.01.patch > > > This is not happening because Lineage is being exposed by MoveTask which > checks for a LoadTableDesc. In case of CTAS the table is created after the > MoveTask. > Proposed solution is to add a flag to CreateTableDesc to track a CTAS > operation, and have the DDLTask expose the lineage if this flag is set. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13129) CliService leaks HMS connection
[
https://issues.apache.org/jira/browse/HIVE-13129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15175771#comment-15175771
]
Naveen Gangam commented on HIVE-13129:
--
It makes sense to attach the HMS connection lifecycle to the service's
lifecycle.
+1 for me pending pre-commit tests.
> CliService leaks HMS connection
> ---
>
> Key: HIVE-13129
> URL: https://issues.apache.org/jira/browse/HIVE-13129
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Affects Versions: 2.1.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-13129.2.patch, HIVE-13129.patch
>
>
> HIVE-12790 fixes the HMS connection leaking. But seems there is one more
> connection from CLIService.
> The init() function in CLIService will get info from DB but we never close
> the HMS connection for this service main thread.
> {noformat}
> // creates connection to HMS and thus *must* occur after kerberos login
> above
> try {
> applyAuthorizationConfigPolicy(hiveConf);
> } catch (Exception e) {
> throw new RuntimeException("Error applying authorization policy on hive
> configuration: "
> + e.getMessage(), e);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-13185) orc.ReaderImp.ensureOrcFooter() method fails on small text files with IndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HIVE-13185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Illya Yalovyy reassigned HIVE-13185:
Assignee: Illya Yalovyy
> orc.ReaderImp.ensureOrcFooter() method fails on small text files with
> IndexOutOfBoundsException
> ---
>
> Key: HIVE-13185
> URL: https://issues.apache.org/jira/browse/HIVE-13185
> Project: Hive
> Issue Type: Bug
> Components: ORC
>Affects Versions: 2.1.0
>Reporter: Illya Yalovyy
>Assignee: Illya Yalovyy
>
> Steps to reproduce:
> 1. Create a Text source table with one line of data:
> {code}
> create table src (id int);
> insert overwrite table src values (1);
> {code}
> 2. Create a target table:
> {code}
> create table trg (id int);
> {code}
> 3. Try to load small text file to the target table:
> {code}
> load data inpath 'user/hive/warehouse/src/00_0' into table trg;
> {code}
> *Error message:*
> {quote}
> FAILED: SemanticException Unable to load data to destination table. Error:
> java.lang.IndexOutOfBoundsException
> {quote}
> *Stack trace:*
> {noformat}
> org.apache.hadoop.hive.ql.parse.SemanticException: Unable to load data to
> destination table. Error: java.lang.IndexOutOfBoundsException
> at
> org.apache.hadoop.hive.ql.parse.LoadSemanticAnalyzer.ensureFileFormatsMatch(LoadSemanticAnalyzer.java:340)
> at
> org.apache.hadoop.hive.ql.parse.LoadSemanticAnalyzer.analyzeInternal(LoadSemanticAnalyzer.java:224)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:242)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:481)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:317)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1190)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1285)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1116)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1104)
> ...
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12270) Add DBTokenStore support to HS2 delegation token
[ https://issues.apache.org/jira/browse/HIVE-12270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-12270: --- Attachment: HIVE-12270.nothrift.patch > Add DBTokenStore support to HS2 delegation token > > > Key: HIVE-12270 > URL: https://issues.apache.org/jira/browse/HIVE-12270 > Project: Hive > Issue Type: New Feature >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-12270.nothrift.patch > > > DBTokenStore was initially introduced by HIVE-3255 in Hive-0.12 and it is > mainly for HMS delegation token. Later in Hive-0.13, the HS2 delegation token > support was introduced by HIVE-5155 but it used MemoryTokenStore as token > store. That the HIVE-9622 uses the shared RawStore (or HMSHandler) to access > the token/keys information in HMS DB directly from HS2 seems not the right > approach to support DBTokenStore in HS2. I think we should use > HiveMetaStoreClient in HS2 instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12270) Add DBTokenStore support to HS2 delegation token
[ https://issues.apache.org/jira/browse/HIVE-12270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-12270: --- Attachment: HIVE-12270.patch > Add DBTokenStore support to HS2 delegation token > > > Key: HIVE-12270 > URL: https://issues.apache.org/jira/browse/HIVE-12270 > Project: Hive > Issue Type: New Feature >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-12270.nothrift.patch, HIVE-12270.patch > > > DBTokenStore was initially introduced by HIVE-3255 in Hive-0.12 and it is > mainly for HMS delegation token. Later in Hive-0.13, the HS2 delegation token > support was introduced by HIVE-5155 but it used MemoryTokenStore as token > store. That the HIVE-9622 uses the shared RawStore (or HMSHandler) to access > the token/keys information in HMS DB directly from HS2 seems not the right > approach to support DBTokenStore in HS2. I think we should use > HiveMetaStoreClient in HS2 instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12270) Add DBTokenStore support to HS2 delegation token
[ https://issues.apache.org/jira/browse/HIVE-12270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-12270: --- Status: Patch Available (was: Open) [~prasadm], [~ashutoshc], [~sershe], you worked on HS2 delegation token, DBTokenStore etc before, could you review the patch? Thanks > Add DBTokenStore support to HS2 delegation token > > > Key: HIVE-12270 > URL: https://issues.apache.org/jira/browse/HIVE-12270 > Project: Hive > Issue Type: New Feature >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-12270.nothrift.patch, HIVE-12270.patch > > > DBTokenStore was initially introduced by HIVE-3255 in Hive-0.12 and it is > mainly for HMS delegation token. Later in Hive-0.13, the HS2 delegation token > support was introduced by HIVE-5155 but it used MemoryTokenStore as token > store. That the HIVE-9622 uses the shared RawStore (or HMSHandler) to access > the token/keys information in HMS DB directly from HS2 seems not the right > approach to support DBTokenStore in HS2. I think we should use > HiveMetaStoreClient in HS2 instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12270) Add DBTokenStore support to HS2 delegation token
[ https://issues.apache.org/jira/browse/HIVE-12270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-12270: --- Attachment: (was: HIVE-12270.nothrift.patch) > Add DBTokenStore support to HS2 delegation token > > > Key: HIVE-12270 > URL: https://issues.apache.org/jira/browse/HIVE-12270 > Project: Hive > Issue Type: New Feature >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-12270.patch > > > DBTokenStore was initially introduced by HIVE-3255 in Hive-0.12 and it is > mainly for HMS delegation token. Later in Hive-0.13, the HS2 delegation token > support was introduced by HIVE-5155 but it used MemoryTokenStore as token > store. That the HIVE-9622 uses the shared RawStore (or HMSHandler) to access > the token/keys information in HMS DB directly from HS2 seems not the right > approach to support DBTokenStore in HS2. I think we should use > HiveMetaStoreClient in HS2 instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12270) Add DBTokenStore support to HS2 delegation token
[ https://issues.apache.org/jira/browse/HIVE-12270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-12270: --- Attachment: (was: HIVE-12270.patch) > Add DBTokenStore support to HS2 delegation token > > > Key: HIVE-12270 > URL: https://issues.apache.org/jira/browse/HIVE-12270 > Project: Hive > Issue Type: New Feature >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-12270.nothrift.patch > > > DBTokenStore was initially introduced by HIVE-3255 in Hive-0.12 and it is > mainly for HMS delegation token. Later in Hive-0.13, the HS2 delegation token > support was introduced by HIVE-5155 but it used MemoryTokenStore as token > store. That the HIVE-9622 uses the shared RawStore (or HMSHandler) to access > the token/keys information in HMS DB directly from HS2 seems not the right > approach to support DBTokenStore in HS2. I think we should use > HiveMetaStoreClient in HS2 instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12270) Add DBTokenStore support to HS2 delegation token
[ https://issues.apache.org/jira/browse/HIVE-12270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-12270: --- Attachment: HIVE-12270.nothrift.patch > Add DBTokenStore support to HS2 delegation token > > > Key: HIVE-12270 > URL: https://issues.apache.org/jira/browse/HIVE-12270 > Project: Hive > Issue Type: New Feature >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-12270.nothrift.patch > > > DBTokenStore was initially introduced by HIVE-3255 in Hive-0.12 and it is > mainly for HMS delegation token. Later in Hive-0.13, the HS2 delegation token > support was introduced by HIVE-5155 but it used MemoryTokenStore as token > store. That the HIVE-9622 uses the shared RawStore (or HMSHandler) to access > the token/keys information in HMS DB directly from HS2 seems not the right > approach to support DBTokenStore in HS2. I think we should use > HiveMetaStoreClient in HS2 instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12270) Add DBTokenStore support to HS2 delegation token
[ https://issues.apache.org/jira/browse/HIVE-12270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-12270: --- Attachment: HIVE-12270.patch Patch has been uploaded to RB https://reviews.apache.org/r/44271/ > Add DBTokenStore support to HS2 delegation token > > > Key: HIVE-12270 > URL: https://issues.apache.org/jira/browse/HIVE-12270 > Project: Hive > Issue Type: New Feature >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-12270.nothrift.patch, HIVE-12270.patch > > > DBTokenStore was initially introduced by HIVE-3255 in Hive-0.12 and it is > mainly for HMS delegation token. Later in Hive-0.13, the HS2 delegation token > support was introduced by HIVE-5155 but it used MemoryTokenStore as token > store. That the HIVE-9622 uses the shared RawStore (or HMSHandler) to access > the token/keys information in HMS DB directly from HS2 seems not the right > approach to support DBTokenStore in HS2. I think we should use > HiveMetaStoreClient in HS2 instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13069) Enable cartesian product merging
[
https://issues.apache.org/jira/browse/HIVE-13069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15175929#comment-15175929
]
Hive QA commented on HIVE-13069:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12790910/HIVE-13069.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 9765 tests
executed
*Failed tests:*
{noformat}
TestSparkCliDriver-groupby3_map.q-sample2.q-auto_join14.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-groupby_map_ppr_multi_distinct.q-table_access_keys_stats.q-groupby4_noskew.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join_rc.q-insert1.q-vectorized_rcfile_columnar.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-ppd_join4.q-join9.q-ppd_join3.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_coltype_literals
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_stats_only_null
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_stats_only_null
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7144/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7144/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7144/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12790910 - PreCommit-HIVE-TRUNK-Build
> Enable cartesian product merging
>
>
> Key: HIVE-13069
> URL: https://issues.apache.org/jira/browse/HIVE-13069
> Project: Hive
> Issue Type: Improvement
> Components: Parser
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13069.01.patch, HIVE-13069.01.patch,
> HIVE-13069.patch
>
>
> Currently we can merge 2-way joins into n-way joins when the joins are
> executed over the same column.
> In turn, CBO might produce plans containing cartesian products if the join
> columns are constant values; after HIVE-12543 went in, this is rather common,
> as those constant columns are correctly pruned. However, currently we do not
> merge a cartesian product with two inputs into a cartesian product with
> multiple inputs, which could result in performance loss.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13069) Enable cartesian product merging
[ https://issues.apache.org/jira/browse/HIVE-13069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13069: --- Resolution: Fixed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) Regenerated q files and pushed to master. Thanks for the review [~ashutoshc]! > Enable cartesian product merging > > > Key: HIVE-13069 > URL: https://issues.apache.org/jira/browse/HIVE-13069 > Project: Hive > Issue Type: Improvement > Components: Parser >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Fix For: 2.1.0 > > Attachments: HIVE-13069.01.patch, HIVE-13069.01.patch, > HIVE-13069.patch > > > Currently we can merge 2-way joins into n-way joins when the joins are > executed over the same column. > In turn, CBO might produce plans containing cartesian products if the join > columns are constant values; after HIVE-12543 went in, this is rather common, > as those constant columns are correctly pruned. However, currently we do not > merge a cartesian product with two inputs into a cartesian product with > multiple inputs, which could result in performance loss. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-5370) format_number udf should take user specifed format as argument
[ https://issues.apache.org/jira/browse/HIVE-5370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15176089#comment-15176089 ] Ashutosh Chauhan commented on HIVE-5370: Comment on RB. > format_number udf should take user specifed format as argument > -- > > Key: HIVE-5370 > URL: https://issues.apache.org/jira/browse/HIVE-5370 > Project: Hive > Issue Type: Improvement > Components: UDF >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu >Priority: Minor > Fix For: 2.1.0 > > Attachments: D13185.1.patch, D13185.2.patch, HIVE-5370.2.patch, > HIVE-5370.patch, HIVE-5370.patch > > > Currently, format_number udf formats the number to #,###,###.##, but it > should also take a user specified format as optional input. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-4897) Hive should handle AlreadyExists on retries when creating tables/partitions
[
https://issues.apache.org/jira/browse/HIVE-4897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15176132#comment-15176132
]
Aihua Xu commented on HIVE-4897:
[~sershe] Do you still notice the issue happens? I guess we may not have this
issue anymore since before we have unsafe concurrent access to the HMS clients
which could lead to this error.
I went through the code, if some error happens when we create table or
partitions, it should roll back properly. Let me know your thoughts.
> Hive should handle AlreadyExists on retries when creating tables/partitions
> ---
>
> Key: HIVE-4897
> URL: https://issues.apache.org/jira/browse/HIVE-4897
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Aihua Xu
> Attachments: HIVE-4897.patch, hive-snippet.log
>
>
> Creating new tables/partitions may fail with an AlreadyExistsException if
> there is an error part way through the creation and the HMS tries again
> without properly cleaning up or checking if this is a retry.
> While partitioning a new table via a script on distributed hive (MetaStore on
> the same machine) there was a long timeout and then:
> {code}
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask.
> AlreadyExistsException(message:Partition already exists:Partition( ...
> {code}
> I am assuming this is due to retry. Perhaps already-exists on retry could be
> handled better.
> A similar error occurred while creating a table through Impala, which issued
> a single createTable call that failed with an AlreadyExistsException. See the
> logs related to table tmp_proc_8_d2b7b0f133be455ca95615818b8a5879_7 in the
> attached hive-snippet.log
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13063) Create UDFs for CHR and REPLACE
[
https://issues.apache.org/jira/browse/HIVE-13063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15176136#comment-15176136
]
Jason Dere commented on HIVE-13063:
---
+1 if the tests look ok.
> Create UDFs for CHR and REPLACE
>
>
> Key: HIVE-13063
> URL: https://issues.apache.org/jira/browse/HIVE-13063
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 1.2.0
>Reporter: Alejandro Fernandez
>Assignee: Alejandro Fernandez
> Fix For: 2.1.0
>
> Attachments: HIVE-13063.patch, Screen Shot 2016-02-17 at 7.20.57
> PM.png, Screen Shot 2016-02-17 at 7.21.07 PM.png
>
>
> Create UDFS for these functions.
> CHR: convert n where n : [0, 256) into the ascii equivalent as a varchar. If
> n is less than 0 or greater than 255, return the empty string. If n is 0,
> return null.
> REPLACE: replace all substrings of 'str' that match 'search' with 'rep'.
> Example. SELECT REPLACE('Hack and Hue', 'H', 'BL');
> Equals 'BLack and BLue'"
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13175) Disallow making external tables transactional
[ https://issues.apache.org/jira/browse/HIVE-13175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15176203#comment-15176203 ] Wei Zheng commented on HIVE-13175: -- Test failures are not related. [~ekoifman] Can you take a look? > Disallow making external tables transactional > - > > Key: HIVE-13175 > URL: https://issues.apache.org/jira/browse/HIVE-13175 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 2.0.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-13175.1.patch, HIVE-13175.2.patch > > > The fact that compactor rewrites contents of ACID tables is in conflict with > what is expected of external tables. > Conversely, end user can write to External table which certainly not what is > expected of ACID table. > So we should explicitly disallow making an external table ACID. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13175) Disallow making external tables transactional
[ https://issues.apache.org/jira/browse/HIVE-13175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15176304#comment-15176304 ] Eugene Koifman commented on HIVE-13175: --- I think it would be good to make the message include db.table in it and be included in ErrorMsg I would make it and "db.table cannot be declared transactions because it's an external table" or something like that. > Disallow making external tables transactional > - > > Key: HIVE-13175 > URL: https://issues.apache.org/jira/browse/HIVE-13175 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 2.0.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-13175.1.patch, HIVE-13175.2.patch > > > The fact that compactor rewrites contents of ACID tables is in conflict with > what is expected of external tables. > Conversely, end user can write to External table which certainly not what is > expected of ACID table. > So we should explicitly disallow making an external table ACID. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10632) Make sure TXN_COMPONENTS gets cleaned up if table is dropped before compaction.
[
https://issues.apache.org/jira/browse/HIVE-10632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15176340#comment-15176340
]
Hive QA commented on HIVE-10632:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12790774/HIVE-10632.6.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 9780 tests
executed
*Failed tests:*
{noformat}
TestSparkCliDriver-groupby3_map.q-sample2.q-auto_join14.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-groupby_map_ppr_multi_distinct.q-table_access_keys_stats.q-groupby4_noskew.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join_rc.q-insert1.q-vectorized_rcfile_columnar.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-ppd_join4.q-join9.q-ppd_join3.q-and-12-more - did not
produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_coltype_literals
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_stats_only_null
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_stats_only_null
org.apache.hadoop.hive.metastore.TestMetaStoreConnectionUrlHook.testUrlHook
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7145/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7145/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7145/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12790774 - PreCommit-HIVE-TRUNK-Build
> Make sure TXN_COMPONENTS gets cleaned up if table is dropped before
> compaction.
> ---
>
> Key: HIVE-10632
> URL: https://issues.apache.org/jira/browse/HIVE-10632
> Project: Hive
> Issue Type: Bug
> Components: Metastore, Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Wei Zheng
>Priority: Critical
> Attachments: HIVE-10632.1.patch, HIVE-10632.2.patch,
> HIVE-10632.3.patch, HIVE-10632.4.patch, HIVE-10632.5.patch, HIVE-10632.6.patch
>
>
> The compaction process will clean up entries in TXNS,
> COMPLETED_TXN_COMPONENTS, TXN_COMPONENTS. If the table/partition is dropped
> before compaction is complete there will be data left in these tables. Need
> to investigate if there are other situations where this may happen and
> address it.
> see HIVE-10595 for additional info
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13182) row process cause ArrayIndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HIVE-13182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15176342#comment-15176342
]
Yongzhi Chen commented on HIVE-13182:
-
The issue is not reproducible in current master branch(2.1.0), the query
finished with returning values:
{noformat}
+-+---+---+--+--+
1 row selected (22.884 seconds)
16/03/02 14:40:43 DEBUG TSaslTransport: writing data length: 117
| a.date_id |a.gu_id| a.sub_mct_id |
a.track_time |
+-+---+---+--+--+
| 2016-02-22 | -0006-d31e--a4b4eb18 | 110456|
2016-02-22 06:56:12 |
+-+---+---+--+--+
{noformat}
> row process cause ArrayIndexOutOfBoundsException
>
>
> Key: HIVE-13182
> URL: https://issues.apache.org/jira/browse/HIVE-13182
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 1.1.0
> Environment: hadoop 2.6.0 hive on mr
>Reporter: tangjunjie
> Attachments: error.txt, rpt_click.hql, sampledata.txt
>
>
> I run query , then cause ArrayIndexOutOfBoundsException,For detail see
> attachment.
> I digg into the code.
> {code:title=GenMapRedUtils.java|borderStyle=solid}
> //parent.getSchema() is (_col0:string,_col3:string) and parent is
> FilterOperator I don't think that is right, it shoulde be
> (_col0:string,_col1:bigint,_col2:bigint,_col3:string)
> TableDesc tt_desc = PlanUtils.getIntermediateFileTableDesc(PlanUtils
> .getFieldSchemasFromRowSchema(parent.getSchema(), "temporarycol"));
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11837) comments do not support unicode characters well.
[ https://issues.apache.org/jira/browse/HIVE-11837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongzhi Chen updated HIVE-11837: Status: Patch Available (was: Reopened) > comments do not support unicode characters well. > > > Key: HIVE-11837 > URL: https://issues.apache.org/jira/browse/HIVE-11837 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 1.1.0, 0.13.1 > Environment: Hadoop 2.7 > Hive 0.13.1 / Hive 1.1.0 > RHEL 6.4 / SLES 11.3 >Reporter: Rudd Chen >Assignee: Rudd Chen >Priority: Minor > Attachments: HIVE-11837.patch > > > the terminal encoding is set to UTF-8, It can display Chinese characters. > then I create a table with a comment in Chinese, both "show create table" and > "desc formatted table" can not display the Chinese characters in the table > comments, meanwhile it can display Chinese characters in column comment.. See > below: > 0: jdbc:hive2://ha-cluster/default> create table tt(id int comment '列中文测试') > comment '表中文测试'; > No rows affected (0.152 seconds) > 0: jdbc:hive2://ha-cluster/default> > 0: jdbc:hive2://ha-cluster/default> > 0: jdbc:hive2://ha-cluster/default> desc formatted tt; > +---+---+-+ > | col_name| data_type > | comment | > +---+---+-+ > | # col_name| data_type > | comment | > | | NULL > | NULL| > | id| int > | 列中文测试 | > | | NULL > | NULL| > | # Detailed Table Information | NULL > | NULL| > | Database: | default > | NULL| > | Owner:| admin > | NULL| > | CreateTime: | Wed Sep 16 11:13:34 CST 2015 > | NULL| > | LastAccessTime: | UNKNOWN > | NULL| > | Protect Mode: | None > | NULL| > | Retention:| 0 > | NULL| > | Location: | hdfs://hacluster/user/hive/warehouse/tt > | NULL| > | Table Type: | MANAGED_TABLE > | NULL| > | Table Parameters: | NULL > | NULL| > | | comment > | \u8868\u4E2D\u6587\u6D4B\u8BD5 | > | | transient_lastDdlTime > | 1442373214 | > | | NULL > | NULL| > | # Storage Information | NULL > | NULL| > | SerDe Library:| > org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe | NULL > | > | InputFormat: | > org.apache.hadoop.hive.ql.io.RCFileInputFormat| NULL > | > | OutputFormat: | > org.apache.hadoop.hive.ql.io.RCFileOutputFormat | NULL > | > | Compressed: | No > | NULL| > | Num Buckets: | -1 > | NULL| > | Bucket Columns: | [] > | NULL| > | Sort Columns: | [] > | NULL
[jira] [Updated] (HIVE-11837) comments do not support unicode characters well.
[ https://issues.apache.org/jira/browse/HIVE-11837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongzhi Chen updated HIVE-11837: Assignee: Rudd Chen > comments do not support unicode characters well. > > > Key: HIVE-11837 > URL: https://issues.apache.org/jira/browse/HIVE-11837 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 0.13.1, 1.1.0 > Environment: Hadoop 2.7 > Hive 0.13.1 / Hive 1.1.0 > RHEL 6.4 / SLES 11.3 >Reporter: Rudd Chen >Assignee: Rudd Chen >Priority: Minor > Attachments: HIVE-11837.patch > > > the terminal encoding is set to UTF-8, It can display Chinese characters. > then I create a table with a comment in Chinese, both "show create table" and > "desc formatted table" can not display the Chinese characters in the table > comments, meanwhile it can display Chinese characters in column comment.. See > below: > 0: jdbc:hive2://ha-cluster/default> create table tt(id int comment '列中文测试') > comment '表中文测试'; > No rows affected (0.152 seconds) > 0: jdbc:hive2://ha-cluster/default> > 0: jdbc:hive2://ha-cluster/default> > 0: jdbc:hive2://ha-cluster/default> desc formatted tt; > +---+---+-+ > | col_name| data_type > | comment | > +---+---+-+ > | # col_name| data_type > | comment | > | | NULL > | NULL| > | id| int > | 列中文测试 | > | | NULL > | NULL| > | # Detailed Table Information | NULL > | NULL| > | Database: | default > | NULL| > | Owner:| admin > | NULL| > | CreateTime: | Wed Sep 16 11:13:34 CST 2015 > | NULL| > | LastAccessTime: | UNKNOWN > | NULL| > | Protect Mode: | None > | NULL| > | Retention:| 0 > | NULL| > | Location: | hdfs://hacluster/user/hive/warehouse/tt > | NULL| > | Table Type: | MANAGED_TABLE > | NULL| > | Table Parameters: | NULL > | NULL| > | | comment > | \u8868\u4E2D\u6587\u6D4B\u8BD5 | > | | transient_lastDdlTime > | 1442373214 | > | | NULL > | NULL| > | # Storage Information | NULL > | NULL| > | SerDe Library:| > org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe | NULL > | > | InputFormat: | > org.apache.hadoop.hive.ql.io.RCFileInputFormat| NULL > | > | OutputFormat: | > org.apache.hadoop.hive.ql.io.RCFileOutputFormat | NULL > | > | Compressed: | No > | NULL| > | Num Buckets: | -1 > | NULL| > | Bucket Columns: | [] > | NULL| > | Sort Columns: | [] > | NULL| > | Storage Des
[jira] [Updated] (HIVE-11837) comments do not support unicode characters well.
[ https://issues.apache.org/jira/browse/HIVE-11837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongzhi Chen updated HIVE-11837: Status: Open (was: Patch Available) > comments do not support unicode characters well. > > > Key: HIVE-11837 > URL: https://issues.apache.org/jira/browse/HIVE-11837 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 1.1.0, 0.13.1 > Environment: Hadoop 2.7 > Hive 0.13.1 / Hive 1.1.0 > RHEL 6.4 / SLES 11.3 >Reporter: Rudd Chen >Assignee: Rudd Chen >Priority: Minor > Attachments: HIVE-11837.patch > > > the terminal encoding is set to UTF-8, It can display Chinese characters. > then I create a table with a comment in Chinese, both "show create table" and > "desc formatted table" can not display the Chinese characters in the table > comments, meanwhile it can display Chinese characters in column comment.. See > below: > 0: jdbc:hive2://ha-cluster/default> create table tt(id int comment '列中文测试') > comment '表中文测试'; > No rows affected (0.152 seconds) > 0: jdbc:hive2://ha-cluster/default> > 0: jdbc:hive2://ha-cluster/default> > 0: jdbc:hive2://ha-cluster/default> desc formatted tt; > +---+---+-+ > | col_name| data_type > | comment | > +---+---+-+ > | # col_name| data_type > | comment | > | | NULL > | NULL| > | id| int > | 列中文测试 | > | | NULL > | NULL| > | # Detailed Table Information | NULL > | NULL| > | Database: | default > | NULL| > | Owner:| admin > | NULL| > | CreateTime: | Wed Sep 16 11:13:34 CST 2015 > | NULL| > | LastAccessTime: | UNKNOWN > | NULL| > | Protect Mode: | None > | NULL| > | Retention:| 0 > | NULL| > | Location: | hdfs://hacluster/user/hive/warehouse/tt > | NULL| > | Table Type: | MANAGED_TABLE > | NULL| > | Table Parameters: | NULL > | NULL| > | | comment > | \u8868\u4E2D\u6587\u6D4B\u8BD5 | > | | transient_lastDdlTime > | 1442373214 | > | | NULL > | NULL| > | # Storage Information | NULL > | NULL| > | SerDe Library:| > org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe | NULL > | > | InputFormat: | > org.apache.hadoop.hive.ql.io.RCFileInputFormat| NULL > | > | OutputFormat: | > org.apache.hadoop.hive.ql.io.RCFileOutputFormat | NULL > | > | Compressed: | No > | NULL| > | Num Buckets: | -1 > | NULL| > | Bucket Columns: | [] > | NULL| > | Sort Columns: | [] > | NULL
[jira] [Commented] (HIVE-13192) OOM error when MapJoinBytesTableContainer memory threshold is too low
[
https://issues.apache.org/jira/browse/HIVE-13192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15176442#comment-15176442
]
Wei Zheng commented on HIVE-13192:
--
HIVE-11587 might be related/helpful for this issue
> OOM error when MapJoinBytesTableContainer memory threshold is too low
> --
>
> Key: HIVE-13192
> URL: https://issues.apache.org/jira/browse/HIVE-13192
> Project: Hive
> Issue Type: Bug
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-13192.1.patch
>
>
> HIVE-11449 covered the scenario with HybridHashTableContainer, but we can
> still run into the error as shown below:
> {code}
> Vertex failed, vertexName=Map 1, vertexId=vertex_1454464706407_0225_1_19,
> diagnostics=[Task failed, taskId=task_1454464706407_0225_1_19_00,
> diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running
> task:java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:172)
> at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:138)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:333)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:177)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:168)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:168)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:163)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.hadoop.hive.serde2.WriteBuffers.nextBufferToWrite(WriteBuffers.java:206)
> at org.apache.hadoop.hive.serde2.WriteBuffers.write(WriteBuffers.java:182)
> at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryUtils.writeVLong(LazyBinaryUtils.java:411)
> at
> org.apache.hadoop.hive.serde2.WriteBuffers.writeVLong(WriteBuffers.java:294)
> at
> org.apache.hadoop.hive.ql.exec.persistence.BytesBytesMultiHashMap.addRecordToList(BytesBytesMultiHashMap.java:540)
> at
> org.apache.hadoop.hive.ql.exec.persistence.BytesBytesMultiHashMap.put(BytesBytesMultiHashMap.java:219)
> at
> org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer.putRow(MapJoinBytesTableContainer.java:285)
> at
> org.apache.hadoop.hive.ql.exec.tez.HashTableLoader.load(HashTableLoader.java:114)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:190)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:216)
> at
> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1051)
> at
> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1055)
> at
> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1055)
> at
> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1055)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:37)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:83)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:316)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:163)
> ... 13 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11837) comments do not support unicode characters well.
[ https://issues.apache.org/jira/browse/HIVE-11837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongzhi Chen updated HIVE-11837: Attachment: HIVE-11837.1.patch The original fix is outdated. Attach a new fix. > comments do not support unicode characters well. > > > Key: HIVE-11837 > URL: https://issues.apache.org/jira/browse/HIVE-11837 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 0.13.1, 1.1.0 > Environment: Hadoop 2.7 > Hive 0.13.1 / Hive 1.1.0 > RHEL 6.4 / SLES 11.3 >Reporter: Rudd Chen >Assignee: Rudd Chen >Priority: Minor > Attachments: HIVE-11837.1.patch, HIVE-11837.patch > > > the terminal encoding is set to UTF-8, It can display Chinese characters. > then I create a table with a comment in Chinese, both "show create table" and > "desc formatted table" can not display the Chinese characters in the table > comments, meanwhile it can display Chinese characters in column comment.. See > below: > 0: jdbc:hive2://ha-cluster/default> create table tt(id int comment '列中文测试') > comment '表中文测试'; > No rows affected (0.152 seconds) > 0: jdbc:hive2://ha-cluster/default> > 0: jdbc:hive2://ha-cluster/default> > 0: jdbc:hive2://ha-cluster/default> desc formatted tt; > +---+---+-+ > | col_name| data_type > | comment | > +---+---+-+ > | # col_name| data_type > | comment | > | | NULL > | NULL| > | id| int > | 列中文测试 | > | | NULL > | NULL| > | # Detailed Table Information | NULL > | NULL| > | Database: | default > | NULL| > | Owner:| admin > | NULL| > | CreateTime: | Wed Sep 16 11:13:34 CST 2015 > | NULL| > | LastAccessTime: | UNKNOWN > | NULL| > | Protect Mode: | None > | NULL| > | Retention:| 0 > | NULL| > | Location: | hdfs://hacluster/user/hive/warehouse/tt > | NULL| > | Table Type: | MANAGED_TABLE > | NULL| > | Table Parameters: | NULL > | NULL| > | | comment > | \u8868\u4E2D\u6587\u6D4B\u8BD5 | > | | transient_lastDdlTime > | 1442373214 | > | | NULL > | NULL| > | # Storage Information | NULL > | NULL| > | SerDe Library:| > org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe | NULL > | > | InputFormat: | > org.apache.hadoop.hive.ql.io.RCFileInputFormat| NULL > | > | OutputFormat: | > org.apache.hadoop.hive.ql.io.RCFileOutputFormat | NULL > | > | Compressed: | No > | NULL| > | Num Buckets: | -1 > | NULL| > | Bucket Columns: | [] > | NULL| > | Sort Columns: | []
[jira] [Updated] (HIVE-11837) comments do not support unicode characters well.
[ https://issues.apache.org/jira/browse/HIVE-11837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongzhi Chen updated HIVE-11837: Status: Patch Available (was: Open) > comments do not support unicode characters well. > > > Key: HIVE-11837 > URL: https://issues.apache.org/jira/browse/HIVE-11837 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 1.1.0, 0.13.1 > Environment: Hadoop 2.7 > Hive 0.13.1 / Hive 1.1.0 > RHEL 6.4 / SLES 11.3 >Reporter: Rudd Chen >Assignee: Rudd Chen >Priority: Minor > Attachments: HIVE-11837.1.patch, HIVE-11837.patch > > > the terminal encoding is set to UTF-8, It can display Chinese characters. > then I create a table with a comment in Chinese, both "show create table" and > "desc formatted table" can not display the Chinese characters in the table > comments, meanwhile it can display Chinese characters in column comment.. See > below: > 0: jdbc:hive2://ha-cluster/default> create table tt(id int comment '列中文测试') > comment '表中文测试'; > No rows affected (0.152 seconds) > 0: jdbc:hive2://ha-cluster/default> > 0: jdbc:hive2://ha-cluster/default> > 0: jdbc:hive2://ha-cluster/default> desc formatted tt; > +---+---+-+ > | col_name| data_type > | comment | > +---+---+-+ > | # col_name| data_type > | comment | > | | NULL > | NULL| > | id| int > | 列中文测试 | > | | NULL > | NULL| > | # Detailed Table Information | NULL > | NULL| > | Database: | default > | NULL| > | Owner:| admin > | NULL| > | CreateTime: | Wed Sep 16 11:13:34 CST 2015 > | NULL| > | LastAccessTime: | UNKNOWN > | NULL| > | Protect Mode: | None > | NULL| > | Retention:| 0 > | NULL| > | Location: | hdfs://hacluster/user/hive/warehouse/tt > | NULL| > | Table Type: | MANAGED_TABLE > | NULL| > | Table Parameters: | NULL > | NULL| > | | comment > | \u8868\u4E2D\u6587\u6D4B\u8BD5 | > | | transient_lastDdlTime > | 1442373214 | > | | NULL > | NULL| > | # Storage Information | NULL > | NULL| > | SerDe Library:| > org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe | NULL > | > | InputFormat: | > org.apache.hadoop.hive.ql.io.RCFileInputFormat| NULL > | > | OutputFormat: | > org.apache.hadoop.hive.ql.io.RCFileOutputFormat | NULL > | > | Compressed: | No > | NULL| > | Num Buckets: | -1 > | NULL| > | Bucket Columns: | [] > | NULL| > | Sort Columns: | [] > | NULL
[jira] [Updated] (HIVE-10632) Make sure TXN_COMPONENTS gets cleaned up if table is dropped before compaction.
[ https://issues.apache.org/jira/browse/HIVE-10632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-10632: - Attachment: HIVE-10632.7.patch patch 7 for test. Fixed the failure for TestMetaStoreConnectionUrlHook.testUrlHook by making txnHandler initialization in AcidEventListener lazy. > Make sure TXN_COMPONENTS gets cleaned up if table is dropped before > compaction. > --- > > Key: HIVE-10632 > URL: https://issues.apache.org/jira/browse/HIVE-10632 > Project: Hive > Issue Type: Bug > Components: Metastore, Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Critical > Attachments: HIVE-10632.1.patch, HIVE-10632.2.patch, > HIVE-10632.3.patch, HIVE-10632.4.patch, HIVE-10632.5.patch, > HIVE-10632.6.patch, HIVE-10632.7.patch > > > The compaction process will clean up entries in TXNS, > COMPLETED_TXN_COMPONENTS, TXN_COMPONENTS. If the table/partition is dropped > before compaction is complete there will be data left in these tables. Need > to investigate if there are other situations where this may happen and > address it. > see HIVE-10595 for additional info -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13189) Consider using Joda DateTimeFormatter instead of SimpleDateFormat in GenericUDFDateAdd
[
https://issues.apache.org/jira/browse/HIVE-13189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15176579#comment-15176579
]
Rajesh Balamohan commented on HIVE-13189:
-
[~varunk44] - I have a patch which is tested with lineitem table in tpch
dataset. Plz let me know if I can post it as a first cut.
> Consider using Joda DateTimeFormatter instead of SimpleDateFormat in
> GenericUDFDateAdd
> --
>
> Key: HIVE-13189
> URL: https://issues.apache.org/jira/browse/HIVE-13189
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Rajesh Balamohan
>Assignee: varun a kumar
>
> Quite an amount was spent by tasks in trying to parse date string in
> GenericUDFDateAdd.
> {noformat}
> java.lang.Thread.State: RUNNABLE
> at java.text.DecimalFormat.subparse(DecimalFormat.java:1467)
> at java.text.DecimalFormat.parse(DecimalFormat.java:1268)
> at java.text.SimpleDateFormat.subParse(SimpleDateFormat.java:2088)
> at java.text.SimpleDateFormat.parse(SimpleDateFormat.java:1455)
> at java.text.DateFormat.parse(DateFormat.java:355)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFDateAdd.evaluate(GenericUDFDateAdd.java:172)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:186)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator$DeferredExprObject.get(ExprNodeGenericFuncEvaluator.java:87)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPGreaterThan.evaluate(GenericUDFOPGreaterThan.java:80)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:186)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:65)
> at
> org.apache.hadoop.hive.ql.exec.FilterOperator.process(FilterOperator.java:108)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:838)
> at
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator.internalForward(CommonJoinOperator.java:644)
> {noformat}
> Joda DateTimeFormatter can be considered for better performance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-12484) Show meta operations on HS2 web UI
[ https://issues.apache.org/jira/browse/HIVE-12484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal reassigned HIVE-12484: -- Assignee: Mohit Sabharwal > Show meta operations on HS2 web UI > -- > > Key: HIVE-12484 > URL: https://issues.apache.org/jira/browse/HIVE-12484 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Reporter: Jimmy Xiang >Assignee: Mohit Sabharwal > > As Mohit pointed out in the review of HIVE-12338, it is nice to show meta > operations on HS2 web UI too. So that we can have an end-to-end picture for > those operations access HMS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13175) Disallow making external tables transactional
[ https://issues.apache.org/jira/browse/HIVE-13175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-13175: - Attachment: HIVE-13175.3.patch patch 3 for test. [~ekoifman] I updated the error message to include db and tbl name, but looks like ErrorMsg is mainly for ql module and nothing in metastore is using that. So I still use the plain error message. > Disallow making external tables transactional > - > > Key: HIVE-13175 > URL: https://issues.apache.org/jira/browse/HIVE-13175 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 2.0.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-13175.1.patch, HIVE-13175.2.patch, > HIVE-13175.3.patch > > > The fact that compactor rewrites contents of ACID tables is in conflict with > what is expected of external tables. > Conversely, end user can write to External table which certainly not what is > expected of ACID table. > So we should explicitly disallow making an external table ACID. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12481) Occasionally "Request is a replay" will be thrown from HS2
[
https://issues.apache.org/jira/browse/HIVE-12481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-12481:
Attachment: HIVE-12481.3.patch
Attached the patch-3 to retry from the JDBC client side:
{{beeline -u
"jdbc:hive2://localhost:1/default;principal=;retries=3"}}
> Occasionally "Request is a replay" will be thrown from HS2
> --
>
> Key: HIVE-12481
> URL: https://issues.apache.org/jira/browse/HIVE-12481
> Project: Hive
> Issue Type: Improvement
> Components: Authentication
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-12481.2.patch, HIVE-12481.3.patch, HIVE-12481.patch
>
>
> We have seen the following exception thrown from HS2 in secured cluster when
> many queries are running simultaneously on single HS2 instance.
> The cause I can guess is that it happens that two queries are submitted at
> the same time and have the same timestamp. For such case, we can add a retry
> for the query.
>
> {noformat}
> 2015-11-18 16:12:33,117 ERROR org.apache.thrift.transport.TSaslTransport:
> SASL negotiation failure
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: Failure unspecified at GSS-API level (Mechanism level: Request
> is a replay (34))]
> at
> com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:177)
> at
> org.apache.thrift.transport.TSaslTransport$SaslParticipant.evaluateChallengeOrResponse(TSaslTransport.java:539)
> at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:283)
> at
> org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
> at
> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:739)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:736)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:356)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1651)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:736)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:268)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: GSSException: Failure unspecified at GSS-API level (Mechanism
> level: Request is a replay (34))
> at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:788)
> at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:342)
> at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285)
> at
> com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:155)
> ... 14 more
> Caused by: KrbException: Request is a replay (34)
> at sun.security.krb5.KrbApReq.authenticate(KrbApReq.java:308)
> at sun.security.krb5.KrbApReq.(KrbApReq.java:144)
> at
> sun.security.jgss.krb5.InitSecContextToken.(InitSecContextToken.java:108)
> at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:771)
> ... 17 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12481) Occasionally "Request is a replay" will be thrown from HS2
[
https://issues.apache.org/jira/browse/HIVE-12481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-12481:
Status: Patch Available (was: Reopened)
> Occasionally "Request is a replay" will be thrown from HS2
> --
>
> Key: HIVE-12481
> URL: https://issues.apache.org/jira/browse/HIVE-12481
> Project: Hive
> Issue Type: Improvement
> Components: Authentication
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-12481.2.patch, HIVE-12481.3.patch, HIVE-12481.patch
>
>
> We have seen the following exception thrown from HS2 in secured cluster when
> many queries are running simultaneously on single HS2 instance.
> The cause I can guess is that it happens that two queries are submitted at
> the same time and have the same timestamp. For such case, we can add a retry
> for the query.
>
> {noformat}
> 2015-11-18 16:12:33,117 ERROR org.apache.thrift.transport.TSaslTransport:
> SASL negotiation failure
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: Failure unspecified at GSS-API level (Mechanism level: Request
> is a replay (34))]
> at
> com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:177)
> at
> org.apache.thrift.transport.TSaslTransport$SaslParticipant.evaluateChallengeOrResponse(TSaslTransport.java:539)
> at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:283)
> at
> org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
> at
> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:739)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:736)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:356)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1651)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:736)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:268)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: GSSException: Failure unspecified at GSS-API level (Mechanism
> level: Request is a replay (34))
> at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:788)
> at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:342)
> at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285)
> at
> com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:155)
> ... 14 more
> Caused by: KrbException: Request is a replay (34)
> at sun.security.krb5.KrbApReq.authenticate(KrbApReq.java:308)
> at sun.security.krb5.KrbApReq.(KrbApReq.java:144)
> at
> sun.security.jgss.krb5.InitSecContextToken.(InitSecContextToken.java:108)
> at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:771)
> ... 17 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11550) ACID queries pollute HiveConf
[ https://issues.apache.org/jira/browse/HIVE-11550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-11550: -- Description: HiveConf is a SessionState level object. Some ACID related logic makes changes to it (which are meant to be per query) but become permanent. See SemanticAnalyzer.checkAcidConstraints() Also note HiveConf.setVar(conf, HiveConf.ConfVars.DYNAMICPARTITIONINGMODE, "nonstrict"); in UpdateDeleteSemancitAnalzyer [~alangates], do you know of other cases or ideas on how to deal with this differently? was: HiveConf is a SessionState level object. Some ACID related logic makes changes to it (which are meant to be per query) but become permanent. See SemanticAnalyzer.checkAcidConstraints() [~alangates], do you know of other cases or ideas on how to deal with this differently? > ACID queries pollute HiveConf > - > > Key: HIVE-11550 > URL: https://issues.apache.org/jira/browse/HIVE-11550 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > > HiveConf is a SessionState level object. Some ACID related logic makes > changes to it (which are meant to be per query) but become permanent. > See SemanticAnalyzer.checkAcidConstraints() > Also note HiveConf.setVar(conf, > HiveConf.ConfVars.DYNAMICPARTITIONINGMODE, "nonstrict"); > in UpdateDeleteSemancitAnalzyer > [~alangates], do you know of other cases or ideas on how to deal with this > differently? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13175) Disallow making external tables transactional
[ https://issues.apache.org/jira/browse/HIVE-13175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15176660#comment-15176660 ] Eugene Koifman commented on HIVE-13175: --- does this work if one does an ALTER TABLE on external table to add transactional=true? > Disallow making external tables transactional > - > > Key: HIVE-13175 > URL: https://issues.apache.org/jira/browse/HIVE-13175 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 2.0.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-13175.1.patch, HIVE-13175.2.patch, > HIVE-13175.3.patch > > > The fact that compactor rewrites contents of ACID tables is in conflict with > what is expected of external tables. > Conversely, end user can write to External table which certainly not what is > expected of ACID table. > So we should explicitly disallow making an external table ACID. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-13171) Add unit test for hs2 webui
[ https://issues.apache.org/jira/browse/HIVE-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho reassigned HIVE-13171: Assignee: Szehon Ho > Add unit test for hs2 webui > --- > > Key: HIVE-13171 > URL: https://issues.apache.org/jira/browse/HIVE-13171 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Reporter: Szehon Ho >Assignee: Szehon Ho > > With more complex changes going into webui, it is hard to manually verify all > the kinds of cases. > With HIVE-12952, HS2 webui now uses jamon, which should be more unit-testable > than plain old jsp. We can perhaps add unit test for the jamon servlets, or > test the new OperationDisplay classes queried by the servlets. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13175) Disallow making external tables transactional
[ https://issues.apache.org/jira/browse/HIVE-13175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15176668#comment-15176668 ] Wei Zheng commented on HIVE-13175: -- Good catch, I should make the same change in handleAlterTableTransactionalProp. Will do that. > Disallow making external tables transactional > - > > Key: HIVE-13175 > URL: https://issues.apache.org/jira/browse/HIVE-13175 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 2.0.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-13175.1.patch, HIVE-13175.2.patch, > HIVE-13175.3.patch > > > The fact that compactor rewrites contents of ACID tables is in conflict with > what is expected of external tables. > Conversely, end user can write to External table which certainly not what is > expected of ACID table. > So we should explicitly disallow making an external table ACID. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13149) Remove some unnecessary HMS connections from HS2
[
https://issues.apache.org/jira/browse/HIVE-13149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15176692#comment-15176692
]
Hive QA commented on HIVE-13149:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12790788/HIVE-13149.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 21 failed/errored test(s), 9753 tests
executed
*Failed tests:*
{noformat}
TestJdbcWithMiniHS2 - did not produce a TEST-*.xml file
TestMsgBusConnection - did not produce a TEST-*.xml file
TestSparkCliDriver-groupby3_map.q-sample2.q-auto_join14.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-groupby_map_ppr_multi_distinct.q-table_access_keys_stats.q-groupby4_noskew.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join_rc.q-insert1.q-vectorized_rcfile_columnar.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-ppd_join4.q-join9.q-ppd_join3.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_stats
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_escape_clusterby1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_explain_rearrange
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_limit_pushdown_negative
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_lock2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_conv
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_rpad
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.metastore.TestMetastoreVersion.testVersionRestriction
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7146/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7146/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7146/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 21 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12790788 - PreCommit-HIVE-TRUNK-Build
> Remove some unnecessary HMS connections from HS2
> -
>
> Key: HIVE-13149
> URL: https://issues.apache.org/jira/browse/HIVE-13149
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-13149.1.patch, HIVE-13149.2.patch
>
>
> In SessionState class, currently we will always try to get a HMS connection
> in {{start(SessionState startSs, boolean isAsync, LogHelper console)}}
> regardless of if the connection will be used later or not.
> When SessionState is accessed by the tasks in TaskRunner.java, although most
> of the tasks other than some like StatsTask, don't need to access HMS.
> Currently a new HMS connection will be established for each Task thread. If
> HiveServer2 is configured to run in parallel and the query involves many
> tasks, then the connections are created but unused.
> {noformat}
> @Override
> public void run() {
> runner = Thread.currentThread();
> try {
> OperationLog.setCurrentOperationLog(operationLog);
> SessionState.start(ss);
> runSequential();
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13149) Remove some unnecessary HMS connections from HS2
[
https://issues.apache.org/jira/browse/HIVE-13149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15176744#comment-15176744
]
Szehon Ho commented on HIVE-13149:
--
Hey Aihua the fix looks good.. I'm just wondering some questions..
1. Do we really need another method 'Hive.setConf' that does the same thing as
Hive.get? I agree it's pretty confusing already, but now the class might be
more confusing with two methods named differently that do the same thing?
2. So now we dont initialize MSC any longer with SessionState.start, when does
MSC get initialized now?
> Remove some unnecessary HMS connections from HS2
> -
>
> Key: HIVE-13149
> URL: https://issues.apache.org/jira/browse/HIVE-13149
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-13149.1.patch, HIVE-13149.2.patch
>
>
> In SessionState class, currently we will always try to get a HMS connection
> in {{start(SessionState startSs, boolean isAsync, LogHelper console)}}
> regardless of if the connection will be used later or not.
> When SessionState is accessed by the tasks in TaskRunner.java, although most
> of the tasks other than some like StatsTask, don't need to access HMS.
> Currently a new HMS connection will be established for each Task thread. If
> HiveServer2 is configured to run in parallel and the query involves many
> tasks, then the connections are created but unused.
> {noformat}
> @Override
> public void run() {
> runner = Thread.currentThread();
> try {
> OperationLog.setCurrentOperationLog(operationLog);
> SessionState.start(ss);
> runSequential();
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-4897) Hive should handle AlreadyExists on retries when creating tables/partitions
[
https://issues.apache.org/jira/browse/HIVE-4897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15176759#comment-15176759
]
Sergey Shelukhin commented on HIVE-4897:
I've seen this error fairly recently. It happens if the response was not
delivered to client due to a network problem (or theoretically due to timing
issue with retry it can also happen, if the retry is done after timeout but
before the corresponding timeout on the server, and the original request
finishes before the retry is processed.
I've also seen it happen when the connection to underlying DB was lost in
commitTxn, but the commit still happened (that one time was due to BoneCP
connection-closing bug, but it could presumably also happen because of a
connection issue). commitTxn fails, but the table is already created.
> Hive should handle AlreadyExists on retries when creating tables/partitions
> ---
>
> Key: HIVE-4897
> URL: https://issues.apache.org/jira/browse/HIVE-4897
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Aihua Xu
> Attachments: HIVE-4897.patch, hive-snippet.log
>
>
> Creating new tables/partitions may fail with an AlreadyExistsException if
> there is an error part way through the creation and the HMS tries again
> without properly cleaning up or checking if this is a retry.
> While partitioning a new table via a script on distributed hive (MetaStore on
> the same machine) there was a long timeout and then:
> {code}
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask.
> AlreadyExistsException(message:Partition already exists:Partition( ...
> {code}
> I am assuming this is due to retry. Perhaps already-exists on retry could be
> handled better.
> A similar error occurred while creating a table through Impala, which issued
> a single createTable call that failed with an AlreadyExistsException. See the
> logs related to table tmp_proc_8_d2b7b0f133be455ca95615818b8a5879_7 in the
> attached hive-snippet.log
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13175) Disallow making external tables transactional
[ https://issues.apache.org/jira/browse/HIVE-13175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-13175: - Attachment: HIVE-13175.4.patch patch 4 for test > Disallow making external tables transactional > - > > Key: HIVE-13175 > URL: https://issues.apache.org/jira/browse/HIVE-13175 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 2.0.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-13175.1.patch, HIVE-13175.2.patch, > HIVE-13175.3.patch, HIVE-13175.4.patch > > > The fact that compactor rewrites contents of ACID tables is in conflict with > what is expected of external tables. > Conversely, end user can write to External table which certainly not what is > expected of ACID table. > So we should explicitly disallow making an external table ACID. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13178) Enhance ORC Schema Evolution to handle more standard data type conversions
[ https://issues.apache.org/jira/browse/HIVE-13178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-13178: Attachment: (was: HIVE-13178.02.patch) > Enhance ORC Schema Evolution to handle more standard data type conversions > -- > > Key: HIVE-13178 > URL: https://issues.apache.org/jira/browse/HIVE-13178 > Project: Hive > Issue Type: Bug > Components: Hive, ORC >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-13178.01.patch > > > Currently, SHORT -> INT -> BIGINT is supported. > Handle ORC data type conversions permitted by Implicit conversion allowed by > TypeIntoUtils.implicitConvertible method. >* STRING_GROUP -> DOUBLE >* STRING_GROUP -> DECIMAL >* DATE_GROUP -> STRING >* NUMERIC_GROUP -> STRING >* STRING_GROUP -> STRING_GROUP >* >* // Upward from "lower" type to "higher" numeric type: >* BYTE -> SHORT -> INT -> BIGINT -> FLOAT -> DOUBLE -> DECIMAL -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13178) Enhance ORC Schema Evolution to handle more standard data type conversions
[ https://issues.apache.org/jira/browse/HIVE-13178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-13178: Attachment: HIVE-13178.02.patch > Enhance ORC Schema Evolution to handle more standard data type conversions > -- > > Key: HIVE-13178 > URL: https://issues.apache.org/jira/browse/HIVE-13178 > Project: Hive > Issue Type: Bug > Components: Hive, ORC >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-13178.01.patch, HIVE-13178.02.patch > > > Currently, SHORT -> INT -> BIGINT is supported. > Handle ORC data type conversions permitted by Implicit conversion allowed by > TypeIntoUtils.implicitConvertible method. >* STRING_GROUP -> DOUBLE >* STRING_GROUP -> DECIMAL >* DATE_GROUP -> STRING >* NUMERIC_GROUP -> STRING >* STRING_GROUP -> STRING_GROUP >* >* // Upward from "lower" type to "higher" numeric type: >* BYTE -> SHORT -> INT -> BIGINT -> FLOAT -> DOUBLE -> DECIMAL -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13182) row process cause ArrayIndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HIVE-13182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15176830#comment-15176830
]
tangjunjie commented on HIVE-13182:
---
Is your intermediate result format is SequenceFile ?
> row process cause ArrayIndexOutOfBoundsException
>
>
> Key: HIVE-13182
> URL: https://issues.apache.org/jira/browse/HIVE-13182
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 1.1.0
> Environment: hadoop 2.6.0 hive on mr
>Reporter: tangjunjie
> Attachments: error.txt, rpt_click.hql, sampledata.txt
>
>
> I run query , then cause ArrayIndexOutOfBoundsException,For detail see
> attachment.
> I digg into the code.
> {code:title=GenMapRedUtils.java|borderStyle=solid}
> //parent.getSchema() is (_col0:string,_col3:string) and parent is
> FilterOperator I don't think that is right, it shoulde be
> (_col0:string,_col1:bigint,_col2:bigint,_col3:string)
> TableDesc tt_desc = PlanUtils.getIntermediateFileTableDesc(PlanUtils
> .getFieldSchemasFromRowSchema(parent.getSchema(), "temporarycol"));
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12994) Implement support for NULLS FIRST/NULLS LAST
[ https://issues.apache.org/jira/browse/HIVE-12994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lefty Leverenz updated HIVE-12994: -- Labels: TODOC2.1 (was: ) > Implement support for NULLS FIRST/NULLS LAST > > > Key: HIVE-12994 > URL: https://issues.apache.org/jira/browse/HIVE-12994 > Project: Hive > Issue Type: New Feature > Components: CBO, Parser, Serializers/Deserializers >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Labels: TODOC2.1 > Fix For: 2.1.0 > > Attachments: HIVE-12994.01.patch, HIVE-12994.02.patch, > HIVE-12994.03.patch, HIVE-12994.04.patch, HIVE-12994.05.patch, > HIVE-12994.06.patch, HIVE-12994.06.patch, HIVE-12994.07.patch, > HIVE-12994.08.patch, HIVE-12994.09.patch, HIVE-12994.10.patch, > HIVE-12994.11.patch, HIVE-12994.12.patch, HIVE-12994.patch > > > From SQL:2003, the NULLS FIRST and NULLS LAST options can be used to > determine whether nulls appear before or after non-null data values when the > ORDER BY clause is used. > SQL standard does not specify the behavior by default. Currently in Hive, > null values sort as if lower than any non-null value; that is, NULLS FIRST is > the default for ASC order, and NULLS LAST for DESC order. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-4570) More information to user on GetOperationStatus in Hive Server2 when query is still executing
[ https://issues.apache.org/jira/browse/HIVE-4570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15176853#comment-15176853 ] Szehon Ho commented on HIVE-4570: - Looks pretty useful, I posted some comments on the rb.. thanks. > More information to user on GetOperationStatus in Hive Server2 when query is > still executing > > > Key: HIVE-4570 > URL: https://issues.apache.org/jira/browse/HIVE-4570 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Amareshwari Sriramadasu >Assignee: Akshay Goyal > Attachments: HIVE-4570.01.patch, HIVE-4570.02.patch, > HIVE-4570.03.patch, HIVE-4570.04.patch > > > Currently in Hive Server2, when the query is still executing only the status > is set as STILL_EXECUTING. > This issue is to give more information to the user such as progress and > running job handles, if possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13199) NDC stopped working in LLAP logging
[ https://issues.apache.org/jira/browse/HIVE-13199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-13199: - Attachment: HIVE-13199.1.patch > NDC stopped working in LLAP logging > --- > > Key: HIVE-13199 > URL: https://issues.apache.org/jira/browse/HIVE-13199 > Project: Hive > Issue Type: Bug > Components: llap, Logging >Affects Versions: 2.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-13199.1.patch > > > NDC context were missing from the log lines. Reason for it is NDC class is > part of log4j-1.2-api (bridge jar). This is added as compile time dependency. > Due to the absence of this jar in llap daemons, the NDC context failed to > initialize. Log4j2 replaced NDC with ThreadContext. Hence we need the bridge > jar. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13199) NDC stopped working in LLAP logging
[ https://issues.apache.org/jira/browse/HIVE-13199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-13199: - Description: NDC context were missing from the log lines. Reason for it is NDC class is part of log4j-1.2-api (bridge jar). This is added as compile time dependency. Due to the absence of this jar in llap daemons, the NDC context failed to initialize. Log4j2 replaced NDC with ThreadContext. Hence we need the bridge jar. was:NDC context were missing from the log lines. Reason for it is NDC class is part of log4j-1.2-api (bridge jar). This is added as compile time dependency. Due to the absence of this jar in llap daemons, the NDC context failed to initialize. Log4j2 replaced NDC with ThreadContext. Hence we need the bridge jar. > NDC stopped working in LLAP logging > --- > > Key: HIVE-13199 > URL: https://issues.apache.org/jira/browse/HIVE-13199 > Project: Hive > Issue Type: Bug > Components: llap, Logging >Affects Versions: 2.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-13199.1.patch > > > NDC context were missing from the log lines. Reason for it is NDC class is > part of log4j-1.2-api (bridge jar). This is added as compile time dependency. > Due to the absence of this jar in llap daemons, the NDC context failed to > initialize. Log4j2 replaced NDC with ThreadContext. Hence we need the bridge > jar. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13199) NDC stopped working in LLAP logging
[ https://issues.apache.org/jira/browse/HIVE-13199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-13199: - Status: Patch Available (was: Open) > NDC stopped working in LLAP logging > --- > > Key: HIVE-13199 > URL: https://issues.apache.org/jira/browse/HIVE-13199 > Project: Hive > Issue Type: Bug > Components: llap, Logging >Affects Versions: 2.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-13199.1.patch > > > NDC context were missing from the log lines. Reason for it is NDC class is > part of log4j-1.2-api (bridge jar). This is added as compile time dependency. > Due to the absence of this jar in llap daemons, the NDC context failed to > initialize. Log4j2 replaced NDC with ThreadContext. Hence we need the bridge > jar. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13199) NDC stopped working in LLAP logging
[ https://issues.apache.org/jira/browse/HIVE-13199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15176863#comment-15176863 ] Prasanth Jayachandran commented on HIVE-13199: -- [~gopalv] Could you take a look this? > NDC stopped working in LLAP logging > --- > > Key: HIVE-13199 > URL: https://issues.apache.org/jira/browse/HIVE-13199 > Project: Hive > Issue Type: Bug > Components: llap, Logging >Affects Versions: 2.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-13199.1.patch > > > NDC context were missing from the log lines. Reason for it is NDC class is > part of log4j-1.2-api (bridge jar). This is added as compile time dependency. > Due to the absence of this jar in llap daemons, the NDC context failed to > initialize. Log4j2 replaced NDC with ThreadContext. Hence we need the bridge > jar. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11837) comments do not support unicode characters well.
[ https://issues.apache.org/jira/browse/HIVE-11837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15176865#comment-15176865 ] Szehon Ho commented on HIVE-11837: -- Looks pretty simple, +1. Guess jira description should be 'show create table does not show unicode characters well'? > comments do not support unicode characters well. > > > Key: HIVE-11837 > URL: https://issues.apache.org/jira/browse/HIVE-11837 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 0.13.1, 1.1.0 > Environment: Hadoop 2.7 > Hive 0.13.1 / Hive 1.1.0 > RHEL 6.4 / SLES 11.3 >Reporter: Rudd Chen >Assignee: Rudd Chen >Priority: Minor > Attachments: HIVE-11837.1.patch, HIVE-11837.patch > > > the terminal encoding is set to UTF-8, It can display Chinese characters. > then I create a table with a comment in Chinese, both "show create table" and > "desc formatted table" can not display the Chinese characters in the table > comments, meanwhile it can display Chinese characters in column comment.. See > below: > 0: jdbc:hive2://ha-cluster/default> create table tt(id int comment '列中文测试') > comment '表中文测试'; > No rows affected (0.152 seconds) > 0: jdbc:hive2://ha-cluster/default> > 0: jdbc:hive2://ha-cluster/default> > 0: jdbc:hive2://ha-cluster/default> desc formatted tt; > +---+---+-+ > | col_name| data_type > | comment | > +---+---+-+ > | # col_name| data_type > | comment | > | | NULL > | NULL| > | id| int > | 列中文测试 | > | | NULL > | NULL| > | # Detailed Table Information | NULL > | NULL| > | Database: | default > | NULL| > | Owner:| admin > | NULL| > | CreateTime: | Wed Sep 16 11:13:34 CST 2015 > | NULL| > | LastAccessTime: | UNKNOWN > | NULL| > | Protect Mode: | None > | NULL| > | Retention:| 0 > | NULL| > | Location: | hdfs://hacluster/user/hive/warehouse/tt > | NULL| > | Table Type: | MANAGED_TABLE > | NULL| > | Table Parameters: | NULL > | NULL| > | | comment > | \u8868\u4E2D\u6587\u6D4B\u8BD5 | > | | transient_lastDdlTime > | 1442373214 | > | | NULL > | NULL| > | # Storage Information | NULL > | NULL| > | SerDe Library:| > org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe | NULL > | > | InputFormat: | > org.apache.hadoop.hive.ql.io.RCFileInputFormat| NULL > | > | OutputFormat: | > org.apache.hadoop.hive.ql.io.RCFileOutputFormat | NULL > | > | Compressed: | No > | NULL| > | Num Buckets: | -1 > | NULL| > | Bucket Columns: | [] > | NULL
[jira] [Commented] (HIVE-12994) Implement support for NULLS FIRST/NULLS LAST
[ https://issues.apache.org/jira/browse/HIVE-12994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15176881#comment-15176881 ] Lefty Leverenz commented on HIVE-12994: --- Doc note: This needs to be documented in the wiki, with version information. * [Syntax of Order By | https://cwiki.apache.org/confluence/display/Hive/LanguageManual+SortBy#LanguageManualSortBy-SyntaxofOrderBy] > Implement support for NULLS FIRST/NULLS LAST > > > Key: HIVE-12994 > URL: https://issues.apache.org/jira/browse/HIVE-12994 > Project: Hive > Issue Type: New Feature > Components: CBO, Parser, Serializers/Deserializers >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Labels: TODOC2.1 > Fix For: 2.1.0 > > Attachments: HIVE-12994.01.patch, HIVE-12994.02.patch, > HIVE-12994.03.patch, HIVE-12994.04.patch, HIVE-12994.05.patch, > HIVE-12994.06.patch, HIVE-12994.06.patch, HIVE-12994.07.patch, > HIVE-12994.08.patch, HIVE-12994.09.patch, HIVE-12994.10.patch, > HIVE-12994.11.patch, HIVE-12994.12.patch, HIVE-12994.patch > > > From SQL:2003, the NULLS FIRST and NULLS LAST options can be used to > determine whether nulls appear before or after non-null data values when the > ORDER BY clause is used. > SQL standard does not specify the behavior by default. Currently in Hive, > null values sort as if lower than any non-null value; that is, NULLS FIRST is > the default for ASC order, and NULLS LAST for DESC order. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13199) NDC stopped working in LLAP logging
[ https://issues.apache.org/jira/browse/HIVE-13199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13199: Reporter: Sergey Shelukhin (was: Prasanth Jayachandran) > NDC stopped working in LLAP logging > --- > > Key: HIVE-13199 > URL: https://issues.apache.org/jira/browse/HIVE-13199 > Project: Hive > Issue Type: Bug > Components: llap, Logging >Affects Versions: 2.1.0 >Reporter: Sergey Shelukhin >Assignee: Prasanth Jayachandran > Attachments: HIVE-13199.1.patch > > > NDC context were missing from the log lines. Reason for it is NDC class is > part of log4j-1.2-api (bridge jar). This is added as compile time dependency. > Due to the absence of this jar in llap daemons, the NDC context failed to > initialize. Log4j2 replaced NDC with ThreadContext. Hence we need the bridge > jar. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13199) NDC stopped working in LLAP logging
[ https://issues.apache.org/jira/browse/HIVE-13199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15176935#comment-15176935 ] Sergey Shelukhin commented on HIVE-13199: - +1.. Does this also affect 2.0? I assume all the logging fixes need to be backported to branch-2 > NDC stopped working in LLAP logging > --- > > Key: HIVE-13199 > URL: https://issues.apache.org/jira/browse/HIVE-13199 > Project: Hive > Issue Type: Bug > Components: llap, Logging >Affects Versions: 2.1.0 >Reporter: Sergey Shelukhin >Assignee: Prasanth Jayachandran > Attachments: HIVE-13199.1.patch > > > NDC context were missing from the log lines. Reason for it is NDC class is > part of log4j-1.2-api (bridge jar). This is added as compile time dependency. > Due to the absence of this jar in llap daemons, the NDC context failed to > initialize. Log4j2 replaced NDC with ThreadContext. Hence we need the bridge > jar. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13192) OOM error when MapJoinBytesTableContainer memory threshold is too low
[
https://issues.apache.org/jira/browse/HIVE-13192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15176998#comment-15176998
]
Hive QA commented on HIVE-13192:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12790792/HIVE-13192.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 9780 tests executed
*Failed tests:*
{noformat}
TestSparkCliDriver-groupby3_map.q-sample2.q-auto_join14.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-groupby_map_ppr_multi_distinct.q-table_access_keys_stats.q-groupby4_noskew.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join_rc.q-insert1.q-vectorized_rcfile_columnar.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-ppd_join4.q-join9.q-ppd_join3.q-and-12-more - did not
produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_mult_tables
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7147/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7147/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7147/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12790792 - PreCommit-HIVE-TRUNK-Build
> OOM error when MapJoinBytesTableContainer memory threshold is too low
> --
>
> Key: HIVE-13192
> URL: https://issues.apache.org/jira/browse/HIVE-13192
> Project: Hive
> Issue Type: Bug
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-13192.1.patch
>
>
> HIVE-11449 covered the scenario with HybridHashTableContainer, but we can
> still run into the error as shown below:
> {code}
> Vertex failed, vertexName=Map 1, vertexId=vertex_1454464706407_0225_1_19,
> diagnostics=[Task failed, taskId=task_1454464706407_0225_1_19_00,
> diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running
> task:java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:172)
> at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:138)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:333)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:177)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:168)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:168)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:163)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.hadoop.hive.serde2.WriteBuffers.nextBufferToWrite(WriteBuffers.java:206)
> at org.apache.hadoop.hive.serde2.WriteBuffers.write(WriteBuffers.java:182)
> at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryUtils.writeVLong(LazyBinaryUtils.java:411)
> at
> org.apache.hadoop.hive.serde2.WriteBuffers.writeVLong(WriteBuffers.java:294)
> at
> org.apache.hadoop.hive.ql.exec.persistence.BytesBytesMultiHashMap.addRecordToList(BytesBytesMultiHashMap.java:540)
> at
> org.apache.hadoop.hive.ql.exec.persistence.BytesBytesMultiHashMap.put(BytesBytesMultiHashMap.java:219)
> at
> org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer.putRow(MapJoinBytesTableContainer.java:285)
> at
> org.apache.hadoop.hive.ql.exec.tez.HashTableLoader.load(HashTableLoader.j
[jira] [Commented] (HIVE-12558) LLAP: output QueryFragmentCounters somewhere
[
https://issues.apache.org/jira/browse/HIVE-12558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15177200#comment-15177200
]
Hive QA commented on HIVE-12558:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12787190/HIVE-12558.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 9753 tests
executed
*Failed tests:*
{noformat}
TestParseNegative - did not produce a TEST-*.xml file
TestSparkCliDriver-groupby3_map.q-sample2.q-auto_join14.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-groupby_map_ppr_multi_distinct.q-table_access_keys_stats.q-groupby4_noskew.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join_rc.q-insert1.q-vectorized_rcfile_columnar.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-ppd_join4.q-join9.q-ppd_join3.q-and-12-more - did not
produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_llap_acid
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_llap_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_llap_uncompressed
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_llap
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_llap_nullscan
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.service.cli.operation.TestOperationLoggingAPIWithTez.testFetchResultsOfLogWithPerformanceMode
org.apache.hive.service.cli.operation.TestOperationLoggingAPIWithTez.testFetchResultsOfLogWithVerboseMode
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7149/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7149/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7149/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 14 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12787190 - PreCommit-HIVE-TRUNK-Build
> LLAP: output QueryFragmentCounters somewhere
>
>
> Key: HIVE-12558
> URL: https://issues.apache.org/jira/browse/HIVE-12558
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Sergey Shelukhin
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-12558.1.patch, HIVE-12558.2.patch,
> HIVE-12558.wip.patch, sample-output.png
>
>
> Right now, LLAP logs counters for every fragment; most of them are IO related
> and could be very useful, they also include table names so that things like
> cache hit ratio, etc., could be calculated for every table.
> We need to output them to some metrics system (preserving the breakdown by
> table, possibly also adding query ID or even stage) so that they'd be usable
> without grep/sed/awk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-5370) format_number udf should take user specifed format as argument
[ https://issues.apache.org/jira/browse/HIVE-5370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated HIVE-5370: -- Status: Open (was: Patch Available) > format_number udf should take user specifed format as argument > -- > > Key: HIVE-5370 > URL: https://issues.apache.org/jira/browse/HIVE-5370 > Project: Hive > Issue Type: Improvement > Components: UDF >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu >Priority: Minor > Fix For: 2.1.0 > > Attachments: D13185.1.patch, D13185.2.patch, HIVE-5370.2.patch, > HIVE-5370.patch, HIVE-5370.patch > > > Currently, format_number udf formats the number to #,###,###.##, but it > should also take a user specified format as optional input. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-5370) format_number udf should take user specifed format as argument
[ https://issues.apache.org/jira/browse/HIVE-5370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated HIVE-5370: -- Attachment: HIVE-5370.3.patch Updated patch with review comments incorporated > format_number udf should take user specifed format as argument > -- > > Key: HIVE-5370 > URL: https://issues.apache.org/jira/browse/HIVE-5370 > Project: Hive > Issue Type: Improvement > Components: UDF >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu >Priority: Minor > Fix For: 2.1.0 > > Attachments: D13185.1.patch, D13185.2.patch, HIVE-5370.2.patch, > HIVE-5370.3.patch, HIVE-5370.patch, HIVE-5370.patch > > > Currently, format_number udf formats the number to #,###,###.##, but it > should also take a user specified format as optional input. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-5370) format_number udf should take user specifed format as argument
[ https://issues.apache.org/jira/browse/HIVE-5370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated HIVE-5370: -- Status: Patch Available (was: Open) > format_number udf should take user specifed format as argument > -- > > Key: HIVE-5370 > URL: https://issues.apache.org/jira/browse/HIVE-5370 > Project: Hive > Issue Type: Improvement > Components: UDF >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu >Priority: Minor > Fix For: 2.1.0 > > Attachments: D13185.1.patch, D13185.2.patch, HIVE-5370.2.patch, > HIVE-5370.3.patch, HIVE-5370.patch, HIVE-5370.patch > > > Currently, format_number udf formats the number to #,###,###.##, but it > should also take a user specified format as optional input. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13163) ORC MemoryManager thread checks are fatal, should WARN
[
https://issues.apache.org/jira/browse/HIVE-13163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15177344#comment-15177344
]
Gopal V commented on HIVE-13163:
Test failures seem unrelated.
{code}
Caused by: MetaException(message:Unable to read from or write to hbase Failed 1
action: RetriesExhaustedException: 1 time, )
at
org.apache.hadoop.hive.metastore.hbase.HBaseStore.createDatabase(HBaseStore.java:157)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB_core(HiveMetaStore.java:580)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:596)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:387)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.(HiveMetaStore.java:345)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:222)
at
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.(SessionHiveMetaStoreClient.java:68)
at
org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3128)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3169)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3149)
at
org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:530)
... 28 more
{code}
> ORC MemoryManager thread checks are fatal, should WARN
> ---
>
> Key: HIVE-13163
> URL: https://issues.apache.org/jira/browse/HIVE-13163
> Project: Hive
> Issue Type: Bug
> Components: ORC
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Gopal V
>Assignee: Gopal V
> Labels: PIG
> Attachments: HIVE-13163.1.patch
>
>
> The MemoryManager is tied to a WriterOptions on create, which can occur in a
> different thread from the writer calls.
> This is unexpected, but safe and needs a warning not a fatal.
> {code}
> /**
>* Light weight thread-safety check for multi-threaded access patterns
>*/
> private void checkOwner() {
> Preconditions.checkArgument(ownerLock.isHeldByCurrentThread(),
> "Owner thread expected %s, got %s",
> ownerLock.getOwner(),
> Thread.currentThread());
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13163) ORC MemoryManager thread checks are fatal, should WARN
[
https://issues.apache.org/jira/browse/HIVE-13163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-13163:
---
Resolution: Fixed
Fix Version/s: 2.1.0
Status: Resolved (was: Patch Available)
Committed to master, thanks [~prasanth_j]
> ORC MemoryManager thread checks are fatal, should WARN
> ---
>
> Key: HIVE-13163
> URL: https://issues.apache.org/jira/browse/HIVE-13163
> Project: Hive
> Issue Type: Bug
> Components: ORC
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Gopal V
>Assignee: Gopal V
> Labels: PIG
> Fix For: 2.1.0
>
> Attachments: HIVE-13163.1.patch
>
>
> The MemoryManager is tied to a WriterOptions on create, which can occur in a
> different thread from the writer calls.
> This is unexpected, but safe and needs a warning not a fatal.
> {code}
> /**
>* Light weight thread-safety check for multi-threaded access patterns
>*/
> private void checkOwner() {
> Preconditions.checkArgument(ownerLock.isHeldByCurrentThread(),
> "Owner thread expected %s, got %s",
> ownerLock.getOwner(),
> Thread.currentThread());
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13108) Operators: SORT BY randomness is not safe with network partitions
[ https://issues.apache.org/jira/browse/HIVE-13108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-13108: --- Resolution: Fixed Fix Version/s: 2.1.0 1.3.0 Release Note: Operators: SORT BY randomness is not safe with network partitions (Gopal V, reviewed by Sergey Shelukhin) Status: Resolved (was: Patch Available) Pushed to master & branch-1, thanks [~sershe]. > Operators: SORT BY randomness is not safe with network partitions > - > > Key: HIVE-13108 > URL: https://issues.apache.org/jira/browse/HIVE-13108 > Project: Hive > Issue Type: Bug > Components: Spark, Tez >Affects Versions: 1.3.0, 1.2.1, 2.0.0, 2.0.1 >Reporter: Gopal V >Assignee: Gopal V > Fix For: 1.3.0, 2.1.0 > > Attachments: HIVE-13108.1.patch > > > SORT BY relies on a transient Random object, which is initialized once per > deserialize operation. > This results in complications during a network partition and when Tez/Spark > reuses a cached plan. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13199) NDC stopped working in LLAP logging
[ https://issues.apache.org/jira/browse/HIVE-13199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15177402#comment-15177402 ] Prasanth Jayachandran commented on HIVE-13199: -- I am not sure yet. HIVE-13110 seems to have committed only to master. This patch should have been part of HIVE-13110. > NDC stopped working in LLAP logging > --- > > Key: HIVE-13199 > URL: https://issues.apache.org/jira/browse/HIVE-13199 > Project: Hive > Issue Type: Bug > Components: llap, Logging >Affects Versions: 2.1.0 >Reporter: Sergey Shelukhin >Assignee: Prasanth Jayachandran > Attachments: HIVE-13199.1.patch > > > NDC context were missing from the log lines. Reason for it is NDC class is > part of log4j-1.2-api (bridge jar). This is added as compile time dependency. > Due to the absence of this jar in llap daemons, the NDC context failed to > initialize. Log4j2 replaced NDC with ThreadContext. Hence we need the bridge > jar. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
