[jira] [Commented] (HIVE-14276) Update protocol version in TOpenSessionReq and TOpenSessionResp
[
https://issues.apache.org/jira/browse/HIVE-14276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15405492#comment-15405492
]
Hive QA commented on HIVE-14276:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12819737/HIVE-14276.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10429 tests
executed
*Failed tests:*
{noformat}
TestMsgBusConnection - did not produce a TEST-*.xml file
TestQueryLifeTimeHook - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_mult_tables_compact
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testDelayedLocalityNodeCommErrorImmediateAllocation
org.apache.hadoop.hive.metastore.TestHiveMetaStoreTxns.stringifyValidTxns
org.apache.hadoop.hive.metastore.TestHiveMetaStoreTxns.testTxnRange
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/745/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/745/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-745/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12819737 - PreCommit-HIVE-MASTER-Build
> Update protocol version in TOpenSessionReq and TOpenSessionResp
> ---
>
> Key: HIVE-14276
> URL: https://issues.apache.org/jira/browse/HIVE-14276
> Project: Hive
> Issue Type: Sub-task
>Reporter: Ziyang Zhao
>Assignee: Ziyang Zhao
> Attachments: HIVE-14276.1.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14409) Allow regex pattern in the output of qtest
[
https://issues.apache.org/jira/browse/HIVE-14409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15405494#comment-15405494
]
Zoltan Haindrich commented on HIVE-14409:
-
[~ajisakaa]
i think a micro dsl inside the q-files would be useful - for cases in which the
application output can't be reliably defined
...but I think that hive's output for explain and such should be:
* stable(never change in repeated executions)
* and be as much as independent from the execution environment (like jdk
version) as possible.
i'm afraid that making the test comparisions smarter may even hide bugs later
on...
i think that burning a few more cycles of application time to produce more
stable explain outputs can even help the end user who is changing their
application environment and looking for any regresions after the changes have
been made.
...when you look at just one line in an output you don't really know that the
source of that line was a list or a set; for a set it's perfectly fine to sort
- but for an array it doesn't really align with application internals.
i've worked toward stabilizing explain outputs lately in HIVE-14285,
HIVE-14287, HIVE-14329
i know there are a few more affected...it seems to me that there are many
explain output writing methods here and there ;) ...i planning to somehow
reduce the number of these and put more pressure on the {{@Explain}} based
output generator - which i think already can produce stable outputs
independently of jvm ; but if it's not - theres only one place where it has to
be fixed.
i know this is a "longer" path...what do you think about it?
> Allow regex pattern in the output of qtest
> --
>
> Key: HIVE-14409
> URL: https://issues.apache.org/jira/browse/HIVE-14409
> Project: Hive
> Issue Type: Improvement
> Components: Test
>Reporter: Akira Ajisaka
>
> We cannot guarantee the output from external libraries (ex. HashMap), so we
> need to make the validation infrastructure to allow a sort within a single
> line.
> HIVE-13894, HIVE-13907, and HIVE-14088 are related.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14204) Optimize loading dynamic partitions
[
https://issues.apache.org/jira/browse/HIVE-14204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-14204:
Attachment: HIVE-14204.7.patch
> Optimize loading dynamic partitions
>
>
> Key: HIVE-14204
> URL: https://issues.apache.org/jira/browse/HIVE-14204
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-14204.1.patch, HIVE-14204.3.patch,
> HIVE-14204.4.patch, HIVE-14204.6.patch, HIVE-14204.7.patch
>
>
> Lots of time is spent in sequential fashion to load dynamic partitioned
> dataset in driver side. E.g simple dynamic partitioned load as follows takes
> 300+ seconds
> {noformat}
> INSERT INTO web_sales_test partition(ws_sold_date_sk) select * from
> tpcds_bin_partitioned_orc_200.web_sales;
> Time taken to load dynamic partitions: 309.22 seconds
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14204) Optimize loading dynamic partitions
[
https://issues.apache.org/jira/browse/HIVE-14204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-14204:
Status: Open (was: Patch Available)
> Optimize loading dynamic partitions
>
>
> Key: HIVE-14204
> URL: https://issues.apache.org/jira/browse/HIVE-14204
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-14204.1.patch, HIVE-14204.3.patch,
> HIVE-14204.4.patch, HIVE-14204.6.patch, HIVE-14204.7.patch
>
>
> Lots of time is spent in sequential fashion to load dynamic partitioned
> dataset in driver side. E.g simple dynamic partitioned load as follows takes
> 300+ seconds
> {noformat}
> INSERT INTO web_sales_test partition(ws_sold_date_sk) select * from
> tpcds_bin_partitioned_orc_200.web_sales;
> Time taken to load dynamic partitions: 309.22 seconds
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14204) Optimize loading dynamic partitions
[
https://issues.apache.org/jira/browse/HIVE-14204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-14204:
Status: Patch Available (was: Open)
> Optimize loading dynamic partitions
>
>
> Key: HIVE-14204
> URL: https://issues.apache.org/jira/browse/HIVE-14204
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-14204.1.patch, HIVE-14204.3.patch,
> HIVE-14204.4.patch, HIVE-14204.6.patch, HIVE-14204.7.patch
>
>
> Lots of time is spent in sequential fashion to load dynamic partitioned
> dataset in driver side. E.g simple dynamic partitioned load as follows takes
> 300+ seconds
> {noformat}
> INSERT INTO web_sales_test partition(ws_sold_date_sk) select * from
> tpcds_bin_partitioned_orc_200.web_sales;
> Time taken to load dynamic partitions: 309.22 seconds
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14204) Optimize loading dynamic partitions
[
https://issues.apache.org/jira/browse/HIVE-14204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15405528#comment-15405528
]
Rajesh Balamohan commented on HIVE-14204:
-
Uploaded revised patch. Removed fetching existing partitions in
loadDynamicPartitions. This can be added in the follow on JIRA.
> Optimize loading dynamic partitions
>
>
> Key: HIVE-14204
> URL: https://issues.apache.org/jira/browse/HIVE-14204
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-14204.1.patch, HIVE-14204.3.patch,
> HIVE-14204.4.patch, HIVE-14204.6.patch, HIVE-14204.7.patch
>
>
> Lots of time is spent in sequential fashion to load dynamic partitioned
> dataset in driver side. E.g simple dynamic partitioned load as follows takes
> 300+ seconds
> {noformat}
> INSERT INTO web_sales_test partition(ws_sold_date_sk) select * from
> tpcds_bin_partitioned_orc_200.web_sales;
> Time taken to load dynamic partitions: 309.22 seconds
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14409) Allow regex pattern in the output of qtest
[ https://issues.apache.org/jira/browse/HIVE-14409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15405558#comment-15405558 ] Peter Vary commented on HIVE-14409: --- Probably already aware of this, but there is a partialReservedPlanMask in QtestUtil, which allows pattern matching for the output lines, and regex could be used to match groups with different orders (for example: http://stackoverflow.com/questions/2596868/regex-matching-against-groups-in-different-order-without-repeating-the-group). So if it is not test specific issue, then it could be used to help the problem - and generated more, as [~kgyrtkirk] already pointed out :) > Allow regex pattern in the output of qtest > -- > > Key: HIVE-14409 > URL: https://issues.apache.org/jira/browse/HIVE-14409 > Project: Hive > Issue Type: Improvement > Components: Test >Reporter: Akira Ajisaka > > We cannot guarantee the output from external libraries (ex. HashMap), so we > need to make the validation infrastructure to allow a sort within a single > line. > HIVE-13894, HIVE-13907, and HIVE-14088 are related. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-14404) Allow delimiterfordsv to use multiple-character delimiters
[ https://issues.apache.org/jira/browse/HIVE-14404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marta Kuczora reassigned HIVE-14404: Assignee: Marta Kuczora (was: Peter Vary) > Allow delimiterfordsv to use multiple-character delimiters > -- > > Key: HIVE-14404 > URL: https://issues.apache.org/jira/browse/HIVE-14404 > Project: Hive > Issue Type: Improvement >Reporter: Stephen Measmer >Assignee: Marta Kuczora > > HIVE-5871 allows for reading multiple character delimiters. Would like the > ability to use outputformat=dsv and define multiple character delimiters. > Today delimiterfordsv only uses on character even if multiple are passes. > For example: > when I use: > beeline>!set outputformat dsv > beeline>!set delimiterfordsv "^-^" > I get: > 111201081253106275^31-Oct-2011 > 00:00:00^Text^201605232823^2016051968232151^201605232823_2016051968232151_0_0_1 > > Would like it to be: > 111201081253106275^-^31-Oct-2011 > 00:00:00^-^Text^-^201605232823^-^2016051968232151^-^201605232823_2016051968232151_0_0_1 > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14123) Add beeline configuration option to show database in the prompt
[ https://issues.apache.org/jira/browse/HIVE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lefty Leverenz updated HIVE-14123: -- Labels: (was: TODOC2.2) > Add beeline configuration option to show database in the prompt > --- > > Key: HIVE-14123 > URL: https://issues.apache.org/jira/browse/HIVE-14123 > Project: Hive > Issue Type: Improvement > Components: Beeline, CLI >Affects Versions: 2.2.0 >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Minor > Fix For: 2.2.0 > > Attachments: HIVE-14123.10.patch, HIVE-14123.2.patch, > HIVE-14123.3.patch, HIVE-14123.4.patch, HIVE-14123.5.patch, > HIVE-14123.6.patch, HIVE-14123.7.patch, HIVE-14123.8.patch, > HIVE-14123.9.patch, HIVE-14123.patch > > > There are several jira issues complaining that, the Beeline does not respect > hive.cli.print.current.db. > This is partially true, since in embedded mode, it uses the > hive.cli.print.current.db to change the prompt, since HIVE-10511. > In beeline mode, I think this function should use a beeline command line > option instead, like for the showHeader option emphasizing, that this is a > client side option. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14123) Add beeline configuration option to show database in the prompt
[ https://issues.apache.org/jira/browse/HIVE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15405592#comment-15405592 ] Lefty Leverenz commented on HIVE-14123: --- Looks good, [~pvary]. I removed the TODOC2.2 label. Thanks for doing the documentation and adding the release note. > Add beeline configuration option to show database in the prompt > --- > > Key: HIVE-14123 > URL: https://issues.apache.org/jira/browse/HIVE-14123 > Project: Hive > Issue Type: Improvement > Components: Beeline, CLI >Affects Versions: 2.2.0 >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Minor > Fix For: 2.2.0 > > Attachments: HIVE-14123.10.patch, HIVE-14123.2.patch, > HIVE-14123.3.patch, HIVE-14123.4.patch, HIVE-14123.5.patch, > HIVE-14123.6.patch, HIVE-14123.7.patch, HIVE-14123.8.patch, > HIVE-14123.9.patch, HIVE-14123.patch > > > There are several jira issues complaining that, the Beeline does not respect > hive.cli.print.current.db. > This is partially true, since in embedded mode, it uses the > hive.cli.print.current.db to change the prompt, since HIVE-10511. > In beeline mode, I think this function should use a beeline command line > option instead, like for the showHeader option emphasizing, that this is a > client side option. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14123) Add beeline configuration option to show database in the prompt
[ https://issues.apache.org/jira/browse/HIVE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15405624#comment-15405624 ] Peter Vary commented on HIVE-14123: --- Thanks for teaching [~leftylev] > Add beeline configuration option to show database in the prompt > --- > > Key: HIVE-14123 > URL: https://issues.apache.org/jira/browse/HIVE-14123 > Project: Hive > Issue Type: Improvement > Components: Beeline, CLI >Affects Versions: 2.2.0 >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Minor > Fix For: 2.2.0 > > Attachments: HIVE-14123.10.patch, HIVE-14123.2.patch, > HIVE-14123.3.patch, HIVE-14123.4.patch, HIVE-14123.5.patch, > HIVE-14123.6.patch, HIVE-14123.7.patch, HIVE-14123.8.patch, > HIVE-14123.9.patch, HIVE-14123.patch > > > There are several jira issues complaining that, the Beeline does not respect > hive.cli.print.current.db. > This is partially true, since in embedded mode, it uses the > hive.cli.print.current.db to change the prompt, since HIVE-10511. > In beeline mode, I think this function should use a beeline command line > option instead, like for the showHeader option emphasizing, that this is a > client side option. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14403) LLAP node specific preemption will only preempt once on a node per AM
[
https://issues.apache.org/jira/browse/HIVE-14403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15405632#comment-15405632
]
Hive QA commented on HIVE-14403:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12821711/HIVE-14403.02.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10432 tests
executed
*Failed tests:*
{noformat}
TestMsgBusConnection - did not produce a TEST-*.xml file
TestQueryLifeTimeHook - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testDelayedLocalityNodeCommErrorImmediateAllocation
org.apache.hadoop.hive.metastore.TestHiveMetaStoreTxns.stringifyValidTxns
org.apache.hadoop.hive.metastore.TestHiveMetaStoreTxns.testTxnRange
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/746/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/746/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-746/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12821711 - PreCommit-HIVE-MASTER-Build
> LLAP node specific preemption will only preempt once on a node per AM
> -
>
> Key: HIVE-14403
> URL: https://issues.apache.org/jira/browse/HIVE-14403
> Project: Hive
> Issue Type: Bug
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
>Priority: Critical
> Attachments: HIVE-14403.01.patch, HIVE-14403.02.patch
>
>
> Query hang reported by [~cartershanklin]
> Turns out that once an AM has preempted a task on a node for locality, it
> will not be able to preempt another task on the same node (specifically for
> local requests)
> Manifests as a query hanging. It's possible for a previous query to interfere
> with a subsequent query since the AM is shared.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14343) HiveDriverRunHookContext's command is null in HS2 mode
[
https://issues.apache.org/jira/browse/HIVE-14343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15405634#comment-15405634
]
Lefty Leverenz commented on HIVE-14343:
---
Commit note: An underscore was used for the JIRA number in the commit
(HIVE_14343 instead of HIVE-14343).
This probably doesn't need to be added to the errata.txt list (HIVE-11704), but
in gmail a search for "14343" didn't find the commit.
> HiveDriverRunHookContext's command is null in HS2 mode
> --
>
> Key: HIVE-14343
> URL: https://issues.apache.org/jira/browse/HIVE-14343
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 2.2.0
>Reporter: Chao Sun
>Assignee: Chao Sun
> Fix For: 2.2.0
>
> Attachments: HIVE-14343.0.patch, HIVE-14343.1.patch
>
>
> Looking at the {{Driver#runInternal(String command, boolean
> alreadyCompiled)}}:
> {code}
> HiveDriverRunHookContext hookContext = new
> HiveDriverRunHookContextImpl(conf, command);
> // Get all the driver run hooks and pre-execute them.
> List driverRunHooks;
> {code}
> The context is initialized with the {{command}} passed in to the method.
> However, this command is always null if {{alreadyCompiled}} is true, which is
> the case for HS2 mode.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12994) Implement support for NULLS FIRST/NULLS LAST
[ https://issues.apache.org/jira/browse/HIVE-12994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lefty Leverenz updated HIVE-12994: -- Labels: (was: TODOC2.1) > Implement support for NULLS FIRST/NULLS LAST > > > Key: HIVE-12994 > URL: https://issues.apache.org/jira/browse/HIVE-12994 > Project: Hive > Issue Type: New Feature > Components: CBO, Parser, Serializers/Deserializers >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Fix For: 2.1.0 > > Attachments: HIVE-12994.01.patch, HIVE-12994.02.patch, > HIVE-12994.03.patch, HIVE-12994.04.patch, HIVE-12994.05.patch, > HIVE-12994.06.patch, HIVE-12994.06.patch, HIVE-12994.07.patch, > HIVE-12994.08.patch, HIVE-12994.09.patch, HIVE-12994.10.patch, > HIVE-12994.11.patch, HIVE-12994.12.patch, HIVE-12994.patch > > > From SQL:2003, the NULLS FIRST and NULLS LAST options can be used to > determine whether nulls appear before or after non-null data values when the > ORDER BY clause is used. > SQL standard does not specify the behavior by default. Currently in Hive, > null values sort as if lower than any non-null value; that is, NULLS FIRST is > the default for ASC order, and NULLS LAST for DESC order. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12994) Implement support for NULLS FIRST/NULLS LAST
[ https://issues.apache.org/jira/browse/HIVE-12994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15405675#comment-15405675 ] Lefty Leverenz commented on HIVE-12994: --- [~jcamachorodriguez] added the documentation to the wiki, so I removed the TODOC2.1 label from this issue. Thanks for the doc, Jesús. > Implement support for NULLS FIRST/NULLS LAST > > > Key: HIVE-12994 > URL: https://issues.apache.org/jira/browse/HIVE-12994 > Project: Hive > Issue Type: New Feature > Components: CBO, Parser, Serializers/Deserializers >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Fix For: 2.1.0 > > Attachments: HIVE-12994.01.patch, HIVE-12994.02.patch, > HIVE-12994.03.patch, HIVE-12994.04.patch, HIVE-12994.05.patch, > HIVE-12994.06.patch, HIVE-12994.06.patch, HIVE-12994.07.patch, > HIVE-12994.08.patch, HIVE-12994.09.patch, HIVE-12994.10.patch, > HIVE-12994.11.patch, HIVE-12994.12.patch, HIVE-12994.patch > > > From SQL:2003, the NULLS FIRST and NULLS LAST options can be used to > determine whether nulls appear before or after non-null data values when the > ORDER BY clause is used. > SQL standard does not specify the behavior by default. Currently in Hive, > null values sort as if lower than any non-null value; that is, NULLS FIRST is > the default for ASC order, and NULLS LAST for DESC order. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-14409) Allow regex pattern in the output of qtest
[ https://issues.apache.org/jira/browse/HIVE-14409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka reassigned HIVE-14409: Assignee: Akira Ajisaka > Allow regex pattern in the output of qtest > -- > > Key: HIVE-14409 > URL: https://issues.apache.org/jira/browse/HIVE-14409 > Project: Hive > Issue Type: Improvement > Components: Test >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka > > We cannot guarantee the output from external libraries (ex. HashMap), so we > need to make the validation infrastructure to allow a sort within a single > line. > HIVE-13894, HIVE-13907, and HIVE-14088 are related. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14411) selecting Hive on Hbase table may cause FileNotFoundException
[
https://issues.apache.org/jira/browse/HIVE-14411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15405760#comment-15405760
]
Rudd Chen commented on HIVE-14411:
--
@SuppressWarnings("rawtypes")
private static Path createDummyFileForEmptyPartition(Path path, JobConf job,
MapWork work,
Path hiveScratchDir, String alias, int sequenceNumber)
throws Exception {
String strPath = path.toString();
// The input file does not exist, replace it by a empty file
PartitionDesc partDesc = work.getPathToPartitionInfo().get(strPath);
if (partDesc.getTableDesc().isNonNative()) {
// if this isn't a hive table we can't create an empty file for it.
return path;
}
Properties props = SerDeUtils.createOverlayedProperties(
partDesc.getTableDesc().getProperties(), partDesc.getProperties());
HiveOutputFormat outFileFormat =
HiveFileFormatUtils.getHiveOutputFormat(job, partDesc);
boolean oneRow = partDesc.getInputFileFormatClass() ==
OneNullRowInputFormat.class;
Path newPath = createEmptyFile(hiveScratchDir, outFileFormat, job,
> selecting Hive on Hbase table may cause FileNotFoundException
> -
>
> Key: HIVE-14411
> URL: https://issues.apache.org/jira/browse/HIVE-14411
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Affects Versions: 1.3.0
>Reporter: Rudd Chen
>
> 1. create a Hbase table hbase_table
> 2. create a external Hive table test_table mapping to the hbase table
> example:
> create 'hbase_t'
> ,{NAME=>'cf',COMPRESSION=>'snappy'},{NUMREGIONS=>15,SPLITALGO=>'HexStringSplit'}
> create external table hbase_t_hive(key1 string,cf_train string,cf_flight
> string,cf_wbsw string,cf_wbxw string,cf_bgrz string,cf_bgtf string)
> stored by 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> with
> serdeproperties("hbase.columns.mapping"=":key,cf:train,cf:flight,cf:wbsw,cf:wbxw,cf:bgrz,cf:bgtf")
> tblproperties("hbase.table.name"="hbase_t");
> create table test3 as select * from hbase_t_hive where 1=2;
>
> if hive.optimize.null.scan=true, it will return an FileNotFoundException
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14411) selecting Hive on Hbase table may cause FileNotFoundException
[
https://issues.apache.org/jira/browse/HIVE-14411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15405763#comment-15405763
]
Rudd Chen commented on HIVE-14411:
--
when we are selecting a non native table, this method would return a strange
path like "-mr-10004default.hbase_t_hive{}", and it would throw an exception
like below when submitting the MapReduce Job.
ava.io.FileNotFoundException: File does not exist:
/user/admin/-mr-10004default.hbase_t_hive{}
at
org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getContentSummaryInt(FSDirStatAndListingOp.java:509)
at
org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getContentSummary(FSDirStatAndListingOp.java:140)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getContentSummary(FSNamesystem.java:3862)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getContentSummary(NameNodeRpcServer.java:1230)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getContentSummary(ClientNamenodeProtocolServerSideTranslatorPB.java:887)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:973)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2089)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2085)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1711)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2083)
at sun.reflect.GeneratedConstructorAccessor83.newInstance(Unknown
Source)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
at
org.apache.hadoop.hdfs.DFSClient.getContentSummary(DFSClient.java:2731)
at
org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:816)
at
org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:812)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getContentSummary(DistributedFileSystem.java:812)
at
org.apache.hadoop.hive.ql.exec.mr.ExecDriver.getInputFileNum(ExecDriver.java:555)
at
org.apache.hadoop.hive.ql.exec.mr.ExecDriver.checkFileNum(ExecDriver.java:535)
at
org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:430)
at
org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:137)
> selecting Hive on Hbase table may cause FileNotFoundException
> -
>
> Key: HIVE-14411
> URL: https://issues.apache.org/jira/browse/HIVE-14411
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Affects Versions: 1.3.0
>Reporter: Rudd Chen
>
> 1. create a Hbase table hbase_table
> 2. create a external Hive table test_table mapping to the hbase table
> example:
> create 'hbase_t'
> ,{NAME=>'cf',COMPRESSION=>'snappy'},{NUMREGIONS=>15,SPLITALGO=>'HexStringSplit'}
> create external table hbase_t_hive(key1 string,cf_train string,cf_flight
> string,cf_wbsw string,cf_wbxw string,cf_bgrz string,cf_bgtf string)
> stored by 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> with
> serdeproperties("hbase.columns.mapping"=":key,cf:train,cf:flight,cf:wbsw,cf:wbxw,cf:bgrz,cf:bgtf")
> tblproperties("hbase.table.name"="hbase_t");
> create table test3 as select * from hbase_t_hive where 1=2;
>
> if hive.optimize.null.scan=true, it will return an FileNotFoundException
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14405) Have tests log to the console along with hive.log
[
https://issues.apache.org/jira/browse/HIVE-14405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15405793#comment-15405793
]
Hive QA commented on HIVE-14405:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12821686/HIVE-14405.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10429 tests
executed
*Failed tests:*
{noformat}
TestMsgBusConnection - did not produce a TEST-*.xml file
TestQueryLifeTimeHook - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_orc_llap_counters
org.apache.hadoop.hive.metastore.TestHiveMetaStoreTxns.stringifyValidTxns
org.apache.hadoop.hive.metastore.TestHiveMetaStoreTxns.testTxnRange
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching
org.apache.hive.service.cli.operation.TestOperationLoggingLayout.testSwitchLogLayout
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/747/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/747/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-747/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12821686 - PreCommit-HIVE-MASTER-Build
> Have tests log to the console along with hive.log
> -
>
> Key: HIVE-14405
> URL: https://issues.apache.org/jira/browse/HIVE-14405
> Project: Hive
> Issue Type: Task
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HIVE-14405.01.patch
>
>
> When running tests from the IDE (not itests), logs end up going to hive.log -
> making it difficult to debug tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14039) HiveServer2: Make the usage of server with JDBC thirft serde enabled, backward compatible for older clients
[
https://issues.apache.org/jira/browse/HIVE-14039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15405966#comment-15405966
]
Hive QA commented on HIVE-14039:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12818692/HIVE-14039.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10429 tests
executed
*Failed tests:*
{noformat}
TestMsgBusConnection - did not produce a TEST-*.xml file
TestQueryLifeTimeHook - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_orc_llap_counters
org.apache.hadoop.hive.metastore.TestHiveMetaStoreTxns.stringifyValidTxns
org.apache.hadoop.hive.metastore.TestHiveMetaStoreTxns.testTxnRange
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/748/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/748/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-748/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12818692 - PreCommit-HIVE-MASTER-Build
> HiveServer2: Make the usage of server with JDBC thirft serde enabled,
> backward compatible for older clients
> ---
>
> Key: HIVE-14039
> URL: https://issues.apache.org/jira/browse/HIVE-14039
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2, JDBC
>Affects Versions: 2.0.1
>Reporter: Vaibhav Gumashta
>Assignee: Ziyang Zhao
> Attachments: HIVE-14039.2.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14395) Add the missing data files to Avro union tests (HIVE-14205 addendum)
[ https://issues.apache.org/jira/browse/HIVE-14395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-14395: --- Fix Version/s: 2.1.1 Thanks, [~ashutoshc]. I also push to 2.1.1 > Add the missing data files to Avro union tests (HIVE-14205 addendum) > > > Key: HIVE-14395 > URL: https://issues.apache.org/jira/browse/HIVE-14395 > Project: Hive > Issue Type: Bug > Components: Test >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang >Priority: Trivial > Fix For: 2.2.0, 2.1.1 > > Attachments: HIVE-14395.patch > > > The union_non_nullable.txt & union_nullable.txt were not checked in for > HIVE-14205. It was my mistake. > It is the reason that testCliDriver_avro_nullable_union & > testNegativeCliDriver_avro_non_nullable_union are failing in current > pre-commit build. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14368) ThriftCLIService.GetOperationStatus should include exception's stack trace to the error message.
[ https://issues.apache.org/jira/browse/HIVE-14368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HIVE-14368: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Integrated into trunk. Thanks Zhihai for the patch. > ThriftCLIService.GetOperationStatus should include exception's stack trace to > the error message. > > > Key: HIVE-14368 > URL: https://issues.apache.org/jira/browse/HIVE-14368 > Project: Hive > Issue Type: Improvement > Components: Thrift API >Reporter: zhihai xu >Assignee: zhihai xu >Priority: Minor > Fix For: 2.2.0 > > Attachments: HIVE-14368.000.patch > > > ThriftCLIService.GetOperationStatus should include exception's stack trace to > the error message. The stack trace will be really helpful for client to debug > failed queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14146) Column comments with "\n" character "corrupts" table metadata
[
https://issues.apache.org/jira/browse/HIVE-14146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-14146:
--
Attachment: HIVE-14146.9.patch
The index comments with newlines are handled now
I had to change several test outputs to match the changes:
- describe_comment_indent.q.out - With multiline comments the extra lines was
not indented appropriately. My patch solves this problem
- create_view_translate.q.out, create_view_partitioned.q.out,
create_view.q.out, alter_view_as_select.q.out - Describing views a little nicer
now - the multiline select is indented
- create_like.q.out - Table parameters containing newlines displayed nicely
The comments in describe database XXX results is not handled yet. The problem
here is that the comment is the 2nd column, so printing the extra lines is not
trivial. Either I would like to leave this as-is, or put this to another jira,
or reorder the fields, so the comment is the last.
{noformat:title=Current}
hive> describe database escape_comments_db;
escape_comments_db a\nbfile:/tmp/hive/warehouse/escape_comments_db.db
petervary USER
{noformat}
It is not too much hassle to do this:
{noformat:title=Proposed}
hive> describe database escape_comments_db;
escape_comments_db file:/tmp/warehouse/escape_comments_db.db
petervary USERa
b
{noformat}
I am not convinced that it is worth to do this, or even if anyone would
understand this:
{noformat:title=Alternative}
hive> describe database escape_comments_db;
escape_comments_db a file:/tmp/hive/warehouse/escape_comments_db.db
petervary USER
b
{noformat}
What do you think?
> Column comments with "\n" character "corrupts" table metadata
> -
>
> Key: HIVE-14146
> URL: https://issues.apache.org/jira/browse/HIVE-14146
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 2.2.0
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-14146.2.patch, HIVE-14146.3.patch,
> HIVE-14146.4.patch, HIVE-14146.5.patch, HIVE-14146.6.patch,
> HIVE-14146.7.patch, HIVE-14146.8.patch, HIVE-14146.9.patch, HIVE-14146.patch
>
>
> Create a table with the following(noting the \n in the COMMENT):
> {noformat}
> CREATE TABLE commtest(first_nm string COMMENT 'Indicates First name\nof an
> individual’);
> {noformat}
> Describe shows that now the metadata is messed up:
> {noformat}
> beeline> describe commtest;
> +---++---+--+
> | col_name | data_type |comment|
> +---++---+--+
> | first_nm | string | Indicates First name |
> | of an individual | NULL | NULL |
> +---++---+--+
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14409) Allow regex pattern in the output of qtest
[
https://issues.apache.org/jira/browse/HIVE-14409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406062#comment-15406062
]
Akira Ajisaka commented on HIVE-14409:
--
Thanks [~kgyrtkirk] and [~pvary] for the comments.
bq. i'm afraid that making the test comparisions smarter may even hide bugs
later on...
Agreed. We should use regex pattern only when we really need to do.
bq. i planning to somehow reduce the number of these and put more pressure on
the @Explain based output generator - which i think already can produce stable
outputs independently of jvm
I'm afraid that {{@Explain}} based output affects performance. If this
annotation can only be used in test, it's great.
bq. Probably already aware of this, but there is a partialReservedPlanMask in
QtestUtil
Thank you for the input!
> Allow regex pattern in the output of qtest
> --
>
> Key: HIVE-14409
> URL: https://issues.apache.org/jira/browse/HIVE-14409
> Project: Hive
> Issue Type: Improvement
> Components: Test
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>
> We cannot guarantee the output from external libraries (ex. HashMap), so we
> need to make the validation infrastructure to allow a sort within a single
> line.
> HIVE-13894, HIVE-13907, and HIVE-14088 are related.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HIVE-14409) Allow regex pattern in the output of qtest
[ https://issues.apache.org/jira/browse/HIVE-14409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406070#comment-15406070 ] Akira Ajisaka edited comment on HIVE-14409 at 8/3/16 3:33 PM: -- Attaching a sample patch which introduces a macro "-- REGEX_PATTERN ". The patch includes a fix for HIVE-13907 using the macro. was (Author: ajisakaa): Attaching a sample patch which introduces a macro "-- REGEX_PATTERN ". The patch includes the fix for HIVE-13907 using the macro. > Allow regex pattern in the output of qtest > -- > > Key: HIVE-14409 > URL: https://issues.apache.org/jira/browse/HIVE-14409 > Project: Hive > Issue Type: Improvement > Components: Test >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka > Attachments: HIVE-14409.00.patch > > > We cannot guarantee the output from external libraries (ex. HashMap), so we > need to make the validation infrastructure to allow a sort within a single > line. > HIVE-13894, HIVE-13907, and HIVE-14088 are related. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14409) Allow regex pattern in the output of qtest
[ https://issues.apache.org/jira/browse/HIVE-14409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HIVE-14409: - Attachment: HIVE-14409.00.patch Attaching a sample patch which introduces a macro "-- REGEX_PATTERN ". The patch includes the fix for HIVE-13907 using the macro. > Allow regex pattern in the output of qtest > -- > > Key: HIVE-14409 > URL: https://issues.apache.org/jira/browse/HIVE-14409 > Project: Hive > Issue Type: Improvement > Components: Test >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka > Attachments: HIVE-14409.00.patch > > > We cannot guarantee the output from external libraries (ex. HashMap), so we > need to make the validation infrastructure to allow a sort within a single > line. > HIVE-13894, HIVE-13907, and HIVE-14088 are related. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14409) Allow regex pattern in the output of qtest
[ https://issues.apache.org/jira/browse/HIVE-14409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HIVE-14409: - Status: Patch Available (was: Open) > Allow regex pattern in the output of qtest > -- > > Key: HIVE-14409 > URL: https://issues.apache.org/jira/browse/HIVE-14409 > Project: Hive > Issue Type: Improvement > Components: Test >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka > Attachments: HIVE-14409.00.patch > > > We cannot guarantee the output from external libraries (ex. HashMap), so we > need to make the validation infrastructure to allow a sort within a single > line. > HIVE-13894, HIVE-13907, and HIVE-14088 are related. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14380) Queries on tables with remote HDFS paths fail in "encryption" checks.
[
https://issues.apache.org/jira/browse/HIVE-14380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergio Peña updated HIVE-14380:
---
Resolution: Fixed
Fix Version/s: 2.2.0
Status: Resolved (was: Patch Available)
Thanks [~mithun] for your contribution. I committed this to master.
> Queries on tables with remote HDFS paths fail in "encryption" checks.
> -
>
> Key: HIVE-14380
> URL: https://issues.apache.org/jira/browse/HIVE-14380
> Project: Hive
> Issue Type: Bug
> Components: Encryption
>Reporter: Mithun Radhakrishnan
>Assignee: Mithun Radhakrishnan
> Fix For: 2.2.0
>
> Attachments: HIVE-14380.1.patch
>
>
> If a table has table/partition locations set to remote HDFS paths, querying
> them will cause the following IAException:
> {noformat}
> 2016-07-26 01:16:27,471 ERROR parse.CalcitePlanner
> (SemanticAnalyzer.java:getMetaData(1867)) -
> org.apache.hadoop.hive.ql.metadata.HiveException: Unable to determine if
> hdfs://foo.ygrid.yahoo.com:8020/projects/my_db/my_table is encrypted:
> java.lang.IllegalArgumentException: Wrong FS:
> hdfs://foo.ygrid.yahoo.com:8020/projects/my_db/my_table, expected:
> hdfs://bar.ygrid.yahoo.com:8020
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.isPathEncrypted(SemanticAnalyzer.java:2204)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getStrongestEncryptedTablePath(SemanticAnalyzer.java:2274)
> ...
> {noformat}
> This is because of the following code in {{SessionState}}:
> {code:title=SessionState.java|borderStyle=solid}
> public HadoopShims.HdfsEncryptionShim getHdfsEncryptionShim() throws
> HiveException {
> if (hdfsEncryptionShim == null) {
> try {
> FileSystem fs = FileSystem.get(sessionConf);
> if ("hdfs".equals(fs.getUri().getScheme())) {
> hdfsEncryptionShim =
> ShimLoader.getHadoopShims().createHdfsEncryptionShim(fs, sessionConf);
> } else {
> LOG.debug("Could not get hdfsEncryptionShim, it is only applicable
> to hdfs filesystem.");
> }
> } catch (Exception e) {
> throw new HiveException(e);
> }
> }
> return hdfsEncryptionShim;
> }
> {code}
> When the {{FileSystem}} instance is created, using the {{sessionConf}}
> implies that the current HDFS is going to be used. This call should instead
> fetch the {{FileSystem}} instance corresponding to the path being checked.
> A fix is forthcoming...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14409) Allow regex pattern in the output of qtest
[ https://issues.apache.org/jira/browse/HIVE-14409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406147#comment-15406147 ] Peter Vary commented on HIVE-14409: --- Thanks for the patch. These flaky tests are a pain. It will be nice to get rid of them. I think, that with this patch we lose the ability to regenerate the out file with maven. Am I right, or did I miss something? One solution could be putting the patterns into the input file, or even better into a 3rd file, and if that file doesn't exists, then we do not have to run the pattern matching for the whole file when it is not necessary > Allow regex pattern in the output of qtest > -- > > Key: HIVE-14409 > URL: https://issues.apache.org/jira/browse/HIVE-14409 > Project: Hive > Issue Type: Improvement > Components: Test >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka > Attachments: HIVE-14409.00.patch > > > We cannot guarantee the output from external libraries (ex. HashMap), so we > need to make the validation infrastructure to allow a sort within a single > line. > HIVE-13894, HIVE-13907, and HIVE-14088 are related. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14408) thread safety issue in fast hashtable
[
https://issues.apache.org/jira/browse/HIVE-14408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406157#comment-15406157
]
Hive QA commented on HIVE-14408:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12821717/HIVE-14408.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10429 tests
executed
*Failed tests:*
{noformat}
TestMsgBusConnection - did not produce a TEST-*.xml file
TestQueryLifeTimeHook - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testDelayedLocalityNodeCommErrorImmediateAllocation
org.apache.hadoop.hive.metastore.TestHiveMetaStoreTxns.stringifyValidTxns
org.apache.hadoop.hive.metastore.TestHiveMetaStoreTxns.testTxnRange
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/749/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/749/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-749/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12821717 - PreCommit-HIVE-MASTER-Build
> thread safety issue in fast hashtable
> -
>
> Key: HIVE-14408
> URL: https://issues.apache.org/jira/browse/HIVE-14408
> Project: Hive
> Issue Type: Bug
>Reporter: Takahiko Saito
>Assignee: Sergey Shelukhin
> Attachments: HIVE-14408.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14380) Queries on tables with remote HDFS paths fail in "encryption" checks.
[
https://issues.apache.org/jira/browse/HIVE-14380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406177#comment-15406177
]
Mithun Radhakrishnan commented on HIVE-14380:
-
Thank you very much, [~spena]. I have a related fix on the metastore
server-side. I hope to make time to raise a JIRA for this soon.
> Queries on tables with remote HDFS paths fail in "encryption" checks.
> -
>
> Key: HIVE-14380
> URL: https://issues.apache.org/jira/browse/HIVE-14380
> Project: Hive
> Issue Type: Bug
> Components: Encryption
>Reporter: Mithun Radhakrishnan
>Assignee: Mithun Radhakrishnan
> Fix For: 2.2.0
>
> Attachments: HIVE-14380.1.patch
>
>
> If a table has table/partition locations set to remote HDFS paths, querying
> them will cause the following IAException:
> {noformat}
> 2016-07-26 01:16:27,471 ERROR parse.CalcitePlanner
> (SemanticAnalyzer.java:getMetaData(1867)) -
> org.apache.hadoop.hive.ql.metadata.HiveException: Unable to determine if
> hdfs://foo.ygrid.yahoo.com:8020/projects/my_db/my_table is encrypted:
> java.lang.IllegalArgumentException: Wrong FS:
> hdfs://foo.ygrid.yahoo.com:8020/projects/my_db/my_table, expected:
> hdfs://bar.ygrid.yahoo.com:8020
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.isPathEncrypted(SemanticAnalyzer.java:2204)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getStrongestEncryptedTablePath(SemanticAnalyzer.java:2274)
> ...
> {noformat}
> This is because of the following code in {{SessionState}}:
> {code:title=SessionState.java|borderStyle=solid}
> public HadoopShims.HdfsEncryptionShim getHdfsEncryptionShim() throws
> HiveException {
> if (hdfsEncryptionShim == null) {
> try {
> FileSystem fs = FileSystem.get(sessionConf);
> if ("hdfs".equals(fs.getUri().getScheme())) {
> hdfsEncryptionShim =
> ShimLoader.getHadoopShims().createHdfsEncryptionShim(fs, sessionConf);
> } else {
> LOG.debug("Could not get hdfsEncryptionShim, it is only applicable
> to hdfs filesystem.");
> }
> } catch (Exception e) {
> throw new HiveException(e);
> }
> }
> return hdfsEncryptionShim;
> }
> {code}
> When the {{FileSystem}} instance is created, using the {{sessionConf}}
> implies that the current HDFS is going to be used. This call should instead
> fetch the {{FileSystem}} instance corresponding to the path being checked.
> A fix is forthcoming...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14403) LLAP node specific preemption will only preempt once on a node per AM
[ https://issues.apache.org/jira/browse/HIVE-14403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406180#comment-15406180 ] Siddharth Seth commented on HIVE-14403: --- Test failures are unrelated. Cannot reproduce TestLlapTaskSchedulerService.testDelayedLocalityNodeCommErrorImmediateAllocation on local runs. Will walk through the test in a separate jira to identify flakiness. > LLAP node specific preemption will only preempt once on a node per AM > - > > Key: HIVE-14403 > URL: https://issues.apache.org/jira/browse/HIVE-14403 > Project: Hive > Issue Type: Bug >Reporter: Siddharth Seth >Assignee: Siddharth Seth >Priority: Critical > Attachments: HIVE-14403.01.patch, HIVE-14403.02.patch > > > Query hang reported by [~cartershanklin] > Turns out that once an AM has preempted a task on a node for locality, it > will not be able to preempt another task on the same node (specifically for > local requests) > Manifests as a query hanging. It's possible for a previous query to interfere > with a subsequent query since the AM is shared. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14403) LLAP node specific preemption will only preempt once on a node per AM
[ https://issues.apache.org/jira/browse/HIVE-14403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-14403: -- Resolution: Fixed Fix Version/s: 2.1.1 Status: Resolved (was: Patch Available) Committed to master and branch-2.1 > LLAP node specific preemption will only preempt once on a node per AM > - > > Key: HIVE-14403 > URL: https://issues.apache.org/jira/browse/HIVE-14403 > Project: Hive > Issue Type: Bug >Reporter: Siddharth Seth >Assignee: Siddharth Seth >Priority: Critical > Fix For: 2.1.1 > > Attachments: HIVE-14403.01.patch, HIVE-14403.02.patch > > > Query hang reported by [~cartershanklin] > Turns out that once an AM has preempted a task on a node for locality, it > will not be able to preempt another task on the same node (specifically for > local requests) > Manifests as a query hanging. It's possible for a previous query to interfere > with a subsequent query since the AM is shared. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12274) Increase width of columns used for general configuration in the metastore.
[
https://issues.apache.org/jira/browse/HIVE-12274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406206#comment-15406206
]
Heye Vöcking commented on HIVE-12274:
-
In order to push this forward, I think it would make sense to focus on this
story and mark the related stories (HIVE-9815 and HIVE-12299) as duplicates
because both of them are subsets of this story.
There is some valuable information in them, in summary:
Regarding the 4000 characters limitation:
It was set to by this
[patch|https://github.com/apache/hive/commit/c21a526b0a752df2a51d20a2729cc8493c228799]
(HIVE-1364) which mentions the "max length on Oracle 9i/10g/11g" as the
reason. However, nowadays the limit can be increased because:
* Oracle DB's varchar2 supports 32767 bytes now, by setting the configuration
parameter MAX_STRING_SIZE to EXTENDED.
([source|http://docs.oracle.com/database/121/SQLRF/sql_elements001.htm#SQLRF55623])
* Postgres supports a max of 1GB for character datatype.
([source|http://www.postgresql.org/docs/8.3/static/datatype-character.html])
* MySQL can support upto 65535 bytes for the entire row. So long as the
PARAM_KEY value + PARAM_VALUE is less than 65535, we should be good.
([source|http://dev.mysql.com/doc/refman/5.0/en/char.html])
* SQL Server's varchar max length is 8000 and can go beyond using
"varchar(max)" with the same limitation as MySQL being 65535 bytes for the
entire row. ([source|http://dev.mysql.com/doc/refman/5.0/en/char.html])
* Derby's varchar can be upto 32672 bytes.
([source|https://db.apache.org/derby/docs/10.7/ref/rrefsqlj41207.html])
Regarding the updating process of existing databases:
After updating the maximum length the metastore database needs to be configured
and restarted with the new settings. Altering MAX_STRING_SIZE will update
database objects and possibly invalidate them, as follows:
# Tables with virtual columns will be updated with new data type metadata for
virtual columns of VARCHAR2(4000), 4000-byte NVARCHAR2, or RAW(2000) type.
# Functional indexes will become unusable if a change to their associated
virtual columns causes the index key to exceed index key length limits.
Attempts to rebuild such indexes will fail with ORA-01450: maximum key length
exceeded.
# Views will be invalidated if they contain VARCHAR2(4000), 4000-byte
NVARCHAR2, or RAW(2000) typed expression columns.
# Materialized views will be updated with new metadata VARCHAR2(4000),
4000-byte NVARCHAR2, and RAW(2000) typed expression columns
*So the limitation could be raised to 32672 bytes, with the caveat that MySQL
and SQL Server limit the row length to 65535 bytes, so that should also be
validated to provide consistency.*
[~teabot], could you update this story with the information?
> Increase width of columns used for general configuration in the metastore.
> --
>
> Key: HIVE-12274
> URL: https://issues.apache.org/jira/browse/HIVE-12274
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Affects Versions: 2.0.0
>Reporter: Elliot West
> Labels: metastore
> Attachments: HIVE-12274.example.ddl.hql
>
>
> This issue is very similar in principle to HIVE-1364. We are hitting a limit
> when processing JSON data that has a large nested schema. The struct
> definition is truncated when inserted into the metastore database column
> {{COLUMNS_V2.YPE_NAME}} as it is greater than 4000 characters in length.
> Given that the purpose of these columns is to hold very loosely defined
> configuration values it seems rather limiting to impose such a relatively low
> length bound. One can imagine that valid use cases will arise where
> reasonable parameter/property values exceed the current limit. Can these
> columns not use CLOB-like types as for example as used by
> {{TBLS.VIEW_EXPANDED_TEXT}}? It would seem that suitable type equivalents
> exist for all targeted database platforms:
> * MySQL: {{mediumtext}}
> * Postgres: {{text}}
> * Oracle: {{CLOB}}
> * Derby: {{LONG VARCHAR}}
> I'd suggest that the candidates for type change are:
> * {{COLUMNS_V2.TYPE_NAME}}
> * {{TABLE_PARAMS.PARAM_VALUE}}
> * {{SERDE_PARAMS.PARAM_VALUE}}
> * {{SD_PARAMS.PARAM_VALUE}}
> Finally, will this limitation persist in the work resulting from HIVE-9452?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9815) Metastore column"SERDE_PARAMS"."PARAM_VALUE" limited to 4000 bytes
[
https://issues.apache.org/jira/browse/HIVE-9815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406217#comment-15406217
]
Heye Vöcking commented on HIVE-9815:
I would suggest to mark this story as a duplicate since it seems to be a subset
of HIVE-12274. Therefore, the focus can be concentrated on HIVE-12274. I added
a comment there, summarizing the information collected in this story.
> Metastore column"SERDE_PARAMS"."PARAM_VALUE" limited to 4000 bytes
> ---
>
> Key: HIVE-9815
> URL: https://issues.apache.org/jira/browse/HIVE-9815
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 0.14.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>Priority: Critical
> Attachments: Hv2.txt
>
>
> Currently, in the hive metastore schema, the length of the column
> SERDE_PARAMS.PARAM_VALUE is set to 4000 bytes. This is not enough for users
> that have a key with a value larger than 4000 bytes. Say something like
> hbase.columns.mapping.
> I am not a database historian but appears that this limitation may have been
> put in place because Oracle's varchar2 was restricted to 4k bytes for a long
> time until recently.
> According to the following documentation, even today Oracle DB's varchar2
> only supports a max size of 4000 unless a configuration parameter
> MAX_STRING_SIZE is set to EXTENDED.
> http://docs.oracle.com/database/121/SQLRF/sql_elements001.htm#SQLRF55623
> {code}
> MAX_STRING_SIZE=EXTENDED
> {code}
> Postgres supports a max of 1GB for character datatype according to
> http://www.postgresql.org/docs/8.3/static/datatype-character.html
> MySQL can support upto 65535 bytes for the entire row. So long as the
> PARAM_KEY value + PARAM_VALUE is less than 65535, we should be good.
> http://dev.mysql.com/doc/refman/5.0/en/char.html
> SQL Server's varchar max length is 8000 and can go beyond using
> "varchar(max)".
> http://dev.mysql.com/doc/refman/5.0/en/char.html
> Derby's varchar can be upto 32672 bytes.
> https://db.apache.org/derby/docs/10.7/ref/rrefsqlj41207.html
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12299) Hive Column Data Type definition in schema limited to 4000 characters - too small
[ https://issues.apache.org/jira/browse/HIVE-12299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406219#comment-15406219 ] Heye Vöcking commented on HIVE-12299: - I would suggest to mark this story as a duplicate since it seems to be a subset of HIVE-12274. Therefore, the focus can be concentrated on HIVE-12274. > Hive Column Data Type definition in schema limited to 4000 characters - too > small > - > > Key: HIVE-12299 > URL: https://issues.apache.org/jira/browse/HIVE-12299 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Lakshmi Ramakrishnan > > The data type definitions in the table schema are limited to 4K characters - > as per the code here: > https://github.com/apache/hive/tree/master/metastore/scripts/upgrade/derby > I checked quickly and all meta stores have similar schema definition for the > column type. > Is there any reason why this limit is low? We had a table that had defined a > struct, which had over 200 columns and the column names were rather verbose > (for readability). This caused a non-obvious failure like > FAILED: IllegalArgumentException Error: : expected at the end of > 'string:array configurable or at least increase it to something much higher? > Additionally, there is no validation error that communicates this limitation > to the user, it required non-trivial debugging and looking into the table > definitions when it failed trying to parse what was essentially a truncated > type. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14117) HS2 UI: List of recent queries shows most recent query last
[ https://issues.apache.org/jira/browse/HIVE-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-14117: -- Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Test failures are unrelated. Committed to master. > HS2 UI: List of recent queries shows most recent query last > --- > > Key: HIVE-14117 > URL: https://issues.apache.org/jira/browse/HIVE-14117 > Project: Hive > Issue Type: Bug >Reporter: Gunther Hagleitner >Assignee: Gunther Hagleitner > Fix For: 2.2.0 > > Attachments: HIVE-14117.1.patch, HIVE-14117.1.patch > > > It's more useful to see the latest one first in your "last n queries" view. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14413) Extend HivePreFilteringRule to traverse inside elements of DNF/CNF and extract more deterministic pieces out
[ https://issues.apache.org/jira/browse/HIVE-14413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-14413: --- Status: Patch Available (was: Open) > Extend HivePreFilteringRule to traverse inside elements of DNF/CNF and > extract more deterministic pieces out > > > Key: HIVE-14413 > URL: https://issues.apache.org/jira/browse/HIVE-14413 > Project: Hive > Issue Type: Improvement > Components: CBO >Affects Versions: 2.2.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14413) Extend HivePreFilteringRule to traverse inside elements of DNF/CNF and extract more deterministic pieces out
[ https://issues.apache.org/jira/browse/HIVE-14413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-14413: --- Attachment: HIVE-14413.patch > Extend HivePreFilteringRule to traverse inside elements of DNF/CNF and > extract more deterministic pieces out > > > Key: HIVE-14413 > URL: https://issues.apache.org/jira/browse/HIVE-14413 > Project: Hive > Issue Type: Improvement > Components: CBO >Affects Versions: 2.2.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-14413.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14413) Extend HivePreFilteringRule to traverse inside elements of DNF/CNF and extract more deterministic pieces out
[ https://issues.apache.org/jira/browse/HIVE-14413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-14413: --- Attachment: (was: HIVE-14413.patch) > Extend HivePreFilteringRule to traverse inside elements of DNF/CNF and > extract more deterministic pieces out > > > Key: HIVE-14413 > URL: https://issues.apache.org/jira/browse/HIVE-14413 > Project: Hive > Issue Type: Improvement > Components: CBO >Affects Versions: 2.2.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-14413.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14413) Extend HivePreFilteringRule to traverse inside elements of DNF/CNF and extract more deterministic pieces out
[ https://issues.apache.org/jira/browse/HIVE-14413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-14413: --- Attachment: HIVE-14413.patch > Extend HivePreFilteringRule to traverse inside elements of DNF/CNF and > extract more deterministic pieces out > > > Key: HIVE-14413 > URL: https://issues.apache.org/jira/browse/HIVE-14413 > Project: Hive > Issue Type: Improvement > Components: CBO >Affects Versions: 2.2.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-14413.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14414) Fix TestHiveMetaStoreTxns UTs
[ https://issues.apache.org/jira/browse/HIVE-14414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-14414: -- Attachment: HIVE-14414.patch [~wzheng] could you review please > Fix TestHiveMetaStoreTxns UTs > - > > Key: HIVE-14414 > URL: https://issues.apache.org/jira/browse/HIVE-14414 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.3.0, 2.2.0, 2.1.1 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-14414.patch > > > Missed updating these tests in HIVE-14350 commit -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14414) Fix TestHiveMetaStoreTxns UTs
[ https://issues.apache.org/jira/browse/HIVE-14414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-14414: -- Status: Patch Available (was: Open) > Fix TestHiveMetaStoreTxns UTs > - > > Key: HIVE-14414 > URL: https://issues.apache.org/jira/browse/HIVE-14414 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.3.0, 2.2.0, 2.1.1 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-14414.patch > > > Missed updating these tests in HIVE-14350 commit -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14397) Queries ran after reopening of tez session launches additional sessions
[
https://issues.apache.org/jira/browse/HIVE-14397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406395#comment-15406395
]
Hive QA commented on HIVE-14397:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12821730/HIVE-14397.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10430 tests
executed
*Failed tests:*
{noformat}
TestMsgBusConnection - did not produce a TEST-*.xml file
TestQueryLifeTimeHook - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.metastore.TestHiveMetaStoreTxns.stringifyValidTxns
org.apache.hadoop.hive.metastore.TestHiveMetaStoreTxns.testTxnRange
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/750/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/750/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-750/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12821730 - PreCommit-HIVE-MASTER-Build
> Queries ran after reopening of tez session launches additional sessions
> ---
>
> Key: HIVE-14397
> URL: https://issues.apache.org/jira/browse/HIVE-14397
> Project: Hive
> Issue Type: Bug
> Components: Tez
>Affects Versions: 2.1.0, 2.2.0
>Reporter: Takahiko Saito
>Assignee: Prasanth Jayachandran
>Priority: Critical
> Attachments: HIVE-14397.1.patch, HIVE-14397.2.patch,
> HIVE-14397.2.patch
>
>
> Say we have configured hive.server2.tez.default.queues with 2 queues q1 and
> q2 with default expiry interval of 5 mins.
> After 5 mins of non-usage the sessions corresponding to queues q1 and q2 will
> be expired. When new set of queries are issue after this expiry, the default
> sessions backed by q1 and q2 and reopened again. Now when we run more queries
> the reopened sessions are not used instead new session is opened.
> At this point there will be 4 sessions running (2 abandoned sessions and 2
> current sessions).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HIVE-10264) Document Replication support on wiki
[ https://issues.apache.org/jira/browse/HIVE-10264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406402#comment-15406402 ] Shannon Ladymon edited comment on HIVE-10264 at 8/3/16 6:48 PM: I added more thorough documentation on the design of replication in the wiki: * [HiveReplicationDevelopment | https://cwiki.apache.org/confluence/display/Hive/HiveReplicationDevelopment] was (Author: sladymon): I added more thorough documentation on the design of replication in the wiki: *[HiveReplicationDevelopment | https://cwiki.apache.org/confluence/display/Hive/HiveReplicationDevelopment] > Document Replication support on wiki > > > Key: HIVE-10264 > URL: https://issues.apache.org/jira/browse/HIVE-10264 > Project: Hive > Issue Type: Sub-task > Components: Import/Export >Affects Versions: 1.2.0 >Reporter: Sushanth Sowmyan >Assignee: Sushanth Sowmyan > Labels: TODOC1.2 > Attachments: BirdsAndBees.pdf, EXIMReplicationReplayProtocol.pdf, > apache_hivedr.0.pdf > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10264) Document Replication support on wiki
[ https://issues.apache.org/jira/browse/HIVE-10264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406402#comment-15406402 ] Shannon Ladymon commented on HIVE-10264: I added more thorough documentation on the design of replication in the wiki: *[HiveReplicationDevelopment | https://cwiki.apache.org/confluence/display/Hive/HiveReplicationDevelopment] > Document Replication support on wiki > > > Key: HIVE-10264 > URL: https://issues.apache.org/jira/browse/HIVE-10264 > Project: Hive > Issue Type: Sub-task > Components: Import/Export >Affects Versions: 1.2.0 >Reporter: Sushanth Sowmyan >Assignee: Sushanth Sowmyan > Labels: TODOC1.2 > Attachments: BirdsAndBees.pdf, EXIMReplicationReplayProtocol.pdf, > apache_hivedr.0.pdf > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14412) Add a timezone-aware timestamp
[ https://issues.apache.org/jira/browse/HIVE-14412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406415#comment-15406415 ] Sergey Shelukhin commented on HIVE-14412: - Doesn't the problem only exist at the time of parsing back/forth, not storage time? When you parse a string into timestamp, it refers to an unambiguous UTC value. If the string has TZ information it's 100% unambiguous, if it doesn't I'm assuming the local tz is implied, perhaps we could clarify the UDFs or add missing ones with explicit TZ. When converting TZ to string the same applies, the TZ is either specified or comes from the environment. Storing the original TZ with the UTC timestamp doesn't resolve the first one, it may make the 2nd one less confusing, but it can also make it more confusing if the dates come in a strange TZ to a different reader. > Add a timezone-aware timestamp > -- > > Key: HIVE-14412 > URL: https://issues.apache.org/jira/browse/HIVE-14412 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Rui Li >Assignee: Rui Li > > Java's Timestamp stores the time elapsed since the epoch. While it's by > itself unambiguous, ambiguity comes when we parse a string into timestamp, or > convert a timestamp to string, causing problems like HIVE-14305. > To solve the issue, I think we should make timestamp aware of timezone. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-14412) Add a timezone-aware timestamp
[ https://issues.apache.org/jira/browse/HIVE-14412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406415#comment-15406415 ] Sergey Shelukhin edited comment on HIVE-14412 at 8/3/16 6:54 PM: - Doesn't the problem only exist at the time of parsing back/forth, not storage time? When you parse a string into timestamp, it refers to an unambiguous UTC value. If the string has TZ information it's 100% unambiguous, if it doesn't I'm assuming the local tz is implied, perhaps we could clarify the UDFs or add missing ones with explicit TZ. When converting TZ to string the same applies, the TZ is either specified or comes from the environment. Storing the original TZ with the UTC timestamp doesn't resolve the first one, it may make the 2nd one less confusing, but it can also make it more confusing if the dates come in a strange TZ to a different reader. If user wants to store original TZ they can store it manually (or at least perhaps it should be explicit in the datatype, if possible). was (Author: sershe): Doesn't the problem only exist at the time of parsing back/forth, not storage time? When you parse a string into timestamp, it refers to an unambiguous UTC value. If the string has TZ information it's 100% unambiguous, if it doesn't I'm assuming the local tz is implied, perhaps we could clarify the UDFs or add missing ones with explicit TZ. When converting TZ to string the same applies, the TZ is either specified or comes from the environment. Storing the original TZ with the UTC timestamp doesn't resolve the first one, it may make the 2nd one less confusing, but it can also make it more confusing if the dates come in a strange TZ to a different reader. > Add a timezone-aware timestamp > -- > > Key: HIVE-14412 > URL: https://issues.apache.org/jira/browse/HIVE-14412 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Rui Li >Assignee: Rui Li > > Java's Timestamp stores the time elapsed since the epoch. While it's by > itself unambiguous, ambiguity comes when we parse a string into timestamp, or > convert a timestamp to string, causing problems like HIVE-14305. > To solve the issue, I think we should make timestamp aware of timezone. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14409) Allow regex pattern in the output of qtest
[ https://issues.apache.org/jira/browse/HIVE-14409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406426#comment-15406426 ] Sergey Shelukhin commented on HIVE-14409: - Judging by the changes in the file, this doesn't scale very well (aside from not being able to overwrite with maven, which is a huge pain esp. if this is used in many tests). For 2 values, it's ok to "or" the orders... What if there are 4 values? I think for explain it's much easier to just enforce order. Or, given that list format is relatively well-defined, it could have a list of order-agnostic patterns, kinda like it has a list of patterns to ignore. Then it can extract the list for such prefixes/patterns and reorder as needed. We already have similar functionality in SORT_QUERY_RESULTS. > Allow regex pattern in the output of qtest > -- > > Key: HIVE-14409 > URL: https://issues.apache.org/jira/browse/HIVE-14409 > Project: Hive > Issue Type: Improvement > Components: Test >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka > Attachments: HIVE-14409.00.patch > > > We cannot guarantee the output from external libraries (ex. HashMap), so we > need to make the validation infrastructure to allow a sort within a single > line. > HIVE-13894, HIVE-13907, and HIVE-14088 are related. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-14409) Allow regex pattern in the output of qtest
[ https://issues.apache.org/jira/browse/HIVE-14409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406426#comment-15406426 ] Sergey Shelukhin edited comment on HIVE-14409 at 8/3/16 7:00 PM: - Judging by the changes in the out files, this doesn't scale very well (aside from not being able to overwrite with maven, which is a huge pain esp. if this is used in many tests). For 2 values, it's ok to "or" the orders... What if there are 4 values? I think for explain it's much easier to just enforce order. Or, given that list format is relatively well-defined, it could have a list of order-agnostic patterns, kinda like it has a list of patterns to ignore. Then it can extract the list for such prefixes/patterns and reorder as needed. We already have similar functionality in SORT_QUERY_RESULTS. was (Author: sershe): Judging by the changes in the file, this doesn't scale very well (aside from not being able to overwrite with maven, which is a huge pain esp. if this is used in many tests). For 2 values, it's ok to "or" the orders... What if there are 4 values? I think for explain it's much easier to just enforce order. Or, given that list format is relatively well-defined, it could have a list of order-agnostic patterns, kinda like it has a list of patterns to ignore. Then it can extract the list for such prefixes/patterns and reorder as needed. We already have similar functionality in SORT_QUERY_RESULTS. > Allow regex pattern in the output of qtest > -- > > Key: HIVE-14409 > URL: https://issues.apache.org/jira/browse/HIVE-14409 > Project: Hive > Issue Type: Improvement > Components: Test >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka > Attachments: HIVE-14409.00.patch > > > We cannot guarantee the output from external libraries (ex. HashMap), so we > need to make the validation infrastructure to allow a sort within a single > line. > HIVE-13894, HIVE-13907, and HIVE-14088 are related. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14407) issues when redirecting CLI output
[ https://issues.apache.org/jira/browse/HIVE-14407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406435#comment-15406435 ] Sergey Shelukhin commented on HIVE-14407: - [~prasanth_j] [~ashutoshc] fyi. I dunno if anyone feels ownership over CLI... it doesn't look like it :) > issues when redirecting CLI output > -- > > Key: HIVE-14407 > URL: https://issues.apache.org/jira/browse/HIVE-14407 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin > > I was running a script the other day and noticed that with > tez.print.exec.summary enabled, the colorful headers are still output to > console (with simple > redirect on Linux), while everything else including > the rows of the same tables goes into the file. Probably needs special > handling like we have for updatable vs non-updatable output for job progress. > Additionally, whereas CLI normally exits after running the script with i > argument, it does not exit when redirecting. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-14371) use datanucleus.rdbms.useColumnDefaultWhenNull when available
[ https://issues.apache.org/jira/browse/HIVE-14371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15404583#comment-15404583 ] Sergey Shelukhin edited comment on HIVE-14371 at 8/3/16 7:07 PM: - DN has been released, we can do a small upgrade here now. was (Author: sershe): DN has been release, we can do a small upgrade here now. > use datanucleus.rdbms.useColumnDefaultWhenNull when available > - > > Key: HIVE-14371 > URL: https://issues.apache.org/jira/browse/HIVE-14371 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin > > We are using a different property to work around postgres defaults issues in > DN 4 right now (HIVE-14322). The above property was just added to DN branches > to address this better; we should use that instead of the current workaround, > once the next DN 4.x version is released. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14204) Optimize loading dynamic partitions
[
https://issues.apache.org/jira/browse/HIVE-14204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406444#comment-15406444
]
Hive QA commented on HIVE-14204:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12821796/HIVE-14204.7.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/752/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/752/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-752/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.8.0_25 ]]
+ export JAVA_HOME=/usr/java/jdk1.8.0_25
+ JAVA_HOME=/usr/java/jdk1.8.0_25
+ export
PATH=/usr/java/jdk1.8.0_25/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/java/jdk1.8.0_25/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-MASTER-Build-752/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
4817d8b..4b6ac73 master -> origin/master
8adb913..f7fdd4e branch-2.1 -> origin/branch-2.1
+ git reset --hard HEAD
HEAD is now at 4817d8b HIVE-14380: Queries on tables with remote HDFS paths
fail in "encryption" checks. (Mithun Radhakrishnan, reviewed by Sergio Pena)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 2 commits, and can be fast-forwarded.
(use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at 4b6ac73 HIVE-14177: HS2 UI - List of recent queries shows most
recent query last (Gunther Hagleitner, reviewed by Szehon Ho)
+ git merge --ff-only origin/master
Already up-to-date.
+ git gc
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12821796 - PreCommit-HIVE-MASTER-Build
> Optimize loading dynamic partitions
>
>
> Key: HIVE-14204
> URL: https://issues.apache.org/jira/browse/HIVE-14204
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-14204.1.patch, HIVE-14204.3.patch,
> HIVE-14204.4.patch, HIVE-14204.6.patch, HIVE-14204.7.patch
>
>
> Lots of time is spent in sequential fashion to load dynamic partitioned
> dataset in driver side. E.g simple dynamic partitioned load as follows takes
> 300+ seconds
> {noformat}
> INSERT INTO web_sales_test partition(ws_sold_date_sk) select * from
> tpcds_bin_partitioned_orc_200.web_sales;
> Time taken to load dynamic partitions: 309.22 seconds
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12971) Hive Support for Kudu
[ https://issues.apache.org/jira/browse/HIVE-12971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406456#comment-15406456 ] ZAK commented on HIVE-12971: could you please update us whether Hive can support Kudu or only we can use Impala? > Hive Support for Kudu > - > > Key: HIVE-12971 > URL: https://issues.apache.org/jira/browse/HIVE-12971 > Project: Hive > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: Lenni Kuff >Assignee: bimal tandel > > JIRA for tracking work related to Hive/Kudu integration. > It would be useful to allow Kudu data to be accessible via Hive. This would > involve creating a Kudu SerDe/StorageHandler and implementing support for > QUERY and DML commands like SELECT, INSERT, UPDATE, and DELETE. Kudu > Input/OutputFormats classes already exist. The work can be staged to support > this functionality incrementally. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14408) thread safety issue in fast hashtable
[ https://issues.apache.org/jira/browse/HIVE-14408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-14408: Resolution: Fixed Fix Version/s: 2.0.2 2.1.1 2.2.0 1.3.0 Status: Resolved (was: Patch Available) Committed to million places > thread safety issue in fast hashtable > - > > Key: HIVE-14408 > URL: https://issues.apache.org/jira/browse/HIVE-14408 > Project: Hive > Issue Type: Bug >Reporter: Takahiko Saito >Assignee: Sergey Shelukhin > Fix For: 1.3.0, 2.2.0, 2.1.1, 2.0.2 > > Attachments: HIVE-14408.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14343) HiveDriverRunHookContext's command is null in HS2 mode
[
https://issues.apache.org/jira/browse/HIVE-14343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406538#comment-15406538
]
Chao Sun commented on HIVE-14343:
-
[~leftylev] Oops my bad. Thanks for pointing out!
> HiveDriverRunHookContext's command is null in HS2 mode
> --
>
> Key: HIVE-14343
> URL: https://issues.apache.org/jira/browse/HIVE-14343
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 2.2.0
>Reporter: Chao Sun
>Assignee: Chao Sun
> Fix For: 2.2.0
>
> Attachments: HIVE-14343.0.patch, HIVE-14343.1.patch
>
>
> Looking at the {{Driver#runInternal(String command, boolean
> alreadyCompiled)}}:
> {code}
> HiveDriverRunHookContext hookContext = new
> HiveDriverRunHookContextImpl(conf, command);
> // Get all the driver run hooks and pre-execute them.
> List driverRunHooks;
> {code}
> The context is initialized with the {{command}} passed in to the method.
> However, this command is always null if {{alreadyCompiled}} is true, which is
> the case for HS2 mode.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14415) Upgrade qtest execution framework to junit4 - TestPerfCliDriver
[
https://issues.apache.org/jira/browse/HIVE-14415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-14415:
Attachment: HIVE-14415.1.patch
#1 preliminary patch...i'm not sure the ptest execution framework will like it
or not... ;)
> Upgrade qtest execution framework to junit4 - TestPerfCliDriver
> ---
>
> Key: HIVE-14415
> URL: https://issues.apache.org/jira/browse/HIVE-14415
> Project: Hive
> Issue Type: Improvement
> Components: Tests
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-14415.1.patch
>
>
> I would like to upgrade the current maven+ant+velocimacro+junit4 qtest
> generation&execution framework to use only junit4 - while (trying) to keep
> all the existing features it provides.
> What I can't really do with the current one: execute easily a single qtests
> from an IDE (as a matter of fact I can...but it's way too complicated; after
> this it won't be a cake-walk either...but it will be a step closer ;)
> I think this change will make it more clear how these tests are configured
> and executed.
> I will do this in two phases, currently i will only change
> {{TestPerfCliDriver}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14397) Queries ran after reopening of tez session launches additional sessions
[ https://issues.apache.org/jira/browse/HIVE-14397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-14397: - Resolution: Fixed Fix Version/s: 2.1.1 2.2.0 Status: Resolved (was: Patch Available) Test failures are unrelated to this patch. Committed patch to master and branch-2.1 > Queries ran after reopening of tez session launches additional sessions > --- > > Key: HIVE-14397 > URL: https://issues.apache.org/jira/browse/HIVE-14397 > Project: Hive > Issue Type: Bug > Components: Tez >Affects Versions: 2.1.0, 2.2.0 >Reporter: Takahiko Saito >Assignee: Prasanth Jayachandran >Priority: Critical > Fix For: 2.2.0, 2.1.1 > > Attachments: HIVE-14397.1.patch, HIVE-14397.2.patch, > HIVE-14397.2.patch > > > Say we have configured hive.server2.tez.default.queues with 2 queues q1 and > q2 with default expiry interval of 5 mins. > After 5 mins of non-usage the sessions corresponding to queues q1 and q2 will > be expired. When new set of queries are issue after this expiry, the default > sessions backed by q1 and q2 and reopened again. Now when we run more queries > the reopened sessions are not used instead new session is opened. > At this point there will be 4 sessions running (2 abandoned sessions and 2 > current sessions). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14415) Upgrade qtest execution framework to junit4 - TestPerfCliDriver
[
https://issues.apache.org/jira/browse/HIVE-14415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-14415:
Attachment: (was: HIVE-14415.1.patch)
> Upgrade qtest execution framework to junit4 - TestPerfCliDriver
> ---
>
> Key: HIVE-14415
> URL: https://issues.apache.org/jira/browse/HIVE-14415
> Project: Hive
> Issue Type: Improvement
> Components: Tests
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-14415.1.patch
>
>
> I would like to upgrade the current maven+ant+velocimacro+junit4 qtest
> generation&execution framework to use only junit4 - while (trying) to keep
> all the existing features it provides.
> What I can't really do with the current one: execute easily a single qtests
> from an IDE (as a matter of fact I can...but it's way too complicated; after
> this it won't be a cake-walk either...but it will be a step closer ;)
> I think this change will make it more clear how these tests are configured
> and executed.
> I will do this in two phases, currently i will only change
> {{TestPerfCliDriver}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14415) Upgrade qtest execution framework to junit4 - TestPerfCliDriver
[
https://issues.apache.org/jira/browse/HIVE-14415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-14415:
Attachment: HIVE-14415.1.patch
> Upgrade qtest execution framework to junit4 - TestPerfCliDriver
> ---
>
> Key: HIVE-14415
> URL: https://issues.apache.org/jira/browse/HIVE-14415
> Project: Hive
> Issue Type: Improvement
> Components: Tests
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-14415.1.patch
>
>
> I would like to upgrade the current maven+ant+velocimacro+junit4 qtest
> generation&execution framework to use only junit4 - while (trying) to keep
> all the existing features it provides.
> What I can't really do with the current one: execute easily a single qtests
> from an IDE (as a matter of fact I can...but it's way too complicated; after
> this it won't be a cake-walk either...but it will be a step closer ;)
> I think this change will make it more clear how these tests are configured
> and executed.
> I will do this in two phases, currently i will only change
> {{TestPerfCliDriver}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14414) Fix TestHiveMetaStoreTxns UTs
[ https://issues.apache.org/jira/browse/HIVE-14414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406598#comment-15406598 ] Wei Zheng commented on HIVE-14414: -- +1 > Fix TestHiveMetaStoreTxns UTs > - > > Key: HIVE-14414 > URL: https://issues.apache.org/jira/browse/HIVE-14414 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.3.0, 2.2.0, 2.1.1 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-14414.patch > > > Missed updating these tests in HIVE-14350 commit -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14416) NPE trying to move results of a subselect into an RDBMS using HPL/SQL
[
https://issues.apache.org/jira/browse/HIVE-14416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Carter Shanklin updated HIVE-14416:
---
Description:
I was trying to use HPL/SQL to move some records from Hive to MySQL using this
script:
{code}
MAP OBJECT remote TO pgtable AT mydbconn;
insert into remote values ( select * from hivetable );
{code}
When I run this script I get this NPE:
{code}
Open connection: jdbc:hive2://hdp250.example.com:1 (285 ms)
Starting query
Query executed successfully (127 ms)
Exception in thread "main" java.lang.NullPointerException
at org.apache.hive.hplsql.Select.getIntoCount(Select.java:405)
at org.apache.hive.hplsql.Select.select(Select.java:88)
at org.apache.hive.hplsql.Exec.visitSelect_stmt(Exec.java:1002)
at org.apache.hive.hplsql.Exec.visitSelect_stmt(Exec.java:52)
at
org.apache.hive.hplsql.HplsqlParser$Select_stmtContext.accept(HplsqlParser.java:14768)
at
org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visitChildren(AbstractParseTreeVisitor.java:70)
at org.apache.hive.hplsql.Exec.visitStmt(Exec.java:994)
at org.apache.hive.hplsql.Exec.visitStmt(Exec.java:52)
at
org.apache.hive.hplsql.HplsqlParser$StmtContext.accept(HplsqlParser.java:1012)
at
org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visitChildren(AbstractParseTreeVisitor.java:70)
at
org.apache.hive.hplsql.HplsqlBaseVisitor.visitBlock(HplsqlBaseVisitor.java:28)
at
org.apache.hive.hplsql.HplsqlParser$BlockContext.accept(HplsqlParser.java:446)
at
org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visitChildren(AbstractParseTreeVisitor.java:70)
at org.apache.hive.hplsql.Exec.visitProgram(Exec.java:901)
at org.apache.hive.hplsql.Exec.visitProgram(Exec.java:52)
at
org.apache.hive.hplsql.HplsqlParser$ProgramContext.accept(HplsqlParser.java:389)
at
org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visit(AbstractParseTreeVisitor.java:42)
at org.apache.hive.hplsql.Exec.run(Exec.java:760)
at org.apache.hive.hplsql.Exec.run(Exec.java:736)
at org.apache.hive.hplsql.Hplsql.main(Hplsql.java:23)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
{code}
This is my configuration:
{code}
hplsql.conn.default
myhiveconn
hplsql.conn.myhiveconn
org.apache.hive.jdbc.HiveDriver;jdbc:hive2://hdp250.example.com:1
hplsql.conn.mydbconn
com.mysql.jdbc.Driver;jdbc:mysql://hdp250.example.com:3306/hive;hive;vagrant
{code}
was:
I was trying to use HPL/SQL to move some records from Hive to Postgres using
this script:
{code}
MAP OBJECT remote TO pgtable AT mydbconn;
insert into remote values ( select * from hivetable );
{code}
When I run this script I get this NPE:
{code}
Open connection: jdbc:hive2://hdp250.example.com:1 (285 ms)
Starting query
Query executed successfully (127 ms)
Exception in thread "main" java.lang.NullPointerException
at org.apache.hive.hplsql.Select.getIntoCount(Select.java:405)
at org.apache.hive.hplsql.Select.select(Select.java:88)
at org.apache.hive.hplsql.Exec.visitSelect_stmt(Exec.java:1002)
at org.apache.hive.hplsql.Exec.visitSelect_stmt(Exec.java:52)
at
org.apache.hive.hplsql.HplsqlParser$Select_stmtContext.accept(HplsqlParser.java:14768)
at
org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visitChildren(AbstractParseTreeVisitor.java:70)
at org.apache.hive.hplsql.Exec.visitStmt(Exec.java:994)
at org.apache.hive.hplsql.Exec.visitStmt(Exec.java:52)
at
org.apache.hive.hplsql.HplsqlParser$StmtContext.accept(HplsqlParser.java:1012)
at
org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visitChildren(AbstractParseTreeVisitor.java:70)
at
org.apache.hive.hplsql.HplsqlBaseVisitor.visitBlock(HplsqlBaseVisitor.java:28)
at
org.apache.hive.hplsql.HplsqlParser$BlockContext.accept(HplsqlParser.java:446)
at
org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visitChildren(AbstractParseTreeVisitor.java:70)
at org.apache.hive.hplsql.Exec.visitProgram(Exec.java:901)
at org.apache.hive.hplsql.Exec.visitProgram(Exec.java:52)
at
org.apache.hive.hplsql.HplsqlParser$ProgramContext.accept(HplsqlParser.java:389)
at
org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visit(AbstractParseTreeVisitor.java:42)
at org.apache.hive.hplsql.Exec.run(Exec.java:760)
at org.apache.hive.hplsql.Exec.run(Exec.java:7
[jira] [Commented] (HIVE-12971) Hive Support for Kudu
[ https://issues.apache.org/jira/browse/HIVE-12971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406637#comment-15406637 ] bimal tandel commented on HIVE-12971: - [~alziyad], You can refer to the github page and check the current state if you would like to use Hive with Kudu. Currently you should be able to create a table, and run any hive queries (including using Hive On Spark). As the Jira is open please report any issues and challenges on the Github page and I will try to help you out. > Hive Support for Kudu > - > > Key: HIVE-12971 > URL: https://issues.apache.org/jira/browse/HIVE-12971 > Project: Hive > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: Lenni Kuff >Assignee: bimal tandel > > JIRA for tracking work related to Hive/Kudu integration. > It would be useful to allow Kudu data to be accessible via Hive. This would > involve creating a Kudu SerDe/StorageHandler and implementing support for > QUERY and DML commands like SELECT, INSERT, UPDATE, and DELETE. Kudu > Input/OutputFormats classes already exist. The work can be staged to support > this functionality incrementally. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14259) remove FileUtils.isSubDir() method
[ https://issues.apache.org/jira/browse/HIVE-14259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-14259: Summary: remove FileUtils.isSubDir() method (was: FileUtils.isSubDir may return incorrect result) > remove FileUtils.isSubDir() method > -- > > Key: HIVE-14259 > URL: https://issues.apache.org/jira/browse/HIVE-14259 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Minor > Attachments: HIVE-14259.1.patch, HIVE-14259.2.patch, > HIVE-14259.3.patch, HIVE-14259.4.patch > > > while I was working on HIVE-12244 i've looked around for utility > methods...i've found this method; but it considers path: `/dir12` inside > `/dir1` > which is not true -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14259) remove FileUtils.isSubDir() method
[ https://issues.apache.org/jira/browse/HIVE-14259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-14259: Status: Open (was: Patch Available) > remove FileUtils.isSubDir() method > -- > > Key: HIVE-14259 > URL: https://issues.apache.org/jira/browse/HIVE-14259 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Minor > Attachments: HIVE-14259.1.patch, HIVE-14259.2.patch, > HIVE-14259.3.patch, HIVE-14259.4.patch > > > while I was working on HIVE-12244 i've looked around for utility > methods...i've found this method; but it considers path: `/dir12` inside > `/dir1` > which is not true -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14259) remove FileUtils.isSubDir() method
[ https://issues.apache.org/jira/browse/HIVE-14259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-14259: Status: Patch Available (was: Open) > remove FileUtils.isSubDir() method > -- > > Key: HIVE-14259 > URL: https://issues.apache.org/jira/browse/HIVE-14259 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Minor > Attachments: HIVE-14259.1.patch, HIVE-14259.2.patch, > HIVE-14259.3.patch, HIVE-14259.4.patch, HIVE-14259.5.patch > > > while I was working on HIVE-12244 i've looked around for utility > methods...i've found this method; but it considers path: `/dir12` inside > `/dir1` > which is not true -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14259) remove FileUtils.isSubDir() method
[ https://issues.apache.org/jira/browse/HIVE-14259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-14259: Attachment: HIVE-14259.5.patch rebased again ;) reviewboard entry: https://reviews.apache.org/r/50772/ > remove FileUtils.isSubDir() method > -- > > Key: HIVE-14259 > URL: https://issues.apache.org/jira/browse/HIVE-14259 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Minor > Attachments: HIVE-14259.1.patch, HIVE-14259.2.patch, > HIVE-14259.3.patch, HIVE-14259.4.patch, HIVE-14259.5.patch > > > while I was working on HIVE-12244 i've looked around for utility > methods...i've found this method; but it considers path: `/dir12` inside > `/dir1` > which is not true -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14378) Data size may be estimated as 0 if no columns are being projected after an operator
[ https://issues.apache.org/jira/browse/HIVE-14378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-14378: Attachment: HIVE-14378.4.patch > Data size may be estimated as 0 if no columns are being projected after an > operator > --- > > Key: HIVE-14378 > URL: https://issues.apache.org/jira/browse/HIVE-14378 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer, Statistics >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-14378.2.patch, HIVE-14378.3.patch, > HIVE-14378.3.patch, HIVE-14378.4.patch, HIVE-14378.patch > > > in those cases we still emit rows.. but they may not have any columns within > it. We shouldn't estimate 0 data size in such cases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14378) Data size may be estimated as 0 if no columns are being projected after an operator
[ https://issues.apache.org/jira/browse/HIVE-14378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-14378: Status: Patch Available (was: Open) > Data size may be estimated as 0 if no columns are being projected after an > operator > --- > > Key: HIVE-14378 > URL: https://issues.apache.org/jira/browse/HIVE-14378 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer, Statistics >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-14378.2.patch, HIVE-14378.3.patch, > HIVE-14378.3.patch, HIVE-14378.4.patch, HIVE-14378.patch > > > in those cases we still emit rows.. but they may not have any columns within > it. We shouldn't estimate 0 data size in such cases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14378) Data size may be estimated as 0 if no columns are being projected after an operator
[ https://issues.apache.org/jira/browse/HIVE-14378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-14378: Status: Open (was: Patch Available) > Data size may be estimated as 0 if no columns are being projected after an > operator > --- > > Key: HIVE-14378 > URL: https://issues.apache.org/jira/browse/HIVE-14378 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer, Statistics >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-14378.2.patch, HIVE-14378.3.patch, > HIVE-14378.3.patch, HIVE-14378.4.patch, HIVE-14378.patch > > > in those cases we still emit rows.. but they may not have any columns within > it. We shouldn't estimate 0 data size in such cases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14146) Column comments with "\n" character "corrupts" table metadata
[
https://issues.apache.org/jira/browse/HIVE-14146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406674#comment-15406674
]
Hive QA commented on HIVE-14146:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12821848/HIVE-14146.9.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 16 failed/errored test(s), 10433 tests
executed
*Failed tests:*
{noformat}
TestMsgBusConnection - did not produce a TEST-*.xml file
TestQueryLifeTimeHook - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_concatenate_indexed_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_table_invalidate_column_stats
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_columnstats_part_coltype
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_with_constraints
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auth
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_show_indexes_edge_cases
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_show_indexes_syntax
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_orc_llap_counters
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_alter_concatenate_indexed_table
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_alter_view_as_select_with_partition
org.apache.hadoop.hive.metastore.TestHiveMetaStoreTxns.stringifyValidTxns
org.apache.hadoop.hive.metastore.TestHiveMetaStoreTxns.testTxnRange
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/753/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/753/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-753/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 16 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12821848 - PreCommit-HIVE-MASTER-Build
> Column comments with "\n" character "corrupts" table metadata
> -
>
> Key: HIVE-14146
> URL: https://issues.apache.org/jira/browse/HIVE-14146
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 2.2.0
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-14146.2.patch, HIVE-14146.3.patch,
> HIVE-14146.4.patch, HIVE-14146.5.patch, HIVE-14146.6.patch,
> HIVE-14146.7.patch, HIVE-14146.8.patch, HIVE-14146.9.patch, HIVE-14146.patch
>
>
> Create a table with the following(noting the \n in the COMMENT):
> {noformat}
> CREATE TABLE commtest(first_nm string COMMENT 'Indicates First name\nof an
> individual’);
> {noformat}
> Describe shows that now the metadata is messed up:
> {noformat}
> beeline> describe commtest;
> +---++---+--+
> | col_name | data_type |comment|
> +---++---+--+
> | first_nm | string | Indicates First name |
> | of an individual | NULL | NULL |
> +---++---+--+
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14415) Upgrade qtest execution framework to junit4 - TestPerfCliDriver
[
https://issues.apache.org/jira/browse/HIVE-14415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-14415:
Status: Patch Available (was: Open)
reviewboard:
https://reviews.apache.org/r/50768/
> Upgrade qtest execution framework to junit4 - TestPerfCliDriver
> ---
>
> Key: HIVE-14415
> URL: https://issues.apache.org/jira/browse/HIVE-14415
> Project: Hive
> Issue Type: Improvement
> Components: Tests
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-14415.1.patch
>
>
> I would like to upgrade the current maven+ant+velocimacro+junit4 qtest
> generation&execution framework to use only junit4 - while (trying) to keep
> all the existing features it provides.
> What I can't really do with the current one: execute easily a single qtests
> from an IDE (as a matter of fact I can...but it's way too complicated; after
> this it won't be a cake-walk either...but it will be a step closer ;)
> I think this change will make it more clear how these tests are configured
> and executed.
> I will do this in two phases, currently i will only change
> {{TestPerfCliDriver}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14419) fix test TestCliDriver/testCliDriver#stats_list_bucket
[ https://issues.apache.org/jira/browse/HIVE-14419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-14419: Status: Patch Available (was: Open) > fix test TestCliDriver/testCliDriver#stats_list_bucket > -- > > Key: HIVE-14419 > URL: https://issues.apache.org/jira/browse/HIVE-14419 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich > Attachments: HIVE-14419.1.patch > > > it looks like somehow this qtest have evaded its update in HIVE-14329 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14419) fix test TestCliDriver/testCliDriver#stats_list_bucket
[ https://issues.apache.org/jira/browse/HIVE-14419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-14419: Attachment: HIVE-14419.1.patch > fix test TestCliDriver/testCliDriver#stats_list_bucket > -- > > Key: HIVE-14419 > URL: https://issues.apache.org/jira/browse/HIVE-14419 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich > Attachments: HIVE-14419.1.patch > > > it looks like somehow this qtest have evaded its update in HIVE-14329 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14419) fix test TestCliDriver/testCliDriver#stats_list_bucket
[ https://issues.apache.org/jira/browse/HIVE-14419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-14419: Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Pushed to master. Thanks, Zoltan! > fix test TestCliDriver/testCliDriver#stats_list_bucket > -- > > Key: HIVE-14419 > URL: https://issues.apache.org/jira/browse/HIVE-14419 > Project: Hive > Issue Type: Improvement > Components: Test >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich > Fix For: 2.2.0 > > Attachments: HIVE-14419.1.patch > > > it looks like somehow this qtest have evaded its update in HIVE-14329 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14419) fix test TestCliDriver/testCliDriver#stats_list_bucket
[ https://issues.apache.org/jira/browse/HIVE-14419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-14419: Component/s: Test > fix test TestCliDriver/testCliDriver#stats_list_bucket > -- > > Key: HIVE-14419 > URL: https://issues.apache.org/jira/browse/HIVE-14419 > Project: Hive > Issue Type: Improvement > Components: Test >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich > Fix For: 2.2.0 > > Attachments: HIVE-14419.1.patch > > > it looks like somehow this qtest have evaded its update in HIVE-14329 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14303) CommonJoinOperator.checkAndGenObject should return directly to avoid NPE if ExecReducer.close is called twice.
[
https://issues.apache.org/jira/browse/HIVE-14303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406753#comment-15406753
]
zhihai xu commented on HIVE-14303:
--
The test failures are not related to my patch, TestHiveMetaStoreTxns and
testCliDriver_orc_llap_counters failed with my change.
TestPigHBaseStorageHandler and TestMiniLlapCliDriver passed in my local build
with my change.
> CommonJoinOperator.checkAndGenObject should return directly to avoid NPE if
> ExecReducer.close is called twice.
> --
>
> Key: HIVE-14303
> URL: https://issues.apache.org/jira/browse/HIVE-14303
> Project: Hive
> Issue Type: Bug
>Reporter: zhihai xu
>Assignee: zhihai xu
> Fix For: 2.2.0
>
> Attachments: HIVE-14303.0.patch, HIVE-14303.1.patch
>
>
> CommonJoinOperator.checkAndGenObject should return directly (after
> {{CommonJoinOperator.closeOp}} was called ) to avoid NPE if ExecReducer.close
> is called twice. ExecReducer.close implements Closeable interface and
> ExecReducer.close can be called multiple time. We saw the following NPE which
> hide the real exception due to this bug.
> {code}
> Error: java.lang.RuntimeException: Hive Runtime Error while closing
> operators: null
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:296)
> at org.apache.hadoop.io.IOUtils.cleanup(IOUtils.java:244)
> at
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:459)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:718)
> at
> org.apache.hadoop.hive.ql.exec.JoinOperator.endGroup(JoinOperator.java:256)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:284)
> ... 8 more
> {code}
> The code from ReduceTask.runOldReducer:
> {code}
> reducer.close(); //line 453
> reducer = null;
>
> out.close(reporter);
> out = null;
> } finally {
> IOUtils.cleanup(LOG, reducer);// line 459
> closeQuietly(out, reporter);
> }
> {code}
> Based on the above stack trace and code, reducer.close() is called twice
> because the exception happened when reducer.close() is called for the first
> time at line 453, the code exit before reducer was set to null.
> NullPointerException is triggered when reducer.close() is called for the
> second time in IOUtils.cleanup at line 459. NullPointerException hide the
> real exception which happened when reducer.close() is called for the first
> time at line 453.
> The reason for NPE is:
> The first reducer.close called CommonJoinOperator.closeOp which clear
> {{storage}}
> {code}
> Arrays.fill(storage, null);
> {code}
> the second reduce.close generated NPE due to null {{storage[alias]}} which is
> set to null by first reducer.close.
> The following reducer log can give more proof:
> {code}
> 2016-07-14 22:24:51,016 INFO [main]
> org.apache.hadoop.hive.ql.exec.JoinOperator: 0 finished. closing...
> 2016-07-14 22:24:51,016 INFO [main]
> org.apache.hadoop.hive.ql.exec.JoinOperator: 0 finished. closing...
> 2016-07-14 22:24:51,016 INFO [main]
> org.apache.hadoop.hive.ql.exec.JoinOperator: SKEWJOINFOLLOWUPJOBS:0
> 2016-07-14 22:24:51,016 INFO [main]
> org.apache.hadoop.hive.ql.exec.SelectOperator: 1 finished. closing...
> 2016-07-14 22:24:51,016 INFO [main]
> org.apache.hadoop.hive.ql.exec.SelectOperator: 2 finished. closing...
> 2016-07-14 22:24:51,016 INFO [main]
> org.apache.hadoop.hive.ql.exec.SelectOperator: 3 finished. closing...
> 2016-07-14 22:24:51,016 INFO [main]
> org.apache.hadoop.hive.ql.exec.FileSinkOperator: 4 finished. closing...
> 2016-07-14 22:24:51,016 INFO [main]
> org.apache.hadoop.hive.ql.exec.FileSinkOperator: FS[4]: records written -
> 53466
> 2016-07-14 22:25:11,555 ERROR [main] ExecReducer: Hit error while closing
> operators - failing tree
> 2016-07-14 22:25:11,649 WARN [main] org.apache.hadoop.mapred.YarnChild:
> Exception running child : java.lang.RuntimeException: Hive Runtime Error
> while closing operators: null
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:296)
> at org.apache.hadoop.io.IOUtils.cleanup(IOUtils.java:244)
> a
[jira] [Updated] (HIVE-13822) TestPerfCliDriver throws warning in StatsSetupConst that JsonParser cannot parse COLUMN_STATS
[
https://issues.apache.org/jira/browse/HIVE-13822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan updated HIVE-13822:
-
Attachment: HIVE-13822.4.patch
> TestPerfCliDriver throws warning in StatsSetupConst that JsonParser cannot
> parse COLUMN_STATS
> --
>
> Key: HIVE-13822
> URL: https://issues.apache.org/jira/browse/HIVE-13822
> Project: Hive
> Issue Type: Bug
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-13822.1.patch, HIVE-13822.2.patch,
> HIVE-13822.3.patch, HIVE-13822.4.patch
>
>
> Thanks to [~jcamachorodriguez] for uncovering this issue as part of
> HIVE-13269. StatsSetupConst.areColumnStatsUptoDate() is used to check whether
> stats are up-to-date. In case of PerfCliDriver, ‘false’ (thus, not
> up-to-date) is returned and the following debug message in the logs:
> {code}
> In StatsSetupConst, JsonParser can not parse COLUMN_STATS. (line 190 in
> StatsSetupConst)
> {code}
> Looks like the issue started happening after HIVE-12261 went in.
> The fix would be to replace
> {color:red}COLUMN_STATS_ACCURATE,true{color}
> with
> {color:green}COLUMN_STATS_ACCURATE,{"COLUMN_STATS":{"key":"true","value":"true"},"BASIC_STATS":"true"}{color}
> where key, value are the column names.
> in data/files/tpcds-perf/metastore_export/csv/TABLE_PARAMS.txt
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13822) TestPerfCliDriver throws warning in StatsSetupConst that JsonParser cannot parse COLUMN_STATS
[
https://issues.apache.org/jira/browse/HIVE-13822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan updated HIVE-13822:
-
Status: Patch Available (was: Open)
> TestPerfCliDriver throws warning in StatsSetupConst that JsonParser cannot
> parse COLUMN_STATS
> --
>
> Key: HIVE-13822
> URL: https://issues.apache.org/jira/browse/HIVE-13822
> Project: Hive
> Issue Type: Bug
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-13822.1.patch, HIVE-13822.2.patch,
> HIVE-13822.3.patch, HIVE-13822.4.patch
>
>
> Thanks to [~jcamachorodriguez] for uncovering this issue as part of
> HIVE-13269. StatsSetupConst.areColumnStatsUptoDate() is used to check whether
> stats are up-to-date. In case of PerfCliDriver, ‘false’ (thus, not
> up-to-date) is returned and the following debug message in the logs:
> {code}
> In StatsSetupConst, JsonParser can not parse COLUMN_STATS. (line 190 in
> StatsSetupConst)
> {code}
> Looks like the issue started happening after HIVE-12261 went in.
> The fix would be to replace
> {color:red}COLUMN_STATS_ACCURATE,true{color}
> with
> {color:green}COLUMN_STATS_ACCURATE,{"COLUMN_STATS":{"key":"true","value":"true"},"BASIC_STATS":"true"}{color}
> where key, value are the column names.
> in data/files/tpcds-perf/metastore_export/csv/TABLE_PARAMS.txt
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13822) TestPerfCliDriver throws warning in StatsSetupConst that JsonParser cannot parse COLUMN_STATS
[
https://issues.apache.org/jira/browse/HIVE-13822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan updated HIVE-13822:
-
Status: Open (was: Patch Available)
> TestPerfCliDriver throws warning in StatsSetupConst that JsonParser cannot
> parse COLUMN_STATS
> --
>
> Key: HIVE-13822
> URL: https://issues.apache.org/jira/browse/HIVE-13822
> Project: Hive
> Issue Type: Bug
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-13822.1.patch, HIVE-13822.2.patch,
> HIVE-13822.3.patch, HIVE-13822.4.patch
>
>
> Thanks to [~jcamachorodriguez] for uncovering this issue as part of
> HIVE-13269. StatsSetupConst.areColumnStatsUptoDate() is used to check whether
> stats are up-to-date. In case of PerfCliDriver, ‘false’ (thus, not
> up-to-date) is returned and the following debug message in the logs:
> {code}
> In StatsSetupConst, JsonParser can not parse COLUMN_STATS. (line 190 in
> StatsSetupConst)
> {code}
> Looks like the issue started happening after HIVE-12261 went in.
> The fix would be to replace
> {color:red}COLUMN_STATS_ACCURATE,true{color}
> with
> {color:green}COLUMN_STATS_ACCURATE,{"COLUMN_STATS":{"key":"true","value":"true"},"BASIC_STATS":"true"}{color}
> where key, value are the column names.
> in data/files/tpcds-perf/metastore_export/csv/TABLE_PARAMS.txt
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14414) Fix TestHiveMetaStoreTxns UTs
[ https://issues.apache.org/jira/browse/HIVE-14414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406789#comment-15406789 ] Eugene Koifman commented on HIVE-14414: --- All Failed Tests |Test Name |Duration |Age| |org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_skewtable |17 sec |1| |org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching |4.6 sec| 1| |org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure |5.1 sec| 1| |org.apache.hive.hcatalog.listener.TestDbNotificationListener.cleanupNotifs 1 |min 0 sec|2| |org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket |9 sec| 5| |org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_schema_evolution_float |52 sec|6| |org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_selectindate |14 sec |43| |org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_avrocountemptytbl |10 sec|43| |org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_order_null |39 sec|43| |org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_join_with_different_encryption_keys |1 min 34 sec| 169| failures are not related since patch only has changes to a singe TestHiveMetaStoreTxns file. > Fix TestHiveMetaStoreTxns UTs > - > > Key: HIVE-14414 > URL: https://issues.apache.org/jira/browse/HIVE-14414 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.3.0, 2.2.0, 2.1.1 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-14414.patch > > > Missed updating these tests in HIVE-14350 commit -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14396) CBO: Calcite Operator To Hive Operator (Calcite Return Path): TestCliDriver count.q failure
[
https://issues.apache.org/jira/browse/HIVE-14396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406796#comment-15406796
]
Hari Sankar Sivarama Subramaniyan commented on HIVE-14396:
--
Looks like a duplicate of HIVE-12803.
> CBO: Calcite Operator To Hive Operator (Calcite Return Path): TestCliDriver
> count.q failure
> ---
>
> Key: HIVE-14396
> URL: https://issues.apache.org/jira/browse/HIVE-14396
> Project: Hive
> Issue Type: Sub-task
> Components: CBO
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>
> Currently there are three different failures
> Set hive.cbo.returnpath.hiveop=true for all cases.
> 1) First case is wrong result for following query
> {code:title=failure 1 Wrong result}
> explain select count(1), count(*), count(a), count(b), count(c), count(d),
> count(distinct a), count(distinct b), count(distinct c), count(distinct d),
> count(distinct a,b), count(distinct b,c), count(distinct c,d), count(distinct
> a,d), count(distinct a,c), count(distinct b,d), count(distinct a,b,c),
> count(distinct b,c,d), count(distinct a,c,d), count(distinct a,b,d),
> count(distinct a,b,c,d) from abcd;
> {code}
> This occurs due to a bug in HiveCalciteUtil.getExprNodes. While looking for
> corresponding expression for a aggregate function's argument wrong index is
> being used.
> 2) Out of bound exception for following
> {code}
> set hive.map.aggr=false
> explain select count(1), count(*), count(a), count(b), count(c), count(d),
> count(distinct a), count(distinct b), count(distinct c), count(distinct d),
> count(distinct a,b), count(distinct b,c), count(distinct c,d), count(distinct
> a,d), count(distinct a,c), count(distinct b,d), count(distinct a,b,c),
> count(distinct b,c,d), count(distinct a,c,d), count(distinct a,b,d),
> count(distinct a,b,c,d) from abcd;
> {code}
> The above happens while converting Calcite Aggregation to Hive's group by
> operator.
> 3) Once the above case with exception is fixed same query with
> hive.map.aggr=false give wrong results. Problem in this case is that while
> creating expression for aggregate function's argument we end up with wrong
> column info from underlying reduce sink operator.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13822) TestPerfCliDriver throws warning in StatsSetupConst that JsonParser cannot parse COLUMN_STATS
[
https://issues.apache.org/jira/browse/HIVE-13822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406797#comment-15406797
]
Hari Sankar Sivarama Subramaniyan commented on HIVE-13822:
--
The failures were file diffs which went away after a rebase. Re uploading the
patch.
> TestPerfCliDriver throws warning in StatsSetupConst that JsonParser cannot
> parse COLUMN_STATS
> --
>
> Key: HIVE-13822
> URL: https://issues.apache.org/jira/browse/HIVE-13822
> Project: Hive
> Issue Type: Bug
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-13822.1.patch, HIVE-13822.2.patch,
> HIVE-13822.3.patch, HIVE-13822.4.patch
>
>
> Thanks to [~jcamachorodriguez] for uncovering this issue as part of
> HIVE-13269. StatsSetupConst.areColumnStatsUptoDate() is used to check whether
> stats are up-to-date. In case of PerfCliDriver, ‘false’ (thus, not
> up-to-date) is returned and the following debug message in the logs:
> {code}
> In StatsSetupConst, JsonParser can not parse COLUMN_STATS. (line 190 in
> StatsSetupConst)
> {code}
> Looks like the issue started happening after HIVE-12261 went in.
> The fix would be to replace
> {color:red}COLUMN_STATS_ACCURATE,true{color}
> with
> {color:green}COLUMN_STATS_ACCURATE,{"COLUMN_STATS":{"key":"true","value":"true"},"BASIC_STATS":"true"}{color}
> where key, value are the column names.
> in data/files/tpcds-perf/metastore_export/csv/TABLE_PARAMS.txt
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-12803) CBO: Calcite Operator To Hive Operator (Calcite Return Path): MiniTezCliDriver count.q failure
[
https://issues.apache.org/jira/browse/HIVE-12803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg reassigned HIVE-12803:
--
Assignee: Vineet Garg
> CBO: Calcite Operator To Hive Operator (Calcite Return Path):
> MiniTezCliDriver count.q failure
> --
>
> Key: HIVE-12803
> URL: https://issues.apache.org/jira/browse/HIVE-12803
> Project: Hive
> Issue Type: Sub-task
> Components: CBO
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Vineet Garg
>
> {code}
> select a, count(distinct b), count(distinct c), sum(d) from abcd group by a;
> {code}
> Set hive.cbo.returnpath.hiveop=true;
> {code}
> java.lang.IndexOutOfBoundsException: Index: 5, Size: 5
> at java.util.ArrayList.rangeCheck(ArrayList.java:635) ~[?:1.7.0_79]
> at java.util.ArrayList.get(ArrayList.java:411) ~[?:1.7.0_79]
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.translator.HiveGBOpConvUtil.genReduceSideGB1NoMapGB(HiveGBOpConvUtil.java:1060)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.translator.HiveGBOpConvUtil.genNoMapSideGBNoSkew(HiveGBOpConvUtil.java:473)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.translator.HiveGBOpConvUtil.translateGB(HiveGBOpConvUtil.java:304)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.translator.HiveOpConverter.visit(HiveOpConverter.java:398)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.translator.HiveOpConverter.dispatch(HiveOpConverter.java:181)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.translator.HiveOpConverter.convert(HiveOpConverter.java:154)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.getOptimizedHiveOPDag(CalcitePlanner.java:688)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:266)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10094)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:231)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:237)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:237)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:471)
> [hive-exec-2.1.0-SNAPSHOT.jar:?]
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:311)
> [hive-exec-2.1.0-SNAPSHOT.jar:?]
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1149)
> [hive-exec-2.1.0-SNAPSHOT.jar:?]
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1237)
> [hive-exec-2.1.0-SNAPSHOT.jar:?]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-12924) CBO: Calcite Operator To Hive Operator (Calcite Return Path): TestCliDriver groupby_ppr_multi_distinct.q failure
[
https://issues.apache.org/jira/browse/HIVE-12924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg reassigned HIVE-12924:
--
Assignee: Vineet Garg (was: Hari Sankar Sivarama Subramaniyan)
> CBO: Calcite Operator To Hive Operator (Calcite Return Path): TestCliDriver
> groupby_ppr_multi_distinct.q failure
>
>
> Key: HIVE-12924
> URL: https://issues.apache.org/jira/browse/HIVE-12924
> Project: Hive
> Issue Type: Sub-task
> Components: CBO
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Vineet Garg
> Attachments: HIVE-12924.1.patch, HIVE-12924.2.patch,
> HIVE-12924.3.patch
>
>
> {code}
> EXPLAIN EXTENDED
> FROM srcpart src
> INSERT OVERWRITE TABLE dest1
> SELECT substr(src.key,1,1), count(DISTINCT substr(src.value,5)),
> concat(substr(src.key,1,1),sum(substr(src.value,5))), sum(DISTINCT
> substr(src.value, 5)), count(DISTINCT src.value)
> WHERE src.ds = '2008-04-08'
> GROUP BY substr(src.key,1,1)
> {code}
> Ended Job = job_local968043618_0742 with errors
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14414) Fix TestHiveMetaStoreTxns UTs
[ https://issues.apache.org/jira/browse/HIVE-14414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-14414: -- Resolution: Fixed Fix Version/s: 2.1.1 2.2.0 1.3.0 Status: Resolved (was: Patch Available) > Fix TestHiveMetaStoreTxns UTs > - > > Key: HIVE-14414 > URL: https://issues.apache.org/jira/browse/HIVE-14414 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.3.0, 2.2.0, 2.1.1 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Fix For: 1.3.0, 2.2.0, 2.1.1 > > Attachments: HIVE-14414.patch > > > Missed updating these tests in HIVE-14350 commit -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14204) Optimize loading dynamic partitions
[
https://issues.apache.org/jira/browse/HIVE-14204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-14204:
Status: Open (was: Patch Available)
Had to rebase after HIVE-14400
> Optimize loading dynamic partitions
>
>
> Key: HIVE-14204
> URL: https://issues.apache.org/jira/browse/HIVE-14204
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-14204.1.patch, HIVE-14204.3.patch,
> HIVE-14204.4.patch, HIVE-14204.6.patch, HIVE-14204.7.patch
>
>
> Lots of time is spent in sequential fashion to load dynamic partitioned
> dataset in driver side. E.g simple dynamic partitioned load as follows takes
> 300+ seconds
> {noformat}
> INSERT INTO web_sales_test partition(ws_sold_date_sk) select * from
> tpcds_bin_partitioned_orc_200.web_sales;
> Time taken to load dynamic partitions: 309.22 seconds
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14204) Optimize loading dynamic partitions
[
https://issues.apache.org/jira/browse/HIVE-14204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-14204:
Attachment: HIVE-14204.8.patch
> Optimize loading dynamic partitions
>
>
> Key: HIVE-14204
> URL: https://issues.apache.org/jira/browse/HIVE-14204
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-14204.1.patch, HIVE-14204.3.patch,
> HIVE-14204.4.patch, HIVE-14204.6.patch, HIVE-14204.7.patch, HIVE-14204.8.patch
>
>
> Lots of time is spent in sequential fashion to load dynamic partitioned
> dataset in driver side. E.g simple dynamic partitioned load as follows takes
> 300+ seconds
> {noformat}
> INSERT INTO web_sales_test partition(ws_sold_date_sk) select * from
> tpcds_bin_partitioned_orc_200.web_sales;
> Time taken to load dynamic partitions: 309.22 seconds
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14204) Optimize loading dynamic partitions
[
https://issues.apache.org/jira/browse/HIVE-14204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-14204:
Attachment: (was: HIVE-14204.8.patch)
> Optimize loading dynamic partitions
>
>
> Key: HIVE-14204
> URL: https://issues.apache.org/jira/browse/HIVE-14204
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-14204.1.patch, HIVE-14204.3.patch,
> HIVE-14204.4.patch, HIVE-14204.6.patch, HIVE-14204.7.patch
>
>
> Lots of time is spent in sequential fashion to load dynamic partitioned
> dataset in driver side. E.g simple dynamic partitioned load as follows takes
> 300+ seconds
> {noformat}
> INSERT INTO web_sales_test partition(ws_sold_date_sk) select * from
> tpcds_bin_partitioned_orc_200.web_sales;
> Time taken to load dynamic partitions: 309.22 seconds
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14368) ThriftCLIService.GetOperationStatus should include exception's stack trace to the error message.
[ https://issues.apache.org/jira/browse/HIVE-14368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406859#comment-15406859 ] zhihai xu commented on HIVE-14368: -- Thanks [~jxiang] for reviewing and committing the patch! > ThriftCLIService.GetOperationStatus should include exception's stack trace to > the error message. > > > Key: HIVE-14368 > URL: https://issues.apache.org/jira/browse/HIVE-14368 > Project: Hive > Issue Type: Improvement > Components: Thrift API >Reporter: zhihai xu >Assignee: zhihai xu >Priority: Minor > Fix For: 2.2.0 > > Attachments: HIVE-14368.000.patch > > > ThriftCLIService.GetOperationStatus should include exception's stack trace to > the error message. The stack trace will be really helpful for client to debug > failed queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14204) Optimize loading dynamic partitions
[
https://issues.apache.org/jira/browse/HIVE-14204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-14204:
Status: Patch Available (was: Open)
> Optimize loading dynamic partitions
>
>
> Key: HIVE-14204
> URL: https://issues.apache.org/jira/browse/HIVE-14204
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-14204.1.patch, HIVE-14204.3.patch,
> HIVE-14204.4.patch, HIVE-14204.6.patch, HIVE-14204.7.patch, HIVE-14204.8.patch
>
>
> Lots of time is spent in sequential fashion to load dynamic partitioned
> dataset in driver side. E.g simple dynamic partitioned load as follows takes
> 300+ seconds
> {noformat}
> INSERT INTO web_sales_test partition(ws_sold_date_sk) select * from
> tpcds_bin_partitioned_orc_200.web_sales;
> Time taken to load dynamic partitions: 309.22 seconds
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14204) Optimize loading dynamic partitions
[
https://issues.apache.org/jira/browse/HIVE-14204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-14204:
Attachment: HIVE-14204.8.patch
> Optimize loading dynamic partitions
>
>
> Key: HIVE-14204
> URL: https://issues.apache.org/jira/browse/HIVE-14204
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-14204.1.patch, HIVE-14204.3.patch,
> HIVE-14204.4.patch, HIVE-14204.6.patch, HIVE-14204.7.patch, HIVE-14204.8.patch
>
>
> Lots of time is spent in sequential fashion to load dynamic partitioned
> dataset in driver side. E.g simple dynamic partitioned load as follows takes
> 300+ seconds
> {noformat}
> INSERT INTO web_sales_test partition(ws_sold_date_sk) select * from
> tpcds_bin_partitioned_orc_200.web_sales;
> Time taken to load dynamic partitions: 309.22 seconds
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14409) Allow regex pattern in the output of qtest
[
https://issues.apache.org/jira/browse/HIVE-14409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406881#comment-15406881
]
Hive QA commented on HIVE-14409:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12821850/HIVE-14409.00.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10433 tests
executed
*Failed tests:*
{noformat}
TestMsgBusConnection - did not produce a TEST-*.xml file
TestQueryLifeTimeHook - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_orc_llap_counters
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testForcedLocalityMultiplePreemptionsSameHost2
org.apache.hadoop.hive.metastore.TestHiveMetaStoreTxns.stringifyValidTxns
org.apache.hadoop.hive.metastore.TestHiveMetaStoreTxns.testTxnRange
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/754/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/754/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-754/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12821850 - PreCommit-HIVE-MASTER-Build
> Allow regex pattern in the output of qtest
> --
>
> Key: HIVE-14409
> URL: https://issues.apache.org/jira/browse/HIVE-14409
> Project: Hive
> Issue Type: Improvement
> Components: Test
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
> Attachments: HIVE-14409.00.patch
>
>
> We cannot guarantee the output from external libraries (ex. HashMap), so we
> need to make the validation infrastructure to allow a sort within a single
> line.
> HIVE-13894, HIVE-13907, and HIVE-14088 are related.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14355) Schema evolution for ORC in llap is broken for int to string conversion
[
https://issues.apache.org/jira/browse/HIVE-14355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406888#comment-15406888
]
Prasanth Jayachandran commented on HIVE-14355:
--
Committed to branch-2.1 as well.
> Schema evolution for ORC in llap is broken for int to string conversion
> ---
>
> Key: HIVE-14355
> URL: https://issues.apache.org/jira/browse/HIVE-14355
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0, 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Critical
> Fix For: 2.2.0, 2.1.1
>
> Attachments: HIVE-14355-java-only.patch, HIVE-14355.1.patch,
> HIVE-14355.2.java-only.patch, HIVE-14355.2.patch,
> HIVE-14355.3.java-only.patch, HIVE-14355.3.patch
>
>
> When schema is evolved from any integer type to string then following
> exceptions are thrown in LLAP (Works fine in Tez). I guess this should happen
> even for other conversions.
> {code}
> hive> create table orc_integer(b bigint) stored as orc;
> hive> insert into orc_integer values(100);
> hive> select count(*) from orc_integer where b=100;
> OK
> 1
> hive> alter table orc_integer change column b b string;
> hive> select count(*) from orc_integer where b=100;
> // FAIL with following exception
> {code}
> {code:title=When vectorization is enabled}
> 2016-07-27T01:48:05,611 INFO [TezTaskRunner ()]
> vector.VectorReduceSinkOperator: RECORDS_OUT_INTERMEDIATE_Map_1:0,
> 2016-07-27T01:48:05,611 ERROR [TezTaskRunner ()] tez.TezProcessor:
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
> Hive Runtime Error while processing row
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:95)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:70)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:393)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:185)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:866)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:86)
> ... 18 more
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.ql.exec.vector.LongColumnVector cannot be cast to
> org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.gen.FilterStringGroupColEqualStringGroupScalarBase.evaluate(FilterStringGroupColEqualStringGroupScalarBase.java:42)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorFilterOperator.process(VectorFilterOperator.java:110)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:879)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:774)
> ... 19 more
> {code}
> {code:title=When vectorization is disabled}
> 2016-07-27T01:52:43,328 INFO [TezTaskRunner
> (1469608604787_0002_26_00_00_0)] exec.ReduceSinkOperator: Using tag = -1
> 2016-07-27T01:52:43,32
[jira] [Updated] (HIVE-14355) Schema evolution for ORC in llap is broken for int to string conversion
[
https://issues.apache.org/jira/browse/HIVE-14355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-14355:
-
Fix Version/s: 2.1.1
> Schema evolution for ORC in llap is broken for int to string conversion
> ---
>
> Key: HIVE-14355
> URL: https://issues.apache.org/jira/browse/HIVE-14355
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0, 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Critical
> Fix For: 2.2.0, 2.1.1
>
> Attachments: HIVE-14355-java-only.patch, HIVE-14355.1.patch,
> HIVE-14355.2.java-only.patch, HIVE-14355.2.patch,
> HIVE-14355.3.java-only.patch, HIVE-14355.3.patch
>
>
> When schema is evolved from any integer type to string then following
> exceptions are thrown in LLAP (Works fine in Tez). I guess this should happen
> even for other conversions.
> {code}
> hive> create table orc_integer(b bigint) stored as orc;
> hive> insert into orc_integer values(100);
> hive> select count(*) from orc_integer where b=100;
> OK
> 1
> hive> alter table orc_integer change column b b string;
> hive> select count(*) from orc_integer where b=100;
> // FAIL with following exception
> {code}
> {code:title=When vectorization is enabled}
> 2016-07-27T01:48:05,611 INFO [TezTaskRunner ()]
> vector.VectorReduceSinkOperator: RECORDS_OUT_INTERMEDIATE_Map_1:0,
> 2016-07-27T01:48:05,611 ERROR [TezTaskRunner ()] tez.TezProcessor:
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
> Hive Runtime Error while processing row
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:95)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:70)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:393)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:185)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:866)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:86)
> ... 18 more
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.ql.exec.vector.LongColumnVector cannot be cast to
> org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.gen.FilterStringGroupColEqualStringGroupScalarBase.evaluate(FilterStringGroupColEqualStringGroupScalarBase.java:42)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorFilterOperator.process(VectorFilterOperator.java:110)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:879)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:774)
> ... 19 more
> {code}
> {code:title=When vectorization is disabled}
> 2016-07-27T01:52:43,328 INFO [TezTaskRunner
> (1469608604787_0002_26_00_00_0)] exec.ReduceSinkOperator: Using tag = -1
> 2016-07-27T01:52:43,328 INFO [TezTaskRunner
> (1469608604787_0002_26_00_00_0)] exe
[jira] [Updated] (HIVE-14421) FS.deleteOnExit holds references to _tmp_space.db files
[ https://issues.apache.org/jira/browse/HIVE-14421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-14421: -- Status: Patch Available (was: Open) > FS.deleteOnExit holds references to _tmp_space.db files > --- > > Key: HIVE-14421 > URL: https://issues.apache.org/jira/browse/HIVE-14421 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-14421.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14421) FS.deleteOnExit holds references to _tmp_space.db files
[ https://issues.apache.org/jira/browse/HIVE-14421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-14421: -- Attachment: HIVE-14421.01.patch [~thejas] - please review. > FS.deleteOnExit holds references to _tmp_space.db files > --- > > Key: HIVE-14421 > URL: https://issues.apache.org/jira/browse/HIVE-14421 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-14421.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14422) LLAP IF: when using LLAP IF from multiple threads in secure cluster, tokens can get mixed up
[ https://issues.apache.org/jira/browse/HIVE-14422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-14422: Attachment: HIVE-14422.patch [~jdere] [~sseth] do you want to take a look? I also added some logging here and there. > LLAP IF: when using LLAP IF from multiple threads in secure cluster, tokens > can get mixed up > - > > Key: HIVE-14422 > URL: https://issues.apache.org/jira/browse/HIVE-14422 > Project: Hive > Issue Type: Bug >Reporter: Jason Dere >Assignee: Sergey Shelukhin > Attachments: HIVE-14422.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14422) LLAP IF: when using LLAP IF from multiple threads in secure cluster, tokens can get mixed up
[ https://issues.apache.org/jira/browse/HIVE-14422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-14422: Reporter: Jason Dere (was: Sergey Shelukhin) > LLAP IF: when using LLAP IF from multiple threads in secure cluster, tokens > can get mixed up > - > > Key: HIVE-14422 > URL: https://issues.apache.org/jira/browse/HIVE-14422 > Project: Hive > Issue Type: Bug >Reporter: Jason Dere >Assignee: Sergey Shelukhin > Attachments: HIVE-14422.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14422) LLAP IF: when using LLAP IF from multiple threads in secure cluster, tokens can get mixed up
[ https://issues.apache.org/jira/browse/HIVE-14422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-14422: Status: Patch Available (was: Open) > LLAP IF: when using LLAP IF from multiple threads in secure cluster, tokens > can get mixed up > - > > Key: HIVE-14422 > URL: https://issues.apache.org/jira/browse/HIVE-14422 > Project: Hive > Issue Type: Bug >Reporter: Jason Dere >Assignee: Sergey Shelukhin > Attachments: HIVE-14422.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
