[jira] [Commented] (HIVE-12529) HiveTxnManager.acquireLocks() should not block forever
[
https://issues.apache.org/jira/browse/HIVE-12529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15031132#comment-15031132
]
Hive QA commented on HIVE-12529:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12774565/HIVE-12529.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 9866 tests executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_drop_with_concurrency
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_insert_into1
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_insert_into2
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_insert_into3
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_insert_into4
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6164/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6164/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6164/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12774565 - PreCommit-HIVE-TRUNK-Build
> HiveTxnManager.acquireLocks() should not block forever
> --
>
> Key: HIVE-12529
> URL: https://issues.apache.org/jira/browse/HIVE-12529
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-12529.2.patch, HIVE-12529.patch
>
>
> Currently, in DbTxnManager this method will block until all competing locks
> have gone away.
> This is not appropriate for all clients. There should be a way to specify a
> max-wait-time.
> It will throw an exception on timeout (given how current method signature is
> written).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12522) Wrong FS error during Tez merge files when warehouse and scratchdir are on different FS
[
https://issues.apache.org/jira/browse/HIVE-12522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15031151#comment-15031151
]
Jason Dere commented on HIVE-12522:
---
All of the test failures have been failing on previous precommit test runs.
> Wrong FS error during Tez merge files when warehouse and scratchdir are on
> different FS
> ---
>
> Key: HIVE-12522
> URL: https://issues.apache.org/jira/browse/HIVE-12522
> Project: Hive
> Issue Type: Bug
> Components: Tez
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-12522.1.patch
>
>
> When hive.merge.tezfiles=true, and the warehouse dir/scratchdir are on
> different filesystems.
> {noformat}
> 2015-11-13 10:22:10,617 ERROR exec.Task (TezTask.java:execute(184)) - Failed
> to execute tez graph.
> java.lang.IllegalArgumentException: Wrong FS:
> wasb://chaoyitezt...@chaoyiteztest.blob.core.windows.net/hive/scratch/chaoyitest/c888f405-3c98-46b1-bf39-e57f067dfe4c/hive_2015-11-13_10-16-10_216_8161037519951665173-1/_tmp.-ext-1,
> expected: hdfs://headnodehost:9000
> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:645)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:193)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.access$000(DistributedFileSystem.java:105)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1136)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1132)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1132)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1423)
> at org.apache.hadoop.hive.ql.exec.tez.DagUtils.createVertex(DagUtils.java:579)
> at
> org.apache.hadoop.hive.ql.exec.tez.DagUtils.createVertex(DagUtils.java:1083)
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.build(TezTask.java:329)
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:156)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
> at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1606)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1367)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1179)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1006)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:996)
> at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:247)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:199)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:410)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:345)
> at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:733)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:677)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:616)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> 2015-11-13 10:22:10,620 INFO hooks.ATSHook (ATSHook.java:(84)) -
> Created ATS Hook
> {noformat}
> When the scratchdir is set to the same FS as the warehouse the problem goes
> away.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HIVE-12533) Unexpected NULL in map join small table
[
https://issues.apache.org/jira/browse/HIVE-12533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan resolved HIVE-12533.
-
Resolution: Duplicate
[~damien.carol] - This was in master branch.
Closing this as it is a duplicate of HIVE-12399.
> Unexpected NULL in map join small table
> ---
>
> Key: HIVE-12533
> URL: https://issues.apache.org/jira/browse/HIVE-12533
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Rajesh Balamohan
>
> {noformat}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Unexpected NULL in map join
> small table
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastHashTableLoader.load(VectorMapJoinFastHashTableLoader.java:110)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:293)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator$1.call(MapJoinOperator.java:174)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator$1.call(MapJoinOperator.java:170)
> at
> org.apache.hadoop.hive.ql.exec.tez.LlapObjectCache.retrieve(LlapObjectCache.java:104)
> ... 5 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unexpected NULL
> in map join small table
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastLongHashTable.putRow(VectorMapJoinFastLongHashTable.java:88)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastTableContainer.putRow(VectorMapJoinFastTableContainer.java:182)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastHashTableLoader.load(VectorMapJoinFastHashTableLoader.java:97)
> ... 9 more
> {noformat}
> \cc [~gopalv]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12020) Revert log4j2 xml configuration to properties based configuration
[ https://issues.apache.org/jira/browse/HIVE-12020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-12020: - Attachment: HIVE-12020.4.patch > Revert log4j2 xml configuration to properties based configuration > - > > Key: HIVE-12020 > URL: https://issues.apache.org/jira/browse/HIVE-12020 > Project: Hive > Issue Type: Sub-task > Components: Logging >Affects Versions: 2.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-12020.1.patch, HIVE-12020.2.patch, > HIVE-12020.3.patch, HIVE-12020.4.patch > > > Log4j 2.4 release brought back properties based configuration. We should > revert XML based configuration and use properties based configuration instead > (less verbose and will be similar to old log4j properties). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12538) After set spark related config, SparkSession never get reused
[ https://issues.apache.org/jira/browse/HIVE-12538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15031109#comment-15031109 ] Jimmy Xiang commented on HIVE-12538: [~nemon], good findings. Do you want to post a patch? Otherwise, I can do it. Thanks. > After set spark related config, SparkSession never get reused > - > > Key: HIVE-12538 > URL: https://issues.apache.org/jira/browse/HIVE-12538 > Project: Hive > Issue Type: Bug > Components: Spark >Affects Versions: 1.3.0 >Reporter: Nemon Lou > > Hive on Spark yarn-cluster mode. > After setting "set spark.yarn.queue=QueueA;" , > run the query "select count(*) from test" 3 times and you will find 3 > different yarn applications. > Two of the yarn applications in FINISHED & SUCCEEDED state,and one in RUNNING > & UNDEFINED state waiting for next work. > And if you submit one more "select count(*) from test" ,the third one will be > in FINISHED & SUCCEEDED state and a new yarn application will start up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11775) Implement limit push down through union all in CBO
[
https://issues.apache.org/jira/browse/HIVE-11775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15031185#comment-15031185
]
Hive QA commented on HIVE-11775:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12774571/HIVE-11775.05.patch
{color:green}SUCCESS:{color} +1 due to 7 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 9866 tests executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_SortUnionTransposeRule
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_constprog_dpp
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_constprog_dpp
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_temp_table
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6165/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6165/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6165/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12774571 - PreCommit-HIVE-TRUNK-Build
> Implement limit push down through union all in CBO
> --
>
> Key: HIVE-11775
> URL: https://issues.apache.org/jira/browse/HIVE-11775
> Project: Hive
> Issue Type: New Feature
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-11775.01.patch, HIVE-11775.02.patch,
> HIVE-11775.03.patch, HIVE-11775.04.patch, HIVE-11775.05.patch
>
>
> Enlightened by HIVE-11684 (Kudos to [~jcamachorodriguez]), we can actually
> push limit down through union all, which reduces the intermediate number of
> rows in union branches.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11927) Implement/Enable constant related optimization rules in Calcite: enable HiveReduceExpressionsRule to fold constants
[ https://issues.apache.org/jira/browse/HIVE-11927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15030862#comment-15030862 ] Pengcheng Xiong commented on HIVE-11927: [~jpullokkaran], I attached a new patch. I would like to skip CBO when there is an empty logical value list. There are two reasons. (1) If we translate logical value list to table "dummy", we can not specify the row limit as 0 in AST. If we do not specify the row limit, the default 1 will be used. (2) There is a physical optimizer "NullScanOptimizer", which will handle the where false case and translate it to a null scan. Thanks. > Implement/Enable constant related optimization rules in Calcite: enable > HiveReduceExpressionsRule to fold constants > --- > > Key: HIVE-11927 > URL: https://issues.apache.org/jira/browse/HIVE-11927 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-11927.01.patch, HIVE-11927.02.patch, > HIVE-11927.03.patch, HIVE-11927.04.patch, HIVE-11927.05.patch, > HIVE-11927.06.patch, HIVE-11927.07.patch, HIVE-11927.08.patch, > HIVE-11927.09.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12479) Vectorization: Vectorized Date UDFs with up-stream Joins
[
https://issues.apache.org/jira/browse/HIVE-12479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15030868#comment-15030868
]
Hive QA commented on HIVE-12479:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12774310/HIVE-12479.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 9865 tests executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testFetchingPartitionsWithDifferentSchemas
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
org.apache.hadoop.hive.metastore.txn.TestCompactionTxnHandler.testRevokeTimedOutWorkers
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6161/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6161/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6161/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12774310 - PreCommit-HIVE-TRUNK-Build
> Vectorization: Vectorized Date UDFs with up-stream Joins
>
>
> Key: HIVE-12479
> URL: https://issues.apache.org/jira/browse/HIVE-12479
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 1.3.0, 2.0.0
>Reporter: Gopal V
>Assignee: Gopal V
> Attachments: HIVE-12479.1.patch, HIVE-12479.tar.gz
>
>
> The row-counts expected with and without vectorization differ.
> The attached small-scale repro case produces 5 rows with vectorized multi-key
> joins and 53 rows without the vectorized join.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11358) LLAP: move LlapConfiguration into HiveConf and document the settings
[
https://issues.apache.org/jira/browse/HIVE-11358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15030918#comment-15030918
]
Hive QA commented on HIVE-11358:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12774456/HIVE-11358.04.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 9865 tests executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6162/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6162/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6162/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12774456 - PreCommit-HIVE-TRUNK-Build
> LLAP: move LlapConfiguration into HiveConf and document the settings
>
>
> Key: HIVE-11358
> URL: https://issues.apache.org/jira/browse/HIVE-11358
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-11358.01.patch, HIVE-11358.02.patch,
> HIVE-11358.03.patch, HIVE-11358.04.patch, HIVE-11358.patch
>
>
> Hive uses HiveConf for configuration. LlapConfiguration should be replaced
> with parameters in HiveConf
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12075) add analyze command to explictly cache file metadata in HBase metastore
[
https://issues.apache.org/jira/browse/HIVE-12075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15031022#comment-15031022
]
Hive QA commented on HIVE-12075:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12774459/HIVE-12075.03.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 9866 tests executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_stats_filemetadata
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_columnstats_partlvl_multiple_part_clause
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
org.apache.hadoop.hive.ql.security.authorization.plugin.TestHiveOperationType.checkHiveOperationTypeMatch
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.jdbc.miniHS2.TestHs2Metrics.testMetrics
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6163/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6163/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6163/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12774459 - PreCommit-HIVE-TRUNK-Build
> add analyze command to explictly cache file metadata in HBase metastore
> ---
>
> Key: HIVE-12075
> URL: https://issues.apache.org/jira/browse/HIVE-12075
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-12075.01.nogen.patch, HIVE-12075.01.patch,
> HIVE-12075.02.patch, HIVE-12075.03.patch, HIVE-12075.nogen.patch,
> HIVE-12075.patch
>
>
> ANALYZE TABLE (spec as usual) CACHE METADATA
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12433) Merge master into spark 11/29/2015 [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-12433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-12433: --- Summary: Merge master into spark 11/29/2015 [Spark Branch] (was: Merge master into spark 11/17/2015 [Spark Branch]) > Merge master into spark 11/29/2015 [Spark Branch] > - > > Key: HIVE-12433 > URL: https://issues.apache.org/jira/browse/HIVE-12433 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Brock Noland >Assignee: Xuefu Zhang > Fix For: spark-branch > > Attachments: HIVE-12433.1-spark.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12433) Merge master into spark 11/17/2015 [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-12433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-12433: --- Summary: Merge master into spark 11/17/2015 [Spark Branch] (was: Merge master into spark 11/29/2015 [Spark Branch]) > Merge master into spark 11/17/2015 [Spark Branch] > - > > Key: HIVE-12433 > URL: https://issues.apache.org/jira/browse/HIVE-12433 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Brock Noland >Assignee: Xuefu Zhang > Fix For: spark-branch > > Attachments: HIVE-12433.1-spark.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12338) Add webui to HiveServer2
[ https://issues.apache.org/jira/browse/HIVE-12338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15031229#comment-15031229 ] Lefty Leverenz commented on HIVE-12338: --- Doc note: This should be documented with a new top-level section in HiveServer2 Clients, and perhaps also a subsection in Setting Up HiveServer2. Three new configuration parameters (*hive.server2.webui.host*, *hive.server2.webui.port*, and *hive.server2.webui.max.threads*) should be documented in the HiveServer2 section of Configuration Properties. * [HiveServer2 Clients | https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients] * [Setting Up HiveServer2 -- How to Configure | https://cwiki.apache.org/confluence/display/Hive/Setting+Up+HiveServer2#SettingUpHiveServer2-HowtoConfigure] * [Configuration Properties -- HiveServer2 | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-HiveServer2] > Add webui to HiveServer2 > > > Key: HIVE-12338 > URL: https://issues.apache.org/jira/browse/HIVE-12338 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > Labels: TODOC2.0 > Fix For: 2.0.0 > > Attachments: HIVE-12338.1.patch, HIVE-12338.2.patch, > HIVE-12338.3.patch, HIVE-12338.4.patch, hs2-conf.png, hs2-logs.png, > hs2-metrics.png, hs2-webui.png > > > A web ui for HiveServer2 can show some useful information such as: > > 1. Sessions, > 2. Queries that are executing on the HS2, their states, starting time, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12301) CBO: Calcite Operator To Hive Operator (Calcite Return Path): fix test failure for udf_percentile.q
[ https://issues.apache.org/jira/browse/HIVE-12301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15031257#comment-15031257 ] Jesus Camacho Rodriguez commented on HIVE-12301: Comments are minor, rest looks good to me. +1 once those are addressed. > CBO: Calcite Operator To Hive Operator (Calcite Return Path): fix test > failure for udf_percentile.q > --- > > Key: HIVE-12301 > URL: https://issues.apache.org/jira/browse/HIVE-12301 > Project: Hive > Issue Type: Sub-task > Components: CBO >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-12301.01.patch, HIVE-12301.02.patch, > HIVE-12301.03.patch > > > The position in argList is mapped to a wrong column from RS operator -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12301) CBO: Calcite Operator To Hive Operator (Calcite Return Path): fix test failure for udf_percentile.q
[ https://issues.apache.org/jira/browse/HIVE-12301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15031249#comment-15031249 ] Jesus Camacho Rodriguez commented on HIVE-12301: [~pxiong], I went through the patch and left some comments in RB. Thanks > CBO: Calcite Operator To Hive Operator (Calcite Return Path): fix test > failure for udf_percentile.q > --- > > Key: HIVE-12301 > URL: https://issues.apache.org/jira/browse/HIVE-12301 > Project: Hive > Issue Type: Sub-task > Components: CBO >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-12301.01.patch, HIVE-12301.02.patch, > HIVE-12301.03.patch > > > The position in argList is mapped to a wrong column from RS operator -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12526) PerfLogger for hive compiler and optimizer
[ https://issues.apache.org/jira/browse/HIVE-12526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15031250#comment-15031250 ] Jesus Camacho Rodriguez commented on HIVE-12526: [~hsubramaniyan], I have seen that you have added a single performance measurement for Calcite planning. I was wondering if we would like to have more precise measurements for Calcite, as we have for the other Hive optimizations. For instance, (1) one per optimization call, or (2) one per optimization block (prejoin reordering, join reordering, postjoin reordering), or (3) -another option-. What do you think? [~ashutoshc], what's your opinion? > PerfLogger for hive compiler and optimizer > -- > > Key: HIVE-12526 > URL: https://issues.apache.org/jira/browse/HIVE-12526 > Project: Hive > Issue Type: Bug >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-12526.1.patch > > > This jira is intended to use the perflogger to track compilation times and > optimization times (calcite, tez compiler, physical compiler) etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-12538) After set spark related config, SparkSession never get reused
[ https://issues.apache.org/jira/browse/HIVE-12538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nemon Lou reassigned HIVE-12538: Assignee: Nemon Lou > After set spark related config, SparkSession never get reused > - > > Key: HIVE-12538 > URL: https://issues.apache.org/jira/browse/HIVE-12538 > Project: Hive > Issue Type: Bug > Components: Spark >Affects Versions: 1.3.0 >Reporter: Nemon Lou >Assignee: Nemon Lou > > Hive on Spark yarn-cluster mode. > After setting "set spark.yarn.queue=QueueA;" , > run the query "select count(*) from test" 3 times and you will find 3 > different yarn applications. > Two of the yarn applications in FINISHED & SUCCEEDED state,and one in RUNNING > & UNDEFINED state waiting for next work. > And if you submit one more "select count(*) from test" ,the third one will be > in FINISHED & SUCCEEDED state and a new yarn application will start up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12338) Add webui to HiveServer2
[ https://issues.apache.org/jira/browse/HIVE-12338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lefty Leverenz updated HIVE-12338: -- Labels: TODOC2.0 (was: ) > Add webui to HiveServer2 > > > Key: HIVE-12338 > URL: https://issues.apache.org/jira/browse/HIVE-12338 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > Labels: TODOC2.0 > Fix For: 2.0.0 > > Attachments: HIVE-12338.1.patch, HIVE-12338.2.patch, > HIVE-12338.3.patch, HIVE-12338.4.patch, hs2-conf.png, hs2-logs.png, > hs2-metrics.png, hs2-webui.png > > > A web ui for HiveServer2 can show some useful information such as: > > 1. Sessions, > 2. Queries that are executing on the HS2, their states, starting time, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12381) analyze table compute stats for table with special characters will wipe out all the table stats
[
https://issues.apache.org/jira/browse/HIVE-12381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15031234#comment-15031234
]
Hive QA commented on HIVE-12381:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12774573/HIVE-12381.04.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 9865 tests executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats19
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_stats3
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.jdbc.miniHS2.TestHs2Metrics.testMetrics
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6166/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6166/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6166/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12774573 - PreCommit-HIVE-TRUNK-Build
> analyze table compute stats for table with special characters will wipe out
> all the table stats
> ---
>
> Key: HIVE-12381
> URL: https://issues.apache.org/jira/browse/HIVE-12381
> Project: Hive
> Issue Type: Bug
> Environment:
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-12381.01.patch, HIVE-12381.02.patch,
> HIVE-12381.03.patch, HIVE-12381.04.patch
>
>
> repo:
> {code}
> drop table `t//`;
> create table `t//` (col string);
> insert into `t//` values(1);
> insert into `t//` values(null);
> analyze table `t//` compute statistics;
> explain select * from `t//`;
> {code}
> The result
> {code}
> Statistics: Num rows: 1 Data size: 5 Basic stats: COMPLETE Column stats: NONE
> {code}
> is wrong
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12538) After set spark related config, SparkSession never get reused
[ https://issues.apache.org/jira/browse/HIVE-12538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nemon Lou updated HIVE-12538: - Attachment: HIVE-12538.patch 1,Fixing the bug : " isSparkConfigUpdated = isSparkRelatedConfig(name)" 2,Setting "isSparkConfigUpdated =false " at session level 3,Taking concurrency into consideration by adding a session level conf lock.As multiple queries can run in a single session. > After set spark related config, SparkSession never get reused > - > > Key: HIVE-12538 > URL: https://issues.apache.org/jira/browse/HIVE-12538 > Project: Hive > Issue Type: Bug > Components: Spark >Affects Versions: 1.3.0 >Reporter: Nemon Lou >Assignee: Nemon Lou > Attachments: HIVE-12538.patch > > > Hive on Spark yarn-cluster mode. > After setting "set spark.yarn.queue=QueueA;" , > run the query "select count(*) from test" 3 times and you will find 3 > different yarn applications. > Two of the yarn applications in FINISHED & SUCCEEDED state,and one in RUNNING > & UNDEFINED state waiting for next work. > And if you submit one more "select count(*) from test" ,the third one will be > in FINISHED & SUCCEEDED state and a new yarn application will start up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12505) Insert overwrite in same encrypted zone silently fails to remove some existing files
[
https://issues.apache.org/jira/browse/HIVE-12505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15031304#comment-15031304
]
Hive QA commented on HIVE-12505:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12774658/HIVE-12505.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 9866 tests executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testFetchingPartitionsWithDifferentSchemas
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.jdbc.miniHS2.TestHs2Metrics.testMetrics
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6167/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6167/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6167/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12774658 - PreCommit-HIVE-TRUNK-Build
> Insert overwrite in same encrypted zone silently fails to remove some
> existing files
>
>
> Key: HIVE-12505
> URL: https://issues.apache.org/jira/browse/HIVE-12505
> Project: Hive
> Issue Type: Bug
> Components: Encryption
>Affects Versions: 1.2.1
>Reporter: Chaoyu Tang
>Assignee: Chaoyu Tang
> Attachments: HIVE-12505.1.patch, HIVE-12505.2.patch, HIVE-12505.patch
>
>
> With HDFS Trash enabled but its encryption zone lower than Hive data
> directory, insert overwrite command silently fails to trash the existing
> files during overwrite, which could lead to unexpected incorrect results
> (more rows returned than expected)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-7214) Support predicate pushdown for complex data types in ORCFile
[ https://issues.apache.org/jira/browse/HIVE-7214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Damien Carol updated HIVE-7214: --- Component/s: ORC > Support predicate pushdown for complex data types in ORCFile > > > Key: HIVE-7214 > URL: https://issues.apache.org/jira/browse/HIVE-7214 > Project: Hive > Issue Type: Improvement > Components: File Formats, ORC >Reporter: Rohini Palaniswamy > Labels: ORC > > Currently ORCFile does not support predicate pushdown for complex datatypes > like map, array and struct while Parquet does. Came across this during > discussion of PIG-3760. Our users have a lot of map and struct (tuple in pig) > columns and most of the filter conditions are on them. Would be great to have > support added for them in ORC -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12500) JDBC driver not overlaying params supplied via properties object when reading params from ZK
[ https://issues.apache.org/jira/browse/HIVE-12500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15031411#comment-15031411 ] Thejas M Nair commented on HIVE-12500: -- +1 pending tests > JDBC driver not overlaying params supplied via properties object when reading > params from ZK > > > Key: HIVE-12500 > URL: https://issues.apache.org/jira/browse/HIVE-12500 > Project: Hive > Issue Type: Bug > Components: JDBC >Affects Versions: 1.3.0, 2.0.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > Attachments: HIVE-12500.1.patch, HIVE-12500.2.patch > > > It makes sense to setup the connection info in one place. Right now part of > connection configuration happens in Utils#parseURL and part in the > HiveConnection constructor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11110) Reorder applyPreJoinOrderingTransforms, add NotNULL/FilterMerge rules, improve Filter selectivity estimation
[
https://issues.apache.org/jira/browse/HIVE-0?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laljo John Pullokkaran updated HIVE-0:
--
Attachment: HIVE-0.27
> Reorder applyPreJoinOrderingTransforms, add NotNULL/FilterMerge rules,
> improve Filter selectivity estimation
>
>
> Key: HIVE-0
> URL: https://issues.apache.org/jira/browse/HIVE-0
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Laljo John Pullokkaran
> Attachments: HIVE-0-10.patch, HIVE-0-11.patch,
> HIVE-0-12.patch, HIVE-0-branch-1.2.patch, HIVE-0.1.patch,
> HIVE-0.13.patch, HIVE-0.14.patch, HIVE-0.15.patch,
> HIVE-0.16.patch, HIVE-0.17.patch, HIVE-0.18.patch,
> HIVE-0.19.patch, HIVE-0.2.patch, HIVE-0.20.patch,
> HIVE-0.21.patch, HIVE-0.22.patch, HIVE-0.23.patch,
> HIVE-0.24.patch, HIVE-0.25.patch, HIVE-0.26.patch, HIVE-0.27,
> HIVE-0.4.patch, HIVE-0.5.patch, HIVE-0.6.patch,
> HIVE-0.7.patch, HIVE-0.8.patch, HIVE-0.9.patch,
> HIVE-0.91.patch, HIVE-0.92.patch, HIVE-0.patch
>
>
> Query
> {code}
> select count(*)
> from store_sales
> ,store_returns
> ,date_dim d1
> ,date_dim d2
> where d1.d_quarter_name = '2000Q1'
>and d1.d_date_sk = ss_sold_date_sk
>and ss_customer_sk = sr_customer_sk
>and ss_item_sk = sr_item_sk
>and ss_ticket_number = sr_ticket_number
>and sr_returned_date_sk = d2.d_date_sk
>and d2.d_quarter_name in ('2000Q1','2000Q2','2000Q3’);
> {code}
> The store_sales table is partitioned on ss_sold_date_sk, which is also used
> in a join clause. The join clause should add a filter “filterExpr:
> ss_sold_date_sk is not null”, which should get pushed the MetaStore when
> fetching the stats. Currently this is not done in CBO planning, which results
> in the stats from __HIVE_DEFAULT_PARTITION__ to be fetched and considered in
> the optimization phase. In particular, this increases the NDV for the join
> columns and may result in wrong planning.
> Including HiveJoinAddNotNullRule in the optimization phase solves this issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12535) Dynamic Hash Join: Key references are cyclic
[
https://issues.apache.org/jira/browse/HIVE-12535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15031217#comment-15031217
]
Jason Dere commented on HIVE-12535:
---
It looks like that output is due to the user-level explain formatting that is
done in common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/Op.java.
After the initial plan is created the MapJoin operator looks like this:
{noformat}
"keys:":{"0":"KEY.reducesinkkey0 (type: int)","1":"KEY.reducesinkkey0 (type:
int)"}
"input vertices:":{"1":"Map 3"}
{noformat}
Because input "0" (which I think is the big table in this case) is not in the
"input vertices" list, it gets resolved during Op.java as the current vertex
("Reducer 2").
So this issue in the explain output is simply cosmetic, but if there is a
similar issue in the vectorizer, it could also be related to the fact that the
input vertices for the MapJoin do not include the big table. Tho I'm not sure
if whether that mapping is supposed to include the big table, someone else may
need to comment on that.
> Dynamic Hash Join: Key references are cyclic
>
>
> Key: HIVE-12535
> URL: https://issues.apache.org/jira/browse/HIVE-12535
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 2.0.0
>Reporter: Gopal V
>Assignee: Jason Dere
> Attachments: philz_26.txt
>
>
> MAPJOIN_4227 is inside "Reducer 2", but refers back to "Reducer 2" in its
> keys. It should say "Map 1" there.
> {code}
> ||<-Reducer 2 [SIMPLE_EDGE] vectorized, llap
>
>
> |
> | Reduce Output Operator [RS_4189]
>
>
> |
> | key expressions:_col0 (type: string), _col1 (type:
> int)
>
> |
> | Map-reduce partition columns:_col0 (type: string),
> _col1 (type: int)
>
> |
> | sort order:++
>
>
> |
> | Statistics:Num rows: 83 Data size: 9213 Basic stats:
> COMPLETE Column stats: COMPLETE
>
> |
> | value expressions:_col2 (type: double)
>
>
> |
> | Group By Operator [OP_4229]
>
>
> |
> | aggregations:["sum(_col2)"]
>
>
> |
> | keys:_col0 (type: string), _col1 (type: int)
>
>
> |
> | outputColumnNames:["_col0","_col1","_col2"]
>
>
> |
> | Statistics:Num rows: 83 Data size: 9213 Basic
> stats: COMPLETE Column stats: COMPLETE
>
> |
> | Select Operator
[jira] [Commented] (HIVE-12515) Clean the SparkCounters related code after remove counter based stats collection[Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-12515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15031226#comment-15031226 ] Xuefu Zhang commented on HIVE-12515: Thanks for working on this, Rui/Chengxiang. I just made a merge from master to Spark. The merge was clean. > Clean the SparkCounters related code after remove counter based stats > collection[Spark Branch] > -- > > Key: HIVE-12515 > URL: https://issues.apache.org/jira/browse/HIVE-12515 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Chengxiang Li >Assignee: Xuefu Zhang > > As SparkCounters is only used to collection stats, after HIVE-12411, we does > not need it anymore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12530) Merge join in mutiple subsquence join and a mapjoin in it in mr model
[ https://issues.apache.org/jira/browse/HIVE-12530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15031251#comment-15031251 ] Feng Yuan commented on HIVE-12530: -- [~vikram.dixit],hi could u please look at this issue? its similar to HIVE-9832,i think,thank you! > Merge join in mutiple subsquence join and a mapjoin in it in mr model > - > > Key: HIVE-12530 > URL: https://issues.apache.org/jira/browse/HIVE-12530 > Project: Hive > Issue Type: Bug > Components: CLI >Affects Versions: 1.2.1 >Reporter: Feng Yuan > Fix For: 2.00 > > > sample hql: > select A.state_date, >A.customer, >A.channel_2, >A.id, >A.pid, >A.type, >A.pv, >A.uv, >A.visits, >if(C.stay_visits is null,0,C.stay_visits) as stay_visits, >A.stay_time, >if(B.bounce is null,0,B.bounce) as bounce > from > (select a.state_date, > a.customer, > b.url as channel_2, > b.id, > b.pid, > b.type, > count(1) as pv, > count(distinct a.gid) uv, > count(distinct a.session_id) as visits, > sum(a.stay_time) as stay_time >from >( select state_date, >customer, >gid, >session_id, >ep, >stay_time > from bdi_fact.mid_pageview_dt0 > where l_date ='$v_date' > )a > join > (select l_date as state_date , > url, > id, > pid, > type, > cid >from bdi_fact.frequency_channel >where l_date ='$v_date' >and type ='2' >and dr='0' > )b >on a.customer=b.cid >where a.ep rlike b.url >group by a.state_date, a.customer, b.url,b.id,b.pid,b.type >)A > > left outer join >( select >c.state_date , >c.customer , >d.url as channel_2, >d.id, >sum(pagedepth) as bounce > from > ( select > t1.state_date , > t1.customer , > t1.session_id, > t1.ep, > t2.pagedepth > from > ( select > state_date , > customer , > session_id, > exit_url as ep > from ods.mid_session_enter_exit_dt0 > where l_date ='$v_date' > )t1 > join > ( select > state_date , > customer , > session_id, > pagedepth > from ods.mid_session_action_dt0 > where l_date ='$v_date' > and pagedepth='1' > )t2 > on t1.customer=t2.customer > and t1.session_id=t2.session_id >)c >join >(select * >from bdi_fact.frequency_channel >where l_date ='$v_date' >and type ='2' >and dr='0' >)d >on c.customer=d.cid >where c.ep rlike d.url >group by c.state_date,c.customer,d.url,d.id > )B > on > A.customer=B.customer > and A.channel_2=B.channel_2 > and A.id=B.id > left outer join > ( > select e.state_date, > e.customer, > f.url as channel_2, > f.id, > f.pid, > f.type, > count(distinct e.session_id) as stay_visits >from >( select state_date, >customer, >gid, >session_id, >ep, >
[jira] [Commented] (HIVE-12506) SHOW CREATE TABLE command creates a table that does not work for RCFile format
[
https://issues.apache.org/jira/browse/HIVE-12506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15031095#comment-15031095
]
Chaoyu Tang commented on HIVE-12506:
The "show create table" output should use SERDE instead of DELIMITED for
row_format and SERDEPROPERTIES for the DELIMITED properties if any. It can
ensure a table with any kind of serde/fileformat to be recreated correctly.
{code}
row_format
: DELIMITED [FIELDS TERMINATED BY char [ESCAPED BY char]] [COLLECTION ITEMS
TERMINATED BY char]
[MAP KEYS TERMINATED BY char] [LINES TERMINATED BY char]
[NULL DEFINED AS char] -- (Note: Available in Hive 0.13 and later)
| SERDE serde_name [WITH SERDEPROPERTIES (property_name=property_value,
property_name=property_value, ...)]
{code}
> SHOW CREATE TABLE command creates a table that does not work for RCFile format
> --
>
> Key: HIVE-12506
> URL: https://issues.apache.org/jira/browse/HIVE-12506
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Affects Versions: 1.1.1
>Reporter: Eric Lin
>Assignee: Chaoyu Tang
>
> See the following test case:
> 1) Create a table with RCFile format:
> {code}
> DROP TABLE IF EXISTS test;
> CREATE TABLE test (a int) PARTITIONED BY (p int)
> ROW FORMAT DELIMITED FIELDS TERMINATED BY '|'
> STORED AS RCFILE;
> {code}
> 2) run "DESC FORMATTED test"
> {code}
> # Storage Information
> SerDe Library:org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe
> InputFormat: org.apache.hadoop.hive.ql.io.RCFileInputFormat
> OutputFormat: org.apache.hadoop.hive.ql.io.RCFileOutputFormat
> {code}
> shows that SerDe used is "ColumnarSerDe"
> 3) run "SHOW CREATE TABLE" and get the output:
> {code}
> CREATE TABLE `test`(
> `a` int)
> PARTITIONED BY (
> `p` int)
> ROW FORMAT DELIMITED
> FIELDS TERMINATED BY '|'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.RCFileInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.RCFileOutputFormat'
> LOCATION
> 'hdfs://node5.lab.cloudera.com:8020/user/hive/warehouse/case_78732.db/test'
> TBLPROPERTIES (
> 'transient_lastDdlTime'='1448343875')
> {code}
> Note that there is no mention of "ColumnarSerDe"
> 4) Drop the table and then create the table again using the output from 3)
> 5) Check the output of "DESC FORMATTED test"
> {code}
> # Storage Information
> SerDe Library:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
> InputFormat: org.apache.hadoop.hive.ql.io.RCFileInputFormat
> OutputFormat: org.apache.hadoop.hive.ql.io.RCFileOutputFormat
> {code}
> The SerDe falls back to "LazySimpleSerDe", which is not correct.
> Any further query tries to INSERT or SELECT this table will fail with errors
> I suspect that we can't specify ROW FORMAT DELIMITED with ROW FORMAT SERDE at
> the same time at table creation, this causes confusion to end users as copy
> table structure using "SHOW CREATE TABLE" will not work.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12506) SHOW CREATE TABLE command creates a table that does not work for RCFile format
[
https://issues.apache.org/jira/browse/HIVE-12506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chaoyu Tang updated HIVE-12506:
---
Attachment: HIVE-12506.patch
> SHOW CREATE TABLE command creates a table that does not work for RCFile format
> --
>
> Key: HIVE-12506
> URL: https://issues.apache.org/jira/browse/HIVE-12506
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Affects Versions: 1.1.1
>Reporter: Eric Lin
>Assignee: Chaoyu Tang
> Attachments: HIVE-12506.patch
>
>
> See the following test case:
> 1) Create a table with RCFile format:
> {code}
> DROP TABLE IF EXISTS test;
> CREATE TABLE test (a int) PARTITIONED BY (p int)
> ROW FORMAT DELIMITED FIELDS TERMINATED BY '|'
> STORED AS RCFILE;
> {code}
> 2) run "DESC FORMATTED test"
> {code}
> # Storage Information
> SerDe Library:org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe
> InputFormat: org.apache.hadoop.hive.ql.io.RCFileInputFormat
> OutputFormat: org.apache.hadoop.hive.ql.io.RCFileOutputFormat
> {code}
> shows that SerDe used is "ColumnarSerDe"
> 3) run "SHOW CREATE TABLE" and get the output:
> {code}
> CREATE TABLE `test`(
> `a` int)
> PARTITIONED BY (
> `p` int)
> ROW FORMAT DELIMITED
> FIELDS TERMINATED BY '|'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.RCFileInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.RCFileOutputFormat'
> LOCATION
> 'hdfs://node5.lab.cloudera.com:8020/user/hive/warehouse/case_78732.db/test'
> TBLPROPERTIES (
> 'transient_lastDdlTime'='1448343875')
> {code}
> Note that there is no mention of "ColumnarSerDe"
> 4) Drop the table and then create the table again using the output from 3)
> 5) Check the output of "DESC FORMATTED test"
> {code}
> # Storage Information
> SerDe Library:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
> InputFormat: org.apache.hadoop.hive.ql.io.RCFileInputFormat
> OutputFormat: org.apache.hadoop.hive.ql.io.RCFileOutputFormat
> {code}
> The SerDe falls back to "LazySimpleSerDe", which is not correct.
> Any further query tries to INSERT or SELECT this table will fail with errors
> I suspect that we can't specify ROW FORMAT DELIMITED with ROW FORMAT SERDE at
> the same time at table creation, this causes confusion to end users as copy
> table structure using "SHOW CREATE TABLE" will not work.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
