[jira] [Commented] (HIVE-12730) MetadataUpdater: provide a mechanism to edit the basic statistics of a table (or a partition)
[
https://issues.apache.org/jira/browse/HIVE-12730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15136197#comment-15136197
]
Hive QA commented on HIVE-12730:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12786422/HIVE-12730.05.patch
{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 22 failed/errored test(s), 10040 tests
executed
*Failed tests:*
{noformat}
TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.org.apache.hadoop.hive.cli.TestMiniTezCliDriver
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_limit
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_optimization
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_hybridgrace_hashjoin_1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge9
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_smb_main
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_union_multiinsert
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_update_two_cols
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_date_1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_decimal_expressions
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_decimal_math_funcs
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_partition_diff_num_cols
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorization_1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorization_10
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorization_6
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testMetastoreProxyUser
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testSaslWithHiveMetaStore
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6894/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6894/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6894/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 22 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12786422 - PreCommit-HIVE-TRUNK-Build
> MetadataUpdater: provide a mechanism to edit the basic statistics of a table
> (or a partition)
> -
>
> Key: HIVE-12730
> URL: https://issues.apache.org/jira/browse/HIVE-12730
> Project: Hive
> Issue Type: New Feature
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-12730.01.patch, HIVE-12730.02.patch,
> HIVE-12730.03.patch, HIVE-12730.04.patch, HIVE-12730.05.patch
>
>
> We would like to provide a way for developers/users to modify the numRows and
> dataSize for a table/partition. Right now although they are part of the table
> properties, they will be set to -1 when the task is not coming from a
> statsTask.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12878) Support Vectorization for TEXTFILE and other formats
[ https://issues.apache.org/jira/browse/HIVE-12878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-12878: Attachment: (was: HIVE-HIVE-12878.01.patch) > Support Vectorization for TEXTFILE and other formats > > > Key: HIVE-12878 > URL: https://issues.apache.org/jira/browse/HIVE-12878 > Project: Hive > Issue Type: New Feature > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > > Support vectorizing when the input format is TEXTFILE and other formats for > better Map Vertex performance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12878) Support Vectorization for TEXTFILE and other formats
[ https://issues.apache.org/jira/browse/HIVE-12878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-12878: Attachment: HIVE-12878.01.patch > Support Vectorization for TEXTFILE and other formats > > > Key: HIVE-12878 > URL: https://issues.apache.org/jira/browse/HIVE-12878 > Project: Hive > Issue Type: New Feature > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-12878.01.patch > > > Support vectorizing when the input format is TEXTFILE and other formats for > better Map Vertex performance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12878) Support Vectorization for TEXTFILE and other formats
[ https://issues.apache.org/jira/browse/HIVE-12878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-12878: Attachment: (was: HIVE-12878.01.patch) > Support Vectorization for TEXTFILE and other formats > > > Key: HIVE-12878 > URL: https://issues.apache.org/jira/browse/HIVE-12878 > Project: Hive > Issue Type: New Feature > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > > Support vectorizing when the input format is TEXTFILE and other formats for > better Map Vertex performance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12878) Support Vectorization for TEXTFILE and other formats
[ https://issues.apache.org/jira/browse/HIVE-12878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-12878: Attachment: HIVE-12878.01.patch > Support Vectorization for TEXTFILE and other formats > > > Key: HIVE-12878 > URL: https://issues.apache.org/jira/browse/HIVE-12878 > Project: Hive > Issue Type: New Feature > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-12878.01.patch > > > Support vectorizing when the input format is TEXTFILE and other formats for > better Map Vertex performance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12878) Support Vectorization for TEXTFILE and other formats
[
https://issues.apache.org/jira/browse/HIVE-12878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15136224#comment-15136224
]
Matt McCline commented on HIVE-12878:
-
Patch submitted is an experiment:
Changed the default on these environment variables (temporarily) to force
vectorization of many queries.
{code}
hive.fetch.task.conversion=none
hive.vectorized.execution.enabled=true
{code}
New environment variables are set so that all vectorized queries either use the
new vectorized versions of deserialize for LazySimple (i.e. TEXTFILE) and
LazyBinarySerDe. Or, we deserialize row-by-row to fill up VectorizedRowBatch.
{code}
hive.vectorized.use.vectorized.input.format=false
hive.vectorized.use.vector.serde.deserialize=true
hive.vectorized.use.row.serde.deserialize=true
{code}
So, MapWork tasks not vectorizing due to input file format should not happen
(except for ACID which only is permitted for vectorized input format...).
> Support Vectorization for TEXTFILE and other formats
>
>
> Key: HIVE-12878
> URL: https://issues.apache.org/jira/browse/HIVE-12878
> Project: Hive
> Issue Type: New Feature
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-12878.01.patch
>
>
> Support vectorizing when the input format is TEXTFILE and other formats for
> better Map Vertex performance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12994) Implement support for NULLS FIRST/NULLS LAST
[
https://issues.apache.org/jira/browse/HIVE-12994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15136230#comment-15136230
]
Hive QA commented on HIVE-12994:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12786516/HIVE-12994.01.patch
{color:green}SUCCESS:{color} +1 due to 7 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 105 failed/errored test(s), 10040 tests
executed
*Failed tests:*
{noformat}
TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_numbuckets_partitioned_table2_h23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_numbuckets_partitioned_table_h23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_partition_clusterby_sortby
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_table_not_sorted
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_index
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ctas
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ctas_colname
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynpart_sort_opt_vectorization
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynpart_sort_optimization
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynpart_sort_optimization2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_explain_logical
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_grouping_window
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_resolution
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_skewtable
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_infer_bucket_sort
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_infer_bucket_sort_convert_join
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_infer_bucket_sort_grouping_operators
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_infer_bucket_sort_multi_insert
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_part7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_lineage2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_lineage3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_analyze
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_order_null
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_coltype_literals
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_pcr
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_pcs
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_pointlookup2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_pointlookup3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_pointlookup4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_vc
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ptf
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ptf_matchpath
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ptf_streaming
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ptfgroupbyjoin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_push_or
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_quotedid_basic
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_semijoin2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_semijoin4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_show_create_table_alter
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_15
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_in
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_in_having
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_notin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_unqualcolumnrefs
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_ppr
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_6_subq
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorized_ptf
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_windowing_streaming
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket5
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucketmapjoin7
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_bucketed_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.org.apache.hadoop.hive.cli.TestMiniTezCliDriver
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_ctas
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.
[jira] [Commented] (HIVE-12987) Add metrics for HS2 active users and SQL operations
[ https://issues.apache.org/jira/browse/HIVE-12987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15136234#comment-15136234 ] Aihua Xu commented on HIVE-12987: - I see. Thanks for explanation. > Add metrics for HS2 active users and SQL operations > --- > > Key: HIVE-12987 > URL: https://issues.apache.org/jira/browse/HIVE-12987 > Project: Hive > Issue Type: Task >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > Attachments: HIVE-12987.1.patch, HIVE-12987.2.patch, > HIVE-12987.2.patch, HIVE-12987.3.patch, HIVE-12987.3.patch > > > HIVE-12271 added metrics for all HS2 operations. Sometimes, users are also > interested in metrics just for SQL operations. > It is useful to track active user count as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11752) Pre-materializing complex CTE queries
[
https://issues.apache.org/jira/browse/HIVE-11752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15136363#comment-15136363
]
Hive QA commented on HIVE-11752:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12786464/HIVE-11752.05.patch
{color:green}SUCCESS:{color} +1 due to 9 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10065 tests
executed
*Failed tests:*
{noformat}
TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6896/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6896/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6896/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12786464 - PreCommit-HIVE-TRUNK-Build
> Pre-materializing complex CTE queries
> -
>
> Key: HIVE-11752
> URL: https://issues.apache.org/jira/browse/HIVE-11752
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Affects Versions: 2.1.0
>Reporter: Navis
>Assignee: Jesus Camacho Rodriguez
>Priority: Minor
> Attachments: HIVE-11752.03.patch, HIVE-11752.04.patch,
> HIVE-11752.05.patch, HIVE-11752.1.patch.txt, HIVE-11752.2.patch.txt
>
>
> Currently, hive regards CTE clauses as a simple alias to the query block,
> which makes redundant works if it's used multiple times in a query. This
> introduces a reference threshold for pre-materializing the CTE clause as a
> volatile table (which is not exists in any form of metastore and just
> accessible from QB).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13000) Hive returns useless parsing error
[
https://issues.apache.org/jira/browse/HIVE-13000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15136370#comment-15136370
]
Hive QA commented on HIVE-13000:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12786462/HIVE-13000.1.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6897/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6897/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6897/
Messages:
{noformat}
This message was trimmed, see log for full details
[INFO] Building jar:
/data/hive-ptest/working/apache-github-source-source/service-rpc/target/hive-service-rpc-2.1.0-SNAPSHOT.jar
[INFO]
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @
hive-service-rpc ---
[INFO]
[INFO] --- maven-jar-plugin:2.2:test-jar (default) @ hive-service-rpc ---
[INFO] Building jar:
/data/hive-ptest/working/apache-github-source-source/service-rpc/target/hive-service-rpc-2.1.0-SNAPSHOT-tests.jar
[INFO]
[INFO] --- maven-install-plugin:2.4:install (default-install) @
hive-service-rpc ---
[INFO] Installing
/data/hive-ptest/working/apache-github-source-source/service-rpc/target/hive-service-rpc-2.1.0-SNAPSHOT.jar
to
/data/hive-ptest/working/maven/org/apache/hive/hive-service-rpc/2.1.0-SNAPSHOT/hive-service-rpc-2.1.0-SNAPSHOT.jar
[INFO] Installing
/data/hive-ptest/working/apache-github-source-source/service-rpc/pom.xml to
/data/hive-ptest/working/maven/org/apache/hive/hive-service-rpc/2.1.0-SNAPSHOT/hive-service-rpc-2.1.0-SNAPSHOT.pom
[INFO] Installing
/data/hive-ptest/working/apache-github-source-source/service-rpc/target/hive-service-rpc-2.1.0-SNAPSHOT-tests.jar
to
/data/hive-ptest/working/maven/org/apache/hive/hive-service-rpc/2.1.0-SNAPSHOT/hive-service-rpc-2.1.0-SNAPSHOT-tests.jar
[INFO]
[INFO]
[INFO] Building Spark Remote Client 2.1.0-SNAPSHOT
[INFO]
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ spark-client ---
[INFO] Deleting
/data/hive-ptest/working/apache-github-source-source/spark-client/target
[INFO] Deleting
/data/hive-ptest/working/apache-github-source-source/spark-client (includes =
[datanucleus.log, derby.log], excludes = [])
[INFO]
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-no-snapshots) @
spark-client ---
[INFO]
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ spark-client
---
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @
spark-client ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory
/data/hive-ptest/working/apache-github-source-source/spark-client/src/main/resources
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ spark-client ---
[INFO] Executing tasks
main:
[INFO] Executed tasks
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ spark-client
---
[INFO] Compiling 28 source files to
/data/hive-ptest/working/apache-github-source-source/spark-client/target/classes
[WARNING]
/data/hive-ptest/working/apache-github-source-source/spark-client/src/main/java/org/apache/hive/spark/client/SparkClientUtilities.java:
/data/hive-ptest/working/apache-github-source-source/spark-client/src/main/java/org/apache/hive/spark/client/SparkClientUtilities.java
uses or overrides a deprecated API.
[WARNING]
/data/hive-ptest/working/apache-github-source-source/spark-client/src/main/java/org/apache/hive/spark/client/SparkClientUtilities.java:
Recompile with -Xlint:deprecation for details.
[WARNING]
/data/hive-ptest/working/apache-github-source-source/spark-client/src/main/java/org/apache/hive/spark/client/rpc/RpcDispatcher.java:
Some input files use unchecked or unsafe operations.
[WARNING]
/data/hive-ptest/working/apache-github-source-source/spark-client/src/main/java/org/apache/hive/spark/client/rpc/RpcDispatcher.java:
Recompile with -Xlint:unchecked for details.
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @
spark-client ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ spark-client ---
[INFO] Executing tasks
main:
[mkdir] Created dir:
/data/hive-ptest/working/apache-github-source-source/spark-client/target/tmp
[mkdir] Created dir:
/data/hive-ptest/working/apache-github-source-source/spark-cli
[jira] [Commented] (HIVE-12987) Add metrics for HS2 active users and SQL operations
[ https://issues.apache.org/jira/browse/HIVE-12987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15136395#comment-15136395 ] Jimmy Xiang commented on HIVE-12987: Sure, I can make Operation::setMetrics private and use onNewState() callback so that we can avoid the new prevState field. However, the SQLOperation needs to emit similar metrics as Operation. We still need to move the shared code in the original setMetrics to a method that can be used by SQLOpertation, so that we don't duplicate code, right? As to the state of SQLOperation, it is accessed from the normal HS2 handler thread and the async pooled processing thread. It is safer to be volatile, isn't it? > Add metrics for HS2 active users and SQL operations > --- > > Key: HIVE-12987 > URL: https://issues.apache.org/jira/browse/HIVE-12987 > Project: Hive > Issue Type: Task >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > Attachments: HIVE-12987.1.patch, HIVE-12987.2.patch, > HIVE-12987.2.patch, HIVE-12987.3.patch, HIVE-12987.3.patch > > > HIVE-12271 added metrics for all HS2 operations. Sometimes, users are also > interested in metrics just for SQL operations. > It is useful to track active user count as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12856) LLAP: update (add/remove) the UDFs available in LLAP when they are changed (refresh periodically)
[
https://issues.apache.org/jira/browse/HIVE-12856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15136462#comment-15136462
]
Hive QA commented on HIVE-12856:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12786557/HIVE-12856.patch
{color:green}SUCCESS:{color} +1 due to 21 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 86 failed/errored test(s), 10028 tests
executed
*Failed tests:*
{noformat}
TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapCliDriver
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_bucket_map_join_tez1
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_bucket_map_join_tez2
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_constprog_dpp
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_dynamic_partition_pruning_2
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_hybridgrace_hashjoin_1
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_hybridgrace_hashjoin_2
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_llapdecider
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_lvj_mapjoin
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_mapjoin_decimal
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_mrr
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_orc_ppd_basic
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_bmj_schema_evolution
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_dml
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_dynpart_hashjoin_1
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_dynpart_hashjoin_2
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_fsstat
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_insert_overwrite_local_directory_1
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_join
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_join_hash
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_join_result_complex
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_join_tests
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_joins_explain
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_multi_union
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_schema_evolution
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_self_join
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_smb_1
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_smb_main
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_union
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_union2
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_union_decimal
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_union_dynamic_partition
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_union_group_by
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_union_multiinsert
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_vector_dynpart_hashjoin_1
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_vector_dynpart_hashjoin_2
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_join_part_col_char
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vectorized_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInEmbed.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInEmbed
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testAlterPartition
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testAlterTable
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testAlterViewParititon
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testColumnStatistics
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testComplexTable
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testComplexTypeApi
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testConcurrentMetastores
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testDBOwner
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testDBOwnerChange
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testDatabase
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testDatabaseLocation
org.apa
[jira] [Updated] (HIVE-13015) Update SLF4j version to 1.7.10
[
https://issues.apache.org/jira/browse/HIVE-13015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-13015:
-
Summary: Update SLF4j version to 1.7.10 (was: Add dependency convergence
for SLF4J)
> Update SLF4j version to 1.7.10
> --
>
> Key: HIVE-13015
> URL: https://issues.apache.org/jira/browse/HIVE-13015
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>
> In some of the recent test runs, we are seeing multiple bindings for SLF4j
> that causes issues with LOG4j2 logger.
> {code}
> SLF4J: Found binding in
> [jar:file:/grid/0/hadoop/yarn/local/usercache/hrt_qa/appcache/application_1454694331819_0001/container_e06_1454694331819_0001_01_02/app/install/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> {code}
> We have added explicit exclusions for slf4j-log4j12 but some library is
> pulling it transitively and it's getting packaged with hive libs. Also hive
> currently uses version 1.7.5 for slf4j. We should add dependency convergence
> for sl4fj and also remove packaging of slf4j-log4j12.*.jar
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13015) Update SLF4j version to 1.7.10
[
https://issues.apache.org/jira/browse/HIVE-13015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-13015:
-
Attachment: HIVE-13015.1.patch
I tried dependency convergence enforcer plugin in maven but that brings in
whole lot of errors due to diverging dependency versions. For now just updating
the slf4j version to 1.7.10 (hadoop, tez and spark uses this version).
> Update SLF4j version to 1.7.10
> --
>
> Key: HIVE-13015
> URL: https://issues.apache.org/jira/browse/HIVE-13015
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-13015.1.patch
>
>
> In some of the recent test runs, we are seeing multiple bindings for SLF4j
> that causes issues with LOG4j2 logger.
> {code}
> SLF4J: Found binding in
> [jar:file:/grid/0/hadoop/yarn/local/usercache/hrt_qa/appcache/application_1454694331819_0001/container_e06_1454694331819_0001_01_02/app/install/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> {code}
> We have added explicit exclusions for slf4j-log4j12 but some library is
> pulling it transitively and it's getting packaged with hive libs. Also hive
> currently uses version 1.7.5 for slf4j. We should add dependency convergence
> for sl4fj and also remove packaging of slf4j-log4j12.*.jar
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13016) ORC FileDump recovery utility fails in Windows
[
https://issues.apache.org/jira/browse/HIVE-13016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15136513#comment-15136513
]
Hive QA commented on HIVE-13016:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12786599/HIVE-13016.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 10052 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6899/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6899/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6899/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12786599 - PreCommit-HIVE-TRUNK-Build
> ORC FileDump recovery utility fails in Windows
> --
>
> Key: HIVE-13016
> URL: https://issues.apache.org/jira/browse/HIVE-13016
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.3.0, 2.1.0
>Reporter: Jason Dere
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-13016.1.patch, HIVE-13016.2.patch
>
>
> org.apache.hive.hcatalog.streaming.TestStreaming.testFileDumpCorruptDataFiles
> org.apache.hive.hcatalog.streaming.TestStreaming.testFileDumpCorruptSideFiles
> java.io.IOException: Unable to move
> file:/E:/hive/hcatalog/streaming/target/tmp/junit4129594478393496260/testing3.db/dimensionTable/delta_001_002/bucket_0
> to
> E:/hive/hcatalog/streaming/target/tmp/E:/hive/hcatalog/streaming/target/tmp/junit4129594478393496260/testing3.db/dimensionTable/delta_001_002/bucket_0
> at
> org.apache.hadoop.hive.ql.io.orc.FileDump.moveFiles(FileDump.java:546)^M
> at
> org.apache.hadoop.hive.ql.io.orc.FileDump.recoverFile(FileDump.java:513)^M
> at
> org.apache.hadoop.hive.ql.io.orc.FileDump.recoverFiles(FileDump.java:428)^M
> at org.apache.hadoop.hive.ql.io.orc.FileDump.main(FileDump.java:125)^M
> at
> org.apache.hive.hcatalog.streaming.TestStreaming.testFileDumpCorruptSideFiles(TestStreaming.java:1523)^M
> Note that FileDump appends the full source path to the backup path when
> trying to recover files (see "E:" in the middle of the destination path).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11526) LLAP: implement LLAP UI as a separate service
[ https://issues.apache.org/jira/browse/HIVE-11526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuya OZAWA updated HIVE-11526: -- Attachment: HIVE-11526.patch I attached first patch. > LLAP: implement LLAP UI as a separate service > - > > Key: HIVE-11526 > URL: https://issues.apache.org/jira/browse/HIVE-11526 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Sergey Shelukhin >Assignee: Yuya OZAWA > Attachments: HIVE-11526.patch, llap_monitor_design.pdf > > > The specifics are vague at this point. > Hadoop metrics can be output, as well as metrics we collect and output in > jmx, as well as those we collect per fragment and log right now. > This service can do LLAP-specific views, and per-query aggregation. > [~gopalv] may have some information on how to reuse existing solutions for > part of the work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11526) LLAP: implement LLAP UI as a separate service
[ https://issues.apache.org/jira/browse/HIVE-11526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15136520#comment-15136520 ] Yuya OZAWA commented on HIVE-11526: --- It can look single node metrics. http://:/webapps/llap/metrics.html port is hive.llap.daemon.web.port(default:15002) > LLAP: implement LLAP UI as a separate service > - > > Key: HIVE-11526 > URL: https://issues.apache.org/jira/browse/HIVE-11526 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Sergey Shelukhin >Assignee: Yuya OZAWA > Attachments: HIVE-11526.patch, llap_monitor_design.pdf > > > The specifics are vague at this point. > Hadoop metrics can be output, as well as metrics we collect and output in > jmx, as well as those we collect per fragment and log right now. > This service can do LLAP-specific views, and per-query aggregation. > [~gopalv] may have some information on how to reuse existing solutions for > part of the work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12857) LLAP: modify the decider to allow using LLAP with whitelisted UDFs
[
https://issues.apache.org/jira/browse/HIVE-12857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15136557#comment-15136557
]
Hive QA commented on HIVE-12857:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12786594/HIVE-12857.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 10039 tests
executed
*Failed tests:*
{noformat}
TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_llap_udf
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_create_function_nonexistent_class
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6900/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6900/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6900/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12786594 - PreCommit-HIVE-TRUNK-Build
> LLAP: modify the decider to allow using LLAP with whitelisted UDFs
> --
>
> Key: HIVE-12857
> URL: https://issues.apache.org/jira/browse/HIVE-12857
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-12857.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13020) Hive Zookeeper Connection From MetaStore and HiveServer2 fails with IBM JDK
[ https://issues.apache.org/jira/browse/HIVE-13020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Greg Senia updated HIVE-13020: -- Attachment: HIVE-13020.patch Patch > Hive Zookeeper Connection From MetaStore and HiveServer2 fails with IBM JDK > --- > > Key: HIVE-13020 > URL: https://issues.apache.org/jira/browse/HIVE-13020 > Project: Hive > Issue Type: Bug > Components: HiveServer2, Metastore, Shims >Affects Versions: 1.2.0, 1.3.0, 1.2.1 > Environment: Linux X86_64 and IBM JDK 8 >Reporter: Greg Senia >Assignee: Greg Senia > Fix For: 1.3.0, 2.0.0, 1.2.2, 2.1.0 > > Attachments: HIVE-13020.patch, hivemetastore_afterpatch.txt, > hivemetastore_beforepatch.txt, hiveserver2_afterpatch.txt, > hiveserver2_beforepatch.txt > > > HiveServer2 and Hive Metastore Zookeeper component is hardcoded to only > support the Oracle/Open JDK. I was performing testing of Hadoop running on > the IBM JDK and discovered this issue and have since drawn up the attached > patch. This looks to resolve the issue in a similar manner as how the Hadoop > core folks handle the IBM JDK. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13020) Hive Zookeeper Connection From MetaStore and HiveServer2 fails with IBM JDK
[ https://issues.apache.org/jira/browse/HIVE-13020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Greg Senia updated HIVE-13020: -- Attachment: hiveserver2_beforepatch.txt hiveserver2_afterpatch.txt hivemetastore_beforepatch.txt hivemetastore_afterpatch.txt Logs showing before and after patching with the provided patch > Hive Zookeeper Connection From MetaStore and HiveServer2 fails with IBM JDK > --- > > Key: HIVE-13020 > URL: https://issues.apache.org/jira/browse/HIVE-13020 > Project: Hive > Issue Type: Bug > Components: HiveServer2, Metastore, Shims >Affects Versions: 1.2.0, 1.3.0, 1.2.1 > Environment: Linux X86_64 and IBM JDK 8 >Reporter: Greg Senia >Assignee: Greg Senia > Fix For: 1.3.0, 2.0.0, 1.2.2, 2.1.0 > > Attachments: HIVE-13020.patch, hivemetastore_afterpatch.txt, > hivemetastore_beforepatch.txt, hiveserver2_afterpatch.txt, > hiveserver2_beforepatch.txt > > > HiveServer2 and Hive Metastore Zookeeper component is hardcoded to only > support the Oracle/Open JDK. I was performing testing of Hadoop running on > the IBM JDK and discovered this issue and have since drawn up the attached > patch. This looks to resolve the issue in a similar manner as how the Hadoop > core folks handle the IBM JDK. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13020) Hive Metastore and HiveServer2 to Zookeeper fails with IBM JDK
[ https://issues.apache.org/jira/browse/HIVE-13020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Greg Senia updated HIVE-13020: -- Labels: hdp ibm ibm-jdk (was: ) > Hive Metastore and HiveServer2 to Zookeeper fails with IBM JDK > -- > > Key: HIVE-13020 > URL: https://issues.apache.org/jira/browse/HIVE-13020 > Project: Hive > Issue Type: Bug > Components: HiveServer2, Metastore, Shims >Affects Versions: 1.2.0, 1.3.0, 1.2.1 > Environment: Linux X86_64 and IBM JDK 8 >Reporter: Greg Senia >Assignee: Greg Senia > Labels: hdp, ibm, ibm-jdk > Fix For: 1.3.0, 2.0.0, 1.2.2, 2.1.0 > > Attachments: HIVE-13020.patch, hivemetastore_afterpatch.txt, > hivemetastore_beforepatch.txt, hiveserver2_afterpatch.txt, > hiveserver2_beforepatch.txt > > > HiveServer2 and Hive Metastore Zookeeper component is hardcoded to only > support the Oracle/Open JDK. I was performing testing of Hadoop running on > the IBM JDK and discovered this issue and have since drawn up the attached > patch. This looks to resolve the issue in a similar manner as how the Hadoop > core folks handle the IBM JDK. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13020) Hive Metastore and HiveServer2 to Zookeeper fails with IBM JDK
[ https://issues.apache.org/jira/browse/HIVE-13020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Greg Senia updated HIVE-13020: -- Summary: Hive Metastore and HiveServer2 to Zookeeper fails with IBM JDK (was: Hive Zookeeper Connection From MetaStore and HiveServer2 fails with IBM JDK) > Hive Metastore and HiveServer2 to Zookeeper fails with IBM JDK > -- > > Key: HIVE-13020 > URL: https://issues.apache.org/jira/browse/HIVE-13020 > Project: Hive > Issue Type: Bug > Components: HiveServer2, Metastore, Shims >Affects Versions: 1.2.0, 1.3.0, 1.2.1 > Environment: Linux X86_64 and IBM JDK 8 >Reporter: Greg Senia >Assignee: Greg Senia > Labels: hdp, ibm, ibm-jdk > Fix For: 1.3.0, 2.0.0, 1.2.2, 2.1.0 > > Attachments: HIVE-13020.patch, hivemetastore_afterpatch.txt, > hivemetastore_beforepatch.txt, hiveserver2_afterpatch.txt, > hiveserver2_beforepatch.txt > > > HiveServer2 and Hive Metastore Zookeeper component is hardcoded to only > support the Oracle/Open JDK. I was performing testing of Hadoop running on > the IBM JDK and discovered this issue and have since drawn up the attached > patch. This looks to resolve the issue in a similar manner as how the Hadoop > core folks handle the IBM JDK. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12918) LLAP should never create embedded metastore when localizing functions
[
https://issues.apache.org/jira/browse/HIVE-12918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15136595#comment-15136595
]
Hive QA commented on HIVE-12918:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12786601/HIVE-12918.03.patch
{color:green}SUCCESS:{color} +1 due to 23 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10037 tests
executed
*Failed tests:*
{noformat}
TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6901/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6901/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6901/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12786601 - PreCommit-HIVE-TRUNK-Build
> LLAP should never create embedded metastore when localizing functions
> -
>
> Key: HIVE-12918
> URL: https://issues.apache.org/jira/browse/HIVE-12918
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 2.1.0
>Reporter: Siddharth Seth
>Assignee: Sergey Shelukhin
> Attachments: HIVE-12918.01.patch, HIVE-12918.02.patch,
> HIVE-12918.03.patch, HIVE-12918.patch
>
>
> {code}
> 16/01/24 21:29:02 INFO service.AbstractService: Service LlapDaemon failed in
> state INITED; cause: java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1552)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:86)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:132)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:104)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3110)
> at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3130)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:3355)
> at
> org.apache.hadoop.hive.llap.daemon.impl.FunctionLocalizer.startLocalizeAllFunctions(FunctionLocalizer.java:88)
> at
> org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon.serviceInit(LlapDaemon.java:244)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon.main(LlapDaemon.java:323)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1550)
> ... 10 more
> Caused by: java.lang.NoClassDefFoundError: org/datanucleus/NucleusContext
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.getClass(MetaStoreUtils.java:1517)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:61)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:568)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:533)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:595)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:387)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.(RetryingHMSHandler.java:78)
> a
[jira] [Assigned] (HIVE-13011) HIve 0.14 has introduce transaction feature. but when inserting a row my API stuck at " kill command"
[
https://issues.apache.org/jira/browse/HIVE-13011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gaurav Tomar reassigned HIVE-13011:
---
Assignee: Gaurav Tomar
> HIve 0.14 has introduce transaction feature. but when inserting a row my API
> stuck at " kill command"
> -
>
> Key: HIVE-13011
> URL: https://issues.apache.org/jira/browse/HIVE-13011
> Project: Hive
> Issue Type: New Feature
> Components: API
>Affects Versions: 0.14.0
>Reporter: Gaurav Tomar
>Assignee: Gaurav Tomar
>
> Hive 0.14 has introduce transactions feature. But we need to configure few
> things before any transaction e.g
> set hive.support.concurrency = true
> set hive.enforce.bucketing = true
> set hive.exec.dynamic.partition.mode = nonstrict
> set hive.txn.manager =org.apache.hadoop.hive.ql.lockmgr.DbTxnManager
> set hive.compactor.initiator.on = true
> set hive.compactor.worker.threads = 1
> to create a table with this command :
> CREATE TABLE college(clg_id int,clg_name string,clg_loc string) clustered by
> (clg_id) into 5 buckets stored as orc TBLPROPERTIES('transactional'='true');
> Now when I am inserting a row into the table using
> INSERT INTO table college
> values(1,'nec','nlr'),(2,'vit','vlr'),(3,'srm','chen'),(4,'lpu','del'),(5,'stanford','uk'),(6,'JNTUA','atp'),(7,'cambridge','us');
> My Hive API stuck at "Kill Command" describe below.
> Query ID = training_20160205183232_2bb88ba5-187b-4f9e-b93a-47c91b87c79a
> Total jobs = 1
> Launching Job 1 out of 1
> Number of reduce tasks is set to 0 since there's no reduce operator
> Starting Job = job_1424080128507_5229, Tracking URL =
> http://hydetamaster:8088/proxy/application_1424080128507_5229/
> Kill Command = /opt/installation/hadoop/bin/hadoop job -kill
> job_1424080128507_5229
> Can anybody please help me out with this ??
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13011) HIve 0.14 has introduce transaction feature. but when inserting a row my API stuck at " kill command"
[
https://issues.apache.org/jira/browse/HIVE-13011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gaurav Tomar updated HIVE-13011:
Assignee: (was: Gaurav Tomar)
> HIve 0.14 has introduce transaction feature. but when inserting a row my API
> stuck at " kill command"
> -
>
> Key: HIVE-13011
> URL: https://issues.apache.org/jira/browse/HIVE-13011
> Project: Hive
> Issue Type: New Feature
> Components: API
>Affects Versions: 0.14.0
>Reporter: Gaurav Tomar
>
> Hive 0.14 has introduce transactions feature. But we need to configure few
> things before any transaction e.g
> set hive.support.concurrency = true
> set hive.enforce.bucketing = true
> set hive.exec.dynamic.partition.mode = nonstrict
> set hive.txn.manager =org.apache.hadoop.hive.ql.lockmgr.DbTxnManager
> set hive.compactor.initiator.on = true
> set hive.compactor.worker.threads = 1
> to create a table with this command :
> CREATE TABLE college(clg_id int,clg_name string,clg_loc string) clustered by
> (clg_id) into 5 buckets stored as orc TBLPROPERTIES('transactional'='true');
> Now when I am inserting a row into the table using
> INSERT INTO table college
> values(1,'nec','nlr'),(2,'vit','vlr'),(3,'srm','chen'),(4,'lpu','del'),(5,'stanford','uk'),(6,'JNTUA','atp'),(7,'cambridge','us');
> My Hive API stuck at "Kill Command" describe below.
> Query ID = training_20160205183232_2bb88ba5-187b-4f9e-b93a-47c91b87c79a
> Total jobs = 1
> Launching Job 1 out of 1
> Number of reduce tasks is set to 0 since there's no reduce operator
> Starting Job = job_1424080128507_5229, Tracking URL =
> http://hydetamaster:8088/proxy/application_1424080128507_5229/
> Kill Command = /opt/installation/hadoop/bin/hadoop job -kill
> job_1424080128507_5229
> Can anybody please help me out with this ??
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12839) Upgrade Hive to Calcite 1.6
[ https://issues.apache.org/jira/browse/HIVE-12839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-12839: --- Attachment: HIVE-12839.05.patch > Upgrade Hive to Calcite 1.6 > --- > > Key: HIVE-12839 > URL: https://issues.apache.org/jira/browse/HIVE-12839 > Project: Hive > Issue Type: Improvement >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-12839.01.patch, HIVE-12839.02.patch, > HIVE-12839.03.patch, HIVE-12839.04.patch, HIVE-12839.05.patch > > > CLEAR LIBRARY CACHE > Upgrade Hive to Calcite 1.6.0-incubating. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12730) MetadataUpdater: provide a mechanism to edit the basic statistics of a table (or a partition)
[ https://issues.apache.org/jira/browse/HIVE-12730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15136641#comment-15136641 ] Pengcheng Xiong commented on HIVE-12730: I could not reproduce the test case failures. [~ashutoshc], could you please review the patch? In the mean time, I will resubmit again to make sure everything works fine. Thanks. > MetadataUpdater: provide a mechanism to edit the basic statistics of a table > (or a partition) > - > > Key: HIVE-12730 > URL: https://issues.apache.org/jira/browse/HIVE-12730 > Project: Hive > Issue Type: New Feature >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-12730.01.patch, HIVE-12730.02.patch, > HIVE-12730.03.patch, HIVE-12730.04.patch, HIVE-12730.05.patch > > > We would like to provide a way for developers/users to modify the numRows and > dataSize for a table/partition. Right now although they are part of the table > properties, they will be set to -1 when the task is not coming from a > statsTask. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12730) MetadataUpdater: provide a mechanism to edit the basic statistics of a table (or a partition)
[ https://issues.apache.org/jira/browse/HIVE-12730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-12730: --- Attachment: HIVE-12730.06.patch > MetadataUpdater: provide a mechanism to edit the basic statistics of a table > (or a partition) > - > > Key: HIVE-12730 > URL: https://issues.apache.org/jira/browse/HIVE-12730 > Project: Hive > Issue Type: New Feature >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-12730.01.patch, HIVE-12730.02.patch, > HIVE-12730.03.patch, HIVE-12730.04.patch, HIVE-12730.05.patch, > HIVE-12730.06.patch > > > We would like to provide a way for developers/users to modify the numRows and > dataSize for a table/partition. Right now although they are part of the table > properties, they will be set to -1 when the task is not coming from a > statsTask. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12592) Expose connection pool tuning props in TxnHandler
[ https://issues.apache.org/jira/browse/HIVE-12592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15136645#comment-15136645 ] Chetna Chaudhari commented on HIVE-12592: - [~ekoifman]: Any progress on this? If not, I would like to take it up, as we are seeing its need in production. > Expose connection pool tuning props in TxnHandler > - > > Key: HIVE-12592 > URL: https://issues.apache.org/jira/browse/HIVE-12592 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > > BoneCP allows various pool tuning options like connection timeout, num > connections, etc > There should be a config based way to set these -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9862) Vectorized execution corrupts timestamp values
[
https://issues.apache.org/jira/browse/HIVE-9862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15136650#comment-15136650
]
Hive QA commented on HIVE-9862:
---
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12786620/HIVE-9862.09.patch
{color:green}SUCCESS:{color} +1 due to 18 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 9766 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6902/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6902/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6902/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12786620 - PreCommit-HIVE-TRUNK-Build
> Vectorized execution corrupts timestamp values
> --
>
> Key: HIVE-9862
> URL: https://issues.apache.org/jira/browse/HIVE-9862
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 1.0.0
>Reporter: Nathan Howell
>Assignee: Matt McCline
> Attachments: HIVE-9862.01.patch, HIVE-9862.02.patch,
> HIVE-9862.03.patch, HIVE-9862.04.patch, HIVE-9862.05.patch,
> HIVE-9862.06.patch, HIVE-9862.07.patch, HIVE-9862.08.patch, HIVE-9862.09.patch
>
>
> Timestamps in the future (year 2250?) and before ~1700 are silently corrupted
> in vectorized execution mode. Simple repro:
> {code}
> hive> DROP TABLE IF EXISTS test;
> hive> CREATE TABLE test(ts TIMESTAMP) STORED AS ORC;
> hive> INSERT INTO TABLE test VALUES ('-12-31 23:59:59');
> hive> SET hive.vectorized.execution.enabled = false;
> hive> SELECT MAX(ts) FROM test;
> -12-31 23:59:59
> hive> SET hive.vectorized.execution.enabled = true;
> hive> SELECT MAX(ts) FROM test;
> 1816-03-30 05:56:07.066277376
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
