[jira] [Updated] (HIVE-27966) Disable flaky testFetchResultsOfLogWithOrientation
[
https://issues.apache.org/jira/browse/HIVE-27966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-27966:
--
Labels: pull-request-available (was: )
> Disable flaky testFetchResultsOfLogWithOrientation
> --
>
> Key: HIVE-27966
> URL: https://issues.apache.org/jira/browse/HIVE-27966
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
> Labels: pull-request-available
>
> {{org.apache.hive.service.cli.operation.TestOperationLoggingAPIWithMr#testFetchResultsOfLogWithOrientation}}
> test is flaky and can easily result into CI failed.
> Here are some affected pipelines:
> http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-4959/3/tests
> http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-4755/15/tests
> http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-4966/1/tests
> http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-4517/25/tests
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-27966) Disable flaky testFetchResultsOfLogWithOrientation
Wechar created HIVE-27966:
-
Summary: Disable flaky testFetchResultsOfLogWithOrientation
Key: HIVE-27966
URL: https://issues.apache.org/jira/browse/HIVE-27966
Project: Hive
Issue Type: Bug
Components: Hive
Reporter: Wechar
Assignee: Wechar
{{org.apache.hive.service.cli.operation.TestOperationLoggingAPIWithMr#testFetchResultsOfLogWithOrientation}}
test is flaky and can easily result into CI failed.
Here are some affected pipelines:
http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-4959/3/tests
http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-4755/15/tests
http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-4966/1/tests
http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-4517/25/tests
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-27965) Table/partition rename takes a long time at PART_COL_STATS for wide tables
[
https://issues.apache.org/jira/browse/HIVE-27965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17799649#comment-17799649
]
Naresh P R commented on HIVE-27965:
---
[~zhangbutao]
Yes, this helps. Thanks for letting me know.
> Table/partition rename takes a long time at PART_COL_STATS for wide tables
> --
>
> Key: HIVE-27965
> URL: https://issues.apache.org/jira/browse/HIVE-27965
> Project: Hive
> Issue Type: Improvement
>Reporter: Naresh P R
>Priority: Major
>
> Partition table rename gets clogged at PART_COL_STATS for wide tables.
> {code:java}
> CREATE TABLE IF NOT EXISTS `PART_COL_STATS` (
> ...
> `DB_NAME` varchar(128) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL,
> `TABLE_NAME` varchar(128) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL,
> `PARTITION_NAME` varchar(767) CHARACTER SET latin1 COLLATE latin1_bin NOT
> NULL,
> ...){code}
> Since PART_COL_STATS holds db_name & table_name, incase of table rename,
> every row in PART_COL_STATS associated with the table should be fetched,
> stored in memory, delete & re-insert with new db/table/partition name.
>
> Remove DB_NAME, TABLE_NAME, PARTITION_NAME from PART_COL_STATS, instead use
> TBL_ID, DB_ID, PART_ID to avoid touching PART_COL_STATS for table/partition
> renames.
> Also TBL_ID, DB_ID, PART_ID can be used for PART_COL_STATS INDEXING.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (HIVE-27964) Support drop stats similar to Impala
[
https://issues.apache.org/jira/browse/HIVE-27964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17799626#comment-17799626
]
Butao Zhang edited comment on HIVE-27964 at 12/22/23 1:28 AM:
--
Thank you for your explanation!
We have a ticket HIVE-27725 about removal {{DB_NAME}} & {{TABLE_NAME}} in
TAB_COL_STATS and PART_COL_STATS, which is under active review.
Do you think HIVE-27725 can help to resolve your problem? Please feel free to
comment the PR HIVE-27725 if you have any suggestion.
Also cc [~dkuzmenko] [~wechar]
Thanks.
was (Author: zhangbutao):
Thank you for your explanation!
We have a ticket HIVE-27725 about removal {{DB_NAME}} and \{{TABLE_NAME in
TAB_COL_STATS and PART_COL_STATS, }}which is under active review.
Do you think HIVE-27725 can help to resolve your problem? Please feel free to
comment the PR HIVE-27725 if you have any suggestion.
Also cc [~dkuzmenko] [~wechar]
Thanks.
> Support drop stats similar to Impala
>
>
> Key: HIVE-27964
> URL: https://issues.apache.org/jira/browse/HIVE-27964
> Project: Hive
> Issue Type: New Feature
>Reporter: Naresh P R
>Priority: Major
>
> Hive should support drop stats similar to impala.
> https://impala.apache.org/docs/build/html/topics/impala_drop_stats.html
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (HIVE-27964) Support drop stats similar to Impala
[
https://issues.apache.org/jira/browse/HIVE-27964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17799626#comment-17799626
]
Butao Zhang edited comment on HIVE-27964 at 12/22/23 1:19 AM:
--
Thank you for your explanation!
We have a ticket HIVE-27725 about removal {{DB_NAME}} and \{{TABLE_NAME in
TAB_COL_STATS and PART_COL_STATS, }}which is under active review.
Do you think HIVE-27725 can help to resolve your problem? Please feel free to
comment the PR HIVE-27725 if you have any suggestion.
Also cc [~dkuzmenko] [~wechar]
Thanks.
was (Author: zhangbutao):
Thank you for your explanation!
We have a ticket HIVE-27725 about removal {{DB_NAME}} and {{TABLE_NAME in
TAB_COL_STATS and PART_COL_STATS, }}which is under active review.
Do you think HIVE-27725 can help to resolve your problem? Please feel free to
comment the PR HIVE-27725 if you have any suggestion.
Thanks.
> Support drop stats similar to Impala
>
>
> Key: HIVE-27964
> URL: https://issues.apache.org/jira/browse/HIVE-27964
> Project: Hive
> Issue Type: New Feature
>Reporter: Naresh P R
>Priority: Major
>
> Hive should support drop stats similar to impala.
> https://impala.apache.org/docs/build/html/topics/impala_drop_stats.html
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-27964) Support drop stats similar to Impala
[
https://issues.apache.org/jira/browse/HIVE-27964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17799626#comment-17799626
]
Butao Zhang commented on HIVE-27964:
Thank you for your explanation!
We have a ticket HIVE-27725 about removal {{DB_NAME}} and {{TABLE_NAME in
TAB_COL_STATS and PART_COL_STATS, }}which is under active review.
Do you think HIVE-27725 can help to resolve your problem? Please feel free to
comment the PR HIVE-27725 if you have any suggestion.
Thanks.
> Support drop stats similar to Impala
>
>
> Key: HIVE-27964
> URL: https://issues.apache.org/jira/browse/HIVE-27964
> Project: Hive
> Issue Type: New Feature
>Reporter: Naresh P R
>Priority: Major
>
> Hive should support drop stats similar to impala.
> https://impala.apache.org/docs/build/html/topics/impala_drop_stats.html
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HIVE-14) selecting fields from a complex object column in transform clause is throwing a Parse Error.
[ https://issues.apache.org/jira/browse/HIVE-14?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HIVE-14. -- Fix Version/s: Not Applicable Resolution: Cannot Reproduce > selecting fields from a complex object column in transform clause is throwing > a Parse Error. > > > Key: HIVE-14 > URL: https://issues.apache.org/jira/browse/HIVE-14 > Project: Hive > Issue Type: Bug > Components: Query Processor >Reporter: Prasad Chakka >Priority: Critical > Fix For: Not Applicable > > > For a thrift table whose columns are complex objects the following query > throws up an error > from ( from cdx select transform(cdx.a.b) as tx using 'mapper' cluster by tx > ) mo insert into output select tx; > the error thrown on the second . of expresion 'cdx.a.b'. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-9187) Hive0.14.0 query fail with Tez0.5.3 and Hadoop2.4.1
[ https://issues.apache.org/jira/browse/HIVE-9187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HIVE-9187. Fix Version/s: Not Applicable Resolution: Won't Fix the versions mentioned are either EOL or deprecated try the latest versions > Hive0.14.0 query fail with Tez0.5.3 and Hadoop2.4.1 > --- > > Key: HIVE-9187 > URL: https://issues.apache.org/jira/browse/HIVE-9187 > Project: Hive > Issue Type: Bug > Environment: CentOS 6 >Reporter: Ethan Hu >Priority: Critical > Fix For: Not Applicable > > > The Operation System，CentOS 6 > Software version, Hadoop 2.4.1, Tez 0.5.3, Hive 0.14.0. > The hive table structure is > CREATE EXTERNAL TABLE IF NOT EXISTS inv_info(optDate string, code int, custno > int, sku int, euno int, totalamount double, price double, type int, > finalamout double) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LOCATION > '/user/hadoop/inv'; > Use Hive local Cli for query, > 1. Use the default engine "mr", execute the Hive QL, it's normal > hive> select count(*) from inv_info; > Automatically selecting local only mode for query > Query ID = root_20141221181313_60e150e1-34dd-4bea-8d4f-1a703c5516d6 > Total jobs = 1 > Launching Job 1 out of 1 > Number of reduce tasks determined at compile time: 1 > In order to change the average load for a reducer (in bytes): > set hive.exec.reducers.bytes.per.reducer= > In order to limit the maximum number of reducers: > set hive.exec.reducers.max= > In order to set a constant number of reducers: > set mapreduce.job.reduces= > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > [jar:file:/webapp/hadoop-2.4.1/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/webapp/apache-hive-0.14.0-bin/lib/hive-jdbc-0.14.0-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an > explanation. > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] > 14/12/21 18:13:50 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 14/12/21 18:13:51 WARN conf.Configuration: > file:/tmp/root/2006970b-1b5e-41e7-a604-4ed7405cc7e9/hive_2014-12-21_18-13-41_747_2372484449278617092-3/-local-10003/jobconf.xml:an > attempt to override final parameter: > mapreduce.job.end-notification.max.retry.interval; Ignoring. > 14/12/21 18:13:51 WARN conf.Configuration: > file:/tmp/root/2006970b-1b5e-41e7-a604-4ed7405cc7e9/hive_2014-12-21_18-13-41_747_2372484449278617092-3/-local-10003/jobconf.xml:an > attempt to override final parameter: > mapreduce.job.end-notification.max.attempts; Ignoring. > Execution log at: > /tmp/root/root_20141221181313_60e150e1-34dd-4bea-8d4f-1a703c5516d6.log > Job running in-process (local Hadoop) > Hadoop job information for null: number of mappers: 0; number of reducers: 0 > 2014-12-21 18:14:00,666 null map = 0%, reduce = 0% > 2014-12-21 18:14:05,365 null map = 100%, reduce = 100% > Ended Job = job_local886866076_0001 > Execution completed successfully > MapredLocal task succeeded > OK > 2146872 > Time taken: 26.281 seconds, Fetched: 1 row(s) > 2. when I set hive.execution.engine=tez; the query is fail, can anyone help > me to find the reason? I looked up for many reference materials, can't > resolve it. > hive> select count(*) from inv_info; > Query ID = root_20141221181515_cbf8f20b-e105-441e-b044-269266088c48 > Total jobs = 1 > Launching Job 1 out of 1 > Status: Running (Executing on YARN cluster with App id > application_1418953370736_0009) > > VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED > KILLED > > Map 1 FAILED 4 004 9 > 3 > Reducer 2 KILLED 1 001 0 > 1 > > VERTICES: 00/02 [>>--] 0%ELAPSED TIME: 29.57 s > > Status: Failed > Vertex failed, vertexName=Map 1, vertexId=vertex_1418953370736_0009_1_00, > diagnostics=[Task failed, taskId=task_1418953370736_0009_1_00_00, > diagnostics=[TaskAttempt 0 failed, info=[Container > container_1418953370736_0009_01_02 finished with diagnostics set to > [Container failed. Exception from container-launch: > org.apache.hadoop.util.Shell$ExitCodeException: > org.apache.hadoop.util.Shell$ExitCodeException: >
[jira] [Updated] (HIVE-12369) Native Vector GROUP BY
[
https://issues.apache.org/jira/browse/HIVE-12369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena updated HIVE-12369:
Issue Type: Improvement (was: Bug)
> Native Vector GROUP BY
> --
>
> Key: HIVE-12369
> URL: https://issues.apache.org/jira/browse/HIVE-12369
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-12369.01.patch, HIVE-12369.02.patch,
> HIVE-12369.05.patch, HIVE-12369.06.patch, HIVE-12369.091.patch,
> HIVE-12369.094.patch, HIVE-12369.095.patch, HIVE-12369.096.patch,
> HIVE-12369.097.patch, HIVE-12369.098.patch, HIVE-12369.099.patch,
> HIVE-12369.0991.patch, HIVE-12369.0992.patch, HIVE-12369.0993.patch,
> HIVE-12369.0994.patch, HIVE-12369.0995.patch, HIVE-12369.0996.patch,
> HIVE-12369.0998.patch
>
>
> Implement Native Vector GroupBy using fast hash table technology developed
> for Native Vector MapJoin, etc.
> Patch is currently limited to a single COUNT aggregation. Or, no aggregation
> also known as duplicate reduction.
> Here are examples of new kinds of classes introduces that stored the count in
> the slot table and don't allocate hash elements:
> {noformat}
> COUNT(column) VectorGroupByHashLongKeySingleCountColumnOperator
> COUNT(key) VectorGroupByHashLongKeySingleCountKeyOperator
> COUNT(*) VectorGroupByHashLongKeySingleCountStarOperator
> {noformat}
> And the duplicate reduction operator a single key. Example:
> {noformat}
> VectorGroupByHashLongKeyDuplicateReductionOperator
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-13157) MetaStoreEventListener.onAlter triggered for INSERT and SELECT
[ https://issues.apache.org/jira/browse/HIVE-13157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HIVE-13157: Affects Version/s: 3.1.3 > MetaStoreEventListener.onAlter triggered for INSERT and SELECT > -- > > Key: HIVE-13157 > URL: https://issues.apache.org/jira/browse/HIVE-13157 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 1.2.1, 3.1.3, 4.0.0 >Reporter: Eugen Stoianovici >Priority: Critical > Labels: obsolete? > > The event onAlter from > org.apache.hadoop.hive.metastore.MetaStoreEventListener is triggered when > INSERT or SELECT statements are executed on the target table. > Furthermore, the value of transient_lastDdl is updated in table properties > for INSERT statements. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-11247) hive升级遇到问题
[
https://issues.apache.org/jira/browse/HIVE-11247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena resolved HIVE-11247.
-
Fix Version/s: Not Applicable
Resolution: Not A Problem
> hive升级遇到问题
> --
>
> Key: HIVE-11247
> URL: https://issues.apache.org/jira/browse/HIVE-11247
> Project: Hive
> Issue Type: Bug
>Reporter: hongyan
>Assignee: hongyan
>Priority: Critical
> Fix For: Not Applicable
>
>
> 现在使用的hive版本0.12.0 hadoop版本是1.1.2 hive升级到1.2.1 之后select报错，我看官网说支持hadoop
> 1.x.y但是在网上查了下说 hive和hadoop版本不兼容问题，求解答
> {noformat}
> Exception in thread "main" java.lang.NoSuchMethodError:
> org.apache.hadoop.mapred.JobConf.unset(Ljava/lang/String;)V
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.pushFilters(HiveInputFormat.java:438)
> at
> org.apache.hadoop.hive.ql.exec.FetchTask.initialize(FetchTask.java:77)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:456)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:308)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1122)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1170)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:213)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-13370) Add test for HIVE-11470
[ https://issues.apache.org/jira/browse/HIVE-13370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HIVE-13370: Priority: Major (was: Blocker) > Add test for HIVE-11470 > --- > > Key: HIVE-13370 > URL: https://issues.apache.org/jira/browse/HIVE-13370 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.3 >Reporter: Sushanth Sowmyan >Assignee: Sushanth Sowmyan >Priority: Major > Attachments: HIVE-13370.patch > > > HIVE-11470 added capability to handle NULL dynamic partitioning keys > properly. However, it did not add a test for the case, we should have one so > we don't have future regressions of the same. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-14044) Newlines in Avro maps cause external table to return corrupt values
[
https://issues.apache.org/jira/browse/HIVE-14044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena updated HIVE-14044:
Priority: Major (was: Critical)
> Newlines in Avro maps cause external table to return corrupt values
> ---
>
> Key: HIVE-14044
> URL: https://issues.apache.org/jira/browse/HIVE-14044
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.2
> Environment: Hive version: 1.1.0-cdh5.5.1 (bundled with cloudera
> 5.5.1)
>Reporter: David Nies
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: test.json, test.schema
>
>
> When {{\n}} characters are contained in Avro files that are used as data
> bases for an external table, the result of {{SELECT}} queries may be corrupt.
> I encountered this error when querying hive both from {{beeline}} and from
> JDBC.
> h3. Steps to reproduce (used files are attached to ticket)
> # Create an {{.avro}} file that contains newline characters in a value of a
> map:
> {code}
> avro-tools fromjson --schema-file test.schema test.json > test.avro
> {code}
> # Copy {{.avro}} file to HDFS
> {code}
> hdfs dfs -copyFromLocal test.avro /some/location/
> {code}
> # Create an external table in beeline containing this {{.avro}}:
> {code}
> beeline> CREATE EXTERNAL TABLE broken_newline_map
> ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
> STORED AS
> INPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
> OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
> LOCATION '/some/location/'
> TBLPROPERTIES ('avro.schema.literal'='
> {
> "type" : "record",
> "name" : "myEntry",
> "namespace" : "myNamespace",
> "fields" : [ {
> "name" : "foo",
> "type" : "long"
> }, {
> "name" : "bar",
> "type" : {
> "type" : "map",
> "values" : "string"
> }
> } ]
> }
> ');
> {code}
> # Now, selecting may return corrupt results:
> {code}
> jdbc:hive2://my-server:1/> select * from broken_newline_map;
> +-+---+--+
> | broken_newline_map.foo | broken_newline_map.bar
> |
> +-+---+--+
> | 1 | {"key2":"value2","key1":"value1\nafter newline"}
> |
> | 2 | {"key2":"new value2","key1":"new value"}
> |
> +-+---+--+
> 2 rows selected (1.661 seconds)
> jdbc:hive2://my-server:1/> select foo, map_keys(bar), map_values(bar)
> from broken_newline_map;
> +---+--+-+--+
> | foo | _c1| _c2 |
> +---+--+-+--+
> | 1 | ["key2","key1"] | ["value2","value1"] |
> | NULL | NULL | NULL|
> | 2 | ["key2","key1"] | ["new value2","new value"] |
> +---+--+-+--+
> 3 rows selected (28.05 seconds)
> {code}
> Obviously, the last result set contains corrupt entries (line 2) and
> incorrect entries (line 1). I also encountered this when doing this query
> with JDBC.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-14668) TestMiniLlapCliDriver. hybridgrace_hashjoin_1 - the first attempt for a couple of tasks fails
[
https://issues.apache.org/jira/browse/HIVE-14668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena updated HIVE-14668:
Affects Version/s: 3.1.3
> TestMiniLlapCliDriver. hybridgrace_hashjoin_1 - the first attempt for a
> couple of tasks fails
> -
>
> Key: HIVE-14668
> URL: https://issues.apache.org/jira/browse/HIVE-14668
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.3
>Reporter: Siddharth Seth
>Priority: Critical
>
> https://issues.apache.org/jira/browse/HIVE-14651 tried changing the max
> attempted task attempts to 1 (i.e. fail fast if a task attempt fails).
> This caused TestMiniLlapCliDriver. hybridgrace_hashjoin_1.
> Turns out there's a first attempt which fails when running some queries in
> this q file. The same attempt succeeds on the next task attempt.
> {code}
> 2016-08-30T12:17:21,430 ERROR [TezTaskRunner
> (1472584599354_0001_43_01_00_0)] tez.MapRecordSource: Failed to close the
> reader; ignoring
> java.io.IOException: java.lang.AssertionError
> at
> org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.rethrowErrorIfAny(LlapInputFormat.java:383)
> ~[hive-llap-server-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.close(LlapInputFormat.java:374)
> ~[hive-llap-server-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.HiveRecordReader.doClose(HiveRecordReader.java:50)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.close(HiveContextAwareRecordReader.java:104)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.close(TezGroupedSplitsInputFormat.java:177)
> ~[tez-mapreduce-0.8.local.jar:0.8.local]
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.close(MRReaderMapred.java:99)
> ~[tez-mapreduce-0.8.local.jar:0.8.local]
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.closeReader(MapRecordSource.java:109)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:73)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:391)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:185)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> ~[tez-runtime-internals-0.8.local.jar:0.8.local]
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> ~[tez-runtime-internals-0.8.local.jar:0.8.local]
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> ~[tez-runtime-internals-0.8.local.jar:0.8.local]
> at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_60]
> at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_60]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> ~[hadoop-common-2.7.2.jar:?]
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> ~[tez-runtime-internals-0.8.local.jar:0.8.local]
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> ~[tez-runtime-internals-0.8.local.jar:0.8.local]
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> ~[tez-common-0.8.local.jar:0.8.local]
> at
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
> ~[hive-llap-server-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_60]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [?:1.8.0_60]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [?:1.8.0_60]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_60]
> Caused by: java.lang.AssertionError
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.performDataRead(OrcEncodedDataReader.java:398)
> ~[hive-llap-server-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:227)
> ~[hive-llap-server-2.2.0-SNAPSHOT.jar:2.2.
[jira] [Updated] (HIVE-13370) Add test for HIVE-11470
[ https://issues.apache.org/jira/browse/HIVE-13370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HIVE-13370: Affects Version/s: 3.1.3 > Add test for HIVE-11470 > --- > > Key: HIVE-13370 > URL: https://issues.apache.org/jira/browse/HIVE-13370 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.3 >Reporter: Sushanth Sowmyan >Assignee: Sushanth Sowmyan >Priority: Blocker > Attachments: HIVE-13370.patch > > > HIVE-11470 added capability to handle NULL dynamic partitioning keys > properly. However, it did not add a test for the case, we should have one so > we don't have future regressions of the same. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-14044) Newlines in Avro maps cause external table to return corrupt values
[
https://issues.apache.org/jira/browse/HIVE-14044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena updated HIVE-14044:
Affects Version/s: 1.2.2
> Newlines in Avro maps cause external table to return corrupt values
> ---
>
> Key: HIVE-14044
> URL: https://issues.apache.org/jira/browse/HIVE-14044
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.2
> Environment: Hive version: 1.1.0-cdh5.5.1 (bundled with cloudera
> 5.5.1)
>Reporter: David Nies
>Assignee: Sahil Takiar
>Priority: Critical
> Attachments: test.json, test.schema
>
>
> When {{\n}} characters are contained in Avro files that are used as data
> bases for an external table, the result of {{SELECT}} queries may be corrupt.
> I encountered this error when querying hive both from {{beeline}} and from
> JDBC.
> h3. Steps to reproduce (used files are attached to ticket)
> # Create an {{.avro}} file that contains newline characters in a value of a
> map:
> {code}
> avro-tools fromjson --schema-file test.schema test.json > test.avro
> {code}
> # Copy {{.avro}} file to HDFS
> {code}
> hdfs dfs -copyFromLocal test.avro /some/location/
> {code}
> # Create an external table in beeline containing this {{.avro}}:
> {code}
> beeline> CREATE EXTERNAL TABLE broken_newline_map
> ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
> STORED AS
> INPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
> OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
> LOCATION '/some/location/'
> TBLPROPERTIES ('avro.schema.literal'='
> {
> "type" : "record",
> "name" : "myEntry",
> "namespace" : "myNamespace",
> "fields" : [ {
> "name" : "foo",
> "type" : "long"
> }, {
> "name" : "bar",
> "type" : {
> "type" : "map",
> "values" : "string"
> }
> } ]
> }
> ');
> {code}
> # Now, selecting may return corrupt results:
> {code}
> jdbc:hive2://my-server:1/> select * from broken_newline_map;
> +-+---+--+
> | broken_newline_map.foo | broken_newline_map.bar
> |
> +-+---+--+
> | 1 | {"key2":"value2","key1":"value1\nafter newline"}
> |
> | 2 | {"key2":"new value2","key1":"new value"}
> |
> +-+---+--+
> 2 rows selected (1.661 seconds)
> jdbc:hive2://my-server:1/> select foo, map_keys(bar), map_values(bar)
> from broken_newline_map;
> +---+--+-+--+
> | foo | _c1| _c2 |
> +---+--+-+--+
> | 1 | ["key2","key1"] | ["value2","value1"] |
> | NULL | NULL | NULL|
> | 2 | ["key2","key1"] | ["new value2","new value"] |
> +---+--+-+--+
> 3 rows selected (28.05 seconds)
> {code}
> Obviously, the last result set contains corrupt entries (line 2) and
> incorrect entries (line 1). I also encountered this when doing this query
> with JDBC.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-13370) Add test for HIVE-11470
[ https://issues.apache.org/jira/browse/HIVE-13370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HIVE-13370: Priority: Minor (was: Major) > Add test for HIVE-11470 > --- > > Key: HIVE-13370 > URL: https://issues.apache.org/jira/browse/HIVE-13370 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.3 >Reporter: Sushanth Sowmyan >Assignee: Sushanth Sowmyan >Priority: Minor > Attachments: HIVE-13370.patch > > > HIVE-11470 added capability to handle NULL dynamic partitioning keys > properly. However, it did not add a test for the case, we should have one so > we don't have future regressions of the same. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-16583) Select query fails with some exception error
[ https://issues.apache.org/jira/browse/HIVE-16583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HIVE-16583. - Fix Version/s: Not Applicable Resolution: Not A Problem reach out to user ml if you need any help, AFAIK the use case works in hive code > Select query fails with some exception error > > > Key: HIVE-16583 > URL: https://issues.apache.org/jira/browse/HIVE-16583 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Atul >Priority: Critical > Fix For: Not Applicable > > > After loading the ORC file to Hive table, My select * query gets aborted with > below error: > Fetching results ran into the following error(s): > Bad status for request TFetchResultsReq(fetchType=0, > operationHandle=TOperationHandle(hasResultSet=True, modifiedRowCount=None, > operationType=0, > operationId=THandleIdentifier(secret='f\xfa\xf32\xd1KB\xbb\xb9\t\xe8\x1c\xd1\x01\xa5\xf2', > guid=')\x17\r4\x7f D\xcf\xa0Z\xff\x8a70\xaa\x93')), orientation=4, > maxRows=100): TFetchResultsResp(status=TStatus(errorCode=0, > errorMessage='java.io.IOException: > org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.ClassCastException', sqlState=None, > infoMessages=['*org.apache.hive.service.cli.HiveSQLException:java.io.IOException: > org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.ClassCastException:25:24', > 'org.apache.hive.service.cli.operation.SQLOperation:getNextRowSet:SQLOperation.java:352', > > 'org.apache.hive.service.cli.operation.OperationManager:getOperationNextRowSet:OperationManager.java:221', > > 'org.apache.hive.service.cli.session.HiveSessionImpl:fetchResults:HiveSessionImpl.java:707', > 'sun.reflect.GeneratedMethodAccessor45:invoke::-1', > 'sun.reflect.DelegatingMethodAccessorImpl:invoke:DelegatingMethodAccessorImpl.java:43', > 'java.lang.reflect.Method:invoke:Method.java:606', > 'org.apache.hive.service.cli.session.HiveSessionProxy:invoke:HiveSessionProxy.java:78', > > 'org.apache.hive.service.cli.session.HiveSessionProxy:access$000:HiveSessionProxy.java:36', > > 'org.apache.hive.service.cli.session.HiveSessionProxy$1:run:HiveSessionProxy.java:63', > 'java.security.AccessController:doPrivileged:AccessController.java:-2', > 'javax.security.auth.Subject:doAs:Subject.java:415', > 'org.apache.hadoop.security.UserGroupInformation:doAs:UserGroupInformation.java:1692', > > 'org.apache.hive.service.cli.session.HiveSessionProxy:invoke:HiveSessionProxy.java:59', > 'com.sun.proxy.$Proxy45:fetchResults::-1', > 'org.apache.hive.service.cli.CLIService:fetchResults:CLIService.java:454', > 'org.apache.hive.service.cli.thrift.ThriftCLIService:FetchResults:ThriftCLIService.java:672', > > 'org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults:getResult:TCLIService.java:1553', > > 'org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults:getResult:TCLIService.java:1538', > 'org.apache.thrift.ProcessFunction:process:ProcessFunction.java:39', > 'org.apache.thrift.TBaseProcessor:process:TBaseProcessor.java:39', > 'org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor:process:HadoopThriftAuthBridge.java:692', > > 'org.apache.thrift.server.TThreadPoolServer$WorkerProcess:run:TThreadPoolServer.java:285', > > 'java.util.concurrent.ThreadPoolExecutor:runWorker:ThreadPoolExecutor.java:1145', > > 'java.util.concurrent.ThreadPoolExecutor$Worker:run:ThreadPoolExecutor.java:615', > 'java.lang.Thread:run:Thread.java:745', > '*java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.ClassCastException:27:2', > 'org.apache.hadoop.hive.ql.exec.FetchTask:fetch:FetchTask.java:154', > 'org.apache.hadoop.hive.ql.Driver:getResults:Driver.java:1720', > 'org.apache.hive.service.cli.operation.SQLOperation:getNextRowSet:SQLOperation.java:347', > > '*org.apache.hadoop.hive.ql.metadata.HiveException:java.lang.ClassCastException:34:7', > > 'org.apache.hadoop.hive.ql.exec.ListSinkOperator:process:ListSinkOperator.java:93', > 'org.apache.hadoop.hive.ql.exec.Operator:forward:Operator.java:838', > 'org.apache.hadoop.hive.ql.exec.SelectOperator:process:SelectOperator.java:88', > 'org.apache.hadoop.hive.ql.exec.Operator:forward:Operator.java:838', > 'org.apache.hadoop.hive.ql.exec.TableScanOperator:process:TableScanOperator.java:97', > > 'org.apache.hadoop.hive.ql.exec.FetchOperator:pushRow:FetchOperator.java:425', > > 'org.apache.hadoop.hive.ql.exec.FetchOperator:pushRow:FetchOperator.java:417', > 'org.apache.hadoop.hive.ql.exec.FetchTask:fetch:FetchTask.java:140', > '*java.lang.ClassCastException:null:0:-1'], statusCode=3), results=None, > hasMoreRows=None) > From UNIX I am getting below error: > Failed with exception > java.io.IOException:org.apache.hadoop.h
[jira] [Updated] (HIVE-16999) Performance bottleneck in the ADD FILE/ARCHIVE commands for an HDFS resource
[
https://issues.apache.org/jira/browse/HIVE-16999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena updated HIVE-16999:
Issue Type: Improvement (was: Bug)
> Performance bottleneck in the ADD FILE/ARCHIVE commands for an HDFS resource
>
>
> Key: HIVE-16999
> URL: https://issues.apache.org/jira/browse/HIVE-16999
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Sailee Jain
>Assignee: Bing Li
>Priority: Critical
>
> Performance bottleneck is found in adding resource[which is lying on HDFS] to
> the distributed cache.
> Commands used are :-
> {code:java}
> 1. ADD ARCHIVE "hdfs://some_dir/archive.tar"
> 2. ADD FILE "hdfs://some_dir/file.txt"
> {code}
> Here is the log corresponding to the archive adding operation:-
> {noformat}
> converting to local hdfs://some_dir/archive.tar
> Added resources: [hdfs://some_dir/archive.tar
> {noformat}
> Hive is downloading the resource to the local filesystem [shown in log by
> "converting to local"].
> {color:#d04437}Ideally there is no need to bring the file to the local
> filesystem when this operation is all about copying the file from one
> location on HDFS to other location on HDFS[distributed cache].{color}
> This adds lot of performance bottleneck when the the resource is a big file
> and all commands need the same resource.
> After debugging around the impacted piece of code is found to be :-
> {code:java}
> public List add_resources(ResourceType t, Collection values,
> boolean convertToUnix)
> throws RuntimeException {
> Set resourceSet = resourceMaps.getResourceSet(t);
> Map> resourcePathMap =
> resourceMaps.getResourcePathMap(t);
> Map> reverseResourcePathMap =
> resourceMaps.getReverseResourcePathMap(t);
> List localized = new ArrayList();
> try {
> for (String value : values) {
> String key;
> {color:#d04437}//get the local path of downloaded jars{color}
> List downloadedURLs = resolveAndDownload(t, value,
> convertToUnix);
> ;
> .
> {code}
> {code:java}
> List resolveAndDownload(ResourceType t, String value, boolean
> convertToUnix) throws URISyntaxException,
> IOException {
> URI uri = createURI(value);
> if (getURLType(value).equals("file")) {
> return Arrays.asList(uri);
> } else if (getURLType(value).equals("ivy")) {
> return dependencyResolver.downloadDependencies(uri);
> } else { // goes here for HDFS
> return Arrays.asList(createURI(downloadResource(value,
> convertToUnix))); // Here when the resource is not local it will download it
> to the local machine.
> }
> }
> {code}
> Here, the function resolveAndDownload() always calls the downloadResource()
> api in case of external filesystem. It should take into consideration the
> fact that - when the resource is on same HDFS then bringing it on local
> machine is not a needed step and can be skipped for better performance.
> Thanks,
> Sailee
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-16766) Hive query with space as filter does not give proper result
[ https://issues.apache.org/jira/browse/HIVE-16766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HIVE-16766: Priority: Major (was: Critical) > Hive query with space as filter does not give proper result > --- > > Key: HIVE-16766 > URL: https://issues.apache.org/jira/browse/HIVE-16766 > Project: Hive > Issue Type: Bug >Affects Versions: 1.1.0 >Reporter: Subash >Assignee: Bing Li >Priority: Major > > Hi Team, > I have used the query as below format and it does not give proper results. > Since there is a split by \s+ in ExecuteStatementOperation class in line 48, > I feel something goes wrong there. Could help me with this, if i am wrong ? > I am using Hive JDBC version 1.1.0 > The sample query is as follows, > select count(1) as cnt from table where col1=" " and col2="D"; -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-16766) Hive query with space as filter does not give proper result
[ https://issues.apache.org/jira/browse/HIVE-16766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HIVE-16766: Affects Version/s: (was: 3.1.3) > Hive query with space as filter does not give proper result > --- > > Key: HIVE-16766 > URL: https://issues.apache.org/jira/browse/HIVE-16766 > Project: Hive > Issue Type: Bug >Affects Versions: 1.1.0 >Reporter: Subash >Assignee: Bing Li >Priority: Critical > > Hi Team, > I have used the query as below format and it does not give proper results. > Since there is a split by \s+ in ExecuteStatementOperation class in line 48, > I feel something goes wrong there. Could help me with this, if i am wrong ? > I am using Hive JDBC version 1.1.0 > The sample query is as follows, > select count(1) as cnt from table where col1=" " and col2="D"; -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-16766) Hive query with space as filter does not give proper result
[ https://issues.apache.org/jira/browse/HIVE-16766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HIVE-16766: Affects Version/s: 3.1.3 1.1.0 > Hive query with space as filter does not give proper result > --- > > Key: HIVE-16766 > URL: https://issues.apache.org/jira/browse/HIVE-16766 > Project: Hive > Issue Type: Bug >Affects Versions: 1.1.0, 3.1.3 >Reporter: Subash >Assignee: Bing Li >Priority: Critical > > Hi Team, > I have used the query as below format and it does not give proper results. > Since there is a split by \s+ in ExecuteStatementOperation class in line 48, > I feel something goes wrong there. Could help me with this, if i am wrong ? > I am using Hive JDBC version 1.1.0 > The sample query is as follows, > select count(1) as cnt from table where col1=" " and col2="D"; -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-18761) FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:java.lang.RuntimeException: java.lang.NullPointerException
[ https://issues.apache.org/jira/browse/HIVE-18761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HIVE-18761. - Fix Version/s: Not Applicable Resolution: Cannot Reproduce the error is from hbase, if you can reproduce the issue, add the details and reopen if it is hive specific > FAILED: Execution Error, return code 1 from > org.apache.hadoop.hive.ql.exec.DDLTask. > MetaException(message:java.lang.RuntimeException: > java.lang.NullPointerException > > > Key: HIVE-18761 > URL: https://issues.apache.org/jira/browse/HIVE-18761 > Project: Hive > Issue Type: Bug >Reporter: chandhirakumar >Priority: Critical > Fix For: Not Applicable > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-18325) Config to do case unaware schema evolution to ORC reader.
[ https://issues.apache.org/jira/browse/HIVE-18325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HIVE-18325: Issue Type: Improvement (was: Bug) > Config to do case unaware schema evolution to ORC reader. > - > > Key: HIVE-18325 > URL: https://issues.apache.org/jira/browse/HIVE-18325 > Project: Hive > Issue Type: Improvement > Components: ORC >Reporter: piyush mukati >Priority: Critical > > in case of orc data reader schema passed by hive are all small cases and if > the column name stored in the file has any uppercase, it will return null > values for those columns even if the data is present in the file. > Column name matching while schema evolution should be case unaware. > we need to pass config for same from hive. the > config(orc.schema.evolution.case.sensitive) in orc will be exposed by > https://issues.apache.org/jira/browse/ORC-264 > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-18412) FileSinkOperator thows NullPointerException
[
https://issues.apache.org/jira/browse/HIVE-18412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena updated HIVE-18412:
Affects Version/s: 3.1.3
1.2.2
> FileSinkOperator thows NullPointerException
>
>
> Key: HIVE-18412
> URL: https://issues.apache.org/jira/browse/HIVE-18412
> Project: Hive
> Issue Type: Bug
> Components: Hive, Transactions
>Affects Versions: 1.2.2, 3.1.3
> Environment: HDP2.6.1, Hive 1.2.1
>Reporter: Benjamin BONNET
>Priority: Blocker
>
> Hi,
> while executing a query (DELETE with a join) on an ACID table, I get a
> NullPointerException in reducer.
> See stack trace below.
> According to FileSinkOperator source code, it seems that buckepMap transient
> field is Null.
> In my opinion, the only circumstance in which this field may be null is when
> the involved FileSinkOperator has been serialized and then deserialized.
> Actually, deserialization lets that transient reference uninitialized.
> I checked source code for more recent versions (including Hive 2.x) but
> everywhere that field may remain uninitialized (if FileSinkOperator is
> serialized/deserialized). So I think that issue may concern any version of
> Hive.
> {noformat}
> ERROR : Vertex failed, vertexName=Reducer 3,
> vertexId=vertex_1513704146031_77754_2_05, diagnostics=[Task failed,
> taskId=task_1513704146031_77754_2_05_00, diagnostics=[TaskAttempt 0
> failed, info=[Error: Failure while running task:java
> .lang.RuntimeException: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row (tag=0)
> {"key":{"reducesinkkey0":{"transactionid":108117,"bucketid":0,"rowid":1114}},"value":{"
> _col0":"2017","_col1":"10"}}
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:173)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:139)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:347)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:194)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:185)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:185)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:181)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row (tag=0)
> {"key":{"reducesinkkey0":{"transactionid":108117,"bucketid":0,"rowid":1114}},"value":{"_col0":"2017"
> ,"_col1":"10"}}
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:284)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:266)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:150)
> ... 14 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row (tag=0)
> {"key":{"reducesinkkey0":{"transactionid":108117,"bucketid":0,"rowid":1114}},"value":{"_col0":"2017","_col1":"10"}}
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:352)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:274)
> ... 16 more
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.findWriterOffset(FileSinkOperator.java:830)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:758)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:841)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:88)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:343)
> ...
[jira] [Updated] (HIVE-16669) Fine tune Compaction to take advantage of Acid 2.0
[ https://issues.apache.org/jira/browse/HIVE-16669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HIVE-16669: Affects Version/s: 3.1.3 > Fine tune Compaction to take advantage of Acid 2.0 > -- > > Key: HIVE-16669 > URL: https://issues.apache.org/jira/browse/HIVE-16669 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.1.3 >Reporter: Eugene Koifman >Priority: Critical > Attachments: HIVE-16669.wip.patch > > > * There is little point using 2.0 vectorized reader since there is no > operator pipeline in compaction > * If minor compaction just concats delete_delta files together, then the 2 > stage compaction should always ensure that we have a limited number of Orc > readers to do the merging and current OrcRawRecordMerger should be fine > * ... -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-19811) the hive server2 web ui is suddenly down,can't open master:100002
[ https://issues.apache.org/jira/browse/HIVE-19811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HIVE-19811: Affects Version/s: 3.1.3 > the hive server2 web ui is suddenly down,can't open master:12 > - > > Key: HIVE-19811 > URL: https://issues.apache.org/jira/browse/HIVE-19811 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.3 > Environment: CDH5 >Reporter: joeyzhang >Priority: Critical > > this is I found in the log about the problem. > > 2018-06-05 20:34:35,622 WARN org.eclipse.jetty.servlet.ServletHandler: > [hiveserver2-web-16298]: /hiveserver2.jsp > java.lang.NullPointerException > at > org.apache.hive.generated.hiveserver2.hiveserver2_jsp._jspService(hiveserver2_jsp.java:145) > at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:98) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) > at > org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:565) > at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:479) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119) > at > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:521) > at > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227) > at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031) > at > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:406) > at > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:186) > at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:965) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117) > at > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:250) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111) > at org.eclipse.jetty.server.Server.handle(Server.java:349) > at > org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:449) > at > org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:910) > at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:634) > at > org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:230) > at > org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:76) > at > org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:609) > at > org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:45) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599) > at > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534) > at java.lang.Thread.run(Thread.java:745) > == > after I restart the hive server,the hive server2 web ui is ok again.but I > don't know what's wrong about the server,I think maybe there is somethin > wrong with jetty，but I don't know much about it.Please help me , thank you. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-20939) Branch 3.1 is unstable
[
https://issues.apache.org/jira/browse/HIVE-20939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena updated HIVE-20939:
Affects Version/s: 3.1.3
> Branch 3.1 is unstable
> --
>
> Key: HIVE-20939
> URL: https://issues.apache.org/jira/browse/HIVE-20939
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.3
>Reporter: Igor Kryvenko
>Assignee: Igor Kryvenko
>Priority: Blocker
> Attachments: HIVE-20939-branch-3.1.patch,
> HIVE-20939.1-branch-3.1.patch
>
>
> The latest Travis build from {{branch-3.1}} is failed.
> Seems like just missed out change the dependencies version of
> {{standalone-metastore}} and {{upgrade-acid}} modules from {{3.1.1}} to
> {{3.1.2-SNAPSHOT}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-21190) hive thrift server may be blocked by session level waiting,caused by udf!
[
https://issues.apache.org/jira/browse/HIVE-21190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena updated HIVE-21190:
Affects Version/s: 3.1.3
1.2.2
> hive thrift server may be blocked by session level waiting,caused by udf!
> -
>
> Key: HIVE-21190
> URL: https://issues.apache.org/jira/browse/HIVE-21190
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 1.2.2, 3.1.3
> Environment: hdp ambari 26
> hive1.2.0
>Reporter: kongxianghe
>Priority: Critical
> Attachments: session_1.jpg, session_2.jpg
>
>
> # caused by an error UDF function！time_waiting(Long sleepSeconds)
> {code}
> public class UDFTimeWaiting extends UDF throws Exception{
> public String evaluate(String sleepSeconds){
> ...
> Thread.sleep(Long.parseLong(sleepSeconds) * 1000);
> return "ok";
> }
> }
> {code}
>
> # in session_1:
> {code}select time_waiting(100);{code}
> # in session_2:
> {code}select 1; or show tables;{code}
> # session_2 will not have any response from thrift server until session_1
> waiting 100 seconds!
> this bug may cause hiveserver come into an unavailable status!
>
> # session_1 run waiting 200s,
> !session_1.jpg!
> # session_2 run at the same time ,but blocked by session_1 , see the
> pic,waiting 197s after session_1 returned then returned
> !session_2.jpg!
>
> # if someone want to attack or do sth ,hiveserver will not be down,but not
> available again!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HIVE-21644) ptests are timing out regularly
[
https://issues.apache.org/jira/browse/HIVE-21644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena resolved HIVE-21644.
-
Fix Version/s: Not Applicable
Resolution: Not A Problem
> ptests are timing out regularly
> ---
>
> Key: HIVE-21644
> URL: https://issues.apache.org/jira/browse/HIVE-21644
> Project: Hive
> Issue Type: Bug
> Components: Test
>Reporter: Jesus Camacho Rodriguez
>Priority: Critical
> Fix For: Not Applicable
>
>
> Over the last few days, I see the same tests timing out in different JIRAs.
> {noformat}
> TestCachedStore - did not produce a TEST-*.xml file (likely timed out)
> (batchId=231)
> TestCatalogCaching - did not produce a TEST-*.xml file (likely timed out)
> (batchId=231)
> TestDataSourceProviderFactory - did not produce a TEST-*.xml file (likely
> timed out) (batchId=231)
> TestDeadline - did not produce a TEST-*.xml file (likely timed out)
> (batchId=231)
> TestHiveMetaStoreGetMetaConf - did not produce a TEST-*.xml file (likely
> timed out) (batchId=231)
> TestHiveMetaStoreTimeout - did not produce a TEST-*.xml file (likely timed
> out) (batchId=231)
> TestHiveMetaStoreTxns - did not produce a TEST-*.xml file (likely timed out)
> (batchId=231)
> TestMarkPartition - did not produce a TEST-*.xml file (likely timed out)
> (batchId=231)
> TestMarkPartitionRemote - did not produce a TEST-*.xml file (likely timed
> out) (batchId=231)
> TestMetaStoreEventListenerOnlyOnCommit - did not produce a TEST-*.xml file
> (likely timed out) (batchId=231)
> TestMetaStoreEventListenerWithOldConf - did not produce a TEST-*.xml file
> (likely timed out) (batchId=231)
> TestMetaStoreInitListener - did not produce a TEST-*.xml file (likely timed
> out) (batchId=231)
> TestMetaStoreListenersError - did not produce a TEST-*.xml file (likely timed
> out) (batchId=231)
> TestMetaStoreSchemaInfo - did not produce a TEST-*.xml file (likely timed
> out) (batchId=231)
> TestObjectStore - did not produce a TEST-*.xml file (likely timed out)
> (batchId=231)
> TestObjectStoreSchemaMethods - did not produce a TEST-*.xml file (likely
> timed out) (batchId=231)
> TestRemoteHiveMetaStoreZKBindHost - did not produce a TEST-*.xml file (likely
> timed out) (batchId=231)
> TestTableIterable - did not produce a TEST-*.xml file (likely timed out)
> (batchId=231)
> {noformat}
> These are the tests comprised in that batch:
> {code}
> 2019-04-23 17:08:04,309 DEBUG [TestExecutor] ExecutionPhase.execute:98
> PBatch: UnitTestBatch
> [name=231_UTBatch_standalone-metastore__metastore-server_20_tests, id=231,
> moduleName=standalone-metastore/metastore-server, batchSize=20,
> isParallel=true, testList=[TestHiveMetaStoreSchemaMethods, TestOldSchema,
> TestCachedStore, TestCatalogCaching, TestDeadline,
> TestMetaStoreListenersError, TestMetaStoreEventListenerOnlyOnCommit,
> TestMetaStoreSchemaInfo, TestMarkPartition, TestMetaStoreInitListener,
> TestHiveMetaStoreGetMetaConf, TestHiveMetaStoreTxns, TestTableIterable,
> TestMarkPartitionRemote, TestMetaStoreEventListenerWithOldConf,
> TestObjectStoreSchemaMethods, TestHiveMetaStoreTimeout,
> TestRemoteHiveMetaStoreZKBindHost, TestObjectStore,
> TestDataSourceProviderFactory]]
> {code}
> We need to fix this, currently it is rather difficult to push any code to
> master.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-22334) There is a permission problem with the add jar package on the command line
[ https://issues.apache.org/jira/browse/HIVE-22334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HIVE-22334: Affects Version/s: 3.1.3 > There is a permission problem with the add jar package on the command line > -- > > Key: HIVE-22334 > URL: https://issues.apache.org/jira/browse/HIVE-22334 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.3 > Environment: hive 3.1 >Reporter: jiazhaoquan >Assignee: weizheng >Priority: Blocker > Attachments: 启动失败文档.jpg > > > The environment is ve3.0;Now add a jar package has the problem of removing > permissions, as follows; > >add jar > >/export/servers/hive-connector-0.9.0-SNAPSHOT-jar-with-dependencies.jar; > >Error: Error while processing statement: Permission denied: Principal > >[name=root, type=USER] does not have following privileges for operation ADD > >[ADMIN] (state=,code=1)` > > I have used Storage Based Authorization in the Metastore Server according to > the configuration;But the add jar package still has permissions issues;Put my > jar package into the lib file, and my hive started until it finally appeared > (closed on the command line).The problem log is in the attachment. > > 环境是hive3.0；现在add一个jar包出现去权限问题，如下； > >add jar > >/export/servers/hive-connector-0.9.0-SNAPSHOT-jar-with-dependencies.jar; > >Error: Error while processing statement: Permission denied: Principal > >[name=root, type=USER] does not have following privileges for operation ADD > >[ADMIN] (state=,code=1)` > 我已经根据配置使用了Storage Based Authorization in the Metastore Server；但是add > jar包还是有权限问题；将我的jar包放入lib文件，我的hive启动到最后出现（命令行出现closed），启动不了hive；问题在附件中； -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-22932) Unable to kill Beeline with Ctrl+C
[ https://issues.apache.org/jira/browse/HIVE-22932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HIVE-22932: Priority: Major (was: Blocker) > Unable to kill Beeline with Ctrl+C > -- > > Key: HIVE-22932 > URL: https://issues.apache.org/jira/browse/HIVE-22932 > Project: Hive > Issue Type: Bug >Reporter: Renukaprasad C >Priority: Major > > Stopped the server and tried to stop the beeline console with "Ctrl+C". But > it unable to kill the process & gets process gets hanged. > Read call got blocked. > Attached the thread dump. > 0: jdbc:hive2://localhost:1> show tables; > Unknown HS2 problem when communicating with Thrift server. > Error: org.apache.thrift.transport.TTransportException: > java.net.SocketException: Broken pipe (state=08S01,code=0) > 0: jdbc:hive2://localhost:1> Interrupting... Please be patient this may > take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > Interrupting... Please be patient this may take some time. > 2020-02-26 17:40:42 > Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.72-b15 mixed mode): > "NonBlockingInputStreamThread" #16 daemon prio=5 os_prio=0 > tid=0x7f0318c10800 nid=0x258c in Object.wait() [0x7f031c193000] >java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0xfe9113c0> (a > jline.internal.NonBlockingInputStream) > at > jline.internal.NonBlockingInputStream.run(NonBlockingInputStream.java:278) > - locked <0xfe9113c0> (a > jline.internal.NonBlockingInputStream) > at java.lang.Thread.run(Thread.java:745) > "Service Thread" #11 daemon prio=9 os_prio=0 tid=0x7f032006c000 > nid=0x257b runnable [0x] >java.lang.Thread.State: RUNNABLE > "C1 CompilerThread3" #10 daemon prio=9 os_prio=0 tid=0x7f0320060800 > nid=0x257a waiting on condition [0x] >java.lang.Thread.State: RUNNABLE > "C2 CompilerThread2" #9 daemon prio=9 os_prio=0 tid=0x7f0320056000 > nid=0x2579 waiting on condition [0x] >java.lang.Thread.State: RUNNABLE > "C2 CompilerThread1" #8 daemon prio=9 os_prio=0 tid=0x7f0320054000 > nid=0x2578 waiting on condition [0x] >java.lang.Thread.State: RUNNABLE > "C2 CompilerThread0" #7 daemon prio=9 os_prio=0 tid=0x7f0320051000 > nid=0x2577 waiting on condition [0x] >java.lang.Thread.State: RUNNABLE > "JDWP Event Helper Thread" #6 daemon prio=10 os_prio=0 tid=0x7f032004f000 > nid=0x2576 runnable [0x] >java.lang.Thread.State: RUNNABLE > "JDWP Transport Listener: dt_socket" #5 daemon prio=10 os_prio=0 > tid=0x7f032004b800 nid=0x2575 runnable [0x] >java.lang.Thread.State: RUNNABLE > "Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x7f0320035800 > nid=0x2574 waiting on condition [0x] >java.lang.Thread.State: RUNNABLE > "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x7f0320003800 nid=0x2572 in > Object.wait() [0x7f0324b1c000] >java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0xfe930770> (a > java.lang.ref.ReferenceQueue$Lock) > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143) > - locked <0xfe930770> (a java.lang.ref.ReferenceQueue$Lock) > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164) > at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209) > "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x006a6000 > nid=0x2571 in Object.wait() [0x7f0324c1d000] >java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0xfe9307b0> (a java.lang.ref.Reference$Lock) >
[jira] [Resolved] (HIVE-23156) NPE if -f is used with HCatCLI
[ https://issues.apache.org/jira/browse/HIVE-23156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HIVE-23156. - Fix Version/s: Not Applicable Resolution: Duplicate Fixed as part of HIVE-24782, resolving as dupe > NPE if -f is used with HCatCLI > -- > > Key: HIVE-23156 > URL: https://issues.apache.org/jira/browse/HIVE-23156 > Project: Hive > Issue Type: Bug > Components: HCatalog >Reporter: Ivan Suller >Priority: Critical > Fix For: Not Applicable > > > After HIVE-22889 if there is no -e cli parameter then a NPE is thrown. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-23360) Unable to create Connection using the Ambari default PORT (working fine if Port is taken as 10000)
[ https://issues.apache.org/jira/browse/HIVE-23360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HIVE-23360: Affects Version/s: 3.1.3 > Unable to create Connection using the Ambari default PORT (working fine if > Port is taken as 1) > -- > > Key: HIVE-23360 > URL: https://issues.apache.org/jira/browse/HIVE-23360 > Project: Hive > Issue Type: Bug > Components: Hive, HiveServer2, JDBC >Affects Versions: 3.1.3 > Environment: # hive version 3.1.0.3.0.1.0-187 > * success -- hive jdbc used 3.0.0 -- > * success -- hive jdbc used 2.0.0 -- > * success -- hive jdbc used 2.1.0 -- > * success -- hive jdbc used 2.2.0 -- > * success -- hive jdbc used 2.1.1 -- > * connection.setUrl( > * > "jdbc:hive2://xen188-hdp30-centos7:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2" > # hive version 1.2.1000.2.6.0.3-8 > * failed -- hive jdbc used 3.0.0 -- > * success -- hive jdbc used 2.0.0 -- > * success -- hive jdbc used 2.1.0 -- > * failed -- hive jdbc used 2.2.0 -- > * failed -- hive jdbc used 2.1.1 -- > * connection.setUrl( > * > "jdbc:hive2://rack167-hdp26-dev:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2" > * ); >Reporter: Sukhpreet Kaur >Priority: Critical > Attachments: HiveConnection.java, MainClass.java, build.gradle > > > Unable to create a connection using the Ambari JDBC URL (port 2181). Working > fine with the default port 1. > Please look into the environment details (found after doing a lot of > hit-and-trial) and attached files to further investigate the issue. > Using a higher hive-jdbc version results in conflicts while deploying the > project. > Our R&D concluded that the issue mainly lies with the zookeeper jar. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-23502) 【hive on spark】 return code 3 from org.apache.hadoop.hive.ql.exec.spark.SparkTask
[ https://issues.apache.org/jira/browse/HIVE-23502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HIVE-23502: Affects Version/s: 3.1.3 > 【hive on spark】 return code 3 from > org.apache.hadoop.hive.ql.exec.spark.SparkTask > - > > Key: HIVE-23502 > URL: https://issues.apache.org/jira/browse/HIVE-23502 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.3 > Environment: hadoop 2.7.2 hive 1.2.1 sclala 2.9.x spark 1.3.1 >Reporter: tom >Priority: Blocker > > Spark UI Log: > > 20/05/19 17:07:11 INFO exec.Utilities: No plan file found: > hdfs://mycluster/tmp/hive/root/a3b20597-61d1-47a9-86b1-dde289fded78/hive_2020-05-19_17-06-53_394_4024151029162597012-1/-mr-10003/c586ae6a-eefb-49fd-92b6-7593e57f0a93/map.xml > 20/05/19 17:07:11 ERROR executor.Executor: Exception in task 0.0 in stage 0.0 > (TID 0) > java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:255) > at > org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:437) > at > org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:430) > at > org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:587) > at org.apache.spark.rdd.HadoopRDD$$anon$1.(HadoopRDD.scala:236) > at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:212) > at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:101) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:244) > at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:244) > at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68) > at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) > at org.apache.spark.scheduler.Task.run(Task.scala:64) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > 20/05/19 17:07:11 INFO executor.CoarseGrainedExecutorBackend: Got assigned > task 1 > 20/05/19 17:07:11 INFO executor.Executor: Running task 0.1 in stage 0.0 (TID > 1) > 20/05/19 17:07:11 INFO rdd.HadoopRDD: Input split: > Paths:/user/hive/warehouse/orginfobig_fq/nd=2014/frcode=410503/fqdate=2014-01-01/part-m-0:0+100InputFormatClass: > org.apache.hadoop.mapred.TextInputFormat > 20/05/19 17:07:11 INFO exec.Utilities: No plan file found: > hdfs://mycluster/tmp/hive/root/a3b20597-61d1-47a9-86b1-dde289fded78/hive_2020-05-19_17-06-53_394_4024151029162597012-1/-mr-10003/c586ae6a-eefb-49fd-92b6-7593e57f0a93/map.xml > 20/05/19 17:07:11 ERROR executor.Executor: Exception in task 0.1 in stage 0.0 > (TID 1) > java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:255) > at > org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:437) > at > org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:430) > at > org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:587) > at org.apache.spark.rdd.HadoopRDD$$anon$1.(HadoopRDD.scala:236) > at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:212) > at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:101) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:244) > at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:244) > at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68) > at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) > at org.apache.spark.scheduler.Task.run(Task.scala:64) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > 20/05/19 17:19:19 INFO storage.BlockManager: Removing broadcast 1 > 20/05/19 17:19:19 INFO storage.BlockManager: Removing block broadcast_1 > 20/05/19 17:19:19 INFO storage.MemoryStore: Block broadcast_1 of size 189144 > dropped from
[jira] [Updated] (HIVE-24819) CombineHiveInputFormat format seems to be returning row count in the multiple of Maps
[ https://issues.apache.org/jira/browse/HIVE-24819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HIVE-24819: Priority: Major (was: Critical) > CombineHiveInputFormat format seems to be returning row count in the multiple > of Maps > -- > > Key: HIVE-24819 > URL: https://issues.apache.org/jira/browse/HIVE-24819 > Project: Hive > Issue Type: Bug > Environment: Apache Hive (version 3.1.0.3.1.0.0-78) > Driver: Hive JDBC (version 3.1.0.3.1.0.0-78) > Transaction isolation: TRANSACTION_REPEATABLE_READ > Beeline version 3.1.0.3.1.0.0-78 by Apache Hive >Reporter: Jitender Kumar >Priority: Major > > Hi Team, > This is the first time I am writing a bug using apache Jira, so pardon me if > I am unintentionally breaking any protocols. > I am facing the following issue (on a multi-node cluster) when I set > hive.tez.input.format to > org.apache.hadoop.hive.ql.io.CombineHiveInputFormat. > Just for demonstration purposes, I will be executing the following query for > multiple cases. > _select count(1) from dbname.personal_data_rc tablesample(1000 rows);_ > *Case1* > mapred.map.tasks=2 > hive.tez.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat > *Output* > 1000 > *Case 2* > mapred.map.tasks=2 > hive.tez.input.format=org.apache.hadoop.hive.ql.io.CombineHiveInputFormat > *Output* > 2000 > *Case 3* > mapred.map.tasks=3 > hive.tez.input.format=org.apache.hadoop.hive.ql.io.CombineHiveInputFormat > *Output* > 3000 > After 3 maps set as default, out remains same, i.e multiple of 3. > Can you help me understand why if I have TABLESAMPLE set to 1000 rows, it is > giving me more number of rows? Is there any other property that must be used > with CombineHiveInputFormat or is it an issue with CombineHiveInputFormat > only? > I have tried to look for a solution but in the end i had to come here. Please > share your inputs ASAP as one of our client is looking for a solution or > explaination regarding this? > For now as a workaround we have changed it to following. > *hive.tez.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat* > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-25435) build fail from master branch
[ https://issues.apache.org/jira/browse/HIVE-25435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HIVE-25435. - Fix Version/s: Not Applicable Resolution: Cannot Reproduce Build is working fine, we have daily builds and each PR runs it, reach out to the dev mailing list for help > build fail from master branch > - > > Key: HIVE-25435 > URL: https://issues.apache.org/jira/browse/HIVE-25435 > Project: Hive > Issue Type: Bug > Environment: operator system: unbuntu for windows10 wsl > mvn: 3.2.5 version > java: 1.8 >Reporter: yandufeng >Priority: Critical > Fix For: Not Applicable > > > when i use mvn clean install -DskipTests for build hive in master > branch,following error appear: > [ERROR] Failed to execute goal > org.apache.maven.plugins:maven-compiler-plugin:3.8.1:compile > (default-compile) on project hive-kryo-registrator: Compilation failure: > Compilation failure: > [ERROR] > /mnt/d/programs/github/hive/kryo-registrator/src/main/java/org/apache/hive/spark/HiveKryoRegistrator.java:[41,18] > org.apache.hive.spark.HiveKryoRegistrator.HiveKeySerializer is not abstract > and does not override abstract method > read(com.esotericsoftware.kryo.Kryo,com.esotericsoftware.kryo.io.Input,java.lang.Class) > in com.esotericsoftware.kryo.Serializer > [ERROR] > /mnt/d/programs/github/hive/kryo-registrator/src/main/java/org/apache/hive/spark/HiveKryoRegistrator.java:[49,20] > name clash: > read(com.esotericsoftware.kryo.Kryo,com.esotericsoftware.kryo.io.Input,java.lang.Class extends org.apache.hadoop.hive.ql.io.HiveKey>) in > org.apache.hive.spark.HiveKryoRegistrator.HiveKeySerializer and > read(com.esotericsoftware.kryo.Kryo,com.esotericsoftware.kryo.io.Input,java.lang.Class) > in com.esotericsoftware.kryo.Serializer have the same erasure, yet neither > overrides the other > [ERROR] > /mnt/d/programs/github/hive/kryo-registrator/src/main/java/org/apache/hive/spark/HiveKryoRegistrator.java:[57,10] > org.apache.hive.spark.HiveKryoRegistrator.BytesWritableSerializer is not > abstract and does not override abstract method > read(com.esotericsoftware.kryo.Kryo,com.esotericsoftware.kryo.io.Input,java.lang.Class) > in com.esotericsoftware.kryo.Serializer > [ERROR] > /mnt/d/programs/github/hive/kryo-registrator/src/main/java/org/apache/hive/spark/HiveKryoRegistrator.java:[64,26] > name clash: > read(com.esotericsoftware.kryo.Kryo,com.esotericsoftware.kryo.io.Input,java.lang.Class extends org.apache.hadoop.io.BytesWritable>) in > org.apache.hive.spark.HiveKryoRegistrator.BytesWritableSerializer and > read(com.esotericsoftware.kryo.Kryo,com.esotericsoftware.kryo.io.Input,java.lang.Class) > in com.esotericsoftware.kryo.Serializer have the same erasure, yet neither > overrides the other > [ERROR] > /mnt/d/programs/github/hive/kryo-registrator/src/main/java/org/apache/hive/spark/NoHashCodeKryoSerializer.java:[51,18] > org.apache.hive.spark.NoHashCodeKryoSerializer.HiveKeySerializer is not > abstract and does not override abstract method > read(com.esotericsoftware.kryo.Kryo,com.esotericsoftware.kryo.io.Input,java.lang.Class) > in com.esotericsoftware.kryo.Serializer > [ERROR] /mnt/d/programs/github/hive/kryo- > registrator/src/main/java/org/apache/hive/spark/NoHashCodeKryoSerializer.java:[58,20] > name clash: > read(com.esotericsoftware.kryo.Kryo,com.esotericsoftware.kryo.io.Input,java.lang.Class extends org.apache.hadoop.hive.ql.io.HiveKey>) in > org.apache.hive.spark.NoHashCodeKryoSerializer.HiveKeySerializer and > read(com.esotericsoftware.kryo.Kryo,com.esotericsoftware.kryo.io.Input,java.lang.Class) > in com.esotericsoftware.kryo.Serializer have the same erasure, yet neither > overrides the other > [ERROR] -> [Help 1] > [ERROR] > [ERROR] To see the full stack trace of the errors, re-run Maven with the -e > switch. > [ERROR] Re-run Maven using the -X switch to enable full debug logging. > [ERROR] > [ERROR] For more information about the errors and possible solutions, please > read the following articles: > [ERROR] [Help 1] > http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException > [ERROR] > [ERROR] After correcting the problems, you can resume the build with the > command > [ERROR] mvn -rf :hive-kryo-registrator -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-25578) Tests are failing because operators can't be closed
[ https://issues.apache.org/jira/browse/HIVE-25578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HIVE-25578: Labels: Test pull-request-available (was: pull-request-available) > Tests are failing because operators can't be closed > --- > > Key: HIVE-25578 > URL: https://issues.apache.org/jira/browse/HIVE-25578 > Project: Hive > Issue Type: Bug >Reporter: Karen Coppage >Priority: Major > Labels: Test, pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > The following qtests are failing consistently > ([example|http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-2667/6/tests/]) > on the master branch: > * TestMiniLlapCliDriver > ([http://ci.hive.apache.org/job/hive-flaky-check/420/]) > ** newline > ** groupby_bigdata > ** input20 > ** input33 > ** rcfile_bigdata > ** remote_script > * TestContribCliDriver > ([http://ci.hive.apache.org/job/hive-flaky-check/421/]) > ** serde_typedbytes5 > The failure reason for all seems to be that operators can't be closed. Not > 100% sure that TestContribCliDriver#serde_typedbytes5 failure is related to > the others – the issue seems to be the same, the error message is a bit > different. > I'm about to disable these as they are blocking all work. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-25578) Tests are failing because operators can't be closed
[ https://issues.apache.org/jira/browse/HIVE-25578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HIVE-25578: Priority: Major (was: Critical) > Tests are failing because operators can't be closed > --- > > Key: HIVE-25578 > URL: https://issues.apache.org/jira/browse/HIVE-25578 > Project: Hive > Issue Type: Bug >Reporter: Karen Coppage >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > The following qtests are failing consistently > ([example|http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-2667/6/tests/]) > on the master branch: > * TestMiniLlapCliDriver > ([http://ci.hive.apache.org/job/hive-flaky-check/420/]) > ** newline > ** groupby_bigdata > ** input20 > ** input33 > ** rcfile_bigdata > ** remote_script > * TestContribCliDriver > ([http://ci.hive.apache.org/job/hive-flaky-check/421/]) > ** serde_typedbytes5 > The failure reason for all seems to be that operators can't be closed. Not > 100% sure that TestContribCliDriver#serde_typedbytes5 failure is related to > the others – the issue seems to be the same, the error message is a bit > different. > I'm about to disable these as they are blocking all work. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-25617) fix CVE-2020-8908
[ https://issues.apache.org/jira/browse/HIVE-25617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17799551#comment-17799551 ] Ayush Saxena commented on HIVE-25617: - Not applicable to master/4.0 release, Hive On Spark no longer exists there, reducing priority and added the affect version > fix CVE-2020-8908 > - > > Key: HIVE-25617 > URL: https://issues.apache.org/jira/browse/HIVE-25617 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.3 >Reporter: lujie >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > see [https://www.cvedetails.com/cve/CVE-2020-8908/] > A temp directory creation vulnerability exists in all versions of Guava, > allowing an attacker with access to the machine to potentially access data in > a temporary directory created by the Guava API > com.google.common.io.Files.createTempDir(). By default, on unix-like systems, > the created directory is world-readable (readable by an attacker with access > to the system). The method in question has been marked > [@deprecated|https://github.com/deprecated] in versions 30.0 and later and > should not be used. For Android developers, we recommend choosing a temporary > directory API provided by Android, such as context.getCacheDir(). For other > Java developers, we recommend migrating to the Java 7 API > java.nio.file.Files.createTempDirectory() which explicitly configures > permissions of 700, or configuring the Java runtime's java.io.tmpdir system > property to point to a location whose permissions are appropriately > configured. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-25617) fix CVE-2020-8908
[ https://issues.apache.org/jira/browse/HIVE-25617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HIVE-25617: Affects Version/s: 3.2.0 > fix CVE-2020-8908 > - > > Key: HIVE-25617 > URL: https://issues.apache.org/jira/browse/HIVE-25617 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.3, 3.2.0 >Reporter: lujie >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > see [https://www.cvedetails.com/cve/CVE-2020-8908/] > A temp directory creation vulnerability exists in all versions of Guava, > allowing an attacker with access to the machine to potentially access data in > a temporary directory created by the Guava API > com.google.common.io.Files.createTempDir(). By default, on unix-like systems, > the created directory is world-readable (readable by an attacker with access > to the system). The method in question has been marked > [@deprecated|https://github.com/deprecated] in versions 30.0 and later and > should not be used. For Android developers, we recommend choosing a temporary > directory API provided by Android, such as context.getCacheDir(). For other > Java developers, we recommend migrating to the Java 7 API > java.nio.file.Files.createTempDirectory() which explicitly configures > permissions of 700, or configuring the Java runtime's java.io.tmpdir system > property to point to a location whose permissions are appropriately > configured. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-25617) fix CVE-2020-8908
[ https://issues.apache.org/jira/browse/HIVE-25617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HIVE-25617: Affects Version/s: 3.1.3 > fix CVE-2020-8908 > - > > Key: HIVE-25617 > URL: https://issues.apache.org/jira/browse/HIVE-25617 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.3 >Reporter: lujie >Priority: Critical > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > see [https://www.cvedetails.com/cve/CVE-2020-8908/] > A temp directory creation vulnerability exists in all versions of Guava, > allowing an attacker with access to the machine to potentially access data in > a temporary directory created by the Guava API > com.google.common.io.Files.createTempDir(). By default, on unix-like systems, > the created directory is world-readable (readable by an attacker with access > to the system). The method in question has been marked > [@deprecated|https://github.com/deprecated] in versions 30.0 and later and > should not be used. For Android developers, we recommend choosing a temporary > directory API provided by Android, such as context.getCacheDir(). For other > Java developers, we recommend migrating to the Java 7 API > java.nio.file.Files.createTempDirectory() which explicitly configures > permissions of 700, or configuring the Java runtime's java.io.tmpdir system > property to point to a location whose permissions are appropriately > configured. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-25617) fix CVE-2020-8908
[ https://issues.apache.org/jira/browse/HIVE-25617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HIVE-25617: Priority: Major (was: Critical) > fix CVE-2020-8908 > - > > Key: HIVE-25617 > URL: https://issues.apache.org/jira/browse/HIVE-25617 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.3 >Reporter: lujie >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > see [https://www.cvedetails.com/cve/CVE-2020-8908/] > A temp directory creation vulnerability exists in all versions of Guava, > allowing an attacker with access to the machine to potentially access data in > a temporary directory created by the Guava API > com.google.common.io.Files.createTempDir(). By default, on unix-like systems, > the created directory is world-readable (readable by an attacker with access > to the system). The method in question has been marked > [@deprecated|https://github.com/deprecated] in versions 30.0 and later and > should not be used. For Android developers, we recommend choosing a temporary > directory API provided by Android, such as context.getCacheDir(). For other > Java developers, we recommend migrating to the Java 7 API > java.nio.file.Files.createTempDirectory() which explicitly configures > permissions of 700, or configuring the Java runtime's java.io.tmpdir system > property to point to a location whose permissions are appropriately > configured. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-26634) [Hive][Spark] EntityNotFoundException ,Database global_temp not found, when connecting hive metastore to aws glue.
[
https://issues.apache.org/jira/browse/HIVE-26634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena resolved HIVE-26634.
-
Fix Version/s: Not Applicable
Resolution: Cannot Reproduce
This is a Spark thing or a AWS thing, resolving!!!
> [Hive][Spark] EntityNotFoundException ,Database global_temp not found, when
> connecting hive metastore to aws glue.
> --
>
> Key: HIVE-26634
> URL: https://issues.apache.org/jira/browse/HIVE-26634
> Project: Hive
> Issue Type: Bug
>Reporter: Mahmood Abu Awwad
>Priority: Blocker
> Fix For: Not Applicable
>
>
> while running our batches using Apache Spark with Hive on EMR cluster, as
> we're using AWS glue as a MetaStore, it seems there is an issue occurs, which
> is
> {code:java}
> EntityNotFoundException ,Database global_temp not found {code}
> {code:java}
> 2022-10-09T10:36:31,262 INFO [573c4ce0-f73c-439b-829d-1f0b25db45ec
> main([])]: ql.Driver (:()) - Completed compiling
> command(queryId=hadoop_20221009103631_214e4b6c-b0f2-496e-b9a8-86831b202736);
> Time taken: 0.02 seconds
> 2022-10-09T10:36:31,262 INFO [573c4ce0-f73c-439b-829d-1f0b25db45ec
> main([])]: reexec.ReExecDriver (:()) - Execution #1 of query
> 2022-10-09T10:36:31,262 INFO [573c4ce0-f73c-439b-829d-1f0b25db45ec
> main([])]: ql.Driver (:()) - Concurrency mode is disabled, not creating a
> lock manager
> 2022-10-09T10:36:31,262 INFO [573c4ce0-f73c-439b-829d-1f0b25db45ec
> main([])]: ql.Driver (:()) - Executing
> command(queryId=hadoop_20221009103631_214e4b6c-b0f2-496e-b9a8-86831b202736):
> show views
> 2022-10-09T10:36:31,263 INFO [573c4ce0-f73c-439b-829d-1f0b25db45ec
> main([])]: ql.Driver (:()) - Starting task [Stage-0:DDL] in serial mode
> 2022-10-09T10:36:32,270 INFO [573c4ce0-f73c-439b-829d-1f0b25db45ec
> main([])]: ql.Driver (:()) - Completed executing
> command(queryId=hadoop_20221009103631_214e4b6c-b0f2-496e-b9a8-86831b202736);
> Time taken: 1.008 seconds
> 2022-10-09T10:36:32,270 INFO [573c4ce0-f73c-439b-829d-1f0b25db45ec
> main([])]: ql.Driver (:()) - OK
> 2022-10-09T10:36:32,270 INFO [573c4ce0-f73c-439b-829d-1f0b25db45ec
> main([])]: ql.Driver (:()) - Concurrency mode is disabled, not creating a
> lock manager
> 2022-10-09T10:36:32,271 INFO [573c4ce0-f73c-439b-829d-1f0b25db45ec
> main([])]: exec.ListSinkOperator (:()) - RECORDS_OUT_INTERMEDIATE:0,
> RECORDS_OUT_OPERATOR_LIST_SINK_0:0,
> 2022-10-09T10:36:32,271 INFO [573c4ce0-f73c-439b-829d-1f0b25db45ec
> main([])]: CliDriver (:()) - Time taken: 1.028 seconds
> 2022-10-09T10:36:32,271 INFO [573c4ce0-f73c-439b-829d-1f0b25db45ec
> main([])]: conf.HiveConf (HiveConf.java:getLogIdVar(5104)) - Using the
> default value passed in for log id: 573c4ce0-f73c-439b-829d-1f0b25db45ec
> 2022-10-09T10:36:32,272 INFO [573c4ce0-f73c-439b-829d-1f0b25db45ec
> main([])]: session.SessionState (SessionState.java:resetThreadName(452)) -
> Resetting thread name to main
> 2022-10-09T10:36:46,512 INFO [main([])]: conf.HiveConf
> (HiveConf.java:getLogIdVar(5104)) - Using the default value passed in for log
> id: 573c4ce0-f73c-439b-829d-1f0b25db45ec
> 2022-10-09T10:36:46,513 INFO [main([])]: session.SessionState
> (SessionState.java:updateThreadName(441)) - Updating thread name to
> 573c4ce0-f73c-439b-829d-1f0b25db45ec main
> 2022-10-09T10:36:46,515 INFO [573c4ce0-f73c-439b-829d-1f0b25db45ec
> main([])]: ql.Driver (:()) - Compiling
> command(queryId=hadoop_20221009103646_f390a868-07d7-49f1-b620-70d40e5e2cff):
> use global_temp
> 2022-10-09T10:36:46,530 INFO [573c4ce0-f73c-439b-829d-1f0b25db45ec
> main([])]: ql.Driver (:()) - Concurrency mode is disabled, not creating a
> lock manager
> 2022-10-09T10:36:46,666 ERROR [573c4ce0-f73c-439b-829d-1f0b25db45ec
> main([])]: ql.Driver (:()) - FAILED: SemanticException [Error 10072]:
> Database does not exist: global_temp
> org.apache.hadoop.hive.ql.parse.SemanticException: Database does not exist:
> global_temp
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.getDatabase(BaseSemanticAnalyzer.java:2171)
> at
> org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.analyzeSwitchDatabase(DDLSemanticAnalyzer.java:1413)
> at
> org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.analyzeInternal(DDLSemanticAnalyzer.java:516)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:285)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:659)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1826)
> at
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1773)
> at
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1768)
> at
> org.apache.ha
[jira] [Resolved] (HIVE-25803) URL Mapping appends default Fs scheme even for LOCAL DIRECTORY ops
[
https://issues.apache.org/jira/browse/HIVE-25803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena resolved HIVE-25803.
-
Fix Version/s: 4.1.0
Assignee: Ayush Saxena (was: Sai Hemanth Gantasala)
Resolution: Fixed
> URL Mapping appends default Fs scheme even for LOCAL DIRECTORY ops
> --
>
> Key: HIVE-25803
> URL: https://issues.apache.org/jira/browse/HIVE-25803
> Project: Hive
> Issue Type: Bug
> Components: Authorization, HiveServer2
>Affects Versions: 4.0.0
>Reporter: Soumitra Sulav
>Assignee: Ayush Saxena
>Priority: Critical
> Labels: pull-request-available
> Fix For: 4.1.0
>
>
> Repro steps:
> Connect to beeline
> {code:java}
> beeline -u
> "jdbc:hive2://quasar-pxlypi-2.quasar-pxlypi.root.hwx.site:10001/;principal=hive/_h...@root.hwx.site;ssl=true;sslTrustStore=/var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_truststore.jks;trustStorePassword=VOAnRk5l4oXsg0upJ1ApscSuNksirOKgyhJvoPv2o4j;transportMode=http;httpPath=cliservice;"
> {code}
>
> Create a test table and run insert on local
> {code:java}
> > create table dual (id int);
> > insert overwrite local directory "/tmp/" select * from dual;
> {code}
> {code:java}
> Error: Error while compiling statement: FAILED: HiveAccessControlException
> Permission denied: user [hrt_qa] does not have [ALL] privilege on
> [hdfs://ns1/tmp] (state=42000,code=4)
> {code}
> It always appends hdfs:// to the path even if the operation is meant for
> local directory.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-25803) URL Mapping appends default Fs scheme even for LOCAL DIRECTORY ops
[
https://issues.apache.org/jira/browse/HIVE-25803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17799547#comment-17799547
]
Ayush Saxena commented on HIVE-25803:
-
Committed to master.
Thanx [~dkuzmenko] for the review!!!
> URL Mapping appends default Fs scheme even for LOCAL DIRECTORY ops
> --
>
> Key: HIVE-25803
> URL: https://issues.apache.org/jira/browse/HIVE-25803
> Project: Hive
> Issue Type: Bug
> Components: Authorization, HiveServer2
>Affects Versions: 4.0.0
>Reporter: Soumitra Sulav
>Assignee: Sai Hemanth Gantasala
>Priority: Critical
> Labels: pull-request-available
>
> Repro steps:
> Connect to beeline
> {code:java}
> beeline -u
> "jdbc:hive2://quasar-pxlypi-2.quasar-pxlypi.root.hwx.site:10001/;principal=hive/_h...@root.hwx.site;ssl=true;sslTrustStore=/var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_truststore.jks;trustStorePassword=VOAnRk5l4oXsg0upJ1ApscSuNksirOKgyhJvoPv2o4j;transportMode=http;httpPath=cliservice;"
> {code}
>
> Create a test table and run insert on local
> {code:java}
> > create table dual (id int);
> > insert overwrite local directory "/tmp/" select * from dual;
> {code}
> {code:java}
> Error: Error while compiling statement: FAILED: HiveAccessControlException
> Permission denied: user [hrt_qa] does not have [ALL] privilege on
> [hdfs://ns1/tmp] (state=42000,code=4)
> {code}
> It always appends hdfs:// to the path even if the operation is meant for
> local directory.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27022) Split removeDuplicateCompletedTxnComponents away from AcidHouseKeeper and onto a separate timer
[ https://issues.apache.org/jira/browse/HIVE-27022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-27022: -- Labels: pull-request-available (was: ) > Split removeDuplicateCompletedTxnComponents away from AcidHouseKeeper and > onto a separate timer > --- > > Key: HIVE-27022 > URL: https://issues.apache.org/jira/browse/HIVE-27022 > Project: Hive > Issue Type: Improvement >Reporter: Jason Phelps >Assignee: Taraka Rama Rao Lethavadla >Priority: Major > Labels: pull-request-available > > The operations and queries involved with > removeDuplicateCompletedTxnComponents are much costlier than the rest of the > AcidHouseKeeper jobs. The default timer for > metastore.acid.housekeeper.interval is 60s [0], which is much too frequent > for removeDuplicateCompletedTxnComponents, but fine for the other operations. > There is no way of lowering the internal for just > removeDuplicateCompletedTxnComponents without impacting the other > AcidHouseKeeper jobs, so I am requesting that it be split into its own > process/timer > [0] > https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java#L286 > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (HIVE-27964) Support drop stats similar to Impala
[
https://issues.apache.org/jira/browse/HIVE-27964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17799522#comment-17799522
]
Naresh P R edited comment on HIVE-27964 at 12/21/23 6:17 PM:
-
Partition table rename gets clogged at PART_COL_STATS for wide tables.
{code:java}
CREATE TABLE IF NOT EXISTS `PART_COL_STATS` (
...
`DB_NAME` varchar(128) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL,
`TABLE_NAME` varchar(128) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL,
`PARTITION_NAME` varchar(767) CHARACTER SET latin1 COLLATE latin1_bin NOT
NULL,
...){code}
Since PART_COL_STATS holds db_name & table_name, incase of table rename, every
row in PART_COL_STATS associated with the table should be fetched, stored in
memory, delete & re-insert with new db/table/partition name.
Instead clearing the stats before rename & computing later would help to speed
up the process.
Just raised another optimization HIVE-27965, to remove DB_NAME, TABLE_NAME,
PARTITION_NAME from PART_COL_STATS & use TBL_ID, DB_ID, PART_ID to avoid
touching PART_COL_STATS for table/partition renames + can be used in indexes as
well.
was (Author: nareshpr):
Partition table rename gets clogged at PART_COL_STATS for wide tables.
{code:java}
CREATE TABLE IF NOT EXISTS `PART_COL_STATS` (
...
`DB_NAME` varchar(128) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL,
`TABLE_NAME` varchar(128) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL,
`PARTITION_NAME` varchar(767) CHARACTER SET latin1 COLLATE latin1_bin NOT
NULL,
...){code}
Since PART_COL_STATS holds db_name & table_name, incase of table rename, every
row in PART_COL_STATS associated with the table should be fetched, stored in
memory, delete & re-insert with new db/table/partition name.
Instead clearing the stats before rename & computing later would help to speed
up the process.
Another optimization i was about to raise is to remove DB_NAME, TABLE_NAME,
PARTITION_NAME from PART_COL_STATS & use TBL_ID, DB_ID, PART_ID to avoid
touching PART_COL_STATS for table/partition renames + can be used in indexes as
well.
> Support drop stats similar to Impala
>
>
> Key: HIVE-27964
> URL: https://issues.apache.org/jira/browse/HIVE-27964
> Project: Hive
> Issue Type: New Feature
>Reporter: Naresh P R
>Priority: Major
>
> Hive should support drop stats similar to impala.
> https://impala.apache.org/docs/build/html/topics/impala_drop_stats.html
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27965) Table/partition rename takes a long time at PART_COL_STATS for wide tables
[
https://issues.apache.org/jira/browse/HIVE-27965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Naresh P R updated HIVE-27965:
--
Description:
Partition table rename gets clogged at PART_COL_STATS for wide tables.
{code:java}
CREATE TABLE IF NOT EXISTS `PART_COL_STATS` (
...
`DB_NAME` varchar(128) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL,
`TABLE_NAME` varchar(128) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL,
`PARTITION_NAME` varchar(767) CHARACTER SET latin1 COLLATE latin1_bin NOT
NULL,
...){code}
Since PART_COL_STATS holds db_name & table_name, incase of table rename, every
row in PART_COL_STATS associated with the table should be fetched, stored in
memory, delete & re-insert with new db/table/partition name.
Remove DB_NAME, TABLE_NAME, PARTITION_NAME from PART_COL_STATS, instead use
TBL_ID, DB_ID, PART_ID to avoid touching PART_COL_STATS for table/partition
renames.
Also TBL_ID, DB_ID, PART_ID can be used for PART_COL_STATS INDEXING.
was:
Partition table rename gets clogged at PART_COL_STATS for wide tables.
{code:java}
CREATE TABLE IF NOT EXISTS `PART_COL_STATS` (
...
`DB_NAME` varchar(128) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL,
`TABLE_NAME` varchar(128) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL,
`PARTITION_NAME` varchar(767) CHARACTER SET latin1 COLLATE latin1_bin NOT
NULL,
...){code}
Since PART_COL_STATS holds db_name & table_name, incase of table rename, every
row in PART_COL_STATS associated with the table should be fetched, stored in
memory, delete & re-insert with new db/table/partition name.
Remove DB_NAME, TABLE_NAME, PARTITION_NAME from PART_COL_STATS & use & use
TBL_ID, DB_ID, PART_ID to avoid touching PART_COL_STATS for table/partition
renames.
Also TBL_ID, DB_ID, PART_ID can be used for PART_COL_STATS INDEXING.
> Table/partition rename takes a long time at PART_COL_STATS for wide tables
> --
>
> Key: HIVE-27965
> URL: https://issues.apache.org/jira/browse/HIVE-27965
> Project: Hive
> Issue Type: Improvement
>Reporter: Naresh P R
>Priority: Major
>
> Partition table rename gets clogged at PART_COL_STATS for wide tables.
> {code:java}
> CREATE TABLE IF NOT EXISTS `PART_COL_STATS` (
> ...
> `DB_NAME` varchar(128) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL,
> `TABLE_NAME` varchar(128) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL,
> `PARTITION_NAME` varchar(767) CHARACTER SET latin1 COLLATE latin1_bin NOT
> NULL,
> ...){code}
> Since PART_COL_STATS holds db_name & table_name, incase of table rename,
> every row in PART_COL_STATS associated with the table should be fetched,
> stored in memory, delete & re-insert with new db/table/partition name.
>
> Remove DB_NAME, TABLE_NAME, PARTITION_NAME from PART_COL_STATS, instead use
> TBL_ID, DB_ID, PART_ID to avoid touching PART_COL_STATS for table/partition
> renames.
> Also TBL_ID, DB_ID, PART_ID can be used for PART_COL_STATS INDEXING.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27965) Table/partition rename takes a long time at PART_COL_STATS for wide tables
[
https://issues.apache.org/jira/browse/HIVE-27965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Naresh P R updated HIVE-27965:
--
Description:
Partition table rename gets clogged at PART_COL_STATS for wide tables.
{code:java}
CREATE TABLE IF NOT EXISTS `PART_COL_STATS` (
...
`DB_NAME` varchar(128) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL,
`TABLE_NAME` varchar(128) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL,
`PARTITION_NAME` varchar(767) CHARACTER SET latin1 COLLATE latin1_bin NOT
NULL,
...){code}
Since PART_COL_STATS holds db_name & table_name, incase of table rename, every
row in PART_COL_STATS associated with the table should be fetched, stored in
memory, delete & re-insert with new db/table/partition name.
Remove DB_NAME, TABLE_NAME, PARTITION_NAME from PART_COL_STATS & use & use
TBL_ID, DB_ID, PART_ID to avoid touching PART_COL_STATS for table/partition
renames.
Also TBL_ID, DB_ID, PART_ID can be used for PART_COL_STATS INDEXING.
was:
Partition table rename gets clogged at PART_COL_STATS for wide tables.
{code:java}
CREATE TABLE IF NOT EXISTS `PART_COL_STATS` (
...
`DB_NAME` varchar(128) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL,
`TABLE_NAME` varchar(128) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL,
`PARTITION_NAME` varchar(767) CHARACTER SET latin1 COLLATE latin1_bin NOT
NULL,
...){code}
Since PART_COL_STATS holds db_name & table_name, incase of table rename, every
row in PART_COL_STATS associated with the table should be fetched, stored in
memory, delete & re-insert with new db/table/partition name.
Remove DB_NAME, TABLE_NAME, PARTITION_NAME from PART_COL_STATS & use PART_ID as
FOREIGN KEY from PARTITIONS to avoid touching PART_COL_STATS for
table/partition renames.
> Table/partition rename takes a long time at PART_COL_STATS for wide tables
> --
>
> Key: HIVE-27965
> URL: https://issues.apache.org/jira/browse/HIVE-27965
> Project: Hive
> Issue Type: Improvement
>Reporter: Naresh P R
>Priority: Major
>
> Partition table rename gets clogged at PART_COL_STATS for wide tables.
> {code:java}
> CREATE TABLE IF NOT EXISTS `PART_COL_STATS` (
> ...
> `DB_NAME` varchar(128) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL,
> `TABLE_NAME` varchar(128) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL,
> `PARTITION_NAME` varchar(767) CHARACTER SET latin1 COLLATE latin1_bin NOT
> NULL,
> ...){code}
> Since PART_COL_STATS holds db_name & table_name, incase of table rename,
> every row in PART_COL_STATS associated with the table should be fetched,
> stored in memory, delete & re-insert with new db/table/partition name.
>
> Remove DB_NAME, TABLE_NAME, PARTITION_NAME from PART_COL_STATS & use & use
> TBL_ID, DB_ID, PART_ID to avoid touching PART_COL_STATS for table/partition
> renames.
> Also TBL_ID, DB_ID, PART_ID can be used for PART_COL_STATS INDEXING.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (HIVE-27964) Support drop stats similar to Impala
[
https://issues.apache.org/jira/browse/HIVE-27964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17799522#comment-17799522
]
Naresh P R edited comment on HIVE-27964 at 12/21/23 6:10 PM:
-
Partition table rename gets clogged at PART_COL_STATS for wide tables.
{code:java}
CREATE TABLE IF NOT EXISTS `PART_COL_STATS` (
...
`DB_NAME` varchar(128) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL,
`TABLE_NAME` varchar(128) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL,
`PARTITION_NAME` varchar(767) CHARACTER SET latin1 COLLATE latin1_bin NOT
NULL,
...){code}
Since PART_COL_STATS holds db_name & table_name, incase of table rename, every
row in PART_COL_STATS associated with the table should be fetched, stored in
memory, delete & re-insert with new db/table/partition name.
Instead clearing the stats before rename & computing later would help to speed
up the process.
Another optimization i was about to raise is to remove DB_NAME, TABLE_NAME,
PARTITION_NAME from PART_COL_STATS & use TBL_ID, DB_ID, PART_ID to avoid
touching PART_COL_STATS for table/partition renames + can be used in indexes as
well.
was (Author: nareshpr):
Partition table rename gets clogged at PART_COL_STATS for wide tables.
{code:java}
CREATE TABLE IF NOT EXISTS `PART_COL_STATS` (
...
`DB_NAME` varchar(128) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL,
`TABLE_NAME` varchar(128) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL,
`PARTITION_NAME` varchar(767) CHARACTER SET latin1 COLLATE latin1_bin NOT
NULL,
...){code}
Since PART_COL_STATS holds db_name & table_name, incase of table rename, every
row in PART_COL_STATS associated with the table should be fetched, stored in
memory, delete & re-insert with new db/table/partition name.
Instead clearing the stats before rename & computing later would help to speed
up the process.
Another optimization i was about to raise is to remove DB_NAME, TABLE_NAME,
PARTITION_NAME from PART_COL_STATS & use PART_ID as FOREIGN KEY from PARTITIONS
to avoid touching PART_COL_STATS for table/partition renames.
> Support drop stats similar to Impala
>
>
> Key: HIVE-27964
> URL: https://issues.apache.org/jira/browse/HIVE-27964
> Project: Hive
> Issue Type: New Feature
>Reporter: Naresh P R
>Priority: Major
>
> Hive should support drop stats similar to impala.
> https://impala.apache.org/docs/build/html/topics/impala_drop_stats.html
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-27965) Table/partition rename takes a long time at PART_COL_STATS for wide tables
Naresh P R created HIVE-27965:
-
Summary: Table/partition rename takes a long time at
PART_COL_STATS for wide tables
Key: HIVE-27965
URL: https://issues.apache.org/jira/browse/HIVE-27965
Project: Hive
Issue Type: Improvement
Reporter: Naresh P R
Partition table rename gets clogged at PART_COL_STATS for wide tables.
{code:java}
CREATE TABLE IF NOT EXISTS `PART_COL_STATS` (
...
`DB_NAME` varchar(128) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL,
`TABLE_NAME` varchar(128) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL,
`PARTITION_NAME` varchar(767) CHARACTER SET latin1 COLLATE latin1_bin NOT
NULL,
...){code}
Since PART_COL_STATS holds db_name & table_name, incase of table rename, every
row in PART_COL_STATS associated with the table should be fetched, stored in
memory, delete & re-insert with new db/table/partition name.
Remove DB_NAME, TABLE_NAME, PARTITION_NAME from PART_COL_STATS & use PART_ID as
FOREIGN KEY from PARTITIONS to avoid touching PART_COL_STATS for
table/partition renames.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (HIVE-27964) Support drop stats similar to Impala
[
https://issues.apache.org/jira/browse/HIVE-27964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17799522#comment-17799522
]
Naresh P R edited comment on HIVE-27964 at 12/21/23 5:47 PM:
-
Partition table rename gets clogged at PART_COL_STATS for wide tables.
{code:java}
CREATE TABLE IF NOT EXISTS `PART_COL_STATS` (
...
`DB_NAME` varchar(128) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL,
`TABLE_NAME` varchar(128) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL,
`PARTITION_NAME` varchar(767) CHARACTER SET latin1 COLLATE latin1_bin NOT
NULL,
...){code}
Since PART_COL_STATS holds db_name & table_name, incase of table rename, every
row in PART_COL_STATS associated with the table should be fetched, stored in
memory, delete & re-insert with new db/table/partition name.
Instead clearing the stats before rename & computing later would help to speed
up the process.
Another optimization i was about to raise is to remove DB_NAME, TABLE_NAME,
PARTITION_NAME from PART_COL_STATS & use PART_ID as FOREIGN KEY from PARTITIONS
to avoid touching PART_COL_STATS for table/partition renames.
was (Author: nareshpr):
Partition table rename gets clogged at PART_COL_STATS for wide tables.
{code:java}
CREATE TABLE IF NOT EXISTS `PART_COL_STATS` (
...
`DB_NAME` varchar(128) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL,
`TABLE_NAME` varchar(128) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL,
`PARTITION_NAME` varchar(767) CHARACTER SET latin1 COLLATE latin1_bin NOT
NULL,
...){code}
Since PART_COL_STATS holds db_name & table_name, incase of table rename, every
row in PART_COL_STATS associated with the table should be fetched, stored in
memory, dropped & re-added with new tableName.
Instead clearing the stats before rename & computing later would help to speed
up the process.
Another optimization i was about to raise is to remove DB_NAME, TABLE_NAME,
PARTITION_NAME from PART_COL_STATS & use PART_ID as FOREIGN KEY from PARTITIONS
to avoid touching PART_COL_STATS for table/partition renames.
> Support drop stats similar to Impala
>
>
> Key: HIVE-27964
> URL: https://issues.apache.org/jira/browse/HIVE-27964
> Project: Hive
> Issue Type: New Feature
>Reporter: Naresh P R
>Priority: Major
>
> Hive should support drop stats similar to impala.
> https://impala.apache.org/docs/build/html/topics/impala_drop_stats.html
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-27964) Support drop stats similar to Impala
[
https://issues.apache.org/jira/browse/HIVE-27964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17799522#comment-17799522
]
Naresh P R commented on HIVE-27964:
---
Partition table rename gets clogged at PART_COL_STATS for wide tables.
{code:java}
CREATE TABLE IF NOT EXISTS `PART_COL_STATS` (
...
`DB_NAME` varchar(128) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL,
`TABLE_NAME` varchar(128) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL,
`PARTITION_NAME` varchar(767) CHARACTER SET latin1 COLLATE latin1_bin NOT
NULL,
...){code}
Since PART_COL_STATS holds db_name & table_name, incase of table rename, every
row in PART_COL_STATS associated with the table should be fetched, stored in
memory, dropped & re-added with new tableName.
Instead clearing the stats before rename & computing later would help to speed
up the process.
Another optimization i was about to raise is to remove DB_NAME, TABLE_NAME,
PARTITION_NAME from PART_COL_STATS & use PART_ID as FOREIGN KEY from PARTITIONS
to avoid touching PART_COL_STATS for table/partition renames.
> Support drop stats similar to Impala
>
>
> Key: HIVE-27964
> URL: https://issues.apache.org/jira/browse/HIVE-27964
> Project: Hive
> Issue Type: New Feature
>Reporter: Naresh P R
>Priority: Major
>
> Hive should support drop stats similar to impala.
> https://impala.apache.org/docs/build/html/topics/impala_drop_stats.html
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-27964) Support drop stats similar to Impala
[ https://issues.apache.org/jira/browse/HIVE-27964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17799436#comment-17799436 ] Butao Zhang commented on HIVE-27964: IMO, many users are used to updating stats in stead of drop stats in Hive. If supporting the drop stats ability, What are the main use cases of this feature in Hive? > Support drop stats similar to Impala > > > Key: HIVE-27964 > URL: https://issues.apache.org/jira/browse/HIVE-27964 > Project: Hive > Issue Type: New Feature >Reporter: Naresh P R >Priority: Major > > Hive should support drop stats similar to impala. > https://impala.apache.org/docs/build/html/topics/impala_drop_stats.html -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27161) MetaException when executing CTAS query in Druid storage handler
[
https://issues.apache.org/jira/browse/HIVE-27161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-27161:
--
Fix Version/s: 4.1.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
Merged to master. Thanks [~dkuzmenko] for review!
> MetaException when executing CTAS query in Druid storage handler
>
>
> Key: HIVE-27161
> URL: https://issues.apache.org/jira/browse/HIVE-27161
> Project: Hive
> Issue Type: Bug
> Components: Druid integration
>Affects Versions: 4.0.0-alpha-1, 4.0.0-alpha-2
>Reporter: Stamatis Zampetakis
>Assignee: Krisztian Kasa
>Priority: Critical
> Labels: pull-request-available
> Fix For: 4.1.0
>
>
> Any kind of CTAS query targeting the Druid storage handler fails with the
> following exception:
> {noformat}
> org.apache.hadoop.hive.ql.metadata.HiveException:
> MetaException(message:LOCATION may not be specified for Druid)
> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:1347)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:1352)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.ddl.table.create.CreateTableOperation.createTableNonReplaceMode(CreateTableOperation.java:158)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.ddl.table.create.CreateTableOperation.execute(CreateTableOperation.java:116)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.ddl.DDLTask.execute(DDLTask.java:84)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:354)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:327)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:244)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:105)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:367)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:205)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:154)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:149)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:185)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:228)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:257)
> ~[hive-cli-4.0.0-SNAPSHOT.jar:?]
> at org.apache.hadoop.hive.cli.CliDriver.processCmd1(CliDriver.java:201)
> ~[hive-cli-4.0.0-SNAPSHOT.jar:?]
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:127)
> ~[hive-cli-4.0.0-SNAPSHOT.jar:?]
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:425)
> ~[hive-cli-4.0.0-SNAPSHOT.jar:?]
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:356)
> ~[hive-cli-4.0.0-SNAPSHOT.jar:?]
> at
> org.apache.hadoop.hive.ql.dataset.QTestDatasetHandler.initDataset(QTestDatasetHandler.java:86)
> ~[hive-it-util-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.dataset.QTestDatasetHandler.beforeTest(QTestDatasetHandler.java:190)
> ~[hive-it-util-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.qoption.QTestOptionDispatcher.beforeTest(QTestOptionDispatcher.java:79)
> ~[hive-it-util-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.QTestUtil.cliInit(QTestUtil.java:607)
> ~[hive-it-util-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:112)
> ~[hive-it-util-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:157)
> ~[hive-it-util-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver(TestMiniDruidCliD
