[jira] [Resolved] (HIVE-28200) Improve get_partitions_by_filter/expr when partition limit enabled
[ https://issues.apache.org/jira/browse/HIVE-28200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng resolved HIVE-28200. Fix Version/s: 4.1.0 Resolution: Fixed Fix has been pushed to master. Thank you for the PR [~wechar] ! > Improve get_partitions_by_filter/expr when partition limit enabled > -- > > Key: HIVE-28200 > URL: https://issues.apache.org/jira/browse/HIVE-28200 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Wechar >Assignee: Wechar >Priority: Major > Labels: pull-request-available > Fix For: 4.1.0 > > > When {{hive.metastore.limit.partition.request}} is configured, HMS would get > the matching partition counts before get the real partition objects. The > count could be a slow query if the input filter or expr is too complex. > In this case, such slow filter will be executed in both counting partition > numbers and fetching real partition objects, which harms the performance and > backend DBMS. > We can make an improvement by getting matched partition names firstly, and > then check limit through the size of partition names, and finally get the > partitions by the partition names. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-28250) Add tez.task-specific configs into whitelist to modify at session level
[ https://issues.apache.org/jira/browse/HIVE-28250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-28250: -- Labels: pull-request-available (was: ) > Add tez.task-specific configs into whitelist to modify at session level > --- > > Key: HIVE-28250 > URL: https://issues.apache.org/jira/browse/HIVE-28250 > Project: Hive > Issue Type: Improvement >Reporter: Raghav Aggarwal >Assignee: Raghav Aggarwal >Priority: Major > Labels: pull-request-available > > When we try to set tez.task-specific configs at runtime, it doesn't allow it > {code:java} > : jdbc:hive2://localhost:1> set > tez.task-specific.launch.cmd-opts.list="Map 1[0]"; > Error: Error while processing statement: Cannot modify > tez.task-specific.launch.cmd-opts.list at runtime. It is not in list of > params that are allowed to be modified at runtime (state=42000,code=1) {code} > Putting this in whitelist will help to debug tez query easily otherwise, > admin has to add the regex in > _hive.security.authorization.sqlstd.confwhitelist.append_ config in Ambari UI > and restart the HS2 process. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work started] (HIVE-28250) Add tez.task-specific configs into whitelist to modify at session level
[ https://issues.apache.org/jira/browse/HIVE-28250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-28250 started by Raghav Aggarwal. -- > Add tez.task-specific configs into whitelist to modify at session level > --- > > Key: HIVE-28250 > URL: https://issues.apache.org/jira/browse/HIVE-28250 > Project: Hive > Issue Type: Improvement >Reporter: Raghav Aggarwal >Assignee: Raghav Aggarwal >Priority: Major > > When we try to set tez.task-specific configs at runtime, it doesn't allow it > {code:java} > : jdbc:hive2://localhost:1> set > tez.task-specific.launch.cmd-opts.list="Map 1[0]"; > Error: Error while processing statement: Cannot modify > tez.task-specific.launch.cmd-opts.list at runtime. It is not in list of > params that are allowed to be modified at runtime (state=42000,code=1) {code} > Putting this in whitelist will help to debug tez query easily otherwise, > admin has to add the regex in > _hive.security.authorization.sqlstd.confwhitelist.append_ config in Ambari UI > and restart the HS2 process. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-28250) Add tez.task-specific configs into whitelist to modify at session level
[ https://issues.apache.org/jira/browse/HIVE-28250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raghav Aggarwal updated HIVE-28250: --- Description: When we try to set tez.task-specific configs at runtime, it doesn't allow it {code:java} : jdbc:hive2://localhost:1> set tez.task-specific.launch.cmd-opts.list="Map 1[0]"; Error: Error while processing statement: Cannot modify tez.task-specific.launch.cmd-opts.list at runtime. It is not in list of params that are allowed to be modified at runtime (state=42000,code=1) {code} Putting this in whitelist will help to debug tez query easily otherwise, admin has to add the regex in _hive.security.authorization.sqlstd.confwhitelist.append_ config in Ambari UI and restart the HS2 process. was: When we try to set tez.task-specific configs at runtime, it doesn't allow it {code:java} : jdbc:hive2://localhost:1> set tez.task-specific.launch.cmd-opts.list="Map 1[0]"; Error: Error while processing statement: Cannot modify tez.task-specific.launch.cmd-opts.list at runtime. It is not in list of params that are allowed to be modified at runtime (state=42000,code=1) {code} Putting this in whitelist will help to debug tez query easily otherwise, admin has to add the regex in _hive.security.authorization.sqlstd.confwhitelist.append_ config in Ambari UI and restart the HS2 process. Better to whitelist by default as we have already whitelisted, the following regex: {code:java} {code} {color:#808080}tez\\.task\\..*{color} > Add tez.task-specific configs into whitelist to modify at session level > --- > > Key: HIVE-28250 > URL: https://issues.apache.org/jira/browse/HIVE-28250 > Project: Hive > Issue Type: Improvement >Reporter: Raghav Aggarwal >Assignee: Raghav Aggarwal >Priority: Major > > When we try to set tez.task-specific configs at runtime, it doesn't allow it > {code:java} > : jdbc:hive2://localhost:1> set > tez.task-specific.launch.cmd-opts.list="Map 1[0]"; > Error: Error while processing statement: Cannot modify > tez.task-specific.launch.cmd-opts.list at runtime. It is not in list of > params that are allowed to be modified at runtime (state=42000,code=1) {code} > Putting this in whitelist will help to debug tez query easily otherwise, > admin has to add the regex in > _hive.security.authorization.sqlstd.confwhitelist.append_ config in Ambari UI > and restart the HS2 process. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-28250) Add tez.task-specific configs into whitelist to modify at session level
[ https://issues.apache.org/jira/browse/HIVE-28250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raghav Aggarwal updated HIVE-28250: --- Description: When we try to set tez.task-specific configs at runtime, it doesn't allow it {code:java} : jdbc:hive2://localhost:1> set tez.task-specific.launch.cmd-opts.list="Map 1[0]"; Error: Error while processing statement: Cannot modify tez.task-specific.launch.cmd-opts.list at runtime. It is not in list of params that are allowed to be modified at runtime (state=42000,code=1) {code} Putting this in whitelist will help to debug tez query easily otherwise, admin has to add the regex in _hive.security.authorization.sqlstd.confwhitelist.append_ config in Ambari UI and restart the HS2 process. Better to whitelist by default as we have already whitelisted, the following regex: {code:java} {code} {color:#808080}tez\\.task\\..*{color} was: When we try to set tez.task-specific configs at runtime, it doesn't allow it {code:java} : jdbc:hive2://localhost:1> set tez.task-specific.launch.cmd-opts.list="Map 1[0]"; Error: Error while processing statement: Cannot modify tez.task-specific.launch.cmd-opts.list at runtime. It is not in list of params that are allowed to be modified at runtime (state=42000,code=1) {code} Putting this in whitelist will help to debug tez query easily otherwise, admin has to add the regex in _hive.security.authorization.sqlstd.confwhitelist.append_ config in Ambari UI and restart the HS2 process. Better to whitelist by default as we have already whitelisted, {_}*tez\\.task\\*{_}{_}{*}..{*}*{_} regex. > Add tez.task-specific configs into whitelist to modify at session level > --- > > Key: HIVE-28250 > URL: https://issues.apache.org/jira/browse/HIVE-28250 > Project: Hive > Issue Type: Improvement >Reporter: Raghav Aggarwal >Assignee: Raghav Aggarwal >Priority: Major > > When we try to set tez.task-specific configs at runtime, it doesn't allow it > {code:java} > : jdbc:hive2://localhost:1> set > tez.task-specific.launch.cmd-opts.list="Map 1[0]"; > Error: Error while processing statement: Cannot modify > tez.task-specific.launch.cmd-opts.list at runtime. It is not in list of > params that are allowed to be modified at runtime (state=42000,code=1) {code} > Putting this in whitelist will help to debug tez query easily otherwise, > admin has to add the regex in > _hive.security.authorization.sqlstd.confwhitelist.append_ config in Ambari UI > and restart the HS2 process. > Better to whitelist by default as we have already whitelisted, the following > regex: > {code:java} > {code} > {color:#808080}tez\\.task\\..*{color} -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-28250) Add tez.task-specific configs into whitelist to modify at session level
Raghav Aggarwal created HIVE-28250: -- Summary: Add tez.task-specific configs into whitelist to modify at session level Key: HIVE-28250 URL: https://issues.apache.org/jira/browse/HIVE-28250 Project: Hive Issue Type: Improvement Reporter: Raghav Aggarwal Assignee: Raghav Aggarwal When we try to set tez.task-specific configs at runtime, it doesn't allow it {code:java} : jdbc:hive2://localhost:1> set tez.task-specific.launch.cmd-opts.list="Map 1[0]"; Error: Error while processing statement: Cannot modify tez.task-specific.launch.cmd-opts.list at runtime. It is not in list of params that are allowed to be modified at runtime (state=42000,code=1) {code} Putting this in whitelist will help to debug tez query easily otherwise, admin has to add the regex in _hive.security.authorization.sqlstd.confwhitelist.append_ config in Ambari UI and restart the HS2 process. Better to whitelist by default as we have already whitelisted, _*tez\\.task\\..**_ regex. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-28250) Add tez.task-specific configs into whitelist to modify at session level
[ https://issues.apache.org/jira/browse/HIVE-28250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raghav Aggarwal updated HIVE-28250: --- Description: When we try to set tez.task-specific configs at runtime, it doesn't allow it {code:java} : jdbc:hive2://localhost:1> set tez.task-specific.launch.cmd-opts.list="Map 1[0]"; Error: Error while processing statement: Cannot modify tez.task-specific.launch.cmd-opts.list at runtime. It is not in list of params that are allowed to be modified at runtime (state=42000,code=1) {code} Putting this in whitelist will help to debug tez query easily otherwise, admin has to add the regex in _hive.security.authorization.sqlstd.confwhitelist.append_ config in Ambari UI and restart the HS2 process. Better to whitelist by default as we have already whitelisted, {_}*tez\\.task\\*{_}{_}{*}..{*}*{_} regex. was: When we try to set tez.task-specific configs at runtime, it doesn't allow it {code:java} : jdbc:hive2://localhost:1> set tez.task-specific.launch.cmd-opts.list="Map 1[0]"; Error: Error while processing statement: Cannot modify tez.task-specific.launch.cmd-opts.list at runtime. It is not in list of params that are allowed to be modified at runtime (state=42000,code=1) {code} Putting this in whitelist will help to debug tez query easily otherwise, admin has to add the regex in _hive.security.authorization.sqlstd.confwhitelist.append_ config in Ambari UI and restart the HS2 process. Better to whitelist by default as we have already whitelisted, _*tez\\.task\\..**_ regex. > Add tez.task-specific configs into whitelist to modify at session level > --- > > Key: HIVE-28250 > URL: https://issues.apache.org/jira/browse/HIVE-28250 > Project: Hive > Issue Type: Improvement >Reporter: Raghav Aggarwal >Assignee: Raghav Aggarwal >Priority: Major > > When we try to set tez.task-specific configs at runtime, it doesn't allow it > {code:java} > : jdbc:hive2://localhost:1> set > tez.task-specific.launch.cmd-opts.list="Map 1[0]"; > Error: Error while processing statement: Cannot modify > tez.task-specific.launch.cmd-opts.list at runtime. It is not in list of > params that are allowed to be modified at runtime (state=42000,code=1) {code} > Putting this in whitelist will help to debug tez query easily otherwise, > admin has to add the regex in > _hive.security.authorization.sqlstd.confwhitelist.append_ config in Ambari UI > and restart the HS2 process. > Better to whitelist by default as we have already whitelisted, > {_}*tez\\.task\\*{_}{_}{*}..{*}*{_} regex. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-28247) Execute immediate 'select count(*) from tbl' throwing ClassCastException in hplsql mode.
[ https://issues.apache.org/jira/browse/HIVE-28247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa resolved HIVE-28247. --- Fix Version/s: 4.1.0 Resolution: Fixed Merged to master. Thanks [~Dayakar] for the patch. > Execute immediate 'select count(*) from tbl' throwing ClassCastException in > hplsql mode. > > > Key: HIVE-28247 > URL: https://issues.apache.org/jira/browse/HIVE-28247 > Project: Hive > Issue Type: Bug > Components: hpl/sql >Reporter: Dayakar M >Assignee: Dayakar M >Priority: Major > Labels: pull-request-available > Fix For: 4.1.0 > > > Execute immediate 'select count(*) from tbl' throwing ClassCastException in > hplsql mode. > > Steps to reproduce: > {noformat} > execute immediate 'SELECT count(*) from result';"{noformat} > StackTrace in HS2 logs: > {noformat} > 2024-05-06T08:45:42,730 ERROR [HiveServer2-Background-Pool: Thread-850] > hplsql.HplSqlOperation: Error running hive query > org.apache.hive.service.cli.HiveSQLException: Error running HPL/SQL operation >at > org.apache.hive.service.cli.operation.hplsql.HplSqlOperation.interpret(HplSqlOperation.java:111) > ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT] >at > org.apache.hive.service.cli.operation.hplsql.HplSqlOperation.access$500(HplSqlOperation.java:54) > ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT] >at > org.apache.hive.service.cli.operation.hplsql.HplSqlOperation$BackgroundWork.lambda$run$0(HplSqlOperation.java:207) > ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT] >at java.security.AccessController.doPrivileged(Native Method) > ~[?:1.8.0_292] >at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_292] >at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899) > ~[hadoop-common-3.3.6.jar:?] >at > org.apache.hive.service.cli.operation.hplsql.HplSqlOperation$BackgroundWork.run(HplSqlOperation.java:219) > ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT] >at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > ~[?:1.8.0_292] >at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_292] >at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > ~[?:1.8.0_292] >at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > ~[?:1.8.0_292] >at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_292] > Caused by: java.lang.ClassCastException: class java.lang.Long cannot be > casted to class java.lang.String >at > org.apache.hive.service.cli.operation.hplsql.HplSqlQueryExecutor$OperationRowResult.get(HplSqlQueryExecutor.java:147) > ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT] >at org.apache.hive.hplsql.executor.QueryResult.column(QueryResult.java:49) > ~[hive-hplsql-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT] >at org.apache.hive.hplsql.Stmt.exec(Stmt.java:1095) > ~[hive-hplsql-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT] >at org.apache.hive.hplsql.Exec.visitExec_stmt(Exec.java:2061) > ~[hive-hplsql-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT] >at org.apache.hive.hplsql.Exec.visitExec_stmt(Exec.java:96) > ~[hive-hplsql-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT] >at > org.apache.hive.hplsql.HplsqlParser$Exec_stmtContext.accept(HplsqlParser.java:10369) > ~[hive-hplsql-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT] >at > org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visitChildren(AbstractParseTreeVisitor.java:46) > ~[antlr4-runtime-4.9.3.jar:4.9.3] >at org.apache.hive.hplsql.Exec.visitStmt(Exec.java:1103) > ~[hive-hplsql-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT] >at org.apache.hive.hplsql.Exec.visitStmt(Exec.java:96) > ~[hive-hplsql-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT] >at > org.apache.hive.hplsql.HplsqlParser$StmtContext.accept(HplsqlParser.java:1054) > ~[hive-hplsql-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT] >at > org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visitChildren(AbstractParseTreeVisitor.java:46) > ~[antlr4-runtime-4.9.3.jar:4.9.3] >at > org.apache.hive.hplsql.HplsqlBaseVisitor.visitBlock(HplsqlBaseVisitor.java:27) > ~[hive-hplsql-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT] >at > org.apache.hive.hplsql.HplsqlParser$BlockContext.accept(HplsqlParser.java:473) > ~[hive-hplsql-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT] >at > org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visitChildren(AbstractParseTreeVisitor.java:46) > ~[antlr4-runtime-4.9.3.jar:4.9.3] >at org.apache.hive.hplsql.Exec.visitProgram(Exec.java:999) > ~[hive-hplsql-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT] >at org.apache.hive.hplsql.Exec.visitProgram(Exec.java:96) > ~[hive-hplsql-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT] >at >
[jira] [Updated] (HIVE-28249) Parquet legacy timezone conversion converts march 1st to 29th feb and fails with not a leap year exception
[ https://issues.apache.org/jira/browse/HIVE-28249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-28249: -- Labels: pull-request-available (was: ) > Parquet legacy timezone conversion converts march 1st to 29th feb and fails > with not a leap year exception > -- > > Key: HIVE-28249 > URL: https://issues.apache.org/jira/browse/HIVE-28249 > Project: Hive > Issue Type: Task >Reporter: Simhadri Govindappa >Assignee: Simhadri Govindappa >Priority: Major > Labels: pull-request-available > > When handling legacy timezone conversions in parquet, 'February 29' year > '200' is an edge case. > This is because, according to this: [https://www.lanl.gov/Caesar/node202.html] > The Julian day for 200 CE/02/29 in the Julian calendar is different from the > Julian day in Gregorian Calendar . > |Date (BC/AD)|Date (CE)|Julian Day|Julian Day| > | | |(Julian Calendar)|(Gregorian Calendar)| > |200 AD/02/28|200 CE/02/28|1794166|1794167| > |200 AD/02/29|200 CE/02/29|1794167|1794168| > |200 AD/03/01|200 CE/03/01|1794168|1794168| > As a result since hive stores timestamp in UTC, when converting 200 CE/03/01 > between timezones, hive runs into an exception and fails with "not a leap > year exception" for 29th Feb 200 even if the actual record inserted was 200 > CE/03/01 in Asia/Singapore timezone. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-28249) Parquet legacy timezone conversion converts march 1st to 29th feb and fails with not a leap year exception
[ https://issues.apache.org/jira/browse/HIVE-28249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simhadri Govindappa updated HIVE-28249: --- Description: When handling legacy timezone conversions in parquet, 'February 29' year '200' is an edge case. This is because, according to this: [https://www.lanl.gov/Caesar/node202.html] The Julian day for 200 CE/02/29 in the Julian calendar is different from the Julian day in Gregorian Calendar . |Date (BC/AD)|Date (CE)|Julian Day|Julian Day| | | |(Julian Calendar)|(Gregorian Calendar)| |200 AD/02/28|200 CE/02/28|1794166|1794167| |200 AD/02/29|200 CE/02/29|1794167|1794168| |200 AD/03/01|200 CE/03/01|1794168|1794168| As a result since hive stores timestamp in UTC, when converting 200 CE/03/01 between timezones, hive runs into an exception and fails with "not a leap year exception" for 29th Feb 200 even if the actual record inserted was 200 CE/03/01 in Asia/Singapore timezone. > Parquet legacy timezone conversion converts march 1st to 29th feb and fails > with not a leap year exception > -- > > Key: HIVE-28249 > URL: https://issues.apache.org/jira/browse/HIVE-28249 > Project: Hive > Issue Type: Task >Reporter: Simhadri Govindappa >Assignee: Simhadri Govindappa >Priority: Major > > When handling legacy timezone conversions in parquet, 'February 29' year > '200' is an edge case. > This is because, according to this: [https://www.lanl.gov/Caesar/node202.html] > The Julian day for 200 CE/02/29 in the Julian calendar is different from the > Julian day in Gregorian Calendar . > |Date (BC/AD)|Date (CE)|Julian Day|Julian Day| > | | |(Julian Calendar)|(Gregorian Calendar)| > |200 AD/02/28|200 CE/02/28|1794166|1794167| > |200 AD/02/29|200 CE/02/29|1794167|1794168| > |200 AD/03/01|200 CE/03/01|1794168|1794168| > As a result since hive stores timestamp in UTC, when converting 200 CE/03/01 > between timezones, hive runs into an exception and fails with "not a leap > year exception" for 29th Feb 200 even if the actual record inserted was 200 > CE/03/01 in Asia/Singapore timezone. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-28249) Parquet legacy timezone conversion converts march 1st to 29th feb and fails with not a leap year exception
Simhadri Govindappa created HIVE-28249: -- Summary: Parquet legacy timezone conversion converts march 1st to 29th feb and fails with not a leap year exception Key: HIVE-28249 URL: https://issues.apache.org/jira/browse/HIVE-28249 Project: Hive Issue Type: Task Reporter: Simhadri Govindappa Assignee: Simhadri Govindappa -- This message was sent by Atlassian Jira (v8.20.10#820010)