[jira] [Created] (HIVE-27964) Support drop stats similar to Impala
Naresh P R created HIVE-27964: - Summary: Support drop stats similar to Impala Key: HIVE-27964 URL: https://issues.apache.org/jira/browse/HIVE-27964 Project: Hive Issue Type: New Feature Reporter: Naresh P R Hive should support drop stats similar to impala. https://impala.apache.org/docs/build/html/topics/impala_drop_stats.html -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-26537) Deprecate older APIs in the HMS
[ https://issues.apache.org/jira/browse/HIVE-26537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17799112#comment-17799112 ] Sai Hemanth Gantasala commented on HIVE-26537: -- [~dkuzmenko] - Thanks for the reminder. Revived the patch. Will get it reviewed again. > Deprecate older APIs in the HMS > --- > > Key: HIVE-26537 > URL: https://issues.apache.org/jira/browse/HIVE-26537 > Project: Hive > Issue Type: Improvement >Affects Versions: 4.0.0-alpha-1, 4.0.0-alpha-2 >Reporter: Sai Hemanth Gantasala >Assignee: Sai Hemanth Gantasala >Priority: Critical > Labels: pull-request-available > Time Spent: 7h 10m > Remaining Estimate: 0h > > This Jira is to track the clean-up(deprecate older APIs and point the HMS > client to the newer APIs) work in the hive metastore server. > More details will be added here soon. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27492) HPL/SQL built-in functions like sysdate not working
[ https://issues.apache.org/jira/browse/HIVE-27492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-27492: -- Labels: pull-request-available (was: ) > HPL/SQL built-in functions like sysdate not working > --- > > Key: HIVE-27492 > URL: https://issues.apache.org/jira/browse/HIVE-27492 > Project: Hive > Issue Type: Bug > Components: hpl/sql >Reporter: Dayakar M >Assignee: Dayakar M >Priority: Major > Labels: pull-request-available > > HPL/SQL built-in functions like sysdate not working > > {noformat} > select sysdate; > / > ERROR : FAILED: SemanticException [Error 10004]: Line 1:7 Invalid table alias > or column reference 'sysdate': (possible column names are: ) > org.apache.hadoop.hive.ql.parse.SemanticException: Line 1:7 Invalid table > alias or column reference 'sysdate': (possible column names are: ){noformat} -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-27386) Avoid duplicate audit log in cleanupHandlerContext
[ https://issues.apache.org/jira/browse/HIVE-27386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wechar resolved HIVE-27386. --- Fix Version/s: 4.0.0-beta-1 Resolution: Fixed > Avoid duplicate audit log in cleanupHandlerContext > -- > > Key: HIVE-27386 > URL: https://issues.apache.org/jira/browse/HIVE-27386 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Wechar >Assignee: Wechar >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0-beta-1 > > Attachments: screenshot-1.png > > > {{HMSHandler#cleanupHandlerContext}} will be called twice if > {{IMetaStoreClient#close}} was called. In this case, where will create two > same audit log: Done cleaning up thread local RawStore. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-27963) Build failure when license-maven-plugin downloads bsd-license.php
[ https://issues.apache.org/jira/browse/HIVE-27963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stamatis Zampetakis resolved HIVE-27963. Fix Version/s: 4.1.0 Resolution: Fixed Fixed in [https://github.com/apache/hive/commit/6671e1a91f288d8552840a2517912e5820d0204b.] Thanks for the patch [~akshatm] and [~ayushtkn] for the review! > Build failure when license-maven-plugin downloads bsd-license.php > - > > Key: HIVE-27963 > URL: https://issues.apache.org/jira/browse/HIVE-27963 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Akshat Mathur >Assignee: Akshat Mathur >Priority: Major > Labels: pull-request-available > Fix For: 4.1.0 > > > Test are failing due to the following issue: > > [ERROR] Failed to execute goal > org.codehaus.mojo:license-maven-plugin:2.1.0:download-licenses > (license-fetch) on project hive-packaging: Execution license-fetch of goal > org.codehaus.mojo:license-maven-plugin:2.1.0:download-licenses failed: URL ' > [http://www.opensource.org/licenses/bsd-license.php] > ' should belong to licenseUrlFileName having key > 'bsd-2-clause-license-bsd-2-clause.html' together with URLs ' > [https://opensource.org/licenses/BSD-2-Clause] > ' -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-27919) Constant reduction in CBO does not work for FROM_UNIXTIME, DATE_ADD, DATE_SUB, TO_UNIX_TIMESTAMP
[ https://issues.apache.org/jira/browse/HIVE-27919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stamatis Zampetakis resolved HIVE-27919. Fix Version/s: 4.1.0 Resolution: Fixed Fixed in https://github.com/apache/hive/commit/36ce858163a19e29eafe4a8d3307191bc28fc175. Thanks for the reviews [~akshatm] [~kkasa]! > Constant reduction in CBO does not work for FROM_UNIXTIME, DATE_ADD, > DATE_SUB, TO_UNIX_TIMESTAMP > > > Key: HIVE-27919 > URL: https://issues.apache.org/jira/browse/HIVE-27919 > Project: Hive > Issue Type: Improvement > Components: CBO >Affects Versions: 4.0.0-beta-1 >Reporter: Stamatis Zampetakis >Assignee: Stamatis Zampetakis >Priority: Major > Labels: pull-request-available > Fix For: 4.1.0 > > > Constant reduction (aka. constant folding) does not work in CBO for > FROM_UNIXTIME, DATE_ADD, and DATE_SUB functions. > Below, we outline the result of EXPLAIN CBO and plain EXPLAIN using some > trivial constant inputs. Notice, that constant reduction is not present in > the CBO plan but it is applied later on by the physical optimizer. > {code:sql} > SELECT FROM_UNIXTIME(1701088643) > {code} > {noformat} > CBO PLAN: > HiveProject(_o__c0=[FROM_UNIXTIME(1701088643)]) > HiveTableScan(table=[[_dummy_database, _dummy_table]], > table:alias=[_dummy_table]) > STAGE PLANS: > Stage: Stage-0 > Fetch Operator > limit: -1 > Processor Tree: > TableScan > alias: _dummy_table > Row Limit Per Split: 1 > Select Operator > expressions: '2023-11-27 04:37:23' (type: string) > outputColumnNames: _col0 > ListSink > {noformat} > {code:sql} > SELECT DATE_ADD('2023-01-01', 1) > {code} > {noformat} > CBO PLAN: > HiveProject(_o__c0=[DATE_ADD(_UTF-16LE'2023-01-01':VARCHAR(2147483647) > CHARACTER SET "UTF-16LE", 1)]) > HiveTableScan(table=[[_dummy_database, _dummy_table]], > table:alias=[_dummy_table]) > STAGE PLANS: > Stage: Stage-0 > Fetch Operator > limit: -1 > Processor Tree: > TableScan > alias: _dummy_table > Row Limit Per Split: 1 > Select Operator > expressions: DATE'2023-01-02' (type: date) > outputColumnNames: _col0 > ListSink > {noformat} > {code:sql} > SELECT DATE_SUB('2023-01-01', 1) > {code} > {noformat} > CBO PLAN: > HiveProject(_o__c0=[DATE_SUB(_UTF-16LE'2023-01-01':VARCHAR(2147483647) > CHARACTER SET "UTF-16LE", 1)]) > HiveTableScan(table=[[_dummy_database, _dummy_table]], > table:alias=[_dummy_table]) > STAGE PLANS: > Stage: Stage-0 > Fetch Operator > limit: -1 > Processor Tree: > TableScan > alias: _dummy_table > Row Limit Per Split: 1 > Select Operator > expressions: DATE'2022-12-31' (type: date) > outputColumnNames: _col0 > ListSink > {noformat} > The reason is that all the functions at the moment are (wrongly) declared as > a dynamic functions and this by default prevents constant folding. The > functions are not dynamic since they don't depend on context variables and > require one or more parameters. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-27963) Build failure when license-maven-plugin downloads bsd-license.php
[ https://issues.apache.org/jira/browse/HIVE-27963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17798902#comment-17798902 ] Akshat Mathur commented on HIVE-27963: -- Thanks [~zabetak], I was running it from hive-packaging, I have updated the PR. It was intresting to checkout this part of the code :), Thank again for your help. > Build failure when license-maven-plugin downloads bsd-license.php > - > > Key: HIVE-27963 > URL: https://issues.apache.org/jira/browse/HIVE-27963 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Akshat Mathur >Assignee: Akshat Mathur >Priority: Major > Labels: pull-request-available > > Test are failing due to the following issue: > > [ERROR] Failed to execute goal > org.codehaus.mojo:license-maven-plugin:2.1.0:download-licenses > (license-fetch) on project hive-packaging: Execution license-fetch of goal > org.codehaus.mojo:license-maven-plugin:2.1.0:download-licenses failed: URL ' > [http://www.opensource.org/licenses/bsd-license.php] > ' should belong to licenseUrlFileName having key > 'bsd-2-clause-license-bsd-2-clause.html' together with URLs ' > [https://opensource.org/licenses/BSD-2-Clause] > ' -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27963) Build failure when license-maven-plugin downloads bsd-license.php
[ https://issues.apache.org/jira/browse/HIVE-27963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stamatis Zampetakis updated HIVE-27963: --- Summary: Build failure when license-maven-plugin downloads bsd-license.php (was: Fix license fetch issue with license-maven-plugin) > Build failure when license-maven-plugin downloads bsd-license.php > - > > Key: HIVE-27963 > URL: https://issues.apache.org/jira/browse/HIVE-27963 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Akshat Mathur >Assignee: Akshat Mathur >Priority: Major > Labels: pull-request-available > > Test are failing due to the following issue: > > [ERROR] Failed to execute goal > org.codehaus.mojo:license-maven-plugin:2.1.0:download-licenses > (license-fetch) on project hive-packaging: Execution license-fetch of goal > org.codehaus.mojo:license-maven-plugin:2.1.0:download-licenses failed: URL ' > [http://www.opensource.org/licenses/bsd-license.php] > ' should belong to licenseUrlFileName having key > 'bsd-2-clause-license-bsd-2-clause.html' together with URLs ' > [https://opensource.org/licenses/BSD-2-Clause] > ' -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (HIVE-27963) Fix license fetch issue with license-maven-plugin
[ https://issues.apache.org/jira/browse/HIVE-27963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17798892#comment-17798892 ] Akshat Mathur edited comment on HIVE-27963 at 12/20/23 10:01 AM: - [~zabetak] I tried adding {code:java} https?://(www\.)?opensource.org/licenses/bsd-license.php https?://(www\.)?opensource.org/licenses/BSD-2-Clause {code} Post which I started getting assembly mismatch errors: {code:java} [ERROR] Failed to execute goal org.apache.maven.plugins:maven-assembly-plugin:3.2.0:single (assemble) on project hive-packaging: Assembly is incorrectly configured: bin: Assembly is incorrectly configured: bin: [ERROR] Assembly: bin is not configured correctly: One or more filters had unmatched criteria. Check debug log for more information. [ERROR] -> [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-assembly-plugin:3.2.0:single (assemble) on project hive-packaging: Assembly is incorrectly configured: bin {code} Do you have any idea around this? was (Author: JIRAUSER298271): [~zabetak] I tried adding {code:java} https?://(www\.)?opensource.org/licenses/bsd-license.php https?://(www\.)?opensource.org/licenses/BSD-2-Clause {code} Post which I started getting assembly mismatch errors: {code:java} [ERROR] Failed to execute goal org.apache.maven.plugins:maven-assembly-plugin:3.2.0:single (assemble) on project hive-packaging: Assembly is incorrectly configured: bin: Assembly is incorrectly configured: bin: [ERROR] Assembly: bin is not configured correctly: One or more filters had unmatched criteria. Check debug log for more information. [ERROR] -> [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-assembly-plugin:3.2.0:single (assemble) on project hive-packaging: Assembly is incorrectly configured: bin {code} Do you have nay idea around this? > Fix license fetch issue with license-maven-plugin > - > > Key: HIVE-27963 > URL: https://issues.apache.org/jira/browse/HIVE-27963 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Akshat Mathur >Assignee: Akshat Mathur >Priority: Major > Labels: pull-request-available > > Test are failing due to the following issue: > > [ERROR] Failed to execute goal > org.codehaus.mojo:license-maven-plugin:2.1.0:download-licenses > (license-fetch) on project hive-packaging: Execution license-fetch of goal > org.codehaus.mojo:license-maven-plugin:2.1.0:download-licenses failed: URL ' > [http://www.opensource.org/licenses/bsd-license.php] > ' should belong to licenseUrlFileName having key > 'bsd-2-clause-license-bsd-2-clause.html' together with URLs ' > [https://opensource.org/licenses/BSD-2-Clause] > ' -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-27963) Fix license fetch issue with license-maven-plugin
[ https://issues.apache.org/jira/browse/HIVE-27963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17798895#comment-17798895 ] Stamatis Zampetakis commented on HIVE-27963: The assembly error that you get is probably unrelated and has to do with how and from where you run the mvn command. I tried mvn clean install -DskipTests -Pdist from the root and the build passed for me. > Fix license fetch issue with license-maven-plugin > - > > Key: HIVE-27963 > URL: https://issues.apache.org/jira/browse/HIVE-27963 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Akshat Mathur >Assignee: Akshat Mathur >Priority: Major > Labels: pull-request-available > > Test are failing due to the following issue: > > [ERROR] Failed to execute goal > org.codehaus.mojo:license-maven-plugin:2.1.0:download-licenses > (license-fetch) on project hive-packaging: Execution license-fetch of goal > org.codehaus.mojo:license-maven-plugin:2.1.0:download-licenses failed: URL ' > [http://www.opensource.org/licenses/bsd-license.php] > ' should belong to licenseUrlFileName having key > 'bsd-2-clause-license-bsd-2-clause.html' together with URLs ' > [https://opensource.org/licenses/BSD-2-Clause] > ' -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-27963) Fix license fetch issue with license-maven-plugin
[ https://issues.apache.org/jira/browse/HIVE-27963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17798892#comment-17798892 ] Akshat Mathur commented on HIVE-27963: -- [~zabetak] I tried adding {code:java} https?://(www\.)?opensource.org/licenses/bsd-license.php https?://(www\.)?opensource.org/licenses/BSD-2-Clause {code} Post which I started getting assembly mismatch errors: {code:java} [ERROR] Failed to execute goal org.apache.maven.plugins:maven-assembly-plugin:3.2.0:single (assemble) on project hive-packaging: Assembly is incorrectly configured: bin: Assembly is incorrectly configured: bin: [ERROR] Assembly: bin is not configured correctly: One or more filters had unmatched criteria. Check debug log for more information. [ERROR] -> [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-assembly-plugin:3.2.0:single (assemble) on project hive-packaging: Assembly is incorrectly configured: bin {code} Do you have nay idea around this? > Fix license fetch issue with license-maven-plugin > - > > Key: HIVE-27963 > URL: https://issues.apache.org/jira/browse/HIVE-27963 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Akshat Mathur >Assignee: Akshat Mathur >Priority: Major > Labels: pull-request-available > > Test are failing due to the following issue: > > [ERROR] Failed to execute goal > org.codehaus.mojo:license-maven-plugin:2.1.0:download-licenses > (license-fetch) on project hive-packaging: Execution license-fetch of goal > org.codehaus.mojo:license-maven-plugin:2.1.0:download-licenses failed: URL ' > [http://www.opensource.org/licenses/bsd-license.php] > ' should belong to licenseUrlFileName having key > 'bsd-2-clause-license-bsd-2-clause.html' together with URLs ' > [https://opensource.org/licenses/BSD-2-Clause] > ' -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-27963) Fix license fetch issue with license-maven-plugin
[ https://issues.apache.org/jira/browse/HIVE-27963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1779#comment-1779 ] Stamatis Zampetakis commented on HIVE-27963: I don't see any changes around the dk.brics.automaton:automaton and com.github.luben:zstd-jni dependencies that use the BSD URLS mentioned above so the only thing that comes to mind is that maybe the content of the respective URL changed recently. > Fix license fetch issue with license-maven-plugin > - > > Key: HIVE-27963 > URL: https://issues.apache.org/jira/browse/HIVE-27963 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Akshat Mathur >Assignee: Akshat Mathur >Priority: Major > Labels: pull-request-available > > Test are failing due to the following issue: > > [ERROR] Failed to execute goal > org.codehaus.mojo:license-maven-plugin:2.1.0:download-licenses > (license-fetch) on project hive-packaging: Execution license-fetch of goal > org.codehaus.mojo:license-maven-plugin:2.1.0:download-licenses failed: URL ' > [http://www.opensource.org/licenses/bsd-license.php] > ' should belong to licenseUrlFileName having key > 'bsd-2-clause-license-bsd-2-clause.html' together with URLs ' > [https://opensource.org/licenses/BSD-2-Clause] > ' -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-27963) Fix license fetch issue with license-maven-plugin
[ https://issues.apache.org/jira/browse/HIVE-27963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17798887#comment-17798887 ] Stamatis Zampetakis commented on HIVE-27963: I suppose adding the following in the configuration of the license-maven-plugin should resolve the problem. {noformat} https?://(www\.)?opensource.org/licenses/bsd-license.php https?://(www\.)?opensource.org/licenses/BSD-2-Clause {noformat} > Fix license fetch issue with license-maven-plugin > - > > Key: HIVE-27963 > URL: https://issues.apache.org/jira/browse/HIVE-27963 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Akshat Mathur >Assignee: Akshat Mathur >Priority: Major > Labels: pull-request-available > > Test are failing due to the following issue: > > [ERROR] Failed to execute goal > org.codehaus.mojo:license-maven-plugin:2.1.0:download-licenses > (license-fetch) on project hive-packaging: Execution license-fetch of goal > org.codehaus.mojo:license-maven-plugin:2.1.0:download-licenses failed: URL ' > [http://www.opensource.org/licenses/bsd-license.php] > ' should belong to licenseUrlFileName having key > 'bsd-2-clause-license-bsd-2-clause.html' together with URLs ' > [https://opensource.org/licenses/BSD-2-Clause] > ' -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-27963) Fix license fetch issue with license-maven-plugin
[ https://issues.apache.org/jira/browse/HIVE-27963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17798886#comment-17798886 ] Stamatis Zampetakis commented on HIVE-27963: The http://www.opensource.org/licenses/bsd-license.php url redirects to https://opensource.org/license/bsd-2-clause/ so I guess the plugin complains because we don't want to store the same license twice. > Fix license fetch issue with license-maven-plugin > - > > Key: HIVE-27963 > URL: https://issues.apache.org/jira/browse/HIVE-27963 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Akshat Mathur >Assignee: Akshat Mathur >Priority: Major > Labels: pull-request-available > > Test are failing due to the following issue: > > [ERROR] Failed to execute goal > org.codehaus.mojo:license-maven-plugin:2.1.0:download-licenses > (license-fetch) on project hive-packaging: Execution license-fetch of goal > org.codehaus.mojo:license-maven-plugin:2.1.0:download-licenses failed: URL ' > [http://www.opensource.org/licenses/bsd-license.php] > ' should belong to licenseUrlFileName having key > 'bsd-2-clause-license-bsd-2-clause.html' together with URLs ' > [https://opensource.org/licenses/BSD-2-Clause] > ' -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Reopened] (HIVE-27858) OOM happens when selecting many columns and JOIN.
[ https://issues.apache.org/jira/browse/HIVE-27858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko reopened HIVE-27858: --- > OOM happens when selecting many columns and JOIN. > -- > > Key: HIVE-27858 > URL: https://issues.apache.org/jira/browse/HIVE-27858 > Project: Hive > Issue Type: Bug > Components: Query Planning >Affects Versions: 4.0.0-beta-1 >Reporter: Ryu Kobayashi >Priority: Critical > Labels: hive-4.0.0-must > Fix For: Not Applicable > > Attachments: HIVE-27858.full.q, HIVE-27858.less.columns.q, ddl.sql, > query.sql > > > OOM happens when executing [^query.sql] using a table in [^ddl.sql]. These > did not happen in Hive 2 previously. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-27858) OOM happens when selecting many columns and JOIN.
[ https://issues.apache.org/jira/browse/HIVE-27858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko resolved HIVE-27858. --- Fix Version/s: Not Applicable Resolution: Abandoned > OOM happens when selecting many columns and JOIN. > -- > > Key: HIVE-27858 > URL: https://issues.apache.org/jira/browse/HIVE-27858 > Project: Hive > Issue Type: Bug > Components: Query Planning >Affects Versions: 4.0.0-beta-1 >Reporter: Ryu Kobayashi >Priority: Critical > Labels: hive-4.0.0-must > Fix For: Not Applicable > > Attachments: HIVE-27858.full.q, HIVE-27858.less.columns.q, ddl.sql, > query.sql > > > OOM happens when executing [^query.sql] using a table in [^ddl.sql]. These > did not happen in Hive 2 previously. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27775) DirectSQL and JDO results are different when fetching partitions by timestamp in DST shift
[ https://issues.apache.org/jira/browse/HIVE-27775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-27775: -- Status: Patch Available (was: Open) > DirectSQL and JDO results are different when fetching partitions by timestamp > in DST shift > -- > > Key: HIVE-27775 > URL: https://issues.apache.org/jira/browse/HIVE-27775 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Affects Versions: 4.0.0-beta-1 >Reporter: Stamatis Zampetakis >Assignee: Zhihua Deng >Priority: Critical > Labels: pull-request-available > > DirectSQL and JDO results are different when fetching partitions by timestamp > in DST shift. > {code:sql} > --! qt:timezone:Europe/Paris > CREATE EXTERNAL TABLE payments (card string) PARTITIONED BY(txn_datetime > TIMESTAMP) STORED AS ORC; > INSERT into payments VALUES('---', '2023-03-26 02:30:00'); > SELECT * FROM payments WHERE txn_datetime = '2023-03-26 02:30:00'; > {code} > The '2023-03-26 02:30:00' is a timestamp that in Europe/Paris timezone falls > exactly in the middle of the DST shift. In this particular timezone this date > time never really exists since we are jumping directly from 02:00:00 to > 03:00:00. However, the TIMESTAMP data type in Hive is timezone agnostic > (https://cwiki.apache.org/confluence/display/Hive/Different+TIMESTAMP+types) > so it is a perfectly valid timestamp that can be inserted in a table and we > must be able to recover it back. > For the SELECT query above, partition pruning kicks in and calls the > ObjectStore#getPartitionsByExpr method in order to fetch the respective > partitions matching the timestamp from HMS. > The tests however reveal that DirectSQL and JDO paths are not returning the > same results leading to an exception when VerifyingObjectStore is used. > According to the error below DirectSQL is able to recover one partition from > HMS (expected) while JDO/ORM returns empty (not expected). > {noformat} > 2023-10-06T03:51:19,406 ERROR [80252df4-3fdc-4971-badf-ad67ce8567c7 main] > metastore.VerifyingObjectStore: Lists are not the same size: SQL 1, ORM 0 > 2023-10-06T03:51:19,409 ERROR [80252df4-3fdc-4971-badf-ad67ce8567c7 main] > metastore.RetryingHMSHandler: MetaException(message:Lists are not the same > size: SQL 1, ORM 0) > at > org.apache.hadoop.hive.metastore.VerifyingObjectStore.verifyLists(VerifyingObjectStore.java:148) > at > org.apache.hadoop.hive.metastore.VerifyingObjectStore.getPartitionsByExpr(VerifyingObjectStore.java:88) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97) > at com.sun.proxy.$Proxy57.getPartitionsByExpr(Unknown Source) > at > org.apache.hadoop.hive.metastore.HMSHandler.get_partitions_spec_by_expr(HMSHandler.java:7330) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:98) > at > org.apache.hadoop.hive.metastore.AbstractHMSHandlerProxy.invoke(AbstractHMSHandlerProxy.java:82) > at com.sun.proxy.$Proxy59.get_partitions_spec_by_expr(Unknown Source) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartitionsSpecByExprInternal(HiveMetaStoreClient.java:2472) > at > org.apache.hadoop.hive.ql.metadata.HiveMetaStoreClientWithLocalCache.getPartitionsSpecByExprInternal(HiveMetaStoreClientWithLocalCache.java:396) > at > org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.getPartitionsSpecByExprInternal(SessionHiveMetaStoreClient.java:2279) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.listPartitionsSpecByExpr(HiveMetaStoreClient.java:2484) > at > org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.listPartitionsSpecByExpr(SessionHiveMetaStoreClient.java:1346) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at
[jira] [Updated] (HIVE-27775) DirectSQL and JDO results are different when fetching partitions by timestamp in DST shift
[ https://issues.apache.org/jira/browse/HIVE-27775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-27775: -- Status: In Progress (was: Patch Available) > DirectSQL and JDO results are different when fetching partitions by timestamp > in DST shift > -- > > Key: HIVE-27775 > URL: https://issues.apache.org/jira/browse/HIVE-27775 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Affects Versions: 4.0.0-beta-1 >Reporter: Stamatis Zampetakis >Assignee: Zhihua Deng >Priority: Critical > Labels: pull-request-available > > DirectSQL and JDO results are different when fetching partitions by timestamp > in DST shift. > {code:sql} > --! qt:timezone:Europe/Paris > CREATE EXTERNAL TABLE payments (card string) PARTITIONED BY(txn_datetime > TIMESTAMP) STORED AS ORC; > INSERT into payments VALUES('---', '2023-03-26 02:30:00'); > SELECT * FROM payments WHERE txn_datetime = '2023-03-26 02:30:00'; > {code} > The '2023-03-26 02:30:00' is a timestamp that in Europe/Paris timezone falls > exactly in the middle of the DST shift. In this particular timezone this date > time never really exists since we are jumping directly from 02:00:00 to > 03:00:00. However, the TIMESTAMP data type in Hive is timezone agnostic > (https://cwiki.apache.org/confluence/display/Hive/Different+TIMESTAMP+types) > so it is a perfectly valid timestamp that can be inserted in a table and we > must be able to recover it back. > For the SELECT query above, partition pruning kicks in and calls the > ObjectStore#getPartitionsByExpr method in order to fetch the respective > partitions matching the timestamp from HMS. > The tests however reveal that DirectSQL and JDO paths are not returning the > same results leading to an exception when VerifyingObjectStore is used. > According to the error below DirectSQL is able to recover one partition from > HMS (expected) while JDO/ORM returns empty (not expected). > {noformat} > 2023-10-06T03:51:19,406 ERROR [80252df4-3fdc-4971-badf-ad67ce8567c7 main] > metastore.VerifyingObjectStore: Lists are not the same size: SQL 1, ORM 0 > 2023-10-06T03:51:19,409 ERROR [80252df4-3fdc-4971-badf-ad67ce8567c7 main] > metastore.RetryingHMSHandler: MetaException(message:Lists are not the same > size: SQL 1, ORM 0) > at > org.apache.hadoop.hive.metastore.VerifyingObjectStore.verifyLists(VerifyingObjectStore.java:148) > at > org.apache.hadoop.hive.metastore.VerifyingObjectStore.getPartitionsByExpr(VerifyingObjectStore.java:88) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97) > at com.sun.proxy.$Proxy57.getPartitionsByExpr(Unknown Source) > at > org.apache.hadoop.hive.metastore.HMSHandler.get_partitions_spec_by_expr(HMSHandler.java:7330) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:98) > at > org.apache.hadoop.hive.metastore.AbstractHMSHandlerProxy.invoke(AbstractHMSHandlerProxy.java:82) > at com.sun.proxy.$Proxy59.get_partitions_spec_by_expr(Unknown Source) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartitionsSpecByExprInternal(HiveMetaStoreClient.java:2472) > at > org.apache.hadoop.hive.ql.metadata.HiveMetaStoreClientWithLocalCache.getPartitionsSpecByExprInternal(HiveMetaStoreClientWithLocalCache.java:396) > at > org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.getPartitionsSpecByExprInternal(SessionHiveMetaStoreClient.java:2279) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.listPartitionsSpecByExpr(HiveMetaStoreClient.java:2484) > at > org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.listPartitionsSpecByExpr(SessionHiveMetaStoreClient.java:1346) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at
[jira] [Commented] (HIVE-27963) Fix license fetch issue with license-maven-plugin
[ https://issues.apache.org/jira/browse/HIVE-27963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17798885#comment-17798885 ] Stamatis Zampetakis commented on HIVE-27963: The plugin complains about a license having the key "bsd-2-clause-license-bsd-2-clause.html". It seems that this is coming from zstd-jni dependency. {noformat} [DEBUG] Downloading license(s) for project com.github.luben:zstd-jni [DEBUG] About to download 'https://opensource.org/licenses/BSD-2-Clause' [DEBUG] Downloading 'https://opensource.org/licenses/BSD-2-Clause' -> '/home/stamatis/Projects/Apache/hive/packaging/target/generated-resources/licenses/bsd-2-clause-license-bsd-2-clause.html' {noformat} > Fix license fetch issue with license-maven-plugin > - > > Key: HIVE-27963 > URL: https://issues.apache.org/jira/browse/HIVE-27963 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Akshat Mathur >Assignee: Akshat Mathur >Priority: Major > Labels: pull-request-available > > Test are failing due to the following issue: > > [ERROR] Failed to execute goal > org.codehaus.mojo:license-maven-plugin:2.1.0:download-licenses > (license-fetch) on project hive-packaging: Execution license-fetch of goal > org.codehaus.mojo:license-maven-plugin:2.1.0:download-licenses failed: URL ' > [http://www.opensource.org/licenses/bsd-license.php] > ' should belong to licenseUrlFileName having key > 'bsd-2-clause-license-bsd-2-clause.html' together with URLs ' > [https://opensource.org/licenses/BSD-2-Clause] > ' -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27775) DirectSQL and JDO results are different when fetching partitions by timestamp in DST shift
[ https://issues.apache.org/jira/browse/HIVE-27775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-27775: -- Status: Patch Available (was: Open) > DirectSQL and JDO results are different when fetching partitions by timestamp > in DST shift > -- > > Key: HIVE-27775 > URL: https://issues.apache.org/jira/browse/HIVE-27775 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Affects Versions: 4.0.0-beta-1 >Reporter: Stamatis Zampetakis >Assignee: Zhihua Deng >Priority: Critical > Labels: pull-request-available > > DirectSQL and JDO results are different when fetching partitions by timestamp > in DST shift. > {code:sql} > --! qt:timezone:Europe/Paris > CREATE EXTERNAL TABLE payments (card string) PARTITIONED BY(txn_datetime > TIMESTAMP) STORED AS ORC; > INSERT into payments VALUES('---', '2023-03-26 02:30:00'); > SELECT * FROM payments WHERE txn_datetime = '2023-03-26 02:30:00'; > {code} > The '2023-03-26 02:30:00' is a timestamp that in Europe/Paris timezone falls > exactly in the middle of the DST shift. In this particular timezone this date > time never really exists since we are jumping directly from 02:00:00 to > 03:00:00. However, the TIMESTAMP data type in Hive is timezone agnostic > (https://cwiki.apache.org/confluence/display/Hive/Different+TIMESTAMP+types) > so it is a perfectly valid timestamp that can be inserted in a table and we > must be able to recover it back. > For the SELECT query above, partition pruning kicks in and calls the > ObjectStore#getPartitionsByExpr method in order to fetch the respective > partitions matching the timestamp from HMS. > The tests however reveal that DirectSQL and JDO paths are not returning the > same results leading to an exception when VerifyingObjectStore is used. > According to the error below DirectSQL is able to recover one partition from > HMS (expected) while JDO/ORM returns empty (not expected). > {noformat} > 2023-10-06T03:51:19,406 ERROR [80252df4-3fdc-4971-badf-ad67ce8567c7 main] > metastore.VerifyingObjectStore: Lists are not the same size: SQL 1, ORM 0 > 2023-10-06T03:51:19,409 ERROR [80252df4-3fdc-4971-badf-ad67ce8567c7 main] > metastore.RetryingHMSHandler: MetaException(message:Lists are not the same > size: SQL 1, ORM 0) > at > org.apache.hadoop.hive.metastore.VerifyingObjectStore.verifyLists(VerifyingObjectStore.java:148) > at > org.apache.hadoop.hive.metastore.VerifyingObjectStore.getPartitionsByExpr(VerifyingObjectStore.java:88) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97) > at com.sun.proxy.$Proxy57.getPartitionsByExpr(Unknown Source) > at > org.apache.hadoop.hive.metastore.HMSHandler.get_partitions_spec_by_expr(HMSHandler.java:7330) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:98) > at > org.apache.hadoop.hive.metastore.AbstractHMSHandlerProxy.invoke(AbstractHMSHandlerProxy.java:82) > at com.sun.proxy.$Proxy59.get_partitions_spec_by_expr(Unknown Source) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartitionsSpecByExprInternal(HiveMetaStoreClient.java:2472) > at > org.apache.hadoop.hive.ql.metadata.HiveMetaStoreClientWithLocalCache.getPartitionsSpecByExprInternal(HiveMetaStoreClientWithLocalCache.java:396) > at > org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.getPartitionsSpecByExprInternal(SessionHiveMetaStoreClient.java:2279) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.listPartitionsSpecByExpr(HiveMetaStoreClient.java:2484) > at > org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.listPartitionsSpecByExpr(SessionHiveMetaStoreClient.java:1346) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at
[jira] [Updated] (HIVE-27775) DirectSQL and JDO results are different when fetching partitions by timestamp in DST shift
[ https://issues.apache.org/jira/browse/HIVE-27775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-27775: -- Status: Open (was: Patch Available) > DirectSQL and JDO results are different when fetching partitions by timestamp > in DST shift > -- > > Key: HIVE-27775 > URL: https://issues.apache.org/jira/browse/HIVE-27775 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Affects Versions: 4.0.0-beta-1 >Reporter: Stamatis Zampetakis >Assignee: Zhihua Deng >Priority: Critical > Labels: pull-request-available > > DirectSQL and JDO results are different when fetching partitions by timestamp > in DST shift. > {code:sql} > --! qt:timezone:Europe/Paris > CREATE EXTERNAL TABLE payments (card string) PARTITIONED BY(txn_datetime > TIMESTAMP) STORED AS ORC; > INSERT into payments VALUES('---', '2023-03-26 02:30:00'); > SELECT * FROM payments WHERE txn_datetime = '2023-03-26 02:30:00'; > {code} > The '2023-03-26 02:30:00' is a timestamp that in Europe/Paris timezone falls > exactly in the middle of the DST shift. In this particular timezone this date > time never really exists since we are jumping directly from 02:00:00 to > 03:00:00. However, the TIMESTAMP data type in Hive is timezone agnostic > (https://cwiki.apache.org/confluence/display/Hive/Different+TIMESTAMP+types) > so it is a perfectly valid timestamp that can be inserted in a table and we > must be able to recover it back. > For the SELECT query above, partition pruning kicks in and calls the > ObjectStore#getPartitionsByExpr method in order to fetch the respective > partitions matching the timestamp from HMS. > The tests however reveal that DirectSQL and JDO paths are not returning the > same results leading to an exception when VerifyingObjectStore is used. > According to the error below DirectSQL is able to recover one partition from > HMS (expected) while JDO/ORM returns empty (not expected). > {noformat} > 2023-10-06T03:51:19,406 ERROR [80252df4-3fdc-4971-badf-ad67ce8567c7 main] > metastore.VerifyingObjectStore: Lists are not the same size: SQL 1, ORM 0 > 2023-10-06T03:51:19,409 ERROR [80252df4-3fdc-4971-badf-ad67ce8567c7 main] > metastore.RetryingHMSHandler: MetaException(message:Lists are not the same > size: SQL 1, ORM 0) > at > org.apache.hadoop.hive.metastore.VerifyingObjectStore.verifyLists(VerifyingObjectStore.java:148) > at > org.apache.hadoop.hive.metastore.VerifyingObjectStore.getPartitionsByExpr(VerifyingObjectStore.java:88) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97) > at com.sun.proxy.$Proxy57.getPartitionsByExpr(Unknown Source) > at > org.apache.hadoop.hive.metastore.HMSHandler.get_partitions_spec_by_expr(HMSHandler.java:7330) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:98) > at > org.apache.hadoop.hive.metastore.AbstractHMSHandlerProxy.invoke(AbstractHMSHandlerProxy.java:82) > at com.sun.proxy.$Proxy59.get_partitions_spec_by_expr(Unknown Source) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartitionsSpecByExprInternal(HiveMetaStoreClient.java:2472) > at > org.apache.hadoop.hive.ql.metadata.HiveMetaStoreClientWithLocalCache.getPartitionsSpecByExprInternal(HiveMetaStoreClientWithLocalCache.java:396) > at > org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.getPartitionsSpecByExprInternal(SessionHiveMetaStoreClient.java:2279) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.listPartitionsSpecByExpr(HiveMetaStoreClient.java:2484) > at > org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.listPartitionsSpecByExpr(SessionHiveMetaStoreClient.java:1346) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at
[jira] [Updated] (HIVE-27948) Wrong results when using materialized views with non-deterministic/dynamic functions
[ https://issues.apache.org/jira/browse/HIVE-27948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-27948: -- Status: Patch Available (was: Open) > Wrong results when using materialized views with non-deterministic/dynamic > functions > > > Key: HIVE-27948 > URL: https://issues.apache.org/jira/browse/HIVE-27948 > Project: Hive > Issue Type: Bug > Components: CBO >Affects Versions: 4.0.0-beta-1 >Reporter: Stamatis Zampetakis >Assignee: Krisztian Kasa >Priority: Critical > Labels: pull-request-available > Attachments: materialized_view_unix_timestamp.q > > > There are certain SQL functions that return different results across > different executions. Usually we refer to these functions as > non-deterministic or dynamic. Some examples are: UNIX_TIMESTAMP(), > CURRENT_TIMESTAMP, CURRENT_DATE, etc. > When a materialized view definition contains such functions the queries that > are using this view may return wrong results. > Consider the following scenario where we populate the employee table with > timestamps representing the future. For making this easily reproable in > self-contained test the timestamps are only a few seconds apart. > {code:sql} > CREATE TABLE EMPS (ENAME STRING, BIRTH_EPOCH_SECS INT) STORED AS ORC > TBLPROPERTIES ('transactional'='true'); > INSERT INTO EMPS > VALUES ('Victor', UNIX_TIMESTAMP()), >('Alex', UNIX_TIMESTAMP() + 2), >('Bob', UNIX_TIMESTAMP() + 5), >('Alice', UNIX_TIMESTAMP() + 10); > CREATE MATERIALIZED VIEW v_emp AS SELECT * FROM EMPS WHERE BIRTH_EPOCH_SECS > <= UNIX_TIMESTAMP(); > {code} > When the materialized view is created it is populated with only the rows that > match the timestamp at the given time. > To demonstrate the problem run the following queries with view based > rewritting disabled and enabled. > {code:sql} > set hive.materializedview.rewriting.sql=false; > SELECT * FROM EMPS WHERE BIRTH_EPOCH_SECS <= UNIX_TIMESTAMP(); > {code} > {noformat} > Victor1702302786 > Alex 1702302788 > Bob 1702302791 > {noformat} > {code:sql} > set hive.materializedview.rewriting.sql=true; > SELECT * FROM EMPS WHERE BIRTH_EPOCH_SECS <= UNIX_TIMESTAMP(); > {code} > {noformat} > Victor1702302786 > Alex 1702302788 > {noformat} > Naturally the second query should return more rows than the first one since > UNIX_TIMESTAMP is constantly growing. However, when view based rewritting is > in use the second query will use the results from the materialized view which > are by now obsolete (missing Bob entry). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27951) hcatalog dynamic partitioning fails with partition already exist error when exist parent partitions path
[ https://issues.apache.org/jira/browse/HIVE-27951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-27951: -- Status: Patch Available (was: Open) > hcatalog dynamic partitioning fails with partition already exist error when > exist parent partitions path > > > Key: HIVE-27951 > URL: https://issues.apache.org/jira/browse/HIVE-27951 > Project: Hive > Issue Type: Bug > Components: HCatalog >Affects Versions: 4.0.0-beta-1 >Reporter: Yi Zhang >Assignee: Yi Zhang >Priority: Critical > Labels: pull-request-available > > if a table have multiple partitions (part1=x1, part2=y1), when insert into a > new partition(part1=x1, part2=y2) hcatalog FileOutputCommitterContainer > throws path already exists error > > reproduce: > create table source(id int, part1 string, part2 string); > create table target(id int) partitioned by (part1 string, part2 string) > insert into table source values (1, "x1", "y1"), (2, "x1", "y2"); > > pig -useHcatalog > A = load 'source' using org.apache.hive.hcatalog.pig.HCatLoader(); > B = filter A by (part2 == 'y1'); > // following succeeds > store B into 'target' USING org.apache.hive.hcatalog.pig.HCatStorer(); > //following fails with duplicate publishing error > C = filter A by (part2 == 'y2'); > store C into 'target' USING org.apache.hive.hcatalog.pig.HCatStorer(); > > > ``` > Partition already present with given partition key values : Data already > exists in /user/hive/warehouse/target_data/part1=x1, duplicate publish not > possible. > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.commitJob(PigOutputCommitter.java:243) > at > org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:286) > > Caused by: org.apache.hive.hcatalog.common.HCatException : 2002 : Partition > already present with given partition key values : Data already exists in > /user/hive/warehouse/target_data/part1=x1, duplicate publish not possible. > at > org.apache.hive.hcatalog.mapreduce.FileOutputCommitterContainer.moveTaskOutputs(FileOutputCommitterContainer.java:564) > at > org.apache.hive.hcatalog.mapreduce.FileOutputCommitterContainer.registerPartitions(FileOutputCommitterContainer.java:949) > at > org.apache.hive.hcatalog.mapreduce.FileOutputCommitterContainer.commitJob(FileOutputCommitterContainer.java:273) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.commitJob(PigOutputCommitter.java:241) > ``` -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-27963) Fix license fetch issue with license-maven-plugin
[ https://issues.apache.org/jira/browse/HIVE-27963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17798884#comment-17798884 ] Stamatis Zampetakis commented on HIVE-27963: By running maven in debug mode (-X) I see the following log messages right before the failure: {noformat} [DEBUG] Downloading license(s) for project dk.brics.automaton:automaton [DEBUG] About to download 'http://www.opensource.org/licenses/bsd-license.php' [DEBUG] Downloading 'http://www.opensource.org/licenses/bsd-license.php' -> '/home/stamatis/Projects/Apache/hive/packaging/target/generated-resources/licenses/bsd-bsd-license.html' {noformat} > Fix license fetch issue with license-maven-plugin > - > > Key: HIVE-27963 > URL: https://issues.apache.org/jira/browse/HIVE-27963 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Akshat Mathur >Assignee: Akshat Mathur >Priority: Major > Labels: pull-request-available > > Test are failing due to the following issue: > > [ERROR] Failed to execute goal > org.codehaus.mojo:license-maven-plugin:2.1.0:download-licenses > (license-fetch) on project hive-packaging: Execution license-fetch of goal > org.codehaus.mojo:license-maven-plugin:2.1.0:download-licenses failed: URL ' > [http://www.opensource.org/licenses/bsd-license.php] > ' should belong to licenseUrlFileName having key > 'bsd-2-clause-license-bsd-2-clause.html' together with URLs ' > [https://opensource.org/licenses/BSD-2-Clause] > ' -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-27963) Fix license fetch issue with license-maven-plugin
[ https://issues.apache.org/jira/browse/HIVE-27963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17798883#comment-17798883 ] Stamatis Zampetakis commented on HIVE-27963: The full stacktrace from the build failure is attached below: {noformat} [ERROR] Failed to execute goal org.codehaus.mojo:license-maven-plugin:2.1.0:download-licenses (license-fetch) on project hive-packaging: Execution license-fetch of goal org.codehaus.mojo:license-maven-plugin:2.1.0:download-licenses failed: URL 'http://www.opensource.org/licenses/bsd-license.php' should belong to licenseUrlFileName having key 'bsd-2-clause-license-bsd-2-clause.html' together with URLs 'https://opensource.org/licenses/BSD-2-Clause' -> [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.codehaus.mojo:license-maven-plugin:2.1.0:download-licenses (license-fetch) on project hive-packaging: Execution license-fetch of goal org.codehaus.mojo:license-maven-plugin:2.1.0:download-licenses failed: URL 'http://www.opensource.org/licenses/bsd-license.php' should belong to licenseUrlFileName having key 'bsd-2-clause-license-bsd-2-clause.html' together with URLs 'https://opensource.org/licenses/BSD-2-Clause' at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81) at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56) at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128) at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305) at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192) at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105) at org.apache.maven.cli.MavenCli.execute (MavenCli.java:957) at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:289) at org.apache.maven.cli.MavenCli.main (MavenCli.java:193) at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke (Method.java:498) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282) at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406) at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347) Caused by: org.apache.maven.plugin.PluginExecutionException: Execution license-fetch of goal org.codehaus.mojo:license-maven-plugin:2.1.0:download-licenses failed: URL 'http://www.opensource.org/licenses/bsd-license.php' should belong to licenseUrlFileName having key 'bsd-2-clause-license-bsd-2-clause.html' together with URLs 'https://opensource.org/licenses/BSD-2-Clause' at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:148) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81) at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56) at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128) at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305) at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192) at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105) at org.apache.maven.cli.MavenCli.execute (MavenCli.java:957) at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:289) at org.apache.maven.cli.MavenCli.main (MavenCli.java:193) at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke (Method.java:498)
[jira] [Updated] (HIVE-27161) MetaException when executing CTAS query in Druid storage handler
[ https://issues.apache.org/jira/browse/HIVE-27161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-27161: -- Status: Patch Available (was: Open) > MetaException when executing CTAS query in Druid storage handler > > > Key: HIVE-27161 > URL: https://issues.apache.org/jira/browse/HIVE-27161 > Project: Hive > Issue Type: Bug > Components: Druid integration >Affects Versions: 4.0.0-alpha-2, 4.0.0-alpha-1 >Reporter: Stamatis Zampetakis >Assignee: Krisztian Kasa >Priority: Critical > Labels: pull-request-available > > Any kind of CTAS query targeting the Druid storage handler fails with the > following exception: > {noformat} > org.apache.hadoop.hive.ql.metadata.HiveException: > MetaException(message:LOCATION may not be specified for Druid) > at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:1347) > ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:1352) > ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at > org.apache.hadoop.hive.ql.ddl.table.create.CreateTableOperation.createTableNonReplaceMode(CreateTableOperation.java:158) > ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at > org.apache.hadoop.hive.ql.ddl.table.create.CreateTableOperation.execute(CreateTableOperation.java:116) > ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at org.apache.hadoop.hive.ql.ddl.DDLTask.execute(DDLTask.java:84) > ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214) > ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105) > ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:354) > ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:327) > ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:244) > ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:105) > ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:367) > ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:205) > ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:154) > ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:149) > ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at > org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:185) > ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at > org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:228) > ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at > org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:257) > ~[hive-cli-4.0.0-SNAPSHOT.jar:?] > at org.apache.hadoop.hive.cli.CliDriver.processCmd1(CliDriver.java:201) > ~[hive-cli-4.0.0-SNAPSHOT.jar:?] > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:127) > ~[hive-cli-4.0.0-SNAPSHOT.jar:?] > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:425) > ~[hive-cli-4.0.0-SNAPSHOT.jar:?] > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:356) > ~[hive-cli-4.0.0-SNAPSHOT.jar:?] > at > org.apache.hadoop.hive.ql.dataset.QTestDatasetHandler.initDataset(QTestDatasetHandler.java:86) > ~[hive-it-util-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at > org.apache.hadoop.hive.ql.dataset.QTestDatasetHandler.beforeTest(QTestDatasetHandler.java:190) > ~[hive-it-util-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at > org.apache.hadoop.hive.ql.qoption.QTestOptionDispatcher.beforeTest(QTestOptionDispatcher.java:79) > ~[hive-it-util-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at org.apache.hadoop.hive.ql.QTestUtil.cliInit(QTestUtil.java:607) > ~[hive-it-util-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at > org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:112) > ~[hive-it-util-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at > org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:157) > ~[hive-it-util-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at > org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver(TestMiniDruidCliDriver.java:60) > ~[test-classes/:?] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > ~[?:1.8.0_261] > at >
[jira] [Resolved] (HIVE-27876) Incorrect query results on tables with ClusterBy & SortBy
[ https://issues.apache.org/jira/browse/HIVE-27876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa resolved HIVE-27876. --- Fix Version/s: 4.1.0 Resolution: Fixed Megred to master. Thanks [~rameshkumar] for the patch and [~aturoczy] for review. > Incorrect query results on tables with ClusterBy & SortBy > - > > Key: HIVE-27876 > URL: https://issues.apache.org/jira/browse/HIVE-27876 > Project: Hive > Issue Type: Bug >Reporter: Naresh P R >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Labels: pull-request-available > Fix For: 4.1.0 > > > Repro: > > {code:java} > create external table test_bucket(age int, name string, dept string) > clustered by (age, name) sorted by (age asc, name asc) into 2 buckets stored > as orc; > insert into test_bucket values (1, 'user1', 'dept1'), ( 2, 'user2' , 'dept2'); > insert into test_bucket values (1, 'user1', 'dept1'), ( 2, 'user2' , 'dept2'); > //empty wrong results > select age, name, count(*) from test_bucket group by age, name having > count(*) > 1; > +--+---+--+ > | age | name | _c2 | > +--+---+--+ > +--+---+--+ > // Workaround > set hive.map.aggr=false; > select age, name, count(*) from test_bucket group by age, name having > count(*) > 1; > +--++--+ > | age | name | _c2 | > +--++--+ > | 1 | user1 | 2 | > | 2 | user2 | 2 | > +--++--+ {code} > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27963) Fix license fetch issue with license-maven-plugin
[ https://issues.apache.org/jira/browse/HIVE-27963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-27963: -- Labels: pull-request-available (was: ) > Fix license fetch issue with license-maven-plugin > - > > Key: HIVE-27963 > URL: https://issues.apache.org/jira/browse/HIVE-27963 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Akshat Mathur >Assignee: Akshat Mathur >Priority: Major > Labels: pull-request-available > > Test are failing due to the following issue: > > [ERROR] Failed to execute goal > org.codehaus.mojo:license-maven-plugin:2.1.0:download-licenses > (license-fetch) on project hive-packaging: Execution license-fetch of goal > org.codehaus.mojo:license-maven-plugin:2.1.0:download-licenses failed: URL ' > [http://www.opensource.org/licenses/bsd-license.php] > ' should belong to licenseUrlFileName having key > 'bsd-2-clause-license-bsd-2-clause.html' together with URLs ' > [https://opensource.org/licenses/BSD-2-Clause] > ' -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-27963) Fix license fetch issue with license-maven-plugin
Akshat Mathur created HIVE-27963: Summary: Fix license fetch issue with license-maven-plugin Key: HIVE-27963 URL: https://issues.apache.org/jira/browse/HIVE-27963 Project: Hive Issue Type: Bug Components: Hive Reporter: Akshat Mathur Test are failing due to the following issue: [ERROR] Failed to execute goal org.codehaus.mojo:license-maven-plugin:2.1.0:download-licenses (license-fetch) on project hive-packaging: Execution license-fetch of goal org.codehaus.mojo:license-maven-plugin:2.1.0:download-licenses failed: URL ' [http://www.opensource.org/licenses/bsd-license.php] ' should belong to licenseUrlFileName having key 'bsd-2-clause-license-bsd-2-clause.html' together with URLs ' [https://opensource.org/licenses/BSD-2-Clause] ' -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-27963) Fix license fetch issue with license-maven-plugin
[ https://issues.apache.org/jira/browse/HIVE-27963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshat Mathur reassigned HIVE-27963: Assignee: Akshat Mathur > Fix license fetch issue with license-maven-plugin > - > > Key: HIVE-27963 > URL: https://issues.apache.org/jira/browse/HIVE-27963 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Akshat Mathur >Assignee: Akshat Mathur >Priority: Major > > Test are failing due to the following issue: > > [ERROR] Failed to execute goal > org.codehaus.mojo:license-maven-plugin:2.1.0:download-licenses > (license-fetch) on project hive-packaging: Execution license-fetch of goal > org.codehaus.mojo:license-maven-plugin:2.1.0:download-licenses failed: URL ' > [http://www.opensource.org/licenses/bsd-license.php] > ' should belong to licenseUrlFileName having key > 'bsd-2-clause-license-bsd-2-clause.html' together with URLs ' > [https://opensource.org/licenses/BSD-2-Clause] > ' -- This message was sent by Atlassian Jira (v8.20.10#820010)