[jira] [Updated] (HIVE-27532) Missing semicolon in show create table output
[
https://issues.apache.org/jira/browse/HIVE-27532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-27532:
--
Labels: pull-request-available (was: )
> Missing semicolon in show create table output
> -
>
> Key: HIVE-27532
> URL: https://issues.apache.org/jira/browse/HIVE-27532
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Soumyakanti Das
>Assignee: Soumyakanti Das
>Priority: Major
> Labels: pull-request-available
>
> When we run SHOW CREATE TABLE on a table with constraints, it doesn't add a
> semi-colon to the create table ddl. Here's an output for tpcds table reason:
> {code:java}
> CREATE TABLE `reason`(
> `r_reason_sk` int,
> `r_reason_id` string,
> `r_reason_desc` string)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
> WITH SERDEPROPERTIES (
> 'field.delim'='|',
> 'serialization.format'='|')
> STORED AS ORC
> TBLPROPERTIES (
> 'transactional'='true',
> 'OBJCAPABILITIES'='EXTREAD,EXTWRITE',
> 'STATS_GENERATED'='TASK',
> 'impala.lastComputeStatsTime'='1674074181',
> 'serialization.null.format'='',
> 'transient_lastDdlTime'='1674073496')
> ALTER TABLE reason ADD CONSTRAINT 2e47abb2-b6c7-450a-8229-395d6b1ff168
> PRIMARY KEY (r_reason_sk) DISABLE NOVALIDATE RELY; {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27632) ClassCast Exception in Vectorization converting decimal64 to decimal
[
https://issues.apache.org/jira/browse/HIVE-27632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-27632:
--
Labels: pull-request-available (was: )
> ClassCast Exception in Vectorization converting decimal64 to decimal
>
>
> Key: HIVE-27632
> URL: https://issues.apache.org/jira/browse/HIVE-27632
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Reporter: Riju Trivedi
>Priority: Major
> Labels: pull-request-available
> Attachments: vectortest.q
>
>
> Attached [^vectortest.q] which fails with the below ClassCast Exception
> {code:java}
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.ql.exec.vector.Decimal64ColumnVector cannot be cast to
> org.apache.hadoop.hive.ql.exec.vector.DecimalColumnVector
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.gen.FilterDecimalColEqualDecimalScalar.evaluate(FilterDecimalColEqualDecimalScalar.java:64)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorFilterOperator.process(VectorFilterOperator.java:125)
> at
> org.apache.hadoop.hive.ql.exec.Operator.vectorForward(Operator.java:919)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:171)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.deliverVectorizedRowBatch(VectorMapOperator.java:809)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:878)
> {code}
> This seems related to HIVE-26208 , which avoids Decimal64 to Decimal
> conversion for the vector expressions that explicitly handle decimal64 types.
> However, in this scenario exception comes from
> `FilterDecimalColEqualDecimalScalar`.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-27589) Iceberg: Branches of Merge/Update statements should be committed atomically
[ https://issues.apache.org/jira/browse/HIVE-27589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17756936#comment-17756936 ] Simhadri Govindappa commented on HIVE-27589: Thanks [~dkuzmenko] , [~krisztiankasa] and [~zhangbutao] :) > Iceberg: Branches of Merge/Update statements should be committed atomically > --- > > Key: HIVE-27589 > URL: https://issues.apache.org/jira/browse/HIVE-27589 > Project: Hive > Issue Type: Task >Reporter: Denys Kuzmenko >Assignee: Simhadri Govindappa >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-27589) Iceberg: Branches of Merge/Update statements should be committed atomically
[ https://issues.apache.org/jira/browse/HIVE-27589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko resolved HIVE-27589. --- Fix Version/s: 4.0.0 Resolution: Fixed > Iceberg: Branches of Merge/Update statements should be committed atomically > --- > > Key: HIVE-27589 > URL: https://issues.apache.org/jira/browse/HIVE-27589 > Project: Hive > Issue Type: Task >Reporter: Denys Kuzmenko >Assignee: Simhadri Govindappa >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (HIVE-27589) Iceberg: Branches of Merge/Update statements should be committed atomically
[ https://issues.apache.org/jira/browse/HIVE-27589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17756875#comment-17756875 ] Denys Kuzmenko edited comment on HIVE-27589 at 8/21/23 12:28 PM: - Merged to master. [~kkasa], [~zhangbutao], thank you for the review! was (Author: dkuzmenko): Merged to master. @kkasa, [~zhangbutao], thank you for the review! > Iceberg: Branches of Merge/Update statements should be committed atomically > --- > > Key: HIVE-27589 > URL: https://issues.apache.org/jira/browse/HIVE-27589 > Project: Hive > Issue Type: Task >Reporter: Denys Kuzmenko >Assignee: Simhadri Govindappa >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-27589) Iceberg: Branches of Merge/Update statements should be committed atomically
[ https://issues.apache.org/jira/browse/HIVE-27589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17756875#comment-17756875 ] Denys Kuzmenko commented on HIVE-27589: --- Merged to master. @kkasa, [~zhangbutao], thank you for the review! > Iceberg: Branches of Merge/Update statements should be committed atomically > --- > > Key: HIVE-27589 > URL: https://issues.apache.org/jira/browse/HIVE-27589 > Project: Hive > Issue Type: Task >Reporter: Denys Kuzmenko >Assignee: Simhadri Govindappa >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-25576) Configurable datetime formatter for unix_timestamp, from_unixtime
[
https://issues.apache.org/jira/browse/HIVE-25576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17756851#comment-17756851
]
Stamatis Zampetakis commented on HIVE-25576:
I added an entry about the hive.datetime.formatter property in the wiki:
https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-Datetime
I updated the from_unixtime and unix_timestamp entries in the wiki:
https://cwiki.apache.org/confluence/display/hive/languagemanual+udf#LanguageManualUDF-DateFunctions
> Configurable datetime formatter for unix_timestamp, from_unixtime
> -
>
> Key: HIVE-25576
> URL: https://issues.apache.org/jira/browse/HIVE-25576
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.1.0, 3.0.0, 3.1.1, 3.1.2, 4.0.0-beta-1
>Reporter: Ashish Sharma
>Assignee: Stamatis Zampetakis
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> HIVE-25403, HIVE-25458 switched the internal implementation of datetime
> formatter for unix_timestamp and from_unixtime from
> {{java.text.SimpleDateFormat}} to {{java.time.format.DateTimeFormatter}} in
> order fix some bugs and inconsistencies when the aforementioned functions are
> combined with other UDFs that have already migrated to use the new modern
> java.time package.
> The two Java formatters present differences in their behavior leading to
> different query results. The supported patterns, between the two formatters,
> are also different something that makes existing queries crash at runtime
> (after upgrade). Adapting to the new behavior of DateTimeFormatter is a
> challenging and time-consuming task for end users especially due to the
> widespread use of the afforementioned unixtime functions.
> Although DateTimeFormatter is a clear improvement over SimpleDateFormat some
> users still want to retain the old behavior for compatibility reasons thus
> introducing a property is necessary for facilitating migration.
> The goal of this ticket is to introduce a new property namely
> {{hive.datetime.formatter}} to control the formatter used by unix_timestamp
> and from_unixtime. By default the new {{DateTimeFormatter}} is used while the
> use of {{SimpleDateFormat}} is discouraged. Eventually, {{SimpleDateFormat}}
> will cease to exist.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HIVE-25576) Configurable datetime formatter for unix_timestamp, from_unixtime
[
https://issues.apache.org/jira/browse/HIVE-25576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stamatis Zampetakis resolved HIVE-25576.
Fix Version/s: 4.0.0
Resolution: Fixed
Fixed in
https://github.com/apache/hive/commit/ed51dfdbdfc109db53b3c631e9f1631e9bb65c34.
[~ashish-kumar-sharma] Thanks for reporting the issue and working on the
initial POC!
[~amansinha], [~jfs], [~hemanth619], many thanks for the reviews and feedback
on the final PR.
> Configurable datetime formatter for unix_timestamp, from_unixtime
> -
>
> Key: HIVE-25576
> URL: https://issues.apache.org/jira/browse/HIVE-25576
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.1.0, 3.0.0, 3.1.1, 3.1.2, 4.0.0-beta-1
>Reporter: Ashish Sharma
>Assignee: Stamatis Zampetakis
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> HIVE-25403, HIVE-25458 switched the internal implementation of datetime
> formatter for unix_timestamp and from_unixtime from
> {{java.text.SimpleDateFormat}} to {{java.time.format.DateTimeFormatter}} in
> order fix some bugs and inconsistencies when the aforementioned functions are
> combined with other UDFs that have already migrated to use the new modern
> java.time package.
> The two Java formatters present differences in their behavior leading to
> different query results. The supported patterns, between the two formatters,
> are also different something that makes existing queries crash at runtime
> (after upgrade). Adapting to the new behavior of DateTimeFormatter is a
> challenging and time-consuming task for end users especially due to the
> widespread use of the afforementioned unixtime functions.
> Although DateTimeFormatter is a clear improvement over SimpleDateFormat some
> users still want to retain the old behavior for compatibility reasons thus
> introducing a property is necessary for facilitating migration.
> The goal of this ticket is to introduce a new property namely
> {{hive.datetime.formatter}} to control the formatter used by unix_timestamp
> and from_unixtime. By default the new {{DateTimeFormatter}} is used while the
> use of {{SimpleDateFormat}} is discouraged. Eventually, {{SimpleDateFormat}}
> will cease to exist.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HIVE-24771) Fix hang of TransactionalKafkaWriterTest
[
https://issues.apache.org/jira/browse/HIVE-24771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stamatis Zampetakis resolved HIVE-24771.
Fix Version/s: 4.0.0
Resolution: Fixed
Test re-enabled in
https://github.com/apache/hive/commit/cd9b42d36d4f24bc6d37ccb8ffad2cadb745483f.
Thanks for the PR [~kokila19] and [~akshatm], [~aturoczy] for the reviews.
> Fix hang of TransactionalKafkaWriterTest
> -
>
> Key: HIVE-24771
> URL: https://issues.apache.org/jira/browse/HIVE-24771
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Kokila N
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: hive.log.gz, jstack.1, jstack.2, jstack.3
>
>
> this test seems to hang randomly - I've launched 3 checks against it - all of
> which started to hang after some time
> http://ci.hive.apache.org/job/hive-flaky-check/187/
> http://ci.hive.apache.org/job/hive-flaky-check/188/
> http://ci.hive.apache.org/job/hive-flaky-check/189/
> {code}
> "main" #1 prio=5 os_prio=0 tid=0x7f1d5400a800 nid=0x31e waiting on
> condition [0x7f1d59381000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x894b3ed8> (a
> java.util.concurrent.CountDownLatch$Sync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:837)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:999)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1308)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
> at
> org.apache.kafka.clients.producer.internals.TransactionalRequestResult.await(TransactionalRequestResult.java:56)
> at
> org.apache.hadoop.hive.kafka.HiveKafkaProducer.flushNewPartitions(HiveKafkaProducer.java:187)
> at
> org.apache.hadoop.hive.kafka.HiveKafkaProducer.flush(HiveKafkaProducer.java:123)
> at
> org.apache.hadoop.hive.kafka.TransactionalKafkaWriter.close(TransactionalKafkaWriter.java:189)
> at
> org.apache.hadoop.hive.kafka.TransactionalKafkaWriterTest.writeAndCommit(TransactionalKafkaWriterTest.java:182)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
>
[jira] [Updated] (HIVE-25576) Configurable datetime formatter for unix_timestamp, from_unixtime
[
https://issues.apache.org/jira/browse/HIVE-25576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stamatis Zampetakis updated HIVE-25576:
---
Affects Version/s: 4.0.0-beta-1
(was: 4.0.0)
> Configurable datetime formatter for unix_timestamp, from_unixtime
> -
>
> Key: HIVE-25576
> URL: https://issues.apache.org/jira/browse/HIVE-25576
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.1.0, 3.0.0, 3.1.1, 3.1.2, 4.0.0-beta-1
>Reporter: Ashish Sharma
>Assignee: Stamatis Zampetakis
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h
> Remaining Estimate: 0h
>
> HIVE-25403, HIVE-25458 switched the internal implementation of datetime
> formatter for unix_timestamp and from_unixtime from
> {{java.text.SimpleDateFormat}} to {{java.time.format.DateTimeFormatter}} in
> order fix some bugs and inconsistencies when the aforementioned functions are
> combined with other UDFs that have already migrated to use the new modern
> java.time package.
> The two Java formatters present differences in their behavior leading to
> different query results. The supported patterns, between the two formatters,
> are also different something that makes existing queries crash at runtime
> (after upgrade). Adapting to the new behavior of DateTimeFormatter is a
> challenging and time-consuming task for end users especially due to the
> widespread use of the afforementioned unixtime functions.
> Although DateTimeFormatter is a clear improvement over SimpleDateFormat some
> users still want to retain the old behavior for compatibility reasons thus
> introducing a property is necessary for facilitating migration.
> The goal of this ticket is to introduce a new property namely
> {{hive.datetime.formatter}} to control the formatter used by unix_timestamp
> and from_unixtime. By default the new {{DateTimeFormatter}} is used while the
> use of {{SimpleDateFormat}} is discouraged. Eventually, {{SimpleDateFormat}}
> will cease to exist.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-25576) Configurable datetime formatter for unix_timestamp, from_unixtime
[
https://issues.apache.org/jira/browse/HIVE-25576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stamatis Zampetakis updated HIVE-25576:
---
Description:
HIVE-25403, HIVE-25458 switched the internal implementation of datetime
formatter for unix_timestamp and from_unixtime from
{{java.text.SimpleDateFormat}} to {{java.time.format.DateTimeFormatter}} in
order fix some bugs and inconsistencies when the aforementioned functions are
combined with other UDFs that have already migrated to use the new modern
java.time package.
The two Java formatters present differences in their behavior leading to
different query results. The supported patterns, between the two formatters,
are also different something that makes existing queries crash at runtime
(after upgrade). Adapting to the new behavior of DateTimeFormatter is a
challenging and time-consuming task for end users especially due to the
widespread use of the afforementioned unixtime functions.
Although DateTimeFormatter is a clear improvement over SimpleDateFormat some
users still want to retain the old behavior for compatibility reasons thus
introducing a property is necessary for facilitating migration.
The goal of this ticket is to introduce a new property namely
{{hive.datetime.formatter}} to control the formatter used by unix_timestamp and
from_unixtime. By default the new {{DateTimeFormatter}} is used while the use
of {{SimpleDateFormat}} is discouraged. Eventually, {{SimpleDateFormat}} will
cease to exist.
was:
*History*
*Hive 1.2* -
VM time zone set to Asia/Bangkok
*Query* - SELECT FROM_UNIXTIME(UNIX_TIMESTAMP('1800-01-01 00:00:00
UTC','-MM-dd HH:mm:ss z'));
*Result* - 1800-01-01 07:00:00
*Implementation details* -
SimpleDateFormat formatter = new SimpleDateFormat(pattern);
Long unixtime = formatter.parse(textval).getTime() / 1000;
Date date = new Date(unixtime * 1000L);
https://docs.oracle.com/javase/8/docs/api/java/util/Date.html . In official
documentation they have mention that "Unfortunately, the API for these
functions was not amenable to internationalization and The corresponding
methods in Date are deprecated" . Due to that this is producing wrong result
*Master branch* -
set hive.local.time.zone=Asia/Bangkok;
*Query* - SELECT FROM_UNIXTIME(UNIX_TIMESTAMP('1800-01-01 00:00:00
UTC','-MM-dd HH:mm:ss z'));
*Result* - 1800-01-01 06:42:04
*Implementation details* -
DateTimeFormatter dtformatter = new DateTimeFormatterBuilder()
.parseCaseInsensitive()
.appendPattern(pattern)
.toFormatter();
ZonedDateTime zonedDateTime =
ZonedDateTime.parse(textval,dtformatter).withZoneSameInstant(ZoneId.of(timezone));
Long dttime = zonedDateTime.toInstant().getEpochSecond();
*Problem*-
Now *SimpleDateFormat* has been replaced with *DateTimeFormatter* which is
giving the correct result but it is not backword compatible. Which is causing
issue at time for migration to new version. Because the older data written is
using Hive 1.x or 2.x is not compatible with *DateTimeFormatter*.
*Solution*
Introduce an config "hive.legacy.timeParserPolicy" with following values -
1. *True*- use *SimpleDateFormat*
2. *False* - use *DateTimeFormatter*
Note: apache spark also face the same issue
https://issues.apache.org/jira/browse/SPARK-30668
> Configurable datetime formatter for unix_timestamp, from_unixtime
> -
>
> Key: HIVE-25576
> URL: https://issues.apache.org/jira/browse/HIVE-25576
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.1.0, 3.0.0, 3.1.1, 3.1.2, 4.0.0
>Reporter: Ashish Sharma
>Assignee: Stamatis Zampetakis
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h
> Remaining Estimate: 0h
>
> HIVE-25403, HIVE-25458 switched the internal implementation of datetime
> formatter for unix_timestamp and from_unixtime from
> {{java.text.SimpleDateFormat}} to {{java.time.format.DateTimeFormatter}} in
> order fix some bugs and inconsistencies when the aforementioned functions are
> combined with other UDFs that have already migrated to use the new modern
> java.time package.
> The two Java formatters present differences in their behavior leading to
> different query results. The supported patterns, between the two formatters,
> are also different something that makes existing queries crash at runtime
> (after upgrade). Adapting to the new behavior of DateTimeFormatter is a
> challenging and time-consuming task for end users especially due to the
> widespread use of the afforementioned unixtime functions.
> Although DateTimeFormatter is a clear improvement over SimpleDateFormat some
> users still want to retain the old behavior for compatibility reasons thus
> introducing a property is necessary for
[jira] [Updated] (HIVE-25576) Configurable datetime formatter for unix_timestamp, from_unixtime
[
https://issues.apache.org/jira/browse/HIVE-25576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stamatis Zampetakis updated HIVE-25576:
---
Summary: Configurable datetime formatter for unix_timestamp, from_unixtime
(was: Add config to parse date with older date format)
> Configurable datetime formatter for unix_timestamp, from_unixtime
> -
>
> Key: HIVE-25576
> URL: https://issues.apache.org/jira/browse/HIVE-25576
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.1.0, 3.0.0, 3.1.1, 3.1.2, 4.0.0
>Reporter: Ashish Sharma
>Assignee: Stamatis Zampetakis
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h
> Remaining Estimate: 0h
>
> *History*
> *Hive 1.2* -
> VM time zone set to Asia/Bangkok
> *Query* - SELECT FROM_UNIXTIME(UNIX_TIMESTAMP('1800-01-01 00:00:00
> UTC','-MM-dd HH:mm:ss z'));
> *Result* - 1800-01-01 07:00:00
> *Implementation details* -
> SimpleDateFormat formatter = new SimpleDateFormat(pattern);
> Long unixtime = formatter.parse(textval).getTime() / 1000;
> Date date = new Date(unixtime * 1000L);
> https://docs.oracle.com/javase/8/docs/api/java/util/Date.html . In official
> documentation they have mention that "Unfortunately, the API for these
> functions was not amenable to internationalization and The corresponding
> methods in Date are deprecated" . Due to that this is producing wrong result
> *Master branch* -
> set hive.local.time.zone=Asia/Bangkok;
> *Query* - SELECT FROM_UNIXTIME(UNIX_TIMESTAMP('1800-01-01 00:00:00
> UTC','-MM-dd HH:mm:ss z'));
> *Result* - 1800-01-01 06:42:04
> *Implementation details* -
> DateTimeFormatter dtformatter = new DateTimeFormatterBuilder()
> .parseCaseInsensitive()
> .appendPattern(pattern)
> .toFormatter();
> ZonedDateTime zonedDateTime =
> ZonedDateTime.parse(textval,dtformatter).withZoneSameInstant(ZoneId.of(timezone));
> Long dttime = zonedDateTime.toInstant().getEpochSecond();
> *Problem*-
> Now *SimpleDateFormat* has been replaced with *DateTimeFormatter* which is
> giving the correct result but it is not backword compatible. Which is causing
> issue at time for migration to new version. Because the older data written is
> using Hive 1.x or 2.x is not compatible with *DateTimeFormatter*.
> *Solution*
> Introduce an config "hive.legacy.timeParserPolicy" with following values -
> 1. *True*- use *SimpleDateFormat*
> 2. *False* - use *DateTimeFormatter*
> Note: apache spark also face the same issue
> https://issues.apache.org/jira/browse/SPARK-30668
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27569) Backport of HIVE-22405: Add ColumnVector support for ProlepticCalendar
[ https://issues.apache.org/jira/browse/HIVE-27569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-27569: Affects Version/s: 3.1.3 (was: 3.2.0) > Backport of HIVE-22405: Add ColumnVector support for ProlepticCalendar > -- > > Key: HIVE-27569 > URL: https://issues.apache.org/jira/browse/HIVE-27569 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.1.3 >Reporter: Shefali Singh >Assignee: Shefali Singh >Priority: Major > Labels: pull-request-available > Fix For: 3.2.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-27569) Backport of HIVE-22405: Add ColumnVector support for ProlepticCalendar
[ https://issues.apache.org/jira/browse/HIVE-27569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan resolved HIVE-27569. - Fix Version/s: 3.2.0 Resolution: Fixed Thanks [~shefali636] for the contribution! > Backport of HIVE-22405: Add ColumnVector support for ProlepticCalendar > -- > > Key: HIVE-27569 > URL: https://issues.apache.org/jira/browse/HIVE-27569 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.2.0 >Reporter: Shefali Singh >Assignee: Shefali Singh >Priority: Major > Labels: pull-request-available > Fix For: 3.2.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-27570) Backport of HIVE-21815: Stats in ORC file are parsed twice
[ https://issues.apache.org/jira/browse/HIVE-27570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan resolved HIVE-27570. - Fix Version/s: 3.2.0 Resolution: Fixed Thanks [~shefali636] for the contribution! > Backport of HIVE-21815: Stats in ORC file are parsed twice > -- > > Key: HIVE-27570 > URL: https://issues.apache.org/jira/browse/HIVE-27570 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.2.0 >Reporter: Shefali Singh >Assignee: Shefali Singh >Priority: Major > Labels: pull-request-available > Fix For: 3.2.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-27571) Backport of HIVE-18702: INSERT OVERWRITE TABLE doesn't clean the table directory before overwriting
[ https://issues.apache.org/jira/browse/HIVE-27571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan resolved HIVE-27571. - Fix Version/s: 3.2.0 Resolution: Fixed Thanks [~shefali636] for the contribution! > Backport of HIVE-18702: INSERT OVERWRITE TABLE doesn't clean the table > directory before overwriting > --- > > Key: HIVE-27571 > URL: https://issues.apache.org/jira/browse/HIVE-27571 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.2.0 >Reporter: Shefali Singh >Assignee: Shefali Singh >Priority: Major > Labels: pull-request-available > Fix For: 3.2.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27571) Backport of HIVE-18702: INSERT OVERWRITE TABLE doesn't clean the table directory before overwriting
[ https://issues.apache.org/jira/browse/HIVE-27571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-27571: Affects Version/s: 3.1.3 (was: 3.2.0) > Backport of HIVE-18702: INSERT OVERWRITE TABLE doesn't clean the table > directory before overwriting > --- > > Key: HIVE-27571 > URL: https://issues.apache.org/jira/browse/HIVE-27571 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.1.3 >Reporter: Shefali Singh >Assignee: Shefali Singh >Priority: Major > Labels: pull-request-available > Fix For: 3.2.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-27572) Backport of HIVE-21296: Dropping varchar partition throw exception
[ https://issues.apache.org/jira/browse/HIVE-27572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan resolved HIVE-27572. - Fix Version/s: 3.2.0 Resolution: Fixed > Backport of HIVE-21296: Dropping varchar partition throw exception > -- > > Key: HIVE-27572 > URL: https://issues.apache.org/jira/browse/HIVE-27572 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.2.0 >Reporter: Shefali Singh >Assignee: Shefali Singh >Priority: Major > Labels: pull-request-available > Fix For: 3.2.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27572) Backport of HIVE-21296: Dropping varchar partition throw exception
[ https://issues.apache.org/jira/browse/HIVE-27572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-27572: Affects Version/s: 3.1.3 (was: 3.2.0) > Backport of HIVE-21296: Dropping varchar partition throw exception > -- > > Key: HIVE-27572 > URL: https://issues.apache.org/jira/browse/HIVE-27572 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.1.3 >Reporter: Shefali Singh >Assignee: Shefali Singh >Priority: Major > Labels: pull-request-available > Fix For: 3.2.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-27572) Backport of HIVE-21296: Dropping varchar partition throw exception
[ https://issues.apache.org/jira/browse/HIVE-27572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17756688#comment-17756688 ] Sankar Hariappan commented on HIVE-27572: - Thanks [~shefali636] for the contribution! > Backport of HIVE-21296: Dropping varchar partition throw exception > -- > > Key: HIVE-27572 > URL: https://issues.apache.org/jira/browse/HIVE-27572 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.1.3 >Reporter: Shefali Singh >Assignee: Shefali Singh >Priority: Major > Labels: pull-request-available > Fix For: 3.2.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
