[
https://issues.apache.org/jira/browse/HIVE-29798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-29798:
--------------------------------
Description:
This was discovered while testing HIVE-28822, repro attached:
{code}
Caused by: java.lang.NumberFormatException: For input string: "NION"
at
java.base/java.lang.NumberFormatException.forInputString(NumberFormatException.java:67)
at java.base/java.lang.Long.parseLong(Long.java:709)
at java.base/java.lang.Long.parseLong(Long.java:832)
at
org.apache.hadoop.hive.ql.io.AcidUtils$ParsedDeltaLight.parse(AcidUtils.java:1081)
at
org.apache.hadoop.hive.ql.io.orc.VectorizedOrcAcidRowBatchReader.<init>(VectorizedOrcAcidRowBatchReader.java:355)
at
org.apache.hadoop.hive.ql.io.orc.VectorizedOrcAcidRowBatchReader.<init>(VectorizedOrcAcidRowBatchReader.java:175)
at
org.apache.hadoop.hive.ql.io.orc.VectorizedOrcAcidRowBatchReader.<init>(VectorizedOrcAcidRowBatchReader.java:170)
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:2008)
at
org.apache.hadoop.hive.ql.io.RecordReaderWrapper.create(RecordReaderWrapper.java:76)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:469)
... 28 more
{code}
this happens after "alter table tbl convert to acid" as well as altering table
properties to transactional=true
2 scenarios I can imagine:
1a) this is a bug, fix in writer path: alter table should take care of
flattening the union subdirs and converting the file structure to an ACID
compliant layout
1b) this is a bug, fix on reader path: considering [a unit
test|https://github.com/apache/hive/blob/9173850b9d8d51b253a0763115a6e5720b040640/ql/src/test/org/apache/hadoop/hive/ql/TestTxnNoBuckets.java#L418-L437],
having the union subdirs in acid tables is totally fine, so no need to flatten
2. this is by design, and tables containing union subdirs must be flattened
BEFORE converting them to ACID
I'm about to handle this as 1b) as it makes to most sense to me
was:
This was discovered while testing HIVE-28822, repro attached:
{code}
Caused by: java.lang.NumberFormatException: For input string: "NION"
at
java.base/java.lang.NumberFormatException.forInputString(NumberFormatException.java:67)
at java.base/java.lang.Long.parseLong(Long.java:709)
at java.base/java.lang.Long.parseLong(Long.java:832)
at
org.apache.hadoop.hive.ql.io.AcidUtils$ParsedDeltaLight.parse(AcidUtils.java:1081)
at
org.apache.hadoop.hive.ql.io.orc.VectorizedOrcAcidRowBatchReader.<init>(VectorizedOrcAcidRowBatchReader.java:355)
at
org.apache.hadoop.hive.ql.io.orc.VectorizedOrcAcidRowBatchReader.<init>(VectorizedOrcAcidRowBatchReader.java:175)
at
org.apache.hadoop.hive.ql.io.orc.VectorizedOrcAcidRowBatchReader.<init>(VectorizedOrcAcidRowBatchReader.java:170)
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:2008)
at
org.apache.hadoop.hive.ql.io.RecordReaderWrapper.create(RecordReaderWrapper.java:76)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:469)
... 28 more
{code}
this happens after "alter table tbl convert to acid" as well as altering table
properties to transactional=true
2 scenarios I can imagine:
1a) this is a bug, fix in writer path: alter table should take care of
flattening the union subdirs and converting the file structure to an ACID
compliant layout
1b) this is a bug, fix on reader path: considering [a unit
test|https://github.com/apache/hive/blob/9173850b9d8d51b253a0763115a6e5720b040640/ql/src/test/org/apache/hadoop/hive/ql/TestTxnNoBuckets.java#L418-L437],
having the union subdirs in acid tables are totally fine, so no need to flatten
2. this is by design, and tables containing union subdirs must be flattened
BEFORE converting them to ACID
I'm about to handle this as 1b) as it makes to most sense to me
> NumberFormatException after converting table with UNION subdirs to ACID
> -----------------------------------------------------------------------
>
> Key: HIVE-29798
> URL: https://issues.apache.org/jira/browse/HIVE-29798
> Project: Hive
> Issue Type: Bug
> Reporter: László Bodor
> Assignee: László Bodor
> Priority: Major
> Attachments: TestUnionAllToAcidConversionReadFails.java
>
>
> This was discovered while testing HIVE-28822, repro attached:
> {code}
> Caused by: java.lang.NumberFormatException: For input string: "NION"
> at
> java.base/java.lang.NumberFormatException.forInputString(NumberFormatException.java:67)
> at java.base/java.lang.Long.parseLong(Long.java:709)
> at java.base/java.lang.Long.parseLong(Long.java:832)
> at
> org.apache.hadoop.hive.ql.io.AcidUtils$ParsedDeltaLight.parse(AcidUtils.java:1081)
> at
> org.apache.hadoop.hive.ql.io.orc.VectorizedOrcAcidRowBatchReader.<init>(VectorizedOrcAcidRowBatchReader.java:355)
> at
> org.apache.hadoop.hive.ql.io.orc.VectorizedOrcAcidRowBatchReader.<init>(VectorizedOrcAcidRowBatchReader.java:175)
> at
> org.apache.hadoop.hive.ql.io.orc.VectorizedOrcAcidRowBatchReader.<init>(VectorizedOrcAcidRowBatchReader.java:170)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:2008)
> at
> org.apache.hadoop.hive.ql.io.RecordReaderWrapper.create(RecordReaderWrapper.java:76)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:469)
> ... 28 more
> {code}
> this happens after "alter table tbl convert to acid" as well as altering
> table properties to transactional=true
> 2 scenarios I can imagine:
> 1a) this is a bug, fix in writer path: alter table should take care of
> flattening the union subdirs and converting the file structure to an ACID
> compliant layout
> 1b) this is a bug, fix on reader path: considering [a unit
> test|https://github.com/apache/hive/blob/9173850b9d8d51b253a0763115a6e5720b040640/ql/src/test/org/apache/hadoop/hive/ql/TestTxnNoBuckets.java#L418-L437],
> having the union subdirs in acid tables is totally fine, so no need to
> flatten
> 2. this is by design, and tables containing union subdirs must be flattened
> BEFORE converting them to ACID
> I'm about to handle this as 1b) as it makes to most sense to me
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
