[
https://issues.apache.org/jira/browse/HIVE-29798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-29798:
--------------------------------
Description:
Update: this change is finally merged together with HIVE-29799, because both
are related to how directory and file layout - coming from a union all query -
can be converted to ACID.
2 issues here:
1) the non-flattened union case
{code}
Caused by: java.lang.NumberFormatException: For input string: "NION"
at
java.base/java.lang.NumberFormatException.forInputString(NumberFormatException.java:67)
at java.base/java.lang.Long.parseLong(Long.java:709)
at java.base/java.lang.Long.parseLong(Long.java:832)
at
org.apache.hadoop.hive.ql.io.AcidUtils$ParsedDeltaLight.parse(AcidUtils.java:1081)
at
org.apache.hadoop.hive.ql.io.orc.VectorizedOrcAcidRowBatchReader.<init>(VectorizedOrcAcidRowBatchReader.java:355)
at
org.apache.hadoop.hive.ql.io.orc.VectorizedOrcAcidRowBatchReader.<init>(VectorizedOrcAcidRowBatchReader.java:175)
at
org.apache.hadoop.hive.ql.io.orc.VectorizedOrcAcidRowBatchReader.<init>(VectorizedOrcAcidRowBatchReader.java:170)
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:2008)
at
org.apache.hadoop.hive.ql.io.RecordReaderWrapper.create(RecordReaderWrapper.java:76)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:469)
... 28 more
{code}
this happens after "alter table tbl convert to acid" as well as altering table
properties to transactional=true
2 scenarios I can imagine:
1a) this is a bug, fix in writer path: alter table should take care of
flattening the union subdirs and converting the file structure to an ACID
compliant layout
1b) this is a bug, fix on reader path: considering [a unit
test|https://github.com/apache/hive/blob/9173850b9d8d51b253a0763115a6e5720b040640/ql/src/test/org/apache/hadoop/hive/ql/TestTxnNoBuckets.java#L418-L437],
having the union subdirs in acid tables is totally fine, so no need to flatten
2. this is by design, and tables containing union subdirs must be flattened
BEFORE converting them to ACID
I'm about to handle this as 1b) as it makes to most sense to me
2. The flattened union case: table containing flattened union files cannot be
converted to ACID
MoveTask.flattenUnionSubdirectories produces <index>000000_0 files (three
numeric parts) which don't match the metastore's ORIGINAL_PATTERN ([0-9][0-9]),
so a subsequent ACID conversion is rejected by
TransactionalValidationListener.validateTableStructureForPath with "Unexpected
data file name format". The tests assert exactly that failure and include a
comment pointing at what should change to make them assert successful
conversion instead.
1. Reconsider the file format after flattening
2. Prepare ACID format patterns to consume these files: 1_000000_0, 2_000000_0
f, 3_000000_0 ...
I would go for 1) first, then worst case 2)
was:
This was discovered while testing HIVE-28822, repro attached:
{code}
Caused by: java.lang.NumberFormatException: For input string: "NION"
at
java.base/java.lang.NumberFormatException.forInputString(NumberFormatException.java:67)
at java.base/java.lang.Long.parseLong(Long.java:709)
at java.base/java.lang.Long.parseLong(Long.java:832)
at
org.apache.hadoop.hive.ql.io.AcidUtils$ParsedDeltaLight.parse(AcidUtils.java:1081)
at
org.apache.hadoop.hive.ql.io.orc.VectorizedOrcAcidRowBatchReader.<init>(VectorizedOrcAcidRowBatchReader.java:355)
at
org.apache.hadoop.hive.ql.io.orc.VectorizedOrcAcidRowBatchReader.<init>(VectorizedOrcAcidRowBatchReader.java:175)
at
org.apache.hadoop.hive.ql.io.orc.VectorizedOrcAcidRowBatchReader.<init>(VectorizedOrcAcidRowBatchReader.java:170)
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:2008)
at
org.apache.hadoop.hive.ql.io.RecordReaderWrapper.create(RecordReaderWrapper.java:76)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:469)
... 28 more
{code}
this happens after "alter table tbl convert to acid" as well as altering table
properties to transactional=true
2 scenarios I can imagine:
1a) this is a bug, fix in writer path: alter table should take care of
flattening the union subdirs and converting the file structure to an ACID
compliant layout
1b) this is a bug, fix on reader path: considering [a unit
test|https://github.com/apache/hive/blob/9173850b9d8d51b253a0763115a6e5720b040640/ql/src/test/org/apache/hadoop/hive/ql/TestTxnNoBuckets.java#L418-L437],
having the union subdirs in acid tables is totally fine, so no need to flatten
2. this is by design, and tables containing union subdirs must be flattened
BEFORE converting them to ACID
I'm about to handle this as 1b) as it makes to most sense to me
> Fix union directory and file layout to ACID conversion issues
> -------------------------------------------------------------
>
> Key: HIVE-29798
> URL: https://issues.apache.org/jira/browse/HIVE-29798
> Project: Hive
> Issue Type: Bug
> Reporter: László Bodor
> Assignee: László Bodor
> Priority: Major
> Labels: pull-request-available
> Attachments: TestUnionAllToAcidConversionReadFails.java
>
>
> Update: this change is finally merged together with HIVE-29799, because both
> are related to how directory and file layout - coming from a union all query
> - can be converted to ACID.
> 2 issues here:
> 1) the non-flattened union case
> {code}
> Caused by: java.lang.NumberFormatException: For input string: "NION"
> at
> java.base/java.lang.NumberFormatException.forInputString(NumberFormatException.java:67)
> at java.base/java.lang.Long.parseLong(Long.java:709)
> at java.base/java.lang.Long.parseLong(Long.java:832)
> at
> org.apache.hadoop.hive.ql.io.AcidUtils$ParsedDeltaLight.parse(AcidUtils.java:1081)
> at
> org.apache.hadoop.hive.ql.io.orc.VectorizedOrcAcidRowBatchReader.<init>(VectorizedOrcAcidRowBatchReader.java:355)
> at
> org.apache.hadoop.hive.ql.io.orc.VectorizedOrcAcidRowBatchReader.<init>(VectorizedOrcAcidRowBatchReader.java:175)
> at
> org.apache.hadoop.hive.ql.io.orc.VectorizedOrcAcidRowBatchReader.<init>(VectorizedOrcAcidRowBatchReader.java:170)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:2008)
> at
> org.apache.hadoop.hive.ql.io.RecordReaderWrapper.create(RecordReaderWrapper.java:76)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:469)
> ... 28 more
> {code}
> this happens after "alter table tbl convert to acid" as well as altering
> table properties to transactional=true
> 2 scenarios I can imagine:
> 1a) this is a bug, fix in writer path: alter table should take care of
> flattening the union subdirs and converting the file structure to an ACID
> compliant layout
> 1b) this is a bug, fix on reader path: considering [a unit
> test|https://github.com/apache/hive/blob/9173850b9d8d51b253a0763115a6e5720b040640/ql/src/test/org/apache/hadoop/hive/ql/TestTxnNoBuckets.java#L418-L437],
> having the union subdirs in acid tables is totally fine, so no need to
> flatten
> 2. this is by design, and tables containing union subdirs must be flattened
> BEFORE converting them to ACID
> I'm about to handle this as 1b) as it makes to most sense to me
> 2. The flattened union case: table containing flattened union files cannot be
> converted to ACID
> MoveTask.flattenUnionSubdirectories produces <index>000000_0 files (three
> numeric parts) which don't match the metastore's ORIGINAL_PATTERN
> ([0-9][0-9]), so a subsequent ACID conversion is rejected by
> TransactionalValidationListener.validateTableStructureForPath with
> "Unexpected data file name format". The tests assert exactly that failure and
> include a comment pointing at what should change to make them assert
> successful conversion instead.
> 1. Reconsider the file format after flattening
> 2. Prepare ACID format patterns to consume these files: 1_000000_0,
> 2_000000_0 f, 3_000000_0 ...
> I would go for 1) first, then worst case 2)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
