abstractdog commented on code in PR #6642:
URL: https://github.com/apache/hive/pull/6642#discussion_r3728881159
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/ParsedOutputFileName.java:
##########
@@ -41,9 +44,9 @@ public class ParsedOutputFileName {
private static final Pattern COPY_FILE_NAME_TO_TASK_ID_REGEX =
Pattern.compile(
"^(.*?)?" + // any prefix
"(\\(.*\\))?" + // taskId prefix
- "([0-9]+)" + // taskId
- "(?:_([0-9]{1,6}))?" + // _<attemptId> (limited to 6 digits)
- "(?:_copy_([0-9]{1,6}))?" + // copy file index
+ "(\\d+)" + // taskId
+ "(?:_(\\d{1,6}))?" + // _<attemptId> (limited to 6 digits)
+ "(?:_copy_(\\d{1,6}|[\\da-fA-F]{8}))?" + // copy suffix: numeric
counter, or 8-hex uniqueness tag
Review Comment:
valid concert, it was addressed by changing the pattern, also introduce unit
tests that failed without properly patching this, I saw 2 different exceptions:
```
ERROR : DDLTask failed, DDL Operation: class
org.apache.hadoop.hive.ql.ddl.table.misc.properties.AlterTableSetPropertiesOperation
org.apache.hadoop.hive.ql.metadata.HiveException: Unable to alter table.
java.lang.IllegalStateException: Unexpected data file name format. Cannot
convert default.t_acid_demo to transactional table. File:
s3a://dw-team-bucket/tmp/t_acid_demo/000000_0_copy_f0796c02aef8435d
at org.apache.hadoop.hive.ql.metadata.Hive.alterTable(Hive.java:1007)
at org.apache.hadoop.hive.ql.metadata.Hive.alterTable(Hive.java:943)
at
org.apache.hadoop.hive.ql.ddl.table.AbstractAlterTableOperation.finalizeAlterTableWithWriteIdOp(AbstractAlterTableOperation.java:163)
at
org.apache.hadoop.hive.ql.ddl.table.AbstractAlterTableOperation.execute(AbstractAlterTableOperation.java:82)
at org.apache.hadoop.hive.ql.ddl.DDLTask.execute(DDLTask.java:84)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:354)
at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:327)
at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:244)
at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:105)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:346)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:191)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:143)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:138)
at
org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:190)
at
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:234)
at
org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:334)
at
java.base/java.security.AccessController.doPrivileged(AccessController.java:714)
at java.base/javax.security.auth.Subject.doAs(Subject.java:525)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1953)
at
org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:354)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
```
and:
```
Caused by: java.lang.IllegalArgumentException: Bucket ID out of range: -1
at
org.apache.hive.com.google.common.base.Preconditions.checkArgument(Preconditions.java:134)
at
org.apache.hadoop.hive.ql.io.BucketCodec$2.encode(BucketCodec.java:103)
at
org.apache.hadoop.hive.ql.io.orc.VectorizedOrcAcidRowBatchReader.computeOffsetAndBucket(VectorizedOrcAcidRowBatchReader.java:797)
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.callInternal(OrcInputFormat.java:1548)
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator$1.run(OrcInputFormat.java:1535)
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator$1.run(OrcInputFormat.java:1532)
at
java.base/java.security.AccessController.doPrivileged(AccessController.java:714)
at java.base/javax.security.auth.Subject.doAs(Subject.java:525)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1953)
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.call(OrcInputFormat.java:1532)
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.call(OrcInputFormat.java:1348)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
... 3 more
```
[TestInsertCopySuffixOnFakeS3.java](https://github.com/apache/hive/pull/6642/changes#diff-f61872cb69734c3f4f2a5e9a271bdcaaa58a9c8a19b54618cb32e27cb5ff7990)
extensively tests different source tables converted to ACID, also
[acid_convert_16hex_copy_tag.q](https://github.com/apache/hive/pull/6642/changes#diff-36d468488649e42ba8d13c4a62fda25936321acb7ee28b379e67f93dc5908249)
was added for the same
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
