abstractdog commented on PR #6642: URL: https://github.com/apache/hive/pull/6642#issuecomment-5204985852
> > The tag is per-query, but a single query can move multiple files with the same basename into the same destination directory, isn't it? > > ``` > > INSERT INTO t SELECT ... UNION ALL SELECT .. > > ``` > > > > > > > > > > > > > > > > > > > > > > > > FS > > ``` > > -ext-10000/HIVE_UNION_SUBDIR_1/000000_0, > > --> 000000_0_copy_<tag> > > -ext-10000/HIVE_UNION_SUBDIR_2/000000_0 > > ``` > > > > > > > > > > > > > > > > > > > > > > > > On master, the exists-probe loop resolves it (000000_0 + 000000_0_copy_1); under the PR both legs compute the same 000000_0_copy_ and collide > > [HIVE-21100](https://issues.apache.org/jira/browse/HIVE-21100) seems to add branch index, so we might be sorted > > ack, this has to be sorted now, because "the exists-probe loop resolves it" is just true to a certain extent, which is still subject to the reported problem, which is the race in multiple places in the copy++ loop: I'm going to address this as well and let you know > > regarding [HIVE-21100](https://issues.apache.org/jira/browse/HIVE-21100) that's another area that might be investigated, because it claims: `// when we move the files to the parent directory. Ex. HIVE_UNION_SUBDIR_1/000000_0 -> 1_000000_0` but there is no guarantee that multiple union queries with flattening enabled don't clash, so how to resolve two final/"flattened" files arriving as `1_000000_0`: this is not the current `Hive.mvFile` bug, but something that has to be sorted out separately, maybe, I'll think about it after thorough investigation it turned out this patch doesn't introduce regression in terms of union, but unions already have their own issues regardless, which are being addressed in the scope of [HIVE-29798](https://issues.apache.org/jira/browse/HIVE-29798) let me explain the 2 distinct union cases, which had to be checked from this patch's point of view: 1. `hive.tez.union.flatten.subdirectories=false` source dir example: ``` sourcePath: s3a://dw-team-bucket/tmp/lbodor/uall_ext_dst/.hive-staging_hive_2026-07-31_02-37-57_520_1261345641459976868-3/-ext-10000/HIVE_UNION_SUBDIR_1 ``` destination dir: ``` 2026-07-31 12:40:19 0 tmp/lbodor/uall_ext_dst/ 2026-07-31 12:42:49 0 tmp/lbodor/uall_ext_dst/HIVE_UNION_SUBDIR_1/ 2026-07-31 12:42:50 736 tmp/lbodor/uall_ext_dst/HIVE_UNION_SUBDIR_1/000000_0_copy_b79125e672424f25 ``` be mindful that in order to preserve this behavior, I had to add an extra `isFile` check before applying the uniqueness tag, as `Hive.mvFile` can be called with source files and source folders, and `HIVE_UNION_SUBDIR_1` is a typical case of the 'folder', in which case the uniqueness logic doesn't kick in, as it's supposed to handle only file collisions 2. `hive.tez.union.flatten.subdirectories=true` the flattening happens in the staging dir, seeing these in the logs: ``` 2026-07-31T02:48:46,165 INFO [HiveServer2-Background-Pool: Thread-1195] exec.MoveTask: This subdirectory has been flattened: s3a://dw-team-bucket/tmp/lbodor/uall_ext_dst/.hive-staging_hive_2026-07-31_02-46-02_314_5209122360622121708-3/-ext-10000/HIVE_UNION_SUBDIR_5 ``` giving flattened files like: ``` 2026-07-31 11:48:22 653 tmp/lbodor/uall_ext_dst/.hive-staging_hive_2026-07-31_02-46-02_314_5209122360622121708-3/-ext-10000/10_000000_0 2026-07-31 11:48:20 653 tmp/lbodor/uall_ext_dst/.hive-staging_hive_2026-07-31_02-46-02_314_5209122360622121708-3/-ext-10000/1_000000_0 2026-07-31 11:48:24 653 tmp/lbodor/uall_ext_dst/.hive-staging_hive_2026-07-31_02-46-02_314_5209122360622121708-3/-ext-10000/2_000000_0 ``` so when we hit the currently touched codepath, due to the uniqueness logic, this is the source/final path: ``` sourcePath: s3a://dw-team-bucket/tmp/lbodor/uall_ext_dst/.hive-staging_hive_2026-07-31_02-46-02_314_5209122360622121708-3/-ext-10000/10_000000_0 destFilePath: s3a://dw-team-bucket/tmp/lbodor/uall_ext_dst/10_000000_0_copy_216e01f1530c4d9d ``` so due to the flattening code, the files don't collide on query level due to the prefix, and they end up as final paths like below: ``` 2026-07-31 11:46:00 0 tmp/lbodor/uall_ext_dst/ 2026-07-31 11:49:37 653 tmp/lbodor/uall_ext_dst/10_000000_0_copy_216e01f1530c4d9d 2026-07-31 11:49:37 653 tmp/lbodor/uall_ext_dst/1_000000_0_copy_216e01f1530c4d9d 2026-07-31 11:49:37 653 tmp/lbodor/uall_ext_dst/2_000000_0_copy_216e01f1530c4d9d 2026-07-31 11:49:37 653 tmp/lbodor/uall_ext_dst/3_000000_0_copy_216e01f1530c4d9d 2026-07-31 11:49:37 653 tmp/lbodor/uall_ext_dst/4_000000_0_copy_216e01f1530c4d9d 2026-07-31 11:49:37 653 tmp/lbodor/uall_ext_dst/5_000000_0_copy_216e01f1530c4d9d 2026-07-31 11:49:37 653 tmp/lbodor/uall_ext_dst/6_000000_0_copy_216e01f1530c4d9d 2026-07-31 11:49:37 653 tmp/lbodor/uall_ext_dst/7_000000_0_copy_216e01f1530c4d9d 2026-07-31 11:49:37 653 tmp/lbodor/uall_ext_dst/8_000000_0_copy_216e01f1530c4d9d 2026-07-31 11:49:37 653 tmp/lbodor/uall_ext_dst/9_000000_0_copy_216e01f1530c4d9d ``` the only problem is that these files are not ACID copy compliant due to the prefix, but it's not because of this patch and is going to be fixed in [HIVE-29798](https://issues.apache.org/jira/browse/HIVE-29798): I mean, on master, a unit test already files while converting flattened files to ACID (see [HIVE-29799](https://issues.apache.org/jira/browse/HIVE-29799), that I'll fixed together with HIVE-29798) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
