r7raul1984 opened a new issue, #13629:
URL: https://github.com/apache/gluten/issues/13629

   ### Backend
   
   VL (Velox)
   
   ### Bug description
   
   
   Recursive Hive scans can include data files stored under hidden directories.
   
   When `mapreduce.input.fileinputformat.input.dir.recursive=true`,
   `HivePartitionConverter` recursively enters subdirectories before filtering
   hidden paths. The final filtering checks only each file's name, so normally
   named files inside `_temporary`, `.backup`, or nested hidden directories
   can be included in the scan.
   
   ### Example directory layout
   
   ```text
   table/
     root-data
     visible/
       nested-data
       _temporary/
         nested-uncommitted-data
       .backup/
         nested-backup-data
     _temporary/
       attempt/
         uncommitted-data
     .backup/
       backup-data
   ```
   
   All files in this example contain a non-empty line of text, such as `1\n`.
   
   ### Expected behavior
   
   With recursive scanning enabled, only these files should be included:
   
   - `root-data`
   - `visible/nested-data`
   
   Hidden directories and their descendants should be skipped, matching
   Hadoop `TextInputFormat`.
   
   ### Actual behavior
   
   `HivePartitionConverter` includes all six files, including four files under
   hidden directories. This can cause scans to read temporary or backup data
   and return unexpected rows.
   
   With recursive scanning disabled, both `HivePartitionConverter` and
   Hadoop `TextInputFormat` include only `root-data`.
   
   ### Reproduction and validation
   
   The file-listing discrepancy was reproduced by compiling the current
   `HivePartitionConverter` source and invoking its file-listing method,
   then comparing the results with Hadoop `TextInputFormat` on the same 
directory.
   This reproduction checks file selection directly; it does not execute a
   native reader or a SQL query.
   
   Before the fix:
   
   | Recursive scanning | Hadoop file count | Gluten file count |
   | --- | --- | --- |
   | Disabled | 1 | 1 |
   | Enabled | 2 | 6 |
   
   A regression test was added under:
   
   ```text
   
gluten-ut/spark35/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveTableScanSuite.scala
   ```
   
   Test name:
   
   ```text
   Gluten - Hive recursive file listing excludes hidden directories
   ```
   
   The test compares file enumeration with Hadoop `TextInputFormat`, including
   visible subdirectories, hidden directories at multiple levels, and hidden 
files.
   
   After filtering hidden paths before descending into directories, the
   regression test passed with Spark 3.5 / Scala 2.12 and the Velox profile:
   one test succeeded, zero failed.
   
   ### Affected code
   
   ```text
   
gluten-substrait/src/main/scala/org/apache/spark/sql/hive/HivePartitionConverter.scala
   ```
   
   Both the initial directory listing and recursive directory traversal need
   to filter hidden paths before descending.
   
   ### Related issues
   
   - https://github.com/apache/gluten/issues/3535
   - https://github.com/apache/gluten/pull/3536
   
   These introduced recursive Hive directory scanning. This issue concerns
   hidden-directory filtering in that scanning path.
   
   This issue description was prepared with assistance from
   OpenAI Codex (gpt-6.1-sol).
   
   ### Gluten version
   
   main branch
   
   ### Spark version
   
   _No response_
   
   ### Spark configurations
   
   ```text
   spark.hadoop.mapreduce.input.fileinputformat.input.dir.recursive=true
   ```
   
   Spark forwards this setting to Hadoop using the `spark.hadoop.` prefix.
   
   The reproduction sets Hadoop configuration
   `mapreduce.input.fileinputformat.input.dir.recursive=true` directly when
   constructing `HivePartitionConverter`. The regression test covers both
   `true` and `false`.
   
   ### System information
   
   _No response_
   
   ### Relevant logs
   
   ```bash
   
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to