jiayuasu opened a new pull request, #2660: URL: https://github.com/apache/sedona/pull/2660
## Did you read the Contributor Guide? - Yes, I have read the [Contributor Rules](https://sedona.apache.org/latest/community/rule/) and [Contributor Development Guide](https://sedona.apache.org/latest/community/develop/) ## Is this PR related to a ticket? - Yes, and the PR name follows the format `[GH-XXX] my subject`. Closes #2659 ## What changes were proposed in this PR? On Databricks Runtime (both Spark 3.5 and 4.0), the method `DataSource.checkAndGlobPathIfNecessary` has all required parameters with no default values. This differs from OSS Apache Spark, where some parameters have defaults. The previous code used Scala named parameters when calling this method, which causes the Scala compiler to emit synthetic `$default$N()` accessor method calls in the bytecode. On Databricks these synthetic accessors do not exist, resulting in a `NoSuchMethodError` at runtime when reading Shapefiles, GeoPackages, or GeoParquet metadata files. This PR changes `SedonaFileIndexHelper` to invoke `DataSource.checkAndGlobPathIfNecessary` via reflection, resolving the method at runtime and passing all arguments explicitly based on the detected parameter count (5 or 6). This avoids any dependency on compiler-generated default parameter accessors and works across OSS Spark 3.x/4.x and Databricks Runtime. **Affected readers (all go through the fixed `SedonaFileIndexHelper.createFileIndex`):** - Shapefile (`ShapefileTable`) - GeoPackage (`GeoPackageTable`) - GeoParquet metadata (`GeoParquetMetadataTable`) ## How was this patch tested? Verified that Scala compilation succeeds across all Spark versions (3.4, 3.5, 4.0, 4.1). The runtime fix is a reflection-based dispatch that will be validated on Databricks. ## Did this PR include necessary documentation updates? - No, this PR does not affect any public API so no need to change the documentation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
