zhangfengcdt opened a new pull request, #1455:
URL: https://github.com/apache/sedona/pull/1455
- Added default Spark configurations in `TestBaseScala` trait.
- Provided a method `defaultSparkConfig` for default configurations.
- Allowed subclasses to override `sparkConfig` to add or modify Spark
configurations.
- Updated initialization of Spark session and Spark context to use the
provided configurations.
- Included comments to explain how to override configurations in subclasses.
- Ensured that default configurations are preserved if not overridden by
subclasses.
## Did you read the Contributor Guide?
- Yes, I have read the [Contributor
Rules](https://sedona.apache.org/latest-snapshot/community/rule/) and
[Contributor Development
Guide](https://sedona.apache.org/latest-snapshot/community/develop/)
## Is this PR related to a JIRA ticket?
- Yes, the URL of the associated JIRA ticket is
https://issues.apache.org/jira/browse/SEDONA-568. The PR name follows the
format `[SEDONA-XXX] my subject`.
## What changes were proposed in this PR?
- Added default Spark configurations in `TestBaseScala` trait.
- Provided a method `defaultSparkConfig` for default configurations.
- Allowed subclasses to override `sparkConfig` to add or modify Spark
configurations.
- Updated initialization of Spark session and Spark context to use the
provided configurations.
- Included comments to explain how to override configurations in subclasses.
- Ensured that default configurations are preserved if not overridden by
subclasses.
After this change, user can either use
- Default Configurations: Provided in defaultSparkConfig.
- Overriding Configurations: Subclasses can override sparkConfig and merge
additional configurations with the default ones using the ++ operator, for
example:
```
class CustomTest extends TestBaseScala {
// Override sparkConfig to provide additional configurations
override def sparkConfig: Map[String, String] = defaultSparkConfig ++ Map(
"spark.sql.warehouse.dir" -> "/custom/warehouse/location",
"spark.some.config.option" -> "some-value"
)
describe("CustomTest") {
it("should use custom Spark configuration") {
assert(sparkSession.conf.get("spark.some.config.option") ==
"some-value")
}
it("should access Spark context") {
assert(sc != null)
}
}
}
```
## How was this patch tested?
All existing unit tests should all pass without any impact.
## Did this PR include necessary documentation updates?
- No, this PR does not affect any public API so no need to change the
documentation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
