AliRana30 opened a new pull request, #798:
URL: https://github.com/apache/wayang/pull/798
## Description
Closes #761
Currently, `SqlToRddOperator` executes JDBC queries and materializes the
entire `ResultSet` into a `java.util.List` in the Spark driver JVM memory
before calling `sc.parallelize(...)`. On large analytical engines (such as
Trino, Presto, BigQuery, and PostgreSQL), this causes out-of-memory bottlenecks
on the driver and prevents Spark from leveraging distributed JDBC execution.
This PR refactors `SqlToRddOperator` to use Spark's distributed
`DataFrameReader` JDBC interface (`executor.ss.read().format("jdbc")`):
- Reads JDBC data directly into a distributed `Dataset<Row>`.
- Converts `Dataset<Row>` to `JavaRDD<Record>` lazily across Spark executors
using a static serializable helper method (`rowToRecord`).
- Sanitizes incoming SQL queries and wraps subqueries in derived tables
`(query) as wayang_subquery`.
- Exposes connection parameter getters on `DatabaseDescriptor`
(`getJdbcUrl()`, `getUser()`, `getPassword()`, `getJdbcDriverClassName()`).
- Adds support for optional JDBC partitioning configurations
(`partitionColumn`, `lowerBound`, `upperBound`, `numPartitions`) and
`fetchsize`.
- Implements `toJson()` and `fromJson()` serialization and load profile
estimator keys.
## Type of Change
- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to not work as expected)
- [ ] Documentation update
## How Has This Been Tested?
- Added `testRowToRecord` and `testJsonSerialization` unit tests to
`SqlToRddOperatorTest`.
- Verified query formatting and schema conversion compatibility with HSQLDB
platform tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
