ReemaAlzaid opened a new pull request, #13599:
URL: https://github.com/apache/gluten/pull/13599
## What changes are proposed in this pull request?
Expand Kafka streaming test coverage for Gluten on Spark 3.5 and add a
bounded Structured Streaming benchmark.
### Kafka tests
Reuse Spark’s `KafkaMicroBatchV2SourceSuite` and `KafkaSourceStressSuite`
through Gluten wrappers, with test dependencies, suite registration, and CI
execution.
Correctness coverage includes:
- Starting offsets and partition changes.
- Restarting queries from checkpoints.
- Processing available records with AvailableNow.
- Limiting offsets per batch with `maxOffsetsPerTrigger`.
- Handling missing data according to `failOnDataLoss`.
Tests verify output correctness and execution plans. Supported reads must
use Gluten’s native Kafka scan. Reads that retain headers must fall back to
Spark and preserve header values.
```mermaid
flowchart TD
A["Spark Kafka test suites"] --> B["Gluten wrappers"]
B --> C["Quick correctness checks"]
B --> D["Separate stress runs"]
C --> E["Verify expected records"]
E --> F["Supported reads: native Kafka scan"]
E --> G["Retained headers: Spark fallback"]
```
For example, the checkpoint restart test writes Kafka records to Parquet,
stops the query, adds records, and restarts from the same checkpoint. It checks
that the output contains all expected records without duplicates and that
supported reads use native scans.
Longer stress runs remain separate from quick correctness checks. Exclusions
are documented individually. The added tests wait for preloaded Kafka records
to be processed before checking results.
### Streaming benchmarks
Add a bounded rate-source benchmark with configurable input rate,
partitions, trigger interval, warm-up, and measurement duration. A no-op sink
excludes console printing from measured work.
Extend the existing Kafka-to-Parquet benchmark with checkpoint handling and
Spark-versus-Gluten comparisons. Report processed rows, throughput in
rows/second, and micro-batch durations in milliseconds. Stop queries and clean
up listeners and resources reliably.
The rate-source benchmark does not read Kafka or validate native Kafka scans.
## How was this patch tested?
- Native Velox and Gluten build completed successfully on Linux.
- Spark 3.5 Kafka correctness suite ran 75 tests: 22 passed and 53 failed.
- Visible failures include unexpected Spark fallback and premature result
checks in two added tests.
- Fixed the premature result checks using Spark’s existing processing-wait
helper. Results after this fix are pending.
- Scala formatting and `git diff --check` passed for the timing fix.
- Native Kafka connector test and stress-suite results have not been
confirmed.
The tests currently show Kafka reads falling back to Spark. This checkout
lacks the Velox Kafka integration from upstream PR #13237, so native Kafka scan
validation is not yet complete.
## Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-6)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]