ReemaAlzaid opened a new pull request, #13599:
URL: https://github.com/apache/gluten/pull/13599

   ## What changes are proposed in this pull request?
   
   Expand Kafka streaming test coverage for Gluten on Spark 3.5 and add a 
bounded Structured Streaming benchmark.
   
   ### Kafka tests
   
   Reuse Spark’s `KafkaMicroBatchV2SourceSuite` and `KafkaSourceStressSuite` 
through Gluten wrappers, with test dependencies, suite registration, and CI 
execution.
   
   Correctness coverage includes:
   - Starting offsets and partition changes.
   - Restarting queries from checkpoints.
   - Processing available records with AvailableNow.
   - Limiting offsets per batch with `maxOffsetsPerTrigger`.
   - Handling missing data according to `failOnDataLoss`.
   
   Tests verify output correctness and execution plans. Supported reads must 
use Gluten’s native Kafka scan. Reads that retain headers must fall back to 
Spark and preserve header values.
   
   ```mermaid
   flowchart TD
       A["Spark Kafka test suites"] --> B["Gluten wrappers"]
       B --> C["Quick correctness checks"]
       B --> D["Separate stress runs"]
       C --> E["Verify expected records"]
       E --> F["Supported reads: native Kafka scan"]
       E --> G["Retained headers: Spark fallback"]
   ```
   
   For example, the checkpoint restart test writes Kafka records to Parquet, 
stops the query, adds records, and restarts from the same checkpoint. It checks 
that the output contains all expected records without duplicates and that 
supported reads use native scans.
   
   Longer stress runs remain separate from quick correctness checks. Exclusions 
are documented individually. The added tests wait for preloaded Kafka records 
to be processed before checking results.
   
   ### Streaming benchmarks
   
   Add a bounded rate-source benchmark with configurable input rate, 
partitions, trigger interval, warm-up, and measurement duration. A no-op sink 
excludes console printing from measured work.
   
   Extend the existing Kafka-to-Parquet benchmark with checkpoint handling and 
Spark-versus-Gluten comparisons. Report processed rows, throughput in 
rows/second, and micro-batch durations in milliseconds. Stop queries and clean 
up listeners and resources reliably.
   
   The rate-source benchmark does not read Kafka or validate native Kafka scans.
   
   ## How was this patch tested?
   
   - Native Velox and Gluten build completed successfully on Linux.
   - Spark 3.5 Kafka correctness suite ran 75 tests: 22 passed and 53 failed.
   - Visible failures include unexpected Spark fallback and premature result 
checks in two added tests.
   - Fixed the premature result checks using Spark’s existing processing-wait 
helper. Results after this fix are pending.
   - Scala formatting and `git diff --check` passed for the timing fix.
   - Native Kafka connector test and stress-suite results have not been 
confirmed.
   
   The tests currently show Kafka reads falling back to Spark. This checkout 
lacks the Velox Kafka integration from upstream PR #13237, so native Kafka scan 
validation is not yet complete.
   
   ## Was this patch authored or co-authored using generative AI tooling?
   
   Generated-by: OpenAI Codex (GPT-6)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to