AliRana30 commented on PR #803: URL: https://github.com/apache/wayang/pull/803#issuecomment-5544586218
cc @zkaoudi @novatechflow Just wanted to highlight the impact of this bug in production pipelines: In cross-platform workflows (e.g., PostgreSQL / BigQuery / Trino ➔ Java Streams / Spark), SQL joins and network transfers are typically the heaviest and most expensive steps. Because there was no plan-time validation, pipelines would spend significant compute time, cluster bandwidth, and cloud warehouse credits running complex joins across large datasets, only to fail at the very final Java step on the first record with a `ClassCastException`. Beyond the wasted compute budget and delayed SLA cycles, if records were accessed by numerical field indices without explicit casts, the flattened tuple structure could have silently corrupted downstream analytics by mapping fields to the wrong offsets. Catching this early during channel conversion and properly preserving `Tuple2<Record, Record>` ensures pipelines fail fast on schema errors and execute reliably across platform boundaries. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
