aglinxinyuan opened a new pull request, #4490: URL: https://github.com/apache/texera/pull/4490
### What changes were proposed in this PR? This PR adds state materialization as a general mechanism for passing state across different regions. - materialize state as a separate storage object alongside result storage - store one serialized state per row - use the same cross-language format on Python and Scala/Java sides - let downstream regions read back both data and state when needed ### Any related issues, documentation, discussions? Closes #4489 ### How was this PR tested? - added a Python round-trip test for materialized state storage in test_iceberg_document.py - added a Scala round-trip test for materialized state storage in IcebergDocumentSpec.scala - ran sbt "WorkflowCore / Test / testOnly org.apache.texera.amber.storage.result.iceberg.IcebergDocumentSpec" successfully - Python test code was added and syntax-checked, but local execution was blocked by Postgres authentication for the Iceberg catalog ### Was this PR authored or co-authored using generative AI tooling? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
