GitHub user carloea2 created a discussion: Proposal: Statement-level Python-to-Texera workflow compiler MVP
## Summary This proposal defines a deliberately narrow MVP that translates supported Python source into an ordinary Texera workflow of Python UDF operators. The implementation will reuse the existing compiler kernel and its working analysis, checking, realization, and rendering machinery. This is a constrained modular composition of that compiler, not a second compiler or a rewrite from scratch. ## MVP source subset The MVP accepts top-level, straight-line Python statements: - imports, assignments, expressions, and supported data dependencies; - calls to imported or library functions remain ordinary Python calls; - each complete statement is the minimum placement unit. The frontend rejects unsupported constructs with a typed diagnostic. The MVP does not admit local function or class definitions, lambdas, or control-flow statements such as `if`, `for`, `while`, `try`, `with`, `match`, and async control flow. It does not decompose expressions within a statement. ## Deterministic placement strategies The same analyzed program can be compiled with either strategy: 1. **One statement per operator**: cut at every legal statement boundary. 2. **SqrtLocContiguous**: target `K = min(statement_count, ceil(sqrt(physical_LOC)))`, then choose deterministic contiguous, acyclic cuts closest to equal physical-LOC partitions. Both strategies operate on a statement-level view of the same dependency graph. They cannot override dependency, transport, or realizability constraints; the checker remains authoritative. ## Architecture ```text source -> one parse: source inventory + existing action forest -> configured analysis passes and facts -> one authoritative dependence graph -> evaluation atoms -> statement placement view -> placement strategy -> coloring expansion -> one checker -> selected internal and boundary realizations -> one renderer -> ordinary Texera workflow ``` An immutable compiler composition selects frontend admission, analysis passes, semantic modules, placement strategy, and target integration. Providers own their fact projectors, and optional capabilities contribute through explicit module interfaces. Adding future support for functions or control flow should change the composition and add modules, not introduce `if mvp` branches or duplicate the graph, checker, or renderer. ## Boundary transport and Amber integration - The existing `PythonValue` boundary realization determines exactly which required values cross each operator boundary. - A shared, generic PyTexera runtime performs export/import using Cloudpickle so aliases, cycles, and supported callables survive the process boundary. - Generated UDFs import that shared runtime instead of embedding a runtime or serializer in every operator. - The result uses Texera's ordinary Python source/tuple operators. - This MVP requires no Amber scheduler, coordinator, recovery, or protocol changes. MOSAIC owns source semantics, dependency analysis, placement, and realization selection. Amber/PyTexera owns the generic execution and transport primitives; it does not learn MOSAIC-specific carriers, colors, or solver rules. ## Validation plan The initial implementation will cover: - positive and negative source-admission tests; - statement-unit closure and deterministic placement tests; - exact required-carrier and Cloudpickle cross-process tests; - semantic parity between the original Python source and the generated workflow; - a real Amber integration test using ordinary Python UDF operators; - a representative straight-line data-science program, with the Wine pipeline expected to produce roughly 7–10 operators under `SqrtLocContiguous`. ## Non-goals This MVP does not include control-flow decomposition, local function/class decomposition, recursion, intra-statement splitting, ML placement, whole-namespace transport, or an Amber engine redesign. Existing complex-case compiler work remains reusable for later modules; it is not being removed or reimplemented for this slice. ## Questions for review 1. Is this straight-line source subset the right boundary for the first system MVP? 2. Should both deterministic placement strategies be exposed initially, or should one-statement-per-operator remain a diagnostic/reference strategy? 3. Is required-value transport through the shared Cloudpickle runtime sufficient for the first integration, with other boundary realizations added later? 4. Are there additional end-to-end cases that should block the MVP before the first PR? GitHub link: https://github.com/apache/texera/discussions/8160 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
