Brb6754 opened a new issue, #8524:
URL: https://github.com/apache/texera/issues/8524
## The problem
Texera records a great deal about every workflow and shows the user almost
none
of it. Two things in particular:
- **Every save.** `workflow_version` holds one inverse JSON patch per
version,
so the entire history of a canvas is recoverable. The user sees a list of
version names and a restore button.
- **Every run.** A run's results are dropped thirty seconds after the
workflow
goes idle, but its runtime statistics are not: they stay in an Iceberg
table
with a row per operator per sample. Rows in and out of every step of every
run
ever made are still there, and nothing surfaces them.
This matters most for the people Texera is built for. Someone who joins a
project has no way to find out what a workflow does beyond reading the canvas
operator by operator, and no way at all to find out what has already been
tried.
Someone who returns to their own workflow after two weeks is in the same
position. And when results change, working out which edit did it means
guessing.
## The proposal
A panel, opened from the workspace menu, that reads what is already recorded
and
answers four questions in plain language:
1. **How does this work?** The flow as one drawing, each pipe as thick as the
rows that run through it, with a step inspector alongside.
2. **What did I change?** Every save as a block on a ribbon, the canvas
rebuilt
at whichever one is picked, and the edits of that save listed in words.
3. **What has been tried?** Every run read as an experiment rather than a log
line, so the same settings run five times count as one thing tried.
4. **Why did my results change?** Two runs compared step by step, naming the
first step whose output differs and the edit behind it.
Two rules I think are worth fixing in the design from the start:
- **Everything asserted is derived and checkable.** A language model is used
only to put already-derived facts into prose. It never decides which step
diverged or which edit caused it. With no model configured the panel loses
its
sentences and nothing else.
- **Ordering is not a change.** A step that returns the same rows in a
different
order is reported as such, not as a difference, because that ordering comes
from how work was split between workers.
## The open design question
Comparing two runs row by row needs the rows, and Texera drops them. The
draft
currently keeps a copy as JSON on the service's own disk, which is the
weakest
part of it and should not survive review.
The rows do not need copying at all. When a run finishes they are already in
an
Iceberg table; they disappear because
`WorkflowService.clearExecutionResources`
clears them once the workflow has been idle for
`executionStateCleanUpInSecs`.
So keeping a run for comparison is a flag, not a write: mark the execution as
retained and let the cleanup skip it, the way it already skips per-user
warehouses via `WarehouseReadGuard.skipWhileDisabled`. That would need a
retention policy alongside it, since nothing would otherwise free the space.
That is probably useful beyond this panel: "keep this execution's results"
is a
thing users ask for on its own.
## Where it stands
A working draft is at #PR, with the panel in the GUI and a `spy-service` that
answers it. I would like to know whether the direction is worth pursuing, and
what the committers think about the retention question, before taking it
further.
Verified against a workflow with 61 saves and 21 runs: all 61 versions
rebuild
and match what `GET /api/version/{wid}/{vid}` returns, one by one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
