Hi team, I have started a discussion regarding the decision to merge the Python Notebook Migration tool in a single PR, despite its large size. The post is a summary of our past discussions about this topic. Anyone can feel free to chime in with thoughts.
Best, Ryan Zhang On Thu, Apr 30, 2026 at 9:03 PM zyratlo (via GitHub) <[email protected]> wrote: > > GitHub user zyratlo edited a discussion: Merge the Python Notebook > Migration Tool in a Single PR > > ## Summary > A new feature currently in the review phase before being merged is the > Python Notebook Migration Tool. The related pages are Issue #4301 and PR > #4486 . As of the time of this writing, the PR has 37 files changed and > 3,342 additions. We have decided to currently merge this large addition to > the code base in a single PR for multiple reasons: > > - The code is not able to be sliced vertically to merge in multiple PR's > - While portions of the code can be merged across multiple PR's, since > we want the tool to still be functional in each PR we cannot divide it > while maintaining functionality > - The code can be merged safely in one PR since it does not interfere or > modify with any other parts of the code base. > - Existing files which are modified are only changed to add > functionality for this tool, and ensures any other pre-existing code is > unchanged > - A full breakdown of the new code can be found below > > ## Architecture > As shown in the architecture diagrams below, a large portion of the new > code is contained within new files. Portions of the front-end and existing > services (such as the LiteLLM proxy) are reused while maintaining their > original purposes are unchanged. > > <img width="1333" height="942" alt="Image" src=" > https://urldefense.com/v3/__https://github.com/user-attachments/assets/ce31d2ed-2aad-402c-9ec7-02d58ec53980__;!!CzAuKJ42GuquVTTmVmPViYEvSg!Kg0y6WvGldZWwU_KxhMLqKCLBZ1xFUO7taYmi_BblZwrSXA3PX3hJCHt9KIyOb7BgeukgYSZYA$ > " /> > > <img width="1332" height="935" alt="Image" src=" > https://urldefense.com/v3/__https://github.com/user-attachments/assets/b34491a3-9033-4e93-a303-07806079ab07__;!!CzAuKJ42GuquVTTmVmPViYEvSg!Kg0y6WvGldZWwU_KxhMLqKCLBZ1xFUO7taYmi_BblZwrSXA3PX3hJCHt9KIyOb7BgeuOk9P3iw$ > " /> > > <img width="1343" height="938" alt="Image" src=" > https://urldefense.com/v3/__https://github.com/user-attachments/assets/a3e6d77a-e369-40dd-85f5-47be4eb2d4f9__;!!CzAuKJ42GuquVTTmVmPViYEvSg!Kg0y6WvGldZWwU_KxhMLqKCLBZ1xFUO7taYmi_BblZwrSXA3PX3hJCHt9KIyOb7BgesDRgCwDA$ > " /> > > ## Breakdown of Code Changes > More detailed breakdown can be found [here]( > https://urldefense.com/v3/__https://docs.google.com/document/d/1HnE2g2fDXnCw6Qm072_cNZyv4zZ4XRL_AIAEgJ_pYXc/edit?usp=sharing__;!!CzAuKJ42GuquVTTmVmPViYEvSg!Kg0y6WvGldZWwU_KxhMLqKCLBZ1xFUO7taYmi_BblZwrSXA3PX3hJCHt9KIyOb7BgetuU3tDJg$ > ) > ### SQL > - `sql/updates/23.sql` > `sql/texera_ddl.sql` > - Add notebook and workflow_notebook_mapping tables to database > > ### Modified Existing Files > #### Config files > - `build.sbt` > - Add NotebookMigrationService > - `common/config/src/main/resources/storage.conf` > > `common/config/src/main/scala/org/apache/texera/amber/config/StorageConfig.scala` > - Added path for Jupyter server > - `frontend/proxy.config.json` > - Add notebook-migration proxy > - `frontend/src/app/app.module.ts` > - Add Jupyter components/services > #### Front-end > - `frontend/src/app/workspace/component/menu/menu.component.ts` > - Added new button for upload popup and calling necessary services for > the tool > - > `frontend/src/app/workspace/component/workflow-editor/mini-map/workspace.component.ts` > - Added spinner and stopwatch for LLM waiting phase > > ### New Files > #### Front-end > - > `frontend/src/app/workspace/component/jupyter-notebook-panel/jupyter-notebook-panel.component.ts` > - Logic to handle the Jupyter iframe (opening, closing, find the > iframe URL, etc.) > - > `frontend/src/app/workspace/service/jupyter-panel/jupyter-panel.service.ts` > - Handles communication between Jupyter iframe and the workflow > - `frontend/src/app/workspace/service/notebook-migration/migration-llm.ts` > - Handles all communication with LiteLLM proxy > - > `frontend/src/app/workspace/service/notebook-migration/notebook-migration.service.ts` > - Handles communication with back-end and mapping logic > > #### Back-end > - `notebook-migration-service/` > - All back-end code is in this directory and no code outside of it has > been modified > > GitHub link: > https://urldefense.com/v3/__https://github.com/apache/texera/discussions/4590__;!!CzAuKJ42GuquVTTmVmPViYEvSg!Kg0y6WvGldZWwU_KxhMLqKCLBZ1xFUO7taYmi_BblZwrSXA3PX3hJCHt9KIyOb7BgeuRPGH_Kg$ > > ---- > This is an automatically sent email for [email protected]. > To unsubscribe, please send an email to: [email protected] > >
