GitHub user Yicong-Huang added a comment to the discussion: Refactor: Decoupling Direct Database Connection From ComputingUnitMaster & ComputingUnitWorker
First, I want to ask, what is the biggest issue of the current architecture? exposing DB credentials could be one, but that's not that fundamental. I need some more justification for the redesign. That aside, I always wanted to go with option B, and that was the reason we initially introduced the workflow compilation service. but there are some blockers: 1. is it secure to expose physical plan to frontend in option B?? Let's be more clear: after compilation, we will have physical plan which contains resolved file URL. Is it safe to expose that to frontend? What it does not have is the execution configuration (those are only available after scheduling, which currently stays in CU master), so computation resources and worker location are less of a concern. 2. I believe during execution, CU also needs to write stuff, including iceberg metadata, runtime logs, etc. How are we going to handle those DB access with option B? GitHub link: https://github.com/apache/texera/discussions/5295#discussioncomment-17113981 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
