Re: [I] [Feature Request]: Data vault 2.0 wizard Industrial ETL production (hop)
mattcasters commented on issue #7077: URL: https://github.com/apache/hop/issues/7077#issuecomment-4573482683 Desired high level architecture: ``` [DV Metadata Model (.json)] ──> [Run Request] ──> [In-Memory Compiling (PipelineMeta)] ──> [Immediate Execution] ``` Phase 1: Establish the Data Vault Model Schema (@HopMetadata) Phase 2: Build the Visual Modeling Perspective (Hop GUI) Phase 3: Develop the Custom Workflow Action (@Action) Phase 4: Debugging and Catalog Optimization Key design principles: Zero Deployments: The model lives as pure metadata. Zero Database Overload: No heavy, expensive SQL joins are forced onto the data warehouse for delta checking. Minimal I/O: Millions of unchanged rows are dropped in memory before they ever hit the network or storage layer. On CDC for satelites: ``` [Source DB / Iceberg] ──> (SQL with ORDER BY BK) ──> [Hop Stream 1] ──┐ ├──> [Merge Rows (diff)] ──> [Filter: Changed/New] ──> [Target Append] [Target Satellite] ──> (SQL with ORDER BY BK) ──> [Hop Stream 2] ───┘ ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] [Feature Request]: Data vault 2.0 wizard Industrial ETL production (hop)
mhamedbenjmaa commented on issue #7077: URL: https://github.com/apache/hop/issues/7077#issuecomment-4416440708 @mattcasters I have to say, I'm starting to notice a pattern here... 4 RFEs, 4 "I like the idea." That's a 100% success rate for me, Matt. Not even Amazon gets those numbers!!!. At this point I'm not sure if I'm contributing to Apache Hop or if Apache Hop is just quietly adopting me one feature request at a time. Either way, I'm here for it. So let's make it official adopt me into this project and together we will build the Death Star of data engineering. Pipelines so powerful, so elegant, that Airflow will tremble, Spark will rethink its life choices, and dbt will go back to school hahah. We're talking galactic-scale data orchestration. You bring 20+ years of Kettle wisdom, I bring 20 years of battle scars and apparently a 100% idea approval rating from the most important voter in the room. The universe won't know what hit it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] [Feature Request]: Data vault 2.0 wizard Industrial ETL production (hop)
mattcasters commented on issue #7077: URL: https://github.com/apache/hop/issues/7077#issuecomment-4377319697 When I said I didn't like the wizard approach I meant that it's better to completely generate and run the workflows and pipelines in the background, only to be made visible for debugging. We need to go metadata driven all the way IMO. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] [Feature Request]: Data vault 2.0 wizard Industrial ETL production (hop)
mattcasters commented on issue #7077: URL: https://github.com/apache/hop/issues/7077#issuecomment-4372379863 I like the idea of model-driven Data Vault 2.0 population. I'm not a fan of any type of wizard in this regard. Personally, if there's an opportunity it's in the possibility to model the vault in Hop metadata, versioned by git as per usual. Now., in a previous life a man called Dan Linstedt asked me to write exactly this for Data Vault 1.0 and I simply didn't have the time or usecase for it even though I liked the idea. It's what's behind whole companies like [VaultSpeed](http://www.vaultspeed.com) so it's no small task to create this. I'd need to think about it and have an incentive before I'd start on this one ;-) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] [Feature Request]: Data vault 2.0 wizard Industrial ETL production (hop)
mhamedbenjmaa commented on issue #7077: URL: https://github.com/apache/hop/issues/7077#issuecomment-4366623650 I asked claude to rewrite it :D what about now ? :D -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] [Feature Request]: Data vault 2.0 wizard Industrial ETL production (hop)
hansva commented on issue #7077: URL: https://github.com/apache/hop/issues/7077#issuecomment-4363273797 This sounds like a discussion and not a feature request -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
