GitHub user marcopistacchio added a comment to the discussion: Hop Web | Airflow HTTP operator
Thank you, @sramazzina. I really appreciate the feedback 👍 > Safer and cleaner. May I ask you to expand on why, in your opinion, it is safer and cleaner architecture to use Hop Run via DockerOperator rather than a remote pipeline engine with HTTPOperator? I am testing 2 deploy options described below. 1. Hop Run via DockerOperator. - Hop Web running in a remote instance, deployed via Docker. - No need to deploy Hop Server. - Airflow running in a separate remote instance. - Persistent Disk volumes to ensure the Hop Project is available in the Airflow instance. - Run Hop Pipelines via Hop Run directly on the Airflow instance. Pros: No need for Hop Server Cons: Airflow and Hop deployments are tightly coupled. For example, must scale the Airflow instance to add resources to the Hop Pipelines. 2. Remote pipeline engine with HTTPOperator - Hop Web and Hop Server running in the same remote instance, deployed via Docker. - Airflow running in a separate remote instance. - Docker volumes between Web + Server containers to ensure the Hop Project is available in the Hop Server. - Run Hop Pipelines via the remote pipeline engine directly on the Hop Server. Pros: Simpler management(in my opinion) of Persistent volumes and a less-coupled deployment architecture. Cons: Added complexity in managing Hop Server deployment. Thanks again for the feedback! GitHub link: https://github.com/apache/hop/discussions/6095#discussioncomment-15116001 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
