Re: [PR] Enhance Edge3 Provider docs [airflow]
pykenny commented on code in PR #49859: URL: https://github.com/apache/airflow/pull/49859#discussion_r2105763692 ## providers/edge3/docs/architecture.rst: ## @@ -0,0 +1,189 @@ + .. Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. + +Edge Provider Architecture +== + +Airflow consist of several components which are connected like in the following diagram. The Edge Worker which is +deployed outside of the central Airflow cluster is connected via HTTP(s) to the API server of the Airflow cluster: + +.. graphviz:: Review Comment: I can use `graphviz` directly instead, since `graphviz` is required library of `diagrams`. (In Airflow's codebase there's only one CLI command script that imports `graphviz`, though) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Enhance Edge3 Provider docs [airflow]
pykenny commented on code in PR #49859: URL: https://github.com/apache/airflow/pull/49859#discussion_r2105763692 ## providers/edge3/docs/architecture.rst: ## @@ -0,0 +1,189 @@ + .. Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. + +Edge Provider Architecture +== + +Airflow consist of several components which are connected like in the following diagram. The Edge Worker which is +deployed outside of the central Airflow cluster is connected via HTTP(s) to the API server of the Airflow cluster: + +.. graphviz:: Review Comment: I can use `graphviz` instead, since `graphviz` is required library of `diagrams`. (In Airflow's codebase there's only one CLI command script that imports `graphviz`, though) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Enhance Edge3 Provider docs [airflow]
pykenny commented on code in PR #49859: URL: https://github.com/apache/airflow/pull/49859#discussion_r2105763692 ## providers/edge3/docs/architecture.rst: ## @@ -0,0 +1,189 @@ + .. Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. + +Edge Provider Architecture +== + +Airflow consist of several components which are connected like in the following diagram. The Edge Worker which is +deployed outside of the central Airflow cluster is connected via HTTP(s) to the API server of the Airflow cluster: + +.. graphviz:: Review Comment: I can use `graphviz` instead, since `graphviz` is required library of `diagrams` (only one CLI command script imports `graphviz`, though) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Enhance Edge3 Provider docs [airflow]
pykenny commented on code in PR #49859: URL: https://github.com/apache/airflow/pull/49859#discussion_r2105763692 ## providers/edge3/docs/architecture.rst: ## @@ -0,0 +1,189 @@ + .. Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. + +Edge Provider Architecture +== + +Airflow consist of several components which are connected like in the following diagram. The Edge Worker which is +deployed outside of the central Airflow cluster is connected via HTTP(s) to the API server of the Airflow cluster: + +.. graphviz:: Review Comment: I can use `graphviz` instead, since `graphviz` is required library of `diagrams`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Enhance Edge3 Provider docs [airflow]
pykenny commented on code in PR #49859: URL: https://github.com/apache/airflow/pull/49859#discussion_r2105763692 ## providers/edge3/docs/architecture.rst: ## @@ -0,0 +1,189 @@ + .. Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. + +Edge Provider Architecture +== + +Airflow consist of several components which are connected like in the following diagram. The Edge Worker which is +deployed outside of the central Airflow cluster is connected via HTTP(s) to the API server of the Airflow cluster: + +.. graphviz:: Review Comment: I can use `graphviz` as well since `graphviz` is required library of `diagrams`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Enhance Edge3 Provider docs [airflow]
jscheffl commented on code in PR #49859: URL: https://github.com/apache/airflow/pull/49859#discussion_r2096387955 ## providers/edge3/docs/architecture.rst: ## @@ -0,0 +1,189 @@ + .. Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. + +Edge Provider Architecture +== + +Airflow consist of several components which are connected like in the following diagram. The Edge Worker which is +deployed outside of the central Airflow cluster is connected via HTTP(s) to the API server of the Airflow cluster: + +.. graphviz:: Review Comment: Mhm, I don't know if the new proposed images are actually "nicer" - I feel like the older ones are good as well. Do we need to change the layout engine for the new images or could we also use GraphViz which is used behind PUML as well? Then we could take-over the existing images and just use Python to attach to enums/interfaces and have a single source but generate the same results. If mixed both "img" and "image" folders are used, I think it would be okay to normalize across repo. Might be good to have oe "normalization" PR and then adding the pre-commit. Would it be meaningful to have a kinf of convention that the python files to generate an image would also resive in "img" folder? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Enhance Edge3 Provider docs [airflow]
pykenny commented on code in PR #49859: URL: https://github.com/apache/airflow/pull/49859#discussion_r2094801210 ## providers/edge3/docs/architecture.rst: ## @@ -0,0 +1,189 @@ + .. Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. + +Edge Provider Architecture +== + +Airflow consist of several components which are connected like in the following diagram. The Edge Worker which is +deployed outside of the central Airflow cluster is connected via HTTP(s) to the API server of the Airflow cluster: + +.. graphviz:: Review Comment: And... do we have (documented) convention for where to place all kinds of assets for rendering documentation? From existing rule Airflow's having both `images` and `img` folders to place images, and that confused me a bit. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Enhance Edge3 Provider docs [airflow]
pykenny commented on code in PR #49859: URL: https://github.com/apache/airflow/pull/49859#discussion_r2095409507 ## providers/edge3/docs/architecture.rst: ## @@ -0,0 +1,189 @@ + .. Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. + +Edge Provider Architecture +== + +Airflow consist of several components which are connected like in the following diagram. The Edge Worker which is +deployed outside of the central Airflow cluster is connected via HTTP(s) to the API server of the Airflow cluster: + +.. graphviz:: Review Comment: Also some initial attempts to plot the two graphviz images with `diagrams` library: ||| |:---:|:---:| | **Architecture** | **Worker State Model** | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Enhance Edge3 Provider docs [airflow]
pykenny commented on code in PR #49859: URL: https://github.com/apache/airflow/pull/49859#discussion_r2094801210 ## providers/edge3/docs/architecture.rst: ## @@ -0,0 +1,189 @@ + .. Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. + +Edge Provider Architecture +== + +Airflow consist of several components which are connected like in the following diagram. The Edge Worker which is +deployed outside of the central Airflow cluster is connected via HTTP(s) to the API server of the Airflow cluster: + +.. graphviz:: Review Comment: And... do we have (documented) convention for where to place a kinds of assets? From existing rule Airflow's having both `images` and `img` folders to place images, and that confused me a bit. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Enhance Edge3 Provider docs [airflow]
pykenny commented on code in PR #49859: URL: https://github.com/apache/airflow/pull/49859#discussion_r2094797540 ## providers/edge3/docs/architecture.rst: ## @@ -0,0 +1,189 @@ + .. Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. + +Edge Provider Architecture +== + +Airflow consist of several components which are connected like in the following diagram. The Edge Worker which is +deployed outside of the central Airflow cluster is connected via HTTP(s) to the API server of the Airflow cluster: + +.. graphviz:: Review Comment: @eladkal I think we need to start with modifying pre commit hooks: https://github.com/apache/airflow/blob/c86a4e1582fde083a4c93ea81955f586009177d1/.pre-commit-config.yaml#L486-L495 Currently, pre-commit hook for updating diagrams only covers scripts under these folders: - Python scripts with `diagram_` prefix under `airflow-core/docs/[SOME_SECTION]/` folders - Python scripts under `docs/images/` folder While provider documentations are now placed within folders that look like `providers/[PROVIDER_TITLE]/docs/`, which are not covered by the path pattern set up in the hook. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Enhance Edge3 Provider docs [airflow]
pykenny commented on code in PR #49859: URL: https://github.com/apache/airflow/pull/49859#discussion_r2094797540 ## providers/edge3/docs/architecture.rst: ## @@ -0,0 +1,189 @@ + .. Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. + +Edge Provider Architecture +== + +Airflow consist of several components which are connected like in the following diagram. The Edge Worker which is +deployed outside of the central Airflow cluster is connected via HTTP(s) to the API server of the Airflow cluster: + +.. graphviz:: Review Comment: @eladkal I think we need to start with modifying pre commit hooks: https://github.com/apache/airflow/blob/c86a4e1582fde083a4c93ea81955f586009177d1/.pre-commit-config.yaml#L486-L495 Currently, pre-commit hook for updating diagrams only covers scripts under these folders: - Python scripts with `diagram_` prefix under `airflow-core/docs/[SOME_SECTION]/` folders - Python scripts under `docs/images/` folder While provider documentations are now placed within folder that looks like `providers/[PROVIDER_TITLE]/docs/`, which is not covered by the path pattern set up in the hook. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Enhance Edge3 Provider docs [airflow]
eladkal commented on code in PR #49859: URL: https://github.com/apache/airflow/pull/49859#discussion_r2079683257 ## providers/edge3/docs/architecture.rst: ## @@ -0,0 +1,189 @@ + .. Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. + +Edge Provider Architecture +== + +Airflow consist of several components which are connected like in the following diagram. The Edge Worker which is +deployed outside of the central Airflow cluster is connected via HTTP(s) to the API server of the Airflow cluster: + +.. graphviz:: Review Comment: @pykenny maybe you'd be interested in giving it a try? As you already handled #46056 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Enhance Edge3 Provider docs [airflow]
jscheffl merged PR #49859: URL: https://github.com/apache/airflow/pull/49859 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Enhance Edge3 Provider docs [airflow]
jscheffl commented on code in PR #49859: URL: https://github.com/apache/airflow/pull/49859#discussion_r2079651743 ## providers/edge3/docs/architecture.rst: ## @@ -0,0 +1,189 @@ + .. Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. + +Edge Provider Architecture +== + +Airflow consist of several components which are connected like in the following diagram. The Edge Worker which is +deployed outside of the central Airflow cluster is connected via HTTP(s) to the API server of the Airflow cluster: + +.. graphviz:: Review Comment: Note that I started with this as a replica of the one in Celery in https://github.com/apache/airflow/blob/main/providers/celery/docs/celery_executor.rst?plain=1#L103 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Enhance Edge3 Provider docs [airflow]
jscheffl commented on code in PR #49859: URL: https://github.com/apache/airflow/pull/49859#discussion_r2079648687 ## providers/edge3/docs/architecture.rst: ## @@ -0,0 +1,189 @@ + .. Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. + +Edge Provider Architecture +== + +Airflow consist of several components which are connected like in the following diagram. The Edge Worker which is +deployed outside of the central Airflow cluster is connected via HTTP(s) to the API server of the Airflow cluster: + +.. graphviz:: Review Comment: Yes we can... would propose this as a future improvement. Until then I'd manually update. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Enhance Edge3 Provider docs [airflow]
eladkal commented on code in PR #49859: URL: https://github.com/apache/airflow/pull/49859#discussion_r2079598081 ## providers/edge3/docs/architecture.rst: ## @@ -0,0 +1,189 @@ + .. Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. + +Edge Provider Architecture +== + +Airflow consist of several components which are connected like in the following diagram. The Edge Worker which is +deployed outside of the central Airflow cluster is connected via HTTP(s) to the API server of the Airflow cluster: + +.. graphviz:: Review Comment: How do we maintain the images created here? I assume we need to remember manually run this code and upload the image manually? In other cases we used diagram lib and automatic update proccess with pre-commit https://github.com/apache/airflow/pull/46056/files would that be applicable here? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Enhance Edge3 Provider docs [airflow]
jscheffl commented on PR #49859: URL: https://github.com/apache/airflow/pull/49859#issuecomment-2848171045 > Ive made some minor corrections to the language, rest is good regarding the CLI commands Thanks for the (native speaker) review! Always good (as non-native speaker)! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Enhance Edge3 Provider docs [airflow]
dheerajturaga commented on code in PR #49859: URL: https://github.com/apache/airflow/pull/49859#discussion_r2072164469 ## providers/edge3/docs/deployment.rst: ## @@ -0,0 +1,172 @@ + .. Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. + +Edge Worker Deployment +== + +Edge Workers can be deployed outside of the central Airflow infrastructure. They +are connected to the Airflow API server via HTTP(s). The Edge Worker is a +lightweight component that can be deployed on any machine that has outbound +HTTP(s) access to the Airflow API server. The Edge Worker is designed to be +lightweight and easy to deploy. It allows you to run Airflow tasks on machines +that are not part of your main data center, e.g. edge servers. This also allows to +deploy only reduced dependencies on the edge worker. + +Here are a few imperative requirements for your workers: + +- ``airflow`` needs to be installed, and the Airflow CLI needs to be in the path. This includes + the Task SDK as well as the edge3 provider package. +- Airflow configuration settings should be homogeneous across the cluster and on the edge site +- Operators that are executed on the Edge Worker need to have their dependencies + met in that context. Please take a look to the respective provider package + documentations +- The worker needs to have access to the ``DAGS_FOLDER``, and you need to + synchronize the filesystems by your own means. A common setup would be to + store your ``DAGS_FOLDER`` in a Git repository and sync it across machines using + Chef, Puppet, Ansible, or whatever you use to configure machines in your + environment. If all your boxes have a common mount point, having your + pipelines files shared there should work as well + + +Minimum Airflow configuration settings for the Edge Worker to make it running is: + +- Section ``[core]`` + + - ``executor``: Executor must be set or added to be ``airflow.providers.edge3.executors.EdgeExecutor`` + - ``internal_api_secret_key``: An encryption key must be set on api-server and Edge Worker component as +shared secret to authenticate traffic. It should be a random string like the fernet key +(but preferably not the same). + +- Section ``[edge]`` + + - ``api_enabled``: Must be set to true. It is disabled intentionally to not expose +API endpoint by default. This is the endpoint the worker connects to. +In a future release a dedicated API server can be started. + - ``api_url``: Must be set to the URL which exposes the api endpoint as it is reachable from the +worker. Typically this looks like ``https://your-hostname-and-port/edge_worker/v1/rpcapi``. + +To kick off a worker, you need to setup Airflow and kick off the worker +subcommand + +.. code-block:: bash + +airflow edge worker + +Your worker should start picking up tasks as soon as they get fired in +its direction. To stop a worker running on a machine you can use: + +.. code-block:: bash + +airflow edge stop + +It will try to stop the worker gracefully by sending ``SIGINT`` signal to main +process as and wait until all running tasks are completed. Also in a console you can use +``Ctrl-C`` to stop the worker. + +If you want to monitor the remote activity and worker, use the UI plugin which +is included in the provider package and install it on the webserver and use the +"Admin" - "Edge Worker Hosts" and "Edge Worker Jobs" pages. +(Note: The plugin is not ported to Airflow 3.0 web UI at time of writing) + +If you want to check status of the worker via CLI you can use the command + +.. code-block:: bash + +airflow edge status + +Some caveats: + +- Tasks can consume resources. Make sure your worker has enough resources to run ``worker_concurrency`` tasks +- Make sure that the ``pool_slots`` of a Tasks matches with the ``worker_concurrency`` of the worker. + See also :ref:`edge_executor:concurrency_slots`. +- Queue names are limited to 256 characters + +See :doc:`apache-airflow:administration-and-deployment/modules_management` for details on how Python and Airflow manage modules. + +.. _deployment:maintenance: + +Worker Maintenance Mode +--- + +Sometimes infrastructure needs to be ma
Re: [PR] Enhance Edge3 Provider docs [airflow]
jscheffl commented on PR #49859: URL: https://github.com/apache/airflow/pull/49859#issuecomment-2845667209 @dheerajturaga FYI as you have your hands on it if the docs make the setup more clear. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
