amoghrajesh opened a new pull request, #34004:
URL: https://github.com/apache/airflow/pull/34004
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
Introducing short_version flag so that the users who build docs can now
provide a short hand name instead of the whole name while building the docs for
specific packages.
Some examples:
```
(new-env) ➜ airflow git:(shortenPackageFilter) ✗ breeze build-docs --help
Usage: breeze build-docs [OPTIONS]
Build documents.
╭─ Doc flags
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --docs-only -d Only build documentation.
│
│ --spellcheck-only -s Only run spell checking.
│
│ --clean-build Clean inventories of Inter-Sphinx documentation
and generated APIs and sphinx artifacts before the build - useful for a clean
build. │
│ --one-pass-only Builds documentation in one pass only. This is
useful for debugging sphinx errors.
│
│ --package-filter List of packages to consider.
│
│ (apache-airflow | docker-stack | helm-chart |
apache-airflow-providers-airbyte | apache-airflow-providers-alibaba |
apache-airflow-providers-amazon | apache-airflow-providers-apache-beam |
│
│ apache-airflow-providers-apache-cassandra |
apache-airflow-providers-apache-drill | apache-airflow-providers-apache-druid |
apache-airflow-providers-apache-flink | apache-airflow-providers-apache-hdfs |
│
│ apache-airflow-providers-apache-hive |
apache-airflow-providers-apache-impala | apache-airflow-providers-apache-kafka
| apache-airflow-providers-apache-kylin | apache-airflow-providers-apache-livy
| │
│ apache-airflow-providers-apache-pig |
apache-airflow-providers-apache-pinot | apache-airflow-providers-apache-spark |
apache-airflow-providers-apache-sqoop | apache-airflow-providers-apprise |
│
│ apache-airflow-providers-arangodb |
apache-airflow-providers-asana | apache-airflow-providers-atlassian-jira |
apache-airflow-providers-celery | apache-airflow-providers-cloudant |
│
│ apache-airflow-providers-cncf-kubernetes |
apache-airflow-providers-common-sql | apache-airflow-providers-daskexecutor |
apache-airflow-providers-databricks | apache-airflow-providers-datadog |
│
│ apache-airflow-providers-dbt-cloud |
apache-airflow-providers-dingding | apache-airflow-providers-discord |
apache-airflow-providers-docker | apache-airflow-providers-elasticsearch |
│
│ apache-airflow-providers-exasol |
apache-airflow-providers-facebook | apache-airflow-providers-ftp |
apache-airflow-providers-github | apache-airflow-providers-google |
apache-airflow-providers-grpc | │
│ apache-airflow-providers-hashicorp |
apache-airflow-providers-http | apache-airflow-providers-imap |
apache-airflow-providers-influxdb | apache-airflow-providers-jdbc |
apache-airflow-providers-jenkins | │
│ apache-airflow-providers-microsoft-azure |
apache-airflow-providers-microsoft-mssql |
apache-airflow-providers-microsoft-psrp |
apache-airflow-providers-microsoft-winrm | apache-airflow-providers-mongo | │
│ apache-airflow-providers-mysql |
apache-airflow-providers-neo4j | apache-airflow-providers-odbc |
apache-airflow-providers-openfaas | apache-airflow-providers-openlineage |
│
│ apache-airflow-providers-opsgenie |
apache-airflow-providers-oracle | apache-airflow-providers-pagerduty |
apache-airflow-providers-papermill | apache-airflow-providers-plexus |
│
│ apache-airflow-providers-postgres |
apache-airflow-providers-presto | apache-airflow-providers-redis |
apache-airflow-providers-salesforce | apache-airflow-providers-samba |
│
│ apache-airflow-providers-segment |
apache-airflow-providers-sendgrid | apache-airflow-providers-sftp |
apache-airflow-providers-singularity | apache-airflow-providers-slack |
│
│ apache-airflow-providers-smtp |
apache-airflow-providers-snowflake | apache-airflow-providers-sqlite |
apache-airflow-providers-ssh | apache-airflow-providers-tableau |
apache-airflow-providers-tabular | │
│ apache-airflow-providers-telegram |
apache-airflow-providers-trino | apache-airflow-providers-vertica |
apache-airflow-providers-yandex | apache-airflow-providers-zendesk)
│
│ --github-repository -g GitHub repository used to pull, push run images.
(TEXT) [default: apache/airflow]
│
│ --builder Buildx builder used to perform `docker buildx
build` commands. (TEXT) [default: autodetect]
│
│ --short-version Build docs by providing short hand for provider
names. Example: Short hand for apache-airflow-providers-cncf-kubernetes will be
cncf.kubernetes │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Common options
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --verbose -v Print verbose information about performed steps.
│
│ --dry-run -D If dry-run is set, commands are only printed, not executed.
│
│ --help -h Show this message and exit.
│
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Selecting two packages - cncf kubernetes and airbyte using short hand
```
(new-env) ➜ airflow git:(shortenPackageFilter) ✗ breeze build-docs
--short-version --package-filter cncf.kubernetes --package-filter airbyte
Good version of Docker: 24.0.2.
Good version of docker-compose: 2.18.1
The following important files are modified in
/Users/adesai/Documents/OSS/airflow since last time image was built:
* setup.py
* setup.cfg
* Dockerfile.ci
* .dockerignore
* generated/provider_dependencies.json
* scripts/docker/common.sh
* scripts/docker/install_additional_dependencies.sh
* scripts/docker/install_airflow.sh
* scripts/docker/install_airflow_dependencies_from_branch_tip.sh
* scripts/docker/install_from_docker_context_files.sh
* scripts/docker/install_mysql.sh
Likely CI image needs rebuild
Do you want to build the image (this works best when you have good
connection and can take usually from 20 seconds to few minutes depending how
old your image is)?
Press y/N/q. Auto-select n in 10 seconds (add `--answer n` to avoid delay
next time): n
The CI image for Python version 3.8 may be outdated
Please run at the earliest convenience:
breeze ci-image build --python 3.8
Running Initialization. Your basic configuration is:
* Airflow home: /root/airflow
* Airflow sources: /opt/airflow
* Airflow core SQL connection:
Using airflow version from current sources
Checking backend and integrations.
Skip fixing ownership of generated files as Host OS is darwin
#################### Available packages ####################
- apache-airflow
- apache-airflow-providers
- apache-airflow-providers-airbyte
- apache-airflow-providers-alibaba
- apache-airflow-providers-amazon
- apache-airflow-providers-apache-beam
- apache-airflow-providers-apache-cassandra
- apache-airflow-providers-apache-drill
- apache-airflow-providers-apache-druid
- apache-airflow-providers-apache-flink
- apache-airflow-providers-apache-hdfs
- apache-airflow-providers-apache-hive
- apache-airflow-providers-apache-impala
- apache-airflow-providers-apache-kafka
- apache-airflow-providers-apache-kylin
- apache-airflow-providers-apache-livy
- apache-airflow-providers-apache-pig
- apache-airflow-providers-apache-pinot
- apache-airflow-providers-apache-spark
- apache-airflow-providers-apache-sqoop
- apache-airflow-providers-apprise
- apache-airflow-providers-arangodb
- apache-airflow-providers-asana
- apache-airflow-providers-atlassian-jira
- apache-airflow-providers-celery
- apache-airflow-providers-cloudant
- apache-airflow-providers-cncf-kubernetes
- apache-airflow-providers-common-sql
- apache-airflow-providers-daskexecutor
- apache-airflow-providers-databricks
- apache-airflow-providers-datadog
- apache-airflow-providers-dbt-cloud
- apache-airflow-providers-dingding
- apache-airflow-providers-discord
- apache-airflow-providers-docker
- apache-airflow-providers-elasticsearch
- apache-airflow-providers-exasol
- apache-airflow-providers-facebook
- apache-airflow-providers-ftp
- apache-airflow-providers-github
- apache-airflow-providers-google
- apache-airflow-providers-grpc
- apache-airflow-providers-hashicorp
- apache-airflow-providers-http
- apache-airflow-providers-imap
- apache-airflow-providers-influxdb
- apache-airflow-providers-jdbc
- apache-airflow-providers-jenkins
- apache-airflow-providers-microsoft-azure
- apache-airflow-providers-microsoft-mssql
- apache-airflow-providers-microsoft-psrp
- apache-airflow-providers-microsoft-winrm
- apache-airflow-providers-mongo
- apache-airflow-providers-mysql
- apache-airflow-providers-neo4j
- apache-airflow-providers-odbc
- apache-airflow-providers-openfaas
- apache-airflow-providers-openlineage
- apache-airflow-providers-opsgenie
- apache-airflow-providers-oracle
- apache-airflow-providers-pagerduty
- apache-airflow-providers-papermill
- apache-airflow-providers-plexus
- apache-airflow-providers-postgres
- apache-airflow-providers-presto
- apache-airflow-providers-redis
- apache-airflow-providers-salesforce
- apache-airflow-providers-samba
- apache-airflow-providers-segment
- apache-airflow-providers-sendgrid
- apache-airflow-providers-sftp
- apache-airflow-providers-singularity
- apache-airflow-providers-slack
- apache-airflow-providers-smtp
- apache-airflow-providers-snowflake
- apache-airflow-providers-sqlite
- apache-airflow-providers-ssh
- apache-airflow-providers-tableau
- apache-airflow-providers-tabular
- apache-airflow-providers-telegram
- apache-airflow-providers-trino
- apache-airflow-providers-vertica
- apache-airflow-providers-yandex
- apache-airflow-providers-zendesk
- docker-stack
- helm-chart
Current package filters:
['apache-airflow-providers-cncf-kubernetes',
'apache-airflow-providers-airbyte']
#################### Fetching inventories ####################
Nothing to do
#################### Documentation will be built for 2 package(s) with 5
parallel jobs ####################
1. apache-airflow-providers-airbyte
2. apache-airflow-providers-cncf-kubernetes
#################### Cleaning documentation files ####################
apache-airflow-providers-airbyte : Cleaning files
apache-airflow-providers-cncf-kubernetes : Cleaning files
#################### Scheduling documentation to build ####################
apache-airflow-providers-airbyte : Scheduling
documentation to build
apache-airflow-providers-cncf-kubernetes : Scheduling
documentation to build
#################### Running docs building ####################
apache-airflow-providers-airbyte : Building
documentation
apache-airflow-providers-cncf-kubernetes : Building
documentation
apache-airflow-providers-airbyte : Running
sphinx. The output is hidden until an error occurs.
apache-airflow-providers-cncf-kubernetes : Running
sphinx. The output is hidden until an error occurs.
^CProcess ForkPoolWorker-4:
Process ForkPoolWorker-3:
Process ForkPoolWorker-5:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/local/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/opt/airflow/docs/build_docs.py", line 572, in <module>
main()
File "/opt/airflow/docs/build_docs.py", line 478, in main
package_build_errors, package_spelling_errors = build_docs_for_packages(
File "/opt/airflow/docs/build_docs.py", line 234, in
build_docs_for_packages
run_in_parallel(
File "/opt/airflow/docs/build_docs.py", line 300, in run_in_parallel
run_docs_build_in_parallel(
File "/opt/airflow/docs/build_docs.py", line 345, in
run_docs_build_in_parallel
result_list = pool.map(perform_docs_build_for_single_package,
doc_build_specifications)
File "/usr/local/lib/python3.8/multiprocessing/pool.py", line 364, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/usr/local/lib/python3.8/multiprocessing/pool.py", line 765, in get
self.wait(timeout)
File "/usr/local/lib/python3.8/multiprocessing/pool.py", line 762, in wait
self._event.wait(timeout)
File "/usr/local/lib/python3.8/threading.py", line 558, in wait
signaled = self._cond.wait(timeout)
File "/usr/local/lib/python3.8/threading.py", line 302, in wait
waiter.acquire()
KeyboardInterrupt
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/opt/airflow/airflow/settings.py", line 362, in dispose_orm
engine.dispose()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py",
line 3076, in dispose
self.pool = self.pool.recreate()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/impl.py",
line 262, in recreate
self._creator,
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py",
line 219, in _creator
return self.__dict__["_creator"]
KeyboardInterrupt
Aborted.
```
closes: #33876
<!-- Please keep an empty line above the dashes. -->
---
**^ Add meaningful description above**
Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)**
for more information.
In case of fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party
License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a
newsfragment file, named `{pr_number}.significant.rst` or
`{issue_number}.significant.rst`, in
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]