[ https://issues.apache.org/jira/browse/AIRFLOW-2871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16572898#comment-16572898 ]
Kaxil Naik commented on AIRFLOW-2871: ------------------------------------- We have also listed this at our confluence page at https://cwiki.apache.org/confluence/display/AIRFLOW/Building+and+deploying+the+docs >The site for the Airflow documentation that used to be located at >pythonhosted.org is now located at http://airflow.incubator.apache.org/ .This >should point to latest stable, while readthedocs.org keeps track of versioned >documentation. > Harden and improve Read the Docs build environment > -------------------------------------------------- > > Key: AIRFLOW-2871 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2871 > Project: Apache Airflow > Issue Type: Bug > Components: docs, Documentation > Reporter: Taylor Edmiston > Assignee: Taylor Edmiston > Priority: Major > Attachments: screenshot-1.png, screenshot-2.png > > > h2. Context > In the process of resolving AIRFLOW-2857 (via [PR > 3703|https://github.com/apache/incubator-airflow/pull/3703]), I noticed some > oddities in our Read the Docs (RTD) build environment especially around > cached dependencies. This motivates hardening and showing some love to our > RTD setup. > h2. Problem > I dug into the RTD build logs for a moment to find some closure on the mock > dependency discussed in PR #3703 above. I think that our RTD environment > possibly has been working by coincidence off of cached dependencies. > {code:java} > python > /home/docs/checkouts/readthedocs.org/user_builds/airflow/envs/latest/bin/pip > install --ignore-installed --cache-dir > /home/docs/checkouts/readthedocs.org/user_builds/airflow/.cache/pip > .[doc,docker,gcp_api,emr]{code} > The directory referenced by that --cache-dir arg earlier in the log happens > to have mock installed already. > {code:java} > python > /home/docs/checkouts/readthedocs.org/user_builds/airflow/envs/latest/bin/pip > install --upgrade --cache-dir > /home/docs/checkouts/readthedocs.org/user_builds/airflow/.cache/pip > Pygments==2.2.0 setuptools<40 docutils==0.13.1 mock==1.0.1 pillow==2.6.1 > alabaster>=0.7,<0.8,!=0.7.5 commonmark==0.5.4 recommonmark==0.4.0 sphinx<1.8 > sphinx-rtd-theme<0.5 readthedocs-sphinx-ext<0.6{code} > Here are some logs where you can see that (view raw): > # Latest successful (Aug. 7, 2018. 9:21 a.m.) - > [7602630|https://readthedocs.org/projects/airflow/builds/7602630/] > # Last unsuccessful before (1) (Aug. 5, 2018. 1:24 p.m.) - > [7593052|https://readthedocs.org/projects/airflow/builds/7593052/] > # Last successful before (2) (July 18, 2018. 3:23 a.m.) - > [7503718|https://readthedocs.org/projects/airflow/builds/7503718/] > # First build (2016) - > [4150778|https://readthedocs.org/projects/airflow/builds/4150778/] > It appears that mock and others have potentially been cached since the first > RTD build in 2016 (4). > These versions like mock==1.0.1 do not appear to be coming from anywhere in > our current config in incubator-airflow; I believe they are installed as > [core dependencies of RTD > itself|https://github.com/rtfd/readthedocs.org/blob/ca7afe6577672e129ccfe63abe33561dc32a6651/readthedocs/doc_builder/python_environments.py#L220-L235]. > Some but not all of these dependencies get upgraded to newer versions further > down in the build. In the case of mock, we were getting lucky that > mock==1.0.1 was a dependency of RTD and our setup inherited that old version > which allowed the docs build to succeed. (There might be other cases of > dependencies like this too.) > h2. Solution > My proposed enhancements to harden and improve our RTD setup are: > * Hardening > ** Set our RTD build to use a virtual environment if it's not already > ** Set our RTD build to ignore packages outside of its virtualenv like > dependencies of RTD itself > ** Specify any dependencies broken by ^ > ** Test wiping a version in the build environment (not sure if this clears > cache dir) > *** > [https://docs.readthedocs.io/en/latest/guides/wipe-environment.html#wiping-a-build-environment] > *** > [https://docs.readthedocs.io/en/latest/builds.html#deleting-a-stale-or-broken-build-environment] > ** Make build.image, python.version, etc explicit in yaml config > *** [https://docs.readthedocs.io/en/latest/yaml-config.html] > ** Test upgrading our RTD environment from CPython 2.x to using CPython 3.x > * Misc > ** Improve RTD project page to have tags and description > ** Lint YAML file > Note: I don't yet have maintainer access for airflow on RTD which I believe > this would require. I am happy to take this issue if I can get that. I have > experience as an admin of another project on RTD (simple-salesforce). > ---- > /cc Everyone who commented in PR #3703 - [~kaxilnaik], [~ashb], [~TaoFeng] -- This message was sent by Atlassian JIRA (v7.6.3#76005)