Hey Stefan (also anyone else who wants to make last quick check): In the latest iteration I think I solved most Linux problems (and added a few more things) - you can check it yourself: https://github.com/PolideaInternal/airflow/tree/ms-travis-ci-tests
- I got rid of .pyc files. The .pyc files and __pychache_ folders will be deleted on entering the container and no new .pyc files will be created (negligible import time penalty, no impact on run performance) - Ownership of the files is fixed automatically in case we build documentation (scripts/ci/ci_docs.sh) - You can also fix ownership for any files manually by (scripts/ci/local_ci_fix_ownership.sh) - You can run all static checks from within the container not only from the host script - You can run all static checks selectively - for the whole package or single python file. - The last static check - check_licence is also dockerised now - no more java/downloaded rat.jar needed locally to run licence check All docs are cleaned up and updated in https://github.com/PolideaInternal/airflow/blob/ms-travis-ci-tests/CONTRIBUTING.md#integration-test-development-environment J. On Wed, Jul 10, 2019 at 8:50 AM Jarek Potiuk <[email protected]> wrote: > Yeah. It's the optimisation I mentioned that's missing now. > > But I realised that you can (and it's easy) run the ci/mypy/flake once you > enter the environment. I will add a few scripts so that you can just run > scripts (and will update the docs). > > The scripts to run on host are mainly for CI purpose now but in the near > future I will use them to automatically run tests during the pre-commit > hooks (and there I have optimisation that the "rebuild" happen only when > needed. And there I have implemented switching to python3 automatically. I > might actually port most of it even now I think. > > Thanks for the feedbac. Very valuable! > > J. > > On Wed, Jul 10, 2019 at 8:18 AM Stefan Seelmann <[email protected]> > wrote: > >> On 7/9/19 7:36 PM, Jarek Potiuk wrote: >> > - The warning with /root/.docker/config.json is nothing to worry >> about >> > (it tries to access your host configuration for credentials - but we >> do not >> > need any credentials for the environment). >> >> Ok, fine >> >> > - The CLEAN_FILES unbound variable was an interesting one - not sure >> why >> > it was not set at the first run and set it on the other (likely >> because >> > CLEAN_FILES is propagated through docker-compose settings and set to >> empty >> > after initialisation) but I protected against such case now >> >> Works now. >> >> > - Running flake/pylint is not supposed to be run from the container >> but >> > from the host. I looked at the docs and realised it was not at all >> clear >> > :). It was obvious for me but not for someone new :). That's big >> value of >> > such tests. I updated CONTRIBUTING.md with a note and also added >> extra >> > protection. >> >> When I just run `./scripts/ci/ci_flake8.sh` it attempts to build the >> Python 2.7 image `apache/airflow:master-python2.7-ci-slim` (I assume >> because my default Python interpreter is 2.7) which fails. >> >> When I run `PYTHON_VERSION=3.6 ./scripts/ci/ci_flake8.sh` it works. >> However on every invocation it rebuilds the "Airflow CI slim image", it >> uses cached layers, but still takes 23 seconds, it that wanted? >> >> Kind Regards, >> Stefan >> >> >> > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> > > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
