+1 (binding) - tested on Py 3.7.3 with example DAGs On Sat, Oct 19, 2019 at 11:30 AM Ash Berlin-Taylor <a...@apache.org> wrote:
> I did all my testing of the RC on 3.7.3 so the bits I testee (core mostly, > didn't test many operators) work > > On 19 October 2019 07:56:40 BST, Jarek Potiuk <jarek.pot...@polidea.com> > wrote: > >It is stated in CONTRIBUTING.md (for v1-10) and CONTRIBUTING.rst (in > >master) - we are working on improving those docs. But indeed it should > >be > >stated in the user documentation :). I will make sure to include that > >in > >our Google Season of Docs initiative. > > > >Yes. It is quite possible to add 3.7 support. We are in the process of > >migrating to another CI system (GitLab CI likely) and there we should > >be > >able to run tests on more python versions. It's not a big effort at > >all. > > > >J. > > > > > >On Sat, Oct 19, 2019 at 5:16 AM Kevin Yang <yrql...@gmail.com> wrote: > > > >> Thank you Jarek for the clarification, that make sense! I might be > >ignorant > >> but do we have an official place stating the python version we > >support? > >> People might not be keep tracking of the Python version in CI and use > >> breeze doc as a reference. Do you think it make sense to update > >breeze doc > >> to reflect what we intend to support to just avoid a bit confusion? > >> > >> As as a side topic, since we've been talking about there's a not > >short > >> timeline for 2.0 release in other threads and the current code base > >may > >> just support 3.7, do you think it make sense to try add the support > >in > >> 1.10.*( optimally we just update a few docs and build a CI test for > >it, and > >> if it cannot just work outta box or requires a lot work we can give > >up > >> eariler :D). > >> > >> Cheers, > >> Kevin Y > >> > >> On Fri, Oct 18, 2019 at 7:26 PM Jarek Potiuk > ><jarek.pot...@polidea.com> > >> wrote: > >> > >> > This is really what's intended. 1.10 supports officialy > >2.7/3.5/3.6. See > >> > for example here where we do not test against 3.7 > >> > https://travis-ci.org/apache/airflow/builds/599638353 > >> > > >> > On the other hand 2.0.*/master should support 3.5, 3.6, 3.7 (and we > >test > >> > against all those versions): for example here - > >> > https://travis-ci.org/apache/airflow/builds/599628309 (never mind > >the > >> > failing kubernetes builds). This should also be quite deliberate > >that if > >> > you want to go > >> > > >> > Then 1.10.* might simply just work in 3.7 and if you need it, it > >should > >> be > >> > as easy as changing ./breeze-complete to add it). But we do not > >have > >> > pre-built images for 3.7 so it will take a long time to build it > >for the > >> > first time from the scratch. > >> > > >> > J. > >> > > >> > > >> > > >> > On Sat, Oct 19, 2019 at 12:43 AM Kevin Yang <yrql...@gmail.com> > >wrote: > >> > > >> > > Just started to play with it for a bit and seems like we got a > >small > >> bit > >> > > inconsistency between what version of python breeze supports. The > >code > >> > > would allow only `2.7, 3.5, 3.6` > >> > > > ><https://github.com/apache/airflow/blob/v1-10-test/breeze-complete#L3> > >> > but > >> > > we claim we support `3.5, 3.6, 3.7` > >> > > > ><https://github.com/apache/airflow/blame/v1-10-test/BREEZE.rst#L532>. > >> > And > >> > > thus `./breeze` will complain about me running in a python3.7 > >env. My > >> > > understanding is that we want to support 2.7, 3.5, 3.6 and 3.7 in > >> 1.10.6 > >> > > until we drop 2.7 support in 2.0, am I right? > >> > > > >> > > On Fri, Oct 18, 2019 at 7:12 AM Ash Berlin-Taylor > ><a...@apache.org> > >> > wrote: > >> > > > >> > > > Hey all, > >> > > > > >> > > > I have cut Airflow 1.10.6 RC1. This email is calling a vote on > >the > >> > > > release, which will last for 96 hours, until Tuesday, October > >22nd at > >> > > 14:30 > >> > > > UTC. (Sorry this is mostly over the weekend again, I've > >extended the > >> > vote > >> > > > by one day to give two working days to test.) > >> > > > > >> > > > Consider this my (binding) +1. > >> > > > > >> > > > Airflow 1.10.6 RC1 is available at: < > >> > > > https://dist.apache.org/repos/dist/dev/airflow/1.10.6rc1/> > >> > > > > >> > > > *apache-airflow-1.10.6rc1-source.tar.gz* is a source release > >that > >> comes > >> > > > with INSTALL instructions. > >> > > > *apache-airflow-1.10.6rc1-bin.tar.gz* is the binary Python > >"sdist" > >> > > release. > >> > > > *apache_airflow-1.10.6rc1-py2.py3-none-any.whl* is the binary > >Python > >> > > > "wheel" release. > >> > > > > >> > > > Public keys are available at: < > >> > > > https://dist.apache.org/repos/dist/release/airflow/KEYS> > >> > > > > >> > > > As per normal the rc1 is available for testing from PyPi. > >> > > > > >> > > > Only votes from PMC members are binding, but members of the > >community > >> > are > >> > > > encouraged to test the release and vote with "(non-binding)". > >> > > > > >> > > > Please note that the version number excludes the `rcX` string, > >so > >> it's > >> > > now > >> > > > simply 1.10.6. This will allow us to rename the artifact > >without > >> > > modifying > >> > > > the artifact checksums when we actually release. > >> > > > > >> > > > Changelog since 1.10.5: > >> > > > > >> > > > Airflow 1.10.6, 2019-10-22 > >> > > > -------------------------- > >> > > > > >> > > > New Features > >> > > > """""""""""" > >> > > > - [AIRFLOW-4908] Implement BigQuery Hooks/Operators for > >> update_dataset, > >> > > > patch_dataset and get_dataset (#5546) > >> > > > - [AIRFLOW-4741] Optionally report task errors to Sentry > >(#5407) > >> > > > - [AIRFLOW-4939] Add default_task_retries config (#5570) > >> > > > - [AIRFLOW-5508] Add config setting to limit which StatsD > >metrics are > >> > > > emitted (#6130) > >> > > > - [AIRFLOW-4222] Add cli autocomplete for bash & zsh (#5789) > >> > > > - [AIRFLOW-3871] Operators template fields can now render > >fields > >> inside > >> > > > objects (#4743) > >> > > > > >> > > > Improvements > >> > > > """""""""""" > >> > > > - [AIRFLOW-5127] Gzip support for > >> CassandraToGoogleCloudStorageOperator > >> > > > (#5738) > >> > > > - [AIRFLOW-5125] Add gzip support for > >> AdlsToGoogleCloudStorageOperator > >> > > > (#5737) > >> > > > - [AIRFLOW-5124] Add gzip support for > >S3ToGoogleCloudStorageOperator > >> > > > (#5736) > >> > > > - [AIRFLOW-5653] Log AirflowSkipException in task instance log > >to > >> make > >> > it > >> > > > clearer why tasks might be skipped (#6330) > >> > > > - [AIRFLOW-5343] Remove legacy SQLAlchmey pessimistic pool > >disconnect > >> > > > handling (#6034) > >> > > > - [AIRFLOW-5561] Relax httplib2 version required for gcp extra > >> (#6194) > >> > > > - [AIRFLOW-5657] Update the upper bound for dill dependency > >(#6334) > >> > > > - [AIRFLOW-5292] Allow ECSOperator to tag tasks (#5891) > >> > > > - [AIRFLOW-4939] Simplify Code for Default Task Retries (#6233) > >> > > > - [AIRFLOW-5126] Read ``aws_session_token`` in extra_config of > >the > >> aws > >> > > > hook (#6303) > >> > > > - [AIRFLOW-5636] Allow adding or overriding existing Operator > >Links > >> > > (#6302) > >> > > > - [AIRFLOW-4965] Handle quote exceptions in GCP AI operators > >(v1.10) > >> > > > (#6304) > >> > > > - [AIRFLOW-3783] Speed up Redshift to S3 UNload with HEADERs > >(#6309) > >> > > > - [AIRFLOW-3388] Add support to Array Jobs for AWS Batch > >Operator > >> > (#6153) > >> > > > - [AIRFLOW-4574] add option to provide private_key in SSHHook > >(#6104) > >> > > > (#6163) > >> > > > - [AIRFLOW-5530] Fix typo in AWS SQS sensors (#6012) > >> > > > - [AIRFLOW-5445] Reduce the required resources for the > >Kubernetes's > >> > > > sidecar (#6062) > >> > > > - [AIRFLOW-5443] Use alpine image in Kubernetes's sidecar > >(#6059) > >> > > > - [AIRFLOW-5344] Add --proxy-user parameter to > >SparkSubmitOperator > >> > > (#5948) > >> > > > - [AIRFLOW-3888] HA for Hive metastore connection (#4708) > >> > > > - [AIRFLOW-5269] Reuse session in Scheduler Job from health > >endpoint > >> > > > (#5873) > >> > > > - [AIRFLOW-5153] Option to force delete non-empty BQ datasets > >(#5768) > >> > > > - [AIRFLOW-4443] Document LatestOnly behavior for external > >trigger > >> > > (#5214) > >> > > > - [AIRFLOW-2891] Make DockerOperator container_name be > >templateable > >> > > (#5696) > >> > > > - [AIRFLOW-2891] allow configurable docker_operator container > >name > >> > > (#5689) > >> > > > - [AIRFLOW-4285] Update task dependency context definition and > >usage > >> > > > (#5079) > >> > > > - [AIRFLOW-5142] Fixed flaky Cassandra test (#5758) > >> > > > - [AIRFLOW-5218] Less polling of AWS Batch job status (#5825) > >> > > > - [AIRFLOW-4956] Fix LocalTaskJob heartbeat log spamming > >(#5589) > >> > > > - [AIRFLOW-3160] Load latest_dagruns asynchronously on home > >page > >> > (#5339) > >> > > > - [AIRFLOW-5560] Allow no confirmation on reset dags in > >`airflow > >> > > backfill` > >> > > > command (#6195) > >> > > > - [AIRFLOW-5280] conn: Remove aws_default's default region name > >> (#5879) > >> > > > - [AIRFLOW-5528] end_of_log_mark should not be a log record > >(#6159) > >> > > > - [AIRFLOW-5526] Update docs configuration due to migration of > >GCP > >> docs > >> > > > (#6154) > >> > > > - [AIRFLOW-4835] Refactor operator render_template (#5461) > >> > > > > >> > > > Bug Fixes > >> > > > """"""""" > >> > > > - [AIRFLOW-5459] Use a dynamic tmp location in Dataflow > >operator > >> > (#6078) > >> > > > - [Airflow 4923] Fix Databricks hook leaks API secret in logs > >(#5635) > >> > > > - [AIRFLOW-5133] Keep original env state in > >> provide_gcp_credential_file > >> > > > (#5747) > >> > > > - [AIRFLOW-5497] Update docstring in > >> > ``airflow/utils/dag_processing.py`` > >> > > > (#6314) > >> > > > - Revert/and then rework "[AIRFLOW-4797] Improve performance > >and > >> > > behaviour > >> > > > of zombie detection (#5511)" to improve performance (#5908) > >> > > > - [AIRFLOW-5634] Don't allow editing of DagModelView (#6308) > >> > > > - [AIRFLOW-4309] Remove Broken Dag error after Dag is deleted > >(#6102) > >> > > > - [AIRFLOW-5387] Fix "show paused" pagination bug (#6100) > >> > > > - [AIRFLOW-5489] Remove unneeded assignment of variable (#6106) > >> > > > - [AIRFLOW-5491] mark_tasks pydoc is incorrect (#6108) > >> > > > - [AIRFLOW-5492] added missing docstrings (#6107) > >> > > > - [AIRFLOW-5503] Fix tree view layout on HDPI screen (#6125) > >> > > > - [AIRFLOW-5481] Allow Deleting Renamed DAGs (#6101) > >> > > > - [AIRFLOW-3857] spark_submit_hook cannot kill driver pod in > >> Kubernetes > >> > > > (#4678) > >> > > > - [AIRFLOW-4391] Fix tooltip for None-State Tasks in 'Recent > >Tasks' > >> > > (#5909) > >> > > > - [AIRFLOW-5554] Require statsd 3.3.0 minimum (#6185) > >> > > > - [AIRFLOW-5306] Fix the display of links when they contain > >special > >> > > > characters (#5904) > >> > > > - [AIRFLOW-3705] Fix PostgresHook get_conn to use > >conn_name_attr > >> > (#5841) > >> > > > - [AIRFLOW-5581] Cleanly shutdown KubernetesJobWatcher for safe > >> > Scheduler > >> > > > shutdown on SIGTERM (#6237) > >> > > > - [AIRFLOW-5634] Don't allow disabled fields to be edited in > >> > DagModelView > >> > > > (#6307) > >> > > > - [AIRFLOW-4833] Allow to set Jinja env options in DAG > >declaration > >> > > (#5943) > >> > > > - [AIRFLOW-5408] Fix env variable name in Kubernetes template > >(#6016) > >> > > > - [AIRFLOW-5102] Worker jobs should terminate themselves if > >they > >> can't > >> > > > heartbeat (#6284) > >> > > > - [AIRFLOW-5572] Clear task reschedules when clearing task > >instances > >> > > > (#6217) > >> > > > - [AIRFLOW-5543] Fix tooltip disappears in tree and graph view > >(RBAC > >> > UI) > >> > > > (#6174) > >> > > > - [AIRFLOW-5444] Fix action_logging so that request.form for > >POST is > >> > > > logged (#6064) > >> > > > - [AIRFLOW-5484] fix PigCliHook has incorrect named parameter > >(#6112) > >> > > > - [AIRFLOW-5342] Fix MSSQL breaking task_instance db migration > >> (#6014) > >> > > > - [AIRFLOW-5556] Add separate config for timeout from scheduler > >dag > >> > > > processing (#6186) > >> > > > - [AIRFLOW-4858] Deprecate "Historical convenience functions" > >in > >> > > > airflow.configuration (#5495) (#6144) > >> > > > - [AIRFLOW-774] Fix long-broken DAG parsing Statsd metrics > >(#6157) > >> > > > - [AIRFLOW-5419] Use ``sudo`` to kill cleared tasks when > >running with > >> > > > impersonation (#6026) (#6176) > >> > > > - [AIRFLOW-5537] Yamllint is not needed as dependency on host > >> > > > - [AIRFLOW-5536] Better handling of temporary output files > >> > > > - [AIRFLOW-5535] Fix name of VERBOSE parameter > >> > > > - [AIRFLOW-5519] Fix sql_to_gcs operator missing multi-level > >default > >> > args > >> > > > by adding apply_defaults decorator (#6146) > >> > > > - [AIRFLOW-5210] Make finding template files more efficient > >(#5815) > >> > > > - [AIRFLOW-5447] Scheduler stalls because second watcher thread > >in > >> > > default > >> > > > args (#6129) > >> > > > > >> > > > Doc-only changes > >> > > > """""""""""""""" > >> > > > - [AIRFLOW-5574] Fix Google Analytics script loading (#6218) > >> > > > - [AIRFLOW-5588] Add Celery's architecture diagram (#6247) > >> > > > - [AIRFLOW-5521] Fix link to GCP documentation (#6150) > >> > > > - [AIRFLOW-5398] Update contrib example DAGs to context manager > >> (#5998) > >> > > > - [AIRFLOW-5268] Apply same DAG naming conventions as in > >literature > >> > > (#5874) > >> > > > - [AIRFLOW-5101] Fix inconsistent owner value in examples > >(#5712) > >> > > > - [AIRFLOW-XXX] Fix typo - AWS DynamoDB Hook (#6319) > >> > > > - [AIRFLOW-XXX] Fix Documentation for adding extra Operator > >Links > >> > (#6301) > >> > > > - [AIRFLOW-XXX] Add section on task lifecycle & correct casing > >in > >> docs > >> > > > (#4681) > >> > > > - [AIRFLOW-XXX] Make it clear that 1.10.5 wasn't accidentally > >omitted > >> > > from > >> > > > UPDATING.md (#6240) > >> > > > - [AIRFLOW-XXX] Improve format in code-block directives (#6242) > >> > > > - [AIRFLOW-XXX] Format Sendgrid docs (#6245) > >> > > > - [AIRFLOW-XXX] Update to new logo (#6066) > >> > > > - [AIRFLOW-XXX] Typo in FAQ - schedule_interval (#6291) > >> > > > - [AIRFLOW-XXX] Add message about breaking change in > >> > > > DAG#get_task_instances in 1.10.4 (#6226) > >> > > > - [AIRFLOW-XXX] Fix incorrect units in docs for metrics using > >Timers > >> > > > (#6152) > >> > > > - [AIRFLOW-XXX] Fix backtick issues in .rst files & Add > >Precommit > >> hook > >> > > > (#6162) > >> > > > - [AIRFLOW-XXX] Update documentation about variables forcing > >answer > >> > > (#6158) > >> > > > - [AIRFLOW-XXX] Add a third way to configure authorization > >(#6134) > >> > > > - [AIRFLOW-XXX] Add example of running pre-commit hooks on > >single > >> file > >> > > > (#6143) > >> > > > - [AIRFLOW-XXX] Add information about default pool to docs > >(#6019) > >> > > > - [AIRFLOW-XXX] Make Breeze The default integration test > >environment > >> > > > (#6001) > >> > > > > >> > > > Misc/Internal > >> > > > """"""""""""" > >> > > > - [AIRFLOW-5687] Upgrade pip to 19.0.2 in CI build pipeline > >(#6358) > >> > > (#6361) > >> > > > - [AIRFLOW-5533] Fixed failing CRON build (#6167) > >> > > > - [AIRFLOW-5130] Use GOOGLE_APPLICATION_CREDENTIALS constant > >from > >> > library > >> > > > (#5744) > >> > > > - [AIRFLOW-5369] Adds interactivity to pre-commits (#5976) > >> > > > - [AIRFLOW-5531] Replace deprecated log.warn() with > >log.warning() > >> > (#6165) > >> > > > - [AIRFLOW-4686] Make dags Pylint compatible (#5753) > >> > > > - [AIRFLOW-4864] Remove calls to load_test_config (#5502) > >> > > > - [AIRFLOW-XXX] Pin version of mypy so we are stable over time > >> (#6198) > >> > > > - [AIRFLOW-XXX] Add tests that got missed from #5127 > >> > > > - [AIRFLOW-4928] Move config parses to class properties inside > >DagBag > >> > > > (#5557) > >> > > > - [AIRFLOW-5003] Making AWS Hooks pylint compatible (#5627) > >> > > > - [AIRFLOW-5580] Add base class for system test (#6229) > >> > > > > >> > > > > >> > > > >> > > >> > > >> > -- > >> > > >> > Jarek Potiuk > >> > Polidea <https://www.polidea.com/> | Principal Software Engineer > >> > > >> > M: +48 660 796 129 <+48660796129> > >> > [image: Polidea] <https://www.polidea.com/> > >> > > >> > > > > > >-- > > > >Jarek Potiuk > >Polidea <https://www.polidea.com/> | Principal Software Engineer > > > >M: +48 660 796 129 <+48660796129> > >[image: Polidea] <https://www.polidea.com/> >
