I did all my testing of the RC on 3.7.3 so the bits I testee (core mostly, didn't test many operators) work
On 19 October 2019 07:56:40 BST, Jarek Potiuk <[email protected]> wrote: >It is stated in CONTRIBUTING.md (for v1-10) and CONTRIBUTING.rst (in >master) - we are working on improving those docs. But indeed it should >be >stated in the user documentation :). I will make sure to include that >in >our Google Season of Docs initiative. > >Yes. It is quite possible to add 3.7 support. We are in the process of >migrating to another CI system (GitLab CI likely) and there we should >be >able to run tests on more python versions. It's not a big effort at >all. > >J. > > >On Sat, Oct 19, 2019 at 5:16 AM Kevin Yang <[email protected]> wrote: > >> Thank you Jarek for the clarification, that make sense! I might be >ignorant >> but do we have an official place stating the python version we >support? >> People might not be keep tracking of the Python version in CI and use >> breeze doc as a reference. Do you think it make sense to update >breeze doc >> to reflect what we intend to support to just avoid a bit confusion? >> >> As as a side topic, since we've been talking about there's a not >short >> timeline for 2.0 release in other threads and the current code base >may >> just support 3.7, do you think it make sense to try add the support >in >> 1.10.*( optimally we just update a few docs and build a CI test for >it, and >> if it cannot just work outta box or requires a lot work we can give >up >> eariler :D). >> >> Cheers, >> Kevin Y >> >> On Fri, Oct 18, 2019 at 7:26 PM Jarek Potiuk ><[email protected]> >> wrote: >> >> > This is really what's intended. 1.10 supports officialy >2.7/3.5/3.6. See >> > for example here where we do not test against 3.7 >> > https://travis-ci.org/apache/airflow/builds/599638353 >> > >> > On the other hand 2.0.*/master should support 3.5, 3.6, 3.7 (and we >test >> > against all those versions): for example here - >> > https://travis-ci.org/apache/airflow/builds/599628309 (never mind >the >> > failing kubernetes builds). This should also be quite deliberate >that if >> > you want to go >> > >> > Then 1.10.* might simply just work in 3.7 and if you need it, it >should >> be >> > as easy as changing ./breeze-complete to add it). But we do not >have >> > pre-built images for 3.7 so it will take a long time to build it >for the >> > first time from the scratch. >> > >> > J. >> > >> > >> > >> > On Sat, Oct 19, 2019 at 12:43 AM Kevin Yang <[email protected]> >wrote: >> > >> > > Just started to play with it for a bit and seems like we got a >small >> bit >> > > inconsistency between what version of python breeze supports. The >code >> > > would allow only `2.7, 3.5, 3.6` >> > > ><https://github.com/apache/airflow/blob/v1-10-test/breeze-complete#L3> >> > but >> > > we claim we support `3.5, 3.6, 3.7` >> > > ><https://github.com/apache/airflow/blame/v1-10-test/BREEZE.rst#L532>. >> > And >> > > thus `./breeze` will complain about me running in a python3.7 >env. My >> > > understanding is that we want to support 2.7, 3.5, 3.6 and 3.7 in >> 1.10.6 >> > > until we drop 2.7 support in 2.0, am I right? >> > > >> > > On Fri, Oct 18, 2019 at 7:12 AM Ash Berlin-Taylor ><[email protected]> >> > wrote: >> > > >> > > > Hey all, >> > > > >> > > > I have cut Airflow 1.10.6 RC1. This email is calling a vote on >the >> > > > release, which will last for 96 hours, until Tuesday, October >22nd at >> > > 14:30 >> > > > UTC. (Sorry this is mostly over the weekend again, I've >extended the >> > vote >> > > > by one day to give two working days to test.) >> > > > >> > > > Consider this my (binding) +1. >> > > > >> > > > Airflow 1.10.6 RC1 is available at: < >> > > > https://dist.apache.org/repos/dist/dev/airflow/1.10.6rc1/> >> > > > >> > > > *apache-airflow-1.10.6rc1-source.tar.gz* is a source release >that >> comes >> > > > with INSTALL instructions. >> > > > *apache-airflow-1.10.6rc1-bin.tar.gz* is the binary Python >"sdist" >> > > release. >> > > > *apache_airflow-1.10.6rc1-py2.py3-none-any.whl* is the binary >Python >> > > > "wheel" release. >> > > > >> > > > Public keys are available at: < >> > > > https://dist.apache.org/repos/dist/release/airflow/KEYS> >> > > > >> > > > As per normal the rc1 is available for testing from PyPi. >> > > > >> > > > Only votes from PMC members are binding, but members of the >community >> > are >> > > > encouraged to test the release and vote with "(non-binding)". >> > > > >> > > > Please note that the version number excludes the `rcX` string, >so >> it's >> > > now >> > > > simply 1.10.6. This will allow us to rename the artifact >without >> > > modifying >> > > > the artifact checksums when we actually release. >> > > > >> > > > Changelog since 1.10.5: >> > > > >> > > > Airflow 1.10.6, 2019-10-22 >> > > > -------------------------- >> > > > >> > > > New Features >> > > > """""""""""" >> > > > - [AIRFLOW-4908] Implement BigQuery Hooks/Operators for >> update_dataset, >> > > > patch_dataset and get_dataset (#5546) >> > > > - [AIRFLOW-4741] Optionally report task errors to Sentry >(#5407) >> > > > - [AIRFLOW-4939] Add default_task_retries config (#5570) >> > > > - [AIRFLOW-5508] Add config setting to limit which StatsD >metrics are >> > > > emitted (#6130) >> > > > - [AIRFLOW-4222] Add cli autocomplete for bash & zsh (#5789) >> > > > - [AIRFLOW-3871] Operators template fields can now render >fields >> inside >> > > > objects (#4743) >> > > > >> > > > Improvements >> > > > """""""""""" >> > > > - [AIRFLOW-5127] Gzip support for >> CassandraToGoogleCloudStorageOperator >> > > > (#5738) >> > > > - [AIRFLOW-5125] Add gzip support for >> AdlsToGoogleCloudStorageOperator >> > > > (#5737) >> > > > - [AIRFLOW-5124] Add gzip support for >S3ToGoogleCloudStorageOperator >> > > > (#5736) >> > > > - [AIRFLOW-5653] Log AirflowSkipException in task instance log >to >> make >> > it >> > > > clearer why tasks might be skipped (#6330) >> > > > - [AIRFLOW-5343] Remove legacy SQLAlchmey pessimistic pool >disconnect >> > > > handling (#6034) >> > > > - [AIRFLOW-5561] Relax httplib2 version required for gcp extra >> (#6194) >> > > > - [AIRFLOW-5657] Update the upper bound for dill dependency >(#6334) >> > > > - [AIRFLOW-5292] Allow ECSOperator to tag tasks (#5891) >> > > > - [AIRFLOW-4939] Simplify Code for Default Task Retries (#6233) >> > > > - [AIRFLOW-5126] Read ``aws_session_token`` in extra_config of >the >> aws >> > > > hook (#6303) >> > > > - [AIRFLOW-5636] Allow adding or overriding existing Operator >Links >> > > (#6302) >> > > > - [AIRFLOW-4965] Handle quote exceptions in GCP AI operators >(v1.10) >> > > > (#6304) >> > > > - [AIRFLOW-3783] Speed up Redshift to S3 UNload with HEADERs >(#6309) >> > > > - [AIRFLOW-3388] Add support to Array Jobs for AWS Batch >Operator >> > (#6153) >> > > > - [AIRFLOW-4574] add option to provide private_key in SSHHook >(#6104) >> > > > (#6163) >> > > > - [AIRFLOW-5530] Fix typo in AWS SQS sensors (#6012) >> > > > - [AIRFLOW-5445] Reduce the required resources for the >Kubernetes's >> > > > sidecar (#6062) >> > > > - [AIRFLOW-5443] Use alpine image in Kubernetes's sidecar >(#6059) >> > > > - [AIRFLOW-5344] Add --proxy-user parameter to >SparkSubmitOperator >> > > (#5948) >> > > > - [AIRFLOW-3888] HA for Hive metastore connection (#4708) >> > > > - [AIRFLOW-5269] Reuse session in Scheduler Job from health >endpoint >> > > > (#5873) >> > > > - [AIRFLOW-5153] Option to force delete non-empty BQ datasets >(#5768) >> > > > - [AIRFLOW-4443] Document LatestOnly behavior for external >trigger >> > > (#5214) >> > > > - [AIRFLOW-2891] Make DockerOperator container_name be >templateable >> > > (#5696) >> > > > - [AIRFLOW-2891] allow configurable docker_operator container >name >> > > (#5689) >> > > > - [AIRFLOW-4285] Update task dependency context definition and >usage >> > > > (#5079) >> > > > - [AIRFLOW-5142] Fixed flaky Cassandra test (#5758) >> > > > - [AIRFLOW-5218] Less polling of AWS Batch job status (#5825) >> > > > - [AIRFLOW-4956] Fix LocalTaskJob heartbeat log spamming >(#5589) >> > > > - [AIRFLOW-3160] Load latest_dagruns asynchronously on home >page >> > (#5339) >> > > > - [AIRFLOW-5560] Allow no confirmation on reset dags in >`airflow >> > > backfill` >> > > > command (#6195) >> > > > - [AIRFLOW-5280] conn: Remove aws_default's default region name >> (#5879) >> > > > - [AIRFLOW-5528] end_of_log_mark should not be a log record >(#6159) >> > > > - [AIRFLOW-5526] Update docs configuration due to migration of >GCP >> docs >> > > > (#6154) >> > > > - [AIRFLOW-4835] Refactor operator render_template (#5461) >> > > > >> > > > Bug Fixes >> > > > """"""""" >> > > > - [AIRFLOW-5459] Use a dynamic tmp location in Dataflow >operator >> > (#6078) >> > > > - [Airflow 4923] Fix Databricks hook leaks API secret in logs >(#5635) >> > > > - [AIRFLOW-5133] Keep original env state in >> provide_gcp_credential_file >> > > > (#5747) >> > > > - [AIRFLOW-5497] Update docstring in >> > ``airflow/utils/dag_processing.py`` >> > > > (#6314) >> > > > - Revert/and then rework "[AIRFLOW-4797] Improve performance >and >> > > behaviour >> > > > of zombie detection (#5511)" to improve performance (#5908) >> > > > - [AIRFLOW-5634] Don't allow editing of DagModelView (#6308) >> > > > - [AIRFLOW-4309] Remove Broken Dag error after Dag is deleted >(#6102) >> > > > - [AIRFLOW-5387] Fix "show paused" pagination bug (#6100) >> > > > - [AIRFLOW-5489] Remove unneeded assignment of variable (#6106) >> > > > - [AIRFLOW-5491] mark_tasks pydoc is incorrect (#6108) >> > > > - [AIRFLOW-5492] added missing docstrings (#6107) >> > > > - [AIRFLOW-5503] Fix tree view layout on HDPI screen (#6125) >> > > > - [AIRFLOW-5481] Allow Deleting Renamed DAGs (#6101) >> > > > - [AIRFLOW-3857] spark_submit_hook cannot kill driver pod in >> Kubernetes >> > > > (#4678) >> > > > - [AIRFLOW-4391] Fix tooltip for None-State Tasks in 'Recent >Tasks' >> > > (#5909) >> > > > - [AIRFLOW-5554] Require statsd 3.3.0 minimum (#6185) >> > > > - [AIRFLOW-5306] Fix the display of links when they contain >special >> > > > characters (#5904) >> > > > - [AIRFLOW-3705] Fix PostgresHook get_conn to use >conn_name_attr >> > (#5841) >> > > > - [AIRFLOW-5581] Cleanly shutdown KubernetesJobWatcher for safe >> > Scheduler >> > > > shutdown on SIGTERM (#6237) >> > > > - [AIRFLOW-5634] Don't allow disabled fields to be edited in >> > DagModelView >> > > > (#6307) >> > > > - [AIRFLOW-4833] Allow to set Jinja env options in DAG >declaration >> > > (#5943) >> > > > - [AIRFLOW-5408] Fix env variable name in Kubernetes template >(#6016) >> > > > - [AIRFLOW-5102] Worker jobs should terminate themselves if >they >> can't >> > > > heartbeat (#6284) >> > > > - [AIRFLOW-5572] Clear task reschedules when clearing task >instances >> > > > (#6217) >> > > > - [AIRFLOW-5543] Fix tooltip disappears in tree and graph view >(RBAC >> > UI) >> > > > (#6174) >> > > > - [AIRFLOW-5444] Fix action_logging so that request.form for >POST is >> > > > logged (#6064) >> > > > - [AIRFLOW-5484] fix PigCliHook has incorrect named parameter >(#6112) >> > > > - [AIRFLOW-5342] Fix MSSQL breaking task_instance db migration >> (#6014) >> > > > - [AIRFLOW-5556] Add separate config for timeout from scheduler >dag >> > > > processing (#6186) >> > > > - [AIRFLOW-4858] Deprecate "Historical convenience functions" >in >> > > > airflow.configuration (#5495) (#6144) >> > > > - [AIRFLOW-774] Fix long-broken DAG parsing Statsd metrics >(#6157) >> > > > - [AIRFLOW-5419] Use ``sudo`` to kill cleared tasks when >running with >> > > > impersonation (#6026) (#6176) >> > > > - [AIRFLOW-5537] Yamllint is not needed as dependency on host >> > > > - [AIRFLOW-5536] Better handling of temporary output files >> > > > - [AIRFLOW-5535] Fix name of VERBOSE parameter >> > > > - [AIRFLOW-5519] Fix sql_to_gcs operator missing multi-level >default >> > args >> > > > by adding apply_defaults decorator (#6146) >> > > > - [AIRFLOW-5210] Make finding template files more efficient >(#5815) >> > > > - [AIRFLOW-5447] Scheduler stalls because second watcher thread >in >> > > default >> > > > args (#6129) >> > > > >> > > > Doc-only changes >> > > > """""""""""""""" >> > > > - [AIRFLOW-5574] Fix Google Analytics script loading (#6218) >> > > > - [AIRFLOW-5588] Add Celery's architecture diagram (#6247) >> > > > - [AIRFLOW-5521] Fix link to GCP documentation (#6150) >> > > > - [AIRFLOW-5398] Update contrib example DAGs to context manager >> (#5998) >> > > > - [AIRFLOW-5268] Apply same DAG naming conventions as in >literature >> > > (#5874) >> > > > - [AIRFLOW-5101] Fix inconsistent owner value in examples >(#5712) >> > > > - [AIRFLOW-XXX] Fix typo - AWS DynamoDB Hook (#6319) >> > > > - [AIRFLOW-XXX] Fix Documentation for adding extra Operator >Links >> > (#6301) >> > > > - [AIRFLOW-XXX] Add section on task lifecycle & correct casing >in >> docs >> > > > (#4681) >> > > > - [AIRFLOW-XXX] Make it clear that 1.10.5 wasn't accidentally >omitted >> > > from >> > > > UPDATING.md (#6240) >> > > > - [AIRFLOW-XXX] Improve format in code-block directives (#6242) >> > > > - [AIRFLOW-XXX] Format Sendgrid docs (#6245) >> > > > - [AIRFLOW-XXX] Update to new logo (#6066) >> > > > - [AIRFLOW-XXX] Typo in FAQ - schedule_interval (#6291) >> > > > - [AIRFLOW-XXX] Add message about breaking change in >> > > > DAG#get_task_instances in 1.10.4 (#6226) >> > > > - [AIRFLOW-XXX] Fix incorrect units in docs for metrics using >Timers >> > > > (#6152) >> > > > - [AIRFLOW-XXX] Fix backtick issues in .rst files & Add >Precommit >> hook >> > > > (#6162) >> > > > - [AIRFLOW-XXX] Update documentation about variables forcing >answer >> > > (#6158) >> > > > - [AIRFLOW-XXX] Add a third way to configure authorization >(#6134) >> > > > - [AIRFLOW-XXX] Add example of running pre-commit hooks on >single >> file >> > > > (#6143) >> > > > - [AIRFLOW-XXX] Add information about default pool to docs >(#6019) >> > > > - [AIRFLOW-XXX] Make Breeze The default integration test >environment >> > > > (#6001) >> > > > >> > > > Misc/Internal >> > > > """"""""""""" >> > > > - [AIRFLOW-5687] Upgrade pip to 19.0.2 in CI build pipeline >(#6358) >> > > (#6361) >> > > > - [AIRFLOW-5533] Fixed failing CRON build (#6167) >> > > > - [AIRFLOW-5130] Use GOOGLE_APPLICATION_CREDENTIALS constant >from >> > library >> > > > (#5744) >> > > > - [AIRFLOW-5369] Adds interactivity to pre-commits (#5976) >> > > > - [AIRFLOW-5531] Replace deprecated log.warn() with >log.warning() >> > (#6165) >> > > > - [AIRFLOW-4686] Make dags Pylint compatible (#5753) >> > > > - [AIRFLOW-4864] Remove calls to load_test_config (#5502) >> > > > - [AIRFLOW-XXX] Pin version of mypy so we are stable over time >> (#6198) >> > > > - [AIRFLOW-XXX] Add tests that got missed from #5127 >> > > > - [AIRFLOW-4928] Move config parses to class properties inside >DagBag >> > > > (#5557) >> > > > - [AIRFLOW-5003] Making AWS Hooks pylint compatible (#5627) >> > > > - [AIRFLOW-5580] Add base class for system test (#6229) >> > > > >> > > > >> > > >> > >> > >> > -- >> > >> > Jarek Potiuk >> > Polidea <https://www.polidea.com/> | Principal Software Engineer >> > >> > M: +48 660 796 129 <+48660796129> >> > [image: Polidea] <https://www.polidea.com/> >> > >> > > >-- > >Jarek Potiuk >Polidea <https://www.polidea.com/> | Principal Software Engineer > >M: +48 660 796 129 <+48660796129> >[image: Polidea] <https://www.polidea.com/>
