Hi Everyone, I'm excited to annoucne that 1.10.4 is almost ready for release.
The release branch already contains 216 commits since 1.10.3 this time around. In order to distinguish it from an actual (apache) release it is: 1. Marked as beta (python package managers do not install beta versions by default - PEP 440) 2. It is not at an official Apache distribution location It can be installed with: pip install 'apache-airflow==1.10.4b2' or downloaded from https://dist.apache.org/repos/dist/dev/airflow/1.10.4b2/ <https://dist.apache.org/repos/dist/dev/airflow/1.10.4b2/> (Don't worry, without asking for `--pre` or specifying the version `pip install apache-airflow` will still get 1.10.3) Included below is the changelog of this release so far. I'd like to try and get this out before the end of the month, so will create an RC in a day or two baring any major problems reported. Cheers, Ash Airflow 1.10.4 -------------- New Features """""""""""" - [AIRFLOW-4456] Add sub-classable BaseBranchOperator (#5231) - [AIRFLOW-4343] Show warning in UI if scheduler is not running (#5127) - [AIRFLOW-4739] Add ability to arbitrarily define kubernetes worker pod labels (#5376) - [AIRFLOW-4348] Add GCP console link in BigQueryOperator (#5195) - [AIRFLOW-4306] Global operator extra links (#5094) - [AIRFLOW-4812] Add batch images annotation (#5433) - [AIRFLOW-4135] Add Google Cloud Build operator and hook (#5251) - [AIRFLOW-4781] Add the ability to specify ports in kubernetesOperator (#5410) - [AIRFLOW-4521] Pause dag also pause its subdags (#5283) - [AIRFLOW-4738] Enforce exampleinclude for example DAGs (#5375) - [AIRFLOW-4326] Airflow AWS SQS Operator (#5110) - [AIRFLOW-3729] Support "DownwardAPI" in env variables for KubernetesPodOperator (#4554) - [AIRFLOW-4585] Implement Kubernetes Pod Mutation Hook (#5359) - [AIRFLOW-161] New redirect route and extra links (#5059) - [AIRFLOW-4420] Backfill respects task_concurrency (#5221) - [AIRFLOW-4147] Add Operator to publish event to Redis (#4967) - [AIRFLOW-3359] Add option to pass customer encryption keys to Dataproc (#4200) - [AIRFLOW-4318] Create Google Cloud Translate Speech Operator (#5102) - [AIRFLOW-3960] Adds Google Cloud Speech operators (#4780) - [AIRFLOW-1684] - Branching based on XCom variable (Docs) (#4365) - [AIRFLOW-1501] Add GoogleCloudStorageDeleteOperator (#5230) - [AIRFLOW-3672] Add support for Mongo DB DNS Seedlist Connection Format (#4481) - [AIRFLOW-4397] add integrations docs manually for gcs sensors (#5204) - [AIRFLOW-4251] Instrument DagRun schedule delay (#5050) - [AIRFLOW-4118] instrument DagRun duration (#4946) - [AIRFLOW-4361] Fix flaky test_integration_run_dag_with_scheduler_failure (#5182) - [AIRFLOW-4361] Fix flaky test_integration_run_dag_with_scheduler_failure (#5140) - [AIRFLOW-4168] Create Google Cloud Video Intelligence Operators (#4985) - [AIRFLOW-4397] Add GCSUploadSessionCompleteSensor (#5166) - [AIRFLOW-4335] Add default num_retries to GCP connection (#5117) - [AIRFLOW-3808] Add cluster_fields to BigQueryHook's create_empty_table (#4654) - [AIRFLOW-4362] Fix test_execution_limited_parallelism (#5141) - [AIRFLOW-4307] Backfill respects concurrency limit (#5128) - [AIRFLOW-4268] Add MsSqlToGoogleCloudStorageOperator (#5077) - [AIRFLOW-4169] Add Google Cloud Vision Detect Operators (#4986) - [AIRFLOW-XXX] Fix WS-2019-0032 (#5384) - [AIRFLOW-XXX] Fix CVE-2019-11358 (#5197) - [AIRFLOW-XXX] Change allowed version of Jinja2 to fix CVE-2019-10906 (#5075) Improvement """"""""""" - [AIRFLOW-4805] Add py_file as templated field in DataflowPythonOperator (#5451) - [AIRFLOW-4838] Surface Athena errors in AWSAthenaOperator (#5467) - [AIRFLOW-4831] conf.has_option no longer throws if section is missing. (#5455) - [AIRFLOW-4829] More descriptive exceptions for EMR sensors (#5452) - [AIRFLOW-4414] AWSAthenaOperator: Push QueryExecutionID to XCom (#5276) - [AIRFLOW-4791] add "schema" keyword arg to SnowflakeOperator (#5415) - [AIRFLOW-4759] Don't error when marking sucessful run as failed (#5435) - [AIRFLOW-4716] Instrument dag loading time duration (#5350) - [AIRFLOW-3958] Support list tasks as upstream in chain (#4779) - [AIRFLOW-4409] Prevent task duration break by null value (#5178) - [AIRFLOW-4418] Add "failed only" option to task modal (#5193) - [AIRFLOW-4740] Accept string ``end_date`` in DAG default_args (#5381) - [AIRFLOW-4423] Improve date handling in mysql to gcs operator. (#5196) - [AIRFLOW-4447] Display task duration as human friendly format in UI (#5218) - [AIRFLOW-4377] Remove needless object conversion in DAG.owner() (#5144) - [AIRFLOW-4766] Add autoscaling option for DataprocClusterCreateOperator (#5425) - [AIRFLOW-4795] Upgrade alembic to latest release. (#5411) - [AIRFLOW-4793] Add signature_name to mlengine operator (#5417) - [AIRFLOW-3211] Reattach to GCP Dataproc jobs upon Airflow restart (#4083) - [AIRFLOW-4750] Log identified zombie task instances (#5389) - [AIRFLOW-3870] STFPOperator: Update log level and return value (#4355) - [AIRFLOW-4759] Batch queries in set_state API. (#5403) - [AIRFLOW-2737] Restore original license header to airflow.api.auth.backend.kerberos_auth - [AIRFLOW-3635] Fix incorrect logic in detele_dag (introduced in PR#4406) (#4445) - [AIRFLOW-3599] Removed Dagbag from delete dag (#4406) - [AIRFLOW-4737] Increase and document celery queue name limit (#5383) - [AIRFLOW-4505] Correct Tag ALL for PY3 (#5275) - [AIRFLOW-4743] Add environment variables support to SSHOperator (#5385) - [AIRFLOW-4725] Fix setup.py PEP440 & Sphinx-PyPI-upload dependency (#5363) - [AIRFLOW-3370] Add stdout output options to Elasticsearch task log handler (#5048) - [AIRFLOW-4396] Provide a link to external Elasticsearch logs in UI. (#5164) - [AIRFLOW-1381] Allow setting host temporary directory in DockerOperator (#5369) - [AIRFLOW-4598] Task retries are not exhausted for K8s executor (#5347) - [AIRFLOW-4218] Support to Provide http args to K8executor while calling k8 python client lib apis (#5060) - [AIRFLOW-4159] Add support for additional static pod labels for K8sExecutor (#5134) - [AIRFLOW-4720] Allow comments in .airflowignore files. (#5355) - [AIRFLOW-4486] Add AWS IAM authenication in MySqlHook (#5334) - [AIRFLOW-4417] Add AWS IAM authenication for PostgresHook (#5223) - [AIRFLOW-3990] Compile regular expressions. (#4813) - [AIRFLOW-4572] Rename prepare_classpath() to prepare_syspath() (#5328) - [AIRFLOW-3869] Raise consistent exception in AirflowConfigParser.getboolean (#4692) - [AIRFLOW-4571] Add headers to templated field for SimpleHttpOperator (#5326) - [AIRFLOW-3867] Rename GCP's subpackage (#4690) - [AIRFLOW-3725] Add private_key to bigquery_hook get_pandas_df (#4549) - [AIRFLOW-4546] Upgrade google-cloud-bigtable. (#5307) - [AIRFLOW-4519] Optimise operator classname sorting in views (#5282) - [AIRFLOW-4503] Support fully pig options (#5271) - [AIRFLOW-4468] add sql_alchemy_max_overflow parameter (#5249) - [AIRFLOW-4467] Add dataproc_jars to templated fields in Dataproc oper… (#5248) - [AIRFLOW-4381] Use get_direct_relative_ids get task relatives (#5147) - [AIRFLOW-3624] Add masterType parameter to MLEngineTrainingOperator (#4428) - [AIRFLOW-3143] Support Auto-Zone in DataprocClusterCreateOperator (#5169) - [AIRFLOW-3874] Improve BigQueryHook.run_with_configuration's location support (#4695) - [AIRFLOW-4399] Avoid duplicated os.path.isfile() check in models.dagbag (#5165) - [AIRFLOW-4031] Allow for key pair auth in snowflake hook (#4875) - [AIRFLOW-3901] add role as optional config parameter for SnowflakeHook (#4721) - [AIRFLOW-3455] add region in snowflake connector (#4285) - [AIRFLOW-4073] add template_ext for AWS Athena operator (#4907) - [AIRFLOW-4093] AWSAthenaOperator: Throw exception if job failed/cancelled/reach max retries (#4919) - [AIRFLOW-4356] Add extra RuntimeEnvironment keys to DataFlowHook (#5149) - [AIRFLOW-4337] Fix docker-compose deprecation warning in CI (#5119) - [AIRFLOW-3603] QuboleOperator: Remove SQLCommand from SparkCmd documentation (#4411) - [AIRFLOW-4328] Fix link to task instances from Pool page (#5124) - [AIRFLOW-4255] Make GCS Hook Backwards compatible (#5089) - [AIRFLOW-4103] Allow uppercase letters in dataflow job names (#4925) - [AIRFLOW-4255] Replace Discovery based api with client based for GCS (#5054) - [AIRFLOW-4311] Remove sleep in localexecutor (#5096) - [AIRFLOW-2836] Minor improvement-contrib.sensors.FileSensor (#3674) - [AIRFLOW-4104] Add type annotations to common classes. (#4926) - [AIRFLOW-3910] Raise exception explicitly in Connection.get_hook() (#4728) - [AIRFLOW-3322] Update QuboleHook to fetch args dynamically from qds_sdk (#4165) - [AIRFLOW-4565] instrument celery executor (#5321) - [AIRFLOW-4573] Import airflow_local_settings after prepare_classpath (#5330) - [AIRFLOW-4448] Don't bake ENV and _cmd into tmp config for non-sudo (#4050) - [AIRFLOW-4295] Make ``method`` attribute case insensitive in HttpHook (#5313) - [AIRFLOW-3703] Add dnsPolicy option for KubernetesPodOperator (#4520) - [AIRFLOW-3057] add prev_*_date_success to template context (#5372) - [AIRFLOW-4336] Stop showing entire GCS files bytes in log for gcs_download_operator (#5151) - [AIRFLOW-4528] Cancel DataProc task on timeout (#5293) Bug fixes """"""""" - [AIRFLOW-4587] Replace self.conn with self.get_conn() in AWSAthenaHook (#5462) - [AIRFLOW-4516] K8s runAsUser and fsGroup cannot be strings (#5429) - [AIRFLOW-4298] Stop Scheduler repeatedly warning "connection invalidated" (#5470) - [AIRFLOW-4559] JenkinsJobTriggerOperator bugfix (#5318) - [AIRFLOW-4841] Pin Sphinx AutoApi to 1.0.0 (#5468) - [AIRFLOW-4479] Include s3_overwrite kwarg in load_bytes method (#5312) - [AIRFLOW-3746] Fix DockerOperator missing container exit (#4583) - [AIRFLOW-4233] Remove Template Extension from Bq to GCS Operator (#5456) - [AIRFLOW-2141][AIRFLOW-3157][AIRFLOW-4170] Serialize non-str value by JSON when importing Variables (#4991) - [AIRFLOW-4826] Remove warning from ``airflow resetdb`` command (#5447) - [AIRFLOW-4148] Fix editing DagRuns when clicking state column (#5436) - [AIRFLOW-4455] dag_details broken for subdags in RBAC UI (#5234) - [AIRFLOW-2955] Fix kubernetes pod operator to set requests and limits on task pods (#4551) - [AIRFLOW-4459] Fix wrong DAG count in /home page when DAG count is zero (#5235) - [AIRFLOW-3876] AttributeError: module 'distutils' has no attribute 'util' - [AIRFLOW-4146] Fix CgroupTaskRunner errors (#5224) - [AIRFLOW-4524] Fix bug with "Ignore \*" toggles in RBAC mode (#5378) - [AIRFLOW-4765] Fix DataProcPigOperator execute method (#5426) - [AIRFLOW-4798] obviate interdependencies for dagbag and TI tests (#5422) - [AIRFLOW-4800] fix GKEClusterHook ctor calls (#5424) - [AIRFLOW-4799] don't mutate self.env in BashOperator execute method (#5421) - [AIRFLOW-4393] Add retry logic when fetching pod status and/or logs in KubernetesPodOperator (#5284) - [AIRFLOW-4174] Fix HttpHook run with backoff (#5213) - [AIRFLOW-4463] Handle divide-by-zero errors in short retry intervals (#5243) - [AIRFLOW-2614] Speed up trigger_dag API call when lots of DAGs in system - [AIRFLOW-4756] add ti.state to ti.start_date as criteria for gantt (#5399) - [AIRFLOW-4760] Fix zip-packaged DAGs disappearing from DagBag when reloaded (#5404) - [AIRFLOW-4731] Fix GCS hook with google-storage-client 1.16 (#5368) - [AIRFLOW-3506] use match_phrase to query log_id in elasticsearch (#4342) - [AIRFLOW-4084] fix ElasticSearch log download (#5177) - [AIRFLOW-4501] Register pendulum datetime converter for sqla+pymysql (#5190) - [AIRFLOW-986] HiveCliHook ignores 'proxy_user' value in a connection's extra parameter (#5305) - [AIRFLOW-4442] fix hive_tblproperties in HiveToDruidTransfer (#5211) - [AIRFLOW-4557] Add gcp_conn_id parameter to get_sqlproxy_runner() of CloudSqlDatabaseHook (#5314) - [AIRFLOW-4545] Upgrade FAB to latest version (#4955) - [AIRFLOW-4492] Change Dataproc Cluster operators to poll Operations (#5269) - [AIRFLOW-4452] Webserver and Scheduler keep crashing because of slackclient update (#5225) - [AIRFLOW-4450] Fix request arguments in has_dag_access (#5220) - [AIRFLOW-4434] Support Impala with the HiveServer2Hook (#5206) - [AIRFLOW-3449] Write local dag parsing logs when remote logging enabled. (#5175) - [AIRFLOW-4300] Fix graph modal call when DAG has not yet run (#5185) - [AIRFLOW-4401] Use managers for Queue synchronization (#5200) - [AIRFLOW-3626] Fixed triggering DAGs contained within zip files (#4439) - [AIRFLOW-3720] Fix missmatch while comparing GCS and S3 files (#4766) - [AIRFLOW-4403] search by ``dag_id`` or ``owners`` in UI (#5184) - [AIRFLOW-4308] Fix TZ-loop around DST on python 3.6+ (#5095) - [AIRFLOW-4324] fix DAG fuzzy search in RBAC UI (#5131) - [AIRFLOW-4297] Temporary hot fix on manage_slas() for 1.10.4 release (#5150) - [AIRFLOW-4299] Upgrade to Celery 4.3.0 to fix crashing workers (#5116) - [AIRFLOW-4291] Correctly render doc_md in DAG graph page (#5121) - [AIRFLOW-4310] Fix incorrect link on Dag Details page (#5122) - [AIRFLOW-4331] Correct filter for Null-state runs from Dag Detail page (#5123) - [AIRFLOW-4294] Fix missing dag & task runs in UI dag_id contains a dot (#5111) - [AIRFLOW-4332] Upgrade sqlalchemy to remove security Vulnerability (#5113) - [AIRFLOW-4312] Add template_fields & template_ext to BigQueryCheckO… (#5097) - [AIRFLOW-4293] Fix downgrade in d4ecb8fbee3_add_schedule_interval_to_dag.py (#5086) - [AIRFLOW-4267] Fix TI duration in Graph View (#5071) - [AIRFLOW-4163] IntervalCheckOperator supports relative diff and not ignore 0 (#4983) - [AIRFLOW-3938] QuboleOperator Fixes and Support for SqlCommand (#4832) - [AIRFLOW-2903] Change default owner to ``airflow`` (#4151) - [aIRFLOW-4136] Fix overwrite of key_file by constructor (#5155) - [AIRFLOW-3241] Remove Invalid template ext in GCS Sensors (#4076) Misc/Internal """"""""""""" - [AIRFLOW-4211] Add tests for WebHDFSHook (#5015) - [AIRFLOW-4320] Add tests for SegmentTrackEventOperator (#5104) - [AIRFLOW-4319] Add tests for Bigquery related Operators (#5101) - [AIRFLOW-4014] Change DatastoreHook and add tests (#4842) - [AIRFLOW-4322] Add test for VerticaOperator (#5107) - [AIRFLOW-4323] Add 2 tests for WinRMOperator (#5108) - [AIRFLOW-3677] Improve CheckOperator test coverage (#4756) - [AIRFLOW-4659] Fix pylint problems for api module (#5398) - [AIRFLOW-4358] Speed up test_jobs by not running tasks (#5162) - [AIRFLOW-4394] Don't test behaviour of BackfillJob from CLI tests (#5160) - [AIRFLOW-3471] Move XCom out of models.py (#4629) - [AIRFLOW-4379] Remove duplicate code & Add validation in gcs_to_gcs.py (#5145) - [AIRFLOW-4259] Move models out of models.py (#5056) - [AIRFLOW-XXX] Speed up building of Cassanda module on Travis (#5233) - [AIRFLOW-4535] Break jobs.py into multiple files (#5303) - [AIRFLOW-1464] Batch update task_instance state (#5323) - [AIRFLOW-4554] Test for sudo command, add some other test docs (#5310) - [AIRFLOW-4419] Refine concurrency check in scheduler (#5194) - [AIRFLOW-4269] Minor acceleration of jobs._process_task_instances() (#5076) - [AIRFLOW-4341] Remove ``View.render()`` already exists in fab.BaseView (#5125) - [AIRFLOW-4342] Use @cached_property instead of re-implementing it each time (#5126) - [AIRFLOW-4256] Remove noqa from migrations (#5055) - [AIRFLOW-4034] Remove unnecessary string formatting with ``**locals()`` (#4861) - [AIRFLOW-3944] Remove code smells (#4762) Doc-only changes """""""""""""""" - [AIRFLOW-3341] FAQ return DAG object example (#4605) - [AIRFLOW-4433] Add missing type in DockerOperator doc string (#5205) - [AIRFLOW-4321] Replace incorrect info of Max Size limit of GCS Object Size (#5106) - [AIRFLOW-XXX] Add information about user list (#5341) - [AIRFLOW-XXX] Clarify documentation related to autodetect parameter in GCS_to_BQ Op (#5294) - [AIRFLOW-XXX] Remove mention of pytz compatibility from timezone documentation (#5316) - [AIRFLOW-XXX] Add missing docs for GoogleCloudStorageDeleteOperator (#5274) - [AIRFLOW-XXX] Remove incorrect note about Scopes of GCP connection (#5242) - [AIRFLOW-XXX] Fix mistakes in docs of Dataproc operators (#5192) - [AIRFLOW-XXX] Link to correct class for timedelta in macros.rst (#5226) - [AIRFLOW-XXX] Add Kamil as committer (#5216) - [AIRFLOW-XXX] Add Joshua and Kevin as committer (#5207) - [AIRFLOW-XXX] Reduce log spam in tests (#5174) - [AIRFLOW-XXX] Speed up tests for PythonSensor (#5158) - [AIRFLOW-XXX] Add Bas Harenslak to committer list (#5157) - [AIRFLOW-XXX] Add Jarek Potiuk to commiter list (#5132) - [AIRFLOW-XXX] Update docstring for SchedulerJob (#5105) - [AIRFLOW-XXX] Fix docstrings for CassandraToGoogleCloudStorageOperator (#5103) - [AIRFLOW-XXX] update SlackWebhookHook and SlackWebhookOperator docstring (#5074) - [AIRFLOW-XXX] Ignore python files under node_modules in docs (#5063) - [AIRFLOW-XXX] Build a universal wheel with LICNESE files (#5052) - [AIRFLOW-XXX] Fix docstrings of SQSHook (#5099) - [AIRFLOW-XXX] Use Py3.7 on readthedocs - [AIRFLOW-4446] Fix typos (#5217)
