If you are using num_runs or run_duration, remove them. We have deprecated run_duration in master (we should probably cherry-pick that in next release) - https://github.com/apache/airflow/blob/30defe130df1399d8ff41a350f714af820113161/UPDATING.md#remove-run_duration (Note in updating.md)
We should remove `num_runs` too and we have a jira open for that - https://jira.apache.org/jira/browse/AIRFLOW-3533 On Thu, Jul 18, 2019 at 11:15 PM James Meickle <[email protected]> wrote: > Yes, it's zombie processes owned by systemd. Here's a writeup with some > various logs: > https://gist.github.com/Eronarn/c5bc2df607d168d6fda3a70700c941d9 > > It's reproducible (lmk if you want to co-pilot on debugging), and we're up > to 3 zombies in the hour or two since our most recent reboot. So it won't > immediately take a cluster down, but it is a steady resource leak that > isn't fixed by restarting the scheduler process. > > On Thu, Jul 18, 2019 at 12:31 PM Ash Berlin-Taylor <[email protected]> wrote: > > > Errk, that's not good. > > > > When you say orphaned do you mean zombie processes owned by pid 1, or > just > > "process that are hanging around?" > > > > https://github.com/apache/airflow/pull/5605 might help if it's the later > > (but I suspect it isn't) > > > > We've also had reports of the scheduler having a memory leak since > ~1.10.2 > > https://issues.apache.org/jira/browse/AIRFLOW-4593 so that may be part > of > > the OOMing too. (If this is the cause of your problem then .4 isn't worse > > than .3 I don't think?) > > > > -ash > > > > > On 18 Jul 2019, at 15:54, James Meickle <[email protected] > .INVALID> > > wrote: > > > > > > Hi folks, > > > > > > Sorry to throw a wrench into this, but we're found that this release is > > > leaving orphaned processes that eventually OOMing the scheduler. 4 > > clusters > > > have this problem post-upgrade, while 1 non-upgraded cluster doesn't > have > > > this problem. > > > > > > In our case we're using Supervisor to manage scheduler restarts. I am > > still > > > investigating whether this is an interaction with our unique Supervisor > > > config, or if it's actually purely an Airflow problem. This switch to > > > multiprocessing.Manager looks like a promising avenue of investigation > > > though: > > > > > > https://github.com/apache/airflow/compare/1.10.3...1.10.4rc3#diff-ced4fd65ce02db58eed692eef6e01d05L495 > > > > > > > > > > > > > > > > > > On Wed, Jul 17, 2019 at 3:35 PM Bolke de Bruin <[email protected]> > > wrote: > > > > > >> +1, binding. > > >> > > >> Thanks ash! > > >> > > >> Verstuurd vanaf mijn iPad > > >> > > >>> Op 17 jul. 2019 om 20:45 heeft Kaxil Naik <[email protected]> het > > >> volgende geschreven: > > >>> > > >>> +1 (binding) Ran DAGs in both UIs. LGTM > > >>> > > >>> Great Job Ash. Appreciate all the effort you put into this. > > >>> > > >>> Regards, > > >>> Kaxil > > >>> > > >>> On Wed, Jul 17, 2019 at 10:30 PM Felix Uellendall > > <[email protected] > > >>> > > >>> wrote: > > >>> > > >>>> +1 (non-binding) flawlessly passed all my tests via rbac and classic > > ui. > > >>>> Also tested it on production-level dags. Great job Ash, thanks :) > > >>>> > > >>>> Kind regards, > > >>>> Felix > > >>>> > > >>>> Sent from ProtonMail mobile > > >>>> > > >>>> -------- Original Message -------- > > >>>>> On Jul 17, 2019, 16:41, Ash Berlin-Taylor wrote: > > >>>>> > > >>>>> Thanks Andrii - I missed it to when reviewing the PR so it's my > fault > > >>>> too. Make the PR against the v[1-10](tel:110)-stable branch please. > > >>>>> > > >>>>> -ash > > >>>>> > > >>>>>> On 17 Jul [2019](tel:2019), at 14:25, Andrii Soldatenko < > > >>>> [email protected]> wrote: > > >>>>>> > > >>>>>> @Ash, i'll fix wrong section bug. > > >>>>>> > > >>>>>> Sorry about that. > > >>>>>> > > >>>>>> On Wed, Jul 17, [2019](tel:2019) at 4:11 PM Robin Edwards < > > >>>> [email protected]> wrote: > > >>>>>> > > >>>>>>> +1 (none binding) - been running in production since RC2. > > >>>>>>> > > >>>>>>> Thanks for all your hard work > > >>>>>>> > > >>>>>>> R > > >>>>>>> > > >>>>>>> On Tue, 16 Jul [2019](tel:2019) at 21:15, Ash Berlin-Taylor < > > >>>> [email protected]> wrote: > > >>>>>>>> > > >>>>>>>> Thanks for testing. > > >>>>>>>> > > >>>>>>>> On 1) everyone should run upgradedb on every upgrade. The > > behaviour > > >> of > > >>>>>>> not running it wasn't great. > > >>>>>>>> > > >>>>>>>> 2) I thought we set deprecation on the ES logging config vars, > > >> except > > >>>> we > > >>>>>>> put the depreciation under the wrong section: > > >>>>>>> > > >>>> > > >> > > > https://github.com/apache/airflow/blob/1.10.4rc3/airflow/configuration.py#L147-L153 > > >>>>>>> :( > > >>>>>>>> > > >>>>>>>> On 16 July [2019 21](tel:201921):06:40 BST, James Meickle < > > >>>> [email protected]> > > >>>>>>> wrote: > > >>>>>>>>> +1 (nonbinding) to the release, it fixes a lot of UI issues > we've > > >>>> been > > >>>>>>>>> seeing lately. Though two notes: > > >>>>>>>>> > > >>>>>>>>> 1) Tasks were unscheduleable until I ran an upgradedb due the > > >> default > > >>>>>>>>> pool > > >>>>>>>>> change. > > >>>>>>>>> > > >>>>>>>>> 2) I got crash loops because I based our custom logging file > off > > of > > >>>> the > > >>>>>>>>> previous version's template. The change to the Elasticsearch > conf > > >>>>>>>>> variables > > >>>>>>>>> broke this, even though we don't directly use Elasticsearch, > > since > > >> it > > >>>>>>>>> was > > >>>>>>>>> now looking for undefined conf variables. It was confusing > until > > I > > >>>>>>>>> hunted > > >>>>>>>>> around the diff and backported the changes in the default > > template > > >> to > > >>>>>>>>> the > > >>>>>>>>> template that lives in our repo. > > >>>>>>>>> > > >>>>>>>>> On Mon, Jul 15, [2019](tel:2019) at 10:17 AM Ash Berlin-Taylor > < > > >>>> [email protected]> > > >>>>>>>>> wrote: > > >>>>>>>>> > > >>>>>>>>>> Hello Airflow community, > > >>>>>>>>>> > > >>>>>>>>>> This email is calling a vote on the release, which will last > for > > >> 72 > > >>>>>>>>> hours > > >>>>>>>>>> (2019-07-08 15:15 Z), and until three binding votes have been > > >> cast. > > >>>>>>>>>> Consider this my (binding) +1. > > >>>>>>>>>> > > >>>>>>>>>> Airflow [1.10.4](tel:1104) RC3 is available at: > > >>>>>>>>>> https://dist.apache.org/repos/dist/dev/airflow/1.10.4rc3/ > > >>>>>>>>>> > > >>>>>>>>>> *apache-airflow-[1.10.4](tel:1104)rc3-source.tar.gz* is a > source > > >>>> release that > > >>>>>>>>> comes > > >>>>>>>>>> with INSTALL instructions. > > >>>>>>>>>> *apache-airflow-[1.10.4](tel:1104)rc3-bin.tar.gz* is the > binary > > >>>> Python "sdist" > > >>>>>>>>> release. > > >>>>>>>>>> *apache_airflow-[1.10.4](tel:1104)rc3-py2.py3-none-any.whl* is > > the > > >>>> binary Python > > >>>>>>>>>> "wheel" release. > > >>>>>>>>>> > > >>>>>>>>>> For connivence of testers the RC is on PYPI too. It can be > > >> installed > > >>>>>>>>> with: > > >>>>>>>>>> > > >>>>>>>>>> pip install 'apache-airflow==[1.10.4](tel:1104)rc3' > > >>>>>>>>>> > > >>>>>>>>>> Public keys are available at: > > >>>>>>>>>> https://dist.apache.org/repos/dist/release/airflow/KEYS > > >>>>>>>>>> > > >>>>>>>>>> Only votes from PMC members are binding (sorry committers), > but > > >>>>>>>>> members of > > >>>>>>>>>> the community are encouraged to test the release and vote with > > >>>>>>>>>> "(non-binding)". > > >>>>>>>>>> > > >>>>>>>>>> Please note that the version number excludes the `rcX` string, > > so > > >>>>>>>>> it's now > > >>>>>>>>>> simply [1.10.4](tel:1104). This will allow us to rename the > > >>>> artefact without > > >>>>>>>>> modifying > > >>>>>>>>>> the checksums when we actually release. > > >>>>>>>>>> > > >>>>>>>>>> [ ] +1 Release this package as Apache Airflow > [1.10.4](tel:1104) > > >>>>>>>>>> [ ] 0 No opinion > > >>>>>>>>>> [ ] -1 Do not release this package because... > > >>>>>>>>>> > > >>>>>>>>>> Changes since RC2: > > >>>>>>>>>> > > >>>>>>>>>> [AIRFLOW-XXX] Update changelog for [1.10.4](tel:1104)rc3 [Ash > > >>>> Berlin-Taylor] > > >>>>>>>>>> [AIRFLOW-XXX] Disable intersphinx loading of `requests` > modules > > >>>>>>>>> (#[5590](tel:5590)) > > >>>>>>>>>> [Ash Berlin-Taylor] > > >>>>>>>>>> [AIRFLOW-[4797](tel:4797)] Improve performance and behaviour > of > > >>>> zombie detection > > >>>>>>>>>> (#[5511](tel:5511)) [Stefan Seelmann] > > >>>>>>>>>> [AIRFLOW-[4911](tel:4911)] Silence the FORBIDDEN errors from > the > > >>>>>>>>> KubernetesExecutor > > >>>>>>>>>> (#[5547](tel:5547)) [Andrii Soldatenko] > > >>>>>>>>>> [AIRFLOW-[4884](tel:4884)] Roll up import_errors in RBAC UI > > >>>> (#[5516](tel:5516)) > > >>>>>>>>> [shuwen-zhang] > > >>>>>>>>>> [AIRFLOW-[4937](tel:4937)] Fix lodash security issue with > > version > > >>>> below [4.17.13](tel:41713) > > >>>>>>>>>> (#[5572](tel:5572)) [Jarek Potiuk] > > >>>>>>>>>> [AIRFLOW-[4934](tel:4934)] Bump Flask to resolve Werkzeug > > ProxyFix > > >>>> (#[5571](tel:5571)) > > >>>>>>>>> [Nathaniel > > >>>>>>>>>> Ritholtz] > > >>>>>>>>>> [AIRFLOW-[4934](tel:4934)] Fix ProxyFix due to Werkzeug > upgrade > > >>>> (#[5563](tel:5563)) > > >>>>>>>>> [Nathaniel > > >>>>>>>>>> Ritholtz] > > >>>>>>>>>> [AIRFLOW-[4136](tel:4136)] fix key_file of hook is overwritten > > by > > >>>> SSHHook > > >>>>>>>>> connection > > >>>>>>>>>> (#[5558](tel:5558)) [eladkal] > > >>>>>>>>>> [AIRFLOW-XXX] Correct BaseSensorOperator docs > > (#[5562](tel:5562)) > > >>>> [Matthew > > >>>>>>>>> Bowden] > > >>>>>>>>>> [AIRFLOW-[3495](tel:3495)] Validate one of query and query_uri > > >>>> passed to > > >>>>>>>>>> DataProcSparkSqlOperator (#[5510](tel:5510)) [eladkal] > > >>>>>>>>>> [AIRFLOW-[4868](tel:4868)] Fix typo in > > kubernetes/docker/build.sh > > >>>> (#[5505](tel:5505)) [Facu] > > >>>>>>>>>> [AIRFLOW-[4925](tel:4925)] Improve css style for Variables > > Import > > >>>> file field > > >>>>>>>>> (#[5552](tel:5552)) > > >>>>>>>>>> [Felix Uellendall] > > >>>>>>>>>> [AIRFLOW-[4906](tel:4906)] Improve debugging for the > > >>>> SparkSubmitHook (#[5542](tel:5542)) > > >>>>>>>>> [Fokko > > >>>>>>>>>> Driesprong] > > >>>>>>>>>> [AIRFLOW-[4904](tel:4904)] Retrieve test config file from > > >>>> $AIRFLOW_TEST_CONFIG > > >>>>>>>>> (#[5540](tel:5540)) > > >>>>>>>>>> [Matthew Rastovac] > > >>>>>>>>>> [AIRFLOW-[4926](tel:4926)] Fix example dags where its > start_date > > >> is > > >>>>>>>>> datetime.utcnow() > > >>>>>>>>>> (#[5553](tel:5553)) [Felix Uellendall] > > >>>>>>>>>> [AIRFLOW-[4587](tel:4587)] Replace self.conn with > > self.get_conn() > > >> in > > >>>>>>>>> AWSAthenaHook > > >>>>>>>>>> (#[5545](tel:5545)) [Bhavika Tekwani] > > >>>>>>>>>> [AIRFLOW-[1740](tel:1740)] Fix xcom creation and update via UI > > >>>> (#[5530](tel:5530)) [Felix > > >>>>>>>>>> Uellendall] > > >>>>>>>>>> [AIRFLOW-[1740](tel:1740)] Fix xcom creation and update via > RBAC > > >> UI > > >>>> (#[5561](tel:5561)) > > >>>>>>>>> [Felix > > >>>>>>>>>> Uellendall] > > >>>>>>>>>> [AIRFLOW-[4920](tel:4920)] Use html.escape instead of > cgi.escape > > >> to > > >>>> fix > > >>>>>>>>>> DeprecationWarning (#[5551](tel:5551)) [Xtreak] > > >>>>>>>>>> [AIRFLOW-[4919](tel:4919)] DataProcJobBaseOperator > > >>>> dataproc_*_properties > > >>>>>>>>> templated > > >>>>>>>>>> (#[5555](tel:5555)) [eladkal] > > >>>>>>>>>> [AIRFLOW-[4900](tel:4900)] Resolve incompatible version of > > >> Werkzeug > > >>>> (#[5535](tel:5535)) > > >>>>>>>>> [Luqman > > >>>>>>>>>> Abdul Mushawwir] > > >>>>>>>>>> [AIRFLOW-[4478](tel:4478)] Lazily instantiate default > resources > > >>>> objects. (#[5259](tel:5259)) > > >>>>>>>>>> [Joshua Carp] > > >>>>>>>>>> [AIRFLOW-[4510](tel:4510)] Don't mutate default_args during > DAG > > >>>> initialization > > >>>>>>>>> (#[5277](tel:5277)) > > >>>>>>>>>> [Abhishek Ray] > > >>>>>>>>>> [AIRFLOW-[3360](tel:3360)] Make the DAGs search respect other > > >>>> querystring > > >>>>>>>>> parameters > > >>>>>>>>>> with url-search-params-polyfill for IE support > > (#[5503](tel:5503)) > > >>>> [Alex] > > >>>>>>>>>> [AIRFLOW-[4896](tel:4896)] Make KubernetesExecutorConfig's > > default > > >>>> args immutable > > >>>>>>>>>> (#[5534](tel:5534)) [Philippe Gagnon] > > >>>>>>>>>> [AIRFLOW-[4494](tel:4494)] Remove `shell=True` in DaskExecutor > > >>>> (#[5273](tel:5273)) [MisLink] > > >>>>>>>>>> [AIRFLOW-[4890](tel:4890)] Fix Log link in TaskInstance's View > > for > > >>>> Non-RBAC > > >>>>>>>>> (#[5525](tel:5525)) > > >>>>>>>>>> [Felix Uellendall] > > >>>>>>>>>> [AIRFLOW-[4892](tel:4892)] Fix connection creation via UIs > > >>>> (#[5527](tel:5527)) [Ash > > >>>>>>>>> Berlin-Taylor] > > >>>>>>>>>> [AIRFLOW-[4862](tel:4862)] Fix bug for earlier change to allow > > >>>> using IP as > > >>>>>>>>> hostname > > >>>>>>>>>> (#[5513](tel:5513)) [Xiaodong] > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> Full Changelog since [1.10.3](tel:1103): > > >>>>>>>>>> > > >>>>>>>>>> Airflow [1.10.4](tel:1104), - [2019-07-18](tel:20190718) > > >>>>>>>>>> ---------------------------- > > >>>>>>>>>> > > >>>>>>>>>> New Features > > >>>>>>>>>> """""""""""" > > >>>>>>>>>> - [AIRFLOW-[4884](tel:4884)] Roll up import_errors in RBAC UI > > >>>> (#[5516](tel:5516)) > > >>>>>>>>>> - [AIRFLOW-[4871](tel:4871)] Allow creating DagRuns via RBAC > UI > > >>>> (#[5507](tel:5507)) > > >>>>>>>>>> - [AIRFLOW-[4591](tel:4591)] Make default_pool a real pool > > >>>> (#[5349](tel:5349)) > > >>>>>>>>>> - [AIRFLOW-[4844](tel:4844)] Add optional > > is_paused_upon_creation > > >>>> argument to DAG > > >>>>>>>>>> (#[5473](tel:5473)) > > >>>>>>>>>> - [AIRFLOW-[4456](tel:4456)] Add sub-classable > > BaseBranchOperator > > >>>> (#[5231](tel:5231)) > > >>>>>>>>>> - [AIRFLOW-[4343](tel:4343)] Show warning in UI if scheduler > is > > >> not > > >>>> running > > >>>>>>>>> (#[5127](tel:5127)) > > >>>>>>>>>> - [AIRFLOW-[4739](tel:4739)] Add ability to arbitrarily define > > >>>> kubernetes worker > > >>>>>>>>> pod > > >>>>>>>>>> labels (#[5376](tel:5376)) > > >>>>>>>>>> - [AIRFLOW-[4348](tel:4348)] Add GCP console link in > > >>>> BigQueryOperator (#[5195](tel:5195)) > > >>>>>>>>>> - [AIRFLOW-[4306](tel:4306)] Global operator extra links > > >>>> (#[5094](tel:5094)) > > >>>>>>>>>> - [AIRFLOW-[4812](tel:4812)] Add batch images annotation > > >>>> (#[5433](tel:5433)) > > >>>>>>>>>> - [AIRFLOW-[4135](tel:4135)] Add Google Cloud Build operator > and > > >>>> hook (#[5251](tel:5251)) > > >>>>>>>>>> - [AIRFLOW-[4781](tel:4781)] Add the ability to specify ports > in > > >>>>>>>>> kubernetesOperator > > >>>>>>>>>> (#[5410](tel:5410)) > > >>>>>>>>>> - [AIRFLOW-[4521](tel:4521)] Pause dag also pause its subdags > > >>>> (#[5283](tel:5283)) > > >>>>>>>>>> - [AIRFLOW-[4738](tel:4738)] Enforce exampleinclude for > example > > >>>> DAGs (#[5375](tel:5375)) > > >>>>>>>>>> - [AIRFLOW-[4326](tel:4326)] Airflow AWS SQS Operator > > >>>> (#[5110](tel:5110)) > > >>>>>>>>>> - [AIRFLOW-[3729](tel:3729)] Support "DownwardAPI" in env > > >> variables > > >>>> for > > >>>>>>>>>> KubernetesPodOperator (#[4554](tel:4554)) > > >>>>>>>>>> - [AIRFLOW-[4585](tel:4585)] Implement Kubernetes Pod Mutation > > >> Hook > > >>>> (#[5359](tel:5359)) > > >>>>>>>>>> - [AIRFLOW-[161](tel:161)] New redirect route and extra links > > >>>> (#[5059](tel:5059)) > > >>>>>>>>>> - [AIRFLOW-[4420](tel:4420)] Backfill respects > task_concurrency > > >>>> (#[5221](tel:5221)) > > >>>>>>>>>> - [AIRFLOW-[4147](tel:4147)] Add Operator to publish event to > > >> Redis > > >>>> (#[4967](tel:4967)) > > >>>>>>>>>> - [AIRFLOW-[3359](tel:3359)] Add option to pass customer > > >> encryption > > >>>> keys to > > >>>>>>>>> Dataproc > > >>>>>>>>>> (#[4200](tel:4200)) > > >>>>>>>>>> - [AIRFLOW-[4318](tel:4318)] Create Google Cloud Translate > > Speech > > >>>> Operator > > >>>>>>>>> (#[5102](tel:5102)) > > >>>>>>>>>> - [AIRFLOW-[3960](tel:3960)] Adds Google Cloud Speech > operators > > >>>> (#[4780](tel:4780)) > > >>>>>>>>>> - [AIRFLOW-[1501](tel:1501)] Add > > GoogleCloudStorageDeleteOperator > > >>>> (#[5230](tel:5230)) > > >>>>>>>>>> - [AIRFLOW-[3672](tel:3672)] Add support for Mongo DB DNS > > Seedlist > > >>>> Connection > > >>>>>>>>> Format > > >>>>>>>>>> (#[4481](tel:4481)) > > >>>>>>>>>> - [AIRFLOW-[4397](tel:4397)] add integrations docs manually > for > > >> gcs > > >>>> sensors > > >>>>>>>>> (#[5204](tel:5204)) > > >>>>>>>>>> - [AIRFLOW-[4251](tel:4251)] Instrument DagRun schedule delay > > >>>> (#[5050](tel:5050)) > > >>>>>>>>>> - [AIRFLOW-[4118](tel:4118)] instrument DagRun duration > > >>>> (#[4946](tel:4946)) > > >>>>>>>>>> - [AIRFLOW-[4361](tel:4361)] Fix flaky > > >>>>>>>>> test_integration_run_dag_with_scheduler_failure > > >>>>>>>>>> (#[5182](tel:5182)) > > >>>>>>>>>> - [AIRFLOW-[4361](tel:4361)] Fix flaky > > >>>>>>>>> test_integration_run_dag_with_scheduler_failure > > >>>>>>>>>> (#[5140](tel:5140)) > > >>>>>>>>>> - [AIRFLOW-[4168](tel:4168)] Create Google Cloud Video > > >> Intelligence > > >>>> Operators > > >>>>>>>>> (#[4985](tel:4985)) > > >>>>>>>>>> - [AIRFLOW-[4397](tel:4397)] Add > GCSUploadSessionCompleteSensor > > >>>> (#[5166](tel:5166)) > > >>>>>>>>>> - [AIRFLOW-[4335](tel:4335)] Add default num_retries to GCP > > >>>> connection (#[5117](tel:5117)) > > >>>>>>>>>> - [AIRFLOW-[3808](tel:3808)] Add cluster_fields to > > BigQueryHook's > > >>>>>>>>> create_empty_table > > >>>>>>>>>> (#[4654](tel:4654)) > > >>>>>>>>>> - [AIRFLOW-[4362](tel:4362)] Fix > > >> test_execution_limited_parallelism > > >>>> (#[5141](tel:5141)) > > >>>>>>>>>> - [AIRFLOW-[4307](tel:4307)] Backfill respects concurrency > limit > > >>>> (#[5128](tel:5128)) > > >>>>>>>>>> - [AIRFLOW-[4268](tel:4268)] Add > > MsSqlToGoogleCloudStorageOperator > > >>>> (#[5077](tel:5077)) > > >>>>>>>>>> - [AIRFLOW-[4169](tel:4169)] Add Google Cloud Vision Detect > > >>>> Operators (#[4986](tel:4986)) > > >>>>>>>>>> - [AIRFLOW-XXX] Fix WS-[2019-0032](tel:20190032) > > >> (#[5384](tel:5384)) > > >>>>>>>>>> - [AIRFLOW-XXX] Fix CVE-[2019-11358](tel:201911358) > > >>>> (#[5197](tel:5197)) > > >>>>>>>>>> - [AIRFLOW-XXX] Change allowed version of Jinja2 to fix > > >>>>>>>>> CVE-[2019-10906](tel:201910906) > > >>>>>>>>>> (#[5075](tel:5075)) > > >>>>>>>>>> > > >>>>>>>>>> Improvement > > >>>>>>>>>> """"""""""" > > >>>>>>>>>> - [AIRFLOW-[4797](tel:4797)] Improve performance and behaviour > > of > > >>>> zombie > > >>>>>>>>> detection > > >>>>>>>>>> (#[5511](tel:5511)) > > >>>>>>>>>> - [AIRFLOW-[4911](tel:4911)] Silence the FORBIDDEN errors from > > the > > >>>>>>>>> KubernetesExecutor > > >>>>>>>>>> (#[5547](tel:5547)) > > >>>>>>>>>> - [AIRFLOW-[3495](tel:3495)] Validate one of query and > query_uri > > >>>> passed to > > >>>>>>>>>> DataProcSparkSqlOperator (#[5510](tel:5510)) > > >>>>>>>>>> - [AIRFLOW-[4925](tel:4925)] Improve css style for Variables > > >> Import > > >>>> file field > > >>>>>>>>> (#[5552](tel:5552)) > > >>>>>>>>>> - [AIRFLOW-[4906](tel:4906)] Improve debugging for the > > >>>> SparkSubmitHook (#[5542](tel:5542)) > > >>>>>>>>>> - [AIRFLOW-[4904](tel:4904)] unittest.cfg name and path can be > > >>>> overriden by > > >>>>>>>>> setting > > >>>>>>>>>> $AIRFLOW_TEST_CONFIG (#[5540](tel:5540)) > > >>>>>>>>>> - [AIRFLOW-[4920](tel:4920)] Use html.escape instead of > > cgi.escape > > >>>> to fix > > >>>>>>>>>> DeprecationWarning (#[5551](tel:5551)) > > >>>>>>>>>> - [AIRFLOW-[4919](tel:4919)] DataProcJobBaseOperator > > >>>> dataproc_*_properties > > >>>>>>>>> templated > > >>>>>>>>>> (#[5555](tel:5555)) > > >>>>>>>>>> - [AIRFLOW-[4478](tel:4478)] Lazily instantiate default > > resources > > >>>> objects. > > >>>>>>>>> (#[5259](tel:5259)) > > >>>>>>>>>> - [AIRFLOW-[4564](tel:4564)] AzureContainerInstance bugfixes > and > > >>>> improvements > > >>>>>>>>> (#[5319](tel:5319)) > > >>>>>>>>>> - [AIRFLOW-[4237](tel:4237)] Including Try Number of Task in > > Gantt > > >>>> Chart (#[5037](tel:5037)) > > >>>>>>>>>> - [AIRFLOW-[4862](tel:4862)] Allow directly using IP address > as > > >>>> hostname for > > >>>>>>>>> webserver > > >>>>>>>>>> logs (#[5501](tel:5501)) > > >>>>>>>>>> - [AIRFLOW-[4857](tel:4857)] Add templated fields to > > >>>> SlackWebhookOperator (#[5490](tel:5490)) > > >>>>>>>>>> - [AIRFLOW-[3502](tel:3502)] Add celery config option for > > setting > > >>>> "pool" (#[4308](tel:4308)) > > >>>>>>>>>> - [AIRFLOW-[3217](tel:3217)] Button to toggle line wrapping in > > log > > >>>> and code views > > >>>>>>>>>> (#[4277](tel:4277)) > > >>>>>>>>>> - [AIRFLOW-[4491](tel:4491)] Add a "Jump to end" button for > logs > > >>>> (#[5266](tel:5266)) > > >>>>>>>>>> - [AIRFLOW-[4422](tel:4422)] Pool utilization stats > > >>>> (#[5453](tel:5453)) > > >>>>>>>>>> - [AIRFLOW-[4805](tel:4805)] Add py_file as templated field in > > >>>>>>>>> DataflowPythonOperator > > >>>>>>>>>> (#[5451](tel:5451)) > > >>>>>>>>>> - [AIRFLOW-[4838](tel:4838)] Surface Athena errors in > > >>>> AWSAthenaOperator (#[5467](tel:5467)) > > >>>>>>>>>> - [AIRFLOW-[4831](tel:4831)] conf.has_option no longer throws > if > > >>>> section is > > >>>>>>>>> missing. > > >>>>>>>>>> (#[5455](tel:5455)) > > >>>>>>>>>> - [AIRFLOW-[4829](tel:4829)] More descriptive exceptions for > EMR > > >>>> sensors (#[5452](tel:5452)) > > >>>>>>>>>> - [AIRFLOW-[4414](tel:4414)] AWSAthenaOperator: Push > > >>>> QueryExecutionID to XCom > > >>>>>>>>> (#[5276](tel:5276)) > > >>>>>>>>>> - [AIRFLOW-[4791](tel:4791)] add "schema" keyword arg to > > >>>> SnowflakeOperator > > >>>>>>>>> (#[5415](tel:5415)) > > >>>>>>>>>> - [AIRFLOW-[4759](tel:4759)] Don't error when marking > sucessful > > >> run > > >>>> as failed > > >>>>>>>>> (#[5435](tel:5435)) > > >>>>>>>>>> - [AIRFLOW-[4716](tel:4716)] Instrument dag loading time > > duration > > >>>> (#[5350](tel:5350)) > > >>>>>>>>>> - [AIRFLOW-[3958](tel:3958)] Support list tasks as upstream in > > >>>> chain (#[4779](tel:4779)) > > >>>>>>>>>> - [AIRFLOW-[4409](tel:4409)] Prevent task duration break by > null > > >>>> value (#[5178](tel:5178)) > > >>>>>>>>>> - [AIRFLOW-[4418](tel:4418)] Add "failed only" option to task > > >> modal > > >>>> (#[5193](tel:5193)) > > >>>>>>>>>> - [AIRFLOW-[4740](tel:4740)] Accept string ``end_date`` in DAG > > >>>> default_args > > >>>>>>>>> (#[5381](tel:5381)) > > >>>>>>>>>> - [AIRFLOW-[4423](tel:4423)] Improve date handling in mysql to > > gcs > > >>>> operator. > > >>>>>>>>> (#[5196](tel:5196)) > > >>>>>>>>>> - [AIRFLOW-[4447](tel:4447)] Display task duration as human > > >>>> friendly format in UI > > >>>>>>>>>> (#[5218](tel:5218)) > > >>>>>>>>>> - [AIRFLOW-[4377](tel:4377)] Remove needless object conversion > > in > > >>>> DAG.owner() > > >>>>>>>>> (#[5144](tel:5144)) > > >>>>>>>>>> - [AIRFLOW-[4766](tel:4766)] Add autoscaling option for > > >>>>>>>>> DataprocClusterCreateOperator > > >>>>>>>>>> (#[5425](tel:5425)) > > >>>>>>>>>> - [AIRFLOW-[4795](tel:4795)] Upgrade alembic to latest > release. > > >>>> (#[5411](tel:5411)) > > >>>>>>>>>> - [AIRFLOW-[4793](tel:4793)] Add signature_name to mlengine > > >>>> operator (#[5417](tel:5417)) > > >>>>>>>>>> - [AIRFLOW-[3211](tel:3211)] Reattach to GCP Dataproc jobs > upon > > >>>> Airflow restart > > >>>>>>>>>> (#[4083](tel:4083)) > > >>>>>>>>>> - [AIRFLOW-[4750](tel:4750)] Log identified zombie task > > instances > > >>>> (#[5389](tel:5389)) > > >>>>>>>>>> - [AIRFLOW-[3870](tel:3870)] STFPOperator: Update log level > and > > >>>> return value > > >>>>>>>>> (#[4355](tel:4355)) > > >>>>>>>>>> - [AIRFLOW-[4759](tel:4759)] Batch queries in set_state API. > > >>>> (#[5403](tel:5403)) > > >>>>>>>>>> - [AIRFLOW-[2737](tel:2737)] Restore original license header > to > > >>>>>>>>>> airflow.api.auth.backend.kerberos_auth > > >>>>>>>>>> - [AIRFLOW-[3635](tel:3635)] Fix incorrect logic in detele_dag > > >>>> (introduced in > > >>>>>>>>> PR#[4406](tel:4406)) > > >>>>>>>>>> (#[4445](tel:4445)) > > >>>>>>>>>> - [AIRFLOW-[3599](tel:3599)] Removed Dagbag from delete dag > > >>>> (#[4406](tel:4406)) > > >>>>>>>>>> - [AIRFLOW-[4737](tel:4737)] Increase and document celery > queue > > >>>> name limit > > >>>>>>>>> (#[5383](tel:5383)) > > >>>>>>>>>> - [AIRFLOW-[4505](tel:4505)] Correct Tag ALL for PY3 > > >>>> (#[5275](tel:5275)) > > >>>>>>>>>> - [AIRFLOW-[4743](tel:4743)] Add environment variables support > > to > > >>>> SSHOperator > > >>>>>>>>> (#[5385](tel:5385)) > > >>>>>>>>>> - [AIRFLOW-[4725](tel:4725)] Fix setup.py PEP[440](tel:440) & > > >>>> Sphinx-PyPI-upload dependency > > >>>>>>>>>> (#[5363](tel:5363)) > > >>>>>>>>>> - [AIRFLOW-[3370](tel:3370)] Add stdout output options to > > >>>> Elasticsearch task log > > >>>>>>>>>> handler (#[5048](tel:5048)) > > >>>>>>>>>> - [AIRFLOW-[4396](tel:4396)] Provide a link to external > > >>>> Elasticsearch logs in UI. > > >>>>>>>>>> (#[5164](tel:5164)) > > >>>>>>>>>> - [AIRFLOW-[1381](tel:1381)] Allow setting host temporary > > >> directory > > >>>> in > > >>>>>>>>> DockerOperator > > >>>>>>>>>> (#[5369](tel:5369)) > > >>>>>>>>>> - [AIRFLOW-[4598](tel:4598)] Task retries are not exhausted > for > > >> K8s > > >>>> executor > > >>>>>>>>> (#[5347](tel:5347)) > > >>>>>>>>>> - [AIRFLOW-[4218](tel:4218)] Support to Provide http args to > > >>>> K8executor while > > >>>>>>>>> calling > > >>>>>>>>>> k8 python client lib apis (#[5060](tel:5060)) > > >>>>>>>>>> - [AIRFLOW-[4159](tel:4159)] Add support for additional static > > pod > > >>>> labels for > > >>>>>>>>>> K8sExecutor (#[5134](tel:5134)) > > >>>>>>>>>> - [AIRFLOW-[4720](tel:4720)] Allow comments in .airflowignore > > >>>> files. (#[5355](tel:5355)) > > >>>>>>>>>> - [AIRFLOW-[4486](tel:4486)] Add AWS IAM authenication in > > >> MySqlHook > > >>>> (#[5334](tel:5334)) > > >>>>>>>>>> - [AIRFLOW-[4417](tel:4417)] Add AWS IAM authenication for > > >>>> PostgresHook (#[5223](tel:5223)) > > >>>>>>>>>> - [AIRFLOW-[3990](tel:3990)] Compile regular expressions. > > >>>> (#[4813](tel:4813)) > > >>>>>>>>>> - [AIRFLOW-[4572](tel:4572)] Rename prepare_classpath() to > > >>>> prepare_syspath() > > >>>>>>>>> (#[5328](tel:5328)) > > >>>>>>>>>> - [AIRFLOW-[3869](tel:3869)] Raise consistent exception in > > >>>>>>>>>> AirflowConfigParser.getboolean (#[4692](tel:4692)) > > >>>>>>>>>> - [AIRFLOW-[4571](tel:4571)] Add headers to templated field > for > > >>>>>>>>> SimpleHttpOperator > > >>>>>>>>>> (#[5326](tel:5326)) > > >>>>>>>>>> - [AIRFLOW-[3867](tel:3867)] Rename GCP's subpackage > > >>>> (#[4690](tel:4690)) > > >>>>>>>>>> - [AIRFLOW-[3725](tel:3725)] Add private_key to bigquery_hook > > >>>> get_pandas_df > > >>>>>>>>> (#[4549](tel:4549)) > > >>>>>>>>>> - [AIRFLOW-[4546](tel:4546)] Upgrade google-cloud-bigtable. > > >>>> (#[5307](tel:5307)) > > >>>>>>>>>> - [AIRFLOW-[4519](tel:4519)] Optimise operator classname > sorting > > >> in > > >>>> views (#[5282](tel:5282)) > > >>>>>>>>>> - [AIRFLOW-[4503](tel:4503)] Support fully pig options > > >>>> (#[5271](tel:5271)) > > >>>>>>>>>> - [AIRFLOW-[4468](tel:4468)] add sql_alchemy_max_overflow > > >> parameter > > >>>> (#[5249](tel:5249)) > > >>>>>>>>>> - [AIRFLOW-[4467](tel:4467)] Add dataproc_jars to templated > > fields > > >>>> in Dataproc > > >>>>>>>>> oper… > > >>>>>>>>>> (#[5248](tel:5248)) > > >>>>>>>>>> - [AIRFLOW-[4381](tel:4381)] Use get_direct_relative_ids get > > task > > >>>> relatives > > >>>>>>>>> (#[5147](tel:5147)) > > >>>>>>>>>> - [AIRFLOW-[3624](tel:3624)] Add masterType parameter to > > >>>> MLEngineTrainingOperator > > >>>>>>>>>> (#[4428](tel:4428)) > > >>>>>>>>>> - [AIRFLOW-[3143](tel:3143)] Support Auto-Zone in > > >>>> DataprocClusterCreateOperator > > >>>>>>>>> (#[5169](tel:5169)) > > >>>>>>>>>> - [AIRFLOW-[3874](tel:3874)] Improve > > >>>> BigQueryHook.run_with_configuration's > > >>>>>>>>> location > > >>>>>>>>>> support (#[4695](tel:4695)) > > >>>>>>>>>> - [AIRFLOW-[4399](tel:4399)] Avoid duplicated os.path.isfile() > > >>>> check in > > >>>>>>>>> models.dagbag > > >>>>>>>>>> (#[5165](tel:5165)) > > >>>>>>>>>> - [AIRFLOW-[4031](tel:4031)] Allow for key pair auth in > > snowflake > > >>>> hook (#[4875](tel:4875)) > > >>>>>>>>>> - [AIRFLOW-[3901](tel:3901)] add role as optional config > > parameter > > >>>> for > > >>>>>>>>> SnowflakeHook > > >>>>>>>>>> (#[4721](tel:4721)) > > >>>>>>>>>> - [AIRFLOW-[3455](tel:3455)] add region in snowflake connector > > >>>> (#[4285](tel:4285)) > > >>>>>>>>>> - [AIRFLOW-[4073](tel:4073)] add template_ext for AWS Athena > > >>>> operator (#[4907](tel:4907)) > > >>>>>>>>>> - [AIRFLOW-[4093](tel:4093)] AWSAthenaOperator: Throw > exception > > if > > >>>> job > > >>>>>>>>>> failed/cancelled/reach max retries (#[4919](tel:4919)) > > >>>>>>>>>> - [AIRFLOW-[4356](tel:4356)] Add extra RuntimeEnvironment keys > > to > > >>>> DataFlowHook > > >>>>>>>>> (#[5149](tel:5149)) > > >>>>>>>>>> - [AIRFLOW-[4337](tel:4337)] Fix docker-compose deprecation > > >> warning > > >>>> in CI (#[5119](tel:5119)) > > >>>>>>>>>> - [AIRFLOW-[3603](tel:3603)] QuboleOperator: Remove SQLCommand > > >> from > > >>>> SparkCmd > > >>>>>>>>>> documentation (#[4411](tel:4411)) > > >>>>>>>>>> - [AIRFLOW-[4328](tel:4328)] Fix link to task instances from > > Pool > > >>>> page (#[5124](tel:5124)) > > >>>>>>>>>> - [AIRFLOW-[4255](tel:4255)] Make GCS Hook Backwards > compatible > > >>>> (#[5089](tel:5089)) > > >>>>>>>>>> - [AIRFLOW-[4103](tel:4103)] Allow uppercase letters in > dataflow > > >>>> job names > > >>>>>>>>> (#[4925](tel:4925)) > > >>>>>>>>>> - [AIRFLOW-[4255](tel:4255)] Replace Discovery based api with > > >>>> client based for > > >>>>>>>>> GCS > > >>>>>>>>>> (#[5054](tel:5054)) > > >>>>>>>>>> - [AIRFLOW-[4311](tel:4311)] Remove sleep in localexecutor > > >>>> (#[5096](tel:5096)) > > >>>>>>>>>> - [AIRFLOW-[2836](tel:2836)] Minor > > >>>> improvement-contrib.sensors.FileSensor (#[3674](tel:3674)) > > >>>>>>>>>> - [AIRFLOW-[4104](tel:4104)] Add type annotations to common > > >>>> classes. (#[4926](tel:4926)) > > >>>>>>>>>> - [AIRFLOW-[3910](tel:3910)] Raise exception explicitly in > > >>>> Connection.get_hook() > > >>>>>>>>>> (#[4728](tel:4728)) > > >>>>>>>>>> - [AIRFLOW-[3322](tel:3322)] Update QuboleHook to fetch args > > >>>> dynamically from > > >>>>>>>>> qds_sdk > > >>>>>>>>>> (#[4165](tel:4165)) > > >>>>>>>>>> - [AIRFLOW-[4565](tel:4565)] instrument celery executor > > >>>> (#[5321](tel:5321)) > > >>>>>>>>>> - [AIRFLOW-[4573](tel:4573)] Import airflow_local_settings > after > > >>>>>>>>> prepare_classpath > > >>>>>>>>>> (#[5330](tel:5330)) > > >>>>>>>>>> - [AIRFLOW-[4448](tel:4448)] Don't bake ENV and _cmd into tmp > > >>>> config for non-sudo > > >>>>>>>>>> (#[4050](tel:4050)) > > >>>>>>>>>> - [AIRFLOW-[4295](tel:4295)] Make ``method`` attribute case > > >>>> insensitive in > > >>>>>>>>> HttpHook > > >>>>>>>>>> (#[5313](tel:5313)) > > >>>>>>>>>> - [AIRFLOW-[3703](tel:3703)] Add dnsPolicy option for > > >>>> KubernetesPodOperator > > >>>>>>>>> (#[4520](tel:4520)) > > >>>>>>>>>> - [AIRFLOW-[3057](tel:3057)] add prev_*_date_success to > template > > >>>> context (#[5372](tel:5372)) > > >>>>>>>>>> - [AIRFLOW-[4336](tel:4336)] Stop showing entire GCS files > bytes > > >> in > > >>>> log for > > >>>>>>>>>> gcs_download_operator (#[5151](tel:5151)) > > >>>>>>>>>> - [AIRFLOW-[4528](tel:4528)] Cancel DataProc task on timeout > > >>>> (#[5293](tel:5293)) > > >>>>>>>>>> > > >>>>>>>>>> Bug fixes > > >>>>>>>>>> """"""""" > > >>>>>>>>>> - [AIRFLOW-[4934](tel:4934)] Fix ProxyFix due to Werkzeug > > upgrade > > >>>> (#[5563](tel:5563)) (#[5571](tel:5571)) > > >>>>>>>>>> - [AIRFLOW-[4136](tel:4136)] fix key_file of hook is > overwritten > > >> by > > >>>> SSHHook > > >>>>>>>>> connection > > >>>>>>>>>> (#[5558](tel:5558)) > > >>>>>>>>>> - [AIRFLOW-[4587](tel:4587)] Replace self.conn with > > >> self.get_conn() > > >>>> in > > >>>>>>>>> AWSAthenaHook > > >>>>>>>>>> (#[5545](tel:5545)) > > >>>>>>>>>> - [AIRFLOW-[1740](tel:1740)] Fix xcom creation and update via > UI > > >>>> (#[5530](tel:5530)) (#[5531](tel:5531)) > > >>>>>>>>>> - [AIRFLOW-[4900](tel:4900)] Resolve incompatible version of > > >>>> Werkzeug (#[5535](tel:5535)) > > >>>>>>>>>> - [AIRFLOW-[4510](tel:4510)] Don't mutate default_args during > > DAG > > >>>> initialization > > >>>>>>>>>> (#[5277](tel:5277)) > > >>>>>>>>>> - [AIRFLOW-[3360](tel:3360)] Make the DAGs search respect > other > > >>>> querystring > > >>>>>>>>> parameters > > >>>>>>>>>> with url-search-params-polyfill for IE support > > (#[5503](tel:5503)) > > >>>>>>>>>> - [AIRFLOW-[4896](tel:4896)] Make KubernetesExecutorConfig's > > >>>> default args > > >>>>>>>>> immutable > > >>>>>>>>>> (#[5534](tel:5534)) > > >>>>>>>>>> - [AIRFLOW-[4494](tel:4494)] Remove ``shell=True`` in > > DaskExecutor > > >>>> (#[5273](tel:5273)) > > >>>>>>>>>> - [AIRFLOW-[4890](tel:4890)] Fix Log link in TaskInstance's > View > > >>>> for Non-RBAC > > >>>>>>>>> (#[5525](tel:5525)) > > >>>>>>>>>> - [AIRFLOW-[4892](tel:4892)] Fix connection creation via UIs > > >>>> (#[5527](tel:5527)) > > >>>>>>>>>> - [AIRFLOW-[4406](tel:4406)] Fix a method name typo: > > >>>> NullFernet.decrpyt to > > >>>>>>>>> decrypt > > >>>>>>>>>> (#[5509](tel:5509)) > > >>>>>>>>>> - [AIRFLOW-[4849](tel:4849)] Add gcp_conn_id to > > cloudsqldatabehook > > >>>> class to use > > >>>>>>>>>> correctly CloudSqlProxyRunner class (#[5478](tel:5478)) > > >>>>>>>>>> - [AIRFLOW-[4769](tel:4769)] Pass gcp_conn_id to BigtableHook > > >>>> (#[5445](tel:5445)) > > >>>>>>>>>> - [AIRFLOW-[4524](tel:4524)] Fix incorrect field names in view > > for > > >>>> Mark > > >>>>>>>>>> Success/Failure (#[5486](tel:5486)) > > >>>>>>>>>> - [AIRFLOW-[3671](tel:3671)] Remove arg ``replace`` of > > >>>> MongoToS3Operator from > > >>>>>>>>>> ``kwargs`` (#[4480](tel:4480)) > > >>>>>>>>>> - [AIRFLOW-[4845](tel:4845)] Fix bug where runAsUser 0 doesn't > > get > > >>>> set in k8s > > >>>>>>>>> security > > >>>>>>>>>> context (#[5474](tel:5474)) > > >>>>>>>>>> - [AIRFLOW-[4354](tel:4354)] Fix exception in "between" date > > >> filter > > >>>> in classic UI > > >>>>>>>>>> (#[5480](tel:5480)) > > >>>>>>>>>> - [AIRFLOW-[4587](tel:4587)] Replace self.conn with > > >> self.get_conn() > > >>>> in > > >>>>>>>>> AWSAthenaHook > > >>>>>>>>>> (#[5462](tel:5462)) > > >>>>>>>>>> - [AIRFLOW-[4516](tel:4516)] K8s runAsUser and fsGroup cannot > be > > >>>> strings (#[5429](tel:5429)) > > >>>>>>>>>> - [AIRFLOW-[4298](tel:4298)] Stop Scheduler repeatedly warning > > >>>> "connection > > >>>>>>>>>> invalidated" (#[5470](tel:5470)) > > >>>>>>>>>> - [AIRFLOW-[4559](tel:4559)] JenkinsJobTriggerOperator bugfix > > >>>> (#[5318](tel:5318)) > > >>>>>>>>>> - [AIRFLOW-[4841](tel:4841)] Pin Sphinx AutoApi to > > >> [1.0.0](tel:100) > > >>>> (#[5468](tel:5468)) > > >>>>>>>>>> - [AIRFLOW-[4479](tel:4479)] Include s3_overwrite kwarg in > > >>>> load_bytes method > > >>>>>>>>> (#[5312](tel:5312)) > > >>>>>>>>>> - [AIRFLOW-[3746](tel:3746)] Fix DockerOperator missing > > container > > >>>> exit (#[4583](tel:4583)) > > >>>>>>>>>> - [AIRFLOW-[4233](tel:4233)] Remove Template Extension from Bq > > to > > >>>> GCS Operator > > >>>>>>>>> (#[5456](tel:5456)) > > >>>>>>>>>> - > > >>>> > > >> > > > [AIRFLOW-[2141](tel:2141)][AIRFLOW-[3157](tel:3157)][AIRFLOW-[4170](tel:4170)] > > >>>> Serialize non-str value > > >>>>>>>>> by > > >>>>>>>>>> JSON when importing Variables (#[4991](tel:4991)) > > >>>>>>>>>> - [AIRFLOW-[4826](tel:4826)] Remove warning from ``airflow > > >>>> resetdb`` command > > >>>>>>>>> (#[5447](tel:5447)) > > >>>>>>>>>> - [AIRFLOW-[4148](tel:4148)] Fix editing DagRuns when clicking > > >>>> state column > > >>>>>>>>> (#[5436](tel:5436)) > > >>>>>>>>>> - [AIRFLOW-[4455](tel:4455)] dag_details broken for subdags in > > >> RBAC > > >>>> UI (#[5234](tel:5234)) > > >>>>>>>>>> - [AIRFLOW-[2955](tel:2955)] Fix kubernetes pod operator to > set > > >>>> requests and > > >>>>>>>>> limits on > > >>>>>>>>>> task pods (#[4551](tel:4551)) > > >>>>>>>>>> - [AIRFLOW-[4459](tel:4459)] Fix wrong DAG count in /home page > > >> when > > >>>> DAG count is > > >>>>>>>>> zero > > >>>>>>>>>> (#[5235](tel:5235)) > > >>>>>>>>>> - [AIRFLOW-[3876](tel:3876)] AttributeError: module > 'distutils' > > >> has > > >>>> no attribute > > >>>>>>>>> 'util' > > >>>>>>>>>> - [AIRFLOW-[4146](tel:4146)] Fix CgroupTaskRunner errors > > >>>> (#[5224](tel:5224)) > > >>>>>>>>>> - [AIRFLOW-[4524](tel:4524)] Fix bug with "Ignore \*" toggles > in > > >>>> RBAC mode > > >>>>>>>>> (#[5378](tel:5378)) > > >>>>>>>>>> - [AIRFLOW-[4765](tel:4765)] Fix DataProcPigOperator execute > > >> method > > >>>> (#[5426](tel:5426)) > > >>>>>>>>>> - [AIRFLOW-[4798](tel:4798)] obviate interdependencies for > > dagbag > > >>>> and TI tests > > >>>>>>>>> (#[5422](tel:5422)) > > >>>>>>>>>> - [AIRFLOW-[4800](tel:4800)] fix GKEClusterHook ctor calls > > >>>> (#[5424](tel:5424)) > > >>>>>>>>>> - [AIRFLOW-[4799](tel:4799)] don't mutate self.env in > > BashOperator > > >>>> execute method > > >>>>>>>>>> (#[5421](tel:5421)) > > >>>>>>>>>> - [AIRFLOW-[4393](tel:4393)] Add retry logic when fetching pod > > >>>> status and/or logs > > >>>>>>>>> in > > >>>>>>>>>> KubernetesPodOperator (#[5284](tel:5284)) > > >>>>>>>>>> - [AIRFLOW-[4174](tel:4174)] Fix HttpHook run with backoff > > >>>> (#[5213](tel:5213)) > > >>>>>>>>>> - [AIRFLOW-[4463](tel:4463)] Handle divide-by-zero errors in > > short > > >>>> retry > > >>>>>>>>> intervals > > >>>>>>>>>> (#[5243](tel:5243)) > > >>>>>>>>>> - [AIRFLOW-[2614](tel:2614)] Speed up trigger_dag API call > when > > >>>> lots of DAGs in > > >>>>>>>>> system > > >>>>>>>>>> - [AIRFLOW-[4756](tel:4756)] add ti.state to ti.start_date as > > >>>> criteria for gantt > > >>>>>>>>>> (#[5399](tel:5399)) > > >>>>>>>>>> - [AIRFLOW-[4760](tel:4760)] Fix zip-packaged DAGs > disappearing > > >>>> from DagBag when > > >>>>>>>>>> reloaded (#[5404](tel:5404)) > > >>>>>>>>>> - [AIRFLOW-[4731](tel:4731)] Fix GCS hook with > > >>>> google-storage-client [1.16](tel:116) (#[5368](tel:5368)) > > >>>>>>>>>> - [AIRFLOW-[3506](tel:3506)] use match_phrase to query log_id > in > > >>>> elasticsearch > > >>>>>>>>> (#[4342](tel:4342)) > > >>>>>>>>>> - [AIRFLOW-[4084](tel:4084)] fix ElasticSearch log download > > >>>> (#[5177](tel:5177)) > > >>>>>>>>>> - [AIRFLOW-[4501](tel:4501)] Register pendulum datetime > > converter > > >>>> for > > >>>>>>>>> sqla+pymysql > > >>>>>>>>>> (#[5190](tel:5190)) > > >>>>>>>>>> - [AIRFLOW-[986](tel:986)] HiveCliHook ignores 'proxy_user' > > value > > >>>> in a > > >>>>>>>>> connection's > > >>>>>>>>>> extra parameter (#[5305](tel:5305)) > > >>>>>>>>>> - [AIRFLOW-[4442](tel:4442)] fix hive_tblproperties in > > >>>> HiveToDruidTransfer > > >>>>>>>>> (#[5211](tel:5211)) > > >>>>>>>>>> - [AIRFLOW-[4557](tel:4557)] Add gcp_conn_id parameter to > > >>>> get_sqlproxy_runner() > > >>>>>>>>> of > > >>>>>>>>>> CloudSqlDatabaseHook (#[5314](tel:5314)) > > >>>>>>>>>> - [AIRFLOW-[4545](tel:4545)] Upgrade FAB to latest version > > >>>> (#[4955](tel:4955)) > > >>>>>>>>>> - [AIRFLOW-[4492](tel:4492)] Change Dataproc Cluster operators > > to > > >>>> poll Operations > > >>>>>>>>>> (#[5269](tel:5269)) > > >>>>>>>>>> - [AIRFLOW-[4452](tel:4452)] Webserver and Scheduler keep > > crashing > > >>>> because of > > >>>>>>>>>> slackclient update (#[5225](tel:5225)) > > >>>>>>>>>> - [AIRFLOW-[4450](tel:4450)] Fix request arguments in > > >>>> has_dag_access (#[5220](tel:5220)) > > >>>>>>>>>> - [AIRFLOW-[4434](tel:4434)] Support Impala with the > > >>>> HiveServer2Hook (#[5206](tel:5206)) > > >>>>>>>>>> - [AIRFLOW-[3449](tel:3449)] Write local dag parsing logs when > > >>>> remote logging > > >>>>>>>>> enabled. > > >>>>>>>>>> (#[5175](tel:5175)) > > >>>>>>>>>> - [AIRFLOW-[4300](tel:4300)] Fix graph modal call when DAG has > > not > > >>>> yet run > > >>>>>>>>> (#[5185](tel:5185)) > > >>>>>>>>>> - [AIRFLOW-[4401](tel:4401)] Use managers for Queue > > >> synchronization > > >>>> (#[5200](tel:5200)) > > >>>>>>>>>> - [AIRFLOW-[3626](tel:3626)] Fixed triggering DAGs contained > > >> within > > >>>> zip files > > >>>>>>>>> (#[4439](tel:4439)) > > >>>>>>>>>> - [AIRFLOW-[3720](tel:3720)] Fix missmatch while comparing GCS > > and > > >>>> S3 files > > >>>>>>>>> (#[4766](tel:4766)) > > >>>>>>>>>> - [AIRFLOW-[4403](tel:4403)] search by ``dag_id`` or > ``owners`` > > in > > >>>> UI (#[5184](tel:5184)) > > >>>>>>>>>> - [AIRFLOW-[4308](tel:4308)] Fix TZ-loop around DST on python > > 3.6+ > > >>>> (#[5095](tel:5095)) > > >>>>>>>>>> - [AIRFLOW-[4324](tel:4324)] fix DAG fuzzy search in RBAC UI > > >>>> (#[5131](tel:5131)) > > >>>>>>>>>> - [AIRFLOW-[4297](tel:4297)] Temporary hot fix on > manage_slas() > > >> for > > >>>> [1.10.4](tel:1104) > > >>>>>>>>> release > > >>>>>>>>>> (#[5150](tel:5150)) > > >>>>>>>>>> - [AIRFLOW-[4299](tel:4299)] Upgrade to Celery > [4.3.0](tel:430) > > to > > >>>> fix crashing workers > > >>>>>>>>> (#[5116](tel:5116)) > > >>>>>>>>>> - [AIRFLOW-[4291](tel:4291)] Correctly render doc_md in DAG > > graph > > >>>> page (#[5121](tel:5121)) > > >>>>>>>>>> - [AIRFLOW-[4310](tel:4310)] Fix incorrect link on Dag Details > > >> page > > >>>> (#[5122](tel:5122)) > > >>>>>>>>>> - [AIRFLOW-[4331](tel:4331)] Correct filter for Null-state > runs > > >>>> from Dag Detail > > >>>>>>>>> page > > >>>>>>>>>> (#[5123](tel:5123)) > > >>>>>>>>>> - [AIRFLOW-[4294](tel:4294)] Fix missing dag & task runs in UI > > >>>> dag_id contains a > > >>>>>>>>> dot > > >>>>>>>>>> (#[5111](tel:5111)) > > >>>>>>>>>> - [AIRFLOW-[4332](tel:4332)] Upgrade sqlalchemy to remove > > security > > >>>> Vulnerability > > >>>>>>>>>> (#[5113](tel:5113)) > > >>>>>>>>>> - [AIRFLOW-[4312](tel:4312)] Add template_fields & > template_ext > > to > > >>>>>>>>> BigQueryCheckO… > > >>>>>>>>>> (#[5097](tel:5097)) > > >>>>>>>>>> - [AIRFLOW-[4293](tel:4293)] Fix downgrade in > > >>>>>>>>>> d4ecb8fbee3_add_schedule_interval_to_dag.py > (#[5086](tel:5086)) > > >>>>>>>>>> - [AIRFLOW-[4267](tel:4267)] Fix TI duration in Graph View > > >>>> (#[5071](tel:5071)) > > >>>>>>>>>> - [AIRFLOW-[4163](tel:4163)] IntervalCheckOperator supports > > >>>> relative diff and not > > >>>>>>>>>> ignore 0 (#[4983](tel:4983)) > > >>>>>>>>>> - [AIRFLOW-[3938](tel:3938)] QuboleOperator Fixes and Support > > for > > >>>> SqlCommand > > >>>>>>>>> (#[4832](tel:4832)) > > >>>>>>>>>> - [AIRFLOW-[2903](tel:2903)] Change default owner to > ``airflow`` > > >>>> (#[4151](tel:4151)) > > >>>>>>>>>> - [aIRFLOW-[4136](tel:4136)] Fix overwrite of key_file by > > >>>> constructor (#[5155](tel:5155)) > > >>>>>>>>>> - [AIRFLOW-[3241](tel:3241)] Remove Invalid template ext in > GCS > > >>>> Sensors (#[4076](tel:4076)) > > >>>>>>>>>> > > >>>>>>>>>> Misc/Internal > > >>>>>>>>>> """"""""""""" > > >>>>>>>>>> - [AIRFLOW-[4937](tel:4937)] Fix lodash security issue with > > >> version > > >>>> below [4.17.13](tel:41713) > > >>>>>>>>>> (#[5572](tel:5572)) (used only in build-pipeline, not runtime) > > >>>>>>>>>> - [AIRFLOW-[4868](tel:4868)] Fix typo in > > >> kubernetes/docker/build.sh > > >>>> (#[5505](tel:5505)) > > >>>>>>>>>> - [AIRFLOW-[4211](tel:4211)] Add tests for WebHDFSHook > > >>>> (#[5015](tel:5015)) > > >>>>>>>>>> - [AIRFLOW-[4320](tel:4320)] Add tests for > > >>>> SegmentTrackEventOperator (#[5104](tel:5104)) > > >>>>>>>>>> - [AIRFLOW-[4319](tel:4319)] Add tests for Bigquery related > > >>>> Operators (#[5101](tel:5101)) > > >>>>>>>>>> - [AIRFLOW-[4014](tel:4014)] Change DatastoreHook and add > tests > > >>>> (#[4842](tel:4842)) > > >>>>>>>>>> - [AIRFLOW-[4322](tel:4322)] Add test for VerticaOperator > > >>>> (#[5107](tel:5107)) > > >>>>>>>>>> - [AIRFLOW-[4323](tel:4323)] Add 2 tests for WinRMOperator > > >>>> (#[5108](tel:5108)) > > >>>>>>>>>> - [AIRFLOW-[3677](tel:3677)] Improve CheckOperator test > coverage > > >>>> (#[4756](tel:4756)) > > >>>>>>>>>> - [AIRFLOW-[4659](tel:4659)] Fix pylint problems for api > module > > >>>> (#[5398](tel:5398)) > > >>>>>>>>>> - [AIRFLOW-[4358](tel:4358)] Speed up test_jobs by not running > > >>>> tasks (#[5162](tel:5162)) > > >>>>>>>>>> - [AIRFLOW-[4394](tel:4394)] Don't test behaviour of > BackfillJob > > >>>> from CLI tests > > >>>>>>>>> (#[5160](tel:5160)) > > >>>>>>>>>> - [AIRFLOW-[3471](tel:3471)] Move XCom out of models.py > > >>>> (#[4629](tel:4629)) > > >>>>>>>>>> - [AIRFLOW-[4379](tel:4379)] Remove duplicate code & Add > > >> validation > > >>>> in > > >>>>>>>>> gcs_to_gcs.py > > >>>>>>>>>> (#[5145](tel:5145)) > > >>>>>>>>>> - [AIRFLOW-[4259](tel:4259)] Move models out of models.py > > >>>> (#[5056](tel:5056)) > > >>>>>>>>>> - [AIRFLOW-XXX] Speed up building of Cassanda module on Travis > > >>>>>>>>> (#[5233](tel:5233)) > > >>>>>>>>>> - [AIRFLOW-[4535](tel:4535)] Break jobs.py into multiple files > > >>>> (#[5303](tel:5303)) > > >>>>>>>>>> - [AIRFLOW-[1464](tel:1464)] Batch update task_instance state > > >>>> (#[5323](tel:5323)) > > >>>>>>>>>> - [AIRFLOW-[4554](tel:4554)] Test for sudo command, add some > > other > > >>>> test docs > > >>>>>>>>> (#[5310](tel:5310)) > > >>>>>>>>>> - [AIRFLOW-[4419](tel:4419)] Refine concurrency check in > > scheduler > > >>>> (#[5194](tel:5194)) > > >>>>>>>>>> - [AIRFLOW-[4269](tel:4269)] Minor acceleration of > > >>>> jobs._process_task_instances() > > >>>>>>>>>> (#[5076](tel:5076)) > > >>>>>>>>>> - [AIRFLOW-[4341](tel:4341)] Remove ``View.render()`` already > > >>>> exists in > > >>>>>>>>> fab.BaseView > > >>>>>>>>>> (#[5125](tel:5125)) > > >>>>>>>>>> - [AIRFLOW-[4342](tel:4342)] Use @cached_property instead of > > >>>> re-implementing it > > >>>>>>>>> each > > >>>>>>>>>> time (#[5126](tel:5126)) > > >>>>>>>>>> - [AIRFLOW-[4256](tel:4256)] Remove noqa from migrations > > >>>> (#[5055](tel:5055)) > > >>>>>>>>>> - [AIRFLOW-[4034](tel:4034)] Remove unnecessary string > > formatting > > >>>> with > > >>>>>>>>> ``**locals()`` > > >>>>>>>>>> (#[4861](tel:4861)) > > >>>>>>>>>> - [AIRFLOW-[3944](tel:3944)] Remove code smells > > >> (#[4762](tel:4762)) > > >>>>>>>>>> > > >>>>>>>>>> Doc-only changes > > >>>>>>>>>> """""""""""""""" > > >>>>>>>>>> - [AIRFLOW-XXX] Correct BaseSensorOperator docs > > >> (#[5562](tel:5562)) > > >>>>>>>>>> - [AIRFLOW-[4926](tel:4926)] Fix example dags where its > > start_date > > >>>> is > > >>>>>>>>>> datetime.utcnow() (#[5553](tel:5553)) > > >>>>>>>>>> - [AIRFLOW-[4860](tel:4860)] Remove Redundant Information in > > >>>> Example Dags (#[5497](tel:5497)) > > >>>>>>>>>> - [AIRFLOW-[4767](tel:4767)] Fix errors in the documentation > of > > >>>> Dataproc Operator > > >>>>>>>>>> (#[5487](tel:5487)) > > >>>>>>>>>> - [AIRFLOW-[1684](tel:1684)] Branching based on XCom variable > > >>>> (Docs) (#[4365](tel:4365)) > > >>>>>>>>>> - [AIRFLOW-[3341](tel:3341)] FAQ return DAG object example > > >>>> (#[4605](tel:4605)) > > >>>>>>>>>> - [AIRFLOW-[4433](tel:4433)] Add missing type in > DockerOperator > > >> doc > > >>>> string > > >>>>>>>>> (#[5205](tel:5205)) > > >>>>>>>>>> - [AIRFLOW-[4321](tel:4321)] Replace incorrect info of Max > Size > > >>>> limit of GCS > > >>>>>>>>> Object > > >>>>>>>>>> Size (#[5106](tel:5106)) > > >>>>>>>>>> - [AIRFLOW-XXX] Add information about user list > > >> (#[5341](tel:5341)) > > >>>>>>>>>> - [AIRFLOW-XXX] Clarify documentation related to autodetect > > >>>> parameter > > >>>>>>>>> in > > >>>>>>>>>> GCS_to_BQ Op (#[5294](tel:5294)) > > >>>>>>>>>> - [AIRFLOW-XXX] Remove mention of pytz compatibility from > > timezone > > >>>>>>>>>> documentation (#[5316](tel:5316)) > > >>>>>>>>>> - [AIRFLOW-XXX] Add missing docs for > > >>>> GoogleCloudStorageDeleteOperator > > >>>>>>>>>> (#[5274](tel:5274)) > > >>>>>>>>>> - [AIRFLOW-XXX] Remove incorrect note about Scopes of GCP > > >> connection > > >>>>>>>>>> (#[5242](tel:5242)) > > >>>>>>>>>> - [AIRFLOW-XXX] Fix mistakes in docs of Dataproc operators > > >>>> (#[5192](tel:5192)) > > >>>>>>>>>> - [AIRFLOW-XXX] Link to correct class for timedelta in > > macros.rst > > >>>>>>>>> (#[5226](tel:5226)) > > >>>>>>>>>> - [AIRFLOW-XXX] Add Kamil as committer (#[5216](tel:5216)) > > >>>>>>>>>> - [AIRFLOW-XXX] Add Joshua and Kevin as committer > > >>>> (#[5207](tel:5207)) > > >>>>>>>>>> - [AIRFLOW-XXX] Reduce log spam in tests (#[5174](tel:5174)) > > >>>>>>>>>> - [AIRFLOW-XXX] Speed up tests for PythonSensor > > >> (#[5158](tel:5158)) > > >>>>>>>>>> - [AIRFLOW-XXX] Add Bas Harenslak to committer list > > >>>> (#[5157](tel:5157)) > > >>>>>>>>>> - [AIRFLOW-XXX] Add Jarek Potiuk to commiter list > > >>>> (#[5132](tel:5132)) > > >>>>>>>>>> - [AIRFLOW-XXX] Update docstring for SchedulerJob > > >>>> (#[5105](tel:5105)) > > >>>>>>>>>> - [AIRFLOW-XXX] Fix docstrings for > > >>>>>>>>> CassandraToGoogleCloudStorageOperator > > >>>>>>>>>> (#[5103](tel:5103)) > > >>>>>>>>>> - [AIRFLOW-XXX] update SlackWebhookHook and > SlackWebhookOperator > > >>>>>>>>> docstring > > >>>>>>>>>> (#[5074](tel:5074)) > > >>>>>>>>>> - [AIRFLOW-XXX] Ignore python files under node_modules in docs > > >>>>>>>>> (#[5063](tel:5063)) > > >>>>>>>>>> - [AIRFLOW-XXX] Build a universal wheel with LICNESE files > > >>>> (#[5052](tel:5052)) > > >>>>>>>>>> - [AIRFLOW-XXX] Fix docstrings of SQSHook (#[5099](tel:5099)) > > >>>>>>>>>> - [AIRFLOW-XXX] Use Py3.7 on readthedocs > > >>>>>>>>>> - [AIRFLOW-[4446](tel:4446)] Fix typos (#[5217](tel:5217)) > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>> > > >>>>>> > > >>>>>> > > >>>>>> -- > > >>>>>> Best regards, > > >>>>>> > > >>>>>> Andrii Soldatenko > > >>>>>> Python Developer > > >>>>>> > > >>>>>> skype: andrii.soldatenko > > >>>>>> [email protected] > > >>>>>> http://asoldatenko.com > > >>> > > >>> > > >>> > > >>> -- > > >>> *Kaxil Naik* > > >>> *Big Data Consultant | DevOps Data Engineer* > > >>> *Certified *Google Cloud Data Engineer | *Certified* Apache Spark & > > Neo4j > > >>> Developer > > >>> *LinkedIn*: https://www.linkedin.com/in/kaxil > > >> > > >> > > > > > -- *Kaxil Naik* *Big Data Consultant | DevOps Data Engineer* *Certified *Google Cloud Data Engineer | *Certified* Apache Spark & Neo4j Developer *LinkedIn*: https://www.linkedin.com/in/kaxil
