Re: execution_date - can we stop the confusion?

What about (aliasing) execution_date to period_start, and next_execution_date to period_end? Would this help any do we think? (Though things like ds and ts might still be confusing? This is probably where the OP got the idea for run_stamped from? One step at a time.) Ash On 27 September 2018

Re: 1.10.1 Release?

hange over time. This should unblock the scheduler. Ash On 30 October 2018 18:43:30 GMT, David Klosowski wrote: >Hi Airflow Devs: > >Is this timezone issue in Airflow version 1.10.0 or only in 1.10.1? > >Thanks. > >Regards, >David > >On Tue, Oct 30, 2018 at 11:11

Re: Apache Airflow Setup Issue

The error is related to python 3.7. `pip install apache-airflow==1.10.1rc2` should fix this. (The release candidate fixes this, so make sure you ask for that explicit version until the release is made) Note the new name - apache-airflow is the new package name. On 17 November 2018 13:03:10 GMT,

Re: [VOTE] Graduate the Apache Airflow as a TLP

;responsibility of the Apache Airflow Project; and be it further > >RESOLVED, that the persons listed immediately below be and >hereby are appointed to serve as the initial members of the >Apache Airflow Project: > >* Alex Guziel >* Alex Van Boxel >* Arthur Wiedmer >* Ash Be

Re: Good practices around restarting scheduler and server of airflow

Is there any indication of which Celery version this is fixed in? I've just started using Airflow and upon setting a start date for a task to the start of the year I've got 7 tasks which are stuck in queued and don't seem to be anywhere. (I suspect if I restarted airflow worker process on the on

Re: [VOTE] Release Airflow 1.8.2 based on Airflow 1.8.2 RC3

pip can be very picky at times the version number of things it installs, specifically when installing from wheels (which are pre-packaged, and recommend because they are much faster to install.) A distribution version number should follow PEP-0440 https://www.python.org/dev/peps/pep-0440/

Re: [webserver] Webserver time_starttransfer very high

rm-up. -ash > On 8 Aug 2017, at 18:52, Victor Monteiro wrote: > > Hi everyone. > > The problem is very straightforward. When doing a request to airflow > webserver, it is taking too much time to send the first byte. > > > > As you can see in the picture, it to

Re: Operators that poll vs Sensors

l) logging.info("Success criteria met. Exiting.") i.e. there's not much difference in effect from an operator that loops and sleeps itself to one which is a Sensor. -ash > On 5 Sep 2017, at 16:14, Richard Baron Penman wrote: > > Hello, > > I noticed some operators

Re: Upgrading to 1.8.2 fails to display variable page

Was it included as an image? If so it was stripped (and would be more useful included as text anyway) -ash > On 7 Sep 2017, at 17:12, David Capwell wrote: > > I just upgraded a test environment from 1.8.0 to 1.8.2 and notice that the > variables page is no longer able to lo

Re: 1.9.0

clude entries for 1.8.2. -ash > On 7 Sep 2017, at 17:28, Andrew Chen wrote: > > Any estimate when branch 1.9 will be cut? > > On Thu, Sep 7, 2017 at 9:25 AM, Maxime Beauchemin < > maximebeauche...@gmail.com> wrote: > >> Go for it! Still trying to wrap up 1.8.2 ...

Re: Terminate task process through UI

lob/b2e1753f5b74ad1b6e0889f7b784ce69623c95ce/airflow/jobs.py#L164-L165> which triggers kill, so it is meant to work. -ash > On 7 Sep 2017, at 20:00, Richard Baron Penman > wrote: > > Is this known to work? > I tried clearing and the on_kill() method wasn't triggered. > > As far as I can tell

Re: 1.9.0 PR mini-rundown

emantic meaning but seems to be the last instance of this string that isn't using the variable. I hope this was helpful. -ash > On 7 Sep 2017, at 20:26, Chris Riccomini wrote: > > Double negative for the win. Rephrase: Highly unlikely K8s will make 1.9.0. > > On Thu,

Re: 1.9.0 test branch has been cut

There are 14 (only 2 of which are open) issues that are marked with a fix version of 1.8.3. What should happen to those? > On 13 Sep 2017, at 20:45, Chris Riccomini wrote: > > Hey all, > > I've cut a 1.9.0 test branch. > > https://github.com/apache/incubator-airflow/tree/v1-9-test > > Here

Re: 1.9.0 test branch has been cut

Unless I'm missing something I don't think I have permission - at least I'm not seeing any Edit links or menu items, perhaps because I didn't report the issues? -ash > On 13 Sep 2017, at 20:52, Chris Riccomini wrote: > > Go ahead and add 1.9.0 as well (we c

Re: Proposal: Set Celery 4.0 as a minimum as Celery 4 is unsupported

+1 from us, we're running on Celery 4.0.2 in production on Airflow 1.8.2 (4.1 wasn't out when we started and haven't upgraded in prod yet) > On 16 Sep 2017, at 16:35, Bolke de Bruin wrote: > > Hi, > > Some refactoring of the Celery config is underway and as some of the options > have changed

Re: Using s3 logging in Airflow 1.9.x

It could be that you have created a custom logging file, but you haven't specified it in your airflow.cfg: ``` logging_config_class=mymodule.LOGGING_CONFIG ``` > On 7 Oct 2017, at 00:11, David Klosowski wrote: > > Hey Airflow Devs: > > How is s3 logging supposed to work in Airflow 1.9.0? >

Re: programatically configure aws_default, emr_default connections?

ncubator-airflow/blob/15feb7dd3f39ba7926ae5817d488e4e54a3d7742/airflow/models.py#L577-L590 <https://github.com/apache/incubator-airflow/blob/15feb7dd3f39ba7926ae5817d488e4e54a3d7742/airflow/models.py#L577-L590> and it has some problems trying to parse uri schemas for things like `google_cloud_platform://` But it might help you

Re: too many long running tasks in sensors

This isn't possible with Sensors as they exist right now. Our solution was to not make it a sensor, but just a normal operator, and set a number of retries and a large retry_delay: checkTask = FindTriggerFileForExecutionPeriod( task_id="trigger_file_for_period", s3_conn_id="s3_default",

Publishing alpha releases to PyPI?

st.apache.org/repos/dist/dev/incubator/airflow/1.9.0alpha1/, and upload it to PyPi. - Update release process to publish all alpha/beta/rc to PyPi in the future. I'd be happy to update https://cwiki.apache.org/confluence/display/AIRFLOW/Releasing+Airflow if we start doing this (but that doesn't currently mention how uploads to pypi happen at all...) Thanks, Ash [1]: https://pip.pypa.io/en/stable/reference/pip_install/#pre-release-versions ---

Two small DAG chart ui fixes

s, or hours? I think both of these were regressions from the switch from Highline to NVD3 charts (around 1.8?). I'd like them in 1.9, but we can always carry a patch for these internally until 1.10 is released :) -ash

Re: ImportError: No module named sendgrid

hich I think is > against the general understanding. > 3. Why sendgrid got special privileges, and why not 100s of other mail > services? The Same concern was raised by Ash in JIRA as well. > 4. Why it was not coded as a plugin. > 5. Why there is a hard dependency to install Sendgrid module

Re: ImportError: No module named sendgrid

d why not 100s of other mail > services? The Same concern was raised by Ash in JIRA as well. > 4. Why it was not coded as a plugin. > 5. Why there is a hard dependency to install Sendgrid module if I am not > using it. > > I think this commit needs to be reverted and a new PR should b

Re: Experimental API

ng default config should not leave the API unprotected. I think the default API auth backend should deny all rather than allow all? -ash > On 30 Oct 2017, at 08:51, Niels Zeilemaker > wrote: > > Hi All, > > I've implemented HTTP Basic Authentication for the experiment API,

Re: Experimental API

to be loaded via JSON? Couldn't that be info be sent on initial page load without needing an extra page load? > On 30 Oct 2017, at 15:44, Andy Hadjigeorgiou wrote: > > Is this experimental API available by default, or does it need a > configuration? > > On Mon, Oct

Re: Experimental API

Good work! Oh, /dag_stats and /task_stats were un-authenticated before too. Not disasterous but not great. -ash > On 30 Oct 2017, at 16:01, Niels Zeilemaker wrote: > > Hi Ash, > > I made a pull request moving the latest runs call to the web api. > > https://github

Re: Airflow 1.9.0 status

I think we need to add https://issues.apache.org/jira/browse/AIRFLOW-1764 https://issues.apache.org/jira/browse/AIRFLOW-1765 to that list as critical security fixes. I'll tackled 1765 tomorrow (UK time) if no one gets round to it before then. -ash > On 30 Oct 2017, at 17:58, Chris R

Re: Docs

Not that it helps fix the "official" site, but http://airflow.readthedocs.io/en/latest/installation.html contains more up-to-date docs. (Though I'm not sure what versions this doc relates to: the "versions" selection in the bottom lef

Re: [VOTE] Airflow 1.9.0rc1

I suspect this might be how we've misconfigured our logging. We will see tomorrow. -ash File "/usr/local/lib/python3.5/dist-packages/airflow/www/views.py", line 712, in log logs = handler.read(ti) AttributeError: 'NoneType' object has no attribute 'read'

Re: Airflow configuration in environment variable not working

x27;s the right stanza) to make it load that too -ash > On 9 Nov 2017, at 08:35, Somasundaram Sekar > wrote: > > Below are the env variables configured > > AIRFLOW__CORE__SQL_ALCHEMY_CONN=mysql://XXX > AIRFLOW__CELERY__BROKER_URL=redis://XX

Re: [VOTE] Airflow 1.9.0rc1

#x27;ve created https://issues.apache.org/jira/browse/AIRFLOW-1795 for that issue. -ash > On 8 Nov 2017, at 18:54, Daniel Huang wrote: > > Still testing this out. > > Put up a small fix for Ash's second exception > https://github.com/apache/incubator-airflow/pull/276

Re: [VOTE] Airflow 1.9.0rc1

ing on the tests for S3Hook which it looks like was kind of lacking. -ash > On 9 Nov 2017, at 10:54, Ash Berlin-Taylor > wrote: > > Thanks for picking this up. Your fix should stop the 500 error, but there's > another problem (which is ultimately user misconfiguration abou

Re: [VOTE] Airflow 1.9.0rc1

to the log task handlers to try loading under the old pattern if none are found with the new style? (The other option is that I just run a migration script to move the old logs into the new place. That sort of only helps me though.) -ash > On 9 Nov 2017, at 11:04, Ash Berlin-Taylor > wr

Re: [VOTE] Airflow 1.9.0rc1

<http://mail-archives.apache.org/mod_mbox/incubator-airflow-dev/201710.mbox/%3cd008c556-c67c-42eb-88c0-cfe440c93...@firemirror.com%3E> To confirm: pip doesn't upgrade to RC/beta unless the user asks for it -- more detail is available in the linked post. -ash > On 9 Nov 2017, at 19:54, Bolke de

Re: [VOTE] Airflow 1.9.0rc1

ifference a lot of time, but it is occasionally useful. -ash > On 9 Nov 2017, at 23:08, Alek Storm wrote: > > It’s not a major difference, but installing from a git repo via pip > requires a completely different syntax, which complicates our tooling, e.g.: > > $ pip instal

Re: Making Airflow Timezone aware

This sounds like an awesome change! I'm happy to review (will take a look tomorrow) but won't be a suitable tester as all our DAGs operate in UTC. -ash > On 13 Nov 2017, at 18:09, Bolke de Bruin wrote: > > Hi All, > > I just want to make you aware that I am c

Re: [VOTE] Airflow 1.9.0rc2

olving that in the ticket, I'd like some feedback on which one people think is the best then I'll make the change. -ash > On 15 Nov 2017, at 13:48, Bolke de Bruin wrote: > > Hi Chris, > > Your changelog seems incomplete (some more fixes were included in RC2). > >

Re: [VOTE] Airflow 1.9.0rc3

-0.5, from me again (Sorry!) We got as far as deploying this to our test cluster and ran into another bug in the S3Hook: `list_keys` doesn't work (and I suspect list_prefixes will be similarly broken, though we don't use it ourselves): File "/usr/local/lib/python3.5/site-packages/airflow/h

Re: [VOTE] Airflow 1.9.0rc4

We're running more tests (day of meetings meant we couldn't do as many as I'd have liked yesterday) but looking good so far. +0.5 (non binding) -ash > On 29 Nov 2017, at 18:31, Chris Riccomini wrote: > > Anyone else? Last day before I close the vote off. > > On

Re: [VOTE] Airflow 1.9.0rc4

tself, neither print(), or `ti.log.info()` Anyone else noticed similar, or is it something specific to our environment? -ash > On 30 Nov 2017, at 09:52, Ash Berlin-Taylor > wrote: > > We're running more tests (day of meetings meant we couldn't do as many as I'd

Re: [VOTE] Airflow 1.9.0rc4

explicitly, so it's log.handlers is an empty list. This in turn means that any log calls end up not writing anything as our FileTaskHandler's self.handler property is None, meaning it doesn't emit anything. Does this ring any bells to anyone? > On 30 Nov 2017, at 15:18, Ash Berlin-Tayl

Re: [VOTE] Airflow 1.9.0rc4

airflow.processor': {'handlers': ['file.processor'], 'level': 'INFO', 'propagate': True}, 'airflow.task': {'handlers': ['file.task

Re: [VOTE] Airflow 1.9.0rc4

ons to fix this: - I can add an explicit handler for .task.raw in my logging config. - We can always call set_context on airflow.task instead of airflow.task.raw - We walk up the .parent links if .propagate is True (possibly stopping once we find any handlers) -ash > On 30 Nov 2017, at 16:02,

Re: [VOTE] Airflow 1.9.0rc4

inside an operator would help me. -ash > On 30 Nov 2017, at 17:03, Chris Riccomini wrote: > > @Ash, do you think this issue is blocking for 1.9.0? > > On Thu, Nov 30, 2017 at 8:22 AM, Ash Berlin-Taylor < > ash_airflowl...@firemirror.com> wrote: > >> Ah, I

Re: [VOTE] Airflow 1.9.0rc4

Yes, that is in my logging config. I think I agree about airflow.task.raw -ash > On 30 Nov 2017, at 18:06, Joy Gao wrote: > > Hi Ash, > > Can you verify that your logging config has root logger set to 'console'? > Something like: > >'

Re: [VOTE] Airflow 1.9.0rc4

that a fix I've got for it works, but I'm not sure it's the best plan long term) -ash > On 30 Nov 2017, at 20:34, Bolke de Bruin wrote: > > Patch here: https://github.com/apache/incubator-airflow/pull/2831 > > Bolke > >> On 30 Nov 2017, at 20:53, Bolke

Re: [VOTE] Airflow 1.9.0rc4

for a better fix to do that instead. -ash > On 30 Nov 2017, at 23:18, Ash Berlin-Taylor wrote: > > Thanks for picking that up so speedily Bolke! > > I've opened a jira ticket for the other problem I found: > https://issues.apache.org/jira/browse/AIRFLOW-1873 > <

Re: Xcom_push is broken - probably after the timezone awareness

encoding like base64. > + By default pickling is still enabled until Airflow 2.0. To disable it > + Set enable_xcom_pickling = False in your Airflow config. Was your test performed against the release candidate or master? I thought the timezone-aware date change was only in the later. -

Re: [VOTE] Airflow 1.9.0rc6

+0.5 from me. Our big test will come on Thursday morning, but looking good so far for the small daily dags we've got are running okay, logs are showing up, and making their way to S3. -ash > On 11 Dec 2017, at 18:50, Chris Riccomini wrote: > > Hey all, > > I have c

Re: [VOTE] Airflow 1.9.0rc6

Sorry, Christmas party season got in the way. Yes +1 from me, I'm happy enough with this release. -ash > On 14 Dec 2017, at 23:57, Chris Riccomini wrote: > > @Ash, it's Thursday. Care to update to +1? :) > > On Thu, Dec 14, 2017 at 10:53 AM, Joy Gao wrote: > >

Re: [RESULT][VOTE] Airflow 1.9.0rc8

Woot! Great news, and thanks for all the time you've spent preparing all the RCs. -ash > On 18 Dec 2017, at 20:13, Chris Riccomini wrote: > > Hello, > > Apache Airflow (incubating) 1.9.0 (based on RC8) has been accepted. > > 5 “+1” binding votes received: > &

Backwards compability - what do we mean? when? how long?

are: 1) Yes to SemVer 2) Strong Yes to Hooks and Operators and anything used directly in DAGs, with a weak yes to including config and other python classes too. My aim here is to start some discussion. If we get to any consensus then after the holidays I'll open a PR to update the docs. Cheers, -ash

Re: How to bes use Google Cloud Storage for logging?

What version are you on? I can't match up the line numbers in this stack trace to either 1.9.0rc8 or 1.9.0rc2 -- both of which show the 'if old_log else log' on line 157 -ash > On 20 Dec 2017, at 15:25, Kevin Lam wrote: > > Thanks Bolke and Feng! > > I see

Re: Fix on_kill command for operators

Without this change does on_kill ever get triggered? It seems like this change is desired behaviour. As per the first comment https://issues.apache.org/jira/browse/AIRFLOW-1623?focusedCommentId=16171819&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16171819

Re: Double Logging to GCS? (And to S3)

(merged in to master) -ash > On 12 Jan 2018, at 16:04, Kevin Lam wrote: > > Hi airflow folks, > > I'm currently using Airflow v1.9 and set up logging to GCS via > http://airflow.readthedocs.io/en/latest/integration.html#gcp. I'm running > into an issue wher

Re: Double Logging to GCS? (And to S3)

Nope, probably not :( -ash > On 13 Jan 2018, at 18:51, Bolke de Bruin wrote: > > We shouldn’t have missed that :-( > >> On 13 Jan 2018, at 19:49, Ash Berlin-Taylor >> wrote: >> >> This is a known bug in 1.9.0 that we didn't manage to fix before re

Re: StackOverflow synonym proposal

Thank you for doing this - been bugging me for a while :) It's been created now too! -ash > On 16 Jan 2018, at 06:58, Baghino, Stefano(AWF) wrote: > > Hi everybody, > > on StackOverflow many people regularly come to get help about Airflow. > Currently on the site ther

Re: Installation of Airflow

what we do) -ash > On 26 Jan 2018, at 08:41, Bolke de Bruin wrote: > > Installation under a different user works fine in a virtualenv. You can also > install system wide and run as normal user (that is how we run it currently). > > Bolke > > Verstuurd vanaf mijn iPad

Re: Airflow Documentation - Readthedocs

"Destroy Documentation" button: > If you would like to DESTROY any existing documentation hosted at > http://pythonhosted.org/airflow Use this button, There is no undo. This should hopefully mean fewer people find the older version of the docs form Google etc. -ash >

Re: max_active_runs

It seems unlikely, but could it be the location of where max_active_runs is specified? In our DAGs we pass it directly as an argument to the DAG() call, not via default_arguments and it behaves itself for us. I think I should check that! -ash > On 14 Feb 2018, at 13:43, Gerard Toons

Re: Scheduler won't schedule past minimum end_date of tasks

lines you linked to. Do we need to add a TIDep https://github.com/apache/incubator-airflow/tree/master/airflow/ti_deps/deps <https://github.com/apache/incubator-airflow/tree/master/airflow/ti_deps/deps> to ensure the exec date is less than the task end date? -ash > On 21 Feb 2018, at

Re: Merging PRs, closing Jira tickets (a.k.a New Committer) guide?

those of you subscribed to the commit list will see) -- I started going through old Jira tickets and closing ones that are no longer an issue or that were fixed+merged but not closed. Github: I don't have a button to close PRs in Github - Is that expected? -ash > On 4 Mar 2018, at 00:

Re: Merging PRs, closing Jira tickets (a.k.a New Committer) guide?

cket asking for this, or is it not possible? -ash > On 6 Mar 2018, at 08:25, Driesprong, Fokko wrote: > > Hi Ash, > > As a committer we don't have any rights on the Github itself. The Github > repo is just a sync of the apache repo. Unfortunately, therefore we

Moving to Github? Re: Merging PRs, closing Jira tickets (a.k.a New Committer) guide?

thread.html/Znkiyqnxqzryecv [2]: http://mail-archives.apache.org/mod_mbox/bookkeeper-dev/201706.mbox/%3CCAO2yDybRq2VUM1JYo_6VT_H8Ca7Lu8af6H-2CZKQzYT6xYGU-g%40mail.gmail.com%3E > On 6 Mar 2018, at 09:57, Ash Berlin-Taylor > wrote: > > Ah that would explain why I don't have a

Re: Moving to Github? Re: Merging PRs, closing Jira tickets (a.k.a New Committer) guide?

ppy with it. > > One downside is around the current release management tooling for Airflow > has bindings with Jira and the "target version" field. > > Max > > On Thu, Mar 8, 2018 at 6:31 AM, Ash Berlin-Taylor < > ash_airflowl...@firemirror.com> wrote: > >

Re: Merging PRs, closing Jira tickets (a.k.a New Committer) guide?

Could someone add my Confluence user (ashb) to the space as well please? I can't seem to edit/create pages right now. Thanks, Ash > On 4 Mar 2018, at 00:59, Sid Anand wrote: > > Hi Ash, > Welcome aboard. > > Firstly, I'm moving this conversation over to the dev l

[VOTE] Migrate to Github as primary repo (a.k.a. Gitbox)

eps can be done now no matter the outcome of the vote -- we just won't get write access to airflow unless we migrate. Ash

[VOTE] [RESULT] Migrate to Github as primary repo (a.k.a. Gitbox)

The vote passed with the following +1's, no -1. 4+ Binding (PPMC) votes Ash Berlin-Taylor, Chris Riccomini, Joy Gao, Maxime Beauchemin 4+ Non-binding (community) votes George Leslie-Waksman, Shah Altaf, Matthew Housley, Beau Barker I have created https://issues.apache.org/jira/browse/AI

Re: Apache Airflow welcome new committer/PMC member : Naik Kaxil (a.k.a. kaxil)

. The vote passed, we just need to do the work summarised in https://issues.apache.org/jira/browse/AIRFLOW-2238 <https://issues.apache.org/jira/browse/AIRFLOW-2238> -ash > On 10 May 2018, at 21:22, Naik Kaxil wrote: > > Hi all, > > My apache account creation has been so

Re: celery problem: cannot override celery_broker_transport_options

To use with the SQLA backend to celery you need to override the options Airflow passes to Celery. Those come from https://github.com/apache/incubator-airflow/blob/v1-10-test/airflow/config_templates/default_celery.py Since you don't want most/all of those options (and there is no way in the con

Re: Moving to Github? Re: Merging PRs, closing Jira tickets (a.k.a New Committer) guide?

p before I get to it I won't be upset :) -ash > On 22 May 2018, at 23:20, Driesprong, Fokko wrote: > > Are we moving to Gitbox? I would really like that. I see a lot of other > project moving to Gitbox at the moment (Flink, Parquet). How's your > schedule Ash? ;) > >

Re: Moving to Github? Re: Merging PRs, closing Jira tickets (a.k.a New Committer) guide?

p before I get to it I won't be upset :) -ash > On 22 May 2018, at 23:20, Driesprong, Fokko wrote: > > Are we moving to Gitbox? I would really like that. I see a lot of other > project moving to Gitbox at the moment (Flink, Parquet). How's your > schedule Ash? ;) > >

Re: celery problem: cannot override celery_broker_transport_options

xtra options. I think the fix is just to move them up to the celery section. -ash > On 24 May 2018, at 07:21, Craig Rodrigues wrote: > > Ash, > > Thanks! You put me on the right track. > Unfortunately, there is a lot of logic in > airflow/config_templates/default_celery

Re: celery problem: cannot override celery_broker_transport_options

// and sqs:// backends. > On 24 May 2018, at 16:13, Craig Rodrigues wrote: > > Ash, > > According to this: > http://docs.celeryproject.org/en/latest/userguide/configuration.html#broker-settings > visibility_timeout is supported by Redis and SQS. > > -- > Craig >

Re: Moving to Github? Re: Merging PRs, closing Jira tickets (a.k.a New Committer) guide?

igrate the repo for us. I'll open a ticket for this early next week. -ash > On 23 May 2018, at 19:04, Sid Anand wrote: > > Welcome back Ash! Firstly, it was great meeting you in person in London in > March, right before you went on Paternity Leave. Hope you are getting some > s

Re: celery problem: cannot override celery_broker_transport_options

equest or release notes which document this. >> >> -- >> Craig >> >> >> On Thu, May 24, 2018 at 8:19 AM Ash Berlin-Taylor < >> ash_airflowl...@firemirror.com> wrote: >> >>> Sounds like >>> https://github.com/apac

Re: conn_id breaking change; once more with feeling

I was involved in the Github discussion about the rename to aws_conn_id, and it prompted me to write http://mail-archives.apache.org/mod_mbox/airflow-dev/201801.mbox/%3cCABYbY7dPS8X6Z4mgbahevQwF5BnYYHXezFo=avoLBNxPzp5=b...@mail.gmail.com%3e

Re: [VOTE] [RESULT] Migrate to Github as primary repo (a.k.a. Gitbox)

I have now requested that the Infra team migrate us over to gitbox/github https://issues.apache.org/jira/browse/INFRA-16602 Once that is completed we should merge https://github.com/apache/incubator-airflow/pull/3413 -ash > On 21 Mar 2018, at 15:59, Chris Riccomini wrote: > > Grea

Re: Airflow London Meetup Group

Hey, I'd be interested in this, and I'm sure I could drag a few colleagues along too :) -ash > On 31 May 2018, at 20:55, Naik Kaxil wrote: > > Hi all, > > We(me and my colleagues at Data Reply UK) are planning to start a meetup for > Airflow users. Me &

Re: Log external module into airflow log

From inside a running task these should already show up in the task instance log. Do they not? Or do you want the logging from parsing the DAG itself? These probably get routed to $AIRFLOW_HOME/logs/scheduler/latest/$dag_id.log (path approximate - from memory) -ash > On 1 Jun 2018, at 19

Re: Is `airflow backfill` disfunctional?

o dig in to it in detail - it's 5pm on a Friday :D Has anyone else noticed anything similar? -ash > On 8 Jun 2018, at 01:22, Tao Feng wrote: > > Thanks everyone for the feedback especially on the background for backfill. > After reading the discussion, I think it would be

Re: Tensorflow Hub - cant seem to load module in Airflow

There's nothing directly in Airflow itself that would cause this kind of issue that I can think of. It depends on what the PythonOperator you are using in the DAG does really. Can you share that code? -ash > On 19 Jun 2018, at 10:01, Andrew Maguire wrote: > > Hi All, > &g

Re: Tensorflow Hub - cant seem to load module in Airflow

that makes sense? You could always try `embed = hub.Module(b"https://tfhub.dev/google/nnlm-en-dim50/1";)` I'm not expecting that to make any difference but doesn't hurt to try. -ash > On 19 Jun 2018, at 10:47, Andrew Maguire wrote: > > sure thing. > >

Re: Tensorflow Hub - cant seem to load module in Airflow

e info in the issue: > > https://github.com/tensorflow/hub/issues/76 > > Seems like maybe Airflow is passing as bytes when it's being expected as a > string..? > > On Tue, Jun 19, 2018 at 11:02 AM Ash Berlin-Taylor < > ash_airflowl...@firemirror.com> wrote: >

Re: Moving to Github? Re: Merging PRs, closing Jira tickets (a.k.a New Committer) guide?

on infra" ¯\_(ツ)_/¯ -ash > On 24 May 2018, at 16:30, Ash Berlin-Taylor > wrote: > > Found some time, opened a PR: > https://github.com/apache/incubator-airflow/pull/3413 > <https://github.com/apache/incubator-airflow/pull/3413> > > Commiters: > &

Re: Apache Airflow 1.10.0b2

I'm preparing a patch that maintains config compat between 1.9 and 1.10 around Celery settings. We renamed them to better names in AIRFLOW-1840 but we should respect the old names for at least one release version. -ash > On 25 Jun 2018, at 22:30, Stefan Seelmann wrote: > > I n

Re: Apache Airflow 1.10.0b2

pat with 1.9 config names. > On 26 Jun 2018, at 09:05, Ash Berlin-Taylor > wrote: > > I'm preparing a patch that maintains config compat between 1.9 and 1.10 > around Celery settings. We renamed them to better names in AIRFLOW-1840 but > we should respect the old nam

Re: Apache Airflow 1.10.0b4

If there are fixes we want to get in should we merge them into the 1-10-test branch? I'm not sure what the release/branching workflow is yet, and I don't think it's written in the wiki anywere (or I'm bad at finding it) -ash > On 1 Jul 2018, at 11:03, Bolke de Br

Re: Initial Design for Supporting fine-grained Connection encryption [Feedback Requested]

A KMS is nothing strictly to do with Kubernetes - both Google Cloud and AWS have a Key Management System. Having extra fields in the JSON relies on extra being JSON which it might not be in every case. Comments in-line Jasper. The end goal of only allowing certain tasks to decrypt connections i

Re: Airflow's JS code (and dependencies) manageable via npm and webpack

Great! Thanks for doing this. I've left some review comments on your PR. -ash > On 9 Jul 2018, at 11:45, Verdan Mahmood wrote: > > ​Hey Guys, ​ > > In an effort to simplify the JS dependencies of Airflow > ​​ > , > ​I've > introduce > ​d​ > npm and

Re: Using large numbers of sensors, resource consumption

It would be nice to have a different kind of sensor (or a flag to the existing ones) so that rather than sitting in a busy loop on an executor they just go back and re-schedule themselves. We've just not gotten around to writing that (we being where I work). -ash > On 10 Jul 2018, at

Re: Using large numbers of sensors, resource consumption

It would be nice to have a different kind of sensor (or a flag to the existing ones) so that rather than sitting in a busy loop on an executor they just go back and re-schedule themselves. We've just not gotten around to writing that (we being where I work). -ash > On 10 Jul 2018, at

Re: [VOTE] Airflow 1.10.0rc1

I'm afraid I won't have any time to test it out before Friday at the earliest. There have been some people using it successfully in the Gitter chat room, so +0.5 (binding) from me. -ash > On 11 Jul 2018, at 22:24, Bolke de Bruin wrote: > > Committers can you please cast

Re: Splitting hooks/operators

lete_loop: 12.4513s The slowest operator test only appears as the 100th slowest test! I'm not against splitting tests if we have answers to the questions :) -ash > On 9 Jul 2018, at 21:56, Taylor Edmiston <mailto:tedmis...@gmail.com>> wrote: > > We discussed the sp

Re: [VOTE] Airflow 1.10.0rc1

Possible blocking regression here - .airflowignore doesn't seem to be respected anymore https://issues.apache.org/jira/browse/AIRFLOW-2729 <https://issues.apache.org/jira/browse/AIRFLOW-2729> -ash > On 12 Jul 2018, at 08:45, Bolke de Bruin wrote: > > Hi Jakob, > >

Re: [VOTE] Airflow 1.10.0rc1

That said I can't reproduce it myself (I will take discussion to the Jira ticket) -ash > On 12 Jul 2018, at 13:36, Ash Berlin-Taylor > wrote: > > Possible blocking regression here - .airflowignore doesn't seem to be > respected anymore > > https://issues.apac

Re: [VOTE] Airflow 1.10.0rc1

e 1.9, so updating my vote to -0.5 (binding), but not blocking/vetoing release if others are okay with it. -ash > On 12 Jul 2018, at 13:46, Ash Berlin-Taylor > wrote: > > That said I can't reproduce it myself (I will take discussion to the Jira > ticket) > > -ash &g

Re: [VOTE] Airflow 1.10.0rc1

orrect email so it doesn't need moderation. Sorry) -ash > On 12 Jul 2018, at 21:54, Bolke de Bruin wrote: > > If a PR is quick, I’m willing to restart the vote for a RC2 (we are 3 +1, 0.5 > -1). > > B. > > Verstuurd vanaf mijn iPad > >> Op 12 jul. 2018 om 1

Re: [VOTE] Airflow 1.10.0rc1

27;t change that behaviour) It doesn't contain unittests as we don't have anything in place to test that. I'm not happy with that situation, but it doesn't make anything worse. It removes duplication between models.py and utils/dag_processing.py too. -ash > On 12 Jul 201

Re: [VOTE] Airflow 1.10.0rc1

ithub.com/apache/incubator-airflow/pull/3603 <https://github.com/apache/incubator-airflow/pull/3603> -ash > On 13 Jul 2018, at 17:36, Bolke de Bruin wrote: > > Example dags are not picked up. If you put a dag in the normal dag folder it > works fine. > > Please crea

Staled Infra ticket to migrate to github - can we chase anyone?

che.org/jira/browse/INFRA-16615> for example). Any ideas? Thanks, Ash

  1   2   3   >