Re: UI not working on Master

> I could reproduce the problem. > > On Tue, Dec 18, 2018 at 2:17 PM Daniel Imberman > wrote: > > > Do we need to install NPM? Should I add npm to the docker image/bootstrap > > script? > > > > On Tue, Dec 18, 2018 at 1:20 PM Tao Feng wrote: > > >

Re: UI not working on Master

On Tue, Dec 18, 2018 at 12:50 PM Daniel Imberman < > daniel.imber...@gmail.com> > wrote: > > > I believe I am using the new UI since I have to login as admin to get > in. I > > can still see certain admin panels, but the primary DAG one doesn't show > > any D

Re: UI not working on Master

as Admin user for new > UI? > > On Tue, Dec 18, 2018 at 11:56 AM Daniel Imberman < > daniel.imber...@gmail.com> > wrote: > > > Hello all, > > > > I've been testing the K8sExecutor and I notice that when I open the UI, > I'm > > unable to see any

UI not working on Master

Hello all, I've been testing the K8sExecutor and I notice that when I open the UI, I'm unable to see any of the DAGs even though they are in the aiflow/dags folder. I'm also still able to launch DAGs via the API. I have not tested this on any of the other executors yet. Is this a known issue?

Re: Call for fixes for Airflow 1.10.2

I'd also say that this https://github.com/apache/incubator-airflow/pull/3683 PR which allows users to bake their DAGs into their docker image is pretty critical (didn't realize this hadn't been merged yet). It's also ready to go if someone could PTAL :) On Thu, Dec 6, 2018 at 7:13 AM Daniel

Re: Call for fixes for Airflow 1.10.2

Thank you for setting this up Kaxil! I know that https://issues.apache.org/jira/plugins/servlet/mobile#issue/AIRFLOW-3281 is pretty necessary as without it a major portion of the git-sync for the k8s executor is broken. Hopefully should be merged soon On Thu, Dec 6, 2018, 6:13 AM James Meickle

Re: Question on Running Airflow 1.10 in Kubernetes

Hi pramiti, We're in the process of allowing baked in images for the k8s executor (should be merged soon/possibly already merged). With this added you can specify the worker image in the airflow.cfg pretty easily the only potential issue with re-launching multiple times a day would be if a DAG

Re: Making Airflow Fault-Tolerant when running Airflow on Kubernetes

Hi Kevin, Have you looked into the KubernetesExecutor? We achieve fault tolerance using the kubernetes resourceVersion to ensure that all state is reproducible. On Wed, Sep 12, 2018 at 1:08 PM Kevin Lam wrote: > Hi all, > > We currently run Airflow as a Deployment in a kubernetes cluster. We

Re: Retiring Airflow Gitter?

+1 on slack, especially if we can have multiple channels. There's something really great about being able to ask questions in specialized mini communities (i.e. #celery-executor or #kubernetes-executor). It also makes it way easier for subject matter experts to respond to relevant questions. On

Re: Retiring Airflow Gitter?

@anirudh @barni could you guys add whoever manages the k8s slack please? On Sat, Sep 1, 2018, 2:30 AM Ash Berlin-Taylor wrote: > I'm not a fan of slack for open source work - it's a walled garden, > signing up is a hurdle (where you have to use a work around of a heroku > app), not to mention

Re: Retiring Airflow Gitter?

I'm with Adam. Slack has been significantly more useful for the k8s community. It's now the case that all k8s executor questions are being fielded on kubernetes.slack.com. Would be great to have an airflow slack instead. On Fri, Aug 31, 2018 at 10:54 AM Trent Robbins wrote: > I'm in favor of

Re: Will redeploying webserver and scheduler in Kubernetes cluster kill running tasks

Also worth mentioning that when you restart the scheduler it will use ETCD and postgres to recreate state so you won't end up re-launching tasks/missing tasks On Thu, Aug 30, 2018, 12:54 PM Eamon Keane wrote: > Great I must give pgbouncer a try. Testing on GKE/cloudsql I quickly ran > into that

Re: [RESULT][VOTE] Airflow 1.10.0rc4

RC4) has been accepted. > > 4 “+1” binding votes received: > > - Ash Berlin-Taylor (binding) > - Naik Kaxil (binding) > - Bolke de Bruin (binding) > - Fokko Driesprong (binding) > > 1 “-1” non-binding: > > - Daniel Imberman (non-binding) > > My next step is to

Re: [VOTE] Airflow 1.10.0rc4

-1 (non-binding), There is a k8s bug fix that should be PRed by @jordan.zucker this weekend (relating to the tracking of resourceVersions). There have also been multiple users requesting the ability to pre-bake docker images which I will make a PR for this weekend. On Thu, Aug 9, 2018 at 11:22

Re: K8S deployment operator proposal

a second operator to implement, it would definitely be > a Helm operator instead of a Kube API operator. > > On Tue, Jul 3, 2018 at 12:47 PM, Daniel Imberman < > daniel.imber...@gmail.com> > wrote: > > > An example of creating a deployment using the k8s model architect

Re: K8S deployment operator proposal

would handle a lot of the boilerplate. On Tue, Jul 3, 2018 at 9:20 AM Daniel Imberman wrote: > Hi all, > > Enclosed is a proposal for a kubernetes deployment management operator. I > think this would be a good addition to current k8s offerings s.t. users can > actually launch persisten

K8S deployment operator proposal

Hi all, Enclosed is a proposal for a kubernetes deployment management operator. I think this would be a good addition to current k8s offerings s.t. users can actually launch persistent applications from airflow DAGs. * What?* Add an operator that monitors a k8s deployment, declaring the task

Re: Master is broken

+1 for merge blocking hooks. It would be great to have safety knowing that any commit I revert to will still pass tests (for rebase testing, etc) On Mon, Jun 11, 2018 at 10:23 PM Alex Tronchin-James 949-412-7220 <(949)%20412-7220> wrote: > Could we adopt some sort of merge-blocking hook that

Re: Dockerised CI and testing environment

Hi Gerardo, I left some comments on the PR. Could you please get the travis tests to pass + rebase your PR? Afterwards I'd be glad to try it out. On Mon, May 21, 2018 at 6:55 AM Gerardo Curiel wrote: > Hello folks, > > I just submitted a PR for using Docker as part of

Re: Airflow Docker Container

om.invalid>: > >> @Erik Erlandson <e...@redhat.com> has had conversations about publishing > > >> docker images with the ASF Legal team. >> Adding him to the thread. >> >> On Mon, May 14, 2018 at 1:07 PM Daniel Imberman < >> daniel.imber...@gm

Airflow Docker Container

Hi everyone, I've started looking into creating an official airflow docker container s.t. users of the KubernetesExecutor could auto-pull from helm charts/deployment yamls/etc. I was wondering what everyone thinks the best way to do this would be? Is there an official apache docker repo? Is there

Re: How to use Kuberenetes executor with Airflow

Hi Goutham, Once we fix a few final bugs with the k8s executor (which should be in the next week). The next step will be to write out documentation for the kubernetes executor (we have one for the Kubernetes Operator but are waiting to hear back from CNCF to cross-post with them). If you look in

Re: Apache Airflow welcome new committer/PMC member : Naik Kaxil (a.k.a. kaxil)

Congratulations! On Wed, May 9, 2018 at 2:33 AM Naik Kaxil wrote: > Hi Everyone! > > Thanks for inviting me to join the Airflow PPMC! > > Brief introduction: I live and work in London, I'm currently working for a > Big Data & Analytics consultancy specialising in Google Cloud

Re: Managed Apache Airflow Service on Google Cloud Platform

+1! Congrats Feng! On Tue, May 1, 2018 at 9:59 AM Feng Lu wrote: > *Hello everyone,I want to let everyone know that today Google Cloud > launched a new managed service based on Apache Airflow - Cloud Composer[1]. > Now that we have launched into public beta, I wanted

Re: Airflow Tasks on Kubernetes with Pre-emptible VMs (Jobs vs. Pods vs ?)

Hi Kevin, I can answer in more detail in about 1 hour but I'll give a few quick points: 1. We decided that jobs are pretty risky when it comes to airflow deployments (imagine a pod that launches a spark job retrying infinitely), however airflow allows you to define a retry policy 2. We

Re: Airflow + Kubernetes + Git Sync

@jordan can you turn delete mode off and post the kubectl describe results for the workers? On Sat, Jan 13, 2018, 3:20 AM Koen Mevissen wrote: > Are you using kubernetes on Google Cloud Platform? (GKE) > > You should be able to capture the logs from your nodes. In case you

[Airflow 1517] Kubernetes Operator PR

Hi guys! We've finished the first feature PR for the airflow kubernetes integration, and I would love to open it up for discussion/review to make sure we meet the needs of the community at large :). Please take a look here and let us know if there's anything you would like us to add/fix!

Re: Dynamic Airflow config reloads

At least for the k8s executor it would be easier to just restart the scheduler pod. I can see some potential issues if some configs are stored in local variables. On Wed, Dec 6, 2017 at 11:45 AM Feng Lu wrote: > Hi, > > It's probably well-known that Airflow only loads

Re: Introducing a "LAUNCHED" state into airflow

t; more people might want to chime in and offer valuable opinions) > > > > > > Jumping right in: I am wondering if are you not duplicating the > “queued” > > > logic for (a.o) pools. Introducing LAUNCHED with the meaning attached > to > > > it that you describe

Re: Introducing a "LAUNCHED" state into airflow

pressure. > > > > Isn’t there a way to ask the k8s cluster how many tasks it has pending > and > > just to execute any queued tasks when it crosses a certain threshold? > Have > > a look a base_executor where it is handling slots and queued tasks. > > > > Che

Re: Kubernetes Executor

Hi Niranda, One thing to note is we're actually re-writing the roadmap document since there have been pretty significant changes since it was released. Here is a more recent talk with an overview of the system and the features: https://www.youtube.com/watch?v=5BU3YPYYRno. We will also be

Re: Automatic DAGs deployment

+1 for this conversation. I know that most of the production airflow instances basically just have a policy of "don't update the DAG files while a job is running." One thing that is difficult with this, however, is that for CeleryExecutor and KubernetesExecutor we don't really have any power

Re: Airflow on ECS

Hi Shoumitra, One thing worth noting is that with the release of the kubernetes executor, we will be using resource versions + the Kubernetes API to take care of some of the current issues with crash handling (basically recreating state from what tasks have been run/are pending within the

Re: Moving on to Lyft

Congratulations, Max! On Thu, Oct 26, 2017 at 9:57 AM Arthur Wiedmer wrote: > Congratulations, Max! > > > > On Thu, Oct 26, 2017 at 8:25 AM, Chris Riccomini > wrote: > > > Congrats, Max! > > > > On Thu, Oct 26, 2017 at 7:32 AM, Andy

Re: Meetup Interest?

+1 We're getting really close on the Kubernetes Executor PR. Would love to discuss final features/architecture to make sure we cover our bases before we try to roll out alpha. From: mw...@newrelic.com Subject: Re: Meetup Interest? +1 for this meetup idea! We don't use Kube+Airflow, but I'd

Re: 1.9.0

@sergey: Chris and Bolke are 100% correct. Still a few major features needed in k8s. Definitely hoping to have an alpha/beta version for the next release if all goes well, though :). On Mon, Sep 11, 2017 at 2:05 PM Chris Riccomini wrote: > I'm waiting to cut 1.9.0 test

Re: Airflow + Kubernetes update meeting

On Wed, Sep 6, 2017 at 2:34 PM Daniel Imberman <daniel.imber...@gmail.com> wrote: > Hey guys! > > @Dan & @Chris: Great! > @Marc: My pleasure :). > > So everything is all set for tomorrow at 10AM PST at > zoom.us/my/sig.big.data. Main points will be to update o

Re: Airflow + Kubernetes update meeting

; On Mon, Sep 4, 2017 at 11:02 PM, Anirudh Ramanathan < > >> ramanath...@google.com> wrote: > >> > >>> Date/time work for me if we get quorum from this group. > >>> > >>> On Thu, Aug 31, 2017 at 7:54 PM, Christopher Bockman < >

Airflow + Kubernetes update meeting

Hey guys! So I wanted to set up a meeting to discuss some of the updates/current work that is going on with both the kubernetes operator and kubernetes executor efforts. There has been some really cool updates/proposals on the design of these two features and I would love to get some community

Re: RFC: Managing task credentials inside KubernetesExecutor

Thank you for posting this to the wiki Feng Lu :). I'm going to propose an overall "airflow + kubernetes update" meeting in a seperate email to discuss with the community at large. Would love it if you could discuss this further at that meeting! Daniel On Wed, Aug 30, 2017 at 10:38 PM Feng Lu

Re: Repurposing the `queue` field for non-celery executors

I think its a really good idea from the kubernetes executor perspective. There are a lot of elements that should be automated when launching a pod (i.e. a bash command wouldn't need the same amount of compute as building a model with scikitlearn) @Alexander > don't you think that you give too

Re: Airflow + Kubernetes Talk video

airbnb.com.invalid> > wrote: > > > Thanks for organizing, and leading this effort in general! > > > > On Fri, Jul 28, 2017 at 2:40 PM, Daniel Imberman < > > daniel.imber...@gmail.com> > > wrote: > > > > > Hi guys! > > >

Airflow + Kubernetes Talk video

Hi guys! Thank you again to everyone who attended the talk yesterday. I've posted the video of the conversation to youtube, and will soon add the video and slides to the airflow Wiki Cheers, Daniel https://www.youtube.com/watch?v=5BU3YPYYRno

Re: Airflow + Kubernetes discussion

2017 at 11:04 AM, Wilson Lian <wwl...@google.com.invalid> > wrote: > > > The call is full :( > > > > On Thu, Jul 27, 2017 at 11:02 AM, Daniel Imberman < > > daniel.imber...@gmail.com > > > wrote: > > > > > Hello everyone, > > >

Re: Airflow + Kubernetes discussion

ll take this message to the right place. > > On Fri, 21 Jul 2017 at 01:55 Maxime Beauchemin < > maximebeauche...@gmail.com > > > > > wrote: > > > > > +1! > > > > > > On Thu, Jul 20, 2017 at 5:53 PM, Feng Lu <fen...@google.c

Invitation: Airflow + Kubernetes discussion @ Thu Jul 27, 2017 11am - 12:30pm (PDT) (dev@airflow.incubator.apache.org)

=INDIVIDUAL;ROLE=REQ-PARTICIPANT;PARTSTAT=NEEDS-ACTION;RSVP= TRUE;CN=criccom...@apache.org;X-NUM-GUESTS=0:mailto:criccom...@apache.org ATTENDEE;CUTYPE=INDIVIDUAL;ROLE=REQ-PARTICIPANT;PARTSTAT=ACCEPTED;RSVP=TRUE ;CN=Daniel Imberman;X-NUM-GUESTS=0:mailto:daniel.imber...@gmail.com ATTENDEE;CUTYPE

Re: Airflow + Kubernetes discussion

>> cell. 9632202950 > >>>> > >>>> > >>>> On Thu, Jul 20, 2017 at 11:00 PM, Chris Riccomini < > >> criccom...@apache.org > >>>> > >>>> wrote: > >>>> > >>>>> I would definitely be up to joi

Airflow + Kubernetes discussion

Hello everyone, Recently there's been a fair amount of discussion regarding the integration of airflow with kubernetes. If there is interest I would love to host an e-meeting to discuss this integration. I can go over the architecture as it stands right now and would love feedback on

Re: Airflow kubernetes executor

@Grant @Gerard: WRT Static workers: I think that the downside of a slightly longer start-up time is significantly less important than the massive upshot in scalability offered by having non-static workers. This also opens up really interesting opportunities (i.e. being able to identify how many

Re: Airflow kubernetes executor

thinking is that a pod/container is just a > temporary > > > > thing that runs a job > > > >and we should be interested in how the job did vs. how the > > > container/pod > > > > ran this. From my little experience with k8s though, I do see that it >

Re: Notifications

Hi Niels, Thank you for setting this up! I think this is a feature that would really useful to a lot of people. Have you considered setting up a "notifier plugin" so that people can create custom notifiers? The API seems pretty consistent so I don't think it would be too much work to add. On

Re: Airflow kubernetes executor

, 2017 at 11:36 AM Daniel Imberman <daniel.imber...@gmail.com> wrote: > Hi Amit, > > For now the design doc is included as an attachment to the original email. > Once I am able to get permission to edit the wiki I would like add it there > but for now I figured that this would

Re: Airflow kubernetes executor

com> wrote: > Hi Daniel, > > I don't see link to design PDF. > > > Amit Kulkarni > Site Reliability Engineer > Mobile: (716)-352-3270 <(716)%20352-3270> > > Payments partner to the platform economy > > On Wed, Jul 5, 2017 at 11:25 AM, Daniel Imberman <

Airflow kubernetes executor

Hello Airflow community! My name is Daniel Imberman, and I have been working on behalf of Bloomberg LP to create an airflow kubernetes executor/operator. We wanted to allow for maximum throughput/scalability, while keeping a lot of the kubernetes details abstracted away from the users. Below I