Re: Duplicate key unique constraint error

Can you share the full python stack trace? On Wed, Sep 12, 2018 at 5:31 PM Abhishek Sinha wrote: > Got the following error on Airflow 1.8.2 version: > > duplicate key value violates unique constraint "task_instance_pkey" > > DETAIL: Key (task_id, dag_id, execution_date)=(PB_BPNZ, master_v2, >

Duplicate key unique constraint error

Got the following error on Airflow 1.8.2 version: duplicate key value violates unique constraint "task_instance_pkey" DETAIL: Key (task_id, dag_id, execution_date)=(PB_BPNZ, master_v2, 2018-09-12 03:00:37) already exists.n [SQL: 'INSERT INTO task_instance (task_id, dag_id, execution_date,

Re: Live Airflow Meetup @ Amsterdam

Looked like a great event Bolke! Sorry if this was covered and I missed it, but are there plans to open-source the Dataflows library that Bas described? Seems like a very useful Airflow supplement. On Wed, Sep 12, 2018 at 11:19 AM Bolke de Bruin wrote: >

Re: Making Airflow Fault-Tolerant when running Airflow on Kubernetes

Hi Daniel, Thanks for the reply! No we haven't looked too deeply into it. Can you elaborate a bit on how that works? With the KubernetesExecutor, if a DAG is in flight and part of airflow go down, it will be able to recover? How do airflow workers reconnect to Pods that were in flight? On Wed,

Re: Making Airflow Fault-Tolerant when running Airflow on Kubernetes

Hi Kevin, Have you looked into the KubernetesExecutor? We achieve fault tolerance using the kubernetes resourceVersion to ensure that all state is reproducible. On Wed, Sep 12, 2018 at 1:08 PM Kevin Lam wrote: > Hi all, > > We currently run Airflow as a Deployment in a kubernetes cluster. We

Making Airflow Fault-Tolerant when running Airflow on Kubernetes

Hi all, We currently run Airflow as a Deployment in a kubernetes cluster. We also use a variant of KubernetesOperator to run our DAGs. We are investigating how to best make Airflow fault-tolerant, in part, due to investigating the use of preemptible vms [1]. *Has there been much discussion about

Live Airflow Meetup @ Amsterdam

https://www.youtube.com/watch?v=MM8tfTrcnfk Enjoy! Bolke

Re: Visual Correlation in Superset

Wrong dev list :-) -s On Wed, Sep 12, 2018 at 8:55 AM Maxime Beauchemin < maximebeauche...@gmail.com> wrote: > The main thing that "scoped filters" and "visual correlation" have in > common is the fact that they imply some sort of cross chart communication. > Christine I tagged you here so you'd

Re: Sep Airflow Bay Area Meetup @ Google

btw how are we doing on the “one day” hackathon? > On 12 Sep 2018, at 16:49, Bolke de Bruin wrote: > > Hi feng, > > I can do “Elegant pipelining with Airflow” recycle of pydata 2018 amsterdam > (that I did together with Fokko). > > Cheers > Bolke > >> On 4 Sep 2018, at 22:13, Feng Lu

Re: Sep Airflow Bay Area Meetup @ Google

Hi feng, I can do “Elegant pipelining with Airflow” recycle of pydata 2018 amsterdam (that I did together with Fokko). Cheers Bolke > On 4 Sep 2018, at 22:13, Feng Lu wrote: > > We are 3 weeks away from the meetup and still have a few lightening talks > open, please take the chance and share

Re: Call for fixes for Airflow 1.10.1

Hi Ash, I've cherry-picked two commits on top of 1.10-test branch: https://github.com/apache/incubator-airflow/commit/4d2f83b19af3489d6c9563d51210a3dab2f38b26 https://github.com/apache/incubator-airflow/commit/0d2bb5cd215bd3e2d4f47033230ea03b14b36634 This was on request of Yuan of the Amazon