Re: Improving the Airflow UI

+1 The react issue was solved some time ago and is fine to use. A challenge might be to keep track of all the licenses of the subcomponents react can pull in. Superset has some experience there. Superset is also based around the same components as airflow (FAB/React) B. Sent from my iPhone >

Re: Improving the Airflow UI

Thank you for starting this discussion/initiative! Love it. Not a frontend guy but react looks fine. +1 for Dan's point of making it more testable. Great opportunity to establish a better UI baseline. Cheers, Kevin Y On Wed, Nov 27, 2019 at 12:29 PM Jarek Potiuk wrote: > Indeed it's been re-li

Re: Improving the Airflow UI

Indeed it's been re-licenced. It was more than licence for Facebook, it was BSD+Patent clause that is still forbidden by Apache . (but React is now OK). In favour of React JS, we have quite good experience. BTW. Just a thought. Maybe for the

Re: Improving the Airflow UI

Good to know that community is still leaning towards ReactJS. I'm also in favor of moving Airflow frontend to ReactJS, and initiated this discussion last year. https://issues.apache.org/jira/browse/AIRFLOW-2598 +1, frontend definitely needs some love from community :) Best, *Verdan Mahmood* On

Re: [DISCUSS] Using shared memory for inter-task communication

If memory is shared across tasks, they are by definition not idempotent, which can be troublesome. What if you have a chain of 3 tasks and the last one failed while operating on the memory that came from task number 2? The whole chain may have to be re-executed, which to me sounds like it's really

Re: Improving the Airflow UI

The issue was before they re-licensed it. Now I believe the issue is put to bed as MIT is Apache compatible. On Wed, Nov 27, 2019 at 7:38 AM Kamil Breguła wrote: > But there is the question, does Apache have additional restrictions on > this issue? > > On Wed, Nov 27, 2019 at 4:30 PM Colin Ingar

Re: [PROPOSAL] Migrate to Pytest

Hi all, Here is a PR that introduces pytest. Your opinions are highly appreciated! ;) https://github.com/apache/airflow/pull/6472 Travis's red due to one test (test_on_kill) but it can be fixed as seen in previous commits. Bests, Tomek On Thu, Oct 31, 2019 at 6:11 PM Chao-Han Tsai wrote: > +1

Re: Improving the Airflow UI

But there is the question, does Apache have additional restrictions on this issue? On Wed, Nov 27, 2019 at 4:30 PM Colin Ingarfield wrote: > > React is currently licensed under MIT. > > https://github.com/facebook/react/blob/master/LICENSE > > https://www.freecodecamp.org/news/facebook-just-chang

Re: Improving the Airflow UI

React is currently licensed under MIT. https://github.com/facebook/react/blob/master/LICENSE https://www.freecodecamp.org/news/facebook-just-changed-the-license-on-react-heres-a-2-minute-explanation-why-5878478913b2/ On 11/27/19, 9:11 AM, "Kamil Breguła" wrote: [EXTERNAL EMAIL] I

Re: Improving the Airflow UI

Is React legal in Apache initiatives already? I heard that this project changed the licenses, but we should watch out for Facebook. https://www.itprotoday.com/devops-and-software-development/apache-foundation-and-facebook-standoff-over-reactjs-license Here is license for Angular: https://angular.i

Re: Improving the Airflow UI

+1 to everything you said, it all sounds like awesome work : ). Hopefully will be easier to make the front-end code testable as well. Another thing to maybe think about in the future is plugin/customization of the UI. E.g. being able to have custom UI widgets for operators that e.g. visualize data

Re: [DISCUSS] Using shared memory for inter-task communication

Yeah. I was thinking about the new landing page/website where we specifically have section "Use cases" and we can describe some actual examples (and counter-examples specifically) :). J. On Wed, Nov 27, 2019 at 11:43 AM Bolke de Bruin wrote: > From our website :-) > > "Airflow *is not* a data s

Re: [DISCUSS] Using shared memory for inter-task communication

>From our website :-) "Airflow *is not* a data streaming solution. Tasks do not move data from one to the other (though tasks can exchange metadata!). Airflow is not in the Spark Streaming or Storm space, it is more comparable to Ooz

Re: [DISCUSS] Using shared memory for inter-task communication

Listening to all those comments, that reaffirms the gut feelings I had. Even if like the idea of optimisations, I think it makes sense to say "it's not an Airflow-domain problem really". I think now that XCom is good what it is for and introducing "generic" data passing mechanism goes way beyond wh

Re: [DISCUSS] Using shared memory for inter-task communication

I agree with Bolke, Airflow is not a data processing tool. Also it should not become one as we already have some awesome solutions like Apache Storm, Flink or Beam. Tomek On Wed, Nov 27, 2019 at 10:24 AM Bolke de Bruin wrote: > My 2 cents: > > I don’t think this makes sense at all as it goes a

Improving the Airflow UI

Hi everyone, We here at Astronomer are thinking about what we'd next like to work on to improve Airflow, and one of the most visible ways we could improve Airflow would be to update the UI, and make it, well, more designed and less grown-over-time :) A non-exhaustive list of things we'd like t

Re: [DISCUSS] Using shared memory for inter-task communication

My 2 cents: I don’t think this makes sense at all as it goes against to core of Airflow: Airflow does not do data processing by itself. So the only think you should share between tasks is meta data and that you do through XCom. We can redesign com if you want but it is also the only viable option

Re: [DISCUSS] Using shared memory for inter-task communication

Agreed on running before we can crawl. The logical way to do this now is to group it as one big task with more resources. With respect to affinity on the same machine, that's basically what it is. I guess this hinges on well your solution can handle workloads with different resource requirements.