RE: [EXT] Airflow and NiFi
User: "I want to change my production flow while it's running, you know, mid stream just route my content to a completely different flow, re-run data mid run through a new set of processors, fork it on the fly, you know, whatever I want anytime I want " NiFi: "Yeah, I can do that" Everyone Else: " I'm sorry... what?" -Original Message- From: Mike Thomsen Sent: Thursday, August 15, 2019 2:00 PM To: dev@nifi.apache.org Subject: [EXT] Airflow and NiFi Does anyone see any areas where the two can complement each other and where we might want to give users the ability to offload processing to Airflow? Curious since our poking around the docs lead us to conclude it was probably more "Airflow vs Spark+Oozie" than really competing with NiFi.
Re: [EXT] Airflow and NiFi
Yeah, pretty much. On Thu, Aug 15, 2019 at 4:02 PM Peter Wicks (pwicks) wrote: > User: "I want to change my production flow while it's running, you know, > mid stream just route my content to a completely different flow, re-run > data mid run through a new set of processors, fork it on the fly, you know, > whatever I want anytime I want " > > NiFi: "Yeah, I can do that" > > Everyone Else: " I'm sorry... what?" > > -Original Message- > From: Mike Thomsen > Sent: Thursday, August 15, 2019 2:00 PM > To: dev@nifi.apache.org > Subject: [EXT] Airflow and NiFi > > Does anyone see any areas where the two can complement each other and > where we might want to give users the ability to offload processing to > Airflow? > Curious since our poking around the docs lead us to conclude it was > probably more "Airflow vs Spark+Oozie" than really competing with NiFi. >
Re: [EXT] Airflow and NiFi
Love it. Airflow I see as an orchestrator of making other systems do things. Nifi i see as very focused on acquiring, manipulating, and routing data between systems. In short, I believe theyre complimentary in a given architecture. Someone always tries to use one to do both of the things but each is better at their sweet spot. Thanks On Thu, Aug 15, 2019 at 4:07 PM Mike Thomsen wrote: > Yeah, pretty much. > > On Thu, Aug 15, 2019 at 4:02 PM Peter Wicks (pwicks) > wrote: > > > User: "I want to change my production flow while it's running, you know, > > mid stream just route my content to a completely different flow, re-run > > data mid run through a new set of processors, fork it on the fly, you > know, > > whatever I want anytime I want " > > > > NiFi: "Yeah, I can do that" > > > > Everyone Else: " I'm sorry... what?" > > > > -Original Message- > > From: Mike Thomsen > > Sent: Thursday, August 15, 2019 2:00 PM > > To: dev@nifi.apache.org > > Subject: [EXT] Airflow and NiFi > > > > Does anyone see any areas where the two can complement each other and > > where we might want to give users the ability to offload processing to > > Airflow? > > Curious since our poking around the docs lead us to conclude it was > > probably more "Airflow vs Spark+Oozie" than really competing with NiFi. > > >
Re: [EXT] Airflow and NiFi
i blame spell check for ‘complimentary’ On Thu, Aug 15, 2019 at 4:11 PM Joe Witt wrote: > Love it. > > Airflow I see as an orchestrator of making other systems do things. Nifi > i see as very focused on acquiring, manipulating, and routing data between > systems. > > In short, I believe theyre complimentary in a given architecture. Someone > always tries to use one to do both of the things but each is better at > their sweet spot. > > Thanks > > On Thu, Aug 15, 2019 at 4:07 PM Mike Thomsen > wrote: > >> Yeah, pretty much. >> >> On Thu, Aug 15, 2019 at 4:02 PM Peter Wicks (pwicks) >> wrote: >> >> > User: "I want to change my production flow while it's running, you know, >> > mid stream just route my content to a completely different flow, re-run >> > data mid run through a new set of processors, fork it on the fly, you >> know, >> > whatever I want anytime I want " >> > >> > NiFi: "Yeah, I can do that" >> > >> > Everyone Else: " I'm sorry... what?" >> > >> > -Original Message- >> > From: Mike Thomsen >> > Sent: Thursday, August 15, 2019 2:00 PM >> > To: dev@nifi.apache.org >> > Subject: [EXT] Airflow and NiFi >> > >> > Does anyone see any areas where the two can complement each other and >> > where we might want to give users the ability to offload processing to >> > Airflow? >> > Curious since our poking around the docs lead us to conclude it was >> > probably more "Airflow vs Spark+Oozie" than really competing with NiFi. >> > >> >
Re: [EXT] Airflow and NiFi
Nifi is used for ETL and focus on transformations and data movement. Airflow is good for orchestration. Both tools have scheduling in common. Airflow is good at where the data is already available and you want to control jobs. If your focus is more like the data integertion, manipulation and transformation and movement then NiFi is a great choice. No doubt there is some overlap but ultimately it comes down to your use case. Thanks, Muazma On Thu, Aug 15, 2019, 1:15 PM Joe Witt wrote: > i blame spell check for ‘complimentary’ > > On Thu, Aug 15, 2019 at 4:11 PM Joe Witt wrote: > > > Love it. > > > > Airflow I see as an orchestrator of making other systems do things. Nifi > > i see as very focused on acquiring, manipulating, and routing data > between > > systems. > > > > In short, I believe theyre complimentary in a given architecture. > Someone > > always tries to use one to do both of the things but each is better at > > their sweet spot. > > > > Thanks > > > > On Thu, Aug 15, 2019 at 4:07 PM Mike Thomsen > > wrote: > > > >> Yeah, pretty much. > >> > >> On Thu, Aug 15, 2019 at 4:02 PM Peter Wicks (pwicks) > > >> wrote: > >> > >> > User: "I want to change my production flow while it's running, you > know, > >> > mid stream just route my content to a completely different flow, > re-run > >> > data mid run through a new set of processors, fork it on the fly, you > >> know, > >> > whatever I want anytime I want " > >> > > >> > NiFi: "Yeah, I can do that" > >> > > >> > Everyone Else: " I'm sorry... what?" > >> > > >> > -Original Message- > >> > From: Mike Thomsen > >> > Sent: Thursday, August 15, 2019 2:00 PM > >> > To: dev@nifi.apache.org > >> > Subject: [EXT] Airflow and NiFi > >> > > >> > Does anyone see any areas where the two can complement each other and > >> > where we might want to give users the ability to offload processing to > >> > Airflow? > >> > Curious since our poking around the docs lead us to conclude it was > >> > probably more "Airflow vs Spark+Oozie" than really competing with > NiFi. > >> > > >> > > >