Re: [VOTE] Release Airflow 1.8.2 based on Airflow 1.8.2 RC2
I think I'm gathering a good picture of what is expected here. I'll try to update the Confluence page as I go. I'm hoping to get started tomorrow and package it early next week. Max On Wed, Jul 19, 2017 at 6:16 PM, siddharth anand wrote: > FYI, can anyone pictorially describe the release process (and post it on > the apache airflow wiki)? I think that would eliminate a lot of confusion > in the future and avoid a rehash of this email thread on the next release. > > -s > > On Wed, Jul 19, 2017 at 10:48 AM, Hitesh Shah wrote: > > > To add, the main source tarball should have instructions to generate the > > sdist and bdist versions. Additionally, as part of the release process if > > the plan is to publish to pypi (after the IPMC vote succeeds), then the > > appropriate bits also need to be verified/voted upon. There are not > exactly > > counted as the official release bits but they do need to be verified as > > part of the voting process to ensure that the bits do indeed map to the > > source release, license/notice files are correct, etc. > > > > thanks > > -- Hitesh > > > > > > On Tue, Jul 18, 2017 at 12:01 AM, Bolke de Bruin > > wrote: > > > > > Thanks Hitesh. We discussed it with John Ament on the IPMC. Python has > > the > > > notion of 3 types of distributions, “source”, “sdist”, “bdist”, > contrary > > to > > > Java that knows only two (source, bdist). We used to vote on “sdist”, > > which > > > was deemed incorrect. > > > > > > So, Max, indeed we need to vote on a tar.gz that contains build > > > instructions in INSTALL to get to “sdist”. The build instructions > should > > > also contain instruction how to run the license checks by Apache Rat. > > Most > > > of the work probably goes in the build instructions and verifying they > > > work, but it should not be much. > > > > > > Any other clarification required? > > > > > > Bolke > > > > > > > > >
Re: Airflow + Kubernetes discussion
+1! On Thu, Jul 20, 2017 at 5:53 PM, Feng Lu wrote: > Would like to join, please kindly invite me. > Thanks! > > On Thu, Jul 20, 2017 at 9:54 AM, Daniel Imberman < > daniel.imber...@gmail.com> > wrote: > > > Hello everyone, > > > > Recently there's been a fair amount of discussion regarding the > integration > > of airflow with kubernetes. If there is interest I would love to host an > > e-meeting to discuss this integration. I can go over the architecture as > it > > stands right now and would love feedback on > improvements/features/design. I > > could also attempt to get one or two members of google's kubernetes team > to > > join to discuss best practices. > > > > I'm currently thinking that next Thursday at 11AM PST over zoom.us, > though > > if there's strong opinions otherwise I'd be glad to propose other times. > > > > Cheers! > > > > Daniel > > >
Re: Airflow + Kubernetes discussion
Would like to join, please kindly invite me. Thanks! On Thu, Jul 20, 2017 at 9:54 AM, Daniel Imberman wrote: > Hello everyone, > > Recently there's been a fair amount of discussion regarding the integration > of airflow with kubernetes. If there is interest I would love to host an > e-meeting to discuss this integration. I can go over the architecture as it > stands right now and would love feedback on improvements/features/design. I > could also attempt to get one or two members of google's kubernetes team to > join to discuss best practices. > > I'm currently thinking that next Thursday at 11AM PST over zoom.us, though > if there's strong opinions otherwise I'd be glad to propose other times. > > Cheers! > > Daniel >
Re: Airflow + Kubernetes discussion
Please send me an invite! On Thu, Jul 20, 2017 at 5:15 PM, Dan Davydov wrote: > I'm in. > > On Thu, Jul 20, 2017 at 4:30 PM, Daniel Imberman < > daniel.imber...@gmail.com> > wrote: > > > Glad to hear that people are interested! I've created a google calendar > > event and messaged everyone in this thread, if anyone else would like to > > join please let me know! > > > > On Thu, Jul 20, 2017 at 1:04 PM Bolke de Bruin > wrote: > > > > > Invite would be nice, I will try to join! > > > > > > > On 20 Jul 2017, at 20:36, Gerard Toonstra > wrote: > > > > > > > > send me an invite too! > > > > > > > > On Thu, Jul 20, 2017 at 8:17 PM, Jeremiah Lowin > > > wrote: > > > > > > > >> I'm interested as well. > > > >> > > > >> On Thu, Jul 20, 2017 at 1:51 PM Marc Bollinger > > > wrote: > > > >> > > > >>> +1 We're in the middle of moving some services to k8s, and have had > > our > > > >>> eye on Airflow. > > > >>> > > > On Jul 20, 2017, at 10:37 AM, Sumit Maheshwari < > > > sumeet.ma...@gmail.com > > > >>> > > > >>> wrote: > > > > > > I would join as well for sure. > > > > > > Thanks, > > > Sumit Maheshwari > > > cell. 9632202950 > > > > > > > > > On Thu, Jul 20, 2017 at 11:00 PM, Chris Riccomini < > > > >> criccom...@apache.org > > > > > > wrote: > > > > > > > I would definitely be up to joining. We're interested in the K8s > > work > > > > that's going on. That time works for me. > > > > > > > > On Thu, Jul 20, 2017 at 9:54 AM, Daniel Imberman < > > > > daniel.imber...@gmail.com> > > > > wrote: > > > > > > > >> Hello everyone, > > > >> > > > >> Recently there's been a fair amount of discussion regarding the > > > > integration > > > >> of airflow with kubernetes. If there is interest I would love to > > > host > > > >>> an > > > >> e-meeting to discuss this integration. I can go over the > > > architecture > > > >>> as > > > > it > > > >> stands right now and would love feedback on > > > > improvements/features/design. I > > > >> could also attempt to get one or two members of google's > > kubernetes > > > >>> team > > > > to > > > >> join to discuss best practices. > > > >> > > > >> I'm currently thinking that next Thursday at 11AM PST over > > zoom.us, > > > > though > > > >> if there's strong opinions otherwise I'd be glad to propose > other > > > >>> times. > > > >> > > > >> Cheers! > > > >> > > > >> Daniel > > > >> > > > > > > > >>> > > > >> > > > > > > > > >
Re: Airflow + Kubernetes discussion
I'm in. On Thu, Jul 20, 2017 at 4:30 PM, Daniel Imberman wrote: > Glad to hear that people are interested! I've created a google calendar > event and messaged everyone in this thread, if anyone else would like to > join please let me know! > > On Thu, Jul 20, 2017 at 1:04 PM Bolke de Bruin wrote: > > > Invite would be nice, I will try to join! > > > > > On 20 Jul 2017, at 20:36, Gerard Toonstra wrote: > > > > > > send me an invite too! > > > > > > On Thu, Jul 20, 2017 at 8:17 PM, Jeremiah Lowin > > wrote: > > > > > >> I'm interested as well. > > >> > > >> On Thu, Jul 20, 2017 at 1:51 PM Marc Bollinger > > wrote: > > >> > > >>> +1 We're in the middle of moving some services to k8s, and have had > our > > >>> eye on Airflow. > > >>> > > On Jul 20, 2017, at 10:37 AM, Sumit Maheshwari < > > sumeet.ma...@gmail.com > > >>> > > >>> wrote: > > > > I would join as well for sure. > > > > Thanks, > > Sumit Maheshwari > > cell. 9632202950 > > > > > > On Thu, Jul 20, 2017 at 11:00 PM, Chris Riccomini < > > >> criccom...@apache.org > > > > wrote: > > > > > I would definitely be up to joining. We're interested in the K8s > work > > > that's going on. That time works for me. > > > > > > On Thu, Jul 20, 2017 at 9:54 AM, Daniel Imberman < > > > daniel.imber...@gmail.com> > > > wrote: > > > > > >> Hello everyone, > > >> > > >> Recently there's been a fair amount of discussion regarding the > > > integration > > >> of airflow with kubernetes. If there is interest I would love to > > host > > >>> an > > >> e-meeting to discuss this integration. I can go over the > > architecture > > >>> as > > > it > > >> stands right now and would love feedback on > > > improvements/features/design. I > > >> could also attempt to get one or two members of google's > kubernetes > > >>> team > > > to > > >> join to discuss best practices. > > >> > > >> I'm currently thinking that next Thursday at 11AM PST over > zoom.us, > > > though > > >> if there's strong opinions otherwise I'd be glad to propose other > > >>> times. > > >> > > >> Cheers! > > >> > > >> Daniel > > >> > > > > > >>> > > >> > > > > >
Re: Airflow + Kubernetes discussion
Glad to hear that people are interested! I've created a google calendar event and messaged everyone in this thread, if anyone else would like to join please let me know! On Thu, Jul 20, 2017 at 1:04 PM Bolke de Bruin wrote: > Invite would be nice, I will try to join! > > > On 20 Jul 2017, at 20:36, Gerard Toonstra wrote: > > > > send me an invite too! > > > > On Thu, Jul 20, 2017 at 8:17 PM, Jeremiah Lowin > wrote: > > > >> I'm interested as well. > >> > >> On Thu, Jul 20, 2017 at 1:51 PM Marc Bollinger > wrote: > >> > >>> +1 We're in the middle of moving some services to k8s, and have had our > >>> eye on Airflow. > >>> > On Jul 20, 2017, at 10:37 AM, Sumit Maheshwari < > sumeet.ma...@gmail.com > >>> > >>> wrote: > > I would join as well for sure. > > Thanks, > Sumit Maheshwari > cell. 9632202950 > > > On Thu, Jul 20, 2017 at 11:00 PM, Chris Riccomini < > >> criccom...@apache.org > > wrote: > > > I would definitely be up to joining. We're interested in the K8s work > > that's going on. That time works for me. > > > > On Thu, Jul 20, 2017 at 9:54 AM, Daniel Imberman < > > daniel.imber...@gmail.com> > > wrote: > > > >> Hello everyone, > >> > >> Recently there's been a fair amount of discussion regarding the > > integration > >> of airflow with kubernetes. If there is interest I would love to > host > >>> an > >> e-meeting to discuss this integration. I can go over the > architecture > >>> as > > it > >> stands right now and would love feedback on > > improvements/features/design. I > >> could also attempt to get one or two members of google's kubernetes > >>> team > > to > >> join to discuss best practices. > >> > >> I'm currently thinking that next Thursday at 11AM PST over zoom.us, > > though > >> if there's strong opinions otherwise I'd be glad to propose other > >>> times. > >> > >> Cheers! > >> > >> Daniel > >> > > > >>> > >> > >
Re: Airflow + Kubernetes discussion
Invite would be nice, I will try to join! > On 20 Jul 2017, at 20:36, Gerard Toonstra wrote: > > send me an invite too! > > On Thu, Jul 20, 2017 at 8:17 PM, Jeremiah Lowin wrote: > >> I'm interested as well. >> >> On Thu, Jul 20, 2017 at 1:51 PM Marc Bollinger wrote: >> >>> +1 We're in the middle of moving some services to k8s, and have had our >>> eye on Airflow. >>> On Jul 20, 2017, at 10:37 AM, Sumit Maheshwari >> >>> wrote: I would join as well for sure. Thanks, Sumit Maheshwari cell. 9632202950 On Thu, Jul 20, 2017 at 11:00 PM, Chris Riccomini < >> criccom...@apache.org wrote: > I would definitely be up to joining. We're interested in the K8s work > that's going on. That time works for me. > > On Thu, Jul 20, 2017 at 9:54 AM, Daniel Imberman < > daniel.imber...@gmail.com> > wrote: > >> Hello everyone, >> >> Recently there's been a fair amount of discussion regarding the > integration >> of airflow with kubernetes. If there is interest I would love to host >>> an >> e-meeting to discuss this integration. I can go over the architecture >>> as > it >> stands right now and would love feedback on > improvements/features/design. I >> could also attempt to get one or two members of google's kubernetes >>> team > to >> join to discuss best practices. >> >> I'm currently thinking that next Thursday at 11AM PST over zoom.us, > though >> if there's strong opinions otherwise I'd be glad to propose other >>> times. >> >> Cheers! >> >> Daniel >> > >>> >>
Re: Passing Args to Python Operator
The correct way to pass kwargs is through op_kwargs.
for log_type in LOG_TYPES:
raw_log_py_operator = PythonOperator(
task_id='{}-logs-copy-to-s3'.format(log_type),
provide_context=True,
python_callable=raw_log_callable,
dag=raw_log_dag,
op_kwargs={'log_type': log_type},
)
See the PythonOperator example,
https://github.com/apache/incubator-airflow/blob/1.8.1/airflow/example_dags/example_python_operator.py#L54
.
On Thu, Jul 20, 2017 at 11:26 AM, Kevin Pamplona wrote:
> I'm trying to pass a custom arg to a python operator, but when trying to
> resolve *kwargs.get('log_type')*, I get 'None'.
>
> Anyone have any issues with this before?
>
> *LOG_TYPES = ['abc_log', 'xyz_log']*
>
> def raw_log_callable(ds, **kwargs):
> exec_date = str(kwargs.get('task_instance').execution_date.date() -
> timedelta(days=1)).split()[0]
> task_logic.main({'' : exec_date, '' :
> *kwargs.get('log_type')*})
>
>
> for log_type in LOG_TYPES:
> raw_log_py_operator = PythonOperator(
> task_id='{}-logs-copy-to-s3'.format(log_type),
> provide_context=True,
> python_callable=raw_log_callable,
> dag=raw_log_dag,
> *log_type=log_type*
> )
>
Re: Airflow + Kubernetes discussion
send me an invite too! On Thu, Jul 20, 2017 at 8:17 PM, Jeremiah Lowin wrote: > I'm interested as well. > > On Thu, Jul 20, 2017 at 1:51 PM Marc Bollinger wrote: > > > +1 We're in the middle of moving some services to k8s, and have had our > > eye on Airflow. > > > > > On Jul 20, 2017, at 10:37 AM, Sumit Maheshwari > > > wrote: > > > > > > I would join as well for sure. > > > > > > Thanks, > > > Sumit Maheshwari > > > cell. 9632202950 > > > > > > > > > On Thu, Jul 20, 2017 at 11:00 PM, Chris Riccomini < > criccom...@apache.org > > > > > > wrote: > > > > > >> I would definitely be up to joining. We're interested in the K8s work > > >> that's going on. That time works for me. > > >> > > >> On Thu, Jul 20, 2017 at 9:54 AM, Daniel Imberman < > > >> daniel.imber...@gmail.com> > > >> wrote: > > >> > > >>> Hello everyone, > > >>> > > >>> Recently there's been a fair amount of discussion regarding the > > >> integration > > >>> of airflow with kubernetes. If there is interest I would love to host > > an > > >>> e-meeting to discuss this integration. I can go over the architecture > > as > > >> it > > >>> stands right now and would love feedback on > > >> improvements/features/design. I > > >>> could also attempt to get one or two members of google's kubernetes > > team > > >> to > > >>> join to discuss best practices. > > >>> > > >>> I'm currently thinking that next Thursday at 11AM PST over zoom.us, > > >> though > > >>> if there's strong opinions otherwise I'd be glad to propose other > > times. > > >>> > > >>> Cheers! > > >>> > > >>> Daniel > > >>> > > >> > > >
Passing Args to Python Operator
I'm trying to pass a custom arg to a python operator, but when trying to
resolve *kwargs.get('log_type')*, I get 'None'.
Anyone have any issues with this before?
*LOG_TYPES = ['abc_log', 'xyz_log']*
def raw_log_callable(ds, **kwargs):
exec_date = str(kwargs.get('task_instance').execution_date.date() -
timedelta(days=1)).split()[0]
task_logic.main({'' : exec_date, '' :
*kwargs.get('log_type')*})
for log_type in LOG_TYPES:
raw_log_py_operator = PythonOperator(
task_id='{}-logs-copy-to-s3'.format(log_type),
provide_context=True,
python_callable=raw_log_callable,
dag=raw_log_dag,
*log_type=log_type*
)
Re: Airflow + Kubernetes discussion
I'm interested as well. On Thu, Jul 20, 2017 at 1:51 PM Marc Bollinger wrote: > +1 We're in the middle of moving some services to k8s, and have had our > eye on Airflow. > > > On Jul 20, 2017, at 10:37 AM, Sumit Maheshwari > wrote: > > > > I would join as well for sure. > > > > Thanks, > > Sumit Maheshwari > > cell. 9632202950 > > > > > > On Thu, Jul 20, 2017 at 11:00 PM, Chris Riccomini > > > wrote: > > > >> I would definitely be up to joining. We're interested in the K8s work > >> that's going on. That time works for me. > >> > >> On Thu, Jul 20, 2017 at 9:54 AM, Daniel Imberman < > >> daniel.imber...@gmail.com> > >> wrote: > >> > >>> Hello everyone, > >>> > >>> Recently there's been a fair amount of discussion regarding the > >> integration > >>> of airflow with kubernetes. If there is interest I would love to host > an > >>> e-meeting to discuss this integration. I can go over the architecture > as > >> it > >>> stands right now and would love feedback on > >> improvements/features/design. I > >>> could also attempt to get one or two members of google's kubernetes > team > >> to > >>> join to discuss best practices. > >>> > >>> I'm currently thinking that next Thursday at 11AM PST over zoom.us, > >> though > >>> if there's strong opinions otherwise I'd be glad to propose other > times. > >>> > >>> Cheers! > >>> > >>> Daniel > >>> > >> >
Re: Airflow + Kubernetes discussion
+1 We're in the middle of moving some services to k8s, and have had our eye on Airflow. > On Jul 20, 2017, at 10:37 AM, Sumit Maheshwari wrote: > > I would join as well for sure. > > Thanks, > Sumit Maheshwari > cell. 9632202950 > > > On Thu, Jul 20, 2017 at 11:00 PM, Chris Riccomini > wrote: > >> I would definitely be up to joining. We're interested in the K8s work >> that's going on. That time works for me. >> >> On Thu, Jul 20, 2017 at 9:54 AM, Daniel Imberman < >> daniel.imber...@gmail.com> >> wrote: >> >>> Hello everyone, >>> >>> Recently there's been a fair amount of discussion regarding the >> integration >>> of airflow with kubernetes. If there is interest I would love to host an >>> e-meeting to discuss this integration. I can go over the architecture as >> it >>> stands right now and would love feedback on >> improvements/features/design. I >>> could also attempt to get one or two members of google's kubernetes team >> to >>> join to discuss best practices. >>> >>> I'm currently thinking that next Thursday at 11AM PST over zoom.us, >> though >>> if there's strong opinions otherwise I'd be glad to propose other times. >>> >>> Cheers! >>> >>> Daniel >>> >>
Re: Airflow + Kubernetes discussion
I would join as well for sure. Thanks, Sumit Maheshwari cell. 9632202950 On Thu, Jul 20, 2017 at 11:00 PM, Chris Riccomini wrote: > I would definitely be up to joining. We're interested in the K8s work > that's going on. That time works for me. > > On Thu, Jul 20, 2017 at 9:54 AM, Daniel Imberman < > daniel.imber...@gmail.com> > wrote: > > > Hello everyone, > > > > Recently there's been a fair amount of discussion regarding the > integration > > of airflow with kubernetes. If there is interest I would love to host an > > e-meeting to discuss this integration. I can go over the architecture as > it > > stands right now and would love feedback on > improvements/features/design. I > > could also attempt to get one or two members of google's kubernetes team > to > > join to discuss best practices. > > > > I'm currently thinking that next Thursday at 11AM PST over zoom.us, > though > > if there's strong opinions otherwise I'd be glad to propose other times. > > > > Cheers! > > > > Daniel > > >
Re: Role Based Access Control for Airflow UI
Hey Max, Want to come down to WePay? We can set up a zoom for those that want to join online, and record it as well to post for the community. Since Joy is just getting started, and it looks like there's going to be a K8s discussion next week, maybe we can shoot for the week after (the week of the 31st of July)? Care to float a few times that week? Cheers, Chris On Thu, Jul 20, 2017 at 9:31 AM, Maxime Beauchemin < maximebeauche...@gmail.com> wrote: > Sounds awesome, count me in! > > * check out the prototype in my fork, I went far enough to hit some > hurdles, try different workarounds. I hooked up the Airflow Bootstrap > template too so that we feel at home in this new UI > * using a single `id` field is a requirement for FAB that airflow doesn't > respect (composite pks), either we add the feature to support that in FAB, > or we align on the Airflow side and modify the models and add a migration > script. This upgrade would require downtime and might be annoying to the > Airflow community, but could help with db performance a bit (smaller > index)... I probably could be convinced either way but I'm leaning on > improving FAB > * I'm a maintainer for FAB so I can help get stuff through there > * React is in limbo at the ASF for licensing reasons, so no React at least > for now > * npm/webpack/ES6, javascript only in `.js` files > * I vote for eslint + eslint-config-airbnb as a set of linting rules for JS > * Keep out of apache (for now), this new app ships as its own pypi package > `airflow-webserver`, have a period of overlap (maintaining 2 web apps) > before ripping out `airflow/www` from the core package > * You need to get in touch with Marty Kausas, an intern at Airbnb who's > been working on a Flask blueprint for improved, more personalized views on > DAGs that we were planning on merging into the main branch eventually. Some > of Marty's idea and code could be merged into this effort. > > These are ideas on how I would proceed personally on this but definitely > everything here is up for discussion. > > Let's meet physically at either WePay or Airbnb. Folks from the community, > let us know on this thread if you want to be part of this effort, we'll be > happy to include you. > > Thanks, > > Max > > On Wed, Jul 19, 2017 at 7:33 PM, Joy Gao wrote: > > > Hey everyone, > > > > I recently transferred to Data Infra team here at WePay to focus on > > Airflow-related initiatives. > > > > Given the RBAC design is mostly hashed out, I'm happy to get this feature > > off the ground for Q3, starting with converting Airflow to Fab, if there > > are no objections. > > > > Cheers, > > Joy > > > > On Thu, Jun 29, 2017 at 7:32 AM, Gurer Kiratli < > > gurer.kira...@airbnb.com.invalid> wrote: > > > > > Hey all, > > > > > > We talked about this internally. We would like to work on this feature > > but > > > given the immediate priorities we are not going to be working on it in > > Q3. > > > Comes end of Q3 we will reevaluate. Likely scenario is we can work on > it > > > late Q4 or Q12018. > > > > > > Cheers, > > > > > > Gurer > > > > > > On Tue, Jun 27, 2017 at 8:08 AM, Chris Riccomini < > criccom...@apache.org> > > > wrote: > > > > > > > I think FAB sounds like the right approach. Waiting to hear back with > > > notes > > > > on AirBNB H2 discussion to see if they want to take this up. > > > > > > > > @Gurer, any idea when this will happen? > > > > > > > > On Thu, Jun 22, 2017 at 1:00 AM, Bolke de Bruin > > > wrote: > > > > > > > > > One downside I see from FAB is that is does not do Business Role > > > mapping > > > > > to FAB role. I would prefer to create groups in IPA/LDAP/AD and > have > > > > those > > > > > map to FAB roles instead of needing to manage that in FAB. > > > > > > > > > > B. > > > > > > > > > > > On 22 Jun 2017, at 09:36, Bolke de Bruin > > wrote: > > > > > > > > > > > > Hi Guys, > > > > > > > > > > > > Thanks for putting the thinking in! It is about time that we get > > this > > > > > moving. > > > > > > > > > > > > The design looks pretty sound. One can argue about the different > > > roles > > > > > that are required, but that will be situation dependent I guess. > > > > > > > > > > > > Implementation wise I would argue together with Max that FAB is a > > > > better > > > > > or best fit. The ER model that is being described is pretty much a > > copy > > > > of > > > > > a normal security model. So a reimplementation of that is 1) > > > significant > > > > > duplication of effort and 2) bound to have bugs that have been > solved > > > in > > > > > the other framework. Moreover, FAB does have integration out of the > > box > > > > > with some enterprisey systems like IPA, ActiveDirectory, and LDAP. > > > > > > > > > > > > So while you argue that using FAB would increase the scope of the > > > > > proposal significantly, but I think that is not true. Using FAB > would > > > > allow > > > > > you to focus on what kind of out-of-the-box permission sets and > roles > > > we > > > > > would
Re: Airflow + Kubernetes discussion
I would definitely be up to joining. We're interested in the K8s work that's going on. That time works for me. On Thu, Jul 20, 2017 at 9:54 AM, Daniel Imberman wrote: > Hello everyone, > > Recently there's been a fair amount of discussion regarding the integration > of airflow with kubernetes. If there is interest I would love to host an > e-meeting to discuss this integration. I can go over the architecture as it > stands right now and would love feedback on improvements/features/design. I > could also attempt to get one or two members of google's kubernetes team to > join to discuss best practices. > > I'm currently thinking that next Thursday at 11AM PST over zoom.us, though > if there's strong opinions otherwise I'd be glad to propose other times. > > Cheers! > > Daniel >
Airflow + Kubernetes discussion
Hello everyone, Recently there's been a fair amount of discussion regarding the integration of airflow with kubernetes. If there is interest I would love to host an e-meeting to discuss this integration. I can go over the architecture as it stands right now and would love feedback on improvements/features/design. I could also attempt to get one or two members of google's kubernetes team to join to discuss best practices. I'm currently thinking that next Thursday at 11AM PST over zoom.us, though if there's strong opinions otherwise I'd be glad to propose other times. Cheers! Daniel
Re: Role Based Access Control for Airflow UI
Sounds awesome, count me in! * check out the prototype in my fork, I went far enough to hit some hurdles, try different workarounds. I hooked up the Airflow Bootstrap template too so that we feel at home in this new UI * using a single `id` field is a requirement for FAB that airflow doesn't respect (composite pks), either we add the feature to support that in FAB, or we align on the Airflow side and modify the models and add a migration script. This upgrade would require downtime and might be annoying to the Airflow community, but could help with db performance a bit (smaller index)... I probably could be convinced either way but I'm leaning on improving FAB * I'm a maintainer for FAB so I can help get stuff through there * React is in limbo at the ASF for licensing reasons, so no React at least for now * npm/webpack/ES6, javascript only in `.js` files * I vote for eslint + eslint-config-airbnb as a set of linting rules for JS * Keep out of apache (for now), this new app ships as its own pypi package `airflow-webserver`, have a period of overlap (maintaining 2 web apps) before ripping out `airflow/www` from the core package * You need to get in touch with Marty Kausas, an intern at Airbnb who's been working on a Flask blueprint for improved, more personalized views on DAGs that we were planning on merging into the main branch eventually. Some of Marty's idea and code could be merged into this effort. These are ideas on how I would proceed personally on this but definitely everything here is up for discussion. Let's meet physically at either WePay or Airbnb. Folks from the community, let us know on this thread if you want to be part of this effort, we'll be happy to include you. Thanks, Max On Wed, Jul 19, 2017 at 7:33 PM, Joy Gao wrote: > Hey everyone, > > I recently transferred to Data Infra team here at WePay to focus on > Airflow-related initiatives. > > Given the RBAC design is mostly hashed out, I'm happy to get this feature > off the ground for Q3, starting with converting Airflow to Fab, if there > are no objections. > > Cheers, > Joy > > On Thu, Jun 29, 2017 at 7:32 AM, Gurer Kiratli < > gurer.kira...@airbnb.com.invalid> wrote: > > > Hey all, > > > > We talked about this internally. We would like to work on this feature > but > > given the immediate priorities we are not going to be working on it in > Q3. > > Comes end of Q3 we will reevaluate. Likely scenario is we can work on it > > late Q4 or Q12018. > > > > Cheers, > > > > Gurer > > > > On Tue, Jun 27, 2017 at 8:08 AM, Chris Riccomini > > wrote: > > > > > I think FAB sounds like the right approach. Waiting to hear back with > > notes > > > on AirBNB H2 discussion to see if they want to take this up. > > > > > > @Gurer, any idea when this will happen? > > > > > > On Thu, Jun 22, 2017 at 1:00 AM, Bolke de Bruin > > wrote: > > > > > > > One downside I see from FAB is that is does not do Business Role > > mapping > > > > to FAB role. I would prefer to create groups in IPA/LDAP/AD and have > > > those > > > > map to FAB roles instead of needing to manage that in FAB. > > > > > > > > B. > > > > > > > > > On 22 Jun 2017, at 09:36, Bolke de Bruin > wrote: > > > > > > > > > > Hi Guys, > > > > > > > > > > Thanks for putting the thinking in! It is about time that we get > this > > > > moving. > > > > > > > > > > The design looks pretty sound. One can argue about the different > > roles > > > > that are required, but that will be situation dependent I guess. > > > > > > > > > > Implementation wise I would argue together with Max that FAB is a > > > better > > > > or best fit. The ER model that is being described is pretty much a > copy > > > of > > > > a normal security model. So a reimplementation of that is 1) > > significant > > > > duplication of effort and 2) bound to have bugs that have been solved > > in > > > > the other framework. Moreover, FAB does have integration out of the > box > > > > with some enterprisey systems like IPA, ActiveDirectory, and LDAP. > > > > > > > > > > So while you argue that using FAB would increase the scope of the > > > > proposal significantly, but I think that is not true. Using FAB would > > > allow > > > > you to focus on what kind of out-of-the-box permission sets and roles > > we > > > > would need and maybe address some issues that FAB lacks (maybe how to > > > deal > > > > with non web access - ie. in DAGs, maybe Kerberos, probably how to > deal > > > > with API calls that are not CRUD). Implementation wise it probably > > > > simplifies what we need to do. Maybe - using Max’s early POC as an > > > example > > > > - we can slowly move over? > > > > > > > > > > On a side note: Im planning to hire 2-3 ppl to work on Airflow > coming > > > > year. Improvement of Security, Enterprise Integration, Revamp UI are > on > > > the > > > > todo list. However, this is not confirmed yet as business priorities > > > might > > > > change. > > > > > > > > > > Bolke. > > > > > > > > > > > > > > >> On 15 Jun 2017, at 2
Re: Deploy procedure for new/modify dags
Hello Arthur, Thanks for your help, In your case I will have to update worker code, not necessarily webserver/scheduler and I will set the option --ship_dag to False. This deployment method imply that I have to pause all my dags, wait my queue is empty and restart my worker to pull and install the new code and dependencies. I have some external dependencies which take time to pip install so my service won’t be available during this time. Am I correct in assuming this? I discovered that we can specify the queue where the scheduler push the tasks and the worker listen to. Can it be a viable solution to create a queue for each commit, to deploy a new set of workers for each commit and to kill the old one when they don’t have anything anymore in their old queue? Germain T. On 19/07/17 07:53, "Arthur Wiedmer" wrote: Hi Germain, As long as the structure of the DAG is not changed (tasks are the same and the dependency graph does not change), there should be no need to restart anything. The scheduler only needs the structure of the DAG to send the right message to celery. Essentially the message tells the worker to run an airflow run command for this dag_id, this task_id and the execution_date. While the webserver for instance might show you an older version of the bash script, the code executed will be the latest available on the worker. You should be able to check this by checking the logs for the task, since the script is usually logged there. I hope this helps, Sincerely, Arthur On Mon, Jul 17, 2017 at 11:56 PM, Germain TANGUY < germain.tan...@dailymotion.com> wrote: > Hello everybody, > > I would like to know what are your procedure to deploy new versions of > your DAGs, especially for dags that have external dependencies (bash > script..etc) > I use CeleryExecutor with multiples workers and so there is an issue of > consistency between workers, schedulers and webserver. > > Today I pause the dags, I wait until all running tasks complete, I restart > all airflow services and unpause the dags. Is there a better way? > > Best regards, > > Germain T. > >
