Re: [DISCUSS] Proposal for adding Telemetry via Scarf

+1, looking forward to having better data!

Re: [DISCUSS] Proposal for adding Telemetry via Scarf

Once we decide to go ahead with this, I think it might be worth for us to also check if we can export metrics regarding which all providers are getting used for the user deployments. That I believe would help us understand the adoption of the provider and would also help in our decision making

Re: [DISCUSS] Proposal for adding Telemetry via Scarf

The webserver is packaged after compiling, so that won't be possible Michal. On Tue, 9 Apr 2024 at 11:02, Michał Modras wrote: > If it is packaged and installed by default, we add the dependency (and its > dependencies) to Airflow's already-not-small dependency tree. If we make it > installed

Re: [DISCUSS] Proposal for adding Telemetry via Scarf

If it is packaged and installed by default, we add the dependency (and its dependencies) to Airflow's already-not-small dependency tree. If we make it installed and enabled by default, would there be an easy way to not just switch it off (e.g. through the env variable), but also not package it at

Re: [DISCUSS] Proposal for adding Telemetry via Scarf

> Other than that I don't mind it being e.g. optional provider. I don't think it is possible to implement it in a provider because it is a js package installed on the webserver; we could implement it as a plugin (Blueprint), but in this case, the user must make an effort to register it. It would

Re: [DISCUSS] Proposal for adding Telemetry via Scarf

My 2 cents: it must be possible to opt-out, preferably it should be possible to deploy Airflow instances without bundling the telemetry library dependencies. Other than that I don't mind it being e.g. optional provider. śr., 3 kwi 2024, 22:42 użytkownik Hussein Awala napisał: > > I'd like to

Re: [DISCUSS] Proposal for adding Telemetry via Scarf

> I'd like to propose, that we start with collecting simple data with limited access: to all the PMC members. We can always expand it to Committers and then expand further to make it invite-only or setup exporting it to a DB like Postgres and

Re: [DISCUSS] Proposal for adding Telemetry via Scarf

Yup, exactly. I believe this would definitely help us take early and informed decisions. > E.g. Had we had this earlier, I believe it would have definitely helped us > more for our past discussions like whether we should continue supporting >

Re: [DISCUSS] Proposal for adding Telemetry via Scarf

+1 to introduce this. I believe this would definitely help us take early and informed decisions. E.g. Had we had this earlier, I believe it would have definitely helped us more for our past discussions like whether we should continue supporting

Re: [DISCUSS] Proposal for adding Telemetry via Scarf

Yup, I had added a link to scarf docs in the original email that referenced opting out and we should even add an Airflow config that puts all config in a single place. Without it we can’t be compliant to all the policies even if we collectively ignore or are unaware of the importance of it.

Re: [DISCUSS] Proposal for adding Telemetry via Scarf

I mentioned opting out just to confirm its importance, and after checking the Scarf documentation it appears to be supported natively by Scarf. For data accessibility, my point was more about raw data, not just aggregated information/insights shared via monthly newsletters, as we do for Airflow

Re: [DISCUSS] Proposal for adding Telemetry via Scarf

Agreed to both your points Hussein but both the points are already covered in my original discussion post - both about opting out and providing data to all the PMC members and providing visibility via Monthly newsletters. Is there anything else you propose to discuss that isn’t covered? On Mon,

Re: [DISCUSS] Proposal for adding Telemetry via Scarf

+1 for the idea in general, but there are two main points to discuss before voting on this: 1. We should provide an option to disable Scarf: As Airflow is not a paid product, we cannot force companies to report their use of this project. Otherwise, some may choose to create their own fork just to

Re: [DISCUSS] Proposal for adding Telemetry via Scarf

Big +1 for Scarf. Transparency is key, so it's important to be super clear about opting out and what's tracked to avoid spooking anyone about IP stuff. Regards Ankit Chaurasia On Mon, Apr 1, 2024 at 10:18 AM Amogh Desai wrote: > > +1 looks like a good tool which could be super helpful. > >

Re: [DISCUSS] Proposal for adding Telemetry via Scarf

+1 looks like a good tool which could be super helpful. * We should have some transparency into the data that is collected or sent * We should have an option to optionally opt-out Thanks & Regards, Amogh Desai On Sun, Mar 31, 2024 at 7:53 AM Wei Lee wrote: > +1 to this. It would be really

Re: [DISCUSS] Proposal for adding Telemetry via Scarf

+1 to this. It would be really useful. As long as we can opt out, I think we’re good. Best, Wei > On Mar 31, 2024, at 12:47 AM, Kaxil Naik wrote: > > Grammar Correction: > > We should assume that those who deploy and upgrade Airflow - actually read >> and take into account what is written in

Re: [DISCUSS] Proposal for adding Telemetry via Scarf

Grammar Correction: We should assume that those who deploy and upgrade Airflow - actually read > and take into account what is written in the release notes - especially if > they have security guys breathing their necks, similarly as we have to > assume they follow CVE announcements about

Re: [DISCUSS] Proposal for adding Telemetry via Scarf

Grammar crrection: On Sat, 30 Mar 2024 at 16:43, Kaxil Naik wrote: > Have this at the end of the email too: but if folks don't read until the > end and quoting Maxime from the use-case blog[1]: > > "I think people often ask ‘how do I contribute to open source?’, ‘I've got > to get into the

Re: [DISCUSS] Proposal for adding Telemetry via Scarf

Have this at the end of the email too: but if folks don't read until the end and quoting Maxime from the use-case blog[1]: "I think people often ask ‘how do I contribute to open source?’, ‘I've got to get into the code’, or ‘ I’ve got to be an engineer.’ Actually, the very simplest thing that you

Re: [DISCUSS] Proposal for adding Telemetry via Scarf

Hi Jarek! I understand the reasons for opt-out from a project view. I just suddenly imagined the situation when an upgrade happens and here comes the data to some third party service - that's a view from a user side of some big company. There could be good alternatives to handle this: 1. Short

Re: [DISCUSS] Proposal for adding Telemetry via Scarf

Hello everyone, it has to be: 1. Opt-in by default to not trigger security guys about new unplanned > activity after regular upgrade. > That's a very good point about security triggering Alexander, but I am not so sure it means that we "have to" do opt-in. There are other ways of communicating

Re: [DISCUSS] Proposal for adding Telemetry via Scarf

-0 But it has to be: 1. Opt-in by default to not trigger security guys about new unplanned activity after regular upgrade. 2. It should not affect anything if access to the internet is restricted which is default for many companies. 3. Be transparent about what it will send to for review.

Re: [DISCUSS] Proposal for adding Telemetry via Scarf

+1 I agree with this proposal. On Sat, 30 Mar 2024, 11:34 Aritra Basu, wrote: > +1 yeah, this sounds useful to me > > -- > Regards, > Aritra Basu > > On Sat, Mar 30, 2024, 6:58 AM Jarek Potiuk wrote: > > > +1. All that sounds reasonable, there are precedents, ASF supports Scarf > > officially.

Re: [DISCUSS] Proposal for adding Telemetry via Scarf

+1 yeah, this sounds useful to me -- Regards, Aritra Basu On Sat, Mar 30, 2024, 6:58 AM Jarek Potiuk wrote: > +1. All that sounds reasonable, there are precedents, ASF supports Scarf > officially. Would be great to have access to such telemetry data. > > On Sat, Mar 30, 2024 at 1:18 AM Kaxil

Re: [DISCUSS] Proposal for adding Telemetry via Scarf

+1. All that sounds reasonable, there are precedents, ASF supports Scarf officially. Would be great to have access to such telemetry data. On Sat, Mar 30, 2024 at 1:18 AM Kaxil Naik wrote: > Hi all, > > I want to propose gathering telemetry for Airflow installations. As the > Airflow community,