You are absolutely right that it will be very hard to "fix" the SLA as it stands while keeping backwards compatibility. My thoughts on the new / updated SLA mechanism will NOT be compatible, but will address the core need for data timeliness.
Hmm, that's an interesting thought on "marking it as deprecated". However, I'm not comfortable with that at the moment. On Wed, Jul 13, 2022 at 8:57 PM Jarek Potiuk <ja...@potiuk.com> wrote: > Cool to see that people still need SLA in Airflow. I see the points > why and they make perfect sense. Glad we discussed it :) > > Still - WDYT do we want to signal current SLA as "deprecated" now > (even if we do not know what will replace it?) - at least as a signal > to stay away from it. > Or do we want to keep the status quo? > > I think it will be hard to "fix" the SLA as it stands now without > breaking a lot of compatibility so we can deprecate it even now (and > we can raise warnings and mention that it will be replaced by > something better). I believe - for example - this was a mistake not > doing that with "experimental" API early enough which got more people > using it and harder to make them stop. > > J. > > On Wed, Jul 13, 2022 at 6:51 PM Vikram Koka > <vik...@astronomer.io.invalid> wrote: > > > > - resending below to keep the same thread as Ping's response. My prior > response and Ping's were sent at the same time, but I did not two two email > threads -- > > > > > > I understand the frustration with the SLA feature as it stands. > > I struggled with trying to understand this early on and finally > understood how they were broken. > > > > Having said that, I believe that Airflow users strongly care about the > timeliness and consequently SLAs of their data. I also believe that these > need to be redefined on top of Datasets in AIP-48 (as Ash mentioned > earlier). > > > > In summary, my view is that this needs to be redefined and fixed in > Airflow, rather than entirely removed from Airflow. > > Having said that, I also believe that people will want to build advanced > SLA capabilities on top of Airflow, but that's completely reasonable in my > book. > > > > On Wed, Jul 13, 2022 at 8:31 PM Ping Zhang <pin...@umich.edu> wrote: > >> > >> Hi Jarek, > >> > >> Thanks for bringing this up. > >> > >> I agree the SLA feature needs some work. However, I think we want an > equivalent SLA feature as it is still very useful. > >> > >> Thanks, > >> > >> Ping > >> > >> > >> On Tue, Jul 12, 2022 at 12:42 AM Jarek Potiuk <ja...@potiuk.com> wrote: > >>> > >>> Hey everyone, > >>> > >>> I keep on being involved in discussions where people are complaining > >>> about how bad and useless the SLA feature of Airflow is. And yeah, I > >>> pretty much agree with it. > >>> > >>> Without getting into details of why it is bad - should we possibly > >>> just, well, deprecate it? I think that would give a much stronger > >>> signal to our users if they keep on getting warnings that the feature > >>> is deprecated and when we officially deprecate it in the docs that > >>> they should not rely on it. > >>> > >>> I also think that possibly we do not have to replace it with an > >>> equivalent/better SLA feature. > >>> > >>> I personally think Airflow on its own should not provide such > >>> SLA/monitoring features, but it should become more of the platform > >>> that provides useful metrics that will enable other - more dedicated > >>> systems - to do the job of monitoring and alerting - and the native > >>> Airflow UI should be more of a "management" than "monitoring". > >>> > >>> With (already approved) Open Telemetry support > >>> > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-49+OpenTelemetry+Support+for+Apache+Airflow > >>> integrating such monitoring solution should become much easier. Also > >>> with possible newer and more sophisticated metrics (such as those > >>> proposed by Ping in > >>> https://lists.apache.org/thread/g52vk2p7l4nf6on436mbdzwrqstld7jl ) > >>> this opens up to more sophisticated usages, that Airflow will never be > >>> able to match with built-in SLA/monitoring features. > >>> > >>> Also even today there are better ways to achieve SLA functionality - > >>> Good and successful story about it has been told by Eden from Fyber at > >>> the Summit: > https://airflowsummit.org/sessions/2022/the-slayer-your-data-pipeline-needs/ > >>> > >>> Making SLA deprecate would give a signal to the users that this is the > >>> long-term, recommended approach. > >>> > >>> J. >
