We cleary want to position ourselves as the second group. Not that we don't
write python code, as a fact we do, but once we see something could be solved
in a generic way and be contributed back to Airflow, we will also propose that.
I understand the use of hooks principle, we also do it, when
Hello Pavan Kumar,
I really like your proposition as this would have facilitated my implementation
of the MSGraphSensor, which is actually implemented as a deferrable using a
triggerer.
You can check here how I did it:
https://github.com/apache/airflow/blob/main/airflow/providers/microsoft/azu
The MSGraphAsyncOperator already handles multi paged responses internally using
the defer mechanism, but the result of this are indexed XComs, which means you
have to expand over them to process it. We sometimes also use jq in
combination with BashOperator to merge those multiple JSON responses
The example I give was a simplified version, and also a continuation of another
DAG process.
The issue I tried to solve in Airflow here for this case (we have also other
use cases where we ran into the same issue) was reading n number of users from
MSGraph, which where updated and had to be syn
Hello,
At our company we have DAG's which have to process lots of paged results (e.g.
XComs results from REST endpoints or multiple files from FTP downloads).
The Xcom isn't the issue, as we use a custom provider which allows us to store
large data on a Persistent Volume Claim without overloadi
Beside the new interface and getting rid off FAB in Airflow 3.0, a cool and
handy feature would be to be able to group multiple DAG's so you could order
them by like domain or whatever grouping you want to achieve.
Okay, you can achieve the same with filtering, and maybe the we could use that
fe
with tests, docs etc. is enough.
On Mon, Mar 11, 2024 at 6:27 PM Blain David wrote:
>
> Hello Jarek,
>
> There is no particular need to add this into a separate provider, I just did
> it as I wanted to deploy it myself, it could perfectly reside within the
> Microsoft Azu
under the
impression there is something special about the SDK you use or
"proprietaredness" (for lack of a better word) - but that seems like
yet-another operator, hook, triggerer in `microsoft.azure`.
Or am I missing something?
J.
On Fri, Mar 1, 2024 at 8:57 AM Blain David w
It's certainly possible to check from where a python method is being called
using traceback.
I do think prohibiting the execute method of an operator being called manually
would be a good idea, I've also came accross this in multiple DAG's and this is
ugly and looks like a hack.
Maybe in the be
> dP4HLfDYw1HeMfylfZkEv0p%2BMO4X3Sn6QJu8hU%3D&reserved=0
> My main concern here is how will provide ongoing mantaince for this
> provider?
> This provider is to handle a service by Microsoft yet Microsoft is not
> in the picture here (as far as I can see)
>
>
> On Sat
Hello everyone,
I've already started a discussion about this on the Airflow discussions:
https://github.com/apache/airflow/discussions/36315
As we have multiple DAG's interacting with MS Graph API endpoints, and as we
want to avoid custom code as much as possible as we have to handle lot's of
11 matches
Mail list logo
