I think there are absolutely use-cases for both. I’m totally fine with saying
“for small/medium use-cases, we come with an in-house system. However for
larger cases, you’ll require spark/Flink/S3. That’s totally in line with PLENTY
of use-cases. This would be especially cool when matched with fa
I believe - for not large data - the direct runner is wholly doable, which
seems in line with airflow patterns. I have, and have spoken with several
others that have, been productive with that runner.
For much larger transfers, the generic operator could accept parameters for
submitting the compu
Thanks, Daniel. The PR was merged:
https://github.com/apache/airflow/pull/10568/files
Best wishes
Ping Zhang
On Mon, Aug 24, 2020 at 9:31 AM Daniel Imberman
wrote:
> Hi Ping,
>
> I think that’s a great idea! Would be glad to help merge this.
>
> via Newton Mail [
> https://cloudmagic.com/k/d/
Very closely related issue https://github.com/apache/airflow/issues/9506
On 5 September 2020 08:01:11 BST, Jarek Potiuk wrote:
>And we have a new addition from Kamil about the need to extend slightly
>plugin mechanism to be able to cover dynamically "Connections",
>"Connection
>Form" and "Extra L
And we have a new addition from Kamil about the need to extend slightly
plugin mechanism to be able to cover dynamically "Connections", "Connection
Form" and "Extra Links" - those are indeed the "core -> Providers"
dependencies that we still have.
They seem to be easy to handle by making provider
