ely use-cases for both. I’m totally
> fine
> > > > >> >with saying “for small/medium use-cases, we come with an in-house
> > > > >> >system. However for larger cases, you’ll require spark/Flink/S3.
> > > > >That’s
> > > > >
ystem. However for larger cases, you’ll require spark/Flink/S3.
> > > >That’s
> > > >> >totally in line with PLENTY of use-cases. This would be especially
> > > >cool
> > > >> >when matched with fast-follow as we could EVEN potentially
t;>
> > >> >> via Newton Mail
> > >>
> > >>[
> > https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.50&pv=10.15.6&source=email_footer_2
> > ]
> > >> >> On Sat, Sep 5, 2020 at 5:11 PM, Austin Ben
gt;> via Newton Mail
> > >>
> > >>[
> > https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.50&pv=10.15.6&source=email_footer_2
> > ]
> > >> >> On Sat, Sep 5, 2020 at 5:11 PM, Austin Bennett
> > >> > wrote:
> > >&g
t; >> >> On Sat, Sep 5, 2020 at 5:11 PM, Austin Bennett
> >> > wrote:
> >> >> I believe - for not large data - the direct runner is wholly
> >doable,
> >> >which
> >> >> seems in line with airflow patterns.
nner is wholly
>doable,
>> >which
>> >> seems in line with airflow patterns. I have, and have spoken with
>> >several
>> >> others that have, been productive with that runner.
>> >>
>> >> For much larger transfers, the generic
s that have, been productive with that runner.
> > >>
> > >> For much larger transfers, the generic operator could accept
> > >parameters for
> > >> submitting the compute to an actual runner. Though, imagining that
> > >> (needing a run
ductive with that runner.
> >>
> >> For much larger transfers, the generic operator could accept
> >parameters for
> >> submitting the compute to an actual runner. Though, imagining that
> >> (needing a runner) would not be the primary use case for such an
;operator.
>>
>>
>> On Tue, Sep 1, 2020, 11:52 PM Tomasz Urbaszek
>wrote:
>>
>> > Austin, you are right, Beam covers all (and more) important IOs.
>> > However, using Apache Beam to design a generic transfer operator
>> > requires Airflow users to h
uld accept parameters for
> submitting the compute to an actual runner. Though, imagining that
> (needing a runner) would not be the primary use case for such an operator.
>
>
> On Tue, Sep 1, 2020, 11:52 PM Tomasz Urbaszek wrote:
>
> > Austin, you are right, Beam covers all
would not be the primary use case for such an operator.
On Tue, Sep 1, 2020, 11:52 PM Tomasz Urbaszek wrote:
> Austin, you are right, Beam covers all (and more) important IOs.
> However, using Apache Beam to design a generic transfer operator
> requires Airflow users to have additiona
t;
> Best,
> Tomek
>
>
> On Wed, Sep 2, 2020 at 12:37 AM Austin Bennett
> wrote:
> >
> > Are there IOs that would be desired for a generic transfer operator that
> > don't exist in: https://beam.apache.org/documentation/io/built-in/ <-
> > th
Austin, you are right, Beam covers all (and more) important IOs.
However, using Apache Beam to design a generic transfer operator
requires Airflow users to have additional resources that will be used
as a runner (Spark, Flink, etc.). Unless you suggest using
DirectRunner?
Can you please tell us
Are there IOs that would be desired for a generic transfer operator that
don't exist in: https://beam.apache.org/documentation/io/built-in/ <-
there is pretty solid coverage?
Beam is getting to the point where even python beam can leverage the java
IOs, which increases the range of
mal" solution but once you
>> admit
>> > you
>> > > > > are
>> > > > > > > not
>> > > > > > > > > going to focus for optimisation, you come with simpler and
>> > easier
>> > > &g
ow operator.
> > > >
> > > > Gerard Casas Saez
> > > > Twitter | Cortex | @casassaez <http://twitter.com/casassaez>
> > > >
> > > >
> > > > On Tue, Sep 1, 2020 at 12:44 PM Austin Bennett <
> > > > whatwo
> > >
> > >
> > > On Tue, Sep 1, 2020 at 12:44 PM Austin Bennett <
> > > whatwouldausti...@gmail.com>
> > > wrote:
> > >
> > > > Are you guys familiar with Beam <https://beam.apache.org>? Esp. if
> not
> > &
ter | Cortex | @casassaez <http://twitter.com/casassaez>
> >
> >
> > On Tue, Sep 1, 2020 at 12:44 PM Austin Bennett <
> > whatwouldausti...@gmail.com>
> > wrote:
> >
> > > Are you guys familiar with Beam <https://beam.apache.org>? Esp. if
might rather straightforward to rely on the
> ecosystem
> > of connectors in that Apache Project to use as the foundations for a
> > generic transfer operator.
> >
> > On Tue, Sep 1, 2020 at 11:05 AM Jarek Potiuk
> > wrote:
> >
> > > +1
> > >
> &
e ecosystem
> of connectors in that Apache Project to use as the foundations for a
> generic transfer operator.
>
> On Tue, Sep 1, 2020 at 11:05 AM Jarek Potiuk
> wrote:
>
> > +1
> >
> > On Tue, Sep 1, 2020 at 1:35 PM Kamil Olszewski <
> > kamil.olszew...@
Sep 1, 2020 at 12:44 PM Austin Bennett
wrote:
> Are you guys familiar with Beam <https://beam.apache.org>? Esp. if not
> doing transforms, it might rather straightforward to rely on the ecosystem
> of connectors in that Apache Project to use as the foundations for a
> generic trans
Are you guys familiar with Beam <https://beam.apache.org>? Esp. if not
doing transforms, it might rather straightforward to rely on the ecosystem
of connectors in that Apache Project to use as the foundations for a
generic transfer operator.
On Tue, Sep 1, 2020 at 11:05 AM Jarek Potiuk
t; > services that Airflow should use (Airflow is an orchestrator, not data
> > > processing solution) but for the kind of quick-start scenarios I
> foresee
> > it
> > > might be most useful, being able to apply simple data transformation on
> > the
> > >
not data
> > processing solution) but for the kind of quick-start scenarios I foresee
> it
> > might be most useful, being able to apply simple data transformation on
> the
> > fly by data scientist might be a big plus.
> >
> > Suggestion: Panda DataFrame as the fo
e as the format of the "data" component
>
> Kamil - you should have access now.
>
> J.
>
>
> On Tue, Aug 18, 2020 at 6:53 PM Kamil Olszewski <
> kamil.olszew...@polidea.com>
> wrote:
>
> > Hello all,
> > in Polidea we have come up with an idea for a
J.
On Tue, Aug 18, 2020 at 6:53 PM Kamil Olszewski
wrote:
> Hello all,
> in Polidea we have come up with an idea for a generic transfer operator
> that would be able to transport data between two destinations of various
> types (file, database, storage, etc.) - please find the l
Hello all,
in Polidea we have come up with an idea for a generic transfer operator
that would be able to transport data between two destinations of various
types (file, database, storage, etc.) - please find the link with a short
doc with POC
<https://docs.google.com/documen
27 matches
Mail list logo
