Hi All,
The implementation of https://s.apache.org/easy-multi-language (with the
dynamic API for Python) was merged and should be available with Beam
2.44.0: https://github.com/apache/beam/pull/23413
Thanks,
Cham
On Fri, Aug 19, 2022 at 3:35 PM Chamikara Jayalath
wrote:
> Hi All,
>
> Thanks
Hi All,
Thanks for the comments so far. Seems like we generally agree on this
proposal.
Please see https://github.com/apache/beam/pull/22802 for a prototype
implementation that adds the following.
* Support for dynamically discovering and registering SchemaTransforms in
the Java expansion
I think the *DiscoverSchemaTransform()* RPC introduced in this proposal and
the ability to easily deploy/use available *SchemaTransforms* using an
expansion service essentially provide the tooling necessary for
implementing such a service. Such a service could even startup expansion
services to
I like that idea, sort of like Kafka’s Schema Service but for transforms?
On Mon, Aug 8, 2022 at 2:45 PM Robert Bradshaw via dev
wrote:
> This is a great idea. I would like to approach this from the
> perspective of making it easy to provide a catalog of well-defined
> transforms for use in
This is a great idea. I would like to approach this from the
perspective of making it easy to provide a catalog of well-defined
transforms for use in expansion services from typical SDKs and also
elsewhere (e.g. for documentation purposes, GUIs, etc.) Ideally
everything about what a transform is
Indeed, there's nothing stopping you from doing codegen where it's useful
but I think it's probably easier to implement codegen from dynamic than it
is to go the other way around (Avro vs Proto)
On Fri, Aug 5, 2022 at 1:15 PM Chamikara Jayalath
wrote:
>
>
> On Fri, Aug 5, 2022 at 12:00 PM Byron
On Fri, Aug 5, 2022 at 12:00 PM Byron Ellis wrote:
> I think there are some practical advantages to having the ability to
> support a dynamic version---at previous places where I've worked having
> Kafka's Schema Service was incredibly useful for data processing (it was a
> Java/Scala shop and
I think there are some practical advantages to having the ability to
support a dynamic version---at previous places where I've worked having
Kafka's Schema Service was incredibly useful for data processing (it was a
Java/Scala shop and we mostly used a "decode to POJO" approach rather than
On Fri, Aug 5, 2022 at 9:44 AM Brian Hulette wrote:
> Thanks Cham! I really like the proposal, I left a few comments. I also had
> one higher-level point I wanted to elevate here:
>
> > Pipeline SDKs can generate user-friendly stub-APIs based on transforms
> registered with an expansion service,
Thanks Cham! I really like the proposal, I left a few comments. I also had
one higher-level point I wanted to elevate here:
> Pipeline SDKs can generate user-friendly stub-APIs based on transforms
registered with an expansion service, eliminating the need to develop
language-specific wrappers.
Hi All,
I believe we can make the multi-language pipelines offering [1] much easier
to use by updating the expansion service to be fully aware of
SchemaTransforms. Additionally this will make it easy to
register/discover/use transforms defined in one SDK from all other SDKs.
Specifically we could
11 matches
Mail list logo
