[
https://issues.apache.org/jira/browse/BEAM-7808?focusedWorklogId=284696&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-284696
]
ASF GitHub Bot logged work on BEAM-7808:
----------------------------------------
Author: ASF GitHub Bot
Created on: 30/Jul/19 04:41
Start Date: 30/Jul/19 04:41
Worklog Time Spent: 10m
Work Description: sheshananda commented on issue #9142: [BEAM-7808] Add
method to convert avro field to beam field
URL: https://github.com/apache/beam/pull/9142#issuecomment-516258507
> LGTM.
>
> Not sure I understand the use case though. You have some Beam rows with a
schema. And you also have an Avro schema that you want to merge (partially)
with that existing Beam schema? Would it make sense to eventually build a more
generic projection framework? Something that would allow you to merge the two
schemas and then pick only the fields you need. And if this happens as part of
some kind of Join implementation then it probably should be supported there as
well in a more generic fashion.
Below is the scenario we are trying:
Kafka Source (AvroSchema based Topic) -> Beam UDF Transform -> Kafka Sink
(AvroSchema based Topic)
We read incoming data stream from Kafka and then apply a Beam Transform.
After tranformation again the data is written to Kafka Sink.
The data that flows between the transform is of type PCollection<Row>.
Now as part of UDF transform, we have exposed a configuration with which
user can provide info on how the incoming Row has to be enriched. This
information is also provided by an avro schema.
So we needed this function which can convert the avro Field to Beam Field
and finally merge it with incoming row.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 284696)
Time Spent: 1h 10m (was: 1h)
> Add method to convert avro field to beam field
> ----------------------------------------------
>
> Key: BEAM-7808
> URL: https://issues.apache.org/jira/browse/BEAM-7808
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
> Reporter: Vishwas
> Assignee: Vishwas
> Priority: Minor
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Currently we have static methods in AvroUtils class, which exposes methods
> like
> toBeamSchema(), toBeamRowStrict(). These methods work only on generic
> record.
> Method is now added which will also support converting avroField to BeamField.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
