Hi Gyorgy,
I guess Your problem might not be directly related to UDFs not UDAFs but
to nested structure of Your data. I did have a problem with processing
nested data and also did not find a way to work this out - especially if
You are not in control of input data structure.
This issues seem very related as nested Rows need some polishing:
- https://github.com/apache/beam/issues/26911
- https://github.com/apache/beam/issues/27733
I have plans to jump on fix for this two, and I guess it would not be
hard, but I do not have any capacity for open source tasks for now.
Anyone would like to take a look at them?
Best
Wiśniowski Piotr
On 28.09.2023 19:31, Balogh, György wrote:
Sorry I was not specific enough. I ment using the SqlTransform
registerUdf and registerUdaf. I use a lot of SQL in my pipeline and I
would prefer using SQL UDFs in many cases over writing beam
transforms. I already have UDFs but I did not find a way to make them
work over nested structures.
Thank you,
Gyorgy
On Thu, Sep 28, 2023 at 5:40 PM Robert Bradshaw via user
<[email protected]> wrote:
Yes, for sure. This is one of the areas Beam excels vs.
more simple tools like SQL. You can write arbitrary code to
iterate over arbitrary structures in the typical
Java/Python/Go/Typescript/Scala/[pick your language] way. In the
Beam nomenclature. UDFs correspond to DoFns and UDAFs correspond
to CombineFns.
On Thu, Sep 28, 2023 at 4:23 AM Balogh, György
<[email protected]> wrote:
Hi,
I've complex nested structure in my input data. Is it possible
to have UDF/UDAF taking nested structure as input? I'm using
java. Outputting nested structure is also a question.
Thank you,
Gyorgy
--
György Balogh
CTO
E [email protected] <mailto:[email protected]>
M +36 30 270 8342 <tel:+36%2030%20270%208342>
A HU, 1117 Budapest, Budafoki út 209.
W www.ultinous.com <http://www.ultinous.com>
--
György Balogh
CTO
E [email protected] <mailto:[email protected]>
M +36 30 270 8342 <tel:+36%2030%20270%208342>
A HU, 1117 Budapest, Budafoki út 209.
W www.ultinous.com <http://www.ultinous.com>
