Re: UDF/UADF over complex structures

[Wiśniowski Piotr]

Hi Gyorgy,

I guess Your problem might not be directly related to UDFs not UDAFs but 
to nested structure of Your data. I did have a problem with processing 
nested data and also did not find a way to work this out - especially if 
You are not in control of input data structure.


This issues seem very related as nested Rows need some polishing:

- https://github.com/apache/beam/issues/26911

- https://github.com/apache/beam/issues/27733

I have plans to jump on fix for this two, and I guess it would not be 
hard, but I do not have any capacity for open source tasks for now. 
Anyone would like to take a look at them?


Best

Wiśniowski Piotr


On 28.09.2023 19:31, Balogh, György wrote:
Sorry I was not specific enough. I ment using the SqlTransform 
registerUdf and registerUdaf. I use a lot of SQL in my pipeline and I 
would prefer using SQL UDFs in many cases over writing beam 
transforms. I already have UDFs but I did not find a way to make them 
work over nested structures.

Thank you,
Gyorgy

On Thu, Sep 28, 2023 at 5:40 PM Robert Bradshaw via user 
 wrote:


Yes, for sure. This is one of the areas Beam excels vs.
more simple tools like SQL. You can write arbitrary code to
iterate over arbitrary structures in the typical
Java/Python/Go/Typescript/Scala/[pick your language] way. In the
Beam nomenclature. UDFs correspond to DoFns and UDAFs correspond
to CombineFns.

On Thu, Sep 28, 2023 at 4:23 AM Balogh, György
 wrote:

Hi,
I've complex nested structure in my input data. Is it possible
to have UDF/UDAF taking nested structure as input? I'm using
java. Outputting nested structure is also a question.
Thank you,
Gyorgy
-- 


György Balogh
CTO
E   gyorgy.bal...@ultinous.com 
M   +36 30 270 8342 
A   HU, 1117 Budapest, Budafoki út 209.
W   www.ultinous.com 



--

György Balogh
CTO
E   gyorgy.bal...@ultinous.com 
M   +36 30 270 8342 
A   HU, 1117 Budapest, Budafoki út 209.
W   www.ultinous.com 

Re: UDF/UADF over complex structures

[Balogh , György]

Sorry I was not specific enough. I ment using the SqlTransform registerUdf
and registerUdaf. I use a lot of SQL in my pipeline and I would prefer
using SQL UDFs in many cases over writing beam transforms. I already have
UDFs but I did not find a way to make them work over nested structures.
Thank you,
Gyorgy

On Thu, Sep 28, 2023 at 5:40 PM Robert Bradshaw via user <
user@beam.apache.org> wrote:

> Yes, for sure. This is one of the areas Beam excels vs. more simple tools
> like SQL. You can write arbitrary code to iterate over arbitrary structures
> in the typical Java/Python/Go/Typescript/Scala/[pick your language] way. In
> the Beam nomenclature. UDFs correspond to DoFns and UDAFs correspond to
> CombineFns.
>
> On Thu, Sep 28, 2023 at 4:23 AM Balogh, György 
> wrote:
>
>> Hi,
>> I've complex nested structure in my input data. Is it possible to have
>> UDF/UDAF taking nested structure as input? I'm using java.
>> Outputting nested structure is also a question.
>> Thank you,
>> Gyorgy
>> --
>>
>> György Balogh
>> CTO
>> E gyorgy.bal...@ultinous.com 
>> M +36 30 270 8342 <+36%2030%20270%208342>
>> A HU, 1117 Budapest, Budafoki út 209.
>> W www.ultinous.com
>>
>

-- 

György Balogh
CTO
E gyorgy.bal...@ultinous.com 
M +36 30 270 8342 <+36%2030%20270%208342>
A HU, 1117 Budapest, Budafoki út 209.
W www.ultinous.com

Re: UDF/UADF over complex structures

[Robert Bradshaw via user]

Yes, for sure. This is one of the areas Beam excels vs. more simple tools
like SQL. You can write arbitrary code to iterate over arbitrary structures
in the typical Java/Python/Go/Typescript/Scala/[pick your language] way. In
the Beam nomenclature. UDFs correspond to DoFns and UDAFs correspond to
CombineFns.

On Thu, Sep 28, 2023 at 4:23 AM Balogh, György  wrote:

> Hi,
> I've complex nested structure in my input data. Is it possible to have
> UDF/UDAF taking nested structure as input? I'm using java.
> Outputting nested structure is also a question.
> Thank you,
> Gyorgy
> --
>
> György Balogh
> CTO
> E gyorgy.bal...@ultinous.com 
> M +36 30 270 8342 <+36%2030%20270%208342>
> A HU, 1117 Budapest, Budafoki út 209.
> W www.ultinous.com
>

UDF/UADF over complex structures

[Balogh , György]

Hi,
I've complex nested structure in my input data. Is it possible to have
UDF/UDAF taking nested structure as input? I'm using java.
Outputting nested structure is also a question.
Thank you,
Gyorgy
-- 

György Balogh
CTO
E gyorgy.bal...@ultinous.com 
M +36 30 270 8342 <+36%2030%20270%208342>
A HU, 1117 Budapest, Budafoki út 209.
W www.ultinous.com