Re: Working around lack of SQL triggers

[deklanw]

Thanks for the reply. 

Sorry for my lateness, I put this project on hold for awhile but I'm looking
back into it.

I think I see what you're saying but I'm not sure how to actually implement
that parameterizable aggregation function. I understand what it would be
conceptually, but I'm new to Scala/Java so actually implementing it seems
daunting. Do you have a link to some code where I can see something similar
I can work from? Preferably many such examples :P

Thanks!



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Working around lack of SQL triggers

[deklanw]

Hi,

Thanks for the reply.

I had already almost completely lost hope in using Flink SQL. You have
confirmed that.

But, like I said, I don't know how to reduce the large amount of boilerplate
I foresee this requiring with the DataStream API. Can you help me with that?
You mention "parameterizable aggregation functions", can you show an
example? I don't know how to do this without reinventing AVG and COUNT over
and over again.

Thanks



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Working around lack of SQL triggers

[deklanw]

I'm not sure how to express my logic simply where early triggers are a
necessity.

My application has large windows (2 weeks~) where early triggering is
absolutely required. But, also, my application has mostly relatively simple
logic which can be expressed in SQL. There's a ton of duplication, like the
following 


```
SELECT A,B,C,
COUNT(*) FILTER (WHERE my_condition) AS total_conditions,
COUNT(*) AS total,
ROUND(COUNT(*) FILTER (WHERE my_condition)/(COUNT(*)), 1) AS
condition_rate,
AVG(D),
AVG(E),
AVG(F)
FROM foo
GROUP BY A,B,C, SESSION(...)
```

Just imagine these kinds of queries duplicated a ton, just varying which
fields are being averaged and grouped by.

This is fairly easy to do with SQL, with some copying and pasting. Just
Ctrl+Fing to give an idea (so far),
COUNT - 50
AVG - 27
GROUP BY - 12

Since Flink doesn't support GROUPING SETS for streaming, I'll need to
duplicate a lot of these queries actually. So this is an underestimation.

Is writing an absolute ton of custom AggregateFunction boilerplate the only
way to solve this problem? Is there no way to abstract this while
maintaining early triggers? I feel like I'm missing something. Is Flink SQL
streaming only for short windows where triggering only at the end of the
window is acceptable?




--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/