[jira] [Comment Edited] (BEAM-3749) support customized trigger/accumulationMode in BeamSql
[
https://issues.apache.org/jira/browse/BEAM-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16390311#comment-16390311
]
Kenneth Knowles edited comment on BEAM-3749 at 3/7/18 9:58 PM:
---
I think this link will clarify:
[https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Window.java#L333]
The trigger is left alone if you use {{Window.into}} when the SQL sets up the
window function. It will not set it to the {{DefaultTrigger}}.
The code that you showed will use tumbling windows with the configured trigger.
was (Author: kenn):
I think this link will clarify:
https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Window.java#L333
The trigger is left alone if you use {{Window.into}} when the SQL sets up the
window function. It will not set it to the {{DefaultTrigger}}.
> support customized trigger/accumulationMode in BeamSql
> --
>
> Key: BEAM-3749
> URL: https://issues.apache.org/jira/browse/BEAM-3749
> Project: Beam
> Issue Type: Improvement
> Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Major
> Fix For: 2.5.0
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> Currently BeamSql use {{DefaultTrigger}} for aggregation operations.
> By adding two options {{withTrigger(Trigger)}} and
> {{withAccumulationMode(AccumulationMode)}}, developers can specify their own
> aggregation strategies with BeamSql.
> [~xumingming] [~kedin] [~kenn] for any comments.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (BEAM-3749) support customized trigger/accumulationMode in BeamSql
[
https://issues.apache.org/jira/browse/BEAM-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16390129#comment-16390129
]
Kenneth Knowles edited comment on BEAM-3749 at 3/7/18 8:13 PM:
---
What I mean is that the windowing itself is set up in the SQL via using
TUMBLE/HOP/SESSION. That is independent of the trigger. If the CLI sets up the
trigger using {{Window.configure()}} and then the SQL set up the WindowFn then
it will work.
was (Author: kenn):
What I mean is that the windowing itself is set up in the SQL via using
TUMBLE/HOP/SESSION. That is independent of the trigger.
> support customized trigger/accumulationMode in BeamSql
> --
>
> Key: BEAM-3749
> URL: https://issues.apache.org/jira/browse/BEAM-3749
> Project: Beam
> Issue Type: Improvement
> Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Major
> Fix For: 2.5.0
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> Currently BeamSql use {{DefaultTrigger}} for aggregation operations.
> By adding two options {{withTrigger(Trigger)}} and
> {{withAccumulationMode(AccumulationMode)}}, developers can specify their own
> aggregation strategies with BeamSql.
> [~xumingming] [~kedin] [~kenn] for any comments.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (BEAM-3749) support customized trigger/accumulationMode in BeamSql
[
https://issues.apache.org/jira/browse/BEAM-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16377862#comment-16377862
]
Xu Mingmin edited comment on BEAM-3749 at 2/27/18 12:55 AM:
{{Window.triggering(trigger).discardingFiredPanes()}} is not valid so far, is
it a work-in-progress feature? It's always
{{Window.into(...).triggering(...)...}}.
Currently TUMBLE/HOP/SESSION don't take Trigger parameter and IMO it's
{{DefaultTrigger}} as no Trigger is set explicitly, +[~kedin] for comments.
was (Author: mingmxu):
{{Window.triggering(trigger).discardingFiredPanes()}} is not valid so far, is
it a work-in-progress feature? It's always
{{Window.into(...).triggering(...)...}}.
Currently TUMBLE/HOP/SESSION doesn't take Trigger parameter and IMO it's
{{DefaultTrigger}} as no Trigger is set explicitly, +[~kedin] for comments.
> support customized trigger/accumulationMode in BeamSql
> --
>
> Key: BEAM-3749
> URL: https://issues.apache.org/jira/browse/BEAM-3749
> Project: Beam
> Issue Type: Improvement
> Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Major
> Fix For: 2.4.0
>
>
> Currently BeamSql use {{DefaultTrigger}} for aggregation operations.
> By adding two options {{withTrigger(Trigger)}} and
> {{withAccumulationMode(AccumulationMode)}}, developers can specify their own
> aggregation strategies with BeamSql.
> [~xumingming] [~kedin] [~kenn] for any comments.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (BEAM-3749) support customized trigger/accumulationMode in BeamSql
[
https://issues.apache.org/jira/browse/BEAM-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16377844#comment-16377844
]
Kenneth Knowles edited comment on BEAM-3749 at 2/27/18 12:39 AM:
-
I don't think the description is quite right - BeamSql doesn't use the
{{DefaultTrigger}}, does it? I think it uses whatever trigger is set up, which
could be anything.
What about this?
{code}
PCollectionTuple inputTables = ...;
input.apply(Window.triggering(trigger).discardingFiredPanes()) // or
accumulatingFiredPanes
.apply(BeamSql.query("..."));
{code}
This should allow the CLI to apply the trigger and accumulation mode outside of
the SQL transform. This will use the continuation trigger the right way
everywhere.
was (Author: kenn):
What about this?
{code}
PCollectionTuple inputTables = ...;
input.apply(Window.triggering(trigger).discardingFiredPanes()) // or
accumulatingFiredPanes
.apply(BeamSql.query("..."));
{code}
This should allow the CLI to apply the trigger and accumulation mode outside of
the SQL transform. This will use the continuation trigger the right way
everywhere.
> support customized trigger/accumulationMode in BeamSql
> --
>
> Key: BEAM-3749
> URL: https://issues.apache.org/jira/browse/BEAM-3749
> Project: Beam
> Issue Type: Improvement
> Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Major
> Fix For: 2.4.0
>
>
> Currently BeamSql use {{DefaultTrigger}} for aggregation operations.
> By adding two options {{withTrigger(Trigger)}} and
> {{withAccumulationMode(AccumulationMode)}}, developers can specify their own
> aggregation strategies with BeamSql.
> [~xumingming] [~kedin] [~kenn] for any comments.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
