Re: Does FlinkKafkaConsumer010 care about consumer group?

[Moiz S Jinia]

Great thanks that was very helpful.

One last question -

> If your job code hasn’t changed across the restores, then it should be
> fine even if you didn’t set the UID.


What kind of code change? What if the operator pipeline is still the same
but there's a some business logic change?


On Wed, Jul 19, 2017 at 1:16 PM, Tzu-Li (Gordon) Tai <tzuli...@apache.org>
wrote:

> Does this mean I can use the same consumer group G1 for the newer version
> A'? And inspite of same consumer group, A' will receive messages from all
> partitions when its started from savepoint?
>
>
> Yes. That’s true. Flink internally uses static partition assignment, and
> the clients are assigned whatever partition states they are restored with.
> The only “conflict” this would introduce is that both jobs will be
> competing offset committing to the same consumer group in Kafka (again,
> this doesn’t affect exactly-once but might mess up other external
> monitoring tools you may be using).
>
> I am using Flink 1.2.1. Does the above plan require setting uid on the
> Kafka source in the job?
>
> That would be a separate topic. Setting the UID of operators explicitly is
> usually always recommended before moving to production. See [1].
>
> If your job code hasn’t changed across the restores, then it should be
> fine even if you didn’t set the UID.
>
>
> [1] https://ci.apache.org/projects/flink/flink-docs-
> release-1.3/ops/production_ready.html
>
> On 19 July 2017 at 3:41:28 PM, Moiz S Jinia (moiz.ji...@gmail.com) wrote:
>
> Does this mean I can use the same consumer group G1 for the newer version
> A'? And inspite of same consumer group, A' will receive messages from all
> partitions when its started from savepoint?
>
> I am using Flink 1.2.1. Does the above plan require setting uid on the
> Kafka source in the job?
>
> Thanks,
> Moiz
>
> On Wed, Jul 19, 2017 at 1:06 PM, Tzu-Li (Gordon) Tai <tzuli...@apache.org>
> wrote:
>
>> Hi!
>>
>> The only occasions which the consumer group is used is:
>> 1. When committing offsets back to Kafka. Since Flink 1.3, this can be
>> disabled completely (both when checkpointing is enabled or disabled). See
>> [1] on details about that.
>> 2. When starting fresh (not starting from some savepoint), if you choose
>> to use GROUP_OFFSETS as the start position, then the consumer group would
>> also be used. If starting from a savepoint, then this is irrelevant. See
>> [2].
>>
>> Note that it actually isn’t used in any critical paths for Flink’s
>> exactly-once processing guarantees, or partition to source parallel
>> instance assignments.
>>
>> So, the desired behavior in which you described is exactly the expected
>> behavior for the Flink Kafka Consumer.
>>
>> Cheers,
>> Gordon
>>
>> [1] https://ci.apache.org/projects/flink/flink-docs-release-
>> 1.3/dev/connectors/kafka.html#kafka-consumers-offset-
>> committing-behaviour-configuration
>> [2] https://ci.apache.org/projects/flink/flink-docs-release-
>> 1.3/dev/connectors/kafka.html#kafka-consumers-start-
>> position-configuration
>>
>> On 19 July 2017 at 3:23:01 PM, Moiz Jinia (moiz.ji...@gmail.com) wrote:
>>
>> Below is a plan for downtime-free upgrade of a Flink job. The downstream
>> consumer of the Flink job is duplicate proof.
>>
>> Scenario 1 -
>> 1. Start Flink job A with consumer group G1 (12 slot job)
>> 2. While job A is running, take a savepoint AS.
>> 3. Start newer version of Flink job A' from savepoint AS with consumer
>> group
>> *G1* (12 slot job again)
>> 4. Stop job A.
>>
>> Scenario 2 -
>> 1. Start Flink job A with consumer group G1 (12 slot job)
>> 2. While job A is running, take a savepoint AS.
>> 3. Start newer version of Flink job A' from savepoint AS with consumer
>> group
>> *G2* (12 slot job again)
>> 4. Stop job A
>>
>> Does it matter what consumer group job A' uses? The desired behavior is
>> that
>> during the window when both A and A' are running, all messages should go
>> to
>> both jobs. (And of course I want that job A' should start consuming from
>> the
>> offsets in the savepoint and not the earliest).
>>
>>
>>
>>
>>
>>
>> --
>> View this message in context: http://apache-flink-user-maili
>> ng-list-archive.2336050.n4.nabble.com/Does-FlinkKafkaCons
>> umer010-care-about-consumer-group-tp14323.html
>> Sent from the Apache Flink User Mailing List archive. mailing list
>> archive at Nabble.com.
>>
>>
>

Re: Does FlinkKafkaConsumer010 care about consumer group?

[Moiz S Jinia]

Does this mean I can use the same consumer group G1 for the newer version
A'? And inspite of same consumer group, A' will receive messages from all
partitions when its started from savepoint?

I am using Flink 1.2.1. Does the above plan require setting uid on the
Kafka source in the job?

Thanks,
Moiz

On Wed, Jul 19, 2017 at 1:06 PM, Tzu-Li (Gordon) Tai 
wrote:

> Hi!
>
> The only occasions which the consumer group is used is:
> 1. When committing offsets back to Kafka. Since Flink 1.3, this can be
> disabled completely (both when checkpointing is enabled or disabled). See
> [1] on details about that.
> 2. When starting fresh (not starting from some savepoint), if you choose
> to use GROUP_OFFSETS as the start position, then the consumer group would
> also be used. If starting from a savepoint, then this is irrelevant. See
> [2].
>
> Note that it actually isn’t used in any critical paths for Flink’s
> exactly-once processing guarantees, or partition to source parallel
> instance assignments.
>
> So, the desired behavior in which you described is exactly the expected
> behavior for the Flink Kafka Consumer.
>
> Cheers,
> Gordon
>
> [1] https://ci.apache.org/projects/flink/flink-docs-
> release-1.3/dev/connectors/kafka.html#kafka-consumers-
> offset-committing-behaviour-configuration
> [2] https://ci.apache.org/projects/flink/flink-docs-
> release-1.3/dev/connectors/kafka.html#kafka-consumers-
> start-position-configuration
>
> On 19 July 2017 at 3:23:01 PM, Moiz Jinia (moiz.ji...@gmail.com) wrote:
>
> Below is a plan for downtime-free upgrade of a Flink job. The downstream
> consumer of the Flink job is duplicate proof.
>
> Scenario 1 -
> 1. Start Flink job A with consumer group G1 (12 slot job)
> 2. While job A is running, take a savepoint AS.
> 3. Start newer version of Flink job A' from savepoint AS with consumer
> group
> *G1* (12 slot job again)
> 4. Stop job A.
>
> Scenario 2 -
> 1. Start Flink job A with consumer group G1 (12 slot job)
> 2. While job A is running, take a savepoint AS.
> 3. Start newer version of Flink job A' from savepoint AS with consumer
> group
> *G2* (12 slot job again)
> 4. Stop job A
>
> Does it matter what consumer group job A' uses? The desired behavior is
> that
> during the window when both A and A' are running, all messages should go
> to
> both jobs. (And of course I want that job A' should start consuming from
> the
> offsets in the savepoint and not the earliest).
>
>
>
>
>
>
> --
> View this message in context: http://apache-flink-user-
> mailing-list-archive.2336050.n4.nabble.com/Does-
> FlinkKafkaConsumer010-care-about-consumer-group-tp14323.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> at Nabble.com.
>
>

Re: Does job restart resume from last known internal checkpoint?

[Moiz S Jinia]

Thanks for that! Yes I indeed did not receive those emails. And my question
is answered.

Moiz

On Fri, Jun 2, 2017 at 12:46 PM, Nico Kruber <n...@data-artisans.com> wrote:

> Hi Moiz,
> didn't Timo's answer cover your questions?
>
> see here in case you didn't receive it:
> https://lists.apache.org/thread.html/
> a1a0d04e7707f4b0ac8b8b2f368110b898b2ba11463d32f9bba73968@
> %3Cuser.flink.apache.org%3E
>
>
> Nico
>
> On Thursday, 1 June 2017 20:30:59 CEST Moiz S Jinia wrote:
> > Bump..
> >
> > On Tue, May 30, 2017 at 10:17 PM, Moiz S Jinia <moiz.ji...@gmail.com>
> wrote:
> > > In a checkpointed Flink job will doing a graceful restart make it
> resume
> > > from last known internal checkpoint? Or are all checkpoints discarded
> when
> > > the job is stopped?
> > >
> > > If discarded, what will be the resume point?
> > >
> > > Moiz
>
>

Re: Does job restart resume from last known internal checkpoint?

[Moiz S Jinia]

Bump..

On Tue, May 30, 2017 at 10:17 PM, Moiz S Jinia <moiz.ji...@gmail.com> wrote:

> In a checkpointed Flink job will doing a graceful restart make it resume
> from last known internal checkpoint? Or are all checkpoints discarded when
> the job is stopped?
>
> If discarded, what will be the resume point?
>
> Moiz
>

Does job restart resume from last known internal checkpoint?

[Moiz S Jinia]

In a checkpointed Flink job will doing a graceful restart make it resume
from last known internal checkpoint? Or are all checkpoints discarded when
the job is stopped?

If discarded, what will be the resume point?

Moiz

Re: Kafka partitions -> task slots? (keyed stream)

[Moiz S Jinia]

I have just 1 job (that has a ProcessFunction with timers).

You're saying that giving more task slots to my job than the number of
partitions on the source topic is not going to help.

This implies that 1 partition cannot be assigned to more than 1 task slot.
That makes sense as otherwise ordering for a partition would not be
guaranteed.

Thanks.

On Tue, May 30, 2017 at 8:43 PM, Stefan Richter <s.rich...@data-artisans.com
> wrote:

> Hi,
>
> it is not restricting the parallelism of your job. Only increasing the
> parallelism of your Job’s sources to more than 5 will not bring any
> improvements. All other operators could still benefit from a higher
> parallelism.
>
> > Am 30.05.2017 um 09:49 schrieb Moiz S Jinia <moiz.ji...@gmail.com>:
> >
> > For a keyed stream (where the key is also the message key in the source
> kafka topic), is the parallelism of the job restricted to the number of
> partitions in the topic?
> >
> > Source topic has 5 partitions, but available task slots are 12. (3 task
> managers each with 4 slots)
> >
> > Moiz
>
>

Kafka partitions -> task slots? (keyed stream)

[Moiz S Jinia]

For a keyed stream (where the key is also the message key in the source
kafka topic), is the parallelism of the job restricted to the number of
partitions in the topic?

Source topic has 5 partitions, but available task slots are 12. (3 task
managers each with 4 slots)

Moiz

Re: Are timers in ProcessFunction fault tolerant?

[Moiz S Jinia]

Thanks Kostas. So even though the timer state is managed separately from
the key state (from runtimeContext) I can safely assume both the states to
be fault tolerant and maintain association with the key of the stream?

On Fri, May 26, 2017 at 1:51 PM, Kostas Kloudas <k.klou...@data-artisans.com
> wrote:

> Hi Moiz,
>
> state.clear() refers to the state that you have registered in your job,
> using the getState()
> from the runtimeContext.
>
> Timers are managed by Flink’s timer service and they are cleaned up by
> Flink itself when
> the job terminates.
>
> Kostas
>
> On May 26, 2017, at 6:41 AM, Moiz S Jinia <moiz.ji...@gmail.com> wrote:
>
> A follow on question. Since the registered timers are part of the managed
> key state, do the timers get cancelled when i call state.clear()?
>
> Moiz
>
> On Thu, May 25, 2017 at 10:20 PM, Moiz S Jinia <moiz.ji...@gmail.com>
> wrote:
>
>> Awesome. Thanks.
>>
>> On Thu, May 25, 2017 at 10:13 PM, Eron Wright <eronwri...@gmail.com>
>> wrote:
>>
>>> Yes, registered timers are stored in managed keyed state and should be
>>> fault-tolerant.
>>>
>>> -Eron
>>>
>>> On Thu, May 25, 2017 at 9:28 AM, Moiz S Jinia <moiz.ji...@gmail.com>
>>> wrote:
>>>
>>>> With a checkpointed RocksDB based state backend, can I expect the
>>>> registered processing timers to be fault tolerant? (along with the managed
>>>> keyed state).
>>>>
>>>> Example -
>>>> A task manager instance owns the key k1 (from a keyed stream) that has
>>>> registered a processing timer with a timestamp thats a day ahead in the
>>>> future. If this instance is killed, and the key is moved to another
>>>> instance, will the onTimer trigger correctly on the other machine at the
>>>> expected time with the same keyed state (for k1)?
>>>>
>>>> Thanks,
>>>> Moiz
>>>>
>>>
>>>
>>
>
>

Re: Are timers in ProcessFunction fault tolerant?

[Moiz S Jinia]

A follow on question. Since the registered timers are part of the managed
key state, do the timers get cancelled when i call state.clear()?

Moiz

On Thu, May 25, 2017 at 10:20 PM, Moiz S Jinia <moiz.ji...@gmail.com> wrote:

> Awesome. Thanks.
>
> On Thu, May 25, 2017 at 10:13 PM, Eron Wright <eronwri...@gmail.com>
> wrote:
>
>> Yes, registered timers are stored in managed keyed state and should be
>> fault-tolerant.
>>
>> -Eron
>>
>> On Thu, May 25, 2017 at 9:28 AM, Moiz S Jinia <moiz.ji...@gmail.com>
>> wrote:
>>
>>> With a checkpointed RocksDB based state backend, can I expect the
>>> registered processing timers to be fault tolerant? (along with the managed
>>> keyed state).
>>>
>>> Example -
>>> A task manager instance owns the key k1 (from a keyed stream) that has
>>> registered a processing timer with a timestamp thats a day ahead in the
>>> future. If this instance is killed, and the key is moved to another
>>> instance, will the onTimer trigger correctly on the other machine at the
>>> expected time with the same keyed state (for k1)?
>>>
>>> Thanks,
>>> Moiz
>>>
>>
>>
>

Re: Are timers in ProcessFunction fault tolerant?

[Moiz S Jinia]

Awesome. Thanks.

On Thu, May 25, 2017 at 10:13 PM, Eron Wright <eronwri...@gmail.com> wrote:

> Yes, registered timers are stored in managed keyed state and should be
> fault-tolerant.
>
> -Eron
>
> On Thu, May 25, 2017 at 9:28 AM, Moiz S Jinia <moiz.ji...@gmail.com>
> wrote:
>
>> With a checkpointed RocksDB based state backend, can I expect the
>> registered processing timers to be fault tolerant? (along with the managed
>> keyed state).
>>
>> Example -
>> A task manager instance owns the key k1 (from a keyed stream) that has
>> registered a processing timer with a timestamp thats a day ahead in the
>> future. If this instance is killed, and the key is moved to another
>> instance, will the onTimer trigger correctly on the other machine at the
>> expected time with the same keyed state (for k1)?
>>
>> Thanks,
>> Moiz
>>
>
>

Are timers in ProcessFunction fault tolerant?

[Moiz S Jinia]

With a checkpointed RocksDB based state backend, can I expect the
registered processing timers to be fault tolerant? (along with the managed
keyed state).

Example -
A task manager instance owns the key k1 (from a keyed stream) that has
registered a processing timer with a timestamp thats a day ahead in the
future. If this instance is killed, and the key is moved to another
instance, will the onTimer trigger correctly on the other machine at the
expected time with the same keyed state (for k1)?

Thanks,
Moiz

Re: Programmatic management of Flink jobs

[Moiz S Jinia]

Thanks. Do you foresee any problems with canceling/starting the jobs
whenever the patterns change?

Moiz

On Wed, May 3, 2017 at 4:57 PM, Aljoscha Krettek <aljos...@apache.org>
wrote:

> Hi,
> Yes, this would work even though it requires canceling/restarting the job
> whenever the patterns change.
>
> Best,
> Aljoscha
>
> On 3. May 2017, at 12:36, Moiz S Jinia <moiz.ji...@gmail.com> wrote:
>
> The kind of program I intend to submit would be one that sets up
> a StreamExecutionEnvironment, connects to a stream from a Kafka topic, and
> uses a PatternStream over the kafka events. I could have the jar for this
> program readily available in the "jobmanager.web.upload.dir", and use the
> REST API for submitting a program with some configuration params.
>
> Does that sound like it'd work or am I missing something?
>
> Moiz
>
> On Wed, May 3, 2017 at 3:23 PM, Moiz S Jinia <moiz.ji...@gmail.com> wrote:
>
>> Not sure I understand Operators. What I need is to have a Pattern that
>> starts consuming from a Kafka stream. And I need the Patterns to come and
>> go.
>>
>> Another option that comes to mind is this -
>> The Patterns I'll need are well known in advance. Only certain parameters
>> such as the time duration of the within clause, and maybe certain filter
>> confitions of the where clause need tweaking. So I could pre-deploy the
>> Patterns (or jobs) and start or stop them (with parameters).
>>
>> Does that sound feasible?
>>
>> On Wed, May 3, 2017 at 3:15 PM, Aljoscha Krettek <aljos...@apache.org>
>> wrote:
>>
>>> What would the pattern be added to. An existing custom operator?
>>>
>>> The REST interface only allows for managing the lifecycle of a job, not
>>> modifying their graph structure.
>>>
>>> On 3. May 2017, at 11:43, Moiz S Jinia <moiz.ji...@gmail.com> wrote:
>>>
>>> Thanks for the references. Looking at the REST API, would adding new
>>> Patterns not work via this?
>>>
>>> https://ci.apache.org/projects/flink/flink-docs-release-1.3/
>>> monitoring/rest_api.html#submitting-programs
>>>
>>> Moiz
>>>
>>> On Wed, May 3, 2017 at 2:52 PM, Aljoscha Krettek <aljos...@apache.org>
>>> wrote:
>>>
>>>> Hi,
>>>> For managing a Job you can either use the bin/flink command-line tool
>>>> or the Rest API [1]. As for dynamically adding patterns, that’s outside of
>>>> the scope of Flink right now. There are, however, some users that
>>>> implemented this on top of Flink, see for example RBEA [2]. The basic idea
>>>> is to use a ConnectedStream where one input is the main input and the other
>>>> input is a control stream that updates the existing patterns.
>>>>
>>>> [1] https://ci.apache.org/projects/flink/flink-docs-release-
>>>> 1.3/monitoring/rest_api.html
>>>> [2] https://techblog.king.com/rbea-scalable-real-time-analytics-king/
>>>>
>>>> On 3. May 2017, at 10:02, Moiz S Jinia <moiz.ji...@gmail.com> wrote:
>>>>
>>>> Is there an API that allows remotely adding, modifying, and cancelling
>>>> Flink jobs? Example - changing the time window of a deployed Pattern,
>>>> adding new Patterns, etc.
>>>>
>>>> Whats the best way to go about this? To the end user the Pattern would
>>>> manifest as rules that can be updated anytime.
>>>>
>>>>
>>>> Moiz
>>>>
>>>>
>>>>
>>>
>>>
>>
>
>

Long running time based Patterns

[Moiz S Jinia]

Does Flink (with a persistent State backend such as RocksDB) work well with
long running Patterns of this type? (running into days)

Pattern.begin("start").followedBy("end").within(Time.days(3))

Is there some gotchas here or things to watch out for?

Thanks,
Moiz

Re: Programmatic management of Flink jobs

[Moiz S Jinia]

Not sure I understand Operators. What I need is to have a Pattern that
starts consuming from a Kafka stream. And I need the Patterns to come and
go.

Another option that comes to mind is this -
The Patterns I'll need are well known in advance. Only certain parameters
such as the time duration of the within clause, and maybe certain filter
confitions of the where clause need tweaking. So I could pre-deploy the
Patterns (or jobs) and start or stop them (with parameters).

Does that sound feasible?

On Wed, May 3, 2017 at 3:15 PM, Aljoscha Krettek <aljos...@apache.org>
wrote:

> What would the pattern be added to. An existing custom operator?
>
> The REST interface only allows for managing the lifecycle of a job, not
> modifying their graph structure.
>
> On 3. May 2017, at 11:43, Moiz S Jinia <moiz.ji...@gmail.com> wrote:
>
> Thanks for the references. Looking at the REST API, would adding new
> Patterns not work via this?
>
> https://ci.apache.org/projects/flink/flink-docs-
> release-1.3/monitoring/rest_api.html#submitting-programs
>
> Moiz
>
> On Wed, May 3, 2017 at 2:52 PM, Aljoscha Krettek <aljos...@apache.org>
> wrote:
>
>> Hi,
>> For managing a Job you can either use the bin/flink command-line tool or
>> the Rest API [1]. As for dynamically adding patterns, that’s outside of the
>> scope of Flink right now. There are, however, some users that implemented
>> this on top of Flink, see for example RBEA [2]. The basic idea is to use a
>> ConnectedStream where one input is the main input and the other input is a
>> control stream that updates the existing patterns.
>>
>> [1] https://ci.apache.org/projects/flink/flink-docs-release-
>> 1.3/monitoring/rest_api.html
>> [2] https://techblog.king.com/rbea-scalable-real-time-analytics-king/
>>
>> On 3. May 2017, at 10:02, Moiz S Jinia <moiz.ji...@gmail.com> wrote:
>>
>> Is there an API that allows remotely adding, modifying, and cancelling
>> Flink jobs? Example - changing the time window of a deployed Pattern,
>> adding new Patterns, etc.
>>
>> Whats the best way to go about this? To the end user the Pattern would
>> manifest as rules that can be updated anytime.
>>
>>
>> Moiz
>>
>>
>>
>
>

Programmatic management of Flink jobs

[Moiz S Jinia]

Is there an API that allows remotely adding, modifying, and cancelling
Flink jobs? Example - changing the time window of a deployed Pattern,
adding new Patterns, etc.

Whats the best way to go about this? To the end user the Pattern would
manifest as rules that can be updated anytime.


Moiz

Re: CEP timeout occurs even for a successful match when using followedBy

[Moiz S Jinia]

Thanks! I downloaded and built 1.3-SNAPSHOT locally and was able to verify
that followedBy now works as I want.

Moiz

On Sat, Apr 29, 2017 at 11:08 PM, Kostas Kloudas <
k.klou...@data-artisans.com> wrote:

> Hi Moiz,
>
> Here are the instructions on how to build Flink from source:
>
> https://ci.apache.org/projects/flink/flink-docs-
> release-1.2/setup/building.html
>
> Kostas
>
> On Apr 29, 2017, at 7:15 PM, Moiz S Jinia <moiz.ji...@gmail.com> wrote:
>
> I meant maven dependencies that i can use by generating them from sources.
>
> On Sat, Apr 29, 2017 at 10:31 PM, Moiz S Jinia <moiz.ji...@gmail.com>
> wrote:
>
>> Ok I'll try that. Its just that I'd rather use a stable version.
>> Are there any instructions for building binaries from latest sources?
>>
>> Moiz
>>
>> On Sat, Apr 29, 2017 at 10:09 PM, Kostas Kloudas <
>> k.klou...@data-artisans.com> wrote:
>>
>>> Hi Moiz,
>>>
>>> The skip-till-next is a big change and backporting it does not seem
>>> feasible.
>>> Also this would require more general changes to the 1.2 to make it
>>> compatible with the previous 1.2 versions.
>>>
>>> If you want you can already use the 1.3 version by downloading the
>>> master branch and writing your
>>> use-case against that. The changes until the final release are going to
>>> be minor hopefully and we can
>>> always help you adjust your program accordingly.
>>>
>>> Hope this helps,
>>> Kostas
>>>
>>> On Apr 29, 2017, at 6:23 PM, Moiz S Jinia <moiz.ji...@gmail.com> wrote:
>>>
>>> Oh ok thats a bit far off. Is there any chance of a backport of
>>> https://issues.apache.org/jira/browse/FLINK-6208 to the 1.2 branch? I
>>> require the SKIP_TILL_NEXT behaviour for a production use case that we want
>>> to use Flink for.
>>>
>>> Moiz
>>>
>>> On Sat, Apr 29, 2017 at 9:49 PM, Kostas Kloudas <
>>> k.klou...@data-artisans.com> wrote:
>>>
>>>> The 1.3 is scheduled for the beginning of June.
>>>>
>>>> Cheers,
>>>> Kostas
>>>>
>>>> On Apr 29, 2017, at 6:16 PM, Moiz S Jinia <moiz.ji...@gmail.com> wrote:
>>>>
>>>> Thanks Dawid!
>>>> Yes thats what i was expecting. I'll give it a try.
>>>>
>>>> When do you expect 1.3.0 stable to be out?
>>>>
>>>> Moiz
>>>>
>>>> On Sat, Apr 29, 2017 at 9:20 PM, Dawid Wysakowicz <
>>>> wysakowicz.da...@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> This is an expected behaviour. After the "ar" event there still may
>>>>> occur other "ar" event that will also trigger a match.
>>>>> To be more generic in all versions prior to 1.3.0 there are two
>>>>> different consuming strategies:
>>>>>
>>>>>- STRICT (the next operator) - that accepts only if the event
>>>>>occurs directly after the previous
>>>>>- SKIP TILL ANY (the followedBy operator) - it accepts any
>>>>>matching event following event if there were already an event that 
>>>>> matched
>>>>>this pattern
>>>>>
>>>>> Because after "ni" event we could match with some other "ar" events,
>>>>> the match is timeouted after 5 seconds.
>>>>>
>>>>> In FLINK-6208 <https://issues.apache.org/jira/browse/FLINK-6208> we
>>>>> introduced third consuming strategy:
>>>>>
>>>>>- SKIP TILL NEXT(this is the strategy for followedBy right now) -
>>>>>the event does not have to occur directly after the previous one but 
>>>>> only
>>>>>one event can be matched
>>>>>
>>>>> and you can still use SKIP TILL ANY by using followedByAny. I believe
>>>>> the SKIP TILL NEXT strategy is the one you expected.
>>>>> You can check it on master branch. We did introduce lots of new
>>>>> features and bugfixes to CEP for 1.3.0 version so any comments,
>>>>> tests or suggestions are welcome.
>>>>>
>>>>>
>>>>> Z pozdrowieniami! / Cheers!
>>>>>
>>>>> Dawid Wysakowicz
>>>>> *Data/Software Engineer*
>>>>> Skype: dawid_wys | Twitter: @OneMoreCoder
>>>>> <http://getindata.com/>
>>>>>
>>>>> 2017-04-29 12:14 GMT+02:00 Moiz S Jinia <moiz.ji...@gmail.com>:
>>>>>
>>>>>> When using "next", this pattern works fine for the both a match as
>>>>>> well as a timeout:
>>>>>>
>>>>>> Pattern<Event, Event> pattern = Pattern.begin("start")
>>>>>> .where(evt -> evt.value.equals("ni"))
>>>>>> .next("last").where(evt -> evt.value.equals("ar")).within
>>>>>> (Time.seconds(5));
>>>>>>
>>>>>> 1. "ni" then "ar" within 5 seconds - triggers match
>>>>>> 2. "ni" then no "ar" within 5 seconds - triggers timeout
>>>>>>
>>>>>> But with "followedBy", this does not behave as expected:
>>>>>>
>>>>>> Pattern<Event, Event> pattern = Pattern.begin("start")
>>>>>> .where(evt -> evt.value.equals("ni"))
>>>>>> .followedBy("last").where(evt ->
>>>>>> evt.value.equals("ar")).within(Time.seconds(5));
>>>>>>
>>>>>> "ni" then "ar" within 5 seconds - triggers match and also triggers
>>>>>> timeout.
>>>>>>
>>>>>> Why is the timeout triggered when using followedBy (when there is a
>>>>>> match)?
>>>>>>
>>>>>> Version - 1.1.5.
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>
>

Re: CEP timeout occurs even for a successful match when using followedBy

[Moiz S Jinia]

I meant maven dependencies that i can use by generating them from sources.

On Sat, Apr 29, 2017 at 10:31 PM, Moiz S Jinia <moiz.ji...@gmail.com> wrote:

> Ok I'll try that. Its just that I'd rather use a stable version.
> Are there any instructions for building binaries from latest sources?
>
> Moiz
>
> On Sat, Apr 29, 2017 at 10:09 PM, Kostas Kloudas <
> k.klou...@data-artisans.com> wrote:
>
>> Hi Moiz,
>>
>> The skip-till-next is a big change and backporting it does not seem
>> feasible.
>> Also this would require more general changes to the 1.2 to make it
>> compatible with the previous 1.2 versions.
>>
>> If you want you can already use the 1.3 version by downloading the master
>> branch and writing your
>> use-case against that. The changes until the final release are going to
>> be minor hopefully and we can
>> always help you adjust your program accordingly.
>>
>> Hope this helps,
>> Kostas
>>
>> On Apr 29, 2017, at 6:23 PM, Moiz S Jinia <moiz.ji...@gmail.com> wrote:
>>
>> Oh ok thats a bit far off. Is there any chance of a backport of
>> https://issues.apache.org/jira/browse/FLINK-6208 to the 1.2 branch? I
>> require the SKIP_TILL_NEXT behaviour for a production use case that we want
>> to use Flink for.
>>
>> Moiz
>>
>> On Sat, Apr 29, 2017 at 9:49 PM, Kostas Kloudas <
>> k.klou...@data-artisans.com> wrote:
>>
>>> The 1.3 is scheduled for the beginning of June.
>>>
>>> Cheers,
>>> Kostas
>>>
>>> On Apr 29, 2017, at 6:16 PM, Moiz S Jinia <moiz.ji...@gmail.com> wrote:
>>>
>>> Thanks Dawid!
>>> Yes thats what i was expecting. I'll give it a try.
>>>
>>> When do you expect 1.3.0 stable to be out?
>>>
>>> Moiz
>>>
>>> On Sat, Apr 29, 2017 at 9:20 PM, Dawid Wysakowicz <
>>> wysakowicz.da...@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> This is an expected behaviour. After the "ar" event there still may
>>>> occur other "ar" event that will also trigger a match.
>>>> To be more generic in all versions prior to 1.3.0 there are two
>>>> different consuming strategies:
>>>>
>>>>- STRICT (the next operator) - that accepts only if the event
>>>>occurs directly after the previous
>>>>- SKIP TILL ANY (the followedBy operator) - it accepts any matching
>>>>event following event if there were already an event that matched this
>>>>pattern
>>>>
>>>> Because after "ni" event we could match with some other "ar" events,
>>>> the match is timeouted after 5 seconds.
>>>>
>>>> In FLINK-6208 <https://issues.apache.org/jira/browse/FLINK-6208> we
>>>> introduced third consuming strategy:
>>>>
>>>>- SKIP TILL NEXT(this is the strategy for followedBy right now) -
>>>>the event does not have to occur directly after the previous one but 
>>>> only
>>>>one event can be matched
>>>>
>>>> and you can still use SKIP TILL ANY by using followedByAny. I believe
>>>> the SKIP TILL NEXT strategy is the one you expected.
>>>> You can check it on master branch. We did introduce lots of new
>>>> features and bugfixes to CEP for 1.3.0 version so any comments,
>>>> tests or suggestions are welcome.
>>>>
>>>>
>>>> Z pozdrowieniami! / Cheers!
>>>>
>>>> Dawid Wysakowicz
>>>> *Data/Software Engineer*
>>>> Skype: dawid_wys | Twitter: @OneMoreCoder
>>>> <http://getindata.com/>
>>>>
>>>> 2017-04-29 12:14 GMT+02:00 Moiz S Jinia <moiz.ji...@gmail.com>:
>>>>
>>>>> When using "next", this pattern works fine for the both a match as
>>>>> well as a timeout:
>>>>>
>>>>> Pattern<Event, Event> pattern = Pattern.begin("start")
>>>>> .where(evt -> evt.value.equals("ni"))
>>>>> .next("last").where(evt -> evt.value.equals("ar")).within
>>>>> (Time.seconds(5));
>>>>>
>>>>> 1. "ni" then "ar" within 5 seconds - triggers match
>>>>> 2. "ni" then no "ar" within 5 seconds - triggers timeout
>>>>>
>>>>> But with "followedBy", this does not behave as expected:
>>>>>
>>>>> Pattern<Event, Event> pattern = Pattern.begin("start")
>>>>> .where(evt -> evt.value.equals("ni"))
>>>>> .followedBy("last").where(evt -> evt.value.equals("ar")).within
>>>>> (Time.seconds(5));
>>>>>
>>>>> "ni" then "ar" within 5 seconds - triggers match and also triggers
>>>>> timeout.
>>>>>
>>>>> Why is the timeout triggered when using followedBy (when there is a
>>>>> match)?
>>>>>
>>>>> Version - 1.1.5.
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>

Re: CEP timeout occurs even for a successful match when using followedBy

[Moiz S Jinia]

Ok I'll try that. Its just that I'd rather use a stable version.
Are there any instructions for building binaries from latest sources?

Moiz

On Sat, Apr 29, 2017 at 10:09 PM, Kostas Kloudas <
k.klou...@data-artisans.com> wrote:

> Hi Moiz,
>
> The skip-till-next is a big change and backporting it does not seem
> feasible.
> Also this would require more general changes to the 1.2 to make it
> compatible with the previous 1.2 versions.
>
> If you want you can already use the 1.3 version by downloading the master
> branch and writing your
> use-case against that. The changes until the final release are going to be
> minor hopefully and we can
> always help you adjust your program accordingly.
>
> Hope this helps,
> Kostas
>
> On Apr 29, 2017, at 6:23 PM, Moiz S Jinia <moiz.ji...@gmail.com> wrote:
>
> Oh ok thats a bit far off. Is there any chance of a backport of
> https://issues.apache.org/jira/browse/FLINK-6208 to the 1.2 branch? I
> require the SKIP_TILL_NEXT behaviour for a production use case that we want
> to use Flink for.
>
> Moiz
>
> On Sat, Apr 29, 2017 at 9:49 PM, Kostas Kloudas <
> k.klou...@data-artisans.com> wrote:
>
>> The 1.3 is scheduled for the beginning of June.
>>
>> Cheers,
>> Kostas
>>
>> On Apr 29, 2017, at 6:16 PM, Moiz S Jinia <moiz.ji...@gmail.com> wrote:
>>
>> Thanks Dawid!
>> Yes thats what i was expecting. I'll give it a try.
>>
>> When do you expect 1.3.0 stable to be out?
>>
>> Moiz
>>
>> On Sat, Apr 29, 2017 at 9:20 PM, Dawid Wysakowicz <
>> wysakowicz.da...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> This is an expected behaviour. After the "ar" event there still may
>>> occur other "ar" event that will also trigger a match.
>>> To be more generic in all versions prior to 1.3.0 there are two
>>> different consuming strategies:
>>>
>>>- STRICT (the next operator) - that accepts only if the event occurs
>>>directly after the previous
>>>- SKIP TILL ANY (the followedBy operator) - it accepts any matching
>>>event following event if there were already an event that matched this
>>>pattern
>>>
>>> Because after "ni" event we could match with some other "ar" events, the
>>> match is timeouted after 5 seconds.
>>>
>>> In FLINK-6208 <https://issues.apache.org/jira/browse/FLINK-6208> we
>>> introduced third consuming strategy:
>>>
>>>- SKIP TILL NEXT(this is the strategy for followedBy right now) -
>>>the event does not have to occur directly after the previous one but only
>>>one event can be matched
>>>
>>> and you can still use SKIP TILL ANY by using followedByAny. I believe
>>> the SKIP TILL NEXT strategy is the one you expected.
>>> You can check it on master branch. We did introduce lots of new features
>>> and bugfixes to CEP for 1.3.0 version so any comments,
>>> tests or suggestions are welcome.
>>>
>>>
>>> Z pozdrowieniami! / Cheers!
>>>
>>> Dawid Wysakowicz
>>> *Data/Software Engineer*
>>> Skype: dawid_wys | Twitter: @OneMoreCoder
>>> <http://getindata.com/>
>>>
>>> 2017-04-29 12:14 GMT+02:00 Moiz S Jinia <moiz.ji...@gmail.com>:
>>>
>>>> When using "next", this pattern works fine for the both a match as well
>>>> as a timeout:
>>>>
>>>> Pattern<Event, Event> pattern = Pattern.begin("start")
>>>> .where(evt -> evt.value.equals("ni"))
>>>> .next("last").where(evt -> evt.value.equals("ar")).within
>>>> (Time.seconds(5));
>>>>
>>>> 1. "ni" then "ar" within 5 seconds - triggers match
>>>> 2. "ni" then no "ar" within 5 seconds - triggers timeout
>>>>
>>>> But with "followedBy", this does not behave as expected:
>>>>
>>>> Pattern<Event, Event> pattern = Pattern.begin("start")
>>>> .where(evt -> evt.value.equals("ni"))
>>>> .followedBy("last").where(evt -> evt.value.equals("ar")).within
>>>> (Time.seconds(5));
>>>>
>>>> "ni" then "ar" within 5 seconds - triggers match and also triggers
>>>> timeout.
>>>>
>>>> Why is the timeout triggered when using followedBy (when there is a
>>>> match)?
>>>>
>>>> Version - 1.1.5.
>>>>
>>>
>>>
>>
>>
>
>

Re: CEP timeout occurs even for a successful match when using followedBy

[Moiz S Jinia]

Oh ok thats a bit far off. Is there any chance of a backport of
https://issues.apache.org/jira/browse/FLINK-6208 to the 1.2 branch? I
require the SKIP_TILL_NEXT behaviour for a production use case that we want
to use Flink for.

Moiz

On Sat, Apr 29, 2017 at 9:49 PM, Kostas Kloudas <k.klou...@data-artisans.com
> wrote:

> The 1.3 is scheduled for the beginning of June.
>
> Cheers,
> Kostas
>
> On Apr 29, 2017, at 6:16 PM, Moiz S Jinia <moiz.ji...@gmail.com> wrote:
>
> Thanks Dawid!
> Yes thats what i was expecting. I'll give it a try.
>
> When do you expect 1.3.0 stable to be out?
>
> Moiz
>
> On Sat, Apr 29, 2017 at 9:20 PM, Dawid Wysakowicz <
> wysakowicz.da...@gmail.com> wrote:
>
>> Hi,
>>
>> This is an expected behaviour. After the "ar" event there still may occur
>> other "ar" event that will also trigger a match.
>> To be more generic in all versions prior to 1.3.0 there are two different
>> consuming strategies:
>>
>>- STRICT (the next operator) - that accepts only if the event occurs
>>directly after the previous
>>- SKIP TILL ANY (the followedBy operator) - it accepts any matching
>>event following event if there were already an event that matched this
>>pattern
>>
>> Because after "ni" event we could match with some other "ar" events, the
>> match is timeouted after 5 seconds.
>>
>> In FLINK-6208 <https://issues.apache.org/jira/browse/FLINK-6208> we
>> introduced third consuming strategy:
>>
>>- SKIP TILL NEXT(this is the strategy for followedBy right now) - the
>>event does not have to occur directly after the previous one but only one
>>event can be matched
>>
>> and you can still use SKIP TILL ANY by using followedByAny. I believe the
>> SKIP TILL NEXT strategy is the one you expected.
>> You can check it on master branch. We did introduce lots of new features
>> and bugfixes to CEP for 1.3.0 version so any comments,
>> tests or suggestions are welcome.
>>
>>
>> Z pozdrowieniami! / Cheers!
>>
>> Dawid Wysakowicz
>> *Data/Software Engineer*
>> Skype: dawid_wys | Twitter: @OneMoreCoder
>> <http://getindata.com/>
>>
>> 2017-04-29 12:14 GMT+02:00 Moiz S Jinia <moiz.ji...@gmail.com>:
>>
>>> When using "next", this pattern works fine for the both a match as well
>>> as a timeout:
>>>
>>> Pattern<Event, Event> pattern = Pattern.begin("start")
>>> .where(evt -> evt.value.equals("ni"))
>>> .next("last").where(evt -> evt.value.equals("ar")).within
>>> (Time.seconds(5));
>>>
>>> 1. "ni" then "ar" within 5 seconds - triggers match
>>> 2. "ni" then no "ar" within 5 seconds - triggers timeout
>>>
>>> But with "followedBy", this does not behave as expected:
>>>
>>> Pattern<Event, Event> pattern = Pattern.begin("start")
>>> .where(evt -> evt.value.equals("ni"))
>>> .followedBy("last").where(evt -> evt.value.equals("ar")).within
>>> (Time.seconds(5));
>>>
>>> "ni" then "ar" within 5 seconds - triggers match and also triggers
>>> timeout.
>>>
>>> Why is the timeout triggered when using followedBy (when there is a
>>> match)?
>>>
>>> Version - 1.1.5.
>>>
>>
>>
>
>

Re: CEP timeout occurs even for a successful match when using followedBy

[Moiz S Jinia]

Thanks Dawid!
Yes thats what i was expecting. I'll give it a try.

When do you expect 1.3.0 stable to be out?

Moiz

On Sat, Apr 29, 2017 at 9:20 PM, Dawid Wysakowicz <
wysakowicz.da...@gmail.com> wrote:

> Hi,
>
> This is an expected behaviour. After the "ar" event there still may occur
> other "ar" event that will also trigger a match.
> To be more generic in all versions prior to 1.3.0 there are two different
> consuming strategies:
>
>- STRICT (the next operator) - that accepts only if the event occurs
>directly after the previous
>- SKIP TILL ANY (the followedBy operator) - it accepts any matching
>event following event if there were already an event that matched this
>pattern
>
> Because after "ni" event we could match with some other "ar" events, the
> match is timeouted after 5 seconds.
>
> In FLINK-6208 <https://issues.apache.org/jira/browse/FLINK-6208> we
> introduced third consuming strategy:
>
>- SKIP TILL NEXT(this is the strategy for followedBy right now) - the
>event does not have to occur directly after the previous one but only one
>event can be matched
>
> and you can still use SKIP TILL ANY by using followedByAny. I believe the
> SKIP TILL NEXT strategy is the one you expected.
> You can check it on master branch. We did introduce lots of new features
> and bugfixes to CEP for 1.3.0 version so any comments,
> tests or suggestions are welcome.
>
>
> Z pozdrowieniami! / Cheers!
>
> Dawid Wysakowicz
>
> *Data/Software Engineer*
>
> Skype: dawid_wys | Twitter: @OneMoreCoder
>
> <http://getindata.com/>
>
> 2017-04-29 12:14 GMT+02:00 Moiz S Jinia <moiz.ji...@gmail.com>:
>
>> When using "next", this pattern works fine for the both a match as well
>> as a timeout:
>>
>> Pattern<Event, Event> pattern = Pattern.begin("start")
>> .where(evt -> evt.value.equals("ni"))
>> .next("last").where(evt -> evt.value.equals("ar")).within
>> (Time.seconds(5));
>>
>> 1. "ni" then "ar" within 5 seconds - triggers match
>> 2. "ni" then no "ar" within 5 seconds - triggers timeout
>>
>> But with "followedBy", this does not behave as expected:
>>
>> Pattern<Event, Event> pattern = Pattern.begin("start")
>> .where(evt -> evt.value.equals("ni"))
>> .followedBy("last").where(evt -> evt.value.equals("ar")).within
>> (Time.seconds(5));
>>
>> "ni" then "ar" within 5 seconds - triggers match and also triggers
>> timeout.
>>
>> Why is the timeout triggered when using followedBy (when there is a
>> match)?
>>
>> Version - 1.1.5.
>>
>
>

CEP timeout occurs even for a successful match when using followedBy

[Moiz S Jinia]

When using "next", this pattern works fine for the both a match as well as
a timeout:

Pattern pattern = Pattern.begin("start")
.where(evt -> evt.value.equals("ni"))
.next("last").where(evt ->
evt.value.equals("ar")).within(Time.seconds(5));

1. "ni" then "ar" within 5 seconds - triggers match
2. "ni" then no "ar" within 5 seconds - triggers timeout

But with "followedBy", this does not behave as expected:

Pattern pattern = Pattern.begin("start")
.where(evt -> evt.value.equals("ni"))
.followedBy("last").where(evt ->
evt.value.equals("ar")).within(Time.seconds(5));

"ni" then "ar" within 5 seconds - triggers match and also triggers timeout.

Why is the timeout triggered when using followedBy (when there is a match)?

Version - 1.1.5.