Re: Implicit Flink Context Documentation

[Padarn Wilson]

Thanks for the clarification. I'll dig in then!

On Mon, 16 Mar 2020, 3:47 pm Piotr Nowojski,  wrote:

> Hi,
>
> We are not maintaining internal docs. We have design docs for newly
> proposed features (previously informal design docs published on dev mailing
> list and recently as FLIP documents [1]), but keyed state is such an old
> concept that dates back so much into the past, that I’m pretty sure it pre
> dates any of that. So you would have to digg through the code if you want
> to understand it.
>
> Piotrek
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
>
> On 13 Mar 2020, at 16:14, Padarn Wilson  wrote:
>
> Thanks Piotr,
>
> Conceptually I understand (and use) the key'ed state quite a lot, but the
> implementation details are what I was looking for.
>
> It looks like
> `org.apache.flink.streaming.api.operators.AbstractStreamOperator#setKeyContextElement1`
> is what I'm looking for though. It would be cool if there were some
> internals design doc however? Quite hard to dig through the code as there
> is a log tied to how the execution of the job actually happens.
>
> Padarn
>
> On Fri, Mar 13, 2020 at 9:43 PM Piotr Nowojski 
> wrote:
>
>> Hi,
>>
>> Please take a look for example here:
>>
>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/state.html#keyed-state
>> And the example in particular
>>
>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/state.html#using-managed-keyed-state
>>
>> The part about "there is a specific key implicitly in context” might be
>> referring to the fact, that for every instance of `CountWindowAverage` that
>> will be running in the cluster, user doesn’t have to set the key context
>> explicility. Flink will set the the key context automatically for the
>> `ValueState> sum;` before any invocation of
>> `CountWindowAverage#flatMap` method.
>>
>> In other words, one parallel instance of `CountWindowAverage` function,
>> for two consecutive invocations of `CountWindowAverage#flatMap` can be
>> referring to different underlying value of `CountWindowAverage#sum` field.
>> For details you could take a look at
>> `org.apache.flink.streaming.api.operators.AbstractStreamOperator#setKeyContextElement1`
>> method and how it’s being used/implemented.
>>
>> I hope that helps.
>>
>> Piotrek
>>
>> On 13 Mar 2020, at 08:20, Padarn Wilson  wrote:
>>
>> Hi Users,
>>
>> I am trying to understand the details of how some aspects of Flink work.
>>
>> While understanding `keyed state` I kept coming up against a claim that 
>> `there
>> is a specific key implicitly in context` I would like to understand how
>> this works, which I'm guessing means understanding the details of the
>> runtime context: Is there any documentation or FLIP someone can recommend
>> on this?
>>
>>
>>
>

Re: Implicit Flink Context Documentation

[Piotr Nowojski]

Hi,

We are not maintaining internal docs. We have design docs for newly proposed 
features (previously informal design docs published on dev mailing list and 
recently as FLIP documents [1]), but keyed state is such an old concept that 
dates back so much into the past, that I’m pretty sure it pre dates any of 
that. So you would have to digg through the code if you want to understand it.

Piotrek

[1] 
https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals 


> On 13 Mar 2020, at 16:14, Padarn Wilson  wrote:
> 
> Thanks Piotr,
> 
> Conceptually I understand (and use) the key'ed state quite a lot, but the 
> implementation details are what I was looking for.
> 
> It looks like 
> `org.apache.flink.streaming.api.operators.AbstractStreamOperator#setKeyContextElement1`
>  is what I'm looking for though. It would be cool if there were some 
> internals design doc however? Quite hard to dig through the code as there is 
> a log tied to how the execution of the job actually happens.
> 
> Padarn
> 
> On Fri, Mar 13, 2020 at 9:43 PM Piotr Nowojski  > wrote:
> Hi,
> 
> Please take a look for example here:
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/state.html#keyed-state
>  
> 
> And the example in particular
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/state.html#using-managed-keyed-state
>  
> 
> 
> The part about "there is a specific key implicitly in context” might be 
> referring to the fact, that for every instance of `CountWindowAverage` that 
> will be running in the cluster, user doesn’t have to set the key context 
> explicility. Flink will set the the key context automatically for the 
> `ValueState> sum;` before any invocation of 
> `CountWindowAverage#flatMap` method.
> 
> In other words, one parallel instance of `CountWindowAverage` function, for 
> two consecutive invocations of `CountWindowAverage#flatMap` can be referring 
> to different underlying value of `CountWindowAverage#sum` field. For details 
> you could take a look at 
> `org.apache.flink.streaming.api.operators.AbstractStreamOperator#setKeyContextElement1`
>  method and how it’s being used/implemented.
> 
> I hope that helps.
> 
> Piotrek
> 
>> On 13 Mar 2020, at 08:20, Padarn Wilson > > wrote:
>> 
>> Hi Users,
>> 
>> I am trying to understand the details of how some aspects of Flink work.
>> 
>> While understanding `keyed state` I kept coming up against a claim that 
>> `there is a specific key implicitly in context` I would like to understand 
>> how this works, which I'm guessing means understanding the details of the 
>> runtime context: Is there any documentation or FLIP someone can recommend on 
>> this?  
> 

Re: Implicit Flink Context Documentation

[Padarn Wilson]

Thanks Piotr,

Conceptually I understand (and use) the key'ed state quite a lot, but the
implementation details are what I was looking for.

It looks like
`org.apache.flink.streaming.api.operators.AbstractStreamOperator#setKeyContextElement1`
is what I'm looking for though. It would be cool if there were some
internals design doc however? Quite hard to dig through the code as there
is a log tied to how the execution of the job actually happens.

Padarn

On Fri, Mar 13, 2020 at 9:43 PM Piotr Nowojski  wrote:

> Hi,
>
> Please take a look for example here:
>
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/state.html#keyed-state
> And the example in particular
>
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/state.html#using-managed-keyed-state
>
> The part about "there is a specific key implicitly in context” might be
> referring to the fact, that for every instance of `CountWindowAverage` that
> will be running in the cluster, user doesn’t have to set the key context
> explicility. Flink will set the the key context automatically for the
> `ValueState> sum;` before any invocation of
> `CountWindowAverage#flatMap` method.
>
> In other words, one parallel instance of `CountWindowAverage` function,
> for two consecutive invocations of `CountWindowAverage#flatMap` can be
> referring to different underlying value of `CountWindowAverage#sum` field.
> For details you could take a look at
> `org.apache.flink.streaming.api.operators.AbstractStreamOperator#setKeyContextElement1`
> method and how it’s being used/implemented.
>
> I hope that helps.
>
> Piotrek
>
> On 13 Mar 2020, at 08:20, Padarn Wilson  wrote:
>
> Hi Users,
>
> I am trying to understand the details of how some aspects of Flink work.
>
> While understanding `keyed state` I kept coming up against a claim that `there
> is a specific key implicitly in context` I would like to understand how
> this works, which I'm guessing means understanding the details of the
> runtime context: Is there any documentation or FLIP someone can recommend
> on this?
>
>
>

Re: Implicit Flink Context Documentation

[Piotr Nowojski]

Hi,

Please take a look for example here:
https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/state.html#keyed-state
 

And the example in particular
https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/state.html#using-managed-keyed-state
 


The part about "there is a specific key implicitly in context” might be 
referring to the fact, that for every instance of `CountWindowAverage` that 
will be running in the cluster, user doesn’t have to set the key context 
explicility. Flink will set the the key context automatically for the 
`ValueState> sum;` before any invocation of 
`CountWindowAverage#flatMap` method.

In other words, one parallel instance of `CountWindowAverage` function, for two 
consecutive invocations of `CountWindowAverage#flatMap` can be referring to 
different underlying value of `CountWindowAverage#sum` field. For details you 
could take a look at 
`org.apache.flink.streaming.api.operators.AbstractStreamOperator#setKeyContextElement1`
 method and how it’s being used/implemented.

I hope that helps.

Piotrek

> On 13 Mar 2020, at 08:20, Padarn Wilson  wrote:
> 
> Hi Users,
> 
> I am trying to understand the details of how some aspects of Flink work.
> 
> While understanding `keyed state` I kept coming up against a claim that 
> `there is a specific key implicitly in context` I would like to understand 
> how this works, which I'm guessing means understanding the details of the 
> runtime context: Is there any documentation or FLIP someone can recommend on 
> this?