Re: Unable to read value from state/Unable to fetch data due to token mismatch for key

2020-07-09 Thread Mohil Khare 
Thanks Reuven for your reply.
Good to know that it is benign.

Regards
Mohil

On Wed, Jul 8, 2020 at 10:19 PM Reuven Lax  wrote:

> This error should be benign. It often means that ownership of the work
> item was moved to a different worker (possibly caused by autoscaling or
> other source of work rebalancing), so the in-progress work item on that
> worker failed. However the work item will be processed successfully on the
> new worker that owns it. This should not cause a persistent failure.
>
> On Wed, Jul 8, 2020 at 9:53 PM Mohil Khare  wrote:
>
>> Hello,
>>
>> I am using beam java sdk 2.19.0 (with enableStreamingEngine set as true)
>> and very heavily use stateful beam processing model.
>> However, sometimes I am seeing the following exception while reading
>> value from state for a key (Please note : here my key is a POJO where
>> fields create a kind of composite key. Also I am using AvroCoder for this
>> key):
>>
>> Caused by: java.util.concurrent.ExecutionException:
>> org.apache.beam.runners.dataflow.worker.KeyTokenInvalidException: Unable to
>> fetch data due to token mismatch for key
>> 0ggadot_static@prosimo.ioHaa552bec-25f2-11ea-8705-267acc424a25H9219bdd5-335f-11ea-bd4f-de07a30b09ca
>> @ OC-AU sydney
>>
>>1.
>>   1. at
>>   
>> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.
>>   AbstractFuture.getDoneValue (AbstractFuture.java:531
>>   
>> 
>>   )
>>   2. at
>>   
>> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.
>>   AbstractFuture.get (AbstractFuture.java:492
>>   
>> 
>>   )
>>   3. at
>>   
>> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.
>>   AbstractFuture$TrustedFuture.get (AbstractFuture.java:83
>>   
>> 
>>   )
>>   4. at
>>   
>> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.
>>   ForwardingFuture.get (ForwardingFuture.java:62
>>   
>> 
>>   )
>>   5. atorg.apache.beam.runners.dataflow.worker.
>>   WindmillStateReader$WrappedFuture.get (WindmillStateReader.java:316
>>   
>> 
>>   )
>>   6. atorg.apache.beam.runners.dataflow.worker.
>>   WindmillStateInternals$WindmillValue.read (
>>   WindmillStateInternals.java:385
>>   
>> 
>>   )
>>
>> Caused by: org.apache.beam.runners.dataflow.worker.KeyTokenInvalidException:
>> Unable to fetch data due to token mismatch for key 
>>
>>1.
>>   1. atorg.apache.beam.runners.dataflow.worker.
>>   WindmillStateReader.consumeResponse (WindmillStateReader.java:482
>>   
>> 
>>   )
>>   2. atorg.apache.beam.runners.dataflow.worker.
>>   WindmillStateReader.startBatchAndBlock (
>>   WindmillStateReader.java:420
>>   
>> 
>>   )
>>   3. atorg.apache.beam.runners.dataflow.worker.
>>   WindmillStateReader$WrappedFuture.get (WindmillStateReader.java:313
>>   
>> 
>>   )
>>
>>
>> Any help to fix this issue would be greatly appreciated.
>>
>> Thanks and Regards
>> Mohil
>>
>

Re: Unable to read value from state/Unable to fetch data due to token mismatch for key

2020-07-08 Thread Reuven Lax 
This error should be benign. It often means that ownership of the work item
was moved to a different worker (possibly caused by autoscaling or other
source of work rebalancing), so the in-progress work item on that worker
failed. However the work item will be processed successfully on the new
worker that owns it. This should not cause a persistent failure.

On Wed, Jul 8, 2020 at 9:53 PM Mohil Khare  wrote:

> Hello,
>
> I am using beam java sdk 2.19.0 (with enableStreamingEngine set as true)
> and very heavily use stateful beam processing model.
> However, sometimes I am seeing the following exception while reading value
> from state for a key (Please note : here my key is a POJO where fields
> create a kind of composite key. Also I am using AvroCoder for this key):
>
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.beam.runners.dataflow.worker.KeyTokenInvalidException: Unable to
> fetch data due to token mismatch for key
> 0ggadot_static@prosimo.ioHaa552bec-25f2-11ea-8705-267acc424a25H9219bdd5-335f-11ea-bd4f-de07a30b09ca
> @ OC-AU sydney
>
>1.
>   1. at
>   
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.
>   AbstractFuture.getDoneValue (AbstractFuture.java:531
>   
> 
>   )
>   2. at
>   
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.
>   AbstractFuture.get (AbstractFuture.java:492
>   
> 
>   )
>   3. at
>   
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.
>   AbstractFuture$TrustedFuture.get (AbstractFuture.java:83
>   
> 
>   )
>   4. at
>   
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.
>   ForwardingFuture.get (ForwardingFuture.java:62
>   
> 
>   )
>   5. atorg.apache.beam.runners.dataflow.worker.
>   WindmillStateReader$WrappedFuture.get (WindmillStateReader.java:316
>   
> 
>   )
>   6. atorg.apache.beam.runners.dataflow.worker.
>   WindmillStateInternals$WindmillValue.read (
>   WindmillStateInternals.java:385
>   
> 
>   )
>
> Caused by: org.apache.beam.runners.dataflow.worker.KeyTokenInvalidException:
> Unable to fetch data due to token mismatch for key 
>
>1.
>   1. atorg.apache.beam.runners.dataflow.worker.
>   WindmillStateReader.consumeResponse (WindmillStateReader.java:482
>   
> 
>   )
>   2. atorg.apache.beam.runners.dataflow.worker.
>   WindmillStateReader.startBatchAndBlock (WindmillStateReader.java:420
>   
> 
>   )
>   3. atorg.apache.beam.runners.dataflow.worker.
>   WindmillStateReader$WrappedFuture.get (WindmillStateReader.java:313
>   
> 
>   )
>
>
> Any help to fix this issue would be greatly appreciated.
>
> Thanks and Regards
> Mohil
>

Unable to read value from state/Unable to fetch data due to token mismatch for key

2020-07-08 Thread Mohil Khare 
Hello,

I am using beam java sdk 2.19.0 (with enableStreamingEngine set as true)
and very heavily use stateful beam processing model.
However, sometimes I am seeing the following exception while reading value
from state for a key (Please note : here my key is a POJO where fields
create a kind of composite key. Also I am using AvroCoder for this key):

Caused by: java.util.concurrent.ExecutionException:
org.apache.beam.runners.dataflow.worker.KeyTokenInvalidException: Unable to
fetch data due to token mismatch for key
0ggadot_static@prosimo.ioHaa552bec-25f2-11ea-8705-267acc424a25H9219bdd5-335f-11ea-bd4f-de07a30b09ca
@ OC-AU sydney

   1.
  1. at
  org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.
  AbstractFuture.getDoneValue (AbstractFuture.java:531
  

  )
  2. at
  org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.
  AbstractFuture.get (AbstractFuture.java:492
  

  )
  3. at
  org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.
  AbstractFuture$TrustedFuture.get (AbstractFuture.java:83
  

  )
  4. at
  org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.
  ForwardingFuture.get (ForwardingFuture.java:62
  

  )
  5. atorg.apache.beam.runners.dataflow.worker.
  WindmillStateReader$WrappedFuture.get (WindmillStateReader.java:316
  

  )
  6. atorg.apache.beam.runners.dataflow.worker.
  WindmillStateInternals$WindmillValue.read (
  WindmillStateInternals.java:385
  

  )

Caused by: org.apache.beam.runners.dataflow.worker.KeyTokenInvalidException:
Unable to fetch data due to token mismatch for key 

   1.
  1. atorg.apache.beam.runners.dataflow.worker.
  WindmillStateReader.consumeResponse (WindmillStateReader.java:482
  

  )
  2. atorg.apache.beam.runners.dataflow.worker.
  WindmillStateReader.startBatchAndBlock (WindmillStateReader.java:420
  

  )
  3. atorg.apache.beam.runners.dataflow.worker.
  WindmillStateReader$WrappedFuture.get (WindmillStateReader.java:313
  

  )


Any help to fix this issue would be greatly appreciated.

Thanks and Regards
Mohil