To further add to this problem, I've now got our ops team to set
transaction.max.timeout.ms on our Kafka brokers to 1 hour (as suggested by
the Flink docs). However the problem persists and I'm still getting
the same error message.
I've confirmed that this config setting is actually set on the Kafka
brokers as well to rule out any mistakes there.
Best,
Tim
On Thu, 12 Nov 2020 at 14:46, Tim Josefsson <tim.josefs...@webstep.se>
wrote:
> Also realized I had a typo in the config dump I did in the previous email
> (the one from the 10th). If I don't do
> Properties producerProps = new Properties();
> producerProps.setProperty("transaction.timeout.ms", "900000");
> Then the value reported from the ProducerConfig is 3600000 and not 60000
> as I had written.
>
>
>
> On Thu, 12 Nov 2020 at 13:37, Tim Josefsson <tim.josefs...@webstep.se>
> wrote:
>
>> Sure, I've attached it to this email. The process seems to restart once
>> the TimeoutException happens so it's repeated a couple of times.
>>
>> Thanks for looking at it!
>>
>> /Tim
>>
>> On Wed, 11 Nov 2020 at 10:37, Aljoscha Krettek <aljos...@apache.org>
>> wrote:
>>
>>> Hmm, could you please post the full stack trace that leads to the
>>> TimeoutException?
>>>
>>> Best,
>>> Aljoscha
>>>
>>> On 10.11.20 17:54, Tim Josefsson wrote:
>>> > Hey Aljoscha,
>>> >
>>> > I'm setting the transaction.timeout.ms when I create the
>>> FlinkKafkaProducer:
>>> >
>>> > I create a Properties object and then set the property and finally add
>>> > those properties when creating the producer.
>>> >
>>> > Properties producerProps = new Properties();
>>> > producerProps.setProperty("transaction.timeout.ms", "900000");
>>> >
>>> > If I don't set that property my I instead get the following config when
>>> > starting the job:
>>> > 11:41:56,345 INFO org.apache.kafka.clients.producer.ProducerConfig
>>> > - ProducerConfig values:
>>> > acks = 1
>>> > [omitted for brevity]
>>> > transaction.timeout.ms = 60000
>>> > transactional.id = Source: Read player events from Kafka -> Map
>>> > Json to HashMap -> Add CanonicalTime as timestamp -> Filter dates not
>>> > needed for backfill -> Sink: Post events to playerEvents
>>> > Kafka-a15b4dd4812495cebdc94e33125ef858-1
>>> > value.serializer = class
>>> > org.apache.kafka.common.serialization.ByteArraySerializer
>>> >
>>> > So I imagine the Producer is picking up the change but it still returns
>>> > errors when running the job.
>>> >
>>> > Best regards,
>>> > Tim
>>> >
>>> >
>>> > On Tue, 10 Nov 2020 at 16:14, Aljoscha Krettek <aljos...@apache.org>
>>> wrote:
>>> >
>>> >> On 10.11.20 11:53, Tim Josefsson wrote:
>>> >>> Also when checking my logs I see the following message:
>>> >>> 11:41:56,345 INFO org.apache.kafka.clients.producer.ProducerConfig
>>> >>> - ProducerConfig values:
>>> >>> acks = 1
>>> >>> [omitted for brevity]
>>> >>> transaction.timeout.ms = 900000
>>> >>> transactional.id = Source: Read player events from Kafka -> Map
>>> >>> Json to HashMap -> Add CanonicalTime as timestamp -> Filter dates
>>> not
>>> >>> needed for backfill -> Sink: Post events to playerEvents
>>> >>> Kafka-a15b4dd4812495cebdc94e33125ef858-1
>>> >>> value.serializer = class
>>> >>> org.apache.kafka.common.serialization.ByteArraySerializer
>>> >>
>>> >> The interesting thing would be to figure out where that
>>> >> `transaction.timeout.ms = 900000` is coming from. The default from
>>> Flink
>>> >> would be 60000, if nothing is configured. Are you specifying that
>>> value,
>>> >> maybe from the commandline or in code?
>>> >>
>>> >> Maybe it's a funny coincidence, but our StreamingKafkaITCase also
>>> >> specifies that timeout value.
>>> >>
>>> >> Best,
>>> >> Aljoscha
>>> >>
>>> >>
>>> >
>>>
>>>
>>
>> --
>>
>> *Tim Josefsson*
>> [image: Webstep GPtW] <http://www.webstep.se/>
>> mobil +46 (0) 707 81 91 12
>> telefon +46 (0) 8 21 40 70
>>
>> tim.josefs...@webstep.se
>> *webstep.se <http://www.webstep.se/>*
>> Suttungs gränd 2
>> 753 19 Uppsala
>> Stockholm | Uppsala | Malmö | Sundsvall | Oslo
>> Bergen | Stavanger | Trondheim | Kristiansand
>> [image: LinkedIn] <http://www.linkedin.com/company/webstep-ab> [image:
>> Facebook] <http://www.facebook.com/webstepAB> [image: Facebook]
>> <http://www.instagram.com/webstep_sverige>
>>
>
>
> --
>
> *Tim Josefsson*
> [image: Webstep GPtW] <http://www.webstep.se/>
> mobil +46 (0) 707 81 91 12
> telefon +46 (0) 8 21 40 70
>
> tim.josefs...@webstep.se
> *webstep.se <http://www.webstep.se/>*
> Suttungs gränd 2
> 753 19 Uppsala
> Stockholm | Uppsala | Malmö | Sundsvall | Oslo
> Bergen | Stavanger | Trondheim | Kristiansand
> [image: LinkedIn] <http://www.linkedin.com/company/webstep-ab> [image:
> Facebook] <http://www.facebook.com/webstepAB> [image: Facebook]
> <http://www.instagram.com/webstep_sverige>
>
--
*Tim Josefsson*
[image: Webstep GPtW] <http://www.webstep.se/>
mobil +46 (0) 707 81 91 12
telefon +46 (0) 8 21 40 70
tim.josefs...@webstep.se
*webstep.se <http://www.webstep.se/>*
Suttungs gränd 2
753 19 Uppsala
Stockholm | Uppsala | Malmö | Sundsvall | Oslo
Bergen | Stavanger | Trondheim | Kristiansand
[image: LinkedIn] <http://www.linkedin.com/company/webstep-ab> [image:
Facebook] <http://www.facebook.com/webstepAB> [image: Facebook]
<http://www.instagram.com/webstep_sverige>
