Also realized I had a typo in the config dump I did in the previous email (the one from the 10th). If I don't do Properties producerProps = new Properties(); producerProps.setProperty("transaction.timeout.ms", "900000"); Then the value reported from the ProducerConfig is 3600000 and not 60000 as I had written.
On Thu, 12 Nov 2020 at 13:37, Tim Josefsson <tim.josefs...@webstep.se> wrote: > Sure, I've attached it to this email. The process seems to restart once > the TimeoutException happens so it's repeated a couple of times. > > Thanks for looking at it! > > /Tim > > On Wed, 11 Nov 2020 at 10:37, Aljoscha Krettek <aljos...@apache.org> > wrote: > >> Hmm, could you please post the full stack trace that leads to the >> TimeoutException? >> >> Best, >> Aljoscha >> >> On 10.11.20 17:54, Tim Josefsson wrote: >> > Hey Aljoscha, >> > >> > I'm setting the transaction.timeout.ms when I create the >> FlinkKafkaProducer: >> > >> > I create a Properties object and then set the property and finally add >> > those properties when creating the producer. >> > >> > Properties producerProps = new Properties(); >> > producerProps.setProperty("transaction.timeout.ms", "900000"); >> > >> > If I don't set that property my I instead get the following config when >> > starting the job: >> > 11:41:56,345 INFO org.apache.kafka.clients.producer.ProducerConfig >> > - ProducerConfig values: >> > acks = 1 >> > [omitted for brevity] >> > transaction.timeout.ms = 60000 >> > transactional.id = Source: Read player events from Kafka -> Map >> > Json to HashMap -> Add CanonicalTime as timestamp -> Filter dates not >> > needed for backfill -> Sink: Post events to playerEvents >> > Kafka-a15b4dd4812495cebdc94e33125ef858-1 >> > value.serializer = class >> > org.apache.kafka.common.serialization.ByteArraySerializer >> > >> > So I imagine the Producer is picking up the change but it still returns >> > errors when running the job. >> > >> > Best regards, >> > Tim >> > >> > >> > On Tue, 10 Nov 2020 at 16:14, Aljoscha Krettek <aljos...@apache.org> >> wrote: >> > >> >> On 10.11.20 11:53, Tim Josefsson wrote: >> >>> Also when checking my logs I see the following message: >> >>> 11:41:56,345 INFO org.apache.kafka.clients.producer.ProducerConfig >> >>> - ProducerConfig values: >> >>> acks = 1 >> >>> [omitted for brevity] >> >>> transaction.timeout.ms = 900000 >> >>> transactional.id = Source: Read player events from Kafka -> Map >> >>> Json to HashMap -> Add CanonicalTime as timestamp -> Filter dates >> not >> >>> needed for backfill -> Sink: Post events to playerEvents >> >>> Kafka-a15b4dd4812495cebdc94e33125ef858-1 >> >>> value.serializer = class >> >>> org.apache.kafka.common.serialization.ByteArraySerializer >> >> >> >> The interesting thing would be to figure out where that >> >> `transaction.timeout.ms = 900000` is coming from. The default from >> Flink >> >> would be 60000, if nothing is configured. Are you specifying that >> value, >> >> maybe from the commandline or in code? >> >> >> >> Maybe it's a funny coincidence, but our StreamingKafkaITCase also >> >> specifies that timeout value. >> >> >> >> Best, >> >> Aljoscha >> >> >> >> >> > >> >> > > -- > > *Tim Josefsson* > [image: Webstep GPtW] <http://www.webstep.se/> > mobil +46 (0) 707 81 91 12 > telefon +46 (0) 8 21 40 70 > > tim.josefs...@webstep.se > *webstep.se <http://www.webstep.se/>* > Suttungs gränd 2 > 753 19 Uppsala > Stockholm | Uppsala | Malmö | Sundsvall | Oslo > Bergen | Stavanger | Trondheim | Kristiansand > [image: LinkedIn] <http://www.linkedin.com/company/webstep-ab> [image: > Facebook] <http://www.facebook.com/webstepAB> [image: Facebook] > <http://www.instagram.com/webstep_sverige> > -- *Tim Josefsson* [image: Webstep GPtW] <http://www.webstep.se/> mobil +46 (0) 707 81 91 12 telefon +46 (0) 8 21 40 70 tim.josefs...@webstep.se *webstep.se <http://www.webstep.se/>* Suttungs gränd 2 753 19 Uppsala Stockholm | Uppsala | Malmö | Sundsvall | Oslo Bergen | Stavanger | Trondheim | Kristiansand [image: LinkedIn] <http://www.linkedin.com/company/webstep-ab> [image: Facebook] <http://www.facebook.com/webstepAB> [image: Facebook] <http://www.instagram.com/webstep_sverige>