Re: Incomplete Beam Schema -> Avro Schema conversion

2022-09-09 Thread Balázs Németh
776 > > On Wed, Aug 17, 2022 at 2:53 PM Balázs Németh > wrote: > >> java.lang.RuntimeException: Unhandled logical type >> beam:logical_type:date:v1 >> at >> org.apache.beam.sdk.schemas.utils.AvroUtils.getFieldSchema(AvroUtils.java:943) >> at

Incomplete Beam Schema -> Avro Schema conversion

2022-08-17 Thread Balázs Németh
java.lang.RuntimeException: Unhandled logical type beam:logical_type:date:v1 at org.apache.beam.sdk.schemas.utils.AvroUtils.getFieldSchema(AvroUtils.java:943) at org.apache.beam.sdk.schemas.utils.AvroUtils.toAvroField(AvroUtils.java:306) at

Re: KafkaIO.java.configuredKafkaCommit() inconsistency

2022-08-08 Thread Balázs Németh
exactly once applications. >> >> >> >> On Mon, Aug 8, 2022 at 3:16 PM Balázs Németh >> wrote: >> >>> I have been reading from Kafka and trying to figure out which offset >>> management would be the best for my

KafkaIO.java.configuredKafkaCommit() inconsistency

2022-08-08 Thread Balázs Németh
I have been reading from Kafka and trying to figure out which offset management would be the best for my use-case. During that I noticed something odd.

Re: Questions regarding contribution: Support for reading Kafka topics from any startReadTime in Java

2022-06-28 Thread Balázs Németh
good to try to come up with some way of describing > this as an invariant like "the observed behavior of startReadTime is the > same even when X Y Z are different". > > Kenn > > On Wed, Jun 15, 2022 at 3:43 PM Balázs Németh > wrote: > >> Well, a sanity c

Re: Questions regarding contribution: Support for reading Kafka topics from any startReadTime in Java

2022-06-15 Thread Balázs Németh
ute. Your logic will start returning a lot of records with > values == now + 1ms. Is that reasonable for users of this API? Maybe, maybe > not. > > -Daniel > > On Wed, Jun 15, 2022 at 6:16 PM Balázs Németh > wrote: > >> Not a single objection means my idea seems ok

Re: Questions regarding contribution: Support for reading Kafka topics from any startReadTime in Java

2022-06-15 Thread Balázs Németh
Not a single objection means my idea seems okay then? :) Balázs Németh ezt írta (időpont: 2022. máj. 26., Cs, 5:59): > https://issues.apache.org/jira/browse/BEAM-14518 > > > https://github.com/apache/beam/blob/fd8546355523f67eaddc22249606fdb982fe4938/sdks/java/io/kafka/src/main/jav

Questions regarding contribution: Support for reading Kafka topics from any startReadTime in Java

2022-05-25 Thread Balázs Németh
https://issues.apache.org/jira/browse/BEAM-14518 https://github.com/apache/beam/blob/fd8546355523f67eaddc22249606fdb982fe4938/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/ConsumerSpEL.java#L180-L198 Right now the 'startReadTime' config for KafkaIO.Read looks up an offset in