Re: Thoughts from a first time contributor

Glad to hear the readme was useful! I lean towards keeping it in GitHub, but agree there should be more cross linking with the Wiki. Reason being is that README content shows up in the Go Package Doc site: eg. https://pkg.go.dev/github.com/apache/beam/sdks/v2/go Granted we don't make 100% the

Re: Bean 2.36.0 + Flink 1.13 appears to be broken

Hey Tomo, Thanks for the tip! It turns out my deployment project (the one that creates the fat jar) and my pipelines project (the one with actual code) had mismatching Beam versions. User error, sorry about that. Thanks for your help, Cristian On Tue, Feb 8, 2022 at 3:32 PM Tomo Suzuki wrote:

Bean 2.36.0 + Flink 1.13 appears to be broken

Hi everyone, I am very excited with the 2.36 release, especially the stopReadOffset addition to the KafkaSourceDescriptors. With it, I can read sections of a topic and create state,effectively having a bounded kafka source, before reading new items that need processing. Unfortunately, running

Beam Website Feedback

Adapt documentation: https://beam.apache.org/documentation/transforms/python/elementwise/pardo/#example-1-pardo-with-a-simple-dofn ## Can be removed > [BEAM-7885]

P1 issues report (70)

This is your daily summary of Beam's current P1 issues, not including flaky tests (https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20statusCategory%20!%3D%20Done%20AND%20priority%20%3D%20P1%20AND%20(labels%20is%20EMPTY%20OR%20labels%20!%3D%20flake). See

Flaky test issue report (50)

This is your daily summary of Beam's current flaky tests (https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20statusCategory%20!%3D%20Done%20AND%20labels%20%3D%20flake) These are P1 issues because they have a major negative impact on the community and make it hard to

Re: KafkaIO.write and Avro

Thanks a lot Moritz. Your suggestion worked immediately. You sort of get on the wrong track since my favorite IDE suggests: .withValueSerializer((Class>) KafkaAvroSerializer.class) ... which simply doesn't even compile for me. incompatible types: java.lang.Class cannot be converted to

Re: KafkaIO.write and Avro

Just having a quick look, it looks like the respective interface in KafkaIO should rather look like this to support KafkaAvroSerializer, which is a Serializer: public Write withValueSerializer(Class> valueSerializer) Thoughts? Cheers, Moritz From: Moritz Mack Date: Tuesday, 8. February

Re: Developing on an M1 Mac

Go supports ARM64 on Darwin since 1.16, which is the minimum version of Go we currently support. See https://go.dev/blog/ports There are definitely some hardcoded paths we'd need to adjust to build boot containers though. Go 1.18 improves things, and since it has the initial run of Go Generics,

Re: KafkaIO.write and Avro

Hi Matt, Unfortunately, the types don’t play well when using KafkaAvroSerializer. It currently requires a cast :/ The following will work: write.withValueSerializer((Class)KafkaAvroSerializer.class)) This seems to be the cause of repeated confusion, so probably worth improving the user

Re: Developing on an M1 Mac

Just for your information: Thanks to that change - i will soon be adding ARM support for Apache Airflow - including building and publishing the images and running our tests (using self-hosted runners). As soon as I get it I will be able to share the code/experiences with you. J On Tue, Feb 8,

Re: Developing on an M1 Mac

For awareness with the just released Beam 2.36.0 Beam works out of the box to develop on a Mac M1. I tried Java and Python pipelines with success running locally on both Flink/Spark runner. I found one issue using zstd and created [1] that was merged today, with this the sdks:core tests and Spark

Re: [ANNOUNCE] Apache Beam 2.36.0 Release

Great work Emily and everyone! I am glad to see that with the dependency updates this is the first Beam release that works correctly out of the box on ARM64, I tried some helloword examples on a Mac M1 with both Java and Python and it works ok. Ismaël On Tue, Feb 8, 2022 at 9:49 AM Jarek

KafkaIO.write and Avro

Dear Beams, When sending Avro values to Kafka, say GenericRecord, you typically specify option value.serializer as "io.confluent.kafka.serializers.KafkaAvroSerializer". This along with a bunch of other options for authentication and so on verifies the schema stored in the Avro record with a

Re: [ANNOUNCE] Apache Beam 2.36.0 Release

Thanks a lot for that Emily! It's been a release we were waiting for at Apache Airflow. I believe It will unblock a number of "modernizations" in our pipeline - Python 3.10, ARM support were quite a bit depending on it (mostly through numpy transitive dependency limitation). Great to see this one