Re: Easy Multi-language via a SchemaTransform-aware Expansion Service

[Chamikara Jayalath via dev]

I think the *DiscoverSchemaTransform()* RPC introduced in this proposal and
the ability to easily deploy/use available *SchemaTransforms* using an
expansion service essentially provide the tooling necessary for
implementing such a service. Such a service could even startup expansion
services to discover/list transforms available in given artifacts (for
example, jar files).

Thanks,
Cham

On Mon, Aug 8, 2022 at 3:48 PM Byron Ellis  wrote:

> I like that idea, sort of like Kafka’s Schema Service but for transforms?
>
> On Mon, Aug 8, 2022 at 2:45 PM Robert Bradshaw via dev <
> dev@beam.apache.org> wrote:
>
>> This is a great idea. I would like to approach this from the
>> perspective of making it easy to provide a catalog of well-defined
>> transforms for use in expansion services from typical SDKs and also
>> elsewhere (e.g. for documentation purposes, GUIs, etc.) Ideally
>> everything about what a transform is (its config, documentation,
>> expectations on inputs, etc.) can be specified programmatically in a
>> way that's much easier to both author and consume than it is now.
>>
>> On Thu, Aug 4, 2022 at 6:51 PM Chamikara Jayalath via dev
>>  wrote:
>> >
>> > Hi All,
>> >
>> > I believe we can make the multi-language pipelines offering [1] much
>> easier to use by updating the expansion service to be fully aware of
>> SchemaTransforms. Additionally this will make it easy to
>> register/discover/use transforms defined in one SDK from all other SDKs.
>> Specifically we could add the following features.
>> >
>> > Expansion service can be used to easily initialize and expand
>> transforms without need for additional code.
>> > Expansion service can be used to easily discover already registered
>> transforms.
>> > Pipeline SDKs can generate user-friendly stub-APIs based on transforms
>> registered with an expansion service, eliminating the need to develop
>> language-specific wrappers.
>> >
>> > Please see here for my proposal:
>> https://s.apache.org/easy-multi-language
>> >
>> > Lemme know if you have any comments/questions/suggestions :)
>> >
>> > Thanks,
>> > Cham
>> >
>> > [1]
>> https://beam.apache.org/documentation/programming-guide/#multi-language-pipelines
>> >
>>
>

Re: Unable to load class 'org.apache.beam.gradle.BeamModulePlugin'

[Pablo Estrada via dev]

Hi there!
Did you try following the instructions here:
https://cwiki.apache.org/confluence/display/BEAM/Set+up+IntelliJ+from+scratch

Or how did you try to load your project into IntelliJ?
Best
-P.

On Mon, Aug 8, 2022 at 8:38 PM 张涛  wrote:

> Hi, I download the source code beam-2.40.0.tar.gz, and my gradle version
> is 7.4.2, JVM is 1.8, like this:
>
> when I used IDEA to open the beam project, the following error occured:
> I'm confused about how to deal with this, I am looking forward for your
> help,  thank you very much!
>

Unable to load class 'org.apache.beam.gradle.BeamModulePlugin'

[张涛]

Hi, I download the source code beam-2.40.0.tar.gz, and my gradle version is 
7.4.2, JVM is 1.8, like this:

when I used IDEA to open the beam project, the following error occured:
I'm confused about how to deal with this, I am looking forward for your help,  
thank you very much!

temp4cj.png
Description: Binary data


temp4cj.png
Description: Binary data

Re: KafkaIO.java.configuredKafkaCommit() inconsistency

[Balázs Németh]

thanks, see https://github.com/apache/beam/issues/22631 +
https://github.com/apache/beam/pull/22633

John Casey via dev  ezt írta (időpont: 2022. aug. 8.,
H, 21:30):

> Which looking at your message again, would imply that the
> configuredKafkaCommit() method shouldn't inspect isolation.level
>
> On Mon, Aug 8, 2022 at 3:27 PM John Casey  wrote:
>
>> .withReadCommitted() doesn't commit messages when read, it instead
>> specifies that the kafka consumer should only read messages that have
>> themselves been committed to kafka.
>>
>> Its use is for exactly once applications.
>>
>>
>>
>> On Mon, Aug 8, 2022 at 3:16 PM Balázs Németh 
>> wrote:
>>
>>> I have been reading from Kafka and trying to figure out which offset
>>> management would be the best for my use-case. During that I noticed
>>> something odd.
>>>
>>>
>>> https://github.com/apache/beam/blob/c9c161d018d116504fd5f83b48853a7071ec9ce4/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L2359-L2362
>>>
>>> private boolean configuredKafkaCommit() {
>>>   return getConsumerConfig().get("isolation.level") ==
>>> "read_committed"
>>>   ||
>>> Boolean.TRUE.equals(getConsumerConfig().get(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG));
>>> }
>>>
>>> https://github.com/apache/beam/blob/c9c161d018d116504fd5f83b48853a7071ec9ce4/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L2292-L2298
>>>
>>> https://github.com/apache/beam/blob/c9c161d018d116504fd5f83b48853a7071ec9ce4/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L2321-L2334
>>>
>>> The name of the method, and how it's being used in the code certainly
>>> suggest that using read_committed isolation level handles and commits
>>> kafka offsets.Seemed strange, but I'm not a Kafka pro, so let's test it.
>>> Well it does not.
>>>
>>> - using ONLY ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG does commit it
>>> - using ONLY commitOffsetsInFinalize() does commit it
>>>
>>> - using ONLY withReadCommitted() does NOT commit it
>>>
>>> Dataflow, 2.40.0 Java SDK, without explicitly enabling SDF-read
>>>
>>> So is it a bug, or what am I missing here?
>>>
>>> If it is indeed a bug, then is it with the read_committed (so it should
>>> commit it although found no explicit documentation about that anywhere), or
>>> having that isolation level shouldn't prefer the commit in the finalize and
>>> that method is wrong?
>>>
>>>
>>>
>>>

Re: Community event cooperation

[Ahmet Altay via dev]

Hi Niko,

Thank you for reaching out. We do have contributors who might be interested
in participating but they might have limited time for participating in an
event. If you can clarify what are you looking for (e.g. speaker? help with
coordination? etc.) people might be able to give you a better answer.

Another note, we have been organizing meetups for a while (with @Danielle
Syse  and @Brittany Hermann  doing
the recent organizations), we could also see if someone from
DolphinScheduler could participate as a speaker in one of those.

A question for my learning, I am not familiar with DolphinScheduler. Are
there users or potential use cases where DolphinScheduler & Beam are used
together?

Thank you!
Ahmet

On Sun, Aug 7, 2022 at 6:13 PM 曾辉  wrote:

> Anyone interested?
>
> 曾辉  于2022年8月4日周四 16:23写道：
>
>> Hey， Developers in the Apache Beam community, How's your day?
>>
>>  I'm Apache DolphinScheduler Community Manager, you can call me Niko,
>> nice to meet you all
>>
>> Apache DolphinScheduler is a worldly renowned data orchestration tool
>> that has largely taken the scheduler market in China. Over 1000 companies,
>> including IBM, Tencent, iFlytek, Meituan, 360, China Unicom, Shein, and SF
>> Express, are relying on its decentralized infrastructure and no-code DAG
>> interface. Apache DolphinScheduler also owns the largest developer
>> community in China and each meetup gathers over 3K attendees.
>>
>> We would love to find partners like Apache Beam to co-host events in the
>> Bay Area, to share our resources with fellow Apache teams.
>>
>> click the link
>> 
>> is an introduction to our community programs. If you are interested in
>> becoming our partner and holding a Meetup together, please contact me in
>> the mail, or schedule a zoom call to discuss the details sometime next
>> week.   looking forward to your reply.
>>
>> I believe that combining the influence of our two sides can let more
>> people know about the Apache Beam open source project.
>>
>> Best，
>> Niko
>>
>

Re: Easy Multi-language via a SchemaTransform-aware Expansion Service

[Byron Ellis via dev]

I like that idea, sort of like Kafka’s Schema Service but for transforms?

On Mon, Aug 8, 2022 at 2:45 PM Robert Bradshaw via dev 
wrote:

> This is a great idea. I would like to approach this from the
> perspective of making it easy to provide a catalog of well-defined
> transforms for use in expansion services from typical SDKs and also
> elsewhere (e.g. for documentation purposes, GUIs, etc.) Ideally
> everything about what a transform is (its config, documentation,
> expectations on inputs, etc.) can be specified programmatically in a
> way that's much easier to both author and consume than it is now.
>
> On Thu, Aug 4, 2022 at 6:51 PM Chamikara Jayalath via dev
>  wrote:
> >
> > Hi All,
> >
> > I believe we can make the multi-language pipelines offering [1] much
> easier to use by updating the expansion service to be fully aware of
> SchemaTransforms. Additionally this will make it easy to
> register/discover/use transforms defined in one SDK from all other SDKs.
> Specifically we could add the following features.
> >
> > Expansion service can be used to easily initialize and expand transforms
> without need for additional code.
> > Expansion service can be used to easily discover already registered
> transforms.
> > Pipeline SDKs can generate user-friendly stub-APIs based on transforms
> registered with an expansion service, eliminating the need to develop
> language-specific wrappers.
> >
> > Please see here for my proposal:
> https://s.apache.org/easy-multi-language
> >
> > Lemme know if you have any comments/questions/suggestions :)
> >
> > Thanks,
> > Cham
> >
> > [1]
> https://beam.apache.org/documentation/programming-guide/#multi-language-pipelines
> >
>

Re: Easy Multi-language via a SchemaTransform-aware Expansion Service

[Robert Bradshaw via dev]

This is a great idea. I would like to approach this from the
perspective of making it easy to provide a catalog of well-defined
transforms for use in expansion services from typical SDKs and also
elsewhere (e.g. for documentation purposes, GUIs, etc.) Ideally
everything about what a transform is (its config, documentation,
expectations on inputs, etc.) can be specified programmatically in a
way that's much easier to both author and consume than it is now.

On Thu, Aug 4, 2022 at 6:51 PM Chamikara Jayalath via dev
 wrote:
>
> Hi All,
>
> I believe we can make the multi-language pipelines offering [1] much easier 
> to use by updating the expansion service to be fully aware of 
> SchemaTransforms. Additionally this will make it easy to 
> register/discover/use transforms defined in one SDK from all other SDKs. 
> Specifically we could add the following features.
>
> Expansion service can be used to easily initialize and expand transforms 
> without need for additional code.
> Expansion service can be used to easily discover already registered 
> transforms.
> Pipeline SDKs can generate user-friendly stub-APIs based on transforms 
> registered with an expansion service, eliminating the need to develop 
> language-specific wrappers.
>
> Please see here for my proposal: https://s.apache.org/easy-multi-language
>
> Lemme know if you have any comments/questions/suggestions :)
>
> Thanks,
> Cham
>
> [1] 
> https://beam.apache.org/documentation/programming-guide/#multi-language-pipelines
>

[RESULT] [VOTE] Vendored Dependencies Release

[Luke Cwik via dev]

I'm happy to announce that we have unanimously approved this release.

There are 3 approving votes, 3 of which are binding:
* Luke Cwik
* Pablo Estrada
* Chamikara Jayalath

There are no disapproving votes.

Thanks everyone!

On Mon, Aug 8, 2022 at 9:47 AM Pablo Estrada  wrote:

> +1
> Thanks!
> -P.
>
> On Mon, Aug 8, 2022 at 9:24 AM Chamikara Jayalath via dev <
> dev@beam.apache.org> wrote:
>
>> +1
>>
>> Thanks,
>> Cham
>>
>> On Fri, Aug 5, 2022 at 1:49 PM Luke Cwik via dev 
>> wrote:
>>
>>> +1
>>>
>>> I verified the signatures of the artifacts, that the jar doesn't contain
>>> classes outside of the org/apache/beam/vendor/grpc/v1p48p1 package and I
>>> tested the artifact against our precommits using
>>> https://github.com/apache/beam/pull/22595
>>>
>>> On Fri, Aug 5, 2022 at 1:42 PM Luke Cwik  wrote:
>>>
 Please review the release of the following artifacts that we vendor:
  * beam-vendor-grpc-1_48_1

 Hi everyone,
 Please review and vote on the release candidate #1 for the version 0.1,
 as follows:
 [ ] +1, Approve the release
 [ ] -1, Do not approve the release (please provide specific comments)


 The complete staging area is available for your review, which includes:
 * the official Apache source release to be deployed to dist.apache.org
 [1], which is signed with the key with fingerprint
 EAD5DE293F4A03DD2E77565589E68A56E371CCA2 [2],
 * all artifacts to be deployed to the Maven Central Repository [3],
 * commit hash "db8db0b6ed0fe1e4891f207f0f7f811798e54db1" [4],

 The vote will be open for at least 72 hours. It is adopted by majority
 approval, with at least 3 PMC affirmative votes.

 Thanks,
 Release Manager

 [1] https://dist.apache.org/repos/dist/dev/beam/vendor/
 [2] https://dist.apache.org/repos/dist/release/beam/KEYS
 [3]
 https://repository.apache.org/content/repositories/orgapachebeam-1277/
 [4]
 https://github.com/apache/beam/commit/db8db0b6ed0fe1e4891f207f0f7f811798e54db1

>>>

Re: KafkaIO.java.configuredKafkaCommit() inconsistency

[John Casey via dev]

Which looking at your message again, would imply that the
configuredKafkaCommit() method shouldn't inspect isolation.level

On Mon, Aug 8, 2022 at 3:27 PM John Casey  wrote:

> .withReadCommitted() doesn't commit messages when read, it instead
> specifies that the kafka consumer should only read messages that have
> themselves been committed to kafka.
>
> Its use is for exactly once applications.
>
>
>
> On Mon, Aug 8, 2022 at 3:16 PM Balázs Németh 
> wrote:
>
>> I have been reading from Kafka and trying to figure out which offset
>> management would be the best for my use-case. During that I noticed
>> something odd.
>>
>>
>> https://github.com/apache/beam/blob/c9c161d018d116504fd5f83b48853a7071ec9ce4/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L2359-L2362
>>
>> private boolean configuredKafkaCommit() {
>>   return getConsumerConfig().get("isolation.level") ==
>> "read_committed"
>>   ||
>> Boolean.TRUE.equals(getConsumerConfig().get(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG));
>> }
>>
>> https://github.com/apache/beam/blob/c9c161d018d116504fd5f83b48853a7071ec9ce4/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L2292-L2298
>>
>> https://github.com/apache/beam/blob/c9c161d018d116504fd5f83b48853a7071ec9ce4/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L2321-L2334
>>
>> The name of the method, and how it's being used in the code certainly
>> suggest that using read_committed isolation level handles and commits
>> kafka offsets.Seemed strange, but I'm not a Kafka pro, so let's test it.
>> Well it does not.
>>
>> - using ONLY ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG does commit it
>> - using ONLY commitOffsetsInFinalize() does commit it
>>
>> - using ONLY withReadCommitted() does NOT commit it
>>
>> Dataflow, 2.40.0 Java SDK, without explicitly enabling SDF-read
>>
>> So is it a bug, or what am I missing here?
>>
>> If it is indeed a bug, then is it with the read_committed (so it should
>> commit it although found no explicit documentation about that anywhere), or
>> having that isolation level shouldn't prefer the commit in the finalize and
>> that method is wrong?
>>
>>
>>
>>

Re: KafkaIO.java.configuredKafkaCommit() inconsistency

[John Casey via dev]

.withReadCommitted() doesn't commit messages when read, it instead
specifies that the kafka consumer should only read messages that have
themselves been committed to kafka.

Its use is for exactly once applications.



On Mon, Aug 8, 2022 at 3:16 PM Balázs Németh  wrote:

> I have been reading from Kafka and trying to figure out which offset
> management would be the best for my use-case. During that I noticed
> something odd.
>
>
> https://github.com/apache/beam/blob/c9c161d018d116504fd5f83b48853a7071ec9ce4/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L2359-L2362
>
> private boolean configuredKafkaCommit() {
>   return getConsumerConfig().get("isolation.level") == "read_committed"
>   ||
> Boolean.TRUE.equals(getConsumerConfig().get(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG));
> }
>
> https://github.com/apache/beam/blob/c9c161d018d116504fd5f83b48853a7071ec9ce4/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L2292-L2298
>
> https://github.com/apache/beam/blob/c9c161d018d116504fd5f83b48853a7071ec9ce4/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L2321-L2334
>
> The name of the method, and how it's being used in the code certainly
> suggest that using read_committed isolation level handles and commits
> kafka offsets.Seemed strange, but I'm not a Kafka pro, so let's test it.
> Well it does not.
>
> - using ONLY ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG does commit it
> - using ONLY commitOffsetsInFinalize() does commit it
>
> - using ONLY withReadCommitted() does NOT commit it
>
> Dataflow, 2.40.0 Java SDK, without explicitly enabling SDF-read
>
> So is it a bug, or what am I missing here?
>
> If it is indeed a bug, then is it with the read_committed (so it should
> commit it although found no explicit documentation about that anywhere), or
> having that isolation level shouldn't prefer the commit in the finalize and
> that method is wrong?
>
>
>
>

KafkaIO.java.configuredKafkaCommit() inconsistency

[Balázs Németh]

I have been reading from Kafka and trying to figure out which offset
management would be the best for my use-case. During that I noticed
something odd.

https://github.com/apache/beam/blob/c9c161d018d116504fd5f83b48853a7071ec9ce4/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L2359-L2362

private boolean configuredKafkaCommit() {
  return getConsumerConfig().get("isolation.level") == "read_committed"
  ||
Boolean.TRUE.equals(getConsumerConfig().get(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG));
}
https://github.com/apache/beam/blob/c9c161d018d116504fd5f83b48853a7071ec9ce4/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L2292-L2298
https://github.com/apache/beam/blob/c9c161d018d116504fd5f83b48853a7071ec9ce4/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L2321-L2334

The name of the method, and how it's being used in the code certainly
suggest that using read_committed isolation level handles and commits kafka
offsets.Seemed strange, but I'm not a Kafka pro, so let's test it. Well it
does not.

- using ONLY ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG does commit it
- using ONLY commitOffsetsInFinalize() does commit it

- using ONLY withReadCommitted() does NOT commit it

Dataflow, 2.40.0 Java SDK, without explicitly enabling SDF-read

So is it a bug, or what am I missing here?

If it is indeed a bug, then is it with the read_committed (so it should
commit it although found no explicit documentation about that anywhere), or
having that isolation level shouldn't prefer the commit in the finalize and
that method is wrong?

Re: [Release] 2.41.0 release update

[Kiley Sok via dev]

There's one new issue found while building the RC. Looking into it now
https://github.com/apache/beam/issues/22621

On Fri, Aug 5, 2022 at 10:21 AM Ahmet Altay  wrote:

> Kiley, do we still have the same blockers? Do you need any help?
>
> On Thu, Aug 4, 2022 at 12:18 PM Kiley Sok via dev 
> wrote:
>
>> Last remaining issue was cherry-picked. There may be one last issue with
>> gRPC that's being investigated.
>>
>> https://github.com/apache/beam/issues/22283
>>
>> On Thu, Jul 28, 2022 at 2:20 PM Kiley Sok  wrote:
>>
>>> Quick update for today:
>>>
>>> I'm still working through the validation tests, but we currently have 2
>>> open issues:
>>> https://github.com/apache/beam/issues/22454
>>> https://github.com/apache/beam/issues/22188
>>>
>>>
>>>
>>> On Wed, Jul 27, 2022 at 5:03 PM Kiley Sok  wrote:
>>>
 Hi all,

 I've cut the release branch:
 https://github.com/apache/beam/tree/release-2.41.0

 There's one known issue
 
  that
 needs to be cherry picked. Please let me know if you have a change that
 needs to go in.

 Thanks,
 Kiley

Re: [VOTE] Vendored Dependencies Release

[Pablo Estrada via dev]

+1
Thanks!
-P.

On Mon, Aug 8, 2022 at 9:24 AM Chamikara Jayalath via dev <
dev@beam.apache.org> wrote:

> +1
>
> Thanks,
> Cham
>
> On Fri, Aug 5, 2022 at 1:49 PM Luke Cwik via dev 
> wrote:
>
>> +1
>>
>> I verified the signatures of the artifacts, that the jar doesn't contain
>> classes outside of the org/apache/beam/vendor/grpc/v1p48p1 package and I
>> tested the artifact against our precommits using
>> https://github.com/apache/beam/pull/22595
>>
>> On Fri, Aug 5, 2022 at 1:42 PM Luke Cwik  wrote:
>>
>>> Please review the release of the following artifacts that we vendor:
>>>  * beam-vendor-grpc-1_48_1
>>>
>>> Hi everyone,
>>> Please review and vote on the release candidate #1 for the version 0.1,
>>> as follows:
>>> [ ] +1, Approve the release
>>> [ ] -1, Do not approve the release (please provide specific comments)
>>>
>>>
>>> The complete staging area is available for your review, which includes:
>>> * the official Apache source release to be deployed to dist.apache.org
>>> [1], which is signed with the key with fingerprint
>>> EAD5DE293F4A03DD2E77565589E68A56E371CCA2 [2],
>>> * all artifacts to be deployed to the Maven Central Repository [3],
>>> * commit hash "db8db0b6ed0fe1e4891f207f0f7f811798e54db1" [4],
>>>
>>> The vote will be open for at least 72 hours. It is adopted by majority
>>> approval, with at least 3 PMC affirmative votes.
>>>
>>> Thanks,
>>> Release Manager
>>>
>>> [1] https://dist.apache.org/repos/dist/dev/beam/vendor/
>>> [2] https://dist.apache.org/repos/dist/release/beam/KEYS
>>> [3]
>>> https://repository.apache.org/content/repositories/orgapachebeam-1277/
>>> [4]
>>> https://github.com/apache/beam/commit/db8db0b6ed0fe1e4891f207f0f7f811798e54db1
>>>
>>

Re: [VOTE] Vendored Dependencies Release

[Chamikara Jayalath via dev]

+1

Thanks,
Cham

On Fri, Aug 5, 2022 at 1:49 PM Luke Cwik via dev 
wrote:

> +1
>
> I verified the signatures of the artifacts, that the jar doesn't contain
> classes outside of the org/apache/beam/vendor/grpc/v1p48p1 package and I
> tested the artifact against our precommits using
> https://github.com/apache/beam/pull/22595
>
> On Fri, Aug 5, 2022 at 1:42 PM Luke Cwik  wrote:
>
>> Please review the release of the following artifacts that we vendor:
>>  * beam-vendor-grpc-1_48_1
>>
>> Hi everyone,
>> Please review and vote on the release candidate #1 for the version 0.1,
>> as follows:
>> [ ] +1, Approve the release
>> [ ] -1, Do not approve the release (please provide specific comments)
>>
>>
>> The complete staging area is available for your review, which includes:
>> * the official Apache source release to be deployed to dist.apache.org
>> [1], which is signed with the key with fingerprint
>> EAD5DE293F4A03DD2E77565589E68A56E371CCA2 [2],
>> * all artifacts to be deployed to the Maven Central Repository [3],
>> * commit hash "db8db0b6ed0fe1e4891f207f0f7f811798e54db1" [4],
>>
>> The vote will be open for at least 72 hours. It is adopted by majority
>> approval, with at least 3 PMC affirmative votes.
>>
>> Thanks,
>> Release Manager
>>
>> [1] https://dist.apache.org/repos/dist/dev/beam/vendor/
>> [2] https://dist.apache.org/repos/dist/release/beam/KEYS
>> [3]
>> https://repository.apache.org/content/repositories/orgapachebeam-1277/
>> [4]
>> https://github.com/apache/beam/commit/db8db0b6ed0fe1e4891f207f0f7f811798e54db1
>>
>

Re: Beam Website Feedback

[Luke Cwik via dev]

Thanks.

On Mon, Aug 8, 2022 at 8:12 AM Peter Simon  wrote:

> Awesome web UI
>
> Peter Simon
>
> *Data Scientist*
>
>
>
> e   peter.si...@fanatical.com
>
> w  fanatical.com
>
> Focus Multimedia Limited.
>
> The Studios, Lea Hall Enterprise Park,
>
> Wheelhouse Road, Brereton, Rugeley,
>
> Staffordshire, WS15 1LH, United Kingdom.
>
> 
>
>

Re: Beam gRPC depedency tracing

[Luke Cwik via dev]

I think you missed Kenn's earlier reply:
https://lists.apache.org/thread/v0nr6mv0rqhd76ox1bwt6qwo4q3g7w58

The vendored gRPC is built by transforming the released gRPC jar. Here is
where in the Beam git history you can find the source for the
transformation:
https://github.com/apache/beam/tree/40293eb52ca914acbbbae51e4b24fa280f2b44f0/vendor/grpc-1_26_0

Kenn


On Sun, Aug 7, 2022 at 1:16 PM JDW J  wrote:

> Anyone?
>
> On Wed, 27 Jul, 2022, 13:21 JDW J,  wrote:
>
>> Team,
>>
>> Consider me a newbie to Beam and Java world in general. I am trying to
>> trace Beam vendor dependency to gRPC-upstream.
>>
>>   4.0.0
>>   org.apache.beam
>>   beam-vendor-grpc-1_26_0
>>   0.3
>>   Apache Beam :: Vendored Dependencies :: gRPC :: 1.26.0
>>   http://beam.apache.org
>>
>>
>> How can I tell what is the exact upstream repo for "Apache Beam ::
>> Vendored Dependencies :: gRPC :: 1.26.0" ?
>>
>> -joji
>>
>

Beam Website Feedback

[Peter Simon]

Awesome web UI

Peter Simon

*Data Scientist*



e   peter.si...@fanatical.com

w  fanatical.com

Focus Multimedia Limited.

The Studios, Lea Hall Enterprise Park,

Wheelhouse Road, Brereton, Rugeley,

Staffordshire, WS15 1LH, United Kingdom.

Beam High Priority Issue Report

[beamactions]

This is your daily summary of Beam's current high priority issues that may need 
attention.

See https://beam.apache.org/contribute/issue-priorities for the meaning and 
expectations around issue priorities.

Unassigned P0 Issues:

https://github.com/apache/beam/issues/22606 [Bug]: The pipeline fails when 
trying to write to kafka


Unassigned P1 Issues:

https://github.com/apache/beam/issues/22440 [Bug]: Python Batch Dataflow 
SideInput LoadTests failing
https://github.com/apache/beam/issues/22321 
PortableRunnerTestWithExternalEnv.test_pardo_large_input is regularly failing 
on jenkins
https://github.com/apache/beam/issues/22303 [Task]: Add tests to Kafka SDF and 
fix known and discovered issues
https://github.com/apache/beam/issues/22299 [Bug]: JDBCIO Write freeze at 
getConnection() in WriteFn
https://github.com/apache/beam/issues/22283 [Bug]: Python Lots of fn runner 
test items cost exactly 5 seconds to run
https://github.com/apache/beam/issues/21794 Dataflow runner creates a new timer 
whenever the output timestamp is change
https://github.com/apache/beam/issues/21713 404s in BigQueryIO don't get output 
to Failed Inserts PCollection
https://github.com/apache/beam/issues/21704 beam_PostCommit_Java_DataflowV2 
failures parent bug
https://github.com/apache/beam/issues/21703 pubsublite.ReadWriteIT failing in 
beam_PostCommit_Java_DataflowV1 and V2
https://github.com/apache/beam/issues/21702 SpannerWriteIT failing in beam 
PostCommit Java V1
https://github.com/apache/beam/issues/21701 beam_PostCommit_Java_DataflowV1 
failing with a variety of flakes and errors
https://github.com/apache/beam/issues/21700 
--dataflowServiceOptions=use_runner_v2 is broken
https://github.com/apache/beam/issues/21696 Flink Tests failure :  
java.lang.NoClassDefFoundError: Could not initialize class 
org.apache.beam.runners.core.construction.SerializablePipelineOptions 
https://github.com/apache/beam/issues/21695 DataflowPipelineResult does not 
raise exception for unsuccessful states.
https://github.com/apache/beam/issues/21694 BigQuery Storage API insert with 
writeResult retry and write to error table
https://github.com/apache/beam/issues/21480 flake: 
FlinkRunnerTest.testEnsureStdoutStdErrIsRestored
https://github.com/apache/beam/issues/21472 Dataflow streaming tests failing 
new AfterSynchronizedProcessingTime test
https://github.com/apache/beam/issues/21471 Flakes: Failed to load cache entry
https://github.com/apache/beam/issues/21470 Test flake: test_split_half_sdf
https://github.com/apache/beam/issues/21469 beam_PostCommit_XVR_Flink flaky: 
Connection refused
https://github.com/apache/beam/issues/21468 
beam_PostCommit_Python_Examples_Dataflow failing
https://github.com/apache/beam/issues/21467 GBK and CoGBK streaming Java load 
tests failing
https://github.com/apache/beam/issues/21465 Kafka commit offset drop data on 
failure for runners that have non-checkpointing shuffle
https://github.com/apache/beam/issues/21463 NPE in Flink Portable 
ValidatesRunner streaming suite
https://github.com/apache/beam/issues/21462 Flake in 
org.apache.beam.sdk.io.mqtt.MqttIOTest.testReadObject: Address already in use
https://github.com/apache/beam/issues/21271 pubsublite.ReadWriteIT flaky in 
beam_PostCommit_Java_DataflowV2  
https://github.com/apache/beam/issues/21270 
org.apache.beam.sdk.transforms.CombineTest$WindowingTests.testWindowedCombineGloballyAsSingletonView
 flaky on Dataflow Runner V2
https://github.com/apache/beam/issues/21268 Race between member variable being 
accessed due to leaking uninitialized state via OutboundObserverFactory
https://github.com/apache/beam/issues/21267 WriteToBigQuery submits a duplicate 
BQ load job if a 503 error code is returned from googleapi
https://github.com/apache/beam/issues/21266 
org.apache.beam.sdk.transforms.ParDoLifecycleTest.testTeardownCalledAfterExceptionInProcessElementStateful
 is flaky in Java ValidatesRunner Flink suite.
https://github.com/apache/beam/issues/21265 
apache_beam.runners.portability.fn_api_runner.translations_test.TranslationsTest.test_run_packable_combine_globally
 'apache_beam.coders.coder_impl._AbstractIterable' object is not reversible
https://github.com/apache/beam/issues/21263 (Broken Pipe induced) Bricked 
Dataflow Pipeline 
https://github.com/apache/beam/issues/21262 Python AfterAny, AfterAll do not 
follow spec
https://github.com/apache/beam/issues/21261 
org.apache.beam.runners.dataflow.worker.fn.logging.BeamFnLoggingServiceTest.testMultipleClientsFailingIsHandledGracefullyByServer
 is flaky
https://github.com/apache/beam/issues/21260 Python DirectRunner does not emit 
data at GC time
https://github.com/apache/beam/issues/21257 Either Create or DirectRunner fails 
to produce all elements to the following transform
https://github.com/apache/beam/issues/21123 Multiple jobs running on Flink 
session cluster reuse the persistent Python environment.
https://github.com/apache/beam/issues/21121