Re: [ANNOUNCE] New Committer: Byron Ellis
Congratulations! On Mon, Oct 16, 2023 at 1:33 PM Chamikara Jayalath via dev < dev@beam.apache.org> wrote: > Congrats Byron! > > On Mon, Oct 16, 2023 at 9:32 AM Kenneth Knowles wrote: > >> Hi all, >> >> Please join me and the rest of the Beam PMC in welcoming a new >> committer: Byron Ellis (b...@apache.org). >> >> Byron has been with Beam for over a year now. You may all know him as the >> guy who just decided to write a Swift SDK :-). In addition to that big >> contribution Byron has also fixed plenty of bugs, prototyped DBT-tyle >> pipeline authoring, and participated in our collective decision-making >> process. >> >> Considering his contributions to the project over this timeframe, the >> Beam PMC trusts Byron with the responsibilities of a Beam committer. [1] >> >> Thank you Byron! And we are looking to see more of your contributions! >> >> Kenn, on behalf of the Apache Beam PMC >> >> [1] >> >> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer >> > -- Regards, Tomo
Re: [DISCUSSION][JAVA] Current state of Java 17 support
> Do we still need to support Java 8 SDK?
Yes, for Google Cloud customers who still use Java 8, I want Apache Beam to
support Java 8. Do you observe any special burden maintaining Java 8?
Regards,
Tomo
On Tue, Nov 29, 2022 at 21:48 Austin Bennett wrote:
> -1 for ongoing Java8 support [ or, said another way, +1 for dropping
> support of Java8 ]
>
> +1 for having tests that run for ANY JDK that we say we support. Is there
> any reason the resources to support are too costly [ or outweigh the
> benefits of additional confidence in ensuring we support what we say we do
> ]? I am not certain on whether this would only be critical for releases,
> or should be done as part of regular CI.
>
> On Tue, Nov 29, 2022 at 8:51 AM Alexey Romanenko
> wrote:
>
>> Hello,
>>
>> I’m sorry if it’s already discussed somewhere but I find myself a little
>> bit lost in the subject.
>> So, I’d like to clarify this - what is a current official state of Java
>> 17 support at Beam?
>>
>> I recall that a great job was done to make Beam compatible with Java 17
>> [1] and Beam already provides “beam_java17_sdk” Docker image [2] but, iiuc,
>> Java 8 is still the default JVM to run all Java tests on Jenkins ("Java
>> PreCommit" in the first order) and there are only limited number of tests
>> that are running with JDK 11 and 17 on Jenkins by dedicated jobs.
>>
>> So, my question would sound like if Beam officially supports Java 17 (and
>> 11), do we need to run all Beam Java SDK related tests (VR and IT test
>> including) against all supported Java SDKs?
>>
>> Do we still need to support Java 8 SDK?
>>
>> In the same time, as we are heading to move everything from Jenkins to
>> GitHub actions, what would be the default JDK there or we will run all
>> Java-related actions against all supported JDKs?
>>
>> —
>> Alexey
>>
>> [1] https://issues.apache.org/jira/browse/BEAM-12240
>> [2] https://hub.docker.com/r/apache/beam_java17_sdk
>>
>>
>>
>> --
Regards,
Tomo
Re: [DISCUSS] Jenkins -> GitHub Actions ?
Kenn, thank you for the summary. SGTM. Looking forward to GitHub Actions. On Mon, Nov 7, 2022 at 12:58 PM Kenneth Knowles wrote: > OK, it seems like there is general consensus. Not too much action on the > document. I will summarize the gaps that don't have an answer in the doc, > and my new opinion of how important they are: > > - [required] Run specific non-default workflow on PR > - [required] View history of a workflow > - [required] Publish nightly snapshots > - [required] Run workflow on dedicated worker pool for performance testing > - [important but not required] Summarize flakiness statistics of one or > all workflows > - [important but not required] History of all/many workflows in a single > view > - [nice to have] History of specific test case (not just the workflow > level) > > Do any of these seem like I got the importance wrong? > > Kenn > > On Mon, Nov 7, 2022 at 9:09 AM Austin Bennett wrote: > >> +1 >> >> Also would help address a good amount of what concerns me that was >> [sorta] raised by >> https://lists.apache.org/thread/7jr99nc5xsb3ft1d75kb0ml32bzw89rv >> >> >> Once we think this is something we want to do, but might be >> blocked/concerned because of lack of definitively comparable features, I'd >> be happy to take a look at what exists in the wider ecosystem or could be >> built. >> >> Cheers - >> >> >> >> On Fri, Oct 21, 2022 at 11:10 AM Ismaël Mejía wrote: >> >>> +1 Github Actions are more intuitive and easy to modify and test for >>> everyone. >>> Also Beam wins because that makes one less system to maintain. >>> >>> Regards, >>> Ismaël >>> >>> On Wed, Oct 19, 2022 at 5:50 PM Danny McCormick via dev >>> wrote: >>> > >>> > Thanks for kicking this conversation off. I'm +1 on migrating, but >>> only once we've found a specific replacement for easy observability (which >>> workflows have been failing lately, and how often) and trigger phrases (for >>> retries and workflows that aren't automatically kicked off but should be >>> run for extra validation, e.g. postcommits). Until we have viable >>> replacements, I don't think we should make the move. Publishing nightly >>> snapshots is eventually also a must to fully migrate, but probably doesn't >>> need to block us from making progress here. >>> > >>> > With those caveats, the reason that I'm +1 on moving is that our >>> Jenkins reliability has been rough. Since I joined the project in January, >>> I can think of 3 different incidents that significantly harmed our ability >>> to do work. >>> > >>> > 1. Jenkins triggers cause multi-day outage - this led to a multi-day >>> code freeze, and we lost our trigger functionality for days afterwards. >>> Investigating/restoring our state ate up a pretty full week for me. >>> > 2. Jenkins plugin cause multi-day outage - this led to multiple days >>> of Jenkins downtime before eventually being resolved by Infra. >>> > 3. Cert issues cause many workers to go down - I don't have a thread >>> for this because I handled most of the investigation the day of, but many >>> of our workers went down for around a day and nobody noticed until queue >>> time reached 6+ hours for each workflow. >>> > >>> > There may be others that I'm overlooking. >>> > >>> > GitHub Actions isn't a magic bullet to fix these problems, but it >>> minimizes the amount of infra that we're maintaining ourselves, increases >>> the isolation between workflows (catastrophic failure is less likely), has >>> uptime guarantees, and is more likely to receive investment going forward >>> (we're likely to get increasing benefits over time for free). We've also >>> done a lot of exploration in this area already, so we're not starting from >>> scratch. >>> > >>> > Thanks, >>> > Danny >>> > >>> > On Wed, Oct 19, 2022 at 11:32 AM Kenneth Knowles >>> wrote: >>> >> >>> >> Hi all, >>> >> >>> >> As you probably noticed, there's a lot of work going on around adding >>> more GitHub Actions workflows. >>> >> >>> >> Can we fully migrate to GitHub Actions? Similar to our GitHub Issues >>> migration (but less user-facing) it would bring us on to "default" >>> infrastructure that more people understand and is maintained by GitHub. >>> >> >>> >> So far we have hit some serious roadblocks. It isn't just a simple >>> migration. We have to weigh doing the work to get there. >>> >> >>> >> I started a document with a table of the things we get from Jenkins >>> that we need to be sure to have for GitHub Actions before we could think >>> about migrating: >>> >> >>> >> https://s.apache.org/beam-jenkins-to-gha >>> >> >>> >> Can you please help me by adding things that we get from Jenkins, and >>> if you know how to get them from GitHub Actions add that too. >>> >> >>> >> Thanks! >>> >> >>> >> Kenn >>> >> -- Regards, Tomo
Re: [ANNOUNCE] New committer: Yi Hu
Congratulations! On Wed, Nov 9, 2022 at 3:00 PM John Casey via dev wrote: > Congrats! this is well deserved YI > > On Wed, Nov 9, 2022 at 2:58 PM Austin Bennett > wrote: > >> Congrats, and Thanks, Yi! >> >> On Wed, Nov 9, 2022 at 11:24 AM Valentyn Tymofieiev via dev < >> dev@beam.apache.org> wrote: >> >>> I am with the Beam PMC on this, congratulations and very well deserved, >>> Yi! >>> >>> On Wed, Nov 9, 2022 at 11:08 AM Byron Ellis via dev >>> wrote: >>> Congratulations! On Wed, Nov 9, 2022 at 11:00 AM Pablo Estrada via dev < dev@beam.apache.org> wrote: > +1 thanks Yi : D > > On Wed, Nov 9, 2022 at 10:47 AM Danny McCormick via dev < > dev@beam.apache.org> wrote: > >> Congrats Yi! I've really appreciated the ways you've consistently >> taken responsibility for improving our team's infra and working through >> sharp edges in the codebase that others have ignored. This is definitely >> well deserved! >> >> Thanks, >> Danny >> >> On Wed, Nov 9, 2022 at 1:37 PM Anand Inguva via dev < >> dev@beam.apache.org> wrote: >> >>> Congratulations Yi! >>> >>> On Wed, Nov 9, 2022 at 1:35 PM Ritesh Ghorse via dev < >>> dev@beam.apache.org> wrote: >>> Congratulations Yi! On Wed, Nov 9, 2022 at 1:34 PM Ahmed Abualsaud via dev < dev@beam.apache.org> wrote: > Congrats Yi! > > On Wed, Nov 9, 2022 at 1:33 PM Sachin Agarwal via dev < > dev@beam.apache.org> wrote: > >> Congratulations Yi! >> >> On Wed, Nov 9, 2022 at 10:32 AM Kenneth Knowles >> wrote: >> >>> Hi all, >>> >>> Please join me and the rest of the Beam PMC in welcoming a new >>> committer: Yi Hu (y...@apache.org) >>> >>> Yi started contributing to Beam in early 2022. Yi's >>> contributions are very diverse! I/Os, performance tests, Jenkins, >>> support >>> for Schema logical types. Not only code but a very large amount of >>> code >>> review. Yi is also noted for picking up smaller issues that >>> normally would >>> be left on the backburner and filing issues that he finds rather >>> than >>> ignoring them. >>> >>> Considering their contributions to the project over this >>> timeframe, the Beam PMC trusts Yi with the responsibilities of a >>> Beam >>> committer. [1] >>> >>> Thank you Yi! And we are looking to see more of your >>> contributions! >>> >>> Kenn, on behalf of the Apache Beam PMC >>> >>> [1] >>> >>> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer >>> >> -- Regards, Tomo
Re: [Dataflow][Guidance] Replacing beam-sdks-java-io-google-cloud-platform with local jar
I don't come up with a solution (I'm not familiar with the method you're using). However I often use "getProtectionDomain()" https://stackoverflow.com/a/56000383/975074 to find the JAR file from a class. This ensures the class you modified is actually used. On Thu, Jul 21, 2022 at 3:35 PM Evan Galpin wrote: > Spoke too soon... still can't seem to get the new behaviour to appear in > dataflow, possibly something is being overridden? > > On Thu, Jul 21, 2022 at 3:15 PM Evan Galpin wrote: > >> Making a shadowJar from "beam-sdks-java-io-google-cloud-platform" looks >> to be working. Added ` id 'com.github.johnrengelman.shadow'` to >> `build.gradle` for "beam-sdks-java-io-google-cloud-platform" in the beam >> source and used the resulting jar as a dependency replacement when >> deploying the job to dataflow. Looks ok. >> >> On Thu, Jul 21, 2022 at 3:02 PM Evan Galpin wrote: >> >>> I believe I have the dependencySubstitution working, but it seems as >>> though the substitution is removing transitive deps of >>> "beam-sdks-java-io-google-cloud-platform", hmm... >>> >>> On Thu, Jul 21, 2022 at 1:15 PM Evan Galpin wrote: >>> Hi all, I'm trying to test a change I've made locally, but by validating it on Dataflow. It works locally, but I want to validate on Dataflow. I've tried a few different attempts at module substitution in the build.gradle config file for the pipeline I'm trying to deploy, but I haven't had any success yet. How might I be able to replace the beam-sdks-java-io-google-cloud-platform module usually installed from maven with a local jar generated from running: "./gradlew :sdk:java:io:google-cloud-platform:jar" Thanks, Evan >>> -- Regards, Tomo
