[GitHub] [beam-site] ibzib commented on pull request #620: Update beam-site for release 2.34.0.
ibzib commented on pull request #620: URL: https://github.com/apache/beam-site/pull/620#issuecomment-956562631 R: @udim -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@beam.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Flaky test issue report (30)
This is your daily summary of Beam's current flaky tests (https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20statusCategory%20!%3D%20Done%20AND%20labels%20%3D%20flake) These are P1 issues because they have a major negative impact on the community and make it hard to determine the quality of the software. https://issues.apache.org/jira/browse/BEAM-13083: XVR Flink and XVR Spark failing (created 2021-10-20) https://issues.apache.org/jira/browse/BEAM-13025: beam_PostCommit_Java_DataflowV2 failing pubsublite.ReadWriteIT (created 2021-10-08) https://issues.apache.org/jira/browse/BEAM-12928: beam_PostCommit_Python36 - CrossLanguageSpannerIOTest - flakey failing (created 2021-09-21) https://issues.apache.org/jira/browse/BEAM-12859: org.apache.beam.runners.dataflow.worker.fn.logging.BeamFnLoggingServiceTest.testMultipleClientsFailingIsHandledGracefullyByServer is flaky (created 2021-09-08) https://issues.apache.org/jira/browse/BEAM-12809: testTwoTimersSettingEachOtherWithCreateAsInputBounded flaky (created 2021-08-26) https://issues.apache.org/jira/browse/BEAM-12794: PortableRunnerTestWithExternalEnv.test_pardo_timers flaky (created 2021-08-24) https://issues.apache.org/jira/browse/BEAM-12766: Already Exists: Dataset apache-beam-testing:python_bq_file_loads_NNN (created 2021-08-16) https://issues.apache.org/jira/browse/BEAM-12540: beam_PostRelease_NightlySnapshot - Task :runners:direct-java:runMobileGamingJavaDirect FAILED (created 2021-06-25) https://issues.apache.org/jira/browse/BEAM-12515: Python PreCommit flaking in PipelineOptionsTest.test_display_data (created 2021-06-18) https://issues.apache.org/jira/browse/BEAM-12322: Python precommit flaky: Failed to read inputs in the data plane (created 2021-05-10) https://issues.apache.org/jira/browse/BEAM-12320: PubsubTableProviderIT.testSQLSelectsArrayAttributes[0] failing in SQL PostCommit (created 2021-05-10) https://issues.apache.org/jira/browse/BEAM-12291: org.apache.beam.runners.flink.ReadSourcePortableTest.testExecution[streaming: false] is flaky (created 2021-05-05) https://issues.apache.org/jira/browse/BEAM-12200: SamzaStoreStateInternalsTest is flaky (created 2021-04-20) https://issues.apache.org/jira/browse/BEAM-12163: Python GHA PreCommits flake with grpc.FutureTimeoutError on SDK harness startup (created 2021-04-13) https://issues.apache.org/jira/browse/BEAM-12061: beam_PostCommit_SQL failing on KafkaTableProviderIT.testFakeNested (created 2021-03-27) https://issues.apache.org/jira/browse/BEAM-11837: Java build flakes: "Memory constraints are impeding performance" (created 2021-02-18) https://issues.apache.org/jira/browse/BEAM-11661: hdfsIntegrationTest flake: network not found (py38 postcommit) (created 2021-01-19) https://issues.apache.org/jira/browse/BEAM-11641: Bigquery Read tests are flaky on Flink runner in Python PostCommit suites (created 2021-01-15) https://issues.apache.org/jira/browse/BEAM-11541: testTeardownCalledAfterExceptionInProcessElement flakes on direct runner. (created 2020-12-30) https://issues.apache.org/jira/browse/BEAM-10955: Flink Java Runner test flake: Could not find Flink job (FlinkJobNotFoundException) (created 2020-09-23) https://issues.apache.org/jira/browse/BEAM-10866: PortableRunnerTestWithSubprocesses.test_register_finalizations flaky on macOS (created 2020-09-09) https://issues.apache.org/jira/browse/BEAM-10485: Failure / flake: ElasticsearchIOTest > testWriteWithIndexFn (created 2020-07-14) https://issues.apache.org/jira/browse/BEAM-9649: beam_python_mongoio_load_test started failing due to mismatched results (created 2020-03-31) https://issues.apache.org/jira/browse/BEAM-8101: Flakes in ParDoLifecycleTest.testTeardownCalledAfterExceptionInStartBundleStateful for Direct, Spark, Flink (created 2019-08-27) https://issues.apache.org/jira/browse/BEAM-8035: WatchTest.testMultiplePollsWithManyResults flake: Outputs must be in timestamp order (sickbayed) (created 2019-08-22) https://issues.apache.org/jira/browse/BEAM-7827: MetricsTest$AttemptedMetricTests.testAllAttemptedMetrics is flaky on DirectRunner (created 2019-07-26) https://issues.apache.org/jira/browse/BEAM-7752: Java Validates DirectRunner: testTeardownCalledAfterExceptionInFinishBundleStateful flaky (created 2019-07-16) https://issues.apache.org/jira/browse/BEAM-6804: [beam_PostCommit_Java] [PubsubReadIT.testReadPublicData] Timeout waiting on Sub (created 2019-03-11) https://issues.apache.org/jira/browse/BEAM-5286: [beam_PostCommit_Java_GradleBuild][org.apache.beam.examples.subprocess.ExampleEchoPipelineTest.testExampleEchoPipeline][Flake] .sh script: text file busy. (created 2018-09-01) https://issues.apache.org/jira/browse/BEAM-5172: org.apache.beam.sdk.io.elasticsearch/ElasticsearchIOTest is flaky (created 2018-08-20)
P1 issues report (55)
This is your daily summary of Beam's current P1 issues, not including flaky tests (https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20statusCategory%20!%3D%20Done%20AND%20priority%20%3D%20P1%20AND%20(labels%20is%20EMPTY%20OR%20labels%20!%3D%20flake). See https://beam.apache.org/contribute/jira-priorities/#p1-critical for the meaning and expectations around P1 issues. https://issues.apache.org/jira/browse/BEAM-13137: make ElasticsearchIO$BoundedElasticsearchSource#getEstimatedSizeBytes deterministic (created 2021-10-28) https://issues.apache.org/jira/browse/BEAM-13118: Nightly build wheels jobs failing (created 2021-10-26) https://issues.apache.org/jira/browse/BEAM-13098: Repeated fields not translated correctly in BigQuery Storage API sink (created 2021-10-21) https://issues.apache.org/jira/browse/BEAM-13087: apache_beam.runners.portability.fn_api_runner.translations_test.TranslationsTest.test_run_packable_combine_globally 'apache_beam.coders.coder_impl._AbstractIterable' object is not reversible (created 2021-10-20) https://issues.apache.org/jira/browse/BEAM-13078: Python DirectRunner does not emit data at GC time (created 2021-10-18) https://issues.apache.org/jira/browse/BEAM-13076: Python AfterAny, AfterAll do not follow spec (created 2021-10-18) https://issues.apache.org/jira/browse/BEAM-13074: Metrics are not reported by the Flink runner (created 2021-10-18) https://issues.apache.org/jira/browse/BEAM-13060: Daily Python SDK build is not publicly accessible (created 2021-10-15) https://issues.apache.org/jira/browse/BEAM-13059: Migrate GKE workloads to Containerd (created 2021-10-15) https://issues.apache.org/jira/browse/BEAM-13058: Upgrade Kubernetes APIs (created 2021-10-15) https://issues.apache.org/jira/browse/BEAM-13056: Add method to fetch ProcessContext FieldAccess (created 2021-10-14) https://issues.apache.org/jira/browse/BEAM-13025: beam_PostCommit_Java_DataflowV2 failing pubsublite.ReadWriteIT (created 2021-10-08) https://issues.apache.org/jira/browse/BEAM-13010: Delete orphaned files (created 2021-10-06) https://issues.apache.org/jira/browse/BEAM-12995: Consumer group with random prefix (created 2021-10-04) https://issues.apache.org/jira/browse/BEAM-12959: Dataflow error in CombinePerKey operation (created 2021-09-26) https://issues.apache.org/jira/browse/BEAM-12867: Either Create or DirectRunner fails to produce all elements to the following transform (created 2021-09-09) https://issues.apache.org/jira/browse/BEAM-12843: (Broken Pipe induced) Bricked Dataflow Pipeline (created 2021-09-06) https://issues.apache.org/jira/browse/BEAM-12818: When writing to GCS, spread prefix of temporary files and reuse autoscaling of the temporary directory (created 2021-08-30) https://issues.apache.org/jira/browse/BEAM-12807: Java creates an incorrect pipeline proto when core-construction-java jar is not in the CLASSPATH (created 2021-08-26) https://issues.apache.org/jira/browse/BEAM-12792: Beam worker only installs --extra_package once (created 2021-08-24) https://issues.apache.org/jira/browse/BEAM-12766: Already Exists: Dataset apache-beam-testing:python_bq_file_loads_NNN (created 2021-08-16) https://issues.apache.org/jira/browse/BEAM-12632: ElasticsearchIO: Enabling both User/Pass auth and SSL overwrites User/Pass (created 2021-07-16) https://issues.apache.org/jira/browse/BEAM-12540: beam_PostRelease_NightlySnapshot - Task :runners:direct-java:runMobileGamingJavaDirect FAILED (created 2021-06-25) https://issues.apache.org/jira/browse/BEAM-12525: SDF BoundedSource seems to execute significantly slower than 'normal' BoundedSource (created 2021-06-22) https://issues.apache.org/jira/browse/BEAM-12505: codecov/patch has poor behavior (created 2021-06-17) https://issues.apache.org/jira/browse/BEAM-12500: Dataflow SocketException (SSLException) error while trying to send message from Cloud Pub/Sub to BigQuery (created 2021-06-16) https://issues.apache.org/jira/browse/BEAM-12484: JdbcIO date conversion is sensitive to OS (created 2021-06-14) https://issues.apache.org/jira/browse/BEAM-12467: java.io.InvalidClassException With Flink Kafka (created 2021-06-09) https://issues.apache.org/jira/browse/BEAM-12279: Implement destination-dependent sharding in FileIO.writeDynamic (created 2021-05-04) https://issues.apache.org/jira/browse/BEAM-12256: PubsubIO.readAvroGenericRecord creates SchemaCoder that fails to decode some Avro logical types (created 2021-04-29) https://issues.apache.org/jira/browse/BEAM-11959: Python Beam SDK Harness hangs when installing pip packages (created 2021-03-11) https://issues.apache.org/jira/browse/BEAM-11906: No trigger early repeatedly for session windows (created 2021-03-01) https://issues.apache.org/jira/browse/BEAM-11875: XmlIO.Read does not handle XML encoding per spec (created 2021-02-26)
Re: Best practices for upgrading installed dependencies on Jenkins VMs?
Looks like while .profile was edited to add in a PATH section pointing to /snap/bin (where go is now installed), it doesn't seem like .profile is executed by the jenkins login shells. On Fri, Oct 29, 2021, 6:23 PM Valentyn Tymofieiev wrote: > > > On Wed, Oct 20, 2021 at 11:16 AM Valentyn Tymofieiev > wrote: > >> >> >> On Wed, Oct 20, 2021 at 11:12 AM Pablo Estrada >> wrote: >> >>> Thanks everyone for investigating and documenting this. I'll use it >>> today : ) >>> >> Dan may be also in the middle of doing this, please coordinate. >> >>> >>> ahem - maybe we should rename the image name/image family names >>> to jenkins-worker-boot-image ? Does anyone foresee issues if we do that? >>> Does jenkins depend on these names in some undocumented way? >>> >> +1. it should 'just work', need to update the wiki after the change. >> Jenkins also did a terminology adjustment. >> > I had to reimage Jenkins workers again, took care of the rename and > changed the instructions. > > I am not sure what is the status of Go Postcommit problem, but noticed > that jenkins worker #1 had a different boot disk. I reimaged all workers > building on top of the latest image from the image family. If Go tests > start failing, we may need to get help from Dan again. > > >> >>> On Tue, Oct 19, 2021 at 1:43 PM Daniel Oliveira >>> wrote: >>> I'm ok with deciding to avoid the "lite" update option, feel free to revise the instructions as it seems appropriate. As for the issue, I fixed it with a workaround that should work until we need to add a new image to the agents, and I'm currently investigating the root cause and prepare a fixed image. That said, I think this issue would have still happened even if we didn't perform the "lite" update. I'm still trying to figure out the exact problem, but it looks to be a PATH issue that wasn't effectively caught by the current process. I won't get into details too much in this thread (see the Jira for that), but essentially everything works in my environment when I SSH into the VMs, but because the location of the "go" command changed in the PATH, it seems to have stopped working for every other user, including the Jenkins agents. I actually did notice that would happen when I was working on the image, but the solution seemed to be to reboot the machine, which I assumed happened already since I shut down the VM to image it. On Tue, Oct 19, 2021 at 12:09 PM Robert Burke wrote: > +1 to only having one way to do things. The Lite option seems liable > to cause more problems since it means it's changes can be blown away if a > new image isn't prepared anyway. > I don't think we are changing the images often enough for it. Perhaps > call it the option to test changes if anything? > > On Tue, Oct 19, 2021, 11:55 AM Valentyn Tymofieiev < > valen...@google.com> wrote: > >> All workers were updated to use jenkins-slave-boot-image-20211011, >> which should have had a go command, but it appears slightly >> misconfigured. >> I reopened BEAM-13037 [1] and added some details there. >> >> I also added instructions to wiki [2] on how to perform an image swap >> and it is actually very straightforward. I think a lesson here is that >> making 'lite' upgrades is brittle as misconfigurations could resurface >> down >> the road when the context of the lite upgrade is no longer fresh in our >> memory. >> >> I suggest we revise the instructions to keep only image swap commands >> and remove the 'lite' update option. +Daniel Oliveira >> , WDYT? In the meantime, we should also >> prepare an image that fixes the misconfiguration. Would you be able to >> help >> with that? Thank you. >> >> [1] https://issues.apache.org/jira/browse/BEAM-13037 >> [2] >> https://cwiki.apache.org/confluence/display/BEAM/Jenkins+Tips#JenkinsTips-HowtoinstallandupgradesoftwareonJenkinsworkers >> >> >> On Tue, Oct 19, 2021 at 8:46 AM Robert Burke >> wrote: >> >>> FYI it looks like all the Go tests are now failing because it can't >>> find the Go command at all. >>> Did a Jenkins image without Go (v1.16+) pre-installed get pushed? >>> >>> On Mon, Oct 18, 2021, 1:45 PM Valentyn Tymofieiev < >>> valen...@google.com> wrote: >>> Thanks Daniel, I can recreate the VMs on new disks. We currently have a set of stopped jenkins workers (named: apache-beam-jenkins-##) and running workers (named: apache-ci-beam-jenkins-##) Are there any concerns about deleting the stopped group of workers? On Mon, Oct 18, 2021 at 11:19 AM Ahmet Altay wrote: > Thank you Daniel, Valentyn! > > On Mon, Oct 18, 2021 at 8:02 AM Daniel Oliveira <
Beam Dependency Check Report (2021-11-01)
ERROR: File 'src/build/dependencyUpdates/beam-dependency-check-report.html' does not exist
