Re: [DISCUSS] propdeps removal and what to do going forward
To clarify: "provided" should have been in the test runtime configuration, but not in the shipped runtime configuration (otherwise dep resolution for users would pull in provided deps, which should not happen) On Thu, Dec 30, 2021 at 10:05 AM Luke Cwik wrote: > During the migration to Gradle 7[1] the propdeps plugin was removed[2] > since there wasn't a newer version that was compatible with Gradle 7 and a > replacement couldn't be found. All existing usages of "provided" were moved > to "compileOnly" and "compileOnly" is being mapped to the "provided" maven > scope in the generated pom files. This has lead to two issues: > 1) provided was also part of the runtime configuration, so we are getting > a few class not found exceptions when running tests [3] > 2) the generated pom.xml will have a bunch of compile time only > annotations added as a provided dependency in the generated pom files[4] > > #1 can be fixed by adding the dependency to both the "compileOnly" and > "runtimeOnly" configurations or by adding dependency to the > "implementation" configuration > #2 will make the pom files messier which can lead to confusion for users > but shouldn't impact existing uses. > > There was a suggestion[4] to completely remove the usage of provided from > the generated pom.xml and have all our previously "provided" dependencies > declared as "implementation" allowing us to solve both #1 and #2 above. > > The largest usage of "provided" in the past was to packages related to the > hadoop ecosystem and afterwards it was for packages such as > junit/hamcrest/aircompressor in sdks/java/core which aren't required to use > the module but can provide additional features if the dependency exists. > > What should we migrate if anything to the "implementation" configuration > or should we try to recreate what we were doing with the "provided" > configuration in the past? > > 1: https://issues.apache.org/jira/browse/BEAM-13430 > 2: https://github.com/apache/beam/pull/16308 > 3: https://issues.apache.org/jira/browse/BEAM-13569 > 4: > https://github.com/apache/beam/blob/fe456b79419d1a67ebf13d7d4b6695fa1aa6204d/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L964 > 5: https://issues.apache.org/jira/browse/BEAM-13504 > >
Flaky test issue report (40)
This is your daily summary of Beam's current flaky tests (https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20statusCategory%20!%3D%20Done%20AND%20labels%20%3D%20flake) These are P1 issues because they have a major negative impact on the community and make it hard to determine the quality of the software. https://issues.apache.org/jira/browse/BEAM-13611: CrossLanguageJdbcIOTest.test_xlang_jdbc_write failing in Python PostCommits (created 2022-01-07) https://issues.apache.org/jira/browse/BEAM-13575: Flink testParDoRequiresStableInput flaky (created 2021-12-28) https://issues.apache.org/jira/browse/BEAM-13525: Java VR (Dataflow, V2, Streaming) failing: ParDoTest$TimestampTests/OnWindowExpirationTests (created 2021-12-22) https://issues.apache.org/jira/browse/BEAM-13519: Java precommit flaky (timing out) (created 2021-12-22) https://issues.apache.org/jira/browse/BEAM-13453: Flake in org.apache.beam.sdk.io.mqtt.MqttIOTest.testReadObject: Address already in use (created 2021-12-13) https://issues.apache.org/jira/browse/BEAM-13367: [beam_PostCommit_Python36] [ apache_beam.io.gcp.experimental.spannerio_read_it_test] Failure summary (created 2021-12-01) https://issues.apache.org/jira/browse/BEAM-13312: org.apache.beam.sdk.transforms.ParDoLifecycleTest.testTeardownCalledAfterExceptionInStartBundle is flaky in Java Spark ValidatesRunner suite (created 2021-11-23) https://issues.apache.org/jira/browse/BEAM-13311: org.apache.beam.sdk.transforms.ParDoLifecycleTest.testTeardownCalledAfterExceptionInProcessElementStateful is flaky in Java ValidatesRunner Flink suite. (created 2021-11-23) https://issues.apache.org/jira/browse/BEAM-13234: Flake in StreamingWordCountIT.test_streaming_wordcount_it (created 2021-11-12) https://issues.apache.org/jira/browse/BEAM-13083: XVR Flink and XVR Spark failing (created 2021-10-20) https://issues.apache.org/jira/browse/BEAM-13025: pubsublite.ReadWriteIT flaky in beam_PostCommit_Java_DataflowV2 (created 2021-10-08) https://issues.apache.org/jira/browse/BEAM-12928: beam_PostCommit_Python36 - CrossLanguageSpannerIOTest - flakey failing (created 2021-09-21) https://issues.apache.org/jira/browse/BEAM-12859: org.apache.beam.runners.dataflow.worker.fn.logging.BeamFnLoggingServiceTest.testMultipleClientsFailingIsHandledGracefullyByServer is flaky (created 2021-09-08) https://issues.apache.org/jira/browse/BEAM-12858: org.apache.beam.sdk.io.gcp.datastore.RampupThrottlingFnTest.testRampupThrottler is flaky (created 2021-09-08) https://issues.apache.org/jira/browse/BEAM-12809: testTwoTimersSettingEachOtherWithCreateAsInputBounded flaky (created 2021-08-26) https://issues.apache.org/jira/browse/BEAM-12794: PortableRunnerTestWithExternalEnv.test_pardo_timers flaky (created 2021-08-24) https://issues.apache.org/jira/browse/BEAM-12793: beam_PostRelease_NightlySnapshot failed (created 2021-08-24) https://issues.apache.org/jira/browse/BEAM-12766: Already Exists: Dataset apache-beam-testing:python_bq_file_loads_NNN (created 2021-08-16) https://issues.apache.org/jira/browse/BEAM-12515: Python PreCommit flaking in PipelineOptionsTest.test_display_data (created 2021-06-18) https://issues.apache.org/jira/browse/BEAM-12322: Python precommit flaky: Failed to read inputs in the data plane (created 2021-05-10) https://issues.apache.org/jira/browse/BEAM-12320: PubsubTableProviderIT.testSQLSelectsArrayAttributes[0] failing in SQL PostCommit (created 2021-05-10) https://issues.apache.org/jira/browse/BEAM-12291: org.apache.beam.runners.flink.ReadSourcePortableTest.testExecution[streaming: false] is flaky (created 2021-05-05) https://issues.apache.org/jira/browse/BEAM-12200: SamzaStoreStateInternalsTest is flaky (created 2021-04-20) https://issues.apache.org/jira/browse/BEAM-12163: Python GHA PreCommits flake with grpc.FutureTimeoutError on SDK harness startup (created 2021-04-13) https://issues.apache.org/jira/browse/BEAM-12061: beam_PostCommit_SQL failing on KafkaTableProviderIT.testFakeNested (created 2021-03-27) https://issues.apache.org/jira/browse/BEAM-11837: Java build flakes: "Memory constraints are impeding performance" (created 2021-02-18) https://issues.apache.org/jira/browse/BEAM-11661: hdfsIntegrationTest flake: network not found (py38 postcommit) (created 2021-01-19) https://issues.apache.org/jira/browse/BEAM-11641: Bigquery Read tests are flaky on Flink runner in Python PostCommit suites (created 2021-01-15) https://issues.apache.org/jira/browse/BEAM-11541: testTeardownCalledAfterExceptionInProcessElement flakes on direct runner. (created 2020-12-30) https://issues.apache.org/jira/browse/BEAM-10955: Flink Java Runner test flake: Could not find Flink job (FlinkJobNotFoundException) (created 2020-09-23) https://issues.apache.org/jira/browse/BEAM-10866: PortableRunnerTestWithSubprocesses.test_register_finalizations f
P1 issues report (66)
This is your daily summary of Beam's current P1 issues, not including flaky tests (https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20statusCategory%20!%3D%20Done%20AND%20priority%20%3D%20P1%20AND%20(labels%20is%20EMPTY%20OR%20labels%20!%3D%20flake). See https://beam.apache.org/contribute/jira-priorities/#p1-critical for the meaning and expectations around P1 issues. https://issues.apache.org/jira/browse/BEAM-13616: Update protobuf-java to 3.19.2 and other vendored dependencies that use protobuf (created 2022-01-08) https://issues.apache.org/jira/browse/BEAM-13615: Bumping up FnApi environment version to 9 in Java, Python SDK (created 2022-01-07) https://issues.apache.org/jira/browse/BEAM-13611: CrossLanguageJdbcIOTest.test_xlang_jdbc_write failing in Python PostCommits (created 2022-01-07) https://issues.apache.org/jira/browse/BEAM-13606: bigtable io doesn't handle non-ok row mutations (created 2022-01-07) https://issues.apache.org/jira/browse/BEAM-13604: NPE while getting null from BigDecimal column (created 2022-01-06) https://issues.apache.org/jira/browse/BEAM-13603: Shared object does not read from cache when using tag (created 2022-01-05) https://issues.apache.org/jira/browse/BEAM-13598: Install Java 17 on Jenkins VM (created 2022-01-04) https://issues.apache.org/jira/browse/BEAM-13582: Beam website precommit mentions broken links, but passes. (created 2021-12-30) https://issues.apache.org/jira/browse/BEAM-13579: Cannot run python_xlang_kafka_taxi_dataflow validation script on 2.35.0 (created 2021-12-29) https://issues.apache.org/jira/browse/BEAM-13522: Spark tests failing PerKeyOrderingTest (created 2021-12-22) https://issues.apache.org/jira/browse/BEAM-13503: BulkIO public constructor: Missing required property: throwWriteErrors (created 2021-12-21) https://issues.apache.org/jira/browse/BEAM-13480: apache_beam.io.gcp.pubsub_integration_test.PubSubIntegrationTest.test_streaming_data_only is flaky (created 2021-12-16) https://issues.apache.org/jira/browse/BEAM-13430: Upgrade Gradle version to 7.3 (created 2021-12-09) https://issues.apache.org/jira/browse/BEAM-13393: GroupIntoBatchesTest is failing (created 2021-12-07) https://issues.apache.org/jira/browse/BEAM-13367: [beam_PostCommit_Python36] [ apache_beam.io.gcp.experimental.spannerio_read_it_test] Failure summary (created 2021-12-01) https://issues.apache.org/jira/browse/BEAM-13314: Revise recommendations to manage Python pipeline dependencies. (created 2021-11-24) https://issues.apache.org/jira/browse/BEAM-13269: Data exceeds database column capacity error while inserting in fixed length string column (created 2021-11-17) https://issues.apache.org/jira/browse/BEAM-13237: org.apache.beam.sdk.transforms.CombineTest$WindowingTests.testWindowedCombineGloballyAsSingletonView flaky on Dataflow Runner V2 (created 2021-11-12) https://issues.apache.org/jira/browse/BEAM-13218: apache_beam.io.gcp.pubsub_integration_test.PubSubIntegrationTest.test_streaming_with_attributes is failing (created 2021-11-10) https://issues.apache.org/jira/browse/BEAM-13203: Potential data loss when using SnsIO.writeAsync (created 2021-11-08) https://issues.apache.org/jira/browse/BEAM-13187: BeamJarExpansionService fails to stage remote artifacts (created 2021-11-03) https://issues.apache.org/jira/browse/BEAM-13164: Race between member variable being accessed due to leaking uninitialized state via OutboundObserverFactory (created 2021-11-01) https://issues.apache.org/jira/browse/BEAM-13087: apache_beam.runners.portability.fn_api_runner.translations_test.TranslationsTest.test_run_packable_combine_globally 'apache_beam.coders.coder_impl._AbstractIterable' object is not reversible (created 2021-10-20) https://issues.apache.org/jira/browse/BEAM-13078: Python DirectRunner does not emit data at GC time (created 2021-10-18) https://issues.apache.org/jira/browse/BEAM-13076: Python AfterAny, AfterAll do not follow spec (created 2021-10-18) https://issues.apache.org/jira/browse/BEAM-13059: Migrate GKE workloads to Containerd (created 2021-10-15) https://issues.apache.org/jira/browse/BEAM-13058: Upgrade Kubernetes APIs (created 2021-10-15) https://issues.apache.org/jira/browse/BEAM-13010: Delete orphaned files (created 2021-10-06) https://issues.apache.org/jira/browse/BEAM-12995: Consumer group with random prefix (created 2021-10-04) https://issues.apache.org/jira/browse/BEAM-12959: Dataflow error in CombinePerKey operation (created 2021-09-26) https://issues.apache.org/jira/browse/BEAM-12867: Either Create or DirectRunner fails to produce all elements to the following transform (created 2021-09-09) https://issues.apache.org/jira/browse/BEAM-12843: (Broken Pipe induced) Bricked Dataflow Pipeline (created 2021-09-06) https://issues.apache.org/jira/browse/BEAM-12807: Java creates an incorrect
P0 (outage) report
This is your daily summary of Beam's current outages. See https://beam.apache.org/contribute/jira-priorities/#p0-outage for the meaning and expectations around P0 issues. BEAM-13583: [Playground] Update Github Actions To Push Actual Tag To Registry (https://issues.apache.org/jira/browse/BEAM-13583) BEAM-13541: Use runtime information to improve CoGroupByKey caching (https://issues.apache.org/jira/browse/BEAM-13541)
