Re: Best practices for upgrading installed dependencies on Jenkins VMs?

[Robert Burke]

SGTM. Thank you very much Daniel!

On Mon, Oct 11, 2021, 7:51 PM Ahmet Altay  wrote:

> Thank you Daniel. Could you please update the wiki once you are done with
> the process?
>
> On Mon, Oct 11, 2021 at 6:22 PM Daniel Oliveira 
> wrote:
>
>> Took me a bit to get to this, sorry. I finally figured out an approach
>> for updating Go and did so and will be updating the image momentarily.
>>
>> I think a more important note is that I tried what Valentyn was
>> considering, which is SSHing into workers and updating the dependency. I'll
>> describe the process below, but the summary is that I did it on one worker
>> with Go so far, saw no problems over the weekend, and would like to
>> continue updating the rest of the workers if there are no objections.
>>
>> Here's a step-by-step of what I did. If we decide to stick with this
>> approach, these instructions can be added to Confluence:
>>
>> 1. Go to the page for the Jenkins agent you want to update [1] and click
>> "Mark this node temporarily offline", leaving a reason such as "Updating X
>> dependency."
>> 2. Wait until there are no more tests running in that agent (under "Build
>> Executor Status" on the left of the page).
>> 3. SSH into the agent and perform the update.
>> 4. Mark the node as online again.
>> 5. Repeat for every worker.
>>
>> And these are some additional steps if you want to immediately run a test
>> suite to check that the update worked correctly. For example in my case, I
>> wanted to check against the Go Postcommit, and it was a good thing I did,
>> because it actually failed the first time and I had to go back in to fix a
>> small oversight I made. So doing this after you update your first worker is
>> probably a good idea before updating the rest:
>>
>> 1. Go to the page for the job you want to run (for example: [2]).
>> 2. Click "Configure" on the left menu.
>> 3. Find the checkmark "Restrict where this project can be run" and change
>> the restriction from "beam" to the specific name of the agent (ex.
>> "apache-beam-jenkins-1").
>> 4. Save and apply that change.
>> 5. Back on the page for the job, click "Build with Parameters" on the
>> left menu.
>> 6. Run the build on "master".
>> 7. Once you're done checking the results, change the restriction for the
>> job back to "beam". (This also gets reset once every 24 hours in case you
>> forget.)
>>
>> I did that on one agent (apache-beam-jenkins-2) on Friday evening when it
>> wasn't too busy, and got Go updated and working. I checked that agent's
>> execution history again today just in case, and it was healthy over
>> the weekend, with no Go-related problems as far as I could see. If there's
>> no objections I'd like to go ahead and continue updating the rest of the
>> workers (I'll do this late at night or over the weekend to avoid disrupting
>> dev work).
>>
>> [1] https://ci-beam.apache.org/computer/apache-beam-jenkins-1/
>> [2] https://ci-beam.apache.org/job/beam_PostCommit_Go/
>>
>> On Mon, Oct 4, 2021 at 6:14 PM Valentyn Tymofieiev 
>> wrote:
>>
>>> I updated the image in [1], but did not change the workers yet to pick
>>> up the new image yet. We can do this once we add Go changes on top of it.
>>>
>>> I am also considering to SSH into every worker and run a one-line
>>> command that adds the dependency that was missing. It seems to be low risk,
>>> and  there is a fall-back plan to re-start the worker using the saved image
>>> - both new and old images are saved and available in Cloud Console.
>>>
>>> Ideally, we should find a way to do a rolling upgrade that a PMC or
>>> committer could trigger without logging into every machine.
>>>
>>> [1]
>>> https://issues.apache.org/jira/browse/BEAM-8152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17424228#comment-17424228
>>>
>>>
>>> On Wed, Sep 22, 2021 at 3:28 PM Daniel Oliveira 
>>> wrote:
>>>
 @Brian Hulette  That button seems like exactly
 what we'd need. Doing it manually would be a pain, but it's probably still
 preferable to causing a bunch of aborted tests.

 @Valentyn Tymofieiev  Collaborating to do both
 updates at once is a great idea! I'll message you directly about it.

 On Wed, Sep 22, 2021 at 2:44 PM Valentyn Tymofieiev <
 valen...@google.com> wrote:

> I am also interested in this updating version of Python on VMs, I need
> to install Python 3.9. Thanks for looking into this.  We can coordinate
> together to make one update instead of two.
>
> On Wed, Sep 22, 2021 at 2:40 PM Brian Hulette 
> wrote:
>
>> I'm not sure about best practices here. Out of curiosity I just poked
>> around in the Jenkins UI (e.g. [1]) and it looks like you can manually
>> "Mark node temporarily offline" when logged in (if you're a committer).
>> According to [2] this will prevent it from picking up new jobs after it's
>> finished the currently executing ones. Doing that manually for every 
>> worker
>> could be a pain 

Flaky test issue report (33)

[Beam Jira Bot]

This is your daily summary of Beam's current flaky tests 
(https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20statusCategory%20!%3D%20Done%20AND%20labels%20%3D%20flake)

These are P1 issues because they have a major negative impact on the community 
and make it hard to determine the quality of the software.

https://issues.apache.org/jira/browse/BEAM-13025: 
beam_PostCommit_Java_DataflowV2 failing pubsublite.ReadWriteIT (created 
2021-10-08)
https://issues.apache.org/jira/browse/BEAM-12928: beam_PostCommit_Python36 
- CrossLanguageSpannerIOTest - flakey failing (created 2021-09-21)
https://issues.apache.org/jira/browse/BEAM-12861: 
apache_beam.ml.gcp.recommendations_ai_test_it.RecommendationAIIT.test_create_catalog_item
  is flaky (created 2021-09-09)
https://issues.apache.org/jira/browse/BEAM-12859: 
org.apache.beam.runners.dataflow.worker.fn.logging.BeamFnLoggingServiceTest.testMultipleClientsFailingIsHandledGracefullyByServer
 is flaky (created 2021-09-08)
https://issues.apache.org/jira/browse/BEAM-12809: 
testTwoTimersSettingEachOtherWithCreateAsInputBounded flaky (created 2021-08-26)
https://issues.apache.org/jira/browse/BEAM-12794: 
PortableRunnerTestWithExternalEnv.test_pardo_timers flaky (created 2021-08-24)
https://issues.apache.org/jira/browse/BEAM-12766: Already Exists: Dataset 
apache-beam-testing:python_bq_file_loads_NNN (created 2021-08-16)
https://issues.apache.org/jira/browse/BEAM-12694: DICOMIoIntegrationTest 
flaky due to store ID (Python PreCommit) (created 2021-07-30)
https://issues.apache.org/jira/browse/BEAM-12540: 
beam_PostRelease_NightlySnapshot - Task 
:runners:direct-java:runMobileGamingJavaDirect FAILED (created 2021-06-25)
https://issues.apache.org/jira/browse/BEAM-12515: Python PreCommit flaking 
in PipelineOptionsTest.test_display_data (created 2021-06-18)
https://issues.apache.org/jira/browse/BEAM-12322: Python precommit flaky: 
Failed to read inputs in the data plane (created 2021-05-10)
https://issues.apache.org/jira/browse/BEAM-12320: 
PubsubTableProviderIT.testSQLSelectsArrayAttributes[0] failing in SQL 
PostCommit (created 2021-05-10)
https://issues.apache.org/jira/browse/BEAM-12291: 
org.apache.beam.runners.flink.ReadSourcePortableTest.testExecution[streaming: 
false] is flaky (created 2021-05-05)
https://issues.apache.org/jira/browse/BEAM-12200: 
SamzaStoreStateInternalsTest is flaky (created 2021-04-20)
https://issues.apache.org/jira/browse/BEAM-12163: Python GHA PreCommits 
flake with grpc.FutureTimeoutError on SDK harness startup (created 2021-04-13)
https://issues.apache.org/jira/browse/BEAM-12061: beam_PostCommit_SQL 
failing on KafkaTableProviderIT.testFakeNested (created 2021-03-27)
https://issues.apache.org/jira/browse/BEAM-11837: Java build flakes: 
"Memory constraints are impeding performance" (created 2021-02-18)
https://issues.apache.org/jira/browse/BEAM-11661: hdfsIntegrationTest 
flake: network not found (py38 postcommit) (created 2021-01-19)
https://issues.apache.org/jira/browse/BEAM-11645: beam_PostCommit_XVR_Flink 
failing (created 2021-01-15)
https://issues.apache.org/jira/browse/BEAM-11641: Bigquery Read tests are 
flaky on Flink runner in Python PostCommit suites (created 2021-01-15)
https://issues.apache.org/jira/browse/BEAM-11541: 
testTeardownCalledAfterExceptionInProcessElement flakes on direct runner. 
(created 2020-12-30)
https://issues.apache.org/jira/browse/BEAM-10955: Flink Java Runner test 
flake: Could not find Flink job (FlinkJobNotFoundException) (created 2020-09-23)
https://issues.apache.org/jira/browse/BEAM-10866: 
PortableRunnerTestWithSubprocesses.test_register_finalizations flaky on macOS 
(created 2020-09-09)
https://issues.apache.org/jira/browse/BEAM-10485: Failure / flake: 
ElasticsearchIOTest > testWriteWithIndexFn (created 2020-07-14)
https://issues.apache.org/jira/browse/BEAM-9649: 
beam_python_mongoio_load_test started failing due to mismatched results 
(created 2020-03-31)
https://issues.apache.org/jira/browse/BEAM-8453: Failure in 
org.apache.beam.sdk.io.jms.JmsIOTest.testCheckpointMarkSafety (created 
2019-10-21)
https://issues.apache.org/jira/browse/BEAM-8101: Flakes in 
ParDoLifecycleTest.testTeardownCalledAfterExceptionInStartBundleStateful for 
Direct, Spark, Flink (created 2019-08-27)
https://issues.apache.org/jira/browse/BEAM-8035: 
WatchTest.testMultiplePollsWithManyResults flake: Outputs must be in timestamp 
order (sickbayed) (created 2019-08-22)
https://issues.apache.org/jira/browse/BEAM-7827: 
MetricsTest$AttemptedMetricTests.testAllAttemptedMetrics is flaky on 
DirectRunner (created 2019-07-26)
https://issues.apache.org/jira/browse/BEAM-7752: Java Validates 
DirectRunner: testTeardownCalledAfterExceptionInFinishBundleStateful flaky 
(created 2019-07-16)
https://issues.apache.org/jira/browse/BEAM-6804: [beam_PostCommit_Java] 
[PubsubReadIT.testReadPublicData] Timeout 

P1 issues report (44)

[Beam Jira Bot]

This is your daily summary of Beam's current P1 issues, not including flaky 
tests 
(https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20statusCategory%20!%3D%20Done%20AND%20priority%20%3D%20P1%20AND%20(labels%20is%20EMPTY%20OR%20labels%20!%3D%20flake).

See https://beam.apache.org/contribute/jira-priorities/#p1-critical for the 
meaning and expectations around P1 issues.

https://issues.apache.org/jira/browse/BEAM-13025: 
beam_PostCommit_Java_DataflowV2 failing pubsublite.ReadWriteIT (created 
2021-10-08)
https://issues.apache.org/jira/browse/BEAM-13010: Delete orphaned files 
(created 2021-10-06)
https://issues.apache.org/jira/browse/BEAM-12995: Consumer group with 
random prefix (created 2021-10-04)
https://issues.apache.org/jira/browse/BEAM-12959: Dataflow error in 
CombinePerKey operation (created 2021-09-26)
https://issues.apache.org/jira/browse/BEAM-12867: Either Create or 
DirectRunner fails to produce all elements to the following transform (created 
2021-09-09)
https://issues.apache.org/jira/browse/BEAM-12843: (Broken Pipe induced) 
Bricked Dataflow Pipeline  (created 2021-09-06)
https://issues.apache.org/jira/browse/BEAM-12818: When writing to GCS, 
spread prefix of temporary files and reuse autoscaling of the temporary 
directory (created 2021-08-30)
https://issues.apache.org/jira/browse/BEAM-12807: Java creates an incorrect 
pipeline proto when core-construction-java jar is not in the CLASSPATH (created 
2021-08-26)
https://issues.apache.org/jira/browse/BEAM-12792: Beam worker only installs 
--extra_package once (created 2021-08-24)
https://issues.apache.org/jira/browse/BEAM-12766: Already Exists: Dataset 
apache-beam-testing:python_bq_file_loads_NNN (created 2021-08-16)
https://issues.apache.org/jira/browse/BEAM-12632: ElasticsearchIO: Enabling 
both User/Pass auth and SSL overwrites User/Pass (created 2021-07-16)
https://issues.apache.org/jira/browse/BEAM-12540: 
beam_PostRelease_NightlySnapshot - Task 
:runners:direct-java:runMobileGamingJavaDirect FAILED (created 2021-06-25)
https://issues.apache.org/jira/browse/BEAM-12525: SDF BoundedSource seems 
to execute significantly slower than 'normal' BoundedSource (created 2021-06-22)
https://issues.apache.org/jira/browse/BEAM-12505: codecov/patch has poor 
behavior (created 2021-06-17)
https://issues.apache.org/jira/browse/BEAM-12500: Dataflow SocketException 
(SSLException) error while trying to send message from Cloud Pub/Sub to 
BigQuery (created 2021-06-16)
https://issues.apache.org/jira/browse/BEAM-12484: JdbcIO date conversion is 
sensitive to OS (created 2021-06-14)
https://issues.apache.org/jira/browse/BEAM-12467: 
java.io.InvalidClassException With Flink Kafka (created 2021-06-09)
https://issues.apache.org/jira/browse/BEAM-12279: Implement 
destination-dependent sharding in FileIO.writeDynamic (created 2021-05-04)
https://issues.apache.org/jira/browse/BEAM-12256: 
PubsubIO.readAvroGenericRecord creates SchemaCoder that fails to decode some 
Avro logical types (created 2021-04-29)
https://issues.apache.org/jira/browse/BEAM-11959: Python Beam SDK Harness 
hangs when installing pip packages (created 2021-03-11)
https://issues.apache.org/jira/browse/BEAM-11906: No trigger early 
repeatedly for session windows (created 2021-03-01)
https://issues.apache.org/jira/browse/BEAM-11875: XmlIO.Read does not 
handle XML encoding per spec (created 2021-02-26)
https://issues.apache.org/jira/browse/BEAM-11828: JmsIO is not 
acknowledging messages correctly (created 2021-02-17)
https://issues.apache.org/jira/browse/BEAM-11755: Cross-language 
consistency (RequiresStableInputs) is quietly broken (at least on portable 
flink runner) (created 2021-02-05)
https://issues.apache.org/jira/browse/BEAM-11578: `dataflow_metrics` 
(python) fails with TypeError (when int overflowing?) (created 2021-01-06)
https://issues.apache.org/jira/browse/BEAM-11148: Kafka 
commitOffsetsInFinalize OOM on Flink (created 2020-10-28)
https://issues.apache.org/jira/browse/BEAM-11017: Timer with dataflow 
runner can be set multiple times (dataflow runner) (created 2020-10-05)
https://issues.apache.org/jira/browse/BEAM-10670: Make non-portable 
Splittable DoFn the only option when executing Java "Read" transforms (created 
2020-08-10)
https://issues.apache.org/jira/browse/BEAM-10617: python 
CombineGlobally().with_fanout() cause duplicate combine results for sliding 
windows (created 2020-07-31)
https://issues.apache.org/jira/browse/BEAM-10569: SpannerIO tests don't 
actually assert anything. (created 2020-07-23)
https://issues.apache.org/jira/browse/BEAM-10529: Kafka XLang fails for 
?empty? key/values (created 2020-07-18)
https://issues.apache.org/jira/browse/BEAM-10288: Quickstart documents are 
out of date (created 2020-06-19)
https://issues.apache.org/jira/browse/BEAM-10244: Populate requirements 
cache fails on poetry-based 

Re: [!SPOOF] Re: [REVIEW] Can you please review my PR https://github.com/apache/beam/pull/15667

[Dmitrii Kuzin]

Thank you for tagging.
But I don’t see activity from reviewers(.
Can you speed up process of the review?
I had pinged the reviewers in PR, but did not get any result.
May be add more reviewers or something else?
Or can I ask the reviewers directly?

Thanks.

From: Dmitrii Kuzin 
Reply-To: "dev@beam.apache.org" 
Date: Thursday, October 7, 2021 at 22:49
To: "dev@beam.apache.org" 
Subject: [!SPOOF] Re: [REVIEW] Can you please review my PR 
https://github.com/apache/beam/pull/15667

Thanks.

From: Valentyn Tymofieiev 
Reply-To: "dev@beam.apache.org" 
Date: Thursday, October 7, 2021 at 19:48
To: "dev@beam.apache.org" 
Subject: Re: [REVIEW] Can you please review my PR 
https://github.com/apache/beam/pull/15667

Hi Dmitrii,

I tagged a couple of folks who recently looked at this code.

Thanks for your contribution.

On Thu, Oct 7, 2021 at 12:15 AM Dmitrii Kuzin 
mailto:dmitrii_ku...@epam.com>> wrote:
Hi!
Can you please review my PR 
https://github.com/apache/beam/pull/15667
 ?

Thanks a lot!
--
DMITRII KUZIN
Lead Software Engineer


Nizhny Novgorod, Russia

Working Remotely Timezone: (UTC+03:00)

EPAM.COM

Beam Dependency Check Report (2021-10-11)

[Apache Jenkins Server]

High Priority Dependency Updates Of Beam Python SDK:


  Dependency Name
  Current Version
  Latest Version
  Release Date Of the Current Used Version
  Release Date Of The Latest Release
  JIRA Issue
  
chromedriver-binary
93.0.4577.63.0
95.0.4638.17.0
2021-09-20
2021-10-04BEAM-10426
dill
0.3.1.1
0.3.4
2019-10-07
2021-06-14BEAM-11167
google-api-core
1.31.3
2.1.0
2021-09-27
2021-10-11BEAM-12784
google-auth
1.35.0
2.3.0
2021-08-23
2021-10-11BEAM-12785
google-cloud-bigtable
1.7.0
2.4.0
2021-04-12
2021-10-04BEAM-8127
google-cloud-datastore
1.15.3
2.2.0
2020-11-16
2021-10-11BEAM-8443
google-cloud-dlp
1.0.0
3.2.4
2020-06-29
2021-10-11BEAM-10344
google-cloud-language
1.3.0
2.2.2
2020-10-26
2021-08-02BEAM-8
google-cloud-pubsub
1.7.0
2.8.0
2020-07-20
2021-09-06BEAM-5539
google-cloud-spanner
1.19.1
3.11.1
2020-11-16
2021-10-11BEAM-10345
google-cloud-videointelligence
1.16.1
2.4.0
2020-11-23
2021-10-11BEAM-11319
google-cloud-vision
1.0.0
2.4.4
2020-03-24
2021-10-04BEAM-9581
ipykernel
5.5.6
6.4.1
None
2021-09-13BEAM-12575
jupyter-client
6.1.12
7.0.6
2021-04-12
2021-10-11BEAM-12786
mock
2.0.0
4.0.3
2019-05-20
2020-12-14BEAM-7369
mypy-protobuf
1.18
2.10
2020-03-24
2021-09-20BEAM-10346
Pillow
7.2.0
8.3.2
2020-10-19
2021-09-06BEAM-11071
pluggy
0.13.1
1.0.0
2021-08-30
2021-08-30BEAM-12819
PyHamcrest
1.10.1
2.0.2
2020-01-20
2020-07-08BEAM-9155
pytest
4.6.11
6.2.5
2020-07-08
2021-09-06BEAM-8606
pytest-timeout
1.4.2
2.0.0
None
2021-10-11BEAM-13029
pytest-xdist
1.34.0
2.4.0
2020-08-17
2021-09-27BEAM-10713
tenacity
5.1.5
8.0.1
2019-11-11
2021-07-19BEAM-8607
High Priority Dependency Updates Of Beam Java SDK:


  Dependency Name
  Current Version
  Latest Version
  Release Date Of the Current Used Version
  Release Date Of The Latest Release
  JIRA Issue
  
com.alibaba:fastjson
1.2.69
1.2.78
2020-05-31
2021-08-09BEAM-8632
com.azure:azure-core
1.9.0
1.21.0
2020-10-02
2021-10-02BEAM-11888
com.azure:azure-identity
1.0.8
1.4.0-beta.1
2020-07-07
2021-09-13BEAM-11814
com.azure:azure-storage-common
12.10.0
12.13.0
2021-01-14
2021-09-15BEAM-11889
com.datastax.cassandra:cassandra-driver-core
3.10.2
4.0.0
2020-08-26
2019-03-18BEAM-8674
com.esotericsoftware:kryo
4.0.2
5.2.0
2018-03-20
2021-07-31BEAM-5809
com.esotericsoftware.kryo:kryo
2.21
2.24.0
2013-02-27
2014-05-04BEAM-5574
com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin
0.33.0
0.39.0
2020-09-14
2021-05-31BEAM-6645
com.github.jk1.dependency-license-report:com.github.jk1.dependency-license-report.gradle.plugin
1.16
2.0
2020-10-26
2021-09-06BEAM-11120
com.github.luben:zstd-jni
1.4.5-2
1.5.0-4
2020-05-30
2021-07-21BEAM-9194
com.github.spotbugs:spotbugs
4.0.6
4.4.1
2020-06-23
2021-09-10BEAM-7792
com.github.spotbugs:spotbugs-annotations
4.0.6
4.4.1
2020-06-23
2021-09-10BEAM-6951
com.google.api.grpc:proto-google-cloud-dlp-v2
1.1.4
3.0.8
2020-05-04
2021-09-29BEAM-11892
com.google.api.grpc:proto-google-cloud-recommendations-ai-v1beta1
0.3.7
0.8.1
2020-11-02
2021-09-22BEAM-12517
com.google.api.grpc:proto-google-cloud-video-intelligence-v1
1.2.0
2.0.9
2020-03-10
2021-10-05BEAM-11894
com.google.api.grpc:proto-google-cloud-vision-v1
1.81.3
2.0.13
2020-04-07
2021-09-24BEAM-11895
com.google.apis:google-api-services-bigquery
v2-rev20210813-1.32.1
v2-rev20210927-1.32.1
2021-08-19
2021-10-02BEAM-8684

Switching Python sdist format

[Yoshiki Obata]

Hello everyone,

I'm working on BEAM-8954[1] which introduces tox isolated_build for
python tests.
Concerning this issue, I want opinions about using .tar.gz as sdist format.

Introducing tox isolated_build leads replacement of
build-requirements.txt to pyproject.toml[2] and we should use
pyproject.toml when creating sdist because we install dependencies
with build-requirements.txt before calling "python setup.py sdist"
PEP 517 based build tools like pypa/build will help to do so, but it
does not allow .zip as sdist format[3].
Therefore I think it would be better to switch sdist format to .tar.gz
when starting to use pyproject.toml.

Are there any obstacles to use .tar.gz?
Please let me know details about adopting .zip as Beam sdist format(I
could not find discussions about this)

Regards,
yoshiki

[1] https://issues.apache.org/jira/browse/BEAM-8954
[2] 
https://tox.wiki/en/latest/config.html?highlight=isolated#conf-isolated_build
[3] https://www.python.org/dev/peps/pep-0517/#source-distributions