Re: [BEAM-13019] Potential breaking change adding matchers to `PAssert.that(pcollection).containsInAnyOrder`

[Chris Gray]

It does work.  We could add it and mark it deprecated with a note to use 
empty()​ instead.

From: Daniel Collins 
Sent: Monday, October 18, 2021 12:35 PM
To: dev 
Subject: Re: [BEAM-13019] Potential breaking change adding matchers to 
`PAssert.that(pcollection).containsInAnyOrder`

Is there another option to add a `containsInAnyOrder()` overload? I wonder if 
this would resolve the ambiguity.



On Mon, Oct 18, 2021 at 12:32 PM Luke Cwik 
mailto:lc...@google.com>> wrote:
It looks like there isn't any pushback. Let us go with the change as you have 
suggested.

On Wed, Oct 13, 2021 at 1:46 PM Luke Cwik 
mailto:lc...@google.com>> wrote:
I'm for the breaking change but if people have lots of push back I would go 
with containsInAnyOrder(SerializableMatcher firstMatcher, 
SerializableMatcher... rest) as my second choice. As a second choice we 
would be able to swap containsInAnyOrder(SerializableMatcher firstMatcher, 
SerializableMatcher... rest) with 
containsInAnyOrder(SerializableMatcher... matchers) and revisit this 
"breaking" change in the future.

Note that there is already an existing method "containsInAnyOrder(Iterable 
expectedElements)" as well. Should we expose 
"containsInAnyOrder(Iterable> matchers)"?

Another option would be to use a different method name but this isn't what 
people typically would expect when coming from typical junit/hamcrest/mockito 
matchers.


On Wed, Oct 13, 2021 at 12:22 PM Chris Gray 
mailto:cg...@ftsinc.com>> wrote:
Hi, I recently noticed that some very useful assertion methods are not 
available to use in the IterableAssert​ interface.  In particular, I wanted to 
add containsInAnyOrder(SerializableMatcher... matchers)​.  I did this in 
https://github.com/apache/beam/pull/15685.   However, this causes some tests to 
not compile because they use containsInAnyOrder()​ to mean empty()​, which 
causes the compiler to give up because the method is now ambiguous with 
containsInAnyOrder(T... elements)​.

It seems like there are a few potential ways forward

  *   Commit the breaking change and tell users to use empty()​ instead of no 
arguments to containsInAnyOrder​.
  *   Change the signature to containsInAnyOrder(SerializableMatcher 
firstMatcher, SerializableMatcher... rest)​
  *   Change the signature to 
containsInAnyOrder(Collection> matchers)​.
  *   Do nothing
  *   Something else

To me, the first two ways forward are the best.  empty()​ is more descriptive 
of what is being asserted than containsInAnyOrder()​.  But if that isn't 
tolerable, then the second should remove the ambiguity.

In any case, please let me know if you have feelings about what should be done.
[https://opengraph.githubassets.com/f93657a3199653b0d47e7f5cc35f08f7fa1fe20691c4e17e13dcdd02e2276bb9/apache/beam/pull/15685]
[BEAM-13019] Add `containsInAnyOrder` with matchers to the `IterableAssert` 
interface by chrismgrayftsinc · Pull Request #15685 · 
apache/beam
This lets an existing assertion be usable in the PAssert class. Thank you for 
your contribution! Follow this checklist to help us incorporate your 
contribution quickly and easily: Choose reviewe...
github.com


Apache beam. question

[azhar mirza]

I have few question regarding apache beam.

if apache beam is using its own stateful processing then which runner will
be used?

if we are using Java SDK which runner is flexible whether is classic apache
flink runner or portable flink runner.

Suppose we are using apache flink runner who does take care take care of
stateful processing for fault tolerance.

How is Back pressure is handled in Apace beam.

if we integrate apache beam with flink and then use apache flink runner
what are the overheads?

Could you please response at the earliest kindly would be helpful.

Thanks
Azhar

Flaky test issue report (30)

[Beam Jira Bot]

This is your daily summary of Beam's current flaky tests 
(https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20statusCategory%20!%3D%20Done%20AND%20labels%20%3D%20flake)

These are P1 issues because they have a major negative impact on the community 
and make it hard to determine the quality of the software.

https://issues.apache.org/jira/browse/BEAM-13025: 
beam_PostCommit_Java_DataflowV2 failing pubsublite.ReadWriteIT (created 
2021-10-08)
https://issues.apache.org/jira/browse/BEAM-12928: beam_PostCommit_Python36 
- CrossLanguageSpannerIOTest - flakey failing (created 2021-09-21)
https://issues.apache.org/jira/browse/BEAM-12859: 
org.apache.beam.runners.dataflow.worker.fn.logging.BeamFnLoggingServiceTest.testMultipleClientsFailingIsHandledGracefullyByServer
 is flaky (created 2021-09-08)
https://issues.apache.org/jira/browse/BEAM-12809: 
testTwoTimersSettingEachOtherWithCreateAsInputBounded flaky (created 2021-08-26)
https://issues.apache.org/jira/browse/BEAM-12794: 
PortableRunnerTestWithExternalEnv.test_pardo_timers flaky (created 2021-08-24)
https://issues.apache.org/jira/browse/BEAM-12766: Already Exists: Dataset 
apache-beam-testing:python_bq_file_loads_NNN (created 2021-08-16)
https://issues.apache.org/jira/browse/BEAM-12540: 
beam_PostRelease_NightlySnapshot - Task 
:runners:direct-java:runMobileGamingJavaDirect FAILED (created 2021-06-25)
https://issues.apache.org/jira/browse/BEAM-12515: Python PreCommit flaking 
in PipelineOptionsTest.test_display_data (created 2021-06-18)
https://issues.apache.org/jira/browse/BEAM-12322: Python precommit flaky: 
Failed to read inputs in the data plane (created 2021-05-10)
https://issues.apache.org/jira/browse/BEAM-12320: 
PubsubTableProviderIT.testSQLSelectsArrayAttributes[0] failing in SQL 
PostCommit (created 2021-05-10)
https://issues.apache.org/jira/browse/BEAM-12291: 
org.apache.beam.runners.flink.ReadSourcePortableTest.testExecution[streaming: 
false] is flaky (created 2021-05-05)
https://issues.apache.org/jira/browse/BEAM-12200: 
SamzaStoreStateInternalsTest is flaky (created 2021-04-20)
https://issues.apache.org/jira/browse/BEAM-12163: Python GHA PreCommits 
flake with grpc.FutureTimeoutError on SDK harness startup (created 2021-04-13)
https://issues.apache.org/jira/browse/BEAM-12061: beam_PostCommit_SQL 
failing on KafkaTableProviderIT.testFakeNested (created 2021-03-27)
https://issues.apache.org/jira/browse/BEAM-11837: Java build flakes: 
"Memory constraints are impeding performance" (created 2021-02-18)
https://issues.apache.org/jira/browse/BEAM-11661: hdfsIntegrationTest 
flake: network not found (py38 postcommit) (created 2021-01-19)
https://issues.apache.org/jira/browse/BEAM-11645: beam_PostCommit_XVR_Flink 
failing (created 2021-01-15)
https://issues.apache.org/jira/browse/BEAM-11641: Bigquery Read tests are 
flaky on Flink runner in Python PostCommit suites (created 2021-01-15)
https://issues.apache.org/jira/browse/BEAM-11541: 
testTeardownCalledAfterExceptionInProcessElement flakes on direct runner. 
(created 2020-12-30)
https://issues.apache.org/jira/browse/BEAM-10955: Flink Java Runner test 
flake: Could not find Flink job (FlinkJobNotFoundException) (created 2020-09-23)
https://issues.apache.org/jira/browse/BEAM-10866: 
PortableRunnerTestWithSubprocesses.test_register_finalizations flaky on macOS 
(created 2020-09-09)
https://issues.apache.org/jira/browse/BEAM-10485: Failure / flake: 
ElasticsearchIOTest > testWriteWithIndexFn (created 2020-07-14)
https://issues.apache.org/jira/browse/BEAM-9649: 
beam_python_mongoio_load_test started failing due to mismatched results 
(created 2020-03-31)
https://issues.apache.org/jira/browse/BEAM-8101: Flakes in 
ParDoLifecycleTest.testTeardownCalledAfterExceptionInStartBundleStateful for 
Direct, Spark, Flink (created 2019-08-27)
https://issues.apache.org/jira/browse/BEAM-8035: 
WatchTest.testMultiplePollsWithManyResults flake: Outputs must be in timestamp 
order (sickbayed) (created 2019-08-22)
https://issues.apache.org/jira/browse/BEAM-7827: 
MetricsTest$AttemptedMetricTests.testAllAttemptedMetrics is flaky on 
DirectRunner (created 2019-07-26)
https://issues.apache.org/jira/browse/BEAM-7752: Java Validates 
DirectRunner: testTeardownCalledAfterExceptionInFinishBundleStateful flaky 
(created 2019-07-16)
https://issues.apache.org/jira/browse/BEAM-6804: [beam_PostCommit_Java] 
[PubsubReadIT.testReadPublicData] Timeout waiting on Sub (created 2019-03-11)
https://issues.apache.org/jira/browse/BEAM-5286: 
[beam_PostCommit_Java_GradleBuild][org.apache.beam.examples.subprocess.ExampleEchoPipelineTest.testExampleEchoPipeline][Flake]
 .sh script: text file busy. (created 2018-09-01)
https://issues.apache.org/jira/browse/BEAM-5172: 
org.apache.beam.sdk.io.elasticsearch/ElasticsearchIOTest is flaky (created 
2018-08-20)

Re: Running python tests in an IDE (IntelliJ, PyCharm)

[Steve Niemitz]

I eventually got it working by `pip install -e .` into my beam dev virtual
env.  Otherwise I was getting a lot of errors around source roots not being
found, weird problems with packages shadowing builtin names (io, etc) and
others.  Are there setup instructions anywhere like the java side for the
"right" way to do it?

On Tue, Oct 19, 2021 at 3:23 PM Svetak Sundhar 
wrote:

> Yes, I was able to. Specifically, are you having trouble running
> individual unit tests? If so, what errors are you getting?
>
>
> Svetak Sundhar
>
>   Technical Solutions Engineer, Data
> s vetaksund...@google.com
>
>
>
> On Tue, Oct 19, 2021 at 2:13 PM Steve Niemitz  wrote:
>
>> Has anyone gotten the beam python sdk tests running in an IDE (I'm
>> specifically trying with IntelliJ/PyCharm)?  I didn't see anything on the
>> wiki about it and haven't gotten it working yet.
>>
>

Re: Best practices for upgrading installed dependencies on Jenkins VMs?

[Robert Burke]

+1 to only having one way to do things. The Lite option seems liable to
cause more problems since it means it's changes can be blown away if a new
image isn't prepared anyway.
I don't think we are changing the images often enough for it.  Perhaps call
it the option to test changes if anything?

On Tue, Oct 19, 2021, 11:55 AM Valentyn Tymofieiev 
wrote:

> All workers were updated to use jenkins-slave-boot-image-20211011, which
> should have had a go command, but it appears slightly misconfigured. I
> reopened BEAM-13037 [1] and added some details there.
>
> I also added instructions to wiki [2] on how to perform an image swap and
> it is actually very straightforward. I think a lesson here is that making
> 'lite' upgrades is brittle as misconfigurations could resurface down the
> road when the context of the lite upgrade is no longer fresh in our memory.
>
> I suggest we revise the instructions to keep only image swap commands and
> remove the 'lite' update option. +Daniel Oliveira ,
> WDYT?  In the meantime, we should also prepare an image that fixes the
> misconfiguration. Would you be able to help with that? Thank you.
>
> [1] https://issues.apache.org/jira/browse/BEAM-13037
> [2]
> https://cwiki.apache.org/confluence/display/BEAM/Jenkins+Tips#JenkinsTips-HowtoinstallandupgradesoftwareonJenkinsworkers
>
>
> On Tue, Oct 19, 2021 at 8:46 AM Robert Burke  wrote:
>
>> FYI it looks like all the Go tests are now failing because it can't find
>> the Go command at all.
>> Did a Jenkins image without Go (v1.16+) pre-installed get pushed?
>>
>> On Mon, Oct 18, 2021, 1:45 PM Valentyn Tymofieiev 
>> wrote:
>>
>>> Thanks Daniel,
>>>
>>> I can recreate the VMs on new disks.
>>>
>>> We currently have a set of stopped jenkins workers (named:
>>> apache-beam-jenkins-##) and running workers (named:
>>> apache-ci-beam-jenkins-##)
>>>
>>> Are there any concerns about deleting the stopped group of workers?
>>>
>>>
>>>
>>> On Mon, Oct 18, 2021 at 11:19 AM Ahmet Altay  wrote:
>>>
 Thank you Daniel, Valentyn!

 On Mon, Oct 18, 2021 at 8:02 AM Daniel Oliveira 
 wrote:

> I performed a light update of both Go and Python (from Valentyn's
> update) on each worker VM over the weekend. I also added additional
> instructions for the light update to Confluence (as an alternative to the
> current instructions).
>
> There is still reason to perform a full update at some point: Valentyn
> updated the VM image from 500 GB to 1000 GB of storage, which requires a
> full update to actually take effect.
>
> On Tue, Oct 12, 2021 at 10:32 AM Valentyn Tymofieiev <
> valen...@google.com> wrote:
>
>> > 3. SSH into the agent and perform the update.
>> So, this would be a 'lite' version of the update, where we make
>> changes to the live worker without recreating worker VM with a new image?
>> We could perhaps document both options, and also make it clear that
>> producing a VM image that has necessary updates is mandatory even if we
>> perform 'lite' updates without recreating the worker.
>> Also, for a lite update, marking the Jenkins offer offline may be
>> optional, as some updates might not be disruptive (such as installing 
>> some
>> software that will not be used immediately).
>>
>>
>>
>> On Mon, Oct 11, 2021 at 7:53 PM Robert Burke 
>> wrote:
>>
>>> SGTM. Thank you very much Daniel!
>>>
>>> On Mon, Oct 11, 2021, 7:51 PM Ahmet Altay  wrote:
>>>
 Thank you Daniel. Could you please update the wiki once you are
 done with the process?

 On Mon, Oct 11, 2021 at 6:22 PM Daniel Oliveira <
 danolive...@google.com> wrote:

> Took me a bit to get to this, sorry. I finally figured out an
> approach for updating Go and did so and will be updating the image
> momentarily.
>
> I think a more important note is that I tried what Valentyn was
> considering, which is SSHing into workers and updating the 
> dependency. I'll
> describe the process below, but the summary is that I did it on one 
> worker
> with Go so far, saw no problems over the weekend, and would like to
> continue updating the rest of the workers if there are no objections.
>
> Here's a step-by-step of what I did. If we decide to stick with
> this approach, these instructions can be added to Confluence:
>
> 1. Go to the page for the Jenkins agent you want to update [1] and
> click "Mark this node temporarily offline", leaving a reason such as
> "Updating X dependency."
> 2. Wait until there are no more tests running in that agent (under
> "Build Executor Status" on the left of the page).
> 3. SSH into the agent and perform the update.
> 4. Mark the node as online again.
> 5. Repeat for every worker.
>

Codecov reports question

[Miguel Anzo Palomo]

Hi, I’m working on adding codecov coverage for go tests on PR's, currently
I have generated the report for the coverage but I have some questions
about how to upload it to codecov [1]. Looking at the current python
actions I see that the coverage report is generated via tox.ini, and I can
see that there is a report uploaded also there [2]. Is that the report and
upload used by codecov? If that's the case, is there a specific filename
that the file needs to have or documentation about it? thanks.

[1] https://github.com/apache/beam/blob/master/.github/codecov.yml
[2]
https://github.com/apache/beam/blob/master/.github/workflows/python_tests.yml#L129


-- 

Miguel Angel Anzo Palomo | WIZELINE

Software Engineer

miguel.a...@wizeline.com

Remote Office

-- 
*This email and its contents (including any attachments) are being sent to
you on the condition of confidentiality and may be protected by legal
privilege. Access to this email by anyone other than the intended recipient
is unauthorized. If you are not the intended recipient, please immediately
notify the sender by replying to this message and delete the material
immediately from your system. Any further use, dissemination, distribution
or reproduction of this email is strictly prohibited. Further, no
representation is made with respect to any content contained in this email.*

Running python tests in an IDE (IntelliJ, PyCharm)

[Steve Niemitz]

Has anyone gotten the beam python sdk tests running in an IDE (I'm
specifically trying with IntelliJ/PyCharm)?  I didn't see anything on the
wiki about it and haven't gotten it working yet.

P1 issues report (51)

[Beam Jira Bot]

This is your daily summary of Beam's current P1 issues, not including flaky 
tests 
(https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20statusCategory%20!%3D%20Done%20AND%20priority%20%3D%20P1%20AND%20(labels%20is%20EMPTY%20OR%20labels%20!%3D%20flake).

See https://beam.apache.org/contribute/jira-priorities/#p1-critical for the 
meaning and expectations around P1 issues.

https://issues.apache.org/jira/browse/BEAM-13078: Python DirectRunner does 
not emit data at GC time (created 2021-10-18)
https://issues.apache.org/jira/browse/BEAM-13076: Python AfterAny, AfterAll 
do not follow spec (created 2021-10-18)
https://issues.apache.org/jira/browse/BEAM-13074: Metrics are not reported 
by the Flink runner (created 2021-10-18)
https://issues.apache.org/jira/browse/BEAM-13060: Daily Python SDK build is 
not publicly accessible (created 2021-10-15)
https://issues.apache.org/jira/browse/BEAM-13059: Migrate GKE workloads to 
Containerd (created 2021-10-15)
https://issues.apache.org/jira/browse/BEAM-13058: Upgrade Kubernetes APIs 
(created 2021-10-15)
https://issues.apache.org/jira/browse/BEAM-13056: Add method to fetch 
ProcessContext FieldAccess (created 2021-10-14)
https://issues.apache.org/jira/browse/BEAM-13053: Avoid runner v2 when 
streaming engine explicitly disabled. (created 2021-10-14)
https://issues.apache.org/jira/browse/BEAM-13025: 
beam_PostCommit_Java_DataflowV2 failing pubsublite.ReadWriteIT (created 
2021-10-08)
https://issues.apache.org/jira/browse/BEAM-13010: Delete orphaned files 
(created 2021-10-06)
https://issues.apache.org/jira/browse/BEAM-12995: Consumer group with 
random prefix (created 2021-10-04)
https://issues.apache.org/jira/browse/BEAM-12959: Dataflow error in 
CombinePerKey operation (created 2021-09-26)
https://issues.apache.org/jira/browse/BEAM-12867: Either Create or 
DirectRunner fails to produce all elements to the following transform (created 
2021-09-09)
https://issues.apache.org/jira/browse/BEAM-12843: (Broken Pipe induced) 
Bricked Dataflow Pipeline  (created 2021-09-06)
https://issues.apache.org/jira/browse/BEAM-12818: When writing to GCS, 
spread prefix of temporary files and reuse autoscaling of the temporary 
directory (created 2021-08-30)
https://issues.apache.org/jira/browse/BEAM-12807: Java creates an incorrect 
pipeline proto when core-construction-java jar is not in the CLASSPATH (created 
2021-08-26)
https://issues.apache.org/jira/browse/BEAM-12792: Beam worker only installs 
--extra_package once (created 2021-08-24)
https://issues.apache.org/jira/browse/BEAM-12766: Already Exists: Dataset 
apache-beam-testing:python_bq_file_loads_NNN (created 2021-08-16)
https://issues.apache.org/jira/browse/BEAM-12632: ElasticsearchIO: Enabling 
both User/Pass auth and SSL overwrites User/Pass (created 2021-07-16)
https://issues.apache.org/jira/browse/BEAM-12540: 
beam_PostRelease_NightlySnapshot - Task 
:runners:direct-java:runMobileGamingJavaDirect FAILED (created 2021-06-25)
https://issues.apache.org/jira/browse/BEAM-12525: SDF BoundedSource seems 
to execute significantly slower than 'normal' BoundedSource (created 2021-06-22)
https://issues.apache.org/jira/browse/BEAM-12505: codecov/patch has poor 
behavior (created 2021-06-17)
https://issues.apache.org/jira/browse/BEAM-12500: Dataflow SocketException 
(SSLException) error while trying to send message from Cloud Pub/Sub to 
BigQuery (created 2021-06-16)
https://issues.apache.org/jira/browse/BEAM-12484: JdbcIO date conversion is 
sensitive to OS (created 2021-06-14)
https://issues.apache.org/jira/browse/BEAM-12467: 
java.io.InvalidClassException With Flink Kafka (created 2021-06-09)
https://issues.apache.org/jira/browse/BEAM-12279: Implement 
destination-dependent sharding in FileIO.writeDynamic (created 2021-05-04)
https://issues.apache.org/jira/browse/BEAM-12256: 
PubsubIO.readAvroGenericRecord creates SchemaCoder that fails to decode some 
Avro logical types (created 2021-04-29)
https://issues.apache.org/jira/browse/BEAM-11959: Python Beam SDK Harness 
hangs when installing pip packages (created 2021-03-11)
https://issues.apache.org/jira/browse/BEAM-11906: No trigger early 
repeatedly for session windows (created 2021-03-01)
https://issues.apache.org/jira/browse/BEAM-11875: XmlIO.Read does not 
handle XML encoding per spec (created 2021-02-26)
https://issues.apache.org/jira/browse/BEAM-11828: JmsIO is not 
acknowledging messages correctly (created 2021-02-17)
https://issues.apache.org/jira/browse/BEAM-11755: Cross-language 
consistency (RequiresStableInputs) is quietly broken (at least on portable 
flink runner) (created 2021-02-05)
https://issues.apache.org/jira/browse/BEAM-11578: `dataflow_metrics` 
(python) fails with TypeError (when int overflowing?) (created 2021-01-06)
https://issues.apache.org/jira/browse/BEAM-11148: Kafka 
commitOffsetsInFinalize OOM on Flink 

Re: [EXTERNAL] [RFC][design/idea] Beam Playground - Interactive Learning for Apache Beam

[Alex Kosolapov]

Hi all,

We are working on developing the Beam Playground experience - a web application 
where users can try out Beam examples without having to install/initialize a 
Beam environment.

The proposed Playground experience is summarized in the mockups [1]. Your 
feedback would be very valuable to make sure we develop a useful application.

Thanks,
Alex

[1]https://docs.google.com/document/d/1v-VOG3c8A-hQj_-fp0VJ9cmGTvp24pr1xXomtDQFbIo/edit?usp=sharing

From: Aizhamal Nurmamat kyzy 
Date: Tuesday, September 21, 2021 at 10:52 AM
To: dev , "u...@beam.apache.org" , 
Pablo Estrada , Damon Douglas , 
Alex Kosolapov , Artur Khanin 
, Ilya Kozyrev 
Subject: [EXTERNAL] [RFC][design/idea] Beam Playground - Interactive Learning 
for Apache Beam


TL:DR: We want to develop a Beam Playground. Please review the design[1] and 
fill up this survey[2] to help us 
prioritize features.


Hi all,


I along with a few community members thought of an idea to develop an 
interactive environment to try out Beam transforms and examples. We are calling 
it Beam Playground. The vision for the Playground is to be a *web application 
where users can try out Beam without having to install / initialize a Beam 
environment*.


We hope that this new tool will make it easier for new users to evaluate and 
adopt Apache Beam, and provide positive developer experience.


The backend architecture and implementation details are described in the design 
document[1].

We would be grateful if you filled out this poll[2] to help us prioritize Beam 
Playground’s features.


Please share your feedback both on the idea and the design doc, and via the 
survey.


We hope to build the Beam Playground in the next few months (see the general 
timeline[3]), and your feedback would be very valuable to make sure we develop 
a useful application.


Thanks!

Aizhamal


[1] 
https://docs.google.com/document/d/1uf58Auags4DBqSU3nZWxfaDH2BcBvKxhmy0Ug-xoewQ/edit?usp=sharing

[2] https://forms.gle/ieknXjFSezqbAmFd6

[3] 
https://docs.google.com/presentation/d/13ETJItGH3QV9hlQXce-47ZEDjOZUNNCJc5PyUUI1EIw/edit#slide=id.gef11b54893_0_0

Re: Best practices for upgrading installed dependencies on Jenkins VMs?

[Robert Burke]

FYI it looks like all the Go tests are now failing because it can't find
the Go command at all.
Did a Jenkins image without Go (v1.16+) pre-installed get pushed?

On Mon, Oct 18, 2021, 1:45 PM Valentyn Tymofieiev 
wrote:

> Thanks Daniel,
>
> I can recreate the VMs on new disks.
>
> We currently have a set of stopped jenkins workers (named:
> apache-beam-jenkins-##) and running workers (named:
> apache-ci-beam-jenkins-##)
>
> Are there any concerns about deleting the stopped group of workers?
>
>
>
> On Mon, Oct 18, 2021 at 11:19 AM Ahmet Altay  wrote:
>
>> Thank you Daniel, Valentyn!
>>
>> On Mon, Oct 18, 2021 at 8:02 AM Daniel Oliveira 
>> wrote:
>>
>>> I performed a light update of both Go and Python (from Valentyn's
>>> update) on each worker VM over the weekend. I also added additional
>>> instructions for the light update to Confluence (as an alternative to the
>>> current instructions).
>>>
>>> There is still reason to perform a full update at some point: Valentyn
>>> updated the VM image from 500 GB to 1000 GB of storage, which requires a
>>> full update to actually take effect.
>>>
>>> On Tue, Oct 12, 2021 at 10:32 AM Valentyn Tymofieiev <
>>> valen...@google.com> wrote:
>>>
 > 3. SSH into the agent and perform the update.
 So, this would be a 'lite' version of the update, where we make changes
 to the live worker without recreating worker VM with a new image? We could
 perhaps document both options, and also make it clear that producing a VM
 image that has necessary updates is mandatory even if we perform 'lite'
 updates without recreating the worker.
 Also, for a lite update, marking the Jenkins offer offline may be
 optional, as some updates might not be disruptive (such as installing some
 software that will not be used immediately).



 On Mon, Oct 11, 2021 at 7:53 PM Robert Burke 
 wrote:

> SGTM. Thank you very much Daniel!
>
> On Mon, Oct 11, 2021, 7:51 PM Ahmet Altay  wrote:
>
>> Thank you Daniel. Could you please update the wiki once you are done
>> with the process?
>>
>> On Mon, Oct 11, 2021 at 6:22 PM Daniel Oliveira <
>> danolive...@google.com> wrote:
>>
>>> Took me a bit to get to this, sorry. I finally figured out an
>>> approach for updating Go and did so and will be updating the image
>>> momentarily.
>>>
>>> I think a more important note is that I tried what Valentyn was
>>> considering, which is SSHing into workers and updating the dependency. 
>>> I'll
>>> describe the process below, but the summary is that I did it on one 
>>> worker
>>> with Go so far, saw no problems over the weekend, and would like to
>>> continue updating the rest of the workers if there are no objections.
>>>
>>> Here's a step-by-step of what I did. If we decide to stick with this
>>> approach, these instructions can be added to Confluence:
>>>
>>> 1. Go to the page for the Jenkins agent you want to update [1] and
>>> click "Mark this node temporarily offline", leaving a reason such as
>>> "Updating X dependency."
>>> 2. Wait until there are no more tests running in that agent (under
>>> "Build Executor Status" on the left of the page).
>>> 3. SSH into the agent and perform the update.
>>> 4. Mark the node as online again.
>>> 5. Repeat for every worker.
>>>
>>> And these are some additional steps if you want to immediately run a
>>> test suite to check that the update worked correctly. For example in my
>>> case, I wanted to check against the Go Postcommit, and it was a good 
>>> thing
>>> I did, because it actually failed the first time and I had to go back 
>>> in to
>>> fix a small oversight I made. So doing this after you update your first
>>> worker is probably a good idea before updating the rest:
>>>
>>> 1. Go to the page for the job you want to run (for example: [2]).
>>> 2. Click "Configure" on the left menu.
>>> 3. Find the checkmark "Restrict where this project can be run" and
>>> change the restriction from "beam" to the specific name of the agent 
>>> (ex.
>>> "apache-beam-jenkins-1").
>>> 4. Save and apply that change.
>>> 5. Back on the page for the job, click "Build with Parameters" on
>>> the left menu.
>>> 6. Run the build on "master".
>>> 7. Once you're done checking the results, change the restriction for
>>> the job back to "beam". (This also gets reset once every 24 hours in 
>>> case
>>> you forget.)
>>>
>>> I did that on one agent (apache-beam-jenkins-2) on Friday evening
>>> when it wasn't too busy, and got Go updated and working. I checked that
>>> agent's execution history again today just in case, and it was healthy 
>>> over
>>> the weekend, with no Go-related problems as far as I could see. If 
>>> there's
>>> no objections I'd like to go ahead and continue 

Re: IO Connector

[Matt Casters]

Thanks Brian,

I couldn't so I noticed with "git status" that something had changed some
gradle files.  Probably an earlier attempt with the older gradle version or
IDEA.  Getting everything back in sync with upstream master all of a sudden
made everything come back to life.  Thanks for sticking with me!
When I'm done I'll document the whole process.

Cheers,
Matt

On Mon, Oct 18, 2021 at 9:10 PM Brian Hulette  wrote:

> Can you run "./gradlew sdks:java:core:compileJava" or "./gradlew
> sdks:java:core:test"? I'm not sure why gradle is failing to find the idea
> task for you - could you share the output of "./gradlew tasks"?
>
> On Mon, Oct 18, 2021 at 1:19 AM Matt Casters <
> matt.cast...@neotechnology.com> wrote:
>
>> Thanks a lot for the advice given last week.
>> Just to circle back: I've updated gradle to a recent version which
>> appears to be 7.1.1. ... to no avail.
>>
>> tasks like:
>>
>> ./gradlew idea
>>
>> result simply in
>>
>> Task 'idea' not found in root project 'beam'.
>>
>> The same goes for the other suggestions.
>>
>> As for Google Auto: is this project still maintained? The docs and so on
>> seem to be getting quite old.
>> The annotation processor in the latest IntelliJ doesn't seem to get
>> picked up even if you configure it manually in the settings.
>> So I'll skip that one for now.
>>
>> Are there any build instructions I can follow for Beam to at least try to
>> build the Java SDK and go from there?
>>
>> Thanks,
>>
>> Matt
>>
>>
>> On Tue, Oct 12, 2021 at 9:58 PM Evan Galpin 
>> wrote:
>>
>>> @Matt have you tried any of the "IDE Tasks" available through gradle?
>>> "./gradlew tasks" from beam top-level will list available tasks, and the
>>> IDE Tasks subsection includes tasks specific to trying to bootstrap or
>>> clean up beam project in either Eclipse or Intellij.  Ex. "./gradlew idea"
>>> should set up the project files for use in Intellij.  There's also
>>> "./gradlew cleanIdea" which may be helpful to you.
>>>
>>> With respect to Google Auto, I've experienced plenty of IDE complaints
>>> around missing types and the like, and those will likely persist until the
>>> code area that you're working on is compiled because the types won't exist
>>> until the sources are generated at pre-processing stage. Not sure if that
>>> was the issue you were having, but if so hopefully this helps.
>>>
>>> Thanks,
>>> Evan
>>>
>>> On Tue, Oct 12, 2021 at 2:51 PM Matt Casters 
>>> wrote:
>>>
 Thanks Chamikara but I'm quite familiar with the Beam API and the
 contribution guide did not answer my questions.

 On Tue, Oct 12, 2021 at 8:49 PM Chamikara Jayalath <
 chamik...@google.com> wrote:

> If you haven't already, going through Beam contribution guide and
> varils links from there might help:
> https://beam.apache.org/contribute/
> Regarding developing I/O connectors, please see the guide here:
> https://beam.apache.org/documentation/io/developing-io-overview/
>
> Thanks,
> Cham
>
> On Tue, Oct 12, 2021 at 6:08 AM Alexey Romanenko <
> aromanenko@gmail.com> wrote:
>
>> Hi Matt,
>>
>> On 12 Oct 2021, at 10:02, Matt Casters <
>> matt.cast...@neotechnology.com> wrote:
>>
>> 1) Setting up my Beam development
>> 
>>  environment
>> for IDEA 2021.2 is something that's going wrong, probably around Gradle
>> configurations.
>>
>> 2) I can't get Google Auto to work in my IDE (IDEA) because of what
>> seems outdated documentation
>> 
>> ?
>>
>>
>> Could you elaborate more what is wrong with 1) and 2) ?
>>
>
>> 3) Since I'm obviously planning to generate a PR at the end of this
>> exercise: what is the suggested code format for Java in the Beam project?
>>
>>
>> Please, run this command before committing your changes:
>>
>> ./gradlew spotlessApply && ./gradlew
>> -PenableCheckerFramework=true checkstyleMain checkstyleTest javadoc
>> spotbugsMain compileJava compileTestJava
>>
>> To save a time, run it only against a package where you did the
>> changes.
>>
>> —
>> Alexey
>>
>>

 --
 Neo4j Chief Solutions Architect
 *✉   *matt.cast...@neo4j.com





-- 
Neo4j Chief Solutions Architect
*✉   *matt.cast...@neo4j.com