Completeness of Beam Java Dependency Check Report

[Tomo Suzuki]

Hi Beam developers,
(I'm thinking to contribute to upgrades of Java dependencies of Beam; I
just read https://beam.apache.org/contribute/dependencies/)

As per the weekly report, Apache Beam Java SDK only has 8 outdated
dependencies based on the criteria. However, it seems many others are not
up-to-date. For example Guava 20.0 used in Beam

is not the latest major release.

Why do some outdated dependencies not appear in this report?

Regards,
Tomo

On Mon, Nov 11, 2019 at 7:05 AM Apache Jenkins Server <
jenk...@builds.apache.org> wrote:

> High Priority Dependency Updates Of Beam Python SDK:
> *Dependency Name* *Current Version* *Latest Version* *Release Date Of the
> Current Used Version* *Release Date Of The Latest Release* *JIRA Issue*
> mock  2.0.0 3.0.5 2019-05-20 2019-05-20
> BEAM-7369 
> oauth2client  3.0.0 4.1.3
> 2018-12-10 2018-12-10 BEAM-6089
> 
> Sphinx  1.8.5 2.2.1 2019-05-20 2019-10-28
> BEAM-7370  High Priority
> Dependency Updates Of Beam Java SDK:
> *Dependency Name* *Current Version* *Latest Version* *Release Date Of the
> Current Used Version* *Release Date Of The Latest Release* *JIRA Issue*
> com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin
> 
> 0.20.0 0.27.0 2019-02-11 2019-10-21 BEAM-6645
> 
> com.github.spotbugs:spotbugs
>  3.1.12
> 4.0.0-beta4 2019-03-01 2019-09-18 BEAM-7792
> 
> com.github.spotbugs:spotbugs-annotations
> 
> 3.1.12 4.0.0-beta4 2019-03-01 2019-09-18 BEAM-6951
> 
> javax.servlet:javax.servlet-api
>  3.1.0
> 4.0.1 2013-04-25 2018-04-20 BEAM-5750
> 
> org.conscrypt:conscrypt-openjdk
>  1.1.3
> 2.2.1 2018-06-04 2019-08-08 BEAM-5748
> 
> org.eclipse.jetty:jetty-server
> 
> 9.2.10.v20150310 10.0.0-alpha0 2015-03-10 2019-07-11 BEAM-5752
> 
> org.eclipse.jetty:jetty-servlet
> 
> 9.2.10.v20150310 10.0.0-alpha0 2015-03-10 2019-07-11 BEAM-5753
> 
> Gradle:  5.2.1 6.0 2019-08-19
> 2019-11-11 BEAM-8002  A
> dependency update is high priority if it satisfies one of following
> criteria:
>
>- It has major versions update available, e.g.
>org.assertj:assertj-core 2.5.0 -> 3.10.0;
>
>
>- It is over 3 minor versions behind the latest version, e.g.
>org.tukaani:xz 1.5 -> 1.8;
>
>
>- The current version is behind the later version for over 180 days,
>e.g. com.google.auto.service:auto-service 2014-10-24 -> 2017-12-11.
>
> In Beam, we make a best-effort attempt at keeping all dependencies
> up-to-date. In the future, issues will be filed and tracked for these
> automatically, but in the meantime you can search for existing issues or
> open a new one. For more information: Beam Dependency Guide
> 
>


-- 
Regards,
Tomo

Re: Completeness of Beam Java Dependency Check Report

[Chamikara Jayalath]

On Mon, Nov 11, 2019 at 10:14 AM Tomo Suzuki  wrote:

> Hi Beam developers,
> (I'm thinking to contribute to upgrades of Java dependencies of Beam; I
> just read https://beam.apache.org/contribute/dependencies/)
>

Thanks that will be great.


>
> As per the weekly report, Apache Beam Java SDK only has 8 outdated
> dependencies based on the criteria. However, it seems many others are not
> up-to-date. For example Guava 20.0 used in Beam
> 
> is not the latest major release.
>
> Why do some outdated dependencies not appear in this report?
>

This is probably since artifact version changed from xy.z to xy.z-jre. +Yifan
Zou  any idea ?


>
> Regards,
> Tomo
>
> On Mon, Nov 11, 2019 at 7:05 AM Apache Jenkins Server <
> jenk...@builds.apache.org> wrote:
>
>> High Priority Dependency Updates Of Beam Python SDK:
>> *Dependency Name* *Current Version* *Latest Version* *Release Date Of
>> the Current Used Version* *Release Date Of The Latest Release* *JIRA
>> Issue*
>> mock  2.0.0 3.0.5 2019-05-20 2019-05-20
>> BEAM-7369 
>> oauth2client  3.0.0 4.1.3
>> 2018-12-10 2018-12-10 BEAM-6089
>> 
>> Sphinx  1.8.5 2.2.1 2019-05-20
>> 2019-10-28 BEAM-7370  High
>> Priority Dependency Updates Of Beam Java SDK:
>> *Dependency Name* *Current Version* *Latest Version* *Release Date Of
>> the Current Used Version* *Release Date Of The Latest Release* *JIRA
>> Issue*
>> com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin
>> 
>> 0.20.0 0.27.0 2019-02-11 2019-10-21 BEAM-6645
>> 
>> com.github.spotbugs:spotbugs
>>  3.1.12
>> 4.0.0-beta4 2019-03-01 2019-09-18 BEAM-7792
>> 
>> com.github.spotbugs:spotbugs-annotations
>> 
>> 3.1.12 4.0.0-beta4 2019-03-01 2019-09-18 BEAM-6951
>> 
>> javax.servlet:javax.servlet-api
>> 
>> 3.1.0 4.0.1 2013-04-25 2018-04-20 BEAM-5750
>> 
>> org.conscrypt:conscrypt-openjdk
>> 
>> 1.1.3 2.2.1 2018-06-04 2019-08-08 BEAM-5748
>> 
>> org.eclipse.jetty:jetty-server
>> 
>> 9.2.10.v20150310 10.0.0-alpha0 2015-03-10 2019-07-11 BEAM-5752
>> 
>> org.eclipse.jetty:jetty-servlet
>> 
>> 9.2.10.v20150310 10.0.0-alpha0 2015-03-10 2019-07-11 BEAM-5753
>> 
>> Gradle:  5.2.1 6.0 2019-08-19
>> 2019-11-11 BEAM-8002  A
>> dependency update is high priority if it satisfies one of following
>> criteria:
>>
>>- It has major versions update available, e.g.
>>org.assertj:assertj-core 2.5.0 -> 3.10.0;
>>
>>
>>- It is over 3 minor versions behind the latest version, e.g.
>>org.tukaani:xz 1.5 -> 1.8;
>>
>>
>>- The current version is behind the later version for over 180 days,
>>e.g. com.google.auto.service:auto-service 2014-10-24 -> 2017-12-11.
>>
>> In Beam, we make a best-effort attempt at keeping all dependencies
>> up-to-date. In the future, issues will be filed and tracked for these
>> automatically, but in the meantime you can search for existing issues or
>> open a new one. For more information: Beam Dependency Guide
>> 
>>
>
>
> --
> Regards,
> Tomo
>

Re: Completeness of Beam Java Dependency Check Report

[Yifan Zou]

Hi,

Thanks for taking care of Beam dependencies. The guava was tracked in
BEAM-5559 . It was
filtered out by the tool because of the target version is x.y-jre.

On the other hand, I checked the logs of dependency job and found that the
high priority dependencies are way less than normal. I think the
gradle-versions-plugin sometimes couldn't get the versions which are
defined in BeamModulePlugin.groovy. Need more investigations on the
dependency management tool.

Yifan

On Mon, Nov 11, 2019 at 10:14 AM Tomo Suzuki  wrote:

> Hi Beam developers,
> (I'm thinking to contribute to upgrades of Java dependencies of Beam; I
> just read https://beam.apache.org/contribute/dependencies/)
>
> As per the weekly report, Apache Beam Java SDK only has 8 outdated
> dependencies based on the criteria. However, it seems many others are not
> up-to-date. For example Guava 20.0 used in Beam
> 
> is not the latest major release.
>
> Why do some outdated dependencies not appear in this report?
>
> Regards,
> Tomo
>
> On Mon, Nov 11, 2019 at 7:05 AM Apache Jenkins Server <
> jenk...@builds.apache.org> wrote:
>
>> High Priority Dependency Updates Of Beam Python SDK:
>> *Dependency Name* *Current Version* *Latest Version* *Release Date Of
>> the Current Used Version* *Release Date Of The Latest Release* *JIRA
>> Issue*
>> mock  2.0.0 3.0.5 2019-05-20 2019-05-20
>> BEAM-7369 
>> oauth2client  3.0.0 4.1.3
>> 2018-12-10 2018-12-10 BEAM-6089
>> 
>> Sphinx  1.8.5 2.2.1 2019-05-20
>> 2019-10-28 BEAM-7370  High
>> Priority Dependency Updates Of Beam Java SDK:
>> *Dependency Name* *Current Version* *Latest Version* *Release Date Of
>> the Current Used Version* *Release Date Of The Latest Release* *JIRA
>> Issue*
>> com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin
>> 
>> 0.20.0 0.27.0 2019-02-11 2019-10-21 BEAM-6645
>> 
>> com.github.spotbugs:spotbugs
>>  3.1.12
>> 4.0.0-beta4 2019-03-01 2019-09-18 BEAM-7792
>> 
>> com.github.spotbugs:spotbugs-annotations
>> 
>> 3.1.12 4.0.0-beta4 2019-03-01 2019-09-18 BEAM-6951
>> 
>> javax.servlet:javax.servlet-api
>> 
>> 3.1.0 4.0.1 2013-04-25 2018-04-20 BEAM-5750
>> 
>> org.conscrypt:conscrypt-openjdk
>> 
>> 1.1.3 2.2.1 2018-06-04 2019-08-08 BEAM-5748
>> 
>> org.eclipse.jetty:jetty-server
>> 
>> 9.2.10.v20150310 10.0.0-alpha0 2015-03-10 2019-07-11 BEAM-5752
>> 
>> org.eclipse.jetty:jetty-servlet
>> 
>> 9.2.10.v20150310 10.0.0-alpha0 2015-03-10 2019-07-11 BEAM-5753
>> 
>> Gradle:  5.2.1 6.0 2019-08-19
>> 2019-11-11 BEAM-8002  A
>> dependency update is high priority if it satisfies one of following
>> criteria:
>>
>>- It has major versions update available, e.g.
>>org.assertj:assertj-core 2.5.0 -> 3.10.0;
>>
>>
>>- It is over 3 minor versions behind the latest version, e.g.
>>org.tukaani:xz 1.5 -> 1.8;
>>
>>
>>- The current version is behind the later version for over 180 days,
>>e.g. com.google.auto.service:auto-service 2014-10-24 -> 2017-12-11.
>>
>> In Beam, we make a best-effort attempt at keeping all dependencies
>> up-to-date. In the future, issues will be filed and tracked for these
>> automatically, but in the meantime you can search for existing issues or
>> open a new one. For more information: Beam Dependency Guide
>> 
>>
>
>
> --
> Regards,
> Tomo
>

Re: Completeness of Beam Java Dependency Check Report

[Tomo Suzuki]

Chamikara and Yifan,
Thank you for the responses! Looking forward to hearing the investigation
result.
In the meantime, I'll explore .test-infra/jenkins/dependency_check
directory.

Re: Completeness of Beam Java Dependency Check Report

[Kenneth Knowles]

BeamModulePlugin just contains lists of versions to ease coordination
across Beam modules, but mostly does not create dependencies. Most of
Beam's modules only depend on a few things there. For example Guava is not
a core dependency, but here is where it is actually depended upon:

$ find . -name build.gradle | xargs grep library.java.guava
./sdks/java/core/build.gradle:  shadowTest library.java.guava_testlib
./sdks/java/extensions/sql/jdbc/build.gradle:  compile library.java.guava
./sdks/java/io/google-cloud-platform/build.gradle:  compile
library.java.guava
./sdks/java/io/kinesis/build.gradle:  testCompile library.java.guava_testlib

These results appear to be misleading. Grepping for 'import
com.google.common', I see this as the actual state of things:

 - GCP connector does not appear to actually depend on Guava in compile
scope
 - The Beam SQL JDBC driver does not appear to actually depend on Guava in
compile scope
 - The Dataflow Java worker does depend on Guava at compile scope but has
incorrect dependencies (and it probably shouldn't)
 - KinesisIO does depend on Guava at compile scope but has incorrect
dependencies (Kinesis libs have Guava on API surface so it is OK here, but
should be correctly declared)
 - ZetaSQL translator does depend on Guava at compile scope but has
incorrect dependencies (ZetaSQL has it on API surface so it is OK here, but
should be correctly declared)

We used to have an analysis that prevented this class of error.

Once the errors are fixed, the guava_version is simply a version that we
have discovered that seems to work for both Kinesis and ZetaSQL, libraries
we do not control. Kinesis producer is built against 18.0. Kinesis client
against 26.0-jre. ZetaSQL against 26.0-android.

(or maybe I messed up in my analysis)

Kenn

On Mon, Nov 11, 2019 at 12:07 PM Tomo Suzuki  wrote:

>
> Chamikara and Yifan,
> Thank you for the responses! Looking forward to hearing the investigation
> result.
> In the meantime, I'll explore .test-infra/jenkins/dependency_check
> directory.
>
>

Re: Completeness of Beam Java Dependency Check Report

[Tomo Suzuki]

Kenn,

Thank you for the analysis. Although Guava was randomly picked up, it's
great learning for me to learn how you analyzed other modules using Guava.

On Mon, Nov 11, 2019 at 4:29 PM Kenneth Knowles  wrote:

> BeamModulePlugin just contains lists of versions to ease coordination
> across Beam modules, but mostly does not create dependencies. Most of
> Beam's modules only depend on a few things there. For example Guava is not
> a core dependency, but here is where it is actually depended upon:
>
> $ find . -name build.gradle | xargs grep library.java.guava
> ./sdks/java/core/build.gradle:  shadowTest library.java.guava_testlib
> ./sdks/java/extensions/sql/jdbc/build.gradle:  compile library.java.guava
> ./sdks/java/io/google-cloud-platform/build.gradle:  compile
> library.java.guava
> ./sdks/java/io/kinesis/build.gradle:  testCompile
> library.java.guava_testlib
>
> These results appear to be misleading. Grepping for 'import
> com.google.common', I see this as the actual state of things:
>
>  - GCP connector does not appear to actually depend on Guava in compile
> scope
>  - The Beam SQL JDBC driver does not appear to actually depend on Guava in
> compile scope
>  - The Dataflow Java worker does depend on Guava at compile scope but has
> incorrect dependencies (and it probably shouldn't)
>  - KinesisIO does depend on Guava at compile scope but has incorrect
> dependencies (Kinesis libs have Guava on API surface so it is OK here, but
> should be correctly declared)
>  - ZetaSQL translator does depend on Guava at compile scope but has
> incorrect dependencies (ZetaSQL has it on API surface so it is OK here, but
> should be correctly declared)
>
> We used to have an analysis that prevented this class of error.
>
> Once the errors are fixed, the guava_version is simply a version that we
> have discovered that seems to work for both Kinesis and ZetaSQL, libraries
> we do not control. Kinesis producer is built against 18.0. Kinesis client
> against 26.0-jre. ZetaSQL against 26.0-android.
>
> (or maybe I messed up in my analysis)
>
> Kenn
>
> On Mon, Nov 11, 2019 at 12:07 PM Tomo Suzuki  wrote:
>
>>
>> Chamikara and Yifan,
>> Thank you for the responses! Looking forward to hearing the investigation
>> result.
>> In the meantime, I'll explore .test-infra/jenkins/dependency_check
>> directory.
>>
>>

-- 
Regards,
Tomo

Re: Completeness of Beam Java Dependency Check Report

[Tomo Suzuki]

Yifan,
I created a ticket to track this finding:
https://issues.apache.org/jira/browse/BEAM-8621 .


On Mon, Nov 11, 2019 at 5:08 PM Tomo Suzuki  wrote:

> Kenn,
>
> Thank you for the analysis. Although Guava was randomly picked up, it's
> great learning for me to learn how you analyzed other modules using Guava.
>
> On Mon, Nov 11, 2019 at 4:29 PM Kenneth Knowles  wrote:
>
>> BeamModulePlugin just contains lists of versions to ease coordination
>> across Beam modules, but mostly does not create dependencies. Most of
>> Beam's modules only depend on a few things there. For example Guava is not
>> a core dependency, but here is where it is actually depended upon:
>>
>> $ find . -name build.gradle | xargs grep library.java.guava
>> ./sdks/java/core/build.gradle:  shadowTest library.java.guava_testlib
>> ./sdks/java/extensions/sql/jdbc/build.gradle:  compile library.java.guava
>> ./sdks/java/io/google-cloud-platform/build.gradle:  compile
>> library.java.guava
>> ./sdks/java/io/kinesis/build.gradle:  testCompile
>> library.java.guava_testlib
>>
>> These results appear to be misleading. Grepping for 'import
>> com.google.common', I see this as the actual state of things:
>>
>>  - GCP connector does not appear to actually depend on Guava in compile
>> scope
>>  - The Beam SQL JDBC driver does not appear to actually depend on Guava
>> in compile scope
>>  - The Dataflow Java worker does depend on Guava at compile scope but has
>> incorrect dependencies (and it probably shouldn't)
>>  - KinesisIO does depend on Guava at compile scope but has incorrect
>> dependencies (Kinesis libs have Guava on API surface so it is OK here, but
>> should be correctly declared)
>>  - ZetaSQL translator does depend on Guava at compile scope but has
>> incorrect dependencies (ZetaSQL has it on API surface so it is OK here, but
>> should be correctly declared)
>>
>> We used to have an analysis that prevented this class of error.
>>
>> Once the errors are fixed, the guava_version is simply a version that we
>> have discovered that seems to work for both Kinesis and ZetaSQL, libraries
>> we do not control. Kinesis producer is built against 18.0. Kinesis client
>> against 26.0-jre. ZetaSQL against 26.0-android.
>>
>> (or maybe I messed up in my analysis)
>>
>> Kenn
>>
>> On Mon, Nov 11, 2019 at 12:07 PM Tomo Suzuki  wrote:
>>
>>>
>>> Chamikara and Yifan,
>>> Thank you for the responses! Looking forward to hearing the
>>> investigation result.
>>> In the meantime, I'll explore .test-infra/jenkins/dependency_check
>>> directory.
>>>
>>>
>
> --
> Regards,
> Tomo
>


-- 
Regards,
Tomo

Re: Completeness of Beam Java Dependency Check Report

[Yifan Zou]

Thanks Tomo. I'll follow up in JIRA.

On Tue, Nov 12, 2019 at 9:44 AM Tomo Suzuki  wrote:

> Yifan,
> I created a ticket to track this finding:
> https://issues.apache.org/jira/browse/BEAM-8621 .
>
>
> On Mon, Nov 11, 2019 at 5:08 PM Tomo Suzuki  wrote:
>
>> Kenn,
>>
>> Thank you for the analysis. Although Guava was randomly picked up, it's
>> great learning for me to learn how you analyzed other modules using Guava.
>>
>> On Mon, Nov 11, 2019 at 4:29 PM Kenneth Knowles  wrote:
>>
>>> BeamModulePlugin just contains lists of versions to ease coordination
>>> across Beam modules, but mostly does not create dependencies. Most of
>>> Beam's modules only depend on a few things there. For example Guava is not
>>> a core dependency, but here is where it is actually depended upon:
>>>
>>> $ find . -name build.gradle | xargs grep library.java.guava
>>> ./sdks/java/core/build.gradle:  shadowTest library.java.guava_testlib
>>> ./sdks/java/extensions/sql/jdbc/build.gradle:  compile library.java.guava
>>> ./sdks/java/io/google-cloud-platform/build.gradle:  compile
>>> library.java.guava
>>> ./sdks/java/io/kinesis/build.gradle:  testCompile
>>> library.java.guava_testlib
>>>
>>> These results appear to be misleading. Grepping for 'import
>>> com.google.common', I see this as the actual state of things:
>>>
>>>  - GCP connector does not appear to actually depend on Guava in compile
>>> scope
>>>  - The Beam SQL JDBC driver does not appear to actually depend on Guava
>>> in compile scope
>>>  - The Dataflow Java worker does depend on Guava at compile scope but
>>> has incorrect dependencies (and it probably shouldn't)
>>>  - KinesisIO does depend on Guava at compile scope but has incorrect
>>> dependencies (Kinesis libs have Guava on API surface so it is OK here, but
>>> should be correctly declared)
>>>  - ZetaSQL translator does depend on Guava at compile scope but has
>>> incorrect dependencies (ZetaSQL has it on API surface so it is OK here, but
>>> should be correctly declared)
>>>
>>> We used to have an analysis that prevented this class of error.
>>>
>>> Once the errors are fixed, the guava_version is simply a version that we
>>> have discovered that seems to work for both Kinesis and ZetaSQL, libraries
>>> we do not control. Kinesis producer is built against 18.0. Kinesis client
>>> against 26.0-jre. ZetaSQL against 26.0-android.
>>>
>>> (or maybe I messed up in my analysis)
>>>
>>> Kenn
>>>
>>> On Mon, Nov 11, 2019 at 12:07 PM Tomo Suzuki  wrote:
>>>

 Chamikara and Yifan,
 Thank you for the responses! Looking forward to hearing the
 investigation result.
 In the meantime, I'll explore .test-infra/jenkins/dependency_check
 directory.


>>
>> --
>> Regards,
>> Tomo
>>
>
>
> --
> Regards,
> Tomo
>

Re: Completeness of Beam Java Dependency Check Report

[Yifan Zou]

The dependency management tool is back. See the latest report

.

On Tue, Nov 12, 2019 at 9:51 AM Yifan Zou  wrote:

> Thanks Tomo. I'll follow up in JIRA.
>
> On Tue, Nov 12, 2019 at 9:44 AM Tomo Suzuki  wrote:
>
>> Yifan,
>> I created a ticket to track this finding:
>> https://issues.apache.org/jira/browse/BEAM-8621 .
>>
>>
>> On Mon, Nov 11, 2019 at 5:08 PM Tomo Suzuki  wrote:
>>
>>> Kenn,
>>>
>>> Thank you for the analysis. Although Guava was randomly picked up, it's
>>> great learning for me to learn how you analyzed other modules using Guava.
>>>
>>> On Mon, Nov 11, 2019 at 4:29 PM Kenneth Knowles  wrote:
>>>
 BeamModulePlugin just contains lists of versions to ease coordination
 across Beam modules, but mostly does not create dependencies. Most of
 Beam's modules only depend on a few things there. For example Guava is not
 a core dependency, but here is where it is actually depended upon:

 $ find . -name build.gradle | xargs grep library.java.guava
 ./sdks/java/core/build.gradle:  shadowTest library.java.guava_testlib
 ./sdks/java/extensions/sql/jdbc/build.gradle:  compile
 library.java.guava
 ./sdks/java/io/google-cloud-platform/build.gradle:  compile
 library.java.guava
 ./sdks/java/io/kinesis/build.gradle:  testCompile
 library.java.guava_testlib

 These results appear to be misleading. Grepping for 'import
 com.google.common', I see this as the actual state of things:

  - GCP connector does not appear to actually depend on Guava in compile
 scope
  - The Beam SQL JDBC driver does not appear to actually depend on Guava
 in compile scope
  - The Dataflow Java worker does depend on Guava at compile scope but
 has incorrect dependencies (and it probably shouldn't)
  - KinesisIO does depend on Guava at compile scope but has incorrect
 dependencies (Kinesis libs have Guava on API surface so it is OK here, but
 should be correctly declared)
  - ZetaSQL translator does depend on Guava at compile scope but has
 incorrect dependencies (ZetaSQL has it on API surface so it is OK here, but
 should be correctly declared)

 We used to have an analysis that prevented this class of error.

 Once the errors are fixed, the guava_version is simply a version that
 we have discovered that seems to work for both Kinesis and ZetaSQL,
 libraries we do not control. Kinesis producer is built against 18.0.
 Kinesis client against 26.0-jre. ZetaSQL against 26.0-android.

 (or maybe I messed up in my analysis)

 Kenn

 On Mon, Nov 11, 2019 at 12:07 PM Tomo Suzuki 
 wrote:

>
> Chamikara and Yifan,
> Thank you for the responses! Looking forward to hearing the
> investigation result.
> In the meantime, I'll explore .test-infra/jenkins/dependency_check
> directory.
>
>
>>>
>>> --
>>> Regards,
>>> Tomo
>>>
>>
>>
>> --
>> Regards,
>> Tomo
>>
>

Re: Completeness of Beam Java Dependency Check Report

[Alexey Romanenko]

Good example about Guava deps, let me go a bit deeper.

> $ find . -name build.gradle | xargs grep library.java.guava
> ./sdks/java/core/build.gradle:  shadowTest library.java.guava_testlib
> ./sdks/java/io/kinesis/build.gradle:  testCompile library.java.guava_testlib


Regarding using non-vendored Guava in KinesisIO (and "java/core" as well), it’s 
all about “library.java.guava_testlib” and 
“com.google.common.testing.EqualsTester” only in particular, which is used for 
tests.
Do we need to vendor “com.google.guava:guava-testlib” for this in this case?

> - KinesisIO does depend on Guava at compile scope but has incorrect 
> dependencies (Kinesis libs have Guava on API surface so it is OK here, but 
> should be correctly declared)


Sorry, but I didn’t understand what do you mean by “but should be correctly 
declared”.
Since Kinesis client libs have own Guava deps and we shade our own guava, so it 
should be fine, no? 

> On 11 Nov 2019, at 22:29, Kenneth Knowles  wrote:
> 
> BeamModulePlugin just contains lists of versions to ease coordination across 
> Beam modules, but mostly does not create dependencies. Most of Beam's modules 
> only depend on a few things there. For example Guava is not a core 
> dependency, but here is where it is actually depended upon:
> 
> $ find . -name build.gradle | xargs grep library.java.guava
> ./sdks/java/core/build.gradle:  shadowTest library.java.guava_testlib
> ./sdks/java/extensions/sql/jdbc/build.gradle:  compile library.java.guava
> ./sdks/java/io/google-cloud-platform/build.gradle:  compile library.java.guava
> ./sdks/java/io/kinesis/build.gradle:  testCompile library.java.guava_testlib
> 
> These results appear to be misleading. Grepping for 'import 
> com.google.common', I see this as the actual state of things:
> 
>  - GCP connector does not appear to actually depend on Guava in compile scope
>  - The Beam SQL JDBC driver does not appear to actually depend on Guava in 
> compile scope
>  - The Dataflow Java worker does depend on Guava at compile scope but has 
> incorrect dependencies (and it probably shouldn't)
>  - KinesisIO does depend on Guava at compile scope but has incorrect 
> dependencies (Kinesis libs have Guava on API surface so it is OK here, but 
> should be correctly declared)
>  - ZetaSQL translator does depend on Guava at compile scope but has incorrect 
> dependencies (ZetaSQL has it on API surface so it is OK here, but should be 
> correctly declared)
> 
> We used to have an analysis that prevented this class of error.
> 
> Once the errors are fixed, the guava_version is simply a version that we have 
> discovered that seems to work for both Kinesis and ZetaSQL, libraries we do 
> not control. Kinesis producer is built against 18.0. Kinesis client against 
> 26.0-jre. ZetaSQL against 26.0-android.
> 
> (or maybe I messed up in my analysis)
> 
> Kenn
> 
> On Mon, Nov 11, 2019 at 12:07 PM Tomo Suzuki  > wrote:
> 
> Chamikara and Yifan,
> Thank you for the responses! Looking forward to hearing the investigation 
> result.
> In the meantime, I'll explore .test-infra/jenkins/dependency_check directory.
> 

Re: Completeness of Beam Java Dependency Check Report

[Kenneth Knowles]

On Thu, Nov 14, 2019 at 8:04 AM Alexey Romanenko 
wrote:

> Good example about Guava deps, let me go a bit deeper.
>
> $ find . -name build.gradle | xargs grep library.java.guava
> ./sdks/java/core/build.gradle:  shadowTest library.java.guava_testlib
>
> ./sdks/java/io/kinesis/build.gradle:  testCompile
> library.java.guava_testlib
>
>
> Regarding using non-vendored Guava in KinesisIO (and "java/core" as well),
> it’s all about *“library.java.guava_testlib” *and
> *“com.google.common.testing.EqualsTester”* only in particular, which is
> used for tests.
> Do we need to vendor “*com.google.guava:guava-testlib*” for this in this
> case?
>

I didn't worry about test scopes because they don't ship to users so much.
It could be useful if there is a runner with a conflict and they want to
run the integration tests.


> - KinesisIO does depend on Guava at compile scope but has incorrect
> dependencies (Kinesis libs have Guava on API surface so it is OK here, but
> should be correctly declared)
>
>
> Sorry, but I didn’t understand what do you mean by “*but should be
> correctly declared*”.
> Since Kinesis client libs have own Guava deps and we shade our own guava,
> so it should be fine, no?
>

I mean that any module with an "import X" should have a dependency on X in
its build.gradle. When you leave it off, the dep analysis (such as Maven's)
calls it out as "used undeclared dependency".

Kenn


>
> On 11 Nov 2019, at 22:29, Kenneth Knowles  wrote:
>
> BeamModulePlugin just contains lists of versions to ease coordination
> across Beam modules, but mostly does not create dependencies. Most of
> Beam's modules only depend on a few things there. For example Guava is not
> a core dependency, but here is where it is actually depended upon:
>
> $ find . -name build.gradle | xargs grep library.java.guava
> ./sdks/java/core/build.gradle:  shadowTest library.java.guava_testlib
> ./sdks/java/extensions/sql/jdbc/build.gradle:  compile library.java.guava
> ./sdks/java/io/google-cloud-platform/build.gradle:  compile
> library.java.guava
> ./sdks/java/io/kinesis/build.gradle:  testCompile
> library.java.guava_testlib
>
> These results appear to be misleading. Grepping for 'import
> com.google.common', I see this as the actual state of things:
>
>  - GCP connector does not appear to actually depend on Guava in compile
> scope
>  - The Beam SQL JDBC driver does not appear to actually depend on Guava in
> compile scope
>  - The Dataflow Java worker does depend on Guava at compile scope but has
> incorrect dependencies (and it probably shouldn't)
>  - KinesisIO does depend on Guava at compile scope but has incorrect
> dependencies (Kinesis libs have Guava on API surface so it is OK here, but
> should be correctly declared)
>  - ZetaSQL translator does depend on Guava at compile scope but has
> incorrect dependencies (ZetaSQL has it on API surface so it is OK here, but
> should be correctly declared)
>
> We used to have an analysis that prevented this class of error.
>
> Once the errors are fixed, the guava_version is simply a version that we
> have discovered that seems to work for both Kinesis and ZetaSQL, libraries
> we do not control. Kinesis producer is built against 18.0. Kinesis client
> against 26.0-jre. ZetaSQL against 26.0-android.
>
> (or maybe I messed up in my analysis)
>
> Kenn
>
> On Mon, Nov 11, 2019 at 12:07 PM Tomo Suzuki  wrote:
>
>>
>> Chamikara and Yifan,
>> Thank you for the responses! Looking forward to hearing the investigation
>> result.
>> In the meantime, I'll explore .test-infra/jenkins/dependency_check
>> directory.
>>
>>
>

Re: Completeness of Beam Java Dependency Check Report

[Tomo Suzuki]

Hi Yifan,

I found resolutionStrategy is interfering gradle-versions-plugin (detail in
BEAM-8654). Would you check this PR?
https://github.com/apache/beam/pull/10127

On Thu, Nov 14, 2019 at 1:42 PM Kenneth Knowles  wrote:

> On Thu, Nov 14, 2019 at 8:04 AM Alexey Romanenko 
> wrote:
>
>> Good example about Guava deps, let me go a bit deeper.
>>
>> $ find . -name build.gradle | xargs grep library.java.guava
>> ./sdks/java/core/build.gradle:  shadowTest library.java.guava_testlib
>>
>> ./sdks/java/io/kinesis/build.gradle:  testCompile
>> library.java.guava_testlib
>>
>>
>> Regarding using non-vendored Guava in KinesisIO (and "java/core" as
>> well), it’s all about *“library.java.guava_testlib” *and
>> *“com.google.common.testing.EqualsTester”* only in particular, which is
>> used for tests.
>> Do we need to vendor “*com.google.guava:guava-testlib*” for this in this
>> case?
>>
>
> I didn't worry about test scopes because they don't ship to users so much.
> It could be useful if there is a runner with a conflict and they want to
> run the integration tests.
>
>
>> - KinesisIO does depend on Guava at compile scope but has incorrect
>> dependencies (Kinesis libs have Guava on API surface so it is OK here, but
>> should be correctly declared)
>>
>>
>> Sorry, but I didn’t understand what do you mean by “*but should be
>> correctly declared*”.
>> Since Kinesis client libs have own Guava deps and we shade our own guava,
>> so it should be fine, no?
>>
>
> I mean that any module with an "import X" should have a dependency on X in
> its build.gradle. When you leave it off, the dep analysis (such as Maven's)
> calls it out as "used undeclared dependency".
>
> Kenn
>
>
>>
>> On 11 Nov 2019, at 22:29, Kenneth Knowles  wrote:
>>
>> BeamModulePlugin just contains lists of versions to ease coordination
>> across Beam modules, but mostly does not create dependencies. Most of
>> Beam's modules only depend on a few things there. For example Guava is not
>> a core dependency, but here is where it is actually depended upon:
>>
>> $ find . -name build.gradle | xargs grep library.java.guava
>> ./sdks/java/core/build.gradle:  shadowTest library.java.guava_testlib
>> ./sdks/java/extensions/sql/jdbc/build.gradle:  compile library.java.guava
>> ./sdks/java/io/google-cloud-platform/build.gradle:  compile
>> library.java.guava
>> ./sdks/java/io/kinesis/build.gradle:  testCompile
>> library.java.guava_testlib
>>
>> These results appear to be misleading. Grepping for 'import
>> com.google.common', I see this as the actual state of things:
>>
>>  - GCP connector does not appear to actually depend on Guava in compile
>> scope
>>  - The Beam SQL JDBC driver does not appear to actually depend on Guava
>> in compile scope
>>  - The Dataflow Java worker does depend on Guava at compile scope but has
>> incorrect dependencies (and it probably shouldn't)
>>  - KinesisIO does depend on Guava at compile scope but has incorrect
>> dependencies (Kinesis libs have Guava on API surface so it is OK here, but
>> should be correctly declared)
>>  - ZetaSQL translator does depend on Guava at compile scope but has
>> incorrect dependencies (ZetaSQL has it on API surface so it is OK here, but
>> should be correctly declared)
>>
>> We used to have an analysis that prevented this class of error.
>>
>> Once the errors are fixed, the guava_version is simply a version that we
>> have discovered that seems to work for both Kinesis and ZetaSQL, libraries
>> we do not control. Kinesis producer is built against 18.0. Kinesis client
>> against 26.0-jre. ZetaSQL against 26.0-android.
>>
>> (or maybe I messed up in my analysis)
>>
>> Kenn
>>
>> On Mon, Nov 11, 2019 at 12:07 PM Tomo Suzuki  wrote:
>>
>>>
>>> Chamikara and Yifan,
>>> Thank you for the responses! Looking forward to hearing the
>>> investigation result.
>>> In the meantime, I'll explore .test-infra/jenkins/dependency_check
>>> directory.
>>>
>>>
>>

-- 
Regards,
Tomo