[jira] [Created] (BEAM-4227) Hidden annotation does not hide methods from help text
Paul Gerver created BEAM-4227:
-
Summary: Hidden annotation does not hide methods from help text
Key: BEAM-4227
URL: https://issues.apache.org/jira/browse/BEAM-4227
Project: Beam
Issue Type: Bug
Components: sdk-java-core
Affects Versions: 2.4.0
Reporter: Paul Gerver
Assignee: Kenneth Knowles
The javadoc for *@Hidden* mentions that it hides methods and/or interfaces, but
the filtering of *@Hidden* for *--help* only works for interfaces while methods
with the annotation are still visible.
For example, in *PipelineOptions* the optionsId getter is set with the *Hidden*
annotation, but when *–help* is executed, it appears in the list:
{noformat}
org.apache.beam.sdk.options.PipelineOptions:
--jobName=
Default: JobNameFactory
Name of the pipeline execution.It must match the regular expression
'[a-z]([-a-z0-9]{0,38}[a-z0-9])?'.It defaults to
ApplicationName-UserName-Date-RandomInteger
--optionsId=
Default: AtomicLongFactory
...
--userAgent=
Default: UserAgentFactory
...{noformat}
A proper fix should be skip methods marked with the Hidden class when
performing print help [1]. Additionally, a test should be added to
*PipelineOptionsReflectorTest* [2] or factory test.
[1]
[https://github.com/apache/beam/blob/19bdad654550958f5749ebfcedc3981240127b05/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java#L581]
[2]
[https://github.com/apache/beam/blob/48f9cce76cf87ffe7073c3ad9c34a9f2f6de4b76/sdks/java/core/src/test/java/org/apache/beam/sdk/options/PipelineOptionsReflectorTest.java#L120]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-3986) Improve setAwsCredentialsProvider documentation
[
https://issues.apache.org/jira/browse/BEAM-3986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16452426#comment-16452426
]
Paul Gerver commented on BEAM-3986:
---
Yes!
> Improve setAwsCredentialsProvider documentation
> ---
>
> Key: BEAM-3986
> URL: https://issues.apache.org/jira/browse/BEAM-3986
> Project: Beam
> Issue Type: Improvement
> Components: io-java-aws
>Affects Versions: 2.4.0
>Reporter: Paul Gerver
>Assignee: Ismaël Mejía
>Priority: Trivial
>
> I tried using --awsCredentialsProvider with a JSON string (i.e.,
> --awsCredentialsProvider='\{"awsAccessKeyId" : "", "awsSecretKey" :
> ""}') and it was not clear from documentation that "@type" and an AWS
> credentials class was required.
> We should improve the javadoc and PipelineOptions to include the following
> description/example:
> @Description("The credential instance that should be used to authenticate
> against AWS services. " + "The option value must contain \"@type\" field and
> an AWS Credentials Provider class name as a value. " + "Refer to
> DefaultAWSCredentialsProviderChain Javadoc for usage help. " + "For example,
> to specify the AWS key ID and secret on the command line, specify the
> following: \{\"@type\": \"AWSStaticCredentialsProvider\",
> \"awsAccessKeyId\":\"\", \"awsSecretKey\":\"\"}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (BEAM-3986) Improve setAwsCredentialsProvider documentation
[
https://issues.apache.org/jira/browse/BEAM-3986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paul Gerver reassigned BEAM-3986:
-
Assignee: Paul Gerver (was: Ismaël Mejía)
> Improve setAwsCredentialsProvider documentation
> ---
>
> Key: BEAM-3986
> URL: https://issues.apache.org/jira/browse/BEAM-3986
> Project: Beam
> Issue Type: Improvement
> Components: io-java-aws
>Affects Versions: 2.4.0
>Reporter: Paul Gerver
>Assignee: Paul Gerver
>Priority: Trivial
>
> I tried using --awsCredentialsProvider with a JSON string (i.e.,
> --awsCredentialsProvider='\{"awsAccessKeyId" : "", "awsSecretKey" :
> ""}') and it was not clear from documentation that "@type" and an AWS
> credentials class was required.
> We should improve the javadoc and PipelineOptions to include the following
> description/example:
> @Description("The credential instance that should be used to authenticate
> against AWS services. " + "The option value must contain \"@type\" field and
> an AWS Credentials Provider class name as a value. " + "Refer to
> DefaultAWSCredentialsProviderChain Javadoc for usage help. " + "For example,
> to specify the AWS key ID and secret on the command line, specify the
> following: \{\"@type\": \"AWSStaticCredentialsProvider\",
> \"awsAccessKeyId\":\"\", \"awsSecretKey\":\"\"}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BEAM-3986) Improve setAwsCredentialsProvider documentation
Paul Gerver created BEAM-3986:
-
Summary: Improve setAwsCredentialsProvider documentation
Key: BEAM-3986
URL: https://issues.apache.org/jira/browse/BEAM-3986
Project: Beam
Issue Type: Improvement
Components: io-java-aws
Affects Versions: 2.4.0
Reporter: Paul Gerver
Assignee: Ismaël Mejía
I tried using --awsCredentialsProvider with a JSON string (i.e.,
--awsCredentialsProvider='\{"awsAccessKeyId" : "", "awsSecretKey" :
""}') and it was not clear from documentation that "@type" and an AWS
credentials class was required.
We should improve the javadoc and PipelineOptions to include the following
description/example:
@Description("The credential instance that should be used to authenticate
against AWS services. " + "The option value must contain \"@type\" field and an
AWS Credentials Provider class name as a value. " + "Refer to
DefaultAWSCredentialsProviderChain Javadoc for usage help. " + "For example, to
specify the AWS key ID and secret on the command line, specify the following:
\{\"@type\": \"AWSStaticCredentialsProvider\", \"awsAccessKeyId\":\"\",
\"awsSecretKey\":\"\"}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (BEAM-3831) Update Google Cloud Core version (which depends on org.json)
[
https://issues.apache.org/jira/browse/BEAM-3831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paul Gerver updated BEAM-3831:
--
Issue Type: Wish (was: Bug)
> Update Google Cloud Core version (which depends on org.json)
>
>
> Key: BEAM-3831
> URL: https://issues.apache.org/jira/browse/BEAM-3831
> Project: Beam
> Issue Type: Wish
> Components: io-java-gcp
>Affects Versions: 2.3.0
>Reporter: Paul Gerver
>Assignee: Chamikara Jayalath
>Priority: Minor
>
> Looking at dependencies pulled in for my application and saw that org.json
> jar file was being used. Looked up the dependency tree and saw the following:
> {noformat}
> [INFO] | +-
> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.3.0:compile
> ...
> [INFO] | | +- com.google.cloud:google-cloud-core:jar:1.0.2:compile
> [INFO] | | | \- org.json:json:jar:20160810:compile{noformat}
> The Apache Foundation has noted that use of software with the json license
> should not be used: [https://www.apache.org/legal/resolved.html#json]
>
> The earliest version to exclude the dependency is v1.15.0:
> https://mvnrepository.com/artifact/com.google.cloud/google-cloud-core/1.15.0
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (BEAM-3831) Update Google Cloud Core version (which depends on org.json)
[
https://issues.apache.org/jira/browse/BEAM-3831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paul Gerver updated BEAM-3831:
--
Description:
Looking at dependencies pulled in for my application and saw that org.json jar
file was being used. Looked up the dependency tree and saw the following:
{noformat}
[INFO] | +-
org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.3.0:compile
...
[INFO] | | +- com.google.cloud:google-cloud-core:jar:1.0.2:compile
[INFO] | | | \- org.json:json:jar:20160810:compile{noformat}
The Apache Foundation has noted that use of software with the json license
should not be used: [https://www.apache.org/legal/resolved.html#json]
The earliest version to exclude the dependency is
[v1.15.0|[https://mvnrepository.com/artifact/com.google.cloud/google-cloud-core/1.15.0|https://mvnrepository.com/artifact/com.google.cloud/google-cloud-core/1.15.0)]]
was:
Looking at dependencies pulled in for my application and saw that org.json jar
file was being used. Looked up the dependency tree and saw the following:
{noformat}
[INFO] | +-
org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.3.0:compile
...
[INFO] | | +- com.google.cloud:google-cloud-core:jar:1.0.2:compile
[INFO] | | | \- org.json:json:jar:20160810:compile{noformat}
The Apache Foundation has noted that use of software with the json license
should not be used: [https://www.apache.org/legal/resolved.html#json]
The earliest version to exclude the dependency is v1.15.0
([https://mvnrepository.com/artifact/com.google.cloud/google-cloud-core/1.15.0)]
> Update Google Cloud Core version (which depends on org.json)
>
>
> Key: BEAM-3831
> URL: https://issues.apache.org/jira/browse/BEAM-3831
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp
>Affects Versions: 2.3.0
>Reporter: Paul Gerver
>Assignee: Chamikara Jayalath
>Priority: Minor
>
> Looking at dependencies pulled in for my application and saw that org.json
> jar file was being used. Looked up the dependency tree and saw the following:
> {noformat}
> [INFO] | +-
> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.3.0:compile
> ...
> [INFO] | | +- com.google.cloud:google-cloud-core:jar:1.0.2:compile
> [INFO] | | | \- org.json:json:jar:20160810:compile{noformat}
> The Apache Foundation has noted that use of software with the json license
> should not be used: [https://www.apache.org/legal/resolved.html#json]
>
> The earliest version to exclude the dependency is
> [v1.15.0|[https://mvnrepository.com/artifact/com.google.cloud/google-cloud-core/1.15.0|https://mvnrepository.com/artifact/com.google.cloud/google-cloud-core/1.15.0)]]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (BEAM-3831) Update Google Cloud Core version (which depends on org.json)
[
https://issues.apache.org/jira/browse/BEAM-3831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paul Gerver updated BEAM-3831:
--
Description:
Looking at dependencies pulled in for my application and saw that org.json jar
file was being used. Looked up the dependency tree and saw the following:
{noformat}
[INFO] | +-
org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.3.0:compile
...
[INFO] | | +- com.google.cloud:google-cloud-core:jar:1.0.2:compile
[INFO] | | | \- org.json:json:jar:20160810:compile{noformat}
The Apache Foundation has noted that use of software with the json license
should not be used: [https://www.apache.org/legal/resolved.html#json]
The earliest version to exclude the dependency is v1.15.0:
https://mvnrepository.com/artifact/com.google.cloud/google-cloud-core/1.15.0
was:
Looking at dependencies pulled in for my application and saw that org.json jar
file was being used. Looked up the dependency tree and saw the following:
{noformat}
[INFO] | +-
org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.3.0:compile
...
[INFO] | | +- com.google.cloud:google-cloud-core:jar:1.0.2:compile
[INFO] | | | \- org.json:json:jar:20160810:compile{noformat}
The Apache Foundation has noted that use of software with the json license
should not be used: [https://www.apache.org/legal/resolved.html#json]
The earliest version to exclude the dependency is
[v1.15.0|[https://mvnrepository.com/artifact/com.google.cloud/google-cloud-core/1.15.0|https://mvnrepository.com/artifact/com.google.cloud/google-cloud-core/1.15.0)]]
> Update Google Cloud Core version (which depends on org.json)
>
>
> Key: BEAM-3831
> URL: https://issues.apache.org/jira/browse/BEAM-3831
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp
>Affects Versions: 2.3.0
>Reporter: Paul Gerver
>Assignee: Chamikara Jayalath
>Priority: Minor
>
> Looking at dependencies pulled in for my application and saw that org.json
> jar file was being used. Looked up the dependency tree and saw the following:
> {noformat}
> [INFO] | +-
> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.3.0:compile
> ...
> [INFO] | | +- com.google.cloud:google-cloud-core:jar:1.0.2:compile
> [INFO] | | | \- org.json:json:jar:20160810:compile{noformat}
> The Apache Foundation has noted that use of software with the json license
> should not be used: [https://www.apache.org/legal/resolved.html#json]
>
> The earliest version to exclude the dependency is v1.15.0:
> https://mvnrepository.com/artifact/com.google.cloud/google-cloud-core/1.15.0
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BEAM-3831) Update Google Cloud Core version (which depends on org.json)
Paul Gerver created BEAM-3831:
-
Summary: Update Google Cloud Core version (which depends on
org.json)
Key: BEAM-3831
URL: https://issues.apache.org/jira/browse/BEAM-3831
Project: Beam
Issue Type: Bug
Components: io-java-gcp
Affects Versions: 2.3.0
Reporter: Paul Gerver
Assignee: Chamikara Jayalath
Looking at dependencies pulled in for my application and saw that org.json jar
file was being used. Looked up the dependency tree and saw the following:
{noformat}
[INFO] | +-
org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.3.0:compile
...
[INFO] | | +- com.google.cloud:google-cloud-core:jar:1.0.2:compile
[INFO] | | | \- org.json:json:jar:20160810:compile{noformat}
The Apache Foundation has noted that use of software with the json license
should not be used: [https://www.apache.org/legal/resolved.html#json]
The earliest version to exclude the dependency is v1.15.0
([https://mvnrepository.com/artifact/com.google.cloud/google-cloud-core/1.15.0)]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (BEAM-3525) TestPipeline serializes PipelineOptions prematurely
[ https://issues.apache.org/jira/browse/BEAM-3525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Gerver closed BEAM-3525. - Resolution: Workaround Fix Version/s: Not applicable PipelineOptions need to be serialized for non-Java SDKs before they get to the runner. As such, options that need to be present in the runner should not be marked with @JsonIgnore. Solution: Remove JsonIgnore annotation from runner configuration options. > TestPipeline serializes PipelineOptions prematurely > --- > > Key: BEAM-3525 > URL: https://issues.apache.org/jira/browse/BEAM-3525 > Project: Beam > Issue Type: Bug > Components: testing >Affects Versions: 2.2.0 >Reporter: Paul Gerver >Assignee: Paul Gerver >Priority: Minor > Fix For: Not applicable > > > The TestPipeline in its run() method now serializes PipelineOptions and adds > in TestValueProvider values before submitting the pipeline to run [1]. > This premature serialization can cause options marked with JsonIgnore > annotations to be dropped before a runner has had a chance to see those > parameters. For example, take the FlinkRunner options, if flinkMaster is > marked with JsonIgnore because that information is only needed by the runner > submitting the job and is not needed during runtime, tests will be run > locally incorrectly instead of against the Flink cluster. > [1] > https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/TestPipeline.java#L354 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-3525) TestPipeline serializes PipelineOptions prematurely
Paul Gerver created BEAM-3525: - Summary: TestPipeline serializes PipelineOptions prematurely Key: BEAM-3525 URL: https://issues.apache.org/jira/browse/BEAM-3525 Project: Beam Issue Type: Bug Components: testing Affects Versions: 2.2.0 Reporter: Paul Gerver Assignee: Paul Gerver The TestPipeline in its run() method now serializes PipelineOptions and adds in TestValueProvider values before submitting the pipeline to run [1]. This premature serialization can cause options marked with JsonIgnore annotations to be dropped before a runner has had a chance to see those parameters. For example, take the FlinkRunner options, if flinkMaster is marked with JsonIgnore because that information is only needed by the runner submitting the job and is not needed during runtime, tests will be run locally incorrectly instead of against the Flink cluster. [1] https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/TestPipeline.java#L354 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (BEAM-3162) Add IBM Streams Runner to compatability matrix
[ https://issues.apache.org/jira/browse/BEAM-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Gerver closed BEAM-3162. - Resolution: Fixed Fix Version/s: Not applicable > Add IBM Streams Runner to compatability matrix > -- > > Key: BEAM-3162 > URL: https://issues.apache.org/jira/browse/BEAM-3162 > Project: Beam > Issue Type: Task > Components: website >Reporter: Paul Gerver >Assignee: Paul Gerver >Priority: Minor > Labels: documentation > Fix For: Not applicable > > Original Estimate: 48h > Remaining Estimate: 48h > > The IBM Streams Runner was announced on 2017-11-02. This task involves adding > a Streams Runner column to the existing Beam capability matrix on the Beam > website. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (BEAM-3162) Add IBM Streams Runner to compatability matrix
[ https://issues.apache.org/jira/browse/BEAM-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16244796#comment-16244796 ] Paul Gerver commented on BEAM-3162: --- I will request Jira access to the Beam dev list to this can be assigned to me. > Add IBM Streams Runner to compatability matrix > -- > > Key: BEAM-3162 > URL: https://issues.apache.org/jira/browse/BEAM-3162 > Project: Beam > Issue Type: Task > Components: website >Reporter: Paul Gerver >Assignee: Reuven Lax >Priority: Minor > Labels: documentation > Original Estimate: 48h > Remaining Estimate: 48h > > The IBM Streams Runner was announced on 2017-11-02. This task involves adding > a Streams Runner column to the existing Beam capability matrix on the Beam > website. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
