[jira] [Created] (BEAM-5508) Upgrade gradle-versions-plugin to 0.20.0
Ted Yu created BEAM-5508: Summary: Upgrade gradle-versions-plugin to 0.20.0 Key: BEAM-5508 URL: https://issues.apache.org/jira/browse/BEAM-5508 Project: Beam Issue Type: Task Components: build-system Reporter: Ted Yu Assignee: Luke Cwik Currently 0.17.0 is used for gradle-versions-plugin This task upgrades to 0.20.0 version https://bintray.com/fooberger/maven/com.github.ben-manes%3Agradle-versions-plugin/0.20.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-4334) The Stream created in LocalFileSystem#matchOne should be closed
[
https://issues.apache.org/jira/browse/BEAM-4334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494404#comment-16494404
]
Ted Yu commented on BEAM-4334:
--
Currently tied up in dayjob.
FYI
> The Stream created in LocalFileSystem#matchOne should be closed
> ---
>
> Key: BEAM-4334
> URL: https://issues.apache.org/jira/browse/BEAM-4334
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Reporter: Ted Yu
>Assignee: Chamikara Jayalath
>Priority: Minor
>
> {code}
> Iterable matchedFiles =
> StreamSupport.stream(files.spliterator(), false)
> .filter(
> {code}
> After result is populated, matchedFiles should be closed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (BEAM-2805) Potential arithmetic overflow in Generator#nextAuctionLengthMs()
[
https://issues.apache.org/jira/browse/BEAM-2805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated BEAM-2805:
-
Description:
{code}
long numEventsForAuctions =
(config.configuration.numInFlightAuctions *
GeneratorConfig.PROPORTION_DENOMINATOR)
/ GeneratorConfig.AUCTION_PROPORTION;
{code}
The multiplication is done on 32-bit integers while long is expected
(numEventsForAuctions).
There is possibility for arithmetic overflow
was:
{code}
long numEventsForAuctions =
(config.configuration.numInFlightAuctions *
GeneratorConfig.PROPORTION_DENOMINATOR)
/ GeneratorConfig.AUCTION_PROPORTION;
{code}
The multiplication is done on 32-bit integers while long is expected
(numEventsForAuctions).
There is possibility for arithmetic overflow
> Potential arithmetic overflow in Generator#nextAuctionLengthMs()
>
>
> Key: BEAM-2805
> URL: https://issues.apache.org/jira/browse/BEAM-2805
> Project: Beam
> Issue Type: Bug
> Components: examples-nexmark
>Reporter: Ted Yu
>Priority: Minor
>
> {code}
> long numEventsForAuctions =
> (config.configuration.numInFlightAuctions *
> GeneratorConfig.PROPORTION_DENOMINATOR)
> / GeneratorConfig.AUCTION_PROPORTION;
> {code}
> The multiplication is done on 32-bit integers while long is expected
> (numEventsForAuctions).
> There is possibility for arithmetic overflow
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (BEAM-3934) BoundedReader should be closed in JavaReadViaImpulse#ReadFromBoundedSourceFn
[
https://issues.apache.org/jira/browse/BEAM-3934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated BEAM-3934:
-
Labels: usability (was: )
> BoundedReader should be closed in JavaReadViaImpulse#ReadFromBoundedSourceFn
>
>
> Key: BEAM-3934
> URL: https://issues.apache.org/jira/browse/BEAM-3934
> Project: Beam
> Issue Type: Bug
> Components: runner-core
>Reporter: Ted Yu
>Assignee: Chamikara Jayalath
>Priority: Minor
> Labels: usability
>
> {code}
> public void readSoruce(ProcessContext ctxt) throws IOException {
> BoundedSource.BoundedReader reader =
> ctxt.element().createReader(ctxt.getPipelineOptions());
> for (boolean more = reader.start(); more; more = reader.advance()) {
> ctxt.outputWithTimestamp(reader.getCurrent(),
> reader.getCurrentTimestamp());
> }
> }
> {code}
> The BoundedSource.BoundedReader instance should be closed before returning
> from the method.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BEAM-4334) The Stream created in LocalFileSystem#matchOne should be closed
Ted Yu created BEAM-4334:
Summary: The Stream created in LocalFileSystem#matchOne should be
closed
Key: BEAM-4334
URL: https://issues.apache.org/jira/browse/BEAM-4334
Project: Beam
Issue Type: Bug
Components: sdk-java-core
Reporter: Ted Yu
Assignee: Kenneth Knowles
{code}
Iterable matchedFiles =
StreamSupport.stream(files.spliterator(), false)
.filter(
{code}
After result is populated, matchedFiles should be closed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-4185) Support building with Java 10
[
https://issues.apache.org/jira/browse/BEAM-4185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16457119#comment-16457119
]
Ted Yu commented on BEAM-4185:
--
Then javadoc errors:
{code}
[javadoc]
/a/beam/sdks/java/javadoc/target/collected/org/apache/beam/sdk/extensions/sql/impl/parser/SqlCreateTable.java:46:
error: cannot find symbol
[javadoc] public class SqlCreateTable extends SqlCreate {
[javadoc] ^
[javadoc] symbol: class SqlCreate
[javadoc]
/a/beam/sdks/java/javadoc/target/collected/org/apache/beam/sdk/extensions/sql/impl/parser/SqlDropTable.java:19:
error: package org.apache.calcite.sql does not exist
[javadoc] import org.apache.calcite.sql.SqlIdentifier;
[javadoc] ^
[javadoc] 100 errors
{code}
> Support building with Java 10
> -
>
> Key: BEAM-4185
> URL: https://issues.apache.org/jira/browse/BEAM-4185
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Ted Yu
>Assignee: Kenneth Knowles
>Priority: Major
>
> Encountered the following in Apache Beam :: SDKs :: Java :: IO :: Solr module
> compiling against Java 10:
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-shade-plugin:3.1.0:shade
> (bundle-and-repackage) on project beam-sdks-java-io-solr: Error creating
> shaded jar: Could not resolve following dependencies:
> [jdk.tools:jdk.tools:jar:1.6 (system)]: Could not resolve dependencies for
> project org.apache.beam:beam-sdks-java-io-solr:jar:2.5.0-SNAPSHOT: Could not
> find artifact jdk.tools:jdk.tools:jar:1.6 at specified path
> /mnt/disk2/a/jdk-10/../lib/tools.jar -> [Help 1]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-4185) Support building with Java 10
[
https://issues.apache.org/jira/browse/BEAM-4185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16457101#comment-16457101
]
Ted Yu commented on BEAM-4185:
--
If I skip the above module, next would be this for Apache Beam :: Runners ::
Flink module:
{code}
[WARNING] An exception occurred while looking for AutoValue extensions. No
extensions will function. This may be due to a corrupt jar file in the
compiler's classpath. Exception: java.util.ServiceConfigurationError:
com.google.auto.value.extension.AutoValueExtension: Provider
com.google.auto.value.extension.memoized.MemoizeExtension could not be
instantiated
[INFO] 1 warning
[INFO] -
[INFO] -
[ERROR] COMPILATION ERROR :
[INFO] -
[ERROR] warnings found and -Werror specified
[INFO] 1 error
{code}
> Support building with Java 10
> -
>
> Key: BEAM-4185
> URL: https://issues.apache.org/jira/browse/BEAM-4185
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Ted Yu
>Assignee: Kenneth Knowles
>Priority: Major
>
> Encountered the following in Apache Beam :: SDKs :: Java :: IO :: Solr module
> compiling against Java 10:
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-shade-plugin:3.1.0:shade
> (bundle-and-repackage) on project beam-sdks-java-io-solr: Error creating
> shaded jar: Could not resolve following dependencies:
> [jdk.tools:jdk.tools:jar:1.6 (system)]: Could not resolve dependencies for
> project org.apache.beam:beam-sdks-java-io-solr:jar:2.5.0-SNAPSHOT: Could not
> find artifact jdk.tools:jdk.tools:jar:1.6 at specified path
> /mnt/disk2/a/jdk-10/../lib/tools.jar -> [Help 1]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BEAM-4185) Support building with Java 10
Ted Yu created BEAM-4185:
Summary: Support building with Java 10
Key: BEAM-4185
URL: https://issues.apache.org/jira/browse/BEAM-4185
Project: Beam
Issue Type: Improvement
Components: sdk-java-core
Reporter: Ted Yu
Assignee: Kenneth Knowles
Encountered the following in Apache Beam :: SDKs :: Java :: IO :: Solr module
compiling against Java 10:
{code}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-shade-plugin:3.1.0:shade (bundle-and-repackage)
on project beam-sdks-java-io-solr: Error creating shaded jar: Could not resolve
following dependencies: [jdk.tools:jdk.tools:jar:1.6 (system)]: Could not
resolve dependencies for project
org.apache.beam:beam-sdks-java-io-solr:jar:2.5.0-SNAPSHOT: Could not find
artifact jdk.tools:jdk.tools:jar:1.6 at specified path
/mnt/disk2/a/jdk-10/../lib/tools.jar -> [Help 1]
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BEAM-3935) FileChannel instance should be closed in ArtifactServiceStager#StagingCallable#get
Ted Yu created BEAM-3935:
Summary: FileChannel instance should be closed in
ArtifactServiceStager#StagingCallable#get
Key: BEAM-3935
URL: https://issues.apache.org/jira/browse/BEAM-3935
Project: Beam
Issue Type: Bug
Components: runner-core
Reporter: Ted Yu
Assignee: Kenneth Knowles
{code}
FileChannel channel = new FileInputStream(file).getChannel();
ByteBuffer readBuffer = ByteBuffer.allocate(bufferSize);
while (!responseObserver.isTerminal() && channel.position() <
channel.size()) {
readBuffer.clear();
channel.read(readBuffer);
{code}
The channel should be closed before returning from get()
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BEAM-3934) BoundedReader should be closed in JavaReadViaImpulse#ReadFromBoundedSourceFn
Ted Yu created BEAM-3934:
Summary: BoundedReader should be closed in
JavaReadViaImpulse#ReadFromBoundedSourceFn
Key: BEAM-3934
URL: https://issues.apache.org/jira/browse/BEAM-3934
Project: Beam
Issue Type: Bug
Components: runner-core
Reporter: Ted Yu
Assignee: Kenneth Knowles
{code}
public void readSoruce(ProcessContext ctxt) throws IOException {
BoundedSource.BoundedReader reader =
ctxt.element().createReader(ctxt.getPipelineOptions());
for (boolean more = reader.start(); more; more = reader.advance()) {
ctxt.outputWithTimestamp(reader.getCurrent(),
reader.getCurrentTimestamp());
}
}
{code}
The BoundedSource.BoundedReader instance should be closed before returning from
the method.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BEAM-3915) Unclosed reader in JavaReadViaImpulse#ReadFromBoundedSourceFn
Ted Yu created BEAM-3915:
Summary: Unclosed reader in
JavaReadViaImpulse#ReadFromBoundedSourceFn
Key: BEAM-3915
URL: https://issues.apache.org/jira/browse/BEAM-3915
Project: Beam
Issue Type: Bug
Components: runner-core
Reporter: Ted Yu
Assignee: Kenneth Knowles
{code}
static class ReadFromBoundedSourceFn extends DoFn, T> {
@ProcessElement
public void readSoruce(ProcessContext ctxt) throws IOException {
BoundedSource.BoundedReader reader =
ctxt.element().createReader(ctxt.getPipelineOptions());
{code}
The reader should be closed upon return from the method.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BEAM-3845) Avoid calling Class#newInstance
Ted Yu created BEAM-3845: Summary: Avoid calling Class#newInstance Key: BEAM-3845 URL: https://issues.apache.org/jira/browse/BEAM-3845 Project: Beam Issue Type: Task Components: project-management Reporter: Ted Yu Assignee: Davor Bonaci Class#newInstance is deprecated starting in Java 9 - https://bugs.openjdk.java.net/browse/JDK-6850612 - because it may throw undeclared checked exceptions. The suggested replacement is getDeclaredConstructor().newInstance(), which wraps the checked exceptions in InvocationException. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-3679) Upgrade calcite to release 1.16
Ted Yu created BEAM-3679: Summary: Upgrade calcite to release 1.16 Key: BEAM-3679 URL: https://issues.apache.org/jira/browse/BEAM-3679 Project: Beam Issue Type: Improvement Components: dsl-sql Reporter: Ted Yu Assignee: Xu Mingmin Currently Beam uses Calcite 1.13.0 This issue is to upgrade to Calcite 1.16.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-3677) Declare proper exception type in catch block of KafkaExactlyOnceSink#initShardWriter
Ted Yu created BEAM-3677:
Summary: Declare proper exception type in catch block of
KafkaExactlyOnceSink#initShardWriter
Key: BEAM-3677
URL: https://issues.apache.org/jira/browse/BEAM-3677
Project: Beam
Issue Type: Bug
Components: sdk-java-extensions
Reporter: Ted Yu
Assignee: Reuven Lax
{code}
} catch (Exception e) {
producer.close();
throw e;
{code}
initShardWriter throws IOException. The catch block shouldn't widen the
exception type.
If the intention is to properly close producer, we can use try-with-resources.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BEAM-3140) Unchecked return value from bis.read() in MutationGroupEncoder#decodeMutation
Ted Yu created BEAM-3140:
Summary: Unchecked return value from bis.read() in
MutationGroupEncoder#decodeMutation
Key: BEAM-3140
URL: https://issues.apache.org/jira/browse/BEAM-3140
Project: Beam
Issue Type: Bug
Components: sdk-java-gcp
Reporter: Ted Yu
Assignee: Chamikara Jayalath
Priority: Minor
{code}
private Mutation decodeMutation(ByteArrayInputStream bis) throws IOException {
Mutation.Op op = Mutation.Op.values()[bis.read()];
if (op == Mutation.Op.DELETE) {
return decodeDelete(bis);
{code}
The return value from bis.read() should be checked before it is used to
dereference values().
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (BEAM-2987) Close of InputStream should be placed in finally block
Ted Yu created BEAM-2987:
Summary: Close of InputStream should be placed in finally block
Key: BEAM-2987
URL: https://issues.apache.org/jira/browse/BEAM-2987
Project: Beam
Issue Type: Bug
Components: sdk-java-extensions
Reporter: Ted Yu
Assignee: Reuven Lax
Priority: Minor
In TikaSource#TikaReader#start():
{code}
try {
parser.parse(is, tikaHandler, tikaMetadata, context);
is.close();
{code}
The close() should be in finally block.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (BEAM-1962) Connection should be closed in case start() throws exception
[
https://issues.apache.org/jira/browse/BEAM-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated BEAM-1962:
-
Description:
In JmsIO#start() :
{code}
try {
Connection connection;
if (spec.getUsername() != null) {
connection =
connectionFactory.createConnection(spec.getUsername(),
spec.getPassword());
} else {
connection = connectionFactory.createConnection();
}
connection.start();
this.connection = connection;
} catch (Exception e) {
throw new IOException("Error connecting to JMS", e);
}
{code}
If start() throws exception, connection should be closed.
was:
In JmsIO#start() :
{code}
try {
Connection connection;
if (spec.getUsername() != null) {
connection =
connectionFactory.createConnection(spec.getUsername(),
spec.getPassword());
} else {
connection = connectionFactory.createConnection();
}
connection.start();
this.connection = connection;
} catch (Exception e) {
throw new IOException("Error connecting to JMS", e);
}
{code}
If start() throws exception, connection should be closed.
> Connection should be closed in case start() throws exception
>
>
> Key: BEAM-1962
> URL: https://issues.apache.org/jira/browse/BEAM-1962
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-extensions
>Reporter: Ted Yu
>Assignee: Jean-Baptiste Onofré
>Priority: Minor
>
> In JmsIO#start() :
> {code}
> try {
> Connection connection;
> if (spec.getUsername() != null) {
> connection =
> connectionFactory.createConnection(spec.getUsername(),
> spec.getPassword());
> } else {
> connection = connectionFactory.createConnection();
> }
> connection.start();
> this.connection = connection;
> } catch (Exception e) {
> throw new IOException("Error connecting to JMS", e);
> }
> {code}
> If start() throws exception, connection should be closed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (BEAM-2805) Potential arithmetic overflow in Generator#nextAuctionLengthMs()
[
https://issues.apache.org/jira/browse/BEAM-2805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated BEAM-2805:
-
Description:
{code}
long numEventsForAuctions =
(config.configuration.numInFlightAuctions *
GeneratorConfig.PROPORTION_DENOMINATOR)
/ GeneratorConfig.AUCTION_PROPORTION;
{code}
The multiplication is done on 32-bit integers while long is expected
(numEventsForAuctions).
There is possibility for arithmetic overflow
was:
{code}
long numEventsForAuctions =
(config.configuration.numInFlightAuctions *
GeneratorConfig.PROPORTION_DENOMINATOR)
/ GeneratorConfig.AUCTION_PROPORTION;
{code}
The multiplication is done on 32-bit integers while long is expected
(numEventsForAuctions).
There is possibility for arithmetic overflow
> Potential arithmetic overflow in Generator#nextAuctionLengthMs()
>
>
> Key: BEAM-2805
> URL: https://issues.apache.org/jira/browse/BEAM-2805
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-extensions
>Reporter: Ted Yu
>Assignee: Reuven Lax
>Priority: Minor
>
> {code}
> long numEventsForAuctions =
> (config.configuration.numInFlightAuctions *
> GeneratorConfig.PROPORTION_DENOMINATOR)
> / GeneratorConfig.AUCTION_PROPORTION;
> {code}
> The multiplication is done on 32-bit integers while long is expected
> (numEventsForAuctions).
> There is possibility for arithmetic overflow
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (BEAM-2775) HadoopInputFormatIOTest fails with NoSuchMethodError
[
https://issues.apache.org/jira/browse/BEAM-2775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated BEAM-2775:
-
Description:
As of commit 8503adbbc3a590cd0dc2939f6a45d335682a9442, I got:
{code}
[ERROR]
testReadObjectCreationWithConfigurationValueTranslation(org.apache.beam.sdk.io.hadoop.inputformat.HadoopInputFormatIOTest)
Time elapsed: 0.007 s <<< ERROR!
java.lang.NoSuchMethodError:
com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;Ljava/lang/Object;)V
at
org.apache.beam.sdk.options.PipelineOptionsFactory.validateMethodsAreEitherBeanMethodOrKnownMethod(PipelineOptionsFactory.java:1255)
at
org.apache.beam.sdk.options.PipelineOptionsFactory.validateClass(PipelineOptionsFactory.java:990)
at
org.apache.beam.sdk.options.PipelineOptionsFactory.validateWellFormed(PipelineOptionsFactory.java:616)
at
org.apache.beam.sdk.options.PipelineOptionsFactory.register(PipelineOptionsFactory.java:550)
at
org.apache.beam.sdk.options.PipelineOptionsFactory.initializeRegistry(PipelineOptionsFactory.java:569)
at
org.apache.beam.sdk.options.PipelineOptionsFactory.(PipelineOptionsFactory.java:525)
at
org.apache.beam.sdk.testing.TestPipeline.testingPipelineOptions(TestPipeline.java:404)
at org.apache.beam.sdk.testing.TestPipeline.create(TestPipeline.java:262)
{code}
Here is related code:
{code}
checkArgument(unknownMethods.isEmpty(),
"Methods %s on [%s] do not conform to being bean properties.",
FluentIterable.from(unknownMethods).transform(ReflectHelpers.METHOD_FORMATTER),
iface.getName());
{code}
This led to test failure of subtests:
{code}
[ERROR]
testReadersGetFractionConsumed(org.apache.beam.sdk.io.hadoop.inputformat.HadoopInputFormatIOTest)
Time elapsed: 0.001 s <<< ERROR!
java.lang.NoClassDefFoundError: Could not initialize class
org.apache.beam.sdk.options.PipelineOptionsFactory
at
org.apache.beam.sdk.testing.TestPipeline.testingPipelineOptions(TestPipeline.java:404)
at org.apache.beam.sdk.testing.TestPipeline.create(TestPipeline.java:262)
at
org.apache.beam.sdk.io.hadoop.inputformat.HadoopInputFormatIOTest.(HadoopInputFormatIOTest.java:69)
{code}
was:
As of commit 8503adbbc3a590cd0dc2939f6a45d335682a9442, I got:
{code}
[ERROR]
testReadObjectCreationWithConfigurationValueTranslation(org.apache.beam.sdk.io.hadoop.inputformat.HadoopInputFormatIOTest)
Time elapsed: 0.007 s <<< ERROR!
java.lang.NoSuchMethodError:
com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;Ljava/lang/Object;)V
at
org.apache.beam.sdk.options.PipelineOptionsFactory.validateMethodsAreEitherBeanMethodOrKnownMethod(PipelineOptionsFactory.java:1255)
at
org.apache.beam.sdk.options.PipelineOptionsFactory.validateClass(PipelineOptionsFactory.java:990)
at
org.apache.beam.sdk.options.PipelineOptionsFactory.validateWellFormed(PipelineOptionsFactory.java:616)
at
org.apache.beam.sdk.options.PipelineOptionsFactory.register(PipelineOptionsFactory.java:550)
at
org.apache.beam.sdk.options.PipelineOptionsFactory.initializeRegistry(PipelineOptionsFactory.java:569)
at
org.apache.beam.sdk.options.PipelineOptionsFactory.(PipelineOptionsFactory.java:525)
at
org.apache.beam.sdk.testing.TestPipeline.testingPipelineOptions(TestPipeline.java:404)
at org.apache.beam.sdk.testing.TestPipeline.create(TestPipeline.java:262)
{code}
Here is related code:
{code}
checkArgument(unknownMethods.isEmpty(),
"Methods %s on [%s] do not conform to being bean properties.",
FluentIterable.from(unknownMethods).transform(ReflectHelpers.METHOD_FORMATTER),
iface.getName());
{code}
This led to test failure of subtests:
{code}
[ERROR]
testReadersGetFractionConsumed(org.apache.beam.sdk.io.hadoop.inputformat.HadoopInputFormatIOTest)
Time elapsed: 0.001 s <<< ERROR!
java.lang.NoClassDefFoundError: Could not initialize class
org.apache.beam.sdk.options.PipelineOptionsFactory
at
org.apache.beam.sdk.testing.TestPipeline.testingPipelineOptions(TestPipeline.java:404)
at org.apache.beam.sdk.testing.TestPipeline.create(TestPipeline.java:262)
at
org.apache.beam.sdk.io.hadoop.inputformat.HadoopInputFormatIOTest.(HadoopInputFormatIOTest.java:69)
{code}
> HadoopInputFormatIOTest fails with NoSuchMethodError
>
>
> Key: BEAM-2775
> URL: https://issues.apache.org/jira/browse/BEAM-2775
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Reporter: Ted Yu
>Assignee: Luke Cwik
>
> As of commit 8503adbbc3a590cd0dc2939f6a45d335682a9442, I got:
> {code}
> [ERROR]
> testReadObjectCreationWithConfigurationValueTranslation(org.apache.beam.sdk.io.hadoop.inputformat.HadoopInputFormatIOTest)
> Time elapsed: 0.007 s <<< ERROR!
> java.lang.NoSuchMethodError:
> com.google.common.base.Preconditions.
[jira] [Updated] (BEAM-1962) Connection should be closed in case start() throws exception
[
https://issues.apache.org/jira/browse/BEAM-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated BEAM-1962:
-
Description:
In JmsIO#start() :
{code}
try {
Connection connection;
if (spec.getUsername() != null) {
connection =
connectionFactory.createConnection(spec.getUsername(),
spec.getPassword());
} else {
connection = connectionFactory.createConnection();
}
connection.start();
this.connection = connection;
} catch (Exception e) {
throw new IOException("Error connecting to JMS", e);
}
{code}
If start() throws exception, connection should be closed.
was:
In JmsIO#start() :
{code}
try {
Connection connection;
if (spec.getUsername() != null) {
connection =
connectionFactory.createConnection(spec.getUsername(),
spec.getPassword());
} else {
connection = connectionFactory.createConnection();
}
connection.start();
this.connection = connection;
} catch (Exception e) {
throw new IOException("Error connecting to JMS", e);
}
{code}
If start() throws exception, connection should be closed.
> Connection should be closed in case start() throws exception
>
>
> Key: BEAM-1962
> URL: https://issues.apache.org/jira/browse/BEAM-1962
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-extensions
>Reporter: Ted Yu
>Assignee: Jean-Baptiste Onofré
>Priority: Minor
>
> In JmsIO#start() :
> {code}
> try {
> Connection connection;
> if (spec.getUsername() != null) {
> connection =
> connectionFactory.createConnection(spec.getUsername(),
> spec.getPassword());
> } else {
> connection = connectionFactory.createConnection();
> }
> connection.start();
> this.connection = connection;
> } catch (Exception e) {
> throw new IOException("Error connecting to JMS", e);
> }
> {code}
> If start() throws exception, connection should be closed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (BEAM-2764) SolrIOTest fails in master branch
[
https://issues.apache.org/jira/browse/BEAM-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16157533#comment-16157533
]
Ted Yu commented on BEAM-2764:
--
lgtm
> SolrIOTest fails in master branch
> -
>
> Key: BEAM-2764
> URL: https://issues.apache.org/jira/browse/BEAM-2764
> Project: Beam
> Issue Type: Test
> Components: sdk-java-extensions
>Reporter: Ted Yu
>Assignee: Cao Manh Dat
>Priority: Minor
>
> From
> https://builds.apache.org/job/beam_Release_NightlySnapshot/org.apache.beam$beam-sdks-java-io-solr/503/testReport/junit/org.apache.beam.sdk.io.solr/SolrIOTest/testSizes/
> :
> {code}
> java.lang.AssertionError:
> Wrong estimated size beyond maximum
> Expected: a value less than <36000L>
> but: <38762L> was greater than <36000L>
> at
> __randomizedtesting.SeedInfo.seed([B412D95ADCA1707D:1747543CE2D1CF75]:0)
> at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
> at org.junit.Assert.assertThat(Assert.java:956)
> at org.apache.beam.sdk.io.solr.SolrIOTest.testSizes(SolrIOTest.java:150)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
> at
> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:239)
> at
> org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:320)
> at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
> at
> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
> at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
> at
> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
> at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
> at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
> at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
> at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequ
[jira] [Updated] (BEAM-2805) Potential arithmetic overflow in Generator#nextAuctionLengthMs()
[
https://issues.apache.org/jira/browse/BEAM-2805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated BEAM-2805:
-
Description:
{code}
long numEventsForAuctions =
(config.configuration.numInFlightAuctions *
GeneratorConfig.PROPORTION_DENOMINATOR)
/ GeneratorConfig.AUCTION_PROPORTION;
{code}
The multiplication is done on 32-bit integers while long is expected
(numEventsForAuctions).
There is possibility for arithmetic overflow
was:
{code}
long numEventsForAuctions =
(config.configuration.numInFlightAuctions *
GeneratorConfig.PROPORTION_DENOMINATOR)
/ GeneratorConfig.AUCTION_PROPORTION;
{code}
The multiplication is done on 32-bit integers while long is expected
(numEventsForAuctions).
There is possibility for arithmetic overflow
> Potential arithmetic overflow in Generator#nextAuctionLengthMs()
>
>
> Key: BEAM-2805
> URL: https://issues.apache.org/jira/browse/BEAM-2805
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-extensions
>Reporter: Ted Yu
>Assignee: Reuven Lax
>Priority: Minor
>
> {code}
> long numEventsForAuctions =
> (config.configuration.numInFlightAuctions *
> GeneratorConfig.PROPORTION_DENOMINATOR)
> / GeneratorConfig.AUCTION_PROPORTION;
> {code}
> The multiplication is done on 32-bit integers while long is expected
> (numEventsForAuctions).
> There is possibility for arithmetic overflow
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (BEAM-2335) Document various maven commands for running tests
[ https://issues.apache.org/jira/browse/BEAM-2335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated BEAM-2335: - Description: In this discussion thread, various maven commands for running / not running selected tests were mentioned: http://search-hadoop.com/m/Beam/gfKHFd9bPDh5WJr1?subj=Re+How+can+I+disable+running+Python+SDK+tests+when+testing+my+Java+change+ We should document these commands under https://beam.apache.org/contribute/testing/ Borisa raised the following questions: how do I execute only one test marked as @NeedsRunner? How do I execute one specific test in java io? How to execute one pecific test in any of the runners? How to use beamTestpipelineoptions with few json examples? Will mvn clean verify execute ALL tests against all runners? For the #1 above, we can create profile which is used run tests in NeedsRunner category. See the following: http://stackoverflow.com/questions/3100924/how-to-run-junit-tests-by-category-in-maven was: In this discussion thread, various maven commands for running / not running selected tests were mentioned: http://search-hadoop.com/m/Beam/gfKHFd9bPDh5WJr1?subj=Re+How+can+I+disable+running+Python+SDK+tests+when+testing+my+Java+change+ We should document these commands under https://beam.apache.org/contribute/testing/ Borisa raised the following questions: how do I execute only one test marked as @NeedsRunner? How do I execute one specific test in java io? How to execute one pecific test in any of the runners? How to use beamTestpipelineoptions with few json examples? Will mvn clean verify execute ALL tests against all runners? For the #1 above, we can create profile which is used run tests in NeedsRunner category. See the following: http://stackoverflow.com/questions/3100924/how-to-run-junit-tests-by-category-in-maven > Document various maven commands for running tests > - > > Key: BEAM-2335 > URL: https://issues.apache.org/jira/browse/BEAM-2335 > Project: Beam > Issue Type: Improvement > Components: testing >Reporter: Ted Yu > Labels: document > > In this discussion thread, various maven commands for running / not running > selected tests were mentioned: > http://search-hadoop.com/m/Beam/gfKHFd9bPDh5WJr1?subj=Re+How+can+I+disable+running+Python+SDK+tests+when+testing+my+Java+change+ > We should document these commands under > https://beam.apache.org/contribute/testing/ > Borisa raised the following questions: > how do I execute only one test marked as @NeedsRunner? > How do I execute one specific test in java io? > How to execute one pecific test in any of the runners? > How to use beamTestpipelineoptions with few json examples? > Will mvn clean verify execute ALL tests against all runners? > For the #1 above, we can create profile which is used run tests in > NeedsRunner category. > See the following: > http://stackoverflow.com/questions/3100924/how-to-run-junit-tests-by-category-in-maven -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (BEAM-2805) Potential arithmetic overflow in Generator#nextAuctionLengthMs()
Ted Yu created BEAM-2805:
Summary: Potential arithmetic overflow in
Generator#nextAuctionLengthMs()
Key: BEAM-2805
URL: https://issues.apache.org/jira/browse/BEAM-2805
Project: Beam
Issue Type: Bug
Components: sdk-java-extensions
Reporter: Ted Yu
Assignee: Reuven Lax
Priority: Minor
{code}
long numEventsForAuctions =
(config.configuration.numInFlightAuctions *
GeneratorConfig.PROPORTION_DENOMINATOR)
/ GeneratorConfig.AUCTION_PROPORTION;
{code}
The multiplication is done on 32-bit integers while long is expected
(numEventsForAuctions).
There is possibility for arithmetic overflow
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (BEAM-2775) HadoopInputFormatIOTest fails with NoSuchMethodError
Ted Yu created BEAM-2775:
Summary: HadoopInputFormatIOTest fails with NoSuchMethodError
Key: BEAM-2775
URL: https://issues.apache.org/jira/browse/BEAM-2775
Project: Beam
Issue Type: Bug
Components: sdk-java-core
Reporter: Ted Yu
Assignee: Luke Cwik
As of commit 8503adbbc3a590cd0dc2939f6a45d335682a9442, I got:
{code}
[ERROR]
testReadObjectCreationWithConfigurationValueTranslation(org.apache.beam.sdk.io.hadoop.inputformat.HadoopInputFormatIOTest)
Time elapsed: 0.007 s <<< ERROR!
java.lang.NoSuchMethodError:
com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;Ljava/lang/Object;)V
at
org.apache.beam.sdk.options.PipelineOptionsFactory.validateMethodsAreEitherBeanMethodOrKnownMethod(PipelineOptionsFactory.java:1255)
at
org.apache.beam.sdk.options.PipelineOptionsFactory.validateClass(PipelineOptionsFactory.java:990)
at
org.apache.beam.sdk.options.PipelineOptionsFactory.validateWellFormed(PipelineOptionsFactory.java:616)
at
org.apache.beam.sdk.options.PipelineOptionsFactory.register(PipelineOptionsFactory.java:550)
at
org.apache.beam.sdk.options.PipelineOptionsFactory.initializeRegistry(PipelineOptionsFactory.java:569)
at
org.apache.beam.sdk.options.PipelineOptionsFactory.(PipelineOptionsFactory.java:525)
at
org.apache.beam.sdk.testing.TestPipeline.testingPipelineOptions(TestPipeline.java:404)
at org.apache.beam.sdk.testing.TestPipeline.create(TestPipeline.java:262)
{code}
Here is related code:
{code}
checkArgument(unknownMethods.isEmpty(),
"Methods %s on [%s] do not conform to being bean properties.",
FluentIterable.from(unknownMethods).transform(ReflectHelpers.METHOD_FORMATTER),
iface.getName());
{code}
This led to test failure of subtests:
{code}
[ERROR]
testReadersGetFractionConsumed(org.apache.beam.sdk.io.hadoop.inputformat.HadoopInputFormatIOTest)
Time elapsed: 0.001 s <<< ERROR!
java.lang.NoClassDefFoundError: Could not initialize class
org.apache.beam.sdk.options.PipelineOptionsFactory
at
org.apache.beam.sdk.testing.TestPipeline.testingPipelineOptions(TestPipeline.java:404)
at org.apache.beam.sdk.testing.TestPipeline.create(TestPipeline.java:262)
at
org.apache.beam.sdk.io.hadoop.inputformat.HadoopInputFormatIOTest.(HadoopInputFormatIOTest.java:69)
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (BEAM-2764) SolrIOTest fails in master branch
Ted Yu created BEAM-2764:
Summary: SolrIOTest fails in master branch
Key: BEAM-2764
URL: https://issues.apache.org/jira/browse/BEAM-2764
Project: Beam
Issue Type: Test
Components: sdk-java-extensions
Reporter: Ted Yu
Assignee: Davor Bonaci
Priority: Minor
>From
>https://builds.apache.org/job/beam_Release_NightlySnapshot/org.apache.beam$beam-sdks-java-io-solr/503/testReport/junit/org.apache.beam.sdk.io.solr/SolrIOTest/testSizes/
> :
{code}
java.lang.AssertionError:
Wrong estimated size beyond maximum
Expected: a value less than <36000L>
but: <38762L> was greater than <36000L>
at
__randomizedtesting.SeedInfo.seed([B412D95ADCA1707D:1747543CE2D1CF75]:0)
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
at org.junit.Assert.assertThat(Assert.java:956)
at org.apache.beam.sdk.io.solr.SolrIOTest.testSizes(SolrIOTest.java:150)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
at
org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:239)
at
org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:320)
at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
at
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evalua
[jira] [Updated] (BEAM-2335) Document various maven commands for running tests
[ https://issues.apache.org/jira/browse/BEAM-2335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated BEAM-2335: - Description: In this discussion thread, various maven commands for running / not running selected tests were mentioned: http://search-hadoop.com/m/Beam/gfKHFd9bPDh5WJr1?subj=Re+How+can+I+disable+running+Python+SDK+tests+when+testing+my+Java+change+ We should document these commands under https://beam.apache.org/contribute/testing/ Borisa raised the following questions: how do I execute only one test marked as @NeedsRunner? How do I execute one specific test in java io? How to execute one pecific test in any of the runners? How to use beamTestpipelineoptions with few json examples? Will mvn clean verify execute ALL tests against all runners? For the #1 above, we can create profile which is used run tests in NeedsRunner category. See the following: http://stackoverflow.com/questions/3100924/how-to-run-junit-tests-by-category-in-maven was: In this discussion thread, various maven commands for running / not running selected tests were mentioned: http://search-hadoop.com/m/Beam/gfKHFd9bPDh5WJr1?subj=Re+How+can+I+disable+running+Python+SDK+tests+when+testing+my+Java+change+ We should document these commands under https://beam.apache.org/contribute/testing/ Borisa raised the following questions: how do I execute only one test marked as @NeedsRunner? How do I execute one specific test in java io? How to execute one pecific test in any of the runners? How to use beamTestpipelineoptions with few json examples? Will mvn clean verify execute ALL tests against all runners? For the #1 above, we can create profile which is used run tests in NeedsRunner category. See the following: http://stackoverflow.com/questions/3100924/how-to-run-junit-tests-by-category-in-maven > Document various maven commands for running tests > - > > Key: BEAM-2335 > URL: https://issues.apache.org/jira/browse/BEAM-2335 > Project: Beam > Issue Type: Improvement > Components: testing >Reporter: Ted Yu > Labels: document > > In this discussion thread, various maven commands for running / not running > selected tests were mentioned: > http://search-hadoop.com/m/Beam/gfKHFd9bPDh5WJr1?subj=Re+How+can+I+disable+running+Python+SDK+tests+when+testing+my+Java+change+ > We should document these commands under > https://beam.apache.org/contribute/testing/ > Borisa raised the following questions: > how do I execute only one test marked as @NeedsRunner? > How do I execute one specific test in java io? > How to execute one pecific test in any of the runners? > How to use beamTestpipelineoptions with few json examples? > Will mvn clean verify execute ALL tests against all runners? > For the #1 above, we can create profile which is used run tests in > NeedsRunner category. > See the following: > http://stackoverflow.com/questions/3100924/how-to-run-junit-tests-by-category-in-maven -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (BEAM-1962) Connection should be closed in case start() throws exception
[
https://issues.apache.org/jira/browse/BEAM-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated BEAM-1962:
-
Description:
In JmsIO#start() :
{code}
try {
Connection connection;
if (spec.getUsername() != null) {
connection =
connectionFactory.createConnection(spec.getUsername(),
spec.getPassword());
} else {
connection = connectionFactory.createConnection();
}
connection.start();
this.connection = connection;
} catch (Exception e) {
throw new IOException("Error connecting to JMS", e);
}
{code}
If start() throws exception, connection should be closed.
was:
In JmsIO#start() :
{code}
try {
Connection connection;
if (spec.getUsername() != null) {
connection =
connectionFactory.createConnection(spec.getUsername(),
spec.getPassword());
} else {
connection = connectionFactory.createConnection();
}
connection.start();
this.connection = connection;
} catch (Exception e) {
throw new IOException("Error connecting to JMS", e);
}
{code}
If start() throws exception, connection should be closed.
> Connection should be closed in case start() throws exception
>
>
> Key: BEAM-1962
> URL: https://issues.apache.org/jira/browse/BEAM-1962
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-extensions
>Reporter: Ted Yu
>Assignee: Jean-Baptiste Onofré
>Priority: Minor
>
> In JmsIO#start() :
> {code}
> try {
> Connection connection;
> if (spec.getUsername() != null) {
> connection =
> connectionFactory.createConnection(spec.getUsername(),
> spec.getPassword());
> } else {
> connection = connectionFactory.createConnection();
> }
> connection.start();
> this.connection = connection;
> } catch (Exception e) {
> throw new IOException("Error connecting to JMS", e);
> }
> {code}
> If start() throws exception, connection should be closed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (BEAM-1962) Connection should be closed in case start() throws exception
[
https://issues.apache.org/jira/browse/BEAM-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated BEAM-1962:
-
Description:
In JmsIO#start() :
{code}
try {
Connection connection;
if (spec.getUsername() != null) {
connection =
connectionFactory.createConnection(spec.getUsername(),
spec.getPassword());
} else {
connection = connectionFactory.createConnection();
}
connection.start();
this.connection = connection;
} catch (Exception e) {
throw new IOException("Error connecting to JMS", e);
}
{code}
If start() throws exception, connection should be closed.
was:
In JmsIO#start() :
{code}
try {
Connection connection;
if (spec.getUsername() != null) {
connection =
connectionFactory.createConnection(spec.getUsername(),
spec.getPassword());
} else {
connection = connectionFactory.createConnection();
}
connection.start();
this.connection = connection;
} catch (Exception e) {
throw new IOException("Error connecting to JMS", e);
}
{code}
If start() throws exception, connection should be closed.
> Connection should be closed in case start() throws exception
>
>
> Key: BEAM-1962
> URL: https://issues.apache.org/jira/browse/BEAM-1962
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-extensions
>Reporter: Ted Yu
>Assignee: Jean-Baptiste Onofré
>Priority: Minor
>
> In JmsIO#start() :
> {code}
> try {
> Connection connection;
> if (spec.getUsername() != null) {
> connection =
> connectionFactory.createConnection(spec.getUsername(),
> spec.getPassword());
> } else {
> connection = connectionFactory.createConnection();
> }
> connection.start();
> this.connection = connection;
> } catch (Exception e) {
> throw new IOException("Error connecting to JMS", e);
> }
> {code}
> If start() throws exception, connection should be closed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (BEAM-2335) Document various maven commands for running tests
[ https://issues.apache.org/jira/browse/BEAM-2335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated BEAM-2335: - Description: In this discussion thread, various maven commands for running / not running selected tests were mentioned: http://search-hadoop.com/m/Beam/gfKHFd9bPDh5WJr1?subj=Re+How+can+I+disable+running+Python+SDK+tests+when+testing+my+Java+change+ We should document these commands under https://beam.apache.org/contribute/testing/ Borisa raised the following questions: how do I execute only one test marked as @NeedsRunner? How do I execute one specific test in java io? How to execute one pecific test in any of the runners? How to use beamTestpipelineoptions with few json examples? Will mvn clean verify execute ALL tests against all runners? For the #1 above, we can create profile which is used run tests in NeedsRunner category. See the following: http://stackoverflow.com/questions/3100924/how-to-run-junit-tests-by-category-in-maven was: In this discussion thread, various maven commands for running / not running selected tests were mentioned: http://search-hadoop.com/m/Beam/gfKHFd9bPDh5WJr1?subj=Re+How+can+I+disable+running+Python+SDK+tests+when+testing+my+Java+change+ We should document these commands under https://beam.apache.org/contribute/testing/ Borisa raised the following questions: how do I execute only one test marked as @NeedsRunner? How do I execute one specific test in java io? How to execute one pecific test in any of the runners? How to use beamTestpipelineoptions with few json examples? Will mvn clean verify execute ALL tests against all runners? For the #1 above, we can create profile which is used run tests in NeedsRunner category. See the following: http://stackoverflow.com/questions/3100924/how-to-run-junit-tests-by-category-in-maven > Document various maven commands for running tests > - > > Key: BEAM-2335 > URL: https://issues.apache.org/jira/browse/BEAM-2335 > Project: Beam > Issue Type: Improvement > Components: testing >Reporter: Ted Yu > Labels: document > > In this discussion thread, various maven commands for running / not running > selected tests were mentioned: > http://search-hadoop.com/m/Beam/gfKHFd9bPDh5WJr1?subj=Re+How+can+I+disable+running+Python+SDK+tests+when+testing+my+Java+change+ > We should document these commands under > https://beam.apache.org/contribute/testing/ > Borisa raised the following questions: > how do I execute only one test marked as @NeedsRunner? > How do I execute one specific test in java io? > How to execute one pecific test in any of the runners? > How to use beamTestpipelineoptions with few json examples? > Will mvn clean verify execute ALL tests against all runners? > For the #1 above, we can create profile which is used run tests in > NeedsRunner category. > See the following: > http://stackoverflow.com/questions/3100924/how-to-run-junit-tests-by-category-in-maven -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (BEAM-1773) Consider allowing Source#validate() to throw exception
[
https://issues.apache.org/jira/browse/BEAM-1773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu resolved BEAM-1773.
--
Resolution: Later
Fix Version/s: Not applicable
> Consider allowing Source#validate() to throw exception
> --
>
> Key: BEAM-1773
> URL: https://issues.apache.org/jira/browse/BEAM-1773
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Ted Yu
>Assignee: Ted Yu
> Labels: backward-incompatible
> Fix For: Not applicable
>
> Attachments: 1773.v4.patch, beam-1773.v1.patch, beam-1773.v2.patch
>
>
> In HDFSFileSource.java :
> {code}
> @Override
> public void validate() {
> ...
> } catch (IOException | InterruptedException e) {
> throw new RuntimeException(e);
> }
> {code}
> Source#validate() should be allowed to throw exception so that we don't
> resort to using RuntimeException.
> Here was related thread on mailing list:
> http://search-hadoop.com/m/Beam/gfKHFOwE0uETxae?subj=Re+why+Source+validate+is+not+declared+to+throw+any+exception
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (BEAM-1962) Connection should be closed in case start() throws exception
[
https://issues.apache.org/jira/browse/BEAM-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated BEAM-1962:
-
Description:
In JmsIO#start() :
{code}
try {
Connection connection;
if (spec.getUsername() != null) {
connection =
connectionFactory.createConnection(spec.getUsername(),
spec.getPassword());
} else {
connection = connectionFactory.createConnection();
}
connection.start();
this.connection = connection;
} catch (Exception e) {
throw new IOException("Error connecting to JMS", e);
}
{code}
If start() throws exception, connection should be closed.
was:
In JmsIO#start() :
{code}
try {
Connection connection;
if (spec.getUsername() != null) {
connection =
connectionFactory.createConnection(spec.getUsername(),
spec.getPassword());
} else {
connection = connectionFactory.createConnection();
}
connection.start();
this.connection = connection;
} catch (Exception e) {
throw new IOException("Error connecting to JMS", e);
}
{code}
If start() throws exception, connection should be closed.
> Connection should be closed in case start() throws exception
>
>
> Key: BEAM-1962
> URL: https://issues.apache.org/jira/browse/BEAM-1962
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-extensions
>Reporter: Ted Yu
>Assignee: Jean-Baptiste Onofré
>Priority: Minor
>
> In JmsIO#start() :
> {code}
> try {
> Connection connection;
> if (spec.getUsername() != null) {
> connection =
> connectionFactory.createConnection(spec.getUsername(),
> spec.getPassword());
> } else {
> connection = connectionFactory.createConnection();
> }
> connection.start();
> this.connection = connection;
> } catch (Exception e) {
> throw new IOException("Error connecting to JMS", e);
> }
> {code}
> If start() throws exception, connection should be closed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (BEAM-2335) Document various maven commands for running tests
[ https://issues.apache.org/jira/browse/BEAM-2335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated BEAM-2335: - Description: In this discussion thread, various maven commands for running / not running selected tests were mentioned: http://search-hadoop.com/m/Beam/gfKHFd9bPDh5WJr1?subj=Re+How+can+I+disable+running+Python+SDK+tests+when+testing+my+Java+change+ We should document these commands under https://beam.apache.org/contribute/testing/ Borisa raised the following questions: how do I execute only one test marked as @NeedsRunner? How do I execute one specific test in java io? How to execute one pecific test in any of the runners? How to use beamTestpipelineoptions with few json examples? Will mvn clean verify execute ALL tests against all runners? For the #1 above, we can create profile which is used run tests in NeedsRunner category. See the following: http://stackoverflow.com/questions/3100924/how-to-run-junit-tests-by-category-in-maven was: In this discussion thread, various maven commands for running / not running selected tests were mentioned: http://search-hadoop.com/m/Beam/gfKHFd9bPDh5WJr1?subj=Re+How+can+I+disable+running+Python+SDK+tests+when+testing+my+Java+change+ We should document these commands under https://beam.apache.org/contribute/testing/ Borisa raised the following questions: how do I execute only one test marked as @NeedsRunner? How do I execute one specific test in java io? How to execute one pecific test in any of the runners? How to use beamTestpipelineoptions with few json examples? Will mvn clean verify execute ALL tests against all runners? For the #1 above, we can create profile which is used run tests in NeedsRunner category. See the following: http://stackoverflow.com/questions/3100924/how-to-run-junit-tests-by-category-in-maven > Document various maven commands for running tests > - > > Key: BEAM-2335 > URL: https://issues.apache.org/jira/browse/BEAM-2335 > Project: Beam > Issue Type: Improvement > Components: testing >Reporter: Ted Yu > Labels: document > > In this discussion thread, various maven commands for running / not running > selected tests were mentioned: > http://search-hadoop.com/m/Beam/gfKHFd9bPDh5WJr1?subj=Re+How+can+I+disable+running+Python+SDK+tests+when+testing+my+Java+change+ > We should document these commands under > https://beam.apache.org/contribute/testing/ > Borisa raised the following questions: > how do I execute only one test marked as @NeedsRunner? > How do I execute one specific test in java io? > How to execute one pecific test in any of the runners? > How to use beamTestpipelineoptions with few json examples? > Will mvn clean verify execute ALL tests against all runners? > For the #1 above, we can create profile which is used run tests in > NeedsRunner category. > See the following: > http://stackoverflow.com/questions/3100924/how-to-run-junit-tests-by-category-in-maven -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (BEAM-1962) Connection should be closed in case start() throws exception
[
https://issues.apache.org/jira/browse/BEAM-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated BEAM-1962:
-
Description:
In JmsIO#start() :
{code}
try {
Connection connection;
if (spec.getUsername() != null) {
connection =
connectionFactory.createConnection(spec.getUsername(),
spec.getPassword());
} else {
connection = connectionFactory.createConnection();
}
connection.start();
this.connection = connection;
} catch (Exception e) {
throw new IOException("Error connecting to JMS", e);
}
{code}
If start() throws exception, connection should be closed.
was:
In JmsIO#start() :
{code}
try {
Connection connection;
if (spec.getUsername() != null) {
connection =
connectionFactory.createConnection(spec.getUsername(),
spec.getPassword());
} else {
connection = connectionFactory.createConnection();
}
connection.start();
this.connection = connection;
} catch (Exception e) {
throw new IOException("Error connecting to JMS", e);
}
{code}
If start() throws exception, connection should be closed.
> Connection should be closed in case start() throws exception
>
>
> Key: BEAM-1962
> URL: https://issues.apache.org/jira/browse/BEAM-1962
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-extensions
>Reporter: Ted Yu
>Assignee: Jean-Baptiste Onofré
>Priority: Minor
>
> In JmsIO#start() :
> {code}
> try {
> Connection connection;
> if (spec.getUsername() != null) {
> connection =
> connectionFactory.createConnection(spec.getUsername(),
> spec.getPassword());
> } else {
> connection = connectionFactory.createConnection();
> }
> connection.start();
> this.connection = connection;
> } catch (Exception e) {
> throw new IOException("Error connecting to JMS", e);
> }
> {code}
> If start() throws exception, connection should be closed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (BEAM-2335) Document various maven commands for running tests
[ https://issues.apache.org/jira/browse/BEAM-2335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated BEAM-2335: - Labels: document (was: ) > Document various maven commands for running tests > - > > Key: BEAM-2335 > URL: https://issues.apache.org/jira/browse/BEAM-2335 > Project: Beam > Issue Type: Improvement > Components: testing >Reporter: Ted Yu > Labels: document > > In this discussion thread, various maven commands for running / not running > selected tests were mentioned: > http://search-hadoop.com/m/Beam/gfKHFd9bPDh5WJr1?subj=Re+How+can+I+disable+running+Python+SDK+tests+when+testing+my+Java+change+ > We should document these commands under > https://beam.apache.org/contribute/testing/ > Borisa raised the following questions: > how do I execute only one test marked as @NeedsRunner? > How do I execute one specific test in java io? > How to execute one pecific test in any of the runners? > How to use beamTestpipelineoptions with few json examples? > Will mvn clean verify execute ALL tests against all runners? > For the #1 above, we can create profile which is used run tests in > NeedsRunner category. > See the following: > http://stackoverflow.com/questions/3100924/how-to-run-junit-tests-by-category-in-maven -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (BEAM-2608) Unclosed BoundedReader in TextIO#ReadTextFn#process()
Ted Yu created BEAM-2608:
Summary: Unclosed BoundedReader in TextIO#ReadTextFn#process()
Key: BEAM-2608
URL: https://issues.apache.org/jira/browse/BEAM-2608
Project: Beam
Issue Type: Bug
Components: sdk-java-core
Reporter: Ted Yu
Assignee: Davor Bonaci
Priority: Minor
{code}
BoundedSource.BoundedReader reader =
source
.createForSubrangeOfFile(metadata, range.getFrom(),
range.getTo())
.createReader(c.getPipelineOptions());
{code}
The reader should be closed upon return.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (BEAM-1962) Connection should be closed in case start() throws exception
[
https://issues.apache.org/jira/browse/BEAM-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated BEAM-1962:
-
Description:
In JmsIO#start() :
{code}
try {
Connection connection;
if (spec.getUsername() != null) {
connection =
connectionFactory.createConnection(spec.getUsername(),
spec.getPassword());
} else {
connection = connectionFactory.createConnection();
}
connection.start();
this.connection = connection;
} catch (Exception e) {
throw new IOException("Error connecting to JMS", e);
}
{code}
If start() throws exception, connection should be closed.
was:
In JmsIO#start() :
{code}
try {
Connection connection;
if (spec.getUsername() != null) {
connection =
connectionFactory.createConnection(spec.getUsername(),
spec.getPassword());
} else {
connection = connectionFactory.createConnection();
}
connection.start();
this.connection = connection;
} catch (Exception e) {
throw new IOException("Error connecting to JMS", e);
}
{code}
If start() throws exception, connection should be closed.
> Connection should be closed in case start() throws exception
>
>
> Key: BEAM-1962
> URL: https://issues.apache.org/jira/browse/BEAM-1962
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-extensions
>Reporter: Ted Yu
>Assignee: Jean-Baptiste Onofré
>Priority: Minor
>
> In JmsIO#start() :
> {code}
> try {
> Connection connection;
> if (spec.getUsername() != null) {
> connection =
> connectionFactory.createConnection(spec.getUsername(),
> spec.getPassword());
> } else {
> connection = connectionFactory.createConnection();
> }
> connection.start();
> this.connection = connection;
> } catch (Exception e) {
> throw new IOException("Error connecting to JMS", e);
> }
> {code}
> If start() throws exception, connection should be closed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (BEAM-2335) Document various maven commands for running tests
[ https://issues.apache.org/jira/browse/BEAM-2335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated BEAM-2335: - Description: In this discussion thread, various maven commands for running / not running selected tests were mentioned: http://search-hadoop.com/m/Beam/gfKHFd9bPDh5WJr1?subj=Re+How+can+I+disable+running+Python+SDK+tests+when+testing+my+Java+change+ We should document these commands under https://beam.apache.org/contribute/testing/ Borisa raised the following questions: how do I execute only one test marked as @NeedsRunner? How do I execute one specific test in java io? How to execute one pecific test in any of the runners? How to use beamTestpipelineoptions with few json examples? Will mvn clean verify execute ALL tests against all runners? For the #1 above, we can create profile which is used run tests in NeedsRunner category. See the following: http://stackoverflow.com/questions/3100924/how-to-run-junit-tests-by-category-in-maven was: In this discussion thread, various maven commands for running / not running selected tests were mentioned: http://search-hadoop.com/m/Beam/gfKHFd9bPDh5WJr1?subj=Re+How+can+I+disable+running+Python+SDK+tests+when+testing+my+Java+change+ We should document these commands under https://beam.apache.org/contribute/testing/ Borisa raised the following questions: how do I execute only one test marked as @NeedsRunner? How do I execute one specific test in java io? How to execute one pecific test in any of the runners? How to use beamTestpipelineoptions with few json examples? Will mvn clean verify execute ALL tests against all runners? For the #1 above, we can create profile which is used run tests in NeedsRunner category. See the following: http://stackoverflow.com/questions/3100924/how-to-run-junit-tests-by-category-in-maven > Document various maven commands for running tests > - > > Key: BEAM-2335 > URL: https://issues.apache.org/jira/browse/BEAM-2335 > Project: Beam > Issue Type: Improvement > Components: testing >Reporter: Ted Yu > > In this discussion thread, various maven commands for running / not running > selected tests were mentioned: > http://search-hadoop.com/m/Beam/gfKHFd9bPDh5WJr1?subj=Re+How+can+I+disable+running+Python+SDK+tests+when+testing+my+Java+change+ > We should document these commands under > https://beam.apache.org/contribute/testing/ > Borisa raised the following questions: > how do I execute only one test marked as @NeedsRunner? > How do I execute one specific test in java io? > How to execute one pecific test in any of the runners? > How to use beamTestpipelineoptions with few json examples? > Will mvn clean verify execute ALL tests against all runners? > For the #1 above, we can create profile which is used run tests in > NeedsRunner category. > See the following: > http://stackoverflow.com/questions/3100924/how-to-run-junit-tests-by-category-in-maven -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (BEAM-1962) Connection should be closed in case start() throws exception
[
https://issues.apache.org/jira/browse/BEAM-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated BEAM-1962:
-
Description:
In JmsIO#start() :
{code}
try {
Connection connection;
if (spec.getUsername() != null) {
connection =
connectionFactory.createConnection(spec.getUsername(),
spec.getPassword());
} else {
connection = connectionFactory.createConnection();
}
connection.start();
this.connection = connection;
} catch (Exception e) {
throw new IOException("Error connecting to JMS", e);
}
{code}
If start() throws exception, connection should be closed.
was:
In JmsIO#start() :
{code}
try {
Connection connection;
if (spec.getUsername() != null) {
connection =
connectionFactory.createConnection(spec.getUsername(),
spec.getPassword());
} else {
connection = connectionFactory.createConnection();
}
connection.start();
this.connection = connection;
} catch (Exception e) {
throw new IOException("Error connecting to JMS", e);
}
{code}
If start() throws exception, connection should be closed.
> Connection should be closed in case start() throws exception
>
>
> Key: BEAM-1962
> URL: https://issues.apache.org/jira/browse/BEAM-1962
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-extensions
>Reporter: Ted Yu
>Assignee: Jean-Baptiste Onofré
>Priority: Minor
>
> In JmsIO#start() :
> {code}
> try {
> Connection connection;
> if (spec.getUsername() != null) {
> connection =
> connectionFactory.createConnection(spec.getUsername(),
> spec.getPassword());
> } else {
> connection = connectionFactory.createConnection();
> }
> connection.start();
> this.connection = connection;
> } catch (Exception e) {
> throw new IOException("Error connecting to JMS", e);
> }
> {code}
> If start() throws exception, connection should be closed.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (BEAM-2335) Document various maven commands for running tests
Ted Yu created BEAM-2335: Summary: Document various maven commands for running tests Key: BEAM-2335 URL: https://issues.apache.org/jira/browse/BEAM-2335 Project: Beam Issue Type: Improvement Components: testing Reporter: Ted Yu Assignee: Davor Bonaci In this discussion thread, various maven commands for running / not running selected tests were mentioned: http://search-hadoop.com/m/Beam/gfKHFd9bPDh5WJr1?subj=Re+How+can+I+disable+running+Python+SDK+tests+when+testing+my+Java+change+ We should document these commands under https://beam.apache.org/contribute/testing/ Borisa raised the following questions: how do I execute only one test marked as @NeedsRunner? How do I execute one specific test in java io? How to execute one pecific test in any of the runners? How to use beamTestpipelineoptions with few json examples? Will mvn clean verify execute ALL tests against all runners? For the #1 above, we can create profile which is used run tests in NeedsRunner category. See the following: http://stackoverflow.com/questions/3100924/how-to-run-junit-tests-by-category-in-maven -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (BEAM-1773) Consider allowing Source#validate() to throw exception
[
https://issues.apache.org/jira/browse/BEAM-1773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003096#comment-16003096
]
Ted Yu commented on BEAM-1773:
--
If there is no good guideline for surfacing validation errors, I can close the
PR.
> Consider allowing Source#validate() to throw exception
> --
>
> Key: BEAM-1773
> URL: https://issues.apache.org/jira/browse/BEAM-1773
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 1773.v4.patch, beam-1773.v1.patch, beam-1773.v2.patch
>
>
> In HDFSFileSource.java :
> {code}
> @Override
> public void validate() {
> ...
> } catch (IOException | InterruptedException e) {
> throw new RuntimeException(e);
> }
> {code}
> Source#validate() should be allowed to throw exception so that we don't
> resort to using RuntimeException.
> Here was related thread on mailing list:
> http://search-hadoop.com/m/Beam/gfKHFOwE0uETxae?subj=Re+why+Source+validate+is+not+declared+to+throw+any+exception
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (BEAM-1962) Connection should be closed in case start() throws exception
[
https://issues.apache.org/jira/browse/BEAM-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated BEAM-1962:
-
Description:
In JmsIO#start() :
{code}
try {
Connection connection;
if (spec.getUsername() != null) {
connection =
connectionFactory.createConnection(spec.getUsername(),
spec.getPassword());
} else {
connection = connectionFactory.createConnection();
}
connection.start();
this.connection = connection;
} catch (Exception e) {
throw new IOException("Error connecting to JMS", e);
}
{code}
If start() throws exception, connection should be closed.
was:
In JmsIO#start() :
{code}
try {
Connection connection;
if (spec.getUsername() != null) {
connection =
connectionFactory.createConnection(spec.getUsername(),
spec.getPassword());
} else {
connection = connectionFactory.createConnection();
}
connection.start();
this.connection = connection;
} catch (Exception e) {
throw new IOException("Error connecting to JMS", e);
}
{code}
If start() throws exception, connection should be closed.
> Connection should be closed in case start() throws exception
>
>
> Key: BEAM-1962
> URL: https://issues.apache.org/jira/browse/BEAM-1962
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-extensions
>Reporter: Ted Yu
>Assignee: Jean-Baptiste Onofré
>Priority: Minor
>
> In JmsIO#start() :
> {code}
> try {
> Connection connection;
> if (spec.getUsername() != null) {
> connection =
> connectionFactory.createConnection(spec.getUsername(),
> spec.getPassword());
> } else {
> connection = connectionFactory.createConnection();
> }
> connection.start();
> this.connection = connection;
> } catch (Exception e) {
> throw new IOException("Error connecting to JMS", e);
> }
> {code}
> If start() throws exception, connection should be closed.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (BEAM-1773) Consider allowing Source#validate() to throw exception
[
https://issues.apache.org/jira/browse/BEAM-1773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15991524#comment-15991524
]
Ted Yu commented on BEAM-1773:
--
Currently working on making tests compile.
For related sub-tests, I am adding "throws ValidationException".
> Consider allowing Source#validate() to throw exception
> --
>
> Key: BEAM-1773
> URL: https://issues.apache.org/jira/browse/BEAM-1773
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 1773.v4.patch, beam-1773.v1.patch, beam-1773.v2.patch
>
>
> In HDFSFileSource.java :
> {code}
> @Override
> public void validate() {
> ...
> } catch (IOException | InterruptedException e) {
> throw new RuntimeException(e);
> }
> {code}
> Source#validate() should be allowed to throw exception so that we don't
> resort to using RuntimeException.
> Here was related thread on mailing list:
> http://search-hadoop.com/m/Beam/gfKHFOwE0uETxae?subj=Re+why+Source+validate+is+not+declared+to+throw+any+exception
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (BEAM-1773) Consider allowing Source#validate() to throw exception
[
https://issues.apache.org/jira/browse/BEAM-1773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15990938#comment-15990938
]
Ted Yu commented on BEAM-1773:
--
[~kenn]:
Can you take a look at the latest patch ?
> Consider allowing Source#validate() to throw exception
> --
>
> Key: BEAM-1773
> URL: https://issues.apache.org/jira/browse/BEAM-1773
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 1773.v4.patch, beam-1773.v1.patch, beam-1773.v2.patch
>
>
> In HDFSFileSource.java :
> {code}
> @Override
> public void validate() {
> ...
> } catch (IOException | InterruptedException e) {
> throw new RuntimeException(e);
> }
> {code}
> Source#validate() should be allowed to throw exception so that we don't
> resort to using RuntimeException.
> Here was related thread on mailing list:
> http://search-hadoop.com/m/Beam/gfKHFOwE0uETxae?subj=Re+why+Source+validate+is+not+declared+to+throw+any+exception
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Resolved] (BEAM-2123) Passing potential null pointer to encode() in StructuredCoder#structuralValue
[
https://issues.apache.org/jira/browse/BEAM-2123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu resolved BEAM-2123.
--
Resolution: Not A Problem
Fix Version/s: Not applicable
> Passing potential null pointer to encode() in StructuredCoder#structuralValue
> -
>
> Key: BEAM-2123
> URL: https://issues.apache.org/jira/browse/BEAM-2123
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Reporter: Ted Yu
>Assignee: Thomas Groh
>Priority: Minor
> Fix For: Not applicable
>
>
> {code}
> public Object structuralValue(T value) {
> if (value != null && consistentWithEquals()) {
> return value;
> } else {
> try {
> ByteArrayOutputStream os = new ByteArrayOutputStream();
> encode(value, os, Context.OUTER);
> {code}
> If value is null, encode() would throw CoderException (I checked
> ByteArrayCoder and KvCoder) which would be caught and converted to
> IllegalArgumentException.
> Looks like structuralValue() can check null value directly and throw
> CoderException. This would result in clearer exception.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (BEAM-2123) Passing potential null pointer to encode() in StructuredCoder#structuralValue
[
https://issues.apache.org/jira/browse/BEAM-2123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15990529#comment-15990529
]
Ted Yu commented on BEAM-2123:
--
Thanks for the confirmation (I didn't see structuralValue() either)
> Passing potential null pointer to encode() in StructuredCoder#structuralValue
> -
>
> Key: BEAM-2123
> URL: https://issues.apache.org/jira/browse/BEAM-2123
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Reporter: Ted Yu
>Assignee: Thomas Groh
>Priority: Minor
>
> {code}
> public Object structuralValue(T value) {
> if (value != null && consistentWithEquals()) {
> return value;
> } else {
> try {
> ByteArrayOutputStream os = new ByteArrayOutputStream();
> encode(value, os, Context.OUTER);
> {code}
> If value is null, encode() would throw CoderException (I checked
> ByteArrayCoder and KvCoder) which would be caught and converted to
> IllegalArgumentException.
> Looks like structuralValue() can check null value directly and throw
> CoderException. This would result in clearer exception.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (BEAM-2123) Passing potential null pointer to encode() in StructuredCoder#structuralValue
[
https://issues.apache.org/jira/browse/BEAM-2123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15990526#comment-15990526
]
Ted Yu commented on BEAM-2123:
--
I looked at
./sdks/java/core/src/test/java/org/apache/beam/sdk/coders/StructuredCoderTest.java
but didn't see NullableBooleanCoder
> Passing potential null pointer to encode() in StructuredCoder#structuralValue
> -
>
> Key: BEAM-2123
> URL: https://issues.apache.org/jira/browse/BEAM-2123
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Reporter: Ted Yu
>Assignee: Thomas Groh
>Priority: Minor
>
> {code}
> public Object structuralValue(T value) {
> if (value != null && consistentWithEquals()) {
> return value;
> } else {
> try {
> ByteArrayOutputStream os = new ByteArrayOutputStream();
> encode(value, os, Context.OUTER);
> {code}
> If value is null, encode() would throw CoderException (I checked
> ByteArrayCoder and KvCoder) which would be caught and converted to
> IllegalArgumentException.
> Looks like structuralValue() can check null value directly and throw
> CoderException. This would result in clearer exception.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (BEAM-2123) Passing potential null pointer to encode() in StructuredCoder#structuralValue
[
https://issues.apache.org/jira/browse/BEAM-2123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15990387#comment-15990387
]
Ted Yu commented on BEAM-2123:
--
I am interested in seeing a counter example where null value produces
meaningful output.
> Passing potential null pointer to encode() in StructuredCoder#structuralValue
> -
>
> Key: BEAM-2123
> URL: https://issues.apache.org/jira/browse/BEAM-2123
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Reporter: Ted Yu
>Assignee: Thomas Groh
>Priority: Minor
>
> {code}
> public Object structuralValue(T value) {
> if (value != null && consistentWithEquals()) {
> return value;
> } else {
> try {
> ByteArrayOutputStream os = new ByteArrayOutputStream();
> encode(value, os, Context.OUTER);
> {code}
> If value is null, encode() would throw CoderException (I checked
> ByteArrayCoder and KvCoder) which would be caught and converted to
> IllegalArgumentException.
> Looks like structuralValue() can check null value directly and throw
> CoderException. This would result in clearer exception.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (BEAM-1773) Consider allowing Source#validate() to throw exception
[
https://issues.apache.org/jira/browse/BEAM-1773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated BEAM-1773:
-
Attachment: (was: 1773.v4.patch)
> Consider allowing Source#validate() to throw exception
> --
>
> Key: BEAM-1773
> URL: https://issues.apache.org/jira/browse/BEAM-1773
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 1773.v4.patch, beam-1773.v1.patch, beam-1773.v2.patch
>
>
> In HDFSFileSource.java :
> {code}
> @Override
> public void validate() {
> ...
> } catch (IOException | InterruptedException e) {
> throw new RuntimeException(e);
> }
> {code}
> Source#validate() should be allowed to throw exception so that we don't
> resort to using RuntimeException.
> Here was related thread on mailing list:
> http://search-hadoop.com/m/Beam/gfKHFOwE0uETxae?subj=Re+why+Source+validate+is+not+declared+to+throw+any+exception
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (BEAM-1773) Consider allowing Source#validate() to throw exception
[
https://issues.apache.org/jira/browse/BEAM-1773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated BEAM-1773:
-
Attachment: 1773.v4.patch
> Consider allowing Source#validate() to throw exception
> --
>
> Key: BEAM-1773
> URL: https://issues.apache.org/jira/browse/BEAM-1773
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 1773.v4.patch, beam-1773.v1.patch, beam-1773.v2.patch
>
>
> In HDFSFileSource.java :
> {code}
> @Override
> public void validate() {
> ...
> } catch (IOException | InterruptedException e) {
> throw new RuntimeException(e);
> }
> {code}
> Source#validate() should be allowed to throw exception so that we don't
> resort to using RuntimeException.
> Here was related thread on mailing list:
> http://search-hadoop.com/m/Beam/gfKHFOwE0uETxae?subj=Re+why+Source+validate+is+not+declared+to+throw+any+exception
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (BEAM-1773) Consider allowing Source#validate() to throw exception
[
https://issues.apache.org/jira/browse/BEAM-1773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated BEAM-1773:
-
Attachment: 1773.v4.patch
See if patch v4 is on right track.
> Consider allowing Source#validate() to throw exception
> --
>
> Key: BEAM-1773
> URL: https://issues.apache.org/jira/browse/BEAM-1773
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 1773.v4.patch, beam-1773.v1.patch, beam-1773.v2.patch
>
>
> In HDFSFileSource.java :
> {code}
> @Override
> public void validate() {
> ...
> } catch (IOException | InterruptedException e) {
> throw new RuntimeException(e);
> }
> {code}
> Source#validate() should be allowed to throw exception so that we don't
> resort to using RuntimeException.
> Here was related thread on mailing list:
> http://search-hadoop.com/m/Beam/gfKHFOwE0uETxae?subj=Re+why+Source+validate+is+not+declared+to+throw+any+exception
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (BEAM-1773) Consider allowing Source#validate() to throw exception
[
https://issues.apache.org/jira/browse/BEAM-1773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15990102#comment-15990102
]
Ted Yu commented on BEAM-1773:
--
Should ValidationException class be created in org.apache.beam.sdk package ?
> Consider allowing Source#validate() to throw exception
> --
>
> Key: BEAM-1773
> URL: https://issues.apache.org/jira/browse/BEAM-1773
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: beam-1773.v1.patch, beam-1773.v2.patch
>
>
> In HDFSFileSource.java :
> {code}
> @Override
> public void validate() {
> ...
> } catch (IOException | InterruptedException e) {
> throw new RuntimeException(e);
> }
> {code}
> Source#validate() should be allowed to throw exception so that we don't
> resort to using RuntimeException.
> Here was related thread on mailing list:
> http://search-hadoop.com/m/Beam/gfKHFOwE0uETxae?subj=Re+why+Source+validate+is+not+declared+to+throw+any+exception
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (BEAM-1773) Consider allowing Source#validate() to throw exception
[
https://issues.apache.org/jira/browse/BEAM-1773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15990101#comment-15990101
]
Ted Yu commented on BEAM-1773:
--
w.r.t. ValidationException, I assume you mean creating ValidationException
class (subclass of IOException) which is declared by the relevant methods.
w.r.t. InterruptedException, allow me to refer to
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java
which has the following:
{code}
} catch (InterruptedException e) {
throw (IOException)new InterruptedIOException("Interrupted: action="
+ action + ", retry policy=" + connectionRetryPolicy).initCause(e);
}
{code}
> Consider allowing Source#validate() to throw exception
> --
>
> Key: BEAM-1773
> URL: https://issues.apache.org/jira/browse/BEAM-1773
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: beam-1773.v1.patch, beam-1773.v2.patch
>
>
> In HDFSFileSource.java :
> {code}
> @Override
> public void validate() {
> ...
> } catch (IOException | InterruptedException e) {
> throw new RuntimeException(e);
> }
> {code}
> Source#validate() should be allowed to throw exception so that we don't
> resort to using RuntimeException.
> Here was related thread on mailing list:
> http://search-hadoop.com/m/Beam/gfKHFOwE0uETxae?subj=Re+why+Source+validate+is+not+declared+to+throw+any+exception
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Comment Edited] (BEAM-1773) Consider allowing Source#validate() to throw exception
[
https://issues.apache.org/jira/browse/BEAM-1773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15990101#comment-15990101
]
Ted Yu edited comment on BEAM-1773 at 4/30/17 2:23 AM:
---
w.r.t. ValidationException, I assume you mean creating ValidationException
class (subclass of IOException) which is declared to be thrown by the relevant
methods.
w.r.t. InterruptedException, allow me to refer to
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java
which has the following:
{code}
} catch (InterruptedException e) {
throw (IOException)new InterruptedIOException("Interrupted: action="
+ action + ", retry policy=" + connectionRetryPolicy).initCause(e);
}
{code}
was (Author: yuzhih...@gmail.com):
w.r.t. ValidationException, I assume you mean creating ValidationException
class (subclass of IOException) which is declared by the relevant methods.
w.r.t. InterruptedException, allow me to refer to
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java
which has the following:
{code}
} catch (InterruptedException e) {
throw (IOException)new InterruptedIOException("Interrupted: action="
+ action + ", retry policy=" + connectionRetryPolicy).initCause(e);
}
{code}
> Consider allowing Source#validate() to throw exception
> --
>
> Key: BEAM-1773
> URL: https://issues.apache.org/jira/browse/BEAM-1773
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: beam-1773.v1.patch, beam-1773.v2.patch
>
>
> In HDFSFileSource.java :
> {code}
> @Override
> public void validate() {
> ...
> } catch (IOException | InterruptedException e) {
> throw new RuntimeException(e);
> }
> {code}
> Source#validate() should be allowed to throw exception so that we don't
> resort to using RuntimeException.
> Here was related thread on mailing list:
> http://search-hadoop.com/m/Beam/gfKHFOwE0uETxae?subj=Re+why+Source+validate+is+not+declared+to+throw+any+exception
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (BEAM-2123) Passing potential null pointer to encode() in StructuredCoder#structuralValue
Ted Yu created BEAM-2123:
Summary: Passing potential null pointer to encode() in
StructuredCoder#structuralValue
Key: BEAM-2123
URL: https://issues.apache.org/jira/browse/BEAM-2123
Project: Beam
Issue Type: Bug
Components: sdk-java-core
Reporter: Ted Yu
Assignee: Davor Bonaci
Priority: Minor
{code}
public Object structuralValue(T value) {
if (value != null && consistentWithEquals()) {
return value;
} else {
try {
ByteArrayOutputStream os = new ByteArrayOutputStream();
encode(value, os, Context.OUTER);
{code}
If value is null, encode() would throw CoderException (I checked ByteArrayCoder
and KvCoder) which would be caught and converted to IllegalArgumentException.
Looks like structuralValue() can check null value directly and throw
CoderException. This would result in clearer exception.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Comment Edited] (BEAM-1773) Consider allowing Source#validate() to throw exception
[
https://issues.apache.org/jira/browse/BEAM-1773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15957130#comment-15957130
]
Ted Yu edited comment on BEAM-1773 at 4/29/17 11:22 PM:
[~kenn] [~j...@nanthrax.net] :
Please kindly provide your feedback.
was (Author: yuzhih...@gmail.com):
[~kenn] [~j...@nanthrax.net]:
Please kindly provide your feedback.
> Consider allowing Source#validate() to throw exception
> --
>
> Key: BEAM-1773
> URL: https://issues.apache.org/jira/browse/BEAM-1773
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: beam-1773.v1.patch, beam-1773.v2.patch
>
>
> In HDFSFileSource.java :
> {code}
> @Override
> public void validate() {
> ...
> } catch (IOException | InterruptedException e) {
> throw new RuntimeException(e);
> }
> {code}
> Source#validate() should be allowed to throw exception so that we don't
> resort to using RuntimeException.
> Here was related thread on mailing list:
> http://search-hadoop.com/m/Beam/gfKHFOwE0uETxae?subj=Re+why+Source+validate+is+not+declared+to+throw+any+exception
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (BEAM-1831) Checking of containment in createdTables may have race condition in StreamingWriteFn
[
https://issues.apache.org/jira/browse/BEAM-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15989162#comment-15989162
]
Ted Yu commented on BEAM-1831:
--
Thanks for checking.
Maybe comment should be added before the second check for this effect.
> Checking of containment in createdTables may have race condition in
> StreamingWriteFn
>
>
> Key: BEAM-1831
> URL: https://issues.apache.org/jira/browse/BEAM-1831
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-gcp
>Reporter: Ted Yu
>Assignee: Reuven Lax
>Priority: Minor
>
> {code}
> public TableReference getOrCreateTable(BigQueryOptions options, String
> tableSpec)
> throws InterruptedException, IOException {
> TableReference tableReference = BigQueryHelpers.parseTableSpec(tableSpec);
> if (createDisposition != createDisposition.CREATE_NEVER
> && !createdTables.contains(tableSpec)) {
> synchronized (createdTables) {
> // Another thread may have succeeded in creating the table in the
> meanwhile, so
> // check again. This check isn't needed for correctness, but we add
> it to prevent
> // every thread from attempting a create and overwhelming our
> BigQuery quota.
> DatasetService datasetService = bqServices.getDatasetService(options);
> if (!createdTables.contains(tableSpec)) {
> {code}
> The first createdTables.contains() check is outside synchronized block.
> At least createdTables should be declared volatile for the double checked
> locking to work.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Closed] (BEAM-1745) Unintended unboxing of potential null pointer in AutoValue_ElasticsearchIO_Write
[
https://issues.apache.org/jira/browse/BEAM-1745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu closed BEAM-1745.
Resolution: Not A Problem
Fix Version/s: Not applicable
> Unintended unboxing of potential null pointer in
> AutoValue_ElasticsearchIO_Write
>
>
> Key: BEAM-1745
> URL: https://issues.apache.org/jira/browse/BEAM-1745
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-extensions
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Fix For: Not applicable
>
>
> {code}
> if (maxBatchSize == null) {
> missing += " maxBatchSize";
> }
> ...
> return new AutoValue_ElasticsearchIO_Write(
> this.connectionConfiguration,
> this.maxBatchSize,
> this.maxBatchSizeBytes);
> {code}
> If maxBatchSize is null, it would be unboxed at the time
> AutoValue_ElasticsearchIO_Write is constructed.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Comment Edited] (BEAM-1773) Consider allowing Source#validate() to throw exception
[
https://issues.apache.org/jira/browse/BEAM-1773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15957130#comment-15957130
]
Ted Yu edited comment on BEAM-1773 at 4/20/17 8:22 PM:
---
[~kenn] [~j...@nanthrax.net]:
Please kindly provide your feedback.
was (Author: yuzhih...@gmail.com):
[~kenn] [~j...@nanthrax.net]:
Please kindly provide your feedback.
> Consider allowing Source#validate() to throw exception
> --
>
> Key: BEAM-1773
> URL: https://issues.apache.org/jira/browse/BEAM-1773
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: beam-1773.v1.patch, beam-1773.v2.patch
>
>
> In HDFSFileSource.java :
> {code}
> @Override
> public void validate() {
> ...
> } catch (IOException | InterruptedException e) {
> throw new RuntimeException(e);
> }
> {code}
> Source#validate() should be allowed to throw exception so that we don't
> resort to using RuntimeException.
> Here was related thread on mailing list:
> http://search-hadoop.com/m/Beam/gfKHFOwE0uETxae?subj=Re+why+Source+validate+is+not+declared+to+throw+any+exception
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (BEAM-1831) Checking of containment in createdTables may have race condition in StreamingWriteFn
[
https://issues.apache.org/jira/browse/BEAM-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated BEAM-1831:
-
Description:
{code}
public TableReference getOrCreateTable(BigQueryOptions options, String
tableSpec)
throws InterruptedException, IOException {
TableReference tableReference = BigQueryHelpers.parseTableSpec(tableSpec);
if (createDisposition != createDisposition.CREATE_NEVER
&& !createdTables.contains(tableSpec)) {
synchronized (createdTables) {
// Another thread may have succeeded in creating the table in the
meanwhile, so
// check again. This check isn't needed for correctness, but we add it
to prevent
// every thread from attempting a create and overwhelming our BigQuery
quota.
DatasetService datasetService = bqServices.getDatasetService(options);
if (!createdTables.contains(tableSpec)) {
{code}
The first createdTables.contains() check is outside synchronized block.
At least createdTables should be declared volatile for the double checked
locking to work.
was:
{code}
public TableReference getOrCreateTable(BigQueryOptions options, String
tableSpec)
throws InterruptedException, IOException {
TableReference tableReference = BigQueryHelpers.parseTableSpec(tableSpec);
if (createDisposition != createDisposition.CREATE_NEVER
&& !createdTables.contains(tableSpec)) {
synchronized (createdTables) {
// Another thread may have succeeded in creating the table in the
meanwhile, so
// check again. This check isn't needed for correctness, but we add it
to prevent
// every thread from attempting a create and overwhelming our BigQuery
quota.
DatasetService datasetService = bqServices.getDatasetService(options);
if (!createdTables.contains(tableSpec)) {
{code}
The first createdTables.contains() check is outside synchronized block.
At least createdTables should be declared volatile for the double checked
locking to work.
> Checking of containment in createdTables may have race condition in
> StreamingWriteFn
>
>
> Key: BEAM-1831
> URL: https://issues.apache.org/jira/browse/BEAM-1831
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-gcp
>Reporter: Ted Yu
>Assignee: Davor Bonaci
>Priority: Minor
>
> {code}
> public TableReference getOrCreateTable(BigQueryOptions options, String
> tableSpec)
> throws InterruptedException, IOException {
> TableReference tableReference = BigQueryHelpers.parseTableSpec(tableSpec);
> if (createDisposition != createDisposition.CREATE_NEVER
> && !createdTables.contains(tableSpec)) {
> synchronized (createdTables) {
> // Another thread may have succeeded in creating the table in the
> meanwhile, so
> // check again. This check isn't needed for correctness, but we add
> it to prevent
> // every thread from attempting a create and overwhelming our
> BigQuery quota.
> DatasetService datasetService = bqServices.getDatasetService(options);
> if (!createdTables.contains(tableSpec)) {
> {code}
> The first createdTables.contains() check is outside synchronized block.
> At least createdTables should be declared volatile for the double checked
> locking to work.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (BEAM-1962) Connection should be closed in case start() throws exception
[
https://issues.apache.org/jira/browse/BEAM-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated BEAM-1962:
-
Description:
In JmsIO#start() :
{code}
try {
Connection connection;
if (spec.getUsername() != null) {
connection =
connectionFactory.createConnection(spec.getUsername(),
spec.getPassword());
} else {
connection = connectionFactory.createConnection();
}
connection.start();
this.connection = connection;
} catch (Exception e) {
throw new IOException("Error connecting to JMS", e);
}
{code}
If start() throws exception, connection should be closed.
was:
In JmsIO#start() :
{code}
try {
Connection connection;
if (spec.getUsername() != null) {
connection =
connectionFactory.createConnection(spec.getUsername(),
spec.getPassword());
} else {
connection = connectionFactory.createConnection();
}
connection.start();
this.connection = connection;
} catch (Exception e) {
throw new IOException("Error connecting to JMS", e);
}
{code}
If start() throws exception, connection should be closed.
> Connection should be closed in case start() throws exception
>
>
> Key: BEAM-1962
> URL: https://issues.apache.org/jira/browse/BEAM-1962
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-extensions
>Reporter: Ted Yu
>Assignee: Jean-Baptiste Onofré
>Priority: Minor
>
> In JmsIO#start() :
> {code}
> try {
> Connection connection;
> if (spec.getUsername() != null) {
> connection =
> connectionFactory.createConnection(spec.getUsername(),
> spec.getPassword());
> } else {
> connection = connectionFactory.createConnection();
> }
> connection.start();
> this.connection = connection;
> } catch (Exception e) {
> throw new IOException("Error connecting to JMS", e);
> }
> {code}
> If start() throws exception, connection should be closed.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (BEAM-1962) Connection should be closed in case start() throws exception
Ted Yu created BEAM-1962:
Summary: Connection should be closed in case start() throws
exception
Key: BEAM-1962
URL: https://issues.apache.org/jira/browse/BEAM-1962
Project: Beam
Issue Type: Bug
Components: sdk-java-extensions
Reporter: Ted Yu
Assignee: Davor Bonaci
Priority: Minor
In JmsIO#start() :
{code}
try {
Connection connection;
if (spec.getUsername() != null) {
connection =
connectionFactory.createConnection(spec.getUsername(),
spec.getPassword());
} else {
connection = connectionFactory.createConnection();
}
connection.start();
this.connection = connection;
} catch (Exception e) {
throw new IOException("Error connecting to JMS", e);
}
{code}
If start() throws exception, connection should be closed.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (BEAM-1773) Consider allowing Source#validate() to throw exception
[
https://issues.apache.org/jira/browse/BEAM-1773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated BEAM-1773:
-
Attachment: beam-1773.v2.patch
> Consider allowing Source#validate() to throw exception
> --
>
> Key: BEAM-1773
> URL: https://issues.apache.org/jira/browse/BEAM-1773
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: beam-1773.v1.patch, beam-1773.v2.patch
>
>
> In HDFSFileSource.java :
> {code}
> @Override
> public void validate() {
> ...
> } catch (IOException | InterruptedException e) {
> throw new RuntimeException(e);
> }
> {code}
> Source#validate() should be allowed to throw exception so that we don't
> resort to using RuntimeException.
> Here was related thread on mailing list:
> http://search-hadoop.com/m/Beam/gfKHFOwE0uETxae?subj=Re+why+Source+validate+is+not+declared+to+throw+any+exception
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (BEAM-1950) Access to static field shouldn't be protected by lock on non-static MicrobatchSource.this
Ted Yu created BEAM-1950:
Summary: Access to static field shouldn't be protected by lock on
non-static MicrobatchSource.this
Key: BEAM-1950
URL: https://issues.apache.org/jira/browse/BEAM-1950
Project: Beam
Issue Type: Bug
Components: runner-spark
Reporter: Ted Yu
Assignee: Amit Sela
Priority: Minor
In MicrobatchSource :
{code}
private synchronized void initReaderCache(long readerCacheInterval) {
if (readerCache == null) {
LOG.info("Creating reader cache. Cache interval = " + readerCacheInterval
+ " ms.");
readerCache =
CacheBuilder.newBuilder()
{code}
readerCache is static.
Access to readerCache shouldn't be protected by lock on non-static
MicrobatchSource.this
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Comment Edited] (BEAM-1773) Consider allowing Source#validate() to throw exception
[
https://issues.apache.org/jira/browse/BEAM-1773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15957130#comment-15957130
]
Ted Yu edited comment on BEAM-1773 at 4/12/17 4:43 PM:
---
[~kenn] [~j...@nanthrax.net]:
Please kindly provide your feedback.
was (Author: yuzhih...@gmail.com):
[~kenn] [~j...@nanthrax.net]:
Please provide your feedback.
> Consider allowing Source#validate() to throw exception
> --
>
> Key: BEAM-1773
> URL: https://issues.apache.org/jira/browse/BEAM-1773
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: beam-1773.v1.patch
>
>
> In HDFSFileSource.java :
> {code}
> @Override
> public void validate() {
> ...
> } catch (IOException | InterruptedException e) {
> throw new RuntimeException(e);
> }
> {code}
> Source#validate() should be allowed to throw exception so that we don't
> resort to using RuntimeException.
> Here was related thread on mailing list:
> http://search-hadoop.com/m/Beam/gfKHFOwE0uETxae?subj=Re+why+Source+validate+is+not+declared+to+throw+any+exception
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Closed] (BEAM-1734) Potentially unbox null pointer in AutoValue_BigQueryIO_Write
[
https://issues.apache.org/jira/browse/BEAM-1734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu closed BEAM-1734.
Resolution: Not A Problem
Fix Version/s: Not applicable
> Potentially unbox null pointer in AutoValue_BigQueryIO_Write
>
>
> Key: BEAM-1734
> URL: https://issues.apache.org/jira/browse/BEAM-1734
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-gcp
>Reporter: Ted Yu
>Priority: Minor
> Labels: newbie, starter
> Fix For: Not applicable
>
>
> {code}
> if (validate == null) {
> missing += " validate";
> }
> ...
> return new AutoValue_BigQueryIO_Write(
> this.jsonTableRef,
> this.tableRefFunction,
> this.jsonSchema,
> this.createDisposition,
> this.writeDisposition,
> this.tableDescription,
> this.validate,
> this.bigQueryServices);
> {code}
> If validate is null, AutoValue_BigQueryIO_Write would try to unbox it since
> the parameter is declared as boolean.
> There is similar problem in AutoValue_BigQueryIO_Read, around line 166
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Comment Edited] (BEAM-1773) Consider allowing Source#validate() to throw exception
[
https://issues.apache.org/jira/browse/BEAM-1773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15957130#comment-15957130
]
Ted Yu edited comment on BEAM-1773 at 4/7/17 4:44 PM:
--
[~kenn] [~j...@nanthrax.net]:
Please provide your feedback.
was (Author: yuzhih...@gmail.com):
[~kenn] [~j...@nanthrax.net]:
Please provide your feedback.
> Consider allowing Source#validate() to throw exception
> --
>
> Key: BEAM-1773
> URL: https://issues.apache.org/jira/browse/BEAM-1773
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: beam-1773.v1.patch
>
>
> In HDFSFileSource.java :
> {code}
> @Override
> public void validate() {
> ...
> } catch (IOException | InterruptedException e) {
> throw new RuntimeException(e);
> }
> {code}
> Source#validate() should be allowed to throw exception so that we don't
> resort to using RuntimeException.
> Here was related thread on mailing list:
> http://search-hadoop.com/m/Beam/gfKHFOwE0uETxae?subj=Re+why+Source+validate+is+not+declared+to+throw+any+exception
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (BEAM-1734) Potentially unbox null pointer in AutoValue_BigQueryIO_Write
[
https://issues.apache.org/jira/browse/BEAM-1734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15957626#comment-15957626
]
Ted Yu commented on BEAM-1734:
--
This seems to be caused by defect in codegen for AutoValue annotation.
Can you remind me where the codegen is ?
Thanks
> Potentially unbox null pointer in AutoValue_BigQueryIO_Write
>
>
> Key: BEAM-1734
> URL: https://issues.apache.org/jira/browse/BEAM-1734
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-gcp
>Reporter: Ted Yu
>Priority: Minor
> Labels: newbie, starter
>
> {code}
> if (validate == null) {
> missing += " validate";
> }
> ...
> return new AutoValue_BigQueryIO_Write(
> this.jsonTableRef,
> this.tableRefFunction,
> this.jsonSchema,
> this.createDisposition,
> this.writeDisposition,
> this.tableDescription,
> this.validate,
> this.bigQueryServices);
> {code}
> If validate is null, AutoValue_BigQueryIO_Write would try to unbox it since
> the parameter is declared as boolean.
> There is similar problem in AutoValue_BigQueryIO_Read, around line 166
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (BEAM-1773) Consider allowing Source#validate() to throw exception
[
https://issues.apache.org/jira/browse/BEAM-1773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15957130#comment-15957130
]
Ted Yu commented on BEAM-1773:
--
[~kenn] [~j...@nanthrax.net]:
Please provide your feedback.
> Consider allowing Source#validate() to throw exception
> --
>
> Key: BEAM-1773
> URL: https://issues.apache.org/jira/browse/BEAM-1773
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: beam-1773.v1.patch
>
>
> In HDFSFileSource.java :
> {code}
> @Override
> public void validate() {
> ...
> } catch (IOException | InterruptedException e) {
> throw new RuntimeException(e);
> }
> {code}
> Source#validate() should be allowed to throw exception so that we don't
> resort to using RuntimeException.
> Here was related thread on mailing list:
> http://search-hadoop.com/m/Beam/gfKHFOwE0uETxae?subj=Re+why+Source+validate+is+not+declared+to+throw+any+exception
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (BEAM-1831) Checking of containment in createdTables may have race condition in StreamingWriteFn
[
https://issues.apache.org/jira/browse/BEAM-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated BEAM-1831:
-
Description:
{code}
public TableReference getOrCreateTable(BigQueryOptions options, String
tableSpec)
throws InterruptedException, IOException {
TableReference tableReference = BigQueryHelpers.parseTableSpec(tableSpec);
if (createDisposition != createDisposition.CREATE_NEVER
&& !createdTables.contains(tableSpec)) {
synchronized (createdTables) {
// Another thread may have succeeded in creating the table in the
meanwhile, so
// check again. This check isn't needed for correctness, but we add it
to prevent
// every thread from attempting a create and overwhelming our BigQuery
quota.
DatasetService datasetService = bqServices.getDatasetService(options);
if (!createdTables.contains(tableSpec)) {
{code}
The first createdTables.contains() check is outside synchronized block.
At least createdTables should be declared volatile for the double checked
locking to work.
was:
{code}
public TableReference getOrCreateTable(BigQueryOptions options, String
tableSpec)
throws InterruptedException, IOException {
TableReference tableReference = BigQueryHelpers.parseTableSpec(tableSpec);
if (createDisposition != createDisposition.CREATE_NEVER
&& !createdTables.contains(tableSpec)) {
synchronized (createdTables) {
// Another thread may have succeeded in creating the table in the
meanwhile, so
// check again. This check isn't needed for correctness, but we add it
to prevent
// every thread from attempting a create and overwhelming our BigQuery
quota.
DatasetService datasetService = bqServices.getDatasetService(options);
if (!createdTables.contains(tableSpec)) {
{code}
The first createdTables.contains() check is outside synchronized block.
At least createdTables should be declared volatile for the double checked
locking to work.
> Checking of containment in createdTables may have race condition in
> StreamingWriteFn
>
>
> Key: BEAM-1831
> URL: https://issues.apache.org/jira/browse/BEAM-1831
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-gcp
>Reporter: Ted Yu
>Assignee: Davor Bonaci
>Priority: Minor
>
> {code}
> public TableReference getOrCreateTable(BigQueryOptions options, String
> tableSpec)
> throws InterruptedException, IOException {
> TableReference tableReference = BigQueryHelpers.parseTableSpec(tableSpec);
> if (createDisposition != createDisposition.CREATE_NEVER
> && !createdTables.contains(tableSpec)) {
> synchronized (createdTables) {
> // Another thread may have succeeded in creating the table in the
> meanwhile, so
> // check again. This check isn't needed for correctness, but we add
> it to prevent
> // every thread from attempting a create and overwhelming our
> BigQuery quota.
> DatasetService datasetService = bqServices.getDatasetService(options);
> if (!createdTables.contains(tableSpec)) {
> {code}
> The first createdTables.contains() check is outside synchronized block.
> At least createdTables should be declared volatile for the double checked
> locking to work.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Comment Edited] (BEAM-1832) Potentially unclosed OutputStream in ApexYarnLauncher
[
https://issues.apache.org/jira/browse/BEAM-1832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15947383#comment-15947383
]
Ted Yu edited comment on BEAM-1832 at 4/4/17 9:02 PM:
--
In the same class:
{code}
public static List getYarnDeployDependencies() throws IOException {
InputStream dependencyTree =
ApexRunner.class.getResourceAsStream("dependency-tree");
{code}
dependencyTree should be closed even if exception is thrown in the while loop.
was (Author: yuzhih...@gmail.com):
In the same class:
{code}
public static List getYarnDeployDependencies() throws IOException {
InputStream dependencyTree =
ApexRunner.class.getResourceAsStream("dependency-tree");
{code}
dependencyTree should be closed even if exception is thrown in the while loop.
> Potentially unclosed OutputStream in ApexYarnLauncher
> -
>
> Key: BEAM-1832
> URL: https://issues.apache.org/jira/browse/BEAM-1832
> Project: Beam
> Issue Type: Bug
> Components: runner-apex
>Reporter: Ted Yu
>Priority: Minor
>
> Here is an example from createJar():
> {code}
> final OutputStream out =
> Files.newOutputStream(zipfs.getPath(JarFile.MANIFEST_NAME));
> if (!manifestFile.exists()) {
> new Manifest().write(out);
> } else {
> FileUtils.copyFile(manifestFile, out);
> }
> out.close();
> {code}
> If FileUtils.copyFile throws IOException, out would be left unclosed.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (BEAM-1773) Consider allowing Source#validate() to throw exception
[
https://issues.apache.org/jira/browse/BEAM-1773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15953890#comment-15953890
]
Ted Yu commented on BEAM-1773:
--
[~kenn] [~j...@nanthrax.net]:
Can you comment on the above proposal ?
> Consider allowing Source#validate() to throw exception
> --
>
> Key: BEAM-1773
> URL: https://issues.apache.org/jira/browse/BEAM-1773
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: beam-1773.v1.patch
>
>
> In HDFSFileSource.java :
> {code}
> @Override
> public void validate() {
> ...
> } catch (IOException | InterruptedException e) {
> throw new RuntimeException(e);
> }
> {code}
> Source#validate() should be allowed to throw exception so that we don't
> resort to using RuntimeException.
> Here was related thread on mailing list:
> http://search-hadoop.com/m/Beam/gfKHFOwE0uETxae?subj=Re+why+Source+validate+is+not+declared+to+throw+any+exception
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Resolved] (BEAM-1760) Potential null dereference in HDFSFileSink#doFinalize
[
https://issues.apache.org/jira/browse/BEAM-1760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu resolved BEAM-1760.
--
Resolution: Later
Assignee: Davor Bonaci (was: Ted Yu)
Fix Version/s: Not applicable
> Potential null dereference in HDFSFileSink#doFinalize
> -
>
> Key: BEAM-1760
> URL: https://issues.apache.org/jira/browse/BEAM-1760
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Reporter: Ted Yu
>Assignee: Davor Bonaci
>Priority: Minor
> Fix For: Not applicable
>
>
> Here is related code:
> {code}
> for (FileStatus s : statuses) {
> String name = s.getPath().getName();
> int pos = name.indexOf('.');
> String ext = pos > 0 ? name.substring(pos) : "";
> fs.rename(
> s.getPath(),
> new Path(s.getPath().getParent(), String.format("part-r-%05d%s",
> i, ext)));
> i++;
> }
> }
> {code}
> We should check whether s.getPath().getParent() is null.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (BEAM-1773) Consider allowing Source#validate() to throw exception
[
https://issues.apache.org/jira/browse/BEAM-1773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated BEAM-1773:
-
Attachment: (was: beam-1773.v1.patch)
> Consider allowing Source#validate() to throw exception
> --
>
> Key: BEAM-1773
> URL: https://issues.apache.org/jira/browse/BEAM-1773
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: beam-1773.v1.patch
>
>
> In HDFSFileSource.java :
> {code}
> @Override
> public void validate() {
> ...
> } catch (IOException | InterruptedException e) {
> throw new RuntimeException(e);
> }
> {code}
> Source#validate() should be allowed to throw exception so that we don't
> resort to using RuntimeException.
> Here was related thread on mailing list:
> http://search-hadoop.com/m/Beam/gfKHFOwE0uETxae?subj=Re+why+Source+validate+is+not+declared+to+throw+any+exception
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (BEAM-1773) Consider allowing Source#validate() to throw exception
[
https://issues.apache.org/jira/browse/BEAM-1773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated BEAM-1773:
-
Attachment: beam-1773.v1.patch
> Consider allowing Source#validate() to throw exception
> --
>
> Key: BEAM-1773
> URL: https://issues.apache.org/jira/browse/BEAM-1773
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: beam-1773.v1.patch
>
>
> In HDFSFileSource.java :
> {code}
> @Override
> public void validate() {
> ...
> } catch (IOException | InterruptedException e) {
> throw new RuntimeException(e);
> }
> {code}
> Source#validate() should be allowed to throw exception so that we don't
> resort to using RuntimeException.
> Here was related thread on mailing list:
> http://search-hadoop.com/m/Beam/gfKHFOwE0uETxae?subj=Re+why+Source+validate+is+not+declared+to+throw+any+exception
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (BEAM-1773) Consider allowing Source#validate() to throw exception
[
https://issues.apache.org/jira/browse/BEAM-1773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15953809#comment-15953809
]
Ted Yu commented on BEAM-1773:
--
I want to get some opinion on the method signature for PTransform#expand().
Is the following acceptable ?
{code}
public abstract OutputT expand(InputT input) throws IOException;
{code}
> Consider allowing Source#validate() to throw exception
> --
>
> Key: BEAM-1773
> URL: https://issues.apache.org/jira/browse/BEAM-1773
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: beam-1773.v1.patch
>
>
> In HDFSFileSource.java :
> {code}
> @Override
> public void validate() {
> ...
> } catch (IOException | InterruptedException e) {
> throw new RuntimeException(e);
> }
> {code}
> Source#validate() should be allowed to throw exception so that we don't
> resort to using RuntimeException.
> Here was related thread on mailing list:
> http://search-hadoop.com/m/Beam/gfKHFOwE0uETxae?subj=Re+why+Source+validate+is+not+declared+to+throw+any+exception
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (BEAM-1773) Consider allowing Source#validate() to throw exception
[
https://issues.apache.org/jira/browse/BEAM-1773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated BEAM-1773:
-
Attachment: beam-1773.v1.patch
> Consider allowing Source#validate() to throw exception
> --
>
> Key: BEAM-1773
> URL: https://issues.apache.org/jira/browse/BEAM-1773
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Ted Yu
>Assignee: Jean-Baptiste Onofré
> Attachments: beam-1773.v1.patch
>
>
> In HDFSFileSource.java :
> {code}
> @Override
> public void validate() {
> ...
> } catch (IOException | InterruptedException e) {
> throw new RuntimeException(e);
> }
> {code}
> Source#validate() should be allowed to throw exception so that we don't
> resort to using RuntimeException.
> Here was related thread on mailing list:
> http://search-hadoop.com/m/Beam/gfKHFOwE0uETxae?subj=Re+why+Source+validate+is+not+declared+to+throw+any+exception
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (BEAM-1773) Consider allowing Source#validate() to throw exception
[
https://issues.apache.org/jira/browse/BEAM-1773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu reassigned BEAM-1773:
Assignee: Ted Yu (was: Jean-Baptiste Onofré)
> Consider allowing Source#validate() to throw exception
> --
>
> Key: BEAM-1773
> URL: https://issues.apache.org/jira/browse/BEAM-1773
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: beam-1773.v1.patch
>
>
> In HDFSFileSource.java :
> {code}
> @Override
> public void validate() {
> ...
> } catch (IOException | InterruptedException e) {
> throw new RuntimeException(e);
> }
> {code}
> Source#validate() should be allowed to throw exception so that we don't
> resort to using RuntimeException.
> Here was related thread on mailing list:
> http://search-hadoop.com/m/Beam/gfKHFOwE0uETxae?subj=Re+why+Source+validate+is+not+declared+to+throw+any+exception
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (BEAM-1760) Potential null dereference in HDFSFileSink#doFinalize
[
https://issues.apache.org/jira/browse/BEAM-1760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15953657#comment-15953657
]
Ted Yu commented on BEAM-1760:
--
[~davor]:
Mind taking a look ?
> Potential null dereference in HDFSFileSink#doFinalize
> -
>
> Key: BEAM-1760
> URL: https://issues.apache.org/jira/browse/BEAM-1760
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
>
> Here is related code:
> {code}
> for (FileStatus s : statuses) {
> String name = s.getPath().getName();
> int pos = name.indexOf('.');
> String ext = pos > 0 ? name.substring(pos) : "";
> fs.rename(
> s.getPath(),
> new Path(s.getPath().getParent(), String.format("part-r-%05d%s",
> i, ext)));
> i++;
> }
> }
> {code}
> We should check whether s.getPath().getParent() is null.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Comment Edited] (BEAM-1773) Consider allowing Source#validate() to throw exception
[
https://issues.apache.org/jira/browse/BEAM-1773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15952662#comment-15952662
]
Ted Yu edited comment on BEAM-1773 at 4/3/17 3:13 PM:
--
[~j...@nanthrax.net]:
If you are busy, I can work on this.
was (Author: yuzhih...@gmail.com):
JB:
If you are busy, I can work on this.
> Consider allowing Source#validate() to throw exception
> --
>
> Key: BEAM-1773
> URL: https://issues.apache.org/jira/browse/BEAM-1773
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Ted Yu
>Assignee: Jean-Baptiste Onofré
>
> In HDFSFileSource.java :
> {code}
> @Override
> public void validate() {
> ...
> } catch (IOException | InterruptedException e) {
> throw new RuntimeException(e);
> }
> {code}
> Source#validate() should be allowed to throw exception so that we don't
> resort to using RuntimeException.
> Here was related thread on mailing list:
> http://search-hadoop.com/m/Beam/gfKHFOwE0uETxae?subj=Re+why+Source+validate+is+not+declared+to+throw+any+exception
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (BEAM-1773) Consider allowing Source#validate() to throw exception
[
https://issues.apache.org/jira/browse/BEAM-1773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15952662#comment-15952662
]
Ted Yu commented on BEAM-1773:
--
JB:
If you are busy, I can work on this.
> Consider allowing Source#validate() to throw exception
> --
>
> Key: BEAM-1773
> URL: https://issues.apache.org/jira/browse/BEAM-1773
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Ted Yu
>Assignee: Jean-Baptiste Onofré
>
> In HDFSFileSource.java :
> {code}
> @Override
> public void validate() {
> ...
> } catch (IOException | InterruptedException e) {
> throw new RuntimeException(e);
> }
> {code}
> Source#validate() should be allowed to throw exception so that we don't
> resort to using RuntimeException.
> Here was related thread on mailing list:
> http://search-hadoop.com/m/Beam/gfKHFOwE0uETxae?subj=Re+why+Source+validate+is+not+declared+to+throw+any+exception
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (BEAM-1832) Potentially unclosed OutputStream in ApexYarnLauncher
[
https://issues.apache.org/jira/browse/BEAM-1832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15947383#comment-15947383
]
Ted Yu commented on BEAM-1832:
--
In the same class:
{code}
public static List getYarnDeployDependencies() throws IOException {
InputStream dependencyTree =
ApexRunner.class.getResourceAsStream("dependency-tree");
{code}
dependencyTree should be closed even if exception is thrown in the while loop.
> Potentially unclosed OutputStream in ApexYarnLauncher
> -
>
> Key: BEAM-1832
> URL: https://issues.apache.org/jira/browse/BEAM-1832
> Project: Beam
> Issue Type: Bug
> Components: runner-apex
>Reporter: Ted Yu
>Priority: Minor
>
> Here is an example from createJar():
> {code}
> final OutputStream out =
> Files.newOutputStream(zipfs.getPath(JarFile.MANIFEST_NAME));
> if (!manifestFile.exists()) {
> new Manifest().write(out);
> } else {
> FileUtils.copyFile(manifestFile, out);
> }
> out.close();
> {code}
> If FileUtils.copyFile throws IOException, out would be left unclosed.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (BEAM-1832) Potentially unclosed OutputStream in ApexYarnLauncher
Ted Yu created BEAM-1832:
Summary: Potentially unclosed OutputStream in ApexYarnLauncher
Key: BEAM-1832
URL: https://issues.apache.org/jira/browse/BEAM-1832
Project: Beam
Issue Type: Bug
Components: runner-apex
Reporter: Ted Yu
Priority: Minor
Here is an example from createJar():
{code}
final OutputStream out =
Files.newOutputStream(zipfs.getPath(JarFile.MANIFEST_NAME));
if (!manifestFile.exists()) {
new Manifest().write(out);
} else {
FileUtils.copyFile(manifestFile, out);
}
out.close();
{code}
If FileUtils.copyFile throws IOException, out would be left unclosed.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (BEAM-1831) Checking of containment in createdTables may have race condition in StreamingWriteFn
Ted Yu created BEAM-1831:
Summary: Checking of containment in createdTables may have race
condition in StreamingWriteFn
Key: BEAM-1831
URL: https://issues.apache.org/jira/browse/BEAM-1831
Project: Beam
Issue Type: Bug
Components: runner-dataflow
Reporter: Ted Yu
Assignee: Davor Bonaci
Priority: Minor
{code}
public TableReference getOrCreateTable(BigQueryOptions options, String
tableSpec)
throws InterruptedException, IOException {
TableReference tableReference = BigQueryHelpers.parseTableSpec(tableSpec);
if (createDisposition != createDisposition.CREATE_NEVER
&& !createdTables.contains(tableSpec)) {
synchronized (createdTables) {
// Another thread may have succeeded in creating the table in the
meanwhile, so
// check again. This check isn't needed for correctness, but we add it
to prevent
// every thread from attempting a create and overwhelming our BigQuery
quota.
DatasetService datasetService = bqServices.getDatasetService(options);
if (!createdTables.contains(tableSpec)) {
{code}
The first createdTables.contains() check is outside synchronized block.
At least createdTables should be declared volatile for the double checked
locking to work.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (BEAM-1773) Consider allowing Source#validate() to throw exception
[
https://issues.apache.org/jira/browse/BEAM-1773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946311#comment-15946311
]
Ted Yu commented on BEAM-1773:
--
[~kenn]:
Do you have other comment ?
> Consider allowing Source#validate() to throw exception
> --
>
> Key: BEAM-1773
> URL: https://issues.apache.org/jira/browse/BEAM-1773
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Ted Yu
>Assignee: Jean-Baptiste Onofré
>
> In HDFSFileSource.java :
> {code}
> @Override
> public void validate() {
> ...
> } catch (IOException | InterruptedException e) {
> throw new RuntimeException(e);
> }
> {code}
> Source#validate() should be allowed to throw exception so that we don't
> resort to using RuntimeException.
> Here was related thread on mailing list:
> http://search-hadoop.com/m/Beam/gfKHFOwE0uETxae?subj=Re+why+Source+validate+is+not+declared+to+throw+any+exception
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (BEAM-1760) Potential null dereference in HDFSFileSink#doFinalize
[
https://issues.apache.org/jira/browse/BEAM-1760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946210#comment-15946210
]
Ted Yu commented on BEAM-1760:
--
[~davor]:
Can you take a look at the PR ?
> Potential null dereference in HDFSFileSink#doFinalize
> -
>
> Key: BEAM-1760
> URL: https://issues.apache.org/jira/browse/BEAM-1760
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
>
> Here is related code:
> {code}
> for (FileStatus s : statuses) {
> String name = s.getPath().getName();
> int pos = name.indexOf('.');
> String ext = pos > 0 ? name.substring(pos) : "";
> fs.rename(
> s.getPath(),
> new Path(s.getPath().getParent(), String.format("part-r-%05d%s",
> i, ext)));
> i++;
> }
> }
> {code}
> We should check whether s.getPath().getParent() is null.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (BEAM-1773) Consider allowing Source#validate() to throw exception
[
https://issues.apache.org/jira/browse/BEAM-1773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15935586#comment-15935586
]
Ted Yu commented on BEAM-1773:
--
Unchecked (runtime) exception: Should indicate a coding error. To fix it
someone needs to go change something in the source.
E.g. Passing null into a method when that's invalid.
Checked exception: Indicates an error that can reasonably occur during program
execution and is generally fixed without changing the code.
E.g. Running out of disk space, network error when sending data over the
wire, invalid user input
Using checked exception(s) would give us more flexibility.
> Consider allowing Source#validate() to throw exception
> --
>
> Key: BEAM-1773
> URL: https://issues.apache.org/jira/browse/BEAM-1773
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Ted Yu
>Assignee: Jean-Baptiste Onofré
>
> In HDFSFileSource.java :
> {code}
> @Override
> public void validate() {
> ...
> } catch (IOException | InterruptedException e) {
> throw new RuntimeException(e);
> }
> {code}
> Source#validate() should be allowed to throw exception so that we don't
> resort to using RuntimeException.
> Here was related thread on mailing list:
> http://search-hadoop.com/m/Beam/gfKHFOwE0uETxae?subj=Re+why+Source+validate+is+not+declared+to+throw+any+exception
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (BEAM-1773) Consider allowing Source#validate() to throw exception
[
https://issues.apache.org/jira/browse/BEAM-1773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15934732#comment-15934732
]
Ted Yu commented on BEAM-1773:
--
Feel free to take it.
Thanks.
> Consider allowing Source#validate() to throw exception
> --
>
> Key: BEAM-1773
> URL: https://issues.apache.org/jira/browse/BEAM-1773
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Ted Yu
>Assignee: Jean-Baptiste Onofré
>
> In HDFSFileSource.java :
> {code}
> @Override
> public void validate() {
> ...
> } catch (IOException | InterruptedException e) {
> throw new RuntimeException(e);
> }
> {code}
> Source#validate() should be allowed to throw exception so that we don't
> resort to using RuntimeException.
> Here was related thread on mailing list:
> http://search-hadoop.com/m/Beam/gfKHFOwE0uETxae?subj=Re+why+Source+validate+is+not+declared+to+throw+any+exception
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (BEAM-1773) Consider allowing Source#validate() to throw exception
Ted Yu created BEAM-1773:
Summary: Consider allowing Source#validate() to throw exception
Key: BEAM-1773
URL: https://issues.apache.org/jira/browse/BEAM-1773
Project: Beam
Issue Type: Improvement
Components: sdk-java-core
Reporter: Ted Yu
Assignee: Davor Bonaci
In HDFSFileSource.java :
{code}
@Override
public void validate() {
...
} catch (IOException | InterruptedException e) {
throw new RuntimeException(e);
}
{code}
Source#validate() should be allowed to throw exception so that we don't resort
to using RuntimeException.
Here was related thread on mailing list:
http://search-hadoop.com/m/Beam/gfKHFOwE0uETxae?subj=Re+why+Source+validate+is+not+declared+to+throw+any+exception
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Comment Edited] (BEAM-1761) Unintended unboxing of potential null pointer in AutoValue_HDFSFileSource
[
https://issues.apache.org/jira/browse/BEAM-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15933746#comment-15933746
]
Ted Yu edited comment on BEAM-1761 at 3/20/17 11:01 PM:
I checked most recent master branch.
AutoValue_HDFSFileSink.java is generated code and HDFSFileSource.java doesn't
contain the above quoted snippet :-)
was (Author: yuzhih...@gmail.com):
I checked most recent master branch.
AutoValue_HDFSFileSink.java is gone and HDFSFileSource.java doesn't contain the
above quoted snippet :-)
> Unintended unboxing of potential null pointer in AutoValue_HDFSFileSource
> -
>
> Key: BEAM-1761
> URL: https://issues.apache.org/jira/browse/BEAM-1761
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-extensions
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Fix For: Not applicable
>
>
> {code}
> if (validateSource == null) {
> missing += " validateSource";
> }
> ...
> return new AutoValue_HDFSFileSource(
> this.filepattern,
> this.formatClass,
> this.coder,
> this.inputConverter,
> this.serializableConfiguration,
> this.serializableSplit,
> this.username,
> this.validateSource);
> {code}
> If validateSource is null, it would be unboxed in call to ctor of
> AutoValue_HDFSFileSource
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Resolved] (BEAM-1761) Unintended unboxing of potential null pointer in AutoValue_HDFSFileSource
[
https://issues.apache.org/jira/browse/BEAM-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu resolved BEAM-1761.
--
Resolution: Not A Problem
Fix Version/s: Not applicable
> Unintended unboxing of potential null pointer in AutoValue_HDFSFileSource
> -
>
> Key: BEAM-1761
> URL: https://issues.apache.org/jira/browse/BEAM-1761
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-extensions
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Fix For: Not applicable
>
>
> {code}
> if (validateSource == null) {
> missing += " validateSource";
> }
> ...
> return new AutoValue_HDFSFileSource(
> this.filepattern,
> this.formatClass,
> this.coder,
> this.inputConverter,
> this.serializableConfiguration,
> this.serializableSplit,
> this.username,
> this.validateSource);
> {code}
> If validateSource is null, it would be unboxed in call to ctor of
> AutoValue_HDFSFileSource
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (BEAM-1761) Unintended unboxing of potential null pointer in AutoValue_HDFSFileSource
[
https://issues.apache.org/jira/browse/BEAM-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15933746#comment-15933746
]
Ted Yu commented on BEAM-1761:
--
I checked most recent master branch.
AutoValue_HDFSFileSink.java is gone and HDFSFileSource.java doesn't contain the
above quoted snippet :-)
> Unintended unboxing of potential null pointer in AutoValue_HDFSFileSource
> -
>
> Key: BEAM-1761
> URL: https://issues.apache.org/jira/browse/BEAM-1761
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-extensions
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
>
> {code}
> if (validateSource == null) {
> missing += " validateSource";
> }
> ...
> return new AutoValue_HDFSFileSource(
> this.filepattern,
> this.formatClass,
> this.coder,
> this.inputConverter,
> this.serializableConfiguration,
> this.serializableSplit,
> this.username,
> this.validateSource);
> {code}
> If validateSource is null, it would be unboxed in call to ctor of
> AutoValue_HDFSFileSource
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (BEAM-1761) Unintended unboxing of potential null pointer in AutoValue_HDFSFileSource
[
https://issues.apache.org/jira/browse/BEAM-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15933732#comment-15933732
]
Ted Yu commented on BEAM-1761:
--
Can you review the PR over BEAM-1568 first ?
I can work on this once that is resolved.
> Unintended unboxing of potential null pointer in AutoValue_HDFSFileSource
> -
>
> Key: BEAM-1761
> URL: https://issues.apache.org/jira/browse/BEAM-1761
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-extensions
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
>
> {code}
> if (validateSource == null) {
> missing += " validateSource";
> }
> ...
> return new AutoValue_HDFSFileSource(
> this.filepattern,
> this.formatClass,
> this.coder,
> this.inputConverter,
> this.serializableConfiguration,
> this.serializableSplit,
> this.username,
> this.validateSource);
> {code}
> If validateSource is null, it would be unboxed in call to ctor of
> AutoValue_HDFSFileSource
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Comment Edited] (BEAM-1761) Unintended unboxing of potential null pointer in AutoValue_HDFSFileSource
[
https://issues.apache.org/jira/browse/BEAM-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15933153#comment-15933153
]
Ted Yu edited comment on BEAM-1761 at 3/20/17 5:42 PM:
---
Similar problem exists in AutoValue_HDFSFileSink.java for this.validate
was (Author: yuzhih...@gmail.com):
Similar problem exists in AutoValue_HDFSFileSink.java
> Unintended unboxing of potential null pointer in AutoValue_HDFSFileSource
> -
>
> Key: BEAM-1761
> URL: https://issues.apache.org/jira/browse/BEAM-1761
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Reporter: Ted Yu
>Assignee: Davor Bonaci
>Priority: Minor
>
> {code}
> if (validateSource == null) {
> missing += " validateSource";
> }
> ...
> return new AutoValue_HDFSFileSource(
> this.filepattern,
> this.formatClass,
> this.coder,
> this.inputConverter,
> this.serializableConfiguration,
> this.serializableSplit,
> this.username,
> this.validateSource);
> {code}
> If validateSource is null, it would be unboxed in call to ctor of
> AutoValue_HDFSFileSource
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (BEAM-1761) Unintended unboxing of potential null pointer in AutoValue_HDFSFileSource
[
https://issues.apache.org/jira/browse/BEAM-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15933153#comment-15933153
]
Ted Yu commented on BEAM-1761:
--
Similar problem exists in AutoValue_HDFSFileSink.java
> Unintended unboxing of potential null pointer in AutoValue_HDFSFileSource
> -
>
> Key: BEAM-1761
> URL: https://issues.apache.org/jira/browse/BEAM-1761
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Reporter: Ted Yu
>Assignee: Davor Bonaci
>Priority: Minor
>
> {code}
> if (validateSource == null) {
> missing += " validateSource";
> }
> ...
> return new AutoValue_HDFSFileSource(
> this.filepattern,
> this.formatClass,
> this.coder,
> this.inputConverter,
> this.serializableConfiguration,
> this.serializableSplit,
> this.username,
> this.validateSource);
> {code}
> If validateSource is null, it would be unboxed in call to ctor of
> AutoValue_HDFSFileSource
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (BEAM-1761) Unintended unboxing of potential null pointer in AutoValue_HDFSFileSource
[
https://issues.apache.org/jira/browse/BEAM-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated BEAM-1761:
-
Description:
{code}
if (validateSource == null) {
missing += " validateSource";
}
...
return new AutoValue_HDFSFileSource(
this.filepattern,
this.formatClass,
this.coder,
this.inputConverter,
this.serializableConfiguration,
this.serializableSplit,
this.username,
this.validateSource);
{code}
If validateSource is null, it would be unboxed in call to ctor of
AutoValue_HDFSFileSource
> Unintended unboxing of potential null pointer in AutoValue_HDFSFileSource
> -
>
> Key: BEAM-1761
> URL: https://issues.apache.org/jira/browse/BEAM-1761
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Reporter: Ted Yu
>Assignee: Davor Bonaci
>Priority: Minor
>
> {code}
> if (validateSource == null) {
> missing += " validateSource";
> }
> ...
> return new AutoValue_HDFSFileSource(
> this.filepattern,
> this.formatClass,
> this.coder,
> this.inputConverter,
> this.serializableConfiguration,
> this.serializableSplit,
> this.username,
> this.validateSource);
> {code}
> If validateSource is null, it would be unboxed in call to ctor of
> AutoValue_HDFSFileSource
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (BEAM-1761) Unintended unboxing of potential null pointer in AutoValue_HDFSFileSource
Ted Yu created BEAM-1761: Summary: Unintended unboxing of potential null pointer in AutoValue_HDFSFileSource Key: BEAM-1761 URL: https://issues.apache.org/jira/browse/BEAM-1761 Project: Beam Issue Type: Bug Components: sdk-java-core Reporter: Ted Yu Assignee: Davor Bonaci Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (BEAM-1760) Potential null dereference in HDFSFileSink#doFinalize
Ted Yu created BEAM-1760:
Summary: Potential null dereference in HDFSFileSink#doFinalize
Key: BEAM-1760
URL: https://issues.apache.org/jira/browse/BEAM-1760
Project: Beam
Issue Type: Bug
Components: sdk-java-core
Reporter: Ted Yu
Assignee: Davor Bonaci
Priority: Minor
Here is related code:
{code}
for (FileStatus s : statuses) {
String name = s.getPath().getName();
int pos = name.indexOf('.');
String ext = pos > 0 ? name.substring(pos) : "";
fs.rename(
s.getPath(),
new Path(s.getPath().getParent(), String.format("part-r-%05d%s", i,
ext)));
i++;
}
}
{code}
We should check whether s.getPath().getParent() is null.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (BEAM-1568) Ineffective null check in IsmFormat#structuralValue
[
https://issues.apache.org/jira/browse/BEAM-1568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated BEAM-1568:
-
Description:
{code}
public Object structuralValue(IsmRecord record) throws Exception {
checkState(record.getKeyComponents().size() == keyComponentCoders.size(),
"Expected the number of key component coders %s "
+ "to match the number of key components %s.",
keyComponentCoders.size(), record.getKeyComponents());
if (record != null && consistentWithEquals()) {
{code}
record is de-referenced before the null check.
was:
{code}
public Object structuralValue(IsmRecord record) throws Exception {
checkState(record.getKeyComponents().size() == keyComponentCoders.size(),
"Expected the number of key component coders %s "
+ "to match the number of key components %s.",
keyComponentCoders.size(), record.getKeyComponents());
if (record != null && consistentWithEquals()) {
{code}
record is de-referenced before the null check.
> Ineffective null check in IsmFormat#structuralValue
> ---
>
> Key: BEAM-1568
> URL: https://issues.apache.org/jira/browse/BEAM-1568
> Project: Beam
> Issue Type: Bug
> Components: runner-dataflow
>Reporter: Ted Yu
>Assignee: Davor Bonaci
>Priority: Minor
>
> {code}
> public Object structuralValue(IsmRecord record) throws Exception {
> checkState(record.getKeyComponents().size() ==
> keyComponentCoders.size(),
> "Expected the number of key component coders %s "
> + "to match the number of key components %s.",
> keyComponentCoders.size(), record.getKeyComponents());
> if (record != null && consistentWithEquals()) {
> {code}
> record is de-referenced before the null check.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Resolved] (BEAM-1574) MongoDbIOTest fails due to inability to download mongodb tar ball
[
https://issues.apache.org/jira/browse/BEAM-1574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu resolved BEAM-1574.
--
Resolution: Cannot Reproduce
Fix Version/s: Not applicable
> MongoDbIOTest fails due to inability to download mongodb tar ball
> -
>
> Key: BEAM-1574
> URL: https://issues.apache.org/jira/browse/BEAM-1574
> Project: Beam
> Issue Type: Test
> Components: sdk-java-extensions
>Reporter: Ted Yu
>Assignee: Davor Bonaci
>Priority: Minor
> Fix For: Not applicable
>
>
> In
> ./sdks/java/io/mongodb/target/surefire-reports/org.apache.beam.sdk.io.mongodb.MongoDbIOTest-output.txt
> :
> {code}
> INFO: Starting MongoDB embedded instance on 41937
> Download PRODUCTION:Linux:B64 START
> Feb 28, 2017 5:48:37 PM de.flapdoodle.embed.process.runtime.Starter prepare
> SEVERE: prepare executable
> java.io.IOException: Could not open inputStream for
> http://fastdl.mongodb.org/linux/mongodb-linux-x86_64-3.0.5.tgz
> at
> de.flapdoodle.embed.process.store.Downloader.downloadInputStream(Downloader.java:131)
> at de.flapdoodle.embed.process.store.Downloader.download(Downloader.java:69)
> at
> de.flapdoodle.embed.process.store.ArtifactStore.checkDistribution(ArtifactStore.java:66)
> at
> de.flapdoodle.embed.process.store.ExtractedArtifactStore.checkDistribution(ExtractedArtifactStore.java:60)
> at de.flapdoodle.embed.process.runtime.Starter.prepare(Starter.java:55)
> at
> org.apache.beam.sdk.io.mongodb.MongoDbIOTest.setup(MongoDbIOTest.java:118)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> {code}
> I got the above on:
> Linux 3.10.0-327.28.2.el7.x86_64 #1 SMP Wed Aug 3 11:11:39 UTC 2016 x86_64
> x86_64 x86_64 GNU/Linux
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
