[GitHub] beam pull request #3714: [BEAM-2684] Fix flaky AmqpIOTest
GitHub user alex-filatov opened a pull request: https://github.com/apache/beam/pull/3714 [BEAM-2684] Fix flaky AmqpIOTest Source of the flakiness is a race between sender and receiver. In a peer-to-peer mode receiver must be running before sender starts sending messages. When this's not the case sender gets stuck in a 'send' method. AMQP broker solves the problem by eliminating sender/receiver startup order requirement. - [x] Make sure there is a [JIRA issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the change (usually before you start working on it). Trivial changes like typos do not require a JIRA issue. Your pull request should address just this issue, without pulling in other changes. - [x] Each commit in the pull request should have a meaningful subject line and body. - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue. - [x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why. - [x] Run `mvn clean verify` to make sure basic checks pass. A more thorough check will be performed on your pull request automatically. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/alex-filatov/beam beam-2684-fix-flaky-amqpiotest Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3714.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3714 commit d9456d6f4f2055d76e7e4f47263d5a8019137944 Author: Alex Filatov Date: 2017-08-10T20:02:37Z [BEAM-2684] Fix flaky AmqpIOTest. Source of the flakiness is a race between sender and receiver. In a peer-to-peer mode receiver must be running before sender starts sending messages. When this's not the case sender gets stuck in a 'send' method. AMQP broker solves the problem by eliminating sender/receiver startup order requirement. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (BEAM-2684) AmqpIOTest is flaky
[
https://issues.apache.org/jira/browse/BEAM-2684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122193#comment-16122193
]
Alex Filatov commented on BEAM-2684:
I can reproduce the issue by commenting out Thread.sleep on line #74 or by
adding one on line #105.
{code}
org.apache.qpid.proton.messenger.impl.MessengerImpl processAllConnectors
SEVERE: Error processing connection
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at
org.apache.qpid.proton.driver.impl.ConnectorImpl.process(ConnectorImpl.java:89)
at
org.apache.qpid.proton.messenger.impl.MessengerImpl.processAllConnectors(MessengerImpl.java:687)
at
org.apache.qpid.proton.messenger.impl.MessengerImpl.waitUntil(MessengerImpl.java:863)
at
org.apache.qpid.proton.messenger.impl.MessengerImpl.waitUntil(MessengerImpl.java:844)
at
org.apache.qpid.proton.messenger.impl.MessengerImpl.send(MessengerImpl.java:417)
at
org.apache.qpid.proton.messenger.impl.MessengerImpl.send(MessengerImpl.java:394)
at org.apache.beam.sdk.io.amqp.AmqpIOTest$1.run(AmqpIOTest.java:82)
{code}
Relevant bits of thread dump:
{code:java}
"direct-runner-worker" #15 prio=5 os_prio=31 tid=0x7fd2c827 nid=0x5303
runnable [0x7aa0]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x0007404c0dd0> (a sun.nio.ch.Util$3)
- locked <0x0007404c0dc0> (a java.util.Collections$UnmodifiableSet)
- locked <0x0007404c0c20> (a sun.nio.ch.KQueueSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at
org.apache.qpid.proton.driver.impl.DriverImpl.doWait(DriverImpl.java:81)
at
org.apache.qpid.proton.messenger.impl.MessengerImpl.waitUntil(MessengerImpl.java:894)
at
org.apache.qpid.proton.messenger.impl.MessengerImpl.waitUntil(MessengerImpl.java:844)
at
org.apache.qpid.proton.messenger.impl.MessengerImpl.recv(MessengerImpl.java:446)
at
org.apache.qpid.proton.messenger.impl.MessengerImpl.recv(MessengerImpl.java:451)
at
org.apache.beam.sdk.io.amqp.AmqpIO$UnboundedAmqpReader.advance(AmqpIO.java:330)
"Thread-1" #11 prio=5 os_prio=31 tid=0x7fd2c7a4f000 nid=0x1407 runnable
[0x7a5f5000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00074000aa98> (a sun.nio.ch.Util$3)
- locked <0x00074000aaa8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00074000aa48> (a sun.nio.ch.KQueueSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at
org.apache.qpid.proton.driver.impl.DriverImpl.doWait(DriverImpl.java:81)
at
org.apache.qpid.proton.messenger.impl.MessengerImpl.waitUntil(MessengerImpl.java:894)
at
org.apache.qpid.proton.messenger.impl.MessengerImpl.waitUntil(MessengerImpl.java:844)
at
org.apache.qpid.proton.messenger.impl.MessengerImpl.send(MessengerImpl.java:417)
at
org.apache.qpid.proton.messenger.impl.MessengerImpl.send(MessengerImpl.java:394)
at org.apache.beam.sdk.io.amqp.AmqpIOTest$1.run(AmqpIOTest.java:82)
{code}
I think root cause is a race between sender and receiver. In the peer-to-peer
mode (without a broker) receiver must be running before sender starts sending
messages. But when it's not the case sender seems to get stuck by suppressing
'Connection refused' exception and then waiting indefinitely for a connection
event on an invalid socket channel.
> AmqpIOTest is flaky
> ---
>
> Key: BEAM-2684
> URL: https://issues.apache.org/jira/browse/BEAM-2684
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-extensions
>Reporter: Eugene Kirpichov
>Assignee: Jean-Baptiste Onofré
>
> This test is often timing out, and has been doing that for a while, causing
> unrelated PRs to fail randomly. I've gotten into the habit of excludin
[GitHub] beam pull request #3656: Fix View.asMultimap() javadoc
GitHub user alex-filatov opened a pull request: https://github.com/apache/beam/pull/3656 Fix View.asMultimap() javadoc Fix return type in the usage example. You can merge this pull request into a Git repository by running: $ git pull https://github.com/alex-filatov/beam fix-view-asmultimap-javadoc Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3656.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3656 commit 0884be5269bdd8a943cbb7a3c3df8013c282129e Author: Alex Filatov Date: 2017-07-28T09:07:07Z Fix View.asMultimap javadoc. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3531: [BEAM-2306] Fail build when @Deprecated is used wit...
GitHub user alex-filatov opened a pull request: https://github.com/apache/beam/pull/3531 [BEAM-2306] Fail build when @Deprecated is used without @deprecated javadoc Add checkstyle check to fail the build when @Deprecated is used without @deprecated javadoc (or vice versa). The check is disabled for existing violations where reason for deprecation and/or alternative is not clear. You can merge this pull request into a Git repository by running: $ git pull https://github.com/alex-filatov/beam beam-2306-check-missing-deprecated Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3531.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3531 commit 0694dcc62a28a98363cdd67a69e3f2c3ce82e072 Author: Alex Filatov Date: 2017-07-10T10:20:49Z [BEAM-2306] Add checkstyle check to fail the build when @Deprecated is used without @deprecated javadoc (or vice versa). The check is disabled for existing violations where reason for deprecation and/or alternative is not clear. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] beam pull request #3475: [BEAM-2544] Fix flaky AvroIOTest
GitHub user alex-filatov opened a pull request: https://github.com/apache/beam/pull/3475 [BEAM-2544] Fix flaky AvroIOTest Source of the flakiness is a race condition in a "write then read" subset of tests. Test pipeline was constructed in a way that write and read operations were performed concurrently instead of sequentially. You can merge this pull request into a Git repository by running: $ git pull https://github.com/alex-filatov/beam beam-2544-fix-flaky-avroittest Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3475.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3475 commit 1b6b38c7069820ca81dd18f44576c3959c35d80a Author: Alex Filatov Date: 2017-06-29T20:23:04Z [BEAM-2544] Fix flaky AvroIOTest by eliminating race condition in "write then read" tests. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (BEAM-2544) AvroIOTest is flaky
Alex Filatov created BEAM-2544: -- Summary: AvroIOTest is flaky Key: BEAM-2544 URL: https://issues.apache.org/jira/browse/BEAM-2544 Project: Beam Issue Type: Bug Components: sdk-java-core Reporter: Alex Filatov Assignee: Davor Bonaci Priority: Minor "Write then read" tests randomly fail. Steps to reproduce: cd /runners/direct-java mvn clean compile mvn surefire:test@validates-runner-tests -Dtest=AvroIOTest Repeat last step until a failure (on my machine failure rate is approx 1/3). Example: [ERROR] testAvroIOWriteAndReadSchemaUpgrade(org.apache.beam.sdk.io.AvroIOTest) Time elapsed: 0.198 s <<< ERROR! java.lang.RuntimeException: java.io.FileNotFoundException: /var/folders/1c/sl733g5s1g7_4mq61_qmbjx4gn/T/junit3332447750239941326/output.avro (No such file or directory) at org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:340) at org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:302) at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:201) at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:64) at org.apache.beam.sdk.Pipeline.run(Pipeline.java:297) at org.apache.beam.sdk.Pipeline.run(Pipeline.java:283) at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:340) at org.apache.beam.sdk.io.AvroIOTest.testAvroIOWriteAndReadSchemaUpgrade(AvroIOTest.java:275) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:239) at org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:321) at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) at org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:321) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.junit.runners.Suite.runChild(Suite.java:128) at org.junit.runners.Suite.runChild(Suite.java:27) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55) at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137) at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107) at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:83) at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75) at org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:157) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:386) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:323) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:143) Caused by: java.io.FileNotFoundExceptio
