[jira] [Updated] (BEAM-6354) Hanging SplittableDoFnTest#testLateData
[ https://issues.apache.org/jira/browse/BEAM-6354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-6354: -- Summary: Hanging SplittableDoFnTest#testLateData (was: Hanging BoundedReadFromUnboundedSourceTest#testTimeBound and SplittableDoFnTest#testLateData) > Hanging SplittableDoFnTest#testLateData > --- > > Key: BEAM-6354 > URL: https://issues.apache.org/jira/browse/BEAM-6354 > Project: Beam > Issue Type: Bug > Components: sdk-java-core >Reporter: Gleb Kanterov >Assignee: Kenneth Knowles >Priority: Major > Fix For: 2.10.0 > > > It seems that they have a similar root cause because both of them use > unbounded streams. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-6492) BoundedReadFromUnboundedSourceTest#testTimeBound hangs since at least 2.8.0
[ https://issues.apache.org/jira/browse/BEAM-6492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-6492: -- Labels: sickbay (was: ) > BoundedReadFromUnboundedSourceTest#testTimeBound hangs since at least 2.8.0 > --- > > Key: BEAM-6492 > URL: https://issues.apache.org/jira/browse/BEAM-6492 > Project: Beam > Issue Type: Bug > Components: sdk-java-core >Affects Versions: 2.8.0, 2.9.0 >Reporter: Gleb Kanterov >Priority: Major > Labels: sickbay > > It seems that they have a similar root cause because both of them use > unbounded streams. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-6354) Hanging SplittableDoFnTest#testLateData
[
https://issues.apache.org/jira/browse/BEAM-6354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16749552#comment-16749552
]
Kenneth Knowles commented on BEAM-6354:
---
Bisect says {{[d13663786cce2851fb83c4e0dee315435518f471] [BEAM-2939]
SplittableDoFn Java SDK API Changes (#6969)}} which sounds about right.
> Hanging SplittableDoFnTest#testLateData
> ---
>
> Key: BEAM-6354
> URL: https://issues.apache.org/jira/browse/BEAM-6354
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Reporter: Gleb Kanterov
>Assignee: Kenneth Knowles
>Priority: Major
> Fix For: 2.10.0
>
>
> It seems that they have a similar root cause because both of them use
> unbounded streams.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (BEAM-6461) beam_precommit_Spotless_Cron flake on format violations
[
https://issues.apache.org/jira/browse/BEAM-6461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles resolved BEAM-6461.
---
Resolution: Fixed
Fix Version/s: Not applicable
Should be healthy at head.
> beam_precommit_Spotless_Cron flake on format violations
> ---
>
> Key: BEAM-6461
> URL: https://issues.apache.org/jira/browse/BEAM-6461
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Reporter: Tyler Akidau
>Assignee: Kenneth Knowles
>Priority: Major
> Fix For: Not applicable
>
>
> From: https://builds.apache.org/job/beam_PreCommit_Spotless_Cron/302/console
>
> *04:01:32* >
> *Task :beam-sdks-java-core:spotlessJava*
> FAILED*04:01:32* *04:01:32* FAILURE: Build failed with an
> exception.*04:01:32* *04:01:32* * What went wrong:*04:01:32* Execution failed
> for task ':beam-sdks-java-core:spotlessJava'.*04:01:32* > The following files
> had format violations:*04:01:32*
> sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/utils/TestPOJOs.java*04:01:32*
>@@ -820,8 +820,7 @@*04:01:32* *04:01:32*
> ··/**·The·schema·for·\{@link·PojoWithNestedArray}.·*/*04:01:32*
> ··public·static·final·Schema·POJO_WITH_NESTED_ARRAY_SCHEMA·=*04:01:32*
>-··Schema.builder().addArrayField("pojos",*04:01:32*
> -··FieldType.array(*04:01:32*
> -··FieldType.row(SIMPLE_POJO_SCHEMA)))*04:01:32*
> -··.build();*04:01:32* +··Schema.builder()*04:01:32*
>
> +··.addArrayField("pojos",·FieldType.array(FieldType.row(SIMPLE_POJO_SCHEMA)))*04:01:32*
>+··.build();*04:01:32*}*04:01:32*
> sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/JavaFieldSchemaTest.java*04:01:32*
>@@ -466,11 +466,10 @@*04:01:32* *04:01:32*
> SimplePOJO·simplePojo·=·createSimple("string");*04:01:32*
> List·simplePojoList·=·ImmutableList.of(simplePojo,·simplePojo);*04:01:32*
>-List>·simplePojoListOfList·=*04:01:32*
> -ImmutableList.of(simplePojoList,·simplePojoList);*04:01:32*
>
> +List>·simplePojoListOfList·=·ImmutableList.of(simplePojoList,·simplePojoList);*04:01:32*
> *04:01:32*
> -PojoWithNestedArray·converted·=·registry.getFromRowFunction(PojoWithNestedArray.class)*04:01:32*
>-.apply(nestedRow);*04:01:32*
> +PojoWithNestedArray·converted·=*04:01:32*
> +registry.getFromRowFunction(PojoWithNestedArray.class).apply(nestedRow);*04:01:32*
>
> assertEquals(simplePojoListOfList,·converted.pojos);*04:01:32*
> ··}*04:01:32* *04:01:32* @@ -487,11 +486,12 @@*04:01:32* *04:01:32*
>SimplePOJO·simplePojo·=·createSimple("string");*04:01:32*
>
> List·simplePojoList·=·ImmutableList.of(simplePojo,·simplePojo);*04:01:32*
>-List>·simplePojoListOfList·=*04:01:32*
> -ImmutableList.of(simplePojoList,·simplePojoList);*04:01:32*
>
> +List>·simplePojoListOfList·=·ImmutableList.of(simplePojoList,·simplePojoList);*04:01:32*
> *04:01:32*
> -Row·converted·=·registry.getToRowFunction(PojoWithNestedArray.class)*04:01:32*
>
> -.apply(new·PojoWithNestedArray(simplePojoListOfList));*04:01:32*
> +Row·converted·=*04:01:32* +registry*04:01:32*
>+.getToRowFunction(PojoWithNestedArray.class)*04:01:32*
>
> +.apply(new·PojoWithNestedArray(simplePojoListOfList));*04:01:32*
>assertEquals(nestedRow,·converted);*04:01:32*
> ··}*04:01:32*}*04:01:32* Run 'gradlew spotlessApply' to fix
> these violations.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-6354) Hanging BoundedReadFromUnboundedSourceTest#testTimeBound and SplittableDoFnTest#testLateData
[ https://issues.apache.org/jira/browse/BEAM-6354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16749530#comment-16749530 ] Kenneth Knowles commented on BEAM-6354: --- I've now confirmed that BoundedReadFromUnboundedSourceTest was broken for 2.9.0 and also 2.8.0. I was mistaken about it working. The tests were not running at that time. I don't think this needs to block the 2.10.0 release. However, SplittableDoFnTest was not broken for 2.8.0 and not broken for 2.9.0. I will bisect that one separately. > Hanging BoundedReadFromUnboundedSourceTest#testTimeBound and > SplittableDoFnTest#testLateData > > > Key: BEAM-6354 > URL: https://issues.apache.org/jira/browse/BEAM-6354 > Project: Beam > Issue Type: Bug > Components: sdk-java-core >Reporter: Gleb Kanterov >Assignee: Kenneth Knowles >Priority: Major > Fix For: 2.10.0 > > > It seems that they have a similar root cause because both of them use > unbounded streams. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-6257) Can we deprecate the side input paths through PAssert?
Kenneth Knowles created BEAM-6257: - Summary: Can we deprecate the side input paths through PAssert? Key: BEAM-6257 URL: https://issues.apache.org/jira/browse/BEAM-6257 Project: Beam Issue Type: Improvement Components: sdk-java-core Reporter: Kenneth Knowles Assignee: Kenneth Knowles PAssert has two distinct paths - one uses GBK with a single-firing trigger, and one uses side inputs. Side inputs are usually a later addition to a runner, while GBK is one of the first primitives (with a single firing it is even simple). Filing this against myself to figure out why the side input version is not deprecated, and if it can be deprecated. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (BEAM-6256) ./gradlew needsRunnerTests fails on fresh clone
[
https://issues.apache.org/jira/browse/BEAM-6256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles reassigned BEAM-6256:
-
Assignee: (was: Luke Cwik)
> ./gradlew needsRunnerTests fails on fresh clone
> ---
>
> Key: BEAM-6256
> URL: https://issues.apache.org/jira/browse/BEAM-6256
> Project: Beam
> Issue Type: Bug
> Components: build-system
>Reporter: Kenneth Knowles
>Priority: Critical
>
> On OSX, this fails on the master branch:
> {{./gradlew needsRunnerTests}}
>
> A few tests fail before it appears to lock up and time out:
> {code:java}
> > Task :beam-runners-direct-java:needsRunnerTests
> org.apache.beam.runners.core.metrics.MetricsPusherTest > test FAILED
> java.lang.AssertionError at MetricsPusherTest.java:71
> org.apache.beam.sdk.io.TFRecordIOTest > testReadInvalidDataMask FAILED
> java.lang.AssertionError
> org.apache.beam.sdk.io.TFRecordIOTest > testReadInvalidLengthMask FAILED
> java.lang.AssertionError
> org.apache.beam.sdk.io.AvroIOTest >
> testContinuouslyWriteAndReadMultipleFilepatterns FAILED
> java.lang.IllegalArgumentException at AvroIOTest.java:453
> org.apache.beam.sdk.io.FileIOTest > testMatchWatchForNewFiles FAILED
> java.lang.IllegalArgumentException at FileIOTest.java:192{code}
> And at this point it hangs.
>
> This should be part of the Java Pre-commit, so it might be OS-specific. A
> bare invocation of these tests should always pass, and pretty quickly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BEAM-6256) ./gradlew needsRunnerTests fails on fresh clone
Kenneth Knowles created BEAM-6256:
-
Summary: ./gradlew needsRunnerTests fails on fresh clone
Key: BEAM-6256
URL: https://issues.apache.org/jira/browse/BEAM-6256
Project: Beam
Issue Type: Bug
Components: build-system
Reporter: Kenneth Knowles
Assignee: Luke Cwik
On OSX, this fails on the master branch:
{{./gradlew needsRunnerTests}}
A few tests fail before it appears to lock up and time out:
{code:java}
> Task :beam-runners-direct-java:needsRunnerTests
org.apache.beam.runners.core.metrics.MetricsPusherTest > test FAILED
java.lang.AssertionError at MetricsPusherTest.java:71
org.apache.beam.sdk.io.TFRecordIOTest > testReadInvalidDataMask FAILED
java.lang.AssertionError
org.apache.beam.sdk.io.TFRecordIOTest > testReadInvalidLengthMask FAILED
java.lang.AssertionError
org.apache.beam.sdk.io.AvroIOTest >
testContinuouslyWriteAndReadMultipleFilepatterns FAILED
java.lang.IllegalArgumentException at AvroIOTest.java:453
org.apache.beam.sdk.io.FileIOTest > testMatchWatchForNewFiles FAILED
java.lang.IllegalArgumentException at FileIOTest.java:192{code}
And at this point it hangs.
This should be part of the Java Pre-commit, so it might be OS-specific. A bare
invocation of these tests should always pass, and pretty quickly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-6250) [beam_SeedJob] Failing with "beam_PostCommit_Go already exists"
[
https://issues.apache.org/jira/browse/BEAM-6250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16724388#comment-16724388
]
Kenneth Knowles commented on BEAM-6250:
---
I clicked in on these, and they are actually not disabled, though I believe
they should be. Previously, I thought the seed job would disable jobs no
longer present. There is probably also something to do with the "previous
names" and preserving history.
I am going to go and do some manual cleanup anyhow, as there actually are
(other) disabled jobs that we can wipe out for sure. If I see two jobs that
look the same and one is more recent, I'll double-check if we are trying to
link their histories (TBH I don't think it matters that much) before I wipe
the old one.
Kenn
> [beam_SeedJob] Failing with "beam_PostCommit_Go already exists"
> ---
>
> Key: BEAM-6250
> URL: https://issues.apache.org/jira/browse/BEAM-6250
> Project: Beam
> Issue Type: Bug
> Components: test-failures, testing
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>Priority: Major
> Labels: currently-failing
>
> _Use this form to file an issue for test failure:_
> * [Jenkins
> Job|https://builds.apache.org/blue/organizations/jenkins/beam_SeedJob/detail/beam_SeedJob/3151/changes/]
> * [Test source
> code|https://github.com/apache/beam/blob/67d0f787b82aa065c2dc1f67608833e9e3e2215c/.test-infra/jenkins/job_PostCommit_Go.groovy#L27]
> Initial investigation:
> The job started failing on the first scheduled run after
> [PR#7286|https://github.com/apache/beam/pull/7286] went in.
> The failure message is:
> {{...}}
> {{Processing DSL script job_PostCommit_Go.groovy}}
> {{java.lang.IllegalArgumentException: beam_PostCommit_Go already exists}}
> The job succeeded on the [test run from the
> PR|https://builds.apache.org/job/beam_SeedJob/3150/], which shows the new job
> with migratedname (beam_PostCommit_Go). The
> [log|https://builds.apache.org/job/beam_SeedJob/3150/console] lists the old
> job name (beam_PostCommit_Go_GradleBuild) under 'Unreferenced items', and I
> can see that the [job still exists and has recent
> runs|https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/]
> It seems that the old jobs were [not properly disabled which is the intended
> behavior|https://github.com/apache/beam/blob/67d0f787b82aa065c2dc1f67608833e9e3e2215c/.test-infra/jenkins/job_00_seed.groovy#L110].
>
> _After you've filled out the above details, please [assign the issue to an
> individual|https://beam.apache.org/contribute/postcommits-guides/index.html#find_specialist].
> Assignee should [treat test failures as
> high-priority|https://beam.apache.org/contribute/postcommits-policies/#assigned-failing-test],
> helping to fix the issue or find a more appropriate owner. See [Apache Beam
> Post-Commit
> Policies|https://beam.apache.org/contribute/postcommits-policies]._
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-6250) [beam_SeedJob] Failing with "beam_PostCommit_Go already exists"
[
https://issues.apache.org/jira/browse/BEAM-6250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16724389#comment-16724389
]
Kenneth Knowles commented on BEAM-6250:
---
I do have this power - will do.
> [beam_SeedJob] Failing with "beam_PostCommit_Go already exists"
> ---
>
> Key: BEAM-6250
> URL: https://issues.apache.org/jira/browse/BEAM-6250
> Project: Beam
> Issue Type: Bug
> Components: test-failures, testing
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>Priority: Major
> Labels: currently-failing
>
> _Use this form to file an issue for test failure:_
> * [Jenkins
> Job|https://builds.apache.org/blue/organizations/jenkins/beam_SeedJob/detail/beam_SeedJob/3151/changes/]
> * [Test source
> code|https://github.com/apache/beam/blob/67d0f787b82aa065c2dc1f67608833e9e3e2215c/.test-infra/jenkins/job_PostCommit_Go.groovy#L27]
> Initial investigation:
> The job started failing on the first scheduled run after
> [PR#7286|https://github.com/apache/beam/pull/7286] went in.
> The failure message is:
> {{...}}
> {{Processing DSL script job_PostCommit_Go.groovy}}
> {{java.lang.IllegalArgumentException: beam_PostCommit_Go already exists}}
> The job succeeded on the [test run from the
> PR|https://builds.apache.org/job/beam_SeedJob/3150/], which shows the new job
> with migratedname (beam_PostCommit_Go). The
> [log|https://builds.apache.org/job/beam_SeedJob/3150/console] lists the old
> job name (beam_PostCommit_Go_GradleBuild) under 'Unreferenced items', and I
> can see that the [job still exists and has recent
> runs|https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/]
> It seems that the old jobs were [not properly disabled which is the intended
> behavior|https://github.com/apache/beam/blob/67d0f787b82aa065c2dc1f67608833e9e3e2215c/.test-infra/jenkins/job_00_seed.groovy#L110].
>
> _After you've filled out the above details, please [assign the issue to an
> individual|https://beam.apache.org/contribute/postcommits-guides/index.html#find_specialist].
> Assignee should [treat test failures as
> high-priority|https://beam.apache.org/contribute/postcommits-policies/#assigned-failing-test],
> helping to fix the issue or find a more appropriate owner. See [Apache Beam
> Post-Commit
> Policies|https://beam.apache.org/contribute/postcommits-policies]._
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BEAM-6280) Failure in PortableRunnerTest.test_error_traceback_includes_user_code
Kenneth Knowles created BEAM-6280:
-
Summary: Failure in
PortableRunnerTest.test_error_traceback_includes_user_code
Key: BEAM-6280
URL: https://issues.apache.org/jira/browse/BEAM-6280
Project: Beam
Issue Type: Bug
Components: sdk-py-core, test-failures
Reporter: Kenneth Knowles
Assignee: Robert Bradshaw
[https://builds.apache.org/job/beam_PreCommit_Python_Cron/732/]
[https://builds.apache.org/job/beam_PreCommit_Python_Cron/732/testReport/apache_beam.runners.portability.portable_runner_test/PortableRunnerTest/test_error_traceback_includes_user_code/]
[https://scans.gradle.com/s/do3hjulee3gaa/console-log?task=:beam-sdks-python:testPython3]
{code:java}
'second' not found in 'Traceback (most recent call last):\n File
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py",
line 466, in test_error_traceback_includes_user_code\np | beam.Create([0])
| beam.Map(first) # pylint: disable=expression-not-assigned\n File
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/apache_beam/pipeline.py",
line 425, in __exit__\nself.run().wait_until_finish()\n File
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/apache_beam/runners/portability/portable_runner.py",
line 314, in wait_until_finish\nself._job_id, self._state,
self._last_error_message()))\nRuntimeError: Pipeline
job-cdcefe6d-1caa-4487-9e63-e971f67ec68c failed in state FAILED: start
\n'{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BEAM-6282) Failure in post-release tests: Intra-project deps on vendored artifacts are incompatible with separate release
Kenneth Knowles created BEAM-6282:
-
Summary: Failure in post-release tests: Intra-project deps on
vendored artifacts are incompatible with separate release
Key: BEAM-6282
URL: https://issues.apache.org/jira/browse/BEAM-6282
Project: Beam
Issue Type: Bug
Components: build-system
Reporter: Kenneth Knowles
Assignee: Kenneth Knowles
https://builds.apache.org/job/beam_PostRelease_NightlySnapshot/467/console
This is a test that runs wordcount against a snapshot release.
{code}
06:03:45 [ERROR] Failed to execute goal
org.codehaus.mojo:exec-maven-plugin:1.6.0:java (default-cli) on project
word-count-beam: An exception occured while executing the Java class.
org/apache/beam/vendor/grpc/v1p13p1/com/google/protobuf/MessageOrBuilder:
org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.MessageOrBuilder ->
[Help 1]
{code}
etc, etc. Still looking through the logs, but I am guessing the v1p13p1
dependency is not available or not set up right.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (BEAM-6279) Failures in ElasticSearchIOTest: testWriteRetryValidRequest (versions 2 & 5) testDefaultRetryPredicate (version 5)
[
https://issues.apache.org/jira/browse/BEAM-6279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles updated BEAM-6279:
--
Issue Type: Bug (was: Improvement)
> Failures in ElasticSearchIOTest: testWriteRetryValidRequest (versions 2 & 5)
> testDefaultRetryPredicate (version 5)
> --
>
> Key: BEAM-6279
> URL: https://issues.apache.org/jira/browse/BEAM-6279
> Project: Beam
> Issue Type: Bug
> Components: io-java-elasticsearch, test-failures
>Reporter: Kenneth Knowles
>Assignee: Etienne Chauchot
>Priority: Critical
> Labels: flake
>
> [https://builds.apache.org/job/beam_PreCommit_Java_Cron/730/]
> [https://scans.gradle.com/s/6rpbvgwx2nk7c]
> I don't see recent changes, so thinking this is a flake.
> testDefaultRetryPredicate:
> {code:java}
> java.lang.AssertionError
> :
> All incoming requests on node [node_s0] should have finished. Expected 0 but
> got 2540 Expected: <0L> but: was <2540L>
> at __randomizedtesting.SeedInfo.seed([4CF46FDC92DC07B8:B39A5DDFBC85274E]:0)
> at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
> at org.junit.Assert.assertThat(Assert.java:956)
> at
> org.elasticsearch.test.InternalTestCluster.lambda$assertRequestsFinished$17(InternalTestCluster.java:2086)
> at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:705)
> at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:679)
> at
> org.elasticsearch.test.InternalTestCluster.assertRequestsFinished(InternalTestCluster.java:2083)
> at
> org.elasticsearch.test.InternalTestCluster.assertAfterTest(InternalTestCluster.java:2062)
> at
> org.elasticsearch.test.ESIntegTestCase.afterInternal(ESIntegTestCase.java:581)
> at
> org.elasticsearch.test.ESIntegTestCase.cleanUpCluster(ESIntegTestCase.java:2054)
> ...
> {code}
> This looks like an internal assertion (aka {{checkState}}) failed and the
> cluster became unhealthy, maybe?
> testWriteRetryValidRequest:
> {code:java}
> org.apache.beam.sdk.Pipeline$PipelineExecutionException
> :
> java.io.IOException: listener timeout after waiting for [9] ms
> Open stacktrace
> Caused by:
> java.io.IOException
> :
> listener timeout after waiting for [9] ms
> Close stacktrace
> at
> org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:899)
> at org.elasticsearch.client.RestClient.performRequest(RestClient.java:227)
> at org.elasticsearch.client.RestClient.performRequest(RestClient.java:321)
> at
> org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO$Write$WriteFn.flushBatch(ElasticsearchIO.java:1285)
> at
> org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO$Write$WriteFn.finishBundle(ElasticsearchIO.java:1260){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BEAM-6281) Jenkins failure in Go SDK PostCommit
Kenneth Knowles created BEAM-6281:
-
Summary: Jenkins failure in Go SDK PostCommit
Key: BEAM-6281
URL: https://issues.apache.org/jira/browse/BEAM-6281
Project: Beam
Issue Type: Bug
Components: build-system, sdk-go, test-failures
Reporter: Kenneth Knowles
Assignee: Robert Burke
https://builds.apache.org/job/beam_PostCommit_Go/2047/console
{code}
*08:43:58* runtime/cgo: pthread_create failed: No space left on device
*08:43:58* SIGABRT: abort*08:43:58* PC=0x7faf428f9428 m=3
sigcode=18446744073709551610
{code}
This looks like the Jenkins worker just beefed - but will you peek at the log
with your Go-knowing eyes and see if there's anything about the run that could
be our fault / we could change to make it not happen?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-6277) KV#getValue is not marked with @Nullable, even though it can be null.
[
https://issues.apache.org/jira/browse/BEAM-6277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16726098#comment-16726098
]
Kenneth Knowles commented on BEAM-6277:
---
The annotations on {{getKey}} and {{getValue}} are actually correct. I am not
sure if the various pluggable type system implementations do it right though.
{{KV}} can be instantiated with any type for {{KeyT}} and
{{ValueT}}. Specifically, since Java 8, you can have {{KeyT := @Nullable
SomeKeyType}} and {{ValueT := SomeValueType}}. So the resulting type is
{{KV<@Nullable SomeKeyType, @Nullable SomeValueType>}}. The nullability is a
property of the key or value type, not of {{KV}} itself. This is important:
{{KV}} never causes nullability.
The annotations on {{of}} are not correct. It should just accept a {{KeyT}} and
{{ValueT}} and whether or not {{null}} is a member of those types should not be
visible to {{KV}}.
Leaving this bug open to fix the annotations on {{of}}, if our analyzers will
let us. (I think findbugs is not updated for this)
> KV#getValue is not marked with @Nullable, even though it can be null.
> -
>
> Key: BEAM-6277
> URL: https://issues.apache.org/jira/browse/BEAM-6277
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Affects Versions: 2.8.0, 2.9.0
>Reporter: Jan Callewaert
>Assignee: Kenneth Knowles
>Priority: Minor
>
> On {{KV#getKey}}, there is the annotation {{@Nullable}}. That annotation is
> missing on {{KV#getValue}}, so I would assume that this means the value
> cannot be null. However, in {{KV#of}}, the parameter {{value}} is annotated
> with {{@Nullable}} and no further {{null}} checks are done.
> I would assume that {{KV#getValue}} should also be marked with {{@Nullable}},
> or that the annotation {{@Nullable}} on the parameter {{value}} in the
> constructor and in {{KV#of}} should be removed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-6276) Performance regression caused by extra calls to TypeDescriptor.getRawType
[ https://issues.apache.org/jira/browse/BEAM-6276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16726099#comment-16726099 ] Kenneth Knowles commented on BEAM-6276: --- Can you point to the regression or describe it here, so there is context? Can you point out where the repeated calls to getRawType are? > Performance regression caused by extra calls to TypeDescriptor.getRawType > - > > Key: BEAM-6276 > URL: https://issues.apache.org/jira/browse/BEAM-6276 > Project: Beam > Issue Type: Bug > Components: sdk-java-core >Reporter: Reuven Lax >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 1h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-6249) Vendored gRPC doesn't seem to work with dataflow
[
https://issues.apache.org/jira/browse/BEAM-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16726057#comment-16726057
]
Kenneth Knowles commented on BEAM-6249:
---
I'll post back here with any news about 2.9.0
> Vendored gRPC doesn't seem to work with dataflow
>
>
> Key: BEAM-6249
> URL: https://issues.apache.org/jira/browse/BEAM-6249
> Project: Beam
> Issue Type: Bug
> Components: runner-dataflow
>Affects Versions: 2.9.0
>Reporter: Steve Niemitz
>Assignee: Tyler Akidau
>Priority: Major
>
> I attempted to migrate an existing pipeline (that worked in 2.8.0) to 2.9.0.
> This pipeline is using the experimental streaming engine
> (–experiments=enable_streaming_engine).
> The pipeline fails to start with these logs:
> {code:java}
> D Unable to load the library
> 'org_apache_beam_vendor_grpc_v1_13_1_netty_tcnative_linux_x86_64', trying
> other loading mechanism.
> D org_apache_beam_vendor_grpc_v1_13_1_netty_tcnative_linux_x86_64 cannot be
> loaded from java.libary.path, now trying export to -Dio.netty.native.workdir:
> /tmp
> D Unable to load the library
> '/tmp/liborg_apache_beam_vendor_grpc_v1_13_1_netty_tcnative_linux_x86_646918605450681921540.so',
> trying other loading mechanism.
> D Unable to load the library 'netty_tcnative_linux_x86_64', trying next
> name...
> D Unable to load the library
> 'org_apache_beam_vendor_grpc_v1_13_1_netty_tcnative_linux_x86_64_fedora',
> trying other loading mechanism.
> D org_apache_beam_vendor_grpc_v1_13_1_netty_tcnative_linux_x86_64_fedora
> cannot be loaded from java.libary.path, now trying export to
> -Dio.netty.native.workdir: /tmp
> D Unable to load the library 'netty_tcnative_linux_x86_64_fedora', trying
> next name...
> D Unable to load the library
> 'org_apache_beam_vendor_grpc_v1_13_1_netty_tcnative_x86_64', trying other
> loading mechanism.
> D org_apache_beam_vendor_grpc_v1_13_1_netty_tcnative_x86_64 cannot be loaded
> from java.libary.path, now trying export to -Dio.netty.native.workdir: /tmp
> D Unable to load the library 'netty_tcnative_x86_64', trying next name...
> D Unable to load the library
> 'org_apache_beam_vendor_grpc_v1_13_1_netty_tcnative', trying other loading
> mechanism.
> D org_apache_beam_vendor_grpc_v1_13_1_netty_tcnative cannot be loaded from
> java.libary.path, now trying export to -Dio.netty.native.workdir: /tmp
> D Unable to load the library 'netty_tcnative', trying next name...
> D Failed to load netty-tcnative; OpenSslEngine will be unavailable, unless
> the application has already loaded the symbols by some other means. See
> http://netty.io/wiki/forked-tomcat-native.html for more information.
> D Failed to initialize netty-tcnative; OpenSslEngine will be unavailable.
> See http://netty.io/wiki/forked-tomcat-native.html for more information.
> I netty-tcnative unavailable (this may be normal)
> I Conscrypt not found (this may be normal)
> I Jetty ALPN unavailable (this may be normal)
> E Uncaught exception in main thread. Exiting with status code 1.
> W Please use a logger instead of System.out or System.err.
> Please switch to using org.slf4j.Logger.
> See: https://cloud.google.com/dataflow/pipelines/logging
> E Uncaught exception in main thread. Exiting with status code 1.
> E java.lang.IllegalStateException: Could not find TLS ALPN provider; no
> working netty-tcnative, Conscrypt, or Jetty NPN/ALPN available
> E at
> org.apache.beam.vendor.grpc.v1_13_1.io.grpc.netty.GrpcSslContexts.defaultSslProvider(GrpcSslContexts.java:256)
>
> E at
> org.apache.beam.vendor.grpc.v1_13_1.io.grpc.netty.GrpcSslContexts.configure(GrpcSslContexts.java:171)
>
> E at
> org.apache.beam.vendor.grpc.v1_13_1.io.grpc.netty.GrpcSslContexts.forClient(GrpcSslContexts.java:120)
>
> E at
> org.apache.beam.runners.dataflow.worker.windmill.GrpcWindmillServer.remoteChannel(GrpcWindmillServer.java:343)
>
> E at
> org.apache.beam.runners.dataflow.worker.windmill.GrpcWindmillServer.initializeWindmillService(GrpcWindmillServer.java:312)
>
> {code}
>
> The interesting part is in the netty load failure, the stack trace is:
> {code:java}
> exception: "java.lang.UnsatisfiedLinkError at
> org.apache.beam.vendor.grpc.v1_13_1.io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:276)
> at
> org.apache.beam.vendor.grpc.v1_13_1.io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:233)
> at
> org.apache.beam.vendor.grpc.v1_13_1.io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:187)
> at
> org.apache.beam.vendor.grpc.v1_13_1.io.netty.util.internal.NativeLibraryLoader.loadFirstAvailable(NativeLibraryLoader.java:85)
> at
>
[jira] [Assigned] (BEAM-6136) Shade conscrypt within the Dataflow legacy worker jar
[ https://issues.apache.org/jira/browse/BEAM-6136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles reassigned BEAM-6136: - Assignee: Ahmet Altay > Shade conscrypt within the Dataflow legacy worker jar > - > > Key: BEAM-6136 > URL: https://issues.apache.org/jira/browse/BEAM-6136 > Project: Beam > Issue Type: Bug > Components: runner-dataflow >Reporter: Luke Cwik >Assignee: Ahmet Altay >Priority: Minor > > org.conscrypt:conscrypt-openjdk:1.1.3:linux-x86_64 relies on JNI which can't > be shaded away since the C code doesn't support automatically figuring out > its post relocation path (like tcnative can). > There is support for building conscrypt with an additional prefix but this > doesn't work well with shading since it requires every user who wants to > shade to build their own copy. > jniutil.cc relies on JNI_JARJAR_PREFIX > https://github.com/google/conscrypt/blob/e3c1c99b3e0c56849cb7150bb95dc211b0ad058c/common/src/jni/main/cpp/conscrypt/jniutil.cc#L62 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-6136) Shade conscrypt within the Dataflow legacy worker jar
[ https://issues.apache.org/jira/browse/BEAM-6136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16725960#comment-16725960 ] Kenneth Knowles commented on BEAM-6136: --- Obsolete / wontfix? > Shade conscrypt within the Dataflow legacy worker jar > - > > Key: BEAM-6136 > URL: https://issues.apache.org/jira/browse/BEAM-6136 > Project: Beam > Issue Type: Bug > Components: runner-dataflow >Reporter: Luke Cwik >Assignee: Ahmet Altay >Priority: Minor > > org.conscrypt:conscrypt-openjdk:1.1.3:linux-x86_64 relies on JNI which can't > be shaded away since the C code doesn't support automatically figuring out > its post relocation path (like tcnative can). > There is support for building conscrypt with an additional prefix but this > doesn't work well with shading since it requires every user who wants to > shade to build their own copy. > jniutil.cc relies on JNI_JARJAR_PREFIX > https://github.com/google/conscrypt/blob/e3c1c99b3e0c56849cb7150bb95dc211b0ad058c/common/src/jni/main/cpp/conscrypt/jniutil.cc#L62 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-6249) Vendored gRPC doesn't seem to work with dataflow
[
https://issues.apache.org/jira/browse/BEAM-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16726051#comment-16726051
]
Kenneth Knowles commented on BEAM-6249:
---
Thanks for the investigation. Thanks to your detailed pointers I've opened a PR
to netty-tcnative to fix their escaping. Our own configuration also had some
unescaped underscores. In the meantime, we've just replaced underscores with
the letter {{p}} on Beam {{master}}.
> Vendored gRPC doesn't seem to work with dataflow
>
>
> Key: BEAM-6249
> URL: https://issues.apache.org/jira/browse/BEAM-6249
> Project: Beam
> Issue Type: Bug
> Components: runner-dataflow
>Affects Versions: 2.9.0
>Reporter: Steve Niemitz
>Assignee: Tyler Akidau
>Priority: Major
>
> I attempted to migrate an existing pipeline (that worked in 2.8.0) to 2.9.0.
> This pipeline is using the experimental streaming engine
> (–experiments=enable_streaming_engine).
> The pipeline fails to start with these logs:
> {code:java}
> D Unable to load the library
> 'org_apache_beam_vendor_grpc_v1_13_1_netty_tcnative_linux_x86_64', trying
> other loading mechanism.
> D org_apache_beam_vendor_grpc_v1_13_1_netty_tcnative_linux_x86_64 cannot be
> loaded from java.libary.path, now trying export to -Dio.netty.native.workdir:
> /tmp
> D Unable to load the library
> '/tmp/liborg_apache_beam_vendor_grpc_v1_13_1_netty_tcnative_linux_x86_646918605450681921540.so',
> trying other loading mechanism.
> D Unable to load the library 'netty_tcnative_linux_x86_64', trying next
> name...
> D Unable to load the library
> 'org_apache_beam_vendor_grpc_v1_13_1_netty_tcnative_linux_x86_64_fedora',
> trying other loading mechanism.
> D org_apache_beam_vendor_grpc_v1_13_1_netty_tcnative_linux_x86_64_fedora
> cannot be loaded from java.libary.path, now trying export to
> -Dio.netty.native.workdir: /tmp
> D Unable to load the library 'netty_tcnative_linux_x86_64_fedora', trying
> next name...
> D Unable to load the library
> 'org_apache_beam_vendor_grpc_v1_13_1_netty_tcnative_x86_64', trying other
> loading mechanism.
> D org_apache_beam_vendor_grpc_v1_13_1_netty_tcnative_x86_64 cannot be loaded
> from java.libary.path, now trying export to -Dio.netty.native.workdir: /tmp
> D Unable to load the library 'netty_tcnative_x86_64', trying next name...
> D Unable to load the library
> 'org_apache_beam_vendor_grpc_v1_13_1_netty_tcnative', trying other loading
> mechanism.
> D org_apache_beam_vendor_grpc_v1_13_1_netty_tcnative cannot be loaded from
> java.libary.path, now trying export to -Dio.netty.native.workdir: /tmp
> D Unable to load the library 'netty_tcnative', trying next name...
> D Failed to load netty-tcnative; OpenSslEngine will be unavailable, unless
> the application has already loaded the symbols by some other means. See
> http://netty.io/wiki/forked-tomcat-native.html for more information.
> D Failed to initialize netty-tcnative; OpenSslEngine will be unavailable.
> See http://netty.io/wiki/forked-tomcat-native.html for more information.
> I netty-tcnative unavailable (this may be normal)
> I Conscrypt not found (this may be normal)
> I Jetty ALPN unavailable (this may be normal)
> E Uncaught exception in main thread. Exiting with status code 1.
> W Please use a logger instead of System.out or System.err.
> Please switch to using org.slf4j.Logger.
> See: https://cloud.google.com/dataflow/pipelines/logging
> E Uncaught exception in main thread. Exiting with status code 1.
> E java.lang.IllegalStateException: Could not find TLS ALPN provider; no
> working netty-tcnative, Conscrypt, or Jetty NPN/ALPN available
> E at
> org.apache.beam.vendor.grpc.v1_13_1.io.grpc.netty.GrpcSslContexts.defaultSslProvider(GrpcSslContexts.java:256)
>
> E at
> org.apache.beam.vendor.grpc.v1_13_1.io.grpc.netty.GrpcSslContexts.configure(GrpcSslContexts.java:171)
>
> E at
> org.apache.beam.vendor.grpc.v1_13_1.io.grpc.netty.GrpcSslContexts.forClient(GrpcSslContexts.java:120)
>
> E at
> org.apache.beam.runners.dataflow.worker.windmill.GrpcWindmillServer.remoteChannel(GrpcWindmillServer.java:343)
>
> E at
> org.apache.beam.runners.dataflow.worker.windmill.GrpcWindmillServer.initializeWindmillService(GrpcWindmillServer.java:312)
>
> {code}
>
> The interesting part is in the netty load failure, the stack trace is:
> {code:java}
> exception: "java.lang.UnsatisfiedLinkError at
> org.apache.beam.vendor.grpc.v1_13_1.io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:276)
> at
> org.apache.beam.vendor.grpc.v1_13_1.io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:233)
> at
>
[jira] [Assigned] (BEAM-6276) Performance regression caused by extra calls to TypeDescriptor.getRawType
[ https://issues.apache.org/jira/browse/BEAM-6276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles reassigned BEAM-6276: - Assignee: Reuven Lax (was: Kenneth Knowles) > Performance regression caused by extra calls to TypeDescriptor.getRawType > - > > Key: BEAM-6276 > URL: https://issues.apache.org/jira/browse/BEAM-6276 > Project: Beam > Issue Type: Bug > Components: sdk-java-core >Reporter: Reuven Lax >Assignee: Reuven Lax >Priority: Major > Time Spent: 1h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-6276) Performance regression caused by extra calls to TypeDescriptor.getRawType
[ https://issues.apache.org/jira/browse/BEAM-6276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16726100#comment-16726100 ] Kenneth Knowles commented on BEAM-6276: --- Assigning to you, since you already opened a PR for this. > Performance regression caused by extra calls to TypeDescriptor.getRawType > - > > Key: BEAM-6276 > URL: https://issues.apache.org/jira/browse/BEAM-6276 > Project: Beam > Issue Type: Bug > Components: sdk-java-core >Reporter: Reuven Lax >Assignee: Reuven Lax >Priority: Major > Time Spent: 1h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-6288) Separate SQL postcommits from core Java SDK postcommits
Kenneth Knowles created BEAM-6288: - Summary: Separate SQL postcommits from core Java SDK postcommits Key: BEAM-6288 URL: https://issues.apache.org/jira/browse/BEAM-6288 Project: Beam Issue Type: Improvement Components: build-system Reporter: Kenneth Knowles Assignee: Kenneth Knowles It would add signal clarity to make the postcommits more granular. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-6288) Separate SQL postcommits from core Java SDK postcommits
[ https://issues.apache.org/jira/browse/BEAM-6288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-6288: -- Priority: Minor (was: Major) > Separate SQL postcommits from core Java SDK postcommits > --- > > Key: BEAM-6288 > URL: https://issues.apache.org/jira/browse/BEAM-6288 > Project: Beam > Issue Type: Improvement > Components: build-system >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > It would add signal clarity to make the postcommits more granular. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-1399) Code coverage numbers are not accurate
[
https://issues.apache.org/jira/browse/BEAM-1399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles updated BEAM-1399:
--
Description:
We've started adding Java Code Coverage numbers to PRs using the jacoco gradle
plugin.
When we ran this in Maven, we got very low coverage reported for things like
the Java SDK core. This was likely because we test the bulk of the SDK not in
the SDK module, but in fact in the DirectRunner and other similar modules.
This is the still the case in gradle. So we'll need to make sure to watch for
it.
This is a good "random improvement" issue for anyone to pick up.
was:
We've started adding Java Code Coverage numbers to PRs using the jacoco plugin.
However, we are getting very low coverage reported for things like the Java SDK
core.
My belief is that this is happening because we test the bulk of the SDK not in
the SDK module , but in fact in the DirectRunner and other similar modules.
JaCoCo has a {{report:aggregate}} target that might do the trick, but with a
few minutes of playing with it I wasn't able to make it work satisfactorily.
Basic work in https://github.com/apache/beam/pull/1800
This is a good "random improvement" issue for anyone to pick up.
> Code coverage numbers are not accurate
> --
>
> Key: BEAM-1399
> URL: https://issues.apache.org/jira/browse/BEAM-1399
> Project: Beam
> Issue Type: Bug
> Components: build-system, sdk-java-core, testing
>Reporter: Daniel Halperin
>Priority: Major
> Labels: newbie, starter
>
> We've started adding Java Code Coverage numbers to PRs using the jacoco
> gradle plugin.
> When we ran this in Maven, we got very low coverage reported for things like
> the Java SDK core. This was likely because we test the bulk of the SDK not in
> the SDK module, but in fact in the DirectRunner and other similar modules.
> This is the still the case in gradle. So we'll need to make sure to watch for
> it.
> This is a good "random improvement" issue for anyone to pick up.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-1132) Jenkins JaCoCo plugin for Beam
[ https://issues.apache.org/jira/browse/BEAM-1132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16727481#comment-16727481 ] Kenneth Knowles commented on BEAM-1132: --- pinging this since I just found it > Jenkins JaCoCo plugin for Beam > -- > > Key: BEAM-1132 > URL: https://issues.apache.org/jira/browse/BEAM-1132 > Project: Beam > Issue Type: New Feature > Components: testing >Reporter: Kenneth Knowles >Assignee: Jason Kuster >Priority: Minor > > Jenkins has a JaCoCo plugin and other Apache projects use it. We should try > it too (and might as well disable coveralls, as I don't know anyone who looks > at it). > If this takes more than 10 minutes to set up, then another option is to just > archive JaCoCo's HTML reports so we can browse them. > Either of these should take just minutes and yield huge benefits in > visibility of where we have really bad coverage. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-6301) gradle jacoco config is broken
Kenneth Knowles created BEAM-6301:
-
Summary: gradle jacoco config is broken
Key: BEAM-6301
URL: https://issues.apache.org/jira/browse/BEAM-6301
Project: Beam
Issue Type: Bug
Components: build-system
Reporter: Kenneth Knowles
Assignee: Kenneth Knowles
This should run tests and generate a coverage report:
{code}
./gradlew jacocoCoverageReport
{code}
But the task does not cause tests to run. It is likely due to the ordering or
organization inside BeamModulePlugin.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-6267) Guava library DOS vulnerability in Beam Java SDK
[ https://issues.apache.org/jira/browse/BEAM-6267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16729074#comment-16729074 ] Kenneth Knowles commented on BEAM-6267: --- I don't think Beam is vulnerable to this, but we should update vendored Guava anyhow. A more interesting question might be what is going to happen with modules that depend on public Guava. I know of Spark and Cassandra but transitively there are probably others. > Guava library DOS vulnerability in Beam Java SDK > > > Key: BEAM-6267 > URL: https://issues.apache.org/jira/browse/BEAM-6267 > Project: Beam > Issue Type: Bug > Components: sdk-java-core >Affects Versions: 2.4.0, 2.9.0 > Environment: Java >Reporter: Abdul Qadeer >Assignee: Kenneth Knowles >Priority: Major > > Guava 11.0 through 24.x are affected by the following vulnerability: > [https://groups.google.com/forum/#!topic/guava-announce/xqWALw4W1vs/discussion] > https://nvd.nist.gov/vuln/detail/2018-10237 > All Beam versions that use these Guava versions need to be updated with > latest fixed Guava library. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (BEAM-1132) Jenkins JaCoCo plugin for Beam
[ https://issues.apache.org/jira/browse/BEAM-1132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles reassigned BEAM-1132: - Assignee: Kenneth Knowles (was: Jason Kuster) > Jenkins JaCoCo plugin for Beam > -- > > Key: BEAM-1132 > URL: https://issues.apache.org/jira/browse/BEAM-1132 > Project: Beam > Issue Type: New Feature > Components: testing >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Minor > > Jenkins has a JaCoCo plugin and other Apache projects use it. We should try > it too (and might as well disable coveralls, as I don't know anyone who looks > at it). > If this takes more than 10 minutes to set up, then another option is to just > archive JaCoCo's HTML reports so we can browse them. > Either of these should take just minutes and yield huge benefits in > visibility of where we have really bad coverage. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-6239) Nexmark benchmark for raw sessionization then stream enrichment
Kenneth Knowles created BEAM-6239: - Summary: Nexmark benchmark for raw sessionization then stream enrichment Key: BEAM-6239 URL: https://issues.apache.org/jira/browse/BEAM-6239 Project: Beam Issue Type: New Feature Components: examples-nexmark Reporter: Kenneth Knowles Assignee: Kenneth Knowles We have BOUNDED_SIDE_INPUT_JOIN that just enriches a stream. Another use case is to sessionize first. I am curious about the different in perf, and how this plays out in SQL. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (BEAM-6272) :beam-sdks-java-io-hadoop-format:test fails due to static port conflict
[ https://issues.apache.org/jira/browse/BEAM-6272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles reassigned BEAM-6272: - Assignee: Kenneth Knowles (was: Chamikara Jayalath) > :beam-sdks-java-io-hadoop-format:test fails due to static port conflict > --- > > Key: BEAM-6272 > URL: https://issues.apache.org/jira/browse/BEAM-6272 > Project: Beam > Issue Type: Bug > Components: io-java-hadoop, test-failures, testing >Reporter: Udi Meiri >Assignee: Kenneth Knowles >Priority: Critical > > This is probably due to two precommits trying using port 7000 at the same > time on the same Jenkins machine (by the same or some other test that also > uses port 7000). > Error in log: > 19:25:15 org.apache.cassandra.exceptions.ConfigurationException: > localhost/127.0.0.1:7000 is in use by another process. Change > listen_address:storage_port in cassandra.yaml to values that do not conflict > with other services > https://builds.apache.org/job/beam_PreCommit_Java_Commit/3181/consoleFull -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-6212) MongoDbIO add option to ignore duplicate inserts
[ https://issues.apache.org/jira/browse/BEAM-6212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-6212: -- Fix Version/s: (was: 2.9.0) 2.10.0 > MongoDbIO add option to ignore duplicate inserts > > > Key: BEAM-6212 > URL: https://issues.apache.org/jira/browse/BEAM-6212 > Project: Beam > Issue Type: Improvement > Components: io-java-mongodb >Affects Versions: 2.8.0 >Reporter: Chaim >Assignee: Jean-Baptiste Onofré >Priority: Major > Fix For: 2.10.0 > > Time Spent: 10m > Remaining Estimate: 0h > > to support failure of pipeline, we need the option to be idempotent, so we > need the option to ignore duplicates -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-6212) MongoDbIO add option to ignore duplicate inserts
[ https://issues.apache.org/jira/browse/BEAM-6212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720375#comment-16720375 ] Kenneth Knowles commented on BEAM-6212: --- The 2.9.0 release is already wrapping up. Since this is not a critical regression, I am going to defer this to version 2.10.0. If this is a critical blocker for your use case, it may merit inclusion into a release on the long-term support branch, aka 2.7.1. > MongoDbIO add option to ignore duplicate inserts > > > Key: BEAM-6212 > URL: https://issues.apache.org/jira/browse/BEAM-6212 > Project: Beam > Issue Type: Improvement > Components: io-java-mongodb >Affects Versions: 2.8.0 >Reporter: Chaim >Assignee: Jean-Baptiste Onofré >Priority: Major > Fix For: 2.10.0 > > Time Spent: 10m > Remaining Estimate: 0h > > to support failure of pipeline, we need the option to be idempotent, so we > need the option to ignore duplicates -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-6228) Website build from source release fails due to git dependency
[
https://issues.apache.org/jira/browse/BEAM-6228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720403#comment-16720403
]
Kenneth Knowles commented on BEAM-6228:
---
Pasted in the output of the build.
> Website build from source release fails due to git dependency
> -
>
> Key: BEAM-6228
> URL: https://issues.apache.org/jira/browse/BEAM-6228
> Project: Beam
> Issue Type: New Feature
> Components: website
>Reporter: Scott Wegner
>Priority: Minor
>
> The website build assumes a git environment, and will fail if built outside
> of git. As a result, we cannot build the website from the source release.
> [~kenn] noticed this during 2.9.0 release validation. See:
> https://lists.apache.org/thread.html/dc816a5f8c82dd4d19e82666209305ec67d46440fe8edba4534f9b63@%3Cdev.beam.apache.org%3E
> {code}
> > Configure project :beam-website
> No git repository found for :beam-website. Accessing grgit will cause an NPE.
> FAILURE: Build failed with an exception.
> * Where:
> Build file '/Users/klk/Downloads/beam-release-2.9.0/website/build.gradle'
> line: 143
> * What went wrong:
> A problem occurred evaluating project ':beam-website'.
> > Cannot get property 'branch' on null object
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (BEAM-6228) Website build from source release fails due to git dependency
[
https://issues.apache.org/jira/browse/BEAM-6228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles updated BEAM-6228:
--
Description:
The website build assumes a git environment, and will fail if built outside of
git. As a result, we cannot build the website from the source release.
[~kenn] noticed this during 2.9.0 release validation. See:
https://lists.apache.org/thread.html/dc816a5f8c82dd4d19e82666209305ec67d46440fe8edba4534f9b63@%3Cdev.beam.apache.org%3E
{code}
> Configure project :beam-website
No git repository found for :beam-website. Accessing grgit will cause an NPE.
FAILURE: Build failed with an exception.
* Where:
Build file '/Users/klk/Downloads/beam-release-2.9.0/website/build.gradle' line:
143
* What went wrong:
A problem occurred evaluating project ':beam-website'.
> Cannot get property 'branch' on null object
{code}
was:
The website build assumes a git environment, and will fail if built outside of
git. As a result, we cannot build the website from the source release.
[~kenn] noticed this during 2.9.0 release validation. See:
https://lists.apache.org/thread.html/dc816a5f8c82dd4d19e82666209305ec67d46440fe8edba4534f9b63@%3Cdev.beam.apache.org%3E
> Website build from source release fails due to git dependency
> -
>
> Key: BEAM-6228
> URL: https://issues.apache.org/jira/browse/BEAM-6228
> Project: Beam
> Issue Type: New Feature
> Components: website
>Reporter: Scott Wegner
>Priority: Minor
>
> The website build assumes a git environment, and will fail if built outside
> of git. As a result, we cannot build the website from the source release.
> [~kenn] noticed this during 2.9.0 release validation. See:
> https://lists.apache.org/thread.html/dc816a5f8c82dd4d19e82666209305ec67d46440fe8edba4534f9b63@%3Cdev.beam.apache.org%3E
> {code}
> > Configure project :beam-website
> No git repository found for :beam-website. Accessing grgit will cause an NPE.
> FAILURE: Build failed with an exception.
> * Where:
> Build file '/Users/klk/Downloads/beam-release-2.9.0/website/build.gradle'
> line: 143
> * What went wrong:
> A problem occurred evaluating project ':beam-website'.
> > Cannot get property 'branch' on null object
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (BEAM-6228) Website build from source release fails due to git dependency
[
https://issues.apache.org/jira/browse/BEAM-6228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles updated BEAM-6228:
--
Description:
The website build assumes a git environment, and will fail if built outside of
git. As a result, we cannot build the website from the source release.
[~kenn] noticed this during 2.9.0 release validation. See:
https://lists.apache.org/thread.html/dc816a5f8c82dd4d19e82666209305ec67d46440fe8edba4534f9b63@%3Cdev.beam.apache.org%3E
{code}
> Configure project :beam-website
No git repository found for :beam-website. Accessing grgit will cause an NPE.
FAILURE: Build failed with an exception.
* Where:
Build file 'website/build.gradle' line: 143
* What went wrong:
A problem occurred evaluating project ':beam-website'.
> Cannot get property 'branch' on null object
{code}
was:
The website build assumes a git environment, and will fail if built outside of
git. As a result, we cannot build the website from the source release.
[~kenn] noticed this during 2.9.0 release validation. See:
https://lists.apache.org/thread.html/dc816a5f8c82dd4d19e82666209305ec67d46440fe8edba4534f9b63@%3Cdev.beam.apache.org%3E
{code}
> Configure project :beam-website
No git repository found for :beam-website. Accessing grgit will cause an NPE.
FAILURE: Build failed with an exception.
* Where:
Build file '/Users/klk/Downloads/beam-release-2.9.0/website/build.gradle' line:
143
* What went wrong:
A problem occurred evaluating project ':beam-website'.
> Cannot get property 'branch' on null object
{code}
> Website build from source release fails due to git dependency
> -
>
> Key: BEAM-6228
> URL: https://issues.apache.org/jira/browse/BEAM-6228
> Project: Beam
> Issue Type: New Feature
> Components: website
>Reporter: Scott Wegner
>Priority: Minor
>
> The website build assumes a git environment, and will fail if built outside
> of git. As a result, we cannot build the website from the source release.
> [~kenn] noticed this during 2.9.0 release validation. See:
> https://lists.apache.org/thread.html/dc816a5f8c82dd4d19e82666209305ec67d46440fe8edba4534f9b63@%3Cdev.beam.apache.org%3E
> {code}
> > Configure project :beam-website
> No git repository found for :beam-website. Accessing grgit will cause an NPE.
> FAILURE: Build failed with an exception.
> * Where:
> Build file 'website/build.gradle' line: 143
> * What went wrong:
> A problem occurred evaluating project ':beam-website'.
> > Cannot get property 'branch' on null object
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (BEAM-6193) Standardize use of insecure vs. secure port in the FnAPI
[ https://issues.apache.org/jira/browse/BEAM-6193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles reassigned BEAM-6193: - Assignee: Robert Bradshaw (was: Kenneth Knowles) > Standardize use of insecure vs. secure port in the FnAPI > > > Key: BEAM-6193 > URL: https://issues.apache.org/jira/browse/BEAM-6193 > Project: Beam > Issue Type: Improvement > Components: beam-model, sdk-java-harness, sdk-py-harness >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > > Possibly the ApiServiceDescriptor should be enriched with this information. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-3660) Port ReadSpannerSchemaTest off DoFnTester
[ https://issues.apache.org/jira/browse/BEAM-3660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16712155#comment-16712155 ] Kenneth Knowles commented on BEAM-3660: --- I've added you to the Contributors role so you can assign tickets to yourself now. > Port ReadSpannerSchemaTest off DoFnTester > - > > Key: BEAM-3660 > URL: https://issues.apache.org/jira/browse/BEAM-3660 > Project: Beam > Issue Type: Sub-task > Components: io-java-gcp >Reporter: Kenneth Knowles >Assignee: Evgeniy Musin >Priority: Major > Labels: beginner, newbie, starter > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (BEAM-3660) Port ReadSpannerSchemaTest off DoFnTester
[ https://issues.apache.org/jira/browse/BEAM-3660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles reassigned BEAM-3660: - Assignee: Evgeniy Musin > Port ReadSpannerSchemaTest off DoFnTester > - > > Key: BEAM-3660 > URL: https://issues.apache.org/jira/browse/BEAM-3660 > Project: Beam > Issue Type: Sub-task > Components: io-java-gcp >Reporter: Kenneth Knowles >Assignee: Evgeniy Musin >Priority: Major > Labels: beginner, newbie, starter > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-6169) Dataflow runner: rateLimit exceeded from BigQuery API in start_bundle
[
https://issues.apache.org/jira/browse/BEAM-6169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles updated BEAM-6169:
--
Component/s: (was: gcp-quota)
sdk-py-core
> Dataflow runner: rateLimit exceeded from BigQuery API in start_bundle
> -
>
> Key: BEAM-6169
> URL: https://issues.apache.org/jira/browse/BEAM-6169
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Affects Versions: 2.8.0
> Environment: Python SDK 2.8, Dataflow runner
>Reporter: Thomas Pilewicz
>Assignee: Kenneth Knowles
>Priority: Minor
>
> When many machines, or too big machines are used, BigQuery returns rateLimit
> exceeded errors in response to the GET requests from
> BigQueryWriteFn.start_bundle.
> Unsure if this causes performance issues, or is just noise. (Produced with a
> pipeline reading strings from Pub/Sub that are ~1000 characters long, and
> writing directly to BigQuery)
> {{... File
> "/usr/local/lib/python2.7/dist-packages/apache_beam/io/gcp/bigquery.py", line
> 1087, in get_or_create_table found_table = self._get_table(project_id,
> dataset_id, table_id) File
> "/usr/local/lib/python2.7/dist-packages/apache_beam/utils/retry.py", line
> 184, in wrapper return fun(*args, **kwargs) File
> "/usr/local/lib/python2.7/dist-packages/apache_beam/io/gcp/bigquery.py", line
> 925, in _get_table response = self.client.tables.Get(request) File
> "/usr/local/lib/python2.7/dist-packages/apache_beam/io/gcp/internal/clients/bigquery/bigquery_v2_client.py",
> line 611, in Get config, request, global_params=global_params) File
> "/usr/local/lib/python2.7/dist-packages/apitools/base/py/base_api.py", line
> 722, in _RunMethod return self.ProcessHttpResponse(method_config,
> http_response, request) File
> "/usr/local/lib/python2.7/dist-packages/apitools/base/py/base_api.py", line
> 728, in ProcessHttpResponse self.__ProcessHttpResponse(method_config,
> http_response, request)) File
> "/usr/local/lib/python2.7/dist-packages/apitools/base/py/base_api.py", line
> 599, in __ProcessHttpResponse http_response, method_config=method_config,
> request=request) HttpForbiddenError: HttpError accessing
> <[https://www.googleapis.com/bigquery/v2/projects//datasets/|https://www.googleapis.com/bigquery/v2/projects/carrefour-170216/datasets/realtime_dashboard/tables/realtime_errors?alt=json]
>
> [|https://www.googleapis.com/bigquery/v2/projects/carrefour-170216/datasets/realtime_dashboard/tables/realtime_errors?alt=json]
>
> [/tables/|https://www.googleapis.com/bigquery/v2/projects/carrefour-170216/datasets/realtime_dashboard/tables/realtime_errors?alt=json]
>
> [|https://www.googleapis.com/bigquery/v2/projects/carrefour-170216/datasets/realtime_dashboard/tables/realtime_errors?alt=json]
>
> [?alt=json|https://www.googleapis.com/bigquery/v2/projects/carrefour-170216/datasets/realtime_dashboard/tables/realtime_errors?alt=json]>:
> response: <
> {'status': '403', 'content-length': '577', 'x-xss-protection': '1;
> mode=block', 'x-content-type-options': 'nosniff', 'transfer-encoding':
> 'chunked', 'expires': 'Sun, 25 Nov 2018 14:36:24 GMT', 'vary': 'Origin,
> X-Origin', 'server': 'GSE', '-content-encoding': 'gzip', 'cache-control':
> 'private, max-age=0', 'date': 'Sun, 25 Nov 2018 14:36:24 GMT',
> 'x-frame-options': 'SAMEORIGIN', 'content-type': 'application/json;
> charset=UTF-8'}
> >, content <{ "error": { "errors": [
> { "domain": "global", "reason": "rateLimitExceeded", "message": "Exceeded
> rate limits: Your user_method exceeded quota for api requests per user per
> method. For more information, see
> https://cloud.google.com/bigquery/troubleshooting-errors;, "locationType":
> "other", "location": "helix_api.method_request" }
> ], "code": 403, "message": "Exceeded rate limits: Your user_method exceeded
> quota for api requests per user per method. For more information, see
> [https://cloud.google.com/bigquery/troubleshooting-errors]"}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (BEAM-6169) Dataflow runner: rateLimit exceeded from BigQuery API in start_bundle
[
https://issues.apache.org/jira/browse/BEAM-6169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles reassigned BEAM-6169:
-
Assignee: Chamikara Jayalath (was: Kenneth Knowles)
> Dataflow runner: rateLimit exceeded from BigQuery API in start_bundle
> -
>
> Key: BEAM-6169
> URL: https://issues.apache.org/jira/browse/BEAM-6169
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Affects Versions: 2.8.0
> Environment: Python SDK 2.8, Dataflow runner
>Reporter: Thomas Pilewicz
>Assignee: Chamikara Jayalath
>Priority: Minor
>
> When many machines, or too big machines are used, BigQuery returns rateLimit
> exceeded errors in response to the GET requests from
> BigQueryWriteFn.start_bundle.
> Unsure if this causes performance issues, or is just noise. (Produced with a
> pipeline reading strings from Pub/Sub that are ~1000 characters long, and
> writing directly to BigQuery)
> {{... File
> "/usr/local/lib/python2.7/dist-packages/apache_beam/io/gcp/bigquery.py", line
> 1087, in get_or_create_table found_table = self._get_table(project_id,
> dataset_id, table_id) File
> "/usr/local/lib/python2.7/dist-packages/apache_beam/utils/retry.py", line
> 184, in wrapper return fun(*args, **kwargs) File
> "/usr/local/lib/python2.7/dist-packages/apache_beam/io/gcp/bigquery.py", line
> 925, in _get_table response = self.client.tables.Get(request) File
> "/usr/local/lib/python2.7/dist-packages/apache_beam/io/gcp/internal/clients/bigquery/bigquery_v2_client.py",
> line 611, in Get config, request, global_params=global_params) File
> "/usr/local/lib/python2.7/dist-packages/apitools/base/py/base_api.py", line
> 722, in _RunMethod return self.ProcessHttpResponse(method_config,
> http_response, request) File
> "/usr/local/lib/python2.7/dist-packages/apitools/base/py/base_api.py", line
> 728, in ProcessHttpResponse self.__ProcessHttpResponse(method_config,
> http_response, request)) File
> "/usr/local/lib/python2.7/dist-packages/apitools/base/py/base_api.py", line
> 599, in __ProcessHttpResponse http_response, method_config=method_config,
> request=request) HttpForbiddenError: HttpError accessing
> <[https://www.googleapis.com/bigquery/v2/projects//datasets/|https://www.googleapis.com/bigquery/v2/projects/carrefour-170216/datasets/realtime_dashboard/tables/realtime_errors?alt=json]
>
> [|https://www.googleapis.com/bigquery/v2/projects/carrefour-170216/datasets/realtime_dashboard/tables/realtime_errors?alt=json]
>
> [/tables/|https://www.googleapis.com/bigquery/v2/projects/carrefour-170216/datasets/realtime_dashboard/tables/realtime_errors?alt=json]
>
> [|https://www.googleapis.com/bigquery/v2/projects/carrefour-170216/datasets/realtime_dashboard/tables/realtime_errors?alt=json]
>
> [?alt=json|https://www.googleapis.com/bigquery/v2/projects/carrefour-170216/datasets/realtime_dashboard/tables/realtime_errors?alt=json]>:
> response: <
> {'status': '403', 'content-length': '577', 'x-xss-protection': '1;
> mode=block', 'x-content-type-options': 'nosniff', 'transfer-encoding':
> 'chunked', 'expires': 'Sun, 25 Nov 2018 14:36:24 GMT', 'vary': 'Origin,
> X-Origin', 'server': 'GSE', '-content-encoding': 'gzip', 'cache-control':
> 'private, max-age=0', 'date': 'Sun, 25 Nov 2018 14:36:24 GMT',
> 'x-frame-options': 'SAMEORIGIN', 'content-type': 'application/json;
> charset=UTF-8'}
> >, content <{ "error": { "errors": [
> { "domain": "global", "reason": "rateLimitExceeded", "message": "Exceeded
> rate limits: Your user_method exceeded quota for api requests per user per
> method. For more information, see
> https://cloud.google.com/bigquery/troubleshooting-errors;, "locationType":
> "other", "location": "helix_api.method_request" }
> ], "code": 403, "message": "Exceeded rate limits: Your user_method exceeded
> quota for api requests per user per method. For more information, see
> [https://cloud.google.com/bigquery/troubleshooting-errors]"}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BEAM-6110) Beam SQL join moves element timestamps to EOW sometimes, not always
Kenneth Knowles created BEAM-6110: - Summary: Beam SQL join moves element timestamps to EOW sometimes, not always Key: BEAM-6110 URL: https://issues.apache.org/jira/browse/BEAM-6110 Project: Beam Issue Type: Bug Components: dsl-sql Reporter: Kenneth Knowles Assignee: Kenneth Knowles When a CoGBK join is chosen, element timestamps move to EOW. When side input or lookup join is chosen, element timestamps are the timestamps from the main input element. In SQL sementics, these timestamps are not observable, as windowing is assigned at the next GROUP BY. So we just need the output to hold the watermark so that nothing becomes late or droppable, which implies we need to use MIN timestamp combiner. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-6109) Beam SQL rejects join of unbounded to bounded, though a side input join does succeed
Kenneth Knowles created BEAM-6109: - Summary: Beam SQL rejects join of unbounded to bounded, though a side input join does succeed Key: BEAM-6109 URL: https://issues.apache.org/jira/browse/BEAM-6109 Project: Beam Issue Type: Bug Components: dsl-sql Reporter: Kenneth Knowles Assignee: Kenneth Knowles Current, verification of supporting triggering and windowing is done before deciding whether to do a side input join. In the case of globally windowed data, this is too strict. We can relax it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-6109) Beam SQL rejects join of unbounded to bounded, though a side input join does succeed
[ https://issues.apache.org/jira/browse/BEAM-6109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-6109: -- Issue Type: New Feature (was: Bug) > Beam SQL rejects join of unbounded to bounded, though a side input join does > succeed > > > Key: BEAM-6109 > URL: https://issues.apache.org/jira/browse/BEAM-6109 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > > Current, verification of supporting triggering and windowing is done before > deciding whether to do a side input join. In the case of globally windowed > data, this is too strict. We can relax it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-6112) SQL could support equijoins on complex expressions, not just column refs
[
https://issues.apache.org/jira/browse/BEAM-6112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles updated BEAM-6112:
--
Description: Currently a join such as {{CAST(A.x AS BIGINT) = B.y}} will
fail, along with other similar simple expressions. We only support joining
directly on immediate column references. It can be worked around by inserting a
WITH clause for the intermediate value.
> SQL could support equijoins on complex expressions, not just column refs
>
>
> Key: BEAM-6112
> URL: https://issues.apache.org/jira/browse/BEAM-6112
> Project: Beam
> Issue Type: New Feature
> Components: dsl-sql
>Reporter: Kenneth Knowles
>Priority: Major
>
> Currently a join such as {{CAST(A.x AS BIGINT) = B.y}} will fail, along with
> other similar simple expressions. We only support joining directly on
> immediate column references. It can be worked around by inserting a WITH
> clause for the intermediate value.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (BEAM-6102) Dataflow cannot deserialize DoFns - incompatible serialVersionUID (JDK or code version mismatch)
[
https://issues.apache.org/jira/browse/BEAM-6102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles updated BEAM-6102:
--
Summary: Dataflow cannot deserialize DoFns - incompatible serialVersionUID
(JDK or code version mismatch) (was: Wordcount broken on master)
> Dataflow cannot deserialize DoFns - incompatible serialVersionUID (JDK or
> code version mismatch)
>
>
> Key: BEAM-6102
> URL: https://issues.apache.org/jira/browse/BEAM-6102
> Project: Beam
> Issue Type: Bug
> Components: runner-dataflow
>Reporter: Ankur Goenka
>Assignee: Luke Cwik
>Priority: Blocker
> Fix For: 2.9.0
>
>
> The wordcount is broken on the master.
> Its failing with serialization error mentioned below
>
> {code:java}
> 11:08:59 AM: Executing task 'WordCount.main()'... Parallel execution is an
> incubating feature. > Task :buildSrc:compileJava NO-SOURCE > Task
> :buildSrc:compileGroovy UP-TO-DATE > Task :buildSrc:processResources
> NO-SOURCE > Task :buildSrc:classes UP-TO-DATE > Task :buildSrc:jar UP-TO-DATE
> > Task :buildSrc:assemble UP-TO-DATE > Task :buildSrc:spotlessGroovy
> UP-TO-DATE > Task :buildSrc:spotlessGroovyCheck UP-TO-DATE > Task
> :buildSrc:spotlessGroovyGradle UP-TO-DATE > Task
> :buildSrc:spotlessGroovyGradleCheck UP-TO-DATE > Task :buildSrc:spotlessCheck
> UP-TO-DATE > Task :buildSrc:compileTestJava NO-SOURCE > Task
> :buildSrc:compileTestGroovy NO-SOURCE > Task :buildSrc:processTestResources
> NO-SOURCE > Task :buildSrc:testClasses UP-TO-DATE > Task :buildSrc:test
> NO-SOURCE > Task :buildSrc:check UP-TO-DATE > Task :buildSrc:build UP-TO-DATE
> Parallel execution with configuration on demand is an incubating feature. >
> Configure project :beam-model-pipeline applyPortabilityNature with default
> configuration for project beam-model-pipeline > Configure project
> :beam-model-job-management applyPortabilityNature with default configuration
> for project beam-model-job-management > Configure project
> :beam-model-fn-execution applyPortabilityNature with default configuration
> for project beam-model-fn-execution > Task
> :beam-examples-java:processResources NO-SOURCE > Task
> :beam-sdks-java-core:generateAvroProtocol NO-SOURCE > Task
> :beam-sdks-java-extensions-google-cloud-platform-core:processResources
> NO-SOURCE > Task :beam-sdks-java-io-google-cloud-platform:processResources
> NO-SOURCE > Task :beam-vendor-grpc-v1_13_1:compileJava NO-SOURCE > Task
> :beam-runners-core-construction-java:processResources NO-SOURCE > Task
> :beam-sdks-java-extensions-protobuf:extractProto UP-TO-DATE > Task
> :beam-model-pipeline:extractProto UP-TO-DATE > Task
> :beam-model-job-management:processResources UP-TO-DATE > Task
> :beam-runners-core-java:processResources NO-SOURCE > Task
> :beam-sdks-java-fn-execution:processResources NO-SOURCE > Task
> :beam-sdks-java-harness:processResources NO-SOURCE > Task
> :beam-vendor-sdks-java-extensions-protobuf:processResources NO-SOURCE > Task
> :beam-sdks-java-extensions-protobuf:processResources NO-SOURCE > Task
> :beam-sdks-java-core:generateAvroJava NO-SOURCE > Task
> :beam-vendor-grpc-v1_13_1:processResources NO-SOURCE > Task
> :beam-model-fn-execution:processResources UP-TO-DATE > Task
> :beam-model-pipeline:processResources UP-TO-DATE > Task
> :beam-runners-local-java-core:processResources NO-SOURCE > Task
> :beam-runners-java-fn-execution:processResources NO-SOURCE > Task
> :beam-runners-direct-java:processResources NO-SOURCE > Task
> :beam-vendor-grpc-v1_13_1:classes UP-TO-DATE > Task
> :beam-sdks-java-core:processResources UP-TO-DATE > Task
> :beam-runners-google-cloud-dataflow-java:processResources UP-TO-DATE > Task
> :beam-vendor-grpc-v1_13_1:shadowJar UP-TO-DATE > Task
> :beam-model-fn-execution:extractIncludeProto UP-TO-DATE > Task
> :beam-model-pipeline:extractIncludeProto UP-TO-DATE > Task
> :beam-model-job-management:extractIncludeProto UP-TO-DATE > Task
> :beam-model-pipeline:generateProto UP-TO-DATE > Task
> :beam-model-pipeline:compileJava UP-TO-DATE > Task
> :beam-model-pipeline:classes UP-TO-DATE > Task :beam-model-pipeline:shadowJar
> UP-TO-DATE > Task :beam-model-pipeline:jar UP-TO-DATE > Task
> :beam-model-job-management:extractProto UP-TO-DATE > Task
> :beam-model-fn-execution:extractProto UP-TO-DATE > Task
> :beam-model-fn-execution:generateProto UP-TO-DATE > Task
> :beam-model-job-management:generateProto UP-TO-DATE > Task
> :beam-model-job-management:compileJava UP-TO-DATE > Task
> :beam-model-job-management:classes UP-TO-DATE > Task
> :beam-model-fn-execution:compileJava UP-TO-DATE > Task
> :beam-model-fn-execution:classes UP-TO-DATE > Task
> :beam-model-job-management:shadowJar UP-TO-DATE > Task
>
[jira] [Created] (BEAM-6104) Stack overflow in javac on SQL generated code
Kenneth Knowles created BEAM-6104:
-
Summary: Stack overflow in javac on SQL generated code
Key: BEAM-6104
URL: https://issues.apache.org/jira/browse/BEAM-6104
Project: Beam
Issue Type: Bug
Components: build-system, dsl-sql
Reporter: Kenneth Knowles
Assignee: Kenneth Knowles
Occasionally, and seemingly nondeterministically, I see this error during build:
{code}
An exception has occurred in the compiler ((version info not available)).
Please file a bug against the Java compiler via the Java bug reporting page
(http://bugreport.java.com) after checking the Bug Database
(http://bugs.java.com) for duplicates. Include your program and the following
diagnostic in your report. Thank you.
java.lang.StackOverflowError
at com.sun.tools.javac.parser.JavacParser.term2(JavacParser.java:925)
at com.sun.tools.javac.parser.JavacParser.term1(JavacParser.java:896)
at com.sun.tools.javac.parser.JavacParser.term(JavacParser.java:852)
at com.sun.tools.javac.parser.JavacParser.term(JavacParser.java:832)
at
com.sun.tools.javac.parser.JavacParser.blockStatement(JavacParser.java:2435)
at
com.sun.tools.javac.parser.JavacParser.blockStatements(JavacParser.java:2339)
at com.sun.tools.javac.parser.JavacParser.block(JavacParser.java:2309)
at com.sun.tools.javac.parser.JavacParser.block(JavacParser.java:2323)
at
com.sun.tools.javac.parser.JavacParser.parseSimpleStatement(JavacParser.java:2482)
at
com.sun.tools.javac.parser.JavacParser.blockStatement(JavacParser.java:2401)
at
com.sun.tools.javac.parser.JavacParser.parseStatementAsBlock(JavacParser.java:2364)
at
com.sun.tools.javac.parser.JavacParser.parseSimpleStatement(JavacParser.java:2486)
at
com.sun.tools.javac.parser.JavacParser.blockStatement(JavacParser.java:2401)
at
com.sun.tools.javac.parser.JavacParser.blockStatements(JavacParser.java:2339)
at com.sun.tools.javac.parser.JavacParser.block(JavacParser.java:2309)
at com.sun.tools.javac.parser.JavacParser.block(JavacParser.java:2323)
at
com.sun.tools.javac.parser.JavacParser.parseSimpleStatement(JavacParser.java:2482)
at
com.sun.tools.javac.parser.JavacParser.blockStatement(JavacParser.java:2401)
at
com.sun.tools.javac.parser.JavacParser.parseStatementAsBlock(JavacParser.java:2364)
at
com.sun.tools.javac.parser.JavacParser.parseSimpleStatement(JavacParser.java:2486)
at
com.sun.tools.javac.parser.JavacParser.blockStatement(JavacParser.java:2401)
at
com.sun.tools.javac.parser.JavacParser.blockStatements(JavacParser.java:2339)
at com.sun.tools.javac.parser.JavacParser.block(JavacParser.java:2309)
at com.sun.tools.javac.parser.JavacParser.block(JavacParser.java:2323)
at
com.sun.tools.javac.parser.JavacParser.parseSimpleStatement(JavacParser.java:2482)
{code}
(and it goes on for pages and pages of the same stack, repeating)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-6102) Wordcount broken on master
[
https://issues.apache.org/jira/browse/BEAM-6102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16693903#comment-16693903
]
Kenneth Knowles commented on BEAM-6102:
---
Definitely a release blocker - the results of
https://builds.apache.org/view/A-D/view/Beam/job/beam_PostRelease_NightlySnapshot/
would imply that the quickstart does not work on Dataflow right now.
> Wordcount broken on master
> --
>
> Key: BEAM-6102
> URL: https://issues.apache.org/jira/browse/BEAM-6102
> Project: Beam
> Issue Type: Bug
> Components: runner-dataflow
>Reporter: Ankur Goenka
>Assignee: Luke Cwik
>Priority: Major
> Fix For: 2.9.0
>
>
> The wordcount is broken on the master.
> Its failing with serialization error mentioned below
>
> {code:java}
> 11:08:59 AM: Executing task 'WordCount.main()'... Parallel execution is an
> incubating feature. > Task :buildSrc:compileJava NO-SOURCE > Task
> :buildSrc:compileGroovy UP-TO-DATE > Task :buildSrc:processResources
> NO-SOURCE > Task :buildSrc:classes UP-TO-DATE > Task :buildSrc:jar UP-TO-DATE
> > Task :buildSrc:assemble UP-TO-DATE > Task :buildSrc:spotlessGroovy
> UP-TO-DATE > Task :buildSrc:spotlessGroovyCheck UP-TO-DATE > Task
> :buildSrc:spotlessGroovyGradle UP-TO-DATE > Task
> :buildSrc:spotlessGroovyGradleCheck UP-TO-DATE > Task :buildSrc:spotlessCheck
> UP-TO-DATE > Task :buildSrc:compileTestJava NO-SOURCE > Task
> :buildSrc:compileTestGroovy NO-SOURCE > Task :buildSrc:processTestResources
> NO-SOURCE > Task :buildSrc:testClasses UP-TO-DATE > Task :buildSrc:test
> NO-SOURCE > Task :buildSrc:check UP-TO-DATE > Task :buildSrc:build UP-TO-DATE
> Parallel execution with configuration on demand is an incubating feature. >
> Configure project :beam-model-pipeline applyPortabilityNature with default
> configuration for project beam-model-pipeline > Configure project
> :beam-model-job-management applyPortabilityNature with default configuration
> for project beam-model-job-management > Configure project
> :beam-model-fn-execution applyPortabilityNature with default configuration
> for project beam-model-fn-execution > Task
> :beam-examples-java:processResources NO-SOURCE > Task
> :beam-sdks-java-core:generateAvroProtocol NO-SOURCE > Task
> :beam-sdks-java-extensions-google-cloud-platform-core:processResources
> NO-SOURCE > Task :beam-sdks-java-io-google-cloud-platform:processResources
> NO-SOURCE > Task :beam-vendor-grpc-v1_13_1:compileJava NO-SOURCE > Task
> :beam-runners-core-construction-java:processResources NO-SOURCE > Task
> :beam-sdks-java-extensions-protobuf:extractProto UP-TO-DATE > Task
> :beam-model-pipeline:extractProto UP-TO-DATE > Task
> :beam-model-job-management:processResources UP-TO-DATE > Task
> :beam-runners-core-java:processResources NO-SOURCE > Task
> :beam-sdks-java-fn-execution:processResources NO-SOURCE > Task
> :beam-sdks-java-harness:processResources NO-SOURCE > Task
> :beam-vendor-sdks-java-extensions-protobuf:processResources NO-SOURCE > Task
> :beam-sdks-java-extensions-protobuf:processResources NO-SOURCE > Task
> :beam-sdks-java-core:generateAvroJava NO-SOURCE > Task
> :beam-vendor-grpc-v1_13_1:processResources NO-SOURCE > Task
> :beam-model-fn-execution:processResources UP-TO-DATE > Task
> :beam-model-pipeline:processResources UP-TO-DATE > Task
> :beam-runners-local-java-core:processResources NO-SOURCE > Task
> :beam-runners-java-fn-execution:processResources NO-SOURCE > Task
> :beam-runners-direct-java:processResources NO-SOURCE > Task
> :beam-vendor-grpc-v1_13_1:classes UP-TO-DATE > Task
> :beam-sdks-java-core:processResources UP-TO-DATE > Task
> :beam-runners-google-cloud-dataflow-java:processResources UP-TO-DATE > Task
> :beam-vendor-grpc-v1_13_1:shadowJar UP-TO-DATE > Task
> :beam-model-fn-execution:extractIncludeProto UP-TO-DATE > Task
> :beam-model-pipeline:extractIncludeProto UP-TO-DATE > Task
> :beam-model-job-management:extractIncludeProto UP-TO-DATE > Task
> :beam-model-pipeline:generateProto UP-TO-DATE > Task
> :beam-model-pipeline:compileJava UP-TO-DATE > Task
> :beam-model-pipeline:classes UP-TO-DATE > Task :beam-model-pipeline:shadowJar
> UP-TO-DATE > Task :beam-model-pipeline:jar UP-TO-DATE > Task
> :beam-model-job-management:extractProto UP-TO-DATE > Task
> :beam-model-fn-execution:extractProto UP-TO-DATE > Task
> :beam-model-fn-execution:generateProto UP-TO-DATE > Task
> :beam-model-job-management:generateProto UP-TO-DATE > Task
> :beam-model-job-management:compileJava UP-TO-DATE > Task
> :beam-model-job-management:classes UP-TO-DATE > Task
> :beam-model-fn-execution:compileJava UP-TO-DATE > Task
> :beam-model-fn-execution:classes UP-TO-DATE > Task
> :beam-model-job-management:shadowJar UP-TO-DATE > Task
> :beam-sdks-java-core:compileJava UP-TO-DATE > Task
>
[jira] [Updated] (BEAM-6102) Wordcount broken on master
[
https://issues.apache.org/jira/browse/BEAM-6102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles updated BEAM-6102:
--
Priority: Blocker (was: Major)
> Wordcount broken on master
> --
>
> Key: BEAM-6102
> URL: https://issues.apache.org/jira/browse/BEAM-6102
> Project: Beam
> Issue Type: Bug
> Components: runner-dataflow
>Reporter: Ankur Goenka
>Assignee: Luke Cwik
>Priority: Blocker
> Fix For: 2.9.0
>
>
> The wordcount is broken on the master.
> Its failing with serialization error mentioned below
>
> {code:java}
> 11:08:59 AM: Executing task 'WordCount.main()'... Parallel execution is an
> incubating feature. > Task :buildSrc:compileJava NO-SOURCE > Task
> :buildSrc:compileGroovy UP-TO-DATE > Task :buildSrc:processResources
> NO-SOURCE > Task :buildSrc:classes UP-TO-DATE > Task :buildSrc:jar UP-TO-DATE
> > Task :buildSrc:assemble UP-TO-DATE > Task :buildSrc:spotlessGroovy
> UP-TO-DATE > Task :buildSrc:spotlessGroovyCheck UP-TO-DATE > Task
> :buildSrc:spotlessGroovyGradle UP-TO-DATE > Task
> :buildSrc:spotlessGroovyGradleCheck UP-TO-DATE > Task :buildSrc:spotlessCheck
> UP-TO-DATE > Task :buildSrc:compileTestJava NO-SOURCE > Task
> :buildSrc:compileTestGroovy NO-SOURCE > Task :buildSrc:processTestResources
> NO-SOURCE > Task :buildSrc:testClasses UP-TO-DATE > Task :buildSrc:test
> NO-SOURCE > Task :buildSrc:check UP-TO-DATE > Task :buildSrc:build UP-TO-DATE
> Parallel execution with configuration on demand is an incubating feature. >
> Configure project :beam-model-pipeline applyPortabilityNature with default
> configuration for project beam-model-pipeline > Configure project
> :beam-model-job-management applyPortabilityNature with default configuration
> for project beam-model-job-management > Configure project
> :beam-model-fn-execution applyPortabilityNature with default configuration
> for project beam-model-fn-execution > Task
> :beam-examples-java:processResources NO-SOURCE > Task
> :beam-sdks-java-core:generateAvroProtocol NO-SOURCE > Task
> :beam-sdks-java-extensions-google-cloud-platform-core:processResources
> NO-SOURCE > Task :beam-sdks-java-io-google-cloud-platform:processResources
> NO-SOURCE > Task :beam-vendor-grpc-v1_13_1:compileJava NO-SOURCE > Task
> :beam-runners-core-construction-java:processResources NO-SOURCE > Task
> :beam-sdks-java-extensions-protobuf:extractProto UP-TO-DATE > Task
> :beam-model-pipeline:extractProto UP-TO-DATE > Task
> :beam-model-job-management:processResources UP-TO-DATE > Task
> :beam-runners-core-java:processResources NO-SOURCE > Task
> :beam-sdks-java-fn-execution:processResources NO-SOURCE > Task
> :beam-sdks-java-harness:processResources NO-SOURCE > Task
> :beam-vendor-sdks-java-extensions-protobuf:processResources NO-SOURCE > Task
> :beam-sdks-java-extensions-protobuf:processResources NO-SOURCE > Task
> :beam-sdks-java-core:generateAvroJava NO-SOURCE > Task
> :beam-vendor-grpc-v1_13_1:processResources NO-SOURCE > Task
> :beam-model-fn-execution:processResources UP-TO-DATE > Task
> :beam-model-pipeline:processResources UP-TO-DATE > Task
> :beam-runners-local-java-core:processResources NO-SOURCE > Task
> :beam-runners-java-fn-execution:processResources NO-SOURCE > Task
> :beam-runners-direct-java:processResources NO-SOURCE > Task
> :beam-vendor-grpc-v1_13_1:classes UP-TO-DATE > Task
> :beam-sdks-java-core:processResources UP-TO-DATE > Task
> :beam-runners-google-cloud-dataflow-java:processResources UP-TO-DATE > Task
> :beam-vendor-grpc-v1_13_1:shadowJar UP-TO-DATE > Task
> :beam-model-fn-execution:extractIncludeProto UP-TO-DATE > Task
> :beam-model-pipeline:extractIncludeProto UP-TO-DATE > Task
> :beam-model-job-management:extractIncludeProto UP-TO-DATE > Task
> :beam-model-pipeline:generateProto UP-TO-DATE > Task
> :beam-model-pipeline:compileJava UP-TO-DATE > Task
> :beam-model-pipeline:classes UP-TO-DATE > Task :beam-model-pipeline:shadowJar
> UP-TO-DATE > Task :beam-model-pipeline:jar UP-TO-DATE > Task
> :beam-model-job-management:extractProto UP-TO-DATE > Task
> :beam-model-fn-execution:extractProto UP-TO-DATE > Task
> :beam-model-fn-execution:generateProto UP-TO-DATE > Task
> :beam-model-job-management:generateProto UP-TO-DATE > Task
> :beam-model-job-management:compileJava UP-TO-DATE > Task
> :beam-model-job-management:classes UP-TO-DATE > Task
> :beam-model-fn-execution:compileJava UP-TO-DATE > Task
> :beam-model-fn-execution:classes UP-TO-DATE > Task
> :beam-model-job-management:shadowJar UP-TO-DATE > Task
> :beam-sdks-java-core:compileJava UP-TO-DATE > Task
> :beam-sdks-java-core:classes UP-TO-DATE > Task
> :beam-model-fn-execution:shadowJar UP-TO-DATE > Task
> :beam-sdks-java-core:shadowJar UP-TO-DATE > Task
> :beam-sdks-java-extensions-protobuf:extractIncludeProto
[jira] [Updated] (BEAM-6102) Wordcount broken on master
[
https://issues.apache.org/jira/browse/BEAM-6102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles updated BEAM-6102:
--
Fix Version/s: 2.9.0
> Wordcount broken on master
> --
>
> Key: BEAM-6102
> URL: https://issues.apache.org/jira/browse/BEAM-6102
> Project: Beam
> Issue Type: Bug
> Components: runner-dataflow
>Reporter: Ankur Goenka
>Assignee: Luke Cwik
>Priority: Blocker
> Fix For: 2.9.0
>
>
> The wordcount is broken on the master.
> Its failing with serialization error mentioned below
>
> {code:java}
> 11:08:59 AM: Executing task 'WordCount.main()'... Parallel execution is an
> incubating feature. > Task :buildSrc:compileJava NO-SOURCE > Task
> :buildSrc:compileGroovy UP-TO-DATE > Task :buildSrc:processResources
> NO-SOURCE > Task :buildSrc:classes UP-TO-DATE > Task :buildSrc:jar UP-TO-DATE
> > Task :buildSrc:assemble UP-TO-DATE > Task :buildSrc:spotlessGroovy
> UP-TO-DATE > Task :buildSrc:spotlessGroovyCheck UP-TO-DATE > Task
> :buildSrc:spotlessGroovyGradle UP-TO-DATE > Task
> :buildSrc:spotlessGroovyGradleCheck UP-TO-DATE > Task :buildSrc:spotlessCheck
> UP-TO-DATE > Task :buildSrc:compileTestJava NO-SOURCE > Task
> :buildSrc:compileTestGroovy NO-SOURCE > Task :buildSrc:processTestResources
> NO-SOURCE > Task :buildSrc:testClasses UP-TO-DATE > Task :buildSrc:test
> NO-SOURCE > Task :buildSrc:check UP-TO-DATE > Task :buildSrc:build UP-TO-DATE
> Parallel execution with configuration on demand is an incubating feature. >
> Configure project :beam-model-pipeline applyPortabilityNature with default
> configuration for project beam-model-pipeline > Configure project
> :beam-model-job-management applyPortabilityNature with default configuration
> for project beam-model-job-management > Configure project
> :beam-model-fn-execution applyPortabilityNature with default configuration
> for project beam-model-fn-execution > Task
> :beam-examples-java:processResources NO-SOURCE > Task
> :beam-sdks-java-core:generateAvroProtocol NO-SOURCE > Task
> :beam-sdks-java-extensions-google-cloud-platform-core:processResources
> NO-SOURCE > Task :beam-sdks-java-io-google-cloud-platform:processResources
> NO-SOURCE > Task :beam-vendor-grpc-v1_13_1:compileJava NO-SOURCE > Task
> :beam-runners-core-construction-java:processResources NO-SOURCE > Task
> :beam-sdks-java-extensions-protobuf:extractProto UP-TO-DATE > Task
> :beam-model-pipeline:extractProto UP-TO-DATE > Task
> :beam-model-job-management:processResources UP-TO-DATE > Task
> :beam-runners-core-java:processResources NO-SOURCE > Task
> :beam-sdks-java-fn-execution:processResources NO-SOURCE > Task
> :beam-sdks-java-harness:processResources NO-SOURCE > Task
> :beam-vendor-sdks-java-extensions-protobuf:processResources NO-SOURCE > Task
> :beam-sdks-java-extensions-protobuf:processResources NO-SOURCE > Task
> :beam-sdks-java-core:generateAvroJava NO-SOURCE > Task
> :beam-vendor-grpc-v1_13_1:processResources NO-SOURCE > Task
> :beam-model-fn-execution:processResources UP-TO-DATE > Task
> :beam-model-pipeline:processResources UP-TO-DATE > Task
> :beam-runners-local-java-core:processResources NO-SOURCE > Task
> :beam-runners-java-fn-execution:processResources NO-SOURCE > Task
> :beam-runners-direct-java:processResources NO-SOURCE > Task
> :beam-vendor-grpc-v1_13_1:classes UP-TO-DATE > Task
> :beam-sdks-java-core:processResources UP-TO-DATE > Task
> :beam-runners-google-cloud-dataflow-java:processResources UP-TO-DATE > Task
> :beam-vendor-grpc-v1_13_1:shadowJar UP-TO-DATE > Task
> :beam-model-fn-execution:extractIncludeProto UP-TO-DATE > Task
> :beam-model-pipeline:extractIncludeProto UP-TO-DATE > Task
> :beam-model-job-management:extractIncludeProto UP-TO-DATE > Task
> :beam-model-pipeline:generateProto UP-TO-DATE > Task
> :beam-model-pipeline:compileJava UP-TO-DATE > Task
> :beam-model-pipeline:classes UP-TO-DATE > Task :beam-model-pipeline:shadowJar
> UP-TO-DATE > Task :beam-model-pipeline:jar UP-TO-DATE > Task
> :beam-model-job-management:extractProto UP-TO-DATE > Task
> :beam-model-fn-execution:extractProto UP-TO-DATE > Task
> :beam-model-fn-execution:generateProto UP-TO-DATE > Task
> :beam-model-job-management:generateProto UP-TO-DATE > Task
> :beam-model-job-management:compileJava UP-TO-DATE > Task
> :beam-model-job-management:classes UP-TO-DATE > Task
> :beam-model-fn-execution:compileJava UP-TO-DATE > Task
> :beam-model-fn-execution:classes UP-TO-DATE > Task
> :beam-model-job-management:shadowJar UP-TO-DATE > Task
> :beam-sdks-java-core:compileJava UP-TO-DATE > Task
> :beam-sdks-java-core:classes UP-TO-DATE > Task
> :beam-model-fn-execution:shadowJar UP-TO-DATE > Task
> :beam-sdks-java-core:shadowJar UP-TO-DATE > Task
> :beam-sdks-java-extensions-protobuf:extractIncludeProto UP-TO-DATE >
[jira] [Updated] (BEAM-6102) Wordcount broken on master
[
https://issues.apache.org/jira/browse/BEAM-6102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles updated BEAM-6102:
--
Component/s: (was: sdk-java-core)
runner-dataflow
> Wordcount broken on master
> --
>
> Key: BEAM-6102
> URL: https://issues.apache.org/jira/browse/BEAM-6102
> Project: Beam
> Issue Type: Bug
> Components: runner-dataflow
>Reporter: Ankur Goenka
>Assignee: Luke Cwik
>Priority: Major
>
> The wordcount is broken on the master.
> Its failing with serialization error mentioned below
>
> {code:java}
> 11:08:59 AM: Executing task 'WordCount.main()'... Parallel execution is an
> incubating feature. > Task :buildSrc:compileJava NO-SOURCE > Task
> :buildSrc:compileGroovy UP-TO-DATE > Task :buildSrc:processResources
> NO-SOURCE > Task :buildSrc:classes UP-TO-DATE > Task :buildSrc:jar UP-TO-DATE
> > Task :buildSrc:assemble UP-TO-DATE > Task :buildSrc:spotlessGroovy
> UP-TO-DATE > Task :buildSrc:spotlessGroovyCheck UP-TO-DATE > Task
> :buildSrc:spotlessGroovyGradle UP-TO-DATE > Task
> :buildSrc:spotlessGroovyGradleCheck UP-TO-DATE > Task :buildSrc:spotlessCheck
> UP-TO-DATE > Task :buildSrc:compileTestJava NO-SOURCE > Task
> :buildSrc:compileTestGroovy NO-SOURCE > Task :buildSrc:processTestResources
> NO-SOURCE > Task :buildSrc:testClasses UP-TO-DATE > Task :buildSrc:test
> NO-SOURCE > Task :buildSrc:check UP-TO-DATE > Task :buildSrc:build UP-TO-DATE
> Parallel execution with configuration on demand is an incubating feature. >
> Configure project :beam-model-pipeline applyPortabilityNature with default
> configuration for project beam-model-pipeline > Configure project
> :beam-model-job-management applyPortabilityNature with default configuration
> for project beam-model-job-management > Configure project
> :beam-model-fn-execution applyPortabilityNature with default configuration
> for project beam-model-fn-execution > Task
> :beam-examples-java:processResources NO-SOURCE > Task
> :beam-sdks-java-core:generateAvroProtocol NO-SOURCE > Task
> :beam-sdks-java-extensions-google-cloud-platform-core:processResources
> NO-SOURCE > Task :beam-sdks-java-io-google-cloud-platform:processResources
> NO-SOURCE > Task :beam-vendor-grpc-v1_13_1:compileJava NO-SOURCE > Task
> :beam-runners-core-construction-java:processResources NO-SOURCE > Task
> :beam-sdks-java-extensions-protobuf:extractProto UP-TO-DATE > Task
> :beam-model-pipeline:extractProto UP-TO-DATE > Task
> :beam-model-job-management:processResources UP-TO-DATE > Task
> :beam-runners-core-java:processResources NO-SOURCE > Task
> :beam-sdks-java-fn-execution:processResources NO-SOURCE > Task
> :beam-sdks-java-harness:processResources NO-SOURCE > Task
> :beam-vendor-sdks-java-extensions-protobuf:processResources NO-SOURCE > Task
> :beam-sdks-java-extensions-protobuf:processResources NO-SOURCE > Task
> :beam-sdks-java-core:generateAvroJava NO-SOURCE > Task
> :beam-vendor-grpc-v1_13_1:processResources NO-SOURCE > Task
> :beam-model-fn-execution:processResources UP-TO-DATE > Task
> :beam-model-pipeline:processResources UP-TO-DATE > Task
> :beam-runners-local-java-core:processResources NO-SOURCE > Task
> :beam-runners-java-fn-execution:processResources NO-SOURCE > Task
> :beam-runners-direct-java:processResources NO-SOURCE > Task
> :beam-vendor-grpc-v1_13_1:classes UP-TO-DATE > Task
> :beam-sdks-java-core:processResources UP-TO-DATE > Task
> :beam-runners-google-cloud-dataflow-java:processResources UP-TO-DATE > Task
> :beam-vendor-grpc-v1_13_1:shadowJar UP-TO-DATE > Task
> :beam-model-fn-execution:extractIncludeProto UP-TO-DATE > Task
> :beam-model-pipeline:extractIncludeProto UP-TO-DATE > Task
> :beam-model-job-management:extractIncludeProto UP-TO-DATE > Task
> :beam-model-pipeline:generateProto UP-TO-DATE > Task
> :beam-model-pipeline:compileJava UP-TO-DATE > Task
> :beam-model-pipeline:classes UP-TO-DATE > Task :beam-model-pipeline:shadowJar
> UP-TO-DATE > Task :beam-model-pipeline:jar UP-TO-DATE > Task
> :beam-model-job-management:extractProto UP-TO-DATE > Task
> :beam-model-fn-execution:extractProto UP-TO-DATE > Task
> :beam-model-fn-execution:generateProto UP-TO-DATE > Task
> :beam-model-job-management:generateProto UP-TO-DATE > Task
> :beam-model-job-management:compileJava UP-TO-DATE > Task
> :beam-model-job-management:classes UP-TO-DATE > Task
> :beam-model-fn-execution:compileJava UP-TO-DATE > Task
> :beam-model-fn-execution:classes UP-TO-DATE > Task
> :beam-model-job-management:shadowJar UP-TO-DATE > Task
> :beam-sdks-java-core:compileJava UP-TO-DATE > Task
> :beam-sdks-java-core:classes UP-TO-DATE > Task
> :beam-model-fn-execution:shadowJar UP-TO-DATE > Task
> :beam-sdks-java-core:shadowJar UP-TO-DATE > Task
>
[jira] [Updated] (BEAM-6096) illegal signature attribute when compiling with JDK 11
[ https://issues.apache.org/jira/browse/BEAM-6096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-6096: -- Component/s: (was: beam-model) sdk-java-core > illegal signature attribute when compiling with JDK 11 > -- > > Key: BEAM-6096 > URL: https://issues.apache.org/jira/browse/BEAM-6096 > Project: Beam > Issue Type: Bug > Components: sdk-java-core >Reporter: Liam Miller-Cushon >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > The following code is invalid, and fails to compile with the JDK 11 javac: > https://github.com/apache/beam/blob/1cbbc4d8755130f137e7d9ef4813e8d3b917cb70/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/WindowMergingFnRunner.java#L132-L144 > The error is: > ``` > error: error while generating class org.apache.beam.fn.harness.WindowMergingFnRunner$MergingViaWindowFnRunner$1> > (illegal signature attribute for type CAP#1) > where CAP#1 is a fresh type-variable: > CAP#1 extends Object from capture of ? > ``` > The code was previously accepted by javac due to a bug, which has been fixed > in JDK 11: https://bugs.openjdk.java.net/browse/JDK-8203436 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (BEAM-6096) illegal signature attribute when compiling with JDK 11
[ https://issues.apache.org/jira/browse/BEAM-6096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles reassigned BEAM-6096: - Assignee: (was: Kenneth Knowles) > illegal signature attribute when compiling with JDK 11 > -- > > Key: BEAM-6096 > URL: https://issues.apache.org/jira/browse/BEAM-6096 > Project: Beam > Issue Type: Sub-task > Components: sdk-java-core >Reporter: Liam Miller-Cushon >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > The following code is invalid, and fails to compile with the JDK 11 javac: > https://github.com/apache/beam/blob/1cbbc4d8755130f137e7d9ef4813e8d3b917cb70/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/WindowMergingFnRunner.java#L132-L144 > The error is: > ``` > error: error while generating class org.apache.beam.fn.harness.WindowMergingFnRunner$MergingViaWindowFnRunner$1> > (illegal signature attribute for type CAP#1) > where CAP#1 is a fresh type-variable: > CAP#1 extends Object from capture of ? > ``` > The code was previously accepted by javac due to a bug, which has been fixed > in JDK 11: https://bugs.openjdk.java.net/browse/JDK-8203436 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-6098) Support side input join on right or left
Kenneth Knowles created BEAM-6098: - Summary: Support side input join on right or left Key: BEAM-6098 URL: https://issues.apache.org/jira/browse/BEAM-6098 Project: Beam Issue Type: Improvement Components: dsl-sql Reporter: Kenneth Knowles Assignee: Kenneth Knowles Currently, side input join is hardcoded as to which of left vs right must be seekable/non-seekable. Making this symmetric would be an incremental improvement. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-6096) illegal signature attribute when compiling with JDK 11
[ https://issues.apache.org/jira/browse/BEAM-6096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692622#comment-16692622 ] Kenneth Knowles commented on BEAM-6096: --- [~lcwik] from git blame it looks like you might be the best person to triage? [~cushon] would you like to submit a fix? :-) > illegal signature attribute when compiling with JDK 11 > -- > > Key: BEAM-6096 > URL: https://issues.apache.org/jira/browse/BEAM-6096 > Project: Beam > Issue Type: Sub-task > Components: sdk-java-core >Reporter: Liam Miller-Cushon >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > The following code is invalid, and fails to compile with the JDK 11 javac: > https://github.com/apache/beam/blob/1cbbc4d8755130f137e7d9ef4813e8d3b917cb70/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/WindowMergingFnRunner.java#L132-L144 > The error is: > ``` > error: error while generating class org.apache.beam.fn.harness.WindowMergingFnRunner$MergingViaWindowFnRunner$1> > (illegal signature attribute for type CAP#1) > where CAP#1 is a fresh type-variable: > CAP#1 extends Object from capture of ? > ``` > The code was previously accepted by javac due to a bug, which has been fixed > in JDK 11: https://bugs.openjdk.java.net/browse/JDK-8203436 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (BEAM-6096) illegal signature attribute when compiling with JDK 11
[ https://issues.apache.org/jira/browse/BEAM-6096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles reassigned BEAM-6096: - Assignee: Liam Miller-Cushon > illegal signature attribute when compiling with JDK 11 > -- > > Key: BEAM-6096 > URL: https://issues.apache.org/jira/browse/BEAM-6096 > Project: Beam > Issue Type: Sub-task > Components: sdk-java-core >Reporter: Liam Miller-Cushon >Assignee: Liam Miller-Cushon >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > The following code is invalid, and fails to compile with the JDK 11 javac: > https://github.com/apache/beam/blob/1cbbc4d8755130f137e7d9ef4813e8d3b917cb70/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/WindowMergingFnRunner.java#L132-L144 > The error is: > ``` > error: error while generating class org.apache.beam.fn.harness.WindowMergingFnRunner$MergingViaWindowFnRunner$1> > (illegal signature attribute for type CAP#1) > where CAP#1 is a fresh type-variable: > CAP#1 extends Object from capture of ? > ``` > The code was previously accepted by javac due to a bug, which has been fixed > in JDK 11: https://bugs.openjdk.java.net/browse/JDK-8203436 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-6097) Add NemoRunner
[ https://issues.apache.org/jira/browse/BEAM-6097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692707#comment-16692707 ] Kenneth Knowles commented on BEAM-6097: --- Done. And I have added you to the "Contributor" Jira role so you can assign yourself any issue you like. > Add NemoRunner > -- > > Key: BEAM-6097 > URL: https://issues.apache.org/jira/browse/BEAM-6097 > Project: Beam > Issue Type: New Feature > Components: runner-ideas >Reporter: Won Wook SONG >Assignee: Kenneth Knowles >Priority: Major > > Add NemoRunner (http://nemo.apache.org) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (BEAM-6097) Add NemoRunner
[ https://issues.apache.org/jira/browse/BEAM-6097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles reassigned BEAM-6097: - Assignee: Won Wook SONG (was: Kenneth Knowles) > Add NemoRunner > -- > > Key: BEAM-6097 > URL: https://issues.apache.org/jira/browse/BEAM-6097 > Project: Beam > Issue Type: New Feature > Components: runner-ideas >Reporter: Won Wook SONG >Assignee: Won Wook SONG >Priority: Major > > Add NemoRunner (http://nemo.apache.org) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-6096) illegal signature attribute when compiling with JDK 11
[ https://issues.apache.org/jira/browse/BEAM-6096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-6096: -- Issue Type: Sub-task (was: Bug) Parent: BEAM-2530 > illegal signature attribute when compiling with JDK 11 > -- > > Key: BEAM-6096 > URL: https://issues.apache.org/jira/browse/BEAM-6096 > Project: Beam > Issue Type: Sub-task > Components: sdk-java-core >Reporter: Liam Miller-Cushon >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > The following code is invalid, and fails to compile with the JDK 11 javac: > https://github.com/apache/beam/blob/1cbbc4d8755130f137e7d9ef4813e8d3b917cb70/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/WindowMergingFnRunner.java#L132-L144 > The error is: > ``` > error: error while generating class org.apache.beam.fn.harness.WindowMergingFnRunner$MergingViaWindowFnRunner$1> > (illegal signature attribute for type CAP#1) > where CAP#1 is a fresh type-variable: > CAP#1 extends Object from capture of ? > ``` > The code was previously accepted by javac due to a bug, which has been fixed > in JDK 11: https://bugs.openjdk.java.net/browse/JDK-8203436 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-6111) org.apache.beam.runners.flink.PortableTimersExecutionTest is very flakey 40/50 runs failed
[ https://issues.apache.org/jira/browse/BEAM-6111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-6111: -- Labels: flake (was: ) > org.apache.beam.runners.flink.PortableTimersExecutionTest is very flakey > 40/50 runs failed > -- > > Key: BEAM-6111 > URL: https://issues.apache.org/jira/browse/BEAM-6111 > Project: Beam > Issue Type: Test > Components: runner-flink >Reporter: Alex Amato >Assignee: Maximilian Michels >Priority: Major > Labels: flake > > I ran this test 50 on master and it failed about 40/50 runs. I have also seen > this failing in jenkins presubmit. > build scan links from java presubmit > https://builds.apache.org/job/beam_PreCommit_Java_Commit/2654/ > build scan links when I ran locally: > [https://scans.gradle.com/s/hzu3bojfdbd5s] > [https://scans.gradle.com/s/txhyj6acebkko] > [https://scans.gradle.com/s/nhju5jthjbdds] > org.junit.runners.model.TestTimedOutException: test timed out after 6 > milliseconds > at java.lang.ClassLoader.findBootstrapClass(Native Method) > at java.lang.ClassLoader.findBootstrapClassOrNull(ClassLoader.java:1015) > at java.lang.ClassLoader.loadClass(ClassLoader.java:413) > at java.lang.ClassLoader.loadClass(ClassLoader.java:411) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:339) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:13) > at org.junit.Assert.assertThat(Assert.java:956) > at org.junit.Assert.assertThat(Assert.java:923) > at > org.apache.beam.runners.flink.PortableTimersExecutionTest.testTimerExecution(PortableTimersExecutionTest.java:191) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) > at > org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at java.lang.Thread.run(Thread.java:748) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (BEAM-6082) [SQL] Nexmark 5, 7 time out
[ https://issues.apache.org/jira/browse/BEAM-6082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles resolved BEAM-6082. --- Resolution: Fixed > [SQL] Nexmark 5, 7 time out > --- > > Key: BEAM-6082 > URL: https://issues.apache.org/jira/browse/BEAM-6082 > Project: Beam > Issue Type: Bug > Components: dsl-sql >Reporter: Anton Kedin >Assignee: Kenneth Knowles >Priority: Blocker > Fix For: 2.9.0 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > SQL Nexmark Queries 5 and 7 are timing out since around October 29: > [https://apache-beam-testing.appspot.com/explore?dashboard=5084698770407424] > > Current suspect is https://github.com/apache/beam/pull/6856 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-6098) Support lookup join on right or left
[ https://issues.apache.org/jira/browse/BEAM-6098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-6098: -- Summary: Support lookup join on right or left (was: Support side input join on right or left) > Support lookup join on right or left > > > Key: BEAM-6098 > URL: https://issues.apache.org/jira/browse/BEAM-6098 > Project: Beam > Issue Type: Improvement > Components: dsl-sql >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Fix For: 2.10.0 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > Currently, side input join is hardcoded as to which of left vs right must be > seekable/non-seekable. Making this symmetric would be an incremental > improvement. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-6098) Support lookup join on right or left
[ https://issues.apache.org/jira/browse/BEAM-6098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-6098: -- Description: Currently, lookup join is hardcoded as to which of left vs right must be seekable/non-seekable. Making this symmetric would be an incremental improvement. (was: Currently, side input join is hardcoded as to which of left vs right must be seekable/non-seekable. Making this symmetric would be an incremental improvement.) > Support lookup join on right or left > > > Key: BEAM-6098 > URL: https://issues.apache.org/jira/browse/BEAM-6098 > Project: Beam > Issue Type: Improvement > Components: dsl-sql >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Fix For: 2.10.0 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > Currently, lookup join is hardcoded as to which of left vs right must be > seekable/non-seekable. Making this symmetric would be an incremental > improvement. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (BEAM-6098) Support lookup join on right or left
[ https://issues.apache.org/jira/browse/BEAM-6098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles resolved BEAM-6098. --- Resolution: Fixed Fix Version/s: 2.10.0 > Support lookup join on right or left > > > Key: BEAM-6098 > URL: https://issues.apache.org/jira/browse/BEAM-6098 > Project: Beam > Issue Type: Improvement > Components: dsl-sql >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Fix For: 2.10.0 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > Currently, lookup join is hardcoded as to which of left vs right must be > seekable/non-seekable. Making this symmetric would be an incremental > improvement. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (BEAM-5817) Nexmark test of joining stream to files
[ https://issues.apache.org/jira/browse/BEAM-5817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles resolved BEAM-5817. --- Resolution: Fixed Fix Version/s: 2.10.0 > Nexmark test of joining stream to files > --- > > Key: BEAM-5817 > URL: https://issues.apache.org/jira/browse/BEAM-5817 > Project: Beam > Issue Type: New Feature > Components: examples-nexmark >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Fix For: 2.10.0 > > Time Spent: 7h 40m > Remaining Estimate: 0h > > Nexmark is a convenient framework for testing the use case of large scale > stream enrichment. One way is joining a stream to files, and it can be tested > via any source that Nexmark supports. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (BEAM-6109) Beam SQL rejects join of unbounded to bounded, though a side input join does succeed
[ https://issues.apache.org/jira/browse/BEAM-6109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles resolved BEAM-6109. --- Resolution: Fixed Fix Version/s: 2.10.0 > Beam SQL rejects join of unbounded to bounded, though a side input join does > succeed > > > Key: BEAM-6109 > URL: https://issues.apache.org/jira/browse/BEAM-6109 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Fix For: 2.10.0 > > > Current, verification of supporting triggering and windowing is done before > deciding whether to do a side input join. In the case of globally windowed > data, this is too strict. We can relax it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (BEAM-5265) Can not test Timer with processing time domain
[ https://issues.apache.org/jira/browse/BEAM-5265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles reassigned BEAM-5265: - Assignee: Jozef Vilcek (was: Kenneth Knowles) > Can not test Timer with processing time domain > -- > > Key: BEAM-5265 > URL: https://issues.apache.org/jira/browse/BEAM-5265 > Project: Beam > Issue Type: Bug > Components: runner-core, runner-direct >Reporter: Jozef Vilcek >Assignee: Jozef Vilcek >Priority: Major > Time Spent: 3.5h > Remaining Estimate: 0h > > I have a stateful DoFn which has a timer on PROCESSING_TIME domain. While > writing tests, I noticed that it does not react to `advanceProcessingTime()` > on tests stream. Problem seems to be here: > [https://github.com/apache/beam/blob/master/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java#L260] > I can only tell that patching this place works for direct runner tests. Not > sure about broader impact on other runners since it is in `runner-core` -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-6082) [SQL] Nexmark 5, 7 time out
[ https://issues.apache.org/jira/browse/BEAM-6082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695556#comment-16695556 ] Kenneth Knowles commented on BEAM-6082: --- So the Dataflow stuff on the graph was just jobs failing and timing out from crash looping, etc? No real slowdown outside of DirectRunner? > [SQL] Nexmark 5, 7 time out > --- > > Key: BEAM-6082 > URL: https://issues.apache.org/jira/browse/BEAM-6082 > Project: Beam > Issue Type: Bug > Components: dsl-sql >Reporter: Anton Kedin >Assignee: Reuven Lax >Priority: Blocker > Fix For: 2.9.0 > > > SQL Nexmark Queries 5 and 7 are timing out since around October 29: > [https://apache-beam-testing.appspot.com/explore?dashboard=5084698770407424] > > Current suspect is https://github.com/apache/beam/pull/6856 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-6114) SQL join selection should be done in planner, not in expansion to PTransform
Kenneth Knowles created BEAM-6114: - Summary: SQL join selection should be done in planner, not in expansion to PTransform Key: BEAM-6114 URL: https://issues.apache.org/jira/browse/BEAM-6114 Project: Beam Issue Type: Improvement Components: dsl-sql Reporter: Kenneth Knowles Assignee: Kenneth Knowles Currently Beam SQL joins all go through a single physical operator which has a single PTransform that does all join algorithms based on properties of its input PCollections as well as the relational algebra. A first step is to make the needed information part of the relational algebra, so it can choose a PTransform based on that, and the PTransforms can be simpler. Second step is to have separate (physical) relational operators for different join algorithms. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-6082) [SQL] Nexmark 5, 7 time out
[ https://issues.apache.org/jira/browse/BEAM-6082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16690131#comment-16690131 ] Kenneth Knowles commented on BEAM-6082: --- So far, I have not reproduced any change in performance going back across a wide variety of commits :-/ > [SQL] Nexmark 5, 7 time out > --- > > Key: BEAM-6082 > URL: https://issues.apache.org/jira/browse/BEAM-6082 > Project: Beam > Issue Type: Bug > Components: dsl-sql >Reporter: Anton Kedin >Assignee: Kenneth Knowles >Priority: Blocker > Fix For: 2.9.0 > > > SQL Nexmark Queries 5 and 7 are timing out since around October 29: > [https://apache-beam-testing.appspot.com/explore?dashboard=5084698770407424] > > Current suspect is https://github.com/apache/beam/pull/6856 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-6082) [SQL] Nexmark 5, 7 time out
[ https://issues.apache.org/jira/browse/BEAM-6082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691223#comment-16691223 ] Kenneth Knowles commented on BEAM-6082: --- The evidence is just the timing of the trouble starting. You can see the charts here: Dataflow: https://apache-beam-testing.appspot.com/explore?dashboard=5670405876482048 Direct: https://apache-beam-testing.appspot.com/explore?dashboard=5084698770407424 For query 5 it is a 5x slowdown on Dataflow and DirectRunner. I don't see the same on Spark and it seems like SQL isn't running on Flink. For query 7 it looks like there was a major slowdown but then no more data. > [SQL] Nexmark 5, 7 time out > --- > > Key: BEAM-6082 > URL: https://issues.apache.org/jira/browse/BEAM-6082 > Project: Beam > Issue Type: Bug > Components: dsl-sql >Reporter: Anton Kedin >Assignee: Kenneth Knowles >Priority: Blocker > Fix For: 2.9.0 > > > SQL Nexmark Queries 5 and 7 are timing out since around October 29: > [https://apache-beam-testing.appspot.com/explore?dashboard=5084698770407424] > > Current suspect is https://github.com/apache/beam/pull/6856 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-6082) [SQL] Nexmark 5, 7 time out
[ https://issues.apache.org/jira/browse/BEAM-6082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16693620#comment-16693620 ] Kenneth Knowles commented on BEAM-6082: --- The Dataflow nexmark runs have been unhealthy and just got fixed by https://github.com/apache/beam/pull/7041. It looks like maybe SQL query 5 in batch is OK while SQL query 5 in streaming is still bad. The signal is really poor here, unfortunately. I haven't done real profiling, just vague association of timing and a little bit of git bisecting (which yielded no info as yet). > [SQL] Nexmark 5, 7 time out > --- > > Key: BEAM-6082 > URL: https://issues.apache.org/jira/browse/BEAM-6082 > Project: Beam > Issue Type: Bug > Components: dsl-sql >Reporter: Anton Kedin >Assignee: Reuven Lax >Priority: Blocker > Fix For: 2.9.0 > > > SQL Nexmark Queries 5 and 7 are timing out since around October 29: > [https://apache-beam-testing.appspot.com/explore?dashboard=5084698770407424] > > Current suspect is https://github.com/apache/beam/pull/6856 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5197) Flaky test: UnboundedSourceWrapperTest$ParameterizedUnboundedSourceWrapperTest
[
https://issues.apache.org/jira/browse/BEAM-5197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699800#comment-16699800
]
Kenneth Knowles commented on BEAM-5197:
---
I tried bumping the timeout on each test case from 30 seconds to 60 seconds and
it did not alleviate the issue.
> Flaky test: UnboundedSourceWrapperTest$ParameterizedUnboundedSourceWrapperTest
> --
>
> Key: BEAM-5197
> URL: https://issues.apache.org/jira/browse/BEAM-5197
> Project: Beam
> Issue Type: Improvement
> Components: runner-flink
>Reporter: Thomas Weise
>Assignee: Kenneth Knowles
>Priority: Critical
> Labels: flake
> Fix For: 2.10.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> {code:java}
> java.lang.NullPointerException
> at
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService$1.compare(TestProcessingTimeService.java:52)
> at
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService$1.compare(TestProcessingTimeService.java:49)
> at java.util.PriorityQueue.siftUpUsingComparator(PriorityQueue.java:670)
> at java.util.PriorityQueue.siftUp(PriorityQueue.java:646)
> at java.util.PriorityQueue.offer(PriorityQueue.java:345)
> at
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService.registerTimer(TestProcessingTimeService.java:93)
> at
> org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper.setNextWatermarkTimer(UnboundedSourceWrapper.java:452)
> at
> org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper.run(UnboundedSourceWrapper.java:225)
> at
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:87)
> at
> org.apache.beam.runners.flink.streaming.UnboundedSourceWrapperTest$ParameterizedUnboundedSourceWrapperTest.testValueEmission(UnboundedSourceWrapperTest.java:153)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-5197) Flaky test: UnboundedSourceWrapperTest$ParameterizedUnboundedSourceWrapperTest
[
https://issues.apache.org/jira/browse/BEAM-5197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699849#comment-16699849
]
Kenneth Knowles commented on BEAM-5197:
---
Perhaps meaningfully, this test completes extremely quickly on my OSX laptop
but times out (due to deadlock or race?) on my Linux desktop.
> Flaky test: UnboundedSourceWrapperTest$ParameterizedUnboundedSourceWrapperTest
> --
>
> Key: BEAM-5197
> URL: https://issues.apache.org/jira/browse/BEAM-5197
> Project: Beam
> Issue Type: Improvement
> Components: runner-flink
>Reporter: Thomas Weise
>Assignee: Kenneth Knowles
>Priority: Critical
> Labels: flake
> Fix For: 2.10.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> {code:java}
> java.lang.NullPointerException
> at
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService$1.compare(TestProcessingTimeService.java:52)
> at
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService$1.compare(TestProcessingTimeService.java:49)
> at java.util.PriorityQueue.siftUpUsingComparator(PriorityQueue.java:670)
> at java.util.PriorityQueue.siftUp(PriorityQueue.java:646)
> at java.util.PriorityQueue.offer(PriorityQueue.java:345)
> at
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService.registerTimer(TestProcessingTimeService.java:93)
> at
> org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper.setNextWatermarkTimer(UnboundedSourceWrapper.java:452)
> at
> org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper.run(UnboundedSourceWrapper.java:225)
> at
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:87)
> at
> org.apache.beam.runners.flink.streaming.UnboundedSourceWrapperTest$ParameterizedUnboundedSourceWrapperTest.testValueEmission(UnboundedSourceWrapperTest.java:153)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (BEAM-5197) Flaky test: UnboundedSourceWrapperTest$ParameterizedUnboundedSourceWrapperTest
[
https://issues.apache.org/jira/browse/BEAM-5197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles reassigned BEAM-5197:
-
Assignee: Kenneth Knowles (was: Maximilian Michels)
> Flaky test: UnboundedSourceWrapperTest$ParameterizedUnboundedSourceWrapperTest
> --
>
> Key: BEAM-5197
> URL: https://issues.apache.org/jira/browse/BEAM-5197
> Project: Beam
> Issue Type: Improvement
> Components: runner-flink
>Reporter: Thomas Weise
>Assignee: Kenneth Knowles
>Priority: Critical
> Labels: flake
> Fix For: 2.10.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> {code:java}
> java.lang.NullPointerException
> at
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService$1.compare(TestProcessingTimeService.java:52)
> at
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService$1.compare(TestProcessingTimeService.java:49)
> at java.util.PriorityQueue.siftUpUsingComparator(PriorityQueue.java:670)
> at java.util.PriorityQueue.siftUp(PriorityQueue.java:646)
> at java.util.PriorityQueue.offer(PriorityQueue.java:345)
> at
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService.registerTimer(TestProcessingTimeService.java:93)
> at
> org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper.setNextWatermarkTimer(UnboundedSourceWrapper.java:452)
> at
> org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper.run(UnboundedSourceWrapper.java:225)
> at
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:87)
> at
> org.apache.beam.runners.flink.streaming.UnboundedSourceWrapperTest$ParameterizedUnboundedSourceWrapperTest.testValueEmission(UnboundedSourceWrapperTest.java:153)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-6129) Flink PortableExecutionTest testExecution is flaky or broken
[ https://issues.apache.org/jira/browse/BEAM-6129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699884#comment-16699884 ] Kenneth Knowles commented on BEAM-6129: --- https://builds.apache.org/job/beam_PreCommit_Java_Cron/622/ https://builds.apache.org/job/beam_PreCommit_Java_Cron/626/ https://builds.apache.org/job/beam_PreCommit_Java_Cron/627/ > Flink PortableExecutionTest testExecution is flaky or broken > - > > Key: BEAM-6129 > URL: https://issues.apache.org/jira/browse/BEAM-6129 > Project: Beam > Issue Type: Bug > Components: runner-flink, test-failures >Reporter: Kenneth Knowles >Priority: Critical > Labels: flake > > Example failure: > https://builds.apache.org/job/beam_PreCommit_Java_Commit/2697/ > Job seems to be in state RUNNING when it should be DONE. I didn't have a > chance to look more deeply. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-6130) Flink RemoteExecutionTest testExecutionWithMultipleStages is flaky
Kenneth Knowles created BEAM-6130: - Summary: Flink RemoteExecutionTest testExecutionWithMultipleStages is flaky Key: BEAM-6130 URL: https://issues.apache.org/jira/browse/BEAM-6130 Project: Beam Issue Type: Bug Components: runner-flink, test-failures Reporter: Kenneth Knowles https://builds.apache.org/job/beam_PreCommit_Java_Cron/620/ https://builds.apache.org/job/beam_PreCommit_Java_Cron/623/ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-6132) GrpcDataServiceTest.testMessageReceivedBySingleClientWhenThereAreMultipleClients
Kenneth Knowles created BEAM-6132: - Summary: GrpcDataServiceTest.testMessageReceivedBySingleClientWhenThereAreMultipleClients Key: BEAM-6132 URL: https://issues.apache.org/jira/browse/BEAM-6132 Project: Beam Issue Type: Bug Components: runner-core, test-failures Reporter: Kenneth Knowles Assignee: Luke Cwik https://builds.apache.org/job/beam_PreCommit_Java_Cron/607/ https://builds.apache.org/job/beam_PreCommit_Java_Cron/620/ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-6131) ElasticSearchIO is flaky: testRead, testWrite, testWritePartialUpdate classMethod
Kenneth Knowles created BEAM-6131: - Summary: ElasticSearchIO is flaky: testRead, testWrite, testWritePartialUpdate classMethod Key: BEAM-6131 URL: https://issues.apache.org/jira/browse/BEAM-6131 Project: Beam Issue Type: Bug Components: runner-flink, test-failures Reporter: Kenneth Knowles https://builds.apache.org/job/beam_PreCommit_Java_Cron/611/ https://builds.apache.org/job/beam_PreCommit_Java_Cron/615/ https://builds.apache.org/job/beam_PreCommit_Java_Cron/622/ I would add that it is hard at a glance to tell which ElasticSearchIO. I would recommend using composition instead of inheritance or scanning tests from another jar. That would make things clearer. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-6131) ElasticSearchIO is flaky: testRead, testWrite, testWritePartialUpdate classMethod
[ https://issues.apache.org/jira/browse/BEAM-6131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699887#comment-16699887 ] Kenneth Knowles commented on BEAM-6131: --- Looked at a couple scans and it looks like it is ES 2 that is flaking. > ElasticSearchIO is flaky: testRead, testWrite, testWritePartialUpdate > classMethod > - > > Key: BEAM-6131 > URL: https://issues.apache.org/jira/browse/BEAM-6131 > Project: Beam > Issue Type: Bug > Components: runner-flink, test-failures >Reporter: Kenneth Knowles >Priority: Critical > Labels: flake > > https://builds.apache.org/job/beam_PreCommit_Java_Cron/611/ > https://builds.apache.org/job/beam_PreCommit_Java_Cron/615/ > https://builds.apache.org/job/beam_PreCommit_Java_Cron/622/ > I would add that it is hard at a glance to tell which ElasticSearchIO. I > would recommend using composition instead of inheritance or scanning tests > from another jar. That would make things clearer. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-6130) Flink RemoteExecutionTest testExecutionWithMultipleStages is flaky
[ https://issues.apache.org/jira/browse/BEAM-6130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699925#comment-16699925 ] Kenneth Knowles commented on BEAM-6130: --- https://builds.apache.org/job/beam_PreCommit_Java_Commit/2706/ > Flink RemoteExecutionTest testExecutionWithMultipleStages is flaky > -- > > Key: BEAM-6130 > URL: https://issues.apache.org/jira/browse/BEAM-6130 > Project: Beam > Issue Type: Bug > Components: runner-flink, test-failures >Reporter: Kenneth Knowles >Priority: Critical > Labels: flake > > https://builds.apache.org/job/beam_PreCommit_Java_Cron/620/ > https://builds.apache.org/job/beam_PreCommit_Java_Cron/623/ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-6129) Flink PortableExecutionTest testExecution is flaky or broken
[ https://issues.apache.org/jira/browse/BEAM-6129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699926#comment-16699926 ] Kenneth Knowles commented on BEAM-6129: --- https://builds.apache.org/job/beam_PreCommit_Java_Commit/2706/ > Flink PortableExecutionTest testExecution is flaky or broken > - > > Key: BEAM-6129 > URL: https://issues.apache.org/jira/browse/BEAM-6129 > Project: Beam > Issue Type: Bug > Components: runner-flink, test-failures >Reporter: Kenneth Knowles >Priority: Critical > Labels: flake > > Example failure: > https://builds.apache.org/job/beam_PreCommit_Java_Commit/2697/ > Job seems to be in state RUNNING when it should be DONE. I didn't have a > chance to look more deeply. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (BEAM-6094) Implement External environment for Portable Beam
[ https://issues.apache.org/jira/browse/BEAM-6094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles reassigned BEAM-6094: - Assignee: Robert Bradshaw (was: Kenneth Knowles) > Implement External environment for Portable Beam > > > Key: BEAM-6094 > URL: https://issues.apache.org/jira/browse/BEAM-6094 > Project: Beam > Issue Type: Improvement > Components: beam-model >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (BEAM-6082) [SQL] Nexmark 5, 7 time out
[ https://issues.apache.org/jira/browse/BEAM-6082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles reassigned BEAM-6082: - Assignee: Reuven Lax (was: Kenneth Knowles) > [SQL] Nexmark 5, 7 time out > --- > > Key: BEAM-6082 > URL: https://issues.apache.org/jira/browse/BEAM-6082 > Project: Beam > Issue Type: Bug > Components: dsl-sql >Reporter: Anton Kedin >Assignee: Reuven Lax >Priority: Blocker > Fix For: 2.9.0 > > > SQL Nexmark Queries 5 and 7 are timing out since around October 29: > [https://apache-beam-testing.appspot.com/explore?dashboard=5084698770407424] > > Current suspect is https://github.com/apache/beam/pull/6856 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-6082) [SQL] Nexmark 5, 7 time out
[ https://issues.apache.org/jira/browse/BEAM-6082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692253#comment-16692253 ] Kenneth Knowles commented on BEAM-6082: --- Super easy. There are instructions on the web site. I copied them to the wiki since they are not really for users: https://cwiki.apache.org/confluence/display/BEAM/Running+Nexmark > [SQL] Nexmark 5, 7 time out > --- > > Key: BEAM-6082 > URL: https://issues.apache.org/jira/browse/BEAM-6082 > Project: Beam > Issue Type: Bug > Components: dsl-sql >Reporter: Anton Kedin >Assignee: Kenneth Knowles >Priority: Blocker > Fix For: 2.9.0 > > > SQL Nexmark Queries 5 and 7 are timing out since around October 29: > [https://apache-beam-testing.appspot.com/explore?dashboard=5084698770407424] > > Current suspect is https://github.com/apache/beam/pull/6856 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5197) Flaky test: UnboundedSourceWrapperTest$ParameterizedUnboundedSourceWrapperTest
[
https://issues.apache.org/jira/browse/BEAM-5197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699657#comment-16699657
]
Kenneth Knowles commented on BEAM-5197:
---
Well, not quite. But here are a few:
https://builds.apache.org/job/beam_PreCommit_Java_Cron/622/
https://builds.apache.org/job/beam_PreCommit_Java_Cron/627/
https://builds.apache.org/job/beam_PreCommit_Java_Cron/631/
https://builds.apache.org/job/beam_PreCommit_Java_Cron/633/
> Flaky test: UnboundedSourceWrapperTest$ParameterizedUnboundedSourceWrapperTest
> --
>
> Key: BEAM-5197
> URL: https://issues.apache.org/jira/browse/BEAM-5197
> Project: Beam
> Issue Type: Improvement
> Components: runner-flink
>Reporter: Thomas Weise
>Assignee: Maximilian Michels
>Priority: Major
> Fix For: 2.10.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> {code:java}
> java.lang.NullPointerException
> at
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService$1.compare(TestProcessingTimeService.java:52)
> at
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService$1.compare(TestProcessingTimeService.java:49)
> at java.util.PriorityQueue.siftUpUsingComparator(PriorityQueue.java:670)
> at java.util.PriorityQueue.siftUp(PriorityQueue.java:646)
> at java.util.PriorityQueue.offer(PriorityQueue.java:345)
> at
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService.registerTimer(TestProcessingTimeService.java:93)
> at
> org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper.setNextWatermarkTimer(UnboundedSourceWrapper.java:452)
> at
> org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper.run(UnboundedSourceWrapper.java:225)
> at
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:87)
> at
> org.apache.beam.runners.flink.streaming.UnboundedSourceWrapperTest$ParameterizedUnboundedSourceWrapperTest.testValueEmission(UnboundedSourceWrapperTest.java:153)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-5197) Flaky test: UnboundedSourceWrapperTest$ParameterizedUnboundedSourceWrapperTest
[
https://issues.apache.org/jira/browse/BEAM-5197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699660#comment-16699660
]
Kenneth Knowles commented on BEAM-5197:
---
I'm seeing it a lot on PRs. If the test is broken and stopping work, and no
culprit for rollback, I think we should {{@Ignore}} it and raise the priority
of this ticket for reinstating it if it is important.
> Flaky test: UnboundedSourceWrapperTest$ParameterizedUnboundedSourceWrapperTest
> --
>
> Key: BEAM-5197
> URL: https://issues.apache.org/jira/browse/BEAM-5197
> Project: Beam
> Issue Type: Improvement
> Components: runner-flink
>Reporter: Thomas Weise
>Assignee: Maximilian Michels
>Priority: Major
> Fix For: 2.10.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> {code:java}
> java.lang.NullPointerException
> at
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService$1.compare(TestProcessingTimeService.java:52)
> at
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService$1.compare(TestProcessingTimeService.java:49)
> at java.util.PriorityQueue.siftUpUsingComparator(PriorityQueue.java:670)
> at java.util.PriorityQueue.siftUp(PriorityQueue.java:646)
> at java.util.PriorityQueue.offer(PriorityQueue.java:345)
> at
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService.registerTimer(TestProcessingTimeService.java:93)
> at
> org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper.setNextWatermarkTimer(UnboundedSourceWrapper.java:452)
> at
> org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper.run(UnboundedSourceWrapper.java:225)
> at
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:87)
> at
> org.apache.beam.runners.flink.streaming.UnboundedSourceWrapperTest$ParameterizedUnboundedSourceWrapperTest.testValueEmission(UnboundedSourceWrapperTest.java:153)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (BEAM-5197) Flaky test: UnboundedSourceWrapperTest$ParameterizedUnboundedSourceWrapperTest
[
https://issues.apache.org/jira/browse/BEAM-5197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles updated BEAM-5197:
--
Priority: Critical (was: Minor)
> Flaky test: UnboundedSourceWrapperTest$ParameterizedUnboundedSourceWrapperTest
> --
>
> Key: BEAM-5197
> URL: https://issues.apache.org/jira/browse/BEAM-5197
> Project: Beam
> Issue Type: Improvement
> Components: runner-flink
>Reporter: Thomas Weise
>Assignee: Maximilian Michels
>Priority: Critical
> Fix For: 2.10.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> {code:java}
> java.lang.NullPointerException
> at
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService$1.compare(TestProcessingTimeService.java:52)
> at
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService$1.compare(TestProcessingTimeService.java:49)
> at java.util.PriorityQueue.siftUpUsingComparator(PriorityQueue.java:670)
> at java.util.PriorityQueue.siftUp(PriorityQueue.java:646)
> at java.util.PriorityQueue.offer(PriorityQueue.java:345)
> at
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService.registerTimer(TestProcessingTimeService.java:93)
> at
> org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper.setNextWatermarkTimer(UnboundedSourceWrapper.java:452)
> at
> org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper.run(UnboundedSourceWrapper.java:225)
> at
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:87)
> at
> org.apache.beam.runners.flink.streaming.UnboundedSourceWrapperTest$ParameterizedUnboundedSourceWrapperTest.testValueEmission(UnboundedSourceWrapperTest.java:153)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (BEAM-5197) Flaky test: UnboundedSourceWrapperTest$ParameterizedUnboundedSourceWrapperTest
[
https://issues.apache.org/jira/browse/BEAM-5197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles updated BEAM-5197:
--
Priority: Minor (was: Major)
> Flaky test: UnboundedSourceWrapperTest$ParameterizedUnboundedSourceWrapperTest
> --
>
> Key: BEAM-5197
> URL: https://issues.apache.org/jira/browse/BEAM-5197
> Project: Beam
> Issue Type: Improvement
> Components: runner-flink
>Reporter: Thomas Weise
>Assignee: Maximilian Michels
>Priority: Minor
> Fix For: 2.10.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> {code:java}
> java.lang.NullPointerException
> at
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService$1.compare(TestProcessingTimeService.java:52)
> at
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService$1.compare(TestProcessingTimeService.java:49)
> at java.util.PriorityQueue.siftUpUsingComparator(PriorityQueue.java:670)
> at java.util.PriorityQueue.siftUp(PriorityQueue.java:646)
> at java.util.PriorityQueue.offer(PriorityQueue.java:345)
> at
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService.registerTimer(TestProcessingTimeService.java:93)
> at
> org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper.setNextWatermarkTimer(UnboundedSourceWrapper.java:452)
> at
> org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper.run(UnboundedSourceWrapper.java:225)
> at
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:87)
> at
> org.apache.beam.runners.flink.streaming.UnboundedSourceWrapperTest$ParameterizedUnboundedSourceWrapperTest.testValueEmission(UnboundedSourceWrapperTest.java:153)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (BEAM-5197) Flaky test: UnboundedSourceWrapperTest$ParameterizedUnboundedSourceWrapperTest
[
https://issues.apache.org/jira/browse/BEAM-5197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles updated BEAM-5197:
--
Labels: flake (was: )
> Flaky test: UnboundedSourceWrapperTest$ParameterizedUnboundedSourceWrapperTest
> --
>
> Key: BEAM-5197
> URL: https://issues.apache.org/jira/browse/BEAM-5197
> Project: Beam
> Issue Type: Improvement
> Components: runner-flink
>Reporter: Thomas Weise
>Assignee: Maximilian Michels
>Priority: Critical
> Labels: flake
> Fix For: 2.10.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> {code:java}
> java.lang.NullPointerException
> at
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService$1.compare(TestProcessingTimeService.java:52)
> at
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService$1.compare(TestProcessingTimeService.java:49)
> at java.util.PriorityQueue.siftUpUsingComparator(PriorityQueue.java:670)
> at java.util.PriorityQueue.siftUp(PriorityQueue.java:646)
> at java.util.PriorityQueue.offer(PriorityQueue.java:345)
> at
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService.registerTimer(TestProcessingTimeService.java:93)
> at
> org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper.setNextWatermarkTimer(UnboundedSourceWrapper.java:452)
> at
> org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper.run(UnboundedSourceWrapper.java:225)
> at
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:87)
> at
> org.apache.beam.runners.flink.streaming.UnboundedSourceWrapperTest$ParameterizedUnboundedSourceWrapperTest.testValueEmission(UnboundedSourceWrapperTest.java:153)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Reopened] (BEAM-5197) Flaky test: UnboundedSourceWrapperTest$ParameterizedUnboundedSourceWrapperTest
[
https://issues.apache.org/jira/browse/BEAM-5197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles reopened BEAM-5197:
---
>From what I can tell, this removed the flakiness and made the test permanently
>failing :-)
> Flaky test: UnboundedSourceWrapperTest$ParameterizedUnboundedSourceWrapperTest
> --
>
> Key: BEAM-5197
> URL: https://issues.apache.org/jira/browse/BEAM-5197
> Project: Beam
> Issue Type: Improvement
> Components: runner-flink
>Reporter: Thomas Weise
>Assignee: Maximilian Michels
>Priority: Major
> Fix For: 2.10.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> {code:java}
> java.lang.NullPointerException
> at
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService$1.compare(TestProcessingTimeService.java:52)
> at
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService$1.compare(TestProcessingTimeService.java:49)
> at java.util.PriorityQueue.siftUpUsingComparator(PriorityQueue.java:670)
> at java.util.PriorityQueue.siftUp(PriorityQueue.java:646)
> at java.util.PriorityQueue.offer(PriorityQueue.java:345)
> at
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService.registerTimer(TestProcessingTimeService.java:93)
> at
> org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper.setNextWatermarkTimer(UnboundedSourceWrapper.java:452)
> at
> org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper.run(UnboundedSourceWrapper.java:225)
> at
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:87)
> at
> org.apache.beam.runners.flink.streaming.UnboundedSourceWrapperTest$ParameterizedUnboundedSourceWrapperTest.testValueEmission(UnboundedSourceWrapperTest.java:153)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-6082) [SQL] Nexmark 5, 7 time out
[ https://issues.apache.org/jira/browse/BEAM-6082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699664#comment-16699664 ] Kenneth Knowles commented on BEAM-6082: --- I've now reproduced this locally so a fix should be forthcoming. I know the suite does work, since I also run it pretty often. It has got to be something to do with the configurations. > [SQL] Nexmark 5, 7 time out > --- > > Key: BEAM-6082 > URL: https://issues.apache.org/jira/browse/BEAM-6082 > Project: Beam > Issue Type: Bug > Components: dsl-sql >Reporter: Anton Kedin >Assignee: Reuven Lax >Priority: Blocker > Fix For: 2.9.0 > > > SQL Nexmark Queries 5 and 7 are timing out since around October 29: > [https://apache-beam-testing.appspot.com/explore?dashboard=5084698770407424] > > Current suspect is https://github.com/apache/beam/pull/6856 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (BEAM-6082) [SQL] Nexmark 5, 7 time out
[ https://issues.apache.org/jira/browse/BEAM-6082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles reassigned BEAM-6082: - Assignee: Kenneth Knowles (was: Reuven Lax) > [SQL] Nexmark 5, 7 time out > --- > > Key: BEAM-6082 > URL: https://issues.apache.org/jira/browse/BEAM-6082 > Project: Beam > Issue Type: Bug > Components: dsl-sql >Reporter: Anton Kedin >Assignee: Kenneth Knowles >Priority: Blocker > Fix For: 2.9.0 > > > SQL Nexmark Queries 5 and 7 are timing out since around October 29: > [https://apache-beam-testing.appspot.com/explore?dashboard=5084698770407424] > > Current suspect is https://github.com/apache/beam/pull/6856 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (BEAM-6082) [SQL] Nexmark 5, 7 time out
[ https://issues.apache.org/jira/browse/BEAM-6082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699611#comment-16699611 ] Kenneth Knowles edited comment on BEAM-6082 at 11/26/18 9:46 PM: - There is still no data for SQL query 7 in Direct runner since 2018-10-29 and Dataflow runner since 2018-10-26. was (Author: kenn): There is still no data for SQL query 7 in Direct or Dataflow runner since 2018-10-29. > [SQL] Nexmark 5, 7 time out > --- > > Key: BEAM-6082 > URL: https://issues.apache.org/jira/browse/BEAM-6082 > Project: Beam > Issue Type: Bug > Components: dsl-sql >Reporter: Anton Kedin >Assignee: Reuven Lax >Priority: Blocker > Fix For: 2.9.0 > > > SQL Nexmark Queries 5 and 7 are timing out since around October 29: > [https://apache-beam-testing.appspot.com/explore?dashboard=5084698770407424] > > Current suspect is https://github.com/apache/beam/pull/6856 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-6082) [SQL] Nexmark 5, 7 time out
[ https://issues.apache.org/jira/browse/BEAM-6082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699627#comment-16699627 ] Kenneth Knowles commented on BEAM-6082: --- Yea browsing it does look like all the SQL runs across all runners do queries 0-5 and then stop and fail silently. > [SQL] Nexmark 5, 7 time out > --- > > Key: BEAM-6082 > URL: https://issues.apache.org/jira/browse/BEAM-6082 > Project: Beam > Issue Type: Bug > Components: dsl-sql >Reporter: Anton Kedin >Assignee: Reuven Lax >Priority: Blocker > Fix For: 2.9.0 > > > SQL Nexmark Queries 5 and 7 are timing out since around October 29: > [https://apache-beam-testing.appspot.com/explore?dashboard=5084698770407424] > > Current suspect is https://github.com/apache/beam/pull/6856 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-6082) [SQL] Nexmark 5, 7 time out
[ https://issues.apache.org/jira/browse/BEAM-6082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699611#comment-16699611 ] Kenneth Knowles commented on BEAM-6082: --- There is still no data for SQL query 7 in Direct or Dataflow runner since 2018-10-29. > [SQL] Nexmark 5, 7 time out > --- > > Key: BEAM-6082 > URL: https://issues.apache.org/jira/browse/BEAM-6082 > Project: Beam > Issue Type: Bug > Components: dsl-sql >Reporter: Anton Kedin >Assignee: Reuven Lax >Priority: Blocker > Fix For: 2.9.0 > > > SQL Nexmark Queries 5 and 7 are timing out since around October 29: > [https://apache-beam-testing.appspot.com/explore?dashboard=5084698770407424] > > Current suspect is https://github.com/apache/beam/pull/6856 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-6082) [SQL] Nexmark 5, 7 time out
[ https://issues.apache.org/jira/browse/BEAM-6082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699610#comment-16699610 ] Kenneth Knowles commented on BEAM-6082: --- Taking a look to make sure it is not broken for any runner right now. I'm not clear now whether the job itself goes red when these fail. All jobs seem OK right now. > [SQL] Nexmark 5, 7 time out > --- > > Key: BEAM-6082 > URL: https://issues.apache.org/jira/browse/BEAM-6082 > Project: Beam > Issue Type: Bug > Components: dsl-sql >Reporter: Anton Kedin >Assignee: Reuven Lax >Priority: Blocker > Fix For: 2.9.0 > > > SQL Nexmark Queries 5 and 7 are timing out since around October 29: > [https://apache-beam-testing.appspot.com/explore?dashboard=5084698770407424] > > Current suspect is https://github.com/apache/beam/pull/6856 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-6082) [SQL] Nexmark 5, 7 time out
[ https://issues.apache.org/jira/browse/BEAM-6082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699681#comment-16699681 ] Kenneth Knowles commented on BEAM-6082: --- This was simply an incorrect translation of numbers to names by me in c4667f071d86205e23782c05a1f647b106d072ff. :((n)(x) > [SQL] Nexmark 5, 7 time out > --- > > Key: BEAM-6082 > URL: https://issues.apache.org/jira/browse/BEAM-6082 > Project: Beam > Issue Type: Bug > Components: dsl-sql >Reporter: Anton Kedin >Assignee: Kenneth Knowles >Priority: Blocker > Fix For: 2.9.0 > > > SQL Nexmark Queries 5 and 7 are timing out since around October 29: > [https://apache-beam-testing.appspot.com/explore?dashboard=5084698770407424] > > Current suspect is https://github.com/apache/beam/pull/6856 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-6082) [SQL] Nexmark 5, 7 time out
[ https://issues.apache.org/jira/browse/BEAM-6082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699696#comment-16699696 ] Kenneth Knowles commented on BEAM-6082: --- I think I will leave this as a blocker as we should cherrypick the fix so perf tests against historical releases work properly. > [SQL] Nexmark 5, 7 time out > --- > > Key: BEAM-6082 > URL: https://issues.apache.org/jira/browse/BEAM-6082 > Project: Beam > Issue Type: Bug > Components: dsl-sql >Reporter: Anton Kedin >Assignee: Kenneth Knowles >Priority: Blocker > Fix For: 2.9.0 > > Time Spent: 20m > Remaining Estimate: 0h > > SQL Nexmark Queries 5 and 7 are timing out since around October 29: > [https://apache-beam-testing.appspot.com/explore?dashboard=5084698770407424] > > Current suspect is https://github.com/apache/beam/pull/6856 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-6082) [SQL] Nexmark 5, 7 time out
[ https://issues.apache.org/jira/browse/BEAM-6082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699692#comment-16699692 ] Kenneth Knowles commented on BEAM-6082: --- Summarizing here: - The data for SQL Query 5 was being attributed to Query 4, which doesn't exist in SQL so it isn't on the dashboards. - The data for SQL Query 7 was being attributed to Query 5, and shows a straight line trend with no regression for production runners. There _is_ a massive regression on the Direct runner, which is expected due to immutability and encodeability enforcement. You can see this regression happened just before I broke the data reporting. > [SQL] Nexmark 5, 7 time out > --- > > Key: BEAM-6082 > URL: https://issues.apache.org/jira/browse/BEAM-6082 > Project: Beam > Issue Type: Bug > Components: dsl-sql >Reporter: Anton Kedin >Assignee: Kenneth Knowles >Priority: Blocker > Fix For: 2.9.0 > > Time Spent: 20m > Remaining Estimate: 0h > > SQL Nexmark Queries 5 and 7 are timing out since around October 29: > [https://apache-beam-testing.appspot.com/explore?dashboard=5084698770407424] > > Current suspect is https://github.com/apache/beam/pull/6856 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
