[jira] [Work logged] (BEAM-9288) Conscrypt shaded dependency
[ https://issues.apache.org/jira/browse/BEAM-9288?focusedWorklogId=394719&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-394719 ] ASF GitHub Bot logged work on BEAM-9288: Author: ASF GitHub Bot Created on: 28/Feb/20 08:28 Start Date: 28/Feb/20 08:28 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #10940: [BEAM-9288] Not bundle conscrypt in gRPC vendor URL: https://github.com/apache/beam/pull/10940#issuecomment-592405805 @lukecwik I have checked that `conscrypt` is only used internally by `ConscryptAlpnSslEngine`, so I agree with you that it should be a runtime dependency. @kennknowles If we change `conscrypt` to runtme dependency, the concern you have will not be a problem any more. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 394719) Time Spent: 2h 10m (was: 2h) > Conscrypt shaded dependency > --- > > Key: BEAM-9288 > URL: https://issues.apache.org/jira/browse/BEAM-9288 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Esun Kim >Assignee: sunjincheng >Priority: Critical > Fix For: 2.20.0 > > Time Spent: 2h 10m > Remaining Estimate: 0h > > Conscrypt is not designed to be shaded properly mainly because of so files. I > happened to see BEAM-9030 (*1) creating a new vendored gRPC shading Conscrypt > (*2) in it. I think this could make a problem when new Conscrypt is brought > by new gcsio depending on gRPC-alts (*4) in a dependency chain. (*5) In this > case, it may have a conflict when finding proper so files for Conscrypt. > *1: https://issues.apache.org/jira/browse/BEAM-9030 > *2: > [https://github.com/apache/beam/blob/e24d1e51cbabe27cb3cc381fd95b334db639c45d/buildSrc/src/main/groovy/org/apache/beam/gradle/GrpcVendoring_1_26_0.groovy#L78] > *3: https://issues.apache.org/jira/browse/BEAM-6136 > *4: [https://mvnrepository.com/artifact/io.grpc/grpc-alts/1.27.0] > *5: https://issues.apache.org/jira/browse/BEAM-8889 > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9345) "Multiple environments cannot be created in detached mode"
[ https://issues.apache.org/jira/browse/BEAM-9345?focusedWorklogId=394732&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-394732 ] ASF GitHub Bot logged work on BEAM-9345: Author: ASF GitHub Bot Created on: 28/Feb/20 08:40 Start Date: 28/Feb/20 08:40 Worklog Time Spent: 10m Work Description: mxm commented on pull request #10999: [BEAM-9345] Fix source of test flakiness in FlinkSubmissionTest URL: https://github.com/apache/beam/pull/10999 The assertion is too strict, as the job submission works asynchronously and the job may only appear in the cluster after submission. I'm also moving the non-static method for better readability of the test. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python37/last
[jira] [Work logged] (BEAM-9401) Flink MiniCluster should bind to localhost
[ https://issues.apache.org/jira/browse/BEAM-9401?focusedWorklogId=394733&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-394733 ] ASF GitHub Bot logged work on BEAM-9401: Author: ASF GitHub Bot Created on: 28/Feb/20 08:41 Start Date: 28/Feb/20 08:41 Worklog Time Spent: 10m Work Description: mxm commented on issue #10995: [BEAM-9401] bind Flink MiniCluster to localhost URL: https://github.com/apache/beam/pull/10995#issuecomment-592410078 Yes, appears to be flaky. Here's the fix: https://github.com/apache/beam/pull/10999 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 394733) Time Spent: 40m (was: 0.5h) > Flink MiniCluster should bind to localhost > -- > > Key: BEAM-9401 > URL: https://issues.apache.org/jira/browse/BEAM-9401 > Project: Beam > Issue Type: Improvement > Components: runner-flink, testing >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > Vulnerability detection services have identified an XSS vulnerability with > the Flink web UI (QUALYS-86175) when I run the Flink MiniCluster on my > workstation. I'm not sure how serious the vulnerability is, but the easiest > way to prevent it would be to bind the the server against localhost. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9384) Add SchemaRegistry.getSchemaCoder to get SchemaCoders for registered types
[ https://issues.apache.org/jira/browse/BEAM-9384?focusedWorklogId=394741&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-394741 ] ASF GitHub Bot logged work on BEAM-9384: Author: ASF GitHub Bot Created on: 28/Feb/20 08:53 Start Date: 28/Feb/20 08:53 Worklog Time Spent: 10m Work Description: iemejia commented on issue #10974: [BEAM-9384] Add SchemaRegistry.getSchemaCoder to get SchemaCoders for registered types URL: https://github.com/apache/beam/pull/10974#issuecomment-592414613 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 394741) Time Spent: 3h 50m (was: 3h 40m) > Add SchemaRegistry.getSchemaCoder to get SchemaCoders for registered types > -- > > Key: BEAM-9384 > URL: https://issues.apache.org/jira/browse/BEAM-9384 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Ismaël Mejía >Assignee: Ismaël Mejía >Priority: Minor > Time Spent: 3h 50m > Remaining Estimate: 0h > > PTransforms that are parameterized by types may be able to infer a valid > SchemaCoder for a given type from the SchemaRegistry (if a Schema for the > given type is available). This method will provide a unified place to do that. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9392) TestStream tests are all flaky
[ https://issues.apache.org/jira/browse/BEAM-9392?focusedWorklogId=394743&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-394743 ] ASF GitHub Bot logged work on BEAM-9392: Author: ASF GitHub Bot Created on: 28/Feb/20 08:54 Start Date: 28/Feb/20 08:54 Worklog Time Spent: 10m Work Description: kamilwu commented on issue #10982: [BEAM-9392] Fix Multi TestStream assertion errors URL: https://github.com/apache/beam/pull/10982#issuecomment-592414688 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 394743) Time Spent: 4h 20m (was: 4h 10m) > TestStream tests are all flaky > -- > > Key: BEAM-9392 > URL: https://issues.apache.org/jira/browse/BEAM-9392 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Pablo Estrada >Assignee: Sam Rohde >Priority: Major > Time Spent: 4h 20m > Remaining Estimate: 0h > > See: > [https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9384) Add SchemaRegistry.getSchemaCoder to get SchemaCoders for registered types
[ https://issues.apache.org/jira/browse/BEAM-9384?focusedWorklogId=394742&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-394742 ] ASF GitHub Bot logged work on BEAM-9384: Author: ASF GitHub Bot Created on: 28/Feb/20 08:54 Start Date: 28/Feb/20 08:54 Worklog Time Spent: 10m Work Description: iemejia commented on issue #10974: [BEAM-9384] Add SchemaRegistry.getSchemaCoder to get SchemaCoders for registered types URL: https://github.com/apache/beam/pull/10974#issuecomment-592414613 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 394742) Time Spent: 4h (was: 3h 50m) > Add SchemaRegistry.getSchemaCoder to get SchemaCoders for registered types > -- > > Key: BEAM-9384 > URL: https://issues.apache.org/jira/browse/BEAM-9384 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Ismaël Mejía >Assignee: Ismaël Mejía >Priority: Minor > Time Spent: 4h > Remaining Estimate: 0h > > PTransforms that are parameterized by types may be able to infer a valid > SchemaCoder for a given type from the SchemaRegistry (if a Schema for the > given type is available). This method will provide a unified place to do that. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9299) Upgrade Flink Runner to 1.8.3 and 1.9.2
[ https://issues.apache.org/jira/browse/BEAM-9299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maximilian Michels updated BEAM-9299: - Fix Version/s: 2.21.0 > Upgrade Flink Runner to 1.8.3 and 1.9.2 > --- > > Key: BEAM-9299 > URL: https://issues.apache.org/jira/browse/BEAM-9299 > Project: Beam > Issue Type: Task > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.21.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > I would like to Upgrade Flink Runner to 18.3 and 1.9.2 due to both the Apache > Flink 1.8.3 and Apache Flink 1.9.2 have been released [1]. > What do you think? > [1] https://dist.apache.org/repos/dist/release/flink/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9299) Upgrade Flink Runner to 1.8.3 and 1.9.2
[ https://issues.apache.org/jira/browse/BEAM-9299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17047339#comment-17047339 ] Maximilian Michels commented on BEAM-9299: -- I think we'll move this to 2.21.0 to allow time for these changes to stabilize. Looking at the PR, I'm in favor of approach (1), removing 1.7 first. There is too much code complexity added now and we're planning to remove 1.7 anyways. > Upgrade Flink Runner to 1.8.3 and 1.9.2 > --- > > Key: BEAM-9299 > URL: https://issues.apache.org/jira/browse/BEAM-9299 > Project: Beam > Issue Type: Task > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 1h 20m > Remaining Estimate: 0h > > I would like to Upgrade Flink Runner to 18.3 and 1.9.2 due to both the Apache > Flink 1.8.3 and Apache Flink 1.9.2 have been released [1]. > What do you think? > [1] https://dist.apache.org/repos/dist/release/flink/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9295) Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
[
https://issues.apache.org/jira/browse/BEAM-9295?focusedWorklogId=394751&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-394751
]
ASF GitHub Bot logged work on BEAM-9295:
Author: ASF GitHub Bot
Created on: 28/Feb/20 09:07
Start Date: 28/Feb/20 09:07
Worklog Time Spent: 10m
Work Description: mxm commented on pull request #10945: [BEAM-9295] Add
Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
URL: https://github.com/apache/beam/pull/10945#discussion_r385577426
##
File path:
runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkExecutionEnvironments.java
##
@@ -311,78 +302,4 @@ private static void applyLatencyTrackingInterval(
long latencyTrackingInterval = options.getLatencyTrackingInterval();
config.setLatencyTrackingInterval(latencyTrackingInterval);
}
-
- /**
- * Remote stream environment that supports job execution with restore from
savepoint.
- *
- * This class can be removed once Flink provides this functionality.
- *
- * TODO: https://issues.apache.org/jira/browse/BEAM-5396
- */
- private static class BeamFlinkRemoteStreamEnvironment extends
RemoteStreamEnvironment {
Review comment:
This cannot be removed yet. The feature is only present in 1.8.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 394751)
Time Spent: 2h 40m (was: 2.5h)
> Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
> ---
>
> Key: BEAM-9295
> URL: https://issues.apache.org/jira/browse/BEAM-9295
> Project: Beam
> Issue Type: New Feature
> Components: runner-flink
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> Apache Flink 1.10 has completed the final release vote, see [1]. So, I would
> like to add Flink 1.10 build target and make Flink Runner compatible with
> Flink 1.10.
> And I appreciate it if you can leave your suggestions or comments!
> [1]
> https://lists.apache.org/thread.html/r97672d4d1e47372cebf23e6643a6cc30a06bfbdf3f277b0be3695b15%40%3Cdev.flink.apache.org%3E
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9295) Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
[ https://issues.apache.org/jira/browse/BEAM-9295?focusedWorklogId=394752&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-394752 ] ASF GitHub Bot logged work on BEAM-9295: Author: ASF GitHub Bot Created on: 28/Feb/20 09:07 Start Date: 28/Feb/20 09:07 Worklog Time Spent: 10m Work Description: mxm commented on pull request #10945: [BEAM-9295] Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 URL: https://github.com/apache/beam/pull/10945#discussion_r385578594 ## File path: runners/flink/1.10/src/test/java/org/apache/beam/runners/flink/FlinkSubmissionTest.java ## @@ -0,0 +1,256 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.beam.runners.flink; + +import static org.hamcrest.MatcherAssert.assertThat; +import static org.hamcrest.Matchers.is; + +import java.io.File; +import java.lang.reflect.Field; +import java.lang.reflect.Modifier; +import java.nio.file.Files; +import java.security.Permission; +import java.util.Collection; +import java.util.Map; +import java.util.concurrent.TimeUnit; +import org.apache.beam.runners.core.construction.resources.PipelineResources; +import org.apache.beam.sdk.Pipeline; +import org.apache.beam.sdk.io.GenerateSequence; +import org.apache.beam.sdk.options.PipelineOptions; +import org.apache.beam.sdk.options.PipelineOptionsFactory; +import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Charsets; +import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList; +import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap; +import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables; +import org.apache.flink.api.common.JobStatus; +import org.apache.flink.client.cli.CliFrontend; +import org.apache.flink.configuration.ConfigConstants; +import org.apache.flink.configuration.Configuration; +import org.apache.flink.configuration.JobManagerOptions; +import org.apache.flink.configuration.RestOptions; +import org.apache.flink.runtime.client.JobStatusMessage; +import org.apache.flink.runtime.minicluster.MiniClusterConfiguration; +import org.apache.flink.runtime.minicluster.RpcServiceSharing; +import org.junit.AfterClass; +import org.junit.BeforeClass; +import org.junit.ClassRule; +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.TemporaryFolder; +import org.junit.rules.Timeout; + +/** + * End-to-end submission test of Beam jobs on a Flink cluster. + * + * This test is copied to 1.10 is becauses the package name of JobStatus has changed in Flink + * 1.10, please refer to Review comment: No need to copy everything because of one change package name. We can load JobStatus dynamically. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 394752) Time Spent: 2h 50m (was: 2h 40m) > Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 > --- > > Key: BEAM-9295 > URL: https://issues.apache.org/jira/browse/BEAM-9295 > Project: Beam > Issue Type: New Feature > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 2h 50m > Remaining Estimate: 0h > > Apache Flink 1.10 has completed the final release vote, see [1]. So, I would > like to add Flink 1.10 build target and make Flink Runner compatible with > Flink 1.10. > And I appreciate it if you can leave your suggestions or comments! > [1] > https://lists.apache.org/thread.html/r97672d4d1e47372cebf23e6643a6cc30a06bfbdf3f277b0be3695b15%40%3Cdev.flink.apache.org%3E -- This message was sent by At
[jira] [Commented] (BEAM-9288) Conscrypt shaded dependency
[ https://issues.apache.org/jira/browse/BEAM-9288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17047343#comment-17047343 ] Ismaël Mejía commented on BEAM-9288: Oh I see that's clearly a mistake I will unlink the issues because they are not related. BEAM-9252 was all about users being unable to shade the Jar and this is solved. This issue is different. My comment about Beam's use of the dependency is related to the fact that it looks like a transitive dependency that gRPC uses, but that our use of gRPC might not use. If that's the case we can go with it even if it is badly shaded, but well I understand if you guys prefer to do it properly, I just wanted to avoid delaying the release if we can go with it as it is. > Conscrypt shaded dependency > --- > > Key: BEAM-9288 > URL: https://issues.apache.org/jira/browse/BEAM-9288 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Esun Kim >Assignee: sunjincheng >Priority: Critical > Fix For: 2.20.0 > > Time Spent: 2h 10m > Remaining Estimate: 0h > > Conscrypt is not designed to be shaded properly mainly because of so files. I > happened to see BEAM-9030 (*1) creating a new vendored gRPC shading Conscrypt > (*2) in it. I think this could make a problem when new Conscrypt is brought > by new gcsio depending on gRPC-alts (*4) in a dependency chain. (*5) In this > case, it may have a conflict when finding proper so files for Conscrypt. > *1: https://issues.apache.org/jira/browse/BEAM-9030 > *2: > [https://github.com/apache/beam/blob/e24d1e51cbabe27cb3cc381fd95b334db639c45d/buildSrc/src/main/groovy/org/apache/beam/gradle/GrpcVendoring_1_26_0.groovy#L78] > *3: https://issues.apache.org/jira/browse/BEAM-6136 > *4: [https://mvnrepository.com/artifact/io.grpc/grpc-alts/1.27.0] > *5: https://issues.apache.org/jira/browse/BEAM-8889 > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-7870) Externally configured KafkaIO / PubsubIO consumer causes coder problems
[ https://issues.apache.org/jira/browse/BEAM-7870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17047344#comment-17047344 ] Maximilian Michels commented on BEAM-7870: -- Sorry, do not have capacity for this at the moment. Feel free to take over this. > Externally configured KafkaIO / PubsubIO consumer causes coder problems > --- > > Key: BEAM-7870 > URL: https://issues.apache.org/jira/browse/BEAM-7870 > Project: Beam > Issue Type: Bug > Components: runner-flink, sdk-java-core >Reporter: Maximilian Michels >Priority: Major > > There are limitations for the consumer to work correctly. The biggest issue > is the structure of KafkaIO itself, which uses a combination of the source > interface and DoFns to generate the desired output. The problem is that the > source interface is natively translated by the Flink Runner to support > unbounded sources in portability, while the DoFn runs in a Java environment. > To transfer data between the two a coder needs to be involved. It happens to > be that the initial read does not immediately drop the KafakRecord structure > which does not work together well with our current assumption of only > supporting "standard coders" present in all SDKs. Only the subsequent DoFn > converts the KafkaRecord structure into a raw KV[byte, byte], but the DoFn > won't have the coder available in its environment. > There are several possible solutions: > 1. Make the DoFn which drops the KafkaRecordCoder a native Java transform in > the Flink Runner > 2. Modify KafkaIO to immediately drop the KafkaRecord structure > 3. Add the KafkaRecordCoder to all SDKs > 4. Add a generic coder, e.g. AvroCoder to all SDKs > For a workaround which uses (3), please see this patch which is not a proper > fix but adds KafkaRecordCoder to the SDK such that it can be used > encode/decode records: > [https://github.com/mxm/beam/commit/b31cf99c75b3972018180d8ccc7e73d311f4cfed] > > See also > [https://github.com/apache/beam/pull/8251|https://github.com/apache/beam/pull/8251:] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (BEAM-9288) Conscrypt shaded dependency
[ https://issues.apache.org/jira/browse/BEAM-9288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17047343#comment-17047343 ] Ismaël Mejía edited comment on BEAM-9288 at 2/28/20 9:09 AM: - Oh I see that's clearly a mistake I will unlink the issues because they are not related. BEAM-9252 was all about users being unable to shade the Jar and this is solved. This issue is different. Conscrypt seems like a transitive dependency that gRPC uses, but that Beam's use of gRPC may not use. If that's the case we can go with it even if it is badly shaded, but well I understand if you guys prefer to do it properly, I just wanted to avoid delaying the release if we can go with it as it is. was (Author: iemejia): Oh I see that's clearly a mistake I will unlink the issues because they are not related. BEAM-9252 was all about users being unable to shade the Jar and this is solved. This issue is different. My comment about Beam's use of the dependency is related to the fact that it looks like a transitive dependency that gRPC uses, but that our use of gRPC might not use. If that's the case we can go with it even if it is badly shaded, but well I understand if you guys prefer to do it properly, I just wanted to avoid delaying the release if we can go with it as it is. > Conscrypt shaded dependency > --- > > Key: BEAM-9288 > URL: https://issues.apache.org/jira/browse/BEAM-9288 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Esun Kim >Assignee: sunjincheng >Priority: Critical > Fix For: 2.20.0 > > Time Spent: 2h 10m > Remaining Estimate: 0h > > Conscrypt is not designed to be shaded properly mainly because of so files. I > happened to see BEAM-9030 (*1) creating a new vendored gRPC shading Conscrypt > (*2) in it. I think this could make a problem when new Conscrypt is brought > by new gcsio depending on gRPC-alts (*4) in a dependency chain. (*5) In this > case, it may have a conflict when finding proper so files for Conscrypt. > *1: https://issues.apache.org/jira/browse/BEAM-9030 > *2: > [https://github.com/apache/beam/blob/e24d1e51cbabe27cb3cc381fd95b334db639c45d/buildSrc/src/main/groovy/org/apache/beam/gradle/GrpcVendoring_1_26_0.groovy#L78] > *3: https://issues.apache.org/jira/browse/BEAM-6136 > *4: [https://mvnrepository.com/artifact/io.grpc/grpc-alts/1.27.0] > *5: https://issues.apache.org/jira/browse/BEAM-8889 > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9392) TestStream tests are all flaky
[ https://issues.apache.org/jira/browse/BEAM-9392?focusedWorklogId=394762&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-394762 ] ASF GitHub Bot logged work on BEAM-9392: Author: ASF GitHub Bot Created on: 28/Feb/20 09:28 Start Date: 28/Feb/20 09:28 Worklog Time Spent: 10m Work Description: kamilwu commented on issue #10982: [BEAM-9392] Fix Multi TestStream assertion errors URL: https://github.com/apache/beam/pull/10982#issuecomment-592427742 `pycommon:docs` task is failing due to: `docstring of apache_beam.testing.test_stream.TestStream:14: WARNING: Unexpected indentation. ` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 394762) Time Spent: 4.5h (was: 4h 20m) > TestStream tests are all flaky > -- > > Key: BEAM-9392 > URL: https://issues.apache.org/jira/browse/BEAM-9392 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Pablo Estrada >Assignee: Sam Rohde >Priority: Major > Time Spent: 4.5h > Remaining Estimate: 0h > > See: > [https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9287) Python Validates runner tests for Unified Worker
[ https://issues.apache.org/jira/browse/BEAM-9287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismaël Mejía updated BEAM-9287: --- Status: Open (was: Triage Needed) > Python Validates runner tests for Unified Worker > > > Key: BEAM-9287 > URL: https://issues.apache.org/jira/browse/BEAM-9287 > Project: Beam > Issue Type: Test > Components: runner-dataflow, testing >Reporter: Ankur Goenka >Assignee: Ankur Goenka >Priority: Major > Fix For: 2.20.0 > > Time Spent: 2h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9402) test_multi_triggered_gbk_side_input is always using DirectRunner
[ https://issues.apache.org/jira/browse/BEAM-9402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismaël Mejía updated BEAM-9402: --- Status: Open (was: Triage Needed) > test_multi_triggered_gbk_side_input is always using DirectRunner > > > Key: BEAM-9402 > URL: https://issues.apache.org/jira/browse/BEAM-9402 > Project: Beam > Issue Type: Bug > Components: sdk-py-core, testing >Reporter: Ankur Goenka >Assignee: Ankur Goenka >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > The option provided in the test overwrite the argument provided during > validates runner test > > > - >> end captured logging << - > == > FAIL: Test a GBK sideinput, with multiple triggering. > -- > Traceback (most recent call last): > File > "/usr/local/google/home/goenka/d/work/beam/beam/sdks/python/apache_beam/transforms/sideinputs_test.py", > line 401, in test_multi_triggered_gbk_side_input > p.run() > File > "/usr/local/google/home/goenka/d/work/beam/beam/sdks/python/apache_beam/testing/test_pipeline.py", > line 112, in run > False if self.not_use_test_runner_api else test_runner_api)) > File > "/usr/local/google/home/goenka/d/work/beam/beam/sdks/python/apache_beam/pipeline.py", > line 483, in run > self._options).run(False) > File > "/usr/local/google/home/goenka/d/work/beam/beam/sdks/python/apache_beam/pipeline.py", > line 809, in from_runner_api > p.transforms_stack = [context.transforms.get_by_id(root_transform_id)] > File > "/usr/local/google/home/goenka/d/work/beam/beam/sdks/python/apache_beam/runners/pipeline_context.py", > line 103, in get_by_id > self._id_to_proto[id], self._pipeline_context) > File > "/usr/local/google/home/goenka/d/work/beam/beam/sdks/python/apache_beam/pipeline.py", > line 1117, in from_runner_api > part = context.transforms.get_by_id(transform_id) > File > "/usr/local/google/home/goenka/d/work/beam/beam/sdks/python/apache_beam/runners/pipeline_context.py", > line 103, in get_by_id > self._id_to_proto[id], self._pipeline_context) > File > "/usr/local/google/home/goenka/d/work/beam/beam/sdks/python/apache_beam/pipeline.py", > line 1104, in from_runner_api > transform = ptransform.PTransform.from_runner_api(proto, context) > File > "/usr/local/google/home/goenka/d/work/beam/beam/sdks/python/apache_beam/transforms/ptransform.py", > line 684, in from_runner_api > context) > File > "/usr/local/google/home/goenka/d/work/beam/beam/sdks/python/apache_beam/testing/test_stream.py", > line 316, in from_runner_api_parameter > output_tags=output_tags) > File > "/usr/local/google/home/goenka/d/work/beam/beam/sdks/python/apache_beam/testing/test_stream.py", > line 200, in __init__ > assert event_tags.issubset(self.output_tags) > AssertionError: > >> begin captured logging << > root: INFO: Missing pipeline option (runner). Executing pipeline using the > default runner: DirectRunner. > - >> end captured logging << - -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9392) TestStream tests are all flaky
[ https://issues.apache.org/jira/browse/BEAM-9392?focusedWorklogId=394774&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-394774 ] ASF GitHub Bot logged work on BEAM-9392: Author: ASF GitHub Bot Created on: 28/Feb/20 09:56 Start Date: 28/Feb/20 09:56 Worklog Time Spent: 10m Work Description: kamilwu commented on issue #10982: [BEAM-9392] Fix Multi TestStream assertion errors URL: https://github.com/apache/beam/pull/10982#issuecomment-592438613 Retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 394774) Time Spent: 4h 40m (was: 4.5h) > TestStream tests are all flaky > -- > > Key: BEAM-9392 > URL: https://issues.apache.org/jira/browse/BEAM-9392 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Pablo Estrada >Assignee: Sam Rohde >Priority: Major > Time Spent: 4h 40m > Remaining Estimate: 0h > > See: > [https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9392) TestStream tests are all flaky
[ https://issues.apache.org/jira/browse/BEAM-9392?focusedWorklogId=394775&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-394775 ] ASF GitHub Bot logged work on BEAM-9392: Author: ASF GitHub Bot Created on: 28/Feb/20 09:58 Start Date: 28/Feb/20 09:58 Worklog Time Spent: 10m Work Description: kamilwu commented on issue #10982: [BEAM-9392] Fix Multi TestStream assertion errors URL: https://github.com/apache/beam/pull/10982#issuecomment-592439340 Retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 394775) Time Spent: 4h 50m (was: 4h 40m) > TestStream tests are all flaky > -- > > Key: BEAM-9392 > URL: https://issues.apache.org/jira/browse/BEAM-9392 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Pablo Estrada >Assignee: Sam Rohde >Priority: Major > Time Spent: 4h 50m > Remaining Estimate: 0h > > See: > [https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9396) Docker image names in Jenkins jobs don't match generated ones
[ https://issues.apache.org/jira/browse/BEAM-9396?focusedWorklogId=394783&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-394783 ] ASF GitHub Bot logged work on BEAM-9396: Author: ASF GitHub Bot Created on: 28/Feb/20 10:02 Start Date: 28/Feb/20 10:02 Worklog Time Spent: 10m Work Description: mwalenia commented on pull request #10985: [BEAM-9396] Match Docker image names between Jenkins jobs and Gradle tasks URL: https://github.com/apache/beam/pull/10985 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 394783) Time Spent: 1.5h (was: 1h 20m) > Docker image names in Jenkins jobs don't match generated ones > - > > Key: BEAM-9396 > URL: https://issues.apache.org/jira/browse/BEAM-9396 > Project: Beam > Issue Type: Bug > Components: testing >Reporter: Michał Walenia >Assignee: Michał Walenia >Priority: Major > Fix For: Not applicable > > Time Spent: 1.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9396) Docker image names in Jenkins jobs don't match generated ones
[ https://issues.apache.org/jira/browse/BEAM-9396?focusedWorklogId=394782&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-394782 ] ASF GitHub Bot logged work on BEAM-9396: Author: ASF GitHub Bot Created on: 28/Feb/20 10:02 Start Date: 28/Feb/20 10:02 Worklog Time Spent: 10m Work Description: mwalenia commented on issue #10985: [BEAM-9396] Match Docker image names between Jenkins jobs and Gradle tasks URL: https://github.com/apache/beam/pull/10985#issuecomment-592440937 Tests passed, merging. @ibzib thanks for review! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 394782) Time Spent: 1h 20m (was: 1h 10m) > Docker image names in Jenkins jobs don't match generated ones > - > > Key: BEAM-9396 > URL: https://issues.apache.org/jira/browse/BEAM-9396 > Project: Beam > Issue Type: Bug > Components: testing >Reporter: Michał Walenia >Assignee: Michał Walenia >Priority: Major > Fix For: Not applicable > > Time Spent: 1h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8972) Add a Jenkins job running Combine load test on Java with Flink in Portability mode
[ https://issues.apache.org/jira/browse/BEAM-8972?focusedWorklogId=394785&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-394785 ] ASF GitHub Bot logged work on BEAM-8972: Author: ASF GitHub Bot Created on: 28/Feb/20 10:03 Start Date: 28/Feb/20 10:03 Worklog Time Spent: 10m Work Description: mwalenia commented on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-592441414 run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 394785) Time Spent: 13h 20m (was: 13h 10m) > Add a Jenkins job running Combine load test on Java with Flink in Portability > mode > -- > > Key: BEAM-8972 > URL: https://issues.apache.org/jira/browse/BEAM-8972 > Project: Beam > Issue Type: Improvement > Components: testing >Reporter: Michał Walenia >Assignee: Michał Walenia >Priority: Minor > Time Spent: 13h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9403) Make RedisIO readAll consistent with other Beam readAll methods
[ https://issues.apache.org/jira/browse/BEAM-9403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismaël Mejía updated BEAM-9403: --- Status: Open (was: Triage Needed) > Make RedisIO readAll consistent with other Beam readAll methods > --- > > Key: BEAM-9403 > URL: https://issues.apache.org/jira/browse/BEAM-9403 > Project: Beam > Issue Type: Improvement > Components: io-java-redis >Reporter: Ismaël Mejía >Assignee: Ismaël Mejía >Priority: Major > > RedisIO.readAll is not consistent with the composable approach of other > readAll transforms in Beam which basically receive a PCollection of read > requests (Read or query/configuration objects) and produce output. This > should be fixed to avoid confusing users. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9403) Make RedisIO readAll consistent with other Beam readAll methods
Ismaël Mejía created BEAM-9403: -- Summary: Make RedisIO readAll consistent with other Beam readAll methods Key: BEAM-9403 URL: https://issues.apache.org/jira/browse/BEAM-9403 Project: Beam Issue Type: Improvement Components: io-java-redis Reporter: Ismaël Mejía Assignee: Ismaël Mejía RedisIO.readAll is not consistent with the composable approach of other readAll transforms in Beam which basically receive a PCollection of read requests (Read or query/configuration objects) and produce output. This should be fixed to avoid confusing users. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9216) Unable to run job server
[
https://issues.apache.org/jira/browse/BEAM-9216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ismaël Mejía updated BEAM-9216:
---
Priority: Minor (was: Blocker)
> Unable to run job server
>
>
> Key: BEAM-9216
> URL: https://issues.apache.org/jira/browse/BEAM-9216
> Project: Beam
> Issue Type: Bug
> Components: build-system
>Affects Versions: 2.16.0, 2.18.0
> Environment: Archlinux Linux 4.19.94-1-lts #1 SMP x86_64 GNU/Linux
> Docker OpenJDK 8 Stretch
>Reporter: Darshan Parab
>Priority: Minor
> Labels: beam, newbie, sparkrunner
>
> Beam throws below mentioned error while following the steps to use Spark
> Portable runner.
> -
> {{>./learning/katas/java/gradlew :runners:spark:job-server:runShadow}}{{}}
> {{}}{{Starting a Gradle Daemon (subsequent builds will be faster)}}
> {{Parallel execution is an incubating feature.}}{{FAILURE: Build failed with
> an exception.}}
> {{* Where:}}
> {{Build file
> '/home/darshan/docker/apache/beam/beam-release-2.18.0/buildSrc/build.gradle'
> line: 32}}
> {{* What went wrong:}}
> {{A problem occurred evaluating project ':buildSrc'.}}
> {{> Could not find method content() for arguments
> [build_9gp7982ce87i1sk76auu30fh3$_run_closure1$_closure6$_closure7@26bd394f]
> on object of type
> org.gradle.api.internal.artifacts.repositories.DefaultMavenArtifactRepository.}}{{*
> Try:}}
> {{Run with --stacktrace option to get the stack trace. Run with --info or
> --debug option to get more log output. Run with --scan to get full
> insights.}}{{* Get more help at
> [https://help.gradle.org|https://help.gradle.org/]}}
> {{-}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-7881) Get rid of jackson to avoid the continuous flow of CVEs in Jackson
[ https://issues.apache.org/jira/browse/BEAM-7881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismaël Mejía updated BEAM-7881: --- Priority: Minor (was: Blocker) > Get rid of jackson to avoid the continuous flow of CVEs in Jackson > -- > > Key: BEAM-7881 > URL: https://issues.apache.org/jira/browse/BEAM-7881 > Project: Beam > Issue Type: Task > Components: sdk-java-core >Affects Versions: 2.14.0 >Reporter: Romain Manni-Bucau >Priority: Minor > > Jackson keeps having CVE on all releases of databind and transitively beam > sdk java core has CVE on all its releases (for the record, when writing this > issue you must use at least jackson-databind 2.9.9.2 but last week it was > 2.9.9.1 and 2.14 didn't get the fix). > Can be neat to get rid of jackson which does not fix this issue for a very > long time now and just use JSON-B or another JSON impl to ensure the CVE is > not usable because beam is there. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8875) Pipeline fails to run remotely
[ https://issues.apache.org/jira/browse/BEAM-8875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismaël Mejía updated BEAM-8875: --- Priority: Minor (was: Blocker) > Pipeline fails to run remotely > -- > > Key: BEAM-8875 > URL: https://issues.apache.org/jira/browse/BEAM-8875 > Project: Beam > Issue Type: Bug > Components: beam-model > Environment: Java 8 >Reporter: Scott McKinney >Priority: Minor > > Beam fails to run remote pipeline when the class loader has URLs other than > _[file://_|file://_/]._ The environment uses the [Manifold > project|http://manifold.systems], which installs a custom URL handler of the > form: _manifoldclass://_, which contributes to the URL classpath, however > Beam throws an exception when it encounters this URL. Instead Beam should > *ignore* or at most warn about URLs it does not recognize. > A similar issue was recently fixed in the Spring framework: > [https://github.com/spring-projects/spring-framework/issues/21950]. Beam > should follow suit. > See external Manifold issue: > [https://github.com/manifold-systems/manifold/issues/143] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8128) Don't deprecate Read for Impulse
[ https://issues.apache.org/jira/browse/BEAM-8128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismaël Mejía updated BEAM-8128: --- Priority: Major (was: Blocker) > Don't deprecate Read for Impulse > > > Key: BEAM-8128 > URL: https://issues.apache.org/jira/browse/BEAM-8128 > Project: Beam > Issue Type: Bug > Components: runner-core, sdk-java-core >Affects Versions: 2.15.0 >Reporter: Romain Manni-Bucau >Priority: Major > > In last beam release, Read.Bounded and Read.Unbounded are deprecated and beam > tends to move to Impulse usage. > This is a huge breaking change since users can't rely on custom pre-runner > pipeline visitor to instrument their pipelines anymore or even identify the > transform accurately anymore. > This issue is about ensure that SDF or not Read.Bounded and Read.Unbounded is > a stable transform matcher and can still be used by user code to identify > inputs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-5526) Make ByteBuddyDoFnInvokerFactory injection strategy configurable + drop the singleton
[ https://issues.apache.org/jira/browse/BEAM-5526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismaël Mejía updated BEAM-5526: --- Priority: Major (was: Blocker) > Make ByteBuddyDoFnInvokerFactory injection strategy configurable + drop the > singleton > - > > Key: BEAM-5526 > URL: https://issues.apache.org/jira/browse/BEAM-5526 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Romain Manni-Bucau >Priority: Major > > org.apache.beam.sdk.transforms.reflect.DoFnInvokers + DoFnInvokerFactory > design is to be a SPI to let user plug their own bytecode manipulation > library, however in practise beam uses ByteBuddyDoFnInvokerFactory as a > singleton which makes all this design useless. > ByteBuddyDoFnInvokerFactory is also not configurable at all - typically the > injection strategy so it assumes it runs in an environment and on a JVM where > it will work - it does not on java 11 for instance. > This ticket is about fixing all these small inconsistency and blocker to tun > on java 11. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8128) Don't deprecate Read for Impulse
[ https://issues.apache.org/jira/browse/BEAM-8128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismaël Mejía updated BEAM-8128: --- Status: Open (was: Triage Needed) > Don't deprecate Read for Impulse > > > Key: BEAM-8128 > URL: https://issues.apache.org/jira/browse/BEAM-8128 > Project: Beam > Issue Type: Bug > Components: runner-core, sdk-java-core >Affects Versions: 2.15.0 >Reporter: Romain Manni-Bucau >Priority: Major > > In last beam release, Read.Bounded and Read.Unbounded are deprecated and beam > tends to move to Impulse usage. > This is a huge breaking change since users can't rely on custom pre-runner > pipeline visitor to instrument their pipelines anymore or even identify the > transform accurately anymore. > This issue is about ensure that SDF or not Read.Bounded and Read.Unbounded is > a stable transform matcher and can still be used by user code to identify > inputs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-8128) Don't deprecate Read for Impulse
[ https://issues.apache.org/jira/browse/BEAM-8128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismaël Mejía resolved BEAM-8128. Fix Version/s: Not applicable Resolution: Invalid Read.Bounded and Read.Unbounded are not marked as deprecated at the moment. > Don't deprecate Read for Impulse > > > Key: BEAM-8128 > URL: https://issues.apache.org/jira/browse/BEAM-8128 > Project: Beam > Issue Type: Bug > Components: runner-core, sdk-java-core >Affects Versions: 2.15.0 >Reporter: Romain Manni-Bucau >Priority: Major > Fix For: Not applicable > > > In last beam release, Read.Bounded and Read.Unbounded are deprecated and beam > tends to move to Impulse usage. > This is a huge breaking change since users can't rely on custom pre-runner > pipeline visitor to instrument their pipelines anymore or even identify the > transform accurately anymore. > This issue is about ensure that SDF or not Read.Bounded and Read.Unbounded is > a stable transform matcher and can still be used by user code to identify > inputs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-7935) TypeError: can't pickle _cffi_backend.CDataGCP objects
[ https://issues.apache.org/jira/browse/BEAM-7935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismaël Mejía updated BEAM-7935: --- Priority: Major (was: Blocker) > TypeError: can't pickle _cffi_backend.CDataGCP objects > -- > > Key: BEAM-7935 > URL: https://issues.apache.org/jira/browse/BEAM-7935 > Project: Beam > Issue Type: Bug > Components: runner-dataflow, sdk-py-core >Affects Versions: 2.13.0 > Environment: Linux(Debian GNU/Linux 9) ( SMP x86_64 GNU/Linux) > Python 2.7.13 > Content of requirements.txt file > -- > apache-beam==2.13.0 > apache-beam[gcp] > google-cloud-core==1.0.3 > google-cloud-iam > google-cloud-bigquery==1.6.1 > PyYAML==3.13 > six==1.12.0 > pyOpenSSL > httplib2==0.12.0 > google-apitools > google-api-python-client > google-cloud-storage==1.17.0 >Reporter: Marappa Ganesan >Priority: Major > > Python Dataflow runner failed with folrowing error > *---* > *p.run().wait_until_finish()* > *File > "/home/marappan/sanmgcppy2env/local/lib/python2.7/site-packages/apache_beam/pipeline.py", > line 416, in run* > *pickler.dump_session(os.path.join(tmpdir, 'main_session.pickle'))* > *File > "/home/marappan/sanmgcppy2env/local/lib/python2.7/site-packages/apache_beam/internal/pickler.py", > line 274, in dump_session* > *dill.dump_session(file_path)* > *File > "/home/marappan/sanmgcppy2env/local/lib/python2.7/site-packages/dill/_dill.py", > line 393, in dump_session* > *pickler.dump(main)* > *File "/usr/lib/python2.7/pickle.py", line 224, in dump* > *self.save(obj)* > *File "/usr/lib/python2.7/pickle.py", line 286, in save* > *f(self, obj) # Call unbound method with explicit self* > *File > "/home/marappan/sanmgcppy2env/local/lib/python2.7/site-packages/apache_beam/internal/pickler.py", > line 149, in save_module* > *return old_save_module(pickler, obj)* > *File > "/home/marappan/sanmgcppy2env/local/lib/python2.7/site-packages/dill/_dill.py", > line 1269, in save_module* > *state=_main_dict)* > *File "/usr/lib/python2.7/pickle.py", line 425, in save_reduce* > *save(state)* > *File "/usr/lib/python2.7/pickle.py", line 286, in save* > *f(self, obj) # Call unbound method with explicit self* > *File > "/home/marappan/sanmgcppy2env/local/lib/python2.7/site-packages/apache_beam/internal/pickler.py", > line 198, in new_save_module_dict* > *return old_save_module_dict(pickler, obj)* > *File > "/home/marappan/sanmgcppy2env/local/lib/python2.7/site-packages/dill/_dill.py", > line 902, in save_module_dict* > *StockPickler.save_dict(pickler, obj)* > *File "/usr/lib/python2.7/pickle.py", line 655, in save_dict* > *self._batch_setitems(obj.iteritems())* > *File "/usr/lib/python2.7/pickle.py", line 687, in _batch_setitems* > *save(v)* > *File "/usr/lib/python2.7/pickle.py", line 331, in save* > *self.save_reduce(obj=obj, *rv)* > *File "/usr/lib/python2.7/pickle.py", line 425, in save_reduce* > *save(state)* > *File "/usr/lib/python2.7/pickle.py", line 286, in save* > *f(self, obj) # Call unbound method with explicit self* > *File > "/home/marappan/sanmgcppy2env/local/lib/python2.7/site-packages/apache_beam/internal/pickler.py", > line 198, in new_save_module_dict* > *return old_save_module_dict(pickler, obj)* > *File > "/home/marappan/sanmgcppy2env/local/lib/python2.7/site-packages/dill/_dill.py", > line 902, in save_module_dict* > *StockPickler.save_dict(pickler, obj)* > *File "/usr/lib/python2.7/pickle.py", line 655, in save_dict* > *self._batch_setitems(obj.iteritems())* > *File "/usr/lib/python2.7/pickle.py", line 687, in _batch_setitems* > *save(v)* > *File "/usr/lib/python2.7/pickle.py", line 331, in save* > *self.save_reduce(obj=obj, *rv)* > *File "/usr/lib/python2.7/pickle.py", line 425, in save_reduce* > *save(state)* > *File "/usr/lib/python2.7/pickle.py", line 286, in save* > *f(self, obj) # Call unbound method with explicit self* > *File > "/home/marappan/sanmgcppy2env/local/lib/python2.7/site-packages/apache_beam/internal/pickler.py", > line 198, in new_save_module_dict* > *return old_save_module_dict(pickler, obj)* > *File > "/home/marappan/sanmgcppy2env/local/lib/python2.7/site-packages/dill/_dill.py", > line 902, in save_module_dict* > *StockPickler.save_dict(pickler, obj)* > *File "/usr/lib/python2.7/pickle.py", line 655, in save_dict* > *self._batch_setitems(obj.iteritems())* > *File "/usr/lib/python2.7/pickle.py", line 687, in _batch_setitems* > *save(v)* > *File "/usr/lib/python2.7/pickle.py", line 331, in save* > *self.save_reduce(obj=obj, *rv)* > *File "/usr/lib/python2.7/pickle.py", line 425, in save_reduce* > *save(state)* > *File "/usr/lib/python
[jira] [Updated] (BEAM-7891) Vendoring packaging is still buggy
[
https://issues.apache.org/jira/browse/BEAM-7891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ismaël Mejía updated BEAM-7891:
---
Priority: Major (was: Blocker)
> Vendoring packaging is still buggy
> --
>
> Key: BEAM-7891
> URL: https://issues.apache.org/jira/browse/BEAM-7891
> Project: Beam
> Issue Type: Bug
> Components: build-system
>Reporter: Romain Manni-Bucau
>Priority: Major
> Fix For: 2.14.0
>
>
> In 2.14 the overlapping bug between modules is still not fixed, it still
> prevents to use beam with some JVM, pollutes a lot shadowing/uber jar
> creation and can prevent beam to run under some classloading setup
> (potentielly in an engine/runner). Here is one example:
>
> {code:java}
> [INFO] [WARNING] beam-vendor-grpc-1_13_1-0.2.jar,
> beam-vendor-sdks-java-extensions-protobuf-2.14.0.jar define 1814 overlapping
> classes:
> [INFO] [WARNING] -
> org.apache.beam.vendor.grpc.v1p13p1.com.google.common.collect.ImmutableMapValues$1
> [INFO] [WARNING] -
> org.apache.beam.vendor.grpc.v1p13p1.com.google.common.util.concurrent.ImmediateFuture$ImmediateCancelledFuture
> [INFO] [WARNING] -
> org.apache.beam.vendor.grpc.v1p13p1.com.google.common.base.Converter$ReverseConverter
> [INFO] [WARNING] -
> org.apache.beam.vendor.grpc.v1p13p1.com.google.common.hash.HashCode$IntHashCode
> [INFO] [WARNING] -
> org.apache.beam.vendor.grpc.v1p13p1.com.google.common.collect.Iterables$8$1
> [INFO] [WARNING] -
> org.apache.beam.vendor.grpc.v1p13p1.com.google.common.collect.HashBiMap
> [INFO] [WARNING] -
> org.apache.beam.vendor.grpc.v1p13p1.com.google.common.cache.CacheBuilderSpec$WriteDurationParser
> [INFO] [WARNING] -
> org.apache.beam.vendor.grpc.v1p13p1.com.google.common.collect.Multiset$Entry
> [INFO] [WARNING] -
> org.apache.beam.vendor.grpc.v1p13p1.com.google.common.graph.AbstractValueGraph
> [INFO] [WARNING] -
> org.apache.beam.vendor.grpc.v1p13p1.com.google.common.util.concurrent.InterruptibleTask{code}
> This task is indeed about fixing the overlappings but also ensuring it can't
> come in 2.15 since all versions are affected since vendoring had been set up
> and it never had been cleanly fixed on all the build.
>
> Thanks
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (BEAM-8647) Remove .mailmap from the sources
[ https://issues.apache.org/jira/browse/BEAM-8647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismaël Mejía reassigned BEAM-8647: -- Assignee: Ismaël Mejía > Remove .mailmap from the sources > > > Key: BEAM-8647 > URL: https://issues.apache.org/jira/browse/BEAM-8647 > Project: Beam > Issue Type: Task > Components: build-system >Reporter: Romain Manni-Bucau >Assignee: Ismaël Mejía >Priority: Major > > Hi, > > .mailmap manipulates individuals data which are considered "personal" (name, > email etc) > AFAIK Apache/Beam is not allowed to do it straight, in particular for EU > citizens (_GDPR)._ > Can the file be removed since it is not used by the beam project (at least > apache/beam repo)? > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8647) Remove .mailmap from the sources
[ https://issues.apache.org/jira/browse/BEAM-8647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismaël Mejía updated BEAM-8647: --- Status: Open (was: Triage Needed) > Remove .mailmap from the sources > > > Key: BEAM-8647 > URL: https://issues.apache.org/jira/browse/BEAM-8647 > Project: Beam > Issue Type: Task > Components: build-system >Reporter: Romain Manni-Bucau >Priority: Major > > Hi, > > .mailmap manipulates individuals data which are considered "personal" (name, > email etc) > AFAIK Apache/Beam is not allowed to do it straight, in particular for EU > citizens (_GDPR)._ > Can the file be removed since it is not used by the beam project (at least > apache/beam repo)? > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8647) Remove .mailmap from the sources
[ https://issues.apache.org/jira/browse/BEAM-8647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17047422#comment-17047422 ] Ismaël Mejía commented on BEAM-8647: Since this looks like a lawyer matter I opened the discussion in [privacy-disc...@apache.org|mailto:privacy-disc...@apache.org] will update once The ASF has clear policies on GDPR. > Remove .mailmap from the sources > > > Key: BEAM-8647 > URL: https://issues.apache.org/jira/browse/BEAM-8647 > Project: Beam > Issue Type: Task > Components: build-system >Reporter: Romain Manni-Bucau >Assignee: Ismaël Mejía >Priority: Major > > Hi, > > .mailmap manipulates individuals data which are considered "personal" (name, > email etc) > AFAIK Apache/Beam is not allowed to do it straight, in particular for EU > citizens (_GDPR)._ > Can the file be removed since it is not used by the beam project (at least > apache/beam repo)? > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-6407) regression: FileIO.writeDynamic() with side inputs fails in DirectRunner
[
https://issues.apache.org/jira/browse/BEAM-6407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ismaël Mejía updated BEAM-6407:
---
Priority: Major (was: Blocker)
> regression: FileIO.writeDynamic() with side inputs fails in DirectRunner
>
>
> Key: BEAM-6407
> URL: https://issues.apache.org/jira/browse/BEAM-6407
> Project: Beam
> Issue Type: Bug
> Components: runner-direct
>Affects Versions: 2.9.0
>Reporter: Niel Markwick
>Assignee: Niel Markwick
>Priority: Major
> Labels: regression
> Fix For: 2.10.0
>
> Attachments: beam-filewriter-demo.tgz
>
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> When FileIO.writeDynamic is used with automatic sharding and a Contextful.Fn
> that uses side inputs for the file naming, DirectRunner (and TestPipeline)
> fail with:
> {{java.lang.IllegalStateException: All PCollectionViews that are consumed
> must be written by some WriteView PTransform: Missing [
> [RunnerPCollectionView]]}}
>
> Example code:
> {code:java}
> PCollectionView outputFileName =
> pipeline.apply(
> "outputDir",
> Create.of("/tmp/testout")).apply(View.asSingleton());
> Contextful.Fn manifestNaming =
> (element, c) ->
> (window, pane, numShards, shardIndex, compression) ->
> c.sideInput(outputFileName)+shardIndex;
> pipeline.apply(FileIO.writeDynamic()
> .by(SerializableFunctions.constant(""))
> .withDestinationCoder(StringUtf8Coder.of())
> .via(TextIO.sink())
> .withTempDirectory("/tmp")
> .withNaming(Contextful.of(
> manifestNaming,
> Requirements.requiresSideInputs(outputFileName;
> {code}
>
> This does not occur in Dataflow-runner
> It does not occur if the ContextFul.Fn is not given side inputs.
> It does not occur if withNumShards(1) is set.
> It did not occur in 2.8.0, and does in 2.9.0 and 2.10.0-SNAPSHOT (as of today)
>
> The cause appears to be due to the DirectRunner using TransformOverrides
> re-writing FileIO sinks to use runner-determined-sharding
> ( see [DirectRunner.java line
> 226|https://github.com/apache/beam/blob/master/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java#L226]
> )
> but I do not know why this started occuring in 2.9.0...
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-7798) After changes in type inference, apache_beam.io.gcp.bigquery_io_read_it_test.BigqueryIOReadIT.test_bigquery_read_1M_python is failing in Python 3.5 postcommits
[
https://issues.apache.org/jira/browse/BEAM-7798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ismaël Mejía updated BEAM-7798:
---
Priority: Major (was: Blocker)
> After changes in type inference,
> apache_beam.io.gcp.bigquery_io_read_it_test.BigqueryIOReadIT.test_bigquery_read_1M_python
> is failing in Python 3.5 postcommits
> ---
>
> Key: BEAM-7798
> URL: https://issues.apache.org/jira/browse/BEAM-7798
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Robert Bradshaw
>Priority: Major
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> {noformat}
> Error Message
> Tuple[t0, t1, ...]: each t must be a type. Got Any.
> Stacktrace
> Traceback (most recent call last):
> File "/usr/lib/python3.5/unittest/case.py", line 58, in testPartExecutor
> yield
> File "/usr/lib/python3.5/unittest/case.py", line 600, in run
> testMethod()
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/io/gcp/bigquery_io_read_it_test.py",
> line 58, in test_bigquery_read_1M_python
> self.run_bigquery_io_read_pipeline('1M')
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/io/gcp/bigquery_io_read_it_test.py",
> line 54, in run_bigquery_io_read_pipeline
> **extra_opts))
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/io/gcp/bigquery_io_read_pipeline.py",
> line 74, in run
> p.run()
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/testing/test_pipeline.py",
> line 107, in run
> else test_runner_api))
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/pipeline.py",
> line 406, in run
> self._options).run(False)
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/pipeline.py",
> line 419, in run
> return self.runner.run_pipeline(self, self._options)
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/runners/direct/test_direct_runner.py",
> line 43, in run_pipeline
> self.result = super(TestDirectRunner, self).run_pipeline(pipeline,
> options)
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/runners/direct/direct_runner.py",
> line 129, in run_pipeline
> return runner.run_pipeline(pipeline, options)
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/runners/direct/direct_runner.py",
> line 355, in run_pipeline
> pipeline.replace_all(_get_transform_overrides(options))
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/pipeline.py",
> line 389, in replace_all
> self._replace(override)
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/pipeline.py",
> line 300, in _replace
> self.visit(TransformUpdater(self))
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/pipeline.py",
> line 447, in visit
> self._root_transform().visit(visitor, self, visited)
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/pipeline.py",
> line 824, in visit
> part.visit(visitor, pipeline, visited)
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/pipeline.py",
> line 824, in visit
> part.visit(visitor, pipeline, visited)
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/pipeline.py",
> line 824, in visit
> part.visit(visitor, pipeline, visited)
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/pipeline.py",
> line 822, in visit
> visitor.enter_composite_transform(self)
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/pipeline.py",
> line 295, in enter_composite_transform
> self._replace_if_needed(transform_node)
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/pipeline.py",
> line 264, in _replace_if_needed
> new_output = replacement_transform.expand(input_node)
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/runners/direct/h
[jira] [Work logged] (BEAM-8542) Add async write to AWS SNS IO & remove retry logic
[ https://issues.apache.org/jira/browse/BEAM-8542?focusedWorklogId=394811&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-394811 ] ASF GitHub Bot logged work on BEAM-8542: Author: ASF GitHub Bot Created on: 28/Feb/20 10:32 Start Date: 28/Feb/20 10:32 Worklog Time Spent: 10m Work Description: aromanenko-dev commented on issue #10078: [BEAM-8542] Change write to async in AWS SNS IO & remove retry logic URL: https://github.com/apache/beam/pull/10078#issuecomment-592054114 @Akshay-Iyangar Thank you for taking care of this! Could you exclude a commit that merges master into feature branch and use rebase instead? Also, please squash all *your* commits but keep Ajo Thomas commit as well (to respect all creds). And finally, please, format all commits with the following pattern - "[BEAM-8542] Commit message" (see [Contribution Guide](https://beam.apache.org/contribute/#make-your-change)) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 394811) Time Spent: 5.5h (was: 5h 20m) > Add async write to AWS SNS IO & remove retry logic > -- > > Key: BEAM-8542 > URL: https://issues.apache.org/jira/browse/BEAM-8542 > Project: Beam > Issue Type: Improvement > Components: io-java-aws >Reporter: Ajo Thomas >Assignee: Ajo Thomas >Priority: Major > Time Spent: 5.5h > Remaining Estimate: 0h > > - While working with SNS IO for one of my work-related projects, I found that > the IO uses synchronous publishes during writes. I had a simple mock pipeline > where I was reading from a kinesis stream and publishing it to SNS using > Beam's SNS IO. For comparison, I also had a lamdba which did the same using > asynchronous publishes but was about 5x faster. Changing the SNS IO to use > async publishes would improve publish latencies. > - SNS IO also has some retry logic which isn't required as SNS clients can > handle retries. The retry logic in the SNS client is user-configurable and > therefore, an explicit retry logic in SNS IO is not required > I have a working version of the IO with these changes, will create a PR > linking this ticket to it once I get some feedback here. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8542) Add async write to AWS SNS IO & remove retry logic
[ https://issues.apache.org/jira/browse/BEAM-8542?focusedWorklogId=394814&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-394814 ] ASF GitHub Bot logged work on BEAM-8542: Author: ASF GitHub Bot Created on: 28/Feb/20 10:32 Start Date: 28/Feb/20 10:32 Worklog Time Spent: 10m Work Description: aromanenko-dev commented on issue #10078: [BEAM-8542] Change write to async in AWS SNS IO & remove retry logic URL: https://github.com/apache/beam/pull/10078#issuecomment-592054114 @Akshay-Iyangar Thank you for taking care of this! Could you exclude a commit that merges master into feature branch and use rebase instead? Also, please squash all **your** commits but keep **Ajo Thomas** commit as well (to respect all creds). And finally, please, format all commits with the following pattern - "[BEAM-8542] Commit message" (see [Contribution Guide](https://beam.apache.org/contribute/#make-your-change)) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 394814) Time Spent: 5h 50m (was: 5h 40m) > Add async write to AWS SNS IO & remove retry logic > -- > > Key: BEAM-8542 > URL: https://issues.apache.org/jira/browse/BEAM-8542 > Project: Beam > Issue Type: Improvement > Components: io-java-aws >Reporter: Ajo Thomas >Assignee: Ajo Thomas >Priority: Major > Time Spent: 5h 50m > Remaining Estimate: 0h > > - While working with SNS IO for one of my work-related projects, I found that > the IO uses synchronous publishes during writes. I had a simple mock pipeline > where I was reading from a kinesis stream and publishing it to SNS using > Beam's SNS IO. For comparison, I also had a lamdba which did the same using > asynchronous publishes but was about 5x faster. Changing the SNS IO to use > async publishes would improve publish latencies. > - SNS IO also has some retry logic which isn't required as SNS clients can > handle retries. The retry logic in the SNS client is user-configurable and > therefore, an explicit retry logic in SNS IO is not required > I have a working version of the IO with these changes, will create a PR > linking this ticket to it once I get some feedback here. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8972) Add a Jenkins job running Combine load test on Java with Flink in Portability mode
[ https://issues.apache.org/jira/browse/BEAM-8972?focusedWorklogId=394812&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-394812 ] ASF GitHub Bot logged work on BEAM-8972: Author: ASF GitHub Bot Created on: 28/Feb/20 10:32 Start Date: 28/Feb/20 10:32 Worklog Time Spent: 10m Work Description: mwalenia commented on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-592452480 Run Load Tests Java Combine Portable Flink Batch This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 394812) Time Spent: 13.5h (was: 13h 20m) > Add a Jenkins job running Combine load test on Java with Flink in Portability > mode > -- > > Key: BEAM-8972 > URL: https://issues.apache.org/jira/browse/BEAM-8972 > Project: Beam > Issue Type: Improvement > Components: testing >Reporter: Michał Walenia >Assignee: Michał Walenia >Priority: Minor > Time Spent: 13.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8542) Add async write to AWS SNS IO & remove retry logic
[ https://issues.apache.org/jira/browse/BEAM-8542?focusedWorklogId=394813&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-394813 ] ASF GitHub Bot logged work on BEAM-8542: Author: ASF GitHub Bot Created on: 28/Feb/20 10:32 Start Date: 28/Feb/20 10:32 Worklog Time Spent: 10m Work Description: aromanenko-dev commented on issue #10078: [BEAM-8542] Change write to async in AWS SNS IO & remove retry logic URL: https://github.com/apache/beam/pull/10078#issuecomment-592054114 @Akshay-Iyangar Thank you for taking care of this! Could you exclude a commit that merges master into feature branch and use rebase instead? Also, please squash all **your** commits but keep **Ajo Thomas** commit as well (to respect all creds). And finally, please, format all commits with the following pattern - "[BEAM-8542] Commit message" (see [Contribution Guide](https://beam.apache.org/contribute/#make-your-change)) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 394813) Time Spent: 5h 40m (was: 5.5h) > Add async write to AWS SNS IO & remove retry logic > -- > > Key: BEAM-8542 > URL: https://issues.apache.org/jira/browse/BEAM-8542 > Project: Beam > Issue Type: Improvement > Components: io-java-aws >Reporter: Ajo Thomas >Assignee: Ajo Thomas >Priority: Major > Time Spent: 5h 40m > Remaining Estimate: 0h > > - While working with SNS IO for one of my work-related projects, I found that > the IO uses synchronous publishes during writes. I had a simple mock pipeline > where I was reading from a kinesis stream and publishing it to SNS using > Beam's SNS IO. For comparison, I also had a lamdba which did the same using > asynchronous publishes but was about 5x faster. Changing the SNS IO to use > async publishes would improve publish latencies. > - SNS IO also has some retry logic which isn't required as SNS clients can > handle retries. The retry logic in the SNS client is user-configurable and > therefore, an explicit retry logic in SNS IO is not required > I have a working version of the IO with these changes, will create a PR > linking this ticket to it once I get some feedback here. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8959) Boolean pipeline options which default to true cannot be set to false
[
https://issues.apache.org/jira/browse/BEAM-8959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17047444#comment-17047444
]
Ismaël Mejía commented on BEAM-8959:
Is this one fixed and just unresolved?
> Boolean pipeline options which default to true cannot be set to false
> -
>
> Key: BEAM-8959
> URL: https://issues.apache.org/jira/browse/BEAM-8959
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Maximilian Michels
>Priority: Critical
> Time Spent: 10m
> Remaining Estimate: 0h
>
> With the included argument parser, any boolean pipeline options which default
> to true cannot be set to false, e.g. {{--enable_metrics=false}}:
> {noformat}
> error: argument --enable_metrics: ignored explicit argument 'false'
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-8916) external_test_it.py is not collected by pytest
[ https://issues.apache.org/jira/browse/BEAM-8916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17047446#comment-17047446 ] Ismaël Mejía commented on BEAM-8916: Is this one fixed and just unresolved? can we close it? > external_test_it.py is not collected by pytest > -- > > Key: BEAM-8916 > URL: https://issues.apache.org/jira/browse/BEAM-8916 > Project: Beam > Issue Type: Bug > Components: sdk-py-core, testing >Reporter: Udi Meiri >Assignee: Chamikara Madhusanka Jayalath >Priority: Critical > Time Spent: 1h 40m > Remaining Estimate: 0h > > pytest only collects tests matching these patterns: > https://github.com/apache/beam/blob/8066d78f0fd2237b718859d4a776511203880df0/sdks/python/pytest.ini#L27 > Please rename the file. (ex: external_integration_test.py) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8453) Failure in org.apache.beam.sdk.io.jms.JmsIOTest.testCheckpointMarkSafety
[
https://issues.apache.org/jira/browse/BEAM-8453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ismaël Mejía updated BEAM-8453:
---
Status: Open (was: Triage Needed)
> Failure in org.apache.beam.sdk.io.jms.JmsIOTest.testCheckpointMarkSafety
>
>
> Key: BEAM-8453
> URL: https://issues.apache.org/jira/browse/BEAM-8453
> Project: Beam
> Issue Type: Bug
> Components: io-java-jms
>Reporter: Kenneth Knowles
>Priority: Critical
> Labels: flake
>
> {code}
> Caused by: java.lang.NullPointerException: (No message provided)
> at
> org.apache.beam.sdk.io.jms.JmsIOTest.lambda$withSlowAcks$2(JmsIOTest.java:463)
> at
> org.apache.beam.sdk.io.jms.JmsIOTest.lambda$proxyMethod$6(JmsIOTest.java:489)
> at com.sun.proxy.$Proxy62.receiveNoWait(Unknown Source)
> at org.apache.beam.sdk.io.jms.JmsIO$UnboundedJmsReader.advance(JmsIO.java:512)
> at
> org.apache.beam.sdk.io.jms.JmsIOTest.testCheckpointMarkSafety(JmsIOTest.java:381)
> ...
> {code}
> stdout:
> {code}
> Oct 21, 2019 9:52:32 PM org.apache.activemq.broker.BrokerService
> doStartPersistenceAdapter
> INFO: Using Persistence Adapter: MemoryPersistenceAdapter
> Oct 21, 2019 9:52:32 PM org.apache.activemq.store.kahadb.plist.PListStoreImpl
> doStart
> INFO:
> PListStore:[/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_Commit@2/src/sdks/java/io/jms/activemq-data/localhost/tmp_storage]
> started
> Oct 21, 2019 9:52:32 PM org.apache.activemq.broker.BrokerService doStartBroker
> INFO: Apache ActiveMQ 5.13.1 (localhost,
> ID:apache-beam-jenkins-8-45641-1571694713139-0:6) is starting
> Oct 21, 2019 9:52:32 PM org.apache.activemq.broker.TransportConnector start
> INFO: Connector vm://localhost started
> Oct 21, 2019 9:52:32 PM org.apache.activemq.broker.BrokerService doStartBroker
> INFO: Apache ActiveMQ 5.13.1 (localhost,
> ID:apache-beam-jenkins-8-45641-1571694713139-0:6) started
> Oct 21, 2019 9:52:32 PM org.apache.activemq.broker.BrokerService doStartBroker
> INFO: For help or more information please see: http://activemq.apache.org
> Oct 21, 2019 9:52:33 PM org.apache.activemq.broker.BrokerService stop
> INFO: Apache ActiveMQ 5.13.1 (localhost,
> ID:apache-beam-jenkins-8-45641-1571694713139-0:6) is shutting down
> Oct 21, 2019 9:52:33 PM org.apache.activemq.broker.TransportConnector stop
> INFO: Connector vm://localhost stopped
> Oct 21, 2019 9:52:33 PM org.apache.activemq.store.kahadb.plist.PListStoreImpl
> doStop
> INFO:
> PListStore:[/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_Commit@2/src/sdks/java/io/jms/activemq-data/localhost/tmp_storage]
> stopped
> Oct 21, 2019 9:52:33 PM org.apache.activemq.broker.BrokerService stop
> INFO: Apache ActiveMQ 5.13.1 (localhost,
> ID:apache-beam-jenkins-8-45641-1571694713139-0:6) uptime 0.512 seconds
> Oct 21, 2019 9:52:33 PM org.apache.activemq.broker.BrokerService stop
> INFO: Apache ActiveMQ 5.13.1 (localhost,
> ID:apache-beam-jenkins-8-45641-1571694713139-0:6) is shutdown
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (BEAM-8453) Failure in org.apache.beam.sdk.io.jms.JmsIOTest.testCheckpointMarkSafety
[
https://issues.apache.org/jira/browse/BEAM-8453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ismaël Mejía resolved BEAM-8453.
Fix Version/s: Not applicable
Resolution: Cannot Reproduce
Seems this was solved by BEAM-7427
Please reopen if you see it happening again.
> Failure in org.apache.beam.sdk.io.jms.JmsIOTest.testCheckpointMarkSafety
>
>
> Key: BEAM-8453
> URL: https://issues.apache.org/jira/browse/BEAM-8453
> Project: Beam
> Issue Type: Bug
> Components: io-java-jms
>Reporter: Kenneth Knowles
>Priority: Critical
> Labels: flake
> Fix For: Not applicable
>
>
> {code}
> Caused by: java.lang.NullPointerException: (No message provided)
> at
> org.apache.beam.sdk.io.jms.JmsIOTest.lambda$withSlowAcks$2(JmsIOTest.java:463)
> at
> org.apache.beam.sdk.io.jms.JmsIOTest.lambda$proxyMethod$6(JmsIOTest.java:489)
> at com.sun.proxy.$Proxy62.receiveNoWait(Unknown Source)
> at org.apache.beam.sdk.io.jms.JmsIO$UnboundedJmsReader.advance(JmsIO.java:512)
> at
> org.apache.beam.sdk.io.jms.JmsIOTest.testCheckpointMarkSafety(JmsIOTest.java:381)
> ...
> {code}
> stdout:
> {code}
> Oct 21, 2019 9:52:32 PM org.apache.activemq.broker.BrokerService
> doStartPersistenceAdapter
> INFO: Using Persistence Adapter: MemoryPersistenceAdapter
> Oct 21, 2019 9:52:32 PM org.apache.activemq.store.kahadb.plist.PListStoreImpl
> doStart
> INFO:
> PListStore:[/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_Commit@2/src/sdks/java/io/jms/activemq-data/localhost/tmp_storage]
> started
> Oct 21, 2019 9:52:32 PM org.apache.activemq.broker.BrokerService doStartBroker
> INFO: Apache ActiveMQ 5.13.1 (localhost,
> ID:apache-beam-jenkins-8-45641-1571694713139-0:6) is starting
> Oct 21, 2019 9:52:32 PM org.apache.activemq.broker.TransportConnector start
> INFO: Connector vm://localhost started
> Oct 21, 2019 9:52:32 PM org.apache.activemq.broker.BrokerService doStartBroker
> INFO: Apache ActiveMQ 5.13.1 (localhost,
> ID:apache-beam-jenkins-8-45641-1571694713139-0:6) started
> Oct 21, 2019 9:52:32 PM org.apache.activemq.broker.BrokerService doStartBroker
> INFO: For help or more information please see: http://activemq.apache.org
> Oct 21, 2019 9:52:33 PM org.apache.activemq.broker.BrokerService stop
> INFO: Apache ActiveMQ 5.13.1 (localhost,
> ID:apache-beam-jenkins-8-45641-1571694713139-0:6) is shutting down
> Oct 21, 2019 9:52:33 PM org.apache.activemq.broker.TransportConnector stop
> INFO: Connector vm://localhost stopped
> Oct 21, 2019 9:52:33 PM org.apache.activemq.store.kahadb.plist.PListStoreImpl
> doStop
> INFO:
> PListStore:[/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_Commit@2/src/sdks/java/io/jms/activemq-data/localhost/tmp_storage]
> stopped
> Oct 21, 2019 9:52:33 PM org.apache.activemq.broker.BrokerService stop
> INFO: Apache ActiveMQ 5.13.1 (localhost,
> ID:apache-beam-jenkins-8-45641-1571694713139-0:6) uptime 0.512 seconds
> Oct 21, 2019 9:52:33 PM org.apache.activemq.broker.BrokerService stop
> INFO: Apache ActiveMQ 5.13.1 (localhost,
> ID:apache-beam-jenkins-8-45641-1571694713139-0:6) is shutdown
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-8386) Java Dependencies out of date
[ https://issues.apache.org/jira/browse/BEAM-8386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismaël Mejía updated BEAM-8386: --- Priority: Major (was: Critical) > Java Dependencies out of date > - > > Key: BEAM-8386 > URL: https://issues.apache.org/jira/browse/BEAM-8386 > Project: Beam > Issue Type: Bug > Components: website >Reporter: Mehran Nazir >Assignee: Kenneth Knowles >Priority: Major > > When navigating to see the "Apache Beam 2.x SDKs" in the Datafloc docs > ([https://cloud.google.com/dataflow/docs/concepts/sdk-worker-dependencies#beam-2x-java]), > you must click through to the Beam website > ([https://beam.apache.org/documentation/sdks/java-dependencies/|https://www.google.com/url?q=https://beam.apache.org/documentation/sdks/java-dependencies/&sa=D&usg=AFQjCNGxu_OwwI-9x88aRB0l7b9G-FnnSQ]). > However, it appears that this hasn't been updated since 2.9.0. > External libraries for 2.16 can be found here > ([https://github.com/apache/beam/blob/release-2.16.0/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L434|https://www.google.com/url?q=https://github.com/apache/beam/blob/release-2.16.0/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy%23L434&sa=D&usg=AFQjCNHODYjJ3BTENZrgUOj82dG_Pl8Usw]). > Additional context found here: > [https://lists.apache.org/thread.html/eb25c9a2c9d26117a6f0ee924902ecd46cd5c2063025918df7080d3a@%3Cdev.beam.apache.org%3E] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-7298) Upgrade zookeeper
[ https://issues.apache.org/jira/browse/BEAM-7298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismaël Mejía updated BEAM-7298: --- Priority: Major (was: Critical) > Upgrade zookeeper > - > > Key: BEAM-7298 > URL: https://issues.apache.org/jira/browse/BEAM-7298 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Etienne Chauchot >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-7296) Upgrade jetty-http
[ https://issues.apache.org/jira/browse/BEAM-7296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismaël Mejía updated BEAM-7296: --- Priority: Major (was: Critical) > Upgrade jetty-http > --- > > Key: BEAM-7296 > URL: https://issues.apache.org/jira/browse/BEAM-7296 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Etienne Chauchot >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-7290) Upgrade hadoop-client
[ https://issues.apache.org/jira/browse/BEAM-7290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismaël Mejía updated BEAM-7290: --- Priority: Blocker (was: Critical) > Upgrade hadoop-client > - > > Key: BEAM-7290 > URL: https://issues.apache.org/jira/browse/BEAM-7290 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Etienne Chauchot >Assignee: Mujuzi Moses >Priority: Blocker > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-7290) Upgrade hadoop-client
[ https://issues.apache.org/jira/browse/BEAM-7290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismaël Mejía updated BEAM-7290: --- Priority: Major (was: Blocker) > Upgrade hadoop-client > - > > Key: BEAM-7290 > URL: https://issues.apache.org/jira/browse/BEAM-7290 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Etienne Chauchot >Assignee: Mujuzi Moses >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-7295) Upgrade javax.mail
[ https://issues.apache.org/jira/browse/BEAM-7295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismaël Mejía updated BEAM-7295: --- Priority: Major (was: Critical) > Upgrade javax.mail > -- > > Key: BEAM-7295 > URL: https://issues.apache.org/jira/browse/BEAM-7295 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Etienne Chauchot >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-7293) Upgrade hadoop-yarn-api
[ https://issues.apache.org/jira/browse/BEAM-7293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismaël Mejía updated BEAM-7293: --- Priority: Major (was: Critical) > Upgrade hadoop-yarn-api > --- > > Key: BEAM-7293 > URL: https://issues.apache.org/jira/browse/BEAM-7293 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Etienne Chauchot >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-7294) Upgrade hadoop-yarn-common
[ https://issues.apache.org/jira/browse/BEAM-7294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismaël Mejía updated BEAM-7294: --- Priority: Major (was: Critical) > Upgrade hadoop-yarn-common > -- > > Key: BEAM-7294 > URL: https://issues.apache.org/jira/browse/BEAM-7294 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Etienne Chauchot >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-7292) Upgrade hadoop-mapreduce-client-core
[ https://issues.apache.org/jira/browse/BEAM-7292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismaël Mejía updated BEAM-7292: --- Priority: Major (was: Critical) > Upgrade hadoop-mapreduce-client-core > > > Key: BEAM-7292 > URL: https://issues.apache.org/jira/browse/BEAM-7292 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Etienne Chauchot >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8972) Add a Jenkins job running Combine load test on Java with Flink in Portability mode
[ https://issues.apache.org/jira/browse/BEAM-8972?focusedWorklogId=394820&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-394820 ] ASF GitHub Bot logged work on BEAM-8972: Author: ASF GitHub Bot Created on: 28/Feb/20 10:44 Start Date: 28/Feb/20 10:44 Worklog Time Spent: 10m Work Description: mwalenia commented on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-592457334 run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 394820) Time Spent: 13h 40m (was: 13.5h) > Add a Jenkins job running Combine load test on Java with Flink in Portability > mode > -- > > Key: BEAM-8972 > URL: https://issues.apache.org/jira/browse/BEAM-8972 > Project: Beam > Issue Type: Improvement > Components: testing >Reporter: Michał Walenia >Assignee: Michał Walenia >Priority: Minor > Time Spent: 13h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-7291) Upgrade hadoop-common
[ https://issues.apache.org/jira/browse/BEAM-7291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismaël Mejía updated BEAM-7291: --- Priority: Major (was: Critical) > Upgrade hadoop-common > - > > Key: BEAM-7291 > URL: https://issues.apache.org/jira/browse/BEAM-7291 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Etienne Chauchot >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-7278) Upgrade some Beam dependencies
[ https://issues.apache.org/jira/browse/BEAM-7278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismaël Mejía updated BEAM-7278: --- Priority: Major (was: Critical) > Upgrade some Beam dependencies > -- > > Key: BEAM-7278 > URL: https://issues.apache.org/jira/browse/BEAM-7278 > Project: Beam > Issue Type: Task > Components: dependencies >Reporter: Etienne Chauchot >Assignee: Mujuzi Moses >Priority: Major > Time Spent: 3h 20m > Remaining Estimate: 0h > > Some dependencies need to be upgraded. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-7278) Upgrade some Beam dependencies
[ https://issues.apache.org/jira/browse/BEAM-7278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismaël Mejía reassigned BEAM-7278: -- Assignee: (was: Mujuzi Moses) > Upgrade some Beam dependencies > -- > > Key: BEAM-7278 > URL: https://issues.apache.org/jira/browse/BEAM-7278 > Project: Beam > Issue Type: Task > Components: dependencies >Reporter: Etienne Chauchot >Priority: Major > Time Spent: 3h 20m > Remaining Estimate: 0h > > Some dependencies need to be upgraded. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-7290) Upgrade hadoop-client
[ https://issues.apache.org/jira/browse/BEAM-7290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismaël Mejía reassigned BEAM-7290: -- Assignee: (was: Mujuzi Moses) > Upgrade hadoop-client > - > > Key: BEAM-7290 > URL: https://issues.apache.org/jira/browse/BEAM-7290 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Etienne Chauchot >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9361) NPE When putting Avro record with enum through SqlTransform
[
https://issues.apache.org/jira/browse/BEAM-9361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17047461#comment-17047461
]
Ryan Skraba commented on BEAM-9361:
---
Interesting! It would be great, of course, if every Avro record was guaranteed
to be representable by a Beam Row.
It looks like there isn't any support for ENUM types in
[Calcite|https://calcite.apache.org/docs/reference.html#data-types] (although
[some databases|https://dev.mysql.com/doc/refman/8.0/en/enum.html] do). Beam
Row doesn't either, but does support [logical
types|https://github.com/apache/beam/blob/d7df9ed14bca07d341bb689053e82674bf0e0243/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java#L375].
What do you think about mapping Avro enums to strings or ints, perhaps
configurable in the toBeamRowXxxx type method? This would have the
consequence of choosing between:
{code}
SELECT name, direction FROM InputStream where direction = "PULL"
{code}
or
{code}
SELECT name, direction FROM InputStream where direction = 0
{code}
(With readability/efficiency tradeoffs!)
> NPE When putting Avro record with enum through SqlTransform
> ---
>
> Key: BEAM-9361
> URL: https://issues.apache.org/jira/browse/BEAM-9361
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql
>Affects Versions: 2.19.0
>Reporter: Niels Basjes
>Priority: Major
>
> I ran into this problem when trying to put my Avro records through the
> SqlTransform.
> I was able to reduce the reproduction path to the code below.
> This code fails on my machine (using Beam 2.19.0) with the following
> NullPointerException
> {code:java}
> org.apache.beam.sdk.extensions.sql.impl.ParseException: Unable to parse
> query SELECT name, direction FROM InputStreamat
> org.apache.beam.sdk.extensions.sql.impl.CalciteQueryPlanner.convertToBeamRel(CalciteQueryPlanner.java:175)
> at
> org.apache.beam.sdk.extensions.sql.impl.BeamSqlEnv.parseQuery(BeamSqlEnv.java:103)
> at
> org.apache.beam.sdk.extensions.sql.SqlTransform.expand(SqlTransform.java:125)
> at
> org.apache.beam.sdk.extensions.sql.SqlTransform.expand(SqlTransform.java:83)
> at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:539)
> at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:490)
> at
> org.apache.beam.sdk.values.PCollectionTuple.apply(PCollectionTuple.java:261)
> at com.bol.analytics.m2.TestAvro2SQL.testAvro2SQL(TestAvro2SQL.java:99)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:319)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
> at
> com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
> at
> com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:230)
> at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:58)
> Caused by:
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.ValidationException:
> java.lang.NullPointerException
> at
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.prepare.PlannerImpl.validate(PlannerImpl.java:217)
> at
> org.apache.beam.sdk.extensions.sq
[jira] [Assigned] (BEAM-3604) MqttIOTest testReadNoClientId failure timeout flaky test
[
https://issues.apache.org/jira/browse/BEAM-3604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ismaël Mejía reassigned BEAM-3604:
--
Assignee: (was: Ismaël Mejía)
> MqttIOTest testReadNoClientId failure timeout flaky test
>
>
> Key: BEAM-3604
> URL: https://issues.apache.org/jira/browse/BEAM-3604
> Project: Beam
> Issue Type: Bug
> Components: io-java-mqtt
>Reporter: Kenneth Knowles
>Priority: Critical
> Labels: flake
> Fix For: Not applicable
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> I've seen failures a bit today. Here is one:
> [https://builds.apache.org/job/beam_PreCommit_Java_GradleBuild/1758/testReport/junit/org.apache.beam.sdk.io.mqtt/MqttIOTest/testReadNoClientId/]
> Filing all flakes as "Critical" priority so we can sickbay or fix.
> Since that build will get GC'd, here is the Standard Error. It looks like
> from that perspective everything went as planned, but perhaps the test has a
> race condition or something?
> {code}
> Feb 01, 2018 11:28:01 PM org.apache.beam.sdk.io.mqtt.MqttIOTest startBroker
> INFO: Finding free network port
> Feb 01, 2018 11:28:01 PM org.apache.beam.sdk.io.mqtt.MqttIOTest startBroker
> INFO: Starting ActiveMQ brokerService on 57986
> Feb 01, 2018 11:28:03 PM org.apache.activemq.broker.BrokerService
> doStartPersistenceAdapter
> INFO: Using Persistence Adapter: MemoryPersistenceAdapter
> Feb 01, 2018 11:28:04 PM org.apache.activemq.broker.BrokerService
> doStartBroker
> INFO: Apache ActiveMQ 5.13.1 (localhost,
> ID:115.98.154.104.bc.googleusercontent.com-38646-1517527683931-0:1) is
> starting
> Feb 01, 2018 11:28:04 PM
> org.apache.activemq.transport.TransportServerThreadSupport doStart
> INFO: Listening for connections at: mqtt://localhost:57986
> Feb 01, 2018 11:28:04 PM org.apache.activemq.broker.TransportConnector start
> INFO: Connector mqtt://localhost:57986 started
> Feb 01, 2018 11:28:04 PM org.apache.activemq.broker.BrokerService
> doStartBroker
> INFO: Apache ActiveMQ 5.13.1 (localhost,
> ID:115.98.154.104.bc.googleusercontent.com-38646-1517527683931-0:1) started
> Feb 01, 2018 11:28:04 PM org.apache.activemq.broker.BrokerService
> doStartBroker
> INFO: For help or more information please see: http://activemq.apache.org
> Feb 01, 2018 11:28:26 PM org.apache.activemq.broker.BrokerService stop
> INFO: Apache ActiveMQ 5.13.1 (localhost,
> ID:115.98.154.104.bc.googleusercontent.com-38646-1517527683931-0:1) is
> shutting down
> Feb 01, 2018 11:28:26 PM org.apache.activemq.broker.TransportConnector stop
> INFO: Connector mqtt://localhost:57986 stopped
> Feb 01, 2018 11:28:26 PM org.apache.activemq.broker.BrokerService stop
> INFO: Apache ActiveMQ 5.13.1 (localhost,
> ID:115.98.154.104.bc.googleusercontent.com-38646-1517527683931-0:1) uptime
> 24.039 seconds
> Feb 01, 2018 11:28:26 PM org.apache.activemq.broker.BrokerService stop
> INFO: Apache ActiveMQ 5.13.1 (localhost,
> ID:115.98.154.104.bc.googleusercontent.com-38646-1517527683931-0:1) is
> shutdown
> Feb 01, 2018 11:28:26 PM org.apache.beam.sdk.io.mqtt.MqttIOTest startBroker
> INFO: Finding free network port
> Feb 01, 2018 11:28:26 PM org.apache.beam.sdk.io.mqtt.MqttIOTest startBroker
> INFO: Starting ActiveMQ brokerService on 46799
> Feb 01, 2018 11:28:26 PM org.apache.activemq.broker.BrokerService
> doStartPersistenceAdapter
> INFO: Using Persistence Adapter: MemoryPersistenceAdapter
> Feb 01, 2018 11:28:26 PM org.apache.activemq.broker.BrokerService
> doStartBroker
> INFO: Apache ActiveMQ 5.13.1 (localhost,
> ID:115.98.154.104.bc.googleusercontent.com-38646-1517527683931-0:2) is
> starting
> Feb 01, 2018 11:28:26 PM
> org.apache.activemq.transport.TransportServerThreadSupport doStart
> INFO: Listening for connections at: mqtt://localhost:46799
> Feb 01, 2018 11:28:26 PM org.apache.activemq.broker.TransportConnector start
> INFO: Connector mqtt://localhost:46799 started
> Feb 01, 2018 11:28:26 PM org.apache.activemq.broker.BrokerService
> doStartBroker
> INFO: Apache ActiveMQ 5.13.1 (localhost,
> ID:115.98.154.104.bc.googleusercontent.com-38646-1517527683931-0:2) started
> Feb 01, 2018 11:28:26 PM org.apache.activemq.broker.BrokerService
> doStartBroker
> INFO: For help or more information please see: http://activemq.apache.org
> Feb 01, 2018 11:28:28 PM org.apache.beam.sdk.io.mqtt.MqttIOTest
> lambda$testRead$1
> INFO: Waiting pipeline connected to the MQTT broker before sending messages
> ...
> Feb 01, 2018 11:28:35 PM org.apache.activemq.broker.BrokerService stop
> INFO: Apache ActiveMQ 5.13.1 (localhost,
> ID:115.98.154.104.bc.googleusercontent.com-38646-1517527683931-0:2) is
> shutting down
> Feb 01, 2018 11:28:35 PM org.apache.activemq.broker.TransportConnector stop
> INFO: Connector mqtt://localhost:46799 stopped
> Feb
[jira] [Updated] (BEAM-3604) MqttIOTest testReadNoClientId failure timeout flaky test
[
https://issues.apache.org/jira/browse/BEAM-3604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ismaël Mejía updated BEAM-3604:
---
Summary: MqttIOTest testReadNoClientId failure timeout flaky test (was:
MqttIOTest testReadNoClientId failure timeout)
> MqttIOTest testReadNoClientId failure timeout flaky test
>
>
> Key: BEAM-3604
> URL: https://issues.apache.org/jira/browse/BEAM-3604
> Project: Beam
> Issue Type: Bug
> Components: io-java-mqtt
>Reporter: Kenneth Knowles
>Assignee: Ismaël Mejía
>Priority: Critical
> Labels: flake
> Fix For: Not applicable
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> I've seen failures a bit today. Here is one:
> [https://builds.apache.org/job/beam_PreCommit_Java_GradleBuild/1758/testReport/junit/org.apache.beam.sdk.io.mqtt/MqttIOTest/testReadNoClientId/]
> Filing all flakes as "Critical" priority so we can sickbay or fix.
> Since that build will get GC'd, here is the Standard Error. It looks like
> from that perspective everything went as planned, but perhaps the test has a
> race condition or something?
> {code}
> Feb 01, 2018 11:28:01 PM org.apache.beam.sdk.io.mqtt.MqttIOTest startBroker
> INFO: Finding free network port
> Feb 01, 2018 11:28:01 PM org.apache.beam.sdk.io.mqtt.MqttIOTest startBroker
> INFO: Starting ActiveMQ brokerService on 57986
> Feb 01, 2018 11:28:03 PM org.apache.activemq.broker.BrokerService
> doStartPersistenceAdapter
> INFO: Using Persistence Adapter: MemoryPersistenceAdapter
> Feb 01, 2018 11:28:04 PM org.apache.activemq.broker.BrokerService
> doStartBroker
> INFO: Apache ActiveMQ 5.13.1 (localhost,
> ID:115.98.154.104.bc.googleusercontent.com-38646-1517527683931-0:1) is
> starting
> Feb 01, 2018 11:28:04 PM
> org.apache.activemq.transport.TransportServerThreadSupport doStart
> INFO: Listening for connections at: mqtt://localhost:57986
> Feb 01, 2018 11:28:04 PM org.apache.activemq.broker.TransportConnector start
> INFO: Connector mqtt://localhost:57986 started
> Feb 01, 2018 11:28:04 PM org.apache.activemq.broker.BrokerService
> doStartBroker
> INFO: Apache ActiveMQ 5.13.1 (localhost,
> ID:115.98.154.104.bc.googleusercontent.com-38646-1517527683931-0:1) started
> Feb 01, 2018 11:28:04 PM org.apache.activemq.broker.BrokerService
> doStartBroker
> INFO: For help or more information please see: http://activemq.apache.org
> Feb 01, 2018 11:28:26 PM org.apache.activemq.broker.BrokerService stop
> INFO: Apache ActiveMQ 5.13.1 (localhost,
> ID:115.98.154.104.bc.googleusercontent.com-38646-1517527683931-0:1) is
> shutting down
> Feb 01, 2018 11:28:26 PM org.apache.activemq.broker.TransportConnector stop
> INFO: Connector mqtt://localhost:57986 stopped
> Feb 01, 2018 11:28:26 PM org.apache.activemq.broker.BrokerService stop
> INFO: Apache ActiveMQ 5.13.1 (localhost,
> ID:115.98.154.104.bc.googleusercontent.com-38646-1517527683931-0:1) uptime
> 24.039 seconds
> Feb 01, 2018 11:28:26 PM org.apache.activemq.broker.BrokerService stop
> INFO: Apache ActiveMQ 5.13.1 (localhost,
> ID:115.98.154.104.bc.googleusercontent.com-38646-1517527683931-0:1) is
> shutdown
> Feb 01, 2018 11:28:26 PM org.apache.beam.sdk.io.mqtt.MqttIOTest startBroker
> INFO: Finding free network port
> Feb 01, 2018 11:28:26 PM org.apache.beam.sdk.io.mqtt.MqttIOTest startBroker
> INFO: Starting ActiveMQ brokerService on 46799
> Feb 01, 2018 11:28:26 PM org.apache.activemq.broker.BrokerService
> doStartPersistenceAdapter
> INFO: Using Persistence Adapter: MemoryPersistenceAdapter
> Feb 01, 2018 11:28:26 PM org.apache.activemq.broker.BrokerService
> doStartBroker
> INFO: Apache ActiveMQ 5.13.1 (localhost,
> ID:115.98.154.104.bc.googleusercontent.com-38646-1517527683931-0:2) is
> starting
> Feb 01, 2018 11:28:26 PM
> org.apache.activemq.transport.TransportServerThreadSupport doStart
> INFO: Listening for connections at: mqtt://localhost:46799
> Feb 01, 2018 11:28:26 PM org.apache.activemq.broker.TransportConnector start
> INFO: Connector mqtt://localhost:46799 started
> Feb 01, 2018 11:28:26 PM org.apache.activemq.broker.BrokerService
> doStartBroker
> INFO: Apache ActiveMQ 5.13.1 (localhost,
> ID:115.98.154.104.bc.googleusercontent.com-38646-1517527683931-0:2) started
> Feb 01, 2018 11:28:26 PM org.apache.activemq.broker.BrokerService
> doStartBroker
> INFO: For help or more information please see: http://activemq.apache.org
> Feb 01, 2018 11:28:28 PM org.apache.beam.sdk.io.mqtt.MqttIOTest
> lambda$testRead$1
> INFO: Waiting pipeline connected to the MQTT broker before sending messages
> ...
> Feb 01, 2018 11:28:35 PM org.apache.activemq.broker.BrokerService stop
> INFO: Apache ActiveMQ 5.13.1 (localhost,
> ID:115.98.154.104.bc.googleusercontent.com-38646-1517527683931-0:2) is
> shutting down
> Feb 01, 2018 11
[jira] [Resolved] (BEAM-3386) Dependency conflict when Calcite is included in a project.
[ https://issues.apache.org/jira/browse/BEAM-3386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismaël Mejía resolved BEAM-3386. Fix Version/s: 2.17.0 Resolution: Fixed > Dependency conflict when Calcite is included in a project. > -- > > Key: BEAM-3386 > URL: https://issues.apache.org/jira/browse/BEAM-3386 > Project: Beam > Issue Type: Bug > Components: dsl-sql >Affects Versions: 2.2.0, 2.3.0, 2.4.0, 2.5.0, 2.6.0 >Reporter: Austin Haas >Assignee: Kai Jiang >Priority: Critical > Fix For: 2.17.0 > > > When Calcite (v. 1.13.0) is included in a project that also includes Beam and > the Beam SQL extension, then the following error is thrown when trying to run > Beam code. > ClassCastException > org.apache.beam.sdk.extensions.sql.impl.planner.BeamRelDataTypeSystem cannot > be cast to org.apache.calcite.rel.type.RelDataTypeSystem > org.apache.calcite.jdbc.CalciteConnectionImpl. > (CalciteConnectionImpl.java:120) > > org.apache.calcite.jdbc.CalciteJdbc41Factory$CalciteJdbc41Connection. > (CalciteJdbc41Factory.java:114) > org.apache.calcite.jdbc.CalciteJdbc41Factory.newConnection > (CalciteJdbc41Factory.java:59) > org.apache.calcite.jdbc.CalciteJdbc41Factory.newConnection > (CalciteJdbc41Factory.java:44) > org.apache.calcite.jdbc.CalciteFactory.newConnection > (CalciteFactory.java:53) > org.apache.calcite.avatica.UnregisteredDriver.connect > (UnregisteredDriver.java:138) > java.sql.DriverManager.getConnection (DriverManager.java:664) > java.sql.DriverManager.getConnection (DriverManager.java:208) > > org.apache.beam.sdks.java.extensions.sql.repackaged.org.apache.calcite.tools.Frameworks.withPrepare > (Frameworks.java:145) > > org.apache.beam.sdks.java.extensions.sql.repackaged.org.apache.calcite.tools.Frameworks.withPlanner > (Frameworks.java:106) > > org.apache.beam.sdks.java.extensions.sql.repackaged.org.apache.calcite.prepare.PlannerImpl.ready > (PlannerImpl.java:140) > > org.apache.beam.sdks.java.extensions.sql.repackaged.org.apache.calcite.prepare.PlannerImpl.parse > (PlannerImpl.java:170) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-3000) No python equivalent of org.apache.beam.sdk.transforms.Sample.any(100)?
[ https://issues.apache.org/jira/browse/BEAM-3000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismaël Mejía updated BEAM-3000: --- Priority: Major (was: Critical) > No python equivalent of org.apache.beam.sdk.transforms.Sample.any(100)? > --- > > Key: BEAM-3000 > URL: https://issues.apache.org/jira/browse/BEAM-3000 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Rodrigo Benenson >Priority: Major > Labels: ccoss2019, starter > > Java's org.apache.beam.sdk.transforms.Sample.any will return a PCollection > with bounded size (filtering strategy). > The closest python eqiuvalent is beam.Sample.FixedSizeGlobally(n) whover, > this version uses a combiner strategy, returning a list with n elements; > which does not scale if n is "bigger than what fits in memory". -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-3200) Streaming Pipeline throws RuntimeException when using DynamicDestinations and Method.FILE_LOADS
[
https://issues.apache.org/jira/browse/BEAM-3200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17047463#comment-17047463
]
Ismaël Mejía commented on BEAM-3200:
This is working now, isn't it? Can we resolve/close this ticket?
> Streaming Pipeline throws RuntimeException when using DynamicDestinations and
> Method.FILE_LOADS
> ---
>
> Key: BEAM-3200
> URL: https://issues.apache.org/jira/browse/BEAM-3200
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp
>Affects Versions: 2.2.0
>Reporter: AJ
>Priority: Critical
> Time Spent: 6h 50m
> Remaining Estimate: 0h
>
> I am trying to use Method.FILE_LOADS for loading data into BQ in my streaming
> pipeline using RC3 release of 2.2.0. I am writing to around 500 tables using
> DynamicDestinations and I am also using
> withCreateDisposition(CreateDisposition.CREATE_IF_NEEDED). Everything works
> fine when the first time bigquery load jobs get triggered. But on subsequent
> triggers pipeline throws a RuntimeException about table not found even though
> I created the pipeline with CreateDisposition.CREATE_IF_NEEDED. The exact
> exception is:
> {code}
> java.lang.RuntimeException: Failed to create load job with id prefix
> 717aed9ed1ef4aa7a616e1132f8b7f6d_a0928cae3d670b32f01ab2d9fe5cc0ee_1_1,
> reached max retries: 3, last failed load job: {
> "configuration" : {
> "load" : {
> "createDisposition" : "CREATE_NEVER",
> "destinationTable" : {
> "datasetId" : ...,
> "projectId" : ...,
> "tableId" :
> },
> "errors" : [ }
> "message" : "Not found: Table ,
> "reason" : "notFound"
> } ],
> {code}
> My theory is all the subsequent load jobs get trigged using CREATE_NEVER
> disposition and
> this might be due to
> https://github.com/apache/beam/blob/release-2.2.0/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteTables.java#L140
> When using DynamicDestinations all the destination tables might not be known
> during the first trigger and hence the pipeline's create disposition should
> be respected.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-3367) The Java SDK Harness should not expect WindowedValueCoders in FnApiDoFnRunner
[ https://issues.apache.org/jira/browse/BEAM-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17047467#comment-17047467 ] Ismaël Mejía commented on BEAM-3367: This is obsolete but maybe related to some discussion [~sunjincheng121] bring in recently, is this necessary or can we resolve/close this ticket? > The Java SDK Harness should not expect WindowedValueCoders in FnApiDoFnRunner > - > > Key: BEAM-3367 > URL: https://issues.apache.org/jira/browse/BEAM-3367 > Project: Beam > Issue Type: Bug > Components: sdk-java-harness >Reporter: Thomas Groh >Priority: Critical > Labels: portability > > Intermediate coders will be the type of the elements, while the windowed > value coders will be embedded within the remote read and write nodes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9392) TestStream tests are all flaky
[ https://issues.apache.org/jira/browse/BEAM-9392?focusedWorklogId=394828&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-394828 ] ASF GitHub Bot logged work on BEAM-9392: Author: ASF GitHub Bot Created on: 28/Feb/20 10:54 Start Date: 28/Feb/20 10:54 Worklog Time Spent: 10m Work Description: kamilwu commented on issue #10982: [BEAM-9392] Fix Multi TestStream assertion errors URL: https://github.com/apache/beam/pull/10982#issuecomment-592461386 Tests passed. @mwalenia could you take a look? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 394828) Time Spent: 5h (was: 4h 50m) > TestStream tests are all flaky > -- > > Key: BEAM-9392 > URL: https://issues.apache.org/jira/browse/BEAM-9392 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Pablo Estrada >Assignee: Sam Rohde >Priority: Major > Time Spent: 5h > Remaining Estimate: 0h > > See: > [https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9392) TestStream tests are all flaky
[ https://issues.apache.org/jira/browse/BEAM-9392?focusedWorklogId=394832&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-394832 ] ASF GitHub Bot logged work on BEAM-9392: Author: ASF GitHub Bot Created on: 28/Feb/20 11:03 Start Date: 28/Feb/20 11:03 Worklog Time Spent: 10m Work Description: mwalenia commented on issue #10982: [BEAM-9392] Fix Multi TestStream assertion errors URL: https://github.com/apache/beam/pull/10982#issuecomment-592464294 LGTM. Thanks for the fix! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 394832) Time Spent: 5h 10m (was: 5h) > TestStream tests are all flaky > -- > > Key: BEAM-9392 > URL: https://issues.apache.org/jira/browse/BEAM-9392 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Pablo Estrada >Assignee: Sam Rohde >Priority: Major > Time Spent: 5h 10m > Remaining Estimate: 0h > > See: > [https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9392) TestStream tests are all flaky
[ https://issues.apache.org/jira/browse/BEAM-9392?focusedWorklogId=394835&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-394835 ] ASF GitHub Bot logged work on BEAM-9392: Author: ASF GitHub Bot Created on: 28/Feb/20 11:04 Start Date: 28/Feb/20 11:04 Worklog Time Spent: 10m Work Description: mwalenia commented on pull request #10982: [BEAM-9392] Fix Multi TestStream assertion errors URL: https://github.com/apache/beam/pull/10982 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 394835) Time Spent: 5h 20m (was: 5h 10m) > TestStream tests are all flaky > -- > > Key: BEAM-9392 > URL: https://issues.apache.org/jira/browse/BEAM-9392 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Pablo Estrada >Assignee: Sam Rohde >Priority: Major > Time Spent: 5h 20m > Remaining Estimate: 0h > > See: > [https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8078) streaming_wordcount_debugging.py is missing a test
[ https://issues.apache.org/jira/browse/BEAM-8078?focusedWorklogId=394845&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-394845 ] ASF GitHub Bot logged work on BEAM-8078: Author: ASF GitHub Bot Created on: 28/Feb/20 11:35 Start Date: 28/Feb/20 11:35 Worklog Time Spent: 10m Work Description: Tesio commented on pull request #10914: [BEAM-8078] streaming_wordcount_debugging.py is missing a test URL: https://github.com/apache/beam/pull/10914#discussion_r385648453 ## File path: sdks/python/apache_beam/examples/streaming_wordcount_debugging.py ## @@ -71,16 +72,14 @@ def process( class AddTimestampFn(beam.DoFn): """A DoFn that attaches timestamps to its elements. - It takes a string of integers and it attaches to each of them - a timestamp of its same value. + It just takes an element and adds a current timestamp. - For example, [120, 225, 312] will result in: - [(120, Timestamp(120)), (225, Timestamp(225)), (312, Timestamp(312))]. + For example, Sometext will result in: + (Sometext, Timestamp(1234567890). Review comment: Fixed that as per comment This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 394845) Time Spent: 1h 50m (was: 1h 40m) > streaming_wordcount_debugging.py is missing a test > -- > > Key: BEAM-8078 > URL: https://issues.apache.org/jira/browse/BEAM-8078 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Udi Meiri >Assignee: Aleksey Vysotin >Priority: Minor > Labels: beginner, easy, newbie, starter > Time Spent: 1h 50m > Remaining Estimate: 0h > > It's example code and should have a basic_test (like the other wordcount > variants in [1]) to at least verify that it runs in the latest Beam release. > [1] > https://github.com/apache/beam/tree/master/sdks/python/apache_beam/examples -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8972) Add a Jenkins job running Combine load test on Java with Flink in Portability mode
[ https://issues.apache.org/jira/browse/BEAM-8972?focusedWorklogId=394847&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-394847 ] ASF GitHub Bot logged work on BEAM-8972: Author: ASF GitHub Bot Created on: 28/Feb/20 11:39 Start Date: 28/Feb/20 11:39 Worklog Time Spent: 10m Work Description: mwalenia commented on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-592476627 Run Load Tests Java Combine Portable Flink Batch This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 394847) Time Spent: 13h 50m (was: 13h 40m) > Add a Jenkins job running Combine load test on Java with Flink in Portability > mode > -- > > Key: BEAM-8972 > URL: https://issues.apache.org/jira/browse/BEAM-8972 > Project: Beam > Issue Type: Improvement > Components: testing >Reporter: Michał Walenia >Assignee: Michał Walenia >Priority: Minor > Time Spent: 13h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-9396) Docker image names in Jenkins jobs don't match generated ones
[ https://issues.apache.org/jira/browse/BEAM-9396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michał Walenia resolved BEAM-9396. -- Resolution: Fixed > Docker image names in Jenkins jobs don't match generated ones > - > > Key: BEAM-9396 > URL: https://issues.apache.org/jira/browse/BEAM-9396 > Project: Beam > Issue Type: Bug > Components: testing >Reporter: Michał Walenia >Assignee: Michał Walenia >Priority: Major > Fix For: Not applicable > > Time Spent: 1.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9384) Add SchemaRegistry.getSchemaCoder to get SchemaCoders for registered types
[ https://issues.apache.org/jira/browse/BEAM-9384?focusedWorklogId=394873&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-394873 ] ASF GitHub Bot logged work on BEAM-9384: Author: ASF GitHub Bot Created on: 28/Feb/20 12:54 Start Date: 28/Feb/20 12:54 Worklog Time Spent: 10m Work Description: iemejia commented on issue #10974: [BEAM-9384] Add SchemaRegistry.getSchemaCoder to get SchemaCoders for registered types URL: https://github.com/apache/beam/pull/10974#issuecomment-592500513 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 394873) Time Spent: 4h 10m (was: 4h) > Add SchemaRegistry.getSchemaCoder to get SchemaCoders for registered types > -- > > Key: BEAM-9384 > URL: https://issues.apache.org/jira/browse/BEAM-9384 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Ismaël Mejía >Assignee: Ismaël Mejía >Priority: Minor > Time Spent: 4h 10m > Remaining Estimate: 0h > > PTransforms that are parameterized by types may be able to infer a valid > SchemaCoder for a given type from the SchemaRegistry (if a Schema for the > given type is available). This method will provide a unified place to do that. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9384) Add SchemaRegistry.getSchemaCoder to get SchemaCoders for registered types
[ https://issues.apache.org/jira/browse/BEAM-9384?focusedWorklogId=394874&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-394874 ] ASF GitHub Bot logged work on BEAM-9384: Author: ASF GitHub Bot Created on: 28/Feb/20 12:55 Start Date: 28/Feb/20 12:55 Worklog Time Spent: 10m Work Description: iemejia commented on issue #10974: [BEAM-9384] Add SchemaRegistry.getSchemaCoder to get SchemaCoders for registered types URL: https://github.com/apache/beam/pull/10974#issuecomment-592500513 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 394874) Time Spent: 4h 20m (was: 4h 10m) > Add SchemaRegistry.getSchemaCoder to get SchemaCoders for registered types > -- > > Key: BEAM-9384 > URL: https://issues.apache.org/jira/browse/BEAM-9384 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Ismaël Mejía >Assignee: Ismaël Mejía >Priority: Minor > Time Spent: 4h 20m > Remaining Estimate: 0h > > PTransforms that are parameterized by types may be able to infer a valid > SchemaCoder for a given type from the SchemaRegistry (if a Schema for the > given type is available). This method will provide a unified place to do that. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9404) Add readAll() method to DynamoDBIO
Ismaël Mejía created BEAM-9404: -- Summary: Add readAll() method to DynamoDBIO Key: BEAM-9404 URL: https://issues.apache.org/jira/browse/BEAM-9404 Project: Beam Issue Type: Improvement Components: io-java-aws Reporter: Ismaël Mejía Assignee: Ismaël Mejía To be consistent with other IOs we should add the readAll() transform for DynamoDBIO. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9404) Add readAll() method to DynamoDBIO
[ https://issues.apache.org/jira/browse/BEAM-9404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismaël Mejía updated BEAM-9404: --- Status: Open (was: Triage Needed) > Add readAll() method to DynamoDBIO > -- > > Key: BEAM-9404 > URL: https://issues.apache.org/jira/browse/BEAM-9404 > Project: Beam > Issue Type: Improvement > Components: io-java-aws >Reporter: Ismaël Mejía >Assignee: Ismaël Mejía >Priority: Minor > > To be consistent with other IOs we should add the readAll() transform for > DynamoDBIO. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9405) Python PostCommit is flaky: 'PortableRunner' object has no attribute 'create_job_service'
Kamil Wasilewski created BEAM-9405:
--
Summary: Python PostCommit is flaky: 'PortableRunner' object has
no attribute 'create_job_service'
Key: BEAM-9405
URL: https://issues.apache.org/jira/browse/BEAM-9405
Project: Beam
Issue Type: Bug
Components: sdk-py-core, test-failures
Reporter: Kamil Wasilewski
Assignee: Chad Dombrova
See: [https://builds.apache.org/job/beam_PostCommit_Python2/]
It seems that it is cause by [this
|https://github.com/apache/beam/commit/1856d8533c879ab236d0593be1f9c7fff41edd7f]commit.
An example log:
{code:java}
:sdks:python:test-suites:portable:py2:crossLanguagePortableWordCount FAILED
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020.
Please upgrade your Python as Python 2.7 is no longer maintained. A future
version of pip will drop support for Python 2.7. More details about Python 2
support in pip, can be found at
https://pip.pypa.io/en/latest/development/release-process/#python-2-support
apache_beam/__init__.py:82: UserWarning: You are using Apache Beam with Python
2. New releases of Apache Beam will soon support Python 3 only.
'You are using Apache Beam with Python 2. '
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
details.
ERROR StatusLogger Log4j2 could not find a logging implementation. Please add
log4j-core to the classpath. Using SimpleLogger to log to the console...
Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python2_PR/src/sdks/python/apache_beam/examples/wordcount_xlang.py",
line 137, in
main()
File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python2_PR/src/sdks/python/apache_beam/examples/wordcount_xlang.py",
line 128, in main
p.runner.create_job_service(pipeline_options)
AttributeError: 'PortableRunner' object has no attribute 'create_job_service'
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (BEAM-9406) Convert KuduIO away from BoundedSource
Ismaël Mejía created BEAM-9406: -- Summary: Convert KuduIO away from BoundedSource Key: BEAM-9406 URL: https://issues.apache.org/jira/browse/BEAM-9406 Project: Beam Issue Type: Improvement Components: io-java-kudu Reporter: Ismaël Mejía Assignee: Ismaël Mejía Convert KuduIO to use the DoFn API instead of BoundedSource to be consistent with recent patterns of use on Beam. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9406) Convert KuduIO away from BoundedSource
[ https://issues.apache.org/jira/browse/BEAM-9406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismaël Mejía updated BEAM-9406: --- Status: Open (was: Triage Needed) > Convert KuduIO away from BoundedSource > -- > > Key: BEAM-9406 > URL: https://issues.apache.org/jira/browse/BEAM-9406 > Project: Beam > Issue Type: Improvement > Components: io-java-kudu >Reporter: Ismaël Mejía >Assignee: Ismaël Mejía >Priority: Minor > > Convert KuduIO to use the DoFn API instead of BoundedSource to be consistent > with recent patterns of use on Beam. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9407) Add readAll() to KuduIO
[ https://issues.apache.org/jira/browse/BEAM-9407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismaël Mejía updated BEAM-9407: --- Status: Open (was: Triage Needed) > Add readAll() to KuduIO > > > Key: BEAM-9407 > URL: https://issues.apache.org/jira/browse/BEAM-9407 > Project: Beam > Issue Type: Improvement > Components: io-java-kudu >Reporter: Ismaël Mejía >Assignee: Ismaël Mejía >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9407) Add readAll() to KuduIO
Ismaël Mejía created BEAM-9407: -- Summary: Add readAll() to KuduIO Key: BEAM-9407 URL: https://issues.apache.org/jira/browse/BEAM-9407 Project: Beam Issue Type: Improvement Components: io-java-kudu Reporter: Ismaël Mejía Assignee: Ismaël Mejía -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9408) Add readAll to HCatalogIO
[ https://issues.apache.org/jira/browse/BEAM-9408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismaël Mejía updated BEAM-9408: --- Status: Open (was: Triage Needed) > Add readAll to HCatalogIO > - > > Key: BEAM-9408 > URL: https://issues.apache.org/jira/browse/BEAM-9408 > Project: Beam > Issue Type: Improvement > Components: io-java-hcatalog >Reporter: Ismaël Mejía >Priority: Trivial > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9408) Add readAll() to HCatalogIO
[ https://issues.apache.org/jira/browse/BEAM-9408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismaël Mejía updated BEAM-9408: --- Summary: Add readAll() to HCatalogIO (was: Add readAll to HCatalogIO) > Add readAll() to HCatalogIO > --- > > Key: BEAM-9408 > URL: https://issues.apache.org/jira/browse/BEAM-9408 > Project: Beam > Issue Type: Improvement > Components: io-java-hcatalog >Reporter: Ismaël Mejía >Priority: Trivial > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9408) Add readAll to HCatalogIO
Ismaël Mejía created BEAM-9408: -- Summary: Add readAll to HCatalogIO Key: BEAM-9408 URL: https://issues.apache.org/jira/browse/BEAM-9408 Project: Beam Issue Type: Improvement Components: io-java-hcatalog Reporter: Ismaël Mejía -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9407) Add ReadAll transform to KuduIO
[ https://issues.apache.org/jira/browse/BEAM-9407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismaël Mejía updated BEAM-9407: --- Summary: Add ReadAll transform to KuduIO (was: Add readAll() to KuduIO ) > Add ReadAll transform to KuduIO > > > Key: BEAM-9407 > URL: https://issues.apache.org/jira/browse/BEAM-9407 > Project: Beam > Issue Type: Improvement > Components: io-java-kudu >Reporter: Ismaël Mejía >Assignee: Ismaël Mejía >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9408) Add ReadAll transform to HCatalogIO
[ https://issues.apache.org/jira/browse/BEAM-9408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismaël Mejía updated BEAM-9408: --- Summary: Add ReadAll transform to HCatalogIO (was: Add readAll() to HCatalogIO) > Add ReadAll transform to HCatalogIO > --- > > Key: BEAM-9408 > URL: https://issues.apache.org/jira/browse/BEAM-9408 > Project: Beam > Issue Type: Improvement > Components: io-java-hcatalog >Reporter: Ismaël Mejía >Priority: Trivial > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9405) Python PostCommit is flaky: 'PortableRunner' object has no attribute 'create_job_service'
[
https://issues.apache.org/jira/browse/BEAM-9405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kamil Wasilewski updated BEAM-9405:
---
Description:
See: [https://builds.apache.org/job/beam_PostCommit_Python2/]
It seems that it is caused by [this
|https://github.com/apache/beam/commit/1856d8533c879ab236d0593be1f9c7fff41edd7f]commit.
An example log:
{code:java}
:sdks:python:test-suites:portable:py2:crossLanguagePortableWordCount FAILED
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020.
Please upgrade your Python as Python 2.7 is no longer maintained. A future
version of pip will drop support for Python 2.7. More details about Python 2
support in pip, can be found at
https://pip.pypa.io/en/latest/development/release-process/#python-2-support
apache_beam/__init__.py:82: UserWarning: You are using Apache Beam with Python
2. New releases of Apache Beam will soon support Python 3 only.
'You are using Apache Beam with Python 2. '
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
details.
ERROR StatusLogger Log4j2 could not find a logging implementation. Please add
log4j-core to the classpath. Using SimpleLogger to log to the console...
Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python2_PR/src/sdks/python/apache_beam/examples/wordcount_xlang.py",
line 137, in
main()
File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python2_PR/src/sdks/python/apache_beam/examples/wordcount_xlang.py",
line 128, in main
p.runner.create_job_service(pipeline_options)
AttributeError: 'PortableRunner' object has no attribute 'create_job_service'
{code}
was:
See: [https://builds.apache.org/job/beam_PostCommit_Python2/]
It seems that it is cause by [this
|https://github.com/apache/beam/commit/1856d8533c879ab236d0593be1f9c7fff41edd7f]commit.
An example log:
{code:java}
:sdks:python:test-suites:portable:py2:crossLanguagePortableWordCount FAILED
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020.
Please upgrade your Python as Python 2.7 is no longer maintained. A future
version of pip will drop support for Python 2.7. More details about Python 2
support in pip, can be found at
https://pip.pypa.io/en/latest/development/release-process/#python-2-support
apache_beam/__init__.py:82: UserWarning: You are using Apache Beam with Python
2. New releases of Apache Beam will soon support Python 3 only.
'You are using Apache Beam with Python 2. '
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
details.
ERROR StatusLogger Log4j2 could not find a logging implementation. Please add
log4j-core to the classpath. Using SimpleLogger to log to the console...
Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python2_PR/src/sdks/python/apache_beam/examples/wordcount_xlang.py",
line 137, in
main()
File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python2_PR/src/sdks/python/apache_beam/examples/wordcount_xlang.py",
line 128, in main
p.runner.create_job_service(pipeline_options)
AttributeError: 'PortableRunner' object has no attribute 'create_job_service'
{code}
> Python PostCommit is flaky: 'PortableRunner' object has no attribute
> 'create_job_service'
> -
>
> Key: BEAM-9405
> URL: https://issues.apache.org/jira/browse/BEAM-9405
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, test-failures
>Reporter: Kamil Wasilewski
>Assignee: Chad Dombrova
>Priority: Major
>
> See: [https://builds.apache.org/job/beam_PostCommit_Python2/]
> It seems that it is caused by [this
> |https://github.com/apache/beam/commit/1856d8533c879ab236d0593be1f9c7fff41edd7f]commit.
> An example log:
> {code:java}
> :sdks:python:test-suites:portable:py2:crossLanguagePortableWordCount FAILED
> DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020.
> Please upgrade your Python as Python 2.7 is no longer maintained. A future
> version of pip will drop support for Python 2.7. More details about Python 2
[jira] [Assigned] (BEAM-7769) Add ReadAll transform for MongoDBIO
[ https://issues.apache.org/jira/browse/BEAM-7769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismaël Mejía reassigned BEAM-7769: -- Assignee: (was: Chaim) > Add ReadAll transform for MongoDBIO > --- > > Key: BEAM-7769 > URL: https://issues.apache.org/jira/browse/BEAM-7769 > Project: Beam > Issue Type: Improvement > Components: io-java-mongodb >Reporter: Ismaël Mejía >Priority: Minor > > To enable composable reads, for example to start reading data in the middle > of a Pipeline we need to offer a ReadAll transform for MongoDBIO, ideally > this should include a refactor of the current Read. For details of how this > can be done take a look at the changes in JdbcIO, RedisIO and SolrIO. > Mailing list discussion: > [https://lists.apache.org/thread.html/7bd4b9533ecfb5236969567f6436f46f4743eda04eaf32ef15fc7ee3@%3Cdev.beam.apache.org%3E] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9409) Convert MongoDBIO away from BoundedSource
[ https://issues.apache.org/jira/browse/BEAM-9409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismaël Mejía updated BEAM-9409: --- Status: Open (was: Triage Needed) > Convert MongoDBIO away from BoundedSource > - > > Key: BEAM-9409 > URL: https://issues.apache.org/jira/browse/BEAM-9409 > Project: Beam > Issue Type: Improvement > Components: io-java-mongodb >Reporter: Ismaël Mejía >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9409) Convert MongoDBIO away from BoundedSource
Ismaël Mejía created BEAM-9409: -- Summary: Convert MongoDBIO away from BoundedSource Key: BEAM-9409 URL: https://issues.apache.org/jira/browse/BEAM-9409 Project: Beam Issue Type: Improvement Components: io-java-mongodb Reporter: Ismaël Mejía -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-9404) Add readAll() method to DynamoDBIO
[ https://issues.apache.org/jira/browse/BEAM-9404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismaël Mejía reassigned BEAM-9404: -- Assignee: (was: Ismaël Mejía) > Add readAll() method to DynamoDBIO > -- > > Key: BEAM-9404 > URL: https://issues.apache.org/jira/browse/BEAM-9404 > Project: Beam > Issue Type: Improvement > Components: io-java-aws >Reporter: Ismaël Mejía >Priority: Minor > > To be consistent with other IOs we should add the readAll() transform for > DynamoDBIO. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9410) Make JdbcIO readAll consistent with other Beam readAll methods
Ismaël Mejía created BEAM-9410: -- Summary: Make JdbcIO readAll consistent with other Beam readAll methods Key: BEAM-9410 URL: https://issues.apache.org/jira/browse/BEAM-9410 Project: Beam Issue Type: Improvement Components: io-java-jdbc Reporter: Ismaël Mejía -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9410) Make JdbcIO readAll consistent with other Beam readAll methods
[ https://issues.apache.org/jira/browse/BEAM-9410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismaël Mejía updated BEAM-9410: --- Status: Open (was: Triage Needed) > Make JdbcIO readAll consistent with other Beam readAll methods > -- > > Key: BEAM-9410 > URL: https://issues.apache.org/jira/browse/BEAM-9410 > Project: Beam > Issue Type: Improvement > Components: io-java-jdbc >Reporter: Ismaël Mejía >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-9407) Add ReadAll transform to KuduIO
[ https://issues.apache.org/jira/browse/BEAM-9407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismaël Mejía reassigned BEAM-9407: -- Assignee: (was: Ismaël Mejía) > Add ReadAll transform to KuduIO > > > Key: BEAM-9407 > URL: https://issues.apache.org/jira/browse/BEAM-9407 > Project: Beam > Issue Type: Improvement > Components: io-java-kudu >Reporter: Ismaël Mejía >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-8647) Remove .mailmap from the sources
[ https://issues.apache.org/jira/browse/BEAM-8647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismaël Mejía reassigned BEAM-8647: -- Assignee: (was: Ismaël Mejía) > Remove .mailmap from the sources > > > Key: BEAM-8647 > URL: https://issues.apache.org/jira/browse/BEAM-8647 > Project: Beam > Issue Type: Task > Components: build-system >Reporter: Romain Manni-Bucau >Priority: Major > > Hi, > > .mailmap manipulates individuals data which are considered "personal" (name, > email etc) > AFAIK Apache/Beam is not allowed to do it straight, in particular for EU > citizens (_GDPR)._ > Can the file be removed since it is not used by the beam project (at least > apache/beam repo)? > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-3921) Scripting extension based on Java Scripting API (JSR-223)
[ https://issues.apache.org/jira/browse/BEAM-3921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismaël Mejía reassigned BEAM-3921: -- Assignee: (was: Ismaël Mejía) > Scripting extension based on Java Scripting API (JSR-223) > - > > Key: BEAM-3921 > URL: https://issues.apache.org/jira/browse/BEAM-3921 > Project: Beam > Issue Type: New Feature > Components: sdk-ideas >Affects Versions: 2.5.0 >Reporter: Ismaël Mejía >Priority: Minor > Time Spent: 2.5h > Remaining Estimate: 0h > > An extension with transforms that package the Java Scripting API (JSR-223) > [1] to allow users to specialize some transforms via a scripting language. It > supports ValueProviders so users can template their scripts also in Dataflow. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-673) Data locality for Read.Bounded
[ https://issues.apache.org/jira/browse/BEAM-673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17047688#comment-17047688 ] Ismaël Mejía commented on BEAM-673: --- Now that we have moved into a DoFn world with (or without) SplittableDoFn I wonder if we should change this issue to be consistent better with he ParDo API. Have something similar have been proposed as part of the ongoing work on portability? A mechanism for transforms to suggest translations to the runner? [~robertwb] [~lcwik] ? > Data locality for Read.Bounded > -- > > Key: BEAM-673 > URL: https://issues.apache.org/jira/browse/BEAM-673 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Amit Sela >Assignee: Ismaël Mejía >Priority: Major > > In some distributed filesystems, such as HDFS, we should be able to hint to > Spark the preferred locations of splits. > Here is an example of how Spark does that for Hadoop RDDs: > https://github.com/apache/spark/blob/branch-1.6/core/src/main/scala/org/apache/spark/rdd/NewHadoopRDD.scala#L249 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (BEAM-673) Data locality for Read.Bounded
[ https://issues.apache.org/jira/browse/BEAM-673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17047688#comment-17047688 ] Ismaël Mejía edited comment on BEAM-673 at 2/28/20 2:29 PM: Now that we have moved into a DoFn world with (or without) SplittableDoFn I wonder if we should change this issue to be consistent better with he ParDo API. Have something similar have been proposed as part of the ongoing work on portability? A mechanism for transforms to suggest translations to the runner as proposed in BEAM-2085 ? [~kenn] [~robertwb] [~lcwik] ? was (Author: iemejia): Now that we have moved into a DoFn world with (or without) SplittableDoFn I wonder if we should change this issue to be consistent better with he ParDo API. Have something similar have been proposed as part of the ongoing work on portability? A mechanism for transforms to suggest translations to the runner as proposed in BEAM-2085 ? [~robertwb] [~lcwik] ? > Data locality for Read.Bounded > -- > > Key: BEAM-673 > URL: https://issues.apache.org/jira/browse/BEAM-673 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Amit Sela >Assignee: Ismaël Mejía >Priority: Major > > In some distributed filesystems, such as HDFS, we should be able to hint to > Spark the preferred locations of splits. > Here is an example of how Spark does that for Hadoop RDDs: > https://github.com/apache/spark/blob/branch-1.6/core/src/main/scala/org/apache/spark/rdd/NewHadoopRDD.scala#L249 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (BEAM-673) Data locality for Read.Bounded
[ https://issues.apache.org/jira/browse/BEAM-673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17047688#comment-17047688 ] Ismaël Mejía edited comment on BEAM-673 at 2/28/20 2:29 PM: Now that we have moved into a DoFn world with (or without) SplittableDoFn I wonder if we should change this issue to be consistent better with he ParDo API. Have something similar have been proposed as part of the ongoing work on portability? A mechanism for transforms to suggest translations to the runner as proposed in BEAM-2085 ? [~robertwb] [~lcwik] ? was (Author: iemejia): Now that we have moved into a DoFn world with (or without) SplittableDoFn I wonder if we should change this issue to be consistent better with he ParDo API. Have something similar have been proposed as part of the ongoing work on portability? A mechanism for transforms to suggest translations to the runner? [~robertwb] [~lcwik] ? > Data locality for Read.Bounded > -- > > Key: BEAM-673 > URL: https://issues.apache.org/jira/browse/BEAM-673 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Amit Sela >Assignee: Ismaël Mejía >Priority: Major > > In some distributed filesystems, such as HDFS, we should be able to hint to > Spark the preferred locations of splits. > Here is an example of how Spark does that for Hadoop RDDs: > https://github.com/apache/spark/blob/branch-1.6/core/src/main/scala/org/apache/spark/rdd/NewHadoopRDD.scala#L249 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9384) Add SchemaRegistry.getSchemaCoder to get SchemaCoders for registered types
[ https://issues.apache.org/jira/browse/BEAM-9384?focusedWorklogId=394932&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-394932 ] ASF GitHub Bot logged work on BEAM-9384: Author: ASF GitHub Bot Created on: 28/Feb/20 15:14 Start Date: 28/Feb/20 15:14 Worklog Time Spent: 10m Work Description: iemejia commented on issue #10974: [BEAM-9384] Add SchemaRegistry.getSchemaCoder to get SchemaCoders for registered types URL: https://github.com/apache/beam/pull/10974#issuecomment-592556811 Finally the tests are green. Can you PTAL @reuvenlax . This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 394932) Time Spent: 4.5h (was: 4h 20m) > Add SchemaRegistry.getSchemaCoder to get SchemaCoders for registered types > -- > > Key: BEAM-9384 > URL: https://issues.apache.org/jira/browse/BEAM-9384 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Ismaël Mejía >Assignee: Ismaël Mejía >Priority: Minor > Time Spent: 4.5h > Remaining Estimate: 0h > > PTransforms that are parameterized by types may be able to infer a valid > SchemaCoder for a given type from the SchemaRegistry (if a Schema for the > given type is available). This method will provide a unified place to do that. -- This message was sent by Atlassian Jira (v8.3.4#803005)
