[jira] [Commented] (BEAM-9239) Dependency conflict with Spark using aws io
[
https://issues.apache.org/jira/browse/BEAM-9239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109347#comment-17109347
]
David McIntosh commented on BEAM-9239:
--
I no longer have access to the project that had this issue but I remember that
not working. There was some sort of conflict with the spark libs so either the
driver would fail to start or it would fail when beam was submitting its job.
Sorry I can't capture any more details on this issue!
> Dependency conflict with Spark using aws io
> ---
>
> Key: BEAM-9239
> URL: https://issues.apache.org/jira/browse/BEAM-9239
> Project: Beam
> Issue Type: Bug
> Components: io-java-aws, runner-spark
>Affects Versions: 2.17.0
>Reporter: David McIntosh
>Priority: P1
>
> Starting with beam 2.17.0 I get this error in the Spark 2.4.4 driver when aws
> io is also used:
> {noformat}
> java.lang.NoSuchMethodError:
> com.fasterxml.jackson.databind.jsontype.TypeSerializer.typeId(Ljava/lang/Object;Lcom/fasterxml/jackson/core/JsonToken;)Lcom/fasterxml/jackson/core/type/WritableTypeId;
> at
> org.apache.beam.sdk.io.aws.options.AwsModule$AWSCredentialsProviderSerializer.serializeWithType(AwsModule.java:163)
> at
> org.apache.beam.sdk.io.aws.options.AwsModule$AWSCredentialsProviderSerializer.serializeWithType(AwsModule.java:134)
> at
> com.fasterxml.jackson.databind.ser.impl.TypeWrappedSerializer.serialize(TypeWrappedSerializer.java:32)
> at
> com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:130)
> at
> com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3559)
> at
> com.fasterxml.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:2927)
> at
> org.apache.beam.sdk.options.ProxyInvocationHandler$Serializer.ensureSerializable(ProxyInvocationHandler.java:721)
> at
> org.apache.beam.sdk.options.ProxyInvocationHandler$Serializer.serialize(ProxyInvocationHandler.java:647)
> at
> org.apache.beam.sdk.options.ProxyInvocationHandler$Serializer.serialize(ProxyInvocationHandler.java:635)
> at
> com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:130)
> at
> com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3559)
> at
> com.fasterxml.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:2927)
> at
> org.apache.beam.runners.core.construction.SerializablePipelineOptions.serializeToJson(SerializablePipelineOptions.java:67)
> at
> org.apache.beam.runners.core.construction.SerializablePipelineOptions.(SerializablePipelineOptions.java:43)
> at
> org.apache.beam.runners.spark.translation.EvaluationContext.(EvaluationContext.java:71)
> at org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:215)
> at org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:90)
> {noformat}
> The cause seems to be that the Spark driver environment uses an older version
> of Jackson. I tried to update jackson on the Spark cluster but that led to
> several other errors.
> The change that started causing this was:
> https://github.com/apache/beam/commit/b68d70a47b68ad84efcd9405c1799002739bd116
> After reverting that change I was able to successfully run my job.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (BEAM-9980) Specific Python version tied tests are to be version-selective
[ https://issues.apache.org/jira/browse/BEAM-9980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yoshiki obata reassigned BEAM-9980: --- Assignee: yoshiki obata > Specific Python version tied tests are to be version-selective > -- > > Key: BEAM-9980 > URL: https://issues.apache.org/jira/browse/BEAM-9980 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: yoshiki obata >Assignee: yoshiki obata >Priority: P2 > > We have some tests that we run only under specific Python versions, > - FlinkValidatesRunner test runs using Python 3.5 > - HDFS Python 3 tests are running only with Python 3.7 > - Cross-language Py3 tests for Spark are running under Python 3.5 > etc... > Python versions to run them under need to be switchable. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-2939) Fn API SDF support
[ https://issues.apache.org/jira/browse/BEAM-2939?focusedWorklogId=434130&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-434130 ] ASF GitHub Bot logged work on BEAM-2939: Author: ASF GitHub Bot Created on: 17/May/20 02:02 Start Date: 17/May/20 02:02 Worklog Time Spent: 10m Work Description: lukecwik merged pull request #11735: URL: https://github.com/apache/beam/pull/11735 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 434130) Time Spent: 33h 20m (was: 33h 10m) > Fn API SDF support > -- > > Key: BEAM-2939 > URL: https://issues.apache.org/jira/browse/BEAM-2939 > Project: Beam > Issue Type: Improvement > Components: beam-model >Reporter: Henning Rohde >Assignee: Luke Cwik >Priority: P2 > Labels: portability > Time Spent: 33h 20m > Remaining Estimate: 0h > > The Fn API should support streaming SDF. Detailed design TBD. > Once design is ready, expand subtasks similarly to BEAM-2822. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-2939) Fn API SDF support
[ https://issues.apache.org/jira/browse/BEAM-2939?focusedWorklogId=434120&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-434120 ] ASF GitHub Bot logged work on BEAM-2939: Author: ASF GitHub Bot Created on: 16/May/20 23:31 Start Date: 16/May/20 23:31 Worklog Time Spent: 10m Work Description: chamikaramj commented on pull request #11735: URL: https://github.com/apache/beam/pull/11735#issuecomment-629719364 LGTM. Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 434120) Time Spent: 33h 10m (was: 33h) > Fn API SDF support > -- > > Key: BEAM-2939 > URL: https://issues.apache.org/jira/browse/BEAM-2939 > Project: Beam > Issue Type: Improvement > Components: beam-model >Reporter: Henning Rohde >Assignee: Luke Cwik >Priority: P2 > Labels: portability > Time Spent: 33h 10m > Remaining Estimate: 0h > > The Fn API should support streaming SDF. Detailed design TBD. > Once design is ready, expand subtasks similarly to BEAM-2822. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-2939) Fn API SDF support
[ https://issues.apache.org/jira/browse/BEAM-2939?focusedWorklogId=434116&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-434116 ] ASF GitHub Bot logged work on BEAM-2939: Author: ASF GitHub Bot Created on: 16/May/20 22:01 Start Date: 16/May/20 22:01 Worklog Time Spent: 10m Work Description: lukecwik commented on pull request #11735: URL: https://github.com/apache/beam/pull/11735#issuecomment-629711625 R: @chamikaramj @ihji This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 434116) Time Spent: 33h (was: 32h 50m) > Fn API SDF support > -- > > Key: BEAM-2939 > URL: https://issues.apache.org/jira/browse/BEAM-2939 > Project: Beam > Issue Type: Improvement > Components: beam-model >Reporter: Henning Rohde >Assignee: Luke Cwik >Priority: P2 > Labels: portability > Time Spent: 33h > Remaining Estimate: 0h > > The Fn API should support streaming SDF. Detailed design TBD. > Once design is ready, expand subtasks similarly to BEAM-2822. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-2939) Fn API SDF support
[
https://issues.apache.org/jira/browse/BEAM-2939?focusedWorklogId=434115&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-434115
]
ASF GitHub Bot logged work on BEAM-2939:
Author: ASF GitHub Bot
Created on: 16/May/20 22:01
Start Date: 16/May/20 22:01
Worklog Time Spent: 10m
Work Description: lukecwik commented on a change in pull request #11735:
URL: https://github.com/apache/beam/pull/11735#discussion_r426195050
##
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/io/Read.java
##
@@ -487,51 +493,54 @@ public void splitRestriction(
@NewTracker
public RestrictionTracker<
-KV, CheckpointT>,
UnboundedSourceValue[]>
+UnboundedSourceRestriction,
UnboundedSourceValue[]>
restrictionTracker(
-@Restriction KV,
CheckpointT> restriction,
+@Restriction UnboundedSourceRestriction
restriction,
PipelineOptions pipelineOptions) {
return new UnboundedSourceAsSDFRestrictionTracker(restriction,
pipelineOptions);
}
@ProcessElement
public ProcessContinuation processElement(
-RestrictionTracker<
-KV, CheckpointT>,
UnboundedSourceValue[]>
+RestrictionTracker,
UnboundedSourceValue[]>
tracker,
ManualWatermarkEstimator watermarkEstimator,
OutputReceiver> receiver,
BundleFinalizer bundleFinalizer)
throws IOException {
- KV, CheckpointT>
initialRestriction =
+ UnboundedSourceRestriction initialRestriction =
tracker.currentRestriction();
UnboundedSourceValue[] out = new UnboundedSourceValue[1];
while (tracker.tryClaim(out)) {
receiver.outputWithTimestamp(
new ValueWithRecordId<>(out[0].getValue(), out[0].getId()),
out[0].getTimestamp());
-
watermarkEstimator.setWatermark(ensureTimestampWithinBounds(out[0].getWatermark()));
}
+ UnboundedSourceRestriction currentRestriction =
+ tracker.currentRestriction();
+
+ // Advance the watermark even if zero elements may have been output.
Review comment:
This is the key part of the change here, moving
`watermarkEstimator.setWatermark` out of the while loop.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 434115)
Time Spent: 32h 50m (was: 32h 40m)
> Fn API SDF support
> --
>
> Key: BEAM-2939
> URL: https://issues.apache.org/jira/browse/BEAM-2939
> Project: Beam
> Issue Type: Improvement
> Components: beam-model
>Reporter: Henning Rohde
>Assignee: Luke Cwik
>Priority: P2
> Labels: portability
> Time Spent: 32h 50m
> Remaining Estimate: 0h
>
> The Fn API should support streaming SDF. Detailed design TBD.
> Once design is ready, expand subtasks similarly to BEAM-2822.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-2939) Fn API SDF support
[ https://issues.apache.org/jira/browse/BEAM-2939?focusedWorklogId=434114&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-434114 ] ASF GitHub Bot logged work on BEAM-2939: Author: ASF GitHub Bot Created on: 16/May/20 22:00 Start Date: 16/May/20 22:00 Worklog Time Spent: 10m Work Description: lukecwik opened a new pull request #11735: URL: https://github.com/apache/beam/pull/11735 Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[ Add batch SnowflakeIO.Read to Java SDK
[
https://issues.apache.org/jira/browse/BEAM-9722?focusedWorklogId=434112&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-434112
]
ASF GitHub Bot logged work on BEAM-9722:
Author: ASF GitHub Bot
Created on: 16/May/20 21:21
Start Date: 16/May/20 21:21
Worklog Time Spent: 10m
Work Description: chamikaramj commented on a change in pull request
#11360:
URL: https://github.com/apache/beam/pull/11360#discussion_r426190761
##
File path:
sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/SnowflakeIO.java
##
@@ -0,0 +1,735 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.snowflake;
+
+import static org.apache.beam.sdk.io.TextIO.readFiles;
+import static
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+
+import com.google.auto.value.AutoValue;
+import com.opencsv.CSVParser;
+import com.opencsv.CSVParserBuilder;
+import java.io.IOException;
+import java.io.Serializable;
+import java.security.PrivateKey;
+import java.sql.Connection;
+import java.sql.SQLException;
+import java.text.SimpleDateFormat;
+import java.util.Date;
+import java.util.List;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import javax.sql.DataSource;
+import net.snowflake.client.jdbc.SnowflakeBasicDataSource;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.io.FileIO;
+import org.apache.beam.sdk.io.FileSystems;
+import org.apache.beam.sdk.io.fs.ResourceId;
+import
org.apache.beam.sdk.io.snowflake.credentials.KeyPairSnowflakeCredentials;
+import
org.apache.beam.sdk.io.snowflake.credentials.OAuthTokenSnowflakeCredentials;
+import org.apache.beam.sdk.io.snowflake.credentials.SnowflakeCredentials;
+import
org.apache.beam.sdk.io.snowflake.credentials.UsernamePasswordSnowflakeCredentials;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.sdk.transforms.Wait;
+import org.apache.beam.sdk.transforms.display.DisplayData;
+import org.apache.beam.sdk.transforms.display.HasDisplayData;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * IO to read and write data on Snowflake.
+ *
+ * SnowflakeIO uses https://docs.snowflake.net/manuals/user-guide/jdbc.html";>Snowflake
+ * JDBC driver under the hood, but data isn't read/written using JDBC
directly. Instead,
+ * SnowflakeIO uses dedicated COPY operations to read/write data
from/to a cloud bucket. By
+ * now only Google Cloud Storage is supported.
+ *
+ * To configure SnowflakeIO to read/write from your Snowflake instance, you
have to provide a
+ * {@link DataSourceConfiguration} using {@link
+ * DataSourceConfiguration#create(SnowflakeCredentials)}, where {@link
SnowflakeCredentials might be
+ * created using {@link
org.apache.beam.sdk.io.snowflake.credentials.SnowflakeCredentialsFactory}}.
+ * Additionally one of {@link DataSourceConfiguration#withServerName(String)}
or {@link
+ * DataSourceConfiguration#withUrl(String)} must be used to tell SnowflakeIO
which instance to use.
+ *
+ * There are also other options available to configure connection to Snowflake:
+ *
+ *
+ * {@link DataSourceConfiguration#withWarehouse(String)} to specify
which Warehouse to use
+ * {@link DataSourceConfiguration#withDatabase(String)} to specify which
Database to connect
+ * to
+ * {@link DataSourceConfiguration#withSchema(String)} to specify which
schema to use
+ * {@link DataSourceConfiguration#withRole(String)} to specify which
role to use
+ * {@link DataSourceConfiguration#withLoginTimeout(Integer)} to specify
the timeout for the
+ * login
+ * {@link DataSourceConfiguration#withPortNumber(Integer)} to specify
custom port of Snowflake
+ *
[jira] [Work logged] (BEAM-9679) Core Transforms | Go SDK Code Katas
[ https://issues.apache.org/jira/browse/BEAM-9679?focusedWorklogId=434108&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-434108 ] ASF GitHub Bot logged work on BEAM-9679: Author: ASF GitHub Bot Created on: 16/May/20 19:24 Start Date: 16/May/20 19:24 Worklog Time Spent: 10m Work Description: damondouglas opened a new pull request #11734: URL: https://github.com/apache/beam/pull/11734 This pull request adds a GroupByKey lesson to the Go SDK katas. I would like to request the following reviewers: (R: @lostluck ) (R: @henryken ) If accepted, please wait until the [Stepik course](https://stepik.org/course/70387) is updated before finally merging this PR. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [x] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[
[jira] [Updated] (BEAM-9745) [beam_PostCommit_Java_PortabilityApi] Various GCP IO tests failing, unable to deserialize Custom DoFns and Custom Coders.
[
https://issues.apache.org/jira/browse/BEAM-9745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chamikara Madhusanka Jayalath updated BEAM-9745:
Fix Version/s: 2.22.0
> [beam_PostCommit_Java_PortabilityApi] Various GCP IO tests failing, unable to
> deserialize Custom DoFns and Custom Coders.
> -
>
> Key: BEAM-9745
> URL: https://issues.apache.org/jira/browse/BEAM-9745
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp, java-fn-execution, sdk-java-harness,
> test-failures
>Reporter: Daniel Oliveira
>Assignee: Kenneth Knowles
>Priority: P0
> Labels: currently-failing
> Fix For: 2.22.0
>
>
> _Use this form to file an issue for test failure:_
> * [Jenkins
> Job|https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi/4657/]
> * [Gradle Build
> Scan|https://scans.gradle.com/s/c3izncsa4u24k/tests/by-project]
> Initial investigation:
> The bug appears to be popping up on BigQuery tests mostly, but also a
> BigTable and a Datastore test.
> Here's an example stacktrace of the two errors, showing _only_ the error
> messages themselves. Source:
> [https://scans.gradle.com/s/c3izncsa4u24k/tests/efn4wciuamvqq-ccxt3jvofvqbe]
> {noformat}
> java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error
> received from SDK harness for instruction -191:
> java.lang.IllegalArgumentException: unable to deserialize Custom DoFn With
> Execution Info
> ...
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.sdk.io.gcp.bigquery.BatchLoads$3
> ...
> Caused by: java.lang.RuntimeException: Error received from SDK harness for
> instruction -191: java.lang.IllegalArgumentException: unable to deserialize
> Custom DoFn With Execution Info
> ...
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.sdk.io.gcp.bigquery.BatchLoads$3
> java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error
> received from SDK harness for instruction -206:
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
>
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
> ...
> Caused by:
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
> ...
> Caused by: java.lang.IllegalArgumentException: unable to deserialize Custom
> Coder Bytes
> ...
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.sdk.io.gcp.bigquery.TableRowJsonCoder
> ...
> Caused by: java.lang.RuntimeException: Error received from SDK harness for
> instruction -206:
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
>
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
> ...
> Caused by:
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
> ...
> Caused by: java.lang.IllegalArgumentException: unable to deserialize Custom
> Coder Bytes
> ...
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.sdk.io.gcp.bigquery.TableRowJsonCoder
> ...
> {noformat}
> Update: Looks like this has been failing as far back as [Apr
> 4|https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi/4566/]
> after a long period where the test was consistently timing out since [Mar
> 31|https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi/4546/].
> So it's hard to narrow down what commit may have caused this. Plus, the test
> was failing due to a completely different BigQuery failure before anyway, so
> it seems like this test will need to be completely fixed from scratch,
> instead of tracking down a specific breaking change.
>
> _After you've filled out the above details, please [assign the issue to an
> individual|https://beam.apache.org/contribute/postcommits-guides/index.html#find_specialist].
> Assignee should [treat test failures as
> high-priority|https://beam.apache.org/contribute/postcommits-policies/#assigned-failing-test],
> helping to fix the issue or find a more appropriate owner. See [Apache Beam
> Post-Commit
> Policies|https://beam.apache.org/contribute/postcommits-policies]._
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (BEAM-9745) [beam_PostCommit_Java_PortabilityApi] Various GCP IO tests failing, unable to deserialize Custom DoFns and Custom Coders.
[
https://issues.apache.org/jira/browse/BEAM-9745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109176#comment-17109176
]
Chamikara Madhusanka Jayalath edited comment on BEAM-9745 at 5/16/20, 7:04 PM:
---
I noticed that we add harness jar to the container in addition to jars received
through the retrieval service.
[https://github.com/apache/beam/blob/master/sdks/java/container/boot.go#L122]
Could it be that we end up with conflicting versions of the same set of classes
due to this ?
was (Author: chamikara):
I noticed that we add harness jar to the container in addition to jars received
through the retrieval service.
[https://github.com/apache/beam/blob/master/sdks/java/container/boot.go#L122]
Could it be that we end up with conflicting versions of the same set of classes
for Dataflow due to this ?
> [beam_PostCommit_Java_PortabilityApi] Various GCP IO tests failing, unable to
> deserialize Custom DoFns and Custom Coders.
> -
>
> Key: BEAM-9745
> URL: https://issues.apache.org/jira/browse/BEAM-9745
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp, java-fn-execution, sdk-java-harness,
> test-failures
>Reporter: Daniel Oliveira
>Assignee: Kenneth Knowles
>Priority: P0
> Labels: currently-failing
>
> _Use this form to file an issue for test failure:_
> * [Jenkins
> Job|https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi/4657/]
> * [Gradle Build
> Scan|https://scans.gradle.com/s/c3izncsa4u24k/tests/by-project]
> Initial investigation:
> The bug appears to be popping up on BigQuery tests mostly, but also a
> BigTable and a Datastore test.
> Here's an example stacktrace of the two errors, showing _only_ the error
> messages themselves. Source:
> [https://scans.gradle.com/s/c3izncsa4u24k/tests/efn4wciuamvqq-ccxt3jvofvqbe]
> {noformat}
> java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error
> received from SDK harness for instruction -191:
> java.lang.IllegalArgumentException: unable to deserialize Custom DoFn With
> Execution Info
> ...
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.sdk.io.gcp.bigquery.BatchLoads$3
> ...
> Caused by: java.lang.RuntimeException: Error received from SDK harness for
> instruction -191: java.lang.IllegalArgumentException: unable to deserialize
> Custom DoFn With Execution Info
> ...
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.sdk.io.gcp.bigquery.BatchLoads$3
> java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error
> received from SDK harness for instruction -206:
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
>
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
> ...
> Caused by:
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
> ...
> Caused by: java.lang.IllegalArgumentException: unable to deserialize Custom
> Coder Bytes
> ...
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.sdk.io.gcp.bigquery.TableRowJsonCoder
> ...
> Caused by: java.lang.RuntimeException: Error received from SDK harness for
> instruction -206:
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
>
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
> ...
> Caused by:
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
> ...
> Caused by: java.lang.IllegalArgumentException: unable to deserialize Custom
> Coder Bytes
> ...
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.sdk.io.gcp.bigquery.TableRowJsonCoder
> ...
> {noformat}
> Update: Looks like this has been failing as far back as [Apr
> 4|https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi/4566/]
> after a long period where the test was consistently timing out since [Mar
> 31|https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi/4546/].
> So it's hard to narrow down what commit may have caused this. Plus, the test
> was failing due to a completely different BigQuery failure before anyway, so
> it seems like this test will need to be completely fixed from scratch,
> i
[jira] [Commented] (BEAM-9745) [beam_PostCommit_Java_PortabilityApi] Various GCP IO tests failing, unable to deserialize Custom DoFns and Custom Coders.
[
https://issues.apache.org/jira/browse/BEAM-9745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109176#comment-17109176
]
Chamikara Madhusanka Jayalath commented on BEAM-9745:
-
I noticed that we add harness jar to the container in addition to jars received
through the retrieval service.
[https://github.com/apache/beam/blob/master/sdks/java/container/boot.go#L122]
Could it be that we end up with conflicting versions of the same set of classes
for Dataflow due to this ?
> [beam_PostCommit_Java_PortabilityApi] Various GCP IO tests failing, unable to
> deserialize Custom DoFns and Custom Coders.
> -
>
> Key: BEAM-9745
> URL: https://issues.apache.org/jira/browse/BEAM-9745
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp, java-fn-execution, sdk-java-harness,
> test-failures
>Reporter: Daniel Oliveira
>Assignee: Kenneth Knowles
>Priority: P0
> Labels: currently-failing
>
> _Use this form to file an issue for test failure:_
> * [Jenkins
> Job|https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi/4657/]
> * [Gradle Build
> Scan|https://scans.gradle.com/s/c3izncsa4u24k/tests/by-project]
> Initial investigation:
> The bug appears to be popping up on BigQuery tests mostly, but also a
> BigTable and a Datastore test.
> Here's an example stacktrace of the two errors, showing _only_ the error
> messages themselves. Source:
> [https://scans.gradle.com/s/c3izncsa4u24k/tests/efn4wciuamvqq-ccxt3jvofvqbe]
> {noformat}
> java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error
> received from SDK harness for instruction -191:
> java.lang.IllegalArgumentException: unable to deserialize Custom DoFn With
> Execution Info
> ...
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.sdk.io.gcp.bigquery.BatchLoads$3
> ...
> Caused by: java.lang.RuntimeException: Error received from SDK harness for
> instruction -191: java.lang.IllegalArgumentException: unable to deserialize
> Custom DoFn With Execution Info
> ...
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.sdk.io.gcp.bigquery.BatchLoads$3
> java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error
> received from SDK harness for instruction -206:
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
>
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
> ...
> Caused by:
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
> ...
> Caused by: java.lang.IllegalArgumentException: unable to deserialize Custom
> Coder Bytes
> ...
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.sdk.io.gcp.bigquery.TableRowJsonCoder
> ...
> Caused by: java.lang.RuntimeException: Error received from SDK harness for
> instruction -206:
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
>
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
> ...
> Caused by:
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
> ...
> Caused by: java.lang.IllegalArgumentException: unable to deserialize Custom
> Coder Bytes
> ...
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.sdk.io.gcp.bigquery.TableRowJsonCoder
> ...
> {noformat}
> Update: Looks like this has been failing as far back as [Apr
> 4|https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi/4566/]
> after a long period where the test was consistently timing out since [Mar
> 31|https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi/4546/].
> So it's hard to narrow down what commit may have caused this. Plus, the test
> was failing due to a completely different BigQuery failure before anyway, so
> it seems like this test will need to be completely fixed from scratch,
> instead of tracking down a specific breaking change.
>
> _After you've filled out the above details, please [assign the issue to an
> individual|https://beam.apache.org/contribute/postcommits-guides/index.html#find_specialist].
> Assignee should [treat test failures as
> high-priority|https://beam.apache.org/contribute/postcommits-policies/#assigned-failing-t
[jira] [Work logged] (BEAM-10017) Expose SocketOptions timeouts in CassandraIO builder
[
https://issues.apache.org/jira/browse/BEAM-10017?focusedWorklogId=434102&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-434102
]
ASF GitHub Bot logged work on BEAM-10017:
-
Author: ASF GitHub Bot
Created on: 16/May/20 18:30
Start Date: 16/May/20 18:30
Worklog Time Spent: 10m
Work Description: nfisher commented on pull request #11732:
URL: https://github.com/apache/beam/pull/11732#issuecomment-629688169
R: @jbonofre
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 434102)
Time Spent: 50m (was: 40m)
> Expose SocketOptions timeouts in CassandraIO builder
>
>
> Key: BEAM-10017
> URL: https://issues.apache.org/jira/browse/BEAM-10017
> Project: Beam
> Issue Type: Improvement
> Components: io-java-cassandra
>Reporter: Nathan Fisher
>Priority: P3
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Currently there are no options to tune the configuration of the CassandraIO
> reader/writer. This can be useful for either slow clusters, large queries, or
> high latency links.
> The intent would be to expose the following configuration elements as setters
> on the CassandraIO builder similar to withKeyspace and other methods.
>
> |{{[SocketOptions|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html]}}|{{[setConnectTimeoutMillis|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html#setConnectTimeoutMillis-int-](int
> connectTimeoutMillis)}}
> Sets the connection timeout in milliseconds.|
> |{{[SocketOptions|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html]}}|{{[setKeepAlive|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html#setKeepAlive-boolean-](boolean
> keepAlive)}}
> Sets whether to enable TCP keepalive.|
> |{{[SocketOptions|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html]}}|{{[setReadTimeoutMillis|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html#setReadTimeoutMillis-int-](int
> readTimeoutMillis)}}
> Sets the per-host read timeout in milliseconds.|
> |{{[SocketOptions|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html]}}|{{[setReceiveBufferSize|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html#setReceiveBufferSize-int-](int
> receiveBufferSize)}}
> Sets a hint to the size of the underlying buffers for incoming network I/O.|
> |{{[SocketOptions|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html]}}|{{[setReuseAddress|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html#setReuseAddress-boolean-](boolean
> reuseAddress)}}
> Sets whether to enable reuse-address.|
> |{{[SocketOptions|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html]}}|{{[setSendBufferSize|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html#setSendBufferSize-int-](int
> sendBufferSize)}}
> Sets a hint to the size of the underlying buffers for outgoing network I/O.|
> |{{[SocketOptions|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html]}}|{{[setSoLinger|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html#setSoLinger-int-](int
> soLinger)}}
> Sets the linger-on-close timeout.|
> |{{[SocketOptions|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html]}}|{{[setTcpNoDelay|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html#setTcpNoDelay-boolean-](boolean
> tcpNoDelay)}}
> Sets whether to disable Nagle's algorithm.|
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-10017) Expose SocketOptions timeouts in CassandraIO builder
[
https://issues.apache.org/jira/browse/BEAM-10017?focusedWorklogId=434100&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-434100
]
ASF GitHub Bot logged work on BEAM-10017:
-
Author: ASF GitHub Bot
Created on: 16/May/20 18:29
Start Date: 16/May/20 18:29
Worklog Time Spent: 10m
Work Description: nfisher edited a comment on pull request #11732:
URL: https://github.com/apache/beam/pull/11732#issuecomment-629687704
Not sure if exposing all of the configuration parameters is desirable. Let
me know what you think.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 434100)
Time Spent: 0.5h (was: 20m)
> Expose SocketOptions timeouts in CassandraIO builder
>
>
> Key: BEAM-10017
> URL: https://issues.apache.org/jira/browse/BEAM-10017
> Project: Beam
> Issue Type: Improvement
> Components: io-java-cassandra
>Reporter: Nathan Fisher
>Priority: P3
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Currently there are no options to tune the configuration of the CassandraIO
> reader/writer. This can be useful for either slow clusters, large queries, or
> high latency links.
> The intent would be to expose the following configuration elements as setters
> on the CassandraIO builder similar to withKeyspace and other methods.
>
> |{{[SocketOptions|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html]}}|{{[setConnectTimeoutMillis|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html#setConnectTimeoutMillis-int-](int
> connectTimeoutMillis)}}
> Sets the connection timeout in milliseconds.|
> |{{[SocketOptions|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html]}}|{{[setKeepAlive|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html#setKeepAlive-boolean-](boolean
> keepAlive)}}
> Sets whether to enable TCP keepalive.|
> |{{[SocketOptions|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html]}}|{{[setReadTimeoutMillis|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html#setReadTimeoutMillis-int-](int
> readTimeoutMillis)}}
> Sets the per-host read timeout in milliseconds.|
> |{{[SocketOptions|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html]}}|{{[setReceiveBufferSize|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html#setReceiveBufferSize-int-](int
> receiveBufferSize)}}
> Sets a hint to the size of the underlying buffers for incoming network I/O.|
> |{{[SocketOptions|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html]}}|{{[setReuseAddress|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html#setReuseAddress-boolean-](boolean
> reuseAddress)}}
> Sets whether to enable reuse-address.|
> |{{[SocketOptions|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html]}}|{{[setSendBufferSize|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html#setSendBufferSize-int-](int
> sendBufferSize)}}
> Sets a hint to the size of the underlying buffers for outgoing network I/O.|
> |{{[SocketOptions|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html]}}|{{[setSoLinger|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html#setSoLinger-int-](int
> soLinger)}}
> Sets the linger-on-close timeout.|
> |{{[SocketOptions|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html]}}|{{[setTcpNoDelay|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html#setTcpNoDelay-boolean-](boolean
> tcpNoDelay)}}
> Sets whether to disable Nagle's algorithm.|
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-10017) Expose SocketOptions timeouts in CassandraIO builder
[
https://issues.apache.org/jira/browse/BEAM-10017?focusedWorklogId=434101&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-434101
]
ASF GitHub Bot logged work on BEAM-10017:
-
Author: ASF GitHub Bot
Created on: 16/May/20 18:29
Start Date: 16/May/20 18:29
Worklog Time Spent: 10m
Work Description: nfisher commented on pull request #11732:
URL: https://github.com/apache/beam/pull/11732#issuecomment-629688041
R: @echauchot
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 434101)
Time Spent: 40m (was: 0.5h)
> Expose SocketOptions timeouts in CassandraIO builder
>
>
> Key: BEAM-10017
> URL: https://issues.apache.org/jira/browse/BEAM-10017
> Project: Beam
> Issue Type: Improvement
> Components: io-java-cassandra
>Reporter: Nathan Fisher
>Priority: P3
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Currently there are no options to tune the configuration of the CassandraIO
> reader/writer. This can be useful for either slow clusters, large queries, or
> high latency links.
> The intent would be to expose the following configuration elements as setters
> on the CassandraIO builder similar to withKeyspace and other methods.
>
> |{{[SocketOptions|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html]}}|{{[setConnectTimeoutMillis|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html#setConnectTimeoutMillis-int-](int
> connectTimeoutMillis)}}
> Sets the connection timeout in milliseconds.|
> |{{[SocketOptions|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html]}}|{{[setKeepAlive|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html#setKeepAlive-boolean-](boolean
> keepAlive)}}
> Sets whether to enable TCP keepalive.|
> |{{[SocketOptions|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html]}}|{{[setReadTimeoutMillis|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html#setReadTimeoutMillis-int-](int
> readTimeoutMillis)}}
> Sets the per-host read timeout in milliseconds.|
> |{{[SocketOptions|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html]}}|{{[setReceiveBufferSize|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html#setReceiveBufferSize-int-](int
> receiveBufferSize)}}
> Sets a hint to the size of the underlying buffers for incoming network I/O.|
> |{{[SocketOptions|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html]}}|{{[setReuseAddress|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html#setReuseAddress-boolean-](boolean
> reuseAddress)}}
> Sets whether to enable reuse-address.|
> |{{[SocketOptions|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html]}}|{{[setSendBufferSize|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html#setSendBufferSize-int-](int
> sendBufferSize)}}
> Sets a hint to the size of the underlying buffers for outgoing network I/O.|
> |{{[SocketOptions|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html]}}|{{[setSoLinger|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html#setSoLinger-int-](int
> soLinger)}}
> Sets the linger-on-close timeout.|
> |{{[SocketOptions|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html]}}|{{[setTcpNoDelay|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html#setTcpNoDelay-boolean-](boolean
> tcpNoDelay)}}
> Sets whether to disable Nagle's algorithm.|
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-10017) Expose SocketOptions timeouts in CassandraIO builder
[
https://issues.apache.org/jira/browse/BEAM-10017?focusedWorklogId=434099&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-434099
]
ASF GitHub Bot logged work on BEAM-10017:
-
Author: ASF GitHub Bot
Created on: 16/May/20 18:26
Start Date: 16/May/20 18:26
Worklog Time Spent: 10m
Work Description: nfisher commented on pull request #11732:
URL: https://github.com/apache/beam/pull/11732#issuecomment-629687704
R: @echauchot
Not sure if exposing all of the configuration parameters is desirable. Let
me know what you think.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 434099)
Time Spent: 20m (was: 10m)
> Expose SocketOptions timeouts in CassandraIO builder
>
>
> Key: BEAM-10017
> URL: https://issues.apache.org/jira/browse/BEAM-10017
> Project: Beam
> Issue Type: Improvement
> Components: io-java-cassandra
>Reporter: Nathan Fisher
>Priority: P3
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Currently there are no options to tune the configuration of the CassandraIO
> reader/writer. This can be useful for either slow clusters, large queries, or
> high latency links.
> The intent would be to expose the following configuration elements as setters
> on the CassandraIO builder similar to withKeyspace and other methods.
>
> |{{[SocketOptions|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html]}}|{{[setConnectTimeoutMillis|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html#setConnectTimeoutMillis-int-](int
> connectTimeoutMillis)}}
> Sets the connection timeout in milliseconds.|
> |{{[SocketOptions|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html]}}|{{[setKeepAlive|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html#setKeepAlive-boolean-](boolean
> keepAlive)}}
> Sets whether to enable TCP keepalive.|
> |{{[SocketOptions|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html]}}|{{[setReadTimeoutMillis|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html#setReadTimeoutMillis-int-](int
> readTimeoutMillis)}}
> Sets the per-host read timeout in milliseconds.|
> |{{[SocketOptions|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html]}}|{{[setReceiveBufferSize|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html#setReceiveBufferSize-int-](int
> receiveBufferSize)}}
> Sets a hint to the size of the underlying buffers for incoming network I/O.|
> |{{[SocketOptions|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html]}}|{{[setReuseAddress|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html#setReuseAddress-boolean-](boolean
> reuseAddress)}}
> Sets whether to enable reuse-address.|
> |{{[SocketOptions|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html]}}|{{[setSendBufferSize|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html#setSendBufferSize-int-](int
> sendBufferSize)}}
> Sets a hint to the size of the underlying buffers for outgoing network I/O.|
> |{{[SocketOptions|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html]}}|{{[setSoLinger|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html#setSoLinger-int-](int
> soLinger)}}
> Sets the linger-on-close timeout.|
> |{{[SocketOptions|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html]}}|{{[setTcpNoDelay|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html#setTcpNoDelay-boolean-](boolean
> tcpNoDelay)}}
> Sets whether to disable Nagle's algorithm.|
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (BEAM-9928) Edit task descriptors with qualified import name | Go SDK Code Katas
[ https://issues.apache.org/jira/browse/BEAM-9928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Damon Douglas resolved BEAM-9928. - Fix Version/s: Not applicable Resolution: Done [https://github.com/apache/beam/pull/11690] was approved and merged. > Edit task descriptors with qualified import name | Go SDK Code Katas > > > Key: BEAM-9928 > URL: https://issues.apache.org/jira/browse/BEAM-9928 > Project: Beam > Issue Type: Sub-task > Components: katas, sdk-go >Reporter: Damon Douglas >Priority: P2 > Fix For: Not applicable > > Time Spent: 50m > Remaining Estimate: 0h > > See original PR comment: > https://github.com/apache/beam/pull/11564#discussion_r419628874 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-10017) Expose SocketOptions timeouts in CassandraIO builder
[
https://issues.apache.org/jira/browse/BEAM-10017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109158#comment-17109158
]
Nathan Fisher commented on BEAM-10017:
--
PR opened here:
[https://github.com/apache/beam/pull/11732]
> Expose SocketOptions timeouts in CassandraIO builder
>
>
> Key: BEAM-10017
> URL: https://issues.apache.org/jira/browse/BEAM-10017
> Project: Beam
> Issue Type: Improvement
> Components: io-java-cassandra
>Reporter: Nathan Fisher
>Priority: P3
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently there are no options to tune the configuration of the CassandraIO
> reader/writer. This can be useful for either slow clusters, large queries, or
> high latency links.
> The intent would be to expose the following configuration elements as setters
> on the CassandraIO builder similar to withKeyspace and other methods.
>
> |{{[SocketOptions|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html]}}|{{[setConnectTimeoutMillis|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html#setConnectTimeoutMillis-int-](int
> connectTimeoutMillis)}}
> Sets the connection timeout in milliseconds.|
> |{{[SocketOptions|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html]}}|{{[setKeepAlive|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html#setKeepAlive-boolean-](boolean
> keepAlive)}}
> Sets whether to enable TCP keepalive.|
> |{{[SocketOptions|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html]}}|{{[setReadTimeoutMillis|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html#setReadTimeoutMillis-int-](int
> readTimeoutMillis)}}
> Sets the per-host read timeout in milliseconds.|
> |{{[SocketOptions|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html]}}|{{[setReceiveBufferSize|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html#setReceiveBufferSize-int-](int
> receiveBufferSize)}}
> Sets a hint to the size of the underlying buffers for incoming network I/O.|
> |{{[SocketOptions|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html]}}|{{[setReuseAddress|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html#setReuseAddress-boolean-](boolean
> reuseAddress)}}
> Sets whether to enable reuse-address.|
> |{{[SocketOptions|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html]}}|{{[setSendBufferSize|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html#setSendBufferSize-int-](int
> sendBufferSize)}}
> Sets a hint to the size of the underlying buffers for outgoing network I/O.|
> |{{[SocketOptions|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html]}}|{{[setSoLinger|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html#setSoLinger-int-](int
> soLinger)}}
> Sets the linger-on-close timeout.|
> |{{[SocketOptions|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html]}}|{{[setTcpNoDelay|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html#setTcpNoDelay-boolean-](boolean
> tcpNoDelay)}}
> Sets whether to disable Nagle's algorithm.|
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8019) Support cross-language transforms for DataflowRunner
[ https://issues.apache.org/jira/browse/BEAM-8019?focusedWorklogId=434097&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-434097 ] ASF GitHub Bot logged work on BEAM-8019: Author: ASF GitHub Bot Created on: 16/May/20 18:13 Start Date: 16/May/20 18:13 Worklog Time Spent: 10m Work Description: chamikaramj opened a new pull request #11733: URL: https://github.com/apache/beam/pull/11733 **Please** add a meaningful description for your change here Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[ Expose SocketOptions timeouts in CassandraIO builder
[ https://issues.apache.org/jira/browse/BEAM-10017?focusedWorklogId=434096&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-434096 ] ASF GitHub Bot logged work on BEAM-10017: - Author: ASF GitHub Bot Created on: 16/May/20 18:08 Start Date: 16/May/20 18:08 Worklog Time Spent: 10m Work Description: nfisher opened a new pull request #11732: URL: https://github.com/apache/beam/pull/11732 This PR is intended to expose all of the available Cassandra client socket options so the client can be tuned for various environments. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[ Windowing katas are failing because timestamps are being calculated in local timezones
[ https://issues.apache.org/jira/browse/BEAM-10018?focusedWorklogId=434095&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-434095 ] ASF GitHub Bot logged work on BEAM-10018: - Author: ASF GitHub Bot Created on: 16/May/20 17:50 Start Date: 16/May/20 17:50 Worklog Time Spent: 10m Work Description: iht commented on pull request #11731: URL: https://github.com/apache/beam/pull/11731#issuecomment-629682959 R: @henryken This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 434095) Time Spent: 20m (was: 10m) > Windowing katas are failing because timestamps are being calculated in local > timezones > -- > > Key: BEAM-10018 > URL: https://issues.apache.org/jira/browse/BEAM-10018 > Project: Beam > Issue Type: Bug > Components: katas >Affects Versions: 2.20.0 >Reporter: Israel Herraiz >Assignee: Israel Herraiz >Priority: P2 > Fix For: 2.21.0 > > Time Spent: 20m > Remaining Estimate: 0h > > The two Python katas about windowing fail because the timestamps for the > elements are calculated based on the local timezone, and my timezone does not > match the timezone hardcoded in the tests. > I am working on a pull request to set the timezones in UTC, so the tests are > always done in UTC, and the timestamps are always set in UTC. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-10018) Windowing katas are failing because timestamps are being calculated in local timezones
[ https://issues.apache.org/jira/browse/BEAM-10018?focusedWorklogId=434094&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-434094 ] ASF GitHub Bot logged work on BEAM-10018: - Author: ASF GitHub Bot Created on: 16/May/20 17:49 Start Date: 16/May/20 17:49 Worklog Time Spent: 10m Work Description: iht opened a new pull request #11731: URL: https://github.com/apache/beam/pull/11731 The two Python katas about windowing fail because the timestamps for the elements are calculated based on the local timezone, and my timezone does not match the timezone hardcoded in the tests, and because parsing from strings using `fromisoformat` was failing. In the first Kata, the timestamp was calculated from time objects, and converted to a timestamp in the local timezone. Thus, the results of the test depended on the configuration of the local timezone in the running system. The tests were hardcoded with a timezone different to mine, and thus I always failed to pass this Kata. The changes in this commit change the type in `Event` to be a `datetime`, the timestamps are set in UTC, and the output in the tests is hardcoded in UTC too. This should ensure that the kata works regardless the timezone configured in the system running the kata. In the second Kata, the code was failing with the following error: ```AttributeError: type object 'datetime.datetime' has no attribute 'fromisoformat'``` I changed the timestamps to be set using the `datetime` constructor, rather than parsing from strings using `fromisoformat`. Both katas now pass with the examples hardcoded in the corresponding tests. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_Validates
[jira] [Created] (BEAM-10018) Windowing katas are failing because timestamps are being calculated in local timezones
Israel Herraiz created BEAM-10018: - Summary: Windowing katas are failing because timestamps are being calculated in local timezones Key: BEAM-10018 URL: https://issues.apache.org/jira/browse/BEAM-10018 Project: Beam Issue Type: Bug Components: katas Affects Versions: 2.20.0 Reporter: Israel Herraiz Fix For: 2.21.0 The two Python katas about windowing fail because the timestamps for the elements are calculated based on the local timezone, and my timezone does not match the timezone hardcoded in the tests. I am working on a pull request to set the timezones in UTC, so the tests are always done in UTC, and the timestamps are always set in UTC. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-10018) Windowing katas are failing because timestamps are being calculated in local timezones
[ https://issues.apache.org/jira/browse/BEAM-10018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Israel Herraiz reassigned BEAM-10018: - Assignee: Israel Herraiz > Windowing katas are failing because timestamps are being calculated in local > timezones > -- > > Key: BEAM-10018 > URL: https://issues.apache.org/jira/browse/BEAM-10018 > Project: Beam > Issue Type: Bug > Components: katas >Affects Versions: 2.20.0 >Reporter: Israel Herraiz >Assignee: Israel Herraiz >Priority: P2 > Fix For: 2.21.0 > > > The two Python katas about windowing fail because the timestamps for the > elements are calculated based on the local timezone, and my timezone does not > match the timezone hardcoded in the tests. > I am working on a pull request to set the timezones in UTC, so the tests are > always done in UTC, and the timestamps are always set in UTC. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-10018) Windowing katas are failing because timestamps are being calculated in local timezones
[ https://issues.apache.org/jira/browse/BEAM-10018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Israel Herraiz updated BEAM-10018: -- Status: Open (was: Triage Needed) > Windowing katas are failing because timestamps are being calculated in local > timezones > -- > > Key: BEAM-10018 > URL: https://issues.apache.org/jira/browse/BEAM-10018 > Project: Beam > Issue Type: Bug > Components: katas >Affects Versions: 2.20.0 >Reporter: Israel Herraiz >Assignee: Israel Herraiz >Priority: P2 > Fix For: 2.21.0 > > > The two Python katas about windowing fail because the timestamps for the > elements are calculated based on the local timezone, and my timezone does not > match the timezone hardcoded in the tests. > I am working on a pull request to set the timezones in UTC, so the tests are > always done in UTC, and the timestamps are always set in UTC. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9819) Extend acceptable httplib2 version range.
[ https://issues.apache.org/jira/browse/BEAM-9819?focusedWorklogId=434089&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-434089 ] ASF GitHub Bot logged work on BEAM-9819: Author: ASF GitHub Bot Created on: 16/May/20 17:17 Start Date: 16/May/20 17:17 Worklog Time Spent: 10m Work Description: chamikaramj commented on pull request #11726: URL: https://github.com/apache/beam/pull/11726#issuecomment-629678744 LGTM. Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 434089) Time Spent: 2h (was: 1h 50m) > Extend acceptable httplib2 version range. > - > > Key: BEAM-9819 > URL: https://issues.apache.org/jira/browse/BEAM-9819 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Priority: P2 > Fix For: 2.22.0 > > Time Spent: 2h > Remaining Estimate: 0h > > A Beam Python 3 user reported an inconvenience when migrating their Python 2 > to Python 3 pipeline due to a bug in httlib2 dependency, where httplib2 on > Python3 doesn’t respect NO_PROXY environment variable. This bug was fixed in > 0.13.1 [1]. Looking at the changelog of httplib2[2], there were more > Python3-specific fixes in recent versions. > In the past we restricted httplib2 version due to a conflict with > googledatastore[3]. We have since then removed[4] a dependency on > googledatastore, and I don't see other reasons to restrict httplib2 to > 0.12.0. > [1] https://github.com/httplib2/httplib2/pull/140 > [2] https://github.com/httplib2/httplib2/blob/master/CHANGELOG > [2] > https://github.com/apache/beam/commit/3b2c90156ddb67f4daddf275172c0b2d4eb1eaf6 > [3] > https://github.com/apache/beam/pull/11175/files#diff-e9d0ab71f74dc10309a29b697ee99330L202 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-10017) Expose SocketOptions timeouts in CassandraIO builder
Nathan Fisher created BEAM-10017:
Summary: Expose SocketOptions timeouts in CassandraIO builder
Key: BEAM-10017
URL: https://issues.apache.org/jira/browse/BEAM-10017
Project: Beam
Issue Type: Improvement
Components: io-java-cassandra
Reporter: Nathan Fisher
Currently there are no options to tune the configuration of the CassandraIO
reader/writer. This can be useful for either slow clusters, large queries, or
high latency links.
The intent would be to expose the following configuration elements as setters
on the CassandraIO builder similar to withKeyspace and other methods.
|{{[SocketOptions|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html]}}|{{[setConnectTimeoutMillis|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html#setConnectTimeoutMillis-int-](int
connectTimeoutMillis)}}
Sets the connection timeout in milliseconds.|
|{{[SocketOptions|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html]}}|{{[setKeepAlive|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html#setKeepAlive-boolean-](boolean
keepAlive)}}
Sets whether to enable TCP keepalive.|
|{{[SocketOptions|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html]}}|{{[setReadTimeoutMillis|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html#setReadTimeoutMillis-int-](int
readTimeoutMillis)}}
Sets the per-host read timeout in milliseconds.|
|{{[SocketOptions|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html]}}|{{[setReceiveBufferSize|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html#setReceiveBufferSize-int-](int
receiveBufferSize)}}
Sets a hint to the size of the underlying buffers for incoming network I/O.|
|{{[SocketOptions|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html]}}|{{[setReuseAddress|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html#setReuseAddress-boolean-](boolean
reuseAddress)}}
Sets whether to enable reuse-address.|
|{{[SocketOptions|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html]}}|{{[setSendBufferSize|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html#setSendBufferSize-int-](int
sendBufferSize)}}
Sets a hint to the size of the underlying buffers for outgoing network I/O.|
|{{[SocketOptions|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html]}}|{{[setSoLinger|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html#setSoLinger-int-](int
soLinger)}}
Sets the linger-on-close timeout.|
|{{[SocketOptions|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html]}}|{{[setTcpNoDelay|https://docs.datastax.com/en/drivers/java/3.8/com/datastax/driver/core/SocketOptions.html#setTcpNoDelay-boolean-](boolean
tcpNoDelay)}}
Sets whether to disable Nagle's algorithm.|
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-9745) [beam_PostCommit_Java_PortabilityApi] Various GCP IO tests failing, unable to deserialize Custom DoFns and Custom Coders.
[
https://issues.apache.org/jira/browse/BEAM-9745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chamikara Madhusanka Jayalath updated BEAM-9745:
Priority: P0 (was: P2)
> [beam_PostCommit_Java_PortabilityApi] Various GCP IO tests failing, unable to
> deserialize Custom DoFns and Custom Coders.
> -
>
> Key: BEAM-9745
> URL: https://issues.apache.org/jira/browse/BEAM-9745
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp, java-fn-execution, sdk-java-harness,
> test-failures
>Reporter: Daniel Oliveira
>Assignee: Kenneth Knowles
>Priority: P0
> Labels: currently-failing
>
> _Use this form to file an issue for test failure:_
> * [Jenkins
> Job|https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi/4657/]
> * [Gradle Build
> Scan|https://scans.gradle.com/s/c3izncsa4u24k/tests/by-project]
> Initial investigation:
> The bug appears to be popping up on BigQuery tests mostly, but also a
> BigTable and a Datastore test.
> Here's an example stacktrace of the two errors, showing _only_ the error
> messages themselves. Source:
> [https://scans.gradle.com/s/c3izncsa4u24k/tests/efn4wciuamvqq-ccxt3jvofvqbe]
> {noformat}
> java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error
> received from SDK harness for instruction -191:
> java.lang.IllegalArgumentException: unable to deserialize Custom DoFn With
> Execution Info
> ...
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.sdk.io.gcp.bigquery.BatchLoads$3
> ...
> Caused by: java.lang.RuntimeException: Error received from SDK harness for
> instruction -191: java.lang.IllegalArgumentException: unable to deserialize
> Custom DoFn With Execution Info
> ...
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.sdk.io.gcp.bigquery.BatchLoads$3
> java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error
> received from SDK harness for instruction -206:
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
>
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
> ...
> Caused by:
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
> ...
> Caused by: java.lang.IllegalArgumentException: unable to deserialize Custom
> Coder Bytes
> ...
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.sdk.io.gcp.bigquery.TableRowJsonCoder
> ...
> Caused by: java.lang.RuntimeException: Error received from SDK harness for
> instruction -206:
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
>
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
> ...
> Caused by:
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
> ...
> Caused by: java.lang.IllegalArgumentException: unable to deserialize Custom
> Coder Bytes
> ...
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.sdk.io.gcp.bigquery.TableRowJsonCoder
> ...
> {noformat}
> Update: Looks like this has been failing as far back as [Apr
> 4|https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi/4566/]
> after a long period where the test was consistently timing out since [Mar
> 31|https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi/4546/].
> So it's hard to narrow down what commit may have caused this. Plus, the test
> was failing due to a completely different BigQuery failure before anyway, so
> it seems like this test will need to be completely fixed from scratch,
> instead of tracking down a specific breaking change.
>
> _After you've filled out the above details, please [assign the issue to an
> individual|https://beam.apache.org/contribute/postcommits-guides/index.html#find_specialist].
> Assignee should [treat test failures as
> high-priority|https://beam.apache.org/contribute/postcommits-policies/#assigned-failing-test],
> helping to fix the issue or find a more appropriate owner. See [Apache Beam
> Post-Commit
> Policies|https://beam.apache.org/contribute/postcommits-policies]._
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-9745) [beam_PostCommit_Java_PortabilityApi] Various GCP IO tests failing, unable to deserialize Custom DoFns and Custom Coders.
[
https://issues.apache.org/jira/browse/BEAM-9745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109113#comment-17109113
]
Chamikara Madhusanka Jayalath commented on BEAM-9745:
-
I've seen this when running x-lang KafkaIO tests on Dataflow and I think this
might be an issue with the SDK Harness is setup not an issue with BigQuery as
Boyuan mentioned. Given that we hope to use Java SDK Harness with with UW for
x-lang in Beam 2.22 we should make this a blocker for Beam 2.22. [~kenn] and
[~altay] can we prioritize this ?
> [beam_PostCommit_Java_PortabilityApi] Various GCP IO tests failing, unable to
> deserialize Custom DoFns and Custom Coders.
> -
>
> Key: BEAM-9745
> URL: https://issues.apache.org/jira/browse/BEAM-9745
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp, java-fn-execution, sdk-java-harness,
> test-failures
>Reporter: Daniel Oliveira
>Assignee: Pablo Estrada
>Priority: P2
> Labels: currently-failing
>
> _Use this form to file an issue for test failure:_
> * [Jenkins
> Job|https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi/4657/]
> * [Gradle Build
> Scan|https://scans.gradle.com/s/c3izncsa4u24k/tests/by-project]
> Initial investigation:
> The bug appears to be popping up on BigQuery tests mostly, but also a
> BigTable and a Datastore test.
> Here's an example stacktrace of the two errors, showing _only_ the error
> messages themselves. Source:
> [https://scans.gradle.com/s/c3izncsa4u24k/tests/efn4wciuamvqq-ccxt3jvofvqbe]
> {noformat}
> java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error
> received from SDK harness for instruction -191:
> java.lang.IllegalArgumentException: unable to deserialize Custom DoFn With
> Execution Info
> ...
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.sdk.io.gcp.bigquery.BatchLoads$3
> ...
> Caused by: java.lang.RuntimeException: Error received from SDK harness for
> instruction -191: java.lang.IllegalArgumentException: unable to deserialize
> Custom DoFn With Execution Info
> ...
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.sdk.io.gcp.bigquery.BatchLoads$3
> java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error
> received from SDK harness for instruction -206:
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
>
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
> ...
> Caused by:
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
> ...
> Caused by: java.lang.IllegalArgumentException: unable to deserialize Custom
> Coder Bytes
> ...
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.sdk.io.gcp.bigquery.TableRowJsonCoder
> ...
> Caused by: java.lang.RuntimeException: Error received from SDK harness for
> instruction -206:
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
>
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
> ...
> Caused by:
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
> ...
> Caused by: java.lang.IllegalArgumentException: unable to deserialize Custom
> Coder Bytes
> ...
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.sdk.io.gcp.bigquery.TableRowJsonCoder
> ...
> {noformat}
> Update: Looks like this has been failing as far back as [Apr
> 4|https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi/4566/]
> after a long period where the test was consistently timing out since [Mar
> 31|https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi/4546/].
> So it's hard to narrow down what commit may have caused this. Plus, the test
> was failing due to a completely different BigQuery failure before anyway, so
> it seems like this test will need to be completely fixed from scratch,
> instead of tracking down a specific breaking change.
>
> _After you've filled out the above details, please [assign the issue to an
> individual|https://beam.apache.org/contribute/postcommits-guides/index.html#find_specialist].
> Assignee should [treat test failures as
> high-priority|https://beam.apache.org/contrib
[jira] [Assigned] (BEAM-9745) [beam_PostCommit_Java_PortabilityApi] Various GCP IO tests failing, unable to deserialize Custom DoFns and Custom Coders.
[
https://issues.apache.org/jira/browse/BEAM-9745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chamikara Madhusanka Jayalath reassigned BEAM-9745:
---
Assignee: Kenneth Knowles (was: Pablo Estrada)
> [beam_PostCommit_Java_PortabilityApi] Various GCP IO tests failing, unable to
> deserialize Custom DoFns and Custom Coders.
> -
>
> Key: BEAM-9745
> URL: https://issues.apache.org/jira/browse/BEAM-9745
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp, java-fn-execution, sdk-java-harness,
> test-failures
>Reporter: Daniel Oliveira
>Assignee: Kenneth Knowles
>Priority: P2
> Labels: currently-failing
>
> _Use this form to file an issue for test failure:_
> * [Jenkins
> Job|https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi/4657/]
> * [Gradle Build
> Scan|https://scans.gradle.com/s/c3izncsa4u24k/tests/by-project]
> Initial investigation:
> The bug appears to be popping up on BigQuery tests mostly, but also a
> BigTable and a Datastore test.
> Here's an example stacktrace of the two errors, showing _only_ the error
> messages themselves. Source:
> [https://scans.gradle.com/s/c3izncsa4u24k/tests/efn4wciuamvqq-ccxt3jvofvqbe]
> {noformat}
> java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error
> received from SDK harness for instruction -191:
> java.lang.IllegalArgumentException: unable to deserialize Custom DoFn With
> Execution Info
> ...
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.sdk.io.gcp.bigquery.BatchLoads$3
> ...
> Caused by: java.lang.RuntimeException: Error received from SDK harness for
> instruction -191: java.lang.IllegalArgumentException: unable to deserialize
> Custom DoFn With Execution Info
> ...
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.sdk.io.gcp.bigquery.BatchLoads$3
> java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error
> received from SDK harness for instruction -206:
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
>
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
> ...
> Caused by:
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
> ...
> Caused by: java.lang.IllegalArgumentException: unable to deserialize Custom
> Coder Bytes
> ...
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.sdk.io.gcp.bigquery.TableRowJsonCoder
> ...
> Caused by: java.lang.RuntimeException: Error received from SDK harness for
> instruction -206:
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
>
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
> ...
> Caused by:
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
> ...
> Caused by: java.lang.IllegalArgumentException: unable to deserialize Custom
> Coder Bytes
> ...
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.sdk.io.gcp.bigquery.TableRowJsonCoder
> ...
> {noformat}
> Update: Looks like this has been failing as far back as [Apr
> 4|https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi/4566/]
> after a long period where the test was consistently timing out since [Mar
> 31|https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi/4546/].
> So it's hard to narrow down what commit may have caused this. Plus, the test
> was failing due to a completely different BigQuery failure before anyway, so
> it seems like this test will need to be completely fixed from scratch,
> instead of tracking down a specific breaking change.
>
> _After you've filled out the above details, please [assign the issue to an
> individual|https://beam.apache.org/contribute/postcommits-guides/index.html#find_specialist].
> Assignee should [treat test failures as
> high-priority|https://beam.apache.org/contribute/postcommits-policies/#assigned-failing-test],
> helping to fix the issue or find a more appropriate owner. See [Apache Beam
> Post-Commit
> Policies|https://beam.apache.org/contribute/postcommits-policies]._
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-8239) Docker options in --environment_config
[
https://issues.apache.org/jira/browse/BEAM-8239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17108991#comment-17108991
]
Maximilian Michels commented on BEAM-8239:
--
To this date there is no option to mount a volume using the environment config.
I think it would be good to revisit BEAM-5440 for the next release.
For now the only option is to use a distributed file system (e.g. HDFS, S3,
Google Storage).
> Docker options in --environment_config
> --
>
> Key: BEAM-8239
> URL: https://issues.apache.org/jira/browse/BEAM-8239
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-harness
>Affects Versions: 2.15.0
>Reporter: Benjamin Tan
>Priority: Major
>
> {{I'm trying to mount a directory by providing additional arguments via
> –environment_config in the PipelineOptions:}}
>
> {{pipeline_options =
> PipelineOptions([{color:#6a8759}"--runner=PortableRunner"{color}{color:#cc7832},{color}
> {color:#6a8759}"--job_endpoint=localhost:8099"{color}{color:#cc7832},{color}
> {color:#6a8759}"--environment_config=-v /tmp:/tmp
> benjamintan-docker-apache.bintray.io/beam/python3:latest"{color}]{color:#cc7832},{color}
> {color:#aa4926}pipeline_type_check{color}={color:#cc7832}True{color})}}
>
> However, the command fails with the following:
>
>
> {{RuntimeError: Pipeline
> BeamApp-benjamintan-091616-839e633f_994659f0-7da9-412e-91e2-f32dd4f24b5c
> failed in state FAILED: java.io.IOException: Received exit code 125 for
> command 'docker run -d --mount
> type=bind,src=/home/benjamintan/.config/gcloud,dst=/root/.config/gcloud
> --network=host --env=DOCKER_MAC_CONTAINER=null -v /tmp:/tmp
> benjamintan-docker-apache.bintray.io/beam/python3:latest --id=7-1
> --logging_endpoint=localhost:41835 --artifact_endpoint=localhost:40063
> --provision_endpoint=localhost:39827 --control_endpoint=localhost:45355'.
> stderr: unknown flag: --idSee 'docker run --help'.}}
>
> However, if I were to copy and paste the `docker run ...` command, the
> command seems OK (no syntax errors)
>
> This seems related to BEAM-5440. It isn't clear if there's a "right" way to
> pass in additional Docker run arguments.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
