[jira] [Updated] (BEAM-10224) Test using a DATE field in an aggregation
[ https://issues.apache.org/jira/browse/BEAM-10224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Qiu updated BEAM-10224: - Labels: beam-fixit (was: ) > Test using a DATE field in an aggregation > - > > Key: BEAM-10224 > URL: https://issues.apache.org/jira/browse/BEAM-10224 > Project: Beam > Issue Type: Task > Components: dsl-sql >Reporter: Brian Hulette >Assignee: Robin Qiu >Priority: P2 > Labels: beam-fixit > > Since logical types are aggregated with their representation type this may be > an issue. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9179) Refactor Beam ZetaSQL type translation code
[ https://issues.apache.org/jira/browse/BEAM-9179?focusedWorklogId=444136&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-444136 ] ASF GitHub Bot logged work on BEAM-9179: Author: ASF GitHub Bot Created on: 11/Jun/20 06:46 Start Date: 11/Jun/20 06:46 Worklog Time Spent: 10m Work Description: robinyqiu opened a new pull request #11988: URL: https://github.com/apache/beam/pull/11988 Simple code cleanup (adding tests, removing dead code, etc.). Does not change functionality. r: @apilloud Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Samza | Spark --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.ap
[jira] [Commented] (BEAM-9178) Support ZetaSQL TIMESTAMP functions in BeamSQL
[ https://issues.apache.org/jira/browse/BEAM-9178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17132956#comment-17132956 ] Robin Qiu commented on BEAM-9178: - The majority of TIMESTAMP functions are already supported. The remaining ones requires support of other data types (e.g. TIME, DATETIME). After those types are implemented we can complete this issue. > Support ZetaSQL TIMESTAMP functions in BeamSQL > -- > > Key: BEAM-9178 > URL: https://issues.apache.org/jira/browse/BEAM-9178 > Project: Beam > Issue Type: New Feature > Components: dsl-sql-zetasql >Reporter: Robin Qiu >Priority: P2 > Time Spent: 3.5h > Remaining Estimate: 0h > > Support *all* TIMESTAMP functions defined in ZetaSQL (BigQuery Standard SQL). > See the full list of functions below: > [https://cloud.google.com/bigquery/docs/reference/standard-sql/timestamp_functions] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-9178) Support ZetaSQL TIMESTAMP functions in BeamSQL
[ https://issues.apache.org/jira/browse/BEAM-9178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Qiu reassigned BEAM-9178: --- Assignee: Zijie Song > Support ZetaSQL TIMESTAMP functions in BeamSQL > -- > > Key: BEAM-9178 > URL: https://issues.apache.org/jira/browse/BEAM-9178 > Project: Beam > Issue Type: New Feature > Components: dsl-sql-zetasql >Reporter: Robin Qiu >Assignee: Zijie Song >Priority: P2 > Time Spent: 3.5h > Remaining Estimate: 0h > > Support *all* TIMESTAMP functions defined in ZetaSQL (BigQuery Standard SQL). > See the full list of functions below: > [https://cloud.google.com/bigquery/docs/reference/standard-sql/timestamp_functions] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-10240) Support ZetaSQL DATETIME functions in BeamSQL
[ https://issues.apache.org/jira/browse/BEAM-10240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Qiu updated BEAM-10240: - Status: Open (was: Triage Needed) > Support ZetaSQL DATETIME functions in BeamSQL > - > > Key: BEAM-10240 > URL: https://issues.apache.org/jira/browse/BEAM-10240 > Project: Beam > Issue Type: New Feature > Components: dsl-sql-zetasql >Reporter: Robin Qiu >Assignee: Zijie Song >Priority: P2 > > Support *all* DATETIME functions defined in ZetaSQL (BigQuery Standard SQL). > See the list of functions below: > [https://github.com/google/zetasql/blob/master/docs/datetime_functions.md|https://github.com/google/zetasql/blob/master/docs/time_functions.md] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-10239) Support ZetaSQL NUMERIC type in BeamSQL
[ https://issues.apache.org/jira/browse/BEAM-10239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Qiu updated BEAM-10239: - Status: Open (was: Triage Needed) > Support ZetaSQL NUMERIC type in BeamSQL > --- > > Key: BEAM-10239 > URL: https://issues.apache.org/jira/browse/BEAM-10239 > Project: Beam > Issue Type: New Feature > Components: dsl-sql-zetasql >Reporter: Robin Qiu >Assignee: Zijie Song >Priority: P2 > > Support NUMERIC type defined in ZetaSQL (BigQuery Standard SQL). See the list > of functions below: > [https://github.com/google/zetasql/blob/master/docs/data-types.md#numeric-type] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-10240) Support ZetaSQL DATETIME functions in BeamSQL
Robin Qiu created BEAM-10240: Summary: Support ZetaSQL DATETIME functions in BeamSQL Key: BEAM-10240 URL: https://issues.apache.org/jira/browse/BEAM-10240 Project: Beam Issue Type: New Feature Components: dsl-sql-zetasql Reporter: Robin Qiu Assignee: Zijie Song Support *all* DATETIME functions defined in ZetaSQL (BigQuery Standard SQL). See the list of functions below: [https://github.com/google/zetasql/blob/master/docs/datetime_functions.md|https://github.com/google/zetasql/blob/master/docs/time_functions.md] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-10239) Support ZetaSQL NUMERIC type in BeamSQL
Robin Qiu created BEAM-10239: Summary: Support ZetaSQL NUMERIC type in BeamSQL Key: BEAM-10239 URL: https://issues.apache.org/jira/browse/BEAM-10239 Project: Beam Issue Type: New Feature Components: dsl-sql-zetasql Reporter: Robin Qiu Assignee: Zijie Song Support NUMERIC type defined in ZetaSQL (BigQuery Standard SQL). See the list of functions below: [https://github.com/google/zetasql/blob/master/docs/data-types.md#numeric-type] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-10238) Improve Beam ZetaSQL type system
Robin Qiu created BEAM-10238: Summary: Improve Beam ZetaSQL type system Key: BEAM-10238 URL: https://issues.apache.org/jira/browse/BEAM-10238 Project: Beam Issue Type: New Feature Components: dsl-sql-zetasql Reporter: Robin Qiu Assignee: Robin Qiu This is an umbrella issue for several improvements we plan to make to the Beam ZetaSQL type system. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-10238) Improve Beam ZetaSQL type system
[ https://issues.apache.org/jira/browse/BEAM-10238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-10238: --- Status: Open (was: Triage Needed) > Improve Beam ZetaSQL type system > > > Key: BEAM-10238 > URL: https://issues.apache.org/jira/browse/BEAM-10238 > Project: Beam > Issue Type: New Feature > Components: dsl-sql-zetasql >Reporter: Robin Qiu >Assignee: Robin Qiu >Priority: P2 > > This is an umbrella issue for several improvements we plan to make to the > Beam ZetaSQL type system. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (BEAM-6764) OutOfMemory Exception in Dataflow Spanner Write Mutations
[
https://issues.apache.org/jira/browse/BEAM-6764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17132934#comment-17132934
]
Mai Nakagawa edited comment on BEAM-6764 at 6/11/20, 5:56 AM:
--
[~jleoirab] - Are you using [Streaming Engine|#dataflow-streaming-engine]? We
faced the same situation while using it. We could work this around by not using
Streaming Engine.
was (Author: nakagawa@gmail.com):
[~jleoirab] - Are you using [Streaming Engine|#dataflow-streaming-engine]]? We
faced the same situation while using it. We could work this around by not using
Streaming Engine.
> OutOfMemory Exception in Dataflow Spanner Write Mutations
> -
>
> Key: BEAM-6764
> URL: https://issues.apache.org/jira/browse/BEAM-6764
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp
>Affects Versions: 2.9.0, 2.10.0, 2.11.0
>Reporter: Joshua
>Priority: P2
> Labels: stale-P2
>
> Since I upgraded my apache beam sdk version to >= 2.9.0, I have been noticing
> OOM exceptions while using the dataflow runner to write mutations to spanner.
> I have been using n1-standard-4 since version 2.9.0. On that version, it
> works. But on higher versions, I get the exception.
>
> The stackdriver logs is provided below
> {code:java}
> java.lang.OutOfMemoryError: Java heap space
> java.util.ArrayList.(ArrayList.java:152)
> org.apache.beam.sdk.io.gcp.spanner.SpannerIO$GatherBundleAndSortFn.initSorter(SpannerIO.java:1056)
> org.apache.beam.sdk.io.gcp.spanner.SpannerIO$GatherBundleAndSortFn.startBundle(SpannerIO.java:1049)
> {code}
> I have a very basic PTransform for writing to Spanner:
> {code:java}
> SpannerIO.write().withInstanceId(options.getSpannerInstanceId())
> .withDatabaseId(options.getSpannerDatabaseId())
> .withProjectId(options.getProject())
> .withFailureMode(SpannerIO.FailureMode.REPORT_FAILURES);
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (BEAM-6764) OutOfMemory Exception in Dataflow Spanner Write Mutations
[
https://issues.apache.org/jira/browse/BEAM-6764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17132934#comment-17132934
]
Mai Nakagawa edited comment on BEAM-6764 at 6/11/20, 5:56 AM:
--
[~jleoirab] - Are you using [Streaming Engine|#dataflow-streaming-engine]]? We
faced the same situation while using it. We could work this around by not using
Streaming Engine.
was (Author: nakagawa@gmail.com):
[~jleoirab] - Are you using [Streaming
Engine|[https://cloud.google.com/dataflow/docs/guides/deploying-a-pipeline#dataflow-streaming-engine]|https://cloud.google.com/dataflow/docs/guides/deploying-a-pipeline#dataflow-streaming-engine]?]?
We faced the same situation while using it. We could work this around by not
using Streaming Engine.
> OutOfMemory Exception in Dataflow Spanner Write Mutations
> -
>
> Key: BEAM-6764
> URL: https://issues.apache.org/jira/browse/BEAM-6764
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp
>Affects Versions: 2.9.0, 2.10.0, 2.11.0
>Reporter: Joshua
>Priority: P2
> Labels: stale-P2
>
> Since I upgraded my apache beam sdk version to >= 2.9.0, I have been noticing
> OOM exceptions while using the dataflow runner to write mutations to spanner.
> I have been using n1-standard-4 since version 2.9.0. On that version, it
> works. But on higher versions, I get the exception.
>
> The stackdriver logs is provided below
> {code:java}
> java.lang.OutOfMemoryError: Java heap space
> java.util.ArrayList.(ArrayList.java:152)
> org.apache.beam.sdk.io.gcp.spanner.SpannerIO$GatherBundleAndSortFn.initSorter(SpannerIO.java:1056)
> org.apache.beam.sdk.io.gcp.spanner.SpannerIO$GatherBundleAndSortFn.startBundle(SpannerIO.java:1049)
> {code}
> I have a very basic PTransform for writing to Spanner:
> {code:java}
> SpannerIO.write().withInstanceId(options.getSpannerInstanceId())
> .withDatabaseId(options.getSpannerDatabaseId())
> .withProjectId(options.getProject())
> .withFailureMode(SpannerIO.FailureMode.REPORT_FAILURES);
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-6764) OutOfMemory Exception in Dataflow Spanner Write Mutations
[
https://issues.apache.org/jira/browse/BEAM-6764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17132934#comment-17132934
]
Mai Nakagawa commented on BEAM-6764:
[~jleoirab] - Are you using [Streaming
Engine|[https://cloud.google.com/dataflow/docs/guides/deploying-a-pipeline#dataflow-streaming-engine]|https://cloud.google.com/dataflow/docs/guides/deploying-a-pipeline#dataflow-streaming-engine]?]?
We faced the same situation while using it. We could work this around by not
using Streaming Engine.
> OutOfMemory Exception in Dataflow Spanner Write Mutations
> -
>
> Key: BEAM-6764
> URL: https://issues.apache.org/jira/browse/BEAM-6764
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp
>Affects Versions: 2.9.0, 2.10.0, 2.11.0
>Reporter: Joshua
>Priority: P2
> Labels: stale-P2
>
> Since I upgraded my apache beam sdk version to >= 2.9.0, I have been noticing
> OOM exceptions while using the dataflow runner to write mutations to spanner.
> I have been using n1-standard-4 since version 2.9.0. On that version, it
> works. But on higher versions, I get the exception.
>
> The stackdriver logs is provided below
> {code:java}
> java.lang.OutOfMemoryError: Java heap space
> java.util.ArrayList.(ArrayList.java:152)
> org.apache.beam.sdk.io.gcp.spanner.SpannerIO$GatherBundleAndSortFn.initSorter(SpannerIO.java:1056)
> org.apache.beam.sdk.io.gcp.spanner.SpannerIO$GatherBundleAndSortFn.startBundle(SpannerIO.java:1049)
> {code}
> I have a very basic PTransform for writing to Spanner:
> {code:java}
> SpannerIO.write().withInstanceId(options.getSpannerInstanceId())
> .withDatabaseId(options.getSpannerDatabaseId())
> .withProjectId(options.getProject())
> .withFailureMode(SpannerIO.FailureMode.REPORT_FAILURES);
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-9179) Refactor Beam ZetaSQL type translation code
[ https://issues.apache.org/jira/browse/BEAM-9179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Qiu updated BEAM-9179: Summary: Refactor Beam ZetaSQL type translation code (was: Simplify types in Beam ZetaSQL) > Refactor Beam ZetaSQL type translation code > --- > > Key: BEAM-9179 > URL: https://issues.apache.org/jira/browse/BEAM-9179 > Project: Beam > Issue Type: Improvement > Components: dsl-sql-zetasql >Reporter: Robin Qiu >Assignee: Robin Qiu >Priority: P2 > Labels: stale-assigned > Time Spent: 1h 20m > Remaining Estimate: 0h > > Currently, how types work in Beam ZetaSQL is very confusing: code that does > type conversion between Calcite/Beam/ZetaSQL type systems are everywhere and > is likely to be not consistent. In addition, the recent migration to > BeamZetaSqlCalcRel makes the situation even worse: it adds another layer for > type conversion and makes a lot of code (for making ZetaSQL functions > manually) obsolete. This makes debugging and supporting new features very > hard. A few examples: > * Support new ZetaSQL types (e.g. DATE/TIME/DATETIME) > * Fix the long-standing precision & timezone issue of TIMESTMAP type > * Support BeamSQL query parameters > Therefore, I propose we work on the following items to make the story of Beam > ZetaSQL type clearer: > * Define a clear mapping between Calcite/Beam/ZetaSQL types and values > * Refactor the codebase to move all type conversion code to a single source > of truth > * Add thorough testing for it > * Remove obsolete code that is added to make functions work before > BeamZetaSqlCalcRel -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9951) Create Go SDK synthetic sources.
[ https://issues.apache.org/jira/browse/BEAM-9951?focusedWorklogId=444123&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-444123 ] ASF GitHub Bot logged work on BEAM-9951: Author: ASF GitHub Bot Created on: 11/Jun/20 05:20 Start Date: 11/Jun/20 05:20 Worklog Time Spent: 10m Work Description: lostluck commented on pull request #11870: URL: https://github.com/apache/beam/pull/11870#issuecomment-642414004 Run Go PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 444123) Time Spent: 4h 50m (was: 4h 40m) > Create Go SDK synthetic sources. > > > Key: BEAM-9951 > URL: https://issues.apache.org/jira/browse/BEAM-9951 > Project: Beam > Issue Type: Sub-task > Components: sdk-go >Reporter: Daniel Oliveira >Assignee: Daniel Oliveira >Priority: P2 > Time Spent: 4h 50m > Remaining Estimate: 0h > > Create synthetic sources for the Go SDK like > [Java|https://github.com/apache/beam/tree/master/sdks/java/io/synthetic/src/main/java/org/apache/beam/sdk/io/synthetic] > and > [Python|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/testing/synthetic_pipeline.py] > have. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-10169) ParDo* functions should declare the correct output N in their error message
[
https://issues.apache.org/jira/browse/BEAM-10169?focusedWorklogId=444119&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-444119
]
ASF GitHub Bot logged work on BEAM-10169:
-
Author: ASF GitHub Bot
Created on: 11/Jun/20 05:06
Start Date: 11/Jun/20 05:06
Worklog Time Spent: 10m
Work Description: lostluck commented on a change in pull request #11976:
URL: https://github.com/apache/beam/pull/11976#discussion_r438475207
##
File path: sdks/go/pkg/beam/core/util/reflectx/functions_test.go
##
@@ -41,3 +45,19 @@ func TestLoadFunction(t *testing.T) {
t.Errorf("got %d, wanted %d", out[0].Int(), testFunction())
}
}
+
+func TestFunctionOutputSize(t *testing.T) {
+ expected := 1
+ received := FunctionOutputSize(testFunction)
+ if received != expected {
+ t.Errorf("got %d, wanted %d", received, expected)
+ }
+}
+
+func TestFunction2OutputSize(t *testing.T) {
Review comment:
Conventionally, if there are multiple cases for a given test, they
should be combined into the same function, and run as a loop.
See https://gobyexample.com/testing for an excellent example.
Otherwise, this function should be named TestFunctionOutputSize_2
##
File path: sdks/go/pkg/beam/pardo.go
##
@@ -414,7 +414,45 @@ func ParDo6(s Scope, dofn interface{}, col PCollection,
opts ...Option) (PCollec
func ParDo7(s Scope, dofn interface{}, col PCollection, opts ...Option)
(PCollection, PCollection, PCollection, PCollection, PCollection, PCollection,
PCollection) {
ret := MustN(TryParDo(s, dofn, col, opts...))
if len(ret) != 7 {
- panic(fmt.Sprintf("expected 7 output. Found: %v", ret))
+ panic(ParDoErrorFormatter(dofn, ParDo7))
}
return ret[0], ret[1], ret[2], ret[3], ret[4], ret[5], ret[6]
}
+
+//ParDoErrorFormatter is a helper function to provide a more concise error
+// message to the users when a DoFn and its ParDo pairing is incorrect.
+func ParDoErrorFormatter(doFn interface{}, parDo interface{}) string {
+ doFnName := reflectx.FunctionName(doFn)
+ doFnOutSize := reflectx.FunctionOutputSize(doFn)
+
+ parDoName := reflectx.FunctionName(parDo)
+ parDoOutSize := reflectx.FunctionOutputSize(parDo)
+
+ useParDo := reflectx.FunctionName(RecommendParDo(doFnOutSize))
+ return fmt.Sprintf("DoFn %v has %v outptus, but %v requires %v outputs,
Use %v instead.", doFnName, doFnOutSize, parDoName, parDoOutSize, useParDo)
+
+}
+
+// recommendParDo takes a in a DoFns emit dimension and recommends the correct
+// ParDo to use.
+func RecommendParDo(emitDim int) interface{} {
+ switch {
+ case emitDim == 0:
+ return ParDo0
+ case emitDim == 1:
+ return ParDo
+ case emitDim == 2:
+ return ParDo2
+ case emitDim == 3:
+ return ParDo3
+ case emitDim == 4:
+ return ParDo4
+ case emitDim == 5:
+ return ParDo5
+ case emitDim == 6:
+ return ParDo6
+ case emitDim == 7:
+ return ParDo7
+ }
+ return ParDoN
+}
Review comment:
What I'd recommend here instead though is to return a string instead of
the function.
```
switch emitDim {
case 0,2,3,4,5,6,7:
return fmt.Sprintf("ParDo%d", emitDim)
case 1:
return "ParDo"
default:
return "ParDoN"
}
```
Easier to read, and see if it's correct. Read [Effective Go for more about
Switches](https://golang.org/doc/effective_go.html#switch)
##
File path: sdks/go/pkg/beam/pardo.go
##
@@ -414,7 +414,45 @@ func ParDo6(s Scope, dofn interface{}, col PCollection,
opts ...Option) (PCollec
func ParDo7(s Scope, dofn interface{}, col PCollection, opts ...Option)
(PCollection, PCollection, PCollection, PCollection, PCollection, PCollection,
PCollection) {
ret := MustN(TryParDo(s, dofn, col, opts...))
if len(ret) != 7 {
- panic(fmt.Sprintf("expected 7 output. Found: %v", ret))
+ panic(ParDoErrorFormatter(dofn, ParDo7))
}
return ret[0], ret[1], ret[2], ret[3], ret[4], ret[5], ret[6]
}
+
+//ParDoErrorFormatter is a helper function to provide a more concise error
+// message to the users when a DoFn and its ParDo pairing is incorrect.
+func ParDoErrorFormatter(doFn interface{}, parDo interface{}) string {
+ doFnName := reflectx.FunctionName(doFn)
+ doFnOutSize := reflectx.FunctionOutputSize(doFn)
+
+ parDoName := reflectx.FunctionName(parDo)
+ parDoOutSize := reflectx.FunctionOutputSize(parDo)
+
+ useParDo := reflectx.FunctionName(RecommendParDo(doFnOutSize))
+ return fmt.Sprintf("DoFn %v has %v outptus, but %v requires %v outputs,
Use %v instead.", doFnName, doFnO
[jira] [Commented] (BEAM-9466) SnowflakeIO in python
[ https://issues.apache.org/jira/browse/BEAM-9466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17132903#comment-17132903 ] Shashanka Balakuntala Srinivasa commented on BEAM-9466: --- Hi [~kenn], there was a discussion in dev-mailer list which can be found here [1] where [~kkucharczyk] mentioned that they were working on a cross platform connector. Hence I parked on this. But if this still can make sense...please let me know i will work on this and complete it at my earliest. [1] - [https://lists.apache.org/thread.html/r4759f0aa5ba830e239589c3dd2724f1f9b145c31fbdd6b0e41719a6b%40%3Cdev.beam.apache.org%3E] > SnowflakeIO in python > - > > Key: BEAM-9466 > URL: https://issues.apache.org/jira/browse/BEAM-9466 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Shashanka Balakuntala Srinivasa >Priority: P3 > > As Elias Djurfeldt mentioned, there is a Snowflake Python library available > [1], using this to create an SnowflakeIO for python > [1] [https://docs.snowflake.net/manuals/user-guide/python-connector.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-10237) Clean up Experimental usage and come up with a graduation policy
[ https://issues.apache.org/jira/browse/BEAM-10237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17132902#comment-17132902 ] Kenneth Knowles commented on BEAM-10237: I think [~iemejia] expressed interest, too. (apologies if I am misremembering) > Clean up Experimental usage and come up with a graduation policy > > > Key: BEAM-10237 > URL: https://issues.apache.org/jira/browse/BEAM-10237 > Project: Beam > Issue Type: Bug > Components: beam-model >Reporter: Ahmet Altay >Priority: P2 > > Beam, APIs use experimental features quite a lot. Experimental features are > growing, but at some point we should consider either removing or graduating > them. > This is a proposal to: > - Track: All experimental features will be continually tracked in the change > log explicitly. > - For features staying in experimental mode longer than 12 months, > --> Graduate: we should review whether the feature meets the usage bar to be > long term maintained, > -- > Deprecate: or should be deprecated to reduce maintenance load. > It would be good to add a github ci action to check new usage of experimental > tags are added to the change log. > /cc [~kenn] [~lcwik] [~robertwb] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (BEAM-3194) Support annotating that a DoFn requires stable / deterministic input for replay/retry
[ https://issues.apache.org/jira/browse/BEAM-3194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Qiu closed BEAM-3194. --- Fix Version/s: 2.21.0 Resolution: Fixed > Support annotating that a DoFn requires stable / deterministic input for > replay/retry > - > > Key: BEAM-3194 > URL: https://issues.apache.org/jira/browse/BEAM-3194 > Project: Beam > Issue Type: New Feature > Components: beam-model, sdk-java-core >Reporter: Kenneth Knowles >Priority: P2 > Labels: stale-P2 > Fix For: 2.21.0 > > Time Spent: 3h 10m > Remaining Estimate: 0h > > See the thread: > https://lists.apache.org/thread.html/5fd81ce371aeaf642665348f8e6940e308e04275dd7072f380f9f945@%3Cdev.beam.apache.org%3E > We need this in order to have truly cross-runner end-to-end exactly once via > replay + idempotence. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-3194) Support annotating that a DoFn requires stable / deterministic input for replay/retry
[ https://issues.apache.org/jira/browse/BEAM-3194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17132892#comment-17132892 ] Robin Qiu commented on BEAM-3194: - [~kenn] Yes the annotation already exists. I will close this JIRA. Feel free to re-open if you disagree. > Support annotating that a DoFn requires stable / deterministic input for > replay/retry > - > > Key: BEAM-3194 > URL: https://issues.apache.org/jira/browse/BEAM-3194 > Project: Beam > Issue Type: New Feature > Components: beam-model, sdk-java-core >Reporter: Kenneth Knowles >Priority: P2 > Labels: stale-P2 > Time Spent: 3h 10m > Remaining Estimate: 0h > > See the thread: > https://lists.apache.org/thread.html/5fd81ce371aeaf642665348f8e6940e308e04275dd7072f380f9f945@%3Cdev.beam.apache.org%3E > We need this in order to have truly cross-runner end-to-end exactly once via > replay + idempotence. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-1261) State API should allow state to be managed in different windows
[ https://issues.apache.org/jira/browse/BEAM-1261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-1261: -- Issue Type: Wish (was: New Feature) > State API should allow state to be managed in different windows > --- > > Key: BEAM-1261 > URL: https://issues.apache.org/jira/browse/BEAM-1261 > Project: Beam > Issue Type: Wish > Components: beam-model >Reporter: Ben Chambers >Priority: P2 > Labels: stale-P2 > > For example, even if the elements are being processed in fixed windows of an > hour, it may be desirable for the state to "roll over" between windows (or be > available to all windows). > It will also be necessary to figure out when this state should be deleted > (TTL? maximum retention?) > Another problem is how to deal with out of order data. If data comes in from > the 10:00 AM window, should its state changes be visible to the data in the > 9:00 AM window? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-2182) AvroCoder is quite overconfident: it claims to be able to create a schema for things it certainly cannot
[
https://issues.apache.org/jira/browse/BEAM-2182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles updated BEAM-2182:
--
Summary: AvroCoder is quite overconfident: it claims to be able to create a
schema for things it certainly cannot (was: AvroCoder is quite overconfident)
> AvroCoder is quite overconfident: it claims to be able to create a schema for
> things it certainly cannot
>
>
> Key: BEAM-2182
> URL: https://issues.apache.org/jira/browse/BEAM-2182
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Reporter: Kenneth Knowles
>Priority: P2
> Labels: stale-P2
>
> AvoCoder claims it can create a schema for:
> {code}
> public static class X {
> public Thread t = Thread.currentThread();
> }
> {code}
> The issue goes all the way to Avro, we think. To be investigated. Meanwhile,
> we need AvroCoder to say no to such things.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-10201) Enhance JsonToRow to add Deadletter Support
[ https://issues.apache.org/jira/browse/BEAM-10201?focusedWorklogId=444088&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-444088 ] ASF GitHub Bot logged work on BEAM-10201: - Author: ASF GitHub Bot Created on: 11/Jun/20 02:30 Start Date: 11/Jun/20 02:30 Worklog Time Spent: 10m Work Description: rezarokni commented on pull request #11929: URL: https://github.com/apache/beam/pull/11929#issuecomment-642368157 @reuvenlax @pabloem The latest commit covers most of the points now, with the exception of the three open questions I raised. - Return of the Schema that failed. @pabloem - Discussion about withExtendedErrMsg @reuvenlax - use of transient with ObjectMapper @reuvenlax This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 444088) Time Spent: 3h (was: 2h 50m) > Enhance JsonToRow to add Deadletter Support > --- > > Key: BEAM-10201 > URL: https://issues.apache.org/jira/browse/BEAM-10201 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core >Reporter: Reza ardeshir rokni >Assignee: Reza ardeshir rokni >Priority: P2 > Time Spent: 3h > Remaining Estimate: 0h > > Current JsonToRow transform does not support Dead Letter pattern on parse > failures. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-10201) Enhance JsonToRow to add Deadletter Support
[
https://issues.apache.org/jira/browse/BEAM-10201?focusedWorklogId=444087&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-444087
]
ASF GitHub Bot logged work on BEAM-10201:
-
Author: ASF GitHub Bot
Created on: 11/Jun/20 02:27
Start Date: 11/Jun/20 02:27
Worklog Time Spent: 10m
Work Description: rezarokni commented on a change in pull request #11929:
URL: https://github.com/apache/beam/pull/11929#discussion_r438508022
##
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/JsonToRow.java
##
@@ -116,4 +131,267 @@ private ObjectMapper objectMapper() {
return this.objectMapper;
}
}
+
+ /**
+ * Enable Dead letter support. If this value is set errors in the parsing
layer are returned as
+ * Row objects within a {@link ParseResult}
+ *
+ * You can access the results by using:
+ *
+ * ParseResult results =
jsonPersons.apply(JsonToRow.withDeadLetter(PERSON_SCHEMA));
+ *
+ * {@link ParseResult#getResults()}
+ *
+ * {@Code PCollection personRows = results.getResults()}
+ *
+ * {@link ParseResult#getFailedToParseLines()}
+ *
+ * {@Code PCollection errorsLines = results.getFailedToParseLines()}
+ *
+ * To access the reason for the failure you will need to first enable
extended error reporting.
+ * {@Code ParseResult results =
+ *
jsonPersons.apply(JsonToRow.withDeadLetter(PERSON_SCHEMA).withExtendedErrorInfo());
}
+ *
+ * {@link ParseResult#getFailedToParseLinesWithErr()}
+ *
+ * {@Code PCollection errorsLinesWithErrMsg =
results.getFailedToParseLines()}
+ *
+ * @return {@link JsonToRowWithErrFn}
+ */
+ @Experimental(Kind.SCHEMAS)
+ public static JsonToRowWithErrFn withDeadLetter(Schema rowSchema) {
+return JsonToRowWithErrFn.forSchema(rowSchema);
+ }
+
+ @AutoValue
+ abstract static class JsonToRowWithErrFn extends
PTransform, ParseResult> {
+
+private Pipeline pipeline;
+
+private PCollection parsedLine;
+private PCollection failedParse;
+private PCollection failedParseWithErr;
+
+private static final String LINE_FIELD_NAME = "line";
+private static final String ERROR_FIELD_NAME = "err";
+
+public static final Schema ERROR_ROW_SCHEMA =
+Schema.of(Field.of(LINE_FIELD_NAME, FieldType.STRING));
+
+public static final Schema ERROR_ROW_WITH_ERR_MSG_SCHEMA =
+Schema.of(
+Field.of(LINE_FIELD_NAME, FieldType.STRING),
+Field.of(ERROR_FIELD_NAME, FieldType.STRING));
+
+static final TupleTag PARSED_LINE = new TupleTag() {};
+static final TupleTag PARSE_ERROR_LINE = new TupleTag() {};
+static final TupleTag PARSE_ERROR_LINE_WITH_MSG = new TupleTag()
{};
+
+public abstract Schema getSchema();
+
+public abstract String getLineFieldName();
+
+public abstract String getErrorFieldName();
+
+public abstract boolean getExtendedErrorInfo();
+
+PCollection deadLetterCollection;
+
+public abstract Builder toBuilder();
+
+@AutoValue.Builder
+public abstract static class Builder {
+ public abstract Builder setSchema(Schema value);
+
+ public abstract Builder setLineFieldName(String value);
+
+ public abstract Builder setErrorFieldName(String value);
+
+ public abstract Builder setExtendedErrorInfo(boolean value);
+
+ public abstract JsonToRowWithErrFn build();
+}
+
+public static JsonToRowWithErrFn forSchema(Schema rowSchema) {
+ // Throw exception if this schema is not supported by RowJson
+ RowJson.verifySchemaSupported(rowSchema);
+ return new AutoValue_JsonToRow_JsonToRowWithErrFn.Builder()
+ .setSchema(rowSchema)
+ .setExtendedErrorInfo(false)
+ .setLineFieldName(LINE_FIELD_NAME)
+ .setErrorFieldName(ERROR_FIELD_NAME)
+ .build();
+}
+
+/**
+ * Adds the error message to the returned error Row.
+ *
+ * @return {@link JsonToRow}
+ */
+public JsonToRowWithErrFn withExtendedErrorInfo() {
+ return this.toBuilder().setExtendedErrorInfo(true).build();
+}
+
+/**
+ * Sets the field name for the line field in the returned Row.
+ *
+ * @return {@link JsonToRow}
+ */
+public JsonToRowWithErrFn setLineField(String lineField) {
+ return this.toBuilder().setLineFieldName(lineField).build();
+}
+
+/**
+ * Adds the error message to the returned error Row.
+ *
+ * @return {@link JsonToRow}
+ */
+public JsonToRowWithErrFn setErrorField(String errorField) {
+ if (!this.getExtendedErrorInfo()) {
+throw new IllegalArgumentException(
+"This option is only available with Extended Error Info.");
+ }
+ return this.toBuilder().setErrorFieldName(errorField).build();
+}
+
+@Override
+public ParseResult expand(PCollection jsonStrings) {
+
+ PCollectionTuple result =
+
[jira] [Work logged] (BEAM-10201) Enhance JsonToRow to add Deadletter Support
[ https://issues.apache.org/jira/browse/BEAM-10201?focusedWorklogId=444086&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-444086 ] ASF GitHub Bot logged work on BEAM-10201: - Author: ASF GitHub Bot Created on: 11/Jun/20 02:26 Start Date: 11/Jun/20 02:26 Worklog Time Spent: 10m Work Description: rezarokni commented on pull request #11929: URL: https://github.com/apache/beam/pull/11929#issuecomment-642367042 @TheNeuralBit If you have time, would be great if you could scan the ParseResult and see if the API would suite all the things PubSubIO needs. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 444086) Time Spent: 2h 40m (was: 2.5h) > Enhance JsonToRow to add Deadletter Support > --- > > Key: BEAM-10201 > URL: https://issues.apache.org/jira/browse/BEAM-10201 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core >Reporter: Reza ardeshir rokni >Assignee: Reza ardeshir rokni >Priority: P2 > Time Spent: 2h 40m > Remaining Estimate: 0h > > Current JsonToRow transform does not support Dead Letter pattern on parse > failures. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-10237) Clean up Experimental usage and come up with a graduation policy
Ahmet Altay created BEAM-10237: -- Summary: Clean up Experimental usage and come up with a graduation policy Key: BEAM-10237 URL: https://issues.apache.org/jira/browse/BEAM-10237 Project: Beam Issue Type: Bug Components: beam-model Reporter: Ahmet Altay Beam, APIs use experimental features quite a lot. Experimental features are growing, but at some point we should consider either removing or graduating them. This is a proposal to: - Track: All experimental features will be continually tracked in the change log explicitly. - For features staying in experimental mode longer than 12 months, --> Graduate: we should review whether the feature meets the usage bar to be long term maintained, -- > Deprecate: or should be deprecated to reduce maintenance load. It would be good to add a github ci action to check new usage of experimental tags are added to the change log. /cc [~kenn] [~lcwik] [~robertwb] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-10232) pip check failure: rsa 4.1
[ https://issues.apache.org/jira/browse/BEAM-10232?focusedWorklogId=444082&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-444082 ] ASF GitHub Bot logged work on BEAM-10232: - Author: ASF GitHub Bot Created on: 11/Jun/20 02:04 Start Date: 11/Jun/20 02:04 Worklog Time Spent: 10m Work Description: ibzib commented on pull request #11987: URL: https://github.com/apache/beam/pull/11987#issuecomment-642360956 R: @aaltay @udim This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 444082) Time Spent: 20m (was: 10m) > pip check failure: rsa 4.1 > -- > > Key: BEAM-10232 > URL: https://issues.apache.org/jira/browse/BEAM-10232 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: P2 > Time Spent: 20m > Remaining Estimate: 0h > > Execution failed for task ':sdks:python:test-suites:tox:py37:mypyPy37'. > google-auth 1.16.1 has requirement rsa<4.1,>=3.1.4, but you have rsa 4.1. > ERROR: InvocationError for command > /home/jenkins/jenkins-slave/workspace/beam_PreCommit_PythonLint_Commit/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-mypy/py37-mypy/bin/pip > check (exited with code 1) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9798) Go SplittableDoFn performance testing and optimizations
[ https://issues.apache.org/jira/browse/BEAM-9798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17132836#comment-17132836 ] Daniel Oliveira commented on BEAM-9798: --- Still something I have to look into. > Go SplittableDoFn performance testing and optimizations > --- > > Key: BEAM-9798 > URL: https://issues.apache.org/jira/browse/BEAM-9798 > Project: Beam > Issue Type: Improvement > Components: sdk-go >Reporter: Daniel Oliveira >Assignee: Daniel Oliveira >Priority: P2 > > Add SDF benchmarks and implement performance optimizations to speed up SDFs. > Pretty straightforward. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-9798) Go SplittableDoFn performance testing and optimizations
[ https://issues.apache.org/jira/browse/BEAM-9798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Oliveira reassigned BEAM-9798: - Assignee: Daniel Oliveira > Go SplittableDoFn performance testing and optimizations > --- > > Key: BEAM-9798 > URL: https://issues.apache.org/jira/browse/BEAM-9798 > Project: Beam > Issue Type: Improvement > Components: sdk-go >Reporter: Daniel Oliveira >Assignee: Daniel Oliveira >Priority: P2 > > Add SDF benchmarks and implement performance optimizations to speed up SDFs. > Pretty straightforward. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-10236) beam_PreCommit_PythonDocker_Commit failing
Kyle Weaver created BEAM-10236: -- Summary: beam_PreCommit_PythonDocker_Commit failing Key: BEAM-10236 URL: https://issues.apache.org/jira/browse/BEAM-10236 Project: Beam Issue Type: Bug Components: test-failures Reporter: Kyle Weaver Task :sdks:python:container:py35:docker FAILED RuntimeError: Could not retrieve licences for packages [['3', 'a', 'h', 'p', 's', 'y']] in Python3.5 environment. Package names are scrambled due to BEAM-10234, but I'm assuming it's pysha3. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9753) Use cmp in fullvalue_test.go
[ https://issues.apache.org/jira/browse/BEAM-9753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17132835#comment-17132835 ] Daniel Oliveira commented on BEAM-9753: --- Still on my backlog. > Use cmp in fullvalue_test.go > > > Key: BEAM-9753 > URL: https://issues.apache.org/jira/browse/BEAM-9753 > Project: Beam > Issue Type: Improvement > Components: sdk-go >Reporter: Daniel Oliveira >Assignee: Daniel Oliveira >Priority: P3 > > We could probably update the comparison helpers in > [fullvalue_test.go|https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/runtime/exec/fullvalue_test.go] > to use cmp options and Transformers instead which would make things much > clearer. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-9643) Add user-facing Go SDF documentation.
[ https://issues.apache.org/jira/browse/BEAM-9643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Oliveira reassigned BEAM-9643: - Assignee: Daniel Oliveira > Add user-facing Go SDF documentation. > - > > Key: BEAM-9643 > URL: https://issues.apache.org/jira/browse/BEAM-9643 > Project: Beam > Issue Type: Sub-task > Components: sdk-go >Reporter: Daniel Oliveira >Assignee: Daniel Oliveira >Priority: P2 > Time Spent: 1.5h > Remaining Estimate: 0h > > This means adding the documentation about how to use SDFs and the contracts > of all the SDF methods to the Go SDK code, as well as updating the Go SDF > design doc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-9643) Add user-facing Go SDF documentation.
[ https://issues.apache.org/jira/browse/BEAM-9643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Oliveira resolved BEAM-9643. --- Fix Version/s: Not applicable Resolution: Fixed > Add user-facing Go SDF documentation. > - > > Key: BEAM-9643 > URL: https://issues.apache.org/jira/browse/BEAM-9643 > Project: Beam > Issue Type: Sub-task > Components: sdk-go >Reporter: Daniel Oliveira >Assignee: Daniel Oliveira >Priority: P2 > Fix For: Not applicable > > Time Spent: 1.5h > Remaining Estimate: 0h > > This means adding the documentation about how to use SDFs and the contracts > of all the SDF methods to the Go SDK code, as well as updating the Go SDF > design doc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-9753) Use cmp in fullvalue_test.go
[ https://issues.apache.org/jira/browse/BEAM-9753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Oliveira reassigned BEAM-9753: - Assignee: Daniel Oliveira > Use cmp in fullvalue_test.go > > > Key: BEAM-9753 > URL: https://issues.apache.org/jira/browse/BEAM-9753 > Project: Beam > Issue Type: Improvement > Components: sdk-go >Reporter: Daniel Oliveira >Assignee: Daniel Oliveira >Priority: P3 > > We could probably update the comparison helpers in > [fullvalue_test.go|https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/runtime/exec/fullvalue_test.go] > to use cmp options and Transformers instead which would make things much > clearer. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-9642) Add SDF execution-time runners
[ https://issues.apache.org/jira/browse/BEAM-9642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Oliveira reassigned BEAM-9642: - Assignee: Daniel Oliveira > Add SDF execution-time runners > -- > > Key: BEAM-9642 > URL: https://issues.apache.org/jira/browse/BEAM-9642 > Project: Beam > Issue Type: Sub-task > Components: sdk-go >Reporter: Daniel Oliveira >Assignee: Daniel Oliveira >Priority: P2 > Time Spent: 6h 10m > Remaining Estimate: 0h > > Adds execution-time SDF runner units to the exec package, and any unit tests > + helpers required. > This is needed to get the expanded SDF URNs to execute in the runner harness. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (BEAM-9642) Add SDF execution-time runners
[ https://issues.apache.org/jira/browse/BEAM-9642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on BEAM-9642 started by Daniel Oliveira. - > Add SDF execution-time runners > -- > > Key: BEAM-9642 > URL: https://issues.apache.org/jira/browse/BEAM-9642 > Project: Beam > Issue Type: Sub-task > Components: sdk-go >Reporter: Daniel Oliveira >Assignee: Daniel Oliveira >Priority: P2 > Time Spent: 6h 10m > Remaining Estimate: 0h > > Adds execution-time SDF runner units to the exec package, and any unit tests > + helpers required. > This is needed to get the expanded SDF URNs to execute in the runner harness. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-9642) Add SDF execution-time runners
[ https://issues.apache.org/jira/browse/BEAM-9642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Oliveira resolved BEAM-9642. --- Fix Version/s: Not applicable Resolution: Fixed > Add SDF execution-time runners > -- > > Key: BEAM-9642 > URL: https://issues.apache.org/jira/browse/BEAM-9642 > Project: Beam > Issue Type: Sub-task > Components: sdk-go >Reporter: Daniel Oliveira >Assignee: Daniel Oliveira >Priority: P2 > Fix For: Not applicable > > Time Spent: 6h 10m > Remaining Estimate: 0h > > Adds execution-time SDF runner units to the exec package, and any unit tests > + helpers required. > This is needed to get the expanded SDF URNs to execute in the runner harness. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-9611) Add a test for Iter/ReIter parameters in ParDos (such as after a GBK).
[ https://issues.apache.org/jira/browse/BEAM-9611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Oliveira reassigned BEAM-9611: - Assignee: Daniel Oliveira > Add a test for Iter/ReIter parameters in ParDos (such as after a GBK). > -- > > Key: BEAM-9611 > URL: https://issues.apache.org/jira/browse/BEAM-9611 > Project: Beam > Issue Type: Test > Components: sdk-go >Reporter: Daniel Oliveira >Assignee: Daniel Oliveira >Priority: P2 > > Some bugs popped up recently because DoFn validation caused an error with > ParDos that had Iter/ReIter elements. That could've been caught if a test > existed that checked the creation of ParDos with Iter/ReIters, so that should > be added, either as a new test or as part of some existing integration test. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9611) Add a test for Iter/ReIter parameters in ParDos (such as after a GBK).
[ https://issues.apache.org/jira/browse/BEAM-9611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17132834#comment-17132834 ] Daniel Oliveira commented on BEAM-9611: --- Still on my backlog. > Add a test for Iter/ReIter parameters in ParDos (such as after a GBK). > -- > > Key: BEAM-9611 > URL: https://issues.apache.org/jira/browse/BEAM-9611 > Project: Beam > Issue Type: Test > Components: sdk-go >Reporter: Daniel Oliveira >Assignee: Daniel Oliveira >Priority: P2 > > Some bugs popped up recently because DoFn validation caused an error with > ParDos that had Iter/ReIter elements. That could've been caught if a test > existed that checked the creation of ParDos with Iter/ReIters, so that should > be added, either as a new test or as part of some existing integration test. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8571) Use SDK version in default Go container tag
[ https://issues.apache.org/jira/browse/BEAM-8571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17132833#comment-17132833 ] Daniel Oliveira commented on BEAM-8571: --- Still something on my backlog. > Use SDK version in default Go container tag > --- > > Key: BEAM-8571 > URL: https://issues.apache.org/jira/browse/BEAM-8571 > Project: Beam > Issue Type: Improvement > Components: sdk-go >Reporter: Kyle Weaver >Assignee: Daniel Oliveira >Priority: P2 > > Currently, the Go SDK uses container `apachebeam/go_sdk:latest` by default > [1]. This causes confusion when using locally built containers [2], > especially since images are automatically pulled, meaning the release image > is used instead of the developer's own image (BEAM-8545). > [1] > [https://github.com/apache/beam/blob/473377ef8f51949983508f70663e75ef0ee24a7f/sdks/go/pkg/beam/options/jobopts/options.go#L111] > [[2] > https://lists.apache.org/thread.html/07131e314e229ec60100eaa2c0cf6dfc206bf2b0f78c3cee9ebb0bda@%3Cdev.beam.apache.org%3E|https://lists.apache.org/thread.html/07131e314e229ec60100eaa2c0cf6dfc206bf2b0f78c3cee9ebb0bda@%3Cdev.beam.apache.org%3E] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-8571) Use SDK version in default Go container tag
[ https://issues.apache.org/jira/browse/BEAM-8571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Oliveira reassigned BEAM-8571: - Assignee: Daniel Oliveira > Use SDK version in default Go container tag > --- > > Key: BEAM-8571 > URL: https://issues.apache.org/jira/browse/BEAM-8571 > Project: Beam > Issue Type: Improvement > Components: sdk-go >Reporter: Kyle Weaver >Assignee: Daniel Oliveira >Priority: P2 > > Currently, the Go SDK uses container `apachebeam/go_sdk:latest` by default > [1]. This causes confusion when using locally built containers [2], > especially since images are automatically pulled, meaning the release image > is used instead of the developer's own image (BEAM-8545). > [1] > [https://github.com/apache/beam/blob/473377ef8f51949983508f70663e75ef0ee24a7f/sdks/go/pkg/beam/options/jobopts/options.go#L111] > [[2] > https://lists.apache.org/thread.html/07131e314e229ec60100eaa2c0cf6dfc206bf2b0f78c3cee9ebb0bda@%3Cdev.beam.apache.org%3E|https://lists.apache.org/thread.html/07131e314e229ec60100eaa2c0cf6dfc206bf2b0f78c3cee9ebb0bda@%3Cdev.beam.apache.org%3E] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-6113) Streamline beam.Init() requirement for Go pipelines.
[ https://issues.apache.org/jira/browse/BEAM-6113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17132830#comment-17132830 ] Daniel Oliveira commented on BEAM-6113: --- Still something I want on my backlog. > Streamline beam.Init() requirement for Go pipelines. > > > Key: BEAM-6113 > URL: https://issues.apache.org/jira/browse/BEAM-6113 > Project: Beam > Issue Type: Improvement > Components: sdk-go >Reporter: Daniel Oliveira >Assignee: Daniel Oliveira >Priority: P3 > Labels: documentation, usability > Time Spent: 1h > Remaining Estimate: 0h > > When writing a Go pipeline it's necessary to call beam.Init() at the > beginning of your code, but right now there is nearly no validation or > checking around that. This makes it very easy for a new user of Beam Go to > accidentally leave it out and not get a clear signal for why their pipeline > isn't being executed as expected. This issue is for tracking progress towards > streamlining this requirement to improve the user experience. > In the short term this can be easily improved by improving documentation and > examples to clearly communicate the necessity of beam.Init(), and also add > log warnings to provide some signal if the user forgot to call it. > In the long term, the best solution to this would be to try avoiding the > problem all together (for example by removing the need for beam.Init(), or > having it called implicitly without relying on users to call it), or to > create a stronger failure state, for example by having pipelines without > beam.Init() immediately break with a clear message notifying the user that it > was not called. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-6113) Streamline beam.Init() requirement for Go pipelines.
[ https://issues.apache.org/jira/browse/BEAM-6113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Oliveira reassigned BEAM-6113: - Assignee: Daniel Oliveira > Streamline beam.Init() requirement for Go pipelines. > > > Key: BEAM-6113 > URL: https://issues.apache.org/jira/browse/BEAM-6113 > Project: Beam > Issue Type: Improvement > Components: sdk-go >Reporter: Daniel Oliveira >Assignee: Daniel Oliveira >Priority: P3 > Labels: documentation, usability > Time Spent: 1h > Remaining Estimate: 0h > > When writing a Go pipeline it's necessary to call beam.Init() at the > beginning of your code, but right now there is nearly no validation or > checking around that. This makes it very easy for a new user of Beam Go to > accidentally leave it out and not get a clear signal for why their pipeline > isn't being executed as expected. This issue is for tracking progress towards > streamlining this requirement to improve the user experience. > In the short term this can be easily improved by improving documentation and > examples to clearly communicate the necessity of beam.Init(), and also add > log warnings to provide some signal if the user forgot to call it. > In the long term, the best solution to this would be to try avoiding the > problem all together (for example by removing the need for beam.Init(), or > having it called implicitly without relying on users to call it), or to > create a stronger failure state, for example by having pipelines without > beam.Init() immediately break with a clear message notifying the user that it > was not called. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9882) Go SplittableDoFn testing
[ https://issues.apache.org/jira/browse/BEAM-9882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17132828#comment-17132828 ] Daniel Oliveira commented on BEAM-9882: --- Still working on this. > Go SplittableDoFn testing > - > > Key: BEAM-9882 > URL: https://issues.apache.org/jira/browse/BEAM-9882 > Project: Beam > Issue Type: Improvement > Components: sdk-go >Reporter: Daniel Oliveira >Assignee: Daniel Oliveira >Priority: P2 > Labels: stale-assigned > > This is a catch-all jira for all tasks needed to fully test SplittableDoFns > in Go. For progress on SplittableDoFns themselves, see BEAM-3301 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9883) Generalize SDF-validating restrictions
[ https://issues.apache.org/jira/browse/BEAM-9883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17132827#comment-17132827 ] Daniel Oliveira commented on BEAM-9883: --- This is still something I want to work on, in my backlog. > Generalize SDF-validating restrictions > -- > > Key: BEAM-9883 > URL: https://issues.apache.org/jira/browse/BEAM-9883 > Project: Beam > Issue Type: Sub-task > Components: sdk-go >Reporter: Daniel Oliveira >Assignee: Daniel Oliveira >Priority: P2 > Labels: stale-assigned > Time Spent: 50m > Remaining Estimate: 0h > > We have some restrictions written for the purpose of validating that SDFs > work in sdf_invokers_test.go, but they can be improved and generalized. The > main improvement is changing the validation approach so that the restriction > keeps track of each method it's had called on it. Then this can be > generalized so that it can be used in upcoming integration tests to validate > which methods are being called. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-2937) Fn API combiner support w/ lifting to PGBK
[ https://issues.apache.org/jira/browse/BEAM-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17132826#comment-17132826 ] Daniel Oliveira commented on BEAM-2937: --- I'll see if I can wrap this up. > Fn API combiner support w/ lifting to PGBK > -- > > Key: BEAM-2937 > URL: https://issues.apache.org/jira/browse/BEAM-2937 > Project: Beam > Issue Type: Improvement > Components: beam-model >Reporter: Henning Rohde >Assignee: Daniel Oliveira >Priority: P2 > Labels: portability, stale-assigned > Time Spent: 3h 50m > Remaining Estimate: 0h > > The FnAPI should support this optimization. Detailed design: > https://s.apache.org/beam-runner-api-combine-model > Once design is ready, expand subtasks similarly to BEAM-2822. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-4678) Support portable combiner lifting in Java Flink Runner
[ https://issues.apache.org/jira/browse/BEAM-4678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Oliveira reassigned BEAM-4678: - Assignee: Kyle Weaver (was: Daniel Oliveira) Kyle, any idea if this is already implemented? It seems likely but I wanna confirm before closing this. > Support portable combiner lifting in Java Flink Runner > -- > > Key: BEAM-4678 > URL: https://issues.apache.org/jira/browse/BEAM-4678 > Project: Beam > Issue Type: Sub-task > Components: runner-flink >Reporter: Daniel Oliveira >Assignee: Kyle Weaver >Priority: P2 > Time Spent: 1.5h > Remaining Estimate: 0h > > Adjust Flink Runner to support portable combiner lifting as described in the > following doc: > https://s.apache.org/beam-runner-api-combine-model -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-10232) pip check failure: rsa 4.1
[ https://issues.apache.org/jira/browse/BEAM-10232?focusedWorklogId=444077&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-444077 ] ASF GitHub Bot logged work on BEAM-10232: - Author: ASF GitHub Bot Created on: 11/Jun/20 01:36 Start Date: 11/Jun/20 01:36 Worklog Time Spent: 10m Work Description: ibzib opened a new pull request #11987: URL: https://github.com/apache/beam/pull/11987 **Please** add a meaningful description for your change here Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Samza | Spark --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [![Build Sta
[jira] [Assigned] (BEAM-4678) Support portable combiner lifting in Java Flink Runner
[ https://issues.apache.org/jira/browse/BEAM-4678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Oliveira reassigned BEAM-4678: - Assignee: Daniel Oliveira > Support portable combiner lifting in Java Flink Runner > -- > > Key: BEAM-4678 > URL: https://issues.apache.org/jira/browse/BEAM-4678 > Project: Beam > Issue Type: Sub-task > Components: runner-flink >Reporter: Daniel Oliveira >Assignee: Daniel Oliveira >Priority: P2 > Time Spent: 1.5h > Remaining Estimate: 0h > > Adjust Flink Runner to support portable combiner lifting as described in the > following doc: > https://s.apache.org/beam-runner-api-combine-model -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9951) Create Go SDK synthetic sources.
[
https://issues.apache.org/jira/browse/BEAM-9951?focusedWorklogId=444073&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-444073
]
ASF GitHub Bot logged work on BEAM-9951:
Author: ASF GitHub Bot
Created on: 11/Jun/20 01:16
Start Date: 11/Jun/20 01:16
Worklog Time Spent: 10m
Work Description: youngoli commented on a change in pull request #11870:
URL: https://github.com/apache/beam/pull/11870#discussion_r438490190
##
File path: sdks/go/pkg/beam/testing/passert/count.go
##
@@ -0,0 +1,52 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+//http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package passert
+
+import (
+ "fmt"
+
+ "github.com/apache/beam/sdks/go/pkg/beam"
+ "github.com/apache/beam/sdks/go/pkg/beam/core/typex"
+)
+
+func Count(s beam.Scope, col beam.PCollection, name string, count int) {
+ s = s.Scope(fmt.Sprintf("passert.Count(%v)", name))
+
+ if typex.IsKV(col.Type()) {
+ col = beam.DropKey(s, col)
+ }
+ counted := beam.Combine(s, &elmCountCombineFn{}, col)
+ Equals(s, counted, count)
Review comment:
I thought so, that's what I was referring to. The error message in
passert.Sum (while better than passert.Equals) is still a bit unclear, just
because it specifically mentions passert.Sum instead of passert.Count.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 444073)
Time Spent: 4h 40m (was: 4.5h)
> Create Go SDK synthetic sources.
>
>
> Key: BEAM-9951
> URL: https://issues.apache.org/jira/browse/BEAM-9951
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-go
>Reporter: Daniel Oliveira
>Assignee: Daniel Oliveira
>Priority: P2
> Time Spent: 4h 40m
> Remaining Estimate: 0h
>
> Create synthetic sources for the Go SDK like
> [Java|https://github.com/apache/beam/tree/master/sdks/java/io/synthetic/src/main/java/org/apache/beam/sdk/io/synthetic]
> and
> [Python|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/testing/synthetic_pipeline.py]
> have.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-10235) Create stats transform that counts number of elements in a PCollection.
[ https://issues.apache.org/jira/browse/BEAM-10235?focusedWorklogId=444072&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-444072 ] ASF GitHub Bot logged work on BEAM-10235: - Author: ASF GitHub Bot Created on: 11/Jun/20 01:11 Start Date: 11/Jun/20 01:11 Worklog Time Spent: 10m Work Description: youngoli commented on pull request #11986: URL: https://github.com/apache/beam/pull/11986#issuecomment-642346309 R: @lostluck This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 444072) Time Spent: 20m (was: 10m) > Create stats transform that counts number of elements in a PCollection. > --- > > Key: BEAM-10235 > URL: https://issues.apache.org/jira/browse/BEAM-10235 > Project: Beam > Issue Type: New Feature > Components: sdk-go >Reporter: Daniel Oliveira >Assignee: Daniel Oliveira >Priority: P2 > Time Spent: 20m > Remaining Estimate: 0h > > In the process of implementing https://github.com/apache/beam/pull/11870 it > seemed useful to turn some of the functionality I was working on into a > general transform. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-10235) Create stats transform that counts number of elements in a PCollection.
[ https://issues.apache.org/jira/browse/BEAM-10235?focusedWorklogId=444071&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-444071 ] ASF GitHub Bot logged work on BEAM-10235: - Author: ASF GitHub Bot Created on: 11/Jun/20 01:10 Start Date: 11/Jun/20 01:10 Worklog Time Spent: 10m Work Description: youngoli opened a new pull request #11986: URL: https://github.com/apache/beam/pull/11986 This transform counts the number of elements in a PTransform. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [x] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Samza | Spark --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [![Build
[jira] [Created] (BEAM-10235) Create stats transform that counts number of elements in a PCollection.
Daniel Oliveira created BEAM-10235: -- Summary: Create stats transform that counts number of elements in a PCollection. Key: BEAM-10235 URL: https://issues.apache.org/jira/browse/BEAM-10235 Project: Beam Issue Type: New Feature Components: sdk-go Reporter: Daniel Oliveira Assignee: Daniel Oliveira In the process of implementing https://github.com/apache/beam/pull/11870 it seemed useful to turn some of the functionality I was working on into a general transform. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-10235) Create stats transform that counts number of elements in a PCollection.
[ https://issues.apache.org/jira/browse/BEAM-10235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-10235: --- Status: Open (was: Triage Needed) > Create stats transform that counts number of elements in a PCollection. > --- > > Key: BEAM-10235 > URL: https://issues.apache.org/jira/browse/BEAM-10235 > Project: Beam > Issue Type: New Feature > Components: sdk-go >Reporter: Daniel Oliveira >Assignee: Daniel Oliveira >Priority: P2 > > In the process of implementing https://github.com/apache/beam/pull/11870 it > seemed useful to turn some of the functionality I was working on into a > general transform. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9852) C-ares status is not ARES_SUCCESS: Misformatted domain name
[
https://issues.apache.org/jira/browse/BEAM-9852?focusedWorklogId=444070&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-444070
]
ASF GitHub Bot logged work on BEAM-9852:
Author: ASF GitHub Bot
Created on: 11/Jun/20 00:47
Start Date: 11/Jun/20 00:47
Worklog Time Spent: 10m
Work Description: ibzib commented on pull request #11972:
URL: https://github.com/apache/beam/pull/11972#issuecomment-642340383
Run PythonDocker PreCommit
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 444070)
Time Spent: 2h 20m (was: 2h 10m)
> C-ares status is not ARES_SUCCESS: Misformatted domain name
> ---
>
> Key: BEAM-9852
> URL: https://issues.apache.org/jira/browse/BEAM-9852
> Project: Beam
> Issue Type: Sub-task
> Components: runner-flink, runner-spark
>Reporter: Kyle Weaver
>Priority: P2
> Labels: portability-flink, portability-spark
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> This affects all portable runners (Flink, Spark, Dataflow Python streaming).
> It does not appear to cause pipelines to fail.
> Exception in thread read_grpc_client_inputs:
> Traceback (most recent call last):
> File "/usr/local/lib/python3.7/threading.py", line 926, in _bootstrap_inner
> self.run()
> File "/usr/local/lib/python3.7/threading.py", line 870, in run
> self._target(*self._args, **self._kwargs)
> File
> "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/data_plane.py",
> line 545, in
> target=lambda: self._read_inputs(elements_iterator),
> File
> "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/data_plane.py",
> line 528, in _read_inputs
> for elements in elements_iterator:
> File "/usr/local/lib/python3.7/site-packages/grpc/channel.py", line 388, in
> __next_
> return self._next()
> File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 365, in
> _next
> raise self
> grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
> status = StatusCode.UNAVAILABLE
> details = "DNS resolution failed"
> debug_error_string =
> "{"created":"@1587426512.443144965","description":"Failed to pick
> subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3876,"referenced_errors":[{"created":"@1587426512.443142363","description":"Resolver
> transient
> failure","file":"src/core/ext/filters/client_channel/resolving_lb_policy.cc","file_line":263,"referenced_errors":[{"created":"@1587426512.443141313","description":"DNS
> resolution
> failed","file":"src/core/ext/filters/client_channel/resolver/dns/c_ares/dns_resolver_ares.cc","file_line":357,"grpc_status":14,"referenced_errors":[{"created":"@1587426512.443136986","description":"C-ares
> status is not ARES_SUCCESS: Misformatted domain
> name","file":"src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_wrapper.cc","file_line":244,"referenced_errors":[
> {"created":"@1587426512.443126564","description":"C-ares status is not
> ARES_SUCCESS: Misformatted domain
> name","file":"src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_wrapper.cc","file_line":244}
> ]}]}]}]}"
> >
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9852) C-ares status is not ARES_SUCCESS: Misformatted domain name
[
https://issues.apache.org/jira/browse/BEAM-9852?focusedWorklogId=444069&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-444069
]
ASF GitHub Bot logged work on BEAM-9852:
Author: ASF GitHub Bot
Created on: 11/Jun/20 00:47
Start Date: 11/Jun/20 00:47
Worklog Time Spent: 10m
Work Description: ibzib edited a comment on pull request #11972:
URL: https://github.com/apache/beam/pull/11972#issuecomment-642336725
PythonDocker is failing with a license issue:
```
File "/tmp/license_scripts/pull_licenses_py.py", line 149, in
py_ver=py_ver, license_list=sorted(','.join(no_licenses)),
how_to=how_to))
RuntimeError: Could not retrieve licences for packages [['3', 'a', 'h', 'p',
's', 'y']] in Python3.5 environment.
```
Package names are scrambled due to BEAM-10234, but I'm assuming it's pysha3.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 444069)
Time Spent: 2h 10m (was: 2h)
> C-ares status is not ARES_SUCCESS: Misformatted domain name
> ---
>
> Key: BEAM-9852
> URL: https://issues.apache.org/jira/browse/BEAM-9852
> Project: Beam
> Issue Type: Sub-task
> Components: runner-flink, runner-spark
>Reporter: Kyle Weaver
>Priority: P2
> Labels: portability-flink, portability-spark
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> This affects all portable runners (Flink, Spark, Dataflow Python streaming).
> It does not appear to cause pipelines to fail.
> Exception in thread read_grpc_client_inputs:
> Traceback (most recent call last):
> File "/usr/local/lib/python3.7/threading.py", line 926, in _bootstrap_inner
> self.run()
> File "/usr/local/lib/python3.7/threading.py", line 870, in run
> self._target(*self._args, **self._kwargs)
> File
> "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/data_plane.py",
> line 545, in
> target=lambda: self._read_inputs(elements_iterator),
> File
> "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/data_plane.py",
> line 528, in _read_inputs
> for elements in elements_iterator:
> File "/usr/local/lib/python3.7/site-packages/grpc/channel.py", line 388, in
> __next_
> return self._next()
> File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 365, in
> _next
> raise self
> grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
> status = StatusCode.UNAVAILABLE
> details = "DNS resolution failed"
> debug_error_string =
> "{"created":"@1587426512.443144965","description":"Failed to pick
> subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3876,"referenced_errors":[{"created":"@1587426512.443142363","description":"Resolver
> transient
> failure","file":"src/core/ext/filters/client_channel/resolving_lb_policy.cc","file_line":263,"referenced_errors":[{"created":"@1587426512.443141313","description":"DNS
> resolution
> failed","file":"src/core/ext/filters/client_channel/resolver/dns/c_ares/dns_resolver_ares.cc","file_line":357,"grpc_status":14,"referenced_errors":[{"created":"@1587426512.443136986","description":"C-ares
> status is not ARES_SUCCESS: Misformatted domain
> name","file":"src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_wrapper.cc","file_line":244,"referenced_errors":[
> {"created":"@1587426512.443126564","description":"C-ares status is not
> ARES_SUCCESS: Misformatted domain
> name","file":"src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_wrapper.cc","file_line":244}
> ]}]}]}]}"
> >
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-10169) ParDo* functions should declare the correct output N in their error message
[ https://issues.apache.org/jira/browse/BEAM-10169?focusedWorklogId=444068&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-444068 ] ASF GitHub Bot logged work on BEAM-10169: - Author: ASF GitHub Bot Created on: 11/Jun/20 00:45 Start Date: 11/Jun/20 00:45 Worklog Time Spent: 10m Work Description: codeBehindMe commented on pull request #11976: URL: https://github.com/apache/beam/pull/11976#issuecomment-642339945 R: @damondouglas This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 444068) Time Spent: 1.5h (was: 1h 20m) > ParDo* functions should declare the correct output N in their error message > --- > > Key: BEAM-10169 > URL: https://issues.apache.org/jira/browse/BEAM-10169 > Project: Beam > Issue Type: Improvement > Components: sdk-go >Reporter: Robert Burke >Assignee: Aaron Tillekeratne >Priority: P3 > Labels: noob, starter > Time Spent: 1.5h > Remaining Estimate: 0h > > User report noted the confusion in the error if you use a DoFn with 0 outputs > with beam.ParDo instead of beam.ParDo0. > In that case, a panic stack trace is followed by the cryptic: "expected 1 > output. Found: []" > We can do better. > While we can't change the return signature dynamically (that's for ParDoN > only), we can instead clearly indicate: > * the DoFn in question. > * the number of outputs the DoFn has > * and recommend using ParDo0, ParDo, ParDo2,...ParDo7, or ParDoN, as > appropriate. > https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/pardo.go#L361 > would need to change as well as any of the specific cases that follow. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9852) C-ares status is not ARES_SUCCESS: Misformatted domain name
[
https://issues.apache.org/jira/browse/BEAM-9852?focusedWorklogId=444066&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-444066
]
ASF GitHub Bot logged work on BEAM-9852:
Author: ASF GitHub Bot
Created on: 11/Jun/20 00:33
Start Date: 11/Jun/20 00:33
Worklog Time Spent: 10m
Work Description: ibzib commented on pull request #11972:
URL: https://github.com/apache/beam/pull/11972#issuecomment-642336725
I'm getting a weird license issue too:
```
File "/tmp/license_scripts/pull_licenses_py.py", line 149, in
py_ver=py_ver, license_list=sorted(','.join(no_licenses)),
how_to=how_to))
RuntimeError: Could not retrieve licences for packages [['3', 'a', 'h', 'p',
's', 'y']] in Python3.5 environment.
```
Package names are scrambled due to BEAM-10234, but I'm assuming it's pysha3.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 444066)
Time Spent: 2h (was: 1h 50m)
> C-ares status is not ARES_SUCCESS: Misformatted domain name
> ---
>
> Key: BEAM-9852
> URL: https://issues.apache.org/jira/browse/BEAM-9852
> Project: Beam
> Issue Type: Sub-task
> Components: runner-flink, runner-spark
>Reporter: Kyle Weaver
>Priority: P2
> Labels: portability-flink, portability-spark
> Time Spent: 2h
> Remaining Estimate: 0h
>
> This affects all portable runners (Flink, Spark, Dataflow Python streaming).
> It does not appear to cause pipelines to fail.
> Exception in thread read_grpc_client_inputs:
> Traceback (most recent call last):
> File "/usr/local/lib/python3.7/threading.py", line 926, in _bootstrap_inner
> self.run()
> File "/usr/local/lib/python3.7/threading.py", line 870, in run
> self._target(*self._args, **self._kwargs)
> File
> "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/data_plane.py",
> line 545, in
> target=lambda: self._read_inputs(elements_iterator),
> File
> "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/data_plane.py",
> line 528, in _read_inputs
> for elements in elements_iterator:
> File "/usr/local/lib/python3.7/site-packages/grpc/channel.py", line 388, in
> __next_
> return self._next()
> File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 365, in
> _next
> raise self
> grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
> status = StatusCode.UNAVAILABLE
> details = "DNS resolution failed"
> debug_error_string =
> "{"created":"@1587426512.443144965","description":"Failed to pick
> subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3876,"referenced_errors":[{"created":"@1587426512.443142363","description":"Resolver
> transient
> failure","file":"src/core/ext/filters/client_channel/resolving_lb_policy.cc","file_line":263,"referenced_errors":[{"created":"@1587426512.443141313","description":"DNS
> resolution
> failed","file":"src/core/ext/filters/client_channel/resolver/dns/c_ares/dns_resolver_ares.cc","file_line":357,"grpc_status":14,"referenced_errors":[{"created":"@1587426512.443136986","description":"C-ares
> status is not ARES_SUCCESS: Misformatted domain
> name","file":"src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_wrapper.cc","file_line":244,"referenced_errors":[
> {"created":"@1587426512.443126564","description":"C-ares status is not
> ARES_SUCCESS: Misformatted domain
> name","file":"src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_wrapper.cc","file_line":244}
> ]}]}]}]}"
> >
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-9217) Update DoFn javadoc for schema type translation
[
https://issues.apache.org/jira/browse/BEAM-9217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brian Hulette updated BEAM-9217:
Labels: (was: beam-fixit)
> Update DoFn javadoc for schema type translation
> ---
>
> Key: BEAM-9217
> URL: https://issues.apache.org/jira/browse/BEAM-9217
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Brian Hulette
>Assignee: Brian Hulette
>Priority: P2
> Time Spent: 20m
> Remaining Estimate: 0h
>
> see https://github.com/apache/beam/pull/10702#discussion_r372516471
> The javadoc for {{@Element}} currently states "the argument type must match
> the input type of this DoFn", which is incorrect. We will translate between
> mismatched types if they both have equivalent schemas.
> cc: [~reuvenlax]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-9045) Implement an Ignite runner using Apache Ignite compute grid
[ https://issues.apache.org/jira/browse/BEAM-9045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17132801#comment-17132801 ] Saikat Maitra commented on BEAM-9045: - Thank you [~kenn] I will share PR with initial implementation to get more feedback. Regards, Saikat > Implement an Ignite runner using Apache Ignite compute grid > --- > > Key: BEAM-9045 > URL: https://issues.apache.org/jira/browse/BEAM-9045 > Project: Beam > Issue Type: New Feature > Components: runner-ideas >Reporter: Saikat Maitra >Assignee: Saikat Maitra >Priority: P2 > > Implement an Ignite runner using Apache Ignite compute grid. > Runner guide [https://beam.apache.org/contribute/runner-guide/] > Capability Matrix > [https://beam.apache.org/documentation/runners/capability-matrix/] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9217) Update DoFn javadoc for schema type translation
[
https://issues.apache.org/jira/browse/BEAM-9217?focusedWorklogId=444065&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-444065
]
ASF GitHub Bot logged work on BEAM-9217:
Author: ASF GitHub Bot
Created on: 11/Jun/20 00:29
Start Date: 11/Jun/20 00:29
Worklog Time Spent: 10m
Work Description: TheNeuralBit commented on pull request #11984:
URL: https://github.com/apache/beam/pull/11984#issuecomment-642335762
R: @reuvenlax
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 444065)
Time Spent: 20m (was: 10m)
> Update DoFn javadoc for schema type translation
> ---
>
> Key: BEAM-9217
> URL: https://issues.apache.org/jira/browse/BEAM-9217
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Brian Hulette
>Assignee: Brian Hulette
>Priority: P2
> Labels: beam-fixit
> Time Spent: 20m
> Remaining Estimate: 0h
>
> see https://github.com/apache/beam/pull/10702#discussion_r372516471
> The javadoc for {{@Element}} currently states "the argument type must match
> the input type of this DoFn", which is incorrect. We will translate between
> mismatched types if they both have equivalent schemas.
> cc: [~reuvenlax]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9217) Update DoFn javadoc for schema type translation
[ https://issues.apache.org/jira/browse/BEAM-9217?focusedWorklogId=444064&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-444064 ] ASF GitHub Bot logged work on BEAM-9217: Author: ASF GitHub Bot Created on: 11/Jun/20 00:29 Start Date: 11/Jun/20 00:29 Worklog Time Spent: 10m Work Description: TheNeuralBit opened a new pull request #11984: URL: https://github.com/apache/beam/pull/11984 Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Samza | Spark --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)[ Implement an Ignite runner using Apache Ignite compute grid
[ https://issues.apache.org/jira/browse/BEAM-9045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saikat Maitra updated BEAM-9045: Labels: (was: stale-assigned) > Implement an Ignite runner using Apache Ignite compute grid > --- > > Key: BEAM-9045 > URL: https://issues.apache.org/jira/browse/BEAM-9045 > Project: Beam > Issue Type: New Feature > Components: runner-ideas >Reporter: Saikat Maitra >Assignee: Saikat Maitra >Priority: P2 > > Implement an Ignite runner using Apache Ignite compute grid. > Runner guide [https://beam.apache.org/contribute/runner-guide/] > Capability Matrix > [https://beam.apache.org/documentation/runners/capability-matrix/] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-10234) Packages missing licenses are not listed correctly
[ https://issues.apache.org/jira/browse/BEAM-10234?focusedWorklogId=444063&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-444063 ] ASF GitHub Bot logged work on BEAM-10234: - Author: ASF GitHub Bot Created on: 11/Jun/20 00:24 Start Date: 11/Jun/20 00:24 Worklog Time Spent: 10m Work Description: ibzib opened a new pull request #11983: URL: https://github.com/apache/beam/pull/11983 I also yapf'd it while I was at it. R: @udim Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Samza | Spark --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [ Update DoFn javadoc for schema type translation
[
https://issues.apache.org/jira/browse/BEAM-9217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brian Hulette updated BEAM-9217:
Labels: beam-fixit (was: )
> Update DoFn javadoc for schema type translation
> ---
>
> Key: BEAM-9217
> URL: https://issues.apache.org/jira/browse/BEAM-9217
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Brian Hulette
>Assignee: Brian Hulette
>Priority: P2
> Labels: beam-fixit
>
> see https://github.com/apache/beam/pull/10702#discussion_r372516471
> The javadoc for {{@Element}} currently states "the argument type must match
> the input type of this DoFn", which is incorrect. We will translate between
> mismatched types if they both have equivalent schemas.
> cc: [~reuvenlax]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (BEAM-9217) Update DoFn javadoc for schema type translation
[
https://issues.apache.org/jira/browse/BEAM-9217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brian Hulette reassigned BEAM-9217:
---
Assignee: Brian Hulette
> Update DoFn javadoc for schema type translation
> ---
>
> Key: BEAM-9217
> URL: https://issues.apache.org/jira/browse/BEAM-9217
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Brian Hulette
>Assignee: Brian Hulette
>Priority: P2
>
> see https://github.com/apache/beam/pull/10702#discussion_r372516471
> The javadoc for {{@Element}} currently states "the argument type must match
> the input type of this DoFn", which is incorrect. We will translate between
> mismatched types if they both have equivalent schemas.
> cc: [~reuvenlax]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-6892) Use temp_location for BQ FILE_LOADS on DirectRunner, and autocreate it in GCS if not specified by user.
[ https://issues.apache.org/jira/browse/BEAM-6892?focusedWorklogId=444057&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-444057 ] ASF GitHub Bot logged work on BEAM-6892: Author: ASF GitHub Bot Created on: 11/Jun/20 00:16 Start Date: 11/Jun/20 00:16 Worklog Time Spent: 10m Work Description: pabloem commented on pull request #11982: URL: https://github.com/apache/beam/pull/11982#issuecomment-642331818 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 444057) Time Spent: 9.5h (was: 9h 20m) > Use temp_location for BQ FILE_LOADS on DirectRunner, and autocreate it in GCS > if not specified by user. > --- > > Key: BEAM-6892 > URL: https://issues.apache.org/jira/browse/BEAM-6892 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Priority: P2 > Time Spent: 9.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-6892) Use temp_location for BQ FILE_LOADS on DirectRunner, and autocreate it in GCS if not specified by user.
[ https://issues.apache.org/jira/browse/BEAM-6892?focusedWorklogId=444059&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-444059 ] ASF GitHub Bot logged work on BEAM-6892: Author: ASF GitHub Bot Created on: 11/Jun/20 00:16 Start Date: 11/Jun/20 00:16 Worklog Time Spent: 10m Work Description: pabloem commented on pull request #11982: URL: https://github.com/apache/beam/pull/11982#issuecomment-642331957 Run Python2_PVR_Flink PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 444059) Time Spent: 9h 50m (was: 9h 40m) > Use temp_location for BQ FILE_LOADS on DirectRunner, and autocreate it in GCS > if not specified by user. > --- > > Key: BEAM-6892 > URL: https://issues.apache.org/jira/browse/BEAM-6892 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Priority: P2 > Time Spent: 9h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-6892) Use temp_location for BQ FILE_LOADS on DirectRunner, and autocreate it in GCS if not specified by user.
[ https://issues.apache.org/jira/browse/BEAM-6892?focusedWorklogId=444058&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-444058 ] ASF GitHub Bot logged work on BEAM-6892: Author: ASF GitHub Bot Created on: 11/Jun/20 00:16 Start Date: 11/Jun/20 00:16 Worklog Time Spent: 10m Work Description: pabloem commented on pull request #11982: URL: https://github.com/apache/beam/pull/11982#issuecomment-642331889 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 444058) Time Spent: 9h 40m (was: 9.5h) > Use temp_location for BQ FILE_LOADS on DirectRunner, and autocreate it in GCS > if not specified by user. > --- > > Key: BEAM-6892 > URL: https://issues.apache.org/jira/browse/BEAM-6892 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Priority: P2 > Time Spent: 9h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-10201) Enhance JsonToRow to add Deadletter Support
[ https://issues.apache.org/jira/browse/BEAM-10201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brian Hulette updated BEAM-10201: - Status: Open (was: Triage Needed) > Enhance JsonToRow to add Deadletter Support > --- > > Key: BEAM-10201 > URL: https://issues.apache.org/jira/browse/BEAM-10201 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core >Reporter: Reza ardeshir rokni >Priority: P2 > Time Spent: 2.5h > Remaining Estimate: 0h > > Current JsonToRow transform does not support Dead Letter pattern on parse > failures. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-10201) Enhance JsonToRow to add Deadletter Support
[ https://issues.apache.org/jira/browse/BEAM-10201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brian Hulette reassigned BEAM-10201: Assignee: Reza ardeshir rokni > Enhance JsonToRow to add Deadletter Support > --- > > Key: BEAM-10201 > URL: https://issues.apache.org/jira/browse/BEAM-10201 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core >Reporter: Reza ardeshir rokni >Assignee: Reza ardeshir rokni >Priority: P2 > Time Spent: 2.5h > Remaining Estimate: 0h > > Current JsonToRow transform does not support Dead Letter pattern on parse > failures. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-10201) Enhance JsonToRow to add Deadletter Support
[ https://issues.apache.org/jira/browse/BEAM-10201?focusedWorklogId=444056&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-444056 ] ASF GitHub Bot logged work on BEAM-10201: - Author: ASF GitHub Bot Created on: 11/Jun/20 00:12 Start Date: 11/Jun/20 00:12 Worklog Time Spent: 10m Work Description: TheNeuralBit commented on pull request #11929: URL: https://github.com/apache/beam/pull/11929#issuecomment-642330786 FYI `PubsubJsonTableProvider` has support for writing to a dead letter pubsub topic: https://github.com/apache/beam/blob/d5dd47b47cbdf0739ac1a28cb8fbd06becbdbae7/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubIOJsonTable.java#L160. Maybe there's some duplicate logic we can remove after this PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 444056) Time Spent: 2.5h (was: 2h 20m) > Enhance JsonToRow to add Deadletter Support > --- > > Key: BEAM-10201 > URL: https://issues.apache.org/jira/browse/BEAM-10201 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core >Reporter: Reza ardeshir rokni >Priority: P2 > Time Spent: 2.5h > Remaining Estimate: 0h > > Current JsonToRow transform does not support Dead Letter pattern on parse > failures. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-10234) Packages missing licenses are not listed correctly
[ https://issues.apache.org/jira/browse/BEAM-10234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-10234: --- Status: Open (was: Triage Needed) > Packages missing licenses are not listed correctly > -- > > Key: BEAM-10234 > URL: https://issues.apache.org/jira/browse/BEAM-10234 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: P2 > > e.g. > RuntimeError: Could not retrieve licences for packages [[',', ',', ',', ',', > ',', ',', ',', '-', '-', '-', '4', 'a', 'a', 'a', 'a', 'a', 'a', 'b', 'b', > 'c', 'c', 'd', 'e', 'e', 'e', 'e', 'e', 'e', 'f', 'f', 'f', 'g', 'g', 'i', > 'i', 'i', 'i', 'i', 'i', 'i', 'k', 'l', 'l', 'm', 'm', 'm', 'm', 'n', 'n', > 'o', 'o', 'o', 'o', 'o', 'o', 'o', 'o', 'p', 'p', 'p', 'p', 'p', 'p', 'p', > 'p', 'p', 'r', 'r', 'r', 'r', 'r', 's', 's', 's', 's', 't', 't', 't', 't', > 't', 't', 't', 'u', 'u', 'u', 'u', 'u', 'w', 'x', 'y', 'y', 'y', 'z']] in > Python3.7 environment. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-10234) Packages missing licenses are not listed correctly
Kyle Weaver created BEAM-10234: -- Summary: Packages missing licenses are not listed correctly Key: BEAM-10234 URL: https://issues.apache.org/jira/browse/BEAM-10234 Project: Beam Issue Type: Bug Components: build-system Reporter: Kyle Weaver Assignee: Kyle Weaver e.g. RuntimeError: Could not retrieve licences for packages [[',', ',', ',', ',', ',', ',', ',', '-', '-', '-', '4', 'a', 'a', 'a', 'a', 'a', 'a', 'b', 'b', 'c', 'c', 'd', 'e', 'e', 'e', 'e', 'e', 'e', 'f', 'f', 'f', 'g', 'g', 'i', 'i', 'i', 'i', 'i', 'i', 'i', 'k', 'l', 'l', 'm', 'm', 'm', 'm', 'n', 'n', 'o', 'o', 'o', 'o', 'o', 'o', 'o', 'o', 'p', 'p', 'p', 'p', 'p', 'p', 'p', 'p', 'p', 'r', 'r', 'r', 'r', 'r', 's', 's', 's', 's', 't', 't', 't', 't', 't', 't', 't', 'u', 'u', 'u', 'u', 'u', 'w', 'x', 'y', 'y', 'y', 'z']] in Python3.7 environment. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-10233) Return error message/information with existing FAILED_ROW data from BigQueryWriteFn (Python SDK)
Tom Hardman created BEAM-10233: -- Summary: Return error message/information with existing FAILED_ROW data from BigQueryWriteFn (Python SDK) Key: BEAM-10233 URL: https://issues.apache.org/jira/browse/BEAM-10233 Project: Beam Issue Type: Improvement Components: io-py-gcp Reporter: Tom Hardman A user may call `apache_beam.io.gcp.bigquery.WriteToBigQuery` to write their streamed data to BQ. I any rows fail to write, this will return a tagged pcollection `BigQueryWriteFn.FAILED_ROWS`. This data includes a tuple `(destination_table, failed_row_payload)`. My suggestion is to include the error information in the `FAILED_ROWS` pcollection. From the source code we can see that we have access to the error information, e.g. that the row failed because field `id` was `invalid` because `this field is not a record`. I think we should surface this to the user. I'm happy to open a PR this myself (as I've already had to overwrite the original code in several projects), but it looks like we'd need a breaking change by either extending the tuple which would cause unpacking issues in existing code, or by returning a different data structure entirely. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-6892) Use temp_location for BQ FILE_LOADS on DirectRunner, and autocreate it in GCS if not specified by user.
[ https://issues.apache.org/jira/browse/BEAM-6892?focusedWorklogId=444054&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-444054 ] ASF GitHub Bot logged work on BEAM-6892: Author: ASF GitHub Bot Created on: 11/Jun/20 00:02 Start Date: 11/Jun/20 00:02 Worklog Time Spent: 10m Work Description: pabloem commented on pull request #11982: URL: https://github.com/apache/beam/pull/11982#issuecomment-642327786 I've tested this on my machine This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 444054) Time Spent: 9h 20m (was: 9h 10m) > Use temp_location for BQ FILE_LOADS on DirectRunner, and autocreate it in GCS > if not specified by user. > --- > > Key: BEAM-6892 > URL: https://issues.apache.org/jira/browse/BEAM-6892 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Priority: P2 > Time Spent: 9h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-10220) Add support for implicit nulls in de/serializing JSON
[ https://issues.apache.org/jira/browse/BEAM-10220?focusedWorklogId=444052&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-444052 ] ASF GitHub Bot logged work on BEAM-10220: - Author: ASF GitHub Bot Created on: 11/Jun/20 00:00 Start Date: 11/Jun/20 00:00 Worklog Time Spent: 10m Work Description: rezarokni commented on pull request #11955: URL: https://github.com/apache/beam/pull/11955#issuecomment-642327425 Thanx for FYI @pabloem . This is awesome! we are also looking to add deadletter pattern to JsonToRow: https://github.com/apache/beam/pull/11929 The null support your adding is great! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 444052) Time Spent: 2h 50m (was: 2h 40m) > Add support for implicit nulls in de/serializing JSON > - > > Key: BEAM-10220 > URL: https://issues.apache.org/jira/browse/BEAM-10220 > Project: Beam > Issue Type: New Feature > Components: extensions-java-json >Reporter: Reuben van Ammers >Assignee: Reuben van Ammers >Priority: P3 > Time Spent: 2h 50m > Remaining Estimate: 0h > > Currently, RowJson.java doesn't have support for deserialising JSON with > fields missing - it fails, even if the beam schema is nullable. This is often > a problem for reading JSON as nulls are very commonly represented by omitting > the field. Additionally, for this reason, not writing nulls is a reasonable > when converting from Beam rows to JSON. > This would help to allow https://issues.apache.org/jira/browse/BEAM-7624 to > be implemented. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-10201) Enhance JsonToRow to add Deadletter Support
[
https://issues.apache.org/jira/browse/BEAM-10201?focusedWorklogId=444053&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-444053
]
ASF GitHub Bot logged work on BEAM-10201:
-
Author: ASF GitHub Bot
Created on: 11/Jun/20 00:01
Start Date: 11/Jun/20 00:01
Worklog Time Spent: 10m
Work Description: rezarokni commented on a change in pull request #11929:
URL: https://github.com/apache/beam/pull/11929#discussion_r438470939
##
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/JsonToRow.java
##
@@ -72,10 +79,44 @@
@Experimental(Kind.SCHEMAS)
public class JsonToRow {
+ private static final String LINE_FIELD_NAME = "line";
+ private static final String ERROR_FIELD_NAME = "err";
+
+ public static final Schema ERROR_ROW_SCHEMA =
+ Schema.of(
+ Field.of(LINE_FIELD_NAME, FieldType.STRING),
+ Field.of(ERROR_FIELD_NAME, FieldType.STRING));
+
+ public static final TupleTag MAIN_TUPLE_TAG = new TupleTag() {};
+ public static final TupleTag DEAD_LETTER_TUPLE_TAG = new
TupleTag() {};
+
Review comment:
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 444053)
Time Spent: 2h 20m (was: 2h 10m)
> Enhance JsonToRow to add Deadletter Support
> ---
>
> Key: BEAM-10201
> URL: https://issues.apache.org/jira/browse/BEAM-10201
> Project: Beam
> Issue Type: New Feature
> Components: sdk-java-core
>Reporter: Reza ardeshir rokni
>Priority: P2
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> Current JsonToRow transform does not support Dead Letter pattern on parse
> failures.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-6892) Use temp_location for BQ FILE_LOADS on DirectRunner, and autocreate it in GCS if not specified by user.
[ https://issues.apache.org/jira/browse/BEAM-6892?focusedWorklogId=444050&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-444050 ] ASF GitHub Bot logged work on BEAM-6892: Author: ASF GitHub Bot Created on: 10/Jun/20 23:59 Start Date: 10/Jun/20 23:59 Worklog Time Spent: 10m Work Description: pabloem commented on pull request #11982: URL: https://github.com/apache/beam/pull/11982#issuecomment-642327079 r: @udim This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 444050) Time Spent: 9h 10m (was: 9h) > Use temp_location for BQ FILE_LOADS on DirectRunner, and autocreate it in GCS > if not specified by user. > --- > > Key: BEAM-6892 > URL: https://issues.apache.org/jira/browse/BEAM-6892 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Priority: P2 > Time Spent: 9h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-6892) Use temp_location for BQ FILE_LOADS on DirectRunner, and autocreate it in GCS if not specified by user.
[ https://issues.apache.org/jira/browse/BEAM-6892?focusedWorklogId=444049&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-444049 ] ASF GitHub Bot logged work on BEAM-6892: Author: ASF GitHub Bot Created on: 10/Jun/20 23:59 Start Date: 10/Jun/20 23:59 Worklog Time Spent: 10m Work Description: pabloem opened a new pull request #11982: URL: https://github.com/apache/beam/pull/11982 Following the logic from Java (https://github.com/apache/beam/blob/b56740f0e8cd80c2873412847d0b336837429fb9/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/options/GcpOptions.java#L332). Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Samza | Spark --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[ Add support for implicit nulls in de/serializing JSON
[ https://issues.apache.org/jira/browse/BEAM-10220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brian Hulette reassigned BEAM-10220: Assignee: Reuben van Ammers > Add support for implicit nulls in de/serializing JSON > - > > Key: BEAM-10220 > URL: https://issues.apache.org/jira/browse/BEAM-10220 > Project: Beam > Issue Type: New Feature > Components: extensions-java-json >Reporter: Reuben van Ammers >Assignee: Reuben van Ammers >Priority: P3 > Time Spent: 2h 40m > Remaining Estimate: 0h > > Currently, RowJson.java doesn't have support for deserialising JSON with > fields missing - it fails, even if the beam schema is nullable. This is often > a problem for reading JSON as nulls are very commonly represented by omitting > the field. Additionally, for this reason, not writing nulls is a reasonable > when converting from Beam rows to JSON. > This would help to allow https://issues.apache.org/jira/browse/BEAM-7624 to > be implemented. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-10220) Add support for implicit nulls in de/serializing JSON
[ https://issues.apache.org/jira/browse/BEAM-10220?focusedWorklogId=444048&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-444048 ] ASF GitHub Bot logged work on BEAM-10220: - Author: ASF GitHub Bot Created on: 10/Jun/20 23:53 Start Date: 10/Jun/20 23:53 Worklog Time Spent: 10m Work Description: pabloem commented on pull request #11955: URL: https://github.com/apache/beam/pull/11955#issuecomment-642325514 and assigned BEAM-7624 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 444048) Time Spent: 2h 40m (was: 2.5h) > Add support for implicit nulls in de/serializing JSON > - > > Key: BEAM-10220 > URL: https://issues.apache.org/jira/browse/BEAM-10220 > Project: Beam > Issue Type: New Feature > Components: extensions-java-json >Reporter: Reuben van Ammers >Priority: P3 > Time Spent: 2h 40m > Remaining Estimate: 0h > > Currently, RowJson.java doesn't have support for deserialising JSON with > fields missing - it fails, even if the beam schema is nullable. This is often > a problem for reading JSON as nulls are very commonly represented by omitting > the field. Additionally, for this reason, not writing nulls is a reasonable > when converting from Beam rows to JSON. > This would help to allow https://issues.apache.org/jira/browse/BEAM-7624 to > be implemented. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-10220) Add support for implicit nulls in de/serializing JSON
[ https://issues.apache.org/jira/browse/BEAM-10220?focusedWorklogId=444047&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-444047 ] ASF GitHub Bot logged work on BEAM-10220: - Author: ASF GitHub Bot Created on: 10/Jun/20 23:53 Start Date: 10/Jun/20 23:53 Worklog Time Spent: 10m Work Description: pabloem commented on pull request #11955: URL: https://github.com/apache/beam/pull/11955#issuecomment-642325342 Added @reubenvanammers as contributor! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 444047) Time Spent: 2.5h (was: 2h 20m) > Add support for implicit nulls in de/serializing JSON > - > > Key: BEAM-10220 > URL: https://issues.apache.org/jira/browse/BEAM-10220 > Project: Beam > Issue Type: New Feature > Components: extensions-java-json >Reporter: Reuben van Ammers >Priority: P3 > Time Spent: 2.5h > Remaining Estimate: 0h > > Currently, RowJson.java doesn't have support for deserialising JSON with > fields missing - it fails, even if the beam schema is nullable. This is often > a problem for reading JSON as nulls are very commonly represented by omitting > the field. Additionally, for this reason, not writing nulls is a reasonable > when converting from Beam rows to JSON. > This would help to allow https://issues.apache.org/jira/browse/BEAM-7624 to > be implemented. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-7624) Missing field support in JsonToRow
[ https://issues.apache.org/jira/browse/BEAM-7624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pablo Estrada reassigned BEAM-7624: --- Assignee: Reuben van Ammers > Missing field support in JsonToRow > -- > > Key: BEAM-7624 > URL: https://issues.apache.org/jira/browse/BEAM-7624 > Project: Beam > Issue Type: Improvement > Components: extensions-java-json >Reporter: Anton Kedin >Assignee: Reuben van Ammers >Priority: P3 > > [JsonToRow|https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/JsonToRow.java] > supports only strict schema matching, so it expects all the fields that are > declared by the schema to be present in the input JSON data. If the field is > missing, [it > crashes|https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/util/RowJsonDeserializer.java#L114-L117]. > In practice it is useful to default to null is such cases. We should make it > configurable in Beam Java SDK at least, and probably in Beam SQL (in Pubsub > Json TableProvider) as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-10220) Add support for implicit nulls in de/serializing JSON
[ https://issues.apache.org/jira/browse/BEAM-10220?focusedWorklogId=444046&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-444046 ] ASF GitHub Bot logged work on BEAM-10220: - Author: ASF GitHub Bot Created on: 10/Jun/20 23:52 Start Date: 10/Jun/20 23:52 Worklog Time Spent: 10m Work Description: pabloem commented on pull request #11955: URL: https://github.com/apache/beam/pull/11955#issuecomment-642325145 fyi @rezarokni @reuvenlax This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 444046) Time Spent: 2h 20m (was: 2h 10m) > Add support for implicit nulls in de/serializing JSON > - > > Key: BEAM-10220 > URL: https://issues.apache.org/jira/browse/BEAM-10220 > Project: Beam > Issue Type: New Feature > Components: extensions-java-json >Reporter: Reuben van Ammers >Priority: P3 > Time Spent: 2h 20m > Remaining Estimate: 0h > > Currently, RowJson.java doesn't have support for deserialising JSON with > fields missing - it fails, even if the beam schema is nullable. This is often > a problem for reading JSON as nulls are very commonly represented by omitting > the field. Additionally, for this reason, not writing nulls is a reasonable > when converting from Beam rows to JSON. > This would help to allow https://issues.apache.org/jira/browse/BEAM-7624 to > be implemented. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-10220) Add support for implicit nulls in de/serializing JSON
[ https://issues.apache.org/jira/browse/BEAM-10220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brian Hulette updated BEAM-10220: - Status: Open (was: Triage Needed) > Add support for implicit nulls in de/serializing JSON > - > > Key: BEAM-10220 > URL: https://issues.apache.org/jira/browse/BEAM-10220 > Project: Beam > Issue Type: New Feature > Components: extensions-java-json >Reporter: Reuben van Ammers >Priority: P3 > Time Spent: 2h 10m > Remaining Estimate: 0h > > Currently, RowJson.java doesn't have support for deserialising JSON with > fields missing - it fails, even if the beam schema is nullable. This is often > a problem for reading JSON as nulls are very commonly represented by omitting > the field. Additionally, for this reason, not writing nulls is a reasonable > when converting from Beam rows to JSON. > This would help to allow https://issues.apache.org/jira/browse/BEAM-7624 to > be implemented. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-10220) Add support for implicit nulls in de/serializing JSON
[ https://issues.apache.org/jira/browse/BEAM-10220?focusedWorklogId=444045&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-444045 ] ASF GitHub Bot logged work on BEAM-10220: - Author: ASF GitHub Bot Created on: 10/Jun/20 23:46 Start Date: 10/Jun/20 23:46 Worklog Time Spent: 10m Work Description: TheNeuralBit commented on pull request #11955: URL: https://github.com/apache/beam/pull/11955#issuecomment-642323660 I went to assign BEAM-7624 to you, but it looks like you need to be added as a Beam contributor first. @pabloem could you add Reuben on jira? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 444045) Time Spent: 2h 10m (was: 2h) > Add support for implicit nulls in de/serializing JSON > - > > Key: BEAM-10220 > URL: https://issues.apache.org/jira/browse/BEAM-10220 > Project: Beam > Issue Type: New Feature > Components: extensions-java-json >Reporter: Reuben van Ammers >Priority: P3 > Time Spent: 2h 10m > Remaining Estimate: 0h > > Currently, RowJson.java doesn't have support for deserialising JSON with > fields missing - it fails, even if the beam schema is nullable. This is often > a problem for reading JSON as nulls are very commonly represented by omitting > the field. Additionally, for this reason, not writing nulls is a reasonable > when converting from Beam rows to JSON. > This would help to allow https://issues.apache.org/jira/browse/BEAM-7624 to > be implemented. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8364) SchemaCoder inconsistent equality behavior for POJO
[ https://issues.apache.org/jira/browse/BEAM-8364?focusedWorklogId=444044&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-444044 ] ASF GitHub Bot logged work on BEAM-8364: Author: ASF GitHub Bot Created on: 10/Jun/20 23:42 Start Date: 10/Jun/20 23:42 Worklog Time Spent: 10m Work Description: TheNeuralBit opened a new pull request #11981: URL: https://github.com/apache/beam/pull/11981 R: @reuvenlax CC: @nevillelyh Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Samza | Spark --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)[ SchemaCoder inconsistent equality behavior for POJO
[
https://issues.apache.org/jira/browse/BEAM-8364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brian Hulette updated BEAM-8364:
Labels: beam-fixit (was: )
> SchemaCoder inconsistent equality behavior for POJO
> ---
>
> Key: BEAM-8364
> URL: https://issues.apache.org/jira/browse/BEAM-8364
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql, sdk-java-core
>Affects Versions: 2.16.0
>Reporter: Neville Li
>Assignee: Brian Hulette
>Priority: P3
> Labels: beam-fixit
> Time Spent: 10m
> Remaining Estimate: 0h
>
> One can create a {{SchemaCoder}} for arbitrary type {{T}} with
> {{SchemaCoder.of(schema, toRowFunction, fromRowFunction)}}. However, in cases
> where {{T}} lacks proper equality behavior, i.e. POJO, the result coder still
> returns true for {{consistentWithEquals}} and {{structuralValue}}s that fail
> equality check.
> This test reproduces the issue.
> {code:java}
> import org.apache.beam.sdk.schemas.Schema;
> import org.apache.beam.sdk.schemas.SchemaCoder;
> import org.apache.beam.sdk.values.Row;
> import org.junit.Test;
> import org.junit.runner.RunWith;
> import org.junit.runners.JUnit4;
> import java.nio.charset.Charset;
> import static org.junit.Assert.*;
> @RunWith(JUnit4.class)
> public class SchemaCoderTest {
> public static class Pojo {
> private final byte[] bytes;
> private final String id;
> public Pojo(byte[] bytes, String id) {
> this.bytes = bytes;
> this.id = id;
> }
> public byte[] getBytes() {
> return bytes;
> }
> public String getId() {
> return id;
> }
> }
> @Test
> public void testCoder() {
> Schema schema =
> Schema.builder().addByteArrayField("bytes").addStringField("id").build();
> SchemaCoder coder = SchemaCoder.of(
> schema,
> t -> Row.withSchema(schema).addValues(t.getBytes(),
> t.getId()).build(),
> r -> new Pojo(r.getBytes("bytes"), r.getString("id")));
> Pojo p1 = new Pojo("hello".getBytes(Charset.forName("UTF-8")), "world");
> Pojo p2 = new Pojo("hello".getBytes(Charset.forName("UTF-8")), "world");
> assertNotEquals(p1, p2); // EXPECTED, p1.equals(p2) == false
> assertFalse(coder.consistentWithEquals()); // FAIL, returns true
> assertEquals(coder.structuralValue(p1), coder.structuralValue(p2)); //
> FAIL
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9564) Remove insecure ssl options from MongoDBIO
[ https://issues.apache.org/jira/browse/BEAM-9564?focusedWorklogId=444042&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-444042 ] ASF GitHub Bot logged work on BEAM-9564: Author: ASF GitHub Bot Created on: 10/Jun/20 23:35 Start Date: 10/Jun/20 23:35 Worklog Time Spent: 10m Work Description: TheNeuralBit opened a new pull request #11980: URL: https://github.com/apache/beam/pull/11980 Adds batching of schema'd PCollections into dataframes based on BatchElements transform. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Samza | Spark --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCo
[jira] [Work logged] (BEAM-9564) Remove insecure ssl options from MongoDBIO
[ https://issues.apache.org/jira/browse/BEAM-9564?focusedWorklogId=444043&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-444043 ] ASF GitHub Bot logged work on BEAM-9564: Author: ASF GitHub Bot Created on: 10/Jun/20 23:35 Start Date: 10/Jun/20 23:35 Worklog Time Spent: 10m Work Description: TheNeuralBit commented on pull request #11980: URL: https://github.com/apache/beam/pull/11980#issuecomment-642320833 CC: @robertwb This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 444043) Time Spent: 2h (was: 1h 50m) > Remove insecure ssl options from MongoDBIO > -- > > Key: BEAM-9564 > URL: https://issues.apache.org/jira/browse/BEAM-9564 > Project: Beam > Issue Type: Improvement > Components: io-java-mongodb >Affects Versions: 2.21.0 >Reporter: Ismaël Mejía >Assignee: Ismaël Mejía >Priority: P1 > Labels: backward-incompatible > Time Spent: 2h > Remaining Estimate: 0h > > The option MongoDBIO.withIgnoreSSLCertificate and > withSSLInvalidHostNameAllowedslInvalidHostNameAllowed() are insecure by > design. We should not encourage users to be able to use them so better to > remove these options. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9852) C-ares status is not ARES_SUCCESS: Misformatted domain name
[
https://issues.apache.org/jira/browse/BEAM-9852?focusedWorklogId=444041&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-444041
]
ASF GitHub Bot logged work on BEAM-9852:
Author: ASF GitHub Bot
Created on: 10/Jun/20 23:33
Start Date: 10/Jun/20 23:33
Worklog Time Spent: 10m
Work Description: ibzib commented on pull request #11972:
URL: https://github.com/apache/beam/pull/11972#issuecomment-642320097
retest this please
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 444041)
Time Spent: 1h 50m (was: 1h 40m)
> C-ares status is not ARES_SUCCESS: Misformatted domain name
> ---
>
> Key: BEAM-9852
> URL: https://issues.apache.org/jira/browse/BEAM-9852
> Project: Beam
> Issue Type: Sub-task
> Components: runner-flink, runner-spark
>Reporter: Kyle Weaver
>Priority: P2
> Labels: portability-flink, portability-spark
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> This affects all portable runners (Flink, Spark, Dataflow Python streaming).
> It does not appear to cause pipelines to fail.
> Exception in thread read_grpc_client_inputs:
> Traceback (most recent call last):
> File "/usr/local/lib/python3.7/threading.py", line 926, in _bootstrap_inner
> self.run()
> File "/usr/local/lib/python3.7/threading.py", line 870, in run
> self._target(*self._args, **self._kwargs)
> File
> "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/data_plane.py",
> line 545, in
> target=lambda: self._read_inputs(elements_iterator),
> File
> "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/data_plane.py",
> line 528, in _read_inputs
> for elements in elements_iterator:
> File "/usr/local/lib/python3.7/site-packages/grpc/channel.py", line 388, in
> __next_
> return self._next()
> File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 365, in
> _next
> raise self
> grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
> status = StatusCode.UNAVAILABLE
> details = "DNS resolution failed"
> debug_error_string =
> "{"created":"@1587426512.443144965","description":"Failed to pick
> subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3876,"referenced_errors":[{"created":"@1587426512.443142363","description":"Resolver
> transient
> failure","file":"src/core/ext/filters/client_channel/resolving_lb_policy.cc","file_line":263,"referenced_errors":[{"created":"@1587426512.443141313","description":"DNS
> resolution
> failed","file":"src/core/ext/filters/client_channel/resolver/dns/c_ares/dns_resolver_ares.cc","file_line":357,"grpc_status":14,"referenced_errors":[{"created":"@1587426512.443136986","description":"C-ares
> status is not ARES_SUCCESS: Misformatted domain
> name","file":"src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_wrapper.cc","file_line":244,"referenced_errors":[
> {"created":"@1587426512.443126564","description":"C-ares status is not
> ARES_SUCCESS: Misformatted domain
> name","file":"src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_wrapper.cc","file_line":244}
> ]}]}]}]}"
> >
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (BEAM-8364) SchemaCoder inconsistent equality behavior for POJO
[
https://issues.apache.org/jira/browse/BEAM-8364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brian Hulette reassigned BEAM-8364:
---
Assignee: Brian Hulette
> SchemaCoder inconsistent equality behavior for POJO
> ---
>
> Key: BEAM-8364
> URL: https://issues.apache.org/jira/browse/BEAM-8364
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql, sdk-java-core
>Affects Versions: 2.16.0
>Reporter: Neville Li
>Assignee: Brian Hulette
>Priority: P3
>
> One can create a {{SchemaCoder}} for arbitrary type {{T}} with
> {{SchemaCoder.of(schema, toRowFunction, fromRowFunction)}}. However, in cases
> where {{T}} lacks proper equality behavior, i.e. POJO, the result coder still
> returns true for {{consistentWithEquals}} and {{structuralValue}}s that fail
> equality check.
> This test reproduces the issue.
> {code:java}
> import org.apache.beam.sdk.schemas.Schema;
> import org.apache.beam.sdk.schemas.SchemaCoder;
> import org.apache.beam.sdk.values.Row;
> import org.junit.Test;
> import org.junit.runner.RunWith;
> import org.junit.runners.JUnit4;
> import java.nio.charset.Charset;
> import static org.junit.Assert.*;
> @RunWith(JUnit4.class)
> public class SchemaCoderTest {
> public static class Pojo {
> private final byte[] bytes;
> private final String id;
> public Pojo(byte[] bytes, String id) {
> this.bytes = bytes;
> this.id = id;
> }
> public byte[] getBytes() {
> return bytes;
> }
> public String getId() {
> return id;
> }
> }
> @Test
> public void testCoder() {
> Schema schema =
> Schema.builder().addByteArrayField("bytes").addStringField("id").build();
> SchemaCoder coder = SchemaCoder.of(
> schema,
> t -> Row.withSchema(schema).addValues(t.getBytes(),
> t.getId()).build(),
> r -> new Pojo(r.getBytes("bytes"), r.getString("id")));
> Pojo p1 = new Pojo("hello".getBytes(Charset.forName("UTF-8")), "world");
> Pojo p2 = new Pojo("hello".getBytes(Charset.forName("UTF-8")), "world");
> assertNotEquals(p1, p2); // EXPECTED, p1.equals(p2) == false
> assertFalse(coder.consistentWithEquals()); // FAIL, returns true
> assertEquals(coder.structuralValue(p1), coder.structuralValue(p2)); //
> FAIL
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (BEAM-6972) LTS backport: CassandraIO is broken because of use of bad relocation of guava
[
https://issues.apache.org/jira/browse/BEAM-6972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles resolved BEAM-6972.
---
Assignee: Kenneth Knowles
Resolution: Fixed
> LTS backport: CassandraIO is broken because of use of bad relocation of guava
> -
>
> Key: BEAM-6972
> URL: https://issues.apache.org/jira/browse/BEAM-6972
> Project: Beam
> Issue Type: Bug
> Components: io-java-cassandra
>Affects Versions: 2.5.0, 2.6.0, 2.7.0, 2.8.0, 2.9.0, 2.10.0, 2.11.0
>Reporter: Arun sethia
>Assignee: Kenneth Knowles
>Priority: P2
> Fix For: 2.7.1
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> While using apache beam to run dataflow job to read data from BigQuery and
> Store/Write to Cassandra with following libaries:
> # beam-sdks-java-io-cassandra - 2.6.0
> # beam-sdks-java-io-jdbc - 2.6.0
> # beam-sdks-java-io-google-cloud-platform - 2.6.0
> # beam-sdks-java-core - 2.6.0
> # google-cloud-dataflow-java-sdk-all - 2.5.0
> # google-api-client -1.25.0
>
> I am getting following error at the time insert/save data to Cassandra.
> {code:java}
> [error] (run-main-0) org.apache.beam.sdk.Pipeline$PipelineExecutionException:
> java.lang.NoSuchMethodError:
> com.datastax.driver.mapping.Mapper.saveAsync(Ljava/lang/Object;)Lorg/apache/beam/repackaged/beam_sdks_java_io_cassandra/com/google/common/util/concurrent/ListenableFuture;
> org.apache.beam.sdk.Pipeline$PipelineExecutionException:
> java.lang.NoSuchMethodError:
> com.datastax.driver.mapping.Mapper.saveAsync(Ljava/lang/Object;)Lorg/apache/beam/repackaged/beam_sdks_java_io_cassandra/com/google/common/util/concurrent/ListenableFuture;
> at
> org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:332)
> at
> org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:302)
> at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:197)
> at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:64)
> at org.apache.beam.sdk.Pipeline.run(Pipeline.java:313)
> at org.apache.beam.sdk.Pipeline.run(Pipeline.java:299){code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (BEAM-3197) Jenkins asfgit is commenting "SUCCESS\n\n--none--" and "FAILURE\n\n--none--" again
[ https://issues.apache.org/jira/browse/BEAM-3197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles closed BEAM-3197. - Fix Version/s: Not applicable Resolution: Fixed > Jenkins asfgit is commenting "SUCCESS\n\n--none--" and "FAILURE\n\n--none--" > again > -- > > Key: BEAM-3197 > URL: https://issues.apache.org/jira/browse/BEAM-3197 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Kenneth Knowles >Priority: P2 > Labels: stale-P2 > Fix For: Not applicable > > > Example: https://github.com/apache/beam/pull/4087 > If I recall, this is because of a parent config getting merged in a > particular way with our own config. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-3194) Support annotating that a DoFn requires stable / deterministic input for replay/retry
[ https://issues.apache.org/jira/browse/BEAM-3194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17132771#comment-17132771 ] Kenneth Knowles commented on BEAM-3194: --- [~robinyqiu] This Jira is "done" in the sense that the annotation exists, right? Perhaps the name is just not quite right. The annotation exists, but various runners have varying levels of support. > Support annotating that a DoFn requires stable / deterministic input for > replay/retry > - > > Key: BEAM-3194 > URL: https://issues.apache.org/jira/browse/BEAM-3194 > Project: Beam > Issue Type: New Feature > Components: beam-model, sdk-java-core >Reporter: Kenneth Knowles >Priority: P2 > Labels: stale-P2 > Time Spent: 3h 10m > Remaining Estimate: 0h > > See the thread: > https://lists.apache.org/thread.html/5fd81ce371aeaf642665348f8e6940e308e04275dd7072f380f9f945@%3Cdev.beam.apache.org%3E > We need this in order to have truly cross-runner end-to-end exactly once via > replay + idempotence. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-3168) Capability matrix contains runners that aren't in the left nav or on master
[ https://issues.apache.org/jira/browse/BEAM-3168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17132770#comment-17132770 ] Kenneth Knowles commented on BEAM-3168: --- [~tysonjh] Some potential low-hanging fruit as you remove Gearpump and Apex: the runners that are still on branches really shouldn't be in the public capability matrix. Not sure if this is just a couple more lines to delete or if it is a rabbit hole. > Capability matrix contains runners that aren't in the left nav or on master > --- > > Key: BEAM-3168 > URL: https://issues.apache.org/jira/browse/BEAM-3168 > Project: Beam > Issue Type: Bug > Components: website >Reporter: Kenneth Knowles >Priority: P2 > Labels: stale-P2 > > Currently the runners on feature branches are in the capability matrix, but > not linked to their summary pages or anything. Now we also have IBM Streams > that could use a link so users can find what the column is about. > I think we can link to either Ongoing Projects or external pages for these. > We may want a left nav item "other runners" for those that we don't want to > highlight as part of what the Beam project builds and ships, so we can > actually put them in context a bit. This might tie in to BEAM-2888. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9679) Core Transforms | Go SDK Code Katas
[ https://issues.apache.org/jira/browse/BEAM-9679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Damon Douglas updated BEAM-9679: Description: A kata devoted to core beam transforms patterns after [https://github.com/apache/beam/tree/master/learning/katas/java/Core%20Transforms] where the take away is an individual's ability to master the following using an Apache Beam pipeline using the Golang SDK. ||Transform||Pull Request||Status|| |Map|[11564|https://github.com/apache/beam/pull/11564]|Closed| |GroupByKey|[11734|https://github.com/apache/beam/pull/11734]|Closed| |CoGroupByKey|[11803|https://github.com/apache/beam/pull/11803]|Closed| |Combine Simple Function|[11866|https://github.com/apache/beam/pull/11866]|Closed| |CombineFn|[11883|https://github.com/apache/beam/pull/11883]|Closed| |CombinePerKey|[11936|https://github.com/apache/beam/pull/11936]|Closed| |Flatten|[11806|https://github.com/apache/beam/pull/11806]|Closed| |Partition|[11979|https://github.com/apache/beam/pull/11979]|Open| |Side Input| | | |Side Output| | | |Branching| | | |Composite Transform| | | |DoFn Additional Parameters| | | was: A kata devoted to core beam transforms patterns after [https://github.com/apache/beam/tree/master/learning/katas/java/Core%20Transforms] where the take away is an individual's ability to master the following using an Apache Beam pipeline using the Golang SDK. ||Transform||Pull Request||Status|| |Map|[11564|https://github.com/apache/beam/pull/11564]|Closed| |GroupByKey|[11734|https://github.com/apache/beam/pull/11734]|Closed| |CoGroupByKey|[11803|https://github.com/apache/beam/pull/11803]|Closed| |Combine Simple Function|[11866|https://github.com/apache/beam/pull/11866]|Closed| |CombineFn|[11883|https://github.com/apache/beam/pull/11883]|Closed| |CombinePerKey|[11936|https://github.com/apache/beam/pull/11936]|Closed| |Flatten|[11806|https://github.com/apache/beam/pull/11806]|Closed| |Partition| | | |Side Input| | | |Side Output| | | |Branching| | | |Composite Transform| | | |DoFn Additional Parameters| | | > Core Transforms | Go SDK Code Katas > --- > > Key: BEAM-9679 > URL: https://issues.apache.org/jira/browse/BEAM-9679 > Project: Beam > Issue Type: Sub-task > Components: katas, sdk-go >Reporter: Damon Douglas >Assignee: Damon Douglas >Priority: P2 > Time Spent: 10.5h > Remaining Estimate: 0h > > A kata devoted to core beam transforms patterns after > [https://github.com/apache/beam/tree/master/learning/katas/java/Core%20Transforms] > where the take away is an individual's ability to master the following using > an Apache Beam pipeline using the Golang SDK. > > ||Transform||Pull Request||Status|| > |Map|[11564|https://github.com/apache/beam/pull/11564]|Closed| > |GroupByKey|[11734|https://github.com/apache/beam/pull/11734]|Closed| > |CoGroupByKey|[11803|https://github.com/apache/beam/pull/11803]|Closed| > |Combine Simple > Function|[11866|https://github.com/apache/beam/pull/11866]|Closed| > |CombineFn|[11883|https://github.com/apache/beam/pull/11883]|Closed| > |CombinePerKey|[11936|https://github.com/apache/beam/pull/11936]|Closed| > |Flatten|[11806|https://github.com/apache/beam/pull/11806]|Closed| > |Partition|[11979|https://github.com/apache/beam/pull/11979]|Open| > |Side Input| | | > |Side Output| | | > |Branching| | | > |Composite Transform| | | > |DoFn Additional Parameters| | | -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9679) Core Transforms | Go SDK Code Katas
[ https://issues.apache.org/jira/browse/BEAM-9679?focusedWorklogId=444037&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-444037 ] ASF GitHub Bot logged work on BEAM-9679: Author: ASF GitHub Bot Created on: 10/Jun/20 22:40 Start Date: 10/Jun/20 22:40 Worklog Time Spent: 10m Work Description: damondouglas opened a new pull request #11979: URL: https://github.com/apache/beam/pull/11979 This pull requests adds a Partition lesson to the Go SDK katas. I would like to request the following reviewers: (R: @lostluck ) (R: @henryken ) If accepted by both reviewers, please wait until the [Stepik course](https://stepik.org/course/70387) is updated before finally merging this PR. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [x] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Samza | Spark --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[ Consider infer:eradicate for null analysis
[ https://issues.apache.org/jira/browse/BEAM-6556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17132769#comment-17132769 ] Kenneth Knowles commented on BEAM-6556: --- I did consider it, but I think my top priority for nullness is checker framework. > Consider infer:eradicate for null analysis > -- > > Key: BEAM-6556 > URL: https://issues.apache.org/jira/browse/BEAM-6556 > Project: Beam > Issue Type: Test > Components: build-system >Reporter: Kenneth Knowles >Priority: P2 > Fix For: Not applicable > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
