[jira] [Work logged] (BEAM-9894) Add batch SnowflakeIO.Write to Java SDK
[
https://issues.apache.org/jira/browse/BEAM-9894?focusedWorklogId=439346=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-439346
]
ASF GitHub Bot logged work on BEAM-9894:
Author: ASF GitHub Bot
Created on: 01/Jun/20 05:42
Start Date: 01/Jun/20 05:42
Worklog Time Spent: 10m
Work Description: purbanow commented on a change in pull request #11794:
URL: https://github.com/apache/beam/pull/11794#discussion_r433050696
##
File path:
sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/SnowflakeIO.java
##
@@ -447,6 +494,346 @@ public void populateDisplayData(DisplayData.Builder
builder) {
}
}
+ /** Implementation of {@link #write()}. */
+ @AutoValue
+ public abstract static class Write extends PTransform,
PDone> {
+@Nullable
+abstract SerializableFunction getDataSourceProviderFn();
+
+@Nullable
+abstract String getTable();
+
+@Nullable
+abstract String getQuery();
+
+@Nullable
+abstract Location getLocation();
+
+@Nullable
+abstract String getFileNameTemplate();
+
+@Nullable
+abstract WriteDisposition getWriteDisposition();
+
+@Nullable
+abstract UserDataMapper getUserDataMapper();
+
+@Nullable
+abstract SnowflakeService getSnowflakeService();
+
+abstract Builder toBuilder();
+
+@AutoValue.Builder
+abstract static class Builder {
+ abstract Builder setDataSourceProviderFn(
+ SerializableFunction dataSourceProviderFn);
+
+ abstract Builder setTable(String table);
+
+ abstract Builder setQuery(String query);
+
+ abstract Builder setLocation(Location location);
+
+ abstract Builder setFileNameTemplate(String fileNameTemplate);
+
+ abstract Builder setUserDataMapper(UserDataMapper userDataMapper);
+
+ abstract Builder setWriteDisposition(WriteDisposition
writeDisposition);
+
+ abstract Builder setSnowflakeService(SnowflakeService
snowflakeService);
+
+ abstract Write build();
+}
+
+/**
+ * Setting information about Snowflake server.
+ *
+ * @param config - An instance of {@link DataSourceConfiguration}.
+ */
+public Write withDataSourceConfiguration(final DataSourceConfiguration
config) {
+ return withDataSourceProviderFn(new
DataSourceProviderFromDataSourceConfiguration(config));
+}
+
+/**
+ * Setting function that will provide {@link DataSourceConfiguration} in
runtime.
+ *
+ * @param dataSourceProviderFn a {@link SerializableFunction}.
+ */
+public Write withDataSourceProviderFn(
+SerializableFunction dataSourceProviderFn) {
+ return toBuilder().setDataSourceProviderFn(dataSourceProviderFn).build();
+}
+
+/**
+ * A table name to be written in Snowflake.
+ *
+ * @param table - String with the name of the table.
+ */
+public Write to(String table) {
+ return toBuilder().setTable(table).build();
+}
+
+/**
+ * A query to be executed in Snowflake.
+ *
+ * @param query - String with query.
+ */
+public Write withQueryTransformation(String query) {
+ return toBuilder().setQuery(query).build();
+}
+
+/**
+ * A location object which contains connection config between Snowflake
and GCP.
+ *
+ * @param location - an instance of {@link Location}.
+ */
+public Write via(Location location) {
+ return toBuilder().setLocation(location).build();
+}
+
+/**
+ * A template name for files saved to GCP.
+ *
+ * @param fileNameTemplate - String with template name for files.
+ */
+public Write withFileNameTemplate(String fileNameTemplate) {
+ return toBuilder().setFileNameTemplate(fileNameTemplate).build();
+}
+
+/**
+ * User-defined function mapping user data into CSV lines.
+ *
+ * @param userDataMapper - an instance of {@link UserDataMapper}.
+ */
+public Write withUserDataMapper(UserDataMapper userDataMapper) {
+ return toBuilder().setUserDataMapper(userDataMapper).build();
+}
+
+/**
+ * A disposition to be used during writing to table phase.
+ *
+ * @param writeDisposition - an instance of {@link WriteDisposition}.
+ */
+public Write withWriteDisposition(WriteDisposition writeDisposition) {
+ return toBuilder().setWriteDisposition(writeDisposition).build();
+}
+
+/**
+ * A snowflake service which is supposed to be used. Note: Currently we
have {@link
+ * SnowflakeServiceImpl} with corresponding {@link
FakeSnowflakeServiceImpl} used for testing.
+ *
+ * @param snowflakeService - an instance of {@link SnowflakeService}.
+ */
+public Write withSnowflakeService(SnowflakeService snowflakeService) {
+ return toBuilder().setSnowflakeService(snowflakeService).build();
+}
+
+@Override
+public PDone
[jira] [Assigned] (BEAM-10161) Fix a typo in avroio_test.py
[ https://issues.apache.org/jira/browse/BEAM-10161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Valentyn Tymofieiev reassigned BEAM-10161: -- Assignee: Valentyn Tymofieiev > Fix a typo in avroio_test.py > > > Key: BEAM-10161 > URL: https://issues.apache.org/jira/browse/BEAM-10161 > Project: Beam > Issue Type: Bug > Components: io-py-avro >Reporter: Valentyn Tymofieiev >Assignee: Valentyn Tymofieiev >Priority: P4 > > Creating this issue for illustrative purposes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-8494) Python 3.8 Support
[ https://issues.apache.org/jira/browse/BEAM-8494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Valentyn Tymofieiev reassigned BEAM-8494: - Assignee: Valentyn Tymofieiev > Python 3.8 Support > -- > > Key: BEAM-8494 > URL: https://issues.apache.org/jira/browse/BEAM-8494 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: Valentyn Tymofieiev >Priority: P2 > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-8494) Python 3.8 Support
[ https://issues.apache.org/jira/browse/BEAM-8494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Valentyn Tymofieiev reassigned BEAM-8494: - Assignee: (was: Valentyn Tymofieiev) > Python 3.8 Support > -- > > Key: BEAM-8494 > URL: https://issues.apache.org/jira/browse/BEAM-8494 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Priority: P2 > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-10161) Fix a typo in avroio_test.py
Valentyn Tymofieiev created BEAM-10161: -- Summary: Fix a typo in avroio_test.py Key: BEAM-10161 URL: https://issues.apache.org/jira/browse/BEAM-10161 Project: Beam Issue Type: Bug Components: io-py-avro Reporter: Valentyn Tymofieiev Creating this issue for illustrative purposes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9653) remove _AvroSource in favor of using _FastAvroSource
[ https://issues.apache.org/jira/browse/BEAM-9653?focusedWorklogId=439295=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-439295 ] ASF GitHub Bot logged work on BEAM-9653: Author: ASF GitHub Bot Created on: 31/May/20 22:02 Start Date: 31/May/20 22:02 Worklog Time Spent: 10m Work Description: stale[bot] commented on pull request #11279: URL: https://github.com/apache/beam/pull/11279#issuecomment-636536791 This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the d...@beam.apache.org list. Thank you for your contributions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 439295) Time Spent: 1h 10m (was: 1h) > remove _AvroSource in favor of using _FastAvroSource > - > > Key: BEAM-9653 > URL: https://issues.apache.org/jira/browse/BEAM-9653 > Project: Beam > Issue Type: Improvement > Components: io-py-avro >Affects Versions: 2.19.0 >Reporter: Sayed Mohammad Hossein Torabi >Priority: P3 > Time Spent: 1h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-3083) BigQueryIO.write() with DynamicDestinations should not call getSchema() on every element
[ https://issues.apache.org/jira/browse/BEAM-3083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chamikara Madhusanka Jayalath updated BEAM-3083: Labels: testlabel (was: ) > BigQueryIO.write() with DynamicDestinations should not call getSchema() on > every element > > > Key: BEAM-3083 > URL: https://issues.apache.org/jira/browse/BEAM-3083 > Project: Beam > Issue Type: Bug > Components: io-java-gcp >Reporter: Eugene Kirpichov >Priority: P2 > Labels: testlabel > > https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/CreateTables.java#L103 > calls getSchema() on every element. This might be very expensive if the user > expects that this will be called about once per destination. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-10027) Support for Kotlin-based Beam Katas
[ https://issues.apache.org/jira/browse/BEAM-10027?focusedWorklogId=439248=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-439248 ] ASF GitHub Bot logged work on BEAM-10027: - Author: ASF GitHub Bot Created on: 31/May/20 14:19 Start Date: 31/May/20 14:19 Worklog Time Spent: 10m Work Description: henryken commented on pull request #11761: URL: https://github.com/apache/beam/pull/11761#issuecomment-636477642 Thanks @rionmonster! @pabloem, please help to merge. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 439248) Time Spent: 13h 20m (was: 13h 10m) > Support for Kotlin-based Beam Katas > --- > > Key: BEAM-10027 > URL: https://issues.apache.org/jira/browse/BEAM-10027 > Project: Beam > Issue Type: Improvement > Components: katas >Reporter: Rion Williams >Assignee: Rion Williams >Priority: P2 > Original Estimate: 8h > Time Spent: 13h 20m > Remaining Estimate: 0h > > Currently, there are a series of examples available demonstrating the use of > Apache Beam with Kotlin. It would be nice to have support for the same Beam > Katas that exist for Python, Go, and Java to also support Kotlin. > The port itself shouldn't be that involved since it can still target the JVM, > so it would likely just require the inclusion for Kotlin dependencies and a > conversion for all of the existing Java examples. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-10027) Support for Kotlin-based Beam Katas
[ https://issues.apache.org/jira/browse/BEAM-10027?focusedWorklogId=439244=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-439244 ] ASF GitHub Bot logged work on BEAM-10027: - Author: ASF GitHub Bot Created on: 31/May/20 13:47 Start Date: 31/May/20 13:47 Worklog Time Spent: 10m Work Description: rionmonster commented on pull request #11761: URL: https://github.com/apache/beam/pull/11761#issuecomment-636473860 @henryken / @pabloem I just pushed up the changes you mentioned regarding the Map placeholders, so they should be good to go now. I've also updated the course on Stepik (and included those updates within the commit as well). I think we are good to go now. Thanks for all of your help Henry! Rion This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 439244) Time Spent: 13h 10m (was: 13h) > Support for Kotlin-based Beam Katas > --- > > Key: BEAM-10027 > URL: https://issues.apache.org/jira/browse/BEAM-10027 > Project: Beam > Issue Type: Improvement > Components: katas >Reporter: Rion Williams >Assignee: Rion Williams >Priority: P2 > Original Estimate: 8h > Time Spent: 13h 10m > Remaining Estimate: 0h > > Currently, there are a series of examples available demonstrating the use of > Apache Beam with Kotlin. It would be nice to have support for the same Beam > Katas that exist for Python, Go, and Java to also support Kotlin. > The port itself shouldn't be that involved since it can still target the JVM, > so it would likely just require the inclusion for Kotlin dependencies and a > conversion for all of the existing Java examples. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-10027) Support for Kotlin-based Beam Katas
[ https://issues.apache.org/jira/browse/BEAM-10027?focusedWorklogId=439242=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-439242 ] ASF GitHub Bot logged work on BEAM-10027: - Author: ASF GitHub Bot Created on: 31/May/20 13:46 Start Date: 31/May/20 13:46 Worklog Time Spent: 10m Work Description: rionmonster commented on a change in pull request #11761: URL: https://github.com/apache/beam/pull/11761#discussion_r432948905 ## File path: learning/katas/kotlin/Core Transforms/Map/MapElements/task-info.yaml ## @@ -0,0 +1,13 @@ +type: edu +files: +- name: src/org/apache/beam/learning/katas/coretransforms/map/mapelements/Task.kt + visible: true + placeholders: Review comment: Done (and checked through the remaining Map tasks to ensure they aligned). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 439242) Time Spent: 13h (was: 12h 50m) > Support for Kotlin-based Beam Katas > --- > > Key: BEAM-10027 > URL: https://issues.apache.org/jira/browse/BEAM-10027 > Project: Beam > Issue Type: Improvement > Components: katas >Reporter: Rion Williams >Assignee: Rion Williams >Priority: P2 > Original Estimate: 8h > Time Spent: 13h > Remaining Estimate: 0h > > Currently, there are a series of examples available demonstrating the use of > Apache Beam with Kotlin. It would be nice to have support for the same Beam > Katas that exist for Python, Go, and Java to also support Kotlin. > The port itself shouldn't be that involved since it can still target the JVM, > so it would likely just require the inclusion for Kotlin dependencies and a > conversion for all of the existing Java examples. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (BEAM-3489) Expose the message id of received messages within PubsubMessage
[ https://issues.apache.org/jira/browse/BEAM-3489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120343#comment-17120343 ] Thinh Ha edited comment on BEAM-3489 at 5/31/20, 12:23 PM: --- Hi [~reuvenlax], thanks for your feedback. I've added a new PR based on your comment: [https://github.com/apache/beam/pull/11873] Let me know if there is a test I should be running to check. was (Author: thinhha): Hi [~reuvenlax], thanks for your feedback. Could you advise what needs to change? In this PR the parseFn is always non-null, e.g. [https://github.com/apache/beam/commit/f1d02e9c6a741088479c9221af4c0250d5678b1f#diff-cfdca1a326496dabb9a075878204894fR471] [https://github.com/apache/beam/commit/f1d02e9c6a741088479c9221af4c0250d5678b1f#diff-cfdca1a326496dabb9a075878204894fR499] which should trigger this branch of code that returns the PubsubMessage [https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/PubsubReader.java#L120] I can think of changing setNeedsAttributes to true in [https://github.com/apache/beam/commit/f1d02e9c6a741088479c9221af4c0250d5678b1f#diff-cfdca1a326496dabb9a075878204894fR472] However, it appears the test against this did not fail? [https://github.com/apache/beam/commit/f1d02e9c6a741088479c9221af4c0250d5678b1f#diff-598fd709155261a38592b0612939551aR73] > Expose the message id of received messages within PubsubMessage > --- > > Key: BEAM-3489 > URL: https://issues.apache.org/jira/browse/BEAM-3489 > Project: Beam > Issue Type: New Feature > Components: io-java-gcp >Reporter: Luke Cwik >Assignee: Thinh Ha >Priority: P3 > Labels: newbie, starter > Time Spent: 8h 50m > Remaining Estimate: 0h > > This task is about passing forward the message id from the pubsub proto to > the java PubsubMessage. > Add a message id field to PubsubMessage. > Update the coder for PubsubMessage to encode the message id. > Update the translation from the Pubsub proto message to the Dataflow message: > https://github.com/apache/beam/blob/2e275264b21db45787833502e5e42907b05e28b8/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubUnboundedSource.java#L976 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3489) Expose the message id of received messages within PubsubMessage
[ https://issues.apache.org/jira/browse/BEAM-3489?focusedWorklogId=439228=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-439228 ] ASF GitHub Bot logged work on BEAM-3489: Author: ASF GitHub Bot Created on: 31/May/20 12:17 Start Date: 31/May/20 12:17 Worklog Time Spent: 10m Work Description: thinhha commented on pull request #11873: URL: https://github.com/apache/beam/pull/11873#issuecomment-636463454 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 439228) Time Spent: 8h 50m (was: 8h 40m) > Expose the message id of received messages within PubsubMessage > --- > > Key: BEAM-3489 > URL: https://issues.apache.org/jira/browse/BEAM-3489 > Project: Beam > Issue Type: New Feature > Components: io-java-gcp >Reporter: Luke Cwik >Assignee: Thinh Ha >Priority: P3 > Labels: newbie, starter > Time Spent: 8h 50m > Remaining Estimate: 0h > > This task is about passing forward the message id from the pubsub proto to > the java PubsubMessage. > Add a message id field to PubsubMessage. > Update the coder for PubsubMessage to encode the message id. > Update the translation from the Pubsub proto message to the Dataflow message: > https://github.com/apache/beam/blob/2e275264b21db45787833502e5e42907b05e28b8/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubUnboundedSource.java#L976 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3489) Expose the message id of received messages within PubsubMessage
[ https://issues.apache.org/jira/browse/BEAM-3489?focusedWorklogId=439226=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-439226 ] ASF GitHub Bot logged work on BEAM-3489: Author: ASF GitHub Bot Created on: 31/May/20 12:16 Start Date: 31/May/20 12:16 Worklog Time Spent: 10m Work Description: thinhha commented on pull request #11873: URL: https://github.com/apache/beam/pull/11873#issuecomment-636463293 Run Dataflow ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 439226) Time Spent: 8h 40m (was: 8.5h) > Expose the message id of received messages within PubsubMessage > --- > > Key: BEAM-3489 > URL: https://issues.apache.org/jira/browse/BEAM-3489 > Project: Beam > Issue Type: New Feature > Components: io-java-gcp >Reporter: Luke Cwik >Assignee: Thinh Ha >Priority: P3 > Labels: newbie, starter > Time Spent: 8h 40m > Remaining Estimate: 0h > > This task is about passing forward the message id from the pubsub proto to > the java PubsubMessage. > Add a message id field to PubsubMessage. > Update the coder for PubsubMessage to encode the message id. > Update the translation from the Pubsub proto message to the Dataflow message: > https://github.com/apache/beam/blob/2e275264b21db45787833502e5e42907b05e28b8/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubUnboundedSource.java#L976 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3489) Expose the message id of received messages within PubsubMessage
[ https://issues.apache.org/jira/browse/BEAM-3489?focusedWorklogId=439225=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-439225 ] ASF GitHub Bot logged work on BEAM-3489: Author: ASF GitHub Bot Created on: 31/May/20 12:15 Start Date: 31/May/20 12:15 Worklog Time Spent: 10m Work Description: thinhha opened a new pull request #11873: URL: https://github.com/apache/beam/pull/11873 Make sure PUBSUB_SERIALIZED_ATTRIBUTES_FN is set if getNeedsMessageId is true. Based on commend on https://issues.apache.org/jira/browse/BEAM-3489?focusedCommentId=17110444=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17110444. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[![Build
[jira] [Work logged] (BEAM-9679) Core Transforms | Go SDK Code Katas
[
https://issues.apache.org/jira/browse/BEAM-9679?focusedWorklogId=439199=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-439199
]
ASF GitHub Bot logged work on BEAM-9679:
Author: ASF GitHub Bot
Created on: 31/May/20 10:07
Start Date: 31/May/20 10:07
Worklog Time Spent: 10m
Work Description: henryken commented on a change in pull request #11866:
URL: https://github.com/apache/beam/pull/11866#discussion_r432930378
##
File path: learning/katas/go/Core Transforms/Combine/Simple
Function/pkg/task/task.go
##
@@ -0,0 +1,26 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+//http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package task
+
+import (
+ "github.com/apache/beam/sdks/go/pkg/beam"
+)
+
+func ApplyTransform(s beam.Scope, input beam.PCollection) beam.PCollection {
+ return beam.Combine(s, func(sum, item int) int {
Review comment:
Not major, but I think "item" can be renamed to "num" or "elem"?
##
File path: learning/katas/go/Core Transforms/Combine/Simple
Function/pkg/task/task.go
##
@@ -0,0 +1,26 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+//http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package task
+
+import (
+ "github.com/apache/beam/sdks/go/pkg/beam"
+)
+
+func ApplyTransform(s beam.Scope, input beam.PCollection) beam.PCollection {
+ return beam.Combine(s, func(sum, item int) int {
Review comment:
Not major, can we exclude the "return" from the answer placeholder to
make it consistent with other similar type of tasks?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 439199)
Time Spent: 6h 20m (was: 6h 10m)
> Core Transforms | Go SDK Code Katas
> ---
>
> Key: BEAM-9679
> URL: https://issues.apache.org/jira/browse/BEAM-9679
> Project: Beam
> Issue Type: Sub-task
> Components: katas, sdk-go
>Reporter: Damon Douglas
>Assignee: Damon Douglas
>Priority: P2
> Time Spent: 6h 20m
> Remaining Estimate: 0h
>
> A kata devoted to core beam transforms patterns after
> [https://github.com/apache/beam/tree/master/learning/katas/java/Core%20Transforms]
> where the take away is an individual's ability to master the following using
> an Apache Beam pipeline using the Golang SDK.
>
> ||Transform||Pull Request||Status||
> |Map|[11564|https://github.com/apache/beam/pull/11564]|Closed|
> |GroupByKey|[11734|https://github.com/apache/beam/pull/11734]|Closed|
> |CoGroupByKey|[11803|https://github.com/apache/beam/pull/11803]|Closed|
> |Combine|[11866|https://github.com/apache/beam/pull/11866]| Open|
> |Flatten|[11806|https://github.com/apache/beam/pull/11806]|Closed|
> |Partition| | |
> |Side Input| | |
> |Side Output| | |
> |Branching| | |
> |Composite Transform| | |
> |DoFn Additional Parameters| | |
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7516) Add a watermark manager for the fn_api_runner
[ https://issues.apache.org/jira/browse/BEAM-7516?focusedWorklogId=439197=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-439197 ] ASF GitHub Bot logged work on BEAM-7516: Author: ASF GitHub Bot Created on: 31/May/20 10:02 Start Date: 31/May/20 10:02 Worklog Time Spent: 10m Work Description: stale[bot] closed pull request #10291: URL: https://github.com/apache/beam/pull/10291 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 439197) Time Spent: 9h 10m (was: 9h) > Add a watermark manager for the fn_api_runner > - > > Key: BEAM-7516 > URL: https://issues.apache.org/jira/browse/BEAM-7516 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Pablo Estrada >Assignee: Pablo Estrada >Priority: P2 > Time Spent: 9h 10m > Remaining Estimate: 0h > > To track watermarks for each stage -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7516) Add a watermark manager for the fn_api_runner
[ https://issues.apache.org/jira/browse/BEAM-7516?focusedWorklogId=439198=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-439198 ] ASF GitHub Bot logged work on BEAM-7516: Author: ASF GitHub Bot Created on: 31/May/20 10:02 Start Date: 31/May/20 10:02 Worklog Time Spent: 10m Work Description: stale[bot] commented on pull request #10291: URL: https://github.com/apache/beam/pull/10291#issuecomment-636449071 This pull request has been closed due to lack of activity. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 439198) Time Spent: 9h 20m (was: 9h 10m) > Add a watermark manager for the fn_api_runner > - > > Key: BEAM-7516 > URL: https://issues.apache.org/jira/browse/BEAM-7516 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Pablo Estrada >Assignee: Pablo Estrada >Priority: P2 > Time Spent: 9h 20m > Remaining Estimate: 0h > > To track watermarks for each stage -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-10027) Support for Kotlin-based Beam Katas
[ https://issues.apache.org/jira/browse/BEAM-10027?focusedWorklogId=439195=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-439195 ] ASF GitHub Bot logged work on BEAM-10027: - Author: ASF GitHub Bot Created on: 31/May/20 09:47 Start Date: 31/May/20 09:47 Worklog Time Spent: 10m Work Description: henryken commented on a change in pull request #11761: URL: https://github.com/apache/beam/pull/11761#discussion_r432929090 ## File path: learning/katas/kotlin/Core Transforms/Map/MapElements/task-info.yaml ## @@ -0,0 +1,13 @@ +type: edu +files: +- name: src/org/apache/beam/learning/katas/coretransforms/map/mapelements/Task.kt + visible: true + placeholders: Review comment: Can you exclude the "return" from the answer placeholder, similar to the other tasks in the Map lesson?  This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 439195) Time Spent: 12h 50m (was: 12h 40m) > Support for Kotlin-based Beam Katas > --- > > Key: BEAM-10027 > URL: https://issues.apache.org/jira/browse/BEAM-10027 > Project: Beam > Issue Type: Improvement > Components: katas >Reporter: Rion Williams >Assignee: Rion Williams >Priority: P2 > Original Estimate: 8h > Time Spent: 12h 50m > Remaining Estimate: 0h > > Currently, there are a series of examples available demonstrating the use of > Apache Beam with Kotlin. It would be nice to have support for the same Beam > Katas that exist for Python, Go, and Java to also support Kotlin. > The port itself shouldn't be that involved since it can still target the JVM, > so it would likely just require the inclusion for Kotlin dependencies and a > conversion for all of the existing Java examples. -- This message was sent by Atlassian Jira (v8.3.4#803005)
