[GitHub] incubator-beam pull request: checkstyle: enforce where line breaks...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-beam/pull/341 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[1/2] incubator-beam git commit: Closes #341
Repository: incubator-beam Updated Branches: refs/heads/master 638cc2071 -> d627266d8 Closes #341 Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/d627266d Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/d627266d Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/d627266d Branch: refs/heads/master Commit: d627266d8d39ff0ec94dc9f3f84893c1026abde7 Parents: 638cc20 99f1925 Author: Dan Halperin Authored: Mon May 16 23:25:14 2016 -0700 Committer: Dan Halperin Committed: Mon May 16 23:25:14 2016 -0700 -- .../examples/complete/game/injector/Injector.java| 10 +- .../runners/direct/InProcessEvaluationContext.java | 6 +++--- .../beam/runners/direct/TransformExecutorTest.java | 6 +++--- .../runners/dataflow/DataflowPipelineRunner.java | 4 ++-- .../dataflow/testing/TestDataflowPipelineRunner.java | 4 ++-- .../runners/dataflow/util/DataflowTransport.java | 4 ++-- .../beam/runners/spark/SparkPipelineRunner.java | 10 +- .../runners/spark/SparkStreamingPipelineOptions.java | 4 ++-- .../spark/translation/SparkContextFactory.java | 4 ++-- .../spark/translation/SparkProcessContext.java | 8 .../streaming/StreamingWindowPipelineDetector.java | 4 ++-- .../src/main/resources/beam/checkstyle.xml | 10 ++ .../org/apache/beam/sdk/coders/CoderFactories.java | 10 +- .../main/java/org/apache/beam/sdk/io/BigQueryIO.java | 6 +++--- .../java/org/apache/beam/sdk/io/FileBasedSource.java | 3 +-- .../beam/sdk/runners/DirectPipelineRunner.java | 15 +++ .../org/apache/beam/sdk/transforms/Partition.java| 4 ++-- .../apache/beam/sdk/transforms/join/CoGbkResult.java | 8 .../apache/beam/sdk/transforms/join/UnionCoder.java | 4 ++-- .../apache/beam/sdk/util/BigQueryTableInserter.java | 8 .../org/apache/beam/sdk/util/InstanceBuilder.java| 6 +++--- .../java/org/apache/beam/sdk/util/StringUtils.java | 4 ++-- .../java/org/apache/beam/sdk/util/Transport.java | 4 ++-- .../org/apache/beam/sdk/values/PCollectionList.java | 4 ++-- .../sdk/transforms/display/DisplayDataMatchers.java | 4 ++-- .../apache/beam/sdk/values/PCollectionListTest.java | 4 ++-- .../java/org/apache/beam/sdk/io/kafka/KafkaIO.java | 8 27 files changed, 87 insertions(+), 79 deletions(-) --
[2/2] incubator-beam git commit: checkstyle.xml: enforce where line breaks happen
checkstyle.xml: enforce where line breaks happen
We should do it consistently throughout the SDK. Only a few
fixups.
Ported from:
https://github.com/checkstyle/checkstyle/blob/ccdc3c09c500ccd31d56e4d6ed9c2d86edf2e9f1/src/main/resources/google_checks.xml#L167
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/99f19257
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/99f19257
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/99f19257
Branch: refs/heads/master
Commit: 99f19257c1634325c4d6ff162f1f999399e0d692
Parents: 638cc20
Author: Dan Halperin
Authored: Mon May 16 14:30:02 2016 -0700
Committer: Dan Halperin
Committed: Mon May 16 23:25:14 2016 -0700
--
.../examples/complete/game/injector/Injector.java| 10 +-
.../runners/direct/InProcessEvaluationContext.java | 6 +++---
.../beam/runners/direct/TransformExecutorTest.java | 6 +++---
.../runners/dataflow/DataflowPipelineRunner.java | 4 ++--
.../dataflow/testing/TestDataflowPipelineRunner.java | 4 ++--
.../runners/dataflow/util/DataflowTransport.java | 4 ++--
.../beam/runners/spark/SparkPipelineRunner.java | 10 +-
.../runners/spark/SparkStreamingPipelineOptions.java | 4 ++--
.../spark/translation/SparkContextFactory.java | 4 ++--
.../spark/translation/SparkProcessContext.java | 8
.../streaming/StreamingWindowPipelineDetector.java | 4 ++--
.../src/main/resources/beam/checkstyle.xml | 10 ++
.../org/apache/beam/sdk/coders/CoderFactories.java | 10 +-
.../main/java/org/apache/beam/sdk/io/BigQueryIO.java | 6 +++---
.../java/org/apache/beam/sdk/io/FileBasedSource.java | 3 +--
.../beam/sdk/runners/DirectPipelineRunner.java | 15 +++
.../org/apache/beam/sdk/transforms/Partition.java| 4 ++--
.../apache/beam/sdk/transforms/join/CoGbkResult.java | 8
.../apache/beam/sdk/transforms/join/UnionCoder.java | 4 ++--
.../apache/beam/sdk/util/BigQueryTableInserter.java | 8
.../org/apache/beam/sdk/util/InstanceBuilder.java| 6 +++---
.../java/org/apache/beam/sdk/util/StringUtils.java | 4 ++--
.../java/org/apache/beam/sdk/util/Transport.java | 4 ++--
.../org/apache/beam/sdk/values/PCollectionList.java | 4 ++--
.../sdk/transforms/display/DisplayDataMatchers.java | 4 ++--
.../apache/beam/sdk/values/PCollectionListTest.java | 4 ++--
.../java/org/apache/beam/sdk/io/kafka/KafkaIO.java | 8
27 files changed, 87 insertions(+), 79 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/99f19257/examples/java8/src/main/java/org/apache/beam/examples/complete/game/injector/Injector.java
--
diff --git
a/examples/java8/src/main/java/org/apache/beam/examples/complete/game/injector/Injector.java
b/examples/java8/src/main/java/org/apache/beam/examples/complete/game/injector/Injector.java
index f8c3e33..034a186 100644
---
a/examples/java8/src/main/java/org/apache/beam/examples/complete/game/injector/Injector.java
+++
b/examples/java8/src/main/java/org/apache/beam/examples/complete/game/injector/Injector.java
@@ -151,8 +151,8 @@ class Injector {
this.teamName = teamName;
this.startTimeInMillis = startTimeInMillis;
// How long until this team is dissolved.
- this.expirationPeriod = random.nextInt(TEAM_EXPIRATION_TIME_IN_MINS) +
-BASE_TEAM_EXPIRATION_TIME_IN_MINS;
+ this.expirationPeriod = random.nextInt(TEAM_EXPIRATION_TIME_IN_MINS)
++ BASE_TEAM_EXPIRATION_TIME_IN_MINS;
this.robot = robot;
// Determine the number of team members.
numMembers = random.nextInt(MEMBERS_PER_TEAM) + BASE_MEMBERS_PER_TEAM;
@@ -204,9 +204,9 @@ class Injector {
long currTime = System.currentTimeMillis();
if ((team.getEndTimeInMillis() < currTime) || team.numMembers() == 0) {
System.out.println("\nteam " + team + " is too old; replacing.");
- System.out.println("start time: " + team.getStartTimeInMillis() +
-", end time: " + team.getEndTimeInMillis() +
-", current time:" + currTime);
+ System.out.println("start time: " + team.getStartTimeInMillis()
++ ", end time: " + team.getEndTimeInMillis()
++ ", current time:" + currTime);
removeTeam(index);
// Add a new team in its stead.
return (addLiveTeam());
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/99f19257/runners/direct-java/src/main/java/org/apache/beam/runners/direct/InProcessEvaluationContext.java
--
diff --git
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/InProcessEvaluationContext.ja
[jira] [Commented] (BEAM-289) Examples Use TypeDescriptors
[ https://issues.apache.org/jira/browse/BEAM-289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15285528#comment-15285528 ] ASF GitHub Bot commented on BEAM-289: - GitHub user eljefe6a opened a pull request: https://github.com/apache/incubator-beam/pull/342 [BEAM-289] Examples Use TypeDescriptors Jira issue BEAM-289 Examples Use TypeDescriptors. Changed example code to use `TypeDescriptors`. These could have used static imports to make the lines shorter, but we opted for the more understandable syntax. You can merge this pull request into a Git repository by running: $ git pull https://github.com/eljefe6a/incubator-beam TypeDescriptorsExamples Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/342.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #342 commit 72ec9b82fb21ee9040b1bee6615fecfb9916e470 Author: Jesse Anderson Date: 2016-05-02T18:39:26Z Make Regex Transform commit e6f8c958a2bb88d8b2582cb9d4391922c15b7141 Author: Jesse Anderson Date: 2016-05-02T22:34:08Z Merge remote-tracking branch 'upstream/master' commit 587eaaec106829002df5df1b38753f811649aa51 Author: Jesse Anderson Date: 2016-05-03T01:08:13Z Fixing checkstyle issues. Added missing Apache license. commit df3045f62c939ef3a777ffbf658088f193144983 Author: Jesse Anderson Date: 2016-05-05T15:46:14Z Added distributed replacement functions. Add replaceAll and replaceFirst. Fixed some JavaDocs. commit 793d22667f485a5cdd49a7d36553c96e6898391c Author: Jesse Anderson Date: 2016-05-05T15:55:58Z Whitespace fixes for check style. commit 9e5a9971131721c988242400643712f5c9671b9e Author: Jesse Anderson Date: 2016-05-09T15:47:56Z Merge remote-tracking branch 'upstream/master' commit 425a4d89692f33f99a68aed511270de0ff9db4ac Author: Jesse Anderson Date: 2016-05-11T19:51:57Z Merge remote-tracking branch 'upstream/master' commit 225b2d0ac2d8808da7756f915cbeb7684e4951a4 Author: Jesse Anderson Date: 2016-05-14T01:20:12Z Merge remote-tracking branch 'upstream/master' commit c1a3bc55b47e5dbd11c2bcb360a2a7f2c4aacfb9 Author: Jesse Anderson Date: 2016-05-16T20:58:08Z Changed Word Counts to use TypeDescriptors. commit adfeb01bafb1c17c6c65b30c45ea3e42473a80e6 Author: Jesse Anderson Date: 2016-05-16T21:09:18Z Updated complete examples to use TypeDescriptors. commit 2a455f3ae029e30644cc7b6c96105f761f903e74 Author: Jesse Anderson Date: 2016-05-16T22:11:57Z Removing Regex transforms from this branch. > Examples Use TypeDescriptors > > > Key: BEAM-289 > URL: https://issues.apache.org/jira/browse/BEAM-289 > Project: Beam > Issue Type: Improvement > Components: examples-java >Reporter: Jesse Anderson >Assignee: Frances Perry > > Change the Java and Java 8 examples to use TypeDescriptors instead of inline > TypeDescriptor creation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-beam pull request: [BEAM-289] Examples Use TypeDescripto...
GitHub user eljefe6a opened a pull request: https://github.com/apache/incubator-beam/pull/342 [BEAM-289] Examples Use TypeDescriptors Jira issue BEAM-289 Examples Use TypeDescriptors. Changed example code to use `TypeDescriptors`. These could have used static imports to make the lines shorter, but we opted for the more understandable syntax. You can merge this pull request into a Git repository by running: $ git pull https://github.com/eljefe6a/incubator-beam TypeDescriptorsExamples Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/342.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #342 commit 72ec9b82fb21ee9040b1bee6615fecfb9916e470 Author: Jesse Anderson Date: 2016-05-02T18:39:26Z Make Regex Transform commit e6f8c958a2bb88d8b2582cb9d4391922c15b7141 Author: Jesse Anderson Date: 2016-05-02T22:34:08Z Merge remote-tracking branch 'upstream/master' commit 587eaaec106829002df5df1b38753f811649aa51 Author: Jesse Anderson Date: 2016-05-03T01:08:13Z Fixing checkstyle issues. Added missing Apache license. commit df3045f62c939ef3a777ffbf658088f193144983 Author: Jesse Anderson Date: 2016-05-05T15:46:14Z Added distributed replacement functions. Add replaceAll and replaceFirst. Fixed some JavaDocs. commit 793d22667f485a5cdd49a7d36553c96e6898391c Author: Jesse Anderson Date: 2016-05-05T15:55:58Z Whitespace fixes for check style. commit 9e5a9971131721c988242400643712f5c9671b9e Author: Jesse Anderson Date: 2016-05-09T15:47:56Z Merge remote-tracking branch 'upstream/master' commit 425a4d89692f33f99a68aed511270de0ff9db4ac Author: Jesse Anderson Date: 2016-05-11T19:51:57Z Merge remote-tracking branch 'upstream/master' commit 225b2d0ac2d8808da7756f915cbeb7684e4951a4 Author: Jesse Anderson Date: 2016-05-14T01:20:12Z Merge remote-tracking branch 'upstream/master' commit c1a3bc55b47e5dbd11c2bcb360a2a7f2c4aacfb9 Author: Jesse Anderson Date: 2016-05-16T20:58:08Z Changed Word Counts to use TypeDescriptors. commit adfeb01bafb1c17c6c65b30c45ea3e42473a80e6 Author: Jesse Anderson Date: 2016-05-16T21:09:18Z Updated complete examples to use TypeDescriptors. commit 2a455f3ae029e30644cc7b6c96105f761f903e74 Author: Jesse Anderson Date: 2016-05-16T22:11:57Z Removing Regex transforms from this branch. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (BEAM-289) Examples Use TypeDescriptors
Jesse Anderson created BEAM-289: --- Summary: Examples Use TypeDescriptors Key: BEAM-289 URL: https://issues.apache.org/jira/browse/BEAM-289 Project: Beam Issue Type: Improvement Components: examples-java Reporter: Jesse Anderson Assignee: Frances Perry Change the Java and Java 8 examples to use TypeDescriptors instead of inline TypeDescriptor creation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-beam pull request: checkstyle: enforce where line breaks...
GitHub user dhalperi opened a pull request: https://github.com/apache/incubator-beam/pull/341 checkstyle: enforce where line breaks happen We should do it consistently throughout the SDK. Only a few fixups. Ported from: https://github.com/checkstyle/checkstyle/blob/ccdc3c09c500ccd31d56e4d6ed9c2d86edf2e9f1/src/main/resources/google_checks.xml#L167 You can merge this pull request into a Git repository by running: $ git pull https://github.com/dhalperi/incubator-beam checkstyle-end-of-line Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/341.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #341 commit 6b03244749be33104000eea7c44ec8c76a644192 Author: Dan Halperin Date: 2016-05-16T21:30:02Z checkstyle: enforce where line breaks happen We should do it consistently throughout the SDK. Only a few fixups. Ported from: https://github.com/checkstyle/checkstyle/blob/ccdc3c09c500ccd31d56e4d6ed9c2d86edf2e9f1/src/main/resources/google_checks.xml#L167 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (BEAM-288) Improve javadoc for UnboundedSource
[ https://issues.apache.org/jira/browse/BEAM-288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15285188#comment-15285188 ] ASF GitHub Bot commented on BEAM-288: - GitHub user mshields822 opened a pull request: https://github.com/apache/incubator-beam/pull/340 [BEAM-288] Improve UnboundedSource javadoc R: @dhalperi You can merge this pull request into a Git repository by running: $ git pull https://github.com/mshields822/incubator-beam beam-288 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/340.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #340 commit 38c083f9df7e0cf75f33f7108eef74e87c5915be Author: Mark Shields Date: 2016-05-16T19:56:53Z Improve UnboundedSource javadoc > Improve javadoc for UnboundedSource > --- > > Key: BEAM-288 > URL: https://issues.apache.org/jira/browse/BEAM-288 > Project: Beam > Issue Type: Bug >Reporter: Mark Shields >Assignee: Mark Shields > > While working on the pubsub source noticed the UnboundedSource and associated > Reader / Checkpoint API needed some important clarrification. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-beam pull request: [BEAM-288] Improve UnboundedSource ja...
GitHub user mshields822 opened a pull request: https://github.com/apache/incubator-beam/pull/340 [BEAM-288] Improve UnboundedSource javadoc R: @dhalperi You can merge this pull request into a Git repository by running: $ git pull https://github.com/mshields822/incubator-beam beam-288 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/340.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #340 commit 38c083f9df7e0cf75f33f7108eef74e87c5915be Author: Mark Shields Date: 2016-05-16T19:56:53Z Improve UnboundedSource javadoc --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (BEAM-117) Implement the API for Static Display Metadata
[ https://issues.apache.org/jira/browse/BEAM-117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15285185#comment-15285185 ] ASF GitHub Bot commented on BEAM-117: - Github user asfgit closed the pull request at: https://github.com/apache/incubator-beam/pull/338 > Implement the API for Static Display Metadata > - > > Key: BEAM-117 > URL: https://issues.apache.org/jira/browse/BEAM-117 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core >Reporter: Ben Chambers >Assignee: Scott Wegner > > As described in the following doc, we would like the SDK to allow associating > display metadata with PTransforms. > https://docs.google.com/document/d/11enEB9JwVp6vO0uOYYTMYTGkr3TdNfELwWqoiUg5ZxM/edit?usp=sharing -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-beam pull request: [BEAM-117] Add labels for SDK display...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-beam/pull/338 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[1/2] incubator-beam git commit: Add labels for SDK display data
Repository: incubator-beam
Updated Branches:
refs/heads/master 0f2e6f37e -> 638cc2071
Add labels for SDK display data
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/62f05d8b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/62f05d8b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/62f05d8b
Branch: refs/heads/master
Commit: 62f05d8b2513fbe7bb5b2ac181f9550634317792
Parents: 0f2e6f3
Author: Scott Wegner
Authored: Wed May 11 14:17:10 2016 -0700
Committer: bchambers
Committed: Mon May 16 12:44:14 2016 -0700
--
.../DataflowPipelineTranslatorTest.java | 2 +
.../java/org/apache/beam/sdk/io/AvroIO.java | 28 +
.../java/org/apache/beam/sdk/io/BigQueryIO.java | 43 +---
.../sdk/io/BoundedReadFromUnboundedSource.java | 9 ++--
.../apache/beam/sdk/io/CompressedSource.java| 9 ++--
.../org/apache/beam/sdk/io/CountingInput.java | 12 --
.../org/apache/beam/sdk/io/DatastoreIO.java | 18 +---
.../org/apache/beam/sdk/io/FileBasedSink.java | 3 +-
.../org/apache/beam/sdk/io/FileBasedSource.java | 3 +-
.../apache/beam/sdk/io/OffsetBasedSource.java | 9 ++--
.../java/org/apache/beam/sdk/io/PubsubIO.java | 42 +++
.../main/java/org/apache/beam/sdk/io/Read.java | 6 ++-
.../java/org/apache/beam/sdk/io/TextIO.java | 25
.../main/java/org/apache/beam/sdk/io/Write.java | 3 +-
.../java/org/apache/beam/sdk/io/XmlSink.java| 6 ++-
.../java/org/apache/beam/sdk/io/XmlSource.java | 9 ++--
.../sdk/transforms/ApproximateQuantiles.java| 6 ++-
.../beam/sdk/transforms/ApproximateUnique.java | 6 ++-
.../org/apache/beam/sdk/transforms/Combine.java | 15 ---
.../apache/beam/sdk/transforms/CombineFns.java | 3 +-
.../org/apache/beam/sdk/transforms/Filter.java | 3 +-
.../apache/beam/sdk/transforms/GroupByKey.java | 3 +-
.../transforms/IntraBundleParallelization.java | 6 ++-
.../apache/beam/sdk/transforms/MapElements.java | 3 +-
.../org/apache/beam/sdk/transforms/Max.java | 3 +-
.../org/apache/beam/sdk/transforms/Min.java | 3 +-
.../org/apache/beam/sdk/transforms/ParDo.java | 3 +-
.../apache/beam/sdk/transforms/Partition.java | 6 ++-
.../org/apache/beam/sdk/transforms/Sample.java | 6 ++-
.../org/apache/beam/sdk/transforms/Top.java | 6 ++-
.../transforms/windowing/CalendarWindows.java | 18 +---
.../sdk/transforms/windowing/FixedWindows.java | 6 ++-
.../beam/sdk/transforms/windowing/Sessions.java | 3 +-
.../transforms/windowing/SlidingWindows.java| 9 ++--
.../beam/sdk/transforms/windowing/Window.java | 18 +---
.../beam/sdk/io/gcp/bigtable/BigtableIO.java| 15 ---
36 files changed, 243 insertions(+), 125 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/62f05d8b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslatorTest.java
--
diff --git
a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslatorTest.java
b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslatorTest.java
index ed7e67d..58c6f75 100644
---
a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslatorTest.java
+++
b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslatorTest.java
@@ -936,6 +936,7 @@ public class DataflowPipelineTranslatorTest implements
Serializable {
.build(),
ImmutableMap.builder()
.put("key", "fn")
+.put("label", "Transform Function")
.put("type", "JAVA_CLASS")
.put("value", fn1.getClass().getName())
.put("shortValue", fn1.getClass().getSimpleName())
@@ -955,6 +956,7 @@ public class DataflowPipelineTranslatorTest implements
Serializable {
ImmutableSet> expectedFn2DisplayData =
ImmutableSet.of(
ImmutableMap.builder()
.put("key", "fn")
+.put("label", "Transform Function")
.put("type", "JAVA_CLASS")
.put("value", fn2.getClass().getName())
.put("shortValue", fn2.getClass().getSimpleName())
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/62f05d8b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java
--
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java
b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java
index 6b9f010..4b40c01 100644
--- a/sdks/java/core/src/main
[2/2] incubator-beam git commit: This closes #338
This closes #338 Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/638cc207 Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/638cc207 Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/638cc207 Branch: refs/heads/master Commit: 638cc20719cb314d9e6b29fa79d45edcf4586dce Parents: 0f2e6f3 62f05d8 Author: bchambers Authored: Mon May 16 12:44:19 2016 -0700 Committer: bchambers Committed: Mon May 16 12:44:19 2016 -0700 -- .../DataflowPipelineTranslatorTest.java | 2 + .../java/org/apache/beam/sdk/io/AvroIO.java | 28 + .../java/org/apache/beam/sdk/io/BigQueryIO.java | 43 +--- .../sdk/io/BoundedReadFromUnboundedSource.java | 9 ++-- .../apache/beam/sdk/io/CompressedSource.java| 9 ++-- .../org/apache/beam/sdk/io/CountingInput.java | 12 -- .../org/apache/beam/sdk/io/DatastoreIO.java | 18 +--- .../org/apache/beam/sdk/io/FileBasedSink.java | 3 +- .../org/apache/beam/sdk/io/FileBasedSource.java | 3 +- .../apache/beam/sdk/io/OffsetBasedSource.java | 9 ++-- .../java/org/apache/beam/sdk/io/PubsubIO.java | 42 +++ .../main/java/org/apache/beam/sdk/io/Read.java | 6 ++- .../java/org/apache/beam/sdk/io/TextIO.java | 25 .../main/java/org/apache/beam/sdk/io/Write.java | 3 +- .../java/org/apache/beam/sdk/io/XmlSink.java| 6 ++- .../java/org/apache/beam/sdk/io/XmlSource.java | 9 ++-- .../sdk/transforms/ApproximateQuantiles.java| 6 ++- .../beam/sdk/transforms/ApproximateUnique.java | 6 ++- .../org/apache/beam/sdk/transforms/Combine.java | 15 --- .../apache/beam/sdk/transforms/CombineFns.java | 3 +- .../org/apache/beam/sdk/transforms/Filter.java | 3 +- .../apache/beam/sdk/transforms/GroupByKey.java | 3 +- .../transforms/IntraBundleParallelization.java | 6 ++- .../apache/beam/sdk/transforms/MapElements.java | 3 +- .../org/apache/beam/sdk/transforms/Max.java | 3 +- .../org/apache/beam/sdk/transforms/Min.java | 3 +- .../org/apache/beam/sdk/transforms/ParDo.java | 3 +- .../apache/beam/sdk/transforms/Partition.java | 6 ++- .../org/apache/beam/sdk/transforms/Sample.java | 6 ++- .../org/apache/beam/sdk/transforms/Top.java | 6 ++- .../transforms/windowing/CalendarWindows.java | 18 +--- .../sdk/transforms/windowing/FixedWindows.java | 6 ++- .../beam/sdk/transforms/windowing/Sessions.java | 3 +- .../transforms/windowing/SlidingWindows.java| 9 ++-- .../beam/sdk/transforms/windowing/Window.java | 18 +--- .../beam/sdk/io/gcp/bigtable/BigtableIO.java| 15 --- 36 files changed, 243 insertions(+), 125 deletions(-) --
[jira] [Created] (BEAM-288) Improve javadoc for UnboundedSource
Mark Shields created BEAM-288: - Summary: Improve javadoc for UnboundedSource Key: BEAM-288 URL: https://issues.apache.org/jira/browse/BEAM-288 Project: Beam Issue Type: Bug Reporter: Mark Shields While working on the pubsub source noticed the UnboundedSource and associated Reader / Checkpoint API needed some important clarrification. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-beam pull request: Add toUnbounded() to get a Unbounded ...
GitHub user peihe opened a pull request: https://github.com/apache/incubator-beam/pull/339 Add toUnbounded() to get a Unbounded Read from a Bounded Read. You can merge this pull request into a Git repository by running: $ git pull https://github.com/peihe/incubator-beam unbounded-read Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/339.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #339 commit d9f25a4540cb9fa049b54cfb38ed60740a0408e4 Author: Pei He Date: 2016-05-09T22:59:58Z Add toUnbounded() to get a Unbounded Read from a Bounded Read. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (BEAM-287) Add new beam groupId
Jean-Baptiste Onofré created BEAM-287:
-
Summary: Add new beam groupId
Key: BEAM-287
URL: https://issues.apache.org/jira/browse/BEAM-287
Project: Beam
Issue Type: Task
Components: build-system
Reporter: Jean-Baptiste Onofré
Assignee: Jean-Baptiste Onofré
Now, all beam modules use the same groupId: {{org.apache.beam}}.
If it's convenient when using {{search.maven.org}}, browsing the repository is
not easy as all artifacts are in the same location.
As proposed initially, I will introduce new groupIds (like
{{org.apache.beam.io}}, {{org.apache.beam.runner}}, etc). We discussed about
that with [~davor] last week.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (BEAM-286) Reorganize flink runner directories
Jean-Baptiste Onofré created BEAM-286: - Summary: Reorganize flink runner directories Key: BEAM-286 URL: https://issues.apache.org/jira/browse/BEAM-286 Project: Beam Issue Type: Task Components: runner-flink Reporter: Jean-Baptiste Onofré Assignee: Jean-Baptiste Onofré The flink runner Maven module uses two sub-modules: runner and examples. It's the only one which use such layout (compare to spark, dataflow or inprocess/direct runners). I will propose a PR to align flink runner module with the other, keeping the examples in a sub-directory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-beam pull request: checkstyle.xml: blacklist more bad im...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-beam/pull/336 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[2/3] incubator-beam git commit: checkstyle.xml: blacklist more bad imports from api client package
checkstyle.xml: blacklist more bad imports from api client package And fixup existing use Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/0a0bb90b Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/0a0bb90b Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/0a0bb90b Branch: refs/heads/master Commit: 0a0bb90b5703c326d46d1c2071e5d10da96c28b3 Parents: ee198d4 Author: Dan Halperin Authored: Fri May 13 18:20:27 2016 -0700 Committer: Dan Halperin Committed: Mon May 16 11:07:07 2016 -0700 -- .../org/apache/beam/runners/dataflow/util/OutputReference.java| 2 +- sdks/java/build-tools/src/main/resources/beam/checkstyle.xml | 2 +- .../core/src/main/java/org/apache/beam/sdk/util/CloudObject.java | 3 ++- 3 files changed, 4 insertions(+), 3 deletions(-) -- http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/0a0bb90b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/OutputReference.java -- diff --git a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/OutputReference.java b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/OutputReference.java index 1b525ac..b0b7811 100644 --- a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/OutputReference.java +++ b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/OutputReference.java @@ -17,7 +17,7 @@ */ package org.apache.beam.runners.dataflow.util; -import static com.google.api.client.util.Preconditions.checkNotNull; +import static com.google.common.base.Preconditions.checkNotNull; import com.google.api.client.json.GenericJson; import com.google.api.client.util.Key; http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/0a0bb90b/sdks/java/build-tools/src/main/resources/beam/checkstyle.xml -- diff --git a/sdks/java/build-tools/src/main/resources/beam/checkstyle.xml b/sdks/java/build-tools/src/main/resources/beam/checkstyle.xml index 2a4f832..b739243 100644 --- a/sdks/java/build-tools/src/main/resources/beam/checkstyle.xml +++ b/sdks/java/build-tools/src/main/resources/beam/checkstyle.xml @@ -110,7 +110,7 @@ page at http://checkstyle.sourceforge.net/config.html --> - + http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/0a0bb90b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/CloudObject.java -- diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/CloudObject.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/CloudObject.java index 8d3ec0b..2772225 100644 --- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/CloudObject.java +++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/CloudObject.java @@ -17,7 +17,8 @@ */ package org.apache.beam.sdk.util; -import static com.google.api.client.util.Preconditions.checkNotNull; + +import static com.google.common.base.Preconditions.checkNotNull; import com.google.api.client.json.GenericJson; import com.google.api.client.util.Key;
[1/3] incubator-beam git commit: checkstyle: ban netty internal classes
Repository: incubator-beam Updated Branches: refs/heads/master ee198d4e9 -> 0f2e6f37e checkstyle: ban netty internal classes Accidentally used Netty's ThreadLocalRandom and want to prevent further uses Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/3433b134 Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/3433b134 Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/3433b134 Branch: refs/heads/master Commit: 3433b134d0551f744556aee38cee55129ec4f79f Parents: 0a0bb90 Author: Dan Halperin Authored: Sun May 15 18:06:21 2016 -0700 Committer: Dan Halperin Committed: Mon May 16 11:07:07 2016 -0700 -- sdks/java/build-tools/src/main/resources/beam/checkstyle.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/3433b134/sdks/java/build-tools/src/main/resources/beam/checkstyle.xml -- diff --git a/sdks/java/build-tools/src/main/resources/beam/checkstyle.xml b/sdks/java/build-tools/src/main/resources/beam/checkstyle.xml index b739243..30eddcc 100644 --- a/sdks/java/build-tools/src/main/resources/beam/checkstyle.xml +++ b/sdks/java/build-tools/src/main/resources/beam/checkstyle.xml @@ -110,7 +110,7 @@ page at http://checkstyle.sourceforge.net/config.html --> - +
[3/3] incubator-beam git commit: Closes #336
Closes #336 Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/0f2e6f37 Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/0f2e6f37 Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/0f2e6f37 Branch: refs/heads/master Commit: 0f2e6f37e39065fab23a122b58e92eb876aebf6e Parents: ee198d4 3433b13 Author: Dan Halperin Authored: Mon May 16 11:59:33 2016 -0700 Committer: Dan Halperin Committed: Mon May 16 11:59:33 2016 -0700 -- .../org/apache/beam/runners/dataflow/util/OutputReference.java| 2 +- sdks/java/build-tools/src/main/resources/beam/checkstyle.xml | 2 +- .../core/src/main/java/org/apache/beam/sdk/util/CloudObject.java | 3 ++- 3 files changed, 4 insertions(+), 3 deletions(-) --
[jira] [Created] (BEAM-285) Make MinLongFn and MaxLongFn mimic SumLongFn and use BinaryCombineLongFn
Mark Shields created BEAM-285: - Summary: Make MinLongFn and MaxLongFn mimic SumLongFn and use BinaryCombineLongFn Key: BEAM-285 URL: https://issues.apache.org/jira/browse/BEAM-285 Project: Beam Issue Type: Bug Components: sdk-java-core Reporter: Mark Shields Assignee: Davor Bonaci Ditto for the other 'optimized accumulator' combiner functions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[2/2] incubator-beam git commit: Closes #337
Closes #337 Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/ee198d4e Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/ee198d4e Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/ee198d4e Branch: refs/heads/master Commit: ee198d4e94b0d549271cde0f64834341b75b0be4 Parents: c0efe56 22dc6a3 Author: Dan Halperin Authored: Mon May 16 11:05:17 2016 -0700 Committer: Dan Halperin Committed: Mon May 16 11:05:17 2016 -0700 -- .../dataflow/DataflowPipelineTranslator.java| 5 -- .../dataflow/internal/BigQueryIOTranslator.java | 72 2 files changed, 77 deletions(-) --
[GitHub] incubator-beam pull request: [BEAM-48] Remove BigQueryIOTranslator
Github user asfgit closed the pull request at: https://github.com/apache/incubator-beam/pull/337 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[1/2] incubator-beam git commit: [BEAM-48] Remove BigQueryIOTranslator
Repository: incubator-beam
Updated Branches:
refs/heads/master c0efe568e -> ee198d4e9
[BEAM-48] Remove BigQueryIOTranslator
This is no longer needed for the Cloud Dataflow service.
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/22dc6a34
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/22dc6a34
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/22dc6a34
Branch: refs/heads/master
Commit: 22dc6a3481a77134b0cc7d9e82865c06b15f6228
Parents: c0efe56
Author: Dan Halperin
Authored: Sun May 15 17:39:57 2016 -0700
Committer: Dan Halperin
Committed: Mon May 16 11:05:16 2016 -0700
--
.../dataflow/DataflowPipelineTranslator.java| 5 --
.../dataflow/internal/BigQueryIOTranslator.java | 72
2 files changed, 77 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/22dc6a34/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
--
diff --git
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
index f58ceff..d822803 100644
---
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
+++
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
@@ -32,7 +32,6 @@ import static
com.google.common.base.Preconditions.checkArgument;
import static com.google.common.base.Preconditions.checkNotNull;
import
org.apache.beam.runners.dataflow.DataflowPipelineRunner.GroupByKeyAndSortValuesOnly;
-import org.apache.beam.runners.dataflow.internal.BigQueryIOTranslator;
import org.apache.beam.runners.dataflow.internal.PubsubIOTranslator;
import org.apache.beam.runners.dataflow.internal.ReadTranslator;
import org.apache.beam.runners.dataflow.options.DataflowPipelineOptions;
@@ -42,7 +41,6 @@ import org.apache.beam.sdk.Pipeline;
import org.apache.beam.sdk.Pipeline.PipelineVisitor;
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.IterableCoder;
-import org.apache.beam.sdk.io.BigQueryIO;
import org.apache.beam.sdk.io.PubsubIO;
import org.apache.beam.sdk.io.Read;
import org.apache.beam.sdk.options.StreamingOptions;
@@ -1012,9 +1010,6 @@ public class DataflowPipelineTranslator {
// IO Translation.
registerTransformTranslator(
-BigQueryIO.Read.Bound.class, new
BigQueryIOTranslator.ReadTranslator());
-
-registerTransformTranslator(
PubsubIO.Read.Bound.class, new PubsubIOTranslator.ReadTranslator());
registerTransformTranslator(
DataflowPipelineRunner.StreamingPubsubIOWrite.class,
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/22dc6a34/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/internal/BigQueryIOTranslator.java
--
diff --git
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/internal/BigQueryIOTranslator.java
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/internal/BigQueryIOTranslator.java
deleted file mode 100755
index b344f0f..000
---
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/internal/BigQueryIOTranslator.java
+++ /dev/null
@@ -1,72 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.dataflow.internal;
-
-import org.apache.beam.runners.dataflow.DataflowPipelineTranslator;
-import org.apache.beam.sdk.io.BigQueryIO;
-import org.apache.beam.sdk.util.PropertyNames;
-
-import com.google.api.services.bigquery.model.TableReference;
-
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-/**
- * BigQ
[GitHub] incubator-beam pull request: Add labels for SDK display data
GitHub user swegner opened a pull request: https://github.com/apache/incubator-beam/pull/338 Add labels for SDK display data Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [ ] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [ ] Replace `` in the title with the actual Jira issue number, if there is one. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/swegner/incubator-beam displaydata-labels Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/338.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #338 commit 998ba7ba369a34654b9e93f662f7b1d9b7d83a58 Author: Scott Wegner Date: 2016-05-11T21:17:10Z Add labels for SDK display data --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[1/2] incubator-beam git commit: Validate display data on transform primitives
Repository: incubator-beam
Updated Branches:
refs/heads/master 9b5cf3247 -> c0efe568e
Validate display data on transform primitives
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/6269b8fa
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/6269b8fa
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/6269b8fa
Branch: refs/heads/master
Commit: 6269b8fa06daa4608f5ce5b11054cca58747a229
Parents: 9b5cf32
Author: Scott Wegner
Authored: Mon May 9 13:31:39 2016 -0700
Committer: Scott Wegner
Committed: Mon May 16 09:41:21 2016 -0700
--
runners/google-cloud-dataflow-java/pom.xml | 10 +++
.../runners/dataflow/io/DataflowAvroIOTest.java | 69
.../dataflow/io/DataflowBigQueryIOTest.java | 29 +++-
.../dataflow/io/DataflowDatastoreIOTest.java| 63 ++
.../dataflow/io/DataflowPubsubIOTest.java | 62 ++
.../runners/dataflow/io/DataflowTextIOTest.java | 15 -
.../DataflowDisplayDataEvaluator.java | 3 +-
.../java/org/apache/beam/sdk/io/BigQueryIO.java | 11
.../java/org/apache/beam/sdk/io/PubsubIO.java | 5 ++
.../org/apache/beam/sdk/io/BigQueryIOTest.java | 6 +-
.../beam/sdk/io/CompressedSourceTest.java | 3 +-
.../sdk/options/ProxyInvocationHandlerTest.java | 6 +-
.../sdk/transforms/ApproximateUniqueTest.java | 3 +-
.../display/DisplayDataEvaluator.java | 28
.../display/DisplayDataEvaluatorTest.java | 24 ++-
.../transforms/display/DisplayDataMatchers.java | 8 +++
.../display/DisplayDataMatchersTest.java| 3 +-
.../sdk/transforms/windowing/WindowTest.java| 4 +-
.../sdk/io/gcp/bigtable/BigtableIOTest.java | 9 +--
19 files changed, 316 insertions(+), 45 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/6269b8fa/runners/google-cloud-dataflow-java/pom.xml
--
diff --git a/runners/google-cloud-dataflow-java/pom.xml
b/runners/google-cloud-dataflow-java/pom.xml
index a1d48ce..f7f1d80 100644
--- a/runners/google-cloud-dataflow-java/pom.xml
+++ b/runners/google-cloud-dataflow-java/pom.xml
@@ -288,11 +288,21 @@
+ org.apache.avro
+ avro
+
+
+
com.google.api-client
google-api-client
+ com.google.apis
+ google-api-services-datastore-protobuf
+
+
+
com.google.oauth-client
google-oauth-client
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/6269b8fa/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/io/DataflowAvroIOTest.java
--
diff --git
a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/io/DataflowAvroIOTest.java
b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/io/DataflowAvroIOTest.java
new file mode 100644
index 000..614affb
--- /dev/null
+++
b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/io/DataflowAvroIOTest.java
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow.io;
+
+import static
org.apache.beam.sdk.transforms.display.DisplayDataMatchers.hasDisplayItem;
+
+import static org.hamcrest.Matchers.hasItem;
+import static org.junit.Assert.assertThat;
+
+import org.apache.beam.runners.dataflow.DataflowPipelineRunner;
+import
org.apache.beam.runners.dataflow.transforms.DataflowDisplayDataEvaluator;
+import org.apache.beam.sdk.io.AvroIO;
+import org.apache.beam.sdk.transforms.display.DisplayData;
+import org.apache.beam.sdk.transforms.display.DisplayDataEvaluator;
+
+import org.apache.avro.Schema;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+import java.util.Set;
+
+/**
+ * {@link DataflowPipelineRunner} specific tests for {@link AvroIO} t
[GitHub] incubator-beam pull request: [BEAM-117] Validate display data regi...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-beam/pull/325 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[2/2] incubator-beam git commit: This closes #325
This closes #325 Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/c0efe568 Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/c0efe568 Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/c0efe568 Branch: refs/heads/master Commit: c0efe568e5291298c1394016a12e7979b37afc44 Parents: 9b5cf32 6269b8f Author: bchambers Authored: Mon May 16 09:55:52 2016 -0700 Committer: bchambers Committed: Mon May 16 09:55:52 2016 -0700 -- runners/google-cloud-dataflow-java/pom.xml | 10 +++ .../runners/dataflow/io/DataflowAvroIOTest.java | 69 .../dataflow/io/DataflowBigQueryIOTest.java | 29 +++- .../dataflow/io/DataflowDatastoreIOTest.java| 63 ++ .../dataflow/io/DataflowPubsubIOTest.java | 62 ++ .../runners/dataflow/io/DataflowTextIOTest.java | 15 - .../DataflowDisplayDataEvaluator.java | 3 +- .../java/org/apache/beam/sdk/io/BigQueryIO.java | 11 .../java/org/apache/beam/sdk/io/PubsubIO.java | 5 ++ .../org/apache/beam/sdk/io/BigQueryIOTest.java | 6 +- .../beam/sdk/io/CompressedSourceTest.java | 3 +- .../sdk/options/ProxyInvocationHandlerTest.java | 6 +- .../sdk/transforms/ApproximateUniqueTest.java | 3 +- .../display/DisplayDataEvaluator.java | 28 .../display/DisplayDataEvaluatorTest.java | 24 ++- .../transforms/display/DisplayDataMatchers.java | 8 +++ .../display/DisplayDataMatchersTest.java| 3 +- .../sdk/transforms/windowing/WindowTest.java| 4 +- .../sdk/io/gcp/bigtable/BigtableIOTest.java | 9 +-- 19 files changed, 316 insertions(+), 45 deletions(-) --
[jira] [Commented] (BEAM-175) Leak garbage collection timers in GlobalWindow
[ https://issues.apache.org/jira/browse/BEAM-175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15284759#comment-15284759 ] Ben Chambers commented on BEAM-175: --- I think in general this seems OK. I like that we're making the behavior explicit rather than trying to guessi t. 1. This may be too much configuration. There may be other/better ways of getting pane indices (eg., once we have a sink API or a state API). We should make sure we understand the use cases before exposing the knob. Especially since ReduceFnRunner is already complicated -- this is just adding more permutations of cases it needs to handle. 2. I worry that the default, at least in the Pane Index case, is actually *less* performant than the ZERO case, which is likely what was desired 90% of the time. If we go this direction, I would propose we change the default. 3. You should flesh this out to address error cases. When do we detect that the user is accessing the PaneIndex with the ZERO behavior specified. What kind of error message do they get? Etc. > Leak garbage collection timers in GlobalWindow > -- > > Key: BEAM-175 > URL: https://issues.apache.org/jira/browse/BEAM-175 > Project: Beam > Issue Type: Bug > Components: runner-core >Reporter: Mark Shields >Assignee: Mark Shields > > Consider the transform: > Window > .into(new GlobalWindows()) > .triggering( > Repeatedly.forever( > AfterProcessingTime.pastFirstElementInPane().plusDelayOf(...))) > .discardingFiredPanes() > This is a common idiom for 'process elements bunched by arrival time'. > Currently we create an end-of-window timer per key, which clearly will only > fire if the pipeline is drained. > Better would be to avoid creating end-of-window timers if there's no state > which needs to be processed at end-of-window (ie at drain if the Global > window). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
