Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3599

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2562) Add integration test for logical operators

[Xu Mingmin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091144#comment-16091144
 ] 

Xu Mingmin commented on BEAM-2562:
--

[~xumingming] can you close this task, thank you!

> Add integration test for logical operators
> --
>
> Key: BEAM-2562
> URL: https://issues.apache.org/jira/browse/BEAM-2562
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: James Xu
>Assignee: Xu Mingmin
>  Labels: dsl_sql_merge
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2562) Add integration test for logical operators

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091139#comment-16091139
 ] 

ASF GitHub Bot commented on BEAM-2562:
--

Github user XuMingmin closed the pull request at:

https://github.com/apache/beam/pull/3560


> Add integration test for logical operators
> --
>
> Key: BEAM-2562
> URL: https://issues.apache.org/jira/browse/BEAM-2562
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: James Xu
>Assignee: Xu Mingmin
>  Labels: dsl_sql_merge
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3560: [BEAM-2562] Add integration test for logical operat...

[XuMingmin]

Github user XuMingmin closed the pull request at:

https://github.com/apache/beam/pull/3560


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-2632) TextIOReadTest create pipelines with non-unique application names

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-2632:
-

 Summary: TextIOReadTest create pipelines with non-unique 
application names
 Key: BEAM-2632
 URL: https://issues.apache.org/jira/browse/BEAM-2632
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Kenneth Knowles
Priority: Trivial


The test {{TextIOReadTest}} uses a loop to create a few tests within a single 
test method. This results in a pipeline with non-unique applied transform nodes.

Perhaps the best way to fix this is to use a JUnit {{Paramaterized}} test 
suite, or multiple. It does seem that the test is basically doing the full 
product of empty/tiny/large with various compression types.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is back to stable : beam_PostCommit_Java_MavenInstall #4394

[Apache Jenkins Server]

See 

Jenkins build is unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3598

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2631) ParDo processing timer need more tests.

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091071#comment-16091071
 ] 

Kenneth Knowles commented on BEAM-2631:
---

[~pei...@gmail.com] would you be interested in contributing some tests?

> ParDo processing timer need more tests.
> ---
>
> Key: BEAM-2631
> URL: https://issues.apache.org/jira/browse/BEAM-2631
> Project: Beam
>  Issue Type: Test
>  Components: beam-model
>Reporter: Pei He
>Assignee: Kenneth Knowles
>
> There is only one test testSimpleProcessingTimerTimer(), and it only runs in 
> direct-runner and is disabled in all other runners because it uses 
> UsesTestStream.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (BEAM-2631) ParDo processing timer need more tests.

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-2631:
-

Assignee: Pei He  (was: Kenneth Knowles)

> ParDo processing timer need more tests.
> ---
>
> Key: BEAM-2631
> URL: https://issues.apache.org/jira/browse/BEAM-2631
> Project: Beam
>  Issue Type: Test
>  Components: beam-model
>Reporter: Pei He
>Assignee: Pei He
>
> There is only one test testSimpleProcessingTimerTimer(), and it only runs in 
> direct-runner and is disabled in all other runners because it uses 
> UsesTestStream.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-2631) ParDo processing timer need more tests.

[Pei He (JIRA)]

Pei He created BEAM-2631:


 Summary: ParDo processing timer need more tests.
 Key: BEAM-2631
 URL: https://issues.apache.org/jira/browse/BEAM-2631
 Project: Beam
  Issue Type: Test
  Components: beam-model
Reporter: Pei He
Assignee: Kenneth Knowles


There is only one test testSimpleProcessingTimerTimer(), and it only runs in 
direct-runner and is disabled in all other runners because it uses 
UsesTestStream.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2648

[Apache Jenkins Server]

See 

[GitHub] beam pull request #3583: [BEAM-2613] add integration test for comparison ope...

[xumingming]

GitHub user xumingming opened a pull request:

https://github.com/apache/beam/pull/3583

[BEAM-2613] add integration test for comparison operators

Summary:

1. added integration test for comparison operators.
2. refactored comparison operator expressions into a separate package: 
`comparison`.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xumingming/beam 
BEAM-2613-integration-test-for-comparison-operators

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3583.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3583


commit f5044071c0ff0e909f2bd24ab3a8ea5921161c4b
Author: James Xu 
Date:   2017-07-13T10:24:55Z

[BEAM-2613] add integration test for comparison operators




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2613) Add integration test for comparison operators

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091051#comment-16091051
 ] 

ASF GitHub Bot commented on BEAM-2613:
--

GitHub user xumingming opened a pull request:

https://github.com/apache/beam/pull/3583

[BEAM-2613] add integration test for comparison operators

Summary:

1. added integration test for comparison operators.
2. refactored comparison operator expressions into a separate package: 
`comparison`.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xumingming/beam 
BEAM-2613-integration-test-for-comparison-operators

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3583.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3583


commit f5044071c0ff0e909f2bd24ab3a8ea5921161c4b
Author: James Xu 
Date:   2017-07-13T10:24:55Z

[BEAM-2613] add integration test for comparison operators




> Add integration test for comparison operators
> -
>
> Key: BEAM-2613
> URL: https://issues.apache.org/jira/browse/BEAM-2613
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: James Xu
>Assignee: James Xu
>  Labels: dsl_sql_merge
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #3421

[Apache Jenkins Server]

See 

[2/3] beam git commit: Splits large TextIOTest into TextIOReadTest and TextIOWriteTest

[kenn]

Splits large TextIOTest into TextIOReadTest and TextIOWriteTest


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/d495d151
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/d495d151
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/d495d151

Branch: refs/heads/master
Commit: d495d1511fe86a2199eb247df95ff0c876803c67
Parents: 0f06eb2
Author: Eugene Kirpichov 
Authored: Fri Jun 23 18:01:53 2017 -0700
Committer: Eugene Kirpichov 
Committed: Mon Jul 17 17:08:00 2017 -0700

--
 .../org/apache/beam/sdk/io/TextIOReadTest.java  |  847 +++
 .../java/org/apache/beam/sdk/io/TextIOTest.java | 1353 +-
 .../org/apache/beam/sdk/io/TextIOWriteTest.java |  604 
 3 files changed, 1460 insertions(+), 1344 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/d495d151/sdks/java/core/src/test/java/org/apache/beam/sdk/io/TextIOReadTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/TextIOReadTest.java 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/TextIOReadTest.java
new file mode 100644
index 000..8b53111
--- /dev/null
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/TextIOReadTest.java
@@ -0,0 +1,847 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io;
+
+import static org.apache.beam.sdk.TestUtils.LINES_ARRAY;
+import static org.apache.beam.sdk.TestUtils.NO_LINES_ARRAY;
+import static org.apache.beam.sdk.io.TextIO.CompressionType.AUTO;
+import static org.apache.beam.sdk.io.TextIO.CompressionType.BZIP2;
+import static org.apache.beam.sdk.io.TextIO.CompressionType.DEFLATE;
+import static org.apache.beam.sdk.io.TextIO.CompressionType.GZIP;
+import static org.apache.beam.sdk.io.TextIO.CompressionType.UNCOMPRESSED;
+import static org.apache.beam.sdk.io.TextIO.CompressionType.ZIP;
+import static 
org.apache.beam.sdk.transforms.display.DisplayDataMatchers.hasDisplayItem;
+import static 
org.apache.beam.sdk.transforms.display.DisplayDataMatchers.hasValue;
+import static org.hamcrest.Matchers.containsInAnyOrder;
+import static org.hamcrest.Matchers.equalTo;
+import static org.hamcrest.Matchers.greaterThan;
+import static org.hamcrest.Matchers.hasItem;
+import static org.hamcrest.Matchers.hasSize;
+import static org.hamcrest.Matchers.startsWith;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertThat;
+import static org.junit.Assert.assertTrue;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Iterables;
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.PrintStream;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.FileVisitResult;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.SimpleFileVisitor;
+import java.nio.file.attribute.BasicFileAttributes;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.zip.GZIPOutputStream;
+import java.util.zip.ZipEntry;
+import java.util.zip.ZipOutputStream;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.io.BoundedSource.BoundedReader;
+import org.apache.beam.sdk.io.TextIO.CompressionType;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.options.ValueProvider;
+import org.apache.beam.sdk.testing.NeedsRunner;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.SourceTestUtils;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.testing.ValidatesRunner;
+import org.apache.beam.sdk.transforms.Create;
+import 

[1/3] beam git commit: Splits large TextIOTest into TextIOReadTest and TextIOWriteTest

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master 0f06eb25b -> 7c3631810


http://git-wip-us.apache.org/repos/asf/beam/blob/d495d151/sdks/java/core/src/test/java/org/apache/beam/sdk/io/TextIOWriteTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/TextIOWriteTest.java 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/TextIOWriteTest.java
new file mode 100644
index 000..a73ed7d
--- /dev/null
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/TextIOWriteTest.java
@@ -0,0 +1,604 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io;
+
+import static com.google.common.base.MoreObjects.firstNonNull;
+import static org.apache.beam.sdk.TestUtils.LINES2_ARRAY;
+import static org.apache.beam.sdk.TestUtils.LINES_ARRAY;
+import static org.apache.beam.sdk.TestUtils.NO_LINES_ARRAY;
+import static 
org.apache.beam.sdk.transforms.display.DisplayDataMatchers.hasDisplayItem;
+import static org.hamcrest.Matchers.containsInAnyOrder;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertThat;
+import static org.junit.Assert.assertTrue;
+
+import com.google.common.base.Function;
+import com.google.common.base.Functions;
+import com.google.common.base.Predicate;
+import com.google.common.base.Predicates;
+import com.google.common.collect.FluentIterable;
+import com.google.common.collect.Iterables;
+import com.google.common.collect.Lists;
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileReader;
+import java.io.IOException;
+import java.nio.file.FileVisitResult;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.SimpleFileVisitor;
+import java.nio.file.attribute.BasicFileAttributes;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.coders.AvroCoder;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.DefaultCoder;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.io.FileBasedSink.WritableByteChannelFactory;
+import org.apache.beam.sdk.io.fs.MatchResult;
+import org.apache.beam.sdk.io.fs.MatchResult.Metadata;
+import org.apache.beam.sdk.io.fs.ResolveOptions;
+import org.apache.beam.sdk.io.fs.ResourceId;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.options.ValueProvider;
+import org.apache.beam.sdk.testing.NeedsRunner;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.sdk.transforms.display.DisplayData;
+import org.apache.beam.sdk.util.CoderUtils;
+import org.apache.beam.sdk.values.PCollection;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.ExpectedException;
+
+/** Tests for {@link TextIO.Write}. */
+public class TextIOWriteTest {
+  private static final String MY_HEADER = "myHeader";
+  private static final String MY_FOOTER = "myFooter";
+
+  private static Path tempFolder;
+
+  @Rule public TestPipeline p = TestPipeline.create();
+
+  @Rule public ExpectedException expectedException = ExpectedException.none();
+
+  @BeforeClass
+  public static void setupClass() throws IOException {
+tempFolder = Files.createTempDirectory("TextIOTest");
+  }
+
+  @AfterClass
+  public static void teardownClass() throws IOException {
+Files.walkFileTree(
+tempFolder,
+new SimpleFileVisitor() {
+  @Override
+  public FileVisitResult visitFile(Path file, BasicFileAttributes 
attrs)
+  throws IOException {
+Files.delete(file);
+return FileVisitResult.CONTINUE;
+  }
+
+  @Override
+  public FileVisitResult postVisitDirectory(Path dir, IOException exc) 
throws IOException {
+Files.delete(dir);
+return 

[GitHub] beam pull request #3442: Splits large TextIOTest into TextIOReadTest and Tex...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3442


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[3/3] beam git commit: This closes #3442: Splits large TextIOTest into TextIOReadTest and TextIOWriteTest

[kenn]

This closes #3442: Splits large TextIOTest into TextIOReadTest and 
TextIOWriteTest


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/7c363181
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/7c363181
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/7c363181

Branch: refs/heads/master
Commit: 7c3631810a604ba58ec16c3b3aa9a346bd6d9f17
Parents: 0f06eb2 d495d15
Author: Kenneth Knowles 
Authored: Mon Jul 17 19:43:20 2017 -0700
Committer: Kenneth Knowles 
Committed: Mon Jul 17 19:43:20 2017 -0700

--
 .../org/apache/beam/sdk/io/TextIOReadTest.java  |  847 +++
 .../java/org/apache/beam/sdk/io/TextIOTest.java | 1353 +-
 .../org/apache/beam/sdk/io/TextIOWriteTest.java |  604 
 3 files changed, 1460 insertions(+), 1344 deletions(-)
--

[2/2] beam git commit: This closes #3560

[lzljs3620320]

This closes #3560


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/2f8ee988
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/2f8ee988
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/2f8ee988

Branch: refs/heads/DSL_SQL
Commit: 2f8ee9884231054e0771897cdfb063b3345d5e67
Parents: bed209e 71eb024
Author: JingsongLi 
Authored: Tue Jul 18 10:55:02 2017 +0800
Committer: JingsongLi 
Committed: Tue Jul 18 10:55:02 2017 +0800

--
 .../apache/beam/dsls/sql/schema/BeamSqlRow.java | 10 +
 .../beam/dsls/sql/schema/BeamSqlRowCoder.java   | 12 +-
 .../beam/dsls/sql/utils/CalciteUtils.java   |  2 +
 ...amSqlArithmeticOperatorsIntegrationTest.java |  4 +-
 ...mSqlBuiltinFunctionsIntegrationTestBase.java |  1 +
 .../BeamSqlLogicalFunctionsIntegrationTest.java | 43 
 .../dsls/sql/schema/BeamSqlRowCoderTest.java|  2 +
 7 files changed, 70 insertions(+), 4 deletions(-)
--

[1/2] beam git commit: [BEAM-2562] Add integration test for logical operators

[lzljs3620320]

Repository: beam
Updated Branches:
  refs/heads/DSL_SQL bed209e41 -> 2f8ee9884


[BEAM-2562] Add integration test for logical operators


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/71eb024f
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/71eb024f
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/71eb024f

Branch: refs/heads/DSL_SQL
Commit: 71eb024fcd39a38bccf026b692d6bd85bccc5909
Parents: bed209e
Author: mingmxu 
Authored: Mon Jul 17 14:40:59 2017 -0700
Committer: JingsongLi 
Committed: Tue Jul 18 10:52:46 2017 +0800

--
 .../apache/beam/dsls/sql/schema/BeamSqlRow.java | 10 +
 .../beam/dsls/sql/schema/BeamSqlRowCoder.java   | 12 +-
 .../beam/dsls/sql/utils/CalciteUtils.java   |  2 +
 ...amSqlArithmeticOperatorsIntegrationTest.java |  4 +-
 ...mSqlBuiltinFunctionsIntegrationTestBase.java |  1 +
 .../BeamSqlLogicalFunctionsIntegrationTest.java | 43 
 .../dsls/sql/schema/BeamSqlRowCoderTest.java|  2 +
 7 files changed, 70 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/71eb024f/dsls/sql/src/main/java/org/apache/beam/dsls/sql/schema/BeamSqlRow.java
--
diff --git 
a/dsls/sql/src/main/java/org/apache/beam/dsls/sql/schema/BeamSqlRow.java 
b/dsls/sql/src/main/java/org/apache/beam/dsls/sql/schema/BeamSqlRow.java
index 082d92a..5c0dbc0 100644
--- a/dsls/sql/src/main/java/org/apache/beam/dsls/sql/schema/BeamSqlRow.java
+++ b/dsls/sql/src/main/java/org/apache/beam/dsls/sql/schema/BeamSqlRow.java
@@ -48,6 +48,8 @@ public class BeamSqlRow implements Serializable {
 SQL_TYPE_TO_JAVA_CLASS.put(Types.DOUBLE, Double.class);
 SQL_TYPE_TO_JAVA_CLASS.put(Types.DECIMAL, BigDecimal.class);
 
+SQL_TYPE_TO_JAVA_CLASS.put(Types.BOOLEAN, Boolean.class);
+
 SQL_TYPE_TO_JAVA_CLASS.put(Types.CHAR, String.class);
 SQL_TYPE_TO_JAVA_CLASS.put(Types.VARCHAR, String.class);
 
@@ -167,6 +169,10 @@ public class BeamSqlRow implements Serializable {
 return (BigDecimal) getFieldValue(fieldName);
   }
 
+  public boolean getBoolean(String fieldName) {
+return (boolean) getFieldValue(fieldName);
+  }
+
   public Object getFieldValue(int fieldIdx) {
 if (nullFields.contains(fieldIdx)) {
   return null;
@@ -215,6 +221,10 @@ public class BeamSqlRow implements Serializable {
 return (BigDecimal) getFieldValue(idx);
   }
 
+  public boolean getBoolean(int idx) {
+return (boolean) getFieldValue(idx);
+  }
+
   public int size() {
 return dataValues.size();
   }

http://git-wip-us.apache.org/repos/asf/beam/blob/71eb024f/dsls/sql/src/main/java/org/apache/beam/dsls/sql/schema/BeamSqlRowCoder.java
--
diff --git 
a/dsls/sql/src/main/java/org/apache/beam/dsls/sql/schema/BeamSqlRowCoder.java 
b/dsls/sql/src/main/java/org/apache/beam/dsls/sql/schema/BeamSqlRowCoder.java
index 8be5212..f2ef857 100644
--- 
a/dsls/sql/src/main/java/org/apache/beam/dsls/sql/schema/BeamSqlRowCoder.java
+++ 
b/dsls/sql/src/main/java/org/apache/beam/dsls/sql/schema/BeamSqlRowCoder.java
@@ -27,6 +27,7 @@ import org.apache.beam.dsls.sql.utils.CalciteUtils;
 import org.apache.beam.sdk.coders.BigDecimalCoder;
 import org.apache.beam.sdk.coders.BigEndianIntegerCoder;
 import org.apache.beam.sdk.coders.BigEndianLongCoder;
+import org.apache.beam.sdk.coders.ByteCoder;
 import org.apache.beam.sdk.coders.Coder;
 import org.apache.beam.sdk.coders.CoderException;
 import org.apache.beam.sdk.coders.CustomCoder;
@@ -49,6 +50,7 @@ public class BeamSqlRowCoder extends CustomCoder {
   private static final DoubleCoder doubleCoder = DoubleCoder.of();
   private static final InstantCoder instantCoder = InstantCoder.of();
   private static final BigDecimalCoder bigDecimalCoder = BigDecimalCoder.of();
+  private static final ByteCoder byteCoder = ByteCoder.of();
 
   public BeamSqlRowCoder(BeamSqlRecordType tableSchema) {
 this.tableSchema = tableSchema;
@@ -70,7 +72,7 @@ public class BeamSqlRowCoder extends CustomCoder {
   intCoder.encode((int) value.getShort(idx), outStream);
   break;
 case TINYINT:
-  intCoder.encode((int) value.getByte(idx), outStream);
+  byteCoder.encode(value.getByte(idx), outStream);
   break;
 case DOUBLE:
   doubleCoder.encode(value.getDouble(idx), outStream);
@@ -95,6 +97,9 @@ public class BeamSqlRowCoder extends CustomCoder {
 case TIMESTAMP:
   longCoder.encode(value.getDate(idx).getTime(), outStream);
   break;
+case BOOLEAN:
+  byteCoder.encode((byte) (value.getBoolean(idx) ? 1 : 0), outStream);
+  break;
 
 default:
   throw 

[GitHub] beam pull request #3582: [BEAM-2630] TestPipeline: construct job/app names b...

[peihe]

GitHub user peihe opened a pull request:

https://github.com/apache/beam/pull/3582

[BEAM-2630] TestPipeline: construct job/app names based on Description in 
junit TestRule.


Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-1234] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-1234` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam fix-test-job-name

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3582.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3582


commit 63dd4a36abf8716affb2e743edcf9a60909ac9b9
Author: Pei He 
Date:   2017-07-17T15:34:27Z

[BEAM-2630] TestPipeline: construct job/app names based on Description in 
junit TestRule.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2630) TestPipeline.getAppName() doesn't work properly when TestPipeline is used as @Rule

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091028#comment-16091028
 ] 

ASF GitHub Bot commented on BEAM-2630:
--

GitHub user peihe opened a pull request:

https://github.com/apache/beam/pull/3582

[BEAM-2630] TestPipeline: construct job/app names based on Description in 
junit TestRule.


Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-1234] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-1234` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam fix-test-job-name

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3582.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3582


commit 63dd4a36abf8716affb2e743edcf9a60909ac9b9
Author: Pei He 
Date:   2017-07-17T15:34:27Z

[BEAM-2630] TestPipeline: construct job/app names based on Description in 
junit TestRule.




> TestPipeline.getAppName() doesn't work properly when TestPipeline is used as 
> @Rule
> --
>
> Key: BEAM-2630
> URL: https://issues.apache.org/jira/browse/BEAM-2630
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Pei He
>Assignee: Pei He
>Priority: Minor
> Fix For: 2.2.0
>
>
> Currently, TestPipeline.getAppName() looks at the stacktraces. However, when 
> @Rule is used, it is invoked from TestPipeline.init(). And, job names of 
> ValidatesRunnerTests all becomes pardo-init for tests in ParDo. (Same for 
> other ValidatesRunnerTests)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-2630) TestPipeline.getAppName() doesn't work properly when TestPipeline is used as @Rule

[Pei He (JIRA)]

Pei He created BEAM-2630:


 Summary: TestPipeline.getAppName() doesn't work properly when 
TestPipeline is used as @Rule
 Key: BEAM-2630
 URL: https://issues.apache.org/jira/browse/BEAM-2630
 Project: Beam
  Issue Type: Bug
  Components: testing
Reporter: Pei He
Assignee: Pei He
Priority: Minor
 Fix For: 2.2.0


Currently, TestPipeline.getAppName() looks at the stacktraces. However, when 
@Rule is used, it is invoked from TestPipeline.init(). And, job names of 
ValidatesRunnerTests all becomes pardo-init for tests in ParDo. (Same for other 
ValidatesRunnerTests)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #4393

[Apache Jenkins Server]

See 

Jenkins build is back to stable : beam_PostCommit_Java_MavenInstall #4392

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2647

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2629) Adding timestamp and id attributes to pubsub messages with no other attributes causes an exception.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090949#comment-16090949
 ] 

ASF GitHub Bot commented on BEAM-2629:
--

GitHub user nkilmer opened a pull request:

https://github.com/apache/beam/pull/3581

[BEAM-2629] Fixed adding timestamp and id attributes to pubsub messages

See the writeup here: https://issues.apache.org/jira/browse/BEAM-2629

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/nkilmer/beam pubsub-attribute-fix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3581.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3581


commit e50e5f6b56eee3acaa17ebb7b840a942ac178c6a
Author: Nigel Kilmer 
Date:   2017-07-18T01:09:57Z

Fixed adding timestamp and id attributes to pubsub messages




> Adding timestamp and id attributes to pubsub messages with no other 
> attributes causes an exception.
> ---
>
> Key: BEAM-2629
> URL: https://issues.apache.org/jira/browse/BEAM-2629
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Affects Versions: 2.2.0
>Reporter: Nigel Kilmer
>Assignee: Nigel Kilmer
>
> I'm trying to use PubsubIO.writeProtos along with a timestamp and id 
> attribute. When PubsubIO tries to write the messages to my topic, it throws a 
> PipelineExecutionException with this cause:
> Caused by: java.lang.UnsupportedOperationException
>   at java.util.AbstractMap.put(AbstractMap.java:209)
>   at 
> org.apache.beam.sdk.io.gcp.pubsub.PubsubJsonClient.publish(PubsubJsonClient.java:148)
>   at 
> org.apache.beam.sdk.io.gcp.pubsub.PubsubUnboundedSink$WriterFn.publishBatch(PubsubUnboundedSink.java:249)
>   at 
> org.apache.beam.sdk.io.gcp.pubsub.PubsubUnboundedSink$WriterFn.processElement(PubsubUnboundedSink.java:285)
> Looks like the problem is that PubsubJsonClient attempts to add the timestamp 
> and id attributes to a map which may be empty and immutable if no other 
> attributes are specified on the message. I will submit a pull request with a 
> fix for this bug shortly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3581: [BEAM-2629] Fixed adding timestamp and id attribute...

[nkilmer]

GitHub user nkilmer opened a pull request:

https://github.com/apache/beam/pull/3581

[BEAM-2629] Fixed adding timestamp and id attributes to pubsub messages

See the writeup here: https://issues.apache.org/jira/browse/BEAM-2629

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/nkilmer/beam pubsub-attribute-fix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3581.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3581


commit e50e5f6b56eee3acaa17ebb7b840a942ac178c6a
Author: Nigel Kilmer 
Date:   2017-07-18T01:09:57Z

Fixed adding timestamp and id attributes to pubsub messages




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-2629) Adding timestamp and id attributes to pubsub messages with no other attributes causes an exception.

[Nigel Kilmer (JIRA)]

Nigel Kilmer created BEAM-2629:
--

 Summary: Adding timestamp and id attributes to pubsub messages 
with no other attributes causes an exception.
 Key: BEAM-2629
 URL: https://issues.apache.org/jira/browse/BEAM-2629
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-gcp
Affects Versions: 2.2.0
Reporter: Nigel Kilmer
Assignee: Nigel Kilmer


I'm trying to use PubsubIO.writeProtos along with a timestamp and id attribute. 
When PubsubIO tries to write the messages to my topic, it throws a 
PipelineExecutionException with this cause:

Caused by: java.lang.UnsupportedOperationException
at java.util.AbstractMap.put(AbstractMap.java:209)
at 
org.apache.beam.sdk.io.gcp.pubsub.PubsubJsonClient.publish(PubsubJsonClient.java:148)
at 
org.apache.beam.sdk.io.gcp.pubsub.PubsubUnboundedSink$WriterFn.publishBatch(PubsubUnboundedSink.java:249)
at 
org.apache.beam.sdk.io.gcp.pubsub.PubsubUnboundedSink$WriterFn.processElement(PubsubUnboundedSink.java:285)

Looks like the problem is that PubsubJsonClient attempts to add the timestamp 
and id attributes to a map which may be empty and immutable if no other 
attributes are specified on the message. I will submit a pull request with a 
fix for this bug shortly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2646

[Apache Jenkins Server]

See 

[beam-site] 01/01: Prepare repository for deployment.

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit c18b82822f0523a45886177b869dde51d9a099a8
Author: Mergebot 
AuthorDate: Tue Jul 18 01:08:14 2017 +

Prepare repository for deployment.
---
 .../runners/capability-matrix/index.html   | 72 +-
 1 file changed, 71 insertions(+), 1 deletion(-)

diff --git a/content/documentation/runners/capability-matrix/index.html 
b/content/documentation/runners/capability-matrix/index.html
index ad89694..6df48e7 100644
--- a/content/documentation/runners/capability-matrix/index.html
+++ b/content/documentation/runners/capability-matrix/index.html
@@ -439,6 +439,41 @@
   
   
   
+Splittable DoFn
+
+
+
+~
+
+
+
+
+~
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+  
+  
+  
 Metrics
 
 
@@ -1468,7 +1503,7 @@
 
 
 
-Yes: fully 
supported
+Yes: fully 
supportedSupport includes autotuning features 
(https://cloud.google.com/dataflow/service/dataflow-service-desc#autotuning-features).
 
 
 
@@ -1494,6 +1529,41 @@
   
   
   
+Splittable DoFn
+
+
+
+Partially: DoFn 
where processing of each element can be split for parallelism, or suspended and 
resumedAllows users to develop DoFn's that process a single 
element in portions ("restrictions"), executed in parallel or sequentially. 
This supersedes the unbounded and bounded `Source` APIs by supporting all of 
their features on a per-element basis. See http://s.apache.org/splittable [...]
+
+
+
+
+Partially: supported 
in streaming modeSupports checkpointing (sequentially 
processing an element over several, potentially infinitely many calls). Does 
not yet support autotuning features of the Source API.
+
+
+
+
+No: implementation 
in streaming mode coming soon
+
+
+
+
+No: not 
implemented
+
+
+
+
+No: implementation 
in streaming mode coming soon
+
+
+
+
+No: not 
implemented
+
+
+  
+  
+  
 Metrics
 
 

-- 
To stop receiving notification emails like this one, please contact
"commits@beam.apache.org" .

[beam-site] branch asf-site updated (1d1840e -> c18b828)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from 1d1840e  Prepare repository for deployment.
 add 6defbe2  Includes Splittable DoFn in the capability matrix
 add 992a3fc  This closes #259
 new c18b828  Prepare repository for deployment.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../runners/capability-matrix/index.html   | 72 +-
 src/_data/capability-matrix.yml| 28 -
 2 files changed, 98 insertions(+), 2 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
['"commits@beam.apache.org" '].

[beam-site] 01/02: Includes Splittable DoFn in the capability matrix

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 6defbe2279ac707ed048a47dca758907d24fffd2
Author: Eugene Kirpichov 
AuthorDate: Fri Jun 23 11:05:10 2017 -0700

Includes Splittable DoFn in the capability matrix
---
 src/_data/capability-matrix.yml | 28 +++-
 1 file changed, 27 insertions(+), 1 deletion(-)

diff --git a/src/_data/capability-matrix.yml b/src/_data/capability-matrix.yml
index 516033c..775e0da 100644
--- a/src/_data/capability-matrix.yml
+++ b/src/_data/capability-matrix.yml
@@ -185,7 +185,7 @@ categories:
   - class: dataflow
 l1: 'Yes'
 l2: fully supported
-l3: 
+l3: Support includes autotuning features 
(https://cloud.google.com/dataflow/service/dataflow-service-desc#autotuning-features).
   - class: flink
 l1: 'Yes'
 l2: fully supported
@@ -202,6 +202,32 @@ categories:
 l1: 'Yes'
 l2: fully supported
 l3: ''
+  - name: Splittable DoFn
+values:
+  - class: model
+l1: 'Partially'
+l2: DoFn where processing of each element can be split for 
parallelism, or suspended and resumed
+l3: Allows users to develop DoFn's that process a single element 
in portions ("restrictions"), executed in parallel or sequentially. This 
supersedes the unbounded and bounded `Source` APIs by supporting all of their 
features on a per-element basis. See http://s.apache.org/splittable-do-fn. 
Design is in progress on achieving parity with Source API regarding progress 
signals.
+  - class: dataflow
+l1: 'Partially'
+l2: supported in streaming mode
+l3: Supports checkpointing (sequentially processing an element 
over several, potentially infinitely many calls). Does not yet support 
autotuning features of the Source API.
+  - class: flink
+l1: 'No'
+l2: implementation in streaming mode coming soon
+l3:
+  - class: spark
+l1: 'No'
+l2: not implemented
+l3:
+  - class: apex
+l1: 'No'
+l2: implementation in streaming mode coming soon
+l3:
+  - class: gearpump
+l1: 'No'
+l2: not implemented
+l3:
   - name: Metrics
 values:
   - class: model

-- 
To stop receiving notification emails like this one, please contact
"commits@beam.apache.org" .

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #3419

[Apache Jenkins Server]

See 

[beam-site] 02/02: This closes #259

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 992a3fca8b8f582b28df41fe15aac55accce06ad
Merge: 1d1840e 6defbe2
Author: Mergebot 
AuthorDate: Tue Jul 18 01:06:19 2017 +

This closes #259

 src/_data/capability-matrix.yml | 28 +++-
 1 file changed, 27 insertions(+), 1 deletion(-)

-- 
To stop receiving notification emails like this one, please contact
"commits@beam.apache.org" .

[beam-site] branch mergebot updated (694c4c8 -> 992a3fc)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


 discard 694c4c8  This closes #257
 discard 8f5046a  [BEAM-2500] List Amazon S3 File System as a planned I/O.
 new 6defbe2  Includes Splittable DoFn in the capability matrix
 new 992a3fc  This closes #259

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (694c4c8)
\
 N -- N -- N   refs/heads/mergebot (992a3fc)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 src/_data/capability-matrix.yml  | 28 +++-
 src/documentation/io/built-in.md |  4 
 2 files changed, 27 insertions(+), 5 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
['"commits@beam.apache.org" '].

Build failed in Jenkins: beam_PerformanceTests_Python #113

[Apache Jenkins Server]

See 


Changes:

[robertwb] Improving labeling of side inputs for Dataflow

[robertwb] [BEAM-1502] GroupByKey should not return bare lists in DirectRunner.

[klk] [BEAM-933] Fix and enable findbugs in Java examples

[klk] Adjust pull request template for Jenkins and mergebot world

[tgroh] Make Dataflow Counter Name Parsing more Robust

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam6 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 0f06eb25bcc9c6bf9fb596a6ddc3a853f339b74d (origin/master)
Commit message: "This closes #3463"
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 0f06eb25bcc9c6bf9fb596a6ddc3a853f339b74d
 > git rev-list 7e4719cd01f412c000bab61f99c687ab5c5491cc # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8620640485434532347.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins150272336230902180.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins7038663871739593923.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): python-gflags==3.1.1 
in /home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
/usr/lib/python2.7/dist-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Cleaning up...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins9007071308420017361.sh
+ pip install --user -e 'sdks/python/[gcp,test]'
Obtaining 
file://
  Running setup.py 
(path:
 egg_info for package from 
file://

:66:
 UserWarning: You are using version 1.5.4 of pip. However, version 7.0.0 is 
recommended.
  _PIP_VERSION, REQUIRED_PIP_VERSION
  

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3596

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-1234) Consider a hint ParDo.withHighFanout()

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090765#comment-16090765
 ] 

ASF GitHub Bot commented on BEAM-1234:
--

GitHub user robertwb opened a pull request:

https://github.com/apache/beam/pull/3580

Let IsBounded take True value.

This is useful for languages like Python that may use this in a conditional 
statement (or allow assignment from True->1/False->0).

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-1234] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-1234` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/robertwb/incubator-beam patch-9

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3580.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3580


commit 0f0d2df88db40ad2b5b3893bf152b6221d68ca50
Author: Robert Bradshaw 
Date:   2017-07-17T23:01:18Z

Let IsBounded take True value.

This is useful for languages like Python that may use this in a conditional 
statement (or allow assignment from True->1/False->0).




> Consider a hint ParDo.withHighFanout()
> --
>
> Key: BEAM-1234
> URL: https://issues.apache.org/jira/browse/BEAM-1234
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Eugene Kirpichov
>Priority: Minor
>
> I'm finding myself again and again suggesting users on StackOverflow to 
> insert fusion breaks after high-fanout ParDo's.
> I think we should just implement this as a hint on ParDo and MapElements 
> transforms, like we have on GroupByKey.fewKeys() or 
> Combine.withHotKeyFanout().
> E.g.: c.apply(ParDo.of(some high-fanout DoFn).withHighFanout()), and a runner 
> that implements fusion could decide to insert a runner-specific fusion break. 
> This somewhat sidesteps the issues in 
> https://issues.apache.org/jira/browse/BEAM-730 and 
> https://lists.apache.org/thread.html/ac34c9ac665a8d9f67b0254015e44c59ea65ecc1360d4014b95d3b2e@%3Cdev.beam.apache.org%3E
>  because every runner can decide how to do the right thing, or is free to 
> ignore the hint.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3463: More robust Counter name parsing

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3463


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: This closes #3463

[tgroh]

Repository: beam
Updated Branches:
  refs/heads/master 04d364d31 -> 0f06eb25b


This closes #3463


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/0f06eb25
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/0f06eb25
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/0f06eb25

Branch: refs/heads/master
Commit: 0f06eb25bcc9c6bf9fb596a6ddc3a853f339b74d
Parents: 04d364d c5ebbff
Author: Thomas Groh 
Authored: Mon Jul 17 16:01:38 2017 -0700
Committer: Thomas Groh 
Committed: Mon Jul 17 16:01:38 2017 -0700

--
 .../beam/runners/dataflow/DataflowMetrics.java  | 30 +++
 .../runners/dataflow/DataflowMetricsTest.java   | 53 +++-
 2 files changed, 59 insertions(+), 24 deletions(-)
--

[GitHub] beam pull request #3580: Let IsBounded take True value.

[robertwb]

GitHub user robertwb opened a pull request:

https://github.com/apache/beam/pull/3580

Let IsBounded take True value.

This is useful for languages like Python that may use this in a conditional 
statement (or allow assignment from True->1/False->0).

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-1234] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-1234` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/robertwb/incubator-beam patch-9

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3580.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3580


commit 0f0d2df88db40ad2b5b3893bf152b6221d68ca50
Author: Robert Bradshaw 
Date:   2017-07-17T23:01:18Z

Let IsBounded take True value.

This is useful for languages like Python that may use this in a conditional 
statement (or allow assignment from True->1/False->0).




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: Make Dataflow Counter Name Parsing more Robust

[tgroh]

Make Dataflow Counter Name Parsing more Robust


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/c5ebbff5
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/c5ebbff5
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/c5ebbff5

Branch: refs/heads/master
Commit: c5ebbff584834d16e3aff9859c90122cf9ed5ef2
Parents: 04d364d
Author: Pablo 
Authored: Wed Jun 28 15:20:53 2017 -0700
Committer: Thomas Groh 
Committed: Mon Jul 17 16:01:38 2017 -0700

--
 .../beam/runners/dataflow/DataflowMetrics.java  | 30 +++
 .../runners/dataflow/DataflowMetricsTest.java   | 53 +++-
 2 files changed, 59 insertions(+), 24 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/c5ebbff5/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowMetrics.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowMetrics.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowMetrics.java
index 31b6cda..330cc7e 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowMetrics.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowMetrics.java
@@ -79,9 +79,14 @@ class DataflowMetrics extends MetricResults {
   private MetricKey metricHashKey(
   com.google.api.services.dataflow.model.MetricUpdate metricUpdate) {
 String fullStepName = metricUpdate.getName().getContext().get("step");
-fullStepName = (dataflowPipelineJob.transformStepNames != null
-? dataflowPipelineJob.transformStepNames
-.inverse().get(fullStepName).getFullName() : fullStepName);
+if (dataflowPipelineJob.transformStepNames == null
+|| 
!dataflowPipelineJob.transformStepNames.inverse().containsKey(fullStepName)) {
+  // If we can't translate internal step names to user step names, we just 
skip them
+  // altogether.
+  return null;
+}
+fullStepName = dataflowPipelineJob.transformStepNames
+.inverse().get(fullStepName).getFullName();
 return MetricKey.create(
 fullStepName,
 MetricName.named(
@@ -119,15 +124,18 @@ class DataflowMetrics extends MetricResults {
 // If the Context of the metric update does not have a namespace, then 
these are not
 // actual metrics counters.
 for (com.google.api.services.dataflow.model.MetricUpdate update : 
metricUpdates) {
-  if (Objects.equal(update.getName().getOrigin(), "user") && 
isMetricTentative(update)
+  if (Objects.equal(update.getName().getOrigin(), "user")
   && update.getName().getContext().containsKey("namespace")) {
-tentativeByName.put(metricHashKey(update), update);
-metricHashKeys.add(metricHashKey(update));
-  } else if (Objects.equal(update.getName().getOrigin(), "user")
-  && update.getName().getContext().containsKey("namespace")
-  && !isMetricTentative(update)) {
-committedByName.put(metricHashKey(update), update);
-metricHashKeys.add(metricHashKey(update));
+MetricKey key = metricHashKey(update);
+if (key == null) {
+  continue;
+}
+metricHashKeys.add(key);
+if (isMetricTentative(update)) {
+  tentativeByName.put(key, update);
+} else {
+  committedByName.put(key, update);
+}
   }
 }
 // Create the lists with the metric result information.

http://git-wip-us.apache.org/repos/asf/beam/blob/c5ebbff5/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowMetricsTest.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowMetricsTest.java
 
b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowMetricsTest.java
index 85a0979..c3c741c 100644
--- 
a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowMetricsTest.java
+++ 
b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowMetricsTest.java
@@ -33,6 +33,7 @@ import com.google.api.services.dataflow.model.Job;
 import com.google.api.services.dataflow.model.JobMetrics;
 import com.google.api.services.dataflow.model.MetricStructuredName;
 import com.google.api.services.dataflow.model.MetricUpdate;
+import com.google.common.collect.HashBiMap;
 import com.google.common.collect.ImmutableList;
 import com.google.common.collect.ImmutableMap;
 import java.io.IOException;
@@ -42,6 +43,7 @@ import 

[GitHub] beam pull request #3575: [BEAM-2625] Update pull request template for mergeb...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3575


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2625) Pull request template could use a little more detail

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090763#comment-16090763
 ] 

ASF GitHub Bot commented on BEAM-2625:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3575


> Pull request template could use a little more detail
> 
>
> Key: BEAM-2625
> URL: https://issues.apache.org/jira/browse/BEAM-2625
> Project: Beam
>  Issue Type: Bug
>  Components: project-management
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Minor
>
> In the mergebot world where we might have contributors curate good commits 
> more frequently, adding that to the PR checklist seems nice.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[2/2] beam git commit: This closes #3575: Adjust pull request template for Jenkins and mergebot world

[kenn]

This closes #3575: Adjust pull request template for Jenkins and mergebot world


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/04d364d3
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/04d364d3
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/04d364d3

Branch: refs/heads/master
Commit: 04d364d31959f044c7ccc7b9fc52884f4ae501d7
Parents: 1996869 4c6fa39
Author: Kenneth Knowles 
Authored: Mon Jul 17 16:00:09 2017 -0700
Committer: Kenneth Knowles 
Committed: Mon Jul 17 16:00:09 2017 -0700

--
 .github/PULL_REQUEST_TEMPLATE.md | 16 +++-
 1 file changed, 7 insertions(+), 9 deletions(-)
--

[1/2] beam git commit: Adjust pull request template for Jenkins and mergebot world

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master 199686907 -> 04d364d31


Adjust pull request template for Jenkins and mergebot world

Adds details about making a good series of commits, while removing advice that
the user do things that Jenkins will do for them.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/4c6fa39f
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/4c6fa39f
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/4c6fa39f

Branch: refs/heads/master
Commit: 4c6fa39f619709ff127ca8418121ad91afa2041b
Parents: 7e4719c
Author: Kenneth Knowles 
Authored: Mon Jul 17 13:06:26 2017 -0700
Committer: Kenneth Knowles 
Committed: Mon Jul 17 15:59:39 2017 -0700

--
 .github/PULL_REQUEST_TEMPLATE.md | 16 +++-
 1 file changed, 7 insertions(+), 9 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/4c6fa39f/.github/PULL_REQUEST_TEMPLATE.md
--
diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md
index 868edd1..750 100644
--- a/.github/PULL_REQUEST_TEMPLATE.md
+++ b/.github/PULL_REQUEST_TEMPLATE.md
@@ -1,12 +1,10 @@
-Be sure to do all of the following to help us incorporate your contribution
-quickly and easily:
+Follow this checklist to help us incorporate your contribution quickly and 
easily:
 
- - [ ] Make sure the PR title is formatted like:
-   `[BEAM-] Description of pull request`
- - [ ] Make sure tests pass via `mvn clean verify`.
- - [ ] Replace `` in the title with the actual Jira issue
-   number, if there is one.
- - [ ] If this contribution is large, please file an Apache
-   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).
+ - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
+ - [ ] Each commit in the pull request should have a meaningful subject line 
and body.
+ - [ ] Format the pull request title like `[BEAM-1234] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-1234` with the appropriate JIRA 
issue.
+ - [ ] Write a pull request description that is detailed enough to understand 
what the pull request does, how, and why.
+ - [ ] Run `mvn clean verify` to make sure basic checks pass. A more thorough 
check will be performed on your pull request automatically.
+ - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
 
 ---

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #4391

[Apache Jenkins Server]

See 

[GitHub] beam pull request #3579: [BEAM-2453] Perform a Multi-step combine in the Dir...

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/3579

[BEAM-2453] Perform a Multi-step combine in the DirectRunner

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
This exercises the entire CombineFn lifecycle for simple combine fns,
expressed as a collection of DoFns.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam combiner_lifting_direct_runner

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3579.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3579


commit 522fa542eeeb7a698982ef42b4d3f77ef642f59e
Author: Thomas Groh 
Date:   2017-06-15T22:53:46Z

Perform a Multi-step combine in the DirectRunner

This exercises the entire CombineFn lifecycle for simple combine fns,
expressed as a collection of DoFns.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2453) The Java DirectRunner should exercise all parts of a CombineFn

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090755#comment-16090755
 ] 

ASF GitHub Bot commented on BEAM-2453:
--

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/3579

[BEAM-2453] Perform a Multi-step combine in the DirectRunner

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
This exercises the entire CombineFn lifecycle for simple combine fns,
expressed as a collection of DoFns.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam combiner_lifting_direct_runner

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3579.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3579


commit 522fa542eeeb7a698982ef42b4d3f77ef642f59e
Author: Thomas Groh 
Date:   2017-06-15T22:53:46Z

Perform a Multi-step combine in the DirectRunner

This exercises the entire CombineFn lifecycle for simple combine fns,
expressed as a collection of DoFns.




> The Java DirectRunner should exercise all parts of a CombineFn
> --
>
> Key: BEAM-2453
> URL: https://issues.apache.org/jira/browse/BEAM-2453
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>
> Specifically it should:
> Create some number of accumulators; add elements to these accumulators, merge 
> the created accumulators, and extract the output.
> This can be performed by replacing the {{Combine.perKey}} composite transform 
> with a multi-step combine {{CombineBundles -> GroupByKey -> 
> MergeAccumulators}}
> Where {{CombineBundles}} is a {{ParDo}} which takes input {{KV}} 
> and produces {{KV}}, outputting in {{FinishBundle}} (this can only 
> be performed if the Combine takes no side inputs or does not have merging 
> windows). {{MergeAccumulators}} takes in {{KV}} and 
> produces {{KV}} by merging all of the accumulators and extracting 
> the output.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (BEAM-933) Findbugs doesn't pass in Java Examples

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles resolved BEAM-933.
--
   Resolution: Fixed
Fix Version/s: 2.2.0

> Findbugs doesn't pass in Java Examples
> --
>
> Key: BEAM-933
> URL: https://issues.apache.org/jira/browse/BEAM-933
> Project: Beam
>  Issue Type: Bug
>  Components: examples-java
>Reporter: Daniel Halperin
>Assignee: Almas Shaikh
>  Labels: newbie, starter
> Fix For: 2.2.0
>
>
> {code}
> [INFO] --- findbugs-maven-plugin:3.0.1:check (default) @ beam-examples-java 
> ---
> [INFO] BugInstance size is 4
> [INFO] Error size is 0
> [INFO] Total bugs: 4
> [INFO] Possible null pointer dereference in 
> org.apache.beam.examples.complete.TfIdf.listInputDocuments(TfIdf$Options) due 
> to return value of called method [org.apache.beam.examples.complete.TfIdf, 
> org.apache.beam.examples.complete.TfIdf] Dereferenced at TfIdf.java:[line 
> 124]Known null at TfIdf.java:[line 124]
> [INFO] Bad attempt to compute absolute value of signed 32-bit hashcode in 
> org.apache.beam.examples.complete.TopWikipediaSessions$ComputeTopSessions$1.processElement(DoFn$ProcessContext)
>  
> [org.apache.beam.examples.complete.TopWikipediaSessions$ComputeTopSessions$1] 
> At TopWikipediaSessions.java:[line 165]
> [INFO] org.apache.beam.examples.complete.TrafficRoutes$StationSpeed defines 
> compareTo(TrafficRoutes$StationSpeed) and uses Object.equals() 
> [org.apache.beam.examples.complete.TrafficRoutes$StationSpeed] At 
> TrafficRoutes.java:[line 113]
> [INFO] 
> org.apache.beam.examples.cookbook.TriggerExample$InsertDelays.processElement(DoFn$ProcessContext)
>  uses the nextDouble method of Random to generate a random integer; using 
> nextInt is more efficient 
> [org.apache.beam.examples.cookbook.TriggerExample$InsertDelays] At 
> TriggerExample.java:[line 479]
> [INFO] 
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3534: Beam-933 Findbugs doesn't pass in Java Examples

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3534


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #3534: [BEAM-933] Fix and enable findbugs in Java examples

[kenn]

This closes #3534: [BEAM-933] Fix and enable findbugs in Java examples


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/19968690
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/19968690
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/19968690

Branch: refs/heads/master
Commit: 19968690767d72fb384d4b3be899b3fb9efd6471
Parents: 02905c2 f6daad4
Author: Kenneth Knowles 
Authored: Mon Jul 17 15:52:24 2017 -0700
Committer: Kenneth Knowles 
Committed: Mon Jul 17 15:52:24 2017 -0700

--
 examples/java/pom.xml   | 12 --
 .../apache/beam/examples/complete/TfIdf.java|  3 ++-
 .../examples/complete/TopWikipediaSessions.java | 24 ++--
 .../beam/examples/complete/TrafficRoutes.java   | 19 
 .../beam/examples/cookbook/TriggerExample.java  |  6 +++--
 5 files changed, 37 insertions(+), 27 deletions(-)
--

[1/2] beam git commit: [BEAM-933] Fix and enable findbugs in Java examples

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master 02905c27b -> 199686907


[BEAM-933] Fix and enable findbugs in Java examples


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/f6daad4f
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/f6daad4f
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/f6daad4f

Branch: refs/heads/master
Commit: f6daad4fc95cb633794c60254c6c335602f1df31
Parents: 02905c2
Author: eralmas7 
Authored: Sun Jul 9 11:50:52 2017 +0530
Committer: Kenneth Knowles 
Committed: Mon Jul 17 15:52:08 2017 -0700

--
 examples/java/pom.xml   | 12 --
 .../apache/beam/examples/complete/TfIdf.java|  3 ++-
 .../examples/complete/TopWikipediaSessions.java | 24 ++--
 .../beam/examples/complete/TrafficRoutes.java   | 19 
 .../beam/examples/cookbook/TriggerExample.java  |  6 +++--
 5 files changed, 37 insertions(+), 27 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/f6daad4f/examples/java/pom.xml
--
diff --git a/examples/java/pom.xml b/examples/java/pom.xml
index ae64a79..12fe06f 100644
--- a/examples/java/pom.xml
+++ b/examples/java/pom.xml
@@ -365,18 +365,6 @@
   
 
   
-
-  
-
-
-  org.codehaus.mojo
-  findbugs-maven-plugin
-  
-true
-  
-
-  
-
 
 
   

http://git-wip-us.apache.org/repos/asf/beam/blob/f6daad4f/examples/java/src/main/java/org/apache/beam/examples/complete/TfIdf.java
--
diff --git 
a/examples/java/src/main/java/org/apache/beam/examples/complete/TfIdf.java 
b/examples/java/src/main/java/org/apache/beam/examples/complete/TfIdf.java
index 7552b94..435ffab 100644
--- a/examples/java/src/main/java/org/apache/beam/examples/complete/TfIdf.java
+++ b/examples/java/src/main/java/org/apache/beam/examples/complete/TfIdf.java
@@ -17,6 +17,7 @@
  */
 package org.apache.beam.examples.complete;
 
+import com.google.common.base.Optional;
 import java.io.File;
 import java.io.IOException;
 import java.net.URI;
@@ -121,7 +122,7 @@ public class TfIdf {
 Set uris = new HashSet<>();
 if (absoluteUri.getScheme().equals("file")) {
   File directory = new File(absoluteUri);
-  for (String entry : directory.list()) {
+  for (String entry : Optional.fromNullable(directory.list()).or(new 
String[] {})) {
 File path = new File(directory, entry);
 uris.add(path.toURI());
   }

http://git-wip-us.apache.org/repos/asf/beam/blob/f6daad4f/examples/java/src/main/java/org/apache/beam/examples/complete/TopWikipediaSessions.java
--
diff --git 
a/examples/java/src/main/java/org/apache/beam/examples/complete/TopWikipediaSessions.java
 
b/examples/java/src/main/java/org/apache/beam/examples/complete/TopWikipediaSessions.java
index 478e2dc..3691e53 100644
--- 
a/examples/java/src/main/java/org/apache/beam/examples/complete/TopWikipediaSessions.java
+++ 
b/examples/java/src/main/java/org/apache/beam/examples/complete/TopWikipediaSessions.java
@@ -162,17 +162,18 @@ public class TopWikipediaSessions {
 public PCollection expand(PCollection input) {
   return input
   .apply(ParDo.of(new ExtractUserAndTimestamp()))
-
-  .apply("SampleUsers", ParDo.of(
-  new DoFn() {
-@ProcessElement
-public void processElement(ProcessContext c) {
-  if (Math.abs(c.element().hashCode()) <= Integer.MAX_VALUE * 
samplingThreshold) {
-c.output(c.element());
-  }
-}
-  }))
-
+  .apply(
+  "SampleUsers",
+  ParDo.of(
+  new DoFn() {
+@ProcessElement
+public void processElement(ProcessContext c) {
+  if (Math.abs((long) c.element().hashCode())
+  <= Integer.MAX_VALUE * samplingThreshold) {
+c.output(c.element());
+  }
+}
+  }))
   .apply(new ComputeSessions())
   .apply("SessionsToStrings", ParDo.of(new SessionsToStringsDoFn()))
   .apply(new TopPerMonth())
@@ -191,7 +192,6 @@ public class TopWikipediaSessions {
 @Default.String(EXPORTED_WIKI_TABLE)
 String getInput();
 void setInput(String value);
-
 @Description("File to output results to")
 @Validation.Required
 String getOutput();

[beam-site] 01/02: [BEAM-2500] List Amazon S3 File System as a planned I/O.

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 8f5046ad89dc9e5daa502daf535b90610830e182
Author: Luke Cwik 
AuthorDate: Wed Jul 12 12:17:16 2017 -0700

[BEAM-2500] List Amazon S3 File System as a planned I/O.
---
 src/documentation/io/built-in.md | 4 
 1 file changed, 4 insertions(+)

diff --git a/src/documentation/io/built-in.md b/src/documentation/io/built-in.md
index 0927d10..2db36e5 100644
--- a/src/documentation/io/built-in.md
+++ b/src/documentation/io/built-in.md
@@ -72,6 +72,10 @@ This table contains I/O transforms that are currently 
planned or in-progress. St
 NameLanguageJIRA
   
   
+Amazon S3 File SystemJava
+https://issues.apache.org/jira/browse/BEAM-2500;>BEAM-2500
+  
+  
 Apache DistributedLogJava
 https://issues.apache.org/jira/browse/BEAM-607;>BEAM-607
   

-- 
To stop receiving notification emails like this one, please contact
"commits@beam.apache.org" .

[beam-site] 02/02: This closes #257

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 694c4c8b242b8d0cfb07ffd0fa95d84f20ed9cc9
Merge: 1d1840e 8f5046a
Author: Mergebot 
AuthorDate: Mon Jul 17 22:29:01 2017 +

This closes #257

 src/documentation/io/built-in.md | 4 
 1 file changed, 4 insertions(+)

-- 
To stop receiving notification emails like this one, please contact
"commits@beam.apache.org" .

[beam-site] branch mergebot updated (350bf24 -> 694c4c8)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from 350bf24  This closes #271
 add 1d1840e  Prepare repository for deployment.
 new 8f5046a  [BEAM-2500] List Amazon S3 File System as a planned I/O.
 new 694c4c8  This closes #257

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/contribute/ptransform-style-guide/index.html | 3 ++-
 content/documentation/runners/direct/index.html  | 2 +-
 src/documentation/io/built-in.md | 4 
 3 files changed, 7 insertions(+), 2 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
['"commits@beam.apache.org" '].

[jira] [Commented] (BEAM-2271) Release guide or pom.xml needs update to avoid releasing Python binary artifacts

[Sourabh Bajaj (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090671#comment-16090671
 ] 

Sourabh Bajaj commented on BEAM-2271:
-

https://github.com/apache/beam/pull/3441 creates a new zip which omits the 
{{.tox }} files etc. but I haven't been able to figure out how to override the 
actual source release file that is being created as I only see one entry for 
the execution.

[~jbonofre] do you have any ideas on what I might be doing wrong in the PR. 

> Release guide or pom.xml needs update to avoid releasing Python binary 
> artifacts
> 
>
> Key: BEAM-2271
> URL: https://issues.apache.org/jira/browse/BEAM-2271
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Daniel Halperin
>Assignee: Sourabh Bajaj
> Fix For: 2.2.0
>
>
> The following directories (and children) were discovered in 2.0.0-RC2 and 
> were present in 0.6.0.
> {code}
> sdks/python: build   dist.eggs   nose-1.3.7-py2.7.egg  (and child 
> contents)
> {code}
> Ideally, these artifacts, which are created during setup and testing, would 
> get created in the {{sdks/python/target/}} subfolder where they will 
> automatically get ignored. More info below.
> For 2.0.0, we will manually remove these files from the source release RC3+. 
> This should be fixed before the next release.
> Here is a list of other paths that get excluded, should they be useful.
> {code}
> 
> 
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/).*${project.build.directory}.*]
> 
> 
>  
> 
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?maven-eclipse\.xml]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?\.project]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?\.classpath]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?[^/]*\.iws]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?\.idea(/.*)?]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?out(/.*)?]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?[^/]*\.ipr]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?[^/]*\.iml]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?\.settings(/.*)?]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?\.externalToolBuilders(/.*)?]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?\.deployables(/.*)?]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?\.wtpmodules(/.*)?]
> 
> 
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?cobertura\.ser]
> 
> 
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?pom\.xml\.releaseBackup]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?release\.properties]
>   
> {code}
> This list is stored inside of this jar, which you can find by tracking 
> maven-assembly-plugin from the root apache pom: 
> https://mvnrepository.com/artifact/org.apache.apache.resources/apache-source-release-assembly-descriptor/1.0.6
> http://svn.apache.org/repos/asf/maven/pom/tags/apache-18/pom.xml



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (BEAM-1502) CombinePerKey returns a list of lists when run locally, but something else when run on Dataflow

[Robert Bradshaw (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Bradshaw closed BEAM-1502.
-
   Resolution: Fixed
Fix Version/s: 2.2.0

> CombinePerKey returns a list of lists when run locally, but something else 
> when run on Dataflow
> ---
>
> Key: BEAM-1502
> URL: https://issues.apache.org/jira/browse/BEAM-1502
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Pablo Estrada
>Assignee: Charles Chen
> Fix For: 2.2.0
>
>
> GroupByKey also has this issue. This has been seen by customers internally 
> and externally:
> http://stackoverflow.com/questions/42276520/dataflow-0-5-5-what-does-object-of-type-unwindowedvalues-has-no-len-mean/42283279#42283279



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2645

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-1502) CombinePerKey returns a list of lists when run locally, but something else when run on Dataflow

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090661#comment-16090661
 ] 

ASF GitHub Bot commented on BEAM-1502:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3578


> CombinePerKey returns a list of lists when run locally, but something else 
> when run on Dataflow
> ---
>
> Key: BEAM-1502
> URL: https://issues.apache.org/jira/browse/BEAM-1502
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Pablo Estrada
>Assignee: Charles Chen
>
> GroupByKey also has this issue. This has been seen by customers internally 
> and externally:
> http://stackoverflow.com/questions/42276520/dataflow-0-5-5-what-does-object-of-type-unwindowedvalues-has-no-len-mean/42283279#42283279



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[2/2] beam git commit: [BEAM-1502] GroupByKey should not return bare lists in DirectRunner.

[robertwb]

[BEAM-1502] GroupByKey should not return bare lists in DirectRunner.

This leads to invalidated expectations on other runners.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/e7059e5c
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/e7059e5c
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/e7059e5c

Branch: refs/heads/master
Commit: e7059e5cb3cd07855582641798c58fc3cf5cd682
Parents: 532256e
Author: Robert Bradshaw 
Authored: Mon Jul 17 13:44:40 2017 -0700
Committer: Robert Bradshaw 
Committed: Mon Jul 17 15:08:02 2017 -0700

--
 .../apache_beam/examples/snippets/snippets.py   |  2 +-
 sdks/python/apache_beam/transforms/core.py  |  2 +-
 sdks/python/apache_beam/transforms/trigger.py   | 21 +++-
 3 files changed, 18 insertions(+), 7 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/e7059e5c/sdks/python/apache_beam/examples/snippets/snippets.py
--
diff --git a/sdks/python/apache_beam/examples/snippets/snippets.py 
b/sdks/python/apache_beam/examples/snippets/snippets.py
index 3a5f9b1..27b8120 100644
--- a/sdks/python/apache_beam/examples/snippets/snippets.py
+++ b/sdks/python/apache_beam/examples/snippets/snippets.py
@@ -1136,7 +1136,7 @@ def model_group_by_key(contents, output_path):
 grouped_words = words_and_counts | beam.GroupByKey()
 # [END model_group_by_key_transform]
 (grouped_words
- | 'count words' >> beam.Map(lambda (word, counts): (word, len(counts)))
+ | 'count words' >> beam.Map(lambda (word, counts): (word, sum(counts)))
  | beam.io.WriteToText(output_path))
 
 

http://git-wip-us.apache.org/repos/asf/beam/blob/e7059e5c/sdks/python/apache_beam/transforms/core.py
--
diff --git a/sdks/python/apache_beam/transforms/core.py 
b/sdks/python/apache_beam/transforms/core.py
index 8018219..92b8737 100644
--- a/sdks/python/apache_beam/transforms/core.py
+++ b/sdks/python/apache_beam/transforms/core.py
@@ -1017,7 +1017,7 @@ class CombineValuesDoFn(DoFn):
self.combinefn.apply(element[1], *args, **kwargs))]
 
 # Add the elements into three accumulators (for testing of merge).
-elements = element[1]
+elements = list(element[1])
 accumulators = []
 for k in range(3):
   if len(elements) <= k:

http://git-wip-us.apache.org/repos/asf/beam/blob/e7059e5c/sdks/python/apache_beam/transforms/trigger.py
--
diff --git a/sdks/python/apache_beam/transforms/trigger.py 
b/sdks/python/apache_beam/transforms/trigger.py
index f77fa1a..c1fbfc5 100644
--- a/sdks/python/apache_beam/transforms/trigger.py
+++ b/sdks/python/apache_beam/transforms/trigger.py
@@ -24,6 +24,7 @@ from abc import ABCMeta
 from abc import abstractmethod
 import collections
 import copy
+import itertools
 
 from apache_beam.coders import observable
 from apache_beam.transforms import combiners
@@ -878,6 +879,17 @@ class _UnwindowedValues(observable.ObservableMixin):
   def __reduce__(self):
 return list, (list(self),)
 
+  def __eq__(self, other):
+if isinstance(other, collections.Iterable):
+  return all(
+  a == b
+  for a, b in itertools.izip_longest(self, other, fillvalue=object()))
+else:
+  return NotImplemented
+
+  def __ne__(self, other):
+return not self == other
+
 
 class DefaultGlobalBatchTriggerDriver(TriggerDriver):
   """Breaks a bundles into window (pane)s according to the default triggering.
@@ -888,11 +900,10 @@ class DefaultGlobalBatchTriggerDriver(TriggerDriver):
 pass
 
   def process_elements(self, state, windowed_values, unused_output_watermark):
-if isinstance(windowed_values, list):
-  unwindowed = [wv.value for wv in windowed_values]
-else:
-  unwindowed = _UnwindowedValues(windowed_values)
-yield WindowedValue(unwindowed, MIN_TIMESTAMP, self.GLOBAL_WINDOW_TUPLE)
+yield WindowedValue(
+_UnwindowedValues(windowed_values),
+MIN_TIMESTAMP,
+self.GLOBAL_WINDOW_TUPLE)
 
   def process_timer(self, window_id, name, time_domain, timestamp, state):
 raise TypeError('Triggers never set or called for batch default 
windowing.')

[GitHub] beam pull request #3578: [BEAM-1502] GroupByKey should not return bare lists...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3578


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: Closes #3578

[robertwb]

Repository: beam
Updated Branches:
  refs/heads/master 532256e88 -> 02905c27b


Closes #3578


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/02905c27
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/02905c27
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/02905c27

Branch: refs/heads/master
Commit: 02905c27bfc59aa90ebe9c929fa060e705ff2fc3
Parents: 532256e e7059e5
Author: Robert Bradshaw 
Authored: Mon Jul 17 15:08:02 2017 -0700
Committer: Robert Bradshaw 
Committed: Mon Jul 17 15:08:02 2017 -0700

--
 .../apache_beam/examples/snippets/snippets.py   |  2 +-
 sdks/python/apache_beam/transforms/core.py  |  2 +-
 sdks/python/apache_beam/transforms/trigger.py   | 21 +++-
 3 files changed, 18 insertions(+), 7 deletions(-)
--

[jira] [Commented] (BEAM-2523) GCP IO exposes protobuf on its API surface, causing user pain

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090651#comment-16090651
 ] 

Kenneth Knowles commented on BEAM-2523:
---

Out of bandwidth for working on this one, but it does matter. You can see all 
three versions in the dependency tree.

> GCP IO exposes protobuf on its API surface, causing user pain
> -
>
> Key: BEAM-2523
> URL: https://issues.apache.org/jira/browse/BEAM-2523
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Kenneth Knowles
> Fix For: 2.2.0
>
>
> Putting the SDK, DataflowRunner, and GCP IO on the same classpath, results in 
> (at least) three versions of protobuf getting pulled in. These should be made 
> to converge. We should consider using maven enforcer, which I think can check 
> this.
> {code}
> [INFO] com.example:foo:jar:0.1
> [INFO] +- org.apache.beam:beam-sdks-java-core:jar:2.0.0:compile
> [INFO] +- 
> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.0.0:compile
> [INFO] |  +- 
> org.apache.beam:beam-sdks-java-extensions-protobuf:jar:2.0.0:compile
> [INFO] |  |  \- (com.google.protobuf:protobuf-java:jar:3.2.0:compile - 
> omitted for duplicate)
> [INFO] |  +- com.google.api.grpc:grpc-google-pubsub-v1:jar:0.1.0:compile
> [INFO] |  |  +- (com.google.protobuf:protobuf-java:jar:3.0.0:compile - 
> omitted for conflict with 3.2.0)
> [INFO] |  |  \- com.google.api.grpc:grpc-google-iam-v1:jar:0.1.0:compile
> [INFO] |  | \- (com.google.protobuf:protobuf-java:jar:3.0.0:compile - 
> omitted for conflict with 3.2.0)
> [INFO] |  +- 
> com.google.cloud.datastore:datastore-v1-proto-client:jar:1.4.0:compile
> [INFO] |  |  +- 
> (com.google.cloud.datastore:datastore-v1-protos:jar:1.3.0:compile - omitted 
> for duplicate)
> [INFO] |  |  +- (com.google.http-client:google-http-client:jar:1.20.0:compile 
> - omitted for conflict with 1.22.0)
> [INFO] |  |  +- 
> com.google.http-client:google-http-client-protobuf:jar:1.20.0:compile
> [INFO] |  |  |  +- 
> (com.google.http-client:google-http-client:jar:1.20.0:compile - omitted for 
> conflict with 1.22.0)
> [INFO] |  |  |  \- (com.google.protobuf:protobuf-java:jar:2.4.1:compile - 
> omitted for conflict with 3.2.0)
> [INFO] |  +- com.google.cloud.datastore:datastore-v1-protos:jar:1.3.0:compile
> [INFO] |  |  +- (com.google.protobuf:protobuf-java:jar:3.0.0:compile - 
> omitted for conflict with 3.2.0)
> [INFO] |  +- com.google.cloud.bigtable:bigtable-protos:jar:0.9.6.2:compile
> [INFO] |  |  +- (com.google.code.findbugs:jsr305:jar:3.0.1:compile - omitted 
> for duplicate)
> [INFO] |  |  +- (com.google.protobuf:protobuf-java:jar:3.2.0:compile - 
> omitted for duplicate)
> {code}
> Incidentally, the dependency plugin stopped supporting the verbose tree, so 
> we can't even visually inspect this except by downgrading.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (BEAM-2523) GCP IO exposes protobuf on its API surface, causing user pain

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-2523:
-

Assignee: (was: Kenneth Knowles)

> GCP IO exposes protobuf on its API surface, causing user pain
> -
>
> Key: BEAM-2523
> URL: https://issues.apache.org/jira/browse/BEAM-2523
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Kenneth Knowles
> Fix For: 2.2.0
>
>
> Putting the SDK, DataflowRunner, and GCP IO on the same classpath, results in 
> (at least) three versions of protobuf getting pulled in. These should be made 
> to converge. We should consider using maven enforcer, which I think can check 
> this.
> {code}
> [INFO] com.example:foo:jar:0.1
> [INFO] +- org.apache.beam:beam-sdks-java-core:jar:2.0.0:compile
> [INFO] +- 
> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.0.0:compile
> [INFO] |  +- 
> org.apache.beam:beam-sdks-java-extensions-protobuf:jar:2.0.0:compile
> [INFO] |  |  \- (com.google.protobuf:protobuf-java:jar:3.2.0:compile - 
> omitted for duplicate)
> [INFO] |  +- com.google.api.grpc:grpc-google-pubsub-v1:jar:0.1.0:compile
> [INFO] |  |  +- (com.google.protobuf:protobuf-java:jar:3.0.0:compile - 
> omitted for conflict with 3.2.0)
> [INFO] |  |  \- com.google.api.grpc:grpc-google-iam-v1:jar:0.1.0:compile
> [INFO] |  | \- (com.google.protobuf:protobuf-java:jar:3.0.0:compile - 
> omitted for conflict with 3.2.0)
> [INFO] |  +- 
> com.google.cloud.datastore:datastore-v1-proto-client:jar:1.4.0:compile
> [INFO] |  |  +- 
> (com.google.cloud.datastore:datastore-v1-protos:jar:1.3.0:compile - omitted 
> for duplicate)
> [INFO] |  |  +- (com.google.http-client:google-http-client:jar:1.20.0:compile 
> - omitted for conflict with 1.22.0)
> [INFO] |  |  +- 
> com.google.http-client:google-http-client-protobuf:jar:1.20.0:compile
> [INFO] |  |  |  +- 
> (com.google.http-client:google-http-client:jar:1.20.0:compile - omitted for 
> conflict with 1.22.0)
> [INFO] |  |  |  \- (com.google.protobuf:protobuf-java:jar:2.4.1:compile - 
> omitted for conflict with 3.2.0)
> [INFO] |  +- com.google.cloud.datastore:datastore-v1-protos:jar:1.3.0:compile
> [INFO] |  |  +- (com.google.protobuf:protobuf-java:jar:3.0.0:compile - 
> omitted for conflict with 3.2.0)
> [INFO] |  +- com.google.cloud.bigtable:bigtable-protos:jar:0.9.6.2:compile
> [INFO] |  |  +- (com.google.code.findbugs:jsr305:jar:3.0.1:compile - omitted 
> for duplicate)
> [INFO] |  |  +- (com.google.protobuf:protobuf-java:jar:3.2.0:compile - 
> omitted for duplicate)
> {code}
> Incidentally, the dependency plugin stopped supporting the verbose tree, so 
> we can't even visually inspect this except by downgrading.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #3418

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2532) BigQueryIO source should avoid expensive JSON schema parsing for every record

[Chamikara Jayalath (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090618#comment-16090618
 ] 

Chamikara Jayalath commented on BEAM-2532:
--

This is not planned for 2.1.0 (which is happening now). So, hopefully, the fix 
will be in 2.2.0. 

> BigQueryIO source should avoid expensive JSON schema parsing for every record
> -
>
> Key: BEAM-2532
> URL: https://issues.apache.org/jira/browse/BEAM-2532
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-gcp
>Affects Versions: 2.0.0
>Reporter: Marian Dvorsky
>Assignee: Chamikara Jayalath
>Priority: Minor
>
> BigQueryIO source converts the schema from JSON for every input row, here:
> https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQuerySourceBase.java#L159
> This is the performance bottleneck in a simple pipeline with BigQueryIO 
> source.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-2628) AvroSource.split() sequentially opens every matched file

[Eugene Kirpichov (JIRA)]

Eugene Kirpichov created BEAM-2628:
--

 Summary: AvroSource.split() sequentially opens every matched file
 Key: BEAM-2628
 URL: https://issues.apache.org/jira/browse/BEAM-2628
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Eugene Kirpichov
Assignee: Eugene Kirpichov


When you do AvroIO.read().from(filepattern), during splitting of AvroSource the 
filepattern gets expanded into N files, and then for each of the N files we do 
this: 
https://github.com/apache/beam/blob/v2.0.0/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroSource.java#L259

This is very slow. E.g. one job was reading 15,000 files, and it took almost 2 
hours to split the source because opening each file and reading schema was 
taking about 0.5s.

I'm not quite sure why we need the file metadata while splitting...



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3520: Improving labeling of side inputs for Dataflow

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3520


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: Closes #3520

[robertwb]

Repository: beam
Updated Branches:
  refs/heads/master 7e4719cd0 -> 532256e88


Closes #3520


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/532256e8
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/532256e8
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/532256e8

Branch: refs/heads/master
Commit: 532256e8811b790fdf25fb4e11b7c2b89383761a
Parents: 7e4719c 7257507
Author: Robert Bradshaw 
Authored: Mon Jul 17 14:33:01 2017 -0700
Committer: Robert Bradshaw 
Committed: Mon Jul 17 14:33:01 2017 -0700

--
 .../runners/dataflow/dataflow_runner.py   | 18 --
 1 file changed, 16 insertions(+), 2 deletions(-)
--

[2/2] beam git commit: Improving labeling of side inputs for Dataflow

[robertwb]

Improving labeling of side inputs for Dataflow


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/7257507d
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/7257507d
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/7257507d

Branch: refs/heads/master
Commit: 7257507d939271a91287837c20fcdde37dc1ddeb
Parents: 7e4719c
Author: Pablo 
Authored: Fri Jul 7 13:49:47 2017 -0700
Committer: Robert Bradshaw 
Committed: Mon Jul 17 14:33:01 2017 -0700

--
 .../runners/dataflow/dataflow_runner.py   | 18 --
 1 file changed, 16 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/7257507d/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
--
diff --git a/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py 
b/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
index 059e139..89c18d4 100644
--- a/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
+++ b/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
@@ -21,6 +21,7 @@ The runner will create a JSON description of the job graph 
and then submit it
 to the Dataflow Service for remote execution by a worker.
 """
 
+from collections import defaultdict
 import logging
 import threading
 import time
@@ -485,11 +486,24 @@ class DataflowRunner(PipelineRunner):
 si_dict = {}
 # We must call self._cache.get_pvalue exactly once due to refcounting.
 si_labels = {}
+full_label_counts = defaultdict(int)
 lookup_label = lambda side_pval: si_labels[side_pval]
 for side_pval in transform_node.side_inputs:
   assert isinstance(side_pval, AsSideInput)
-  si_label = 'SideInput-' + self._get_unique_step_name()
-  si_full_label = '%s/%s' % (transform_node.full_label, si_label)
+  step_number = self._get_unique_step_name()
+  si_label = 'SideInput-' + step_number
+  pcollection_label = '%s.%s' % (
+  side_pval.pvalue.producer.full_label.split('/')[-1],
+  side_pval.pvalue.tag if side_pval.pvalue.tag else 'out')
+  si_full_label = '%s/%s(%s.%s)' % (transform_node.full_label,
+side_pval.__class__.__name__,
+pcollection_label,
+full_label_counts[pcollection_label])
+
+  # Count the number of times the same PCollection is a side input
+  # to the same ParDo.
+  full_label_counts[pcollection_label] += 1
+
   self._add_singleton_step(
   si_label, si_full_label, side_pval.pvalue.tag,
   self._cache.get_pvalue(side_pval.pvalue))

[jira] [Commented] (BEAM-2532) BigQueryIO source should avoid expensive JSON schema parsing for every record

[Neville Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090599#comment-16090599
 ] 

Neville Li commented on BEAM-2532:
--

Would love to see a fix in the next release. This is a big performance 
regression for us since we use BigQuery heavily.

> BigQueryIO source should avoid expensive JSON schema parsing for every record
> -
>
> Key: BEAM-2532
> URL: https://issues.apache.org/jira/browse/BEAM-2532
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-gcp
>Affects Versions: 2.0.0
>Reporter: Marian Dvorsky
>Assignee: Chamikara Jayalath
>Priority: Minor
>
> BigQueryIO source converts the schema from JSON for every input row, here:
> https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQuerySourceBase.java#L159
> This is the performance bottleneck in a simple pipeline with BigQueryIO 
> source.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-1502) CombinePerKey returns a list of lists when run locally, but something else when run on Dataflow

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090545#comment-16090545
 ] 

ASF GitHub Bot commented on BEAM-1502:
--

GitHub user robertwb opened a pull request:

https://github.com/apache/beam/pull/3578

[BEAM-1502] GroupByKey should not return bare lists in DirectRunner.

This leads to invalidated expectations on other runners.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/robertwb/incubator-beam stream-iter

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3578.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3578


commit 61a3179e38d4eb19eb4815564b9ca494591b8945
Author: Robert Bradshaw 
Date:   2017-07-17T20:44:40Z

[BEAM-1502] GroupByKey should not return bare lists in DirectRunner.

This leads to invalidated expectations on other runners.




> CombinePerKey returns a list of lists when run locally, but something else 
> when run on Dataflow
> ---
>
> Key: BEAM-1502
> URL: https://issues.apache.org/jira/browse/BEAM-1502
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Pablo Estrada
>Assignee: Charles Chen
>
> GroupByKey also has this issue. This has been seen by customers internally 
> and externally:
> http://stackoverflow.com/questions/42276520/dataflow-0-5-5-what-does-object-of-type-unwindowedvalues-has-no-len-mean/42283279#42283279



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3578: [BEAM-1502] GroupByKey should not return bare lists...

[robertwb]

GitHub user robertwb opened a pull request:

https://github.com/apache/beam/pull/3578

[BEAM-1502] GroupByKey should not return bare lists in DirectRunner.

This leads to invalidated expectations on other runners.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/robertwb/incubator-beam stream-iter

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3578.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3578


commit 61a3179e38d4eb19eb4815564b9ca494591b8945
Author: Robert Bradshaw 
Date:   2017-07-17T20:44:40Z

[BEAM-1502] GroupByKey should not return bare lists in DirectRunner.

This leads to invalidated expectations on other runners.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-2627) GCP core has split package between its namespace and sdk.util

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-2627:
-

 Summary: GCP core has split package between its namespace and 
sdk.util
 Key: BEAM-2627
 URL: https://issues.apache.org/jira/browse/BEAM-2627
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-extensions
Reporter: Kenneth Knowles
Priority: Minor


Generally bad practice and probably done out of expedience. We might clean this 
up at some point, since {{util}} is officially not user facing.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2626) Java SDK harness has split packages, abuses runners/core namespace

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090533#comment-16090533
 ] 

ASF GitHub Bot commented on BEAM-2626:
--

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/3577

[BEAM-2626] Fix split package in SDK harness

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---

The Java SDK harness defined classes both in its own namespace 
org.apache.beam.fn.harness and the org.apache.beam.runners.core namespace, 
resulting in a split package across multiple jars.

Generally not a good practice since it breaks encapsulation and not 
actually useful as a hack since that method of breaking encapsulation isn't 
always compatible with classloader setups.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam harness-split-package

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3577.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3577


commit f1b4700f32c5ea39559145d6f5db3909439f6c80
Author: Kenneth Knowles 
Date:   2017-07-17T20:46:46Z

Fix split package in SDK harness

The Java SDK harness defined classes both in its own namespace
org.apache.beam.fn.harness and the org.apache.beam.runners.core namespace,
resulting in a split package across multiple jars.




> Java SDK harness has split packages, abuses runners/core namespace
> --
>
> Key: BEAM-2626
> URL: https://issues.apache.org/jira/browse/BEAM-2626
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3577: [BEAM-2626] Fix split package in SDK harness

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/3577

[BEAM-2626] Fix split package in SDK harness

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---

The Java SDK harness defined classes both in its own namespace 
org.apache.beam.fn.harness and the org.apache.beam.runners.core namespace, 
resulting in a split package across multiple jars.

Generally not a good practice since it breaks encapsulation and not 
actually useful as a hack since that method of breaking encapsulation isn't 
always compatible with classloader setups.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam harness-split-package

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3577.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3577


commit f1b4700f32c5ea39559145d6f5db3909439f6c80
Author: Kenneth Knowles 
Date:   2017-07-17T20:46:46Z

Fix split package in SDK harness

The Java SDK harness defined classes both in its own namespace
org.apache.beam.fn.harness and the org.apache.beam.runners.core namespace,
resulting in a split package across multiple jars.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2618) Add a GroupByKeyTest where the inputs are windowed into SlidingWindows

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090525#comment-16090525
 ] 

ASF GitHub Bot commented on BEAM-2618:
--

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/3576

[BEAM-2618][BEAM-2619] Add GroupByKey tests for Multiple & Merging windows

This gives explicit coverage to a GroupByKey where the elements are in
multiple windows, or in merging windows.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam group_by_key_coverage

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3576.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3576


commit f6956f7a72e661fd1ee2397f76f36da941e5af76
Author: Thomas Groh 
Date:   2017-07-17T20:38:11Z

Add GroupByKey tests for Multiple & Merging windows

This gives explicit coverage to a GroupByKey where the elements are in
multiple windows, or in merging windows.




> Add a GroupByKeyTest where the inputs are windowed into SlidingWindows
> --
>
> Key: BEAM-2618
> URL: https://issues.apache.org/jira/browse/BEAM-2618
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3576: [BEAM-2618][BEAM-2619] Add GroupByKey tests for Mul...

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/3576

[BEAM-2618][BEAM-2619] Add GroupByKey tests for Multiple & Merging windows

This gives explicit coverage to a GroupByKey where the elements are in
multiple windows, or in merging windows.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam group_by_key_coverage

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3576.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3576


commit f6956f7a72e661fd1ee2397f76f36da941e5af76
Author: Thomas Groh 
Date:   2017-07-17T20:38:11Z

Add GroupByKey tests for Multiple & Merging windows

This gives explicit coverage to a GroupByKey where the elements are in
multiple windows, or in merging windows.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3595

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-2626) Java SDK harness has split packages, abuses runners/core namespace

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-2626:
-

 Summary: Java SDK harness has split packages, abuses runners/core 
namespace
 Key: BEAM-2626
 URL: https://issues.apache.org/jira/browse/BEAM-2626
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Kenneth Knowles
Assignee: Kenneth Knowles






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is unstable: beam_PostCommit_Java_MavenInstall #4390

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2625) Pull request template could use a little more detail

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090466#comment-16090466
 ] 

ASF GitHub Bot commented on BEAM-2625:
--

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/3575

[BEAM-2625] Update pull request template

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam pr-template

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3575.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3575


commit ae422e741d1b2d29ef8fb7955a10960737943bb2
Author: Kenneth Knowles 
Date:   2017-07-17T20:06:26Z

Update pull request template




> Pull request template could use a little more detail
> 
>
> Key: BEAM-2625
> URL: https://issues.apache.org/jira/browse/BEAM-2625
> Project: Beam
>  Issue Type: Bug
>  Components: project-management
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Minor
>
> In the mergebot world where we might have contributors curate good commits 
> more frequently, adding that to the PR checklist seems nice.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3575: [BEAM-2625] Update pull request template

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/3575

[BEAM-2625] Update pull request template

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam pr-template

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3575.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3575


commit ae422e741d1b2d29ef8fb7955a10960737943bb2
Author: Kenneth Knowles 
Date:   2017-07-17T20:06:26Z

Update pull request template




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-2625) Pull request template could use a little more detail

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-2625:
-

 Summary: Pull request template could use a little more detail
 Key: BEAM-2625
 URL: https://issues.apache.org/jira/browse/BEAM-2625
 Project: Beam
  Issue Type: Bug
  Components: project-management
Reporter: Kenneth Knowles
Assignee: Kenneth Knowles
Priority: Minor


In the mergebot world where we might have contributors curate good commits more 
frequently, adding that to the PR checklist seems nice.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-1542) Need Source/Sink for Spanner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090432#comment-16090432
 ] 

ASF GitHub Bot commented on BEAM-1542:
--

GitHub user mairbek opened a pull request:

https://github.com/apache/beam/pull/3574

[BEAM-1542] Specifies a User Agent in Spanner Client

R: @jkff 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mairbek/beam useragent

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3574.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3574


commit 393f50874d4b5cf8f3ced2dcc248d1a0920bf031
Author: Mairbek Khadikov 
Date:   2017-07-17T19:37:47Z

Inline buildSpannerOptions method

commit a12a5526901eb5a6b69bbc342abcbbcff46b3e41
Author: Mairbek Khadikov 
Date:   2017-07-17T19:39:45Z

Specify user agent.




> Need Source/Sink for Spanner
> 
>
> Key: BEAM-1542
> URL: https://issues.apache.org/jira/browse/BEAM-1542
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-gcp
>Reporter: Guy Molinari
>Assignee: Mairbek Khadikov
>
> Is there a source/sink for Spanner in the works?   If not I would gladly give 
> this a shot.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3574: [BEAM-1542] Specifies a User Agent in Spanner Clien...

[mairbek]

GitHub user mairbek opened a pull request:

https://github.com/apache/beam/pull/3574

[BEAM-1542] Specifies a User Agent in Spanner Client

R: @jkff 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mairbek/beam useragent

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3574.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3574


commit 393f50874d4b5cf8f3ced2dcc248d1a0920bf031
Author: Mairbek Khadikov 
Date:   2017-07-17T19:37:47Z

Inline buildSpannerOptions method

commit a12a5526901eb5a6b69bbc342abcbbcff46b3e41
Author: Mairbek Khadikov 
Date:   2017-07-17T19:39:45Z

Specify user agent.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2644

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #3417

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-1265) Add streaming support to Python DirectRunner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090219#comment-16090219
 ] 

ASF GitHub Bot commented on BEAM-1265:
--

Github user charlesccychen closed the pull request at:

https://github.com/apache/beam/pull/3157


> Add streaming support to Python DirectRunner
> 
>
> Key: BEAM-1265
> URL: https://issues.apache.org/jira/browse/BEAM-1265
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Charles Chen
>
> Continue the work started in https://issues.apache.org/jira/browse/BEAM-428



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3157: [BEAM-1265] Do not merge: streaming scratch work

[charlesccychen]

Github user charlesccychen closed the pull request at:

https://github.com/apache/beam/pull/3157


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3594

[Apache Jenkins Server]

See 

[jira] [Closed] (BEAM-2609) refine UDF interface

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin closed BEAM-2609.

   Resolution: Fixed
Fix Version/s: 2.2.0

> refine UDF interface
> 
>
> Key: BEAM-2609
> URL: https://issues.apache.org/jira/browse/BEAM-2609
> Project: Beam
>  Issue Type: Task
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>  Labels: dsl_sql_merge
> Fix For: 2.2.0
>
>
> Some notes to improve UDF implementation:
> 1. let UDF to extend an interface;
> 2. keep one single method in a class;
> [~takidau] [~xumingming]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2271) Release guide or pom.xml needs update to avoid releasing Python binary artifacts

[Sourabh Bajaj (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sourabh Bajaj updated BEAM-2271:

Fix Version/s: (was: 2.1.0)
   2.2.0

> Release guide or pom.xml needs update to avoid releasing Python binary 
> artifacts
> 
>
> Key: BEAM-2271
> URL: https://issues.apache.org/jira/browse/BEAM-2271
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Daniel Halperin
>Assignee: Sourabh Bajaj
> Fix For: 2.2.0
>
>
> The following directories (and children) were discovered in 2.0.0-RC2 and 
> were present in 0.6.0.
> {code}
> sdks/python: build   dist.eggs   nose-1.3.7-py2.7.egg  (and child 
> contents)
> {code}
> Ideally, these artifacts, which are created during setup and testing, would 
> get created in the {{sdks/python/target/}} subfolder where they will 
> automatically get ignored. More info below.
> For 2.0.0, we will manually remove these files from the source release RC3+. 
> This should be fixed before the next release.
> Here is a list of other paths that get excluded, should they be useful.
> {code}
> 
> 
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/).*${project.build.directory}.*]
> 
> 
>  
> 
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?maven-eclipse\.xml]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?\.project]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?\.classpath]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?[^/]*\.iws]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?\.idea(/.*)?]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?out(/.*)?]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?[^/]*\.ipr]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?[^/]*\.iml]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?\.settings(/.*)?]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?\.externalToolBuilders(/.*)?]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?\.deployables(/.*)?]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?\.wtpmodules(/.*)?]
> 
> 
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?cobertura\.ser]
> 
> 
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?pom\.xml\.releaseBackup]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?release\.properties]
>   
> {code}
> This list is stored inside of this jar, which you can find by tracking 
> maven-assembly-plugin from the root apache pom: 
> https://mvnrepository.com/artifact/org.apache.apache.resources/apache-source-release-assembly-descriptor/1.0.6
> http://svn.apache.org/repos/asf/maven/pom/tags/apache-18/pom.xml



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2624) File-based sinks should produce a PCollection of written filenames

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090128#comment-16090128
 ] 

ASF GitHub Bot commented on BEAM-2624:
--

GitHub user reuvenlax opened a pull request:

https://github.com/apache/beam/pull/3573

[BEAM-2624] Allow access to created filenames from WriteFiles



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/reuvenlax/incubator-beam fbs_results

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3573.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3573


commit 11cd153b97c72d546cb941378416a7bf3c613f0d
Author: Reuven Lax 
Date:   2017-07-17T17:25:41Z

Support new output type on WriteFiles allowing access to all produced 
filenames.




> File-based sinks should produce a PCollection of written filenames
> --
>
> Key: BEAM-2624
> URL: https://issues.apache.org/jira/browse/BEAM-2624
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Davor Bonaci
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3573: [BEAM-2624] Allow access to created filenames from ...

[reuvenlax]

GitHub user reuvenlax opened a pull request:

https://github.com/apache/beam/pull/3573

[BEAM-2624] Allow access to created filenames from WriteFiles



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/reuvenlax/incubator-beam fbs_results

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3573.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3573


commit 11cd153b97c72d546cb941378416a7bf3c613f0d
Author: Reuven Lax 
Date:   2017-07-17T17:25:41Z

Support new output type on WriteFiles allowing access to all produced 
filenames.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (BEAM-2624) File-based sinks should produce a PCollection of written filenames

[Reuven Lax (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Reuven Lax updated BEAM-2624:
-
Component/s: (was: sdk-java-gcp)

> File-based sinks should produce a PCollection of written filenames
> --
>
> Key: BEAM-2624
> URL: https://issues.apache.org/jira/browse/BEAM-2624
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Davor Bonaci
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-2624) File-based sinks should produce a PCollection of written filenames

[Reuven Lax (JIRA)]

Reuven Lax created BEAM-2624:


 Summary: File-based sinks should produce a PCollection of written 
filenames
 Key: BEAM-2624
 URL: https://issues.apache.org/jira/browse/BEAM-2624
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core, sdk-java-gcp
Reporter: Reuven Lax
Assignee: Davor Bonaci






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2643

[Apache Jenkins Server]

See