date:20170509

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Flink #2761

2017-05-09 Thread Apache Jenkins Server

See

[jira] [Commented] (BEAM-2006) window support

2017-05-09 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-2006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16004146#comment-16004146
 ] 

ASF GitHub Bot commented on BEAM-2006:
--

GitHub user XuMingmin opened a pull request:

https://github.com/apache/beam/pull/3039

[BEAM-2006] window support

Add support for aggregation: _global_, _HOP_, _TUMBLE_, _SESSION_, only 
aggregation function `COUNT`


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/XuMingmin/beam BEAM-2006-2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3039.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3039






> window support 
> ---
>
> Key: BEAM-2006
> URL: https://issues.apache.org/jira/browse/BEAM-2006
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Blocker
>
> There're several methods, TUMBLE/HOP/SESSION, introduced in Calcite 1.12, 
> which represent a window-aggregation operation. 
> In BeamSQL, it's expected to leverage these methods to determine window 
> function, set trigger strategy, also handle event_time/watermark properly. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3039: [BEAM-2006] window support

2017-05-09 Thread XuMingmin

GitHub user XuMingmin opened a pull request:

https://github.com/apache/beam/pull/3039

[BEAM-2006] window support

Add support for aggregation: _global_, _HOP_, _TUMBLE_, _SESSION_, only 
aggregation function `COUNT`


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/XuMingmin/beam BEAM-2006-2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3039.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3039






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (BEAM-2244) Move runner-facing Metrics classes to runners core

2017-05-09 Thread Davor Bonaci (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-2244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-2244:
---
Fix Version/s: 2.0.0

> Move runner-facing Metrics classes to runners core
> --
>
> Key: BEAM-2244
> URL: https://issues.apache.org/jira/browse/BEAM-2244
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Ben Chambers
>Assignee: Davor Bonaci
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2241) Correctly mark top level classes and functions as private

2017-05-09 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-2241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16004133#comment-16004133
 ] 

ASF GitHub Bot commented on BEAM-2241:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3027


> Correctly mark top level classes and functions as private
> -
>
> Key: BEAM-2241
> URL: https://issues.apache.org/jira/browse/BEAM-2241
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Chamikara Jayalath
>Assignee: Chamikara Jayalath
> Fix For: 2.0.0
>
>
> Seems like some of the top level classes and function of Python SDK are 
> unnecessarily public (does not start with _ ). We should update Python SDK so 
> that top level classes and functions that are only used within the 
> corresponding modules are private. Additionally, any top level classes and 
> functions that are not expected to be used by end-users 
> (pipeline/source/ptransform authors) should be clearly documented.  



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3027: [BEAM-2241] Mark PipelineVisitor and AppliedPTransf...

2017-05-09 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3027


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: Mark PipelineVisitor and AppliedPTransform as internal.

2017-05-09 Thread chamikara

Repository: beam
Updated Branches:
  refs/heads/master 8312b6f59 -> b1a2dbeb1


Mark PipelineVisitor and AppliedPTransform as internal.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/12c9ba1a
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/12c9ba1a
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/12c9ba1a

Branch: refs/heads/master
Commit: 12c9ba1a95204daaf089de93b615ae9cd8a6c4fa
Parents: 8312b6f
Author: Maria Garcia Herrero 
Authored: Tue May 9 17:43:33 2017 -0700
Committer: chamik...@google.com 
Committed: Tue May 9 23:03:29 2017 -0700

--
 sdks/python/apache_beam/pipeline.py | 10 ++
 1 file changed, 6 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/12c9ba1a/sdks/python/apache_beam/pipeline.py
--
diff --git a/sdks/python/apache_beam/pipeline.py 
b/sdks/python/apache_beam/pipeline.py
index ed896c4..83c7287 100644
--- a/sdks/python/apache_beam/pipeline.py
+++ b/sdks/python/apache_beam/pipeline.py
@@ -361,9 +361,10 @@ class Pipeline(object):
 
 
 class PipelineVisitor(object):
-  """Visitor pattern class used to traverse a DAG of transforms.
+  """For internal use only; no backwards-compatibility guarantees.
 
-  This is an internal class used for bookkeeping by a Pipeline.
+  Visitor pattern class used to traverse a DAG of transforms
+  (used internally by Pipeline for bookeeping purposes).
   """
 
   def visit_value(self, value, producer_node):
@@ -390,9 +391,10 @@ class PipelineVisitor(object):
 
 
 class AppliedPTransform(object):
-  """A transform node representing an instance of applying a PTransform.
+  """For internal use only; no backwards-compatibility guarantees.
 
-  This is an internal class used for bookkeeping by a Pipeline.
+  A transform node representing an instance of applying a PTransform
+  (used internally by Pipeline for bookeeping purposes).
   """
 
   def __init__(self, parent, transform, full_label, inputs):

[2/2] beam git commit: This closes #3027

2017-05-09 Thread chamikara

This closes #3027


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/b1a2dbeb
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/b1a2dbeb
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/b1a2dbeb

Branch: refs/heads/master
Commit: b1a2dbeb112694fe9f75df1956c6a6dfdb125d80
Parents: 8312b6f 12c9ba1
Author: chamik...@google.com 
Authored: Tue May 9 23:03:44 2017 -0700
Committer: chamik...@google.com 
Committed: Tue May 9 23:03:44 2017 -0700

--
 sdks/python/apache_beam/pipeline.py | 10 ++
 1 file changed, 6 insertions(+), 4 deletions(-)
--

[jira] [Commented] (BEAM-1438) The default behavior for the Write transform doesn't work well with the Dataflow streaming runner

2017-05-09 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-1438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16004106#comment-16004106
 ] 

ASF GitHub Bot commented on BEAM-1438:
--

GitHub user reuvenlax reopened a pull request:

https://github.com/apache/beam/pull/1952

BEAM-1438 Auto shard streaming sinks

If a Write requests runner-determined sharding, per-bundle sharding is the 
default but performs poorly in Dataflow's streaming runner. Instead, the runner 
statically picks a sharding based on the number of workers.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/reuvenlax/incubator-beam 
streaming_auto_shard_write

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1952.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1952


commit f4dfbb206382d3ea73881727aa8b0f74eaf98ef4
Author: Kenneth Knowles 
Date:   2017-05-03T02:31:22Z

Annotate internal methods of PCollection

commit c1b26a1b53c334ab171fad60501ba67593fde5d2
Author: Kenneth Knowles 
Date:   2017-05-03T02:48:38Z

Annotate internal pieces of sdks.transforms

commit 49cf433c5c08f3cc91512aa9544a36a5d3e84333
Author: Kenneth Knowles 
Date:   2017-05-03T02:59:32Z

Tighten access control and internal annotations for triggers

commit 9b8a4e5c4b876d4459c64a9bffee613aeae72fb2
Author: Kenneth Knowles 
Date:   2017-05-03T03:05:34Z

The transforms.reflect package is not for users

commit fe51cc0d1a8aa14adbee81b220f9ca8a442f26fe
Author: Kenneth Knowles 
Date:   2017-05-03T03:05:45Z

Annotate internal-only bits of Java sdk.runners

commit 58298d866fe9d1f4fcaf2ccda3078809f4d55b27
Author: Kenneth Knowles 
Date:   2017-05-03T17:10:07Z

Tighten access in sdk.options

commit 362d0be79222ad67f1639d54434c1505ef76752b
Author: Kenneth Knowles 
Date:   2017-05-03T17:13:15Z

Annotate internal methods on Pipeline

commit f43b61af4d5a3ee77a610d8b11ef80d421c34501
Author: Kenneth Knowles 
Date:   2017-05-04T13:10:45Z

This closes #2852: Tighten up access and use internal annotations a bit in 
the Java SDK

  Annotate internal methods on Pipeline
  Tighten access in sdk.options
  Annotate internal-only bits of Java sdk.runners
  The transforms.reflect package is not for users
  Tighten access control and internal annotations for triggers
  Annotate internal pieces of sdks.transforms
  Annotate internal methods of PCollection

commit 1f1c897264ea7ab050c8644344f6e2648af9ae4a
Author: Luke Cwik 
Date:   2017-05-04T00:17:11Z

[BEAM-2165] Update Apex to support serializing/deserializing custom user 
types configured via Jackson modules

commit 02b72d6644c07b72a4c977a6cb16d59ec5a0ed8c
Author: Luke Cwik 
Date:   2017-05-04T14:16:29Z

[BEAM-2165] Update Apex to support serializing/deserializing custom user 
types configured via Jackson modules

This closes #2880

commit e5729b58330a05e7be510710d0027c004704946b
Author: Luke Cwik 
Date:   2017-05-04T00:19:00Z

[BEAM-2165] Update Dataflow to support serializing/deserializing custom 
user types configured via Jackson modules

This also updates the runner harness and existing tests to use a properly 
constructed ObjectMapper for PipelineOptions.

commit 749b33f0b74a9bcd3daf03ea7f9b4579baec2651
Author: Luke Cwik 
Date:   2017-05-04T14:27:17Z

[BEAM-2165] Update Dataflow to support serializing/deserializing custom 
user types configured via Jackson modules

This closes #2881

commit f53e5d43d58c79ab9f3d04e112e6f05ad9dfe42f
Author: Luke Cwik 
Date:   2017-05-04T00:12:20Z

[BEAM-2165] Update Flink to support serializing/deserializing custom user 
types configured via Jackson modules

commit 3c5891b31d8dbeafad0a6ffbea33afb92c01c374
Author: Luke Cwik 
Date:   2017-05-04T14:29:28Z

[BEAM-2165] Update Flink to support serializing/deserializing custom user 
types configured via Jackson modules

This closes #2879

commit cc654f02e8670ea789aee67508c569e7547ef11f
Author: Luke Cwik 
Date:   2017-05-03T20:48:07Z

[BEAM-1871] Migrate ReleaseInfo away from Google API client GenericJson

commit 98e92a0b8a4655a05fce4ae699f5bb93fe74f1de
Author: Luke Cwik 
Date:   2017-05-04T14:41:15Z

[BEAM-1871] Migrate ReleaseInfo away from Google API client GenericJson

This closes #2868

commit 8a2dcdb6f9d4839c864a2c46c4b5254d0c7d4760
Author: Dan Halperin 
Date:   2017-05-03T18:52:02Z

DataflowRunner: integration test GCP-IO

Triggered under `-DskipITs=false -Pdataflow-runner`

commit e1d4aa96338959a556c8b815ccb6b1aae118ad15
Author: Dan Halperin 
Date:   2017-05-04T14:59:38Z

This closes #2870

commit 1671708340fb9fc57cdc91c3bbacdff3ae6af4af
Author: yangping.wu 
Date:   2017-05-04T06:04:08Z

[BEAM-1491]Identify HADOOP_

[GitHub] beam pull request #1952: BEAM-1438 Auto shard streaming sinks

2017-05-09 Thread reuvenlax

GitHub user reuvenlax reopened a pull request:

https://github.com/apache/beam/pull/1952

BEAM-1438 Auto shard streaming sinks

If a Write requests runner-determined sharding, per-bundle sharding is the 
default but performs poorly in Dataflow's streaming runner. Instead, the runner 
statically picks a sharding based on the number of workers.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/reuvenlax/incubator-beam 
streaming_auto_shard_write

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1952.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1952


commit f4dfbb206382d3ea73881727aa8b0f74eaf98ef4
Author: Kenneth Knowles 
Date:   2017-05-03T02:31:22Z

Annotate internal methods of PCollection

commit c1b26a1b53c334ab171fad60501ba67593fde5d2
Author: Kenneth Knowles 
Date:   2017-05-03T02:48:38Z

Annotate internal pieces of sdks.transforms

commit 49cf433c5c08f3cc91512aa9544a36a5d3e84333
Author: Kenneth Knowles 
Date:   2017-05-03T02:59:32Z

Tighten access control and internal annotations for triggers

commit 9b8a4e5c4b876d4459c64a9bffee613aeae72fb2
Author: Kenneth Knowles 
Date:   2017-05-03T03:05:34Z

The transforms.reflect package is not for users

commit fe51cc0d1a8aa14adbee81b220f9ca8a442f26fe
Author: Kenneth Knowles 
Date:   2017-05-03T03:05:45Z

Annotate internal-only bits of Java sdk.runners

commit 58298d866fe9d1f4fcaf2ccda3078809f4d55b27
Author: Kenneth Knowles 
Date:   2017-05-03T17:10:07Z

Tighten access in sdk.options

commit 362d0be79222ad67f1639d54434c1505ef76752b
Author: Kenneth Knowles 
Date:   2017-05-03T17:13:15Z

Annotate internal methods on Pipeline

commit f43b61af4d5a3ee77a610d8b11ef80d421c34501
Author: Kenneth Knowles 
Date:   2017-05-04T13:10:45Z

This closes #2852: Tighten up access and use internal annotations a bit in 
the Java SDK

  Annotate internal methods on Pipeline
  Tighten access in sdk.options
  Annotate internal-only bits of Java sdk.runners
  The transforms.reflect package is not for users
  Tighten access control and internal annotations for triggers
  Annotate internal pieces of sdks.transforms
  Annotate internal methods of PCollection

commit 1f1c897264ea7ab050c8644344f6e2648af9ae4a
Author: Luke Cwik 
Date:   2017-05-04T00:17:11Z

[BEAM-2165] Update Apex to support serializing/deserializing custom user 
types configured via Jackson modules

commit 02b72d6644c07b72a4c977a6cb16d59ec5a0ed8c
Author: Luke Cwik 
Date:   2017-05-04T14:16:29Z

[BEAM-2165] Update Apex to support serializing/deserializing custom user 
types configured via Jackson modules

This closes #2880

commit e5729b58330a05e7be510710d0027c004704946b
Author: Luke Cwik 
Date:   2017-05-04T00:19:00Z

[BEAM-2165] Update Dataflow to support serializing/deserializing custom 
user types configured via Jackson modules

This also updates the runner harness and existing tests to use a properly 
constructed ObjectMapper for PipelineOptions.

commit 749b33f0b74a9bcd3daf03ea7f9b4579baec2651
Author: Luke Cwik 
Date:   2017-05-04T14:27:17Z

[BEAM-2165] Update Dataflow to support serializing/deserializing custom 
user types configured via Jackson modules

This closes #2881

commit f53e5d43d58c79ab9f3d04e112e6f05ad9dfe42f
Author: Luke Cwik 
Date:   2017-05-04T00:12:20Z

[BEAM-2165] Update Flink to support serializing/deserializing custom user 
types configured via Jackson modules

commit 3c5891b31d8dbeafad0a6ffbea33afb92c01c374
Author: Luke Cwik 
Date:   2017-05-04T14:29:28Z

[BEAM-2165] Update Flink to support serializing/deserializing custom user 
types configured via Jackson modules

This closes #2879

commit cc654f02e8670ea789aee67508c569e7547ef11f
Author: Luke Cwik 
Date:   2017-05-03T20:48:07Z

[BEAM-1871] Migrate ReleaseInfo away from Google API client GenericJson

commit 98e92a0b8a4655a05fce4ae699f5bb93fe74f1de
Author: Luke Cwik 
Date:   2017-05-04T14:41:15Z

[BEAM-1871] Migrate ReleaseInfo away from Google API client GenericJson

This closes #2868

commit 8a2dcdb6f9d4839c864a2c46c4b5254d0c7d4760
Author: Dan Halperin 
Date:   2017-05-03T18:52:02Z

DataflowRunner: integration test GCP-IO

Triggered under `-DskipITs=false -Pdataflow-runner`

commit e1d4aa96338959a556c8b815ccb6b1aae118ad15
Author: Dan Halperin 
Date:   2017-05-04T14:59:38Z

This closes #2870

commit 1671708340fb9fc57cdc91c3bbacdff3ae6af4af
Author: yangping.wu 
Date:   2017-05-04T06:04:08Z

[BEAM-1491]Identify HADOOP_CONF_DIR(or YARN_CONF_DIR) environment variables

commit 588f57a1e6771883df84d06087a93fa4fc4baa54
Author: Luke Cwik 
Date:   2017-05-04T15:48:23Z

[BEAM-1491]Identify HADOOP_CONF_DIR(or YARN_CONF_DIR) environment variables

This closes #2890

[jira] [Commented] (BEAM-2079) Support TextIO as SQL source/sink

2017-05-09 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-2079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16004091#comment-16004091
 ] 

ASF GitHub Bot commented on BEAM-2079:
--

Github user xumingming closed the pull request at:

https://github.com/apache/beam/pull/2942


> Support TextIO as SQL source/sink
> -
>
> Key: BEAM-2079
> URL: https://issues.apache.org/jira/browse/BEAM-2079
> Project: Beam
>  Issue Type: Task
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: James Xu
>
> Use TextIO as source/sink to access text-format data in Beam SQL.
> It's an implementation of 
> {{https://github.com/apache/beam/blob/DSL_SQL/dsls/sql/src/main/java/org/apache/beam/dsls/sql/schema/BaseBeamTable.java}},
>  and can refer to 
> {{https://github.com/apache/beam/blob/DSL_SQL/dsls/sql/src/main/java/org/apache/beam/dsls/sql/schema/kafka/BeamKafkaTable.java}}
>  as an example.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2942: [BEAM-2079] Support TextIO as SQL source/sink

2017-05-09 Thread xumingming

Github user xumingming closed the pull request at:

https://github.com/apache/beam/pull/2942


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: [BEAM-2079] Support TextIO as SQL source/sink

2017-05-09 Thread jbonofre

Repository: beam
Updated Branches:
  refs/heads/DSL_SQL 3e678a75a -> 7e918a76e


[BEAM-2079] Support TextIO as SQL source/sink


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/59abc764
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/59abc764
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/59abc764

Branch: refs/heads/DSL_SQL
Commit: 59abc7640866f58427817c026af15df30947f3a6
Parents: 3e678a7
Author: James Xu 
Authored: Thu May 4 11:44:14 2017 +0800
Committer: Jean-Baptiste OnofrÃ© 
Committed: Wed May 10 07:24:01 2017 +0200

--
 dsls/sql/pom.xml|   5 +
 .../dsls/sql/schema/text/BeamTextCSVTable.java  |  66 +++
 .../schema/text/BeamTextCSVTableIOReader.java   | 123 +
 .../schema/text/BeamTextCSVTableIOWriter.java   |  76 
 .../dsls/sql/schema/text/BeamTextTable.java |  45 +
 .../beam/dsls/sql/schema/text/package-info.java |  22 +++
 .../sql/schema/text/BeamTextCSVTableTest.java   | 179 +++
 7 files changed, 516 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/59abc764/dsls/sql/pom.xml
--
diff --git a/dsls/sql/pom.xml b/dsls/sql/pom.xml
index 6139ada..15692e9 100644
--- a/dsls/sql/pom.xml
+++ b/dsls/sql/pom.xml
@@ -204,5 +204,10 @@
   hamcrest-all
   test
 
+
+  org.apache.commons
+  commons-csv
+  1.4
+
   
 

http://git-wip-us.apache.org/repos/asf/beam/blob/59abc764/dsls/sql/src/main/java/org/apache/beam/dsls/sql/schema/text/BeamTextCSVTable.java
--
diff --git 
a/dsls/sql/src/main/java/org/apache/beam/dsls/sql/schema/text/BeamTextCSVTable.java
 
b/dsls/sql/src/main/java/org/apache/beam/dsls/sql/schema/text/BeamTextCSVTable.java
new file mode 100644
index 000..b9e6b81
--- /dev/null
+++ 
b/dsls/sql/src/main/java/org/apache/beam/dsls/sql/schema/text/BeamTextCSVTable.java
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.dsls.sql.schema.text;
+
+import org.apache.beam.dsls.sql.schema.BeamSQLRow;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PDone;
+import org.apache.calcite.rel.type.RelProtoDataType;
+import org.apache.commons.csv.CSVFormat;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * {@code BeamTextCSVTable} is a {@code BeamTextTable} which formatted in CSV.
+ *
+ * 
+ * {@link CSVFormat} itself has many dialects, check its javadoc for more info.
+ * 
+ */
+public class BeamTextCSVTable extends BeamTextTable {
+  private static final Logger LOG = LoggerFactory
+  .getLogger(BeamTextCSVTable.class);
+
+  private CSVFormat csvFormat;
+
+  /**
+   * CSV table with {@link CSVFormat#DEFAULT DEFAULT} format.
+   */
+  public BeamTextCSVTable(RelProtoDataType protoDataType, String filePattern)  
{
+this(protoDataType, filePattern, CSVFormat.DEFAULT);
+  }
+
+  public BeamTextCSVTable(RelProtoDataType protoDataType, String filePattern,
+  CSVFormat csvFormat) {
+super(protoDataType, filePattern);
+this.csvFormat = csvFormat;
+  }
+
+  @Override
+  public PTransform> buildIOReader() {
+return new BeamTextCSVTableIOReader(beamSqlRecordType, filePattern, 
csvFormat);
+  }
+
+  @Override
+  public PTransform, PDone> buildIOWriter() {
+return new BeamTextCSVTableIOWriter(beamSqlRecordType, filePattern, 
csvFormat);
+  }
+}

http://git-wip-us.apache.org/repos/asf/beam/blob/59abc764/dsls/sql/src/main/java/org/apache/beam/dsls/sql/schema/text/BeamTextCSVTableIOReader.java
--
diff --git 
a/dsls/sql/src/main/java/org/apache/beam/dsls/sql/schema/text/BeamTextCSVTableIOReader.java
 
b/dsls/sql/src/main/java/org/apache/beam/dsls/sql/schema/text/BeamTextCSVTableIOReader.java
new file mode 100644
index 0

[2/2] beam git commit: [BEAM-2079] This closes #2942

2017-05-09 Thread jbonofre

[BEAM-2079] This closes #2942


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/7e918a76
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/7e918a76
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/7e918a76

Branch: refs/heads/DSL_SQL
Commit: 7e918a76e409e5c849f2878702746935f4d7bf90
Parents: 3e678a7 59abc76
Author: Jean-Baptiste OnofrÃ© 
Authored: Wed May 10 07:24:40 2017 +0200
Committer: Jean-Baptiste OnofrÃ© 
Committed: Wed May 10 07:24:40 2017 +0200

--
 dsls/sql/pom.xml|   5 +
 .../dsls/sql/schema/text/BeamTextCSVTable.java  |  66 +++
 .../schema/text/BeamTextCSVTableIOReader.java   | 123 +
 .../schema/text/BeamTextCSVTableIOWriter.java   |  76 
 .../dsls/sql/schema/text/BeamTextTable.java |  45 +
 .../beam/dsls/sql/schema/text/package-info.java |  22 +++
 .../sql/schema/text/BeamTextCSVTableTest.java   | 179 +++
 7 files changed, 516 insertions(+)
--

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #3746

2017-05-09 Thread Apache Jenkins Server

See

Build failed in Jenkins: beam_PostCommit_Python_Verify #2177

2017-05-09 Thread Apache Jenkins Server

See 


Changes:

[tgroh] Mark PValue and PValueBase Internal

--
[...truncated 1.76 MB...]
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s14", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": "_equal"
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "assert_that/Match.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s13"
}, 
"serialized_fn": "", 
"user_name": "assert_that/Match"
  }
}
  ], 
  "type": "JOB_TYPE_BATCH"
}
root: INFO: Create job: 
root: INFO: Created job with id: [2017-05-09_21_43_51-2632386391657042642]
root: INFO: To access the Dataflow monitoring console, please navigate to 
https://console.developers.google.com/project/apache-beam-testing/dataflow/job/2017-05-09_21_43_51-2632386391657042642
root: INFO: Job 2017-05-09_21_43_51-2632386391657042642 is in state 
JOB_STATE_RUNNING
root: INFO: 2017-05-10T04:43:51.632Z: JOB_MESSAGE_WARNING: (24881d79ea4a53be): 
Setting the number of workers (1) disables autoscaling for this job. If you are 
trying to cap autoscaling, consider only setting max_num_workers. If you want 
to disable autoscaling altogether, the documented way is to explicitly use 
autoscalingAlgorithm=NONE.
root: INFO: 2017-05-10T04:43:52.695Z: JOB_MESSAGE_DETAILED: (cf18b6e5c174977b): 
Checking required Cloud APIs are enabled.
root: INFO: 2017-05-10T04:43:54.885Z: JOB_MESSAGE_DEBUG: (cf18b6e5c174915c): 
Combiner lifting skipped for step assert_that/Group/GroupByKey: GroupByKey not 
followed by a combiner.
root: INFO: 2017-05-10T04:43:54.888Z: JOB_MESSAGE_DETAILED: (cf18b6e5c1749b66): 
Expanding GroupByKey operations into optimizable parts.
root: INFO: 2017-05-10T04:43:54.890Z: JOB_MESSAGE_DETAILED: (cf18b6e5c1749570): 
Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
root: INFO: 2017-05-10T04:43:54.895Z: JOB_MESSAGE_DEBUG: (cf18b6e5c1749984): 
Annotating graph with Autotuner information.
root: INFO: 2017-05-10T04:43:54.908Z: JOB_MESSAGE_DETAILED: (cf18b6e5c1749d98): 
Fusing adjacent ParDo, Read, Write, and Flatten operations
root: INFO: 2017-05-10T04:43:54.917Z: JOB_MESSAGE_DETAILED: (cf18b6e5c1749bb6): 
Unzipping flatten s10 for input s8.out
root: INFO: 2017-05-10T04:43:54.920Z: JOB_MESSAGE_DETAILED: (cf18b6e5c17495c0): 
Fusing unzipped copy of assert_that/Group/GroupByKey/Reify, through flatten , 
into producer assert_that/Group/pair_with_0
root: INFO: 2017-05-10T04:43:54.923Z: JOB_MESSAGE_DETAILED: (cf18b6e5c1749fca): 
Fusing consumer assert_that/Group/Map(_merge_tagged_vals_under_key) into 
assert_that/Group/GroupByKey/GroupByWindow
root: INFO: 2017-05-10T04:43:54.925Z: JOB_MESSAGE_DETAILED: (cf18b6e5c17499d4): 
Fusing consumer assert_that/Match into assert_that/Unkey
root: INFO: 2017-05-10T04:43:54.930Z: JOB_MESSAGE_DETAILED: (cf18b6e5c17493de): 
Fusing consumer assert_that/Unkey into 
assert_that/Group/Map(_merge_tagged_vals_under_key)
root: INFO: 2017-05-10T04:43:54.937Z: JOB_MESSAGE_DETAILED: (cf18b6e5c17497f2): 
Fusing consumer assert_t

[jira] [Commented] (BEAM-659) WindowFn#isCompatible should provide a meaningful reason

2017-05-09 Thread Kenneth Knowles (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16004040#comment-16004040
 ] 

Kenneth Knowles commented on BEAM-659:
--

Since a risk-free PR for this is available, adding to 2.0.0. It can be punted 
if we cross the finish line early. (how about those metaphors...)

> WindowFn#isCompatible should provide a meaningful reason
> 
>
> Key: BEAM-659
> URL: https://issues.apache.org/jira/browse/BEAM-659
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Huafeng Wang
>Priority: Minor
>  Labels: easy, easyfix, starter
> Fix For: 2.0.0
>
>
> {{WindowFn#isCompatible(...)}} returns a true/false answer without any 
> explanation of why; it suffers from a form of [boolean 
> blindness|https://existentialtype.wordpress.com/2011/03/15/boolean-blindness/].
> We alleviated this for {{Coder#isDeterministic()}} by switching to 
> {{Coder#verifyDeterministic()}} which can raise an exception describing why 
> it is not deterministic (it could just as easily have been a return value).
> We should do the same for {{WindowFn#isCompatible(...)}}.
> See here: 
> http://stackoverflow.com/questions/39617897/inputs-to-flatten-had-incompatible-window-windowfns-when-cogroupbykey-with-calen
> (a decent {{toString}} would go a long way, for immediate and trivial 
> improvement)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-659) WindowFn#isCompatible should provide a meaningful reason

2017-05-09 Thread Kenneth Knowles (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-659:
-
Fix Version/s: 2.0.0

> WindowFn#isCompatible should provide a meaningful reason
> 
>
> Key: BEAM-659
> URL: https://issues.apache.org/jira/browse/BEAM-659
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Huafeng Wang
>Priority: Minor
>  Labels: easy, easyfix, starter
> Fix For: 2.0.0
>
>
> {{WindowFn#isCompatible(...)}} returns a true/false answer without any 
> explanation of why; it suffers from a form of [boolean 
> blindness|https://existentialtype.wordpress.com/2011/03/15/boolean-blindness/].
> We alleviated this for {{Coder#isDeterministic()}} by switching to 
> {{Coder#verifyDeterministic()}} which can raise an exception describing why 
> it is not deterministic (it could just as easily have been a return value).
> We should do the same for {{WindowFn#isCompatible(...)}}.
> See here: 
> http://stackoverflow.com/questions/39617897/inputs-to-flatten-had-incompatible-window-windowfns-when-cogroupbykey-with-calen
> (a decent {{toString}} would go a long way, for immediate and trivial 
> improvement)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2222) Clean up readme files

2017-05-09 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16004034#comment-16004034
 ] 

ASF GitHub Bot commented on BEAM-:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam-site/pull/237


> Clean up readme files
> -
>
> Key: BEAM-
> URL: https://issues.apache.org/jira/browse/BEAM-
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Ahmet Altay
>Assignee: Ahmet Altay
> Fix For: 2.0.0
>
>
> Move content from readme.md's to website.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[2/3] beam-site git commit: Regenerate website

2017-05-09 Thread altay

Regenerate website


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/aa01f116
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/aa01f116
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/aa01f116

Branch: refs/heads/asf-site
Commit: aa01f1161bbbfc0060ab76ebad6bfcee7826e963
Parents: 8ed3f24
Author: Ahmet Altay 
Authored: Tue May 9 21:24:35 2017 -0700
Committer: Ahmet Altay 
Committed: Tue May 9 21:24:35 2017 -0700

--
 content/documentation/runners/flink/index.html  |   2 +-
 .../sdks/java-extensions/index.html | 237 +++
 content/documentation/sdks/java/index.html  |   9 +
 3 files changed, 247 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam-site/blob/aa01f116/content/documentation/runners/flink/index.html
--
diff --git a/content/documentation/runners/flink/index.html 
b/content/documentation/runners/flink/index.html
index 64ae583..deb04c8 100644
--- a/content/documentation/runners/flink/index.html
+++ b/content/documentation/runners/flink/index.html
@@ -287,7 +287,7 @@
 
 
 
-See the reference documentation for the  FlinkPipelineOptionshttps://github.com/apache/beam/blob/master/sdks/python/apache_beam/utils/pipeline_options.py";>PipelineOptions
 interface (and its subinterfaces) for the complete list of pipeline 
configuration options.
+See the reference documentation for the  FlinkPipelineOptionshttps://github.com/apache/beam/blob/master/sdks/python/apache_beam/options/pipeline_options.py";>PipelineOptions
 interface (and its subinterfaces) for the complete list of pipeline 
configuration options.
 
 Additional information and 
caveats
 

http://git-wip-us.apache.org/repos/asf/beam-site/blob/aa01f116/content/documentation/sdks/java-extensions/index.html
--
diff --git a/content/documentation/sdks/java-extensions/index.html 
b/content/documentation/sdks/java-extensions/index.html
new file mode 100644
index 000..ee93238
--- /dev/null
+++ b/content/documentation/sdks/java-extensions/index.html
@@ -0,0 +1,237 @@
+
+
+
+  
+  
+  
+  
+
+  Beam Java SDK Extensions
+  
+
+  
+  
+  https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js";>
+  
+  
+  https://beam.apache.org/documentation/sdks/java-extensions/"; 
data-proofer-ignore>
+  https://beam.apache.org/feed.xml";>
+  
+
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new 
Date();a=s.createElement(o),
+
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+
+ga('create', 'UA-73650088-1', 'auto');
+ga('send', 'pageview');
+
+  
+  
+
+
+
+  
+
+
+  
+
+  
+
+  
+  
+Toggle navigation
+
+
+
+  
+
+
+  
+
+ Get Started 
+ 
+ Beam 
Overview
+Quickstart - Java
+Quickstart - Python
+ 
+ Example Walkthroughs
+ WordCount
+ Mobile Gaming
+  
+  Resources
+  Downloads
+  Support
+ 
+   
+
+ Documentation 
+ 
+ Using the 
Documentation
+ 
+ Beam Concepts
+ Programming Guide
+ Additional 
Resources
+ 
+  Pipeline Fundamentals
+  Design Your 
Pipeline
+  Create Your 
Pipeline
+  Test 
Your Pipeline
+  Pipeline I/O
+  
+ SDKs
+ Java 
SDK
+ Java SDK API Reference 
+
+Python SDK
+Python 
SDK API Reference 
+
+ 
+ Runners
+ Capability Matrix
+ Direct 
Runner
+ Apache 
Apex Runner
+ Apache 
Flink Runner
+ Apache 
Spark Runner
+ Cloud 
Dataflow Runner
+ 
+   
+
+ Contribute 
+ 
+ Get Started 
Contributing
+
+Guides
+ Contribution Guide
+Testing Guide
+Release Guide
+PTransform Style 
Guide
+
+Technical Refer

[GitHub] beam-site pull request #237: [BEAM-2222] Move extension README files to webs...

2017-05-09 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/beam-site/pull/237


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[3/3] beam-site git commit: This closes #237

2017-05-09 Thread altay

This closes #237


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/acd643af
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/acd643af
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/acd643af

Branch: refs/heads/asf-site
Commit: acd643afbddeae5491d8d3af455249557e32c9c4
Parents: 47ad185 aa01f11
Author: Ahmet Altay 
Authored: Tue May 9 21:24:35 2017 -0700
Committer: Ahmet Altay 
Committed: Tue May 9 21:24:35 2017 -0700

--
 content/documentation/runners/flink/index.html  |   2 +-
 .../sdks/java-extensions/index.html | 237 +++
 content/documentation/sdks/java/index.html  |   9 +
 src/documentation/runners/flink.md  |   2 +-
 src/documentation/sdks/java-extensions.md   |  59 +
 src/documentation/sdks/java.md  |   8 +
 6 files changed, 315 insertions(+), 2 deletions(-)
--

[1/3] beam-site git commit: Move extension README files to website

2017-05-09 Thread altay

Repository: beam-site
Updated Branches:
  refs/heads/asf-site 47ad18557 -> acd643afb


Move extension README files to website


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/8ed3f247
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/8ed3f247
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/8ed3f247

Branch: refs/heads/asf-site
Commit: 8ed3f247b8600b7651c5186ba145473e97ed
Parents: 47ad185
Author: Ahmet Altay 
Authored: Tue May 9 17:47:01 2017 -0700
Committer: Ahmet Altay 
Committed: Tue May 9 21:21:25 2017 -0700

--
 src/documentation/runners/flink.md|  2 +-
 src/documentation/sdks/java-extensions.md | 59 ++
 src/documentation/sdks/java.md|  8 
 3 files changed, 68 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam-site/blob/8ed3f247/src/documentation/runners/flink.md
--
diff --git a/src/documentation/runners/flink.md 
b/src/documentation/runners/flink.md
index f844c6b..685917e 100644
--- a/src/documentation/runners/flink.md
+++ b/src/documentation/runners/flink.md
@@ -138,7 +138,7 @@ When executing your pipeline with the Flink Runner, you can 
set these pipeline o
 
 
 
-See the reference documentation for the  [FlinkPipelineOptions]({{ site.baseurl 
}}/documentation/sdks/javadoc/{{ site.release_latest 
}}/index.html?org/apache/beam/runners/flink/FlinkPipelineOptions.html)[PipelineOptions](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/utils/pipeline_options.py)
 interface (and its subinterfaces) for the complete list of pipeline 
configuration options.
+See the reference documentation for the  [FlinkPipelineOptions]({{ site.baseurl 
}}/documentation/sdks/javadoc/{{ site.release_latest 
}}/index.html?org/apache/beam/runners/flink/FlinkPipelineOptions.html)[PipelineOptions](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/options/pipeline_options.py)
 interface (and its subinterfaces) for the complete list of pipeline 
configuration options.
 
 ## Additional information and caveats
 

http://git-wip-us.apache.org/repos/asf/beam-site/blob/8ed3f247/src/documentation/sdks/java-extensions.md
--
diff --git a/src/documentation/sdks/java-extensions.md 
b/src/documentation/sdks/java-extensions.md
new file mode 100644
index 000..a4694af
--- /dev/null
+++ b/src/documentation/sdks/java-extensions.md
@@ -0,0 +1,59 @@
+---
+layout: default
+title: "Beam Java SDK Extensions"
+permalink: /documentation/sdks/java-extensions/
+---
+# Apache Beam Java SDK Extensions
+
+## Join-library
+
+Join-library provides inner join, outer left join, and outer right join 
functions. The aim
+is to simplify the most common cases of join to a simple function call.
+
+The functions are generic and support joins of any Beam-supported types.
+Input to the join functions are `PCollections` of `Key` / `Value`s. Both
+the left and right `PCollection`s need the same type for the key. All the join
+functions return a `Key` / `Value` where `Key` is the join key and value is
+a `Key` / `Value` where the key is the left value and right is the value.
+
+For outer joins, the user must provide a value that represents `null` because 
`null`
+cannot be serialized.
+
+Example usage:
+
+```
+PCollection> leftPcollection = ...
+PCollection> rightPcollection = ...
+
+PCollection>> joinedPcollection =
+  Join.innerJoin(leftPcollection, rightPcollection);
+```
+
+
+## Sorter
+
+This module provides the `SortValues` transform, which takes a 
`PCollection>>>` and produces a `PCollection>>>` where, for each primary key `K` the paired 
`Iterable>` has been sorted by the byte encoding of secondary key 
(`K2`). It is an efficient and scalable sorter for iterables, even if they are 
large (do not fit in memory).
+
+### Caveats
+
+- This transform performs value-only sorting; the iterable accompanying each 
key is sorted, but *there is no relationship between different keys*, as Beam 
does not support any defined relationship between different elements in a 
`PCollection`.
+* Each `Iterable>` is sorted on a single worker using local memory 
and disk. This means that `SortValues` may be a performance and/or scalability 
bottleneck when used in different pipelines. For example, users are discouraged 
from using `SortValues` on a `PCollection` of a single element to globally sort 
a large `PCollection`. A (rough) estimate of the number of bytes of disk space 
utilized if sorting spills to disk is `numRecords * 
(numSecondaryKeyBytesPerRecord + numValueBytesPerRecord + 16) * 3`.
+
+### Options
+
+* The user can customize the temporary location used if sorting requires 
spilling to disk a

[GitHub] beam pull request #3038: [BEAM-659] Cherry-pick #2985, WindowFn error messag...

2017-05-09 Thread kennknowles

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/3038

[BEAM-659] Cherry-pick #2985, WindowFn error message improvement, to 
release-2.0.0

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam windowfn-cherrypick

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3038.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3038


commit 3ab5dfcf585a2b212fe024ec56334e2bf88707a9
Author: huafengw 
Date:   2017-05-09T03:21:44Z

[BEAM-659] WindowFn#isCompatible should provide a meaningful reason




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-659) WindowFn#isCompatible should provide a meaningful reason

2017-05-09 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16004033#comment-16004033
 ] 

ASF GitHub Bot commented on BEAM-659:
-

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/3038

[BEAM-659] Cherry-pick #2985, WindowFn error message improvement, to 
release-2.0.0

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam windowfn-cherrypick

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3038.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3038


commit 3ab5dfcf585a2b212fe024ec56334e2bf88707a9
Author: huafengw 
Date:   2017-05-09T03:21:44Z

[BEAM-659] WindowFn#isCompatible should provide a meaningful reason




> WindowFn#isCompatible should provide a meaningful reason
> 
>
> Key: BEAM-659
> URL: https://issues.apache.org/jira/browse/BEAM-659
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Huafeng Wang
>Priority: Minor
>  Labels: easy, easyfix, starter
>
> {{WindowFn#isCompatible(...)}} returns a true/false answer without any 
> explanation of why; it suffers from a form of [boolean 
> blindness|https://existentialtype.wordpress.com/2011/03/15/boolean-blindness/].
> We alleviated this for {{Coder#isDeterministic()}} by switching to 
> {{Coder#verifyDeterministic()}} which can raise an exception describing why 
> it is not deterministic (it could just as easily have been a return value).
> We should do the same for {{WindowFn#isCompatible(...)}}.
> See here: 
> http://stackoverflow.com/questions/39617897/inputs-to-flatten-had-incompatible-window-windowfns-when-cogroupbykey-with-calen
> (a decent {{toString}} would go a long way, for immediate and trivial 
> improvement)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Flink #2757

2017-05-09 Thread Apache Jenkins Server

See

[GitHub] beam pull request #3037: Cherry-pick #3033 into release-2.0.0

2017-05-09 Thread tgroh

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/3037

Cherry-pick #3033 into release-2.0.0

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
Mark PValue and PValueBase Internal

These should not be referred to as their abstract types by users, and
PipelineRunners won't interact with PValues that aren't PCollections or
PCollectionViews.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam pvalues_internal-rls

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3037.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3037


commit 8f80c0d232b30e70b45e68aad8d1233262dfcfc8
Author: Thomas Groh 
Date:   2017-05-10T01:32:14Z

Mark PValue and PValueBase Internal

These should not be referred to as their abstract types by users, and
PipelineRunners won't interact with PValues that aren't PCollections or
PCollectionViews.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1340) Remove or make private public bits of the SDK that shouldn't be public

2017-05-09 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-1340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16004005#comment-16004005
 ] 

ASF GitHub Bot commented on BEAM-1340:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3033


> Remove or make private public bits of the SDK that shouldn't be public
> --
>
> Key: BEAM-1340
> URL: https://issues.apache.org/jira/browse/BEAM-1340
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core, sdk-java-extensions
>Reporter: Kenneth Knowles
>Priority: Blocker
>  Labels: backward-incompatible
> Fix For: 2.0.0
>
>
> This JIRA is for the many small changes that do not merit their own JIRA 
> towards getting the SDK's API surface right. For example, removal of 
> `DoFn.InputProvider` and `DoFn.OutputReceiver`.
> While the above is not quite backwards incompatible, succeeding at this task 
> surely will be.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3033: [BEAM-1340] Mark PValue and PValueBase Internal

2017-05-09 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3033


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: This closes #3033

2017-05-09 Thread tgroh

Repository: beam
Updated Branches:
  refs/heads/master e2be6961d -> 8312b6f59


This closes #3033


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/8312b6f5
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/8312b6f5
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/8312b6f5

Branch: refs/heads/master
Commit: 8312b6f5962ae6a3ed6a87ba061e34bd6696b009
Parents: e2be696 3499c70
Author: Thomas Groh 
Authored: Tue May 9 21:01:52 2017 -0700
Committer: Thomas Groh 
Committed: Tue May 9 21:01:52 2017 -0700

--
 .../main/java/org/apache/beam/sdk/values/PValue.java | 12 +++-
 .../java/org/apache/beam/sdk/values/PValueBase.java  | 15 +--
 2 files changed, 12 insertions(+), 15 deletions(-)
--

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #2756

2017-05-09 Thread Apache Jenkins Server

See

[2/2] beam git commit: Mark PValue and PValueBase Internal

2017-05-09 Thread tgroh

Mark PValue and PValueBase Internal

These should not be referred to as their abstract types by users, and
PipelineRunners won't interact with PValues that aren't PCollections or
PCollectionViews.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/3499c70a
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/3499c70a
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/3499c70a

Branch: refs/heads/master
Commit: 3499c70a2fde0124dfc48ae20281c2261ef9018a
Parents: e2be696
Author: Thomas Groh 
Authored: Tue May 9 18:32:14 2017 -0700
Committer: Thomas Groh 
Committed: Tue May 9 21:01:52 2017 -0700

--
 .../main/java/org/apache/beam/sdk/values/PValue.java | 12 +++-
 .../java/org/apache/beam/sdk/values/PValueBase.java  | 15 +--
 2 files changed, 12 insertions(+), 15 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/3499c70a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PValue.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PValue.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PValue.java
index d9f6920..1089028 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PValue.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PValue.java
@@ -18,13 +18,15 @@
 package org.apache.beam.sdk.values;
 
 import java.util.Map;
+import org.apache.beam.sdk.annotations.Internal;
 import org.apache.beam.sdk.transforms.PTransform;
 
 /**
- * The interface for values that can be input to and output from {@link 
PTransform PTransforms}.
+ * For internal use. No backwards compatibility guarantees.
  *
- * It is recommended to extend {@link PValueBase}
+ * A primitive value within Beam.
  */
+@Internal
 public interface PValue extends POutput, PInput {
 
   /**
@@ -45,13 +47,13 @@ public interface PValue extends POutput, PInput {
* After building, finalizes this {@code PValue} to make it ready for being 
used as an input to a
* {@link org.apache.beam.sdk.transforms.PTransform}.
*
-   * Automatically invoked whenever {@code apply()} is invoked on this 
{@code PValue}, after
-   * {@link PValue#finishSpecifying(PInput, PTransform)} has been called on 
each component {@link
-   * PValue}, so users do not normally call this explicitly.
+   * Automatically invoked whenever {@code apply()} is invoked on this 
{@code PValue}. Users
+   * should not normally call this explicitly.
*
* @param upstreamInput the {@link PInput} the {@link PTransform} was 
applied to to produce this
* output
* @param upstreamTransform the {@link PTransform} that produced this {@link 
PValue}
*/
+  @Internal
   void finishSpecifying(PInput upstreamInput, PTransform 
upstreamTransform);
 }

http://git-wip-us.apache.org/repos/asf/beam/blob/3499c70a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PValueBase.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PValueBase.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PValueBase.java
index 4de0589..6f638d7 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PValueBase.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PValueBase.java
@@ -22,22 +22,17 @@ import static 
com.google.common.base.Preconditions.checkState;
 import java.util.Collections;
 import java.util.Map;
 import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.annotations.Internal;
 import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.util.NameUtils;
 
 /**
- * A {@link PValueBase} is an abstract base class that provides
- * sensible default implementations for methods of {@link PValue}.
- * In particular, this includes functionality for getting/setting:
+ * For internal use. No backwards compatibility guarantees.
  *
- * 
- *The {@link Pipeline} that the {@link PValue} is part of.
- *Whether the {@link PValue} has bee finalized (as an input
- * or an output), after which its properties can no longer be changed.
- * 
- *
- * For internal use.
+ * An abstract base class that provides default implementations for some 
methods of
+ * {@link PValue}.
  */
+@Internal
 public abstract class PValueBase implements PValue {
 
   private final Pipeline pipeline;

[GitHub] beam pull request #3036: [BEAM-2241] Renames some python classes and functio...

2017-05-09 Thread chamikaramj

GitHub user chamikaramj opened a pull request:

https://github.com/apache/beam/pull/3036

[BEAM-2241] Renames some python classes and functions that were 
unnecessarily public

Adds a note to documentation of classes that are public but should be only 
used internally by the SDK (non-user facing classes).

Marks some of the modules as experimental.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chamikaramj/beam update_public_api

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3036.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3036


commit e6f90ec7b8bd59bd6809edc1aa95e2e894dd2b84
Author: chamik...@google.com 
Date:   2017-05-10T02:56:14Z

Renames some python classes and functions that were unnecessarily public.

Adds a note to documentation of classes that are public but should be only 
used internally by the SDK (non-user facing classes).

Marks some of the modules as experimental.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2241) Correctly mark top level classes and functions as private

2017-05-09 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-2241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003995#comment-16003995
 ] 

ASF GitHub Bot commented on BEAM-2241:
--

GitHub user chamikaramj opened a pull request:

https://github.com/apache/beam/pull/3036

[BEAM-2241] Renames some python classes and functions that were 
unnecessarily public

Adds a note to documentation of classes that are public but should be only 
used internally by the SDK (non-user facing classes).

Marks some of the modules as experimental.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chamikaramj/beam update_public_api

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3036.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3036


commit e6f90ec7b8bd59bd6809edc1aa95e2e894dd2b84
Author: chamik...@google.com 
Date:   2017-05-10T02:56:14Z

Renames some python classes and functions that were unnecessarily public.

Adds a note to documentation of classes that are public but should be only 
used internally by the SDK (non-user facing classes).

Marks some of the modules as experimental.




> Correctly mark top level classes and functions as private
> -
>
> Key: BEAM-2241
> URL: https://issues.apache.org/jira/browse/BEAM-2241
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Chamikara Jayalath
>Assignee: Chamikara Jayalath
> Fix For: 2.0.0
>
>
> Seems like some of the top level classes and function of Python SDK are 
> unnecessarily public (does not start with _ ). We should update Python SDK so 
> that top level classes and functions that are only used within the 
> corresponding modules are private. Additionally, any top level classes and 
> functions that are not expected to be used by end-users 
> (pipeline/source/ptransform authors) should be clearly documented.  



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2245) Remove user-facing Timer.cancel() until further notice

2017-05-09 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-2245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003990#comment-16003990
 ] 

ASF GitHub Bot commented on BEAM-2245:
--

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/3035

[BEAM-2245] Remove Timer.cancel() from user-facing API

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---

R: @aljoscha @JingsongLi 
CC: @davorbonaci ideally for FSR, but not required since it is experimental 
and seemingly no one needs it

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam no-more-cancel

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3035.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3035


commit fe204d5083bdd31a50e0c83f9c1db399e95cd1a3
Author: Kenneth Knowles 
Date:   2017-05-10T03:46:43Z

Remove Timer.cancel() from user-facing API




> Remove user-facing Timer.cancel() until further notice
> --
>
> Key: BEAM-2245
> URL: https://issues.apache.org/jira/browse/BEAM-2245
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3035: [BEAM-2245] Remove Timer.cancel() from user-facing ...

2017-05-09 Thread kennknowles

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/3035

[BEAM-2245] Remove Timer.cancel() from user-facing API

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---

R: @aljoscha @JingsongLi 
CC: @davorbonaci ideally for FSR, but not required since it is experimental 
and seemingly no one needs it

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam no-more-cancel

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3035.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3035


commit fe204d5083bdd31a50e0c83f9c1db399e95cd1a3
Author: Kenneth Knowles 
Date:   2017-05-10T03:46:43Z

Remove Timer.cancel() from user-facing API




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-2245) Remove user-facing Timer.cancel() until further notice

2017-05-09 Thread Kenneth Knowles (JIRA)

Kenneth Knowles created BEAM-2245:
-

 Summary: Remove user-facing Timer.cancel() until further notice
 Key: BEAM-2245
 URL: https://issues.apache.org/jira/browse/BEAM-2245
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-core
Reporter: Kenneth Knowles
Assignee: Kenneth Knowles
 Fix For: 2.0.0






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: beam_PostCommit_Python_Verify #2176

2017-05-09 Thread Apache Jenkins Server

See 


--
[...truncated 1.75 MB...]
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "assert_that/Unkey.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s12"
}, 
"serialized_fn": "", 
"user_name": "assert_that/Unkey"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s14", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": "_equal"
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "assert_that/Match.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s13"
}, 
"serialized_fn": "", 
"user_name": "assert_that/Match"
  }
}
  ], 
  "type": "JOB_TYPE_BATCH"
}
root: INFO: Create job: 
root: INFO: Created job with id: [2017-05-09_20_33_20-14479349599173966395]
root: INFO: To access the Dataflow monitoring console, please navigate to 
https://console.developers.google.com/project/apache-beam-testing/dataflow/job/2017-05-09_20_33_20-14479349599173966395
root: INFO: Job 2017-05-09_20_33_20-14479349599173966395 is in state 
JOB_STATE_RUNNING
root: INFO: 2017-05-10T03:33:20.159Z: JOB_MESSAGE_WARNING: (c8f0fbbedd757e2f): 
Setting the number of workers (1) disables autoscaling for this job. If you are 
trying to cap autoscaling, consider only setting max_num_workers. If you want 
to disable autoscaling altogether, the documented way is to explicitly use 
autoscalingAlgorithm=NONE.
root: INFO: 2017-05-10T03:33:21.409Z: JOB_MESSAGE_DETAILED: (d1eb603b16feebaa): 
Checking required Cloud APIs are enabled.
root: INFO: 2017-05-10T03:33:23.486Z: JOB_MESSAGE_DEBUG: (d1eb603b16feeed4): 
Combiner lifting skipped for step assert_that/Group/GroupByKey: GroupByKey not 
followed by a combiner.
root: INFO: 2017-05-10T03:33:23.488Z: JOB_MESSAGE_DETAILED: (d1eb603b16fee53a): 
Expanding GroupByKey operations into optimizable parts.
root: INFO: 2017-05-10T03:33:23.491Z: JOB_MESSAGE_DETAILED: (d1eb603b16feeba0): 
Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
root: INFO: 2017-05-10T03:33:23.496Z: JOB_MESSAGE_DEBUG: (d1eb603b16fee86c): 
Annotating graph with Autotuner information.
root: INFO: 2017-05-10T03:33:23.508Z: JOB_MESSAGE_DETAILED: (d1eb603b16fee538): 
Fusing adjacent ParDo, Read, Write, and Flatten operations
root: INFO: 2017-05-10T03:33:23.516Z: JOB_MESSAGE_DETAILED: (d1eb603b16fee86a): 
Unzipping flatten s10 for input s8.out
root: INFO: 2017-05-10T03:33:23.518Z: JOB_MESSAGE_DETAILED: (d1eb603b16feeed0): 
Fusing unzipped copy of assert_that/Group/GroupByKey/Reify, through flatten , 
into producer assert_that/Group/pair_with_0
root: INFO: 2017-05-10T03:33:23.520Z: JOB_MESSAGE_DETAILE

[jira] [Resolved] (BEAM-2236) Move test utilities out of python core

2017-05-09 Thread Ahmet Altay (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-2236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay resolved BEAM-2236.
---
Resolution: Fixed

> Move test utilities out of python core
> --
>
> Key: BEAM-2236
> URL: https://issues.apache.org/jira/browse/BEAM-2236
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Mark Liu
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[1/3] beam git commit: [BEAM-2236] Cherry pick #3017 - Move test utilities out of python core

2017-05-09 Thread altay

Repository: beam
Updated Branches:
  refs/heads/release-2.0.0 aa8c9d10f -> 0cc8b9ed7


http://git-wip-us.apache.org/repos/asf/beam/blob/2ecebd22/sdks/python/apache_beam/tests/data/standard_coders.yaml
--
diff --git a/sdks/python/apache_beam/tests/data/standard_coders.yaml 
b/sdks/python/apache_beam/tests/data/standard_coders.yaml
deleted file mode 100644
index 790cacb..000
--- a/sdks/python/apache_beam/tests/data/standard_coders.yaml
+++ /dev/null
@@ -1,196 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements.  See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License.  You may obtain a copy of the License at
-#
-#http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-# This file is broken into multiple sections delimited by ---. Each section 
specifies a set of
-# reference encodings for a single standardized coder used in a specific 
context.
-#
-# Each section contains up to 3 properties:
-#
-#   coder: a common coder spec. Currently, a URN and URNs for component coders 
as necessary.
-#   nested: a boolean meaning whether the coder was used in the nested 
context. Missing means to
-#   test both contexts, a shorthand for when the coder is invariant 
across context.
-#   examples: a map of {encoded bytes: original JSON object} encoded with the 
coder in the context.
-# The LHS (key) is a byte array encoded as a JSON-escaped string. 
The RHS (value) is
-# one of a few standard JSON types such as numbers, strings, dicts 
that map naturally
-# to the type encoded by the coder.
-#
-# These choices were made to strike a balance between portability, ease of 
use, and simple
-# legibility of this file itself.
-#
-# It is expected that future work will move the `coder` field into a format 
that it would be
-# represented by the Runner API, so that it can be understood by all SDKs and 
harnesses.
-#
-# If a coder is marked non-deterministic in the coder spec, then only the 
decoding should be validated.
-
-
-
-coder:
-  urn: "urn:beam:coders:bytes:0.1"
-nested: false
-examples:
-  "abc": abc
-  "ab\0c": "ab\0c"
-

-
-coder:
-  urn: "urn:beam:coders:bytes:0.1"
-nested: true
-examples:
-  "\u0003abc": abc
-  "\u0004ab\0c": "ab\0c"
-  "\u00c8\u0001   10|   20|   30|   40|   50|   60|
   70|   80|   90|  100|  110|  120|  130|  140|
  150|  160|  170|  180|  190|  200|":
-  "   10|   20|   30|   40|   50|   60|
   70|   80|   90|  100|  110|  120|  130|  140|
  150|  160|  170|  180|  190|  200|"
-

-
-coder:
-  urn: "urn:beam:coders:varint:0.1"
-examples:
-  "\0": 0
-  "\u0001": 1
-  "\u000A": 10
-  "\u00c8\u0001": 200
-  "\u00e8\u0007": 1000
-  "\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u0001": -1
-

-
-coder:
-  urn: "urn:beam:coders:kv:0.1"
-  components: [{urn: "urn:beam:coders:bytes:0.1"},
-   {urn: "urn:beam:coders:varint:0.1"}]
-examples:
-  "\u0003abc\0": {key: abc, value: 0}
-  "\u0004ab\0c\u000A": {key: "ab\0c", value: 10}
-

-
-coder:
-  urn: "urn:beam:coders:kv:0.1"
-  components: [{urn: "urn:beam:coders:bytes:0.1"},
-   {urn: "urn:beam:coders:bytes:0.1"}]
-nested: false
-examples:
-  "\u0003abcdef": {key: abc, value: def}
-  "\u0004ab\0cde\0f": {key: "ab\0c", value: "de\0f"}
-

-
-coder:
-  urn: "urn:beam:coders:kv:0.1"
-  components: [{urn: "urn:beam:coders:bytes:0.1"},
-   {urn: "urn:beam:coders:bytes:0.1"}]
-nested: true
-examples:
-  "\u0003abc\u0003def": {key: abc, value: def}
-  "\u0004ab\0c\u0004de\0f": {key: "ab\0c", value: "de\0f"}
-

-
-coder:
-  urn: "urn:beam:coders:interval_window:0.1"
-examples:
-  "\u0080\u\u0001\u0052\u009a\u00a4\u009b\u0068\u0080\u00dd\u00db\u0001" : 
{end: 1454293425000, span: 360}
-  
"\u0080\u\u0001\u0053\u0034\u00ec\u0074\u00e8\u0080\u0090\u00fb\u00d3\u0009"
 : {end: 1456881825000, span: 259200}
-  "\u007f\u00df\u003b\u0064\u005a\u001c\u00ad\u0076\u00ed\u0002" : {end: 
-9223372036854410, span: 365}
-  "\u0080\u0020\u00c4\u009b\u00a5\u00e3\u0053\u00f7\u" : {end: 
9223372036854775, span: 0}
-

-
-coder:
-  urn: "urn:beam:coders:stream:0.1"
-  components: [{urn: "urn:beam:coders:varint:0.1"}]
-examples:
-

[2/3] beam git commit: [BEAM-2236] Cherry pick #3017 - Move test utilities out of python core

2017-05-09 Thread altay

[BEAM-2236] Cherry pick #3017 - Move test utilities out of python core


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/2ecebd22
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/2ecebd22
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/2ecebd22

Branch: refs/heads/release-2.0.0
Commit: 2ecebd22e3071d0ba6d3647273367d16582ca852
Parents: aa8c9d1
Author: Mark Liu 
Authored: Tue May 9 16:41:46 2017 -0700
Committer: Ahmet Altay 
Committed: Tue May 9 20:08:55 2017 -0700

--
 .../apache_beam/coders/standard_coders_test.py  |   2 +-
 .../examples/complete/autocomplete_test.py  |   2 +-
 .../examples/complete/estimate_pi_test.py   |   2 +-
 .../complete/game/hourly_team_score_test.py |   2 +-
 .../examples/complete/game/user_score_test.py   |   2 +-
 .../apache_beam/examples/complete/tfidf_test.py |   2 +-
 .../complete/top_wikipedia_sessions_test.py |   2 +-
 .../cookbook/bigquery_side_input_test.py|   2 +-
 .../cookbook/bigquery_tornadoes_it_test.py  |   4 +-
 .../cookbook/bigquery_tornadoes_test.py |   2 +-
 .../examples/cookbook/coders_test.py|   2 +-
 .../examples/cookbook/combiners_test.py |   2 +-
 .../examples/cookbook/custom_ptransform_test.py |   2 +-
 .../examples/cookbook/filters_test.py   |   2 +-
 .../apache_beam/examples/snippets/snippets.py   |   2 +-
 .../examples/snippets/snippets_test.py  |   2 +-
 .../apache_beam/examples/wordcount_it_test.py   |   6 +-
 sdks/python/apache_beam/io/avroio_test.py   |   2 +-
 .../python/apache_beam/io/concat_source_test.py |   2 +-
 .../apache_beam/io/filebasedsource_test.py  |   6 +-
 sdks/python/apache_beam/io/fileio_test.py   |   2 +-
 .../io/gcp/datastore/v1/helper_test.py  |   2 +-
 .../io/gcp/tests/bigquery_matcher.py|   2 +-
 .../io/gcp/tests/bigquery_matcher_test.py   |   2 +-
 sdks/python/apache_beam/io/sources_test.py  |   2 +-
 sdks/python/apache_beam/io/textio_test.py   |   2 +-
 sdks/python/apache_beam/io/tfrecordio_test.py   |   2 +-
 sdks/python/apache_beam/pipeline_test.py|   2 +-
 sdks/python/apache_beam/pvalue_test.py  |   2 +-
 .../runners/dataflow/dataflow_runner_test.py|   4 +-
 sdks/python/apache_beam/test_pipeline.py| 163 ---
 sdks/python/apache_beam/test_pipeline_test.py   | 112 ---
 sdks/python/apache_beam/testing/__init__.py |  16 ++
 .../apache_beam/testing/data/privatekey.p12 | Bin 0 -> 2452 bytes
 .../testing/data/standard_coders.yaml   | 196 +++
 .../apache_beam/testing/pipeline_verifiers.py   | 146 ++
 .../testing/pipeline_verifiers_test.py  | 148 ++
 .../python/apache_beam/testing/test_pipeline.py | 163 +++
 .../apache_beam/testing/test_pipeline_test.py   | 112 +++
 sdks/python/apache_beam/testing/test_stream.py  | 163 +++
 .../apache_beam/testing/test_stream_test.py |  83 
 sdks/python/apache_beam/testing/test_utils.py   |  69 +++
 sdks/python/apache_beam/tests/__init__.py   |  16 --
 .../apache_beam/tests/data/privatekey.p12   | Bin 2452 -> 0 bytes
 .../apache_beam/tests/data/standard_coders.yaml | 196 ---
 .../apache_beam/tests/pipeline_verifiers.py | 146 --
 .../tests/pipeline_verifiers_test.py| 148 --
 sdks/python/apache_beam/tests/test_utils.py |  69 ---
 .../apache_beam/transforms/combiners_test.py|   2 +-
 .../apache_beam/transforms/create_test.py   |   2 +-
 .../apache_beam/transforms/ptransform_test.py   |   2 +-
 .../apache_beam/transforms/sideinputs_test.py   |   2 +-
 .../apache_beam/transforms/trigger_test.py  |   2 +-
 sdks/python/apache_beam/transforms/util_test.py |   2 +-
 .../apache_beam/transforms/window_test.py   |   2 +-
 .../transforms/write_ptransform_test.py |   2 +-
 .../typehints/typed_pipeline_test.py|   4 +-
 sdks/python/apache_beam/utils/test_stream.py| 163 ---
 .../apache_beam/utils/test_stream_test.py   |  83 
 sdks/python/setup.py|   2 +-
 60 files changed, 1143 insertions(+), 1143 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/2ecebd22/sdks/python/apache_beam/coders/standard_coders_test.py
--
diff --git a/sdks/python/apache_beam/coders/standard_coders_test.py 
b/sdks/python/apache_beam/coders/standard_coders_test.py
index 4a48ed9..885e88f 100644
--- a/sdks/python/apache_beam/coders/standard_coders_test.py
+++ b/sdks/python/apache_beam/coders/standard_coders_test.py
@@ -34,7 +34,7 @@ from apache_beam.transforms.window import IntervalWindow
 from apache_beam.transforms i

[3/3] beam git commit: This closes #3031

2017-05-09 Thread altay

This closes #3031


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/0cc8b9ed
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/0cc8b9ed
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/0cc8b9ed

Branch: refs/heads/release-2.0.0
Commit: 0cc8b9ed7f9e0af27a5801e5fd2a6cffd4049afe
Parents: aa8c9d1 2ecebd2
Author: Ahmet Altay 
Authored: Tue May 9 20:09:17 2017 -0700
Committer: Ahmet Altay 
Committed: Tue May 9 20:09:17 2017 -0700

--
 .../apache_beam/coders/standard_coders_test.py  |   2 +-
 .../examples/complete/autocomplete_test.py  |   2 +-
 .../examples/complete/estimate_pi_test.py   |   2 +-
 .../complete/game/hourly_team_score_test.py |   2 +-
 .../examples/complete/game/user_score_test.py   |   2 +-
 .../apache_beam/examples/complete/tfidf_test.py |   2 +-
 .../complete/top_wikipedia_sessions_test.py |   2 +-
 .../cookbook/bigquery_side_input_test.py|   2 +-
 .../cookbook/bigquery_tornadoes_it_test.py  |   4 +-
 .../cookbook/bigquery_tornadoes_test.py |   2 +-
 .../examples/cookbook/coders_test.py|   2 +-
 .../examples/cookbook/combiners_test.py |   2 +-
 .../examples/cookbook/custom_ptransform_test.py |   2 +-
 .../examples/cookbook/filters_test.py   |   2 +-
 .../apache_beam/examples/snippets/snippets.py   |   2 +-
 .../examples/snippets/snippets_test.py  |   2 +-
 .../apache_beam/examples/wordcount_it_test.py   |   6 +-
 sdks/python/apache_beam/io/avroio_test.py   |   2 +-
 .../python/apache_beam/io/concat_source_test.py |   2 +-
 .../apache_beam/io/filebasedsource_test.py  |   6 +-
 sdks/python/apache_beam/io/fileio_test.py   |   2 +-
 .../io/gcp/datastore/v1/helper_test.py  |   2 +-
 .../io/gcp/tests/bigquery_matcher.py|   2 +-
 .../io/gcp/tests/bigquery_matcher_test.py   |   2 +-
 sdks/python/apache_beam/io/sources_test.py  |   2 +-
 sdks/python/apache_beam/io/textio_test.py   |   2 +-
 sdks/python/apache_beam/io/tfrecordio_test.py   |   2 +-
 sdks/python/apache_beam/pipeline_test.py|   2 +-
 sdks/python/apache_beam/pvalue_test.py  |   2 +-
 .../runners/dataflow/dataflow_runner_test.py|   4 +-
 sdks/python/apache_beam/test_pipeline.py| 163 ---
 sdks/python/apache_beam/test_pipeline_test.py   | 112 ---
 sdks/python/apache_beam/testing/__init__.py |  16 ++
 .../apache_beam/testing/data/privatekey.p12 | Bin 0 -> 2452 bytes
 .../testing/data/standard_coders.yaml   | 196 +++
 .../apache_beam/testing/pipeline_verifiers.py   | 146 ++
 .../testing/pipeline_verifiers_test.py  | 148 ++
 .../python/apache_beam/testing/test_pipeline.py | 163 +++
 .../apache_beam/testing/test_pipeline_test.py   | 112 +++
 sdks/python/apache_beam/testing/test_stream.py  | 163 +++
 .../apache_beam/testing/test_stream_test.py |  83 
 sdks/python/apache_beam/testing/test_utils.py   |  69 +++
 sdks/python/apache_beam/tests/__init__.py   |  16 --
 .../apache_beam/tests/data/privatekey.p12   | Bin 2452 -> 0 bytes
 .../apache_beam/tests/data/standard_coders.yaml | 196 ---
 .../apache_beam/tests/pipeline_verifiers.py | 146 --
 .../tests/pipeline_verifiers_test.py| 148 --
 sdks/python/apache_beam/tests/test_utils.py |  69 ---
 .../apache_beam/transforms/combiners_test.py|   2 +-
 .../apache_beam/transforms/create_test.py   |   2 +-
 .../apache_beam/transforms/ptransform_test.py   |   2 +-
 .../apache_beam/transforms/sideinputs_test.py   |   2 +-
 .../apache_beam/transforms/trigger_test.py  |   2 +-
 sdks/python/apache_beam/transforms/util_test.py |   2 +-
 .../apache_beam/transforms/window_test.py   |   2 +-
 .../transforms/write_ptransform_test.py |   2 +-
 .../typehints/typed_pipeline_test.py|   4 +-
 sdks/python/apache_beam/utils/test_stream.py| 163 ---
 .../apache_beam/utils/test_stream_test.py   |  83 
 sdks/python/setup.py|   2 +-
 60 files changed, 1143 insertions(+), 1143 deletions(-)
--

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Dataflow #3090

2017-05-09 Thread Apache Jenkins Server

See

[jira] [Commented] (BEAM-2233) Java 8 examples should separate runners into distinct profiles, like Java 7 examples

2017-05-09 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-2233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003941#comment-16003941
 ] 

ASF GitHub Bot commented on BEAM-2233:
--

Github user kennknowles closed the pull request at:

https://github.com/apache/beam/pull/3013


> Java 8 examples should separate runners into distinct profiles, like Java 7 
> examples
> 
>
> Key: BEAM-2233
> URL: https://issues.apache.org/jira/browse/BEAM-2233
> Project: Beam
>  Issue Type: Bug
>  Components: examples-java
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3013: [BEAM-2233,BEAM-2223] Cherry pick #3002: fixing Jav...

2017-05-09 Thread kennknowles

Github user kennknowles closed the pull request at:

https://github.com/apache/beam/pull/3013


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #3015: Cherry-pick #2987: javadoc improvements

2017-05-09 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3015


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #3015: Cherry-pick #2987: javadoc improvements

2017-05-09 Thread kenn

This closes #3015: Cherry-pick #2987: javadoc improvements

  Revise javadoc for sdk, state, options, annotations, window, (and misc)


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/aa8c9d10
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/aa8c9d10
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/aa8c9d10

Branch: refs/heads/release-2.0.0
Commit: aa8c9d10f27ecf974f797a2a940c1bb306d95e05
Parents: 46fa490 4950e32
Author: Kenneth Knowles 
Authored: Tue May 9 19:42:40 2017 -0700
Committer: Kenneth Knowles 
Committed: Tue May 9 19:42:40 2017 -0700

--
 .../main/java/org/apache/beam/sdk/Pipeline.java | 57 +
 .../org/apache/beam/sdk/PipelineResult.java | 11 ++-
 .../org/apache/beam/sdk/PipelineRunner.java | 10 +--
 .../beam/sdk/annotations/Experimental.java  | 25 +++---
 .../apache/beam/sdk/annotations/Internal.java   |  6 +-
 .../org/apache/beam/sdk/coders/AtomicCoder.java |  6 +-
 .../main/java/org/apache/beam/sdk/io/Read.java  |  6 +-
 .../DefaultPipelineOptionsRegistrar.java|  3 +-
 .../beam/sdk/options/PipelineOptionSpec.java|  4 +-
 .../apache/beam/sdk/options/ValueProvider.java  | 15 ++--
 .../org/apache/beam/sdk/state/BagState.java |  6 +-
 .../apache/beam/sdk/state/CombiningState.java   | 23 +++---
 .../apache/beam/sdk/state/GroupingState.java| 17 ++--
 .../org/apache/beam/sdk/state/MapState.java | 27 ++
 .../apache/beam/sdk/state/ReadableState.java| 16 ++--
 .../org/apache/beam/sdk/state/SetState.java | 14 ++--
 .../java/org/apache/beam/sdk/state/State.java   |  6 +-
 .../org/apache/beam/sdk/state/StateBinder.java  |  1 -
 .../org/apache/beam/sdk/state/StateContext.java |  1 -
 .../apache/beam/sdk/state/StateContexts.java|  5 +-
 .../org/apache/beam/sdk/state/StateSpecs.java   | 86 ++--
 .../org/apache/beam/sdk/state/TimeDomain.java   |  8 +-
 .../java/org/apache/beam/sdk/state/Timer.java   | 12 ++-
 .../org/apache/beam/sdk/state/TimerSpecs.java   |  4 +-
 .../org/apache/beam/sdk/state/ValueState.java   |  8 +-
 .../beam/sdk/state/WatermarkHoldState.java  |  4 +-
 .../beam/sdk/transforms/windowing/AfterAll.java |  4 +-
 .../sdk/transforms/windowing/AfterFirst.java|  3 +-
 .../sdk/transforms/windowing/AfterPane.java |  3 +-
 .../windowing/AfterProcessingTime.java  |  4 +-
 .../transforms/windowing/DefaultTrigger.java|  8 +-
 .../sdk/transforms/windowing/GlobalWindows.java |  5 +-
 .../transforms/windowing/InvalidWindows.java|  2 +-
 .../MergeOverlappingIntervalWindows.java|  6 +-
 .../beam/sdk/transforms/windowing/Never.java|  2 +-
 .../windowing/NonMergingWindowFn.java   |  2 +-
 .../transforms/windowing/OrFinallyTrigger.java  |  5 +-
 .../sdk/transforms/windowing/Repeatedly.java|  4 +-
 .../beam/sdk/transforms/windowing/Sessions.java | 13 +--
 .../transforms/windowing/TimestampCombiner.java |  5 ++
 .../beam/sdk/transforms/windowing/Window.java   | 12 +--
 .../beam/sdk/transforms/windowing/WindowFn.java | 10 +--
 .../transforms/windowing/WindowMappingFn.java   | 12 ++-
 .../java/org/apache/beam/sdk/values/PBegin.java | 26 +++---
 44 files changed, 271 insertions(+), 236 deletions(-)
--

[1/2] beam git commit: Revise javadoc for sdk, state, options, annotations, window, (and misc)

2017-05-09 Thread kenn

Repository: beam
Updated Branches:
  refs/heads/release-2.0.0 46fa490a5 -> aa8c9d10f


Revise javadoc for sdk, state, options, annotations, window, (and misc)


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/4950e328
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/4950e328
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/4950e328

Branch: refs/heads/release-2.0.0
Commit: 4950e3280e9d0467356e5be66bbbc5143bfcfa72
Parents: fb6130f
Author: Kenneth Knowles 
Authored: Mon May 8 21:28:37 2017 -0700
Committer: Kenneth Knowles 
Committed: Tue May 9 15:28:18 2017 -0700

--
 .../main/java/org/apache/beam/sdk/Pipeline.java | 57 +
 .../org/apache/beam/sdk/PipelineResult.java | 11 ++-
 .../org/apache/beam/sdk/PipelineRunner.java | 10 +--
 .../beam/sdk/annotations/Experimental.java  | 25 +++---
 .../apache/beam/sdk/annotations/Internal.java   |  6 +-
 .../org/apache/beam/sdk/coders/AtomicCoder.java |  6 +-
 .../main/java/org/apache/beam/sdk/io/Read.java  |  6 +-
 .../DefaultPipelineOptionsRegistrar.java|  3 +-
 .../beam/sdk/options/PipelineOptionSpec.java|  4 +-
 .../apache/beam/sdk/options/ValueProvider.java  | 15 ++--
 .../org/apache/beam/sdk/state/BagState.java |  6 +-
 .../apache/beam/sdk/state/CombiningState.java   | 23 +++---
 .../apache/beam/sdk/state/GroupingState.java| 17 ++--
 .../org/apache/beam/sdk/state/MapState.java | 27 ++
 .../apache/beam/sdk/state/ReadableState.java| 16 ++--
 .../org/apache/beam/sdk/state/SetState.java | 14 ++--
 .../java/org/apache/beam/sdk/state/State.java   |  6 +-
 .../org/apache/beam/sdk/state/StateBinder.java  |  1 -
 .../org/apache/beam/sdk/state/StateContext.java |  1 -
 .../apache/beam/sdk/state/StateContexts.java|  5 +-
 .../org/apache/beam/sdk/state/StateSpecs.java   | 86 ++--
 .../org/apache/beam/sdk/state/TimeDomain.java   |  8 +-
 .../java/org/apache/beam/sdk/state/Timer.java   | 12 ++-
 .../org/apache/beam/sdk/state/TimerSpecs.java   |  4 +-
 .../org/apache/beam/sdk/state/ValueState.java   |  8 +-
 .../beam/sdk/state/WatermarkHoldState.java  |  4 +-
 .../beam/sdk/transforms/windowing/AfterAll.java |  4 +-
 .../sdk/transforms/windowing/AfterFirst.java|  3 +-
 .../sdk/transforms/windowing/AfterPane.java |  3 +-
 .../windowing/AfterProcessingTime.java  |  4 +-
 .../transforms/windowing/DefaultTrigger.java|  8 +-
 .../sdk/transforms/windowing/GlobalWindows.java |  5 +-
 .../transforms/windowing/InvalidWindows.java|  2 +-
 .../MergeOverlappingIntervalWindows.java|  6 +-
 .../beam/sdk/transforms/windowing/Never.java|  2 +-
 .../windowing/NonMergingWindowFn.java   |  2 +-
 .../transforms/windowing/OrFinallyTrigger.java  |  5 +-
 .../sdk/transforms/windowing/Repeatedly.java|  4 +-
 .../beam/sdk/transforms/windowing/Sessions.java | 13 +--
 .../transforms/windowing/TimestampCombiner.java |  5 ++
 .../beam/sdk/transforms/windowing/Window.java   | 12 +--
 .../beam/sdk/transforms/windowing/WindowFn.java | 10 +--
 .../transforms/windowing/WindowMappingFn.java   | 12 ++-
 .../java/org/apache/beam/sdk/values/PBegin.java | 26 +++---
 44 files changed, 271 insertions(+), 236 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/4950e328/sdks/java/core/src/main/java/org/apache/beam/sdk/Pipeline.java
--
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/Pipeline.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/Pipeline.java
index f4da6ad..83496a5 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/Pipeline.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/Pipeline.java
@@ -53,19 +53,14 @@ import org.slf4j.LoggerFactory;
 
 /**
  * A {@link Pipeline} manages a directed acyclic graph of {@link PTransform 
PTransforms}, and the
- * {@link PCollection PCollections} that the {@link PTransform}s consume and 
produce.
- *
- * A {@link Pipeline} is initialized with a {@link PipelineRunner} that 
will later
- * execute the {@link Pipeline}.
- *
- * {@link Pipeline Pipelines} are independent, so they can be constructed 
and executed
- * concurrently.
+ * {@link PCollection PCollections} that the {@link PTransform PTransforms} 
consume and produce.
  *
  * Each {@link Pipeline} is self-contained and isolated from any other
  * {@link Pipeline}. The {@link PValue PValues} that are inputs and outputs of 
each of a
  * {@link Pipeline Pipeline's} {@link PTransform PTransforms} are also owned 
by that
  * {@link Pipeline}. A {@link PValue} owned by one {@link Pipeline} can be 
read only by
- * {@link PTransform PTransforms} also owned by that {@link Pipeline}.
+ * {@link PTransform PTransforms} also owned by that {@link Pipeline}. {@link 
Pipeli

Build failed in Jenkins: beam_PostCommit_Python_Verify #2175

2017-05-09 Thread Apache Jenkins Server

See 


Changes:

[altay] [BEAM-2236] Move test utilities out of python core

--
[...truncated 1.74 MB...]
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": ""
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "assert_that/Unkey.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s14"
}, 
"serialized_fn": "", 
"user_name": "assert_that/Unkey"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s16", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": "_equal"
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "assert_that/Match.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s15"
}, 
"serialized_fn": "", 
"user_name": "assert_that/Match"
  }
}
  ], 
  "type": "JOB_TYPE_BATCH"
}
root: INFO: Create job: 
root: INFO: Created job with id: [2017-05-09_19_19_22-8297893128687954825]
root: INFO: To access the Dataflow monitoring console, please navigate to 
https://console.developers.google.com/project/apache-beam-testing/dataflow/job/2017-05-09_19_19_22-8297893128687954825
root: INFO: Job 2017-05-09_19_19_22-8297893128687954825 is in state 
JOB_STATE_RUNNING
root: INFO: 2017-05-10T02:19:22.469Z: JOB_MESSAGE_WARNING: (7328

[jira] [Commented] (BEAM-2242) Apache Beam Java modules do not correctly shade test artifacts

2017-05-09 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-2242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003915#comment-16003915
 ] 

ASF GitHub Bot commented on BEAM-2242:
--

GitHub user lukecwik opened a pull request:

https://github.com/apache/beam/pull/3034

[BEAM-2242] Ensure that the test jar is shaded correctly

This required listing the maven jar plugin earlier within the file so that 
the jar/test jar ran before shade within the package phase.

I also update the maven shade plugin version in archetypes to pickup the 
fixes in the service file transformer and dropped the usage of finalName in the 
shade plugin so all artifacts are named the same way.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lukecwik/incubator-beam beam2242

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3034.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3034


commit ea938d06f7f431c94d5635e9b1b885283c5cf604
Author: Luke Cwik 
Date:   2017-05-10T02:07:36Z

[BEAM-2242] Ensure that the test jar is shaded correctly

This required listing the maven jar plugin earlier within the file so that 
the jar/test jar ran before shade within the package phase.

I also update the maven shade plugin version in archetypes to pickup the 
fixes in the service file transformer and dropped the usage of finalName in the 
shade plugin so all artifacts are named the same way.




> Apache Beam Java modules do not correctly shade test artifacts
> --
>
> Key: BEAM-2242
> URL: https://issues.apache.org/jira/browse/BEAM-2242
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Blocker
> Fix For: 2.0.0
>
>
> javap dump of TextIOTest.class
> Not the unshaded references to org.apache.commons.compress.*
> {code}
> Compiled from "TextIOTest.java"
> public class org.apache.beam.sdk.io.TextIOTest {
> ...
>   private static java.io.File writeToFile(java.lang.String[], 
> java.lang.String, org.apache.beam.sdk.io.TextIO$CompressionType) throws 
> java.io.IOException;
> descriptor: 
> ([Ljava/lang/String;Ljava/lang/String;Lorg/apache/beam/sdk/io/TextIO$CompressionType;)Ljava/io/File;
> Code:
>0: getstatic #6  // Field 
> tempFolder:Ljava/nio/file/Path;
>3: aload_1
>4: invokeinterface #7,  2// InterfaceMethod 
> java/nio/file/Path.resolve:(Ljava/lang/String;)Ljava/nio/file/Path;
>9: invokeinterface #8,  1// InterfaceMethod 
> java/nio/file/Path.toFile:()Ljava/io/File;
>   14: astore_3
>   15: new   #9  // class java/io/FileOutputStream
>   18: dup
>   19: aload_3
>   20: invokespecial #10 // Method 
> java/io/FileOutputStream."":(Ljava/io/File;)V
>   23: astore4
>   25: getstatic #11 // Field 
> org/apache/beam/sdk/io/TextIOTest$4.$SwitchMap$org$apache$beam$sdk$io$TextIO$CompressionType:[I
>   28: aload_2
>   29: invokevirtual #12 // Method 
> org/apache/beam/sdk/io/TextIO$CompressionType.ordinal:()I
>   32: iaload
>   33: tableswitch   { // 1 to 5
>  1: 68
>  2: 71
>  3: 85
>  4: 99
>  5: 131
>default: 145
>   }
>   68: goto  157
>   71: new   #13 // class 
> java/util/zip/GZIPOutputStream
>   74: dup
>   75: aload 4
>   77: invokespecial #14 // Method 
> java/util/zip/GZIPOutputStream."":(Ljava/io/OutputStream;)V
>   80: astore4
>   82: goto  157
>   85: new   #15 // class 
> org/apache/commons/compress/compressors/bzip2/BZip2CompressorOutputStream
>   88: dup
>   89: aload 4
>   91: invokespecial #16 // Method 
> org/apache/commons/compress/compressors/bzip2/BZip2Compres

[GitHub] beam pull request #3034: [BEAM-2242] Ensure that the test jar is shaded corr...

2017-05-09 Thread lukecwik

GitHub user lukecwik opened a pull request:

https://github.com/apache/beam/pull/3034

[BEAM-2242] Ensure that the test jar is shaded correctly

This required listing the maven jar plugin earlier within the file so that 
the jar/test jar ran before shade within the package phase.

I also update the maven shade plugin version in archetypes to pickup the 
fixes in the service file transformer and dropped the usage of finalName in the 
shade plugin so all artifacts are named the same way.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lukecwik/incubator-beam beam2242

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3034.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3034


commit ea938d06f7f431c94d5635e9b1b885283c5cf604
Author: Luke Cwik 
Date:   2017-05-10T02:07:36Z

[BEAM-2242] Ensure that the test jar is shaded correctly

This required listing the maven jar plugin earlier within the file so that 
the jar/test jar ran before shade within the package phase.

I also update the maven shade plugin version in archetypes to pickup the 
fixes in the service file transformer and dropped the usage of finalName in the 
shade plugin so all artifacts are named the same way.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-2244) Move runner-facing Metrics classes to runners core

2017-05-09 Thread Ben Chambers (JIRA)

Ben Chambers created BEAM-2244:
--

 Summary: Move runner-facing Metrics classes to runners core
 Key: BEAM-2244
 URL: https://issues.apache.org/jira/browse/BEAM-2244
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Ben Chambers
Assignee: Davor Bonaci






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: beam_PostCommit_Python_Verify #2174

2017-05-09 Thread Apache Jenkins Server

See 


Changes:

[ccy] Update Apache Beam Python version to 2.1.0.dev

[davor] Register TestSparkPipelineOptions only in src/test to avoid hard

[davor] Add per-runner profile to Java 8 examples

[altay] Remove trailing whitespace

[davor] Shade dependencies in sdks/core

[davor] Fix checkstyle error

[davor] Include 'sun.reflect' in GcpCoreApiSurfaceTest

--
[...truncated 1.75 MB...]
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "assert_that/Unkey.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s12"
}, 
"serialized_fn": "", 
"user_name": "assert_that/Unkey"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s14", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": "_equal"
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "assert_that/Match.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s13"
}, 
"serialized_fn": "", 
"user_name": "assert_that/Match"
  }
}
  ], 
  "type": "JOB_TYPE_BATCH"
}
root: INFO: Create job: 
root: INFO: Created job with id: [2017-05-09_18_38_19-10569987656720048055]
root: INFO: To access the Dataflow monitoring console, please navigate to 
https://console.developers.google.com/project/apache-beam-testing/dataflow/job/2017-05-09_18_38_19-10569987656720048055
root: INFO: Job 2017-05-09_18_38_19-10569987656720048055 is in state 
JOB_STATE_RUNNING
root: INFO: 2017-05-10T01:38:19.388Z: JOB_MESSAGE_WARNING: (92b023c5870fd5b3): 
Setting the number of workers (1) disables autoscaling for this job. If you are 
trying to cap autoscaling, consider only setting max_num_workers. If you want 
to disable autoscaling altogether, the documented way is to explicitly use 
autoscalingAlgorithm=NONE.
root: INFO: 2017-05-10T01:38:21.145Z: JOB_MESS

[jira] [Commented] (BEAM-1340) Remove or make private public bits of the SDK that shouldn't be public

2017-05-09 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-1340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003897#comment-16003897
 ] 

ASF GitHub Bot commented on BEAM-1340:
--

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/3033

[BEAM-1340] Mark PValue and PValueBase Internal

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
These should not be referred to as their abstract types by users, and
PipelineRunners won't interact with PValues that aren't PCollections or
PCollectionViews.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam pvalues_internal

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3033.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3033


commit 26dd294da2c79a45d62a1572479a8c3f60209b92
Author: Thomas Groh 
Date:   2017-05-10T01:32:14Z

Mark PValue and PValueBase Internal

These should not be referred to as their abstract types by users, and
PipelineRunners won't interact with PValues that aren't PCollections or
PCollectionViews.




> Remove or make private public bits of the SDK that shouldn't be public
> --
>
> Key: BEAM-1340
> URL: https://issues.apache.org/jira/browse/BEAM-1340
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core, sdk-java-extensions
>Reporter: Kenneth Knowles
>Priority: Blocker
>  Labels: backward-incompatible
> Fix For: 2.0.0
>
>
> This JIRA is for the many small changes that do not merit their own JIRA 
> towards getting the SDK's API surface right. For example, removal of 
> `DoFn.InputProvider` and `DoFn.OutputReceiver`.
> While the above is not quite backwards incompatible, succeeding at this task 
> surely will be.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3033: [BEAM-1340] Mark PValue and PValueBase Internal

2017-05-09 Thread tgroh

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/3033

[BEAM-1340] Mark PValue and PValueBase Internal

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
These should not be referred to as their abstract types by users, and
PipelineRunners won't interact with PValues that aren't PCollections or
PCollectionViews.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam pvalues_internal

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3033.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3033


commit 26dd294da2c79a45d62a1572479a8c3f60209b92
Author: Thomas Groh 
Date:   2017-05-10T01:32:14Z

Mark PValue and PValueBase Internal

These should not be referred to as their abstract types by users, and
PipelineRunners won't interact with PValues that aren't PCollections or
PCollectionViews.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #3031: [BEAM-2236] Move test utilities out of python core

2017-05-09 Thread markflyhigh

GitHub user markflyhigh opened a pull request:

https://github.com/apache/beam/pull/3031

[BEAM-2236] Move test utilities out of python core

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---

Move test utils from python core to `apache_beam/testing/`

cherry-pick changes from master.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markflyhigh/incubator-beam 
move-test-utils-out-core

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3031.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3031


commit a573c8eb784a097d1f492672fe40c7196f00de05
Author: Mark Liu 
Date:   2017-05-09T23:41:46Z

[BEAM-2236] Move test utilities out of python core




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Flink #2755

2017-05-09 Thread Apache Jenkins Server

See

[GitHub] beam pull request #3032: Update Coder and DirectRunner documentation

2017-05-09 Thread tgroh

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/3032

Update Coder and DirectRunner documentation

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam public_docs

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3032.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3032


commit d4c3b97e9a73a51529f392397b15943f19ca9871
Author: Thomas Groh 
Date:   2017-05-10T01:04:01Z

Improve DirectRunner Javadoc

commit 6610e3099f40b17823cc282f58f016330e618447
Author: Thomas Groh 
Date:   2017-05-10T01:09:12Z

Update Coder Documentation

commit 2f94491fb1525d0ec7260d8a19ea662d2791722e
Author: Thomas Groh 
Date:   2017-05-10T01:32:14Z

Mark PValue and PValueBase Internal

These should not be referred to as their abstract types by users, and
PipelineRunners won't interact with PValues that aren't PCollections or
PCollectionViews.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2236) Move test utilities out of python core

2017-05-09 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-2236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003896#comment-16003896
 ] 

ASF GitHub Bot commented on BEAM-2236:
--

GitHub user markflyhigh opened a pull request:

https://github.com/apache/beam/pull/3031

[BEAM-2236] Move test utilities out of python core

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---

Move test utils from python core to `apache_beam/testing/`

cherry-pick changes from master.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markflyhigh/incubator-beam 
move-test-utils-out-core

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3031.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3031


commit a573c8eb784a097d1f492672fe40c7196f00de05
Author: Mark Liu 
Date:   2017-05-09T23:41:46Z

[BEAM-2236] Move test utilities out of python core




> Move test utilities out of python core
> --
>
> Key: BEAM-2236
> URL: https://issues.apache.org/jira/browse/BEAM-2236
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Mark Liu
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is back to normal : beam_PostCommit_Java_MavenInstall #3742

2017-05-09 Thread Apache Jenkins Server

See

[jira] [Commented] (BEAM-2222) Clean up readme files

2017-05-09 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003886#comment-16003886
 ] 

ASF GitHub Bot commented on BEAM-:
--

GitHub user aaltay opened a pull request:

https://github.com/apache/beam/pull/3030

[BEAM-] Remove Readme files

extensions -> moving to website @
https://github.com/apache/beam-site/pull/237

jdbc/test -> obsolete content, removed.

R: @davorbonaci 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aaltay/beam readme

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3030.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3030


commit 5de8f4dbf798ee4b31d7917a47617eab9e51acda
Author: Ahmet Altay 
Date:   2017-05-10T01:21:39Z

Remove Readme files.

extensions -> moving to website @
https://github.com/apache/beam-site/pull/237

jdbc/test -> obsolete content, removed.




> Clean up readme files
> -
>
> Key: BEAM-
> URL: https://issues.apache.org/jira/browse/BEAM-
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Ahmet Altay
>Assignee: Ahmet Altay
> Fix For: 2.0.0
>
>
> Move content from readme.md's to website.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3030: [BEAM-2222] Remove Readme files

2017-05-09 Thread aaltay

GitHub user aaltay opened a pull request:

https://github.com/apache/beam/pull/3030

[BEAM-] Remove Readme files

extensions -> moving to website @
https://github.com/apache/beam-site/pull/237

jdbc/test -> obsolete content, removed.

R: @davorbonaci 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aaltay/beam readme

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3030.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3030


commit 5de8f4dbf798ee4b31d7917a47617eab9e51acda
Author: Ahmet Altay 
Date:   2017-05-10T01:21:39Z

Remove Readme files.

extensions -> moving to website @
https://github.com/apache/beam-site/pull/237

jdbc/test -> obsolete content, removed.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2236) Move test utilities out of python core

2017-05-09 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-2236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003873#comment-16003873
 ] 

ASF GitHub Bot commented on BEAM-2236:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3017


> Move test utilities out of python core
> --
>
> Key: BEAM-2236
> URL: https://issues.apache.org/jira/browse/BEAM-2236
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Mark Liu
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[3/3] beam git commit: This closes #3017

2017-05-09 Thread altay

This closes #3017


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/e2be6961
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/e2be6961
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/e2be6961

Branch: refs/heads/master
Commit: e2be6961d07f1c7efe73dbc00280c8e5c9c94dfa
Parents: 78a4b8e c028349
Author: Ahmet Altay 
Authored: Tue May 9 18:14:57 2017 -0700
Committer: Ahmet Altay 
Committed: Tue May 9 18:14:57 2017 -0700

--
 .../apache_beam/coders/standard_coders_test.py  |   2 +-
 .../examples/complete/autocomplete_test.py  |   2 +-
 .../examples/complete/estimate_pi_test.py   |   2 +-
 .../complete/game/hourly_team_score_test.py |   2 +-
 .../examples/complete/game/user_score_test.py   |   2 +-
 .../apache_beam/examples/complete/tfidf_test.py |   2 +-
 .../complete/top_wikipedia_sessions_test.py |   2 +-
 .../cookbook/bigquery_side_input_test.py|   2 +-
 .../cookbook/bigquery_tornadoes_it_test.py  |   4 +-
 .../cookbook/bigquery_tornadoes_test.py |   2 +-
 .../examples/cookbook/coders_test.py|   2 +-
 .../examples/cookbook/combiners_test.py |   2 +-
 .../examples/cookbook/custom_ptransform_test.py |   2 +-
 .../examples/cookbook/filters_test.py   |   2 +-
 .../apache_beam/examples/snippets/snippets.py   |   2 +-
 .../examples/snippets/snippets_test.py  |   2 +-
 .../apache_beam/examples/wordcount_it_test.py   |   6 +-
 sdks/python/apache_beam/io/avroio_test.py   |   2 +-
 .../python/apache_beam/io/concat_source_test.py |   2 +-
 .../apache_beam/io/filebasedsource_test.py  |   6 +-
 sdks/python/apache_beam/io/fileio_test.py   |   2 +-
 .../io/gcp/datastore/v1/helper_test.py  |   2 +-
 .../io/gcp/tests/bigquery_matcher.py|   2 +-
 .../io/gcp/tests/bigquery_matcher_test.py   |   2 +-
 sdks/python/apache_beam/io/sources_test.py  |   2 +-
 sdks/python/apache_beam/io/textio_test.py   |   2 +-
 sdks/python/apache_beam/io/tfrecordio_test.py   |   2 +-
 sdks/python/apache_beam/pipeline_test.py|   2 +-
 sdks/python/apache_beam/pvalue_test.py  |   2 +-
 .../runners/dataflow/dataflow_runner_test.py|   4 +-
 sdks/python/apache_beam/test_pipeline.py| 163 ---
 sdks/python/apache_beam/test_pipeline_test.py   | 112 ---
 sdks/python/apache_beam/testing/__init__.py |  16 ++
 .../apache_beam/testing/data/privatekey.p12 | Bin 0 -> 2452 bytes
 .../testing/data/standard_coders.yaml   | 196 +++
 .../apache_beam/testing/pipeline_verifiers.py   | 146 ++
 .../testing/pipeline_verifiers_test.py  | 148 ++
 .../python/apache_beam/testing/test_pipeline.py | 163 +++
 .../apache_beam/testing/test_pipeline_test.py   | 112 +++
 sdks/python/apache_beam/testing/test_stream.py  | 163 +++
 .../apache_beam/testing/test_stream_test.py |  83 
 sdks/python/apache_beam/testing/test_utils.py   |  69 +++
 sdks/python/apache_beam/tests/__init__.py   |  16 --
 .../apache_beam/tests/data/privatekey.p12   | Bin 2452 -> 0 bytes
 .../apache_beam/tests/data/standard_coders.yaml | 196 ---
 .../apache_beam/tests/pipeline_verifiers.py | 146 --
 .../tests/pipeline_verifiers_test.py| 148 --
 sdks/python/apache_beam/tests/test_utils.py |  69 ---
 .../apache_beam/transforms/combiners_test.py|   2 +-
 .../apache_beam/transforms/create_test.py   |   2 +-
 .../apache_beam/transforms/ptransform_test.py   |   2 +-
 .../apache_beam/transforms/sideinputs_test.py   |   2 +-
 .../apache_beam/transforms/trigger_test.py  |   2 +-
 sdks/python/apache_beam/transforms/util_test.py |   2 +-
 .../apache_beam/transforms/window_test.py   |   2 +-
 .../transforms/write_ptransform_test.py |   2 +-
 .../typehints/typed_pipeline_test.py|   4 +-
 sdks/python/apache_beam/utils/test_stream.py| 163 ---
 .../apache_beam/utils/test_stream_test.py   |  83 
 sdks/python/setup.py|   2 +-
 60 files changed, 1143 insertions(+), 1143 deletions(-)
--

[GitHub] beam pull request #3017: [BEAM-2236] Move test utilities out of python core

2017-05-09 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3017


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/3] beam git commit: [BEAM-2236] Move test utilities out of python core

2017-05-09 Thread altay

Repository: beam
Updated Branches:
  refs/heads/master 78a4b8e4f -> e2be6961d


http://git-wip-us.apache.org/repos/asf/beam/blob/c028349c/sdks/python/apache_beam/tests/data/standard_coders.yaml
--
diff --git a/sdks/python/apache_beam/tests/data/standard_coders.yaml 
b/sdks/python/apache_beam/tests/data/standard_coders.yaml
deleted file mode 100644
index 790cacb..000
--- a/sdks/python/apache_beam/tests/data/standard_coders.yaml
+++ /dev/null
@@ -1,196 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements.  See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License.  You may obtain a copy of the License at
-#
-#http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-# This file is broken into multiple sections delimited by ---. Each section 
specifies a set of
-# reference encodings for a single standardized coder used in a specific 
context.
-#
-# Each section contains up to 3 properties:
-#
-#   coder: a common coder spec. Currently, a URN and URNs for component coders 
as necessary.
-#   nested: a boolean meaning whether the coder was used in the nested 
context. Missing means to
-#   test both contexts, a shorthand for when the coder is invariant 
across context.
-#   examples: a map of {encoded bytes: original JSON object} encoded with the 
coder in the context.
-# The LHS (key) is a byte array encoded as a JSON-escaped string. 
The RHS (value) is
-# one of a few standard JSON types such as numbers, strings, dicts 
that map naturally
-# to the type encoded by the coder.
-#
-# These choices were made to strike a balance between portability, ease of 
use, and simple
-# legibility of this file itself.
-#
-# It is expected that future work will move the `coder` field into a format 
that it would be
-# represented by the Runner API, so that it can be understood by all SDKs and 
harnesses.
-#
-# If a coder is marked non-deterministic in the coder spec, then only the 
decoding should be validated.
-
-
-
-coder:
-  urn: "urn:beam:coders:bytes:0.1"
-nested: false
-examples:
-  "abc": abc
-  "ab\0c": "ab\0c"
-

-
-coder:
-  urn: "urn:beam:coders:bytes:0.1"
-nested: true
-examples:
-  "\u0003abc": abc
-  "\u0004ab\0c": "ab\0c"
-  "\u00c8\u0001   10|   20|   30|   40|   50|   60|
   70|   80|   90|  100|  110|  120|  130|  140|
  150|  160|  170|  180|  190|  200|":
-  "   10|   20|   30|   40|   50|   60|
   70|   80|   90|  100|  110|  120|  130|  140|
  150|  160|  170|  180|  190|  200|"
-

-
-coder:
-  urn: "urn:beam:coders:varint:0.1"
-examples:
-  "\0": 0
-  "\u0001": 1
-  "\u000A": 10
-  "\u00c8\u0001": 200
-  "\u00e8\u0007": 1000
-  "\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u0001": -1
-

-
-coder:
-  urn: "urn:beam:coders:kv:0.1"
-  components: [{urn: "urn:beam:coders:bytes:0.1"},
-   {urn: "urn:beam:coders:varint:0.1"}]
-examples:
-  "\u0003abc\0": {key: abc, value: 0}
-  "\u0004ab\0c\u000A": {key: "ab\0c", value: 10}
-

-
-coder:
-  urn: "urn:beam:coders:kv:0.1"
-  components: [{urn: "urn:beam:coders:bytes:0.1"},
-   {urn: "urn:beam:coders:bytes:0.1"}]
-nested: false
-examples:
-  "\u0003abcdef": {key: abc, value: def}
-  "\u0004ab\0cde\0f": {key: "ab\0c", value: "de\0f"}
-

-
-coder:
-  urn: "urn:beam:coders:kv:0.1"
-  components: [{urn: "urn:beam:coders:bytes:0.1"},
-   {urn: "urn:beam:coders:bytes:0.1"}]
-nested: true
-examples:
-  "\u0003abc\u0003def": {key: abc, value: def}
-  "\u0004ab\0c\u0004de\0f": {key: "ab\0c", value: "de\0f"}
-

-
-coder:
-  urn: "urn:beam:coders:interval_window:0.1"
-examples:
-  "\u0080\u\u0001\u0052\u009a\u00a4\u009b\u0068\u0080\u00dd\u00db\u0001" : 
{end: 1454293425000, span: 360}
-  
"\u0080\u\u0001\u0053\u0034\u00ec\u0074\u00e8\u0080\u0090\u00fb\u00d3\u0009"
 : {end: 1456881825000, span: 259200}
-  "\u007f\u00df\u003b\u0064\u005a\u001c\u00ad\u0076\u00ed\u0002" : {end: 
-9223372036854410, span: 365}
-  "\u0080\u0020\u00c4\u009b\u00a5\u00e3\u0053\u00f7\u" : {end: 
9223372036854775, span: 0}
-

-
-coder:
-  urn: "urn:beam:coders:stream:0.1"
-  components: [{urn: "urn:beam:coders:varint:0.1"}]
-examples:
-  "\0\0\

[2/3] beam git commit: [BEAM-2236] Move test utilities out of python core

2017-05-09 Thread altay

[BEAM-2236] Move test utilities out of python core


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/c028349c
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/c028349c
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/c028349c

Branch: refs/heads/master
Commit: c028349c5ba9d47b6d1d3fb1d765d851c54d2cc1
Parents: 78a4b8e
Author: Mark Liu 
Authored: Tue May 9 16:41:46 2017 -0700
Committer: Ahmet Altay 
Committed: Tue May 9 18:14:54 2017 -0700

--
 .../apache_beam/coders/standard_coders_test.py  |   2 +-
 .../examples/complete/autocomplete_test.py  |   2 +-
 .../examples/complete/estimate_pi_test.py   |   2 +-
 .../complete/game/hourly_team_score_test.py |   2 +-
 .../examples/complete/game/user_score_test.py   |   2 +-
 .../apache_beam/examples/complete/tfidf_test.py |   2 +-
 .../complete/top_wikipedia_sessions_test.py |   2 +-
 .../cookbook/bigquery_side_input_test.py|   2 +-
 .../cookbook/bigquery_tornadoes_it_test.py  |   4 +-
 .../cookbook/bigquery_tornadoes_test.py |   2 +-
 .../examples/cookbook/coders_test.py|   2 +-
 .../examples/cookbook/combiners_test.py |   2 +-
 .../examples/cookbook/custom_ptransform_test.py |   2 +-
 .../examples/cookbook/filters_test.py   |   2 +-
 .../apache_beam/examples/snippets/snippets.py   |   2 +-
 .../examples/snippets/snippets_test.py  |   2 +-
 .../apache_beam/examples/wordcount_it_test.py   |   6 +-
 sdks/python/apache_beam/io/avroio_test.py   |   2 +-
 .../python/apache_beam/io/concat_source_test.py |   2 +-
 .../apache_beam/io/filebasedsource_test.py  |   6 +-
 sdks/python/apache_beam/io/fileio_test.py   |   2 +-
 .../io/gcp/datastore/v1/helper_test.py  |   2 +-
 .../io/gcp/tests/bigquery_matcher.py|   2 +-
 .../io/gcp/tests/bigquery_matcher_test.py   |   2 +-
 sdks/python/apache_beam/io/sources_test.py  |   2 +-
 sdks/python/apache_beam/io/textio_test.py   |   2 +-
 sdks/python/apache_beam/io/tfrecordio_test.py   |   2 +-
 sdks/python/apache_beam/pipeline_test.py|   2 +-
 sdks/python/apache_beam/pvalue_test.py  |   2 +-
 .../runners/dataflow/dataflow_runner_test.py|   4 +-
 sdks/python/apache_beam/test_pipeline.py| 163 ---
 sdks/python/apache_beam/test_pipeline_test.py   | 112 ---
 sdks/python/apache_beam/testing/__init__.py |  16 ++
 .../apache_beam/testing/data/privatekey.p12 | Bin 0 -> 2452 bytes
 .../testing/data/standard_coders.yaml   | 196 +++
 .../apache_beam/testing/pipeline_verifiers.py   | 146 ++
 .../testing/pipeline_verifiers_test.py  | 148 ++
 .../python/apache_beam/testing/test_pipeline.py | 163 +++
 .../apache_beam/testing/test_pipeline_test.py   | 112 +++
 sdks/python/apache_beam/testing/test_stream.py  | 163 +++
 .../apache_beam/testing/test_stream_test.py |  83 
 sdks/python/apache_beam/testing/test_utils.py   |  69 +++
 sdks/python/apache_beam/tests/__init__.py   |  16 --
 .../apache_beam/tests/data/privatekey.p12   | Bin 2452 -> 0 bytes
 .../apache_beam/tests/data/standard_coders.yaml | 196 ---
 .../apache_beam/tests/pipeline_verifiers.py | 146 --
 .../tests/pipeline_verifiers_test.py| 148 --
 sdks/python/apache_beam/tests/test_utils.py |  69 ---
 .../apache_beam/transforms/combiners_test.py|   2 +-
 .../apache_beam/transforms/create_test.py   |   2 +-
 .../apache_beam/transforms/ptransform_test.py   |   2 +-
 .../apache_beam/transforms/sideinputs_test.py   |   2 +-
 .../apache_beam/transforms/trigger_test.py  |   2 +-
 sdks/python/apache_beam/transforms/util_test.py |   2 +-
 .../apache_beam/transforms/window_test.py   |   2 +-
 .../transforms/write_ptransform_test.py |   2 +-
 .../typehints/typed_pipeline_test.py|   4 +-
 sdks/python/apache_beam/utils/test_stream.py| 163 ---
 .../apache_beam/utils/test_stream_test.py   |  83 
 sdks/python/setup.py|   2 +-
 60 files changed, 1143 insertions(+), 1143 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/c028349c/sdks/python/apache_beam/coders/standard_coders_test.py
--
diff --git a/sdks/python/apache_beam/coders/standard_coders_test.py 
b/sdks/python/apache_beam/coders/standard_coders_test.py
index 4a48ed9..885e88f 100644
--- a/sdks/python/apache_beam/coders/standard_coders_test.py
+++ b/sdks/python/apache_beam/coders/standard_coders_test.py
@@ -34,7 +34,7 @@ from apache_beam.transforms.window import IntervalWindow
 from apache_beam.transforms import window
 
 STANDARD_CO

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Flink #2754

2017-05-09 Thread Apache Jenkins Server

See

[GitHub] beam pull request #3029: Remove verifyDeterministic from StructuredCoder

2017-05-09 Thread tgroh

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/3029

Remove verifyDeterministic from StructuredCoder

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
This is replaced within all uses in the Core SDK by
Coder.verifyDeterministic(Coder, String, List), which is a
general utility.

I believe that this change was already authored but lost in a merge.

@kennknowles @lukecwik @davorbonaci 

This is opened against the release branch to determine if it's worth 
merging. It cherry-picks cleanly into master, and I can create the appropriate 
PR if desired.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam coder_arguments

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3029.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3029


commit 68875b48894fa5d3743ebb0696266e9600c64ef3
Author: Thomas Groh 
Date:   2017-05-10T01:09:37Z

Remove verifyDeterministic from StructuredCoder

This is replaced within all uses in the Core SDK by
Coder.verifyDeterministic(Coder, String, List), which is a
general utility.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (BEAM-2243) org.apache.beam.GcpCoreApiSurfaceTest.testApiSurface fails at release-2.0.0 head

2017-05-09 Thread Davor Bonaci (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci resolved BEAM-2243.

Resolution: Fixed

> org.apache.beam.GcpCoreApiSurfaceTest.testApiSurface fails at release-2.0.0 
> head
> 
>
> Key: BEAM-2243
> URL: https://issues.apache.org/jira/browse/BEAM-2243
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Blocker
> Fix For: 2.0.0
>
>
> {code}
> java.lang.AssertionError: 
> Expected: API surface to include only:
>   Classes in package "org.apache.beam"
>   Classes in package "com.google.api.client"
>   Classes in package "com.google.api.services.storage"
>   Classes in package "com.google.auth"
>   Classes in package "com.fasterxml.jackson.annotation"
>   Classes in package "com.fasterxml.jackson.core"
>   Classes in package "com.fasterxml.jackson.databind"
>   Classes in package "org.apache.avro"
>   Classes in package "org.hamcrest"
>   Classes in package "org.codehaus.jackson"
>   Classes in package "org.joda.time"
>   Classes in package "org.junit"
>   
>  but: The following disallowed classes appeared on the API surface:
>   interface sun.reflect.CallerSensitive exposed via:
>   interface sun.reflect.CallerSensitive
>   class 
> org.apache.beam.sdk.repackaged.net.bytebuddy.dynamic.loading.MultipleParentClassLoader
>   at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
>   at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:8)
>   at 
> org.apache.beam.GcpCoreApiSurfaceTest.testApiSurface(GcpCoreApiSurfaceTest.java:55)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3026: Include sun.reflect in allowed api surface packages...

2017-05-09 Thread vikkyrk

Github user vikkyrk closed the pull request at:

https://github.com/apache/beam/pull/3026


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #3028: Shade protobuf-java in runners/google-cloud-dataflo...

2017-05-09 Thread vikkyrk

GitHub user vikkyrk opened a pull request:

https://github.com/apache/beam/pull/3028

Shade protobuf-java in runners/google-cloud-dataflow-java

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vikkyrk/incubator-beam df_shade

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3028.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3028


commit c668642ab5b4e5df0b0db93d6325191cdfd11b88
Author: Vikas Kedigehalli 
Date:   2017-05-09T23:47:06Z

shade protobuf-java in runners/google-cloud-dataflow-java




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2241) Correctly mark top level classes and functions as private

2017-05-09 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-2241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003853#comment-16003853
 ] 

ASF GitHub Bot commented on BEAM-2241:
--

GitHub user mariapython opened a pull request:

https://github.com/apache/beam/pull/3027

[BEAM-2241] Mark PipelineVisitor and AppliedPTransform as internal.




You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mariapython/incubator-beam borrar

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3027.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3027


commit d1f7f5893fea217358a2f552070a504b3d80bbc2
Author: Maria Garcia Herrero 
Date:   2017-05-10T00:43:33Z

Mark PipelineVisitor and AppliedPTransform as internal.




> Correctly mark top level classes and functions as private
> -
>
> Key: BEAM-2241
> URL: https://issues.apache.org/jira/browse/BEAM-2241
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Chamikara Jayalath
>Assignee: Chamikara Jayalath
> Fix For: 2.0.0
>
>
> Seems like some of the top level classes and function of Python SDK are 
> unnecessarily public (does not start with _ ). We should update Python SDK so 
> that top level classes and functions that are only used within the 
> corresponding modules are private. Additionally, any top level classes and 
> functions that are not expected to be used by end-users 
> (pipeline/source/ptransform authors) should be clearly documented.  



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3027: [BEAM-2241] Mark PipelineVisitor and AppliedPTransf...

2017-05-09 Thread mariapython

GitHub user mariapython opened a pull request:

https://github.com/apache/beam/pull/3027

[BEAM-2241] Mark PipelineVisitor and AppliedPTransform as internal.




You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mariapython/incubator-beam borrar

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3027.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3027


commit d1f7f5893fea217358a2f552070a504b3d80bbc2
Author: Maria Garcia Herrero 
Date:   2017-05-10T00:43:33Z

Mark PipelineVisitor and AppliedPTransform as internal.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #2753

2017-05-09 Thread Apache Jenkins Server

See

[jira] [Commented] (BEAM-2222) Clean up readme files

2017-05-09 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003844#comment-16003844
 ] 

ASF GitHub Bot commented on BEAM-:
--

GitHub user aaltay opened a pull request:

https://github.com/apache/beam-site/pull/237

[BEAM-] Move extension README files to website

R: @hadarhg @melap 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aaltay/beam-site asf-site-2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam-site/pull/237.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #237


commit ab3cbb7c685abcb44b156c03af50ff4f5f885851
Author: Ahmet Altay 
Date:   2017-05-10T00:47:01Z

Move extension README files to website




> Clean up readme files
> -
>
> Key: BEAM-
> URL: https://issues.apache.org/jira/browse/BEAM-
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Ahmet Altay
>Assignee: Ahmet Altay
> Fix For: 2.0.0
>
>
> Move content from readme.md's to website.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3026: Include sun.reflect in allowed api surface packages...

2017-05-09 Thread vikkyrk

GitHub user vikkyrk opened a pull request:

https://github.com/apache/beam/pull/3026

Include sun.reflect in allowed api surface packages for

GcpCoreApiSurfaceTest

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vikkyrk/incubator-beam release-2.0.0

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3026.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3026


commit c7b3df656aee00ff7c09db1a47bb19262f759c06
Author: Vikas Kedigehalli 
Date:   2017-05-10T00:44:31Z

Include sun.reflect in allowed api surface packages for
GcpCoreApiSurfaceTest




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam-site pull request #237: [BEAM-2222] Move extension README files to webs...

2017-05-09 Thread aaltay

GitHub user aaltay opened a pull request:

https://github.com/apache/beam-site/pull/237

[BEAM-] Move extension README files to website

R: @hadarhg @melap 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aaltay/beam-site asf-site-2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam-site/pull/237.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #237


commit ab3cbb7c685abcb44b156c03af50ff4f5f885851
Author: Ahmet Altay 
Date:   2017-05-10T00:47:01Z

Move extension README files to website




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1871) Thin Java SDK Core

2017-05-09 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-1871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003828#comment-16003828
 ] 

ASF GitHub Bot commented on BEAM-1871:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3004


> Thin Java SDK Core
> --
>
> Key: BEAM-1871
> URL: https://issues.apache.org/jira/browse/BEAM-1871
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Daniel Halperin
>Assignee: Vikas Kedigehalli
> Fix For: 2.0.0
>
>
> Before first stable release we need to thin out {{sdk-java-core}} module. 
> Some candidates for removal, but not a non-exhaustive list:
> {{sdk/io}}
> * anything BigQuery related
> * anything PubSub related
> * everything Protobuf related
> * TFRecordIO
> * XMLSink
> {{sdk/util}}
> * Everything GCS related
> * Everything Backoff related
> * Everything Google API related: ResponseInterceptors, RetryHttpBackoff, etc.
> * Everything CloudObject-related
> * Pubsub stuff
> {{sdk/coders}}
> * JAXBCoder
> * TableRowJsoNCoder



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3004: [BEAM-1871] [BEAM-1019 ] Shade dependencies in sdks...

2017-05-09 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3004


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[3/4] beam git commit: Shade dependencies in sdks/core

2017-05-09 Thread davor

Shade dependencies in sdks/core


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/62f2173a
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/62f2173a
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/62f2173a

Branch: refs/heads/master
Commit: 62f2173a1c03ba0433ab4941b7546f2d789b998f
Parents: c567550
Author: Vikas Kedigehalli 
Authored: Tue May 9 11:41:29 2017 -0700
Committer: Davor Bonaci 
Committed: Tue May 9 17:37:05 2017 -0700

--
 sdks/java/core/pom.xml  | 74 
 .../org/apache/beam/sdk/util/ApiSurface.java| 10 ++-
 2 files changed, 83 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/62f2173a/sdks/java/core/pom.xml
--
diff --git a/sdks/java/core/pom.xml b/sdks/java/core/pom.xml
index 7ed6c8a..ef1873d 100644
--- a/sdks/java/core/pom.xml
+++ b/sdks/java/core/pom.xml
@@ -120,6 +120,80 @@
   
 
   
+
+  
+org.apache.maven.plugins
+maven-shade-plugin
+
+  
+bundle-and-repackage
+package
+
+  shade
+
+
+  
+
+  com.google.guava:guava
+  com.google.protobuf:protobuf-java
+  net.bytebuddy:byte-buddy
+  org.apache.commons:*
+
+  
+  
+
+  *:*
+  
+META-INF/*.SF
+META-INF/*.DSA
+META-INF/*.RSA
+  
+
+  
+  
+
+  com.google.common
+  
+  
+org.apache.beam.sdk.repackaged.com.google.common
+  
+
+
+  com.google.thirdparty
+  
+  
+org.apache.beam.sdk.repackaged.com.google.thirdparty
+  
+
+
+  com.google.protobuf
+  
+  
+org.apache.beam.sdk.repackaged.com.google.protobuf
+  
+
+
+  net.bytebuddy
+  
+  
+org.apache.beam.sdk.repackaged.net.bytebuddy
+  
+
+
+  org.apache.commons
+  
+  
+org.apache.beam.sdk.repackaged.org.apache.commons
+  
+
+  
+  
+
+  
+
+  
+
+  
 
   
 

http://git-wip-us.apache.org/repos/asf/beam/blob/62f2173a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ApiSurface.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ApiSurface.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ApiSurface.java
index 8c2b988..685efef 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ApiSurface.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ApiSurface.java
@@ -352,7 +352,15 @@ public class ApiSurface {
 
 Set> newRootClasses = Sets.newHashSet();
 for (ClassPath.ClassInfo classInfo : 
classPath.getTopLevelClassesRecursive(packageName)) {
-  Class clazz = classInfo.load();
+  Class clazz = null;
+  try {
+clazz = classInfo.load();
+  } catch (NoClassDefFoundError e) {
+// TODO(BEAM-2231): Ignore any NoClassDefFoundError errors as a 
workaround.
+LOG.warn("Failed to load class: {}", classInfo.toString(), e);
+continue;
+  }
+
   if (exposed(clazz.getModifiers())) {
 newRootClasses.add(clazz);
   }

[4/4] beam git commit: This closes #3004

2017-05-09 Thread davor

This closes #3004


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/78a4b8e4
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/78a4b8e4
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/78a4b8e4

Branch: refs/heads/master
Commit: 78a4b8e4fb014d40f7f6ea3117931f185b9e742b
Parents: c567550 9443ee8
Author: Davor Bonaci 
Authored: Tue May 9 17:38:48 2017 -0700
Committer: Davor Bonaci 
Committed: Tue May 9 17:38:48 2017 -0700

--
 sdks/java/core/pom.xml  | 74 
 .../org/apache/beam/sdk/util/ApiSurface.java| 10 ++-
 .../extensions/gcp/GcpCoreApiSurfaceTest.java   |  3 +-
 3 files changed, 85 insertions(+), 2 deletions(-)
--

[2/4] beam git commit: Fix checkstyle error

2017-05-09 Thread davor

Fix checkstyle error


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/098e6f2b
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/098e6f2b
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/098e6f2b

Branch: refs/heads/master
Commit: 098e6f2b0195fb650d326edb8159ec3688b585f9
Parents: 62f2173
Author: Vikas Kedigehalli 
Authored: Tue May 9 14:28:24 2017 -0700
Committer: Davor Bonaci 
Committed: Tue May 9 17:37:05 2017 -0700

--
 .../core/src/main/java/org/apache/beam/sdk/util/ApiSurface.java| 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/098e6f2b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ApiSurface.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ApiSurface.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ApiSurface.java
index 685efef..735190b 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ApiSurface.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ApiSurface.java
@@ -356,7 +356,7 @@ public class ApiSurface {
   try {
 clazz = classInfo.load();
   } catch (NoClassDefFoundError e) {
-// TODO(BEAM-2231): Ignore any NoClassDefFoundError errors as a 
workaround.
+// TODO: Ignore any NoClassDefFoundError errors as a workaround. 
(BEAM-2231)
 LOG.warn("Failed to load class: {}", classInfo.toString(), e);
 continue;
   }

[1/4] beam git commit: Include 'sun.reflect' in GcpCoreApiSurfaceTest

2017-05-09 Thread davor

Repository: beam
Updated Branches:
  refs/heads/master c567550de -> 78a4b8e4f


Include 'sun.reflect' in GcpCoreApiSurfaceTest


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/9443ee87
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/9443ee87
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/9443ee87

Branch: refs/heads/master
Commit: 9443ee878d77d979dc070b74b0690b569f1635d7
Parents: 098e6f2
Author: Vikas Kedigehalli 
Authored: Tue May 9 17:26:12 2017 -0700
Committer: Davor Bonaci 
Committed: Tue May 9 17:37:05 2017 -0700

--
 .../org/apache/beam/sdk/extensions/gcp/GcpCoreApiSurfaceTest.java | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/9443ee87/sdks/java/extensions/google-cloud-platform-core/src/test/java/org/apache/beam/sdk/extensions/gcp/GcpCoreApiSurfaceTest.java
--
diff --git 
a/sdks/java/extensions/google-cloud-platform-core/src/test/java/org/apache/beam/sdk/extensions/gcp/GcpCoreApiSurfaceTest.java
 
b/sdks/java/extensions/google-cloud-platform-core/src/test/java/org/apache/beam/sdk/extensions/gcp/GcpCoreApiSurfaceTest.java
index a8772c3..625c248 100644
--- 
a/sdks/java/extensions/google-cloud-platform-core/src/test/java/org/apache/beam/sdk/extensions/gcp/GcpCoreApiSurfaceTest.java
+++ 
b/sdks/java/extensions/google-cloud-platform-core/src/test/java/org/apache/beam/sdk/extensions/gcp/GcpCoreApiSurfaceTest.java
@@ -50,7 +50,8 @@ public class GcpCoreApiSurfaceTest {
 "org.codehaus.jackson",
 // via Avro
 "org.joda.time",
-"org.junit");
+"org.junit",
+"sun.reflect");
 
 assertThat(
 ApiSurface.getSdkApiSurface(getClass().getClassLoader()), 
containsOnlyPackages(allowed));

[jira] [Resolved] (BEAM-2223) java8 examples are not running

2017-05-09 Thread Davor Bonaci (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci resolved BEAM-2223.

Resolution: Fixed

> java8 examples are not running
> --
>
> Key: BEAM-2223
> URL: https://issues.apache.org/jira/browse/BEAM-2223
> Project: Beam
>  Issue Type: Bug
>  Components: examples-java
>Reporter: Ahmet Altay
>Assignee: Kenneth Knowles
> Fix For: 2.0.0
>
>
> Could not run java8 examples any more with:
> {code}
> mvn compile exec:java 
> -Dexec.mainClass=org.apache.beam.examples.complete.game.UserScore 
> -Dexec.args="--project= --dataset= 
> --tempLocation="
> {code}
> Fails with:
> {code}
> java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:294)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NoClassDefFoundError: org/hamcrest/Matcher
>   at java.lang.ClassLoader.defineClass1(Native Method)
>   at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
>   at 
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>   at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
>   at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
>   at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
>   at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>   at java.lang.Class.getDeclaredMethods0(Native Method)
>   at java.lang.Class.privateGetDeclaredMethods(Class.java:2703)
>   at java.lang.Class.privateGetPublicMethods(Class.java:2904)
>   at java.lang.Class.privateGetPublicMethods(Class.java:2913)
>   at java.lang.Class.getMethods(Class.java:1617)
>   at sun.misc.ProxyGenerator.generateClassFile(ProxyGenerator.java:451)
>   at sun.misc.ProxyGenerator.generateProxyClass(ProxyGenerator.java:339)
>   at java.lang.reflect.Proxy$ProxyClassFactory.apply(Proxy.java:639)
>   at java.lang.reflect.Proxy$ProxyClassFactory.apply(Proxy.java:557)
>   at java.lang.reflect.WeakCache$Factory.get(WeakCache.java:230)
>   at java.lang.reflect.WeakCache.get(WeakCache.java:127)
>   at java.lang.reflect.Proxy.getProxyClass0(Proxy.java:419)
>   at java.lang.reflect.Proxy.getProxyClass(Proxy.java:371)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.validateWellFormed(PipelineOptionsFactory.java:606)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.register(PipelineOptionsFactory.java:544)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.initializeRegistry(PipelineOptionsFactory.java:570)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.(PipelineOptionsFactory.java:519)
>   at 
> org.apache.beam.examples.complete.game.UserScore.main(UserScore.java:226)
>   ... 6 more
> Caused by: java.lang.ClassNotFoundException: org.hamcrest.Matcher
>   at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>   ... 35 more
> {code}
> cc: [~kenn][~tgroh][~vikasrk][~dhalp...@google.com]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BEAM-2233) Java 8 examples should separate runners into distinct profiles, like Java 7 examples

2017-05-09 Thread Davor Bonaci (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-2233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci resolved BEAM-2233.

Resolution: Fixed

> Java 8 examples should separate runners into distinct profiles, like Java 7 
> examples
> 
>
> Key: BEAM-2233
> URL: https://issues.apache.org/jira/browse/BEAM-2233
> Project: Beam
>  Issue Type: Bug
>  Components: examples-java
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Flink #2752

2017-05-09 Thread Apache Jenkins Server

See

Jenkins build is unstable: beam_PostCommit_Java_MavenInstall #3740

2017-05-09 Thread Apache Jenkins Server

See

[2/3] beam git commit: Add per-runner profile to Java 8 examples

2017-05-09 Thread davor

Add per-runner profile to Java 8 examples


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/2521c35e
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/2521c35e
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/2521c35e

Branch: refs/heads/release-2.0.0
Commit: 2521c35ee3de17b88f4b3f10a74d38f58b6a4c09
Parents: adb83a8
Author: Kenneth Knowles 
Authored: Tue May 9 13:02:46 2017 -0700
Committer: Davor Bonaci 
Committed: Tue May 9 17:21:32 2017 -0700

--
 examples/java8/pom.xml | 83 +
 1 file changed, 54 insertions(+), 29 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/2521c35e/examples/java8/pom.xml
--
diff --git a/examples/java8/pom.xml b/examples/java8/pom.xml
index c777e19..7e60e41 100644
--- a/examples/java8/pom.xml
+++ b/examples/java8/pom.xml
@@ -41,21 +41,11 @@
 
   
 
 
-  include-runners
+  direct-runner
   
 true
   
@@ -63,41 +53,72 @@
 
   org.apache.beam
   beam-runners-direct-java
-  ${project.version}
   runtime
-  true
 
+  
+
 
+
+
+  apex-runner
+  
 
   org.apache.beam
-  beam-runners-flink_2.10
-  ${project.version}
+  beam-runners-apex
   runtime
-  true
 
+
+
+  org.apache.httpcomponents
+  httpclient
+  4.3.5
+  runtime
+  
+
+  commons-codec
+  commons-codec
+
+  
+
+  
+
 
+
+
+  flink-runner
+  
 
   org.apache.beam
-  beam-runners-google-cloud-dataflow-java
-  ${project.version}
+  beam-runners-flink_2.10
   runtime
-  true
 
+  
+
 
+
+
+  spark-runner
+  
 
   org.apache.beam
   beam-runners-spark
-  ${project.version}
   runtime
-  true
 
-
+
+  org.apache.spark
+  spark-streaming_2.10
+  ${spark.version}
+  runtime
+
 
   org.apache.spark
   spark-core_2.10
   ${spark.version}
   runtime
-  true
   
 
   org.slf4j
@@ -105,13 +126,17 @@
 
   
 
+  
+
 
+
+
+  dataflow-runner
+  
 
-  org.apache.spark
-  spark-streaming_2.10
-  ${spark.version}
+  org.apache.beam
+  beam-runners-google-cloud-dataflow-java
   runtime
-  true

[1/3] beam git commit: Register TestSparkPipelineOptions only in src/test to avoid hard hamcrest dep

2017-05-09 Thread davor

Repository: beam
Updated Branches:
  refs/heads/release-2.0.0 46e3bf989 -> 0a8a2f4b6


Register TestSparkPipelineOptions only in src/test to avoid hard hamcrest dep


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/adb83a81
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/adb83a81
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/adb83a81

Branch: refs/heads/release-2.0.0
Commit: adb83a810b6112e3b6590516f1d40184827bbf3e
Parents: 46e3bf9
Author: Kenneth Knowles 
Authored: Tue May 9 12:44:19 2017 -0700
Committer: Davor Bonaci 
Committed: Tue May 9 17:21:31 2017 -0700

--
 .../runners/spark/SparkRunnerRegistrar.java |  4 +--
 .../beam/runners/spark/TestSparkRunner.java | 29 
 .../runners/spark/SparkRunnerRegistrarTest.java |  2 +-
 .../TestSparkPipelineOptionsRegistrar.java  | 36 
 4 files changed, 52 insertions(+), 19 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/adb83a81/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkRunnerRegistrar.java
--
diff --git 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkRunnerRegistrar.java
 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkRunnerRegistrar.java
index e2e5ceb..325c86d 100644
--- 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkRunnerRegistrar.java
+++ 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkRunnerRegistrar.java
@@ -54,9 +54,7 @@ public final class SparkRunnerRegistrar {
   public static class Options implements PipelineOptionsRegistrar {
 @Override
 public Iterable> getPipelineOptions() {
-  return ImmutableList.>of(
-  SparkPipelineOptions.class,
-  TestSparkPipelineOptions.class);
+  return ImmutableList.>of(SparkPipelineOptions.class);
 }
   }
 }

http://git-wip-us.apache.org/repos/asf/beam/blob/adb83a81/runners/spark/src/main/java/org/apache/beam/runners/spark/TestSparkRunner.java
--
diff --git 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/TestSparkRunner.java
 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/TestSparkRunner.java
index 6d10b75..eccee57 100644
--- 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/TestSparkRunner.java
+++ 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/TestSparkRunner.java
@@ -83,26 +83,25 @@ import org.slf4j.LoggerFactory;
 public final class TestSparkRunner extends PipelineRunner 
{
 
   private static final Logger LOG = 
LoggerFactory.getLogger(TestSparkRunner.class);
-  private final TestSparkPipelineOptions testSparkPipelineOptions;
-
+  private final PipelineOptions options;
   private SparkRunner delegate;
-  private boolean isForceStreaming;
 
-  private TestSparkRunner(TestSparkPipelineOptions options) {
+  private TestSparkRunner(PipelineOptions options) {
 this.delegate = SparkRunner.fromOptions(options);
-this.isForceStreaming = options.isForceStreaming();
-this.testSparkPipelineOptions = options;
+this.options = options;
   }
 
   public static TestSparkRunner fromOptions(PipelineOptions options) {
-// Default options suffice to set it up as a test runner
-TestSparkPipelineOptions sparkOptions =
-PipelineOptionsValidator.validate(TestSparkPipelineOptions.class, 
options);
-return new TestSparkRunner(sparkOptions);
+return new TestSparkRunner(options);
   }
 
   @Override
   public SparkPipelineResult run(Pipeline pipeline) {
+// Default options suffice to set it up as a test runner
+TestSparkPipelineOptions testSparkOptions =
+PipelineOptionsValidator.validate(TestSparkPipelineOptions.class, 
options);
+
+boolean isForceStreaming = testSparkOptions.isForceStreaming();
 // if the pipeline forces execution as a streaming pipeline,
 // and the source is an adapted unbounded source (as bounded),
 // read it as unbounded source via UnboundedReadFromBoundedSource.
@@ -116,13 +115,13 @@ public final class TestSparkRunner extends 
PipelineRunner {
 MetricsAccumulator.clear();
 GlobalWatermarkHolder.clear();
 
-LOG.info("About to run test pipeline " + 
testSparkPipelineOptions.getJobName());
+LOG.info("About to run test pipeline " + options.getJobName());
 
 // if the pipeline was executed in streaming mode, validate aggregators.
 if (isForceStreaming) {
   try {
 result = delegate.run(pipeline);
-awaitWatermarksOrTimeout(testSparkPipelineOptions, result);
+awaitWatermarksOrTimeout(testSparkOptions, result);
 result.stop();
 PipelineResult.State finishState = result.getState();
 // assert

[3/3] beam git commit: This closes #3013

2017-05-09 Thread davor

This closes #3013


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/0a8a2f4b
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/0a8a2f4b
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/0a8a2f4b

Branch: refs/heads/release-2.0.0
Commit: 0a8a2f4b6007be828bff77dc190bca4c4117648e
Parents: 46e3bf9 2521c35
Author: Davor Bonaci 
Authored: Tue May 9 17:21:38 2017 -0700
Committer: Davor Bonaci 
Committed: Tue May 9 17:21:38 2017 -0700

--
 examples/java8/pom.xml  | 83 +---
 .../runners/spark/SparkRunnerRegistrar.java |  4 +-
 .../beam/runners/spark/TestSparkRunner.java | 29 ---
 .../runners/spark/SparkRunnerRegistrarTest.java |  2 +-
 .../TestSparkPipelineOptionsRegistrar.java  | 36 +
 5 files changed, 106 insertions(+), 48 deletions(-)
--

[jira] [Commented] (BEAM-2178) Update hadoop-file-system README.md

2017-05-09 Thread Ahmet Altay (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-2178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003816#comment-16003816
 ] 

Ahmet Altay commented on BEAM-2178:
---

I removed {{sdks/java/io/hadoop/README.md}} with 
https://github.com/apache/beam/pull/3014/files and migrated the content to the 
web site with https://github.com/apache/beam-site/pull/235

[~jbonofre], what else you were planning to do related to this issue?

> Update hadoop-file-system README.md
> ---
>
> Key: BEAM-2178
> URL: https://issues.apache.org/jira/browse/BEAM-2178
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Jean-Baptiste Onofré
>Assignee: Ahmet Altay
> Fix For: 2.0.0
>
>
> The {{README.md}} in {{hadoop-file-system}} is still the one from {{HdfsIO}} 
> which is no more valid.
> With my tests on HDFS, I'm preparing a PR to update this {{README.md}} 
> explaining how to use Hadoop file system in Beam, and the configuration part 
> (using command line options or {{HADOOP_CONF_DIR}}).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #3741

2017-05-09 Thread Apache Jenkins Server

See 


Changes:

[tgroh] Shade JSR305 in the DirectRunner

[ccy] Update Apache Beam Python version to 2.1.0.dev

--
[...truncated 904.87 KB...]
at 
org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:600)
... 27 more

2017-05-10T00:13:57.870 [ERROR] 
testE2EV1ReadWithGQLQueryWithNoLimit(org.apache.beam.sdk.io.gcp.datastore.V1ReadIT)
  Time elapsed: 270.197 s  <<< ERROR!
java.lang.RuntimeException: Failed to create a workflow job: 
(7b39eb478db06daa): The workflow could not be created. Causes: 
(21cc78349bf78dfb): Too many running jobs. Project apache-beam-testing is 
running 25 jobs and project limit for active jobs is 25. To fix this, cancel an 
existing workflow via the UI, wait for a workflow to finish or contact 
dataflow-feedb...@google.com to request an increase in quota.
at 
org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:613)
at 
org.apache.beam.runners.dataflow.TestDataflowRunner.run(TestDataflowRunner.java:100)
at 
org.apache.beam.runners.dataflow.TestDataflowRunner.run(TestDataflowRunner.java:92)
at 
org.apache.beam.runners.dataflow.TestDataflowRunner.run(TestDataflowRunner.java:53)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:295)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:281)
at 
org.apache.beam.sdk.io.gcp.datastore.V1ReadIT.testE2EV1ReadWithGQLQuery(V1ReadIT.java:139)
at 
org.apache.beam.sdk.io.gcp.datastore.V1ReadIT.testE2EV1ReadWithGQLQueryWithNoLimit(V1ReadIT.java:101)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at 
org.apache.maven.surefire.junitcore.pc.Scheduler$1.run(Scheduler.java:393)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 
429 Too Many Requests
{
  "code" : 429,
  "errors" : [ {
"domain" : "global",
"message" : "(7b39eb478db06daa): The workflow could not be created. Causes: 
(21cc78349bf78dfb): Too many running jobs. Project apache-beam-testing is 
running 25 jobs and project limit for active jobs is 25. To fix this, cancel an 
existing workflow via the UI, wait for a workflow to finish or contact 
dataflow-feedb...@google.com to request an increase in quota.",
"reason" : "rateLimitExceeded"
  } ],
  "message" : "(7b39eb478db06daa): The workflow could not be created. Causes: 
(21cc78349bf78dfb): Too many running jobs. Project apache-beam-testing is 
running 25 jobs and project limit for active jobs is 25. To fix this, cancel an 
existing workflow via the UI, wait for a workflow to finish or contact 
dataflow-feedb...@google.com to request an increase in quota.",
  "status" : "RESOURCE_EXHAUSTED"
}
at 
com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
at 
com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
at 
com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
at 
com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:321)
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1065)

[1/2] beam git commit: Remove trailing whitespace

2017-05-09 Thread altay

Repository: beam
Updated Branches:
  refs/heads/master b9ea415b6 -> c567550de


Remove trailing whitespace


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/dae9fc71
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/dae9fc71
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/dae9fc71

Branch: refs/heads/master
Commit: dae9fc71e526a26966bd9296fa1f64e84361037e
Parents: b9ea415
Author: Sourabh Bajaj 
Authored: Tue May 9 17:02:57 2017 -0700
Committer: Ahmet Altay 
Committed: Tue May 9 17:12:57 2017 -0700

--
 README.md | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/dae9fc71/README.md
--
diff --git a/README.md b/README.md
index 1202c54..52c056f 100644
--- a/README.md
+++ b/README.md
@@ -71,8 +71,8 @@ Have ideas for new Runners? See the 
[JIRA](https://issues.apache.org/jira/browse
 
 Please refer to the 
[Quickstart](http://beam.apache.org/get-started/quickstart/) available on our 
website.
 
-If you'd like to build and install the whole project from the source 
distribution, you may need some additional tools installed 
-in your system. In a Debian-based distribution: 
+If you'd like to build and install the whole project from the source 
distribution, you may need some additional tools installed
+in your system. In a Debian-based distribution:
 
 ```
 sudo apt-get install \

[GitHub] beam pull request #3025: Remove trailing whitespace

2017-05-09 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3025


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #3025

2017-05-09 Thread altay

This closes #3025


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/c567550d
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/c567550d
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/c567550d

Branch: refs/heads/master
Commit: c567550de72eaea257c6eecb84bf1fcde981c4d2
Parents: b9ea415 dae9fc7
Author: Ahmet Altay 
Authored: Tue May 9 17:13:01 2017 -0700
Committer: Ahmet Altay 
Committed: Tue May 9 17:13:01 2017 -0700

--
 README.md | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--

[jira] [Created] (BEAM-2243) org.apache.beam.GcpCoreApiSurfaceTest.testApiSurface fails at release-2.0.0 head

2017-05-09 Thread Luke Cwik (JIRA)

Luke Cwik created BEAM-2243:
---

 Summary: org.apache.beam.GcpCoreApiSurfaceTest.testApiSurface 
fails at release-2.0.0 head
 Key: BEAM-2243
 URL: https://issues.apache.org/jira/browse/BEAM-2243
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-gcp
Reporter: Luke Cwik
Assignee: Luke Cwik
Priority: Blocker
 Fix For: 2.0.0


{code}
java.lang.AssertionError: 

Expected: API surface to include only:
Classes in package "org.apache.beam"
Classes in package "com.google.api.client"
Classes in package "com.google.api.services.storage"
Classes in package "com.google.auth"
Classes in package "com.fasterxml.jackson.annotation"
Classes in package "com.fasterxml.jackson.core"
Classes in package "com.fasterxml.jackson.databind"
Classes in package "org.apache.avro"
Classes in package "org.hamcrest"
Classes in package "org.codehaus.jackson"
Classes in package "org.joda.time"
Classes in package "org.junit"

 but: The following disallowed classes appeared on the API surface:
interface sun.reflect.CallerSensitive exposed via:
interface sun.reflect.CallerSensitive
class 
org.apache.beam.sdk.repackaged.net.bytebuddy.dynamic.loading.MultipleParentClassLoader
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:8)
at 
org.apache.beam.GcpCoreApiSurfaceTest.testApiSurface(GcpCoreApiSurfaceTest.java:55)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-2242) Apache Beam Java modules do not correctly shade test artifacts

2017-05-09 Thread Luke Cwik (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-2242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik updated BEAM-2242:

Description: 
javap dump of TextIOTest.class

Not the unshaded references to org.apache.commons.compress.*

{code}
Compiled from "TextIOTest.java"
public class org.apache.beam.sdk.io.TextIOTest {
...
  private static java.io.File writeToFile(java.lang.String[], java.lang.String, 
org.apache.beam.sdk.io.TextIO$CompressionType) throws java.io.IOException;
descriptor: 
([Ljava/lang/String;Ljava/lang/String;Lorg/apache/beam/sdk/io/TextIO$CompressionType;)Ljava/io/File;
Code:
   0: getstatic #6  // Field 
tempFolder:Ljava/nio/file/Path;
   3: aload_1
   4: invokeinterface #7,  2// InterfaceMethod 
java/nio/file/Path.resolve:(Ljava/lang/String;)Ljava/nio/file/Path;
   9: invokeinterface #8,  1// InterfaceMethod 
java/nio/file/Path.toFile:()Ljava/io/File;
  14: astore_3
  15: new   #9  // class java/io/FileOutputStream
  18: dup
  19: aload_3
  20: invokespecial #10 // Method 
java/io/FileOutputStream."":(Ljava/io/File;)V
  23: astore4
  25: getstatic #11 // Field 
org/apache/beam/sdk/io/TextIOTest$4.$SwitchMap$org$apache$beam$sdk$io$TextIO$CompressionType:[I
  28: aload_2
  29: invokevirtual #12 // Method 
org/apache/beam/sdk/io/TextIO$CompressionType.ordinal:()I
  32: iaload
  33: tableswitch   { // 1 to 5
 1: 68
 2: 71
 3: 85
 4: 99
 5: 131
   default: 145
  }
  68: goto  157
  71: new   #13 // class 
java/util/zip/GZIPOutputStream
  74: dup
  75: aload 4
  77: invokespecial #14 // Method 
java/util/zip/GZIPOutputStream."":(Ljava/io/OutputStream;)V
  80: astore4
  82: goto  157
  85: new   #15 // class 
org/apache/commons/compress/compressors/bzip2/BZip2CompressorOutputStream
  88: dup
  89: aload 4
  91: invokespecial #16 // Method 
org/apache/commons/compress/compressors/bzip2/BZip2CompressorOutputStream."":(Ljava/io/OutputStream;)V
  94: astore4
  96: goto  157
  99: new   #17 // class 
java/util/zip/ZipOutputStream
 102: dup
 103: aload 4
 105: invokespecial #18 // Method 
java/util/zip/ZipOutputStream."":(Ljava/io/OutputStream;)V
 108: astore5
 110: aload 5
 112: new   #19 // class java/util/zip/ZipEntry
 115: dup
 116: ldc   #20 // String entry
 118: invokespecial #21 // Method 
java/util/zip/ZipEntry."":(Ljava/lang/String;)V
 121: invokevirtual #22 // Method 
java/util/zip/ZipOutputStream.putNextEntry:(Ljava/util/zip/ZipEntry;)V
 124: aload 5
 126: astore4
 128: goto  157
 131: new   #23 // class 
org/apache/commons/compress/compressors/deflate/DeflateCompressorOutputStream
{code}

  was:
javap dump of TextIOTest.class

Not the unshaded references to org.apache.commons.compress.*

```
Compiled from "TextIOTest.java"
public class org.apache.beam.sdk.io.TextIOTest {
...
  private static java.io.File writeToFile(java.lang.String[], java.lang.String, 
org.apache.beam.sdk.io.TextIO$CompressionType) throws java.io.IOException;
descriptor: 
([Ljava/lang/String;Ljava/lang/String;Lorg/apache/beam/sdk/io/TextIO$CompressionType;)Ljava/io/File;
Code:
   0: getstatic #6  // Field 
tempFolder:Ljava/nio/file/Path;
   3: aload_1
   4: invokeinterface #7,  2// InterfaceMethod 
java/nio/file/Path.resolve:(Ljava/lang/String;)Ljava/nio/file/Path;
   9: invokeinterface #8,  1// InterfaceMethod 
java/nio/file/Path.toFile:()Ljava/io/File;
  14: astore_3
  15: new   #9  // class java/io/FileOutputStream
  18: dup
  19: aload_3
  20: invokespecial #10 // Method 
java/io/FileOutputStream."":(Ljava/io/File;)V
  23: astore4
  25: getstatic #11 // Field 
org/apache/beam/sdk/io/TextIOTest$4.$SwitchMap$org$apache$beam$sdk$io$TextIO$CompressionType:[I
  28: aload_2
  29: invokevirtual #12 // Method 
org/apache/beam/sdk/io/TextIO$CompressionType.ordinal:()I
  32: iaload
  33: tableswitch   { // 1 to 5
 1: 68
 2: 71
 3: 85
 4: 99
 5: 131
   default: 145
  }
  68: goto  157
  71: new

Build failed in Jenkins: beam_PostCommit_Python_Verify #2173

2017-05-09 Thread Apache Jenkins Server

See 


Changes:

[klk] Revise javadoc for sdk, state, options, annotations, window, (and misc)

[altay] Remove templates from wordcount example

[altay] Remove hadoop io readme

[tgroh] Shade JSR305 in the DirectRunner

--
[...truncated 579.48 KB...]
  File "/usr/lib/python2.7/pickle.py", line 600, in save_list
self._batch_appends(iter(obj))
  File "/usr/lib/python2.7/pickle.py", line 633, in _batch_appends
save(x)
  File "/usr/lib/python2.7/pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python2.7/pickle.py", line 419, in save_reduce
save(state)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File 
"
 line 163, in new_save_module_dict
return old_save_module_dict(pickler, obj)
  File 
"
 line 841, in save_module_dict
StockPickler.save_dict(pickler, obj)
  File "/usr/lib/python2.7/pickle.py", line 649, in save_dict
self._batch_setitems(obj.iteritems())
  File "/usr/lib/python2.7/pickle.py", line 681, in _batch_setitems
save(v)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 600, in save_list
self._batch_appends(iter(obj))
  File "/usr/lib/python2.7/pickle.py", line 633, in _batch_appends
save(x)
  File "/usr/lib/python2.7/pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python2.7/pickle.py", line 419, in save_reduce
save(state)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File 
"
 line 163, in new_save_module_dict
return old_save_module_dict(pickler, obj)
  File 
"
 line 841, in save_module_dict
StockPickler.save_dict(pickler, obj)
  File "/usr/lib/python2.7/pickle.py", line 649, in save_dict
self._batch_setitems(obj.iteritems())
  File "/usr/lib/python2.7/pickle.py", line 681, in _batch_setitems
save(v)
  File "/usr/lib/python2.7/pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python2.7/pickle.py", line 419, in save_reduce
save(state)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File 
"
 line 163, in new_save_module_dict
return old_save_module_dict(pickler, obj)
  File 
"
 line 841, in save_module_dict
StockPickler.save_dict(pickler, obj)
  File "/usr/lib/python2.7/pickle.py", line 649, in save_dict
self._batch_setitems(obj.iteritems())
  File "/usr/lib/python2.7/pickle.py", line 681, in _batch_setitems
save(v)
  File "/usr/lib/python2.7/pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python2.7/pickle.py", line 419, in save_reduce
save(state)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 548, in save_tuple
save(element)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File 
"
 line 163, in new_save_module_dict
return old_save_module_dict(pickler, obj)
  File 
"
 line 841, in save_module_dict
StockPickler.save_dict(pickler, obj)
  File "/usr/lib/python2.7/pickle.py", line 649, in save_dict
self._batch_setitems(obj.iteritems())
  File "/usr/lib/python2.7/pickle.py", line 672, in _batch_setitems
tmp.append(items.next())
  File 
"
 line 276, in signalhandler
raise TimedOutException()
TimedOutException: 'test_iterable_side_input 
(apache_beam.transforms.sideinputs_test.SideInputsTest)'

==
ERROR: test_multi_valued_singleton_s

[GitHub] beam pull request #3002: [BEAM-2233,BEAM-2223] Adjust TestSparkPipelineOptio...

2017-05-09 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3002


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

1 2 3 4 5 >

1 - 100 of 461 matches

Mail list logo