[jira] [Commented] (BEAM-2378) extend join-library to support FULL OUTER JOIN

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030517#comment-16030517
 ] 

ASF GitHub Bot commented on BEAM-2378:
--

GitHub user xumingming opened a pull request:

https://github.com/apache/beam/pull/3267

[BEAM-2378] support FULL OUTER JOIN

@davorbonaci can you take a look at this please?

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xumingming/beam BEAM-2378-full-outer-join

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3267.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3267


commit 86555d68b86f7b62cea8523e1a97aa99b677d7eb
Author: James Xu 
Date:   2017-05-31T02:28:55Z

[BEAM-2378] support FULL OUTER JOIN




> extend join-library to support FULL OUTER JOIN
> --
>
> Key: BEAM-2378
> URL: https://issues.apache.org/jira/browse/BEAM-2378
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: James Xu
>Assignee: James Xu
>
> Currently INNER, LEFT OUTER, RIGHT OUTER are supported, would be nice to 
> support FULL OUTER JOIN.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3267: [BEAM-2378] support FULL OUTER JOIN

[xumingming]

GitHub user xumingming opened a pull request:

https://github.com/apache/beam/pull/3267

[BEAM-2378] support FULL OUTER JOIN

@davorbonaci can you take a look at this please?

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xumingming/beam BEAM-2378-full-outer-join

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3267.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3267


commit 86555d68b86f7b62cea8523e1a97aa99b677d7eb
Author: James Xu 
Date:   2017-05-31T02:28:55Z

[BEAM-2378] support FULL OUTER JOIN




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2390) allow user to use .setTimePartitioning in BigQueryIO.write

[Eric Johston (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030499#comment-16030499
 ] 

Eric Johston commented on BEAM-2390:


I've created a pull request with the functionality I believe is needed:

https://github.com/apache/beam/pull/3266



> allow user to use .setTimePartitioning in BigQueryIO.write
> --
>
> Key: BEAM-2390
> URL: https://issues.apache.org/jira/browse/BEAM-2390
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model-runner-api
>Affects Versions: 2.0.0
>Reporter: Eric Johston
>Assignee: Kenneth Knowles
>  Labels: easyfix, features, newbie
> Fix For: 2.0.0
>
>   Original Estimate: 2h
>  Remaining Estimate: 2h
>
> Currently when writing to a table with BigQueryIO sink, there is no way to 
> create a new table that is date partitioned. This would be very useful, since 
> currently the only way to do this is  by manually creating a table ahead of 
> time. We should be able to leverage the automatic table creation 
> functionality for date partitioned tables.
> The best way to do this would be to have a withTimePartitioning method in the 
> BigQueryIO class.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2390) allow user to use .setTimePartitioning in BigQueryIO.write

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030497#comment-16030497
 ] 

ASF GitHub Bot commented on BEAM-2390:
--

GitHub user ericjohnston1989 opened a pull request:

https://github.com/apache/beam/pull/3266

[BEAM-2390] added time partitioning to BigQueryIO

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ericjohnston1989/beam BigQueryTimePartitioning

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3266.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3266


commit fa376ef6342b8bcfe80da580fc75150aac891919
Author: Eric Johnston 
Date:   2017-05-31T02:01:59Z

added time partitioning to BigQueryIO




> allow user to use .setTimePartitioning in BigQueryIO.write
> --
>
> Key: BEAM-2390
> URL: https://issues.apache.org/jira/browse/BEAM-2390
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model-runner-api
>Affects Versions: 2.0.0
>Reporter: Eric Johston
>Assignee: Kenneth Knowles
>  Labels: easyfix, features, newbie
> Fix For: 2.0.0
>
>   Original Estimate: 2h
>  Remaining Estimate: 2h
>
> Currently when writing to a table with BigQueryIO sink, there is no way to 
> create a new table that is date partitioned. This would be very useful, since 
> currently the only way to do this is  by manually creating a table ahead of 
> time. We should be able to leverage the automatic table creation 
> functionality for date partitioned tables.
> The best way to do this would be to have a withTimePartitioning method in the 
> BigQueryIO class.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3266: [BEAM-2390] added time partitioning to BigQueryIO

[ericjohnston1989]

GitHub user ericjohnston1989 opened a pull request:

https://github.com/apache/beam/pull/3266

[BEAM-2390] added time partitioning to BigQueryIO

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ericjohnston1989/beam BigQueryTimePartitioning

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3266.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3266


commit fa376ef6342b8bcfe80da580fc75150aac891919
Author: Eric Johnston 
Date:   2017-05-31T02:01:59Z

added time partitioning to BigQueryIO




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (BEAM-1962) Connection should be closed in case start() throws exception

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated BEAM-1962:
-
Description: 
In JmsIO#start() :

{code}
  try {
Connection connection;
if (spec.getUsername() != null) {
  connection =
  connectionFactory.createConnection(spec.getUsername(), 
spec.getPassword());
} else {
  connection = connectionFactory.createConnection();
}
connection.start();
this.connection = connection;
  } catch (Exception e) {
throw new IOException("Error connecting to JMS", e);
  }
{code}
If start() throws exception, connection should be closed.

  was:
In JmsIO#start() :
{code}
  try {
Connection connection;
if (spec.getUsername() != null) {
  connection =
  connectionFactory.createConnection(spec.getUsername(), 
spec.getPassword());
} else {
  connection = connectionFactory.createConnection();
}
connection.start();
this.connection = connection;
  } catch (Exception e) {
throw new IOException("Error connecting to JMS", e);
  }
{code}
If start() throws exception, connection should be closed.


> Connection should be closed in case start() throws exception
> 
>
> Key: BEAM-1962
> URL: https://issues.apache.org/jira/browse/BEAM-1962
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Ted Yu
>Assignee: Jean-Baptiste Onofré
>Priority: Minor
>
> In JmsIO#start() :
> {code}
>   try {
> Connection connection;
> if (spec.getUsername() != null) {
>   connection =
>   connectionFactory.createConnection(spec.getUsername(), 
> spec.getPassword());
> } else {
>   connection = connectionFactory.createConnection();
> }
> connection.start();
> this.connection = connection;
>   } catch (Exception e) {
> throw new IOException("Error connecting to JMS", e);
>   }
> {code}
> If start() throws exception, connection should be closed.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Closed] (BEAM-2248) KafkaIO support to use start read time to set start offset

[Jingsong Lee (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jingsong Lee closed BEAM-2248.
--
   Resolution: Fixed
Fix Version/s: 2.1.0

> KafkaIO support to use start read time to set start offset
> --
>
> Key: BEAM-2248
> URL: https://issues.apache.org/jira/browse/BEAM-2248
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: Jingsong Lee
>Assignee: Jingsong Lee
> Fix For: 2.1.0
>
>
> This Kafka 0.10.x adds support for a searchable index for each topic based 
> off of message timestamps. It enables consumer support for offset lookup by 
> timestamp.
> So we can add a start read time to set start offset.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #3968

[Apache Jenkins Server]

See 


Changes:

[lcwik] [BEAM-1544] Java cross-JDK version tests on Jenkins

--
[...truncated 3.07 MB...]
2017-05-31T01:47:00.862 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/sling/org.apache.sling.commons.json/2.0.6/org.apache.sling.commons.json-2.0.6.jar
 (47 KB at 55.5 KB/sec)
2017-05-31T01:47:00.863 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-runtime_2.10/1.2.1/flink-runtime_2.10-1.2.1-tests.jar
2017-05-31T01:47:00.960 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-core/1.2.1/flink-core-1.2.1-tests.jar
 (716 KB at 761.5 KB/sec)
2017-05-31T01:47:00.960 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-streaming-java_2.10/1.2.1/flink-streaming-java_2.10-1.2.1-tests.jar
2017-05-31T01:47:01.047 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-streaming-java_2.10/1.2.1/flink-streaming-java_2.10-1.2.1.jar
 (3035 KB at 2954.4 KB/sec)
2017-05-31T01:47:01.047 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils_2.10/1.2.1/flink-test-utils_2.10-1.2.1.jar
2017-05-31T01:47:01.048 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/2.10.4/scala-library-2.10.4.jar
 (6960 KB at 6769.8 KB/sec)
2017-05-31T01:47:01.048 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-junit/1.2.1/flink-test-utils-junit-1.2.1.jar
2017-05-31T01:47:01.078 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-junit/1.2.1/flink-test-utils-junit-1.2.1.jar
 (24 KB at 21.7 KB/sec)
2017-05-31T01:47:01.078 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-test/2.8.0/curator-test-2.8.0.jar
2017-05-31T01:47:01.097 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-streaming-java_2.10/1.2.1/flink-streaming-java_2.10-1.2.1-tests.jar
 (953 KB at 884.7 KB/sec)
2017-05-31T01:47:01.106 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-test/2.8.0/curator-test-2.8.0.jar
 (39 KB at 35.9 KB/sec)
2017-05-31T01:47:01.130 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-runtime_2.10/1.2.1/flink-runtime_2.10-1.2.1-tests.jar
 (2432 KB at 2190.7 KB/sec)
2017-05-31T01:47:01.248 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils_2.10/1.2.1/flink-test-utils_2.10-1.2.1.jar
 (2366 KB at 1926.5 KB/sec)
2017-05-31T01:47:01.599 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-shaded-hadoop2/1.2.1/flink-shaded-hadoop2-1.2.1.jar
 (17860 KB at 11274.9 KB/sec)
2017-05-31T01:47:01.612 [INFO] 
2017-05-31T01:47:01.612 [INFO] --- maven-clean-plugin:3.0.0:clean 
(default-clean) @ beam-runners-flink_2.10 ---
2017-05-31T01:47:01.614 [INFO] Deleting 

 (includes = [**/*.pyc, **/*.egg-info/, **/sdks/python/LICENSE, 
**/sdks/python/NOTICE, **/sdks/python/README.md], excludes = [])
2017-05-31T01:47:01.769 [INFO] 
2017-05-31T01:47:01.769 [INFO] --- maven-enforcer-plugin:1.4.1:enforce 
(enforce) @ beam-runners-flink_2.10 ---
2017-05-31T01:47:04.053 [INFO] 
2017-05-31T01:47:04.053 [INFO] --- maven-enforcer-plugin:1.4.1:enforce 
(enforce-banned-dependencies) @ beam-runners-flink_2.10 ---
2017-05-31T01:47:04.129 [INFO] 
2017-05-31T01:47:04.129 [INFO] --- maven-remote-resources-plugin:1.5:process 
(process-resource-bundles) @ beam-runners-flink_2.10 ---
2017-05-31T01:47:04.778 [INFO] 
2017-05-31T01:47:04.779 [INFO] --- maven-resources-plugin:3.0.2:resources 
(default-resources) @ beam-runners-flink_2.10 ---
2017-05-31T01:47:04.780 [INFO] Using 'UTF-8' encoding to copy filtered 
resources.
2017-05-31T01:47:04.780 [INFO] Copying 1 resource
2017-05-31T01:47:04.780 [INFO] Copying 3 resources
2017-05-31T01:47:04.870 [INFO] 
2017-05-31T01:47:04.870 [INFO] --- maven-compiler-plugin:3.6.1:compile 
(default-compile) @ beam-runners-flink_2.10 ---
2017-05-31T01:47:04.886 [INFO] Changes detected - recompiling the module!
2017-05-31T01:47:04.886 [INFO] Compiling 75 source files to 

2017-05-31T01:47:05.617 [WARNING] bootstrap class path not set in conjunction 
with -source 1.7
2017-05-31T01:47:05.617 [INFO] 
:
 Some input files use or override a deprecated API.
2017-05-31T01:47:05.617 [INFO] 

[GitHub] beam pull request #3265: Property names cleanup

[jkff]

GitHub user jkff opened a pull request:

https://github.com/apache/beam/pull/3265

Property names cleanup

This also updates Dataflow worker to a version that doesn't depend on 
PropertyNames which are not used in Dataflow runner itself. So, from now on, if 
a PropertyNames entry is unused, it can be deleted (but renaming is still not 
safe).

R: @kennknowles 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jkff/incubator-beam property-names-cleanup

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3265.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3265


commit a073f36a26ef6f4fed1718c3143351bdb47b4aeb
Author: Eugene Kirpichov 
Date:   2017-05-26T23:11:20Z

Removes unused PropertyNames

commit d2693ec252317d168a6b5338c79617f4b6e6df6d
Author: Eugene Kirpichov 
Date:   2017-05-31T01:04:36Z

Bumps dataflow container version to 0530




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #3966

[Apache Jenkins Server]

See 


Changes:

[altay] Clean up pyc files before running tests

--
[...truncated 3.07 MB...]
2017-05-31T01:02:26.354 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/sling/org.apache.sling.commons.json/2.0.6/org.apache.sling.commons.json-2.0.6.jar
 (47 KB at 34.5 KB/sec)
2017-05-31T01:02:26.354 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-runtime_2.10/1.2.1/flink-runtime_2.10-1.2.1-tests.jar
2017-05-31T01:02:26.458 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-core/1.2.1/flink-core-1.2.1-tests.jar
 (716 KB at 490.3 KB/sec)
2017-05-31T01:02:26.458 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-streaming-java_2.10/1.2.1/flink-streaming-java_2.10-1.2.1-tests.jar
2017-05-31T01:02:26.641 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-streaming-java_2.10/1.2.1/flink-streaming-java_2.10-1.2.1-tests.jar
 (953 KB at 580.0 KB/sec)
2017-05-31T01:02:26.641 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils_2.10/1.2.1/flink-test-utils_2.10-1.2.1.jar
2017-05-31T01:02:26.714 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/2.10.4/scala-library-2.10.4.jar
 (6960 KB at 4055.6 KB/sec)
2017-05-31T01:02:26.714 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-junit/1.2.1/flink-test-utils-junit-1.2.1.jar
2017-05-31T01:02:26.744 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-junit/1.2.1/flink-test-utils-junit-1.2.1.jar
 (24 KB at 13.2 KB/sec)
2017-05-31T01:02:26.744 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-test/2.8.0/curator-test-2.8.0.jar
2017-05-31T01:02:26.747 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-streaming-java_2.10/1.2.1/flink-streaming-java_2.10-1.2.1.jar
 (3035 KB at 1734.8 KB/sec)
2017-05-31T01:02:26.777 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-test/2.8.0/curator-test-2.8.0.jar
 (39 KB at 21.9 KB/sec)
2017-05-31T01:02:26.832 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-runtime_2.10/1.2.1/flink-runtime_2.10-1.2.1-tests.jar
 (2432 KB at 1325.9 KB/sec)
2017-05-31T01:02:26.894 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils_2.10/1.2.1/flink-test-utils_2.10-1.2.1.jar
 (2366 KB at 1247.7 KB/sec)
2017-05-31T01:02:27.302 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-shaded-hadoop2/1.2.1/flink-shaded-hadoop2-1.2.1.jar
 (17860 KB at 7751.5 KB/sec)
2017-05-31T01:02:27.318 [INFO] 
2017-05-31T01:02:27.318 [INFO] --- maven-clean-plugin:3.0.0:clean 
(default-clean) @ beam-runners-flink_2.10 ---
2017-05-31T01:02:27.319 [INFO] Deleting 

 (includes = [**/*.pyc, **/*.egg-info/, **/sdks/python/LICENSE, 
**/sdks/python/NOTICE, **/sdks/python/README.md], excludes = [])
2017-05-31T01:02:27.372 [INFO] 
2017-05-31T01:02:27.372 [INFO] --- maven-enforcer-plugin:1.4.1:enforce 
(enforce) @ beam-runners-flink_2.10 ---
2017-05-31T01:02:29.653 [INFO] 
2017-05-31T01:02:29.653 [INFO] --- maven-enforcer-plugin:1.4.1:enforce 
(enforce-banned-dependencies) @ beam-runners-flink_2.10 ---
2017-05-31T01:02:29.720 [INFO] 
2017-05-31T01:02:29.720 [INFO] --- maven-remote-resources-plugin:1.5:process 
(process-resource-bundles) @ beam-runners-flink_2.10 ---
2017-05-31T01:02:30.295 [INFO] 
2017-05-31T01:02:30.295 [INFO] --- maven-resources-plugin:3.0.2:resources 
(default-resources) @ beam-runners-flink_2.10 ---
2017-05-31T01:02:30.296 [INFO] Using 'UTF-8' encoding to copy filtered 
resources.
2017-05-31T01:02:30.296 [INFO] Copying 1 resource
2017-05-31T01:02:30.297 [INFO] Copying 3 resources
2017-05-31T01:02:30.413 [INFO] 
2017-05-31T01:02:30.413 [INFO] --- maven-compiler-plugin:3.6.1:compile 
(default-compile) @ beam-runners-flink_2.10 ---
2017-05-31T01:02:30.433 [INFO] Changes detected - recompiling the module!
2017-05-31T01:02:30.433 [INFO] Compiling 75 source files to 

2017-05-31T01:02:31.364 [WARNING] bootstrap class path not set in conjunction 
with -source 1.7
2017-05-31T01:02:31.364 [INFO] 
:
 Some input files use or override a deprecated API.
2017-05-31T01:02:31.364 [INFO] 

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Flink #2980

[Apache Jenkins Server]

See 

[jira] [Assigned] (BEAM-2392) Avoid use of proto builder clone

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik reassigned BEAM-2392:
---

Assignee: Nigel Kilmer  (was: Daniel Halperin)

> Avoid use of proto builder clone
> 
>
> Key: BEAM-2392
> URL: https://issues.apache.org/jira/browse/BEAM-2392
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-gcp
>Affects Versions: 2.1.0
>Reporter: Nigel Kilmer
>Assignee: Nigel Kilmer
>Priority: Minor
>
> BigtableServiceImpl uses the clone method of the MutateRowResponse proto 
> builder here:
> https://github.com/apache/beam/blob/04e3261818aed0c129e7c715e371463bf5b5c1b8/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableServiceImpl.java#L212
> This method is not generated by the Google-internal Java proto generator, so 
> I had to change this to get it to work with an internal project. Are you 
> interested in adding this change to the main repository for compatibility, or 
> would you prefer to keep the cleaner version that uses clone?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2392) Avoid use of proto builder clone

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030453#comment-16030453
 ] 

Luke Cwik commented on BEAM-2392:
-

How did clone not work for you?

If it seems like a bug, we would gladly take a fix.

> Avoid use of proto builder clone
> 
>
> Key: BEAM-2392
> URL: https://issues.apache.org/jira/browse/BEAM-2392
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-gcp
>Affects Versions: 2.1.0
>Reporter: Nigel Kilmer
>Assignee: Daniel Halperin
>Priority: Minor
>
> BigtableServiceImpl uses the clone method of the MutateRowResponse proto 
> builder here:
> https://github.com/apache/beam/blob/04e3261818aed0c129e7c715e371463bf5b5c1b8/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableServiceImpl.java#L212
> This method is not generated by the Google-internal Java proto generator, so 
> I had to change this to get it to work with an internal project. Are you 
> interested in adding this change to the main repository for compatibility, or 
> would you prefer to keep the cleaner version that uses clone?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3264: Fetch only relevant PR

[jasonkuster]

GitHub user jasonkuster opened a pull request:

https://github.com/apache/beam/pull/3264

Fetch only relevant PR

Per https://github.com/jenkinsci/ghprb-plugin/pull/521 we can be much more 
efficient about how we fetch PRs initially.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jasonkuster/beam patch-3

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3264.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3264


commit 08122080365df8f48402ce3f9c921dfd3559c520
Author: jasonkuster 
Date:   2017-05-31T00:59:09Z

Fetch only relevant PR

Per https://github.com/jenkinsci/ghprb-plugin/pull/521 we can be much more 
efficient about how we fetch PRs initially.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #3967

[Apache Jenkins Server]

See 


--
[...truncated 3.06 MB...]
2017-05-31T00:55:54.857 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/twitter/chill_2.10/0.7.4/chill_2.10-0.7.4.jar
 (217 KB at 139.2 KB/sec)
2017-05-31T00:55:54.857 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-runtime_2.10/1.2.1/flink-runtime_2.10-1.2.1-tests.jar
2017-05-31T00:55:55.111 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-core/1.2.1/flink-core-1.2.1-tests.jar
 (716 KB at 396.4 KB/sec)
2017-05-31T00:55:55.111 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-streaming-java_2.10/1.2.1/flink-streaming-java_2.10-1.2.1-tests.jar
2017-05-31T00:55:55.231 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/2.10.4/scala-library-2.10.4.jar
 (6960 KB at 3611.5 KB/sec)
2017-05-31T00:55:55.231 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils_2.10/1.2.1/flink-test-utils_2.10-1.2.1.jar
2017-05-31T00:55:55.411 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-streaming-java_2.10/1.2.1/flink-streaming-java_2.10-1.2.1-tests.jar
 (953 KB at 452.5 KB/sec)
2017-05-31T00:55:55.411 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-junit/1.2.1/flink-test-utils-junit-1.2.1.jar
2017-05-31T00:55:55.418 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-streaming-java_2.10/1.2.1/flink-streaming-java_2.10-1.2.1.jar
 (3035 KB at 1436.0 KB/sec)
2017-05-31T00:55:55.418 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-test/2.8.0/curator-test-2.8.0.jar
2017-05-31T00:55:55.423 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-runtime_2.10/1.2.1/flink-runtime_2.10-1.2.1-tests.jar
 (2432 KB at 1148.1 KB/sec)
2017-05-31T00:55:55.455 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-test/2.8.0/curator-test-2.8.0.jar
 (39 KB at 18.1 KB/sec)
2017-05-31T00:55:55.468 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-junit/1.2.1/flink-test-utils-junit-1.2.1.jar
 (24 KB at 10.6 KB/sec)
2017-05-31T00:55:55.682 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils_2.10/1.2.1/flink-test-utils_2.10-1.2.1.jar
 (2366 KB at 995.2 KB/sec)
2017-05-31T00:55:55.992 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-shaded-hadoop2/1.2.1/flink-shaded-hadoop2-1.2.1.jar
 (17860 KB at 6644.1 KB/sec)
2017-05-31T00:55:56.007 [INFO] 
2017-05-31T00:55:56.007 [INFO] --- maven-clean-plugin:3.0.0:clean 
(default-clean) @ beam-runners-flink_2.10 ---
2017-05-31T00:55:56.009 [INFO] Deleting 

 (includes = [**/*.pyc, **/*.egg-info/, **/sdks/python/LICENSE, 
**/sdks/python/NOTICE, **/sdks/python/README.md], excludes = [])
2017-05-31T00:55:56.099 [INFO] 
2017-05-31T00:55:56.099 [INFO] --- maven-enforcer-plugin:1.4.1:enforce 
(enforce) @ beam-runners-flink_2.10 ---
2017-05-31T00:55:58.307 [INFO] 
2017-05-31T00:55:58.307 [INFO] --- maven-enforcer-plugin:1.4.1:enforce 
(enforce-banned-dependencies) @ beam-runners-flink_2.10 ---
2017-05-31T00:55:58.376 [INFO] 
2017-05-31T00:55:58.376 [INFO] --- maven-remote-resources-plugin:1.5:process 
(process-resource-bundles) @ beam-runners-flink_2.10 ---
2017-05-31T00:55:58.927 [INFO] 
2017-05-31T00:55:58.927 [INFO] --- maven-resources-plugin:3.0.2:resources 
(default-resources) @ beam-runners-flink_2.10 ---
2017-05-31T00:55:58.928 [INFO] Using 'UTF-8' encoding to copy filtered 
resources.
2017-05-31T00:55:58.929 [INFO] Copying 1 resource
2017-05-31T00:55:58.930 [INFO] Copying 3 resources
2017-05-31T00:55:59.045 [INFO] 
2017-05-31T00:55:59.045 [INFO] --- maven-compiler-plugin:3.6.1:compile 
(default-compile) @ beam-runners-flink_2.10 ---
2017-05-31T00:55:59.063 [INFO] Changes detected - recompiling the module!
2017-05-31T00:55:59.064 [INFO] Compiling 75 source files to 

2017-05-31T00:56:01.873 [WARNING] bootstrap class path not set in conjunction 
with -source 1.7
2017-05-31T00:56:01.874 [INFO] 
:
 Some input files use or override a deprecated API.
2017-05-31T00:56:01.874 [INFO] 
:
 Recompile with -Xlint:deprecation for details.
2017-05-31T00:56:01.874 [INFO] 

[jira] [Commented] (BEAM-1544) Move Java cross-JDK tests from Travis to Jenkins

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030439#comment-16030439
 ] 

ASF GitHub Bot commented on BEAM-1544:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3184


> Move Java cross-JDK tests from Travis to Jenkins
> 
>
> Key: BEAM-1544
> URL: https://issues.apache.org/jira/browse/BEAM-1544
> Project: Beam
>  Issue Type: Task
>  Components: build-system, testing
>Reporter: Mark Liu
>Assignee: Mark Liu
>
> JDK versions we test in Travis:
> JDK 1.8 (latest)
> JDK 1.7 (latest)
> OpenJDK 7 (on Ubuntu only)
> OpenJDK 8 (on Ubuntu only)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3184: [BEAM-1544] Java cross-JDK version tests on Jenkins

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3184


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: [BEAM-1544] Java cross-JDK version tests on Jenkins

[lcwik]

Repository: beam
Updated Branches:
  refs/heads/master f9431a390 -> 2fa24d89c


[BEAM-1544] Java cross-JDK version tests on Jenkins


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/2f9a38e5
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/2f9a38e5
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/2f9a38e5

Branch: refs/heads/master
Commit: 2f9a38e5fa7afa9cd4751980e6cbb989e260aaa2
Parents: f9431a3
Author: Mark Liu 
Authored: Fri May 19 11:24:15 2017 -0700
Committer: Luke Cwik 
Committed: Tue May 30 17:48:53 2017 -0700

--
 .../jenkins/common_job_properties.groovy|  4 +-
 ..._beam_PostCommit_Java_JDKVersionsTest.groovy | 60 
 2 files changed, 63 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/2f9a38e5/.test-infra/jenkins/common_job_properties.groovy
--
diff --git a/.test-infra/jenkins/common_job_properties.groovy 
b/.test-infra/jenkins/common_job_properties.groovy
index f47ab28..5ba12fe 100644
--- a/.test-infra/jenkins/common_job_properties.groovy
+++ b/.test-infra/jenkins/common_job_properties.groovy
@@ -188,7 +188,9 @@ class common_job_properties {
 // Disable archiving the built artifacts by default, as this is slow and 
flaky.
 // We can usually recreate them easily, and we can also opt-in individual 
jobs
 // to artifact archiving.
-context.archivingDisabled(true)
+if (context.metaClass.respondsTo(context, 'archivingDisabled', boolean)) {
+  context.archivingDisabled(true)
+}
   }
 
   // Sets common config for PreCommit jobs.

http://git-wip-us.apache.org/repos/asf/beam/blob/2f9a38e5/.test-infra/jenkins/job_beam_PostCommit_Java_JDKVersionsTest.groovy
--
diff --git 
a/.test-infra/jenkins/job_beam_PostCommit_Java_JDKVersionsTest.groovy 
b/.test-infra/jenkins/job_beam_PostCommit_Java_JDKVersionsTest.groovy
new file mode 100644
index 000..f23e741
--- /dev/null
+++ b/.test-infra/jenkins/job_beam_PostCommit_Java_JDKVersionsTest.groovy
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import common_job_properties
+
+// This job runs the Java postcommit tests cross multiple JDK versions.
+matrixJob('beam_PostCommit_Java_JDK_Versions_Test') {
+  description('Runs postcommit tests on the Java SDK in multiple Jdk 
versions.')
+
+  // Set common parameters.
+  common_job_properties.setTopLevelMainJobProperties(delegate)
+
+  // Set JDK versions.
+  axes {
+label('label', 'beam')
+jdk('JDK 1.7 (latest)',
+'OpenJDK 7 (on Ubuntu only)',
+'OpenJDK 8 (on Ubuntu only)')
+  }
+
+  // Sets that this is a PostCommit job.
+  common_job_properties.setPostCommit(
+  delegate,
+  '0 */6 * * *',
+  false)
+
+  // Allows triggering this build against pull requests.
+  common_job_properties.enablePhraseTriggeringFromPullRequest(
+  delegate,
+  'Java JDK Version Test',
+  'Run Java JDK Version Test')
+
+  // Maven build for this job.
+  steps {
+maven {
+  // Set maven parameters.
+  common_job_properties.setMavenConfig(delegate)
+
+  // Maven build project.
+  // Skip beam-sdks-python since this test is only apply to Java.
+  // TODO[BEAM-2322,BEAM-2323,BEAM-2324]: Re-enable beam-runners-apex once 
the build is passed.
+  goals('-B -e -P dataflow-runner clean install -pl 
\'!org.apache.beam:beam-sdks-python,!org.apache.beam:beam-runners-apex\' 
-DskipITs=false -DintegrationTestPipelineOptions=\'[ 
"--project=apache-beam-testing", 
"--tempRoot=gs://temp-storage-for-end-to-end-tests", 
"--runner=TestDataflowRunner" ]\'')
+}
+  }
+}

[2/2] beam git commit: [BEAM-1544] Java cross-JDK version tests on Jenkins\n\nThis closes #3184

[lcwik]

[BEAM-1544] Java cross-JDK version tests on Jenkins\n\nThis closes #3184


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/2fa24d89
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/2fa24d89
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/2fa24d89

Branch: refs/heads/master
Commit: 2fa24d89c83f287b55c7aeeeb6a53bc48194adbc
Parents: f9431a3 2f9a38e
Author: Luke Cwik 
Authored: Tue May 30 17:49:22 2017 -0700
Committer: Luke Cwik 
Committed: Tue May 30 17:49:22 2017 -0700

--
 .../jenkins/common_job_properties.groovy|  4 +-
 ..._beam_PostCommit_Java_JDKVersionsTest.groovy | 60 
 2 files changed, 63 insertions(+), 1 deletion(-)
--

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall_Windows #74

[Apache Jenkins Server]

See 


Changes:

[jbonofre] [BEAM-2379] Avoid reading projectId from environment variable in 
tests.

[lcwik] [BEAM-1347] Remove the usage of a thread local on a potentially hot path

[kirpichov] [BEAM-2248] KafkaIO support to use start read time to set start 
offset

[altay] Clean up pyc files before running tests

--
[...truncated 2.53 MB...]
2017-05-31T00:39:01.701 [INFO] 
2017-05-31T00:39:01.701 [INFO] --- groovy-maven-plugin:2.0:execute 
(find-supported-python-for-clean) @ beam-sdks-python ---
2017-05-31T00:39:01.706 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-slf4j/1.8/gossip-slf4j-1.8.pom
2017-05-31T00:39:01.885 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-slf4j/1.8/gossip-slf4j-1.8.pom
 (2 KB at 8.1 KB/sec)
2017-05-31T00:39:01.890 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip/1.8/gossip-1.8.pom
2017-05-31T00:39:02.070 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip/1.8/gossip-1.8.pom
 (12 KB at 62.5 KB/sec)
2017-05-31T00:39:02.075 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/forge/forge-parent/9/forge-parent-9.pom
2017-05-31T00:39:02.254 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/forge/forge-parent/9/forge-parent-9.pom
 (13 KB at 71.7 KB/sec)
2017-05-31T00:39:02.259 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-core/1.8/gossip-core-1.8.pom
2017-05-31T00:39:02.353 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-core/1.8/gossip-core-1.8.pom
 (3 KB at 23.1 KB/sec)
2017-05-31T00:39:02.360 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-bootstrap/1.8/gossip-bootstrap-1.8.pom
2017-05-31T00:39:02.533 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-bootstrap/1.8/gossip-bootstrap-1.8.pom
 (2 KB at 9.1 KB/sec)
2017-05-31T00:39:02.538 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/guava/guava/14.0.1/guava-14.0.1.pom
2017-05-31T00:39:02.630 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/guava/guava/14.0.1/guava-14.0.1.pom
 (6 KB at 57.1 KB/sec)
2017-05-31T00:39:02.634 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/guava/guava-parent/14.0.1/guava-parent-14.0.1.pom
2017-05-31T00:39:02.725 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/guava/guava-parent/14.0.1/guava-parent-14.0.1.pom
 (3 KB at 27.4 KB/sec)
2017-05-31T00:39:02.731 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-classworlds/2.4.2/plexus-classworlds-2.4.2.pom
2017-05-31T00:39:02.825 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-classworlds/2.4.2/plexus-classworlds-2.4.2.pom
 (4 KB at 36.1 KB/sec)
2017-05-31T00:39:02.832 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-interpolation/1.16/plexus-interpolation-1.16.pom
2017-05-31T00:39:03.004 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-interpolation/1.16/plexus-interpolation-1.16.pom
 (2 KB at 5.8 KB/sec)
2017-05-31T00:39:03.010 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/eclipse/aether/aether-api/0.9.0.M2/aether-api-0.9.0.M2.pom
2017-05-31T00:39:03.021 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/eclipse/aether/aether-api/0.9.0.M2/aether-api-0.9.0.M2.pom
 (2 KB at 154.2 KB/sec)
2017-05-31T00:39:03.026 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-api/2.0/gmaven-adapter-api-2.0.pom
2017-05-31T00:39:03.199 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-api/2.0/gmaven-adapter-api-2.0.pom
 (2 KB at 10.2 KB/sec)
2017-05-31T00:39:03.204 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-impl/2.0/gmaven-adapter-impl-2.0.pom
2017-05-31T00:39:03.296 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-impl/2.0/gmaven-adapter-impl-2.0.pom
 (3 KB at 28.9 KB/sec)
2017-05-31T00:39:03.301 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/groovy/groovy-all/2.1.5/groovy-all-2.1.5.pom
2017-05-31T00:39:03.475 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/groovy/groovy-all/2.1.5/groovy-all-2.1.5.pom
 (18 KB at 101.2 KB/sec)
2017-05-31T00:39:03.481 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant/1.8.4/ant-1.8.4.pom
2017-05-31T00:39:03.657 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant/1.8.4/ant-1.8.4.pom (10 
KB at 53.6 KB/sec)
2017-05-31T00:39:03.662 [INFO] Downloading: 

[jira] [Created] (BEAM-2392) Avoid use of proto builder clone

[Nigel Kilmer (JIRA)]

Nigel Kilmer created BEAM-2392:
--

 Summary: Avoid use of proto builder clone
 Key: BEAM-2392
 URL: https://issues.apache.org/jira/browse/BEAM-2392
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-gcp
Affects Versions: 2.1.0
Reporter: Nigel Kilmer
Assignee: Daniel Halperin
Priority: Minor


BigtableServiceImpl uses the clone method of the MutateRowResponse proto 
builder here:

https://github.com/apache/beam/blob/04e3261818aed0c129e7c715e371463bf5b5c1b8/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableServiceImpl.java#L212

This method is not generated by the Google-internal Java proto generator, so I 
had to change this to get it to work with an internal project. Are you 
interested in adding this change to the main repository for compatibility, or 
would you prefer to keep the cleaner version that uses clone?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Flink #2978

[Apache Jenkins Server]

See 

[GitHub] beam pull request #3262: Clean up pyc files before running tests

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3262


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: Clean up pyc files before running tests

[altay]

Repository: beam
Updated Branches:
  refs/heads/master 94d677dc9 -> f9431a390


Clean up pyc files before running tests


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/2ddaaeac
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/2ddaaeac
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/2ddaaeac

Branch: refs/heads/master
Commit: 2ddaaeac1462e88bb447802f7574d4a4cee12ed0
Parents: 94d677d
Author: Sourabh Bajaj 
Authored: Tue May 30 14:19:25 2017 -0700
Committer: Ahmet Altay 
Committed: Tue May 30 16:45:06 2017 -0700

--
 sdks/python/tox.ini | 6 ++
 1 file changed, 6 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/2ddaaeac/sdks/python/tox.ini
--
diff --git a/sdks/python/tox.ini b/sdks/python/tox.ini
index 917e907..2166f6a 100644
--- a/sdks/python/tox.ini
+++ b/sdks/python/tox.ini
@@ -31,6 +31,8 @@ deps =
   grpcio-tools==1.3.5
 commands =
   python --version
+  # Clean up all previous python generated files.
+  - find apache_beam -type f -name '*.pyc' -delete
   pip install -e .[test]
   python apache_beam/examples/complete/autocomplete_test.py
   python setup.py test
@@ -50,6 +52,8 @@ deps =
 whitelist_externals=find
 commands =
   python --version
+  # Clean up all previous python generated files.
+  - find apache_beam -type f -name '*.pyc' -delete
   # Clean up all previous cython generated files.
   - find apache_beam -type f -name '*.c' -delete
   - find apache_beam -type f -name '*.so' -delete
@@ -72,6 +76,8 @@ deps =
 commands =
   pip install -e .[test,gcp]
   python --version
+  # Clean up all previous python generated files.
+  - find apache_beam -type f -name '*.pyc' -delete
   python apache_beam/examples/complete/autocomplete_test.py
   python setup.py test
 passenv = TRAVIS*

[2/2] beam git commit: This closes #3262

[altay]

This closes #3262


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/f9431a39
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/f9431a39
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/f9431a39

Branch: refs/heads/master
Commit: f9431a3900692252a7c68d8c3c1d27b83a0c5d07
Parents: 94d677d 2ddaaea
Author: Ahmet Altay 
Authored: Tue May 30 16:46:42 2017 -0700
Committer: Ahmet Altay 
Committed: Tue May 30 16:46:42 2017 -0700

--
 sdks/python/tox.ini | 6 ++
 1 file changed, 6 insertions(+)
--

[jira] [Created] (BEAM-2391) Migrate to gcloud java core for default GCP project id detection

[Luke Cwik (JIRA)]

Luke Cwik created BEAM-2391:
---

 Summary: Migrate to gcloud java core for default GCP project id 
detection
 Key: BEAM-2391
 URL: https://issues.apache.org/jira/browse/BEAM-2391
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-gcp
Reporter: Luke Cwik
Assignee: Daniel Halperin
Priority: Minor


This was exposed within the gcloud java core library with the following issue:
https://github.com/GoogleCloudPlatform/google-cloud-java/issues/1207



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3263: Flink*DoFnFunction: fix check for single-output dof...

[dhalperi]

GitHub user dhalperi opened a pull request:

https://github.com/apache/beam/pull/3263

Flink*DoFnFunction: fix check for single-output dofns

Fixes Findbugs and (presumably) increases efficiency by using the right 
OutputManager.

R: @aljoscha but since this is a PostCommit break we may merge sooner if 
tests pass.

R: @tgroh @kennknowles 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/beam flink-output-manager

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3263.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3263


commit 159003267a0e05c8c90cb5bbb3565980514da9d5
Author: Dan Halperin 
Date:   2017-05-30T23:12:23Z

Flink*DoFnFunction: fix check for single-output dofns

Fixes Findbugs and (presumably) increases efficiency




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #3965

[Apache Jenkins Server]

See 


Changes:

[kirpichov] [BEAM-2248] KafkaIO support to use start read time to set start 
offset

--
[...truncated 3.06 MB...]
2017-05-30T23:05:43.338 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/sling/org.apache.sling.commons.json/2.0.6/org.apache.sling.commons.json-2.0.6.jar
 (47 KB at 52.0 KB/sec)
2017-05-30T23:05:43.338 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-runtime_2.10/1.2.1/flink-runtime_2.10-1.2.1-tests.jar
2017-05-30T23:05:43.367 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/2.10.4/scala-library-2.10.4.jar
 (6960 KB at 7491.2 KB/sec)
2017-05-30T23:05:43.368 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-streaming-java_2.10/1.2.1/flink-streaming-java_2.10-1.2.1-tests.jar
2017-05-30T23:05:43.411 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-core/1.2.1/flink-core-1.2.1-tests.jar
 (716 KB at 735.7 KB/sec)
2017-05-30T23:05:43.411 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils_2.10/1.2.1/flink-test-utils_2.10-1.2.1.jar
2017-05-30T23:05:43.466 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-streaming-java_2.10/1.2.1/flink-streaming-java_2.10-1.2.1-tests.jar
 (953 KB at 926.9 KB/sec)
2017-05-30T23:05:43.466 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-junit/1.2.1/flink-test-utils-junit-1.2.1.jar
2017-05-30T23:05:43.494 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-junit/1.2.1/flink-test-utils-junit-1.2.1.jar
 (24 KB at 21.8 KB/sec)
2017-05-30T23:05:43.494 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-test/2.8.0/curator-test-2.8.0.jar
2017-05-30T23:05:43.527 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-test/2.8.0/curator-test-2.8.0.jar
 (39 KB at 35.8 KB/sec)
2017-05-30T23:05:43.616 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-streaming-java_2.10/1.2.1/flink-streaming-java_2.10-1.2.1.jar
 (3035 KB at 2575.7 KB/sec)
2017-05-30T23:05:43.636 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils_2.10/1.2.1/flink-test-utils_2.10-1.2.1.jar
 (2366 KB at 1974.7 KB/sec)
2017-05-30T23:05:43.637 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-runtime_2.10/1.2.1/flink-runtime_2.10-1.2.1-tests.jar
 (2432 KB at 2028.1 KB/sec)
2017-05-30T23:05:43.997 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-shaded-hadoop2/1.2.1/flink-shaded-hadoop2-1.2.1.jar
 (17860 KB at 11448.3 KB/sec)
2017-05-30T23:05:44.009 [INFO] 
2017-05-30T23:05:44.009 [INFO] --- maven-clean-plugin:3.0.0:clean 
(default-clean) @ beam-runners-flink_2.10 ---
2017-05-30T23:05:44.011 [INFO] Deleting 

 (includes = [**/*.pyc, **/*.egg-info/, **/sdks/python/LICENSE, 
**/sdks/python/NOTICE, **/sdks/python/README.md], excludes = [])
2017-05-30T23:05:44.066 [INFO] 
2017-05-30T23:05:44.066 [INFO] --- maven-enforcer-plugin:1.4.1:enforce 
(enforce) @ beam-runners-flink_2.10 ---
2017-05-30T23:05:46.342 [INFO] 
2017-05-30T23:05:46.342 [INFO] --- maven-enforcer-plugin:1.4.1:enforce 
(enforce-banned-dependencies) @ beam-runners-flink_2.10 ---
2017-05-30T23:05:46.411 [INFO] 
2017-05-30T23:05:46.411 [INFO] --- maven-remote-resources-plugin:1.5:process 
(process-resource-bundles) @ beam-runners-flink_2.10 ---
2017-05-30T23:05:46.979 [INFO] 
2017-05-30T23:05:46.979 [INFO] --- maven-resources-plugin:3.0.2:resources 
(default-resources) @ beam-runners-flink_2.10 ---
2017-05-30T23:05:46.980 [INFO] Using 'UTF-8' encoding to copy filtered 
resources.
2017-05-30T23:05:46.981 [INFO] Copying 1 resource
2017-05-30T23:05:46.981 [INFO] Copying 3 resources
2017-05-30T23:05:47.124 [INFO] 
2017-05-30T23:05:47.124 [INFO] --- maven-compiler-plugin:3.6.1:compile 
(default-compile) @ beam-runners-flink_2.10 ---
2017-05-30T23:05:47.138 [INFO] Changes detected - recompiling the module!
2017-05-30T23:05:47.138 [INFO] Compiling 75 source files to 

2017-05-30T23:05:47.930 [WARNING] bootstrap class path not set in conjunction 
with -source 1.7
2017-05-30T23:05:47.930 [INFO] 
:
 Some input files use or override a deprecated API.
2017-05-30T23:05:47.930 [INFO] 

[jira] [Created] (BEAM-2390) allow user to use .setTimePartitioning in BigQueryIO.write

[Eric Johston (JIRA)]

Eric Johston created BEAM-2390:
--

 Summary: allow user to use .setTimePartitioning in BigQueryIO.write
 Key: BEAM-2390
 URL: https://issues.apache.org/jira/browse/BEAM-2390
 Project: Beam
  Issue Type: Improvement
  Components: beam-model-runner-api
Affects Versions: 2.0.0
Reporter: Eric Johston
Assignee: Kenneth Knowles
 Fix For: 2.0.0


Currently when writing to a table with BigQueryIO sink, there is no way to 
create a new table that is date partitioned. This would be very useful, since 
currently the only way to do this is  by manually creating a table ahead of 
time. We should be able to leverage the automatic table creation functionality 
for date partitioned tables.

The best way to do this would be to have a withTimePartitioning method in the 
BigQueryIO class.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #3964

[Apache Jenkins Server]

See 


Changes:

[lcwik] [BEAM-1347] Remove the usage of a thread local on a potentially hot path

--
[...truncated 3.07 MB...]
2017-05-30T22:31:16.427 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-core/1.2.1/flink-core-1.2.1-tests.jar
2017-05-30T22:31:16.439 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/sling/org.apache.sling.commons.json/2.0.6/org.apache.sling.commons.json-2.0.6.jar
 (47 KB at 34.8 KB/sec)
2017-05-30T22:31:16.440 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-runtime_2.10/1.2.1/flink-runtime_2.10-1.2.1-tests.jar
2017-05-30T22:31:16.620 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-core/1.2.1/flink-core-1.2.1-tests.jar
 (716 KB at 470.0 KB/sec)
2017-05-30T22:31:16.620 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-streaming-java_2.10/1.2.1/flink-streaming-java_2.10-1.2.1-tests.jar
2017-05-30T22:31:16.719 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/2.10.4/scala-library-2.10.4.jar
 (6960 KB at 4290.6 KB/sec)
2017-05-30T22:31:16.719 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils_2.10/1.2.1/flink-test-utils_2.10-1.2.1.jar
2017-05-30T22:31:16.817 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-streaming-java_2.10/1.2.1/flink-streaming-java_2.10-1.2.1-tests.jar
 (953 KB at 554.0 KB/sec)
2017-05-30T22:31:16.817 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-junit/1.2.1/flink-test-utils-junit-1.2.1.jar
2017-05-30T22:31:16.847 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-junit/1.2.1/flink-test-utils-junit-1.2.1.jar
 (24 KB at 13.1 KB/sec)
2017-05-30T22:31:16.847 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-test/2.8.0/curator-test-2.8.0.jar
2017-05-30T22:31:16.879 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-test/2.8.0/curator-test-2.8.0.jar
 (39 KB at 21.9 KB/sec)
2017-05-30T22:31:16.916 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-streaming-java_2.10/1.2.1/flink-streaming-java_2.10-1.2.1.jar
 (3035 KB at 1668.1 KB/sec)
2017-05-30T22:31:16.964 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-runtime_2.10/1.2.1/flink-runtime_2.10-1.2.1-tests.jar
 (2432 KB at 1302.4 KB/sec)
2017-05-30T22:31:17.003 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils_2.10/1.2.1/flink-test-utils_2.10-1.2.1.jar
 (2366 KB at 1241.2 KB/sec)
2017-05-30T22:31:17.342 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-shaded-hadoop2/1.2.1/flink-shaded-hadoop2-1.2.1.jar
 (17860 KB at 7955.2 KB/sec)
2017-05-30T22:31:17.356 [INFO] 
2017-05-30T22:31:17.356 [INFO] --- maven-clean-plugin:3.0.0:clean 
(default-clean) @ beam-runners-flink_2.10 ---
2017-05-30T22:31:17.358 [INFO] Deleting 

 (includes = [**/*.pyc, **/*.egg-info/, **/sdks/python/LICENSE, 
**/sdks/python/NOTICE, **/sdks/python/README.md], excludes = [])
2017-05-30T22:31:17.738 [INFO] 
2017-05-30T22:31:17.738 [INFO] --- maven-enforcer-plugin:1.4.1:enforce 
(enforce) @ beam-runners-flink_2.10 ---
2017-05-30T22:31:20.054 [INFO] 
2017-05-30T22:31:20.054 [INFO] --- maven-enforcer-plugin:1.4.1:enforce 
(enforce-banned-dependencies) @ beam-runners-flink_2.10 ---
2017-05-30T22:31:20.123 [INFO] 
2017-05-30T22:31:20.123 [INFO] --- maven-remote-resources-plugin:1.5:process 
(process-resource-bundles) @ beam-runners-flink_2.10 ---
2017-05-30T22:31:20.657 [INFO] 
2017-05-30T22:31:20.657 [INFO] --- maven-resources-plugin:3.0.2:resources 
(default-resources) @ beam-runners-flink_2.10 ---
2017-05-30T22:31:20.658 [INFO] Using 'UTF-8' encoding to copy filtered 
resources.
2017-05-30T22:31:20.659 [INFO] Copying 1 resource
2017-05-30T22:31:20.659 [INFO] Copying 3 resources
2017-05-30T22:31:20.747 [INFO] 
2017-05-30T22:31:20.747 [INFO] --- maven-compiler-plugin:3.6.1:compile 
(default-compile) @ beam-runners-flink_2.10 ---
2017-05-30T22:31:20.761 [INFO] Changes detected - recompiling the module!
2017-05-30T22:31:20.762 [INFO] Compiling 75 source files to 

2017-05-30T22:31:21.964 [WARNING] bootstrap class path not set in conjunction 
with -source 1.7
2017-05-30T22:31:21.965 [INFO] 
:
 Some input files use or override a deprecated API.

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Flink #2977

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2248) KafkaIO support to use start read time to set start offset

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030232#comment-16030232
 ] 

ASF GitHub Bot commented on BEAM-2248:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3044


> KafkaIO support to use start read time to set start offset
> --
>
> Key: BEAM-2248
> URL: https://issues.apache.org/jira/browse/BEAM-2248
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: Jingsong Lee
>Assignee: Jingsong Lee
>
> This Kafka 0.10.x adds support for a searchable index for each topic based 
> off of message timestamps. It enables consumer support for offset lookup by 
> timestamp.
> So we can add a start read time to set start offset.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3044: [BEAM-2248] KafkaIO support to use start read time ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3044


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: [BEAM-2248] KafkaIO support to use start read time to set start offset

[jkff]

Repository: beam
Updated Branches:
  refs/heads/master 49067b164 -> 94d677dc9


[BEAM-2248] KafkaIO support to use start read time to set start offset


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/d29e353e
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/d29e353e
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/d29e353e

Branch: refs/heads/master
Commit: d29e353ea53349e3c94285fdf5b29318252087d1
Parents: 49067b1
Author: JingsongLi 
Authored: Wed May 10 19:49:04 2017 +0800
Committer: Eugene Kirpichov 
Committed: Tue May 30 14:58:20 2017 -0700

--
 sdks/java/io/kafka/pom.xml  |   2 +-
 .../apache/beam/sdk/io/kafka/ConsumerSpEL.java  |  56 +
 .../org/apache/beam/sdk/io/kafka/KafkaIO.java   |  44 ++-
 .../apache/beam/sdk/io/kafka/KafkaIOTest.java   | 122 ---
 4 files changed, 205 insertions(+), 19 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/d29e353e/sdks/java/io/kafka/pom.xml
--
diff --git a/sdks/java/io/kafka/pom.xml b/sdks/java/io/kafka/pom.xml
index f6f0385..29350cc 100644
--- a/sdks/java/io/kafka/pom.xml
+++ b/sdks/java/io/kafka/pom.xml
@@ -30,7 +30,7 @@
   Library to read Kafka topics.
 
   
-0.9.0.1
+0.10.1.0
   
 
   

http://git-wip-us.apache.org/repos/asf/beam/blob/d29e353e/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/ConsumerSpEL.java
--
diff --git 
a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/ConsumerSpEL.java
 
b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/ConsumerSpEL.java
index 8fe17c1..8cdad22 100644
--- 
a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/ConsumerSpEL.java
+++ 
b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/ConsumerSpEL.java
@@ -17,12 +17,18 @@
  */
 package org.apache.beam.sdk.io.kafka;
 
+import static com.google.common.base.Preconditions.checkArgument;
+
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Iterables;
 import java.lang.reflect.InvocationTargetException;
 import java.lang.reflect.Method;
 import java.util.Collection;
+import java.util.Map;
 import org.apache.kafka.clients.consumer.Consumer;
 import org.apache.kafka.clients.consumer.ConsumerRecord;
 import org.apache.kafka.common.TopicPartition;
+import org.joda.time.Instant;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 import org.springframework.expression.Expression;
@@ -51,13 +57,28 @@ class ConsumerSpEL {
   private Method timestampMethod;
   private boolean hasRecordTimestamp = false;
 
+  private Method offsetGetterMethod;
+  private Method offsetsForTimesMethod;
+  private boolean hasOffsetsForTimes = false;
+
   public ConsumerSpEL() {
 try {
+  // It is supported by Kafka Client 0.10.0.0 onwards.
   timestampMethod = ConsumerRecord.class.getMethod("timestamp", 
(Class[]) null);
   hasRecordTimestamp = timestampMethod.getReturnType().equals(Long.TYPE);
 } catch (NoSuchMethodException | SecurityException e) {
   LOG.debug("Timestamp for Kafka message is not available.");
 }
+
+try {
+  // It is supported by Kafka Client 0.10.1.0 onwards.
+  offsetGetterMethod = 
Class.forName("org.apache.kafka.clients.consumer.OffsetAndTimestamp")
+  .getMethod("offset", (Class[]) null);
+  offsetsForTimesMethod = Consumer.class.getMethod("offsetsForTimes", 
Map.class);
+  hasOffsetsForTimes = 
offsetsForTimesMethod.getReturnType().equals(Map.class);
+} catch (NoSuchMethodException | SecurityException | 
ClassNotFoundException e) {
+  LOG.debug("OffsetsForTimes is not available.");
+}
   }
 
   public void evaluateSeek2End(Consumer consumer, TopicPartition 
topicPartitions) {
@@ -88,4 +109,39 @@ class ConsumerSpEL {
 }
 return timestamp;
   }
+
+  public boolean hasOffsetsForTimes() {
+return hasOffsetsForTimes;
+  }
+
+  /**
+   * Look up the offset for the given partition by timestamp.
+   * Throws RuntimeException if there are no messages later than timestamp or 
if this partition
+   * does not support timestamp based offset.
+   */
+  @SuppressWarnings("unchecked")
+  public long offsetForTime(Consumer consumer, TopicPartition 
topicPartition, Instant time) {
+
+checkArgument(hasOffsetsForTimes,
+"This Kafka Client must support Consumer.OffsetsForTimes().");
+
+Map timestampsToSearch =
+ImmutableMap.of(topicPartition, time.getMillis());
+try {
+  Map offsetsByTimes = (Map) offsetsForTimesMethod.invoke(consumer, 
timestampsToSearch);
+  Object offsetAndTimestamp = 

[2/2] beam git commit: This closes #3044: [BEAM-2248] KafkaIO support to use start read time to set start offset

[jkff]

This closes #3044: [BEAM-2248] KafkaIO support to use start read time to set 
start offset


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/94d677dc
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/94d677dc
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/94d677dc

Branch: refs/heads/master
Commit: 94d677dc97330b424ce960310f23e7992f67bfd6
Parents: 49067b1 d29e353
Author: Eugene Kirpichov 
Authored: Tue May 30 14:58:28 2017 -0700
Committer: Eugene Kirpichov 
Committed: Tue May 30 14:58:28 2017 -0700

--
 sdks/java/io/kafka/pom.xml  |   2 +-
 .../apache/beam/sdk/io/kafka/ConsumerSpEL.java  |  56 +
 .../org/apache/beam/sdk/io/kafka/KafkaIO.java   |  44 ++-
 .../apache/beam/sdk/io/kafka/KafkaIOTest.java   | 122 ---
 4 files changed, 205 insertions(+), 19 deletions(-)
--

[jira] [Updated] (BEAM-2373) AvroSource: Premature End of stream Exception on SnappyCompressorInputStream

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin updated BEAM-2373:
--
Fix Version/s: 2.1.0

> AvroSource: Premature End of stream Exception on SnappyCompressorInputStream
> 
>
> Key: BEAM-2373
> URL: https://issues.apache.org/jira/browse/BEAM-2373
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.0.0
>Reporter: Michael Luckey
>Assignee: Michael Luckey
>Priority: Critical
> Fix For: 2.1.0
>
>
> During processing we encountered on some of our snappy encoded avro input 
> files
> {noformat}
> Exception in thread "main" java.lang.RuntimeException: java.io.IOException: 
> Premature end of stream
>  at 
> org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:330)
>  at 
> org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:292)
>  at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:200)
>  at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:63)
>  at org.apache.beam.sdk.Pipeline.run(Pipeline.java:295)
>  at org.apache.beam.sdk.Pipeline.run(Pipeline.java:281)
> Caused by: java.io.IOException: Premature end of stream
>  at 
> org.apache.beam.sdk.repackaged.org.apache.commons.compress.compressors.snappy.SnappyCompressorInputStream.expandLiteral(SnappyCompressorInputStream.java:310)
>  at 
> org.apache.beam.sdk.repackaged.org.apache.commons.compress.compressors.snappy.SnappyCompressorInputStream.fill(SnappyCompressorInputStream.java:169)
>  at 
> org.apache.beam.sdk.repackaged.org.apache.commons.compress.compressors.snappy.SnappyCompressorInputStream.read(SnappyCompressorInputStream.java:134)
>  at 
> org.apache.avro.io.BinaryDecoder$InputStreamByteSource.tryReadRaw(BinaryDecoder.java:839)
>  at 
> org.apache.avro.io.BinaryDecoder$ByteSource.compactAndFill(BinaryDecoder.java:692)
>  at org.apache.avro.io.BinaryDecoder.ensureBounds(BinaryDecoder.java:471)
>  at org.apache.avro.io.BinaryDecoder.readInt(BinaryDecoder.java:128)
>  at org.apache.avro.io.BinaryDecoder.readIndex(BinaryDecoder.java:423)
>  at org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:290)
>  at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
>  at org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:267)
>  at 
> org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:178)
>  at 
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:152)
>  at 
> org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:240)
>  at 
> org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:230)
>  at 
> org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:174)
>  at 
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:152)
>  at 
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:144)
>  at 
> org.apache.beam.sdk.io.AvroSource$AvroBlock.readNextRecord(AvroSource.java:579)
>  at 
> org.apache.beam.sdk.io.BlockBasedSource$BlockBasedReader.readNextRecord(BlockBasedSource.java:198)
>  at 
> org.apache.beam.sdk.io.FileBasedSource$FileBasedReader.advanceImpl(FileBasedSource.java:479)
>  at 
> org.apache.beam.sdk.io.OffsetBasedSource$OffsetBasedReader.advance(OffsetBasedSource.java:277)
>  at 
> org.apache.beam.runners.direct.BoundedReadEvaluatorFactory$BoundedReadEvaluator.processElement(BoundedReadEvaluatorFactory.java:148)
>  at 
> org.apache.beam.runners.direct.TransformExecutor.processElements(TransformExecutor.java:146)
>  at 
> org.apache.beam.runners.direct.TransformExecutor.run(TransformExecutor.java:110)
>  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>  at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  at java.lang.Thread.run(Thread.java:745)
> {noformat}
> This seems to be caused by a bug in apache.commons.compress:1.9, which was 
> addressed here:
> https://github.com/apache/commons-compress/commit/9ae37525134089dd0c9ee1cf8738192b70e0fc07
> Used a pipeline using AvroIO, on spark and direct, both on hdfs and local 
> file system.
> In our short tests we got it running without exceptions by either:
> * upgrading to commons.compress:1.14
> * applying the patch to the 1.9er code of SnappyCompressorInputStream
> Impacts on other components were not tested, of course :(



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-2389) GcpCoreApiSurfaceTest isn't testing right module

[Daniel Halperin (JIRA)]

Daniel Halperin created BEAM-2389:
-

 Summary: GcpCoreApiSurfaceTest isn't testing right module
 Key: BEAM-2389
 URL: https://issues.apache.org/jira/browse/BEAM-2389
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-gcp
Affects Versions: 2.0.0
Reporter: Daniel Halperin
 Fix For: 2.1.0


It looks like a clone of {{SdkApiSurfaceTest}} that was not updated, outside of 
being renamed, now that it's in a new module. Even the java package of the test 
is wrong.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Flink #2976

[Apache Jenkins Server]

See 

[GitHub] beam pull request #3262: Clean up pyc files before running tests

[sb2nov]

GitHub user sb2nov opened a pull request:

https://github.com/apache/beam/pull/3262

Clean up pyc files before running tests

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---

R: @aaltay PTAL
cc: @robertwb @mariapython 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sb2nov/beam BEAM-remove-old-pyc-files

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3262.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3262


commit 3ba89e736306ef6be7c97025e0ff3655a8f6d52c
Author: Sourabh Bajaj 
Date:   2017-05-30T21:19:25Z

Clean up pyc files before running tests




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #3963

[Apache Jenkins Server]

See 


Changes:

[jbonofre] [BEAM-2379] Avoid reading projectId from environment variable in 
tests.

--
[...truncated 3.07 MB...]
2017-05-30T21:24:31.124 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-core/1.2.1/flink-core-1.2.1-tests.jar
2017-05-30T21:24:31.141 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/2.10.4/scala-library-2.10.4.jar
 (6960 KB at 7279.7 KB/sec)
2017-05-30T21:24:31.141 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-runtime_2.10/1.2.1/flink-runtime_2.10-1.2.1-tests.jar
2017-05-30T21:24:31.143 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/sling/org.apache.sling.commons.json/2.0.6/org.apache.sling.commons.json-2.0.6.jar
 (47 KB at 48.8 KB/sec)
2017-05-30T21:24:31.143 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-streaming-java_2.10/1.2.1/flink-streaming-java_2.10-1.2.1-tests.jar
2017-05-30T21:24:31.210 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-core/1.2.1/flink-core-1.2.1-tests.jar
 (716 KB at 698.4 KB/sec)
2017-05-30T21:24:31.210 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils_2.10/1.2.1/flink-test-utils_2.10-1.2.1.jar
2017-05-30T21:24:31.252 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-streaming-java_2.10/1.2.1/flink-streaming-java_2.10-1.2.1-tests.jar
 (953 KB at 893.0 KB/sec)
2017-05-30T21:24:31.252 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-junit/1.2.1/flink-test-utils-junit-1.2.1.jar
2017-05-30T21:24:31.283 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-junit/1.2.1/flink-test-utils-junit-1.2.1.jar
 (24 KB at 21.0 KB/sec)
2017-05-30T21:24:31.283 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-test/2.8.0/curator-test-2.8.0.jar
2017-05-30T21:24:31.313 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-test/2.8.0/curator-test-2.8.0.jar
 (39 KB at 34.5 KB/sec)
2017-05-30T21:24:31.374 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-runtime_2.10/1.2.1/flink-runtime_2.10-1.2.1-tests.jar
 (2432 KB at 2045.1 KB/sec)
2017-05-30T21:24:31.384 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-streaming-java_2.10/1.2.1/flink-streaming-java_2.10-1.2.1.jar
 (3035 KB at 2530.6 KB/sec)
2017-05-30T21:24:31.444 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils_2.10/1.2.1/flink-test-utils_2.10-1.2.1.jar
 (2366 KB at 1879.0 KB/sec)
2017-05-30T21:24:31.646 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-shaded-hadoop2/1.2.1/flink-shaded-hadoop2-1.2.1.jar
 (17860 KB at 12215.7 KB/sec)
2017-05-30T21:24:31.662 [INFO] 
2017-05-30T21:24:31.662 [INFO] --- maven-clean-plugin:3.0.0:clean 
(default-clean) @ beam-runners-flink_2.10 ---
2017-05-30T21:24:31.665 [INFO] Deleting 

 (includes = [**/*.pyc, **/*.egg-info/, **/sdks/python/LICENSE, 
**/sdks/python/NOTICE, **/sdks/python/README.md], excludes = [])
2017-05-30T21:24:31.841 [INFO] 
2017-05-30T21:24:31.841 [INFO] --- maven-enforcer-plugin:1.4.1:enforce 
(enforce) @ beam-runners-flink_2.10 ---
2017-05-30T21:24:34.139 [INFO] 
2017-05-30T21:24:34.139 [INFO] --- maven-enforcer-plugin:1.4.1:enforce 
(enforce-banned-dependencies) @ beam-runners-flink_2.10 ---
2017-05-30T21:24:35.271 [INFO] 
2017-05-30T21:24:35.271 [INFO] --- maven-remote-resources-plugin:1.5:process 
(process-resource-bundles) @ beam-runners-flink_2.10 ---
2017-05-30T21:24:36.147 [INFO] 
2017-05-30T21:24:36.147 [INFO] --- maven-resources-plugin:3.0.2:resources 
(default-resources) @ beam-runners-flink_2.10 ---
2017-05-30T21:24:36.148 [INFO] Using 'UTF-8' encoding to copy filtered 
resources.
2017-05-30T21:24:36.148 [INFO] Copying 1 resource
2017-05-30T21:24:36.149 [INFO] Copying 3 resources
2017-05-30T21:24:36.269 [INFO] 
2017-05-30T21:24:36.269 [INFO] --- maven-compiler-plugin:3.6.1:compile 
(default-compile) @ beam-runners-flink_2.10 ---
2017-05-30T21:24:36.343 [INFO] Changes detected - recompiling the module!
2017-05-30T21:24:36.343 [INFO] Compiling 75 source files to 

2017-05-30T21:24:37.485 [WARNING] bootstrap class path not set in conjunction 
with -source 1.7
2017-05-30T21:24:37.486 [INFO] 
:
 Some input files use or override a deprecated API.

[GitHub] beam pull request #3261: Rephrases messages on RetryHttpRequestInitializer

[jkff]

GitHub user jkff opened a pull request:

https://github.com/apache/beam/pull/3261

Rephrases messages on RetryHttpRequestInitializer

"will NOT retry" messages are confusing some customers into thinking
that an entire high-level operation inside a Beam job, involving this
request as an implementation detail, will not be retried (e.g. a BigQuery
import etc.)

In reality there's many levels of retries and
RetryHttpRequestInitializer can not be aware of them. Retrying at a
higher level may or may not happen, and it's up to higher-level
components to log that (they usually do).

R: @lukecwik 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jkff/incubator-beam retry-message

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3261.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3261


commit ab9b77db3dd263f432b6535d96495bb36fa4fe0e
Author: Eugene Kirpichov 
Date:   2017-05-30T21:16:22Z

Rephrases messages on RetryHttpRequestInitializer

"will NOT retry" messages are confusing some customers into thinking
that an entire high-level operation inside a Beam job, involving this
request as an implementation detail, will not be retried (e.g. a BigQuery
import etc.)

In reality there's many levels of retries and
RetryHttpRequestInitializer can not be aware of them. Retrying at a
higher level may or may not happen, and it's up to higher-level
components to log that (they usually do).




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1347) Basic Java harness capable of understanding process bundle tasks and sending data over the Fn Api

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030144#comment-16030144
 ] 

ASF GitHub Bot commented on BEAM-1347:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3260


> Basic Java harness capable of understanding process bundle tasks and sending 
> data over the Fn Api
> -
>
> Key: BEAM-1347
> URL: https://issues.apache.org/jira/browse/BEAM-1347
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model-fn-api
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>
> Create a basic Java harness capable of understanding process bundle requests 
> and able to stream data over the Fn Api.
> Overview: https://s.apache.org/beam-fn-api



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3260: [BEAM-1347] Remove the usage of a thread local on a...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3260


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: [BEAM-1347] Remove the usage of a thread local on a potentially hot path

[lcwik]

Repository: beam
Updated Branches:
  refs/heads/master 2d3e9fe75 -> 49067b164


[BEAM-1347] Remove the usage of a thread local on a potentially hot path


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/60779e2e
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/60779e2e
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/60779e2e

Branch: refs/heads/master
Commit: 60779e2ecd76f1cb4766050e4560765c1bc3c19b
Parents: 2d3e9fe
Author: Luke Cwik 
Authored: Tue May 30 13:15:31 2017 -0700
Committer: Luke Cwik 
Committed: Tue May 30 14:15:23 2017 -0700

--
 .../fn/harness/logging/BeamFnLoggingClient.java | 36 +++-
 1 file changed, 19 insertions(+), 17 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/60779e2e/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/logging/BeamFnLoggingClient.java
--
diff --git 
a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/logging/BeamFnLoggingClient.java
 
b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/logging/BeamFnLoggingClient.java
index c8d11ed..d56ee6d 100644
--- 
a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/logging/BeamFnLoggingClient.java
+++ 
b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/logging/BeamFnLoggingClient.java
@@ -38,7 +38,6 @@ import java.util.concurrent.Future;
 import java.util.concurrent.LinkedBlockingDeque;
 import java.util.concurrent.TimeUnit;
 import java.util.function.BiFunction;
-import java.util.function.Consumer;
 import java.util.function.Function;
 import java.util.logging.Formatter;
 import java.util.logging.Handler;
@@ -179,11 +178,14 @@ public class BeamFnLoggingClient implements AutoCloseable 
{
 private final BlockingDeque bufferedLogEntries =
 new LinkedBlockingDeque<>(MAX_BUFFERED_LOG_ENTRY_COUNT);
 private final Future bufferedLogWriter;
-private final ThreadLocal logEntryHandler;
+/**
+ * Safe object publishing is not required since we only care if the thread 
that set
+ * this field is equal to the thread also attempting to add a log entry.
+ */
+private Thread logEntryHandlerThread;
 
 private LogRecordHandler(ExecutorService executorService) {
   bufferedLogWriter = executorService.submit(this);
-  logEntryHandler = new ThreadLocal<>();
 }
 
 @Override
@@ -204,19 +206,18 @@ public class BeamFnLoggingClient implements AutoCloseable 
{
 builder.setTrace(getStackTraceAsString(record.getThrown()));
   }
   // The thread that sends log records should never perform a blocking 
publish and
-  // only insert log records best effort. We detect which thread is logging
-  // by using the thread local, defaulting to the blocking publish.
-  MoreObjects.firstNonNull(
-  logEntryHandler.get(), 
this::blockingPublish).accept(builder.build());
-}
-
-/** Blocks caller till enough space exists to publish this log entry. */
-private void blockingPublish(BeamFnApi.LogEntry logEntry) {
-  try {
-bufferedLogEntries.put(logEntry);
-  } catch (InterruptedException e) {
-Thread.currentThread().interrupt();
-throw new RuntimeException(e);
+  // only insert log records best effort.
+  if (Thread.currentThread() != logEntryHandlerThread) {
+// Blocks caller till enough space exists to publish this log entry.
+try {
+  bufferedLogEntries.put(builder.build());
+} catch (InterruptedException e) {
+  Thread.currentThread().interrupt();
+  throw new RuntimeException(e);
+}
+  } else {
+// Never blocks caller, will drop log message if buffer is full.
+bufferedLogEntries.offer(builder.build());
   }
 }
 
@@ -225,7 +226,8 @@ public class BeamFnLoggingClient implements AutoCloseable {
   // Logging which occurs in this thread will attempt to publish log 
entries into the
   // above handler which should never block if the queue is full otherwise
   // this thread will get stuck.
-  logEntryHandler.set(bufferedLogEntries::offer);
+  logEntryHandlerThread = Thread.currentThread();
+
   List additionalLogEntries =
   new ArrayList<>(MAX_BUFFERED_LOG_ENTRY_COUNT);
   try {

[2/2] beam git commit: [BEAM-1347] Remove the usage of a thread local on a potentially hot path

[lcwik]

[BEAM-1347] Remove the usage of a thread local on a potentially hot path

This closes #3260


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/49067b16
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/49067b16
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/49067b16

Branch: refs/heads/master
Commit: 49067b1642aa5ae01ec393ff9b2e0971f401fb5e
Parents: 2d3e9fe 60779e2
Author: Luke Cwik 
Authored: Tue May 30 14:15:31 2017 -0700
Committer: Luke Cwik 
Committed: Tue May 30 14:15:31 2017 -0700

--
 .../fn/harness/logging/BeamFnLoggingClient.java | 36 +++-
 1 file changed, 19 insertions(+), 17 deletions(-)
--

[jira] [Commented] (BEAM-1347) Basic Java harness capable of understanding process bundle tasks and sending data over the Fn Api

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030061#comment-16030061
 ] 

ASF GitHub Bot commented on BEAM-1347:
--

GitHub user lukecwik opened a pull request:

https://github.com/apache/beam/pull/3260

[BEAM-1347] Remove the usage of a thread local on a potentially hot path

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`.
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lukecwik/incubator-beam remove_thread_local

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3260.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3260


commit 25478d15b8907c834ad1bfa21ce1d40af5c67435
Author: Luke Cwik 
Date:   2017-05-30T20:15:31Z

[BEAM-1347] Remove the usage of a thread local on a potentially hot path




> Basic Java harness capable of understanding process bundle tasks and sending 
> data over the Fn Api
> -
>
> Key: BEAM-1347
> URL: https://issues.apache.org/jira/browse/BEAM-1347
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model-fn-api
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>
> Create a basic Java harness capable of understanding process bundle requests 
> and able to stream data over the Fn Api.
> Overview: https://s.apache.org/beam-fn-api



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Flink #2975

[Apache Jenkins Server]

See 

[GitHub] beam pull request #3260: [BEAM-1347] Remove the usage of a thread local on a...

[lukecwik]

GitHub user lukecwik opened a pull request:

https://github.com/apache/beam/pull/3260

[BEAM-1347] Remove the usage of a thread local on a potentially hot path

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`.
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lukecwik/incubator-beam remove_thread_local

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3260.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3260


commit 25478d15b8907c834ad1bfa21ce1d40af5c67435
Author: Luke Cwik 
Date:   2017-05-30T20:15:31Z

[BEAM-1347] Remove the usage of a thread local on a potentially hot path




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (BEAM-2379) SpannerIO tests are failing

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-2379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré updated BEAM-2379:
---
Fix Version/s: (was: Not applicable)
   2.1.0

> SpannerIO tests are failing
> ---
>
> Key: BEAM-2379
> URL: https://issues.apache.org/jira/browse/BEAM-2379
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions, testing
>Reporter: Jean-Baptiste Onofré
>Assignee: Mairbek Khadikov
> Fix For: 2.1.0
>
>
> When performing a complete {{mvn clean install}}, the build is failing on 
> {{SpannerIO}} tests:
> {code}
> [ERROR] Errors: 
> [ERROR]   SpannerIOTest.batching:129 » UserCode 
> java.lang.IllegalArgumentException: A pr...
> [ERROR]   SpannerIOTest.batchingGroups:155 » UserCode 
> java.lang.IllegalArgumentException...
> [ERROR]   SpannerIOTest.noBatching:178 » UserCode 
> java.lang.IllegalArgumentException: A ...
> [ERROR]   SpannerIOTest.singleMutationPipeline » UncheckedExecution 
> org.apache.beam.sdk
> {code}
> These tests fail with the same reason, here's the complete stack trace:
> {code}
> [ERROR] batchingGroups(org.apache.beam.sdk.io.gcp.spanner.SpannerIOTest)  
> Time elapsed: 0.004 s  <<< ERROR!
> org.apache.beam.sdk.util.UserCodeException: 
> java.lang.IllegalArgumentException: A project ID is required for this service 
> but could not be determined from the builder or the environment.  Please set 
> a project ID using the builder.
> at 
> org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:36)
> at 
> org.apache.beam.sdk.io.gcp.spanner.SpannerIO$SpannerWriteFn$DoFnInvoker.invokeSetup(Unknown
>  Source)
> at 
> org.apache.beam.sdk.transforms.DoFnTester.initializeState(DoFnTester.java:745)
> at 
> org.apache.beam.sdk.transforms.DoFnTester.startBundle(DoFnTester.java:219)
> at 
> org.apache.beam.sdk.transforms.DoFnTester.processBundle(DoFnTester.java:183)
> at 
> org.apache.beam.sdk.io.gcp.spanner.SpannerIOTest.batchingGroups(SpannerIOTest.java:155)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at 
> org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:321)
> at 
> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:239)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runners.Suite.runChild(Suite.java:128)
> at org.junit.runners.Suite.runChild(Suite.java:27)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at 
> org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55)
> at 
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137)
> at 
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107)
> at 
> 

[jira] [Resolved] (BEAM-2379) SpannerIO tests are failing

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-2379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré resolved BEAM-2379.

Resolution: Fixed

> SpannerIO tests are failing
> ---
>
> Key: BEAM-2379
> URL: https://issues.apache.org/jira/browse/BEAM-2379
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions, testing
>Reporter: Jean-Baptiste Onofré
>Assignee: Mairbek Khadikov
> Fix For: 2.1.0
>
>
> When performing a complete {{mvn clean install}}, the build is failing on 
> {{SpannerIO}} tests:
> {code}
> [ERROR] Errors: 
> [ERROR]   SpannerIOTest.batching:129 » UserCode 
> java.lang.IllegalArgumentException: A pr...
> [ERROR]   SpannerIOTest.batchingGroups:155 » UserCode 
> java.lang.IllegalArgumentException...
> [ERROR]   SpannerIOTest.noBatching:178 » UserCode 
> java.lang.IllegalArgumentException: A ...
> [ERROR]   SpannerIOTest.singleMutationPipeline » UncheckedExecution 
> org.apache.beam.sdk
> {code}
> These tests fail with the same reason, here's the complete stack trace:
> {code}
> [ERROR] batchingGroups(org.apache.beam.sdk.io.gcp.spanner.SpannerIOTest)  
> Time elapsed: 0.004 s  <<< ERROR!
> org.apache.beam.sdk.util.UserCodeException: 
> java.lang.IllegalArgumentException: A project ID is required for this service 
> but could not be determined from the builder or the environment.  Please set 
> a project ID using the builder.
> at 
> org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:36)
> at 
> org.apache.beam.sdk.io.gcp.spanner.SpannerIO$SpannerWriteFn$DoFnInvoker.invokeSetup(Unknown
>  Source)
> at 
> org.apache.beam.sdk.transforms.DoFnTester.initializeState(DoFnTester.java:745)
> at 
> org.apache.beam.sdk.transforms.DoFnTester.startBundle(DoFnTester.java:219)
> at 
> org.apache.beam.sdk.transforms.DoFnTester.processBundle(DoFnTester.java:183)
> at 
> org.apache.beam.sdk.io.gcp.spanner.SpannerIOTest.batchingGroups(SpannerIOTest.java:155)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at 
> org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:321)
> at 
> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:239)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runners.Suite.runChild(Suite.java:128)
> at org.junit.runners.Suite.runChild(Suite.java:27)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at 
> org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55)
> at 
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137)
> at 
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107)
> at 
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:83)
> 

[jira] [Updated] (BEAM-2379) SpannerIO tests are failing

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-2379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré updated BEAM-2379:
---
Component/s: testing

> SpannerIO tests are failing
> ---
>
> Key: BEAM-2379
> URL: https://issues.apache.org/jira/browse/BEAM-2379
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions, testing
>Reporter: Jean-Baptiste Onofré
>Assignee: Mairbek Khadikov
> Fix For: 2.1.0
>
>
> When performing a complete {{mvn clean install}}, the build is failing on 
> {{SpannerIO}} tests:
> {code}
> [ERROR] Errors: 
> [ERROR]   SpannerIOTest.batching:129 » UserCode 
> java.lang.IllegalArgumentException: A pr...
> [ERROR]   SpannerIOTest.batchingGroups:155 » UserCode 
> java.lang.IllegalArgumentException...
> [ERROR]   SpannerIOTest.noBatching:178 » UserCode 
> java.lang.IllegalArgumentException: A ...
> [ERROR]   SpannerIOTest.singleMutationPipeline » UncheckedExecution 
> org.apache.beam.sdk
> {code}
> These tests fail with the same reason, here's the complete stack trace:
> {code}
> [ERROR] batchingGroups(org.apache.beam.sdk.io.gcp.spanner.SpannerIOTest)  
> Time elapsed: 0.004 s  <<< ERROR!
> org.apache.beam.sdk.util.UserCodeException: 
> java.lang.IllegalArgumentException: A project ID is required for this service 
> but could not be determined from the builder or the environment.  Please set 
> a project ID using the builder.
> at 
> org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:36)
> at 
> org.apache.beam.sdk.io.gcp.spanner.SpannerIO$SpannerWriteFn$DoFnInvoker.invokeSetup(Unknown
>  Source)
> at 
> org.apache.beam.sdk.transforms.DoFnTester.initializeState(DoFnTester.java:745)
> at 
> org.apache.beam.sdk.transforms.DoFnTester.startBundle(DoFnTester.java:219)
> at 
> org.apache.beam.sdk.transforms.DoFnTester.processBundle(DoFnTester.java:183)
> at 
> org.apache.beam.sdk.io.gcp.spanner.SpannerIOTest.batchingGroups(SpannerIOTest.java:155)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at 
> org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:321)
> at 
> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:239)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runners.Suite.runChild(Suite.java:128)
> at org.junit.runners.Suite.runChild(Suite.java:27)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at 
> org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55)
> at 
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137)
> at 
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107)
> at 
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:83)
> 

[jira] [Commented] (BEAM-2379) SpannerIO tests are failing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030040#comment-16030040
 ] 

ASF GitHub Bot commented on BEAM-2379:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3259


> SpannerIO tests are failing
> ---
>
> Key: BEAM-2379
> URL: https://issues.apache.org/jira/browse/BEAM-2379
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Jean-Baptiste Onofré
>Assignee: Mairbek Khadikov
> Fix For: Not applicable
>
>
> When performing a complete {{mvn clean install}}, the build is failing on 
> {{SpannerIO}} tests:
> {code}
> [ERROR] Errors: 
> [ERROR]   SpannerIOTest.batching:129 » UserCode 
> java.lang.IllegalArgumentException: A pr...
> [ERROR]   SpannerIOTest.batchingGroups:155 » UserCode 
> java.lang.IllegalArgumentException...
> [ERROR]   SpannerIOTest.noBatching:178 » UserCode 
> java.lang.IllegalArgumentException: A ...
> [ERROR]   SpannerIOTest.singleMutationPipeline » UncheckedExecution 
> org.apache.beam.sdk
> {code}
> These tests fail with the same reason, here's the complete stack trace:
> {code}
> [ERROR] batchingGroups(org.apache.beam.sdk.io.gcp.spanner.SpannerIOTest)  
> Time elapsed: 0.004 s  <<< ERROR!
> org.apache.beam.sdk.util.UserCodeException: 
> java.lang.IllegalArgumentException: A project ID is required for this service 
> but could not be determined from the builder or the environment.  Please set 
> a project ID using the builder.
> at 
> org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:36)
> at 
> org.apache.beam.sdk.io.gcp.spanner.SpannerIO$SpannerWriteFn$DoFnInvoker.invokeSetup(Unknown
>  Source)
> at 
> org.apache.beam.sdk.transforms.DoFnTester.initializeState(DoFnTester.java:745)
> at 
> org.apache.beam.sdk.transforms.DoFnTester.startBundle(DoFnTester.java:219)
> at 
> org.apache.beam.sdk.transforms.DoFnTester.processBundle(DoFnTester.java:183)
> at 
> org.apache.beam.sdk.io.gcp.spanner.SpannerIOTest.batchingGroups(SpannerIOTest.java:155)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at 
> org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:321)
> at 
> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:239)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runners.Suite.runChild(Suite.java:128)
> at org.junit.runners.Suite.runChild(Suite.java:27)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at 
> org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55)
> at 
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137)
> at 
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107)
> at 

[GitHub] beam pull request #3259: [BEAM-2379] Avoid reading projectId from environmen...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3259


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: [BEAM-2379] Avoid reading projectId from environment variable in tests. SpannerOptions.Builder requires projectId to be not-null.

[jbonofre]

Repository: beam
Updated Branches:
  refs/heads/master aff406b89 -> 2d3e9fe75


[BEAM-2379] Avoid reading projectId from environment variable in tests. 
SpannerOptions.Builder requires projectId to be not-null.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/84a831e3
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/84a831e3
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/84a831e3

Branch: refs/heads/master
Commit: 84a831e3f3ea7f476a28778468cb084f5eecac68
Parents: aff406b
Author: Mairbek Khadikov 
Authored: Tue May 30 11:54:55 2017 -0700
Committer: Jean-Baptiste Onofré 
Committed: Tue May 30 21:51:20 2017 +0200

--
 .../beam/sdk/io/gcp/spanner/SpannerIO.java  | 27 +++-
 1 file changed, 15 insertions(+), 12 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/84a831e3/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIO.java
--
diff --git 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIO.java
 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIO.java
index c5325bb..5058d13 100644
--- 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIO.java
+++ 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIO.java
@@ -149,14 +149,6 @@ public class SpannerIO {
   abstract Write build();
 }
 
-SpannerOptions getSpannerOptions() {
-  SpannerOptions.Builder builder = SpannerOptions.newBuilder();
-  if (getServiceFactory() != null) {
-builder.setServiceFactory(getServiceFactory());
-  }
-  return builder.build();
-}
-
 /**
  * Returns a new {@link SpannerIO.Write} that will write to the specified 
Cloud Spanner project.
  *
@@ -259,10 +251,10 @@ public class SpannerIO {
 
 @Setup
 public void setup() throws Exception {
-  spanner = spec.getSpannerOptions().getService();
-  dbClient =
-  spanner.getDatabaseClient(
-  DatabaseId.of(projectId(), spec.getInstanceId(), 
spec.getDatabaseId()));
+  SpannerOptions spannerOptions = getSpannerOptions();
+  spanner = spannerOptions.getService();
+  dbClient = spanner.getDatabaseClient(
+  DatabaseId.of(projectId(), spec.getInstanceId(), 
spec.getDatabaseId()));
   mutations = new ArrayList<>();
   batchSizeBytes = 0;
 }
@@ -299,6 +291,17 @@ public class SpannerIO {
   spanner = null;
 }
 
+private SpannerOptions getSpannerOptions() {
+  SpannerOptions.Builder spannerOptionsBuider = 
SpannerOptions.newBuilder();
+  if (spec.getServiceFactory() != null) {
+spannerOptionsBuider.setServiceFactory(spec.getServiceFactory());
+  }
+  if (spec.getProjectId() != null) {
+spannerOptionsBuider.setProjectId(spec.getProjectId());
+  }
+  return spannerOptionsBuider.build();
+}
+
 /**
  * Writes a batch of mutations to Cloud Spanner.
  *

[2/2] beam git commit: [BEAM-2379] This closes #3259

[jbonofre]

[BEAM-2379] This closes #3259


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/2d3e9fe7
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/2d3e9fe7
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/2d3e9fe7

Branch: refs/heads/master
Commit: 2d3e9fe75bf3a6524a1f72db13da0e42ea15beab
Parents: aff406b 84a831e
Author: Jean-Baptiste Onofré 
Authored: Tue May 30 22:08:05 2017 +0200
Committer: Jean-Baptiste Onofré 
Committed: Tue May 30 22:08:05 2017 +0200

--
 .../beam/sdk/io/gcp/spanner/SpannerIO.java  | 27 +++-
 1 file changed, 15 insertions(+), 12 deletions(-)
--

[jira] [Created] (BEAM-2388) Built-In I/Os page - "discoverability"/"supported with adapters" section

[Stephen Sisk (JIRA)]

Stephen Sisk created BEAM-2388:
--

 Summary: Built-In I/Os page - "discoverability"/"supported with 
adapters" section
 Key: BEAM-2388
 URL: https://issues.apache.org/jira/browse/BEAM-2388
 Project: Beam
  Issue Type: Improvement
  Components: website
Reporter: Stephen Sisk


>From email discussion: [1]

The recent inclusion of HadoopInputFormatIO (HiFiIO) gave Beam users
the option to ‘easily’ include data stores that support the
Hadoop-based partitioning scheme.  ...  there are visibility issues since end 
users would need to repackage and discover the supported (and tested) 
HiFi-based IOs that won’t be explicit in the code base.

...

We might add a section to the Built-in IO Transforms page that covers 
"non-native but readable" IOs (better name suggestions appreciated :) - that 
could include a list of data stores that jdbc/jms/hifio support and link to 
HIFIO's info on how to use them. (That might also be a good place to document 
the performance tradeoffs of using HIFIO)

[1]https://lists.apache.org/thread.html/063b1dee9df43a0ba195c3921dd022b812a4e382be8173e32f84bf66@%3Cdev.beam.apache.org%3E



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2379) SpannerIO tests are failing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16029971#comment-16029971
 ] 

ASF GitHub Bot commented on BEAM-2379:
--

GitHub user mairbek opened a pull request:

https://github.com/apache/beam/pull/3259

 [BEAM-2379] Avoid reading projectId from environment variable in tests.

SpannerOptions.Builder requires projectId to be not-null.

Was green because I had GOOGLE_APPLICATION_CREDENTIALS configured on my dev 
machine. Same for Jenkins, I guess.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mairbek/beam fixtest

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3259.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3259


commit 4a4551a0cebfbec3f5f63b115f4e21e94163b979
Author: Mairbek Khadikov 
Date:   2017-05-30T18:54:55Z

Avoid reading projectId from environment variable in tests.

SpannerOptions.Builder requires projectId to be not-null.




> SpannerIO tests are failing
> ---
>
> Key: BEAM-2379
> URL: https://issues.apache.org/jira/browse/BEAM-2379
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Jean-Baptiste Onofré
>Assignee: Mairbek Khadikov
> Fix For: Not applicable
>
>
> When performing a complete {{mvn clean install}}, the build is failing on 
> {{SpannerIO}} tests:
> {code}
> [ERROR] Errors: 
> [ERROR]   SpannerIOTest.batching:129 » UserCode 
> java.lang.IllegalArgumentException: A pr...
> [ERROR]   SpannerIOTest.batchingGroups:155 » UserCode 
> java.lang.IllegalArgumentException...
> [ERROR]   SpannerIOTest.noBatching:178 » UserCode 
> java.lang.IllegalArgumentException: A ...
> [ERROR]   SpannerIOTest.singleMutationPipeline » UncheckedExecution 
> org.apache.beam.sdk
> {code}
> These tests fail with the same reason, here's the complete stack trace:
> {code}
> [ERROR] batchingGroups(org.apache.beam.sdk.io.gcp.spanner.SpannerIOTest)  
> Time elapsed: 0.004 s  <<< ERROR!
> org.apache.beam.sdk.util.UserCodeException: 
> java.lang.IllegalArgumentException: A project ID is required for this service 
> but could not be determined from the builder or the environment.  Please set 
> a project ID using the builder.
> at 
> org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:36)
> at 
> org.apache.beam.sdk.io.gcp.spanner.SpannerIO$SpannerWriteFn$DoFnInvoker.invokeSetup(Unknown
>  Source)
> at 
> org.apache.beam.sdk.transforms.DoFnTester.initializeState(DoFnTester.java:745)
> at 
> org.apache.beam.sdk.transforms.DoFnTester.startBundle(DoFnTester.java:219)
> at 
> org.apache.beam.sdk.transforms.DoFnTester.processBundle(DoFnTester.java:183)
> at 
> org.apache.beam.sdk.io.gcp.spanner.SpannerIOTest.batchingGroups(SpannerIOTest.java:155)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at 
> org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:321)
> at 
> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:239)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at 

[GitHub] beam pull request #3259: [BEAM-2379] Avoid reading projectId from environmen...

[mairbek]

GitHub user mairbek opened a pull request:

https://github.com/apache/beam/pull/3259

 [BEAM-2379] Avoid reading projectId from environment variable in tests.

SpannerOptions.Builder requires projectId to be not-null.

Was green because I had GOOGLE_APPLICATION_CREDENTIALS configured on my dev 
machine. Same for Jenkins, I guess.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mairbek/beam fixtest

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3259.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3259


commit 4a4551a0cebfbec3f5f63b115f4e21e94163b979
Author: Mairbek Khadikov 
Date:   2017-05-30T18:54:55Z

Avoid reading projectId from environment variable in tests.

SpannerOptions.Builder requires projectId to be not-null.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #3962

[Apache Jenkins Server]

See 


--
[...truncated 3.06 MB...]
2017-05-30T19:14:26.312 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-core/1.2.1/flink-core-1.2.1-tests.jar
2017-05-30T19:14:26.341 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/sling/org.apache.sling.commons.json/2.0.6/org.apache.sling.commons.json-2.0.6.jar
 (47 KB at 52.2 KB/sec)
2017-05-30T19:14:26.341 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-runtime_2.10/1.2.1/flink-runtime_2.10-1.2.1-tests.jar
2017-05-30T19:14:26.411 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-core/1.2.1/flink-core-1.2.1-tests.jar
 (716 KB at 741.0 KB/sec)
2017-05-30T19:14:26.411 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-streaming-java_2.10/1.2.1/flink-streaming-java_2.10-1.2.1-tests.jar
2017-05-30T19:14:26.428 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/2.10.4/scala-library-2.10.4.jar
 (6960 KB at 7079.7 KB/sec)
2017-05-30T19:14:26.429 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils_2.10/1.2.1/flink-test-utils_2.10-1.2.1.jar
2017-05-30T19:14:26.516 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-streaming-java_2.10/1.2.1/flink-streaming-java_2.10-1.2.1-tests.jar
 (953 KB at 889.7 KB/sec)
2017-05-30T19:14:26.516 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-junit/1.2.1/flink-test-utils-junit-1.2.1.jar
2017-05-30T19:14:26.544 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-junit/1.2.1/flink-test-utils-junit-1.2.1.jar
 (24 KB at 20.9 KB/sec)
2017-05-30T19:14:26.544 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-test/2.8.0/curator-test-2.8.0.jar
2017-05-30T19:14:26.571 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-runtime_2.10/1.2.1/flink-runtime_2.10-1.2.1-tests.jar
 (2432 KB at 2159.6 KB/sec)
2017-05-30T19:14:26.572 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-streaming-java_2.10/1.2.1/flink-streaming-java_2.10-1.2.1.jar
 (3035 KB at 2692.3 KB/sec)
2017-05-30T19:14:26.601 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-test/2.8.0/curator-test-2.8.0.jar
 (39 KB at 33.7 KB/sec)
2017-05-30T19:14:26.666 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils_2.10/1.2.1/flink-test-utils_2.10-1.2.1.jar
 (2366 KB at 1937.5 KB/sec)
2017-05-30T19:14:26.962 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-shaded-hadoop2/1.2.1/flink-shaded-hadoop2-1.2.1.jar
 (17860 KB at 11772.8 KB/sec)
2017-05-30T19:14:26.974 [INFO] 
2017-05-30T19:14:26.974 [INFO] --- maven-clean-plugin:3.0.0:clean 
(default-clean) @ beam-runners-flink_2.10 ---
2017-05-30T19:14:26.975 [INFO] Deleting 

 (includes = [**/*.pyc, **/*.egg-info/, **/sdks/python/LICENSE, 
**/sdks/python/NOTICE, **/sdks/python/README.md], excludes = [])
2017-05-30T19:14:27.050 [INFO] 
2017-05-30T19:14:27.050 [INFO] --- maven-enforcer-plugin:1.4.1:enforce 
(enforce) @ beam-runners-flink_2.10 ---
2017-05-30T19:14:29.217 [INFO] 
2017-05-30T19:14:29.217 [INFO] --- maven-enforcer-plugin:1.4.1:enforce 
(enforce-banned-dependencies) @ beam-runners-flink_2.10 ---
2017-05-30T19:14:29.286 [INFO] 
2017-05-30T19:14:29.286 [INFO] --- maven-remote-resources-plugin:1.5:process 
(process-resource-bundles) @ beam-runners-flink_2.10 ---
2017-05-30T19:14:30.011 [INFO] 
2017-05-30T19:14:30.011 [INFO] --- maven-resources-plugin:3.0.2:resources 
(default-resources) @ beam-runners-flink_2.10 ---
2017-05-30T19:14:30.013 [INFO] Using 'UTF-8' encoding to copy filtered 
resources.
2017-05-30T19:14:30.013 [INFO] Copying 1 resource
2017-05-30T19:14:30.013 [INFO] Copying 3 resources
2017-05-30T19:14:30.102 [INFO] 
2017-05-30T19:14:30.102 [INFO] --- maven-compiler-plugin:3.6.1:compile 
(default-compile) @ beam-runners-flink_2.10 ---
2017-05-30T19:14:30.116 [INFO] Changes detected - recompiling the module!
2017-05-30T19:14:30.117 [INFO] Compiling 75 source files to 

2017-05-30T19:14:30.914 [WARNING] bootstrap class path not set in conjunction 
with -source 1.7
2017-05-30T19:14:30.914 [INFO] 
:
 Some input files use or override a deprecated API.
2017-05-30T19:14:30.914 [INFO] 

[jira] [Created] (BEAM-2387) Add Top Natural and Reversed

[Ahmet Altay (JIRA)]

Ahmet Altay created BEAM-2387:
-

 Summary: Add Top Natural and Reversed
 Key: BEAM-2387
 URL: https://issues.apache.org/jira/browse/BEAM-2387
 Project: Beam
  Issue Type: Bug
  Components: sdk-py
Reporter: Ahmet Altay


See https://github.com/apache/beam/pull/3139 for changes in Java SDK.

cc: [~wtanaka]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2379) SpannerIO tests are failing

[Daniel Halperin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16029965#comment-16029965
 ] 

Daniel Halperin commented on BEAM-2379:
---

Feel free to rollback if fix is not coming quickly. [~jbonofre]

> SpannerIO tests are failing
> ---
>
> Key: BEAM-2379
> URL: https://issues.apache.org/jira/browse/BEAM-2379
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Jean-Baptiste Onofré
>Assignee: Mairbek Khadikov
> Fix For: Not applicable
>
>
> When performing a complete {{mvn clean install}}, the build is failing on 
> {{SpannerIO}} tests:
> {code}
> [ERROR] Errors: 
> [ERROR]   SpannerIOTest.batching:129 » UserCode 
> java.lang.IllegalArgumentException: A pr...
> [ERROR]   SpannerIOTest.batchingGroups:155 » UserCode 
> java.lang.IllegalArgumentException...
> [ERROR]   SpannerIOTest.noBatching:178 » UserCode 
> java.lang.IllegalArgumentException: A ...
> [ERROR]   SpannerIOTest.singleMutationPipeline » UncheckedExecution 
> org.apache.beam.sdk
> {code}
> These tests fail with the same reason, here's the complete stack trace:
> {code}
> [ERROR] batchingGroups(org.apache.beam.sdk.io.gcp.spanner.SpannerIOTest)  
> Time elapsed: 0.004 s  <<< ERROR!
> org.apache.beam.sdk.util.UserCodeException: 
> java.lang.IllegalArgumentException: A project ID is required for this service 
> but could not be determined from the builder or the environment.  Please set 
> a project ID using the builder.
> at 
> org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:36)
> at 
> org.apache.beam.sdk.io.gcp.spanner.SpannerIO$SpannerWriteFn$DoFnInvoker.invokeSetup(Unknown
>  Source)
> at 
> org.apache.beam.sdk.transforms.DoFnTester.initializeState(DoFnTester.java:745)
> at 
> org.apache.beam.sdk.transforms.DoFnTester.startBundle(DoFnTester.java:219)
> at 
> org.apache.beam.sdk.transforms.DoFnTester.processBundle(DoFnTester.java:183)
> at 
> org.apache.beam.sdk.io.gcp.spanner.SpannerIOTest.batchingGroups(SpannerIOTest.java:155)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at 
> org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:321)
> at 
> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:239)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runners.Suite.runChild(Suite.java:128)
> at org.junit.runners.Suite.runChild(Suite.java:27)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at 
> org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55)
> at 
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137)
> at 
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107)
> at 
> 

[jira] [Created] (BEAM-2386) Change regex used for splitting words

[Ahmet Altay (JIRA)]

Ahmet Altay created BEAM-2386:
-

 Summary: Change regex used for splitting words
 Key: BEAM-2386
 URL: https://issues.apache.org/jira/browse/BEAM-2386
 Project: Beam
  Issue Type: Bug
  Components: sdk-py
Reporter: Ahmet Altay
Priority: Minor


Regex used in splitting words ({{[A-Za-z\']+}}) only works on latin input, 
change it to make it work on non-latin inputs.

For example, see Java version:
https://github.com/apache/beam/blob/367fcb28d544934797d25cb34d54136b2d7d6e99/examples/java/src/main/java/org/apache/beam/examples/common/ExampleUtils.java#L75



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2379) SpannerIO tests are failing

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-2379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16029918#comment-16029918
 ] 

Jean-Baptiste Onofré commented on BEAM-2379:


By the way, it also impacts contributors (not only Jenkins).

[~mkhadikov] I worked on a fix (for my machine), if you want to take a look. Up 
to you !

> SpannerIO tests are failing
> ---
>
> Key: BEAM-2379
> URL: https://issues.apache.org/jira/browse/BEAM-2379
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Jean-Baptiste Onofré
>Assignee: Mairbek Khadikov
> Fix For: Not applicable
>
>
> When performing a complete {{mvn clean install}}, the build is failing on 
> {{SpannerIO}} tests:
> {code}
> [ERROR] Errors: 
> [ERROR]   SpannerIOTest.batching:129 » UserCode 
> java.lang.IllegalArgumentException: A pr...
> [ERROR]   SpannerIOTest.batchingGroups:155 » UserCode 
> java.lang.IllegalArgumentException...
> [ERROR]   SpannerIOTest.noBatching:178 » UserCode 
> java.lang.IllegalArgumentException: A ...
> [ERROR]   SpannerIOTest.singleMutationPipeline » UncheckedExecution 
> org.apache.beam.sdk
> {code}
> These tests fail with the same reason, here's the complete stack trace:
> {code}
> [ERROR] batchingGroups(org.apache.beam.sdk.io.gcp.spanner.SpannerIOTest)  
> Time elapsed: 0.004 s  <<< ERROR!
> org.apache.beam.sdk.util.UserCodeException: 
> java.lang.IllegalArgumentException: A project ID is required for this service 
> but could not be determined from the builder or the environment.  Please set 
> a project ID using the builder.
> at 
> org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:36)
> at 
> org.apache.beam.sdk.io.gcp.spanner.SpannerIO$SpannerWriteFn$DoFnInvoker.invokeSetup(Unknown
>  Source)
> at 
> org.apache.beam.sdk.transforms.DoFnTester.initializeState(DoFnTester.java:745)
> at 
> org.apache.beam.sdk.transforms.DoFnTester.startBundle(DoFnTester.java:219)
> at 
> org.apache.beam.sdk.transforms.DoFnTester.processBundle(DoFnTester.java:183)
> at 
> org.apache.beam.sdk.io.gcp.spanner.SpannerIOTest.batchingGroups(SpannerIOTest.java:155)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at 
> org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:321)
> at 
> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:239)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runners.Suite.runChild(Suite.java:128)
> at org.junit.runners.Suite.runChild(Suite.java:27)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at 
> org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55)
> at 
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137)
> at 
> 

[jira] [Commented] (BEAM-2379) SpannerIO tests are failing

[Mairbek Khadikov (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16029909#comment-16029909
 ] 

Mairbek Khadikov commented on BEAM-2379:


Will send a fix PR

> SpannerIO tests are failing
> ---
>
> Key: BEAM-2379
> URL: https://issues.apache.org/jira/browse/BEAM-2379
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Jean-Baptiste Onofré
>Assignee: Mairbek Khadikov
> Fix For: Not applicable
>
>
> When performing a complete {{mvn clean install}}, the build is failing on 
> {{SpannerIO}} tests:
> {code}
> [ERROR] Errors: 
> [ERROR]   SpannerIOTest.batching:129 » UserCode 
> java.lang.IllegalArgumentException: A pr...
> [ERROR]   SpannerIOTest.batchingGroups:155 » UserCode 
> java.lang.IllegalArgumentException...
> [ERROR]   SpannerIOTest.noBatching:178 » UserCode 
> java.lang.IllegalArgumentException: A ...
> [ERROR]   SpannerIOTest.singleMutationPipeline » UncheckedExecution 
> org.apache.beam.sdk
> {code}
> These tests fail with the same reason, here's the complete stack trace:
> {code}
> [ERROR] batchingGroups(org.apache.beam.sdk.io.gcp.spanner.SpannerIOTest)  
> Time elapsed: 0.004 s  <<< ERROR!
> org.apache.beam.sdk.util.UserCodeException: 
> java.lang.IllegalArgumentException: A project ID is required for this service 
> but could not be determined from the builder or the environment.  Please set 
> a project ID using the builder.
> at 
> org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:36)
> at 
> org.apache.beam.sdk.io.gcp.spanner.SpannerIO$SpannerWriteFn$DoFnInvoker.invokeSetup(Unknown
>  Source)
> at 
> org.apache.beam.sdk.transforms.DoFnTester.initializeState(DoFnTester.java:745)
> at 
> org.apache.beam.sdk.transforms.DoFnTester.startBundle(DoFnTester.java:219)
> at 
> org.apache.beam.sdk.transforms.DoFnTester.processBundle(DoFnTester.java:183)
> at 
> org.apache.beam.sdk.io.gcp.spanner.SpannerIOTest.batchingGroups(SpannerIOTest.java:155)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at 
> org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:321)
> at 
> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:239)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runners.Suite.runChild(Suite.java:128)
> at org.junit.runners.Suite.runChild(Suite.java:27)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at 
> org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55)
> at 
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137)
> at 
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107)
> at 
> 

[jira] [Commented] (BEAM-2379) SpannerIO tests are failing

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16029903#comment-16029903
 ] 

Luke Cwik commented on BEAM-2379:
-

Decided to keep this open to fix the SpannerIO test issue and instead just 
linked a bug which is the cause of why this was not caught in Jenkins.

> SpannerIO tests are failing
> ---
>
> Key: BEAM-2379
> URL: https://issues.apache.org/jira/browse/BEAM-2379
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Jean-Baptiste Onofré
>Assignee: Mairbek Khadikov
> Fix For: Not applicable
>
>
> When performing a complete {{mvn clean install}}, the build is failing on 
> {{SpannerIO}} tests:
> {code}
> [ERROR] Errors: 
> [ERROR]   SpannerIOTest.batching:129 » UserCode 
> java.lang.IllegalArgumentException: A pr...
> [ERROR]   SpannerIOTest.batchingGroups:155 » UserCode 
> java.lang.IllegalArgumentException...
> [ERROR]   SpannerIOTest.noBatching:178 » UserCode 
> java.lang.IllegalArgumentException: A ...
> [ERROR]   SpannerIOTest.singleMutationPipeline » UncheckedExecution 
> org.apache.beam.sdk
> {code}
> These tests fail with the same reason, here's the complete stack trace:
> {code}
> [ERROR] batchingGroups(org.apache.beam.sdk.io.gcp.spanner.SpannerIOTest)  
> Time elapsed: 0.004 s  <<< ERROR!
> org.apache.beam.sdk.util.UserCodeException: 
> java.lang.IllegalArgumentException: A project ID is required for this service 
> but could not be determined from the builder or the environment.  Please set 
> a project ID using the builder.
> at 
> org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:36)
> at 
> org.apache.beam.sdk.io.gcp.spanner.SpannerIO$SpannerWriteFn$DoFnInvoker.invokeSetup(Unknown
>  Source)
> at 
> org.apache.beam.sdk.transforms.DoFnTester.initializeState(DoFnTester.java:745)
> at 
> org.apache.beam.sdk.transforms.DoFnTester.startBundle(DoFnTester.java:219)
> at 
> org.apache.beam.sdk.transforms.DoFnTester.processBundle(DoFnTester.java:183)
> at 
> org.apache.beam.sdk.io.gcp.spanner.SpannerIOTest.batchingGroups(SpannerIOTest.java:155)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at 
> org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:321)
> at 
> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:239)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runners.Suite.runChild(Suite.java:128)
> at org.junit.runners.Suite.runChild(Suite.java:27)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at 
> org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55)
> at 
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137)
> at 
> 

[jira] [Reopened] (BEAM-2379) SpannerIO tests are failing

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik reopened BEAM-2379:
-

> SpannerIO tests are failing
> ---
>
> Key: BEAM-2379
> URL: https://issues.apache.org/jira/browse/BEAM-2379
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Jean-Baptiste Onofré
>Assignee: Mairbek Khadikov
> Fix For: Not applicable
>
>
> When performing a complete {{mvn clean install}}, the build is failing on 
> {{SpannerIO}} tests:
> {code}
> [ERROR] Errors: 
> [ERROR]   SpannerIOTest.batching:129 » UserCode 
> java.lang.IllegalArgumentException: A pr...
> [ERROR]   SpannerIOTest.batchingGroups:155 » UserCode 
> java.lang.IllegalArgumentException...
> [ERROR]   SpannerIOTest.noBatching:178 » UserCode 
> java.lang.IllegalArgumentException: A ...
> [ERROR]   SpannerIOTest.singleMutationPipeline » UncheckedExecution 
> org.apache.beam.sdk
> {code}
> These tests fail with the same reason, here's the complete stack trace:
> {code}
> [ERROR] batchingGroups(org.apache.beam.sdk.io.gcp.spanner.SpannerIOTest)  
> Time elapsed: 0.004 s  <<< ERROR!
> org.apache.beam.sdk.util.UserCodeException: 
> java.lang.IllegalArgumentException: A project ID is required for this service 
> but could not be determined from the builder or the environment.  Please set 
> a project ID using the builder.
> at 
> org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:36)
> at 
> org.apache.beam.sdk.io.gcp.spanner.SpannerIO$SpannerWriteFn$DoFnInvoker.invokeSetup(Unknown
>  Source)
> at 
> org.apache.beam.sdk.transforms.DoFnTester.initializeState(DoFnTester.java:745)
> at 
> org.apache.beam.sdk.transforms.DoFnTester.startBundle(DoFnTester.java:219)
> at 
> org.apache.beam.sdk.transforms.DoFnTester.processBundle(DoFnTester.java:183)
> at 
> org.apache.beam.sdk.io.gcp.spanner.SpannerIOTest.batchingGroups(SpannerIOTest.java:155)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at 
> org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:321)
> at 
> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:239)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runners.Suite.runChild(Suite.java:128)
> at org.junit.runners.Suite.runChild(Suite.java:27)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at 
> org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55)
> at 
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137)
> at 
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107)
> at 
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:83)
> at 
> 

[jira] [Resolved] (BEAM-2379) SpannerIO tests are failing

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-2379.
-
   Resolution: Duplicate
Fix Version/s: Not applicable

> SpannerIO tests are failing
> ---
>
> Key: BEAM-2379
> URL: https://issues.apache.org/jira/browse/BEAM-2379
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Jean-Baptiste Onofré
>Assignee: Mairbek Khadikov
> Fix For: Not applicable
>
>
> When performing a complete {{mvn clean install}}, the build is failing on 
> {{SpannerIO}} tests:
> {code}
> [ERROR] Errors: 
> [ERROR]   SpannerIOTest.batching:129 » UserCode 
> java.lang.IllegalArgumentException: A pr...
> [ERROR]   SpannerIOTest.batchingGroups:155 » UserCode 
> java.lang.IllegalArgumentException...
> [ERROR]   SpannerIOTest.noBatching:178 » UserCode 
> java.lang.IllegalArgumentException: A ...
> [ERROR]   SpannerIOTest.singleMutationPipeline » UncheckedExecution 
> org.apache.beam.sdk
> {code}
> These tests fail with the same reason, here's the complete stack trace:
> {code}
> [ERROR] batchingGroups(org.apache.beam.sdk.io.gcp.spanner.SpannerIOTest)  
> Time elapsed: 0.004 s  <<< ERROR!
> org.apache.beam.sdk.util.UserCodeException: 
> java.lang.IllegalArgumentException: A project ID is required for this service 
> but could not be determined from the builder or the environment.  Please set 
> a project ID using the builder.
> at 
> org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:36)
> at 
> org.apache.beam.sdk.io.gcp.spanner.SpannerIO$SpannerWriteFn$DoFnInvoker.invokeSetup(Unknown
>  Source)
> at 
> org.apache.beam.sdk.transforms.DoFnTester.initializeState(DoFnTester.java:745)
> at 
> org.apache.beam.sdk.transforms.DoFnTester.startBundle(DoFnTester.java:219)
> at 
> org.apache.beam.sdk.transforms.DoFnTester.processBundle(DoFnTester.java:183)
> at 
> org.apache.beam.sdk.io.gcp.spanner.SpannerIOTest.batchingGroups(SpannerIOTest.java:155)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at 
> org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:321)
> at 
> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:239)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runners.Suite.runChild(Suite.java:128)
> at org.junit.runners.Suite.runChild(Suite.java:27)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at 
> org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55)
> at 
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137)
> at 
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107)
> at 
> 

[jira] [Updated] (BEAM-2131) Need Jenkins tests run outside of Google environments

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik updated BEAM-2131:

Description: Now that TravisCI no longer runs, we no longer have coverage 
for running tests which execute outside of a Google environment which means 
that application default credentials and google project will always be found 
and we will never test code paths for developers who have never setup any kind 
of Google cloud integration on their development machine.  (was: Now that 
TravisCI no longer runs, we no longer have coverage for running tests which 
execute outside of a Google environment which means that application default 
credentials will always be found and we will never test code paths for 
developers who have never setup any kind of Google cloud integration on their 
development machine.)

> Need Jenkins tests run outside of Google environments
> -
>
> Key: BEAM-2131
> URL: https://issues.apache.org/jira/browse/BEAM-2131
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-gcp
>Reporter: Luke Cwik
>Priority: Minor
>
> Now that TravisCI no longer runs, we no longer have coverage for running 
> tests which execute outside of a Google environment which means that 
> application default credentials and google project will always be found and 
> we will never test code paths for developers who have never setup any kind of 
> Google cloud integration on their development machine.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-2131) Need Jenkins tests run outside of Google environments

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik updated BEAM-2131:

Summary: Need Jenkins tests run outside of Google environments  (was: 
Google application default credentials always found, need tests run outside of 
Google environments)

> Need Jenkins tests run outside of Google environments
> -
>
> Key: BEAM-2131
> URL: https://issues.apache.org/jira/browse/BEAM-2131
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-gcp
>Reporter: Luke Cwik
>Priority: Minor
>
> Now that TravisCI no longer runs, we no longer have coverage for running 
> tests which execute outside of a Google environment which means that 
> application default credentials will always be found and we will never test 
> code paths for developers who have never setup any kind of Google cloud 
> integration on their development machine.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-2385) IO authoring overview - improve steps for beginners

[Stephen Sisk (JIRA)]

Stephen Sisk created BEAM-2385:
--

 Summary: IO authoring overview - improve steps for beginners
 Key: BEAM-2385
 URL: https://issues.apache.org/jira/browse/BEAM-2385
 Project: Beam
  Issue Type: Improvement
  Components: website
Reporter: Stephen Sisk
Assignee: Stephen Sisk


The I/O Authoring guide has "suggested steps for implementers" - it'd be good 
to add a couple things to that list:
1. Open a JIRA ticket
2. Update the In-Progress I/O Transforms list on the website to include the I/O

It might also be good to call out the procedure of opening a new branch for an 
I/O



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall_Windows #73

[Apache Jenkins Server]

See 


Changes:

[aljoscha.krettek] [BEAM-2380] Forward additional outputs to DoFnRunner in 
Flink Batch

[aljoscha.krettek] Fix flushing of pushed-back data in Flink Runner on +Inf 
watermark

[dhalperi] fix javadoc of View

--
[...truncated 351.53 KB...]
2017-05-30T18:03:26.271 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/errorprone/error_prone_annotations/2.0.15/error_prone_annotations-2.0.15.pom
2017-05-30T18:03:26.302 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/errorprone/error_prone_annotations/2.0.15/error_prone_annotations-2.0.15.pom
 (2 KB at 51.5 KB/sec)
2017-05-30T18:03:26.318 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/errorprone/error_prone_parent/2.0.15/error_prone_parent-2.0.15.pom
2017-05-30T18:03:26.349 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/errorprone/error_prone_parent/2.0.15/error_prone_parent-2.0.15.pom
 (6 KB at 161.5 KB/sec)
2017-05-30T18:03:26.349 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/code/findbugs/jsr305/3.0.1/jsr305-3.0.1.pom
2017-05-30T18:03:26.380 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/code/findbugs/jsr305/3.0.1/jsr305-3.0.1.pom
 (5 KB at 135.0 KB/sec)
2017-05-30T18:03:26.380 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-context/1.2.0/grpc-context-1.2.0.pom
2017-05-30T18:03:26.412 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-context/1.2.0/grpc-context-1.2.0.pom
 (2 KB at 53.3 KB/sec)
2017-05-30T18:03:26.412 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/instrumentation/instrumentation-api/0.3.0/instrumentation-api-0.3.0.pom
2017-05-30T18:03:26.443 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/instrumentation/instrumentation-api/0.3.0/instrumentation-api-0.3.0.pom
 (2 KB at 47.2 KB/sec)
2017-05-30T18:03:26.443 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-protobuf/1.2.0/grpc-protobuf-1.2.0.pom
2017-05-30T18:03:26.474 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-protobuf/1.2.0/grpc-protobuf-1.2.0.pom
 (3 KB at 83.6 KB/sec)
2017-05-30T18:03:26.474 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/protobuf/protobuf-java-util/3.2.0/protobuf-java-util-3.2.0.pom
2017-05-30T18:03:26.505 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/protobuf/protobuf-java-util/3.2.0/protobuf-java-util-3.2.0.pom
 (5 KB at 133.9 KB/sec)
2017-05-30T18:03:26.505 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/code/gson/gson/2.7/gson-2.7.pom
2017-05-30T18:03:26.537 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/code/gson/gson/2.7/gson-2.7.pom 
(2 KB at 44.1 KB/sec)
2017-05-30T18:03:26.537 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/code/gson/gson-parent/2.7/gson-parent-2.7.pom
2017-05-30T18:03:26.568 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/code/gson/gson-parent/2.7/gson-parent-2.7.pom
 (4 KB at 112.8 KB/sec)
2017-05-30T18:03:26.568 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-protobuf-lite/1.2.0/grpc-protobuf-lite-1.2.0.pom
2017-05-30T18:03:26.599 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-protobuf-lite/1.2.0/grpc-protobuf-lite-1.2.0.pom
 (3 KB at 66.9 KB/sec)
2017-05-30T18:03:26.599 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-stub/1.2.0/grpc-stub-1.2.0.pom
2017-05-30T18:03:26.630 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-stub/1.2.0/grpc-stub-1.2.0.pom
 (3 KB at 65.5 KB/sec)
2017-05-30T18:03:26.646 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/protobuf/protobuf-java/3.2.0/protobuf-java-3.2.0.jar
2017-05-30T18:03:26.646 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-core/1.2.0/grpc-core-1.2.0.jar
2017-05-30T18:03:26.646 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/errorprone/error_prone_annotations/2.0.15/error_prone_annotations-2.0.15.jar
2017-05-30T18:03:26.646 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/code/findbugs/jsr305/3.0.1/jsr305-3.0.1.jar
2017-05-30T18:03:26.646 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-context/1.2.0/grpc-context-1.2.0.jar
2017-05-30T18:03:26.677 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-context/1.2.0/grpc-context-1.2.0.jar
 (20 KB at 637.0 KB/sec)
2017-05-30T18:03:26.677 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/instrumentation/instrumentation-api/0.3.0/instrumentation-api-0.3.0.jar
2017-05-30T18:03:26.677 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/code/findbugs/jsr305/3.0.1/jsr305-3.0.1.jar
 (20 

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #3961

[Apache Jenkins Server]

See 


Changes:

[dhalperi] fix javadoc of View

--
[...truncated 3.07 MB...]
2017-05-30T17:43:25.032 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-core/1.2.1/flink-core-1.2.1-tests.jar
2017-05-30T17:43:25.049 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/sling/org.apache.sling.commons.json/2.0.6/org.apache.sling.commons.json-2.0.6.jar
 (47 KB at 30.4 KB/sec)
2017-05-30T17:43:25.049 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-runtime_2.10/1.2.1/flink-runtime_2.10-1.2.1-tests.jar
2017-05-30T17:43:25.167 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-core/1.2.1/flink-core-1.2.1-tests.jar
 (716 KB at 432.3 KB/sec)
2017-05-30T17:43:25.167 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-streaming-java_2.10/1.2.1/flink-streaming-java_2.10-1.2.1-tests.jar
2017-05-30T17:43:25.451 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-streaming-java_2.10/1.2.1/flink-streaming-java_2.10-1.2.1-tests.jar
 (953 KB at 491.4 KB/sec)
2017-05-30T17:43:25.451 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-runtime_2.10/1.2.1/flink-runtime_2.10-1.2.1-tests.jar
 (2432 KB at 1253.4 KB/sec)
2017-05-30T17:43:25.451 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils_2.10/1.2.1/flink-test-utils_2.10-1.2.1.jar
2017-05-30T17:43:25.451 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-junit/1.2.1/flink-test-utils-junit-1.2.1.jar
2017-05-30T17:43:25.482 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-junit/1.2.1/flink-test-utils-junit-1.2.1.jar
 (24 KB at 11.7 KB/sec)
2017-05-30T17:43:25.482 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-test/2.8.0/curator-test-2.8.0.jar
2017-05-30T17:43:25.516 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-test/2.8.0/curator-test-2.8.0.jar
 (39 KB at 19.4 KB/sec)
2017-05-30T17:43:25.754 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-streaming-java_2.10/1.2.1/flink-streaming-java_2.10-1.2.1.jar
 (3035 KB at 1352.7 KB/sec)
2017-05-30T17:43:25.760 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/2.10.4/scala-library-2.10.4.jar
 (6960 KB at 3094.4 KB/sec)
2017-05-30T17:43:25.836 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils_2.10/1.2.1/flink-test-utils_2.10-1.2.1.jar
 (2366 KB at 1017.9 KB/sec)
2017-05-30T17:43:26.572 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-shaded-hadoop2/1.2.1/flink-shaded-hadoop2-1.2.1.jar
 (17860 KB at 5834.5 KB/sec)
2017-05-30T17:43:26.590 [INFO] 
2017-05-30T17:43:26.590 [INFO] --- maven-clean-plugin:3.0.0:clean 
(default-clean) @ beam-runners-flink_2.10 ---
2017-05-30T17:43:26.592 [INFO] Deleting 

 (includes = [**/*.pyc, **/*.egg-info/, **/sdks/python/LICENSE, 
**/sdks/python/NOTICE, **/sdks/python/README.md], excludes = [])
2017-05-30T17:43:26.669 [INFO] 
2017-05-30T17:43:26.669 [INFO] --- maven-enforcer-plugin:1.4.1:enforce 
(enforce) @ beam-runners-flink_2.10 ---
2017-05-30T17:43:30.121 [INFO] 
2017-05-30T17:43:30.121 [INFO] --- maven-enforcer-plugin:1.4.1:enforce 
(enforce-banned-dependencies) @ beam-runners-flink_2.10 ---
2017-05-30T17:43:30.670 [INFO] 
2017-05-30T17:43:30.670 [INFO] --- maven-remote-resources-plugin:1.5:process 
(process-resource-bundles) @ beam-runners-flink_2.10 ---
2017-05-30T17:43:31.781 [INFO] 
2017-05-30T17:43:31.781 [INFO] --- maven-resources-plugin:3.0.2:resources 
(default-resources) @ beam-runners-flink_2.10 ---
2017-05-30T17:43:31.782 [INFO] Using 'UTF-8' encoding to copy filtered 
resources.
2017-05-30T17:43:31.783 [INFO] Copying 1 resource
2017-05-30T17:43:31.783 [INFO] Copying 3 resources
2017-05-30T17:43:31.958 [INFO] 
2017-05-30T17:43:31.958 [INFO] --- maven-compiler-plugin:3.6.1:compile 
(default-compile) @ beam-runners-flink_2.10 ---
2017-05-30T17:43:31.981 [INFO] Changes detected - recompiling the module!
2017-05-30T17:43:31.991 [INFO] Compiling 75 source files to 

2017-05-30T17:43:33.457 [WARNING] bootstrap class path not set in conjunction 
with -source 1.7
2017-05-30T17:43:33.457 [INFO] 
:
 Some input files use or override a deprecated API.
2017-05-30T17:43:33.457 [INFO] 

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #3960

[Apache Jenkins Server]

See 


Changes:

[aljoscha.krettek] [BEAM-2380] Forward additional outputs to DoFnRunner in 
Flink Batch

[aljoscha.krettek] Fix flushing of pushed-back data in Flink Runner on +Inf 
watermark

--
[...truncated 3.07 MB...]
2017-05-30T17:26:13.014 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-core/1.2.1/flink-core-1.2.1-tests.jar
2017-05-30T17:26:13.041 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/sling/org.apache.sling.commons.json/2.0.6/org.apache.sling.commons.json-2.0.6.jar
 (47 KB at 52.8 KB/sec)
2017-05-30T17:26:13.041 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-runtime_2.10/1.2.1/flink-runtime_2.10-1.2.1-tests.jar
2017-05-30T17:26:13.092 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-core/1.2.1/flink-core-1.2.1-tests.jar
 (716 KB at 764.8 KB/sec)
2017-05-30T17:26:13.092 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-streaming-java_2.10/1.2.1/flink-streaming-java_2.10-1.2.1-tests.jar
2017-05-30T17:26:13.206 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/2.10.4/scala-library-2.10.4.jar
 (6960 KB at 6628.0 KB/sec)
2017-05-30T17:26:13.206 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils_2.10/1.2.1/flink-test-utils_2.10-1.2.1.jar
2017-05-30T17:26:13.233 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-streaming-java_2.10/1.2.1/flink-streaming-java_2.10-1.2.1-tests.jar
 (953 KB at 884.7 KB/sec)
2017-05-30T17:26:13.233 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-junit/1.2.1/flink-test-utils-junit-1.2.1.jar
2017-05-30T17:26:13.248 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-streaming-java_2.10/1.2.1/flink-streaming-java_2.10-1.2.1.jar
 (3035 KB at 2778.6 KB/sec)
2017-05-30T17:26:13.248 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-test/2.8.0/curator-test-2.8.0.jar
2017-05-30T17:26:13.260 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-runtime_2.10/1.2.1/flink-runtime_2.10-1.2.1-tests.jar
 (2432 KB at 2202.6 KB/sec)
2017-05-30T17:26:13.261 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-junit/1.2.1/flink-test-utils-junit-1.2.1.jar
 (24 KB at 20.8 KB/sec)
2017-05-30T17:26:13.277 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-test/2.8.0/curator-test-2.8.0.jar
 (39 KB at 34.7 KB/sec)
2017-05-30T17:26:13.407 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils_2.10/1.2.1/flink-test-utils_2.10-1.2.1.jar
 (2366 KB at 1891.1 KB/sec)
2017-05-30T17:26:13.790 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-shaded-hadoop2/1.2.1/flink-shaded-hadoop2-1.2.1.jar
 (17860 KB at 10929.9 KB/sec)
2017-05-30T17:26:13.802 [INFO] 
2017-05-30T17:26:13.802 [INFO] --- maven-clean-plugin:3.0.0:clean 
(default-clean) @ beam-runners-flink_2.10 ---
2017-05-30T17:26:13.804 [INFO] Deleting 

 (includes = [**/*.pyc, **/*.egg-info/, **/sdks/python/LICENSE, 
**/sdks/python/NOTICE, **/sdks/python/README.md], excludes = [])
2017-05-30T17:26:13.954 [INFO] 
2017-05-30T17:26:13.954 [INFO] --- maven-enforcer-plugin:1.4.1:enforce 
(enforce) @ beam-runners-flink_2.10 ---
2017-05-30T17:26:16.121 [INFO] 
2017-05-30T17:26:16.121 [INFO] --- maven-enforcer-plugin:1.4.1:enforce 
(enforce-banned-dependencies) @ beam-runners-flink_2.10 ---
2017-05-30T17:26:16.442 [INFO] 
2017-05-30T17:26:16.442 [INFO] --- maven-remote-resources-plugin:1.5:process 
(process-resource-bundles) @ beam-runners-flink_2.10 ---
2017-05-30T17:26:18.890 [INFO] 
2017-05-30T17:26:18.890 [INFO] --- maven-resources-plugin:3.0.2:resources 
(default-resources) @ beam-runners-flink_2.10 ---
2017-05-30T17:26:18.892 [INFO] Using 'UTF-8' encoding to copy filtered 
resources.
2017-05-30T17:26:18.909 [INFO] Copying 1 resource
2017-05-30T17:26:18.910 [INFO] Copying 3 resources
2017-05-30T17:26:19.029 [INFO] 
2017-05-30T17:26:19.029 [INFO] --- maven-compiler-plugin:3.6.1:compile 
(default-compile) @ beam-runners-flink_2.10 ---
2017-05-30T17:26:19.091 [INFO] Changes detected - recompiling the module!
2017-05-30T17:26:19.091 [INFO] Compiling 75 source files to 

2017-05-30T17:26:19.999 [WARNING] bootstrap class path not set in conjunction 
with -source 1.7
2017-05-30T17:26:19.999 [INFO] 

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Flink #2973

[Apache Jenkins Server]

See 

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Flink #2972

[Apache Jenkins Server]

See 

[GitHub] beam pull request #3257: fix javadoc of View

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3257


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: fix javadoc of View

[dhalperi]

Repository: beam
Updated Branches:
  refs/heads/master 711faffef -> aff406b89


fix javadoc of View


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/dd79d5aa
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/dd79d5aa
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/dd79d5aa

Branch: refs/heads/master
Commit: dd79d5aac3b44a6cb9569d6fb4f30e10335ab66d
Parents: 711faff
Author: James Xu 
Authored: Tue May 30 11:27:32 2017 +0800
Committer: Dan Halperin 
Committed: Tue May 30 09:11:39 2017 -0700

--
 .../core/src/main/java/org/apache/beam/sdk/transforms/View.java | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/dd79d5aa/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/View.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/View.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/View.java
index bcbdb24..073c750 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/View.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/View.java
@@ -116,16 +116,17 @@ import org.apache.beam.sdk.values.PCollectionViews;
  * {@code
  * PCollection pages = ... // pages fit into memory
  * PCollection urlVisits = ... // very large collection
- * final PCollectionView> = urlToPage
+ * final PCollectionView> urlToPageView = pages
  * .apply(WithKeys.of( ... )) // extract the URL from the page
  * .apply(View.asMap());
  *
  * PCollection PageVisits = urlVisits
- * .apply(ParDo.withSideInputs(urlToPage)
+ * .apply(ParDo.withSideInputs(urlToPageView)
  * .of(new DoFn() {
  * {@literal @}Override
  * void processElement(ProcessContext context) {
  *   UrlVisit urlVisit = context.element();
+ *   Map urlToPage = context.sideInput(urlToPageView);
  *   Page page = urlToPage.get(urlVisit.getUrl());
  *   c.output(new PageVisit(page, urlVisit.getVisitData()));
  * }

[jira] [Commented] (BEAM-2380) Flink Batch Runner does not forward additional outputs to operator

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16029604#comment-16029604
 ] 

ASF GitHub Bot commented on BEAM-2380:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3258


> Flink Batch Runner does not forward additional outputs to operator
> --
>
> Key: BEAM-2380
> URL: https://issues.apache.org/jira/browse/BEAM-2380
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 2.1.0
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>Priority: Blocker
> Fix For: 2.1.0
>
>
> Recent changes to {{SimpleDoFnRunner}} made the output tag validation more 
> strict in {{outputWindowedValue()}}. This surfaced a bug in the Flink Batch 
> runner: it doesn't forward the additional output tags to the 
> {{SimpleDoFnRunner}}.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[2/3] beam git commit: Fix flushing of pushed-back data in Flink Runner on +Inf watermark

[dhalperi]

Fix flushing of pushed-back data in Flink Runner on +Inf watermark

Before, we only flushed all pushed-back data when receiving a +Inf
watermark on the side input. It can happen that we receive that
watermark before getting any data on the main input. This changes
DoFnOperator to also flush when receiving a main-input watermark and we
happen to have already received the +Inf watermark on the side input.

Some tests where Flaky because of this. One example was
ViewTest.testWindowedSideInputFixedToFixedWithDefault().


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/838035a4
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/838035a4
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/838035a4

Branch: refs/heads/master
Commit: 838035a4069b152143859e9b34570b15254d69b3
Parents: 9afe395
Author: Aljoscha Krettek 
Authored: Tue May 30 15:19:27 2017 +0200
Committer: Aljoscha Krettek 
Committed: Tue May 30 15:23:55 2017 +0200

--
 .../wrappers/streaming/DoFnOperator.java| 77 +++-
 1 file changed, 60 insertions(+), 17 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/838035a4/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
--
diff --git 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
index d2ab7e1..e473046 100644
--- 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
+++ 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
@@ -19,6 +19,7 @@ package 
org.apache.beam.runners.flink.translation.wrappers.streaming;
 
 import static org.apache.flink.util.Preconditions.checkArgument;
 
+import com.google.common.base.Joiner;
 import com.google.common.base.Optional;
 import com.google.common.collect.Iterables;
 import java.io.DataInputStream;
@@ -129,6 +130,8 @@ public class DoFnOperator
 
   protected transient long currentInputWatermark;
 
+  protected transient long currentSideInputWatermark;
+
   protected transient long currentOutputWatermark;
 
   private transient StateTag> pushedBackTag;
@@ -197,6 +200,7 @@ public class DoFnOperator
 super.open();
 
 setCurrentInputWatermark(BoundedWindow.TIMESTAMP_MIN_VALUE.getMillis());
+
setCurrentSideInputWatermark(BoundedWindow.TIMESTAMP_MIN_VALUE.getMillis());
 setCurrentOutputWatermark(BoundedWindow.TIMESTAMP_MIN_VALUE.getMillis());
 
 sideInputReader = NullSideInputReader.of(sideInputs);
@@ -308,6 +312,21 @@ public class DoFnOperator
   @Override
   public void close() throws Exception {
 super.close();
+
+// sanity check: these should have been flushed out by +Inf watermarks
+if (pushbackStateInternals != null) {
+  BagState pushedBack =
+  pushbackStateInternals.state(StateNamespaces.global(), 
pushedBackTag);
+
+  Iterable pushedBackContents = pushedBack.read();
+  if (pushedBackContents != null) {
+if (!Iterables.isEmpty(pushedBackContents)) {
+  String pushedBackString = Joiner.on(",").join(pushedBackContents);
+  throw new RuntimeException(
+  "Leftover pushed-back data: " + pushedBackString + ". This 
indicates a bug.");
+}
+  }
+}
 doFnInvoker.invokeTeardown();
   }
 
@@ -457,36 +476,56 @@ public class DoFnOperator
   }
   pushbackDoFnRunner.finishBundle();
 }
+
+// We do the check here because we are guaranteed to at least get the +Inf 
watermark on the
+// main input when the job finishes.
+if (currentSideInputWatermark >= 
BoundedWindow.TIMESTAMP_MAX_VALUE.getMillis()) {
+  // this means we will never see any more side input
+  // we also do the check here because we might have received the 
side-input MAX watermark
+  // before receiving any main-input data
+  emitAllPushedBackData();
+}
   }
 
   @Override
   public void processWatermark2(Watermark mark) throws Exception {
-if (mark.getTimestamp() == BoundedWindow.TIMESTAMP_MAX_VALUE.getMillis()) {
+setCurrentSideInputWatermark(mark.getTimestamp());
+if (mark.getTimestamp() >= BoundedWindow.TIMESTAMP_MAX_VALUE.getMillis()) {
   // this means we will never see any more side input
-  pushbackDoFnRunner.startBundle();
+  emitAllPushedBackData();
 
-  

[GitHub] beam pull request #3258: [BEAM-2380] Forward additional outputs to DoFnRunne...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3258


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[3/3] beam git commit: This closes #3258

[dhalperi]

This closes #3258


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/711faffe
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/711faffe
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/711faffe

Branch: refs/heads/master
Commit: 711faffefeb91052efc0e3f45f7c9fb42df05151
Parents: 2cb4b03 838035a
Author: Dan Halperin 
Authored: Tue May 30 08:58:36 2017 -0700
Committer: Dan Halperin 
Committed: Tue May 30 08:58:36 2017 -0700

--
 .../functions/FlinkDoFnFunction.java|  8 +-
 .../functions/FlinkStatefulDoFnFunction.java|  8 +-
 .../wrappers/streaming/DoFnOperator.java| 77 +++-
 3 files changed, 70 insertions(+), 23 deletions(-)
--

[1/3] beam git commit: [BEAM-2380] Forward additional outputs to DoFnRunner in Flink Batch

[dhalperi]

Repository: beam
Updated Branches:
  refs/heads/master 2cb4b03de -> 711faffef


[BEAM-2380] Forward additional outputs to DoFnRunner in Flink Batch


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/9afe395b
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/9afe395b
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/9afe395b

Branch: refs/heads/master
Commit: 9afe395bbddd2382c5222dd3145a0be3cdf7077a
Parents: 2cb4b03
Author: Aljoscha Krettek 
Authored: Tue May 30 10:56:56 2017 +0200
Committer: Aljoscha Krettek 
Committed: Tue May 30 10:57:40 2017 +0200

--
 .../flink/translation/functions/FlinkDoFnFunction.java   | 8 +---
 .../translation/functions/FlinkStatefulDoFnFunction.java | 8 +---
 2 files changed, 10 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/9afe395b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDoFnFunction.java
--
diff --git 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDoFnFunction.java
 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDoFnFunction.java
index 9205bce..42a8833 100644
--- 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDoFnFunction.java
+++ 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDoFnFunction.java
@@ -17,7 +17,8 @@
  */
 package org.apache.beam.runners.flink.translation.functions;
 
-import java.util.Collections;
+import com.google.common.collect.Lists;
+import java.util.List;
 import java.util.Map;
 import org.apache.beam.runners.core.DoFnRunner;
 import org.apache.beam.runners.core.DoFnRunners;
@@ -97,13 +98,14 @@ public class FlinkDoFnFunction
   new FlinkDoFnFunction.MultiDoFnOutputManager((Collector) out, 
outputMap);
 }
 
+List additionalOutputTags = 
Lists.newArrayList(outputMap.keySet());
+
 DoFnRunner doFnRunner = DoFnRunners.simpleRunner(
 serializedOptions.getPipelineOptions(), doFn,
 new FlinkSideInputReader(sideInputs, runtimeContext),
 outputManager,
 mainOutputTag,
-// see SimpleDoFnRunner, just use it to limit number of additional 
outputs
-Collections.emptyList(),
+additionalOutputTags,
 new FlinkNoOpStepContext(),
 windowingStrategy);
 

http://git-wip-us.apache.org/repos/asf/beam/blob/9afe395b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkStatefulDoFnFunction.java
--
diff --git 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkStatefulDoFnFunction.java
 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkStatefulDoFnFunction.java
index 6517bf2..b075768 100644
--- 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkStatefulDoFnFunction.java
+++ 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkStatefulDoFnFunction.java
@@ -19,8 +19,9 @@ package org.apache.beam.runners.flink.translation.functions;
 
 import static org.apache.flink.util.Preconditions.checkArgument;
 
-import java.util.Collections;
+import com.google.common.collect.Lists;
 import java.util.Iterator;
+import java.util.List;
 import java.util.Map;
 import org.apache.beam.runners.core.DoFnRunner;
 import org.apache.beam.runners.core.DoFnRunners;
@@ -114,13 +115,14 @@ public class FlinkStatefulDoFnFunction
 timerInternals.advanceProcessingTime(Instant.now());
 timerInternals.advanceSynchronizedProcessingTime(Instant.now());
 
+List additionalOutputTags = 
Lists.newArrayList(outputMap.keySet());
+
 DoFnRunner, OutputT> doFnRunner = DoFnRunners.simpleRunner(
 serializedOptions.getPipelineOptions(), dofn,
 new FlinkSideInputReader(sideInputs, runtimeContext),
 outputManager,
 mainOutputTag,
-// see SimpleDoFnRunner, just use it to limit number of additional 
outputs
-Collections.emptyList(),
+additionalOutputTags,
 new FlinkNoOpStepContext() {
   @Override
   public StateInternals stateInternals() {

[jira] [Updated] (BEAM-2384) TRUNCATE math function

[tarush grover (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

tarush grover updated BEAM-2384:

Summary: TRUNCATE math function  (was: TRUNCATE)

> TRUNCATE math function
> --
>
> Key: BEAM-2384
> URL: https://issues.apache.org/jira/browse/BEAM-2384
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: tarush grover
>Assignee: tarush grover
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-2384) TRUNCATE

[tarush grover (JIRA)]

tarush grover created BEAM-2384:
---

 Summary: TRUNCATE
 Key: BEAM-2384
 URL: https://issues.apache.org/jira/browse/BEAM-2384
 Project: Beam
  Issue Type: Sub-task
  Components: dsl-sql
Reporter: tarush grover
Assignee: tarush grover






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-2383) ROUND math function

[tarush grover (JIRA)]

tarush grover created BEAM-2383:
---

 Summary: ROUND math function
 Key: BEAM-2383
 URL: https://issues.apache.org/jira/browse/BEAM-2383
 Project: Beam
  Issue Type: Sub-task
  Components: dsl-sql
Reporter: tarush grover
Assignee: tarush grover






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-2382) Introduce PubSubService

[JIRA]

Jean-Baptiste Onofré created BEAM-2382:
--

 Summary: Introduce PubSubService
 Key: BEAM-2382
 URL: https://issues.apache.org/jira/browse/BEAM-2382
 Project: Beam
  Issue Type: Wish
  Components: sdk-java-extensions
Reporter: Jean-Baptiste Onofré
Assignee: Jean-Baptiste Onofré


As we have for BigQuery, Cassandra, and other IOs, it would be great to provide 
a way to mock PubSub for users (especially for the tests).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-2382) Introduce PubSubService in the PubSubIO

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-2382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré updated BEAM-2382:
---
Summary: Introduce PubSubService in the PubSubIO  (was: Introduce 
PubSubService)

> Introduce PubSubService in the PubSubIO
> ---
>
> Key: BEAM-2382
> URL: https://issues.apache.org/jira/browse/BEAM-2382
> Project: Beam
>  Issue Type: Wish
>  Components: sdk-java-extensions
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>
> As we have for BigQuery, Cassandra, and other IOs, it would be great to 
> provide a way to mock PubSub for users (especially for the tests).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2378) extend join-library to support FULL OUTER JOIN

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16029540#comment-16029540
 ] 

Davor Bonaci commented on BEAM-2378:


SGTM; please proceed as you see fit.

> extend join-library to support FULL OUTER JOIN
> --
>
> Key: BEAM-2378
> URL: https://issues.apache.org/jira/browse/BEAM-2378
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: James Xu
>
> Currently INNER, LEFT OUTER, RIGHT OUTER are supported, would be nice to 
> support FULL OUTER JOIN.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-2378) extend join-library to support FULL OUTER JOIN

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci reassigned BEAM-2378:
--

Assignee: James Xu

> extend join-library to support FULL OUTER JOIN
> --
>
> Key: BEAM-2378
> URL: https://issues.apache.org/jira/browse/BEAM-2378
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: James Xu
>Assignee: James Xu
>
> Currently INNER, LEFT OUTER, RIGHT OUTER are supported, would be nice to 
> support FULL OUTER JOIN.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #2969

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall_Windows #72

[Apache Jenkins Server]

See 


--
[...truncated 351.32 KB...]
2017-05-30T12:04:24.973 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-core/1.2.0/grpc-core-1.2.0.pom
 (3 KB at 81.3 KB/sec)
2017-05-30T12:04:24.973 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/errorprone/error_prone_annotations/2.0.15/error_prone_annotations-2.0.15.pom
2017-05-30T12:04:25.004 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/errorprone/error_prone_annotations/2.0.15/error_prone_annotations-2.0.15.pom
 (2 KB at 51.5 KB/sec)
2017-05-30T12:04:25.004 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/errorprone/error_prone_parent/2.0.15/error_prone_parent-2.0.15.pom
2017-05-30T12:04:25.035 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/errorprone/error_prone_parent/2.0.15/error_prone_parent-2.0.15.pom
 (6 KB at 161.5 KB/sec)
2017-05-30T12:04:25.035 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/code/findbugs/jsr305/3.0.1/jsr305-3.0.1.pom
2017-05-30T12:04:25.066 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/code/findbugs/jsr305/3.0.1/jsr305-3.0.1.pom
 (5 KB at 135.0 KB/sec)
2017-05-30T12:04:25.066 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-context/1.2.0/grpc-context-1.2.0.pom
2017-05-30T12:04:25.098 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-context/1.2.0/grpc-context-1.2.0.pom
 (2 KB at 53.3 KB/sec)
2017-05-30T12:04:25.098 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/instrumentation/instrumentation-api/0.3.0/instrumentation-api-0.3.0.pom
2017-05-30T12:04:25.129 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/instrumentation/instrumentation-api/0.3.0/instrumentation-api-0.3.0.pom
 (2 KB at 47.2 KB/sec)
2017-05-30T12:04:25.129 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-protobuf/1.2.0/grpc-protobuf-1.2.0.pom
2017-05-30T12:04:25.160 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-protobuf/1.2.0/grpc-protobuf-1.2.0.pom
 (3 KB at 83.6 KB/sec)
2017-05-30T12:04:25.160 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/protobuf/protobuf-java-util/3.2.0/protobuf-java-util-3.2.0.pom
2017-05-30T12:04:25.191 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/protobuf/protobuf-java-util/3.2.0/protobuf-java-util-3.2.0.pom
 (5 KB at 133.9 KB/sec)
2017-05-30T12:04:25.191 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/code/gson/gson/2.7/gson-2.7.pom
2017-05-30T12:04:25.223 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/code/gson/gson/2.7/gson-2.7.pom 
(2 KB at 44.1 KB/sec)
2017-05-30T12:04:25.223 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/code/gson/gson-parent/2.7/gson-parent-2.7.pom
2017-05-30T12:04:25.254 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/code/gson/gson-parent/2.7/gson-parent-2.7.pom
 (4 KB at 112.8 KB/sec)
2017-05-30T12:04:25.254 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-protobuf-lite/1.2.0/grpc-protobuf-lite-1.2.0.pom
2017-05-30T12:04:25.285 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-protobuf-lite/1.2.0/grpc-protobuf-lite-1.2.0.pom
 (3 KB at 66.9 KB/sec)
2017-05-30T12:04:25.285 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-stub/1.2.0/grpc-stub-1.2.0.pom
2017-05-30T12:04:25.316 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-stub/1.2.0/grpc-stub-1.2.0.pom
 (3 KB at 65.5 KB/sec)
2017-05-30T12:04:25.316 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/protobuf/protobuf-java/3.2.0/protobuf-java-3.2.0.jar
2017-05-30T12:04:25.316 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-core/1.2.0/grpc-core-1.2.0.jar
2017-05-30T12:04:25.316 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/errorprone/error_prone_annotations/2.0.15/error_prone_annotations-2.0.15.jar
2017-05-30T12:04:25.316 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-context/1.2.0/grpc-context-1.2.0.jar
2017-05-30T12:04:25.316 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/code/findbugs/jsr305/3.0.1/jsr305-3.0.1.jar
2017-05-30T12:04:25.363 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/errorprone/error_prone_annotations/2.0.15/error_prone_annotations-2.0.15.jar
 (11 KB at 225.8 KB/sec)
2017-05-30T12:04:25.363 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/instrumentation/instrumentation-api/0.3.0/instrumentation-api-0.3.0.jar
2017-05-30T12:04:25.363 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/code/findbugs/jsr305/3.0.1/jsr305-3.0.1.jar
 (20 KB at 414.4 KB/sec)
2017-05-30T12:04:25.363 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #2968

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2380) Flink Batch Runner does not forward additional outputs to operator

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16029188#comment-16029188
 ] 

ASF GitHub Bot commented on BEAM-2380:
--

GitHub user aljoscha opened a pull request:

https://github.com/apache/beam/pull/3258

[BEAM-2380] Forward additional outputs to DoFnRunner in Flink Batch

R: @iemejia or anyone, really  

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aljoscha/beam 
jira-2380-fix-flink-batch-add-outputs

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3258.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3258


commit 9afe395bbddd2382c5222dd3145a0be3cdf7077a
Author: Aljoscha Krettek 
Date:   2017-05-30T08:56:56Z

[BEAM-2380] Forward additional outputs to DoFnRunner in Flink Batch




> Flink Batch Runner does not forward additional outputs to operator
> --
>
> Key: BEAM-2380
> URL: https://issues.apache.org/jira/browse/BEAM-2380
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 2.1.0
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>Priority: Blocker
> Fix For: 2.1.0
>
>
> Recent changes to {{SimpleDoFnRunner}} made the output tag validation more 
> strict in {{outputWindowedValue()}}. This surfaced a bug in the Flink Batch 
> runner: it doesn't forward the additional output tags to the 
> {{SimpleDoFnRunner}}.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3258: [BEAM-2380] Forward additional outputs to DoFnRunne...

[aljoscha]

GitHub user aljoscha opened a pull request:

https://github.com/apache/beam/pull/3258

[BEAM-2380] Forward additional outputs to DoFnRunner in Flink Batch

R: @iemejia or anyone, really 😃 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aljoscha/beam 
jira-2380-fix-flink-batch-add-outputs

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3258.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3258


commit 9afe395bbddd2382c5222dd3145a0be3cdf7077a
Author: Aljoscha Krettek 
Date:   2017-05-30T08:56:56Z

[BEAM-2380] Forward additional outputs to DoFnRunner in Flink Batch




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (BEAM-2159) CAST operator support

[tarush grover (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

tarush grover reassigned BEAM-2159:
---

Assignee: tarush grover

> CAST operator support
> -
>
> Key: BEAM-2159
> URL: https://issues.apache.org/jira/browse/BEAM-2159
> Project: Beam
>  Issue Type: Task
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: tarush grover
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-2380) Flink Batch Runner does not forward additional outputs to operator

[Aljoscha Krettek (JIRA)]

Aljoscha Krettek created BEAM-2380:
--

 Summary: Flink Batch Runner does not forward additional outputs to 
operator
 Key: BEAM-2380
 URL: https://issues.apache.org/jira/browse/BEAM-2380
 Project: Beam
  Issue Type: Bug
  Components: runner-flink
Affects Versions: 2.1.0
Reporter: Aljoscha Krettek
Assignee: Aljoscha Krettek
Priority: Blocker
 Fix For: 2.1.0


Recent changes to {{SimpleDoFnRunner}} made the output tag validation more 
strict in {{outputWindowedValue()}}. This surfaced a bug in the Flink Batch 
runner: it doesn't forward the additional output tags to the 
{{SimpleDoFnRunner}}.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-16) Make Spark RDDs readable as PCollections

[tarush grover (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-16?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16029185#comment-16029185
 ] 

tarush grover commented on BEAM-16:
---

[~amitsela] Can I take this task?

> Make Spark RDDs readable as PCollections
> 
>
> Key: BEAM-16
> URL: https://issues.apache.org/jira/browse/BEAM-16
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-spark
>Reporter: Amit Sela
>Priority: Minor
>
> This could be done by implementing a SparkSource.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2331) SQRT Function

[tarush grover (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16029151#comment-16029151
 ] 

tarush grover commented on BEAM-2331:
-

This task is implemented with the ABS function and its pull request has been 
created.

https://github.com/apache/beam/pull/3236

> SQRT Function
> -
>
> Key: BEAM-2331
> URL: https://issues.apache.org/jira/browse/BEAM-2331
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: tarush grover
>Assignee: tarush grover
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2329) ABS Function

[tarush grover (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16029022#comment-16029022
 ] 

tarush grover commented on BEAM-2329:
-

Pull request has been created. 

https://github.com/apache/beam/pull/3236

> ABS Function
> 
>
> Key: BEAM-2329
> URL: https://issues.apache.org/jira/browse/BEAM-2329
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: tarush grover
>Assignee: tarush grover
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #2967

[Apache Jenkins Server]

See