date:20170401

[jira] [Commented] (BEAM-1857) Add Neo4jIO

2017-04-01 Thread JIRA


[ 
https://issues.apache.org/jira/browse/BEAM-1857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15952534#comment-15952534
 ] 

Jean-Baptiste Onofré commented on BEAM-1857:


Warning: neo4j is under GPL license which is a Category X for Apache. So, we 
have to be careful in the artifacts we embed/provide to be compliant with 
Apache license.

> Add Neo4jIO
> ---
>
> Key: BEAM-1857
> URL: https://issues.apache.org/jira/browse/BEAM-1857
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: kassem shehady
>Assignee: Jean-Baptiste Onofré
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-1857) Add Neo4jIO

2017-04-01 Thread JIRA


 [ 
https://issues.apache.org/jira/browse/BEAM-1857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré reassigned BEAM-1857:
--

Assignee: Jean-Baptiste Onofré  (was: Davor Bonaci)

> Add Neo4jIO
> ---
>
> Key: BEAM-1857
> URL: https://issues.apache.org/jira/browse/BEAM-1857
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: kassem shehady
>Assignee: Jean-Baptiste Onofré
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1581) JSON source and sink

2017-04-01 Thread Aviem Zur (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-1581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aviem Zur updated BEAM-1581:

Description: 
JSON source and sink to read/write JSON files.
Similarly to {{XmlSource}}/{{XmlSink}}, these be a {{JsonSource}}/{{JonSink}} 
which are a {{FileBaseSource}}/{{FileBasedSink}}.
Consider using methods/code (or refactor these) found in {{AsJsons}} and 
{{ParseJsons}}

The {{PCollection}} of objects the user passes to the transform should be 
embedded in a valid JSON file
The most common pattern for this is a large object with an array member which 
holds all the data objects and other members for metadata.
Examples of public JSON APIs: https://www.sitepoint.com/10-example-json-files/
Another pattern used is a file which is simply a JSON array of objects.

  was:
JSON source and sink to read/write JSON files.
Similarly to {{XmlSource}}/{{XmlSink}}, these be a {{JsonSource}}/{{JonSink}} 
which are a {{FileBaseSource}}/{{FileBasedSink}}.
Consider using methods/code (or refactor these) found in {{AsJsons}} and 
{{ParseJsons}}

The {{PCollection}} of objects the user passes to the transform should be 
embedded in a valid JSON file
The most common pattern for this is a large object with an array member which 
holds all the data objects and other members for metadata.
Examples of public JSON APIs: https://www.sitepoint.com/10-example-json-files/
Another pattern used in integration is a file which is simply a JSON array of 
objects.


> JSON source and sink
> 
>
> Key: BEAM-1581
> URL: https://issues.apache.org/jira/browse/BEAM-1581
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: Aviem Zur
>Assignee: Aviem Zur
>
> JSON source and sink to read/write JSON files.
> Similarly to {{XmlSource}}/{{XmlSink}}, these be a {{JsonSource}}/{{JonSink}} 
> which are a {{FileBaseSource}}/{{FileBasedSink}}.
> Consider using methods/code (or refactor these) found in {{AsJsons}} and 
> {{ParseJsons}}
> The {{PCollection}} of objects the user passes to the transform should be 
> embedded in a valid JSON file
> The most common pattern for this is a large object with an array member which 
> holds all the data objects and other members for metadata.
> Examples of public JSON APIs: https://www.sitepoint.com/10-example-json-files/
> Another pattern used is a file which is simply a JSON array of objects.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1581) JSON source and sink

2017-04-01 Thread Aviem Zur (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-1581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aviem Zur updated BEAM-1581:

Description: 
JSON source and sink to read/write JSON files.
Similarly to {{XmlSource}}/{{XmlSink}}, these be a {{JsonSource}}/{{JonSink}} 
which are a {{FileBaseSource}}/{{FileBasedSink}}.
Consider using methods/code (or refactor these) found in {{AsJsons}} and 
{{ParseJsons}}

The {{PCollection}} of objects the user passes to the transform should be 
embedded in a valid JSON file
The most common pattern for this is a large object with an array member which 
holds all the data objects and other members for metadata.
Examples of public JSON APIs: https://www.sitepoint.com/10-example-json-files/
Another pattern used in integration is a file which is simply a JSON array of 
objects.

  was:
JSON source and sink to read/write JSON files.
Similarly to {{XmlSource}}/{{XmlSink}}, these be a {{JsonSource}}/{{JonSink}} 
which are a {{FileBaseSource}}/{{FileBasedSink}}.
Consider using methods/code (or refactor these) found in {{AsJsons}} and 
{{ParseJsons}}

The {{PCollection}} of objects the user passes to the transform should be 
embedded in a valid JSON file, this normal pattern for this is a large objects 
with an array member which holds all the data objects and other members for 
metadata.
Examples of public JSON APIs: https://www.sitepoint.com/10-example-json-files/
Another pattern used in integration is a file which is simply a JSON array of 
objects.


> JSON source and sink
> 
>
> Key: BEAM-1581
> URL: https://issues.apache.org/jira/browse/BEAM-1581
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: Aviem Zur
>Assignee: Aviem Zur
>
> JSON source and sink to read/write JSON files.
> Similarly to {{XmlSource}}/{{XmlSink}}, these be a {{JsonSource}}/{{JonSink}} 
> which are a {{FileBaseSource}}/{{FileBasedSink}}.
> Consider using methods/code (or refactor these) found in {{AsJsons}} and 
> {{ParseJsons}}
> The {{PCollection}} of objects the user passes to the transform should be 
> embedded in a valid JSON file
> The most common pattern for this is a large object with an array member which 
> holds all the data objects and other members for metadata.
> Examples of public JSON APIs: https://www.sitepoint.com/10-example-json-files/
> Another pattern used in integration is a file which is simply a JSON array of 
> objects.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1581) JSON source and sink

2017-04-01 Thread Aviem Zur (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-1581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aviem Zur updated BEAM-1581:

Description: 
JSON source and sink to read/write JSON files.
Similarly to {{XmlSource}}/{{XmlSink}}, these be a {{JsonSource}}/{{JonSink}} 
which are a {{FileBaseSource}}/{{FileBasedSink}}.
Consider using methods/code (or refactor these) found in {{AsJsons}} and 
{{ParseJsons}}

The {{PCollection}} of objects the user passes to the transform should be 
embedded in a valid JSON file, this normal pattern for this is a large objects 
with an array member which holds all the data objects and other members for 
metadata.
Examples of public JSON APIs: https://www.sitepoint.com/10-example-json-files/
Another pattern used in integration is a file which is simply a JSON array of 
objects.

  was:
JSON source and sink to read/write JSON files.
Similarly to {{XmlSource}}/{{XmlSink}}, these be a {{JsonSource}}/{{JonSink}} 
which are a {{FileBaseSource}}/{{FileBasedSink}}.
Consider using methods/code (or refactor these) found in {{AsJsons}} and 
{{ParseJsons}}

The {{PCollection}} of objects the user passes to the transform should be 
embedded in a valid JSON file, this normal pattern for this is a large objects 
with an array member which holds all the data objects and other members for 
metadata.
Examples of public JSON APIs: https://www.sitepoint.com/10-example-json-files/
Another common pattern used in integration is a file which is simply a JSON 
array of objects.


> JSON source and sink
> 
>
> Key: BEAM-1581
> URL: https://issues.apache.org/jira/browse/BEAM-1581
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: Aviem Zur
>Assignee: Aviem Zur
>
> JSON source and sink to read/write JSON files.
> Similarly to {{XmlSource}}/{{XmlSink}}, these be a {{JsonSource}}/{{JonSink}} 
> which are a {{FileBaseSource}}/{{FileBasedSink}}.
> Consider using methods/code (or refactor these) found in {{AsJsons}} and 
> {{ParseJsons}}
> The {{PCollection}} of objects the user passes to the transform should be 
> embedded in a valid JSON file, this normal pattern for this is a large 
> objects with an array member which holds all the data objects and other 
> members for metadata.
> Examples of public JSON APIs: https://www.sitepoint.com/10-example-json-files/
> Another pattern used in integration is a file which is simply a JSON array of 
> objects.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1581) JSON source and sink

2017-04-01 Thread Aviem Zur (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-1581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aviem Zur updated BEAM-1581:

Description: 
JSON source and sink to read/write JSON files.
Similarly to {{XmlSource}}/{{XmlSink}}, these be a {{JsonSource}}/{{JonSink}} 
which are a {{FileBaseSource}}/{{FileBasedSink}}.
Consider using methods/code (or refactor these) found in {{AsJsons}} and 
{{ParseJsons}}

The {{PCollection}} of objects the user passes to the transform should be 
embedded in a valid JSON file, this normal pattern for this is a large objects 
with an array member which holds all the data objects and other members for 
metadata.
Examples of public JSON APIs: https://www.sitepoint.com/10-example-json-files/
Another common pattern used in integration is a file which is simply a JSON 
array of objects.

  was:
JSON source and sink to read/write JSON files.
Similarly to {{XmlSource}}/{{XmlSink}}, these be a {{JsonSource}}/{{JonSink}} 
which are a {{FileBaseSource}}/{{FileBasedSink}}.
Consider using methods/code (or refactor these) found in {{AsJsons}} and 
{{ParseJsons}}


> JSON source and sink
> 
>
> Key: BEAM-1581
> URL: https://issues.apache.org/jira/browse/BEAM-1581
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: Aviem Zur
>Assignee: Aviem Zur
>
> JSON source and sink to read/write JSON files.
> Similarly to {{XmlSource}}/{{XmlSink}}, these be a {{JsonSource}}/{{JonSink}} 
> which are a {{FileBaseSource}}/{{FileBasedSink}}.
> Consider using methods/code (or refactor these) found in {{AsJsons}} and 
> {{ParseJsons}}
> The {{PCollection}} of objects the user passes to the transform should be 
> embedded in a valid JSON file, this normal pattern for this is a large 
> objects with an array member which holds all the data objects and other 
> members for metadata.
> Examples of public JSON APIs: https://www.sitepoint.com/10-example-json-files/
> Another common pattern used in integration is a file which is simply a JSON 
> array of objects.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-1858) improve error message when Create.of() is called with an empty iterator

2017-04-01 Thread Aviem Zur (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-1858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aviem Zur reassigned BEAM-1858:
---

Assignee: Davor Bonaci  (was: Frances Perry)

> improve error message when Create.of() is called with an empty iterator
> ---
>
> Key: BEAM-1858
> URL: https://issues.apache.org/jira/browse/BEAM-1858
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Wesley Tanaka
>Assignee: Davor Bonaci
>
> The current error copy states:
> "java.lang.IllegalArgumentException: Elements must be provided to construct 
> the default Create Coder. To Create an empty PCollection, either call 
> Create.empty(Coder), or call 'withCoder(Coder)' on the result PTransform"
> This is potentially confusing for two reasons:
> 1. "the default Create Coder" assumes a high level of knowledge of how Create 
> class works
> 2. since "Create" is a common word, it may not be immediately clear that 
> Create is referring to org.apache.beam.sdk.transforms.Create as opposed to 
> the possibility that there might be a compound noun in Beam model called 
> "Create Coder"



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1858) improve error message when Create.of() is called with an empty iterator

2017-04-01 Thread Aviem Zur (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-1858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aviem Zur updated BEAM-1858:

Priority: Trivial  (was: Major)

> improve error message when Create.of() is called with an empty iterator
> ---
>
> Key: BEAM-1858
> URL: https://issues.apache.org/jira/browse/BEAM-1858
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Wesley Tanaka
>Assignee: Davor Bonaci
>Priority: Trivial
>
> The current error copy states:
> "java.lang.IllegalArgumentException: Elements must be provided to construct 
> the default Create Coder. To Create an empty PCollection, either call 
> Create.empty(Coder), or call 'withCoder(Coder)' on the result PTransform"
> This is potentially confusing for two reasons:
> 1. "the default Create Coder" assumes a high level of knowledge of how Create 
> class works
> 2. since "Create" is a common word, it may not be immediately clear that 
> Create is referring to org.apache.beam.sdk.transforms.Create as opposed to 
> the possibility that there might be a compound noun in Beam model called 
> "Create Coder"



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1858) improve error message when Create.of() is called with an empty iterator

2017-04-01 Thread Aviem Zur (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-1858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aviem Zur updated BEAM-1858:

Component/s: (was: beam-model)
 sdk-java-core

> improve error message when Create.of() is called with an empty iterator
> ---
>
> Key: BEAM-1858
> URL: https://issues.apache.org/jira/browse/BEAM-1858
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Wesley Tanaka
>Assignee: Frances Perry
>
> The current error copy states:
> "java.lang.IllegalArgumentException: Elements must be provided to construct 
> the default Create Coder. To Create an empty PCollection, either call 
> Create.empty(Coder), or call 'withCoder(Coder)' on the result PTransform"
> This is potentially confusing for two reasons:
> 1. "the default Create Coder" assumes a high level of knowledge of how Create 
> class works
> 2. since "Create" is a common word, it may not be immediately clear that 
> Create is referring to org.apache.beam.sdk.transforms.Create as opposed to 
> the possibility that there might be a compound noun in Beam model called 
> "Create Coder"



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2402: Add example to BoundedCountingInput javadoc

2017-04-01 Thread wtanaka

GitHub user wtanaka opened a pull request:

https://github.com/apache/beam/pull/2402

Add example to BoundedCountingInput javadoc



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/wtanaka/beam example

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2402.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2402


commit 3345c2bae09a2f3653ffb81e2771bac97e2ad5c2
Author: wtanaka.com 
Date:   2017-04-02T02:44:14Z

Add example to BoundedCountingInput javadoc




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Spark #1472

2017-04-01 Thread Apache Jenkins Server

See

Build failed in Jenkins: beam_PerformanceTests_Dataflow #258

2017-04-01 Thread Apache Jenkins Server

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam5 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Pruning obsolete local branches
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* +refs/pull/*:refs/remotes/origin/pr/* 
 > --prune
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision ea33e3373a88a3ea286ea3f49f7a2b3c12aefaeb (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f ea33e3373a88a3ea286ea3f49f7a2b3c12aefaeb
 > git rev-list ea33e3373a88a3ea286ea3f49f7a2b3c12aefaeb # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Dataflow] $ /bin/bash -xe 
/tmp/hudson3763396364928959556.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Dataflow] $ /bin/bash -xe 
/tmp/hudson1022013905470363437.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Dataflow] $ /bin/bash -xe 
/tmp/hudson9191405340086227596.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): python-gflags==3.1.1 
in /home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
/usr/lib/python2.7/dist-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.11 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Cleaning up...
[beam_PerformanceTests_Dataflow] $ /bin/bash -xe 
/tmp/hudson1643192461275503455.sh
+ python PerfKitBenchmarker/pkb.py --project=apache-beam-testing 
--dpb_log_level=INFO --maven_binary=/home/jenkins/tools/maven/latest/bin/mvn 
--bigquery_table=beam_performance.pkb_results --official=true 
--benchmarks=dpb_wordcount_benchmark 
--dpb_dataflow_staging_location=gs://temp-storage-for-perf-tests/staging 
--dpb_wordcount_input=dataflow-samples/shakespeare/kinglear.txt 
--config_override=dpb_wordcount_benchmark.dpb_service.service_type=dataflow
WARNING:root:File resource loader root perfkitbenchmarker/data/ycsb is not a 
directory.
2017-04-02 00:00:19,385 aa4f9d99 MainThread INFO Verbose logging to: 
/tmp/perfkitbenchmarker/runs/aa4f9d99/pkb.log
2017-04-02 00:00:19,385 aa4f9d99 MainThread INFO PerfKitBenchmarker 
version: v1.11.0-29-g8c92c13
2017-04-02 00:00:19,386 aa4f9d99 MainThread INFO Flag values:
--maven_binary=/home/jenkins/tools/maven/latest/bin/mvn
--project=apache-beam-testing

[jira] [Created] (BEAM-1858) improve error message when Create.of() is called with an empty iterator

2017-04-01 Thread Wesley Tanaka (JIRA)

Wesley Tanaka created BEAM-1858:
---

 Summary: improve error message when Create.of() is called with an 
empty iterator
 Key: BEAM-1858
 URL: https://issues.apache.org/jira/browse/BEAM-1858
 Project: Beam
  Issue Type: Improvement
  Components: beam-model
Reporter: Wesley Tanaka
Assignee: Frances Perry


The current error copy states:

"java.lang.IllegalArgumentException: Elements must be provided to construct the 
default Create Coder. To Create an empty PCollection, either call 
Create.empty(Coder), or call 'withCoder(Coder)' on the result PTransform"

This is potentially confusing for two reasons:

1. "the default Create Coder" assumes a high level of knowledge of how Create 
class works
2. since "Create" is a common word, it may not be immediately clear that Create 
is referring to org.apache.beam.sdk.transforms.Create as opposed to the 
possibility that there might be a compound noun in Beam model called "Create 
Coder"



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2401: New copy for error when Create called w/zero items

2017-04-01 Thread wtanaka

GitHub user wtanaka opened a pull request:

https://github.com/apache/beam/pull/2401

New copy for error when Create called w/zero items



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/wtanaka/beam master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2401.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2401


commit 05eb175506c8ffc2dc6f16c3bf38566c884beee1
Author: wtanaka.com 
Date:   2017-04-01T21:29:31Z

New copy for error when Create called w/zero items




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1843) 'PDone' object has no attribute 'to_runner_api'

2017-04-01 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-1843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15952367#comment-15952367
 ] 

ASF GitHub Bot commented on BEAM-1843:
--

GitHub user robertwb opened a pull request:

https://github.com/apache/beam/pull/2400

[BEAM-1843] Avoid Runner API translation of pipelines with PDone.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/robertwb/incubator-beam pdone

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2400.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2400


commit c4ac34dc40387b752c73a7540c09cda2ceff6833
Author: Robert Bradshaw 
Date:   2017-04-01T19:42:40Z

Avoid Runner API translation of pipelines with PDone.




> 'PDone' object has no attribute 'to_runner_api'
> ---
>
> Key: BEAM-1843
> URL: https://issues.apache.org/jira/browse/BEAM-1843
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Robert Bradshaw
>
> Post commit failure with 
> (https://builds.apache.org/view/Beam/job/beam_PostCommit_Python_Verify/1676/consoleFull):
> ...
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/pipeline.py",
>  line 512, in 
> for tag, out in self.outputs.items()},
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/runners/pipeline_context.py",
>  line 52, in get_id
> self._id_to_proto[id] = obj.to_runner_api(self._pipeline_context)
> AttributeError: 'PDone' object has no attribute 'to_runner_api'



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2400: [BEAM-1843] Avoid Runner API translation of pipelin...

2017-04-01 Thread robertwb

GitHub user robertwb opened a pull request:

https://github.com/apache/beam/pull/2400

[BEAM-1843] Avoid Runner API translation of pipelines with PDone.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/robertwb/incubator-beam pdone

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2400.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2400


commit c4ac34dc40387b752c73a7540c09cda2ceff6833
Author: Robert Bradshaw 
Date:   2017-04-01T19:42:40Z

Avoid Runner API translation of pipelines with PDone.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (BEAM-302) Add Scio Scala DSL to Beam

2017-04-01 Thread Neville Li (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neville Li reassigned BEAM-302:
---

Assignee: (was: Neville Li)

> Add Scio Scala DSL to Beam
> --
>
> Key: BEAM-302
> URL: https://issues.apache.org/jira/browse/BEAM-302
> Project: Beam
>  Issue Type: Wish
>  Components: sdk-ideas
>Reporter: Jean-Baptiste Onofré
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Spark #1471

2017-04-01 Thread Apache Jenkins Server

See

Build failed in Jenkins: beam_PerformanceTests_Dataflow #257

2017-04-01 Thread Apache Jenkins Server

See 


--
[...truncated 270.29 KB...]
 * [new ref] refs/pull/928/head -> origin/pr/928/head
 * [new ref] refs/pull/929/head -> origin/pr/929/head
 * [new ref] refs/pull/93/head -> origin/pr/93/head
 * [new ref] refs/pull/930/head -> origin/pr/930/head
 * [new ref] refs/pull/930/merge -> origin/pr/930/merge
 * [new ref] refs/pull/931/head -> origin/pr/931/head
 * [new ref] refs/pull/931/merge -> origin/pr/931/merge
 * [new ref] refs/pull/932/head -> origin/pr/932/head
 * [new ref] refs/pull/932/merge -> origin/pr/932/merge
 * [new ref] refs/pull/933/head -> origin/pr/933/head
 * [new ref] refs/pull/933/merge -> origin/pr/933/merge
 * [new ref] refs/pull/934/head -> origin/pr/934/head
 * [new ref] refs/pull/934/merge -> origin/pr/934/merge
 * [new ref] refs/pull/935/head -> origin/pr/935/head
 * [new ref] refs/pull/936/head -> origin/pr/936/head
 * [new ref] refs/pull/936/merge -> origin/pr/936/merge
 * [new ref] refs/pull/937/head -> origin/pr/937/head
 * [new ref] refs/pull/937/merge -> origin/pr/937/merge
 * [new ref] refs/pull/938/head -> origin/pr/938/head
 * [new ref] refs/pull/939/head -> origin/pr/939/head
 * [new ref] refs/pull/94/head -> origin/pr/94/head
 * [new ref] refs/pull/940/head -> origin/pr/940/head
 * [new ref] refs/pull/940/merge -> origin/pr/940/merge
 * [new ref] refs/pull/941/head -> origin/pr/941/head
 * [new ref] refs/pull/941/merge -> origin/pr/941/merge
 * [new ref] refs/pull/942/head -> origin/pr/942/head
 * [new ref] refs/pull/942/merge -> origin/pr/942/merge
 * [new ref] refs/pull/943/head -> origin/pr/943/head
 * [new ref] refs/pull/943/merge -> origin/pr/943/merge
 * [new ref] refs/pull/944/head -> origin/pr/944/head
 * [new ref] refs/pull/945/head -> origin/pr/945/head
 * [new ref] refs/pull/945/merge -> origin/pr/945/merge
 * [new ref] refs/pull/946/head -> origin/pr/946/head
 * [new ref] refs/pull/946/merge -> origin/pr/946/merge
 * [new ref] refs/pull/947/head -> origin/pr/947/head
 * [new ref] refs/pull/947/merge -> origin/pr/947/merge
 * [new ref] refs/pull/948/head -> origin/pr/948/head
 * [new ref] refs/pull/948/merge -> origin/pr/948/merge
 * [new ref] refs/pull/949/head -> origin/pr/949/head
 * [new ref] refs/pull/949/merge -> origin/pr/949/merge
 * [new ref] refs/pull/95/head -> origin/pr/95/head
 * [new ref] refs/pull/95/merge -> origin/pr/95/merge
 * [new ref] refs/pull/950/head -> origin/pr/950/head
 * [new ref] refs/pull/951/head -> origin/pr/951/head
 * [new ref] refs/pull/951/merge -> origin/pr/951/merge
 * [new ref] refs/pull/952/head -> origin/pr/952/head
 * [new ref] refs/pull/952/merge -> origin/pr/952/merge
 * [new ref] refs/pull/953/head -> origin/pr/953/head
 * [new ref] refs/pull/954/head -> origin/pr/954/head
 * [new ref] refs/pull/954/merge -> origin/pr/954/merge
 * [new ref] refs/pull/955/head -> origin/pr/955/head
 * [new ref] refs/pull/955/merge -> origin/pr/955/merge
 * [new ref] refs/pull/956/head -> origin/pr/956/head
 * [new ref] refs/pull/957/head -> origin/pr/957/head
 * [new ref] refs/pull/958/head -> origin/pr/958/head
 * [new ref] refs/pull/959/head -> origin/pr/959/head
 * [new ref] refs/pull/959/merge -> origin/pr/959/merge
 * [new ref] refs/pull/96/head -> origin/pr/96/head
 * [new ref] refs/pull/96/merge -> origin/pr/96/merge
 * [new ref] refs/pull/960/head -> origin/pr/960/head
 * [new ref] refs/pull/960/merge -> origin/pr/960/merge
 * [new ref] refs/pull/961/head -> origin/pr/961/head
 * [new ref] refs/pull/962/head -> origin/pr/962/head
 * [new ref] refs/pull/962/merge -> origin/pr/962/merge
 * [new ref] refs/pull/963/head -> origin/pr/963/head
 * [new ref] refs/pull/963/merge -> origin/pr/963/merge
 * [new ref] refs/pull/964/head -> origin/pr/964/head
 * [new ref] refs/pull/965/head -> origin/pr/965/head
 * [new ref] refs/pull/965/merge -> origin/pr/965/merge
 * [new ref] refs/pull/966/head -> origin/pr/966/head
 * [new ref] refs/pull/967/head -> origin/pr/967/head
 * [new ref] refs/pull/967/merge -> origin/pr/967/merge
 * [new ref] refs/pull/968/head -> origin/pr/968/head
 * [new ref] refs/pull/968/merge -> origin/pr/968/merge
 * [new ref] refs/pull/969/head -> origin/pr/969/head
 * [new ref] refs/pull/969/merge -> origin/pr/969/merge
 * [new ref] refs/pull/97/head -> origin/pr/97/head
 * [new ref]

[jira] [Created] (BEAM-1857) Add Neo4jIO

2017-04-01 Thread kassem shehady (JIRA)

kassem shehady created BEAM-1857:


 Summary: Add Neo4jIO
 Key: BEAM-1857
 URL: https://issues.apache.org/jira/browse/BEAM-1857
 Project: Beam
  Issue Type: New Feature
  Components: sdk-java-extensions
Reporter: kassem shehady
Assignee: Davor Bonaci






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1848) GroupByKey stuck with more than one worker on Dataflow

2017-04-01 Thread peay (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-1848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15952249#comment-15952249
 ] 

peay commented on BEAM-1848:


Indeed, the issue is gone. Thanks for looking into it so quickly -- I hadn't 
found those issues.

> GroupByKey stuck with more than one worker on Dataflow
> --
>
> Key: BEAM-1848
> URL: https://issues.apache.org/jira/browse/BEAM-1848
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow, sdk-java-core
>Affects Versions: 0.6.0
>Reporter: peay
>Assignee: Kenneth Knowles
>Priority: Blocker
>
> I have a simple pipeline which has a sliding window, a {{GroupByKey}} and 
> then a simple stateful {{DoFn}}. I run in batch mode ({{--streaming=false}}) 
> on Dataflow. 
> On a very small dataset  of a couple KBs, I can run the pipeline to 
> completion. Dataflow does show "successful".  On a larger dataset (but still 
> very small, 100s MB read by source), the pipeline stays stuck, no matter how 
> long I wait. In addition, it never really gets stuck at the same point. I 
> expect about 340k output records, and never get more than 70k before it gets 
> stuck.
> Dataflow always autoscales from 1 to 8 workers, which is my limit.
> Run A: after 30mins+: no elements added out of {{GroupByKey}}, but logs have 
> repeating occurrences of
> {code}
> Proposing dynamic split of work unit xx;aa;bb at 
> {"position":{"shufflePosition":"AAD_AP8A_wEAAQ"}}
> Refusing to split  [ShufflePosition(base64:AAD_AP8A_wD_AAE), 
> ShufflePosition(base64:AQD_AP8A_wD_AAE))> at 
> ShufflePosition(base64:AAD_AP8A_wEAAQ): unstarted   
> Refused to split GroupingShuffleReader  [ShufflePosition(base64:AAD_AP8A_wD_AAE), 
> ShufflePosition(base64:AQD_AP8A_wD_AAE))> at 
> ShufflePosition(base64:AAD_AP8A_wEAAQ)
> {code}
> Run B: after a couple minutes, elements get added to output of 
> {{GroupByKey}}, up to 56,128 and then stays stuck doing nothing, but logs 
> have repeating occurrences of
> {code}
> Proposing dynamic split of work unit xx;ccc;dd at 
> {"position":{"shufflePosition":"AQD_AP8A_wEAAQ"}}
> Refusing to split  [ShufflePosition(base64:AQD_AP8A_wD_AAE), 
> ShufflePosition(base64:AgD_AP8A_wD_AAE))> at 
> ShufflePosition(base64:AQD_AP8A_wEAAQ): unstarted
> Refused to split GroupingShuffleReader  [ShufflePosition(base64:AQD_AP8A_wD_AAE), 
> ShufflePosition(base64:AgD_AP8A_wD_AAE))> at 
> ShufflePosition(base64:AQD_AP8A_wEAAQ)
> {code}
> Run C: after 10mins: elements get added to output of {{GroupByKey}}, up to 
> 70,262 and then stays stuck doing nothing. No logs as above as far as I can 
> find.
> I've run this about a dozen times and it always gets stuck. I am trying out 
> right now to run the pipeline with the worker limit set to one, and 
> {{GroupByKey}} has output 150k so far, still increasing. This seems like a 
> workaround, but using one worker only is not ideal.
> cc [~dhalp...@google.com]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: beam_PerformanceTests_Dataflow #256

2017-04-01 Thread Apache Jenkins Server

See 


Changes:

[aviemzur] [BEAM-1337] Infer state coders

[kirpichov] [BEAM-1855] Support Splittable DoFn on Flink Runner

--
[...truncated 271.44 KB...]
 * [new ref] refs/pull/928/head -> origin/pr/928/head
 * [new ref] refs/pull/929/head -> origin/pr/929/head
 * [new ref] refs/pull/93/head -> origin/pr/93/head
 * [new ref] refs/pull/930/head -> origin/pr/930/head
 * [new ref] refs/pull/930/merge -> origin/pr/930/merge
 * [new ref] refs/pull/931/head -> origin/pr/931/head
 * [new ref] refs/pull/931/merge -> origin/pr/931/merge
 * [new ref] refs/pull/932/head -> origin/pr/932/head
 * [new ref] refs/pull/932/merge -> origin/pr/932/merge
 * [new ref] refs/pull/933/head -> origin/pr/933/head
 * [new ref] refs/pull/933/merge -> origin/pr/933/merge
 * [new ref] refs/pull/934/head -> origin/pr/934/head
 * [new ref] refs/pull/934/merge -> origin/pr/934/merge
 * [new ref] refs/pull/935/head -> origin/pr/935/head
 * [new ref] refs/pull/936/head -> origin/pr/936/head
 * [new ref] refs/pull/936/merge -> origin/pr/936/merge
 * [new ref] refs/pull/937/head -> origin/pr/937/head
 * [new ref] refs/pull/937/merge -> origin/pr/937/merge
 * [new ref] refs/pull/938/head -> origin/pr/938/head
 * [new ref] refs/pull/939/head -> origin/pr/939/head
 * [new ref] refs/pull/94/head -> origin/pr/94/head
 * [new ref] refs/pull/940/head -> origin/pr/940/head
 * [new ref] refs/pull/940/merge -> origin/pr/940/merge
 * [new ref] refs/pull/941/head -> origin/pr/941/head
 * [new ref] refs/pull/941/merge -> origin/pr/941/merge
 * [new ref] refs/pull/942/head -> origin/pr/942/head
 * [new ref] refs/pull/942/merge -> origin/pr/942/merge
 * [new ref] refs/pull/943/head -> origin/pr/943/head
 * [new ref] refs/pull/943/merge -> origin/pr/943/merge
 * [new ref] refs/pull/944/head -> origin/pr/944/head
 * [new ref] refs/pull/945/head -> origin/pr/945/head
 * [new ref] refs/pull/945/merge -> origin/pr/945/merge
 * [new ref] refs/pull/946/head -> origin/pr/946/head
 * [new ref] refs/pull/946/merge -> origin/pr/946/merge
 * [new ref] refs/pull/947/head -> origin/pr/947/head
 * [new ref] refs/pull/947/merge -> origin/pr/947/merge
 * [new ref] refs/pull/948/head -> origin/pr/948/head
 * [new ref] refs/pull/948/merge -> origin/pr/948/merge
 * [new ref] refs/pull/949/head -> origin/pr/949/head
 * [new ref] refs/pull/949/merge -> origin/pr/949/merge
 * [new ref] refs/pull/95/head -> origin/pr/95/head
 * [new ref] refs/pull/95/merge -> origin/pr/95/merge
 * [new ref] refs/pull/950/head -> origin/pr/950/head
 * [new ref] refs/pull/951/head -> origin/pr/951/head
 * [new ref] refs/pull/951/merge -> origin/pr/951/merge
 * [new ref] refs/pull/952/head -> origin/pr/952/head
 * [new ref] refs/pull/952/merge -> origin/pr/952/merge
 * [new ref] refs/pull/953/head -> origin/pr/953/head
 * [new ref] refs/pull/954/head -> origin/pr/954/head
 * [new ref] refs/pull/954/merge -> origin/pr/954/merge
 * [new ref] refs/pull/955/head -> origin/pr/955/head
 * [new ref] refs/pull/955/merge -> origin/pr/955/merge
 * [new ref] refs/pull/956/head -> origin/pr/956/head
 * [new ref] refs/pull/957/head -> origin/pr/957/head
 * [new ref] refs/pull/958/head -> origin/pr/958/head
 * [new ref] refs/pull/959/head -> origin/pr/959/head
 * [new ref] refs/pull/959/merge -> origin/pr/959/merge
 * [new ref] refs/pull/96/head -> origin/pr/96/head
 * [new ref] refs/pull/96/merge -> origin/pr/96/merge
 * [new ref] refs/pull/960/head -> origin/pr/960/head
 * [new ref] refs/pull/960/merge -> origin/pr/960/merge
 * [new ref] refs/pull/961/head -> origin/pr/961/head
 * [new ref] refs/pull/962/head -> origin/pr/962/head
 * [new ref] refs/pull/962/merge -> origin/pr/962/merge
 * [new ref] refs/pull/963/head -> origin/pr/963/head
 * [new ref] refs/pull/963/merge -> origin/pr/963/merge
 * [new ref] refs/pull/964/head -> origin/pr/964/head
 * [new ref] refs/pull/965/head -> origin/pr/965/head
 * [new ref] refs/pull/965/merge -> origin/pr/965/merge
 * [new ref] refs/pull/966/head -> origin/pr/966/head
 * [new ref] refs/pull/967/head -> origin/pr/967/head
 * [new ref] refs/pull/967/merge -> origin/pr/967/merge
 * [new ref] refs/pull/968/head -> origin/pr/968/head
 * [new ref] refs/pull/968/merge -> origin/pr/968/merge
 * [new ref] refs/pull/969/head -> origin/pr/969/head
 * [new

[jira] [Assigned] (BEAM-1603) Enable programmatic execution of spark pipelines.

2017-04-01 Thread Aviem Zur (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-1603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aviem Zur reassigned BEAM-1603:
---

Assignee: (was: Aviem Zur)

> Enable programmatic execution of spark pipelines.
> -
>
> Key: BEAM-1603
> URL: https://issues.apache.org/jira/browse/BEAM-1603
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark, testing
>Reporter: Jason Kuster
>
> In order to enable execution of Spark Integration Tests against a cluster, it 
> is necessary to have the ability to execute Spark pipelines via maven, rather 
> than spark-submit. The minimum necessary is to enable this in the 
> TestSparkRunner.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1856) HDFSFileSink class do not use the same configuration in master and slave

2017-04-01 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-1856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15952151#comment-15952151
 ] 

ASF GitHub Bot commented on BEAM-1856:
--

GitHub user 397090770 opened a pull request:

https://github.com/apache/beam/pull/2399

[BEAM-1856]

As described in 
[BEAM-1856](https://issues.apache.org/jira/browse/BEAM-1856)  `HDFSFileSink` 
class do not use the same configuration in master thread and slave thread.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/397090770/beam master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2399.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2399


commit d1526d49563bba4820a43ca8e08ee7e8d1845d07
Author: iteblog 
Date:   2017-04-01T09:41:05Z

BEAM-1856

BEAM-1856




> HDFSFileSink class do not use the same configuration in master and slave
> 
>
> Key: BEAM-1856
> URL: https://issues.apache.org/jira/browse/BEAM-1856
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 0.6.0
>Reporter: yangping wu
>Assignee: Davor Bonaci
>
> I have a code snippet as follow:
> {code}
> Read.Bounded> from = 
> Read.from(HDFSFileSource.from(options.getInputFile(), TextInputFormat.class, 
> LongWritable.class, Text.class));
> PCollection> data = p.apply(from);
> data.apply(MapElements.via(new SimpleFunction, 
> String>() {
> @Override
> public String apply(KV input) {
> return input.getValue() + "\t" + input.getValue();
> }
> })).apply(Write.to(HDFSFileSink.toText(options.getOutputFile(;
> {code}
> and submit job like this:
> {code}
> spark-submit --class org.apache.beam.examples.WordCountHDFS --master 
> yarn-client   \
>  ./target/word-count-beam-bundled-0.1.jar 
>  \
>  --runner=SparkRunner 
>  \
>  --inputFile=hdfs://master/tmp/input/ 
>  \
>  --outputFile=/tmp/output/
> {code}
> Then {{HDFSFileSink.validate}} function will check whether the local 
> filesystem (not HDFS) exists {{/tmp/output/}} directory.
> But the final result will store in {{hdfs://master/tmp/output/}} directory in 
> HDFS filesystem.
> The reason is {{HDFSFileSink}} class do not use the same configuration in 
> master thread and slave thread.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2399: [BEAM-1856]

2017-04-01 Thread 397090770

GitHub user 397090770 opened a pull request:

https://github.com/apache/beam/pull/2399

[BEAM-1856]

As described in 
[BEAM-1856](https://issues.apache.org/jira/browse/BEAM-1856)  `HDFSFileSink` 
class do not use the same configuration in master thread and slave thread.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/397090770/beam master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2399.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2399


commit d1526d49563bba4820a43ca8e08ee7e8d1845d07
Author: iteblog 
Date:   2017-04-01T09:41:05Z

BEAM-1856

BEAM-1856




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1856) HDFSFileSink class do not use the same configuration in master and slave

2017-04-01 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-1856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15952146#comment-15952146
 ] 

ASF GitHub Bot commented on BEAM-1856:
--

Github user 397090770 closed the pull request at:

https://github.com/apache/beam/pull/2398


> HDFSFileSink class do not use the same configuration in master and slave
> 
>
> Key: BEAM-1856
> URL: https://issues.apache.org/jira/browse/BEAM-1856
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 0.6.0
>Reporter: yangping wu
>Assignee: Davor Bonaci
>
> I have a code snippet as follow:
> {code}
> Read.Bounded> from = 
> Read.from(HDFSFileSource.from(options.getInputFile(), TextInputFormat.class, 
> LongWritable.class, Text.class));
> PCollection> data = p.apply(from);
> data.apply(MapElements.via(new SimpleFunction, 
> String>() {
> @Override
> public String apply(KV input) {
> return input.getValue() + "\t" + input.getValue();
> }
> })).apply(Write.to(HDFSFileSink.toText(options.getOutputFile(;
> {code}
> and submit job like this:
> {code}
> spark-submit --class org.apache.beam.examples.WordCountHDFS --master 
> yarn-client   \
>  ./target/word-count-beam-bundled-0.1.jar 
>  \
>  --runner=SparkRunner 
>  \
>  --inputFile=hdfs://master/tmp/input/ 
>  \
>  --outputFile=/tmp/output/
> {code}
> Then {{HDFSFileSink.validate}} function will check whether the local 
> filesystem (not HDFS) exists {{/tmp/output/}} directory.
> But the final result will store in {{hdfs://master/tmp/output/}} directory in 
> HDFS filesystem.
> The reason is {{HDFSFileSink}} class do not use the same configuration in 
> master thread and slave thread.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2398: [BEAM-1856]HDFSFileSink class do not use the same c...

2017-04-01 Thread 397090770

Github user 397090770 closed the pull request at:

https://github.com/apache/beam/pull/2398


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1856) HDFSFileSink class do not use the same configuration in master and slave

2017-04-01 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-1856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15952145#comment-15952145
 ] 

ASF GitHub Bot commented on BEAM-1856:
--

GitHub user 397090770 opened a pull request:

https://github.com/apache/beam/pull/2398

[BEAM-1856]HDFSFileSink class do not use the same configuration in master 
and slave

As described in 
[BEAM-1856](https://issues.apache.org/jira/browse/BEAM-1856),`HDFSFileSink` 
class do not use the same configuration in master thread and slave thread.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/apache/beam release-0.6.0

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2398.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2398


commit d375cfa126fd7be9c34f39c2b9b856f324bf
Author: Ahmet Altay 
Date:   2017-03-07T21:08:34Z

Update Beam version in the Maven archetypes

commit e6dd96ed86837bb0997f25d74322c463a72c8d5d
Author: Ahmet Altay 
Date:   2017-03-07T21:12:21Z

Update dataflow container version to beam-0.6.0

commit 1c91d0e598c5dd62a9c34cc0bf4bfcc9a6f2faae
Author: Ahmet Altay 
Date:   2017-03-07T21:44:54Z

[maven-release-plugin] prepare release v0.6.0-RC1

commit 8ab36fa2d5dfc237c7619c278848e31f3412a0e1
Author: Ahmet Altay 
Date:   2017-03-08T21:37:57Z

Revert "[maven-release-plugin] prepare release v0.6.0-RC1"

This reverts commit 1c91d0e598c5dd62a9c34cc0bf4bfcc9a6f2faae.

commit d5257658f094fe8c2a8668027bbdd4a26396ba0b
Author: Ismaël Mejía 
Date:   2017-03-06T08:13:31Z

Change Json parsing from gson to jackson for ElasticsearchIO

commit 730def77ebf216295d953b13722f21185f674ccc
Author: Mark Liu 
Date:   2017-03-07T22:26:37Z

[BEAM-1646] Remove duplicated bigquery dependency

commit 92c5b5bd732d9fc019fa6820afcc31a92a026bbf
Author: Davor Bonaci 
Date:   2017-03-07T19:57:38Z

Add ServicesResourceTransformer to all shading configuration

This ensures that files in META-INF/services aren't overwritten. Instead, 
they are concatenated.

This is critical to ensure PipelineOptionsRegistrar, RunnerRegistrar, 
IOChannelFactoryRegistrar
and FileSystemRegistrar work well for users.

commit 5e2afa29a3a0fe93e662b2fe7173c1641c253cd5
Author: Kenneth Knowles 
Date:   2017-03-02T22:29:56Z

Explicitly GBK before stateful ParDo in Dataflow batch

commit 3518b12fbfc984fcfe12e12ba06809e57744f006
Author: Tibor Kiss 
Date:   2017-03-08T11:18:39Z

[BEAM-1649] Fix unresolved references in Python SDK

commit f572328ce23e70adee8001e3d10f1479bd9a380d
Author: Ahmet Altay 
Date:   2017-03-08T21:42:36Z

Update dataflow container version to beam-0.6.0

commit edaf3ac9f57c208bb7ce444d409b0909ef2d1a67
Author: Ahmet Altay 
Date:   2017-03-08T21:43:28Z

Update py-sdk version to release version.

commit b25a0369dc5e5e4eacdc6e40b1e81452f9685579
Author: Ahmet Altay 
Date:   2017-03-08T22:07:45Z

This closes #2202

commit 99be42e5e6ea5438d80b66d3345615a2754f2ed9
Author: Ahmet Altay 
Date:   2017-03-08T23:12:32Z

[maven-release-plugin] prepare branch release-0.6.0

commit e1ea41274bab5347eac14a299231d97d239c924c
Author: Ahmet Altay 
Date:   2017-03-08T23:13:40Z

Revert "[maven-release-plugin] prepare branch release-0.6.0"

This reverts commit 99be42e5e6ea5438d80b66d3345615a2754f2ed9.

commit d2d4f0aad805b4e06371cb1ad6c29a6183236f7b
Author: Ahmet Altay 
Date:   2017-03-09T00:32:54Z

[maven-release-plugin] prepare release v0.6.0-RC1

commit dc64c2fc0487e3549f48a710b8a7c4fb7bd0c788
Author: Ahmet Altay 
Date:   2017-03-09T00:34:41Z

[maven-release-plugin] rollback changes from release preparation of 
v0.6.0-RC1

commit a18b5b1648489f14fd7a621f345e4d21c09b437f
Author: Aljoscha Krettek 
Date:   2017-03-10T07:29:27Z

Move GC timer checking to StatefulDoFnRunner.CleanupTimer

commit 8fa718db5bc14efd1beefc2c757c331a5bdbf927
Author: Aljoscha Krettek 
Date:   2017-03-10T10:07:00Z

Introduce Flink-specific state GC implementations

We now set the GC timer for window.maxTimestamp() + 1 to ensure that a
user timer set for window.maxTimestamp() still has all state.

This also adds tests for late data dropping and state GC specifically
for the Flink DoFnOperator.

commit 86522157a79fd9a753436312ff8b746cb5740135
Author: Aljoscha Krettek 
Date:   2017-03-10T14:25:26Z

Properly deal with late

[GitHub] beam pull request #2398: [BEAM-1856]HDFSFileSink class do not use the same c...

2017-04-01 Thread 397090770

GitHub user 397090770 opened a pull request:

https://github.com/apache/beam/pull/2398

[BEAM-1856]HDFSFileSink class do not use the same configuration in master 
and slave

As described in 
[BEAM-1856](https://issues.apache.org/jira/browse/BEAM-1856),`HDFSFileSink` 
class do not use the same configuration in master thread and slave thread.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/apache/beam release-0.6.0

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2398.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2398


commit d375cfa126fd7be9c34f39c2b9b856f324bf
Author: Ahmet Altay 
Date:   2017-03-07T21:08:34Z

Update Beam version in the Maven archetypes

commit e6dd96ed86837bb0997f25d74322c463a72c8d5d
Author: Ahmet Altay 
Date:   2017-03-07T21:12:21Z

Update dataflow container version to beam-0.6.0

commit 1c91d0e598c5dd62a9c34cc0bf4bfcc9a6f2faae
Author: Ahmet Altay 
Date:   2017-03-07T21:44:54Z

[maven-release-plugin] prepare release v0.6.0-RC1

commit 8ab36fa2d5dfc237c7619c278848e31f3412a0e1
Author: Ahmet Altay 
Date:   2017-03-08T21:37:57Z

Revert "[maven-release-plugin] prepare release v0.6.0-RC1"

This reverts commit 1c91d0e598c5dd62a9c34cc0bf4bfcc9a6f2faae.

commit d5257658f094fe8c2a8668027bbdd4a26396ba0b
Author: IsmaÃ«l MejÃa 
Date:   2017-03-06T08:13:31Z

Change Json parsing from gson to jackson for ElasticsearchIO

commit 730def77ebf216295d953b13722f21185f674ccc
Author: Mark Liu 
Date:   2017-03-07T22:26:37Z

[BEAM-1646] Remove duplicated bigquery dependency

commit 92c5b5bd732d9fc019fa6820afcc31a92a026bbf
Author: Davor Bonaci 
Date:   2017-03-07T19:57:38Z

Add ServicesResourceTransformer to all shading configuration

This ensures that files in META-INF/services aren't overwritten. Instead, 
they are concatenated.

This is critical to ensure PipelineOptionsRegistrar, RunnerRegistrar, 
IOChannelFactoryRegistrar
and FileSystemRegistrar work well for users.

commit 5e2afa29a3a0fe93e662b2fe7173c1641c253cd5
Author: Kenneth Knowles 
Date:   2017-03-02T22:29:56Z

Explicitly GBK before stateful ParDo in Dataflow batch

commit 3518b12fbfc984fcfe12e12ba06809e57744f006
Author: Tibor Kiss 
Date:   2017-03-08T11:18:39Z

[BEAM-1649] Fix unresolved references in Python SDK

commit f572328ce23e70adee8001e3d10f1479bd9a380d
Author: Ahmet Altay 
Date:   2017-03-08T21:42:36Z

Update dataflow container version to beam-0.6.0

commit edaf3ac9f57c208bb7ce444d409b0909ef2d1a67
Author: Ahmet Altay 
Date:   2017-03-08T21:43:28Z

Update py-sdk version to release version.

commit b25a0369dc5e5e4eacdc6e40b1e81452f9685579
Author: Ahmet Altay 
Date:   2017-03-08T22:07:45Z

This closes #2202

commit 99be42e5e6ea5438d80b66d3345615a2754f2ed9
Author: Ahmet Altay 
Date:   2017-03-08T23:12:32Z

[maven-release-plugin] prepare branch release-0.6.0

commit e1ea41274bab5347eac14a299231d97d239c924c
Author: Ahmet Altay 
Date:   2017-03-08T23:13:40Z

Revert "[maven-release-plugin] prepare branch release-0.6.0"

This reverts commit 99be42e5e6ea5438d80b66d3345615a2754f2ed9.

commit d2d4f0aad805b4e06371cb1ad6c29a6183236f7b
Author: Ahmet Altay 
Date:   2017-03-09T00:32:54Z

[maven-release-plugin] prepare release v0.6.0-RC1

commit dc64c2fc0487e3549f48a710b8a7c4fb7bd0c788
Author: Ahmet Altay 
Date:   2017-03-09T00:34:41Z

[maven-release-plugin] rollback changes from release preparation of 
v0.6.0-RC1

commit a18b5b1648489f14fd7a621f345e4d21c09b437f
Author: Aljoscha Krettek 
Date:   2017-03-10T07:29:27Z

Move GC timer checking to StatefulDoFnRunner.CleanupTimer

commit 8fa718db5bc14efd1beefc2c757c331a5bdbf927
Author: Aljoscha Krettek 
Date:   2017-03-10T10:07:00Z

Introduce Flink-specific state GC implementations

We now set the GC timer for window.maxTimestamp() + 1 to ensure that a
user timer set for window.maxTimestamp() still has all state.

This also adds tests for late data dropping and state GC specifically
for the Flink DoFnOperator.

commit 86522157a79fd9a753436312ff8b746cb5740135
Author: Aljoscha Krettek 
Date:   2017-03-10T14:25:26Z

Properly deal with late processing-time timers

commit 2b92b0d851bcc5aedcc40ebf02ad4f39f3d67514
Author: Ahmet Altay 
Date:   2017-03-10T21:42:17Z

Add README to python tarball.

And, delete test created files, to avoid them being

[jira] [Created] (BEAM-1856) HDFSFileSink class do not use the same configuration in master and slave

2017-04-01 Thread yangping wu (JIRA)

yangping wu created BEAM-1856:
-

 Summary: HDFSFileSink class do not use the same configuration in 
master and slave
 Key: BEAM-1856
 URL: https://issues.apache.org/jira/browse/BEAM-1856
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Affects Versions: 0.6.0
Reporter: yangping wu
Assignee: Davor Bonaci


I have a code snippet as follow:
{code}
Read.Bounded> from = 
Read.from(HDFSFileSource.from(options.getInputFile(), TextInputFormat.class, 
LongWritable.class, Text.class));
PCollection> data = p.apply(from);
data.apply(MapElements.via(new SimpleFunction, String>() 
{
@Override
public String apply(KV input) {
return input.getValue() + "\t" + input.getValue();
}
})).apply(Write.to(HDFSFileSink.toText(options.getOutputFile(;
{code}
and submit job like this:
{code}
spark-submit --class org.apache.beam.examples.WordCountHDFS --master 
yarn-client   \
 ./target/word-count-beam-bundled-0.1.jar   
   \
 --runner=SparkRunner   
   \
 --inputFile=hdfs://master/tmp/input/   
   \
 --outputFile=/tmp/output/
{code}
Then {{HDFSFileSink.validate}} function will check whether the local filesystem 
(not HDFS) exists {{/tmp/output/}} directory.
But the final result will store in {{hdfs://master/tmp/output/}} directory in 
HDFS filesystem.
The reason is {{HDFSFileSink}} class do not use the same configuration in 
master thread and slave thread.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1491) HDFSFileSource should be able to read the HADOOP_CONF_DIR(YARN_CONF_DIR) environmen variable

2017-04-01 Thread yangping wu (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-1491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

yangping wu updated BEAM-1491:
--
Affects Version/s: (was: 0.5.0)
   0.6.0

> HDFSFileSource should be able to read the HADOOP_CONF_DIR(YARN_CONF_DIR) 
> environmen variable
> 
>
> Key: BEAM-1491
> URL: https://issues.apache.org/jira/browse/BEAM-1491
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Affects Versions: 0.6.0
>Reporter: yangping wu
>Assignee: Jean-Baptiste Onofré
>
> Currently, if we want to read file store on HDFS, we will do it as follow:
> {code} 
> PRead.Bounded> from = 
> Read.from(HDFSFileSource.from("hdfs://hadoopserver:8020/tmp/data.txt", 
> TextInputFormat.class, LongWritable.class, Text.class));
> PCollection> data = p.apply(from);
> {code}
> or
> {code}
> Configuration conf = new Configuration();
> conf.set("fs.default.name", "hdfs://hadoopserver:8020");
> PRead.Bounded> from = 
> Read.from(HDFSFileSource.from("/tmp/data.txt", TextInputFormat.class, 
> LongWritable.class, Text.class).withConfiguration(conf));
> PCollection> data = p.apply(from);
> {code}
> As we have seen above, we must be set {{hdfs://hadoopserver:8020}} in the 
> file path
> if we can initialize {{conf}} by reading 
> {{HADOOP_CONF_DIR}}({{YARN_CONF_DIR}}) environmen variable, then we can read 
> HDFS file like this:
> {code}
> PRead.Bounded> from = 
> Read.from(HDFSFileSource.from("/tmp/data.txt", TextInputFormat.class, 
> LongWritable.class, Text.class));
> PCollection> data = p.apply(from);
> {code}
> note we don't specify {{hdfs://hadoopserver:8020}} prefix, because the 
> program read it from  {{HADOOP_CONF_DIR}}({{YARN_CONF_DIR}}) environmen, and 
> the program will read file from HDFS.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1491) HDFSFileSource should be able to read the HADOOP_CONF_DIR(YARN_CONF_DIR) environmen variable

2017-04-01 Thread yangping wu (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-1491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

yangping wu updated BEAM-1491:
--
Description: 
Currently, if we want to read file store on HDFS, we will do it as follow:
{code} 
PRead.Bounded> from = 
Read.from(HDFSFileSource.from("hdfs://hadoopserver:8020/tmp/data.txt", 
TextInputFormat.class, LongWritable.class, Text.class));
PCollection> data = p.apply(from);
{code}
or
{code}
Configuration conf = new Configuration();
conf.set("fs.default.name", "hdfs://hadoopserver:8020");
PRead.Bounded> from = 
Read.from(HDFSFileSource.from("/tmp/data.txt", TextInputFormat.class, 
LongWritable.class, Text.class).withConfiguration(conf));
PCollection> data = p.apply(from);
{code}
As we have seen above, we must be set {{hdfs://hadoopserver:8020}} in the file 
path
if we can initialize {{conf}} by reading {{HADOOP_CONF_DIR}}({{YARN_CONF_DIR}}) 
environmen variable, then we can read HDFS file like this:
{code}
PRead.Bounded> from = 
Read.from(HDFSFileSource.from("/tmp/data.txt", TextInputFormat.class, 
LongWritable.class, Text.class));
PCollection> data = p.apply(from);
{code}
note we don't specify {{hdfs://hadoopserver:8020}} prefix, because the program 
read it from  {{HADOOP_CONF_DIR}}({{YARN_CONF_DIR}}) environmen, and the 
program will read file from HDFS.

  was:
Currently, if we want to read file store on HDFS, we will do it as follow:
{code} PCollection> resultCollection = 
p.apply(HDFSFileSource.readFrom(
"hdfs://hadoopserver:8020/tmp/data.txt",
TextInputFormat.class, LongWritable.class, Text.class));
{code}
As we have seen above, we must be set {{hdfs://hadoopserver:8020}} in the file 
path, and we cann't set any variables when read file, because in 
[HDFSFileSource.java|https://github.com/apache/beam/blob/master/sdks/java/io/hdfs/src/main/java/org/apache/beam/sdk/io/hdfs/HDFSFileSource.java#L310]
 we initialize {{job}} instance as follow:
{code}
this.job = Job.getInstance();
{code}
we should initialize {{job}} instance by configure:
{code}
this.job = Job.getInstance(conf);
{code}
where {{conf}} is instance of {{Configuration}}, and we initialize {{conf}} by 
reading {{HADOOP_CONF_DIR}}({{YARN_CONF_DIR}}) environmen variable,then we can 
read HDFS file like this:
{code} PCollection> resultCollection = 
p.apply(HDFSFileSource.readFrom(
"/tmp/data.txt",
TextInputFormat.class, LongWritable.class, Text.class));
{code}
note we don't specify {{hdfs://hadoopserver:8020}} prefix, because the program 
read it from  {{HADOOP_CONF_DIR}}({{YARN_CONF_DIR}}) environmen.


> HDFSFileSource should be able to read the HADOOP_CONF_DIR(YARN_CONF_DIR) 
> environmen variable
> 
>
> Key: BEAM-1491
> URL: https://issues.apache.org/jira/browse/BEAM-1491
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Affects Versions: 0.5.0
>Reporter: yangping wu
>Assignee: Jean-Baptiste Onofré
>
> Currently, if we want to read file store on HDFS, we will do it as follow:
> {code} 
> PRead.Bounded> from = 
> Read.from(HDFSFileSource.from("hdfs://hadoopserver:8020/tmp/data.txt", 
> TextInputFormat.class, LongWritable.class, Text.class));
> PCollection> data = p.apply(from);
> {code}
> or
> {code}
> Configuration conf = new Configuration();
> conf.set("fs.default.name", "hdfs://hadoopserver:8020");
> PRead.Bounded> from = 
> Read.from(HDFSFileSource.from("/tmp/data.txt", TextInputFormat.class, 
> LongWritable.class, Text.class).withConfiguration(conf));
> PCollection> data = p.apply(from);
> {code}
> As we have seen above, we must be set {{hdfs://hadoopserver:8020}} in the 
> file path
> if we can initialize {{conf}} by reading 
> {{HADOOP_CONF_DIR}}({{YARN_CONF_DIR}}) environmen variable, then we can read 
> HDFS file like this:
> {code}
> PRead.Bounded> from = 
> Read.from(HDFSFileSource.from("/tmp/data.txt", TextInputFormat.class, 
> LongWritable.class, Text.class));
> PCollection> data = p.apply(from);
> {code}
> note we don't specify {{hdfs://hadoopserver:8020}} prefix, because the 
> program read it from  {{HADOOP_CONF_DIR}}({{YARN_CONF_DIR}}) environmen, and 
> the program will read file from HDFS.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is back to stable : beam_PostCommit_Java_MavenInstall #3126

2017-04-01 Thread Apache Jenkins Server

See

[jira] [Resolved] (BEAM-1855) Support Splittable DoFn in Flink runner

2017-04-01 Thread Eugene Kirpichov (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-1855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Kirpichov resolved BEAM-1855.

   Resolution: Fixed
Fix Version/s: First stable release

> Support Splittable DoFn in Flink runner
> ---
>
> Key: BEAM-1855
> URL: https://issues.apache.org/jira/browse/BEAM-1855
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Eugene Kirpichov
>Assignee: Aljoscha Krettek
> Fix For: First stable release
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[1/2] beam git commit: [BEAM-1855] Support Splittable DoFn on Flink Runner

2017-04-01 Thread jkff

Repository: beam
Updated Branches:
  refs/heads/master e31ca8b0d -> ea33e3373


[BEAM-1855] Support Splittable DoFn on Flink Runner


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/5824bb4b
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/5824bb4b
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/5824bb4b

Branch: refs/heads/master
Commit: 5824bb4b5700b5230f569c570d5e8ed2d11cedf2
Parents: e31ca8b
Author: Aljoscha Krettek 
Authored: Mon Mar 13 21:23:14 2017 +0100
Committer: Eugene Kirpichov 
Committed: Sat Apr 1 01:10:55 2017 -0700

--
 runners/flink/runner/pom.xml|   4 +-
 .../flink/FlinkStreamingPipelineTranslator.java |  37 ++
 .../FlinkStreamingTransformTranslators.java | 341 ++-
 .../streaming/SplittableDoFnOperator.java   | 150 
 .../beam/sdk/transforms/SplittableDoFnTest.java |   3 +-
 5 files changed, 448 insertions(+), 87 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/5824bb4b/runners/flink/runner/pom.xml
--
diff --git a/runners/flink/runner/pom.xml b/runners/flink/runner/pom.xml
index 3b35c8e..f26aeb0 100644
--- a/runners/flink/runner/pom.xml
+++ b/runners/flink/runner/pom.xml
@@ -88,9 +88,9 @@
 org.apache.beam.sdk.testing.FlattenWithHeterogeneousCoders,
 org.apache.beam.sdk.testing.UsesSetState,
 org.apache.beam.sdk.testing.UsesMapState,
-org.apache.beam.sdk.testing.UsesSplittableParDo,
 org.apache.beam.sdk.testing.UsesAttemptedMetrics,
-org.apache.beam.sdk.testing.UsesCommittedMetrics
+org.apache.beam.sdk.testing.UsesCommittedMetrics,
+org.apache.beam.sdk.testing.UsesTestStream
   
   none
   true

http://git-wip-us.apache.org/repos/asf/beam/blob/5824bb4b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/FlinkStreamingPipelineTranslator.java
--
diff --git 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/FlinkStreamingPipelineTranslator.java
 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/FlinkStreamingPipelineTranslator.java
index d50d6bf..0cedf66 100644
--- 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/FlinkStreamingPipelineTranslator.java
+++ 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/FlinkStreamingPipelineTranslator.java
@@ -18,8 +18,12 @@
 package org.apache.beam.runners.flink;
 
 import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Iterables;
+import java.util.List;
 import java.util.Map;
+import org.apache.beam.runners.core.SplittableParDo;
 import org.apache.beam.runners.core.construction.PTransformMatchers;
+import org.apache.beam.runners.core.construction.ReplacementOutputs;
 import 
org.apache.beam.runners.core.construction.SingleInputOutputOverrideFactory;
 import org.apache.beam.sdk.Pipeline;
 import org.apache.beam.sdk.options.PipelineOptions;
@@ -29,9 +33,13 @@ import org.apache.beam.sdk.runners.PTransformOverrideFactory;
 import org.apache.beam.sdk.runners.TransformHierarchy;
 import org.apache.beam.sdk.transforms.Combine;
 import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
 import org.apache.beam.sdk.transforms.View;
 import org.apache.beam.sdk.util.InstanceBuilder;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionTuple;
 import org.apache.beam.sdk.values.PValue;
+import org.apache.beam.sdk.values.TaggedPValue;
 import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
@@ -66,6 +74,8 @@ class FlinkStreamingPipelineTranslator extends 
FlinkPipelineTranslator {
 Map transformOverrides =
 ImmutableMap.builder()
 .put(
+PTransformMatchers.splittableParDoMulti(), new 
SplittableParDoOverrideFactory())
+.put(
 PTransformMatchers.classEqualTo(View.AsIterable.class),
 new ReflectiveOneToOneOverrideFactory(
 FlinkStreamingViewOverrides.StreamingViewAsIterable.class, 
flinkRunner))
@@ -228,4 +238,31 @@ class FlinkStreamingPipelineTranslator extends 
FlinkPipelineTranslator {
 }
   }
 
+  /**
+   * A {@link PTransformOverrideFactory} that overrides a
+   * https://s.apache.org/splittable-do-fn;>Splittable DoFn with
+   * {@link

[GitHub] beam pull request #2235: Initial version of Flink Streaming Splittable DoFn ...

2017-04-01 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2235


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #2235

2017-04-01 Thread jkff

This closes #2235


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/ea33e337
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/ea33e337
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/ea33e337

Branch: refs/heads/master
Commit: ea33e3373a88a3ea286ea3f49f7a2b3c12aefaeb
Parents: e31ca8b 5824bb4
Author: Eugene Kirpichov 
Authored: Sat Apr 1 01:11:53 2017 -0700
Committer: Eugene Kirpichov 
Committed: Sat Apr 1 01:11:53 2017 -0700

--
 runners/flink/runner/pom.xml|   4 +-
 .../flink/FlinkStreamingPipelineTranslator.java |  37 ++
 .../FlinkStreamingTransformTranslators.java | 341 ++-
 .../streaming/SplittableDoFnOperator.java   | 150 
 .../beam/sdk/transforms/SplittableDoFnTest.java |   3 +-
 5 files changed, 448 insertions(+), 87 deletions(-)
--

[jira] [Created] (BEAM-1855) Support Splittable DoFn in Flink runner

2017-04-01 Thread Eugene Kirpichov (JIRA)

Eugene Kirpichov created BEAM-1855:
--

 Summary: Support Splittable DoFn in Flink runner
 Key: BEAM-1855
 URL: https://issues.apache.org/jira/browse/BEAM-1855
 Project: Beam
  Issue Type: New Feature
  Components: runner-flink
Reporter: Eugene Kirpichov
Assignee: Aljoscha Krettek






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BEAM-1337) Use our coder infrastructure for coders for state

2017-04-01 Thread Aviem Zur (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-1337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aviem Zur resolved BEAM-1337.
-
   Resolution: Fixed
Fix Version/s: First stable release

> Use our coder infrastructure for coders for state
> -
>
> Key: BEAM-1337
> URL: https://issues.apache.org/jira/browse/BEAM-1337
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Aviem Zur
>Priority: Minor
> Fix For: First stable release
>
>
> Today the user must explicitly provide any coders needed for serializing 
> state data. We'd rather use the coder registry and infer the coder.
> Currently all factory methods in {{StateSpecs}} take a coder argument. For 
> example:
> {code}
> StateSpecs.value(coderForT);
> {code}
> We could leverage the coder registry and provide different overloads:
> TypeDescriptor:
> {code}
> StateSpecs.value(typeDescriptorForT); 
> {code}
> Reflection:
> {code}
> StateSpec.value();
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1337) Use our coder infrastructure for coders for state

2017-04-01 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-1337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15952101#comment-15952101
 ] 

ASF GitHub Bot commented on BEAM-1337:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2133


> Use our coder infrastructure for coders for state
> -
>
> Key: BEAM-1337
> URL: https://issues.apache.org/jira/browse/BEAM-1337
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Aviem Zur
>Priority: Minor
>
> Today the user must explicitly provide any coders needed for serializing 
> state data. We'd rather use the coder registry and infer the coder.
> Currently all factory methods in {{StateSpecs}} take a coder argument. For 
> example:
> {code}
> StateSpecs.value(coderForT);
> {code}
> We could leverage the coder registry and provide different overloads:
> TypeDescriptor:
> {code}
> StateSpecs.value(typeDescriptorForT); 
> {code}
> Reflection:
> {code}
> StateSpec.value();
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[1/2] beam git commit: [BEAM-1337] Infer state coders

2017-04-01 Thread aviemzur

Repository: beam
Updated Branches:
  refs/heads/master 03dce6dcc -> e31ca8b0d


[BEAM-1337] Infer state coders


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/42e690e8
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/42e690e8
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/42e690e8

Branch: refs/heads/master
Commit: 42e690e84a9f05d508f2528b1444b26ce031e080
Parents: 03dce6d
Author: Aviem Zur 
Authored: Wed Mar 1 07:27:57 2017 +0200
Committer: Aviem Zur 
Committed: Sat Apr 1 10:27:14 2017 +0300

--
 .../org/apache/beam/sdk/transforms/ParDo.java   |  62 ++
 .../apache/beam/sdk/util/state/StateSpec.java   |  15 +
 .../apache/beam/sdk/util/state/StateSpecs.java  | 264 -
 .../apache/beam/sdk/transforms/ParDoTest.java   | 578 +++
 4 files changed, 902 insertions(+), 17 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/42e690e8/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java
index 664fbc3..3de845b 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java
@@ -22,6 +22,8 @@ import static 
com.google.common.base.Preconditions.checkArgument;
 import com.google.common.collect.ImmutableList;
 import com.google.common.collect.ImmutableMap;
 import java.io.Serializable;
+import java.lang.reflect.ParameterizedType;
+import java.lang.reflect.Type;
 import java.util.Arrays;
 import java.util.Collections;
 import java.util.List;
@@ -29,6 +31,7 @@ import java.util.Map;
 import org.apache.beam.sdk.Pipeline;
 import org.apache.beam.sdk.coders.CannotProvideCoderException;
 import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.CoderRegistry;
 import org.apache.beam.sdk.runners.PipelineRunner;
 import org.apache.beam.sdk.transforms.display.DisplayData;
 import org.apache.beam.sdk.transforms.display.DisplayData.Builder;
@@ -41,6 +44,7 @@ import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
 import org.apache.beam.sdk.transforms.windowing.WindowFn;
 import org.apache.beam.sdk.util.NameUtils;
 import org.apache.beam.sdk.util.SerializableUtils;
+import org.apache.beam.sdk.util.state.StateSpec;
 import org.apache.beam.sdk.values.PCollection;
 import org.apache.beam.sdk.values.PCollectionTuple;
 import org.apache.beam.sdk.values.PCollectionView;
@@ -434,6 +438,59 @@ public class ParDo {
 return DisplayData.item("fn", fn.getClass()).withLabel("Transform 
Function");
   }
 
+  private static void finishSpecifyingStateSpecs(
+  DoFn fn,
+  CoderRegistry coderRegistry,
+  Coder inputCoder) {
+DoFnSignature signature = DoFnSignatures.getSignature(fn.getClass());
+Map stateDeclarations = 
signature.stateDeclarations();
+for (DoFnSignature.StateDeclaration stateDeclaration : 
stateDeclarations.values()) {
+  try {
+StateSpec stateSpec = (StateSpec) 
stateDeclaration.field().get(fn);
+stateSpec.offerCoders(codersForStateSpecTypes(stateDeclaration, 
coderRegistry, inputCoder));
+stateSpec.finishSpecifying();
+  } catch (IllegalAccessException e) {
+throw new RuntimeException(e);
+  }
+}
+  }
+
+  /**
+   * Try to provide coders for as many of the type arguments of given
+   * {@link DoFnSignature.StateDeclaration} as possible.
+   */
+  private static  Coder[] codersForStateSpecTypes(
+  DoFnSignature.StateDeclaration stateDeclaration,
+  CoderRegistry coderRegistry,
+  Coder inputCoder) {
+Type stateType = stateDeclaration.stateType().getType();
+
+if (!(stateType instanceof ParameterizedType)) {
+  // No type arguments means no coders to infer.
+  return new Coder[0];
+}
+
+Type[] typeArguments = ((ParameterizedType) 
stateType).getActualTypeArguments();
+Coder[] coders = new Coder[typeArguments.length];
+
+for (int i = 0; i < typeArguments.length; i++) {
+  Type typeArgument = typeArguments[i];
+  TypeDescriptor typeDescriptor = TypeDescriptor.of(typeArgument);
+  try {
+coders[i] = coderRegistry.getDefaultCoder(typeDescriptor);
+  } catch (CannotProvideCoderException e) {
+try {
+  coders[i] = coderRegistry.getDefaultCoder(
+  typeDescriptor, inputCoder.getEncodedTypeDescriptor(), 
inputCoder);
+} catch (CannotProvideCoderException ignored) {
+  // Since not all type arguments will have a registered coder we 
ignore this exception.
+}

[GitHub] beam pull request #2133: [BEAM-1337] Infer state coders

2017-04-01 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2133


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #2133

2017-04-01 Thread aviemzur

This closes #2133


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/e31ca8b0
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/e31ca8b0
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/e31ca8b0

Branch: refs/heads/master
Commit: e31ca8b0d05e47c2588d5db29c92bac49aa410da
Parents: 03dce6d 42e690e
Author: Aviem Zur 
Authored: Sat Apr 1 10:28:25 2017 +0300
Committer: Aviem Zur 
Committed: Sat Apr 1 10:28:25 2017 +0300

--
 .../org/apache/beam/sdk/transforms/ParDo.java   |  62 ++
 .../apache/beam/sdk/util/state/StateSpec.java   |  15 +
 .../apache/beam/sdk/util/state/StateSpecs.java  | 264 -
 .../apache/beam/sdk/transforms/ParDoTest.java   | 578 +++
 4 files changed, 902 insertions(+), 17 deletions(-)
--

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #3125

2017-04-01 Thread Apache Jenkins Server

See

[jira] [Updated] (BEAM-1491) HDFSFileSource should be able to read the HADOOP_CONF_DIR(YARN_CONF_DIR) environmen variable

2017-04-01 Thread yangping wu (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-1491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

yangping wu updated BEAM-1491:
--
Description: 
Currently, if we want to read file store on HDFS, we will do it as follow:
{code} PCollection> resultCollection = 
p.apply(HDFSFileSource.readFrom(
"hdfs://hadoopserver:8020/tmp/data.txt",
TextInputFormat.class, LongWritable.class, Text.class));
{code}
As we have seen above, we must be set {{hdfs://hadoopserver:8020}} in the file 
path, and we cann't set any variables when read file, because in 
[HDFSFileSource.java|https://github.com/apache/beam/blob/master/sdks/java/io/hdfs/src/main/java/org/apache/beam/sdk/io/hdfs/HDFSFileSource.java#L310]
 we initialize {{job}} instance as follow:
{code}
this.job = Job.getInstance();
{code}
we should initialize {{job}} instance by configure:
{code}
this.job = Job.getInstance(conf);
{code}
where {{conf}} is instance of {{Configuration}}, and we initialize {{conf}} by 
reading {{HADOOP_CONF_DIR}}({{YARN_CONF_DIR}}) environmen variable,then we can 
read HDFS file like this:
{code} PCollection> resultCollection = 
p.apply(HDFSFileSource.readFrom(
"/tmp/data.txt",
TextInputFormat.class, LongWritable.class, Text.class));
{code}
note we don't specify {{hdfs://hadoopserver:8020}} prefix, because the program 
read it from  {{HADOOP_CONF_DIR}}({{YARN_CONF_DIR}}) environmen.

  was:
Currently, if we want to read file store on HDFS, we will do it as follow:
{code} PCollection> resultCollection = 
p.apply(HDFSFileSource.readFrom(
"hdfs://hadoopserver:8020/tmp/data.txt",
TextInputFormat.class, LongWritable.class, Text.class));
{code}
As we have seen above, we must be set {{hdfs://hadoopserver:8020}} in the file 
path, and we cann't set any variables when read file, because in 
[HDFSFileSource.java|https://github.com/apache/beam/blob/master/sdks/java/io/hdfs/src/main/java/org/apache/beam/sdk/io/hdfs/HDFSFileSource.java#L310]
 we initialize {{job}} instance as follow:
{code}
this.job = Job.getInstance();
{code}
we should initialize {{job}} instance by configure:
{code}
this.job = Job.getInstance(conf);
{code}
where {{conf}} is instance of {{Configuration}}, and we initialize {{conf}} by 
reading {{HADOOP_CONF}}({{YARN_CONF}}) environmen variable,then we can read 
HDFS file like this:
{code} PCollection> resultCollection = 
p.apply(HDFSFileSource.readFrom(
"/tmp/data.txt",
TextInputFormat.class, LongWritable.class, Text.class));
{code}
note we don't specify {{hdfs://hadoopserver:8020}} prefix, because the program 
read it from  {{HADOOP_CONF}}({{YARN_CONF}}) environmen.


> HDFSFileSource should be able to read the HADOOP_CONF_DIR(YARN_CONF_DIR) 
> environmen variable
> 
>
> Key: BEAM-1491
> URL: https://issues.apache.org/jira/browse/BEAM-1491
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Affects Versions: 0.5.0
>Reporter: yangping wu
>Assignee: Jean-Baptiste Onofré
>
> Currently, if we want to read file store on HDFS, we will do it as follow:
> {code} PCollection> resultCollection = 
> p.apply(HDFSFileSource.readFrom(
> "hdfs://hadoopserver:8020/tmp/data.txt",
> TextInputFormat.class, LongWritable.class, Text.class));
> {code}
> As we have seen above, we must be set {{hdfs://hadoopserver:8020}} in the 
> file path, and we cann't set any variables when read file, because in 
> [HDFSFileSource.java|https://github.com/apache/beam/blob/master/sdks/java/io/hdfs/src/main/java/org/apache/beam/sdk/io/hdfs/HDFSFileSource.java#L310]
>  we initialize {{job}} instance as follow:
> {code}
> this.job = Job.getInstance();
> {code}
> we should initialize {{job}} instance by configure:
> {code}
> this.job = Job.getInstance(conf);
> {code}
> where {{conf}} is instance of {{Configuration}}, and we initialize {{conf}} 
> by reading {{HADOOP_CONF_DIR}}({{YARN_CONF_DIR}}) environmen variable,then we 
> can read HDFS file like this:
> {code} PCollection> resultCollection = 
> p.apply(HDFSFileSource.readFrom(
> "/tmp/data.txt",
> TextInputFormat.class, LongWritable.class, Text.class));
> {code}
> note we don't specify {{hdfs://hadoopserver:8020}} prefix, because the 
> program read it from  {{HADOOP_CONF_DIR}}({{YARN_CONF_DIR}}) environmen.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: beam_PerformanceTests_Dataflow #255

2017-04-01 Thread Apache Jenkins Server

See 


Changes:

[kirpichov] KafkaIO : Add withTopic() api that takes single topic.

[tgroh] Separate View Evaluation and Overrides

[robertwb] Only encode PCollection outputs in Runner API protos.

[robertwb] Ensure transforms are picklable before materializing to protos.

[robertwb] Fix side inputs on dataflow runner.

--
[...truncated 271.57 KB...]
 * [new ref] refs/pull/928/head -> origin/pr/928/head
 * [new ref] refs/pull/929/head -> origin/pr/929/head
 * [new ref] refs/pull/93/head -> origin/pr/93/head
 * [new ref] refs/pull/930/head -> origin/pr/930/head
 * [new ref] refs/pull/930/merge -> origin/pr/930/merge
 * [new ref] refs/pull/931/head -> origin/pr/931/head
 * [new ref] refs/pull/931/merge -> origin/pr/931/merge
 * [new ref] refs/pull/932/head -> origin/pr/932/head
 * [new ref] refs/pull/932/merge -> origin/pr/932/merge
 * [new ref] refs/pull/933/head -> origin/pr/933/head
 * [new ref] refs/pull/933/merge -> origin/pr/933/merge
 * [new ref] refs/pull/934/head -> origin/pr/934/head
 * [new ref] refs/pull/934/merge -> origin/pr/934/merge
 * [new ref] refs/pull/935/head -> origin/pr/935/head
 * [new ref] refs/pull/936/head -> origin/pr/936/head
 * [new ref] refs/pull/936/merge -> origin/pr/936/merge
 * [new ref] refs/pull/937/head -> origin/pr/937/head
 * [new ref] refs/pull/937/merge -> origin/pr/937/merge
 * [new ref] refs/pull/938/head -> origin/pr/938/head
 * [new ref] refs/pull/939/head -> origin/pr/939/head
 * [new ref] refs/pull/94/head -> origin/pr/94/head
 * [new ref] refs/pull/940/head -> origin/pr/940/head
 * [new ref] refs/pull/940/merge -> origin/pr/940/merge
 * [new ref] refs/pull/941/head -> origin/pr/941/head
 * [new ref] refs/pull/941/merge -> origin/pr/941/merge
 * [new ref] refs/pull/942/head -> origin/pr/942/head
 * [new ref] refs/pull/942/merge -> origin/pr/942/merge
 * [new ref] refs/pull/943/head -> origin/pr/943/head
 * [new ref] refs/pull/943/merge -> origin/pr/943/merge
 * [new ref] refs/pull/944/head -> origin/pr/944/head
 * [new ref] refs/pull/945/head -> origin/pr/945/head
 * [new ref] refs/pull/945/merge -> origin/pr/945/merge
 * [new ref] refs/pull/946/head -> origin/pr/946/head
 * [new ref] refs/pull/946/merge -> origin/pr/946/merge
 * [new ref] refs/pull/947/head -> origin/pr/947/head
 * [new ref] refs/pull/947/merge -> origin/pr/947/merge
 * [new ref] refs/pull/948/head -> origin/pr/948/head
 * [new ref] refs/pull/948/merge -> origin/pr/948/merge
 * [new ref] refs/pull/949/head -> origin/pr/949/head
 * [new ref] refs/pull/949/merge -> origin/pr/949/merge
 * [new ref] refs/pull/95/head -> origin/pr/95/head
 * [new ref] refs/pull/95/merge -> origin/pr/95/merge
 * [new ref] refs/pull/950/head -> origin/pr/950/head
 * [new ref] refs/pull/951/head -> origin/pr/951/head
 * [new ref] refs/pull/951/merge -> origin/pr/951/merge
 * [new ref] refs/pull/952/head -> origin/pr/952/head
 * [new ref] refs/pull/952/merge -> origin/pr/952/merge
 * [new ref] refs/pull/953/head -> origin/pr/953/head
 * [new ref] refs/pull/954/head -> origin/pr/954/head
 * [new ref] refs/pull/954/merge -> origin/pr/954/merge
 * [new ref] refs/pull/955/head -> origin/pr/955/head
 * [new ref] refs/pull/955/merge -> origin/pr/955/merge
 * [new ref] refs/pull/956/head -> origin/pr/956/head
 * [new ref] refs/pull/957/head -> origin/pr/957/head
 * [new ref] refs/pull/958/head -> origin/pr/958/head
 * [new ref] refs/pull/959/head -> origin/pr/959/head
 * [new ref] refs/pull/959/merge -> origin/pr/959/merge
 * [new ref] refs/pull/96/head -> origin/pr/96/head
 * [new ref] refs/pull/96/merge -> origin/pr/96/merge
 * [new ref] refs/pull/960/head -> origin/pr/960/head
 * [new ref] refs/pull/960/merge -> origin/pr/960/merge
 * [new ref] refs/pull/961/head -> origin/pr/961/head
 * [new ref] refs/pull/962/head -> origin/pr/962/head
 * [new ref] refs/pull/962/merge -> origin/pr/962/merge
 * [new ref] refs/pull/963/head -> origin/pr/963/head
 * [new ref] refs/pull/963/merge -> origin/pr/963/merge
 * [new ref] refs/pull/964/head -> origin/pr/964/head
 * [new ref] refs/pull/965/head -> origin/pr/965/head
 * [new ref] refs/pull/965/merge -> origin/pr/965/merge
 * [new ref] refs/pull/966/head -> origin/pr/966/head
 * [new ref] refs/pull/967/head -> origin/pr/967/head
 * [new ref] refs/pull/967/merge -> origin/pr/967/merge

46 matches

Mail list logo