[jira] [Commented] (BEAM-1859) sorter extension depends on hadoop but does not declare as such in repository artifact

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-1859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15953000#comment-15953000
 ] 

Jean-Baptiste Onofré commented on BEAM-1859:


This approach allows you to match your Hadoop version as you provide it 
explicitly. It's actually more flexible ;)

Even if you don't use Hadoop infrastructure, some extensions are using the 
Hadoop libraries (dealing with hadoop format for instance).

A potential good improvement in your case would be for us to provide a BoM 
(Bill of Material): it's basically a pom describing some default dependencies 
set.

> sorter extension depends on hadoop but does not declare as such in repository 
> artifact
> --
>
> Key: BEAM-1859
> URL: https://issues.apache.org/jira/browse/BEAM-1859
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Affects Versions: 0.6.0
>Reporter: Wesley Tanaka
>Assignee: Davor Bonaci
> Fix For: Not applicable
>
>
> When SortValues is used via 
> {{org.apache.beam:beam-sdks-java-extensions-sorter:0.6.0}}, this exception is 
> raised:
> {noformat}
> Caused by: java.lang.NoClassDefFoundError: 
> org/apache/hadoop/conf/Configuration
>   at 
> org.apache.beam.sdk.extensions.sorter.BufferedExternalSorter.create(BufferedExternalSorter.java:98)
>   at 
> org.apache.beam.sdk.extensions.sorter.SortValues$SortValuesDoFn.processElement(SortValues.java:153)
> Caused by: java.lang.ClassNotFoundException: 
> org.apache.hadoop.conf.Configuration
>   at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>   at 
> org.apache.beam.sdk.extensions.sorter.BufferedExternalSorter.create(BufferedExternalSorter.java:98)
>   at 
> org.apache.beam.sdk.extensions.sorter.SortValues$SortValuesDoFn.processElement(SortValues.java:153)
>   at 
> org.apache.beam.sdk.extensions.sorter.SortValues$SortValuesDoFn$auxiliary$uK25yOmK.invokeProcessElement(Unknown
>  Source)
>   at 
> org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:198)
>   at 
> org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:159)
>   at 
> org.apache.beam.runners.core.PushbackSideInputDoFnRunner.processElement(PushbackSideInputDoFnRunner.java:111)
>   at 
> org.apache.beam.runners.core.PushbackSideInputDoFnRunner.processElementInReadyWindows(PushbackSideInputDoFnRunner.java:77)
>   at 
> org.apache.beam.runners.direct.ParDoEvaluator.processElement(ParDoEvaluator.java:134)
>   at 
> org.apache.beam.runners.direct.DoFnLifecycleManagerRemovingTransformEvaluator.processElement(DoFnLifecycleManagerRemovingTransformEvaluator.java:51)
>   at 
> org.apache.beam.runners.direct.TransformExecutor.processElements(TransformExecutor.java:139)
>   at 
> org.apache.beam.runners.direct.TransformExecutor.run(TransformExecutor.java:107)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> {noformat}
> I think the issue is that beam-sdks-java-extensions-sorter should declare 
> that it depends on that hadoop library but does not?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (BEAM-1859) sorter extension depends on hadoop but does not declare as such in repository artifact

[Wesley Tanaka (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15952999#comment-15952999
 ] 

Wesley Tanaka edited comment on BEAM-1859 at 4/3/17 5:06 AM:
-

Adding org.apache.hadoop:hadoop-core:0.20.2 as a dependency explicitly does 
resolve the issue, thanks, I'll just do that; I didn't know that it was a best 
practice to assume it was already installed.

In case it helps to know it, my use case is one of learning the Beam API, not 
of trying to actually accomplish something with it:

* I am trying to learn the Beam API
* So I am trying to create different toy composite PTransforms
* and I'd like to speed up the code/test/debug cycle relative to uploading code 
into a cluster
* so, despite this being nonsensical w.r.t. the actual use of Beam, I am trying 
to hack together some code to get DirectRunner to read lines from stdin and 
write lines to stdout and run the same code against different inputs to see how 
it behaves.

In case it's also interesting to know, in my actual use case, I don't actually 
have Hadoop setup, I'm using Beam with only Flink and Kafka at the moment.


was (Author: wtanaka):
Adding org.apache.hadoop:hadoop-core:0.20.2 as a dependency explicitly does 
resolve the issue, thanks, I'll just do that; I didn't know that it was a best 
practice to assume it was already installed.

In case it helps to know it, my use case is one of learning the Beam API, not 
of trying to actually accomplish something with it:

* I am trying to learn the Beam API
* So I am trying to create different toy composite PTransforms
* and I'd like to speed up the code/test/debug cycle relative to uploading code 
into a cluster
* so, despite this being nonsensical w.r.t. the actual use of Beam, I am trying 
to hack together some code to get DirectRunner to read lines from stdin and 
write lines to stdout and run the same code against different inputs to see how 
it behaves.

> sorter extension depends on hadoop but does not declare as such in repository 
> artifact
> --
>
> Key: BEAM-1859
> URL: https://issues.apache.org/jira/browse/BEAM-1859
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Affects Versions: 0.6.0
>Reporter: Wesley Tanaka
>Assignee: Davor Bonaci
> Fix For: Not applicable
>
>
> When SortValues is used via 
> {{org.apache.beam:beam-sdks-java-extensions-sorter:0.6.0}}, this exception is 
> raised:
> {noformat}
> Caused by: java.lang.NoClassDefFoundError: 
> org/apache/hadoop/conf/Configuration
>   at 
> org.apache.beam.sdk.extensions.sorter.BufferedExternalSorter.create(BufferedExternalSorter.java:98)
>   at 
> org.apache.beam.sdk.extensions.sorter.SortValues$SortValuesDoFn.processElement(SortValues.java:153)
> Caused by: java.lang.ClassNotFoundException: 
> org.apache.hadoop.conf.Configuration
>   at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>   at 
> org.apache.beam.sdk.extensions.sorter.BufferedExternalSorter.create(BufferedExternalSorter.java:98)
>   at 
> org.apache.beam.sdk.extensions.sorter.SortValues$SortValuesDoFn.processElement(SortValues.java:153)
>   at 
> org.apache.beam.sdk.extensions.sorter.SortValues$SortValuesDoFn$auxiliary$uK25yOmK.invokeProcessElement(Unknown
>  Source)
>   at 
> org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:198)
>   at 
> org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:159)
>   at 
> org.apache.beam.runners.core.PushbackSideInputDoFnRunner.processElement(PushbackSideInputDoFnRunner.java:111)
>   at 
> org.apache.beam.runners.core.PushbackSideInputDoFnRunner.processElementInReadyWindows(PushbackSideInputDoFnRunner.java:77)
>   at 
> org.apache.beam.runners.direct.ParDoEvaluator.processElement(ParDoEvaluator.java:134)
>   at 
> org.apache.beam.runners.direct.DoFnLifecycleManagerRemovingTransformEvaluator.processElement(DoFnLifecycleManagerRemovingTransformEvaluator.java:51)
>   at 
> org.apache.beam.runners.direct.TransformExecutor.processElements(TransformExecutor.java:139)
>   at 
> org.apache.beam.runners.direct.TransformExecutor.run(TransformExecutor.java:107)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> 

[jira] [Commented] (BEAM-1859) sorter extension depends on hadoop but does not declare as such in repository artifact

[Wesley Tanaka (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15952999#comment-15952999
 ] 

Wesley Tanaka commented on BEAM-1859:
-

Adding org.apache.hadoop:hadoop-core:0.20.2 as a dependency explicitly does 
resolve the issue, thanks, I'll just do that; I didn't know that it was a best 
practice to assume it was already installed.

In case it helps to know it, my use case is one of learning the Beam API, not 
of trying to actually accomplish something with it:

* I am trying to learn the Beam API
* So I am trying to create different toy composite PTransforms
* and I'd like to speed up the code/test/debug cycle relative to uploading code 
into a cluster
* so, despite this being nonsensical w.r.t. the actual use of Beam, I am trying 
to hack together some code to get DirectRunner to read lines from stdin and 
write lines to stdout and run the same code against different inputs to see how 
it behaves.

> sorter extension depends on hadoop but does not declare as such in repository 
> artifact
> --
>
> Key: BEAM-1859
> URL: https://issues.apache.org/jira/browse/BEAM-1859
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Affects Versions: 0.6.0
>Reporter: Wesley Tanaka
>Assignee: Davor Bonaci
> Fix For: Not applicable
>
>
> When SortValues is used via 
> {{org.apache.beam:beam-sdks-java-extensions-sorter:0.6.0}}, this exception is 
> raised:
> {noformat}
> Caused by: java.lang.NoClassDefFoundError: 
> org/apache/hadoop/conf/Configuration
>   at 
> org.apache.beam.sdk.extensions.sorter.BufferedExternalSorter.create(BufferedExternalSorter.java:98)
>   at 
> org.apache.beam.sdk.extensions.sorter.SortValues$SortValuesDoFn.processElement(SortValues.java:153)
> Caused by: java.lang.ClassNotFoundException: 
> org.apache.hadoop.conf.Configuration
>   at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>   at 
> org.apache.beam.sdk.extensions.sorter.BufferedExternalSorter.create(BufferedExternalSorter.java:98)
>   at 
> org.apache.beam.sdk.extensions.sorter.SortValues$SortValuesDoFn.processElement(SortValues.java:153)
>   at 
> org.apache.beam.sdk.extensions.sorter.SortValues$SortValuesDoFn$auxiliary$uK25yOmK.invokeProcessElement(Unknown
>  Source)
>   at 
> org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:198)
>   at 
> org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:159)
>   at 
> org.apache.beam.runners.core.PushbackSideInputDoFnRunner.processElement(PushbackSideInputDoFnRunner.java:111)
>   at 
> org.apache.beam.runners.core.PushbackSideInputDoFnRunner.processElementInReadyWindows(PushbackSideInputDoFnRunner.java:77)
>   at 
> org.apache.beam.runners.direct.ParDoEvaluator.processElement(ParDoEvaluator.java:134)
>   at 
> org.apache.beam.runners.direct.DoFnLifecycleManagerRemovingTransformEvaluator.processElement(DoFnLifecycleManagerRemovingTransformEvaluator.java:51)
>   at 
> org.apache.beam.runners.direct.TransformExecutor.processElements(TransformExecutor.java:139)
>   at 
> org.apache.beam.runners.direct.TransformExecutor.run(TransformExecutor.java:107)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> {noformat}
> I think the issue is that beam-sdks-java-extensions-sorter should declare 
> that it depends on that hadoop library but does not?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BEAM-1859) sorter extension depends on hadoop but does not declare as such in repository artifact

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci resolved BEAM-1859.

   Resolution: Not A Problem
Fix Version/s: Not applicable

> sorter extension depends on hadoop but does not declare as such in repository 
> artifact
> --
>
> Key: BEAM-1859
> URL: https://issues.apache.org/jira/browse/BEAM-1859
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Affects Versions: 0.6.0
>Reporter: Wesley Tanaka
>Assignee: Davor Bonaci
> Fix For: Not applicable
>
>
> When SortValues is used via 
> {{org.apache.beam:beam-sdks-java-extensions-sorter:0.6.0}}, this exception is 
> raised:
> {noformat}
> Caused by: java.lang.NoClassDefFoundError: 
> org/apache/hadoop/conf/Configuration
>   at 
> org.apache.beam.sdk.extensions.sorter.BufferedExternalSorter.create(BufferedExternalSorter.java:98)
>   at 
> org.apache.beam.sdk.extensions.sorter.SortValues$SortValuesDoFn.processElement(SortValues.java:153)
> Caused by: java.lang.ClassNotFoundException: 
> org.apache.hadoop.conf.Configuration
>   at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>   at 
> org.apache.beam.sdk.extensions.sorter.BufferedExternalSorter.create(BufferedExternalSorter.java:98)
>   at 
> org.apache.beam.sdk.extensions.sorter.SortValues$SortValuesDoFn.processElement(SortValues.java:153)
>   at 
> org.apache.beam.sdk.extensions.sorter.SortValues$SortValuesDoFn$auxiliary$uK25yOmK.invokeProcessElement(Unknown
>  Source)
>   at 
> org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:198)
>   at 
> org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:159)
>   at 
> org.apache.beam.runners.core.PushbackSideInputDoFnRunner.processElement(PushbackSideInputDoFnRunner.java:111)
>   at 
> org.apache.beam.runners.core.PushbackSideInputDoFnRunner.processElementInReadyWindows(PushbackSideInputDoFnRunner.java:77)
>   at 
> org.apache.beam.runners.direct.ParDoEvaluator.processElement(ParDoEvaluator.java:134)
>   at 
> org.apache.beam.runners.direct.DoFnLifecycleManagerRemovingTransformEvaluator.processElement(DoFnLifecycleManagerRemovingTransformEvaluator.java:51)
>   at 
> org.apache.beam.runners.direct.TransformExecutor.processElements(TransformExecutor.java:139)
>   at 
> org.apache.beam.runners.direct.TransformExecutor.run(TransformExecutor.java:107)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> {noformat}
> I think the issue is that beam-sdks-java-extensions-sorter should declare 
> that it depends on that hadoop library but does not?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1859) sorter extension depends on hadoop but does not declare as such in repository artifact

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15952988#comment-15952988
 ] 

Davor Bonaci commented on BEAM-1859:


"beam-sdks-java-extensions-sorter" depends on the Hadoop codebase in the 
"provided" scope:

{code}
  
2.7.1
  
[...]

  org.apache.hadoop
  hadoop-mapreduce-client-core
  ${hadoop.version}
  provided



  org.apache.hadoop
  hadoop-common
  ${hadoop.version}
  provided

{code}

It is intended for the caller to provide these dependencies -- manually 
providing the dependency, as [~jbonofre] suggested, should solve the problem.

Many Hadoop dependencies are marked as "provided" because they tend to be 
available on a Hadoop cluster by default -- it is a recommended practice in the 
Hadoop ecosystem not to include such dependencies with user code to avoid 
conflicts. On the other hand, when running locally with a Direct Runner, such 
dependencies tend not to be available, causing the issue you just saw.

Therefore, this specific issue in the sorter extension is "Working as 
Intended". Separately, it is debatable should the direct runner try to mimic a 
real cluster more reliably -- the answer is not clear to me yet, and I think 
this is worth a dev@ discussion.

> sorter extension depends on hadoop but does not declare as such in repository 
> artifact
> --
>
> Key: BEAM-1859
> URL: https://issues.apache.org/jira/browse/BEAM-1859
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Affects Versions: 0.6.0
>Reporter: Wesley Tanaka
>Assignee: Davor Bonaci
>
> When SortValues is used via 
> {{org.apache.beam:beam-sdks-java-extensions-sorter:0.6.0}}, this exception is 
> raised:
> {noformat}
> Caused by: java.lang.NoClassDefFoundError: 
> org/apache/hadoop/conf/Configuration
>   at 
> org.apache.beam.sdk.extensions.sorter.BufferedExternalSorter.create(BufferedExternalSorter.java:98)
>   at 
> org.apache.beam.sdk.extensions.sorter.SortValues$SortValuesDoFn.processElement(SortValues.java:153)
> Caused by: java.lang.ClassNotFoundException: 
> org.apache.hadoop.conf.Configuration
>   at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>   at 
> org.apache.beam.sdk.extensions.sorter.BufferedExternalSorter.create(BufferedExternalSorter.java:98)
>   at 
> org.apache.beam.sdk.extensions.sorter.SortValues$SortValuesDoFn.processElement(SortValues.java:153)
>   at 
> org.apache.beam.sdk.extensions.sorter.SortValues$SortValuesDoFn$auxiliary$uK25yOmK.invokeProcessElement(Unknown
>  Source)
>   at 
> org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:198)
>   at 
> org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:159)
>   at 
> org.apache.beam.runners.core.PushbackSideInputDoFnRunner.processElement(PushbackSideInputDoFnRunner.java:111)
>   at 
> org.apache.beam.runners.core.PushbackSideInputDoFnRunner.processElementInReadyWindows(PushbackSideInputDoFnRunner.java:77)
>   at 
> org.apache.beam.runners.direct.ParDoEvaluator.processElement(ParDoEvaluator.java:134)
>   at 
> org.apache.beam.runners.direct.DoFnLifecycleManagerRemovingTransformEvaluator.processElement(DoFnLifecycleManagerRemovingTransformEvaluator.java:51)
>   at 
> org.apache.beam.runners.direct.TransformExecutor.processElements(TransformExecutor.java:139)
>   at 
> org.apache.beam.runners.direct.TransformExecutor.run(TransformExecutor.java:107)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> {noformat}
> I think the issue is that beam-sdks-java-extensions-sorter should declare 
> that it depends on that hadoop library but does not?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BEAM-1843) 'PDone' object has no attribute 'to_runner_api'

[Ahmet Altay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay resolved BEAM-1843.
---
   Resolution: Fixed
Fix Version/s: Not applicable

> 'PDone' object has no attribute 'to_runner_api'
> ---
>
> Key: BEAM-1843
> URL: https://issues.apache.org/jira/browse/BEAM-1843
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Robert Bradshaw
> Fix For: Not applicable
>
>
> Post commit failure with 
> (https://builds.apache.org/view/Beam/job/beam_PostCommit_Python_Verify/1676/consoleFull):
> ...
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/pipeline.py",
>  line 512, in 
> for tag, out in self.outputs.items()},
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/runners/pipeline_context.py",
>  line 52, in get_id
> self._id_to_proto[id] = obj.to_runner_api(self._pipeline_context)
> AttributeError: 'PDone' object has no attribute 'to_runner_api'



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1859) sorter extension depends on hadoop but does not declare as such in repository artifact

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-1859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15952967#comment-15952967
 ] 

Jean-Baptiste Onofré commented on BEAM-1859:


As a workaround, can you try to add hadoop client in your Maven dependencies 
set ?

> sorter extension depends on hadoop but does not declare as such in repository 
> artifact
> --
>
> Key: BEAM-1859
> URL: https://issues.apache.org/jira/browse/BEAM-1859
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Affects Versions: 0.6.0
>Reporter: Wesley Tanaka
>Assignee: Davor Bonaci
>
> When SortValues is used via 
> {{org.apache.beam:beam-sdks-java-extensions-sorter:0.6.0}}, this exception is 
> raised:
> {noformat}
> Caused by: java.lang.NoClassDefFoundError: 
> org/apache/hadoop/conf/Configuration
>   at 
> org.apache.beam.sdk.extensions.sorter.BufferedExternalSorter.create(BufferedExternalSorter.java:98)
>   at 
> org.apache.beam.sdk.extensions.sorter.SortValues$SortValuesDoFn.processElement(SortValues.java:153)
> Caused by: java.lang.ClassNotFoundException: 
> org.apache.hadoop.conf.Configuration
>   at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>   at 
> org.apache.beam.sdk.extensions.sorter.BufferedExternalSorter.create(BufferedExternalSorter.java:98)
>   at 
> org.apache.beam.sdk.extensions.sorter.SortValues$SortValuesDoFn.processElement(SortValues.java:153)
>   at 
> org.apache.beam.sdk.extensions.sorter.SortValues$SortValuesDoFn$auxiliary$uK25yOmK.invokeProcessElement(Unknown
>  Source)
>   at 
> org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:198)
>   at 
> org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:159)
>   at 
> org.apache.beam.runners.core.PushbackSideInputDoFnRunner.processElement(PushbackSideInputDoFnRunner.java:111)
>   at 
> org.apache.beam.runners.core.PushbackSideInputDoFnRunner.processElementInReadyWindows(PushbackSideInputDoFnRunner.java:77)
>   at 
> org.apache.beam.runners.direct.ParDoEvaluator.processElement(ParDoEvaluator.java:134)
>   at 
> org.apache.beam.runners.direct.DoFnLifecycleManagerRemovingTransformEvaluator.processElement(DoFnLifecycleManagerRemovingTransformEvaluator.java:51)
>   at 
> org.apache.beam.runners.direct.TransformExecutor.processElements(TransformExecutor.java:139)
>   at 
> org.apache.beam.runners.direct.TransformExecutor.run(TransformExecutor.java:107)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> {noformat}
> I think the issue is that beam-sdks-java-extensions-sorter should declare 
> that it depends on that hadoop library but does not?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1859) sorter extension depends on hadoop but does not declare as such in repository artifact

[Wesley Tanaka (JIRA)]

Wesley Tanaka created BEAM-1859:
---

 Summary: sorter extension depends on hadoop but does not declare 
as such in repository artifact
 Key: BEAM-1859
 URL: https://issues.apache.org/jira/browse/BEAM-1859
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-extensions
Affects Versions: 0.6.0
Reporter: Wesley Tanaka
Assignee: Davor Bonaci


When SortValues is used via 
{{org.apache.beam:beam-sdks-java-extensions-sorter:0.6.0}}, this exception is 
raised:

{noformat}
Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration
at 
org.apache.beam.sdk.extensions.sorter.BufferedExternalSorter.create(BufferedExternalSorter.java:98)
at 
org.apache.beam.sdk.extensions.sorter.SortValues$SortValuesDoFn.processElement(SortValues.java:153)
Caused by: java.lang.ClassNotFoundException: 
org.apache.hadoop.conf.Configuration
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at 
org.apache.beam.sdk.extensions.sorter.BufferedExternalSorter.create(BufferedExternalSorter.java:98)
at 
org.apache.beam.sdk.extensions.sorter.SortValues$SortValuesDoFn.processElement(SortValues.java:153)
at 
org.apache.beam.sdk.extensions.sorter.SortValues$SortValuesDoFn$auxiliary$uK25yOmK.invokeProcessElement(Unknown
 Source)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:198)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:159)
at 
org.apache.beam.runners.core.PushbackSideInputDoFnRunner.processElement(PushbackSideInputDoFnRunner.java:111)
at 
org.apache.beam.runners.core.PushbackSideInputDoFnRunner.processElementInReadyWindows(PushbackSideInputDoFnRunner.java:77)
at 
org.apache.beam.runners.direct.ParDoEvaluator.processElement(ParDoEvaluator.java:134)
at 
org.apache.beam.runners.direct.DoFnLifecycleManagerRemovingTransformEvaluator.processElement(DoFnLifecycleManagerRemovingTransformEvaluator.java:51)
at 
org.apache.beam.runners.direct.TransformExecutor.processElements(TransformExecutor.java:139)
at 
org.apache.beam.runners.direct.TransformExecutor.run(TransformExecutor.java:107)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{noformat}

I think the issue is that beam-sdks-java-extensions-sorter should declare that 
it depends on that hadoop library but does not?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1151) BigQueryIO.Write has no way of handling failures

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15952953#comment-15952953
 ] 

ASF GitHub Bot commented on BEAM-1151:
--

GitHub user reuvenlax opened a pull request:

https://github.com/apache/beam/pull/2404

[BEAM-1151] Add interface for accessing failed BigQuery inserts

This PR adds a new return type to BigQueryIO.Write, to replace PDone. This 
enables future features such as accessing failed inserts as a PCollection. 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/reuvenlax/incubator-beam 
prepare_for_dead_letter

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2404.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2404


commit 44dd618da08711d20614ad0e05f0e871e12b4b32
Author: Reuven Lax 
Date:   2017-04-03T03:12:05Z

Add new return type to BigQueryIO.Write that allows for accessing failed 
inserts. Accessing failed inserts is not yet implemented - this PR is to ensure 
that the interface is there before we cut off backwards-incompatible changes.




> BigQueryIO.Write has no way of handling failures
> 
>
> Key: BEAM-1151
> URL: https://issues.apache.org/jira/browse/BEAM-1151
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 0.4.0
>Reporter: Reuven Lax
>Assignee: Reuven Lax
> Fix For: 0.3.0-incubating
>
>
> BigQueryIO.Write in streaming mode uses BigQuery's streaming insert API. 
> Inserts can fail for a variety of reasons, and the current connector provides 
> no way of managing those failures. This PR allows better failure handling.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2404: [BEAM-1151] Add interface for accessing failed BigQ...

[reuvenlax]

GitHub user reuvenlax opened a pull request:

https://github.com/apache/beam/pull/2404

[BEAM-1151] Add interface for accessing failed BigQuery inserts

This PR adds a new return type to BigQueryIO.Write, to replace PDone. This 
enables future features such as accessing failed inserts as a PCollection. 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/reuvenlax/incubator-beam 
prepare_for_dead_letter

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2404.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2404


commit 44dd618da08711d20614ad0e05f0e871e12b4b32
Author: Reuven Lax 
Date:   2017-04-03T03:12:05Z

Add new return type to BigQueryIO.Write that allows for accessing failed 
inserts. Accessing failed inserts is not yet implemented - this PR is to ensure 
that the interface is there before we cut off backwards-incompatible changes.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Spark #1479

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_Dataflow #262

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam5 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Pruning obsolete local branches
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* +refs/pull/*:refs/remotes/origin/pr/* 
 > --prune
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision fe1d4124e7d51ec45869fb9e95cc8243f5891553 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f fe1d4124e7d51ec45869fb9e95cc8243f5891553
 > git rev-list fe1d4124e7d51ec45869fb9e95cc8243f5891553 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Dataflow] $ /bin/bash -xe 
/tmp/hudson6512253486991864100.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Dataflow] $ /bin/bash -xe 
/tmp/hudson7385739790240899376.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Dataflow] $ /bin/bash -xe 
/tmp/hudson5191121802013883002.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): python-gflags==3.1.1 
in /home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
/usr/lib/python2.7/dist-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.11 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Cleaning up...
[beam_PerformanceTests_Dataflow] $ /bin/bash -xe 
/tmp/hudson3439536893932770801.sh
+ python PerfKitBenchmarker/pkb.py --project=apache-beam-testing 
--dpb_log_level=INFO --maven_binary=/home/jenkins/tools/maven/latest/bin/mvn 
--bigquery_table=beam_performance.pkb_results --official=true 
--benchmarks=dpb_wordcount_benchmark 
--dpb_dataflow_staging_location=gs://temp-storage-for-perf-tests/staging 
--dpb_wordcount_input=dataflow-samples/shakespeare/kinglear.txt 
--config_override=dpb_wordcount_benchmark.dpb_service.service_type=dataflow
WARNING:root:File resource loader root perfkitbenchmarker/data/ycsb is not a 
directory.
2017-04-03 00:01:56,243 acc615e0 MainThread INFO Verbose logging to: 
/tmp/perfkitbenchmarker/runs/acc615e0/pkb.log
2017-04-03 00:01:56,244 acc615e0 MainThread INFO PerfKitBenchmarker 
version: v1.11.0-29-g8c92c13
2017-04-03 00:01:56,244 acc615e0 MainThread INFO Flag values:
--maven_binary=/home/jenkins/tools/maven/latest/bin/mvn
--project=apache-beam-testing

[jira] [Commented] (BEAM-1676) SdkCoreApiSurfaceTest Failed When Directory Contains Space

[Jason Kuster (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15952864#comment-15952864
 ] 

Jason Kuster commented on BEAM-1676:


Briefly chatted with [~dhalp...@google.com] -- he seemed to be under the 
impression that grabbing the next version of Guava was going to be difficult 
for Java8-related issues. Dan, can you elaborate and provide guidance here?

> SdkCoreApiSurfaceTest Failed When Directory Contains Space
> --
>
> Key: BEAM-1676
> URL: https://issues.apache.org/jira/browse/BEAM-1676
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Mark Liu
>Assignee: Stas Levin
>
> Test failed if build directory contains space. For example: "~/dir with 
> space/beam/..."
> The failure happened on Jenkins and can be reproduced from local.
> GcpApiSurfaceTest may have same problem.
> error is:
> {code}
> Failed tests: 
>   SdkCoreApiSurfaceTest.testSdkApiSurface:59 
> Expected: API surface to include only:
>   Classes in package "org.apache.beam"
>   Classes in package "com.google.api.client"
>   Classes in package "com.google.api.services.bigquery"
>   Classes in package "com.google.api.services.cloudresourcemanager"
>   Classes in package "com.google.api.services.pubsub"
>   Classes in package "com.google.api.services.storage"
>   Classes in package "com.google.auth"
>   Classes in package "com.google.protobuf"
>   Classes in package "com.fasterxml.jackson.annotation"
>   Classes in package "com.fasterxml.jackson.core"
>   Classes in package "com.fasterxml.jackson.databind"
>   Classes in package "org.apache.avro"
>   Classes in package "org.hamcrest"
>   Classes in package "org.codehaus.jackson"
>   Classes in package "org.joda.time"
>   Classes in package "org.junit"
>   
>  but: The following white-listed scopes did not have matching classes on 
> the API surface:
>   No Classes in package "com.fasterxml.jackson.annotation"
>   No Classes in package "com.fasterxml.jackson.core"
>   No Classes in package "com.fasterxml.jackson.databind"
>   No Classes in package "com.google.api.client"
>   No Classes in package "com.google.api.services.bigquery"
>   No Classes in package "com.google.api.services.cloudresourcemanager"
>   No Classes in package "com.google.api.services.pubsub"
>   No Classes in package "com.google.api.services.storage"
>   No Classes in package "com.google.auth"
>   No Classes in package "com.google.protobuf"
>   No Classes in package "org.apache.avro"
>   No Classes in package "org.apache.beam"
>   No Classes in package "org.codehaus.jackson"
>   No Classes in package "org.hamcrest"
>   No Classes in package "org.joda.time"
>   No Classes in package "org.junit"
> {code}
> Job link from Jenkins:
> https://builds.apache.org/job/beam_PostCommit_Java_Version_Test/14/
> One of the Jenkins job uses "JDK 1.8 (latest)" which is also part of project 
> directory.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Spark #1478

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-1773) Consider allowing Source#validate() to throw exception

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15952662#comment-15952662
 ] 

Ted Yu commented on BEAM-1773:
--

JB:
If you are busy, I can work on this. 

> Consider allowing Source#validate() to throw exception
> --
>
> Key: BEAM-1773
> URL: https://issues.apache.org/jira/browse/BEAM-1773
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Ted Yu
>Assignee: Jean-Baptiste Onofré
>
> In HDFSFileSource.java :
> {code}
>   @Override
>   public void validate() {
> ...
>   } catch (IOException | InterruptedException e) {
> throw new RuntimeException(e);
>   }
> {code}
> Source#validate() should be allowed to throw exception so that we don't 
> resort to using RuntimeException.
> Here was related thread on mailing list:
> http://search-hadoop.com/m/Beam/gfKHFOwE0uETxae?subj=Re+why+Source+validate+is+not+declared+to+throw+any+exception



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: beam_PerformanceTests_Dataflow #260

[Apache Jenkins Server]

See 


Changes:

[staslevin] [BEAM-1048] Added a read duration metric to SparkUnboundedSource.

--
[...truncated 240.24 KB...]
 * [new ref] refs/pull/2289/head -> origin/pr/2289/head
 * [new ref] refs/pull/2289/merge -> origin/pr/2289/merge
 * [new ref] refs/pull/2290/head -> origin/pr/2290/head
 * [new ref] refs/pull/2290/merge -> origin/pr/2290/merge
 * [new ref] refs/pull/2291/head -> origin/pr/2291/head
 * [new ref] refs/pull/2291/merge -> origin/pr/2291/merge
 * [new ref] refs/pull/2292/head -> origin/pr/2292/head
 * [new ref] refs/pull/2292/merge -> origin/pr/2292/merge
 * [new ref] refs/pull/2293/head -> origin/pr/2293/head
 * [new ref] refs/pull/2293/merge -> origin/pr/2293/merge
 * [new ref] refs/pull/2294/head -> origin/pr/2294/head
 * [new ref] refs/pull/2294/merge -> origin/pr/2294/merge
 * [new ref] refs/pull/2295/head -> origin/pr/2295/head
 * [new ref] refs/pull/2295/merge -> origin/pr/2295/merge
 * [new ref] refs/pull/2296/head -> origin/pr/2296/head
 * [new ref] refs/pull/2296/merge -> origin/pr/2296/merge
 * [new ref] refs/pull/2297/head -> origin/pr/2297/head
 * [new ref] refs/pull/2297/merge -> origin/pr/2297/merge
 * [new ref] refs/pull/2298/head -> origin/pr/2298/head
 * [new ref] refs/pull/2298/merge -> origin/pr/2298/merge
 * [new ref] refs/pull/2299/head -> origin/pr/2299/head
 * [new ref] refs/pull/2300/head -> origin/pr/2300/head
 * [new ref] refs/pull/2300/merge -> origin/pr/2300/merge
 * [new ref] refs/pull/2301/head -> origin/pr/2301/head
 * [new ref] refs/pull/2301/merge -> origin/pr/2301/merge
 * [new ref] refs/pull/2302/head -> origin/pr/2302/head
 * [new ref] refs/pull/2303/head -> origin/pr/2303/head
 * [new ref] refs/pull/2303/merge -> origin/pr/2303/merge
 * [new ref] refs/pull/2304/head -> origin/pr/2304/head
 * [new ref] refs/pull/2304/merge -> origin/pr/2304/merge
 * [new ref] refs/pull/2305/head -> origin/pr/2305/head
 * [new ref] refs/pull/2305/merge -> origin/pr/2305/merge
 * [new ref] refs/pull/2306/head -> origin/pr/2306/head
 * [new ref] refs/pull/2306/merge -> origin/pr/2306/merge
 * [new ref] refs/pull/2307/head -> origin/pr/2307/head
 * [new ref] refs/pull/2307/merge -> origin/pr/2307/merge
 * [new ref] refs/pull/2308/head -> origin/pr/2308/head
 * [new ref] refs/pull/2308/merge -> origin/pr/2308/merge
 * [new ref] refs/pull/2309/head -> origin/pr/2309/head
 * [new ref] refs/pull/2309/merge -> origin/pr/2309/merge
 * [new ref] refs/pull/2310/head -> origin/pr/2310/head
 * [new ref] refs/pull/2310/merge -> origin/pr/2310/merge
 * [new ref] refs/pull/2311/head -> origin/pr/2311/head
 * [new ref] refs/pull/2311/merge -> origin/pr/2311/merge
 * [new ref] refs/pull/2312/head -> origin/pr/2312/head
 * [new ref] refs/pull/2312/merge -> origin/pr/2312/merge
 * [new ref] refs/pull/2313/head -> origin/pr/2313/head
 * [new ref] refs/pull/2313/merge -> origin/pr/2313/merge
 * [new ref] refs/pull/2314/head -> origin/pr/2314/head
 * [new ref] refs/pull/2314/merge -> origin/pr/2314/merge
 * [new ref] refs/pull/2315/head -> origin/pr/2315/head
 * [new ref] refs/pull/2315/merge -> origin/pr/2315/merge
 * [new ref] refs/pull/2316/head -> origin/pr/2316/head
 * [new ref] refs/pull/2316/merge -> origin/pr/2316/merge
 * [new ref] refs/pull/2317/head -> origin/pr/2317/head
 * [new ref] refs/pull/2317/merge -> origin/pr/2317/merge
 * [new ref] refs/pull/2318/head -> origin/pr/2318/head
 * [new ref] refs/pull/2318/merge -> origin/pr/2318/merge
 * [new ref] refs/pull/2319/head -> origin/pr/2319/head
 * [new ref] refs/pull/2319/merge -> origin/pr/2319/merge
 * [new ref] refs/pull/2320/head -> origin/pr/2320/head
 * [new ref] refs/pull/2321/head -> origin/pr/2321/head
 * [new ref] refs/pull/2321/merge -> origin/pr/2321/merge
 * [new ref] refs/pull/2322/head -> origin/pr/2322/head
 * [new ref] refs/pull/2322/merge -> origin/pr/2322/merge
 * [new ref] refs/pull/2323/head -> origin/pr/2323/head
 * [new ref] refs/pull/2323/merge -> origin/pr/2323/merge
 * [new ref] refs/pull/2324/head -> origin/pr/2324/head
 * [new ref] refs/pull/2325/head -> origin/pr/2325/head
 * [new ref] refs/pull/2325/merge -> origin/pr/2325/merge
 * [new ref] refs/pull/2326/head -> origin/pr/2326/head
 * [new ref] refs/pull/2326/merge -> origin/pr/2326/merge
 * [new ref] refs/pull/2327/head -> 

[jira] [Commented] (BEAM-1777) If PipelineEnforcement throws an exception after Pipeline.run() fails, it overwrites the original failure

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15952626#comment-15952626
 ] 

ASF GitHub Bot commented on BEAM-1777:
--

GitHub user staslev opened a pull request:

https://github.com/apache/beam/pull/2403

[BEAM-1777] In certain circumstances PipelineEnforcement shades exceptions.

…ine's exceptions.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/staslev/beam 
BEAM-1777-PipelineEnforcement-shades-run-exceptions

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2403.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2403


commit 5ecf911f2f826594d829d4820c5e26e6705b8afe
Author: Stas Levin 
Date:   2017-04-02T11:09:43Z

[BEAM-1777] In certain circumstances PipelineEnforcement shades pipeline's 
exceptions.




> If PipelineEnforcement throws an exception after Pipeline.run() fails, it 
> overwrites the original failure
> -
>
> Key: BEAM-1777
> URL: https://issues.apache.org/jira/browse/BEAM-1777
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Stas Levin
>
> The exception is thrown out of the finally block after the original exception 
> was already thrown, and is not properly suppressed.
> If the Pipeline throws an exception in construction (including validation 
> failures), this can cause {{AbandonedNodeEnforcement}} to attempt to traverse 
> a {{Pipeline}} which is invalid, which throws another exception. That 
> exception is improperly propagated instead of the original failure, which 
> complicates debugging.
> An example is using an unkeyed input PCollection to a Stateful ParDo. The 
> validation will fail, but the error message will suggest that a node has 
> incompletely specified outputs (which is a consequence rather than a cause of 
> having an invalid transform).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2403: [BEAM-1777] In certain circumstances PipelineEnforc...

[staslev]

GitHub user staslev opened a pull request:

https://github.com/apache/beam/pull/2403

[BEAM-1777] In certain circumstances PipelineEnforcement shades exceptions.

…ine's exceptions.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/staslev/beam 
BEAM-1777-PipelineEnforcement-shades-run-exceptions

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2403.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2403


commit 5ecf911f2f826594d829d4820c5e26e6705b8afe
Author: Stas Levin 
Date:   2017-04-02T11:09:43Z

[BEAM-1777] In certain circumstances PipelineEnforcement shades pipeline's 
exceptions.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1048) Spark Runner streaming batch duration does not include duration of reading from source

[Stas Levin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15952616#comment-15952616
 ] 

Stas Levin commented on BEAM-1048:
--

While batch duration is still (and probably won't be) reported as part of what 
is known as the "batch duration", the PR linked to this issue adds a separate 
metric that reports the max read duration per-batch.

To get a sense of the entire "batch duration", i.e., read duration + process 
duration, one can sum the read and process delay metrics.
This is not a bullet proof solution, as technically there is no guarantee that 
at a given point in time, the read duration and process delay metrics actually 
refer to the same batch id, that said, this should provide a general idea of 
how an application is doing vis-à-vis "batch duration".

> Spark Runner streaming batch duration does not include duration of reading 
> from source 
> ---
>
> Key: BEAM-1048
> URL: https://issues.apache.org/jira/browse/BEAM-1048
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 0.4.0
>Reporter: Kobi Salant
>Assignee: Stas Levin
> Fix For: First stable release
>
>
> Spark Runner streaming batch duration does not include duration of reading 
> from source this is because we perform rdd.count in SparkUnboundedSourcewhich 
> that invokes a regular spark job outside the streaming context.
> We do it for reporting the batch size both for UI and back pressure



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BEAM-1048) Spark Runner streaming batch duration does not include duration of reading from source

[Stas Levin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stas Levin resolved BEAM-1048.
--
Resolution: Workaround

> Spark Runner streaming batch duration does not include duration of reading 
> from source 
> ---
>
> Key: BEAM-1048
> URL: https://issues.apache.org/jira/browse/BEAM-1048
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 0.4.0
>Reporter: Kobi Salant
>Assignee: Stas Levin
> Fix For: First stable release
>
>
> Spark Runner streaming batch duration does not include duration of reading 
> from source this is because we perform rdd.count in SparkUnboundedSourcewhich 
> that invokes a regular spark job outside the streaming context.
> We do it for reporting the batch size both for UI and back pressure



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1048) Spark Runner streaming batch duration does not include duration of reading from source

[Stas Levin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stas Levin updated BEAM-1048:
-
Fix Version/s: First stable release

> Spark Runner streaming batch duration does not include duration of reading 
> from source 
> ---
>
> Key: BEAM-1048
> URL: https://issues.apache.org/jira/browse/BEAM-1048
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 0.4.0
>Reporter: Kobi Salant
>Assignee: Stas Levin
> Fix For: First stable release
>
>
> Spark Runner streaming batch duration does not include duration of reading 
> from source this is because we perform rdd.count in SparkUnboundedSourcewhich 
> that invokes a regular spark job outside the streaming context.
> We do it for reporting the batch size both for UI and back pressure



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1048) Spark Runner streaming batch duration does not include duration of reading from source

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15952613#comment-15952613
 ] 

ASF GitHub Bot commented on BEAM-1048:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2073


> Spark Runner streaming batch duration does not include duration of reading 
> from source 
> ---
>
> Key: BEAM-1048
> URL: https://issues.apache.org/jira/browse/BEAM-1048
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 0.4.0
>Reporter: Kobi Salant
>Assignee: Stas Levin
>
> Spark Runner streaming batch duration does not include duration of reading 
> from source this is because we perform rdd.count in SparkUnboundedSourcewhich 
> that invokes a regular spark job outside the streaming context.
> We do it for reporting the batch size both for UI and back pressure



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2073: [BEAM-1048] Added a per-batch read duration metric ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2073


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #2073

[staslevin]

This closes #2073


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/fe1d4124
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/fe1d4124
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/fe1d4124

Branch: refs/heads/master
Commit: fe1d4124e7d51ec45869fb9e95cc8243f5891553
Parents: ea33e33 38e00f3
Author: Stas Levin 
Authored: Sun Apr 2 13:07:28 2017 +0300
Committer: Stas Levin 
Committed: Sun Apr 2 13:07:28 2017 +0300

--
 .../runners/spark/io/SparkUnboundedSource.java  | 36 ++--
 .../spark/stateful/StateSpecFunctions.java  | 21 +---
 2 files changed, 50 insertions(+), 7 deletions(-)
--

[1/2] beam git commit: [BEAM-1048] Added a read duration metric to SparkUnboundedSource.

[staslevin]

Repository: beam
Updated Branches:
  refs/heads/master ea33e3373 -> fe1d4124e


[BEAM-1048] Added a read duration metric to SparkUnboundedSource.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/38e00f3f
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/38e00f3f
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/38e00f3f

Branch: refs/heads/master
Commit: 38e00f3f01e0125f6ac929da0a9d1c93fcc6787d
Parents: ea33e33
Author: Stas Levin 
Authored: Sun Apr 2 08:46:14 2017 +0300
Committer: Stas Levin 
Committed: Sun Apr 2 13:07:02 2017 +0300

--
 .../runners/spark/io/SparkUnboundedSource.java  | 36 ++--
 .../spark/stateful/StateSpecFunctions.java  | 21 +---
 2 files changed, 50 insertions(+), 7 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/38e00f3f/runners/spark/src/main/java/org/apache/beam/runners/spark/io/SparkUnboundedSource.java
--
diff --git 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/io/SparkUnboundedSource.java
 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/io/SparkUnboundedSource.java
index 162bca4..6b34590 100644
--- 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/io/SparkUnboundedSource.java
+++ 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/io/SparkUnboundedSource.java
@@ -18,6 +18,8 @@
 
 package org.apache.beam.runners.spark.io;
 
+import java.io.Closeable;
+import java.io.IOException;
 import java.io.Serializable;
 import java.util.Collections;
 import org.apache.beam.runners.spark.SparkPipelineOptions;
@@ -32,6 +34,10 @@ import 
org.apache.beam.runners.spark.util.GlobalWatermarkHolder.SparkWatermarks;
 import org.apache.beam.sdk.io.Source;
 import org.apache.beam.sdk.io.UnboundedSource;
 import org.apache.beam.sdk.io.UnboundedSource.CheckpointMark;
+import org.apache.beam.sdk.metrics.Gauge;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.metrics.MetricsContainer;
+import org.apache.beam.sdk.metrics.MetricsEnvironment;
 import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
 import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
 import org.apache.beam.sdk.util.WindowedValue;
@@ -105,7 +111,8 @@ public class SparkUnboundedSource {
 JavaDStream metadataDStream = mapWithStateDStream.map(new 
Tuple2MetadataFunction());
 
 // register ReadReportDStream to report information related to this read.
-new ReadReportDStream(metadataDStream.dstream(), id, getSourceName(source, 
id)).register();
+new ReadReportDStream(metadataDStream.dstream(), id, getSourceName(source, 
id), stepName)
+.register();
 
 // output the actual (deserialized) stream.
 WindowedValue.FullWindowedValueCoder coder =
@@ -148,18 +155,25 @@ public class SparkUnboundedSource {
* Updates {@link MetricsAccumulator} with metrics reported in the 
read.
*/
   private static class ReadReportDStream extends DStream {
+
+private static final String READ_DURATION_MILLIS = "readDurationMillis";
+private static final String NAMESPACE = "spark-runner.io";
+
 private final DStream parent;
 private final int inputDStreamId;
 private final String sourceName;
+private final String stepName;
 
 ReadReportDStream(
 DStream parent,
 int inputDStreamId,
-String sourceName) {
+String sourceName,
+String stepName) {
   super(parent.ssc(), JavaSparkContext$.MODULE$.fakeClassTag());
   this.parent = parent;
   this.inputDStreamId = inputDStreamId;
   this.sourceName = sourceName;
+  this.stepName = stepName;
 }
 
 @Override
@@ -182,6 +196,7 @@ public class SparkUnboundedSource {
   SparkWatermarks sparkWatermark = null;
   Instant globalLowWatermarkForBatch = BoundedWindow.TIMESTAMP_MIN_VALUE;
   Instant globalHighWatermarkForBatch = BoundedWindow.TIMESTAMP_MIN_VALUE;
+  long maxReadDuration = 0;
   if (parentRDDOpt.isDefined()) {
 JavaRDD parentRDD = parentRDDOpt.get().toJavaRDD();
 for (Metadata metadata: parentRDD.collect()) {
@@ -196,6 +211,16 @@ public class SparkUnboundedSource {
   globalHighWatermarkForBatch.isBefore(partitionHighWatermark)
   ? partitionHighWatermark : globalHighWatermarkForBatch;
   // Update metrics reported in the read
+  final Gauge gauge = Metrics.gauge(NAMESPACE, READ_DURATION_MILLIS);
+  final MetricsContainer container = 
metadata.getMetricsContainer().getContainer(stepName);
+  try (Closeable ignored = 
MetricsEnvironment.scopedMetricsContainer(container)) {
+final long readDurationMillis = metadata.getReadDurationMillis();

[jira] [Updated] (BEAM-25) Add user-ready API for interacting with state

[Wesley Tanaka (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-25?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wesley Tanaka updated BEAM-25:
--
Description: 
Our current state API is targeted at runner implementers, not pipeline authors. 
As such it has many capabilities that are not necessary nor desirable for 
simple use cases of stateful ParDo (such as dynamic state tag creation). 
Implement a simple state intended for user access.

(Details of our current thoughts in forthcoming design doc: 
https://s.apache.org/beam-state)

  was:
Our current state API is targeted at runner implementers, not pipeline authors. 
As such it has many capabilities that are not necessary nor desirable for 
simple use cases of stateful ParDo (such as dynamic state tag creation). 
Implement a simple state intended for user access.

(Details of our current thoughts in forthcoming design doc)


> Add user-ready API for interacting with state
> -
>
> Key: BEAM-25
> URL: https://issues.apache.org/jira/browse/BEAM-25
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>  Labels: State
> Fix For: 0.6.0
>
>
> Our current state API is targeted at runner implementers, not pipeline 
> authors. As such it has many capabilities that are not necessary nor 
> desirable for simple use cases of stateful ParDo (such as dynamic state tag 
> creation). Implement a simple state intended for user access.
> (Details of our current thoughts in forthcoming design doc: 
> https://s.apache.org/beam-state)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1851) Sample.fixedSizedGlobally documentation should include single worker memory constraint

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15952581#comment-15952581
 ] 

Davor Bonaci commented on BEAM-1851:


Thanks [~mdshalda] -- your contribution is much appreciated!

> Sample.fixedSizedGlobally documentation should include single worker memory 
> constraint
> --
>
> Key: BEAM-1851
> URL: https://issues.apache.org/jira/browse/BEAM-1851
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Affects Versions: 0.6.0
>Reporter: Mark Shalda
>Assignee: Mark Shalda
>Priority: Trivial
>  Labels: doc
>
> Sample.any mentions in the documentation:
> " All of the elements of the output {@code PCollection} should fit into main 
> memory of a single worker machine.  This operation does not run in parallel."
> This should also be mentioned for fixedSizedGlobally.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-1851) Sample.fixedSizedGlobally documentation should include single worker memory constraint

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci reassigned BEAM-1851:
--

Assignee: Mark Shalda  (was: Davor Bonaci)

> Sample.fixedSizedGlobally documentation should include single worker memory 
> constraint
> --
>
> Key: BEAM-1851
> URL: https://issues.apache.org/jira/browse/BEAM-1851
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Affects Versions: 0.6.0
>Reporter: Mark Shalda
>Assignee: Mark Shalda
>Priority: Trivial
>  Labels: doc
>
> Sample.any mentions in the documentation:
> " All of the elements of the output {@code PCollection} should fit into main 
> memory of a single worker machine.  This operation does not run in parallel."
> This should also be mentioned for fixedSizedGlobally.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1852) dead links on beam website

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15952579#comment-15952579
 ] 

Davor Bonaci commented on BEAM-1852:


Not sure -- we'll see if it persists.

> dead links on beam website
> --
>
> Key: BEAM-1852
> URL: https://issues.apache.org/jira/browse/BEAM-1852
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Stephen Sisk
>Assignee: Davor Bonaci
>
> Output from rake: 
> Checking 270 external links...
> Ran on 98 files!
> - ./content/subdir/blog/2017/02/01/graduation-media-recap.html
>   *  External link 
> https://globenewswire.com/news-release/2017/01/10/904692/0/en/The-Apache-Software-Foundation-Announces-Apache-Beam-as-a-Top-Level-Project.html
>  failed: response code 0 means something's wrong.
>  It's possible libcurl couldn't connect to the server or perhaps 
> the request timed out.
>  Sometimes, making too many requests at once also breaks things.
>  Either way, the return message (if any) from the server is: Peer 
> certificate cannot be authenticated with given CA certificates
> [stephen note: I could not confirm this manually, but that error seems.. 
> interesting]
> - ./content/subdir/contribute/contribution-guide/index.html
>   *  External link https://www.apache.org/licenses/icla.txt failed: 404 No 
> error
> [stephen note: manually confirmed this]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[3/3] beam-site git commit: This closes #198

[davor]

This closes #198


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/787d5706
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/787d5706
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/787d5706

Branch: refs/heads/asf-site
Commit: 787d5706dbe48b0b1815ac02936a095ee5191189
Parents: 0ee0324 8de1bcc
Author: Davor Bonaci 
Authored: Sun Apr 2 09:51:18 2017 +0200
Committer: Davor Bonaci 
Committed: Sun Apr 2 09:51:18 2017 +0200

--
 content/documentation/runners/capability-matrix/index.html | 4 ++--
 src/_data/capability-matrix.yml| 4 ++--
 2 files changed, 4 insertions(+), 4 deletions(-)
--

[1/3] beam-site git commit: Update Capability Matrix: Flink supports state/timers in batch mode

[davor]

Repository: beam-site
Updated Branches:
  refs/heads/asf-site 0ee032422 -> 787d5706d


Update Capability Matrix: Flink supports state/timers in batch mode


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/6261b1ad
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/6261b1ad
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/6261b1ad

Branch: refs/heads/asf-site
Commit: 6261b1ad5095cb80fd5635d64f2c40fd223e5e8c
Parents: 0ee0324
Author: Aljoscha Krettek 
Authored: Fri Mar 31 11:55:00 2017 +0200
Committer: Aljoscha Krettek 
Committed: Fri Mar 31 11:55:00 2017 +0200

--
 src/_data/capability-matrix.yml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam-site/blob/6261b1ad/src/_data/capability-matrix.yml
--
diff --git a/src/_data/capability-matrix.yml b/src/_data/capability-matrix.yml
index 89f89a2..1dc2fad 100644
--- a/src/_data/capability-matrix.yml
+++ b/src/_data/capability-matrix.yml
@@ -209,7 +209,7 @@ categories:
   - class: flink
 l1: 'Partially'
 l2: streaming, non-merging windows
-l3: State is supported in streaming mode for non-merging windows. 
SetState and MapState are not yet supported.
+l3: State is supported for non-merging windows. SetState and 
MapState are not yet supported.
   - class: spark
 l1: 'No'
 l2: not implemented
@@ -572,7 +572,7 @@ categories:
   - class: flink
 l1: 'Partially'
 l2: streaming, non-merging windows
-l3: The Flink runner support timers in non-merging windows when 
run in streaming mode.
+l3: The Flink Runner supports timers in non-merging windows.
   - class: spark
 l1: 'No'
 l2: not implemented

[2/3] beam-site git commit: Regenerate website

[davor]

Regenerate website


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/8de1bcc6
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/8de1bcc6
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/8de1bcc6

Branch: refs/heads/asf-site
Commit: 8de1bcc66bf5e9db6e05cd8f2fceb9905a7291bf
Parents: 6261b1a
Author: Davor Bonaci 
Authored: Sun Apr 2 09:51:17 2017 +0200
Committer: Davor Bonaci 
Committed: Sun Apr 2 09:51:17 2017 +0200

--
 content/documentation/runners/capability-matrix/index.html | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam-site/blob/8de1bcc6/content/documentation/runners/capability-matrix/index.html
--
diff --git a/content/documentation/runners/capability-matrix/index.html 
b/content/documentation/runners/capability-matrix/index.html
index 04174c7..b1cca24 100644
--- a/content/documentation/runners/capability-matrix/index.html
+++ b/content/documentation/runners/capability-matrix/index.html
@@ -1372,7 +1372,7 @@
 
 
 
-Partially: 
streaming, non-merging windowsState is supported in 
streaming mode for non-merging windows. SetState and MapState are not yet 
supported.
+Partially: 
streaming, non-merging windowsState is supported for 
non-merging windows. SetState and MapState are not yet supported.
 
 
 
@@ -1866,7 +1866,7 @@
 
 
 
-Partially: 
streaming, non-merging windowsThe Flink runner support 
timers in non-merging windows when run in streaming mode.
+Partially: 
streaming, non-merging windowsThe Flink Runner supports 
timers in non-merging windows.
 
 
 

[GitHub] beam-site pull request #198: Update Capability Matrix: Flink supports state/...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam-site/pull/198


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (BEAM-1854) Replace the magic number 42 in PAssert.

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci reassigned BEAM-1854:
--

Assignee: Pei He  (was: Davor Bonaci)

> Replace the magic number 42 in PAssert.
> ---
>
> Key: BEAM-1854
> URL: https://issues.apache.org/jira/browse/BEAM-1854
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Pei He
>Assignee: Pei He
>Priority: Minor
>
> PAssert used integer 42 as the global key. It causes some confusion when 
> debugging ValidatesRunner tests.
> final int combinedKey = 42;
> https://github.com/apache/beam/find/master#L935



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1856) HDFSFileSink class do not use the same configuration in master and slave

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15952559#comment-15952559
 ] 

Davor Bonaci commented on BEAM-1856:


Thanks [~397090770] for this contribution!

> HDFSFileSink class do not use the same configuration in master and slave
> 
>
> Key: BEAM-1856
> URL: https://issues.apache.org/jira/browse/BEAM-1856
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Affects Versions: 0.6.0
>Reporter: yangping wu
>Assignee: yangping wu
>
> I have a code snippet as follow:
> {code}
> Read.Bounded> from = 
> Read.from(HDFSFileSource.from(options.getInputFile(), TextInputFormat.class, 
> LongWritable.class, Text.class));
> PCollection> data = p.apply(from);
> data.apply(MapElements.via(new SimpleFunction, 
> String>() {
> @Override
> public String apply(KV input) {
> return input.getValue() + "\t" + input.getValue();
> }
> })).apply(Write.to(HDFSFileSink.toText(options.getOutputFile(;
> {code}
> and submit job like this:
> {code}
> spark-submit --class org.apache.beam.examples.WordCountHDFS --master 
> yarn-client   \
>  ./target/word-count-beam-bundled-0.1.jar 
>  \
>  --runner=SparkRunner 
>  \
>  --inputFile=hdfs://master/tmp/input/ 
>  \
>  --outputFile=/tmp/output/
> {code}
> Then {{HDFSFileSink.validate}} function will check whether the local 
> filesystem (not HDFS) exists {{/tmp/output/}} directory.
> But the final result will store in {{hdfs://master/tmp/output/}} directory in 
> HDFS filesystem.
> The reason is {{HDFSFileSink}} class do not use the same configuration in 
> master thread and slave thread.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-1856) HDFSFileSink class do not use the same configuration in master and slave

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci reassigned BEAM-1856:
--

Assignee: yangping wu  (was: Davor Bonaci)

> HDFSFileSink class do not use the same configuration in master and slave
> 
>
> Key: BEAM-1856
> URL: https://issues.apache.org/jira/browse/BEAM-1856
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Affects Versions: 0.6.0
>Reporter: yangping wu
>Assignee: yangping wu
>
> I have a code snippet as follow:
> {code}
> Read.Bounded> from = 
> Read.from(HDFSFileSource.from(options.getInputFile(), TextInputFormat.class, 
> LongWritable.class, Text.class));
> PCollection> data = p.apply(from);
> data.apply(MapElements.via(new SimpleFunction, 
> String>() {
> @Override
> public String apply(KV input) {
> return input.getValue() + "\t" + input.getValue();
> }
> })).apply(Write.to(HDFSFileSink.toText(options.getOutputFile(;
> {code}
> and submit job like this:
> {code}
> spark-submit --class org.apache.beam.examples.WordCountHDFS --master 
> yarn-client   \
>  ./target/word-count-beam-bundled-0.1.jar 
>  \
>  --runner=SparkRunner 
>  \
>  --inputFile=hdfs://master/tmp/input/ 
>  \
>  --outputFile=/tmp/output/
> {code}
> Then {{HDFSFileSink.validate}} function will check whether the local 
> filesystem (not HDFS) exists {{/tmp/output/}} directory.
> But the final result will store in {{hdfs://master/tmp/output/}} directory in 
> HDFS filesystem.
> The reason is {{HDFSFileSink}} class do not use the same configuration in 
> master thread and slave thread.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1856) HDFSFileSink class do not use the same configuration in master and slave

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-1856:
---
Component/s: (was: sdk-java-core)
 sdk-java-extensions

> HDFSFileSink class do not use the same configuration in master and slave
> 
>
> Key: BEAM-1856
> URL: https://issues.apache.org/jira/browse/BEAM-1856
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Affects Versions: 0.6.0
>Reporter: yangping wu
>Assignee: Davor Bonaci
>
> I have a code snippet as follow:
> {code}
> Read.Bounded> from = 
> Read.from(HDFSFileSource.from(options.getInputFile(), TextInputFormat.class, 
> LongWritable.class, Text.class));
> PCollection> data = p.apply(from);
> data.apply(MapElements.via(new SimpleFunction, 
> String>() {
> @Override
> public String apply(KV input) {
> return input.getValue() + "\t" + input.getValue();
> }
> })).apply(Write.to(HDFSFileSink.toText(options.getOutputFile(;
> {code}
> and submit job like this:
> {code}
> spark-submit --class org.apache.beam.examples.WordCountHDFS --master 
> yarn-client   \
>  ./target/word-count-beam-bundled-0.1.jar 
>  \
>  --runner=SparkRunner 
>  \
>  --inputFile=hdfs://master/tmp/input/ 
>  \
>  --outputFile=/tmp/output/
> {code}
> Then {{HDFSFileSink.validate}} function will check whether the local 
> filesystem (not HDFS) exists {{/tmp/output/}} directory.
> But the final result will store in {{hdfs://master/tmp/output/}} directory in 
> HDFS filesystem.
> The reason is {{HDFSFileSink}} class do not use the same configuration in 
> master thread and slave thread.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-1858) improve error message when Create.of() is called with an empty iterator

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci reassigned BEAM-1858:
--

Assignee: Wesley Tanaka  (was: Davor Bonaci)

> improve error message when Create.of() is called with an empty iterator
> ---
>
> Key: BEAM-1858
> URL: https://issues.apache.org/jira/browse/BEAM-1858
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Wesley Tanaka
>Assignee: Wesley Tanaka
>Priority: Trivial
>
> The current error copy states:
> "java.lang.IllegalArgumentException: Elements must be provided to construct 
> the default Create Coder. To Create an empty PCollection, either call 
> Create.empty(Coder), or call 'withCoder(Coder)' on the result PTransform"
> This is potentially confusing for two reasons:
> 1. "the default Create Coder" assumes a high level of knowledge of how Create 
> class works
> 2. since "Create" is a common word, it may not be immediately clear that 
> Create is referring to org.apache.beam.sdk.transforms.Create as opposed to 
> the possibility that there might be a compound noun in Beam model called 
> "Create Coder"



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1858) improve error message when Create.of() is called with an empty iterator

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15952557#comment-15952557
 ] 

Davor Bonaci commented on BEAM-1858:


Thanks for your contribution, [~wtanaka].

> improve error message when Create.of() is called with an empty iterator
> ---
>
> Key: BEAM-1858
> URL: https://issues.apache.org/jira/browse/BEAM-1858
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Wesley Tanaka
>Assignee: Wesley Tanaka
>Priority: Trivial
>
> The current error copy states:
> "java.lang.IllegalArgumentException: Elements must be provided to construct 
> the default Create Coder. To Create an empty PCollection, either call 
> Create.empty(Coder), or call 'withCoder(Coder)' on the result PTransform"
> This is potentially confusing for two reasons:
> 1. "the default Create Coder" assumes a high level of knowledge of how Create 
> class works
> 2. since "Create" is a common word, it may not be immediately clear that 
> Create is referring to org.apache.beam.sdk.transforms.Create as opposed to 
> the possibility that there might be a compound noun in Beam model called 
> "Create Coder"



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1680) Uber-JIRA: Extend Python-SDK's unit test coverage

[Tibor Kiss (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tibor Kiss updated BEAM-1680:
-
Description: 
Some components of the Python-SDK has low coverage rate.
As of now ([ea33e33|https://github.com/apache/beam/tree/ea33e33]) the test 
coverage looks as follows (excluding autocomplete_test.py):
{code}
Ran 1050 tests in 145.653s

OK (skipped=14)
py27gcp runtests: commands[5] | coverage report
Name
   Stmts   Miss  Cover
--
apache_beam/__init__.py 
  12  192%
apache_beam/coders/__init__.py  
   2  0   100%
apache_beam/coders/coder_impl.py
 414 2394%
apache_beam/coders/coders.py
 392 6883%
apache_beam/coders/observable.py
  10  0   100%
apache_beam/coders/slow_stream.py   
 105  694%
apache_beam/coders/typecoders.py
  69  691%
apache_beam/error.py
   7  0   100%
apache_beam/internal/__init__.py
   0  0   100%
apache_beam/internal/gcp/__init__.py
   0  0   100%
apache_beam/internal/gcp/auth.py
  75 3948%
apache_beam/internal/gcp/json_value.py  
  64  888%
apache_beam/internal/pickler.py 
  96 1090%
apache_beam/internal/util.py
  30  0   100%
apache_beam/io/__init__.py  
  16  288%
apache_beam/io/avroio.py
 139 1490%
apache_beam/io/concat_source.py 
 134 1093%
apache_beam/io/filebasedsource.py   
 112  397%
apache_beam/io/fileio.py
 188 5571%
apache_beam/io/filesystem.py
 184 2388%
apache_beam/io/filesystems_util.py  
   7  0   100%
apache_beam/io/gcp/__init__.py  
   0  0   100%
apache_beam/io/gcp/bigquery.py  
 418 2993%
apache_beam/io/gcp/datastore/__init__.py
   0  0   100%
apache_beam/io/gcp/datastore/v1/__init__.py 
   0  0   100%
apache_beam/io/gcp/datastore/v1/datastoreio.py  
 170 3381%
apache_beam/io/gcp/datastore/v1/fake_datastore.py   
  38  295%
apache_beam/io/gcp/datastore/v1/helper.py   
 126  894%
apache_beam/io/gcp/datastore/v1/query_splitter.py   
 102  793%
apache_beam/io/gcp/gcsfilesystem.py 
 109 1190%
apache_beam/io/gcp/gcsio.py 
 413 4689%
apache_beam/io/gcp/internal/__init__.py 
   0  0   100%
apache_beam/io/gcp/internal/clients/__init__.py 
   0  0   100%
apache_beam/io/gcp/internal/clients/bigquery/__init__.py
   9  278%
apache_beam/io/gcp/internal/clients/bigquery/bigquery_v2_client.py  
 116 6544%
apache_beam/io/gcp/internal/clients/bigquery/bigquery_v2_messages.py
 459  0   100%
apache_beam/io/gcp/internal/clients/storage/__init__.py 
   9  278%
apache_beam/io/gcp/internal/clients/storage/storage_v1_client.py
 165 7157%
apache_beam/io/gcp/internal/clients/storage/storage_v1_messages.py  
 507  0   100%
apache_beam/io/gcp/pubsub.py
  27  485%
apache_beam/io/iobase.py
 151 2782%

[jira] [Commented] (BEAM-1680) Uber-JIRA: Extend Python-SDK's unit test coverage

[Tibor Kiss (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15952554#comment-15952554
 ] 

Tibor Kiss commented on BEAM-1680:
--

Updated the coverage results to use GCP profile

> Uber-JIRA: Extend Python-SDK's unit test coverage
> -
>
> Key: BEAM-1680
> URL: https://issues.apache.org/jira/browse/BEAM-1680
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Tibor Kiss
>Assignee: Tibor Kiss
>Priority: Minor
>
> Some components of the Python-SDK has low coverage rate.
> As of now ([ea33e33|https://github.com/apache/beam/tree/ea33e33]) the test 
> coverage looks as follows (excluding autocomplete_test.py):
> {code}
> Ran 1050 tests in 145.653s
> OK (skipped=14)
> py27gcp runtests: commands[5] | coverage report
> Name  
>  Stmts   Miss  Cover
> --
> apache_beam/__init__.py   
> 12  192%
> apache_beam/coders/__init__.py
>  2  0   100%
> apache_beam/coders/coder_impl.py  
>414 2394%
> apache_beam/coders/coders.py  
>392 6883%
> apache_beam/coders/coders_test.py 
> 54  394%
> apache_beam/coders/coders_test_common.py  
>179  398%
> apache_beam/coders/observable.py  
> 10  0   100%
> apache_beam/coders/observable_test.py 
> 28  293%
> apache_beam/coders/proto2_coder_test_messages_pb2.py  
> 51  0   100%
> apache_beam/coders/slow_coders_test.py
> 15  380%
> apache_beam/coders/slow_stream.py 
>105  694%
> apache_beam/coders/standard_coders_test.py
> 72 1579%
> apache_beam/coders/stream_test.py 
>127 1489%
> apache_beam/coders/typecoders.py  
> 69  691%
> apache_beam/error.py  
>  7  0   100%
> apache_beam/internal/__init__.py  
>  0  0   100%
> apache_beam/internal/gcp/__init__.py  
>  0  0   100%
> apache_beam/internal/gcp/auth.py  
> 75 3948%
> apache_beam/internal/gcp/auth_test.py 
> 17  288%
> apache_beam/internal/gcp/json_value.py
> 64  888%
> apache_beam/internal/gcp/json_value_test.py   
> 49  394%
> apache_beam/internal/module_test.py   
> 23  0   100%
> apache_beam/internal/pickler.py   
> 96 1090%
> apache_beam/internal/pickler_test.py  
> 30  197%
> apache_beam/internal/util.py  
> 30  0   100%
> apache_beam/internal/util_test.py 
> 32  197%
> apache_beam/io/__init__.py
> 16  288%
> apache_beam/io/avroio.py  
>139 1490%
> apache_beam/io/avroio_test.py 
>200 1792%
> apache_beam/io/concat_source.py   
>134 1093%
> apache_beam/io/concat_source_test.py  
>132  894%
> apache_beam/io/filebasedsource.py 
>112  397%
> apache_beam/io/filebasedsource_test.py
>478  499%
> apache_beam/io/fileio.py  
>188 5571%
> apache_beam/io/fileio_test.py 
>154  299%
> 

[jira] [Updated] (BEAM-1680) Uber-JIRA: Extend Python-SDK's unit test coverage

[Tibor Kiss (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tibor Kiss updated BEAM-1680:
-
Description: 
Some components of the Python-SDK has low coverage rate.
As of now ([ea33e33|https://github.com/apache/beam/tree/ea33e33]) the test 
coverage looks as follows (excluding autocomplete_test.py):
{code}
Ran 1050 tests in 145.653s

OK (skipped=14)
py27gcp runtests: commands[5] | coverage report
Name
   Stmts   Miss  Cover
--
apache_beam/__init__.py 
  12  192%
apache_beam/coders/__init__.py  
   2  0   100%
apache_beam/coders/coder_impl.py
 414 2394%
apache_beam/coders/coders.py
 392 6883%
apache_beam/coders/coders_test.py   
  54  394%
apache_beam/coders/coders_test_common.py
 179  398%
apache_beam/coders/observable.py
  10  0   100%
apache_beam/coders/observable_test.py   
  28  293%
apache_beam/coders/proto2_coder_test_messages_pb2.py
  51  0   100%
apache_beam/coders/slow_coders_test.py  
  15  380%
apache_beam/coders/slow_stream.py   
 105  694%
apache_beam/coders/standard_coders_test.py  
  72 1579%
apache_beam/coders/stream_test.py   
 127 1489%
apache_beam/coders/typecoders.py
  69  691%
apache_beam/error.py
   7  0   100%
apache_beam/internal/__init__.py
   0  0   100%
apache_beam/internal/gcp/__init__.py
   0  0   100%
apache_beam/internal/gcp/auth.py
  75 3948%
apache_beam/internal/gcp/auth_test.py   
  17  288%
apache_beam/internal/gcp/json_value.py  
  64  888%
apache_beam/internal/gcp/json_value_test.py 
  49  394%
apache_beam/internal/module_test.py 
  23  0   100%
apache_beam/internal/pickler.py 
  96 1090%
apache_beam/internal/pickler_test.py
  30  197%
apache_beam/internal/util.py
  30  0   100%
apache_beam/internal/util_test.py   
  32  197%
apache_beam/io/__init__.py  
  16  288%
apache_beam/io/avroio.py
 139 1490%
apache_beam/io/avroio_test.py   
 200 1792%
apache_beam/io/concat_source.py 
 134 1093%
apache_beam/io/concat_source_test.py
 132  894%
apache_beam/io/filebasedsource.py   
 112  397%
apache_beam/io/filebasedsource_test.py  
 478  499%
apache_beam/io/fileio.py
 188 5571%
apache_beam/io/fileio_test.py   
 154  299%
apache_beam/io/filesystem.py
 184 2388%
apache_beam/io/filesystems_util.py  
   7  0   100%
apache_beam/io/gcp/__init__.py  
   0  0   100%
apache_beam/io/gcp/bigquery.py  
 418 2993%
apache_beam/io/gcp/bigquery_test.py 
 434 1197%
apache_beam/io/gcp/datastore/__init__.py
   0  0   100%

Build failed in Jenkins: beam_PerformanceTests_Dataflow #259

[Apache Jenkins Server]

See 


--
[...truncated 221.39 KB...]
 x [deleted] (none) -> origin/pr/983/merge
 x [deleted] (none) -> origin/pr/984/head
 x [deleted] (none) -> origin/pr/984/merge
 x [deleted] (none) -> origin/pr/985/head
 x [deleted] (none) -> origin/pr/985/merge
 x [deleted] (none) -> origin/pr/986/head
 x [deleted] (none) -> origin/pr/986/merge
 x [deleted] (none) -> origin/pr/987/head
 x [deleted] (none) -> origin/pr/988/head
 x [deleted] (none) -> origin/pr/988/merge
 x [deleted] (none) -> origin/pr/989/head
 x [deleted] (none) -> origin/pr/989/merge
 x [deleted] (none) -> origin/pr/99/head
 x [deleted] (none) -> origin/pr/99/merge
 x [deleted] (none) -> origin/pr/990/head
 x [deleted] (none) -> origin/pr/990/merge
 x [deleted] (none) -> origin/pr/991/head
 x [deleted] (none) -> origin/pr/991/merge
 x [deleted] (none) -> origin/pr/992/head
 x [deleted] (none) -> origin/pr/992/merge
 x [deleted] (none) -> origin/pr/993/head
 x [deleted] (none) -> origin/pr/993/merge
 x [deleted] (none) -> origin/pr/994/head
 x [deleted] (none) -> origin/pr/994/merge
 x [deleted] (none) -> origin/pr/995/head
 x [deleted] (none) -> origin/pr/995/merge
 x [deleted] (none) -> origin/pr/996/head
 x [deleted] (none) -> origin/pr/996/merge
 x [deleted] (none) -> origin/pr/997/head
 x [deleted] (none) -> origin/pr/997/merge
 x [deleted] (none) -> origin/pr/998/head
 x [deleted] (none) -> origin/pr/999/head
 x [deleted] (none) -> origin/pr/999/merge
remote: Counting objects: 1047, done.
remote: Compressing objects:   2% (1/36)   remote: Compressing objects: 
  5% (2/36)   remote: Compressing objects:   8% (3/36)   
remote: Compressing objects:  11% (4/36)   remote: Compressing objects: 
 13% (5/36)   remote: Compressing objects:  16% (6/36)   
remote: Compressing objects:  19% (7/36)   remote: Compressing objects: 
 22% (8/36)   remote: Compressing objects:  25% (9/36)   
remote: Compressing objects:  27% (10/36)   remote: Compressing 
objects:  30% (11/36)   remote: Compressing objects:  33% (12/36)   
remote: Compressing objects:  36% (13/36)   remote: Compressing 
objects:  38% (14/36)   remote: Compressing objects:  41% (15/36)   
remote: Compressing objects:  44% (16/36)   remote: Compressing 
objects:  47% (17/36)   remote: Compressing objects:  50% (18/36)   
remote: Compressing objects:  52% (19/36)   remote: Compressing 
objects:  55% (20/36)   remote: Compressing objects:  58% (21/36)   
remote: Compressing objects:  61% (22/36)   remote: Compressing 
objects:  63% (23/36)   remote: Compressing objects:  66% (24/36)   
remote: Compressing objects:  69% (25/36)   remote: Compressing 
objects:  72% (26/36)   remote: Compressing objects:  75% (27/36)   
remote: Compressing objects:  77% (28/36)   remote: Compressing 
objects:  80% (29/36)   remote: Compressing objects:  83% (30/36)   
remote: Compressing objects:  86% (31/36)   remote: Compressing 
objects:  88% (32/36)   remote: Compressing objects:  91% (33/36)   
remote: Compressing objects:  94% (34/36)   remote: Compressing 
objects:  97% (35/36)   remote: Compressing objects: 100% (36/36)   
remote: Compressing objects: 100% (36/36), done.
Receiving objects:   0% (1/1047)   Receiving objects:   1% (11/1047)   
Receiving objects:   2% (21/1047)   Receiving objects:   3% (32/1047)   
Receiving objects:   4% (42/1047)   Receiving objects:   5% (53/1047)   
Receiving objects:   6% (63/1047)   Receiving objects:   7% (74/1047)   
Receiving objects:   8% (84/1047)   Receiving objects:   9% (95/1047)   
Receiving objects:  10% (105/1047)   Receiving objects:  11% (116/1047)   
Receiving objects:  12% (126/1047)   Receiving objects:  13% (137/1047)   
Receiving objects:  14% (147/1047)   Receiving objects:  15% (158/1047)   
Receiving objects:  16% (168/1047)   Receiving objects:  17% (178/1047)   
Receiving objects:  18% (189/1047)   Receiving objects:  19% (199/1047)   
Receiving objects:  20% (210/1047)   Receiving objects:  21% (220/1047)   
Receiving objects:  22% (231/1047)   Receiving objects:  23% (241/1047)   
Receiving objects:  24% (252/1047)   Receiving objects:  25% (262/1047)   
Receiving objects:  26% (273/1047)   Receiving objects:  27% (283/1047)   
Receiving