Jenkins build is back to normal : beam_PostCommit_Java_GradleBuild #1103

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PostCommit_Py_VR_Dataflow #644

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PerformanceTests_JDBC #891

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-2660) Set PubsubIO batch size using builder

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2660?focusedWorklogId=127671&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127671
 ]

ASF GitHub Bot logged work on BEAM-2660:


Author: ASF GitHub Bot
Created on: 26/Jul/18 06:22
Start Date: 26/Jul/18 06:22
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #3619: [BEAM-2660] Set 
PubsubIO batch size using builder
URL: https://github.com/apache/beam/pull/3619#issuecomment-407989768
 
 
   @aromanenko-dev it somewhat is. It turns out that Dataflow has it's own 
implementation of the PubSub source, and so this PR will not change any 
behavior for Dataflow - only for non-Dataflow runners. This PR is still a good 
one I believe, however I want to make sure that you know it will not affect the 
Dataflow runner.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127671)
Time Spent: 2h  (was: 1h 50m)

> Set PubsubIO batch size using builder
> -
>
> Key: BEAM-2660
> URL: https://issues.apache.org/jira/browse/BEAM-2660
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-gcp
>Reporter: Carl McGraw
>Assignee: Chamikara Jayalath
>Priority: Major
>  Labels: gcp, java, pubsub, sdk
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> PubsubIO doesn't allow users to set the publish batch size. Instead the value 
> is hard coded in both the BoundedPubsubWriter and the UnboundedPubsubSink. 
> google's pub/sub is bound to a maximum of 10mb per request size. My company 
> has run into problems with events that are individually smaller than 1mb, but 
> when batched in the 100 or 2000 default batch sizes causes pubsub to fail to 
> send the event.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PreCommit_Java_Cron #148

[Apache Jenkins Server]

See 


Changes:

[altay] Remove reference to dataflow-distribution.properties

[lcwik] [BEAM-4629] Output the names of the failing licenses as part of the

[aaltay] [BEAM-4859] Enable Python VR tests in streaming in postcommit task

--
[...truncated 17.58 MB...]
INFO: 2018-07-26T06:18:06.230Z: Fusing adjacent ParDo, Read, Write, and 
Flatten operations
Jul 26, 2018 6:18:09 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-26T06:18:06.279Z: Unzipping flatten s13 for input 
s12.org.apache.beam.sdk.values.PCollection.:364#1d275f544daf228c
Jul 26, 2018 6:18:09 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-26T06:18:06.320Z: Fusing unzipped copy of 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Add void 
key/AddKeys/Map, through flatten 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/Flatten.PCollections,
 into producer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/DropShardNum
Jul 26, 2018 6:18:09 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-26T06:18:06.394Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/ExpandIterable
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/GroupByWindow
Jul 26, 2018 6:18:09 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-26T06:18:06.440Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Pair
 with random key
Jul 26, 2018 6:18:09 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-26T06:18:06.477Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify
Jul 26, 2018 6:18:09 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-26T06:18:06.529Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read
Jul 26, 2018 6:18:09 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-26T06:18:06.577Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign
Jul 26, 2018 6:18:09 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-26T06:18:06.625Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Drop 
key/Values/Map into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/ExpandIterable
Jul 26, 2018 6:18:09 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-26T06:18:06.677Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Gather 
bundles into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Drop 
key/Values/Map
Jul 26, 2018 6:18:09 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-26T06:18:06.725Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Pair
 with random key into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Gather 
bundles
Jul 26, 2018 6:18:09 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-26T06:18:06.759Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read
Jul 26, 2018 6:18:09 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-26T06:18:06.792Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/Reify
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/W

[jira] [Work logged] (BEAM-4076) Schema followups

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4076?focusedWorklogId=127667&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127667
 ]

ASF GitHub Bot logged work on BEAM-4076:


Author: ASF GitHub Bot
Created on: 26/Jul/18 06:15
Start Date: 26/Jul/18 06:15
Worklog Time Spent: 10m 
  Work Description: reuvenlax opened a new pull request #6072: [BEAM-4076] 
Fix schemas on Dataflow and FnApi
URL: https://github.com/apache/beam/pull/6072
 
 
   Two issues:
 1. For Dataflow, the problem is that the Dataflow replacement for 
ParDo.Single incorrectly broke the correspondence between mainOutputTupleTag 
and the actual outputs. It generated a new PCollection, which by default 
generates a new TupleTag, and no longer matched the output TupleTag. Fix this 
by making sure that the new PCollection has the same tag.
   
   2. For FnApiRunner, the problem is that input/output coders might be 
WindowedValueCoders, preventing us from recognizing a SchemaCoder. Detect this 
case and extract the value coder.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127667)
Time Spent: 16h  (was: 15h 50m)

> Schema followups
> 
>
> Key: BEAM-4076
> URL: https://issues.apache.org/jira/browse/BEAM-4076
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model, dsl-sql, sdk-java-core
>Reporter: Kenneth Knowles
>Priority: Major
>  Time Spent: 16h
>  Remaining Estimate: 0h
>
> This umbrella bug contains subtasks with followups for Beam schemas, which 
> were moved from SQL to the core Java SDK and made to be type-name-based 
> rather than coder based.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5027) Schemas do not work on Dataflow runner of FnApi Runner

[Reuven Lax (JIRA)]

Reuven Lax created BEAM-5027:


 Summary: Schemas do not work on Dataflow runner of FnApi Runner
 Key: BEAM-5027
 URL: https://issues.apache.org/jira/browse/BEAM-5027
 Project: Beam
  Issue Type: Sub-task
  Components: sdk-java-core
Reporter: Reuven Lax
Assignee: Reuven Lax






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4658) Update pipeline representation in runner support libraries to handle timers

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4658?focusedWorklogId=127656&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127656
 ]

ASF GitHub Bot logged work on BEAM-4658:


Author: ASF GitHub Bot
Created on: 26/Jul/18 06:09
Start Date: 26/Jul/18 06:09
Worklog Time Spent: 10m 
  Work Description: youngoli commented on a change in pull request #6050: 
[BEAM-4658] Update pipeline representation in runner support libraries to 
handle timers.
URL: https://github.com/apache/beam/pull/6050#discussion_r205290440
 
 

 ##
 File path: 
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/Networks.java
 ##
 @@ -193,29 +201,55 @@ public final NodeT apply(NodeT input) {
 "Only networks without self loops are supported, given %s",
 network);
 
-// Linked hashset will prevent duplicates from appearing and will maintain 
insertion order.
-LinkedHashSet nodes = new LinkedHashSet<>(network.nodes().size());
-Queue processingOrder = new ArrayDeque<>();
-// Add all the roots
-for (NodeT node : network.nodes()) {
-  if (network.inDegree(node) == 0) {
-processingOrder.add(node);
-  }
-}
+// Uses the following algorithm:
 
 Review comment:
   Using this algorithm for topological order doesn't seem appropriate. From 
what I could understand from the paper, this algorithm generates a vertex 
sequence that makes it easy to remove feedback arcs (i.e. eliminate any 
cycles), but doesn't actually maintain topological order.
   
   As a concrete example, if you had a graph that looked like so:
   ```
   Source1 --\
  ---> A ---> B ---> Sink
   Source2 --/
   ```
   
   Topologically I would expect the result to be: `Source1, Source2, A, B, 
Sink`. But instead this algorithm would produce `Source1, Source2, B, A, Sink` 
because B is considered to have the higher delta than A (delta(B) = 0, delta(A) 
= -1).


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127656)
Time Spent: 50m  (was: 40m)

> Update pipeline representation in runner support libraries to handle timers
> ---
>
> Key: BEAM-4658
> URL: https://issues.apache.org/jira/browse/BEAM-4658
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-core
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Expose relevant timer information on *ExecutableStage* to runners
> Update fusion logic to handle timers.
> Provide execution time interface to fire timers into *RemoteBundle*s and also 
> to receive new timers that are being set.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4658) Update pipeline representation in runner support libraries to handle timers

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4658?focusedWorklogId=127657&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127657
 ]

ASF GitHub Bot logged work on BEAM-4658:


Author: ASF GitHub Bot
Created on: 26/Jul/18 06:09
Start Date: 26/Jul/18 06:09
Worklog Time Spent: 10m 
  Work Description: youngoli commented on a change in pull request #6050: 
[BEAM-4658] Update pipeline representation in runner support libraries to 
handle timers.
URL: https://github.com/apache/beam/pull/6050#discussion_r205264448
 
 

 ##
 File path: 
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/ExecutableStage.java
 ##
 @@ -104,7 +118,7 @@
*   The {@link PTransform#getSubtransformsList()} is empty. This 
ensures that executable
*   stages are treated as primitive transforms.
*   The only {@link PCollection} in the {@link 
PTransform#getInputsMap()} is the result of
-   *   {@link #getInputPCollection()}.
+   *   {@link #getInputPCollection()} and {@link #getSideInputs()}.
 
 Review comment:
   The way this is worded makes it sound like PTransform.getInputsMap has only 
one PCollection, which sounds odd if it's the result of both those functions. 
Is that correct, or does it actually contain multiple PCollections now?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127657)

> Update pipeline representation in runner support libraries to handle timers
> ---
>
> Key: BEAM-4658
> URL: https://issues.apache.org/jira/browse/BEAM-4658
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-core
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Expose relevant timer information on *ExecutableStage* to runners
> Update fusion logic to handle timers.
> Provide execution time interface to fire timers into *RemoteBundle*s and also 
> to receive new timers that are being set.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4658) Update pipeline representation in runner support libraries to handle timers

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4658?focusedWorklogId=127658&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127658
 ]

ASF GitHub Bot logged work on BEAM-4658:


Author: ASF GitHub Bot
Created on: 26/Jul/18 06:09
Start Date: 26/Jul/18 06:09
Worklog Time Spent: 10m 
  Work Description: youngoli commented on a change in pull request #6050: 
[BEAM-4658] Update pipeline representation in runner support libraries to 
handle timers.
URL: https://github.com/apache/beam/pull/6050#discussion_r205256708
 
 

 ##
 File path: 
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ParDoTranslation.java
 ##
 @@ -337,11 +337,22 @@ public static TupleTagList 
getAdditionalOutputTags(AppliedPTransform ap
 ptransform.getSpec().getUrn());
 ParDoPayload payload = 
ParDoPayload.parseFrom(ptransform.getSpec().getPayload());
 return components.getPcollectionsOrThrow(
-ptransform.getInputsOrThrow(getMainInputId(ptransform, payload)));
+ptransform.getInputsOrThrow(getMainInputName(ptransform, payload)));
 
 Review comment:
   Since you added a version of getMainInputName that retrieves the payload, 
what about using that here?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127658)
Time Spent: 1h  (was: 50m)

> Update pipeline representation in runner support libraries to handle timers
> ---
>
> Key: BEAM-4658
> URL: https://issues.apache.org/jira/browse/BEAM-4658
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-core
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Expose relevant timer information on *ExecutableStage* to runners
> Update fusion logic to handle timers.
> Provide execution time interface to fire timers into *RemoteBundle*s and also 
> to receive new timers that are being set.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4658) Update pipeline representation in runner support libraries to handle timers

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4658?focusedWorklogId=127659&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127659
 ]

ASF GitHub Bot logged work on BEAM-4658:


Author: ASF GitHub Bot
Created on: 26/Jul/18 06:09
Start Date: 26/Jul/18 06:09
Worklog Time Spent: 10m 
  Work Description: youngoli commented on a change in pull request #6050: 
[BEAM-4658] Update pipeline representation in runner support libraries to 
handle timers.
URL: https://github.com/apache/beam/pull/6050#discussion_r205274669
 
 

 ##
 File path: 
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/GreedyPCollectionFusers.java
 ##
 @@ -150,41 +154,51 @@ private static boolean canFuseParDo(
   // is never possible.
   return false;
 }
-if (!pipeline.getSideInputs(parDo).isEmpty()) {
-  // At execution time, a Runner is required to only provide inputs to a 
PTransform that, at
-  // the time the PTransform processes them, the associated window is 
ready in all side inputs
-  // that the PTransform consumes. For an arbitrary stage, it is 
significantly complex for the
-  // runner to determine this for each input. As a result, we break fusion 
to simplify this
-  // inspection. In general, a ParDo which consumes side inputs cannot be 
fused into an
-  // executable stage alongside any transforms which are upstream of any 
of its side inputs.
-  return false;
-} else {
-  try {
-ParDoPayload payload = 
ParDoPayload.parseFrom(parDo.getTransform().getSpec().getPayload());
-if (payload.getStateSpecsCount() > 0 || payload.getTimerSpecsCount() > 
0) {
-  // Inputs to a ParDo that uses State or Timers must be 
key-partitioned, and elements for
-  // a key must execute serially. To avoid checking if the rest of the 
stage is
-  // key-partitioned and preserves keys, these ParDos do not fuse into 
an existing stage.
-  return false;
-}
-  } catch (InvalidProtocolBufferException e) {
-throw new IllegalArgumentException(e);
+try {
+  ParDoPayload payload = 
ParDoPayload.parseFrom(parDo.getTransform().getSpec().getPayload());
+  if (Maps.filterKeys(
+  parDo.getTransform().getInputsMap(), s -> 
payload.getTimerSpecsMap().containsKey(s))
+  .values()
+  .contains(candidate.getId())) {
+// Allow fusion across timer PCollections because they are a self loop.
+return true;
+  } else if (payload.getStateSpecsCount() > 0 || 
payload.getTimerSpecsCount() > 0) {
+// Inputs to a ParDo that uses State or Timers must be 
key-partitioned, and elements for
+// a key must execute serially. To avoid checking if the rest of the 
stage is
+// key-partitioned and preserves keys, these ParDos do not fuse into 
an existing stage.
+return false;
+  } else if (!pipeline.getSideInputs(parDo).isEmpty()) {
+// At execution time, a Runner is required to only provide inputs to a 
PTransform that, at
+// the time the PTransform processes them, the associated window is 
ready in all side inputs
+// that the PTransform consumes. For an arbitrary stage, it is 
significantly complex for the
+// runner to determine this for each input. As a result, we break 
fusion to simplify this
+// inspection. In general, a ParDo which consumes side inputs cannot 
be fused into an
+// executable stage alongside any transforms which are upstream of any 
of its side inputs.
+return false;
   }
+} catch (InvalidProtocolBufferException e) {
+  throw new IllegalArgumentException(e);
 }
 return true;
   }
 
   private static boolean parDoCompatibility(
   PTransformNode parDo, PTransformNode other, QueryablePipeline pipeline) {
-// This is a convenience rather than a strict requirement. In general, a 
ParDo that consumes
-// side inputs can be fused with other transforms in the same environment 
which are not
-// upstream of any of the side inputs.
-return pipeline.getSideInputs(parDo).isEmpty()
-// Since we lack the ability to mark upstream transforms as key 
preserving, we
-// purposefully break fusion here to provide runners the opportunity 
to insert a
-// grouping operation
-&& pipeline.getUserStates(parDo).isEmpty()
-&& compatibleEnvironments(parDo, other, pipeline);
+// Implicitly true if we are attempting to fuse against oneself. This is 
for timer PCollection which create a loop.
+return parDo.equals(other)
+// This is a convenience rather than a strict requirement. In general, 
a ParDo that consumes
+// side inputs can be fused with other transforms in the same 
environment which are not
+// upstream of any of the side input

[jira] [Commented] (BEAM-5026) Portable flink wordcount fails sometimes due to non-existent source path in FileBasedSink._check_state_for_finalize_write

[Thomas Weise (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16556936#comment-16556936
 ] 

Thomas Weise commented on BEAM-5026:


I have also seen this intermittently.

 

> Portable flink wordcount fails sometimes due to non-existent source path in 
> FileBasedSink._check_state_for_finalize_write
> -
>
> Key: BEAM-5026
> URL: https://issues.apache.org/jira/browse/BEAM-5026
> Project: Beam
>  Issue Type: Bug
>  Components: examples-python, runner-flink, sdk-py-core
>Affects Versions: 2.6.0
>Reporter: Ryan Williams
>Assignee: Ryan Williams
>Priority: Minor
>
> Running portable flink wordcount locally:
> In one terminal:
> {code:java}
> ./gradlew :beam-runners-flink_2.11-job-server:runShadow{code}
> In another:
> {code:java}
> python -m apache_beam.examples.wordcount --harness_docker_image  
> --input /etc/profile --output /tmp/py-wordcount-direct 
> --experiments=beam_fn_api --runner=PortableRunner 
> --job_endpoint=localhost:8099 --sdk_location=container{code}
> Typically, the first time I run this for a given job-server instance, I see a 
> failure like this ([full 
> output|https://gist.github.com/ryan-williams/a96bf259898b6260cd4f00b8a232057c#file-gistfile1-txt-L3460]):
> {code:java}
> File "apache_beam/runners/common.py", line 661, in 
> apache_beam.runners.common._OutputProcessor.process_outputs
> def process_outputs(self, windowed_input_element, results):
> File "apache_beam/runners/common.py", line 676, in 
> apache_beam.runners.common._OutputProcessor.process_outputs
> for result in results:
> File "/usr/local/lib/python2.7/site-packages/apache_beam/io/iobase.py", line 
> 1074, in 
> return (window.TimestampedValue(v, window.MAX_TIMESTAMP) for v in outputs)
> File 
> "/usr/local/lib/python2.7/site-packages/apache_beam/io/filebasedsink.py", 
> line 271, in finalize_write
> self._check_state_for_finalize_write(writer_results, num_shards))
> File 
> "/usr/local/lib/python2.7/site-packages/apache_beam/io/filebasedsink.py", 
> line 249, in _check_state_for_finalize_write
> src, dst))
> BeamIOError: src and dst files do not exist. src: 
> /tmp/beam-temp-py-wordcount-direct-6a0d8862908c11e88de80251/5cfa9f22-9246-41fb-adef-ca04d5a5fe50.py-wordcount-direct,
>  dst: /tmp/py-wordcount-direct-0-of-1 with exceptions None [while 
> running 'write/Write/WriteImpl/FinalizeWrite'] with exceptions None
> {code}
> This is after a fix to [a slightly earlier failure in {{FileBasedSink}} 
> documented on 
> BEAM-4742|https://issues.apache.org/jira/browse/BEAM-4742?focusedCommentId=16545622&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16545622]
>  which I've been working on in 
> [#5903|https://github.com/apache/beam/pull/5903].
> It typically occurs only on the first run of wordcount against a given 
> job-server instance.
> I'm curious whether others see this, whether it's some race condition in the 
> FileBasedSink, LocalFileSystem, my macbook's disk, or somewhere else, or 
> whether some temporary directory is getting created on the first run (for 
> each job-server) that explains why subsequent wordcount runs succeed, etc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4749) fastavro breaks macos tests

[Ryan Williams (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16556918#comment-16556918
 ] 

Ryan Williams commented on BEAM-4749:
-

Great, thanks! Shall I take out some of the import-guards you added? Or do we 
still want them for any reason?

> fastavro breaks macos tests
> ---
>
> Key: BEAM-4749
> URL: https://issues.apache.org/jira/browse/BEAM-4749
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Ahmet Altay
>Assignee: Ryan Williams
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Recent addition of the fastavro dependency is breaking python linter in 
> macos. At least for some cases, because it requires a compiler.
> Could we optionally depend on fastavro, and fallback to regular avro package?
>  
> Log:
> *> Task :beam-sdks-python:lintPy27*
> ERROR: invocation failed (exit code 1), logfile: 
> /Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/log/py27-lint-2.log
> ERROR: actionid: py27-lint
> msg: installpkg
> cmdargs: 
> ['/Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/bin/python',
>  '/Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/bin/pip', 
> 'install', 
> '/Users/relax/beam-gradle/beam/sdks/python/target/.tox/dist/apache-beam-2.6.0.dev0.zip[test]']
>  
> Processing ./target/.tox/dist/apache-beam-2.6.0.dev0.zip
> Collecting avro<2.0.0,>=1.8.1 (from apache-beam==2.6.0.dev0)
> Collecting crcmod<2.0,>=1.7 (from apache-beam==2.6.0.dev0)
> Collecting dill==0.2.6 (from apache-beam==2.6.0.dev0)
> Collecting fastavro==0.19.7 (from apache-beam==2.6.0.dev0)
>   Using cached 
> [https://files.pythonhosted.org/packages/a7/0a/b08ba5cef63c675e8442c2bf1cbcef90c8b9f824be2202d492f0cedb0913/fastavro-0.19.7.tar.gz]
> Collecting grpcio<2,>=1.8 (from apache-beam==2.6.0.dev0)
>   Using cached 
> [https://files.pythonhosted.org/packages/66/89/4a90caabd51c17686cbb48a9bbe8c592c4be929c0d2542d2ffde76b0d671/grpcio-1.13.0-cp27-cp27m-macosx_10_12_x86_64.whl]
> Collecting hdfs<3.0.0,>=2.1.0 (from apache-beam==2.6.0.dev0)
> Collecting httplib2<=0.11.3,>=0.8 (from apache-beam==2.6.0.dev0)
> Collecting mock<3.0.0,>=1.0.1 (from apache-beam==2.6.0.dev0)
>   Using cached 
> [https://files.pythonhosted.org/packages/e6/35/f187bdf23be87092bd0f1200d43d23076cee4d0dec109f195173fd3ebc79/mock-2.0.0-py2.py3-none-any.whl]
> Collecting oauth2client<5,>=2.0.1 (from apache-beam==2.6.0.dev0)
>   Using cached 
> [https://files.pythonhosted.org/packages/82/d8/3eab58811282ac7271a081ba5c0d4b875ce786ca68ce43e2a62ade32e9a8/oauth2client-4.1.2-py2.py3-none-any.whl]
> Collecting protobuf<4,>=3.5.0.post1 (from apache-beam==2.6.0.dev0)
>   Using cached 
> [https://files.pythonhosted.org/packages/4f/56/a21f2d077ceae7fd521c0ed31fb8bb1c7f13ffbb09bf7dd27de6cf6bad08/protobuf-3.6.0-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl]
> Collecting pytz<=2018.4,>=2018.3 (from apache-beam==2.6.0.dev0)
>   Using cached 
> [https://files.pythonhosted.org/packages/dc/83/15f7833b70d3e067ca91467ca245bae0f6fe56ddc7451aa0dc5606b120f2/pytz-2018.4-py2.py3-none-any.whl]
> Collecting pyyaml<4.0.0,>=3.12 (from apache-beam==2.6.0.dev0)
>   Using cached 
> [https://files.pythonhosted.org/packages/4a/85/db5a2df477072b2902b0eb892feb37d88ac635d36245a72a6a69b23b383a/PyYAML-3.12.tar.gz]
> Collecting pyvcf<0.7.0,>=0.6.8 (from apache-beam==2.6.0.dev0)
> Requirement already satisfied: six<1.12,>=1.9 in 
> ./target/.tox/py27-lint/lib/python2.7/site-packages (from 
> apache-beam==2.6.0.dev0) (1.11.0)
> Collecting typing<3.7.0,>=3.6.0 (from apache-beam==2.6.0.dev0)
>   Using cached 
> [https://files.pythonhosted.org/packages/0d/4d/4e5985d075d241d686a1663fa1f88b61d544658d08c1375c7c6aac32afc3/typing-3.6.4-py2-none-any.whl]
> Collecting futures<4.0.0,>=3.1.1 (from apache-beam==2.6.0.dev0)
>   Using cached 
> [https://files.pythonhosted.org/packages/2d/99/b2c4e9d5a30f6471e410a146232b4118e697fa3ffc06d6a65efde84debd0/futures-3.2.0-py2-none-any.whl]
> Requirement already satisfied: future<1.0.0,>=0.16.0 in 
> ./target/.tox/py27-lint/lib/python2.7/site-packages (from 
> apache-beam==2.6.0.dev0) (0.16.0)
> Collecting nose>=1.3.7 (from apache-beam==2.6.0.dev0)
>   Using cached 
> [https://files.pythonhosted.org/packages/99/4f/13fb671119e65c4dce97c60e67d3fd9e6f7f809f2b307e2611f4701205cb/nose-1.3.7-py2-none-any.whl]
> Collecting pyhamcrest<2.0,>=1.9 (from apache-beam==2.6.0.dev0)
>   Using cached 
> [https://files.pythonhosted.org/packages/9a/d5/d37fd731b7d0e91afcc84577edeccf4638b4f9b82f5ffe2f8b62e2ddc609/PyHamcrest-1.9.0-py2.py3-none-any.whl]
> Requirement already satisfied: setuptools>=18.0 in 
> ./target/.tox/py27-lint/lib/python2.7/site-packages (from 
> fastavro==0.19.7->apache-beam==2.6.0.dev0) (39.2.0

[jira] [Created] (BEAM-5026) Portable flink wordcount fails sometimes due to non-existent source path in FileBasedSink._check_state_for_finalize_write

[Ryan Williams (JIRA)]

Ryan Williams created BEAM-5026:
---

 Summary: Portable flink wordcount fails sometimes due to 
non-existent source path in FileBasedSink._check_state_for_finalize_write
 Key: BEAM-5026
 URL: https://issues.apache.org/jira/browse/BEAM-5026
 Project: Beam
  Issue Type: Bug
  Components: examples-python, runner-flink, sdk-py-core
Affects Versions: 2.6.0
Reporter: Ryan Williams
Assignee: Ryan Williams


Running portable flink wordcount locally:

In one terminal:
{code:java}
./gradlew :beam-runners-flink_2.11-job-server:runShadow{code}
In another:
{code:java}
python -m apache_beam.examples.wordcount --harness_docker_image  --input 
/etc/profile --output /tmp/py-wordcount-direct --experiments=beam_fn_api 
--runner=PortableRunner --job_endpoint=localhost:8099 
--sdk_location=container{code}
Typically, the first time I run this for a given job-server instance, I see a 
failure like this ([full 
output|https://gist.github.com/ryan-williams/a96bf259898b6260cd4f00b8a232057c#file-gistfile1-txt-L3460]):
{code:java}
File "apache_beam/runners/common.py", line 661, in 
apache_beam.runners.common._OutputProcessor.process_outputs
def process_outputs(self, windowed_input_element, results):
File "apache_beam/runners/common.py", line 676, in 
apache_beam.runners.common._OutputProcessor.process_outputs
for result in results:
File "/usr/local/lib/python2.7/site-packages/apache_beam/io/iobase.py", line 
1074, in 
return (window.TimestampedValue(v, window.MAX_TIMESTAMP) for v in outputs)
File "/usr/local/lib/python2.7/site-packages/apache_beam/io/filebasedsink.py", 
line 271, in finalize_write
self._check_state_for_finalize_write(writer_results, num_shards))
File "/usr/local/lib/python2.7/site-packages/apache_beam/io/filebasedsink.py", 
line 249, in _check_state_for_finalize_write
src, dst))
BeamIOError: src and dst files do not exist. src: 
/tmp/beam-temp-py-wordcount-direct-6a0d8862908c11e88de80251/5cfa9f22-9246-41fb-adef-ca04d5a5fe50.py-wordcount-direct,
 dst: /tmp/py-wordcount-direct-0-of-1 with exceptions None [while 
running 'write/Write/WriteImpl/FinalizeWrite'] with exceptions None
{code}

This is after a fix to [a slightly earlier failure in {{FileBasedSink}} 
documented on 
BEAM-4742|https://issues.apache.org/jira/browse/BEAM-4742?focusedCommentId=16545622&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16545622]
 which I've been working on in [#5903|https://github.com/apache/beam/pull/5903].

It typically occurs only on the first run of wordcount against a given 
job-server instance.

I'm curious whether others see this, whether it's some race condition in the 
FileBasedSink, LocalFileSystem, my macbook's disk, or somewhere else, or 
whether some temporary directory is getting created on the first run (for each 
job-server) that explains why subsequent wordcount runs succeed, etc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4747) Python LocalFileSystem directory-creation semantics

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4747?focusedWorklogId=127643&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127643
 ]

ASF GitHub Bot logged work on BEAM-4747:


Author: ASF GitHub Bot
Created on: 26/Jul/18 05:06
Start Date: 26/Jul/18 05:06
Worklog Time Spent: 10m 
  Work Description: ryan-williams commented on issue #5903: [BEAM-4747] 
mkdirs if they don't exist in localfilesystem
URL: https://github.com/apache/beam/pull/5903#issuecomment-407977514
 
 
   OK, I moved the dir-creation to apply on `create` but not `open`, added 
similar logic to `rename` and `copy`, and tried to add tests for all of them in 
a way that made sense. lmk what you think!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127643)
Time Spent: 40m  (was: 0.5h)

> Python LocalFileSystem directory-creation semantics
> ---
>
> Key: BEAM-4747
> URL: https://issues.apache.org/jira/browse/BEAM-4747
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Affects Versions: 2.5.0
>Reporter: Ryan Williams
>Assignee: Ryan Williams
>Priority: Minor
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Coming out of discussion on 
> [BEAM-4742|https://issues.apache.org/jira/browse/BEAM-4742] / 
> [#5903|https://github.com/apache/beam/pull/5903] is a question of whether 
> {{LocalFileSystem.{open,create,copy,rename}}} should create 
> intermediate (destination) directories, or fail with {{IOError}}'s (as the 
> stdlib {{os}} module generally will).
> If the semantics of {{LocalFileSystem}} should mimic those of distributed 
> filesystems (in the spirit of [recent discussion about {{DirectRunner}} being 
> more like a local simulation of a distributed runner than a production-grade 
> local 
> runner|https://www.mail-archive.com/dev@beam.apache.org/msg08410.html]), then 
> this makes sense, and it sounds like [~lcwik] and [~angoenka] are in favor of 
> this interpretation.
> I'll repurpose [#5903|https://github.com/apache/beam/pull/5903] to this end 
> unless I hear otherwise.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4843) Incorrect docs on FileSystems.delete

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4843?focusedWorklogId=127633&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127633
 ]

ASF GitHub Bot logged work on BEAM-4843:


Author: ASF GitHub Bot
Created on: 26/Jul/18 03:05
Start Date: 26/Jul/18 03:05
Worklog Time Spent: 10m 
  Work Description: ryan-williams opened a new pull request #6071: 
[BEAM-4843] remove outdated FileSystem deletion docs
URL: https://github.com/apache/beam/pull/6071
 
 
   As noted in the JIRA, `DeleteOptions` doesn't exist (and afaict never did?)
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | --- | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127633)
Time Spent: 10m
Remaining Estimate: 0h

> Incorrect docs on FileSystems.delete
> 
>
> Key: BEAM-4843
> URL: https://issues.apache.org/jira/browse/BEAM-4843
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.6.0
>Reporter: Ryan Williams
>Assignee: Ryan Williams
>Priority: Minor
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> [The docs on {{FileSystems.delete}} 
> say|https://github.com/apache/beam/blob/b5e8335d982ee69d9f788f65f27356cddd5293d1/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileSystems.java#L332-L333]:
> bq. It is allowed but not recommended to delete directories recursively. 
> Callers depends on {@link FileSystems} and uses {@code DeleteOptions}.
> However, the function actually takes a {{MoveOptions...}} param, there's 
> never been a {{DeleteOptions}} afaict, and there is no way to recursively 
> delete a {{ResourceId}}.
> The docs should be fixed, at a minimum; actually supporting recursive delete 
> would also be nice.



--
This message wa

[jira] [Work logged] (BEAM-3906) Get Python Wheel Validation Automated

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3906?focusedWorklogId=127630&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127630
 ]

ASF GitHub Bot logged work on BEAM-3906:


Author: ASF GitHub Bot
Created on: 26/Jul/18 02:54
Start Date: 26/Jul/18 02:54
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on a change in pull request #4943: 
[BEAM-3906] Automate Validation Aganist Python Wheel
URL: https://github.com/apache/beam/pull/4943#discussion_r205318051
 
 

 ##
 File path: 
release/src/main/python-release/run_release_candidate_python_mobile_gaming.sh
 ##
 @@ -0,0 +1,187 @@
+#!/bin/bash
+#
+#Licensed to the Apache Software Foundation (ASF) under one or more
+#contributor license agreements.  See the NOTICE file distributed with
+#this work for additional information regarding copyright ownership.
+#The ASF licenses this file to You under the Apache License, Version 2.0
+#(the "License"); you may not use this file except in compliance with
+#the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+#Unless required by applicable law or agreed to in writing, software
+#distributed under the License is distributed on an "AS IS" BASIS,
+#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#See the License for the specific language governing permissions and
+#limitations under the License.
+#
+
+#  This file will verify Apache/Beam release candidate python by following 
steps:
+#
+#  1. Create a new virtualenv and install the SDK
+#  2. Run UserScore examples with DirectRunner
+#  3. Run UserScore examples with DataflowRunner
+#  4. Run HourlyTeamScore on DirectRunner
+#  5. Run HourlyTeamScore on DataflowRunner
+#
+
+set -e
+set -v
+
+source release/src/main/python-release/python_release_automation_utils.sh
+
+# Assign default values
+BEAM_PYTHON_SDK=$BEAM_PYTHON_SDK_ZIP
+
+
+###
+# Remove temp directory when complete.
+# Globals:
+#   TMPDIR
+# Arguments:
+#   None
+###
+function complete() {
+  print_separator "Validation $1"
+  rm -rf $TMPDIR
+}
+
+
+###
+# Download files from RC staging location, install python sdk
+# Globals:
+#   BEAM_PYTHON_SDK
+# Arguments:
+#   None
+###
+function install_sdk() {
+  print_separator "Creating new virtualenv and installing the SDK"
+  virtualenv temp_virtualenv
+  . temp_virtualenv/bin/activate
+  gcloud_version=$(gcloud --version | head -1 | awk '{print $4}')
+  if [[ "$gcloud_version" < "189" ]]; then
+update_gcloud
+  fi
+  pip install google-compute-engine
+  pip install $BEAM_PYTHON_SDK[gcp]
+}
+
+
+###
+# Run UserScore with DirectRunner
+# Globals:
+#   USERSCORE_OUTPUT_PREFIX, DATASET, BUCKET_NAME
+# Arguments:
+#   None
+###
+function verify_userscore_direct() {
+  print_separator "Running userscore example with DirectRunner"
+  output_file_name="$USERSCORE_OUTPUT_PREFIX-direct-runner.txt"
+  python -m apache_beam.examples.complete.game.user_score \
+--output=$output_file_name \
+--project=$PROJECT_ID \
+--dataset=$DATASET \
+--input=gs://$BUCKET_NAME/5000_gaming_data.csv
+
+  verify_user_score "direct"
 
 Review comment:
   These lines call the helper functions in the 
python_release_automation_utils.sh to verify results of example pipelines (e.g 
verifying whether output files were created properly, checking if expected 
keywords show in the results, etc).


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127630)
Time Spent: 26h 50m  (was: 26h 40m)

> Get Python Wheel Validation Automated
> -
>
> Key: BEAM-3906
> URL: https://issues.apache.org/jira/browse/BEAM-3906
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python, testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 26h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3906) Get Python Wheel Validation Automated

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3906?focusedWorklogId=127629&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127629
 ]

ASF GitHub Bot logged work on BEAM-3906:


Author: ASF GitHub Bot
Created on: 26/Jul/18 02:53
Start Date: 26/Jul/18 02:53
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on a change in pull request #4943: 
[BEAM-3906] Automate Validation Aganist Python Wheel
URL: https://github.com/apache/beam/pull/4943#discussion_r205318051
 
 

 ##
 File path: 
release/src/main/python-release/run_release_candidate_python_mobile_gaming.sh
 ##
 @@ -0,0 +1,187 @@
+#!/bin/bash
+#
+#Licensed to the Apache Software Foundation (ASF) under one or more
+#contributor license agreements.  See the NOTICE file distributed with
+#this work for additional information regarding copyright ownership.
+#The ASF licenses this file to You under the Apache License, Version 2.0
+#(the "License"); you may not use this file except in compliance with
+#the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+#Unless required by applicable law or agreed to in writing, software
+#distributed under the License is distributed on an "AS IS" BASIS,
+#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#See the License for the specific language governing permissions and
+#limitations under the License.
+#
+
+#  This file will verify Apache/Beam release candidate python by following 
steps:
+#
+#  1. Create a new virtualenv and install the SDK
+#  2. Run UserScore examples with DirectRunner
+#  3. Run UserScore examples with DataflowRunner
+#  4. Run HourlyTeamScore on DirectRunner
+#  5. Run HourlyTeamScore on DataflowRunner
+#
+
+set -e
+set -v
+
+source release/src/main/python-release/python_release_automation_utils.sh
+
+# Assign default values
+BEAM_PYTHON_SDK=$BEAM_PYTHON_SDK_ZIP
+
+
+###
+# Remove temp directory when complete.
+# Globals:
+#   TMPDIR
+# Arguments:
+#   None
+###
+function complete() {
+  print_separator "Validation $1"
+  rm -rf $TMPDIR
+}
+
+
+###
+# Download files from RC staging location, install python sdk
+# Globals:
+#   BEAM_PYTHON_SDK
+# Arguments:
+#   None
+###
+function install_sdk() {
+  print_separator "Creating new virtualenv and installing the SDK"
+  virtualenv temp_virtualenv
+  . temp_virtualenv/bin/activate
+  gcloud_version=$(gcloud --version | head -1 | awk '{print $4}')
+  if [[ "$gcloud_version" < "189" ]]; then
+update_gcloud
+  fi
+  pip install google-compute-engine
+  pip install $BEAM_PYTHON_SDK[gcp]
+}
+
+
+###
+# Run UserScore with DirectRunner
+# Globals:
+#   USERSCORE_OUTPUT_PREFIX, DATASET, BUCKET_NAME
+# Arguments:
+#   None
+###
+function verify_userscore_direct() {
+  print_separator "Running userscore example with DirectRunner"
+  output_file_name="$USERSCORE_OUTPUT_PREFIX-direct-runner.txt"
+  python -m apache_beam.examples.complete.game.user_score \
+--output=$output_file_name \
+--project=$PROJECT_ID \
+--dataset=$DATASET \
+--input=gs://$BUCKET_NAME/5000_gaming_data.csv
+
+  verify_user_score "direct"
 
 Review comment:
   These lines calls the helper functions in the 
python_release_automation_utils.sh to verify results of example pipelines (e.g 
verifying whether output files were created properly, checking if expected 
keywords show in the results, etc).


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127629)
Time Spent: 26h 40m  (was: 26.5h)

> Get Python Wheel Validation Automated
> -
>
> Key: BEAM-3906
> URL: https://issues.apache.org/jira/browse/BEAM-3906
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python, testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 26h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4749) fastavro breaks macos tests

[Ahmet Altay (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16556773#comment-16556773
 ] 

Ahmet Altay commented on BEAM-4749:
---

Thank you! It works for me. I can install and unit tests pass.

> fastavro breaks macos tests
> ---
>
> Key: BEAM-4749
> URL: https://issues.apache.org/jira/browse/BEAM-4749
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Ahmet Altay
>Assignee: Ryan Williams
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Recent addition of the fastavro dependency is breaking python linter in 
> macos. At least for some cases, because it requires a compiler.
> Could we optionally depend on fastavro, and fallback to regular avro package?
>  
> Log:
> *> Task :beam-sdks-python:lintPy27*
> ERROR: invocation failed (exit code 1), logfile: 
> /Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/log/py27-lint-2.log
> ERROR: actionid: py27-lint
> msg: installpkg
> cmdargs: 
> ['/Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/bin/python',
>  '/Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/bin/pip', 
> 'install', 
> '/Users/relax/beam-gradle/beam/sdks/python/target/.tox/dist/apache-beam-2.6.0.dev0.zip[test]']
>  
> Processing ./target/.tox/dist/apache-beam-2.6.0.dev0.zip
> Collecting avro<2.0.0,>=1.8.1 (from apache-beam==2.6.0.dev0)
> Collecting crcmod<2.0,>=1.7 (from apache-beam==2.6.0.dev0)
> Collecting dill==0.2.6 (from apache-beam==2.6.0.dev0)
> Collecting fastavro==0.19.7 (from apache-beam==2.6.0.dev0)
>   Using cached 
> [https://files.pythonhosted.org/packages/a7/0a/b08ba5cef63c675e8442c2bf1cbcef90c8b9f824be2202d492f0cedb0913/fastavro-0.19.7.tar.gz]
> Collecting grpcio<2,>=1.8 (from apache-beam==2.6.0.dev0)
>   Using cached 
> [https://files.pythonhosted.org/packages/66/89/4a90caabd51c17686cbb48a9bbe8c592c4be929c0d2542d2ffde76b0d671/grpcio-1.13.0-cp27-cp27m-macosx_10_12_x86_64.whl]
> Collecting hdfs<3.0.0,>=2.1.0 (from apache-beam==2.6.0.dev0)
> Collecting httplib2<=0.11.3,>=0.8 (from apache-beam==2.6.0.dev0)
> Collecting mock<3.0.0,>=1.0.1 (from apache-beam==2.6.0.dev0)
>   Using cached 
> [https://files.pythonhosted.org/packages/e6/35/f187bdf23be87092bd0f1200d43d23076cee4d0dec109f195173fd3ebc79/mock-2.0.0-py2.py3-none-any.whl]
> Collecting oauth2client<5,>=2.0.1 (from apache-beam==2.6.0.dev0)
>   Using cached 
> [https://files.pythonhosted.org/packages/82/d8/3eab58811282ac7271a081ba5c0d4b875ce786ca68ce43e2a62ade32e9a8/oauth2client-4.1.2-py2.py3-none-any.whl]
> Collecting protobuf<4,>=3.5.0.post1 (from apache-beam==2.6.0.dev0)
>   Using cached 
> [https://files.pythonhosted.org/packages/4f/56/a21f2d077ceae7fd521c0ed31fb8bb1c7f13ffbb09bf7dd27de6cf6bad08/protobuf-3.6.0-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl]
> Collecting pytz<=2018.4,>=2018.3 (from apache-beam==2.6.0.dev0)
>   Using cached 
> [https://files.pythonhosted.org/packages/dc/83/15f7833b70d3e067ca91467ca245bae0f6fe56ddc7451aa0dc5606b120f2/pytz-2018.4-py2.py3-none-any.whl]
> Collecting pyyaml<4.0.0,>=3.12 (from apache-beam==2.6.0.dev0)
>   Using cached 
> [https://files.pythonhosted.org/packages/4a/85/db5a2df477072b2902b0eb892feb37d88ac635d36245a72a6a69b23b383a/PyYAML-3.12.tar.gz]
> Collecting pyvcf<0.7.0,>=0.6.8 (from apache-beam==2.6.0.dev0)
> Requirement already satisfied: six<1.12,>=1.9 in 
> ./target/.tox/py27-lint/lib/python2.7/site-packages (from 
> apache-beam==2.6.0.dev0) (1.11.0)
> Collecting typing<3.7.0,>=3.6.0 (from apache-beam==2.6.0.dev0)
>   Using cached 
> [https://files.pythonhosted.org/packages/0d/4d/4e5985d075d241d686a1663fa1f88b61d544658d08c1375c7c6aac32afc3/typing-3.6.4-py2-none-any.whl]
> Collecting futures<4.0.0,>=3.1.1 (from apache-beam==2.6.0.dev0)
>   Using cached 
> [https://files.pythonhosted.org/packages/2d/99/b2c4e9d5a30f6471e410a146232b4118e697fa3ffc06d6a65efde84debd0/futures-3.2.0-py2-none-any.whl]
> Requirement already satisfied: future<1.0.0,>=0.16.0 in 
> ./target/.tox/py27-lint/lib/python2.7/site-packages (from 
> apache-beam==2.6.0.dev0) (0.16.0)
> Collecting nose>=1.3.7 (from apache-beam==2.6.0.dev0)
>   Using cached 
> [https://files.pythonhosted.org/packages/99/4f/13fb671119e65c4dce97c60e67d3fd9e6f7f809f2b307e2611f4701205cb/nose-1.3.7-py2-none-any.whl]
> Collecting pyhamcrest<2.0,>=1.9 (from apache-beam==2.6.0.dev0)
>   Using cached 
> [https://files.pythonhosted.org/packages/9a/d5/d37fd731b7d0e91afcc84577edeccf4638b4f9b82f5ffe2f8b62e2ddc609/PyHamcrest-1.9.0-py2.py3-none-any.whl]
> Requirement already satisfied: setuptools>=18.0 in 
> ./target/.tox/py27-lint/lib/python2.7/site-packages (from 
> fastavro==0.19.7->apache-beam==2.6.0.dev0) (39.2.0)
> Requirement already satisfied: enum34>=1.0.4 in 

[jira] [Work logged] (BEAM-4747) Python LocalFileSystem directory-creation semantics

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4747?focusedWorklogId=127623&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127623
 ]

ASF GitHub Bot logged work on BEAM-4747:


Author: ASF GitHub Bot
Created on: 26/Jul/18 01:40
Start Date: 26/Jul/18 01:40
Worklog Time Spent: 10m 
  Work Description: ryan-williams commented on issue #5903: [BEAM-4747] 
mkdirs if they don't exist in localfilesystem
URL: https://github.com/apache/beam/pull/5903#issuecomment-407948763
 
 
   Sorry I dropped this, looking into it now!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127623)
Time Spent: 0.5h  (was: 20m)

> Python LocalFileSystem directory-creation semantics
> ---
>
> Key: BEAM-4747
> URL: https://issues.apache.org/jira/browse/BEAM-4747
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Affects Versions: 2.5.0
>Reporter: Ryan Williams
>Assignee: Ryan Williams
>Priority: Minor
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Coming out of discussion on 
> [BEAM-4742|https://issues.apache.org/jira/browse/BEAM-4742] / 
> [#5903|https://github.com/apache/beam/pull/5903] is a question of whether 
> {{LocalFileSystem.{open,create,copy,rename}}} should create 
> intermediate (destination) directories, or fail with {{IOError}}'s (as the 
> stdlib {{os}} module generally will).
> If the semantics of {{LocalFileSystem}} should mimic those of distributed 
> filesystems (in the spirit of [recent discussion about {{DirectRunner}} being 
> more like a local simulation of a distributed runner than a production-grade 
> local 
> runner|https://www.mail-archive.com/dev@beam.apache.org/msg08410.html]), then 
> this makes sense, and it sounds like [~lcwik] and [~angoenka] are in favor of 
> this interpretation.
> I'll repurpose [#5903|https://github.com/apache/beam/pull/5903] to this end 
> unless I hear otherwise.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4711) LocalFileSystem.delete doesn't support globbing

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4711?focusedWorklogId=127624&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127624
 ]

ASF GitHub Bot logged work on BEAM-4711:


Author: ASF GitHub Bot
Created on: 26/Jul/18 01:40
Start Date: 26/Jul/18 01:40
Worklog Time Spent: 10m 
  Work Description: ryan-williams commented on issue #5863: [BEAM-4711] fix 
globbing in LocalFileSystem.delete
URL: https://github.com/apache/beam/pull/5863#issuecomment-407948787
 
 
   I never pinged for review here; @lukecwik do you care to take a look?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127624)
Time Spent: 20m  (was: 10m)

> LocalFileSystem.delete doesn't support globbing
> ---
>
> Key: BEAM-4711
> URL: https://issues.apache.org/jira/browse/BEAM-4711
> Project: Beam
>  Issue Type: Task
>  Components: sdk-py-core
>Affects Versions: 2.5.0
>Reporter: Ryan Williams
>Assignee: Ryan Williams
>Priority: Minor
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> I attempted to run {{wordcount_it_test:WordCountIT.test_wordcount_it}} 
> locally with {{DirectRunner}}:
> {code}
> python setup.py nosetests \
>   --tests 
> apache_beam.examples.wordcount_it_test:WordCountIT.test_wordcount_it \
>   --test-pipeline-options="--output=foo"
> {code}
> It failed in [the {{delete_files}} cleanup 
> command|https://github.com/apache/beam/blob/a58f1ffaafb0e2ebcc73a1c5abfb05a15ec6a84b/sdks/python/apache_beam/examples/wordcount_it_test.py#L64]:
> {code}
> root: WARNING: Retry with exponential backoff: waiting for 11.1454450937 
> seconds before retrying delete_files because we caught exception: 
> BeamIOError: Delete operation failed with exceptions 
> {'foo/1530557644/results*': IOError(OSError(2, 'No such file or directory'),)}
>  Traceback for above exception (most recent call last):
>   File "/Users/ryan/c/beam/sdks/python/apache_beam/utils/retry.py", line 184, 
> in wrapper
> return fun(*args, **kwargs)
>   File "/Users/ryan/c/beam/sdks/python/apache_beam/testing/test_utils.py", 
> line 136, in delete_files
> FileSystems.delete(file_paths)
>   File "/Users/ryan/c/beam/sdks/python/apache_beam/io/filesystems.py", line 
> 282, in delete
> return filesystem.delete(paths)
>   File "/Users/ryan/c/beam/sdks/python/apache_beam/io/localfilesystem.py", 
> line 304, in delete
> raise BeamIOError("Delete operation failed", exceptions)
> {code}
> The line:
> {code}
> self.addCleanup(delete_files, [output + '*'])
> {code}
> works as expected in GCS, and deletes a test's output-directory, but it fails 
> in on the local-filesystem, which doesn't expand globs before attempting to 
> delete paths.
> It would be good to make these consistent, presumably by adding glob-support 
> to {{LocalFileSystem}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3906) Get Python Wheel Validation Automated

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3906?focusedWorklogId=127622&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127622
 ]

ASF GitHub Bot logged work on BEAM-3906:


Author: ASF GitHub Bot
Created on: 26/Jul/18 01:37
Start Date: 26/Jul/18 01:37
Worklog Time Spent: 10m 
  Work Description: aaltay commented on a change in pull request #4943: 
[BEAM-3906] Automate Validation Aganist Python Wheel
URL: https://github.com/apache/beam/pull/4943#discussion_r205308970
 
 

 ##
 File path: 
release/src/main/python-release/run_release_candidate_python_mobile_gaming.sh
 ##
 @@ -0,0 +1,187 @@
+#!/bin/bash
+#
+#Licensed to the Apache Software Foundation (ASF) under one or more
+#contributor license agreements.  See the NOTICE file distributed with
+#this work for additional information regarding copyright ownership.
+#The ASF licenses this file to You under the Apache License, Version 2.0
+#(the "License"); you may not use this file except in compliance with
+#the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+#Unless required by applicable law or agreed to in writing, software
+#distributed under the License is distributed on an "AS IS" BASIS,
+#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#See the License for the specific language governing permissions and
+#limitations under the License.
+#
+
+#  This file will verify Apache/Beam release candidate python by following 
steps:
+#
+#  1. Create a new virtualenv and install the SDK
+#  2. Run UserScore examples with DirectRunner
+#  3. Run UserScore examples with DataflowRunner
+#  4. Run HourlyTeamScore on DirectRunner
+#  5. Run HourlyTeamScore on DataflowRunner
+#
+
+set -e
+set -v
+
+source release/src/main/python-release/python_release_automation_utils.sh
+
+# Assign default values
+BEAM_PYTHON_SDK=$BEAM_PYTHON_SDK_ZIP
+
+
+###
+# Remove temp directory when complete.
+# Globals:
+#   TMPDIR
+# Arguments:
+#   None
+###
+function complete() {
+  print_separator "Validation $1"
+  rm -rf $TMPDIR
+}
+
+
+###
+# Download files from RC staging location, install python sdk
+# Globals:
+#   BEAM_PYTHON_SDK
+# Arguments:
+#   None
+###
+function install_sdk() {
+  print_separator "Creating new virtualenv and installing the SDK"
+  virtualenv temp_virtualenv
+  . temp_virtualenv/bin/activate
+  gcloud_version=$(gcloud --version | head -1 | awk '{print $4}')
+  if [[ "$gcloud_version" < "189" ]]; then
+update_gcloud
+  fi
+  pip install google-compute-engine
+  pip install $BEAM_PYTHON_SDK[gcp]
+}
+
+
+###
+# Run UserScore with DirectRunner
+# Globals:
+#   USERSCORE_OUTPUT_PREFIX, DATASET, BUCKET_NAME
+# Arguments:
+#   None
+###
+function verify_userscore_direct() {
+  print_separator "Running userscore example with DirectRunner"
+  output_file_name="$USERSCORE_OUTPUT_PREFIX-direct-runner.txt"
+  python -m apache_beam.examples.complete.game.user_score \
+--output=$output_file_name \
+--project=$PROJECT_ID \
+--dataset=$DATASET \
+--input=gs://$BUCKET_NAME/5000_gaming_data.csv
+
+  verify_user_score "direct"
 
 Review comment:
   What does this line do? (Similar comment for a few other calls inside 
functions to the wrapping function.)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127622)
Time Spent: 26.5h  (was: 26h 20m)

> Get Python Wheel Validation Automated
> -
>
> Key: BEAM-3906
> URL: https://issues.apache.org/jira/browse/BEAM-3906
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python, testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 26.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4857) Beam Dependency Update Request: groupA:artifactA 4.0

[Beam JIRA Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Beam JIRA Bot updated BEAM-4857:

Description: 


2018-07-24 15:50:08.854551

Please review and upgrade the groupA:artifactA to the latest version 
4.0 

cc: 

2018-07-24 15:54:13.532422

Please review and upgrade the groupA:artifactA to the latest version 
4.0 

cc: 

2018-07-24 16:14:13.608317

Please review and upgrade the groupA:artifactA to the latest version 
4.0 

cc: 

2018-07-25 11:38:36.010963

Please review and upgrade the groupA:artifactA to the latest version 
4.0 

cc: 

2018-07-26 01:30:29.331974

Please review and upgrade the groupA:artifactA to the latest version 
4.0 
 
cc: 

  was:


2018-07-24 15:50:08.854551

Please review and upgrade the groupA:artifactA to the latest version 
4.0 

cc: 

2018-07-24 15:54:13.532422

Please review and upgrade the groupA:artifactA to the latest version 
4.0 

cc: 

2018-07-24 16:14:13.608317

Please review and upgrade the groupA:artifactA to the latest version 
4.0 

cc: 

2018-07-25 11:38:36.010963

Please review and upgrade the groupA:artifactA to the latest version 
4.0 

cc: 


> Beam Dependency Update Request: groupA:artifactA 4.0
> 
>
> Key: BEAM-4857
> URL: https://issues.apache.org/jira/browse/BEAM-4857
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Assignee: yifan zou
>Priority: Major
>
> 2018-07-24 15:50:08.854551
> Please review and upgrade the groupA:artifactA to the latest version 
> 4.0 
> cc: 
> 2018-07-24 15:54:13.532422
> Please review and upgrade the groupA:artifactA to the latest version 
> 4.0 
> cc: 
> 2018-07-24 16:14:13.608317
> Please review and upgrade the groupA:artifactA to the latest version 
> 4.0 
> cc: 
> 2018-07-25 11:38:36.010963
> Please review and upgrade the groupA:artifactA to the latest version 
> 4.0 
> cc: 
> 2018-07-26 01:30:29.331974
> Please review and upgrade the groupA:artifactA to the latest version 
> 4.0 
>  
> cc: 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4749) fastavro breaks macos tests

[Ryan Williams (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16556754#comment-16556754
 ] 

Ryan Williams commented on BEAM-4749:
-

[fastavro#253|https://github.com/fastavro/fastavro/issues/253] is ostensibly 
fixed; I'm working on testing it but it would be great if you could as well 
[~altay] since you have state on what environments were having issues

> fastavro breaks macos tests
> ---
>
> Key: BEAM-4749
> URL: https://issues.apache.org/jira/browse/BEAM-4749
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Ahmet Altay
>Assignee: Ryan Williams
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Recent addition of the fastavro dependency is breaking python linter in 
> macos. At least for some cases, because it requires a compiler.
> Could we optionally depend on fastavro, and fallback to regular avro package?
>  
> Log:
> *> Task :beam-sdks-python:lintPy27*
> ERROR: invocation failed (exit code 1), logfile: 
> /Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/log/py27-lint-2.log
> ERROR: actionid: py27-lint
> msg: installpkg
> cmdargs: 
> ['/Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/bin/python',
>  '/Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/bin/pip', 
> 'install', 
> '/Users/relax/beam-gradle/beam/sdks/python/target/.tox/dist/apache-beam-2.6.0.dev0.zip[test]']
>  
> Processing ./target/.tox/dist/apache-beam-2.6.0.dev0.zip
> Collecting avro<2.0.0,>=1.8.1 (from apache-beam==2.6.0.dev0)
> Collecting crcmod<2.0,>=1.7 (from apache-beam==2.6.0.dev0)
> Collecting dill==0.2.6 (from apache-beam==2.6.0.dev0)
> Collecting fastavro==0.19.7 (from apache-beam==2.6.0.dev0)
>   Using cached 
> [https://files.pythonhosted.org/packages/a7/0a/b08ba5cef63c675e8442c2bf1cbcef90c8b9f824be2202d492f0cedb0913/fastavro-0.19.7.tar.gz]
> Collecting grpcio<2,>=1.8 (from apache-beam==2.6.0.dev0)
>   Using cached 
> [https://files.pythonhosted.org/packages/66/89/4a90caabd51c17686cbb48a9bbe8c592c4be929c0d2542d2ffde76b0d671/grpcio-1.13.0-cp27-cp27m-macosx_10_12_x86_64.whl]
> Collecting hdfs<3.0.0,>=2.1.0 (from apache-beam==2.6.0.dev0)
> Collecting httplib2<=0.11.3,>=0.8 (from apache-beam==2.6.0.dev0)
> Collecting mock<3.0.0,>=1.0.1 (from apache-beam==2.6.0.dev0)
>   Using cached 
> [https://files.pythonhosted.org/packages/e6/35/f187bdf23be87092bd0f1200d43d23076cee4d0dec109f195173fd3ebc79/mock-2.0.0-py2.py3-none-any.whl]
> Collecting oauth2client<5,>=2.0.1 (from apache-beam==2.6.0.dev0)
>   Using cached 
> [https://files.pythonhosted.org/packages/82/d8/3eab58811282ac7271a081ba5c0d4b875ce786ca68ce43e2a62ade32e9a8/oauth2client-4.1.2-py2.py3-none-any.whl]
> Collecting protobuf<4,>=3.5.0.post1 (from apache-beam==2.6.0.dev0)
>   Using cached 
> [https://files.pythonhosted.org/packages/4f/56/a21f2d077ceae7fd521c0ed31fb8bb1c7f13ffbb09bf7dd27de6cf6bad08/protobuf-3.6.0-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl]
> Collecting pytz<=2018.4,>=2018.3 (from apache-beam==2.6.0.dev0)
>   Using cached 
> [https://files.pythonhosted.org/packages/dc/83/15f7833b70d3e067ca91467ca245bae0f6fe56ddc7451aa0dc5606b120f2/pytz-2018.4-py2.py3-none-any.whl]
> Collecting pyyaml<4.0.0,>=3.12 (from apache-beam==2.6.0.dev0)
>   Using cached 
> [https://files.pythonhosted.org/packages/4a/85/db5a2df477072b2902b0eb892feb37d88ac635d36245a72a6a69b23b383a/PyYAML-3.12.tar.gz]
> Collecting pyvcf<0.7.0,>=0.6.8 (from apache-beam==2.6.0.dev0)
> Requirement already satisfied: six<1.12,>=1.9 in 
> ./target/.tox/py27-lint/lib/python2.7/site-packages (from 
> apache-beam==2.6.0.dev0) (1.11.0)
> Collecting typing<3.7.0,>=3.6.0 (from apache-beam==2.6.0.dev0)
>   Using cached 
> [https://files.pythonhosted.org/packages/0d/4d/4e5985d075d241d686a1663fa1f88b61d544658d08c1375c7c6aac32afc3/typing-3.6.4-py2-none-any.whl]
> Collecting futures<4.0.0,>=3.1.1 (from apache-beam==2.6.0.dev0)
>   Using cached 
> [https://files.pythonhosted.org/packages/2d/99/b2c4e9d5a30f6471e410a146232b4118e697fa3ffc06d6a65efde84debd0/futures-3.2.0-py2-none-any.whl]
> Requirement already satisfied: future<1.0.0,>=0.16.0 in 
> ./target/.tox/py27-lint/lib/python2.7/site-packages (from 
> apache-beam==2.6.0.dev0) (0.16.0)
> Collecting nose>=1.3.7 (from apache-beam==2.6.0.dev0)
>   Using cached 
> [https://files.pythonhosted.org/packages/99/4f/13fb671119e65c4dce97c60e67d3fd9e6f7f809f2b307e2611f4701205cb/nose-1.3.7-py2-none-any.whl]
> Collecting pyhamcrest<2.0,>=1.9 (from apache-beam==2.6.0.dev0)
>   Using cached 
> [https://files.pythonhosted.org/packages/9a/d5/d37fd731b7d0e91afcc84577edeccf4638b4f9b82f5ffe2f8b62e2ddc609/PyHamcrest-1.9.0-py2.py3-none-any.whl]
> Requirement already satisfied: setuptools>=18.0 in 

[jira] [Work logged] (BEAM-4629) RAT failures are hard to debug

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4629?focusedWorklogId=127621&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127621
 ]

ASF GitHub Bot logged work on BEAM-4629:


Author: ASF GitHub Bot
Created on: 26/Jul/18 01:29
Start Date: 26/Jul/18 01:29
Worklog Time Spent: 10m 
  Work Description: aaltay closed pull request #6069: [BEAM-4629] Output 
the names of the files with missing/unknown licenses as part of the build 
process.
URL: https://github.com/apache/beam/pull/6069
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/build.gradle b/build.gradle
index dfbb6033065..17ea8915dbc 100644
--- a/build.gradle
+++ b/build.gradle
@@ -15,7 +15,6 @@
  * See the License for the specific language governing permissions and
  * limitations under the License.
  */
-
 // See build_rules.gradle for documentation on default build tasks
 // and properties that are enabled in addition to natures
 // that can be applied to configure a project for certain common
@@ -147,8 +146,11 @@ rat {
 exclusions.addAll(gitIgnoreExcludes)
   }
 
-  plainOutput = true
-  xmlOutput = false
+  // Combining verbose with only XML output has each failing license logged.
+  // See https://github.com/eskatos/creadur-rat-gradle/issues/8 for further 
details.
+  verbose = true
+  plainOutput = false
+  xmlOutput = true
   htmlOutput = false
   failOnError = true
   excludes = exclusions


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127621)
Time Spent: 0.5h  (was: 20m)

> RAT failures are hard to debug
> --
>
> Key: BEAM-4629
> URL: https://issues.apache.org/jira/browse/BEAM-4629
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Ahmet Altay
>Assignee: Luke Cwik
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> RAT failures produces an error with a reference to log file that is not 
> accessible.
> For example:
> > Task :rat
> Build cache key for task ':rat' is 7a8f8655183c948a3f7a087e2f7c412b
> Caching disabled for task ':rat': Caching has not been enabled for the task
> Task ':rat' is not up-to-date because:
>   No history is available.
> Rat TXT report: 
> file:/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild@2/src/build/reports/rat/rat-report.txt
> This is not useful unless, the person has access to the jenkins VM. Tests 
> should include a copy of the failed rat-report.txt.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Merge pull request #6069 from lukecwik/gradle2

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit dd5de56908f31a80bc05bae9e6b7ae0251ae9b54
Merge: 31fea5d 68e0da3
Author: Ahmet Altay 
AuthorDate: Wed Jul 25 18:29:29 2018 -0700

Merge pull request #6069 from lukecwik/gradle2

[BEAM-4629] Output the names of the files with missing/unknown licenses as 
part of the build process.

 build.gradle | 8 +---
 1 file changed, 5 insertions(+), 3 deletions(-)

[beam] branch master updated (31fea5d -> dd5de56)

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 31fea5d  [BEAM-4859] Enable Python VR tests in streaming in postcommit 
task (#6053)
 add 68e0da3  [BEAM-4629] Output the names of the failing licenses as part 
of the build process.
 new dd5de56  Merge pull request #6069 from lukecwik/gradle2

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 build.gradle | 8 +---
 1 file changed, 5 insertions(+), 3 deletions(-)

[jira] [Work logged] (BEAM-4859) Run Python ValidatesRunner tests in streaming

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4859?focusedWorklogId=127620&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127620
 ]

ASF GitHub Bot logged work on BEAM-4859:


Author: ASF GitHub Bot
Created on: 26/Jul/18 01:26
Start Date: 26/Jul/18 01:26
Worklog Time Spent: 10m 
  Work Description: aaltay closed pull request #6053: [BEAM-4859] Enable 
Python VR tests in streaming in postcommit task
URL: https://github.com/apache/beam/pull/6053
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/.test-infra/jenkins/job_PostCommit_Python_ValidatesRunner_Dataflow.groovy 
b/.test-infra/jenkins/job_PostCommit_Python_ValidatesRunner_Dataflow.groovy
index 6ebda53e867..cfdd01bf996 100644
--- a/.test-infra/jenkins/job_PostCommit_Python_ValidatesRunner_Dataflow.groovy
+++ b/.test-infra/jenkins/job_PostCommit_Python_ValidatesRunner_Dataflow.groovy
@@ -32,7 +32,8 @@ 
PostcommitJobBuilder.postCommitJob('beam_PostCommit_Py_VR_Dataflow', 'Run Python
   steps {
 gradle {
   rootBuildScriptDir(commonJobProperties.checkoutDir)
-  tasks(':beam-sdks-python:validatesRunnerTests')
+  tasks(':beam-sdks-python:validatesRunnerBatchTests')
+  tasks(':beam-sdks-python:validatesRunnerStreamingTests')
   commonJobProperties.setGradleSwitches(delegate)
 }
   }
diff --git a/sdks/python/apache_beam/transforms/ptransform_test.py 
b/sdks/python/apache_beam/transforms/ptransform_test.py
index e23fad7a78f..907ee04c079 100644
--- a/sdks/python/apache_beam/transforms/ptransform_test.py
+++ b/sdks/python/apache_beam/transforms/ptransform_test.py
@@ -180,7 +180,9 @@ def 
test_do_with_multiple_outputs_maintains_unique_name(self):
 assert_that(r2.m, equal_to([3, 4, 5]), label='r2')
 pipeline.run()
 
-  @attr('ValidatesRunner')
+  # TODO(BEAM-3544): Disable this test in streaming temporarily.
+  # Remove sickbay-streaming tag after it's resolved.
+  @attr('ValidatesRunner', 'sickbay-streaming')
   def test_read_metrics(self):
 from apache_beam.io.utils import CountingSource
 
diff --git a/sdks/python/apache_beam/transforms/sideinputs_test.py 
b/sdks/python/apache_beam/transforms/sideinputs_test.py
index 1d5883481b6..6b93b8e9137 100644
--- a/sdks/python/apache_beam/transforms/sideinputs_test.py
+++ b/sdks/python/apache_beam/transforms/sideinputs_test.py
@@ -146,7 +146,9 @@ def my_fn(k, s):
 assert_that(result, equal_to([(1, 'empty'), (2, 'empty')]))
 pipeline.run()
 
-  @attr('ValidatesRunner')
+  # TODO(BEAM-5025): Disable this test in streaming temporarily.
+  # Remove sickbay-streaming tag after it's fixed.
+  @attr('ValidatesRunner', 'sickbay-streaming')
   def test_multi_valued_singleton_side_input(self):
 pipeline = self.create_pipeline()
 pcol = pipeline | 'start' >> beam.Create([1, 2])
diff --git a/sdks/python/build.gradle b/sdks/python/build.gradle
index a227af59fa2..0cbd61df83d 100644
--- a/sdks/python/build.gradle
+++ b/sdks/python/build.gradle
@@ -223,20 +223,30 @@ task portableWordCount(dependsOn: 'installGcpTest') {
   }
 }
 
-task postCommitVRTests(dependsOn: 'installGcpTest') {
+task postCommitITTests(dependsOn: 'installGcpTest') {
   doLast {
 exec {
   executable 'sh'
-  args '-c', ". ${envdir}/bin/activate && ./scripts/run_postcommit.sh IT"
+  args '-c', ". ${envdir}/bin/activate && ./scripts/run_postcommit.sh IT 
batch"
 }
   }
 }
 
-task validatesRunnerTests(dependsOn: 'installGcpTest') {
+task validatesRunnerBatchTests(dependsOn: 'installGcpTest') {
   doLast {
 exec {
   executable 'sh'
-  args '-c', ". ${envdir}/bin/activate && ./scripts/run_postcommit.sh 
ValidatesRunner"
+  args '-c', ". ${envdir}/bin/activate && ./scripts/run_postcommit.sh 
ValidatesRunner batch"
+}
+  }
+}
+
+task validatesRunnerStreamingTests(dependsOn: 'installGcpTest') {
+  doLast {
+exec {
+  executable 'sh'
+  // TODO(BEAM-3544,BEAM-5025): Disable tests with sickbay-streaming tag.
+  args '-c', ". ${envdir}/bin/activate && ./scripts/run_postcommit.sh 
ValidatesRunner,'!sickbay-streaming' streaming"
 }
   }
 }
@@ -254,7 +264,7 @@ task postCommit() {
   dependsOn "preCommit"
   dependsOn "localWordCount"
   dependsOn "hdfsIntegrationTest"
-  dependsOn "postCommitVRTests"
+  dependsOn "postCommitITTests"
 }
 
 task dependencyUpdates(dependsOn: ':dependencyUpdates') {
diff --git a/sdks/python/scripts/run_postcommit.sh 
b/sdks/python/scripts/run_postcommit.sh
index d6e31aaccff..c13ef505fa9 100755
--- a/sdks/python/scripts/run_postcommit.sh
+++ b/sdks/python/scripts/run_postcommit.sh
@@ -16,6 +16,8 @@
 #limitations under the License.
 #
 
+#

[beam] branch master updated: [BEAM-4859] Enable Python VR tests in streaming in postcommit task (#6053)

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new 31fea5d  [BEAM-4859] Enable Python VR tests in streaming in postcommit 
task (#6053)
31fea5d is described below

commit 31fea5d86444fe879afba43cf199e7cb7a7a5e73
Author: Mark Liu 
AuthorDate: Wed Jul 25 18:25:58 2018 -0700

[BEAM-4859] Enable Python VR tests in streaming in postcommit task (#6053)

* [BEAM-4859] Enable Python VR tests in streaming in Jenkins postcommit
* Increase integration test timeout since streaming takes longer time
* Disable test_read_metrics in streaming due to BEAM-3544
* Disable test_multi_valued_singleton_side_input in streaming due to 
BEAM-5025
---
 ...stCommit_Python_ValidatesRunner_Dataflow.groovy |  3 +-
 .../apache_beam/transforms/ptransform_test.py  |  4 +-
 .../apache_beam/transforms/sideinputs_test.py  |  4 +-
 sdks/python/build.gradle   | 20 --
 sdks/python/scripts/run_postcommit.sh  | 84 ++
 5 files changed, 78 insertions(+), 37 deletions(-)

diff --git 
a/.test-infra/jenkins/job_PostCommit_Python_ValidatesRunner_Dataflow.groovy 
b/.test-infra/jenkins/job_PostCommit_Python_ValidatesRunner_Dataflow.groovy
index 6ebda53..cfdd01b 100644
--- a/.test-infra/jenkins/job_PostCommit_Python_ValidatesRunner_Dataflow.groovy
+++ b/.test-infra/jenkins/job_PostCommit_Python_ValidatesRunner_Dataflow.groovy
@@ -32,7 +32,8 @@ 
PostcommitJobBuilder.postCommitJob('beam_PostCommit_Py_VR_Dataflow', 'Run Python
   steps {
 gradle {
   rootBuildScriptDir(commonJobProperties.checkoutDir)
-  tasks(':beam-sdks-python:validatesRunnerTests')
+  tasks(':beam-sdks-python:validatesRunnerBatchTests')
+  tasks(':beam-sdks-python:validatesRunnerStreamingTests')
   commonJobProperties.setGradleSwitches(delegate)
 }
   }
diff --git a/sdks/python/apache_beam/transforms/ptransform_test.py 
b/sdks/python/apache_beam/transforms/ptransform_test.py
index e23fad7..907ee04 100644
--- a/sdks/python/apache_beam/transforms/ptransform_test.py
+++ b/sdks/python/apache_beam/transforms/ptransform_test.py
@@ -180,7 +180,9 @@ class PTransformTest(unittest.TestCase):
 assert_that(r2.m, equal_to([3, 4, 5]), label='r2')
 pipeline.run()
 
-  @attr('ValidatesRunner')
+  # TODO(BEAM-3544): Disable this test in streaming temporarily.
+  # Remove sickbay-streaming tag after it's resolved.
+  @attr('ValidatesRunner', 'sickbay-streaming')
   def test_read_metrics(self):
 from apache_beam.io.utils import CountingSource
 
diff --git a/sdks/python/apache_beam/transforms/sideinputs_test.py 
b/sdks/python/apache_beam/transforms/sideinputs_test.py
index 1d58834..6b93b8e 100644
--- a/sdks/python/apache_beam/transforms/sideinputs_test.py
+++ b/sdks/python/apache_beam/transforms/sideinputs_test.py
@@ -146,7 +146,9 @@ class SideInputsTest(unittest.TestCase):
 assert_that(result, equal_to([(1, 'empty'), (2, 'empty')]))
 pipeline.run()
 
-  @attr('ValidatesRunner')
+  # TODO(BEAM-5025): Disable this test in streaming temporarily.
+  # Remove sickbay-streaming tag after it's fixed.
+  @attr('ValidatesRunner', 'sickbay-streaming')
   def test_multi_valued_singleton_side_input(self):
 pipeline = self.create_pipeline()
 pcol = pipeline | 'start' >> beam.Create([1, 2])
diff --git a/sdks/python/build.gradle b/sdks/python/build.gradle
index a227af5..0cbd61d 100644
--- a/sdks/python/build.gradle
+++ b/sdks/python/build.gradle
@@ -223,20 +223,30 @@ task portableWordCount(dependsOn: 'installGcpTest') {
   }
 }
 
-task postCommitVRTests(dependsOn: 'installGcpTest') {
+task postCommitITTests(dependsOn: 'installGcpTest') {
   doLast {
 exec {
   executable 'sh'
-  args '-c', ". ${envdir}/bin/activate && ./scripts/run_postcommit.sh IT"
+  args '-c', ". ${envdir}/bin/activate && ./scripts/run_postcommit.sh IT 
batch"
 }
   }
 }
 
-task validatesRunnerTests(dependsOn: 'installGcpTest') {
+task validatesRunnerBatchTests(dependsOn: 'installGcpTest') {
   doLast {
 exec {
   executable 'sh'
-  args '-c', ". ${envdir}/bin/activate && ./scripts/run_postcommit.sh 
ValidatesRunner"
+  args '-c', ". ${envdir}/bin/activate && ./scripts/run_postcommit.sh 
ValidatesRunner batch"
+}
+  }
+}
+
+task validatesRunnerStreamingTests(dependsOn: 'installGcpTest') {
+  doLast {
+exec {
+  executable 'sh'
+  // TODO(BEAM-3544,BEAM-5025): Disable tests with sickbay-streaming tag.
+  args '-c', ". ${envdir}/bin/activate && ./scripts/run_postcommit.sh 
ValidatesRunner,'!sickbay-streaming' streaming"
 }
   }
 }
@@ -254,7 +264,7 @@ task postCommit() {
   dependsOn "preCommit"
   dependsOn "localWordCount"
   dependsOn "hdfsIntegrationTest"
-  dependsOn "postCommitVRTests"
+  dependsOn "postCommitITTests"
 }
 
 task dependencyUpdates(depen

Jenkins build is back to normal : beam_Dependency_Check #120

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-4859) Run Python ValidatesRunner tests in streaming

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4859?focusedWorklogId=127614&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127614
 ]

ASF GitHub Bot logged work on BEAM-4859:


Author: ASF GitHub Bot
Created on: 26/Jul/18 00:54
Start Date: 26/Jul/18 00:54
Worklog Time Spent: 10m 
  Work Description: markflyhigh commented on issue #6053: [BEAM-4859] 
Enable Python VR tests in streaming in postcommit task
URL: https://github.com/apache/beam/pull/6053#issuecomment-407941652
 
 
   Run Python Dataflow ValidatesRunner


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127614)
Time Spent: 4.5h  (was: 4h 20m)

> Run Python ValidatesRunner tests in streaming
> -
>
> Key: BEAM-4859
> URL: https://issues.apache.org/jira/browse/BEAM-4859
> Project: Beam
>  Issue Type: Test
>  Components: testing
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Major
>  Time Spent: 4.5h
>  Remaining Estimate: 0h
>
> Automated way to run existing Python ValidatesRunner tests in streaming. 
> Gradle should also be able to run them, so Jenkins automation will be much 
> simpler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4859) Run Python ValidatesRunner tests in streaming

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4859?focusedWorklogId=127612&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127612
 ]

ASF GitHub Bot logged work on BEAM-4859:


Author: ASF GitHub Bot
Created on: 26/Jul/18 00:50
Start Date: 26/Jul/18 00:50
Worklog Time Spent: 10m 
  Work Description: markflyhigh commented on issue #6053: [BEAM-4859] 
Enable Python VR tests in streaming in postcommit task
URL: https://github.com/apache/beam/pull/6053#issuecomment-407941095
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127612)
Time Spent: 4h 10m  (was: 4h)

> Run Python ValidatesRunner tests in streaming
> -
>
> Key: BEAM-4859
> URL: https://issues.apache.org/jira/browse/BEAM-4859
> Project: Beam
>  Issue Type: Test
>  Components: testing
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Major
>  Time Spent: 4h 10m
>  Remaining Estimate: 0h
>
> Automated way to run existing Python ValidatesRunner tests in streaming. 
> Gradle should also be able to run them, so Jenkins automation will be much 
> simpler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4859) Run Python ValidatesRunner tests in streaming

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4859?focusedWorklogId=127613&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127613
 ]

ASF GitHub Bot logged work on BEAM-4859:


Author: ASF GitHub Bot
Created on: 26/Jul/18 00:50
Start Date: 26/Jul/18 00:50
Worklog Time Spent: 10m 
  Work Description: markflyhigh removed a comment on issue #6053: 
[BEAM-4859] Enable Python VR tests in streaming in postcommit task
URL: https://github.com/apache/beam/pull/6053#issuecomment-407920485
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127613)
Time Spent: 4h 20m  (was: 4h 10m)

> Run Python ValidatesRunner tests in streaming
> -
>
> Key: BEAM-4859
> URL: https://issues.apache.org/jira/browse/BEAM-4859
> Project: Beam
>  Issue Type: Test
>  Components: testing
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Major
>  Time Spent: 4h 20m
>  Remaining Estimate: 0h
>
> Automated way to run existing Python ValidatesRunner tests in streaming. 
> Gradle should also be able to run them, so Jenkins automation will be much 
> simpler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4859) Run Python ValidatesRunner tests in streaming

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4859?focusedWorklogId=127611&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127611
 ]

ASF GitHub Bot logged work on BEAM-4859:


Author: ASF GitHub Bot
Created on: 26/Jul/18 00:47
Start Date: 26/Jul/18 00:47
Worklog Time Spent: 10m 
  Work Description: markflyhigh commented on issue #6053: [BEAM-4859] 
Enable Python VR tests in streaming in postcommit task
URL: https://github.com/apache/beam/pull/6053#issuecomment-407940561
 
 
   Will rerun VR test and it should pass this time.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127611)
Time Spent: 4h  (was: 3h 50m)

> Run Python ValidatesRunner tests in streaming
> -
>
> Key: BEAM-4859
> URL: https://issues.apache.org/jira/browse/BEAM-4859
> Project: Beam
>  Issue Type: Test
>  Components: testing
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Major
>  Time Spent: 4h
>  Remaining Estimate: 0h
>
> Automated way to run existing Python ValidatesRunner tests in streaming. 
> Gradle should also be able to run them, so Jenkins automation will be much 
> simpler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_Dependency_Check #119

[Apache Jenkins Server]

See 

--
[...truncated 3.90 KB...]

# unset PYTHONHOME if set
if ! [ -z "${PYTHONHOME+_}" ] ; then
_OLD_VIRTUAL_PYTHONHOME="$PYTHONHOME"
unset PYTHONHOME
fi

if [ -z "${VIRTUAL_ENV_DISABLE_PROMPT-}" ] ; then
_OLD_VIRTUAL_PS1="$PS1"
if [ "x" != x ] ; then
PS1="$PS1"
else
PS1="(`basename \"$VIRTUAL_ENV\"`) $PS1"
fi
export PS1
fi
basename "$VIRTUAL_ENV"

# Make sure to unalias pydoc if it's already there
alias pydoc 2>/dev/null >/dev/null && unalias pydoc

pydoc () {
python -m pydoc "$@"
}

# This should detect bash and zsh, which have a hash command that must
# be called to get it to forget past commands.  Without forgetting
# past commands the $PATH changes we made may not be respected
if [ -n "${BASH-}" ] || [ -n "${ZSH_VERSION-}" ] ; then
hash -r 2>/dev/null
fi

mkdir -p $WORKSPACE/src/build/dependencyUpdates
rm -f $WORKSPACE/src/build/dependencyUpdates/test_java_report.txt

echo "The following dependencies have later release versions:" > 
$WORKSPACE/src/build/dependencyUpdates/test_java_report.txt
echo ' - groupA:artifactA [1.0 -> 4.0]' > 
$WORKSPACE/src/build/dependencyUpdates/test_java_report.txt
echo ' - groupB:artifactB [1.0 -> 1.1]' > 
$WORKSPACE/src/build/dependencyUpdates/test_java_report.txt
cat $WORKSPACE/src/build/dependencyUpdates/test_java_report.txt
 - groupB:artifactB [1.0 -> 1.1]
+ bash .test-infra/jenkins/dependency_check/generate_report.sh

REPORT_DESCRIPTION="
 A dependency update is high priority if it satisfies one of following 
criteria: 

 It has major versions update available, e.g. org.assertj:assertj-core 
2.5.0 -> 3.10.0; 


 It is over 3 minor versions behind the latest version, e.g. org.tukaani:xz 
1.5 -> 1.8; 


 The current version is behind the later version for over 180 days, e.g. 
com.google.auto.service:auto-service 2014-10-24 -> 2017-12-11. 

 In Beam, we make a best-effort attempt at keeping all dependencies 
up-to-date.
 In the future, issues will be filed and tracked for these automatically,
 but in the meantime you can search for existing issues or open a new one.

 For more information: https://beam.apache.org/contribute/dependencies/\";> Beam Dependency 
Guide "


# Virtualenv for the rest of the script to run setup
/usr/bin/virtualenv dependency/check
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
. dependency/check/bin/activate
# This file must be used with "source bin/activate" *from bash*
# you cannot run it directly

deactivate () {
unset -f pydoc >/dev/null 2>&1

# reset old environment variables
# ! [ -z ${VAR+_} ] returns true if VAR is declared at all
if ! [ -z "${_OLD_VIRTUAL_PATH+_}" ] ; then
PATH="$_OLD_VIRTUAL_PATH"
export PATH
unset _OLD_VIRTUAL_PATH
fi
if ! [ -z "${_OLD_VIRTUAL_PYTHONHOME+_}" ] ; then
PYTHONHOME="$_OLD_VIRTUAL_PYTHONHOME"
export PYTHONHOME
unset _OLD_VIRTUAL_PYTHONHOME
fi

# This should detect bash and zsh, which have a hash command that must
# be called to get it to forget past commands.  Without forgetting
# past commands the $PATH changes we made may not be respected
if [ -n "${BASH-}" ] || [ -n "${ZSH_VERSION-}" ] ; then
hash -r 2>/dev/null
fi

if ! [ -z "${_OLD_VIRTUAL_PS1+_}" ] ; then
PS1="$_OLD_VIRTUAL_PS1"
export PS1
unset _OLD_VIRTUAL_PS1
fi

unset VIRTUAL_ENV
if [ ! "${1-}" = "nondestructive" ] ; then
# Self destruct!
unset -f deactivate
fi
}

# unset irrelevant variables
deactivate nondestructive

VIRTUAL_ENV="
export VIRTUAL_ENV

_OLD_VIRTUAL_PATH="$PATH"
PATH="$VIRTUAL_ENV/bin:$PATH"
export PATH

# unset PYTHONHOME if set
if ! [ -z "${PYTHONHOME+_}" ] ; then
_OLD_VIRTUAL_PYTHONHOME="$PYTHONHOME"
unset PYTHONHOME
fi

if [ -z "${VIRTUAL_ENV_DISABLE_PROMPT-}" ] ; then
_OLD_VIRTUAL_PS1="$PS1"
if [ "x" != x ] ; then
PS1="$PS1"
else
PS1="(`basename \"$VIRTUAL_ENV\"`) $PS1"
fi
export PS1
fi
basename "$VIRTUAL_ENV"

# Make sure to unalias pydoc if it's already there
alias pydoc 2>/dev/null >/dev/null && unalias pydoc

pydoc () {
python -m pydoc "$@"
}

# This should detect bash and zsh, which have a hash command that must
# be called to get it to forget past commands.  Without forgetting
# past commands the $PATH changes we made may not be respected
if [ -n "${BASH-}" ] || [ -n "${ZSH_VERSION-}" ] ; then
hash -r 2>/dev/null
fi
pip install --upgrade google-cloud-bigquery
Collecting

[jira] [Work logged] (BEAM-4859) Run Python ValidatesRunner tests in streaming

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4859?focusedWorklogId=127610&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127610
 ]

ASF GitHub Bot logged work on BEAM-4859:


Author: ASF GitHub Bot
Created on: 26/Jul/18 00:44
Start Date: 26/Jul/18 00:44
Worklog Time Spent: 10m 
  Work Description: aaltay commented on issue #6053: [BEAM-4859] Enable 
Python VR tests in streaming in postcommit task
URL: https://github.com/apache/beam/pull/6053#issuecomment-407940101
 
 
   Should we re-run the tests or merge as it is?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127610)
Time Spent: 3h 50m  (was: 3h 40m)

> Run Python ValidatesRunner tests in streaming
> -
>
> Key: BEAM-4859
> URL: https://issues.apache.org/jira/browse/BEAM-4859
> Project: Beam
>  Issue Type: Test
>  Components: testing
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Major
>  Time Spent: 3h 50m
>  Remaining Estimate: 0h
>
> Automated way to run existing Python ValidatesRunner tests in streaming. 
> Gradle should also be able to run them, so Jenkins automation will be much 
> simpler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4859) Run Python ValidatesRunner tests in streaming

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4859?focusedWorklogId=127608&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127608
 ]

ASF GitHub Bot logged work on BEAM-4859:


Author: ASF GitHub Bot
Created on: 26/Jul/18 00:40
Start Date: 26/Jul/18 00:40
Worklog Time Spent: 10m 
  Work Description: markflyhigh commented on issue #6053: [BEAM-4859] 
Enable Python VR tests in streaming in postcommit task
URL: https://github.com/apache/beam/pull/6053#issuecomment-407939601
 
 
   30 of 31 tests passed. The failure one is actually failed but since it's a 
streaming job, the pipeline didn't stop until test hit timeout. Created 
BEAM-5025 to tracking it and will disable in streaming VR suite temporarily.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127608)
Time Spent: 3.5h  (was: 3h 20m)

> Run Python ValidatesRunner tests in streaming
> -
>
> Key: BEAM-4859
> URL: https://issues.apache.org/jira/browse/BEAM-4859
> Project: Beam
>  Issue Type: Test
>  Components: testing
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Major
>  Time Spent: 3.5h
>  Remaining Estimate: 0h
>
> Automated way to run existing Python ValidatesRunner tests in streaming. 
> Gradle should also be able to run them, so Jenkins automation will be much 
> simpler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4859) Run Python ValidatesRunner tests in streaming

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4859?focusedWorklogId=127609&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127609
 ]

ASF GitHub Bot logged work on BEAM-4859:


Author: ASF GitHub Bot
Created on: 26/Jul/18 00:40
Start Date: 26/Jul/18 00:40
Worklog Time Spent: 10m 
  Work Description: markflyhigh edited a comment on issue #6053: 
[BEAM-4859] Enable Python VR tests in streaming in postcommit task
URL: https://github.com/apache/beam/pull/6053#issuecomment-407939601
 
 
   30 of 31 tests passed. The failure one is actually failed but since it's a 
streaming job, the pipeline didn't stop until test hit timeout. Created 
[BEAM-5025](https://issues.apache.org/jira/browse/BEAM-5025) to tracking it and 
will disable in streaming VR suite temporarily.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127609)
Time Spent: 3h 40m  (was: 3.5h)

> Run Python ValidatesRunner tests in streaming
> -
>
> Key: BEAM-4859
> URL: https://issues.apache.org/jira/browse/BEAM-4859
> Project: Beam
>  Issue Type: Test
>  Components: testing
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Major
>  Time Spent: 3h 40m
>  Remaining Estimate: 0h
>
> Automated way to run existing Python ValidatesRunner tests in streaming. 
> Gradle should also be able to run them, so Jenkins automation will be much 
> simpler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5025) Fix test_multi_valued_singleton_side_input on DataflowRunner in streaming

[Mark Liu (JIRA)]

Mark Liu created BEAM-5025:
--

 Summary: Fix test_multi_valued_singleton_side_input on 
DataflowRunner in streaming
 Key: BEAM-5025
 URL: https://issues.apache.org/jira/browse/BEAM-5025
 Project: Beam
  Issue Type: Bug
  Components: sdk-py-core, testing
Reporter: Mark Liu
Assignee: Ahmet Altay


test_multi_valued_singleton_side_input is a VR test and failed to run on 
DataflowRunner in streaming. 

Example failure link from pantheon:
https://pantheon.corp.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-07-25_16_16_40-5301315089771762711?project=apache-beam-testing

log:
  File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow_PR/src/sdks/python/apache_beam/pvalue.py",
 line 422, in _from_runtime_iterable
ValueError: PCollection of size 2 with more than one element accessed as a 
singleton view. First two elements encountered are "3", "4".

We should reenable this test in beam_PostCommit_Py_VR_Dataflow after fix it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-4637) Flaky post-commit test org.apache.beam.examples.subprocess.ExampleEchoPipelineTest.testExampleEchoPipeline

[Batkhuyag Batsaikhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Batkhuyag Batsaikhan resolved BEAM-4637.

   Resolution: Cannot Reproduce
Fix Version/s: 2.6.0

> Flaky post-commit test 
> org.apache.beam.examples.subprocess.ExampleEchoPipelineTest.testExampleEchoPipeline
> --
>
> Key: BEAM-4637
> URL: https://issues.apache.org/jira/browse/BEAM-4637
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-java
>Reporter: Mikhail Gryzykhin
>Assignee: Batkhuyag Batsaikhan
>Priority: Critical
>  Labels: flake
> Fix For: 2.6.0
>
>
> Post commit test failed with "Text file busy" exception.
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/916/testReport/junit/org.apache.beam.examples.subprocess/ExampleEchoPipelineTest/testExampleEchoPipeline/]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-4637) Flaky post-commit test org.apache.beam.examples.subprocess.ExampleEchoPipelineTest.testExampleEchoPipeline

[Batkhuyag Batsaikhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Batkhuyag Batsaikhan reassigned BEAM-4637:
--

Assignee: Batkhuyag Batsaikhan  (was: Reuven Lax)

> Flaky post-commit test 
> org.apache.beam.examples.subprocess.ExampleEchoPipelineTest.testExampleEchoPipeline
> --
>
> Key: BEAM-4637
> URL: https://issues.apache.org/jira/browse/BEAM-4637
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-java
>Reporter: Mikhail Gryzykhin
>Assignee: Batkhuyag Batsaikhan
>Priority: Critical
>  Labels: flake
> Fix For: 2.6.0
>
>
> Post commit test failed with "Text file busy" exception.
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/916/testReport/junit/org.apache.beam.examples.subprocess/ExampleEchoPipelineTest/testExampleEchoPipeline/]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-4637) Flaky post-commit test org.apache.beam.examples.subprocess.ExampleEchoPipelineTest.testExampleEchoPipeline

[Batkhuyag Batsaikhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Batkhuyag Batsaikhan reassigned BEAM-4637:
--

Assignee: Reuven Lax  (was: Batkhuyag Batsaikhan)

> Flaky post-commit test 
> org.apache.beam.examples.subprocess.ExampleEchoPipelineTest.testExampleEchoPipeline
> --
>
> Key: BEAM-4637
> URL: https://issues.apache.org/jira/browse/BEAM-4637
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-java
>Reporter: Mikhail Gryzykhin
>Assignee: Reuven Lax
>Priority: Critical
>  Labels: flake
> Fix For: 2.6.0
>
>
> Post commit test failed with "Text file busy" exception.
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/916/testReport/junit/org.apache.beam.examples.subprocess/ExampleEchoPipelineTest/testExampleEchoPipeline/]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4637) Flaky post-commit test org.apache.beam.examples.subprocess.ExampleEchoPipelineTest.testExampleEchoPipeline

[Batkhuyag Batsaikhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16556454#comment-16556454
 ] 

Batkhuyag Batsaikhan commented on BEAM-4637:


I have ran this test locally 1000 times and it succeeded. It seems like this 
test is failing on Jenkins rarely, from Jenkins history, I see only one 
failure. I don't think we need to investigate it for now, unless this test 
fails again,

> Flaky post-commit test 
> org.apache.beam.examples.subprocess.ExampleEchoPipelineTest.testExampleEchoPipeline
> --
>
> Key: BEAM-4637
> URL: https://issues.apache.org/jira/browse/BEAM-4637
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-java
>Reporter: Mikhail Gryzykhin
>Assignee: Batkhuyag Batsaikhan
>Priority: Critical
>  Labels: flake
>
> Post commit test failed with "Text file busy" exception.
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/916/testReport/junit/org.apache.beam.examples.subprocess/ExampleEchoPipelineTest/testExampleEchoPipeline/]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-1124) Python ValidateRunner Test test_multi_valued_singleton_side_input Break Postcommit

[Mark Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16556453#comment-16556453
 ] 

Mark Liu commented on BEAM-1124:


The fix should work. This test is passing for a while.

Latest green Jenkins run for VR test:
https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Py_VR_Dataflow/642/

> Python ValidateRunner Test test_multi_valued_singleton_side_input Break 
> Postcommit
> --
>
> Key: BEAM-1124
> URL: https://issues.apache.org/jira/browse/BEAM-1124
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow, sdk-py-core, testing
>Reporter: Mark Liu
>Assignee: Ahmet Altay
>Priority: Major
> Fix For: Not applicable
>
>
> Python test_multi_valued_singleton_side_input test, a ValidatesRunner test 
> that running on dataflow service, failed and broke 
> postcommit(https://builds.apache.org/view/Beam/job/beam_PostCommit_Python_Verify/853/).
> Here is the stack trace:
> {code}
> Traceback (most recent call last):
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/dataflow_test.py",
>  line 186, in test_multi_valued_singleton_side_input
> pipeline.run()
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/pipeline.py",
>  line 159, in run
> return self.runner.run(self)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/runners/dataflow_runner.py",
>  line 195, in run
> % getattr(self, 'last_error_msg', None), self.result)
> DataflowRuntimeException: Dataflow pipeline failed:
> (99aeafa7a8dffcc7): Traceback (most recent call last):
>   File 
> "/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py", line 
> 514, in do_work
> work_executor.execute()
>   File "dataflow_worker/executor.py", line 892, in 
> dataflow_worker.executor.MapTaskExecutor.execute 
> (dataflow_worker/executor.c:24008)
> op.start()
>   File "dataflow_worker/executor.py", line 456, in 
> dataflow_worker.executor.DoOperation.start (dataflow_worker/executor.c:13870)
> def start(self):
>   File "dataflow_worker/executor.py", line 483, in 
> dataflow_worker.executor.DoOperation.start (dataflow_worker/executor.c:13685)
> self.dofn_runner = common.DoFnRunner(
>   File "apache_beam/runners/common.py", line 89, in 
> apache_beam.runners.common.DoFnRunner.__init__ 
> (apache_beam/runners/common.c:3469)
> args, kwargs, [side_input[global_window]
>   File 
> "/usr/local/lib/python2.7/dist-packages/apache_beam/transforms/sideinputs.py",
>  line 192, in __getitem__
> _FilteringIterable(self._iterable, target_window), self._view_options)
>   File "/usr/local/lib/python2.7/dist-packages/apache_beam/pvalue.py", line 
> 279, in _from_runtime_iterable
> 'PCollection with more than one element accessed as '
> ValueError: PCollection with more than one element accessed as a singleton 
> view.
> {code}
> Worker logs in here:
> https://builds.apache.org/view/Beam/job/beam_PostCommit_Python_Verify/853/console
> In order to temporarily ignore this test in postcommit, we can comment out 
> annotation "@attr('ValidatesRunner')" of this test. Then it will only run as 
> a unit test (execute by DirectRunner), but not run as a ValidatesRunner test.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_JDBC #890

[Apache Jenkins Server]

See 


Changes:

[daniel.o.programmer] [BEAM-3709] Implementing new combine URNs in python.

[kedin] [SQL] Enable running BeamSqlLine from gradle

[pablo] Removing scoped metrics container

[pablo] Remove old style metrics context management

[garrettjonesgoogle] Bumping versions that were missed in #5988

[altay] Remove reference to dataflow-distribution.properties

[lcwik] [BEAM-4176] Initial implementation for running portable runner tests

[pablo] Fix Java Nightly Snapshot Failures

--
[...truncated 419.38 KB...]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.postgresql.core.PGStream.(PGStream.java:69)
at 
org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:156)
at 
org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
at org.postgresql.jdbc.PgConnection.(PgConnection.java:195)
at org.postgresql.Driver.makeConnection(Driver.java:452)
at org.postgresql.Driver.connect(Driver.java:254)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at 
org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:94)
at 
org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:79)
at 
org.apache.commons.dbcp2.DataSourceConnectionFactory.createConnection(DataSourceConnectionFactory.java:44)
at 
org.apache.commons.dbcp2.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:256)
at 
org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:868)
at 
org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
at 
org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
at 
org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:134)
at org.apache.beam.sdk.io.jdbc.JdbcIO$ReadFn.setup(JdbcIO.java:641)
at 
org.apache.beam.sdk.io.jdbc.JdbcIO$ReadFn$DoFnInvoker.invokeSetup(Unknown 
Source)
at 
com.google.cloud.dataflow.worker.DoFnInstanceManagers$ConcurrentQueueInstanceManager.deserializeCopy(DoFnInstanceManagers.java:63)
at 
com.google.cloud.dataflow.worker.DoFnInstanceManagers$ConcurrentQueueInstanceManager.peek(DoFnInstanceManagers.java:45)
at 
com.google.cloud.dataflow.worker.UserParDoFnFactory.create(UserParDoFnFactory.java:94)
at 
com.google.cloud.dataflow.worker.DefaultParDoFnFactory.create(DefaultParDoFnFactory.java:74)
at 
com.google.cloud.dataflow.worker.IntrinsicMapTaskExecutorFactory.createParDoOperation(IntrinsicMapTaskExecutorFactory.java:262)
at 
com.google.cloud.dataflow.worker.IntrinsicMapTaskExecutorFactory.access$000(IntrinsicMapTaskExecutorFactory.java:84)
at 
com.google.cloud.dataflow.worker.IntrinsicMapTaskExecutorFactory$1.typedApply(IntrinsicMapTaskExecutorFactory.java:181)
at 
com.google.cloud.dataflow.worker.IntrinsicMapTaskExecutorFactory$1.typedApply(IntrinsicMapTaskExecutorFactory.java:163)
at 
com.google.cloud.dataflow.worker.graph.Networks$TypeSafeNodeFunction.apply(Networks.java:63)
at 
com.google.cloud.dataflow.worker.graph.Networks$TypeSafeNodeFunction.apply(Networks.java:50)
at 
com.google.cloud.dataflow.worker.graph.Networks.replaceDirectedNetworkNodes(Networks.java:87)
at 
com.google.cloud.dataflow.worker.IntrinsicMapTaskExecutorFactory.create(IntrinsicMapTaskExecutorFactory.java:123)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:336)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:290)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:134)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:114)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:101)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Workflow failed. Causes: 
S06:JdbcIO.Read/Create.Values/Read(CreateSource)+JdbcIO.Read/JdbcIO.ReadAll/ParDo(Read)+JdbcIO.Read/JdbcIO.ReadAll/JdbcIO.Reparallelize/Consume/ParDo(Anonymous)+JdbcIO.Read/JdbcIO.ReadAll/JdbcIO.Reparallelize/View.AsIterable/ParDo(ToIsmRecordForGlobalWindow)
 failed., A 

[jira] [Work logged] (BEAM-4864) run_validatescontainer.sh in Python sdk has hard-coded bucket of sadness

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4864?focusedWorklogId=127607&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127607
 ]

ASF GitHub Bot logged work on BEAM-4864:


Author: ASF GitHub Bot
Created on: 26/Jul/18 00:19
Start Date: 26/Jul/18 00:19
Worklog Time Spent: 10m 
  Work Description: holdenk commented on issue #6061: [BEAM-4864][WIP] Make 
run_validatescontainer.sh not quite as hard coded.
URL: https://github.com/apache/beam/pull/6061#issuecomment-407936532
 
 
   Jenkins retest this please.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127607)
Time Spent: 20m  (was: 10m)

> run_validatescontainer.sh in Python sdk has hard-coded bucket of sadness
> 
>
> Key: BEAM-4864
> URL: https://issues.apache.org/jira/browse/BEAM-4864
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-harness
>Reporter: holdenk
>Assignee: holdenk
>Priority: Trivial
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> The run_validatescontainer.sh script looks amazing! However I could not 
> validate my container, and this made me sad. We can make it configurable and 
> then people can validate their container changes more easily :)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4833) Add support for users specifying a requirements.txt for their Python portable container

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4833?focusedWorklogId=127605&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127605
 ]

ASF GitHub Bot logged work on BEAM-4833:


Author: ASF GitHub Bot
Created on: 26/Jul/18 00:18
Start Date: 26/Jul/18 00:18
Worklog Time Spent: 10m 
  Work Description: holdenk commented on issue #6005: [BEAM-4833] Add 
support for user req.txt for portable python
URL: https://github.com/apache/beam/pull/6005#issuecomment-407936350
 
 
   re-ping, sorry for the WIP title, was for the first draft (now that it's 
refactored into 2 files ready for review).


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127605)
Time Spent: 1h  (was: 50m)

> Add support for users specifying a requirements.txt for their Python portable 
> container
> ---
>
> Key: BEAM-4833
> URL: https://issues.apache.org/jira/browse/BEAM-4833
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: holdenk
>Assignee: holdenk
>Priority: Minor
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> It's pretty common that Python scripts require extra dependencies, even the 
> tensorflow model analysis TFMA example requires a different version of TF 
> than the one we install by default. While users can roll their own container 
> or edit the Dockerfile, it would probably be useful to provide an easier path 
> to integrating their dependencies.
> While we support automatically installing the dependencies at runtime on the 
> workers, this can be very slow, especially for things like tensorflow, arrow, 
> or other numeric heavy code.
> Another alternative could be a simple script to augment the existing base 
> image.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PreCommit_Python_Cron #150

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-4751) Finish io futurize stage 2: fix the missing pylint3 check in tox.ini

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4751?focusedWorklogId=127598&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127598
 ]

ASF GitHub Bot logged work on BEAM-4751:


Author: ASF GitHub Bot
Created on: 26/Jul/18 00:13
Start Date: 26/Jul/18 00:13
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on a change in pull request #5916: 
[BEAM-4751] fix missing pylint3 check for io subpackage
URL: https://github.com/apache/beam/pull/5916#discussion_r205298057
 
 

 ##
 File path: sdks/python/apache_beam/io/concat_source_test.py
 ##
 @@ -76,7 +76,12 @@ def read(self, range_tracker):
   # For testing
   def __eq__(self, other):
 return (type(self) == type(other)
-and self._start == other._start and self._end == other._end)
+and self._start == other._start
+and self._end == other._end
+and self._split_freq == other._split_freq)
+
+  def __hash__(self):
 
 Review comment:
   I think this will be dead code. I suggest:
   ```
   # To maintain compatibility with Python 3
   __hash__ = None
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127598)
Time Spent: 1h 40m  (was: 1.5h)

> Finish io futurize stage 2: fix the missing pylint3 check in tox.ini
> 
>
> Key: BEAM-4751
> URL: https://issues.apache.org/jira/browse/BEAM-4751
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Matthias Feys
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> Finish io futurize stage 2: fix the missing pylint3 check in tox.ini



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4751) Finish io futurize stage 2: fix the missing pylint3 check in tox.ini

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4751?focusedWorklogId=127595&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127595
 ]

ASF GitHub Bot logged work on BEAM-4751:


Author: ASF GitHub Bot
Created on: 26/Jul/18 00:13
Start Date: 26/Jul/18 00:13
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on a change in pull request #5916: 
[BEAM-4751] fix missing pylint3 check for io subpackage
URL: https://github.com/apache/beam/pull/5916#discussion_r205289181
 
 

 ##
 File path: sdks/python/apache_beam/io/hadoopfilesystem_test.py
 ##
 @@ -49,6 +49,9 @@ def __init__(self, path, mode='', type='FILE'):
   def __eq__(self, other):
 return self.stat == other.stat and self.getvalue() == self.getvalue()
 
+  def __hash__(self):
 
 Review comment:
   I think this will be dead code. I suggest:
   ```
   # To maintain compatibility with Python 3
   __hash__ = None
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127595)
Time Spent: 1.5h  (was: 1h 20m)

> Finish io futurize stage 2: fix the missing pylint3 check in tox.ini
> 
>
> Key: BEAM-4751
> URL: https://issues.apache.org/jira/browse/BEAM-4751
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Matthias Feys
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Finish io futurize stage 2: fix the missing pylint3 check in tox.ini



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4751) Finish io futurize stage 2: fix the missing pylint3 check in tox.ini

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4751?focusedWorklogId=127599&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127599
 ]

ASF GitHub Bot logged work on BEAM-4751:


Author: ASF GitHub Bot
Created on: 26/Jul/18 00:13
Start Date: 26/Jul/18 00:13
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on a change in pull request #5916: 
[BEAM-4751] fix missing pylint3 check for io subpackage
URL: https://github.com/apache/beam/pull/5916#discussion_r205293839
 
 

 ##
 File path: sdks/python/apache_beam/io/vcfio.py
 ##
 @@ -215,6 +219,10 @@ def __eq__(self, other):
 return ((self.name, self.genotype, self.phaseset, self.info) ==
 (other.name, other.genotype, other.phaseset, other.info))
 
+  def __hash__(self):
+return hash((self.name, self.genotype,
 
 Review comment:
   `self.genotype` is unhashable object. I think this would be dead code, so I 
suggest instead:
   ```
   # To maintain compatibility with Python 3
   __hash__ = None
   ```
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127599)
Time Spent: 1h 50m  (was: 1h 40m)

> Finish io futurize stage 2: fix the missing pylint3 check in tox.ini
> 
>
> Key: BEAM-4751
> URL: https://issues.apache.org/jira/browse/BEAM-4751
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Matthias Feys
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> Finish io futurize stage 2: fix the missing pylint3 check in tox.ini



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4751) Finish io futurize stage 2: fix the missing pylint3 check in tox.ini

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4751?focusedWorklogId=127596&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127596
 ]

ASF GitHub Bot logged work on BEAM-4751:


Author: ASF GitHub Bot
Created on: 26/Jul/18 00:13
Start Date: 26/Jul/18 00:13
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on a change in pull request #5916: 
[BEAM-4751] fix missing pylint3 check for io subpackage
URL: https://github.com/apache/beam/pull/5916#discussion_r205290053
 
 

 ##
 File path: sdks/python/apache_beam/io/vcfio.py
 ##
 @@ -123,6 +124,9 @@ def __eq__(self, other):
 return (isinstance(other, Variant) and
 vars(self) == vars(other))
 
+  def __hash__(self):
+return hash((type(self), vars(self)))
 
 Review comment:
   `vars(self)` is  unhashable since it's a `dict`. 
   I think this would be dead code, so I suggest instead:
   ```
   # To maintain compatibility with Python 3
   __hash__ = None
   ```
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127596)

> Finish io futurize stage 2: fix the missing pylint3 check in tox.ini
> 
>
> Key: BEAM-4751
> URL: https://issues.apache.org/jira/browse/BEAM-4751
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Matthias Feys
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Finish io futurize stage 2: fix the missing pylint3 check in tox.ini



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4751) Finish io futurize stage 2: fix the missing pylint3 check in tox.ini

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4751?focusedWorklogId=127597&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127597
 ]

ASF GitHub Bot logged work on BEAM-4751:


Author: ASF GitHub Bot
Created on: 26/Jul/18 00:13
Start Date: 26/Jul/18 00:13
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on a change in pull request #5916: 
[BEAM-4751] fix missing pylint3 check for io subpackage
URL: https://github.com/apache/beam/pull/5916#discussion_r205289077
 
 

 ##
 File path: sdks/python/apache_beam/io/gcp/pubsub.py
 ##
 @@ -77,6 +77,9 @@ def __eq__(self, other):
 self.payload == other.payload and
 self.attributes == other.attributes)
 
+  def __hash__(self):
+return hash((type(self), self.payload, self.attributes))
 
 Review comment:
   Let's simplify this to `hash(self.payload)`.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127597)
Time Spent: 1h 40m  (was: 1.5h)

> Finish io futurize stage 2: fix the missing pylint3 check in tox.ini
> 
>
> Key: BEAM-4751
> URL: https://issues.apache.org/jira/browse/BEAM-4751
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Matthias Feys
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> Finish io futurize stage 2: fix the missing pylint3 check in tox.ini



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4751) Finish io futurize stage 2: fix the missing pylint3 check in tox.ini

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4751?focusedWorklogId=127594&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127594
 ]

ASF GitHub Bot logged work on BEAM-4751:


Author: ASF GitHub Bot
Created on: 26/Jul/18 00:13
Start Date: 26/Jul/18 00:13
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on a change in pull request #5916: 
[BEAM-4751] fix missing pylint3 check for io subpackage
URL: https://github.com/apache/beam/pull/5916#discussion_r205286297
 
 

 ##
 File path: sdks/python/apache_beam/io/filebasedsink.py
 ##
 @@ -373,6 +373,9 @@ def __eq__(self, other):
 # pylint: disable=unidiomatic-typecheck
 return type(self) == type(other) and self.__dict__ == other.__dict__
 
+  def __hash__(self):
+return hash((type(self), frozenset(self.__dict__.items(
 
 Review comment:
   I think this will be dead code. I suggest:
   ```
   # To maintain compatibility with Python 3
   __hash__ = None
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127594)
Time Spent: 1.5h  (was: 1h 20m)

> Finish io futurize stage 2: fix the missing pylint3 check in tox.ini
> 
>
> Key: BEAM-4751
> URL: https://issues.apache.org/jira/browse/BEAM-4751
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Matthias Feys
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Finish io futurize stage 2: fix the missing pylint3 check in tox.ini



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #1102

[Apache Jenkins Server]

See 


Changes:

[daniel.o.programmer] [BEAM-3709] Implementing new combine URNs in python.

[kedin] [SQL] Enable running BeamSqlLine from gradle

[lcwik] [BEAM-4866] Fix missing licenses.

[pablo] Removing scoped metrics container

[pablo] Remove old style metrics context management

[garrettjonesgoogle] Bumping versions that were missed in #5988

[lcwik] [BEAM-4176] Initial implementation for running portable runner tests

[pablo] Fix Java Nightly Snapshot Failures

--
[...truncated 12.34 MB...]
Jul 26, 2018 12:10:43 AM org.apache.beam.sdk.io.gcp.bigquery.BatchLoads$4 
getTempFilePrefix
INFO: Writing BigQuery temporary files to 
gs://temp-storage-for-end-to-end-tests/BigQueryWriteTemp/beam_load_bigqueryreadwriteit0testsqltypesjenkins072600104229ade608_73a584f9275d4859ad6806a95b4b2de2/
 before loading them.
Jul 26, 2018 12:10:43 AM org.apache.beam.sdk.io.gcp.bigquery.TableRowWriter 

INFO: Opening TableRowWriter to 
gs://temp-storage-for-end-to-end-tests/BigQueryWriteTemp/beam_load_bigqueryreadwriteit0testsqltypesjenkins072600104229ade608_73a584f9275d4859ad6806a95b4b2de2/eddb4a55-bf2e-45f6-857d-f6f35b12a842.
Jul 26, 2018 12:10:43 AM org.apache.beam.sdk.io.gcp.bigquery.WriteTables 
load
INFO: Loading 1 files into {datasetId=integ_test, 
projectId=apache-beam-testing, 
tableId=BigQueryReadWriteIT_testSQLTypes_2018_07_26_00_10_38_290_7331836372753424765}
 using job 
{jobId=beam_load_bigqueryreadwriteit0testsqltypesjenkins072600104229ade608_73a584f9275d4859ad6806a95b4b2de2_f0dfc4a06d17a7a0e4f125e401a1f502_1_0-0,
 location=US, projectId=apache-beam-testing}, attempt 0

org.apache.beam.sdk.extensions.sql.PubsubToBigqueryIT > testSimpleInsert 
STANDARD_ERROR
Jul 26, 2018 12:10:44 AM 
org.apache.beam.sdk.io.gcp.pubsub.PubsubUnboundedSource createRandomSubscription
WARNING: Created subscription 
projects/apache-beam-testing/subscriptions/integ-test-PubsubToBigqueryIT-testSimpleInsert-2018-07-26-00-10-38-841-events-895187437625804595_beam_-2462694031356663951
 to topic 
projects/apache-beam-testing/topics/integ-test-PubsubToBigqueryIT-testSimpleInsert-2018-07-26-00-10-38-841-events-895187437625804595.
 Note this subscription WILL NOT be deleted when the pipeline terminates

org.apache.beam.sdk.extensions.sql.meta.provider.bigquery.BigQueryReadWriteIT > 
testSQLTypes STANDARD_ERROR
Jul 26, 2018 12:10:44 AM 
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl startJob
INFO: Started BigQuery job: 
{jobId=beam_load_bigqueryreadwriteit0testsqltypesjenkins072600104229ade608_73a584f9275d4859ad6806a95b4b2de2_f0dfc4a06d17a7a0e4f125e401a1f502_1_0-0,
 location=US, projectId=apache-beam-testing}.
bq show -j --format=prettyjson --project_id=apache-beam-testing 
beam_load_bigqueryreadwriteit0testsqltypesjenkins072600104229ade608_73a584f9275d4859ad6806a95b4b2de2_f0dfc4a06d17a7a0e4f125e401a1f502_1_0-0
Jul 26, 2018 12:10:44 AM org.apache.beam.sdk.io.gcp.bigquery.WriteTables 
load
INFO: Load job 
{jobId=beam_load_bigqueryreadwriteit0testsqltypesjenkins072600104229ade608_73a584f9275d4859ad6806a95b4b2de2_f0dfc4a06d17a7a0e4f125e401a1f502_1_0-0,
 location=US, projectId=apache-beam-testing} started
Jul 26, 2018 12:10:44 AM 
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl pollJob
INFO: Still waiting for BigQuery job 
beam_load_bigqueryreadwriteit0testsqltypesjenkins072600104229ade608_73a584f9275d4859ad6806a95b4b2de2_f0dfc4a06d17a7a0e4f125e401a1f502_1_0-0,
 currently in status {"state":"RUNNING"}
bq show -j --format=prettyjson --project_id=apache-beam-testing 
beam_load_bigqueryreadwriteit0testsqltypesjenkins072600104229ade608_73a584f9275d4859ad6806a95b4b2de2_f0dfc4a06d17a7a0e4f125e401a1f502_1_0-0
Jul 26, 2018 12:10:45 AM 
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl pollJob
INFO: BigQuery job 
{jobId=beam_load_bigqueryreadwriteit0testsqltypesjenkins072600104229ade608_73a584f9275d4859ad6806a95b4b2de2_f0dfc4a06d17a7a0e4f125e401a1f502_1_0-0,
 location=US, projectId=apache-beam-testing} completed in state DONE
Jul 26, 2018 12:10:45 AM org.apache.beam.sdk.io.gcp.bigquery.WriteTables 
load
INFO: Load job 
{jobId=beam_load_bigqueryreadwriteit0testsqltypesjenkins072600104229ade608_73a584f9275d4859ad6806a95b4b2de2_f0dfc4a06d17a7a0e4f125e401a1f502_1_0-0,
 location=US, projectId=apache-beam-testing} succeeded. Statistics: 
{"creationTime":"1532563843896","endTime":"1532563845062","load":{"badRecords":"0","inputFileBytes":"243","inputFiles":"1","outputBytes":"82","outputRows":"1"},"startTime":"1532563844243"}

org.apache.beam.sdk.extensions.sql.meta.provider.bigquery.BigQueryReadWriteIT > 
testSQLRead STANDARD_ERROR
Jul 26, 2018 12:10:46 AM 
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$D

[jira] [Work logged] (BEAM-4849) Support running Beam Samza jobs in Yarn

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4849?focusedWorklogId=127593&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127593
 ]

ASF GitHub Bot logged work on BEAM-4849:


Author: ASF GitHub Bot
Created on: 26/Jul/18 00:10
Start Date: 26/Jul/18 00:10
Worklog Time Spent: 10m 
  Work Description: xinyuiscool commented on issue #6029: [BEAM-4849] 
Support running Beam Samza jobs in Yarn
URL: https://github.com/apache/beam/pull/6029#issuecomment-407935248
 
 
   Run Java PreCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127593)
Time Spent: 1h 10m  (was: 1h)

> Support running Beam Samza jobs in Yarn
> ---
>
> Key: BEAM-4849
> URL: https://issues.apache.org/jira/browse/BEAM-4849
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-samza
>Reporter: Xinyu Liu
>Assignee: Xinyu Liu
>Priority: Major
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> We need to some util classes and script to run beam job containers in Yarn. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4849) Support running Beam Samza jobs in Yarn

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4849?focusedWorklogId=127592&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127592
 ]

ASF GitHub Bot logged work on BEAM-4849:


Author: ASF GitHub Bot
Created on: 26/Jul/18 00:10
Start Date: 26/Jul/18 00:10
Worklog Time Spent: 10m 
  Work Description: xinyuiscool commented on issue #6029: [BEAM-4849] 
Support running Beam Samza jobs in Yarn
URL: https://github.com/apache/beam/pull/6029#issuecomment-407935202
 
 
   Pretty weird, my changes are very localized to the Samza Runner, and it 
shouldn't affect examples. Could it be some intermittent failure? Let me try 
running precommit.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127592)
Time Spent: 1h  (was: 50m)

> Support running Beam Samza jobs in Yarn
> ---
>
> Key: BEAM-4849
> URL: https://issues.apache.org/jira/browse/BEAM-4849
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-samza
>Reporter: Xinyu Liu
>Assignee: Xinyu Liu
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> We need to some util classes and script to run beam job containers in Yarn. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PreCommit_Java_Cron #147

[Apache Jenkins Server]

See 


Changes:

[daniel.o.programmer] [BEAM-3709] Implementing new combine URNs in python.

[kedin] [SQL] Enable running BeamSqlLine from gradle

[lcwik] [BEAM-4866] Fix missing licenses.

[pablo] Removing scoped metrics container

[pablo] Remove old style metrics context management

[garrettjonesgoogle] Bumping versions that were missed in #5988

[lcwik] [BEAM-4176] Initial implementation for running portable runner tests

[pablo] Fix Java Nightly Snapshot Failures

--
[...truncated 11.55 MB...]
Jul 26, 2018 12:10:35 AM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: SQLPlan>
LogicalProject(EXPR$0=[CARDINALITY($1)])
  BeamIOSourceRel(table=[[beam, PCOLLECTION]])

Jul 26, 2018 12:10:35 AM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: BEAMPlan>
BeamCalcRel(expr#0..1=[{inputs}], expr#2=[CARDINALITY($t1)], EXPR$0=[$t2])
  BeamIOSourceRel(table=[[beam, PCOLLECTION]])


org.apache.beam.sdk.extensions.sql.BeamSqlDslArrayTest > 
testSelectRowsFromArrayOfRows STANDARD_ERROR
Jul 26, 2018 12:10:35 AM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: SQL:
SELECT `PCOLLECTION`.`f_arrayOfRows`
FROM `beam`.`PCOLLECTION` AS `PCOLLECTION`
Jul 26, 2018 12:10:35 AM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: SQLPlan>
LogicalProject(f_arrayOfRows=[$1])
  BeamIOSourceRel(table=[[beam, PCOLLECTION]])

Jul 26, 2018 12:10:35 AM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: BEAMPlan>
BeamCalcRel(expr#0..1=[{inputs}], f_arrayOfRows=[$t1])
  BeamIOSourceRel(table=[[beam, PCOLLECTION]])


org.apache.beam.sdk.extensions.sql.BeamSqlDslArrayTest > testUnnestLiteral 
STANDARD_ERROR
Jul 26, 2018 12:10:35 AM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: SQL:
SELECT `EXPR$0`.`EXPR$0`
FROM UNNEST(ARRAY['a', 'b', 'c']) AS `EXPR$0`
Jul 26, 2018 12:10:35 AM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: SQLPlan>
LogicalProject(EXPR$0=[$0])
  Uncollect
LogicalProject(EXPR$0=[ARRAY('a', 'b', 'c')])
  LogicalValues(tuples=[[{ 0 }]])

Jul 26, 2018 12:10:35 AM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: BEAMPlan>
BeamCalcRel(expr#0=[{inputs}], EXPR$0=[$t0])
  BeamUncollectRel
BeamCalcRel(expr#0=[{inputs}], expr#1=['a'], expr#2=['b'], 
expr#3=['c'], expr#4=[ARRAY($t1, $t2, $t3)], EXPR$0=[$t4])
  BeamValuesRel(tuples=[[{ 0 }]])


org.apache.beam.sdk.extensions.sql.BeamSqlDslArrayTest > testUnnestNamedLiteral 
STANDARD_ERROR
Jul 26, 2018 12:10:35 AM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: SQL:
SELECT `t`.`f_string`
FROM UNNEST(ARRAY['a', 'b', 'c']) AS `t` (`f_string`)
Jul 26, 2018 12:10:35 AM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: SQLPlan>
LogicalProject(f_string=[$0])
  Uncollect
LogicalProject(EXPR$0=[ARRAY('a', 'b', 'c')])
  LogicalValues(tuples=[[{ 0 }]])

Jul 26, 2018 12:10:35 AM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: BEAMPlan>
BeamCalcRel(expr#0=[{inputs}], f_string=[$t0])
  BeamUncollectRel
BeamCalcRel(expr#0=[{inputs}], expr#1=['a'], expr#2=['b'], 
expr#3=['c'], expr#4=[ARRAY($t1, $t2, $t3)], EXPR$0=[$t4])
  BeamValuesRel(tuples=[[{ 0 }]])


org.apache.beam.sdk.extensions.sql.BeamSqlDslArrayTest > 
testSelectSingleRowFromArrayOfRows STANDARD_ERROR
Jul 26, 2018 12:10:35 AM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: SQL:
SELECT `PCOLLECTION`.`f_arrayOfRows`[1]
FROM `beam`.`PCOLLECTION` AS `PCOLLECTION`
Jul 26, 2018 12:10:35 AM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: SQLPlan>
LogicalProject(EXPR$0$0=[ITEM($1, 1).f_rowString], EXPR$0$1=[ITEM($1, 
1).f_rowInt])
  BeamIOSourceRel(table=[[beam, PCOLLECTION]])

Jul 26, 2018 12:10:35 AM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: BEAMPlan>
BeamCalcRel(expr#0..1=[{inputs}], expr#2=[1], expr#3=[ITEM($t1, $t2)], 
expr#4=[$t3.f_rowString], expr#5=[$t3.f_rowInt], EXPR$0$0=[$t4], EXPR$0$1=[$t5])
  BeamIOSourceRel(table=[[beam, PCOLLECTION]])


org.apache.beam.sdk.extensions.sql.BeamSqlDslArrayTest > testProjectArrayField 
STANDARD_ERROR
Jul 26, 2018 12:10:35 AM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: SQL:
SELECT `PCOLLECTION`.`f_int`, `PCOLLECTION`.`f_stringArr`
FROM `beam`.`PCOLLECTION` AS `PCOLLECTION`
Jul 26, 2018 12:10:35 AM 

[jira] [Work logged] (BEAM-4859) Run Python ValidatesRunner tests in streaming

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4859?focusedWorklogId=127591&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127591
 ]

ASF GitHub Bot logged work on BEAM-4859:


Author: ASF GitHub Bot
Created on: 26/Jul/18 00:08
Start Date: 26/Jul/18 00:08
Worklog Time Spent: 10m 
  Work Description: aaltay commented on issue #6053: [BEAM-4859] Enable 
Python VR tests in streaming in postcommit task
URL: https://github.com/apache/beam/pull/6053#issuecomment-407934848
 
 
   Please ping me after tests pass.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127591)
Time Spent: 3h 20m  (was: 3h 10m)

> Run Python ValidatesRunner tests in streaming
> -
>
> Key: BEAM-4859
> URL: https://issues.apache.org/jira/browse/BEAM-4859
> Project: Beam
>  Issue Type: Test
>  Components: testing
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Major
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
> Automated way to run existing Python ValidatesRunner tests in streaming. 
> Gradle should also be able to run them, so Jenkins automation will be much 
> simpler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Go_GradleBuild #522

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PreCommit_Go_Cron #153

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-5024) Deprecate Dataflow SDK Distribution

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5024?focusedWorklogId=127590&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127590
 ]

ASF GitHub Bot logged work on BEAM-5024:


Author: ASF GitHub Bot
Created on: 26/Jul/18 00:00
Start Date: 26/Jul/18 00:00
Worklog Time Spent: 10m 
  Work Description: aaltay closed pull request #6067: [BEAM-5024] Remove 
reference to dataflow-distribution.properties
URL: https://github.com/apache/beam/pull/6067
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunnerInfo.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunnerInfo.java
index 75b07559554..9a3fa14b2f0 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunnerInfo.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunnerInfo.java
@@ -34,8 +34,6 @@
 
   private static final String APACHE_BEAM_DISTRIBUTION_PROPERTIES_PATH =
   "/org/apache/beam/runners/dataflow/dataflow.properties";
-  private static final String DATAFLOW_DISTRIBUTION_PROPERTIES_PATH =
-  "/org/apache/beam/runners/dataflow/dataflow-distribution.properties";
   private static final String FNAPI_ENVIRONMENT_MAJOR_VERSION_KEY =
   "fnapi.environment.major.version";
   private static final String LEGACY_ENVIRONMENT_MAJOR_VERSION_KEY =
@@ -48,10 +46,7 @@
 static {
   Properties properties;
   try {
-properties = load(DATAFLOW_DISTRIBUTION_PROPERTIES_PATH);
-if (properties == null) {
-  properties = load(APACHE_BEAM_DISTRIBUTION_PROPERTIES_PATH);
-}
+properties = load(APACHE_BEAM_DISTRIBUTION_PROPERTIES_PATH);
 if (properties == null) {
   // Print a warning if we can not load either the Dataflow 
distribution properties
   // or the


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127590)
Time Spent: 40m  (was: 0.5h)

> Deprecate Dataflow SDK Distribution
> ---
>
> Key: BEAM-5024
> URL: https://issues.apache.org/jira/browse/BEAM-5024
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Reporter: Ahmet Altay
>Assignee: Ahmet Altay
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> GCP decided to deprecate Dataflow SDK distribution, in favor of Apache Beam 
> SDK distribution.
> All in Beam!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Merge pull request #6067 from aaltay/removeprops

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit e5e5d54fc4d9f0e65e7f0bf26e6285ce1b4070dd
Merge: 92be85a bc789fb
Author: Ahmet Altay 
AuthorDate: Wed Jul 25 17:00:21 2018 -0700

Merge pull request #6067 from aaltay/removeprops

[BEAM-5024] Remove reference to dataflow-distribution.properties

 .../java/org/apache/beam/runners/dataflow/DataflowRunnerInfo.java  | 7 +--
 1 file changed, 1 insertion(+), 6 deletions(-)

[beam] branch master updated (92be85a -> e5e5d54)

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 92be85a  Merge pull request #6058 from pabloem/remove-smc
 add bc789fb  Remove reference to dataflow-distribution.properties
 new e5e5d54  Merge pull request #6067 from aaltay/removeprops

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../java/org/apache/beam/runners/dataflow/DataflowRunnerInfo.java  | 7 +--
 1 file changed, 1 insertion(+), 6 deletions(-)

Build failed in Jenkins: beam_PostCommit_Py_VR_Dataflow #643

[Apache Jenkins Server]

See 


Changes:

[daniel.o.programmer] [BEAM-3709] Implementing new combine URNs in python.

[kedin] [SQL] Enable running BeamSqlLine from gradle

[lcwik] [BEAM-4866] Fix missing licenses.

[pablo] Removing scoped metrics container

[pablo] Remove old style metrics context management

[garrettjonesgoogle] Bumping versions that were missed in #5988

[lcwik] [BEAM-4176] Initial implementation for running portable runner tests

[pablo] Fix Java Nightly Snapshot Failures

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam15 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 92be85af47bd64fbdb5fbc2a1c2d45fa147dee08 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 92be85af47bd64fbdb5fbc2a1c2d45fa147dee08
Commit message: "Merge pull request #6058 from pabloem/remove-smc"
 > git rev-list --no-walk f2a1fd4307929aa938067058205a5ff79dc0207f # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[Gradle] - Launching build.
[src] $ 
 
--info --continue --max-workers=12 -Dorg.gradle.jvmargs=-Xms2g 
-Dorg.gradle.jvmargs=-Xmx4g :beam-sdks-python:validatesRunnerBatchTests 
:beam-sdks-python:validatesRunnerStreamingTests
Initialized native services in: /home/jenkins/.gradle/native
To honour the JVM settings for this build a new JVM will be forked. Please 
consider using the daemon: 
https://docs.gradle.org/4.8/userguide/gradle_daemon.html.
Starting process 'Gradle build daemon'. Working directory: 
/home/jenkins/.gradle/daemon/4.8 Command: 
/usr/local/asfpackages/java/jdk1.8.0_152/bin/java -Xmx4g -Dfile.encoding=UTF-8 
-Duser.country=US -Duser.language=en -Duser.variant -cp 
/home/jenkins/.gradle/wrapper/dists/gradle-4.8-bin/divx0s2uj4thofgytb7gf9fsi/gradle-4.8/lib/gradle-launcher-4.8.jar
 org.gradle.launcher.daemon.bootstrap.GradleDaemon 4.8
Successfully started process 'Gradle build daemon'
An attempt to start the daemon took 0.968 secs.
The client will now receive all logging from the daemon (pid: 3853). The daemon 
log file: /home/jenkins/.gradle/daemon/4.8/daemon-3853.out.log
Closing daemon's stdin at end of input.
The daemon will no longer process any standard input.
Daemon will be stopped at the end of the build stopping after processing
Using 12 worker leases.
Starting Build
Parallel execution is an incubating feature.

> Configure project :buildSrc
Evaluating project ':buildSrc' using build file 
'
Invalidating in-memory cache of 
/home/jenkins/.gradle/caches/4.8/fileHashes/fileHashes.bin
Invalidating in-memory cache of 
/home/jenkins/.gradle/caches/4.8/fileHashes/resourceHashesCache.bin
file or directory 
'
 not found
Selected primary task 'build' from project :
file or directory 
'
 not found
Using local directory build cache for build ':buildSrc' (location = 
/home/jenkins/.gradle/caches/build-cache-1, removeUnusedEntriesAfter = 7 days).
:compileJava (Thread[Task worker for ':buildSrc' Thread 3,5,main]) started.

> Task :buildSrc:compileJava NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:compileJava' as it has no source files and no previous 
output files.
:compileJava (Thread[Task worker for ':buildSrc' Thread 3,5,main]) completed. 
Took 0.023 secs.
:compileGroovy (Thread[Task worker for ':buildSrc' Thread 3,5,main]) started.

> Task :buildSrc:compileGroovy
Build cache key for task ':buildSrc:compileGroovy' is 
826ebfcf5d4a04956f8f2af76b19af37
Task ':buildSrc:compileGroovy' is not up-to-date because:
  No history is available.
Origin for task 

[jira] [Work logged] (BEAM-4833) Add support for users specifying a requirements.txt for their Python portable container

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4833?focusedWorklogId=127589&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127589
 ]

ASF GitHub Bot logged work on BEAM-4833:


Author: ASF GitHub Bot
Created on: 25/Jul/18 23:59
Start Date: 25/Jul/18 23:59
Worklog Time Spent: 10m 
  Work Description: aaltay commented on issue #6005: [BEAM-4833][WIP] Add 
support for user req.txt for portable python
URL: https://github.com/apache/beam/pull/6005#issuecomment-407933424
 
 
   I am skipping the review for now since it is tagged WIP. Please ping when it 
is ready.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127589)
Time Spent: 50m  (was: 40m)

> Add support for users specifying a requirements.txt for their Python portable 
> container
> ---
>
> Key: BEAM-4833
> URL: https://issues.apache.org/jira/browse/BEAM-4833
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: holdenk
>Assignee: holdenk
>Priority: Minor
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> It's pretty common that Python scripts require extra dependencies, even the 
> tensorflow model analysis TFMA example requires a different version of TF 
> than the one we install by default. While users can roll their own container 
> or edit the Dockerfile, it would probably be useful to provide an easier path 
> to integrating their dependencies.
> While we support automatically installing the dependencies at runtime on the 
> workers, this can be very slow, especially for things like tensorflow, arrow, 
> or other numeric heavy code.
> Another alternative could be a simple script to augment the existing base 
> image.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4094) Remove ScopedMetricsContainer from Python SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4094?focusedWorklogId=127583&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127583
 ]

ASF GitHub Bot logged work on BEAM-4094:


Author: ASF GitHub Bot
Created on: 25/Jul/18 23:41
Start Date: 25/Jul/18 23:41
Worklog Time Spent: 10m 
  Work Description: charlesccychen commented on issue #6058: [BEAM-4094] 
Removing ScopedMetricsContainer from Python SDK
URL: https://github.com/apache/beam/pull/6058#issuecomment-407930325
 
 
   LGTM.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127583)
Time Spent: 1h 10m  (was: 1h)

> Remove ScopedMetricsContainer from Python SDK
> -
>
> Key: BEAM-4094
> URL: https://issues.apache.org/jira/browse/BEAM-4094
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Major
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> With the conversion to a single context provider, ScopedMetricsContainer is 
> not necessary. Should remove it once dataflow / beam code no longer use it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4094) Remove ScopedMetricsContainer from Python SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4094?focusedWorklogId=127584&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127584
 ]

ASF GitHub Bot logged work on BEAM-4094:


Author: ASF GitHub Bot
Created on: 25/Jul/18 23:41
Start Date: 25/Jul/18 23:41
Worklog Time Spent: 10m 
  Work Description: charlesccychen closed pull request #6058: [BEAM-4094] 
Removing ScopedMetricsContainer from Python SDK
URL: https://github.com/apache/beam/pull/6058
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/python/apache_beam/metrics/execution.pxd 
b/sdks/python/apache_beam/metrics/execution.pxd
index af0c30c137e..74b34fbc595 100644
--- a/sdks/python/apache_beam/metrics/execution.pxd
+++ b/sdks/python/apache_beam/metrics/execution.pxd
@@ -23,10 +23,3 @@ cdef class MetricsContainer(object):
   cdef public object counters
   cdef public object distributions
   cdef public object gauges
-
-
-cdef class ScopedMetricsContainer(object):
-  cpdef enter(self)
-  cpdef exit(self)
-  cdef list _stack
-  cdef MetricsContainer _container
diff --git a/sdks/python/apache_beam/metrics/execution.py 
b/sdks/python/apache_beam/metrics/execution.py
index 5bbb9e1ed36..789d85bc34a 100644
--- a/sdks/python/apache_beam/metrics/execution.py
+++ b/sdks/python/apache_beam/metrics/execution.py
@@ -122,38 +122,16 @@ class _MetricsEnvironment(object):
   def __init__(self):
 self.METRICS_SUPPORTED = False
 self._METRICS_SUPPORTED_LOCK = threading.Lock()
-self.PER_THREAD = threading.local()
-self.set_container_stack()
-
-  def set_container_stack(self):
-if not hasattr(self.PER_THREAD, 'container'):
-  self.PER_THREAD.container = []
-
-  def container_stack(self):
-self.set_container_stack()
-return self.PER_THREAD.container
 
   def set_metrics_supported(self, supported):
-self.set_container_stack()
 with self._METRICS_SUPPORTED_LOCK:
   self.METRICS_SUPPORTED = supported
 
-  def _old_style_container(self):
-"""Gets the current MetricsContainer based on the container stack.
-
-The container stack is the old method, and will be deprecated. Should
-rely on StateSampler instead."""
-self.set_container_stack()
-index = len(self.PER_THREAD.container) - 1
-if index < 0:
-  return None
-return self.PER_THREAD.container[index]
-
   def current_container(self):
 """Returns the current MetricsContainer."""
 sampler = statesampler.get_current_tracker()
 if sampler is None:
-  return self._old_style_container()
+  return None
 return sampler.current_state().metrics_container
 
 
@@ -234,27 +212,6 @@ def to_runner_api(self):
 )
 
 
-class ScopedMetricsContainer(object):
-
-  def __init__(self, container=None):
-self._stack = MetricsEnvironment.container_stack()
-self._container = container
-
-  def enter(self):
-if self._container:
-  self._stack.append(self._container)
-
-  def exit(self):
-if self._container:
-  self._stack.pop()
-
-  def __enter__(self):
-self.enter()
-
-  def __exit__(self, type, value, traceback):
-self.exit()
-
-
 class MetricUpdates(object):
   """Contains updates for several metrics.
 
diff --git a/sdks/python/apache_beam/runners/common.pxd 
b/sdks/python/apache_beam/runners/common.pxd
index 4bb226492ba..a53f604e7e8 100644
--- a/sdks/python/apache_beam/runners/common.pxd
+++ b/sdks/python/apache_beam/runners/common.pxd
@@ -18,7 +18,6 @@
 cimport cython
 
 from apache_beam.utils.windowed_value cimport WindowedValue
-from apache_beam.metrics.execution cimport ScopedMetricsContainer
 from apache_beam.transforms.cy_dataflow_distribution_counter cimport 
DataflowDistributionCounter
 
 from libc.stdint cimport int64_t
diff --git a/sdks/python/apache_beam/runners/worker/operations.pxd 
b/sdks/python/apache_beam/runners/worker/operations.pxd
index 5a36bba2ed8..f27de8d0170 100644
--- a/sdks/python/apache_beam/runners/worker/operations.pxd
+++ b/sdks/python/apache_beam/runners/worker/operations.pxd
@@ -20,7 +20,6 @@ cimport cython
 from apache_beam.runners.common cimport Receiver
 from apache_beam.runners.worker cimport opcounters
 from apache_beam.utils.windowed_value cimport WindowedValue
-from apache_beam.metrics.execution cimport ScopedMetricsContainer
 
 
 cdef WindowedValue _globally_windowed_value
@@ -46,7 +45,6 @@ cdef class Operation(object):
   cdef object consumers
   cdef readonly counter_factory
   cdef public metrics_container
-  cdef public ScopedMetricsContainer scoped_metrics_container
   # Public for access by Fn harness operations.
   # TODO(robertwb): Cythonize FnHarness.
   cdef public list receivers
diff --git a/sdks/python/ap

[jira] [Work logged] (BEAM-4094) Remove ScopedMetricsContainer from Python SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4094?focusedWorklogId=127582&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127582
 ]

ASF GitHub Bot logged work on BEAM-4094:


Author: ASF GitHub Bot
Created on: 25/Jul/18 23:41
Start Date: 25/Jul/18 23:41
Worklog Time Spent: 10m 
  Work Description: charlesccychen commented on issue #6058: [BEAM-4094] 
Removing ScopedMetricsContainer from Python SDK
URL: https://github.com/apache/beam/pull/6058#issuecomment-407930312
 
 
   Thank you Pablo! This is a great cleanup.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127582)
Time Spent: 1h  (was: 50m)

> Remove ScopedMetricsContainer from Python SDK
> -
>
> Key: BEAM-4094
> URL: https://issues.apache.org/jira/browse/BEAM-4094
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> With the conversion to a single context provider, ScopedMetricsContainer is 
> not necessary. Should remove it once dataflow / beam code no longer use it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Merge pull request #6058 from pabloem/remove-smc

[ccy]

This is an automated email from the ASF dual-hosted git repository.

ccy pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 92be85af47bd64fbdb5fbc2a1c2d45fa147dee08
Merge: e3df55d 633db7d
Author: Charles Chen 
AuthorDate: Wed Jul 25 16:41:34 2018 -0700

Merge pull request #6058 from pabloem/remove-smc

[BEAM-4094] Removing ScopedMetricsContainer from Python SDK

 sdks/python/apache_beam/metrics/execution.pxd  |  7 
 sdks/python/apache_beam/metrics/execution.py   | 45 +-
 sdks/python/apache_beam/runners/common.pxd |  1 -
 .../apache_beam/runners/worker/operations.pxd  |  2 -
 .../apache_beam/runners/worker/operations.py   |  4 --
 5 files changed, 1 insertion(+), 58 deletions(-)

[jira] [Work logged] (BEAM-4094) Remove ScopedMetricsContainer from Python SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4094?focusedWorklogId=127580&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127580
 ]

ASF GitHub Bot logged work on BEAM-4094:


Author: ASF GitHub Bot
Created on: 25/Jul/18 23:40
Start Date: 25/Jul/18 23:40
Worklog Time Spent: 10m 
  Work Description: charlesccychen commented on issue #6058: [BEAM-4094] 
Removing ScopedMetricsContainer from Python SDK
URL: https://github.com/apache/beam/pull/6058#issuecomment-407930112
 
 
   Run postcommits


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127580)
Time Spent: 40m  (was: 0.5h)

> Remove ScopedMetricsContainer from Python SDK
> -
>
> Key: BEAM-4094
> URL: https://issues.apache.org/jira/browse/BEAM-4094
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> With the conversion to a single context provider, ScopedMetricsContainer is 
> not necessary. Should remove it once dataflow / beam code no longer use it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (e3df55d -> 92be85a)

[ccy]

This is an automated email from the ASF dual-hosted git repository.

ccy pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from e3df55d  Fix Java Nightly Snapshot Failures
 add d0cb44d  Removing scoped metrics container
 add 633db7d  Remove old style metrics context management
 new 92be85a  Merge pull request #6058 from pabloem/remove-smc

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/python/apache_beam/metrics/execution.pxd  |  7 
 sdks/python/apache_beam/metrics/execution.py   | 45 +-
 sdks/python/apache_beam/runners/common.pxd |  1 -
 .../apache_beam/runners/worker/operations.pxd  |  2 -
 .../apache_beam/runners/worker/operations.py   |  4 --
 5 files changed, 1 insertion(+), 58 deletions(-)

[jira] [Work logged] (BEAM-4094) Remove ScopedMetricsContainer from Python SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4094?focusedWorklogId=127581&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127581
 ]

ASF GitHub Bot logged work on BEAM-4094:


Author: ASF GitHub Bot
Created on: 25/Jul/18 23:40
Start Date: 25/Jul/18 23:40
Worklog Time Spent: 10m 
  Work Description: charlesccychen removed a comment on issue #6058: 
[BEAM-4094] Removing ScopedMetricsContainer from Python SDK
URL: https://github.com/apache/beam/pull/6058#issuecomment-407930112
 
 
   Run postcommits


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127581)
Time Spent: 50m  (was: 40m)

> Remove ScopedMetricsContainer from Python SDK
> -
>
> Key: BEAM-4094
> URL: https://issues.apache.org/jira/browse/BEAM-4094
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> With the conversion to a single context provider, ScopedMetricsContainer is 
> not necessary. Should remove it once dataflow / beam code no longer use it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4094) Remove ScopedMetricsContainer from Python SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4094?focusedWorklogId=127579&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127579
 ]

ASF GitHub Bot logged work on BEAM-4094:


Author: ASF GitHub Bot
Created on: 25/Jul/18 23:40
Start Date: 25/Jul/18 23:40
Worklog Time Spent: 10m 
  Work Description: charlesccychen removed a comment on issue #6058: 
[BEAM-4094] Removing ScopedMetricsContainer from Python SDK
URL: https://github.com/apache/beam/pull/6058#issuecomment-407930040
 
 
   Run Python Postcommits


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127579)
Time Spent: 0.5h  (was: 20m)

> Remove ScopedMetricsContainer from Python SDK
> -
>
> Key: BEAM-4094
> URL: https://issues.apache.org/jira/browse/BEAM-4094
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> With the conversion to a single context provider, ScopedMetricsContainer is 
> not necessary. Should remove it once dataflow / beam code no longer use it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4094) Remove ScopedMetricsContainer from Python SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4094?focusedWorklogId=127578&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127578
 ]

ASF GitHub Bot logged work on BEAM-4094:


Author: ASF GitHub Bot
Created on: 25/Jul/18 23:39
Start Date: 25/Jul/18 23:39
Worklog Time Spent: 10m 
  Work Description: charlesccychen commented on issue #6058: [BEAM-4094] 
Removing ScopedMetricsContainer from Python SDK
URL: https://github.com/apache/beam/pull/6058#issuecomment-407930040
 
 
   Run Python Postcommits


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127578)
Time Spent: 20m  (was: 10m)

> Remove ScopedMetricsContainer from Python SDK
> -
>
> Key: BEAM-4094
> URL: https://issues.apache.org/jira/browse/BEAM-4094
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> With the conversion to a single context provider, ScopedMetricsContainer is 
> not necessary. Should remove it once dataflow / beam code no longer use it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4629) RAT failures are hard to debug

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4629?focusedWorklogId=127572&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127572
 ]

ASF GitHub Bot logged work on BEAM-4629:


Author: ASF GitHub Bot
Created on: 25/Jul/18 23:22
Start Date: 25/Jul/18 23:22
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #6069: [BEAM-4629] Output 
the names of the files with missing/unknown licenses as part of the build 
process.
URL: https://github.com/apache/beam/pull/6069#issuecomment-407927022
 
 
   R: @aaltay 
   CC: @alanmyrvold 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127572)
Time Spent: 20m  (was: 10m)

> RAT failures are hard to debug
> --
>
> Key: BEAM-4629
> URL: https://issues.apache.org/jira/browse/BEAM-4629
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Ahmet Altay
>Assignee: Luke Cwik
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> RAT failures produces an error with a reference to log file that is not 
> accessible.
> For example:
> > Task :rat
> Build cache key for task ':rat' is 7a8f8655183c948a3f7a087e2f7c412b
> Caching disabled for task ':rat': Caching has not been enabled for the task
> Task ':rat' is not up-to-date because:
>   No history is available.
> Rat TXT report: 
> file:/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild@2/src/build/reports/rat/rat-report.txt
> This is not useful unless, the person has access to the jenkins VM. Tests 
> should include a copy of the failed rat-report.txt.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4629) RAT failures are hard to debug

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4629?focusedWorklogId=127571&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127571
 ]

ASF GitHub Bot logged work on BEAM-4629:


Author: ASF GitHub Bot
Created on: 25/Jul/18 23:21
Start Date: 25/Jul/18 23:21
Worklog Time Spent: 10m 
  Work Description: lukecwik opened a new pull request #6069: [BEAM-4629] 
Output the names of the failing licenses as part of the build process.
URL: https://github.com/apache/beam/pull/6069
 
 
   An example build output with a failing license:
   ```
   > Task :rat FAILED
   Unapproved/unknown license: 
/usr/local/google/home/lcwik/git/beam/build.gradle
   
   FAILURE: Build failed with an exception.
   
   * What went wrong:
   Execution failed for task ':rat'.
   > Found 1 files with unapproved/unknown licenses. See 
file:/usr/local/google/home/lcwik/git/beam/build/reports/rat/rat-report.xml
   ```
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | --- | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127571)
Time Spent: 10m
Remaining Estimate: 0h

> RAT failures are hard to debug
> --
>
> Key: BEAM-4629
> URL: https://i

[jira] [Assigned] (BEAM-4629) RAT failures are hard to debug

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik reassigned BEAM-4629:
---

Assignee: Luke Cwik  (was: Alan Myrvold)

> RAT failures are hard to debug
> --
>
> Key: BEAM-4629
> URL: https://issues.apache.org/jira/browse/BEAM-4629
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Ahmet Altay
>Assignee: Luke Cwik
>Priority: Major
>
> RAT failures produces an error with a reference to log file that is not 
> accessible.
> For example:
> > Task :rat
> Build cache key for task ':rat' is 7a8f8655183c948a3f7a087e2f7c412b
> Caching disabled for task ':rat': Caching has not been enabled for the task
> Task ':rat' is not up-to-date because:
>   No history is available.
> Rat TXT report: 
> file:/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild@2/src/build/reports/rat/rat-report.txt
> This is not useful unless, the person has access to the jenkins VM. Tests 
> should include a copy of the failed rat-report.txt.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated: Fix Java Nightly Snapshot Failures

[pabloem]

This is an automated email from the ASF dual-hosted git repository.

pabloem pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new e3df55d  Fix Java Nightly Snapshot Failures
e3df55d is described below

commit e3df55da5b903440a84b19185d442b6abbc1dc36
Author: Yifan Zou 
AuthorDate: Wed Jul 25 14:59:54 2018 -0700

Fix Java Nightly Snapshot Failures
---
 release/src/main/groovy/MobileGamingCommands.groovy | 2 --
 1 file changed, 2 deletions(-)

diff --git a/release/src/main/groovy/MobileGamingCommands.groovy 
b/release/src/main/groovy/MobileGamingCommands.groovy
index 3357381..bf8d049 100644
--- a/release/src/main/groovy/MobileGamingCommands.groovy
+++ b/release/src/main/groovy/MobileGamingCommands.groovy
@@ -126,7 +126,6 @@ class MobileGamingCommands {
 return [project: testScripts.gcpProject(),
   dataset: testScripts.bqDataset(),
   topic: 
"projects/${testScripts.gcpProject()}/topics/${testScripts.pubsubTopic()}",
-  output: "gs://${testScripts.gcsBucket()}/java-leaderboard-result.txt",
   leaderBoardTableName: "leaderboard_${runner}",
   teamWindowDuration: 5]
   }
@@ -135,7 +134,6 @@ class MobileGamingCommands {
 return [project: testScripts.gcpProject(),
   dataset: testScripts.bqDataset(),
   topic: 
"projects/${testScripts.gcpProject()}/topics/${testScripts.pubsubTopic()}",
-  output: "gs://${testScripts.gcsBucket()}/java-leaderboard-result.txt",
   fixedWindowDuration: 5,
   userActivityWindowDuration: 5,
   sessionGap: 1,

[jira] [Resolved] (BEAM-4867) Run :rat for all parts of repository as pre-commit

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-4867.
-
   Resolution: Duplicate
Fix Version/s: Not applicable

> Run :rat for all parts of repository as pre-commit
> --
>
> Key: BEAM-4867
> URL: https://issues.apache.org/jira/browse/BEAM-4867
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Mikhail Gryzykhin
>Assignee: Jason Kuster
>Priority: Major
> Fix For: Not applicable
>
>
> The idea comes from:
> [https://issues.apache.org/jira/projects/BEAM/issues/BEAM-4866]
> Apparently, we do not run pre-commits for all of our paths and not all 
> pre-commits are always relevant. However we still want to verify licence 
> headers an whole repo.
>  
> This can be done as extracting :rat as a separate job and run it on each 
> commit regardless of path.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-1457) Enable rat plugin and findbugs plugin in default build

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik reassigned BEAM-1457:
---

Assignee: Jean-Baptiste Onofré

> Enable rat plugin and findbugs plugin in default build
> --
>
> Key: BEAM-1457
> URL: https://issues.apache.org/jira/browse/BEAM-1457
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system
>Reporter: Aviem Zur
>Assignee: Jean-Baptiste Onofré
>Priority: Major
> Fix For: Not applicable
>
>
> Today, maven rat plugin and findbugs plugin only run when `release` profile 
> is specified.
> Since these plugins do not add a large amount of time compared to the normal 
> build, and their checks are required to pass to approve pull requests - let's 
> enable them by default.
> [Original dev list 
> discussion|https://lists.apache.org/thread.html/e1f80e54b44b4a39630d978abe79fb6a6cecf71d9821ee1881b47afb@%3Cdev.beam.apache.org%3E]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-1457) Enable rat plugin and findbugs plugin in default build

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-1457.
-
   Resolution: Fixed
Fix Version/s: Not applicable

During the migration to gradle the rat plugin now runs as part of the 
precommits.

> Enable rat plugin and findbugs plugin in default build
> --
>
> Key: BEAM-1457
> URL: https://issues.apache.org/jira/browse/BEAM-1457
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system
>Reporter: Aviem Zur
>Priority: Major
> Fix For: Not applicable
>
>
> Today, maven rat plugin and findbugs plugin only run when `release` profile 
> is specified.
> Since these plugins do not add a large amount of time compared to the normal 
> build, and their checks are required to pass to approve pull requests - let's 
> enable them by default.
> [Original dev list 
> discussion|https://lists.apache.org/thread.html/e1f80e54b44b4a39630d978abe79fb6a6cecf71d9821ee1881b47afb@%3Cdev.beam.apache.org%3E]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4859) Run Python ValidatesRunner tests in streaming

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4859?focusedWorklogId=127557&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127557
 ]

ASF GitHub Bot logged work on BEAM-4859:


Author: ASF GitHub Bot
Created on: 25/Jul/18 22:51
Start Date: 25/Jul/18 22:51
Worklog Time Spent: 10m 
  Work Description: markflyhigh commented on issue #6053: [BEAM-4859] 
Enable Python VR tests in streaming in postcommit task
URL: https://github.com/apache/beam/pull/6053#issuecomment-407921409
 
 
   Run Python Dataflow ValidatesRunner


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127557)
Time Spent: 3h 10m  (was: 3h)

> Run Python ValidatesRunner tests in streaming
> -
>
> Key: BEAM-4859
> URL: https://issues.apache.org/jira/browse/BEAM-4859
> Project: Beam
>  Issue Type: Test
>  Components: testing
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Major
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> Automated way to run existing Python ValidatesRunner tests in streaming. 
> Gradle should also be able to run them, so Jenkins automation will be much 
> simpler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4859) Run Python ValidatesRunner tests in streaming

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4859?focusedWorklogId=127555&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127555
 ]

ASF GitHub Bot logged work on BEAM-4859:


Author: ASF GitHub Bot
Created on: 25/Jul/18 22:47
Start Date: 25/Jul/18 22:47
Worklog Time Spent: 10m 
  Work Description: markflyhigh commented on issue #6053: [BEAM-4859] 
Enable Python VR tests in streaming in postcommit task
URL: https://github.com/apache/beam/pull/6053#issuecomment-407920485
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127555)
Time Spent: 3h  (was: 2h 50m)

> Run Python ValidatesRunner tests in streaming
> -
>
> Key: BEAM-4859
> URL: https://issues.apache.org/jira/browse/BEAM-4859
> Project: Beam
>  Issue Type: Test
>  Components: testing
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Major
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> Automated way to run existing Python ValidatesRunner tests in streaming. 
> Gradle should also be able to run them, so Jenkins automation will be much 
> simpler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4859) Run Python ValidatesRunner tests in streaming

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4859?focusedWorklogId=127554&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127554
 ]

ASF GitHub Bot logged work on BEAM-4859:


Author: ASF GitHub Bot
Created on: 25/Jul/18 22:47
Start Date: 25/Jul/18 22:47
Worklog Time Spent: 10m 
  Work Description: markflyhigh removed a comment on issue #6053: 
[BEAM-4859] Enable Python VR tests in streaming in postcommit task
URL: https://github.com/apache/beam/pull/6053#issuecomment-407828152
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127554)
Time Spent: 2h 50m  (was: 2h 40m)

> Run Python ValidatesRunner tests in streaming
> -
>
> Key: BEAM-4859
> URL: https://issues.apache.org/jira/browse/BEAM-4859
> Project: Beam
>  Issue Type: Test
>  Components: testing
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Major
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> Automated way to run existing Python ValidatesRunner tests in streaming. 
> Gradle should also be able to run them, so Jenkins automation will be much 
> simpler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4859) Run Python ValidatesRunner tests in streaming

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4859?focusedWorklogId=127553&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127553
 ]

ASF GitHub Bot logged work on BEAM-4859:


Author: ASF GitHub Bot
Created on: 25/Jul/18 22:46
Start Date: 25/Jul/18 22:46
Worklog Time Spent: 10m 
  Work Description: markflyhigh removed a comment on issue #6053: 
[BEAM-4859] Enable Python VR tests in streaming in postcommit task
URL: https://github.com/apache/beam/pull/6053#issuecomment-407630401
 
 
   Run Python PostCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127553)
Time Spent: 2h 40m  (was: 2.5h)

> Run Python ValidatesRunner tests in streaming
> -
>
> Key: BEAM-4859
> URL: https://issues.apache.org/jira/browse/BEAM-4859
> Project: Beam
>  Issue Type: Test
>  Components: testing
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Major
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> Automated way to run existing Python ValidatesRunner tests in streaming. 
> Gradle should also be able to run them, so Jenkins automation will be much 
> simpler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4808) Add an integration test for BeamSqlLine

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4808?focusedWorklogId=127552&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127552
 ]

ASF GitHub Bot logged work on BEAM-4808:


Author: ASF GitHub Bot
Created on: 25/Jul/18 22:44
Start Date: 25/Jul/18 22:44
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on issue #6006: [BEAM-4808][SQL] 
add e2e test for BeamSqlLine.
URL: https://github.com/apache/beam/pull/6006#issuecomment-407919874
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127552)
Time Spent: 5.5h  (was: 5h 20m)

> Add an integration test for BeamSqlLine
> ---
>
> Key: BEAM-4808
> URL: https://issues.apache.org/jira/browse/BEAM-4808
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 5.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4849) Support running Beam Samza jobs in Yarn

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4849?focusedWorklogId=127551&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127551
 ]

ASF GitHub Bot logged work on BEAM-4849:


Author: ASF GitHub Bot
Created on: 25/Jul/18 22:40
Start Date: 25/Jul/18 22:40
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #6029: [BEAM-4849] Support 
running Beam Samza jobs in Yarn
URL: https://github.com/apache/beam/pull/6029#issuecomment-407919159
 
 
   For some reason after attempting to get the precommit to pass it has been 
consistently failing on
   ```
   Task :beam-examples-java:compileTestJava FAILED
   
/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_Phrase/src/examples/java/src/test/java/org/apache/beam/examples/cookbook/BigQueryTornadoesIT.java:22:
 error: cannot access BigqueryMatcher
   import org.apache.beam.sdk.io.gcp.testing.BigqueryMatcher;
^
 bad class file: 
/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_Phrase/src/sdks/java/io/google-cloud-platform/build/libs/beam-sdks-java-io-google-cloud-platform-2.7.0-SNAPSHOT-tests.jar(/org/apache/beam/sdk/io/gcp/testing/BigqueryMatcher.class)
   unable to access file: java.util.zip.ZipException: invalid stored block 
lengths
   Please remove or make sure it appears in the correct subdirectory of the 
classpath.
   1 error
   :beam-examples-java:compileTestJava (Thread[Task worker for ':' Thread 
8,5,main]) completed. Took 16.434 secs.
   ```
   Any ideas?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127551)
Time Spent: 50m  (was: 40m)

> Support running Beam Samza jobs in Yarn
> ---
>
> Key: BEAM-4849
> URL: https://issues.apache.org/jira/browse/BEAM-4849
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-samza
>Reporter: Xinyu Liu
>Assignee: Xinyu Liu
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> We need to some util classes and script to run beam job containers in Yarn. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostRelease_NightlySnapshot #303

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-5024) Deprecate Dataflow SDK Distribution

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5024?focusedWorklogId=127549&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127549
 ]

ASF GitHub Bot logged work on BEAM-5024:


Author: ASF GitHub Bot
Created on: 25/Jul/18 22:32
Start Date: 25/Jul/18 22:32
Worklog Time Spent: 10m 
  Work Description: aaltay commented on issue #6067: [BEAM-5024] Remove 
reference to dataflow-distribution.properties
URL: https://github.com/apache/beam/pull/6067#issuecomment-407917522
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127549)
Time Spent: 0.5h  (was: 20m)

> Deprecate Dataflow SDK Distribution
> ---
>
> Key: BEAM-5024
> URL: https://issues.apache.org/jira/browse/BEAM-5024
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Reporter: Ahmet Altay
>Assignee: Ahmet Altay
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> GCP decided to deprecate Dataflow SDK distribution, in favor of Apache Beam 
> SDK distribution.
> All in Beam!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5024) Deprecate Dataflow SDK Distribution

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5024?focusedWorklogId=127547&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127547
 ]

ASF GitHub Bot logged work on BEAM-5024:


Author: ASF GitHub Bot
Created on: 25/Jul/18 22:32
Start Date: 25/Jul/18 22:32
Worklog Time Spent: 10m 
  Work Description: aaltay commented on issue #6067: [BEAM-5024] Remove 
reference to dataflow-distribution.properties
URL: https://github.com/apache/beam/pull/6067#issuecomment-407917497
 
 
   Two failed tests are:
   :beam-sdks-java-extensions-protobuf:extractIncludeTestProto FAILED
   :beam-sdks-java-io-elasticsearch-tests-2:compileTestJava FAILED
   
   I do not know how they are related to this change.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127547)
Time Spent: 20m  (was: 10m)

> Deprecate Dataflow SDK Distribution
> ---
>
> Key: BEAM-5024
> URL: https://issues.apache.org/jira/browse/BEAM-5024
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Reporter: Ahmet Altay
>Assignee: Ahmet Altay
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> GCP decided to deprecate Dataflow SDK distribution, in favor of Apache Beam 
> SDK distribution.
> All in Beam!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=127546&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127546
 ]

ASF GitHub Bot logged work on BEAM-4176:


Author: ASF GitHub Bot
Created on: 25/Jul/18 22:31
Start Date: 25/Jul/18 22:31
Worklog Time Spent: 10m 
  Work Description: lukecwik closed pull request #5935: [BEAM-4176] Initial 
implementation for running portable runner tests
URL: https://github.com/apache/beam/pull/5935
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/runners/flink/job-server/build.gradle 
b/runners/flink/job-server/build.gradle
index 552e04c3d74..c76a130d430 100644
--- a/runners/flink/job-server/build.gradle
+++ b/runners/flink/job-server/build.gradle
@@ -15,6 +15,7 @@
  * See the License for the specific language governing permissions and
  * limitations under the License.
  */
+import groovy.json.JsonOutput
 
 apply plugin: org.apache.beam.gradle.BeamModulePlugin
 applyJavaNature(
@@ -23,16 +24,32 @@ applyJavaNature(
   },
 )
 
+/*
+ * We need to rely on manually specifying these evaluationDependsOn to ensure 
that
+ * the following projects are evaluated before we evaluate this project. This 
is because
+ * we are attempting to reference the "sourceSets.test.output" directly.
+ */
+evaluationDependsOn(":beam-sdks-java-core")
+evaluationDependsOn(":beam-runners-core-java")
+
 description = "Apache Beam :: Runners :: Flink :: Job Server"
 
 apply plugin: "application"
 
 mainClassName = "org.apache.beam.runners.flink.FlinkJobServerDriver"
 
+configurations {
+  validatesRunner
+}
+
 dependencies {
   compile project(path: ":beam-runners-flink_2.11", configuration: "shadow")
+  validatesRunner project(path: ":beam-runners-flink_2.11", configuration: 
"shadowTest")
+  validatesRunner project(path: ":beam-sdks-java-core", configuration: 
"shadowTest")
+  validatesRunner project(path: ":beam-runners-core-java", configuration: 
"shadowTest")
+  validatesRunner project(path: ":beam-runners-reference-java", configuration: 
"shadowTest")
   compile project(path: 
":beam-sdks-java-extensions-google-cloud-platform-core", configuration: 
"shadow")
-//  TODO: Enable AWS and HDPS file system.
+//  TODO: Enable AWS and HDFS file system.
 }
 
 // NOTE: runShadow must be used in order to run the job server. The standard 
run
@@ -45,3 +62,59 @@ runShadow {
   // Enable remote debugging.
   jvmArgs = ["-Xdebug", 
"-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005"]
 }
+
+class PortableValidatesRunnerConfig {
+  // Task name for validate runner case.
+  String name
+  // Fully qualified JobServerClass name to use.
+  String jobServerDriver
+  // A string representing the jobServer Configuration.
+  String jobServerConfig
+  // Flag to include tests for streaming or batch.
+  boolean streaming
+}
+
+def createPortableValidatesRunnerTask = {
+  def config = it ? it as PortableValidatesRunnerConfig : new 
PortableValidatesRunnerConfig()
+  tasks.create(name: config.name, type: Test) {
+group = "Verification"
+description = "Validates the PortableRunner with JobServer 
${config.jobServerDriver}"
+systemProperty "beamTestPipelineOptions", JsonOutput.toJson([
+
"--runner=org.apache.beam.runners.reference.testing.TestPortableRunner",
+"--jobServerDriver=${config.jobServerDriver}",
+config.jobServerConfig ? 
"--jobServerConfig=${config.jobServerConfig}" : "",
+])
+classpath = configurations.validatesRunner
+testClassesDirs = 
files(project(":beam-sdks-java-core").sourceSets.test.output.classesDirs, 
project(":beam-runners-core-java").sourceSets.test.output.classesDirs)
+maxParallelForks 1
+if (config.streaming) {
+  useJUnit {
+includeCategories 'org.apache.beam.sdk.testing.ValidatesRunner'
+excludeCategories 
'org.apache.beam.sdk.testing.FlattenWithHeterogeneousCoders'
+excludeCategories 'org.apache.beam.sdk.testing.LargeKeys$Above100MB'
+excludeCategories 'org.apache.beam.sdk.testing.UsesCommittedMetrics'
+excludeCategories 'org.apache.beam.sdk.testing.UsesImpulse'
+excludeCategories 'org.apache.beam.sdk.testing.UsesSchema'
+excludeCategories 'org.apache.beam.sdk.testing.UsesTestStream'
+  }
+} else {
+  useJUnit {
+includeCategories 'org.apache.beam.sdk.testing.ValidatesRunner'
+excludeCategories 
'org.apache.beam.sdk.testing.FlattenWithHeterogeneousCoders'
+excludeCategories 'org.apache.beam.sdk.testing.LargeKeys$Above100MB'
+excludeCategories 'org.apache.beam.sdk.testing.UsesCommittedMetrics'
+excludeCategories 'org.apache.beam.sdk.t

[beam] branch master updated: [BEAM-4176] Initial implementation for running portable runner tests (#5935)

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new 75e3986  [BEAM-4176] Initial implementation for running portable 
runner tests (#5935)
75e3986 is described below

commit 75e3986171ce68a6fc4c42689698576c407f05b1
Author: Ankur 
AuthorDate: Wed Jul 25 15:31:35 2018 -0700

[BEAM-4176] Initial implementation for running portable runner tests (#5935)

* TestPortableRunner for validate runner tests

* Dynamically assign jobServer host and port.

* Documentation formatting

* Disable Metrics for Flink Portable Runner

* Change to gradle files and check for pipeline completion for flink 
validates runner test

* Review Comments

* Adding javadoc for default option factory

* Update lambda to handle 0 method args.

* Fix simple typo

* Use varargs in array based method declaration.

* Fixing Log message
---
 runners/flink/job-server/build.gradle  |  75 +++-
 .../beam/runners/flink/FlinkJobInvocation.java |   2 +-
 .../beam/runners/flink/FlinkJobServerDriver.java   |  95 +++-
 runners/reference/java/build.gradle|   5 ++
 .../testing/TestPortablePipelineOptions.java   |  65 ++
 .../reference/testing/TestPortableRunner.java  | 100 +
 6 files changed, 317 insertions(+), 25 deletions(-)

diff --git a/runners/flink/job-server/build.gradle 
b/runners/flink/job-server/build.gradle
index 552e04c..c76a130 100644
--- a/runners/flink/job-server/build.gradle
+++ b/runners/flink/job-server/build.gradle
@@ -15,6 +15,7 @@
  * See the License for the specific language governing permissions and
  * limitations under the License.
  */
+import groovy.json.JsonOutput
 
 apply plugin: org.apache.beam.gradle.BeamModulePlugin
 applyJavaNature(
@@ -23,16 +24,32 @@ applyJavaNature(
   },
 )
 
+/*
+ * We need to rely on manually specifying these evaluationDependsOn to ensure 
that
+ * the following projects are evaluated before we evaluate this project. This 
is because
+ * we are attempting to reference the "sourceSets.test.output" directly.
+ */
+evaluationDependsOn(":beam-sdks-java-core")
+evaluationDependsOn(":beam-runners-core-java")
+
 description = "Apache Beam :: Runners :: Flink :: Job Server"
 
 apply plugin: "application"
 
 mainClassName = "org.apache.beam.runners.flink.FlinkJobServerDriver"
 
+configurations {
+  validatesRunner
+}
+
 dependencies {
   compile project(path: ":beam-runners-flink_2.11", configuration: "shadow")
+  validatesRunner project(path: ":beam-runners-flink_2.11", configuration: 
"shadowTest")
+  validatesRunner project(path: ":beam-sdks-java-core", configuration: 
"shadowTest")
+  validatesRunner project(path: ":beam-runners-core-java", configuration: 
"shadowTest")
+  validatesRunner project(path: ":beam-runners-reference-java", configuration: 
"shadowTest")
   compile project(path: 
":beam-sdks-java-extensions-google-cloud-platform-core", configuration: 
"shadow")
-//  TODO: Enable AWS and HDPS file system.
+//  TODO: Enable AWS and HDFS file system.
 }
 
 // NOTE: runShadow must be used in order to run the job server. The standard 
run
@@ -45,3 +62,59 @@ runShadow {
   // Enable remote debugging.
   jvmArgs = ["-Xdebug", 
"-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005"]
 }
+
+class PortableValidatesRunnerConfig {
+  // Task name for validate runner case.
+  String name
+  // Fully qualified JobServerClass name to use.
+  String jobServerDriver
+  // A string representing the jobServer Configuration.
+  String jobServerConfig
+  // Flag to include tests for streaming or batch.
+  boolean streaming
+}
+
+def createPortableValidatesRunnerTask = {
+  def config = it ? it as PortableValidatesRunnerConfig : new 
PortableValidatesRunnerConfig()
+  tasks.create(name: config.name, type: Test) {
+group = "Verification"
+description = "Validates the PortableRunner with JobServer 
${config.jobServerDriver}"
+systemProperty "beamTestPipelineOptions", JsonOutput.toJson([
+
"--runner=org.apache.beam.runners.reference.testing.TestPortableRunner",
+"--jobServerDriver=${config.jobServerDriver}",
+config.jobServerConfig ? 
"--jobServerConfig=${config.jobServerConfig}" : "",
+])
+classpath = configurations.validatesRunner
+testClassesDirs = 
files(project(":beam-sdks-java-core").sourceSets.test.output.classesDirs, 
project(":beam-runners-core-java").sourceSets.test.output.classesDirs)
+maxParallelForks 1
+if (config.streaming) {
+  useJUnit {
+includeCategories 'org.apache.beam.sdk.testing.ValidatesRunner'
+excludeCategories 
'org.apache.beam.sdk.testing.FlattenWithHeterogeneousCoders'
+excludeCategories 'org.apache.beam.sdk.test

[jira] [Work logged] (BEAM-4808) Add an integration test for BeamSqlLine

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4808?focusedWorklogId=127541&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127541
 ]

ASF GitHub Bot logged work on BEAM-4808:


Author: ASF GitHub Bot
Created on: 25/Jul/18 22:22
Start Date: 25/Jul/18 22:22
Worklog Time Spent: 10m 
  Work Description: amaliujia removed a comment on issue #6006: 
[BEAM-4808][SQL] add e2e test for BeamSqlLine.
URL: https://github.com/apache/beam/pull/6006#issuecomment-406489289
 
 
   run java postcommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127541)
Time Spent: 5h 10m  (was: 5h)

> Add an integration test for BeamSqlLine
> ---
>
> Key: BEAM-4808
> URL: https://issues.apache.org/jira/browse/BEAM-4808
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 5h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4808) Add an integration test for BeamSqlLine

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4808?focusedWorklogId=127542&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127542
 ]

ASF GitHub Bot logged work on BEAM-4808:


Author: ASF GitHub Bot
Created on: 25/Jul/18 22:22
Start Date: 25/Jul/18 22:22
Worklog Time Spent: 10m 
  Work Description: amaliujia removed a comment on issue #6006: 
[BEAM-4808][SQL] add e2e test for BeamSqlLine.
URL: https://github.com/apache/beam/pull/6006#issuecomment-406450624
 
 
   run java postcommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127542)
Time Spent: 5h 20m  (was: 5h 10m)

> Add an integration test for BeamSqlLine
> ---
>
> Key: BEAM-4808
> URL: https://issues.apache.org/jira/browse/BEAM-4808
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 5h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4808) Add an integration test for BeamSqlLine

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4808?focusedWorklogId=127540&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127540
 ]

ASF GitHub Bot logged work on BEAM-4808:


Author: ASF GitHub Bot
Created on: 25/Jul/18 22:22
Start Date: 25/Jul/18 22:22
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on issue #6006: [BEAM-4808][SQL] 
add e2e test for BeamSqlLine.
URL: https://github.com/apache/beam/pull/6006#issuecomment-407915409
 
 
   run java postcommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127540)
Time Spent: 5h  (was: 4h 50m)

> Add an integration test for BeamSqlLine
> ---
>
> Key: BEAM-4808
> URL: https://issues.apache.org/jira/browse/BEAM-4808
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5024) Deprecate Dataflow SDK Distribution

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5024?focusedWorklogId=127536&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127536
 ]

ASF GitHub Bot logged work on BEAM-5024:


Author: ASF GitHub Bot
Created on: 25/Jul/18 22:15
Start Date: 25/Jul/18 22:15
Worklog Time Spent: 10m 
  Work Description: aaltay commented on issue #6067: [BEAM-5024] Remove 
reference to dataflow-distribution.properties
URL: https://github.com/apache/beam/pull/6067#issuecomment-407914126
 
 
   Added a JIRA. Thank you both.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127536)
Time Spent: 10m
Remaining Estimate: 0h

> Deprecate Dataflow SDK Distribution
> ---
>
> Key: BEAM-5024
> URL: https://issues.apache.org/jira/browse/BEAM-5024
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Reporter: Ahmet Altay
>Assignee: Ahmet Altay
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> GCP decided to deprecate Dataflow SDK distribution, in favor of Apache Beam 
> SDK distribution.
> All in Beam!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5024) Deprecate Dataflow SDK Distribution

[Ahmet Altay (JIRA)]

Ahmet Altay created BEAM-5024:
-

 Summary: Deprecate Dataflow SDK Distribution
 Key: BEAM-5024
 URL: https://issues.apache.org/jira/browse/BEAM-5024
 Project: Beam
  Issue Type: Improvement
  Components: runner-dataflow
Reporter: Ahmet Altay
Assignee: Ahmet Altay


GCP decided to deprecate Dataflow SDK distribution, in favor of Apache Beam SDK 
distribution.

All in Beam!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4849) Support running Beam Samza jobs in Yarn

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4849?focusedWorklogId=127535&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127535
 ]

ASF GitHub Bot logged work on BEAM-4849:


Author: ASF GitHub Bot
Created on: 25/Jul/18 22:14
Start Date: 25/Jul/18 22:14
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #6029: [BEAM-4849] Support 
running Beam Samza jobs in Yarn
URL: https://github.com/apache/beam/pull/6029#issuecomment-407913910
 
 
   Run Java PreCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127535)
Time Spent: 40m  (was: 0.5h)

> Support running Beam Samza jobs in Yarn
> ---
>
> Key: BEAM-4849
> URL: https://issues.apache.org/jira/browse/BEAM-4849
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-samza
>Reporter: Xinyu Liu
>Assignee: Xinyu Liu
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> We need to some util classes and script to run beam job containers in Yarn. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5022) Move beam-sdks-java-fn-execution#createPortableValidatesRunnerTask to BeamModulePlugin

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik updated BEAM-5022:

Description: 
Move beam-sdks-java-fn-execution#createPortableValidatesRunnerTask to 
BeamModulePlugin So that it can be used by other portable runners tests.

 

Also Create an interface TestJobserverDriver and make the drivers extend it 
instead of using reflection start the Jobserver.

  was:
Move beam-sdks-java-fn-execution#createPortableValidatesRunnerTask to 
BeamGradlePlugin So that it can be used by other portable runners tests.

 

Also Create an interface TestJobserverDriver and make the drivers extend it 
instead of using reflection start the Jobserver.


> Move beam-sdks-java-fn-execution#createPortableValidatesRunnerTask to 
> BeamModulePlugin
> --
>
> Key: BEAM-5022
> URL: https://issues.apache.org/jira/browse/BEAM-5022
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system, runner-flink
>Reporter: Ankur Goenka
>Assignee: Ankur Goenka
>Priority: Major
>
> Move beam-sdks-java-fn-execution#createPortableValidatesRunnerTask to 
> BeamModulePlugin So that it can be used by other portable runners tests.
>  
> Also Create an interface TestJobserverDriver and make the drivers extend it 
> instead of using reflection start the Jobserver.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)