[jira] [Commented] (BEAM-1081) annotations should support custom messages and classes

[Javier A Gonzalez Trejo (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640946#comment-16640946
 ] 

Javier A Gonzalez Trejo commented on BEAM-1081:
---

Hi.

I having trouble on how this issue should be resolved. I modified the 
annotation test code to use a class as follows.

def test_deprecated_with_since_current_message_class(self):
    with warnings.catch_warnings(record=True) as w:
  @deprecated(since='v.1', current='multiply', extra_message='Do this')
  class class_test_deprecated_with_since_current_message:
    fooo = 'lol'
    def foo(self):
  return 'lol'
  foo = class_test_deprecated_with_since_current_message()
  self.check_annotation(
  warning=w, warning_size=1,
  warning_type=DeprecationWarning,
  fnc_name='class_test_deprecated_with_since_current_message',
  annotation_type='deprecated',
  label_check_list=[('since', True),
    ('instead', True),
    ('Do this', True)])

And it seems to work. My question is. How the annotation decorators must behave 
using them against a class?

> annotations should support custom messages and classes
> --
>
> Key: BEAM-1081
> URL: https://issues.apache.org/jira/browse/BEAM-1081
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Ahmet Altay
>Priority: Minor
>  Labels: newbie, starter
>
> Update 
> https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/utils/annotations.py
>  to add 2 new features:
> 1. ability to customize message
> 2. ability to tag classes (not only functions)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4937) Beam Dependency Update Request: org.codehaus.groovy:groovy-all 3.0.0-alpha-3

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4937?focusedWorklogId=152015=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-152015
 ]

ASF GitHub Bot logged work on BEAM-4937:


Author: ASF GitHub Bot
Created on: 07/Oct/18 00:16
Start Date: 07/Oct/18 00:16
Worklog Time Spent: 10m 
  Work Description: stale[bot] commented on issue #6171: Do Not Merge 
[BEAM-4937] upgrade groovy-all
URL: https://github.com/apache/beam/pull/6171#issuecomment-427615531
 
 
   This pull request has been marked as stale due to 60 days of inactivity. It 
will be closed in 1 week if no further activity occurs. If you think that’s 
incorrect or this pull request requires a review, please simply write any 
comment. If closed, you can revive the PR at any time and @mention a reviewer 
or discuss it on the d...@beam.apache.org list. Thank you for your 
contributions.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 152015)
Time Spent: 1h 20m  (was: 1h 10m)

> Beam Dependency Update Request: org.codehaus.groovy:groovy-all 3.0.0-alpha-3
> 
>
> Key: BEAM-4937
> URL: https://issues.apache.org/jira/browse/BEAM-4937
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Assignee: yifan zou
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> 2018-07-25 20:26:45.609047
> Please review and upgrade the org.codehaus.groovy:groovy-all to the 
> latest version 3.0.0-alpha-3 
>  
> cc: 
> 2018-08-06 12:11:13.609965
> Please review and upgrade the org.codehaus.groovy:groovy-all to the 
> latest version 3.0.0-alpha-3 
>  
> cc: 
> 2018-08-13 12:12:40.581247
> Please review and upgrade the org.codehaus.groovy:groovy-all to the 
> latest version 3.0.0-alpha-3 
>  
> cc: 
> 2018-08-20 12:13:41.389656
> Please review and upgrade the org.codehaus.groovy:groovy-all to the 
> latest version 3.0.0-alpha-3 
>  
> cc: 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch asf-site updated: Publishing website 2018/10/07 00:03:21 at commit b2ad757

[git-site-role]

This is an automated email from the ASF dual-hosted git repository.

git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/asf-site by this push:
 new b84b825  Publishing website 2018/10/07 00:03:21 at commit b2ad757
b84b825 is described below

commit b84b825e43adee9bd3a424a62f238774fd323621
Author: jenkins 
AuthorDate: Sun Oct 7 00:03:21 2018 +

Publishing website 2018/10/07 00:03:21 at commit b2ad757

[beam] branch asf-site updated: Publishing website 2018/10/07 00:02:34 at commit 324f0b3

[git-site-role]

This is an automated email from the ASF dual-hosted git repository.

git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/asf-site by this push:
 new abd4390  Publishing website 2018/10/07 00:02:34 at commit 324f0b3
abd4390 is described below

commit abd439043b61a5d08e7682d7a3224b4e95e4ea41
Author: jenkins 
AuthorDate: Sun Oct 7 00:02:35 2018 +

Publishing website 2018/10/07 00:02:34 at commit 324f0b3

Build failed in Jenkins: beam_PreCommit_Website_Cron #148

[Apache Jenkins Server]

See 


--
[...truncated 170.42 KB...]
  *  External link http://images/logos/runners/spark.png failed: response code 
0 means something's wrong.
 It's possible libcurl couldn't connect to the server or perhaps 
the request timed out.
 Sometimes, making too many requests at once also breaks things.
 Either way, the return message (if any) from the server is: 
Couldn't resolve host name
  *  External link http://images/logos/sdks/go.png failed: response code 0 
means something's wrong.
 It's possible libcurl couldn't connect to the server or perhaps 
the request timed out.
 Sometimes, making too many requests at once also breaks things.
 Either way, the return message (if any) from the server is: 
Couldn't resolve host name
  *  External link http://images/logos/sdks/java.png failed: response code 0 
means something's wrong.
 It's possible libcurl couldn't connect to the server or perhaps 
the request timed out.
 Sometimes, making too many requests at once also breaks things.
 Either way, the return message (if any) from the server is: 
Couldn't resolve host name
  *  External link http://images/logos/sdks/python.png failed: response code 0 
means something's wrong.
 It's possible libcurl couldn't connect to the server or perhaps 
the request timed out.
 Sometimes, making too many requests at once also breaks things.
 Either way, the return message (if any) from the server is: 
Couldn't resolve host name
  *  External link http://images/logos/sdks/scala.png failed: response code 0 
means something's wrong.
 It's possible libcurl couldn't connect to the server or perhaps 
the request timed out.
 Sometimes, making too many requests at once also breaks things.
 Either way, the return message (if any) from the server is: 
Couldn't resolve host name
  *  http:// is an invalid URL (line 77)
 

  
- ./generated-content/get-started/downloads/index.html
  *  http:// is an invalid URL (line 77)
 

  
- ./generated-content/get-started/index.html
  *  External link http://get-started/beam-overview failed: response code 0 
means something's wrong.
 It's possible libcurl couldn't connect to the server or perhaps 
the request timed out.
 Sometimes, making too many requests at once also breaks things.
 Either way, the return message (if any) from the server is: 
Couldn't resolve host name
  *  External link http://get-started/mobile-gaming-example failed: response 
code 0 means something's wrong.
 It's possible libcurl couldn't connect to the server or perhaps 
the request timed out.
 Sometimes, making too many requests at once also breaks things.
 Either way, the return message (if any) from the server is: 
Couldn't resolve host name
  *  http:// is an invalid URL (line 77)
 

  
- ./generated-content/get-started/mobile-gaming-example/index.html
  *  External link http://documentation/programming-guide/ failed: response 
code 0 means something's wrong.
 It's possible libcurl couldn't connect to the server or perhaps 
the request timed out.
 Sometimes, making too many requests at once also breaks things.
 Either way, the return message (if any) from the server is: 
Couldn't resolve host name
  *  External link http://documentation/programming-guide/ failed: response 
code 0 means something's wrong.
 It's possible libcurl couldn't connect to the server or perhaps 
the request timed out.
 Sometimes, making too many requests at once also breaks things.
 Either way, the return message (if any) from the server is: 
Couldn't resolve host name
  *  External link http://images/gaming-example-basic.png failed: response code 
0 means something's wrong.
 It's possible libcurl couldn't connect to the server or perhaps 
the request timed out.
 Sometimes, making too many requests at once also breaks things.
 Either way, the return message (if any) from the server is: 
Couldn't resolve host name
  *  External link http://images/gaming-example-event-time-narrow.gif failed: 
response code 0 means something's wrong.
 It's possible libcurl couldn't connect to the server or perhaps 
the request timed out.
 Sometimes, making too many requests at once also breaks things.
 Either way, the return message (if any) from the server is: 
Couldn't resolve host name
  *  External link http://images/gaming-example-proc-time-narrow.gif failed: 
response code 0 means something's wrong.
 It's possible libcurl couldn't connect to the server or perhaps 
the request timed out.
 

[jira] [Work logged] (BEAM-5653) Dataflow FnApi Worker overrides some of Coders due to coder ID generation collision.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5653?focusedWorklogId=152014=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-152014
 ]

ASF GitHub Bot logged work on BEAM-5653:


Author: ASF GitHub Bot
Created on: 07/Oct/18 00:01
Start Date: 07/Oct/18 00:01
Worklog Time Spent: 10m 
  Work Description: pabloem commented on issue #6578:  [BEAM-5653] Allow 
initialize SdkComponents with existing Components list.
URL: https://github.com/apache/beam/pull/6578#issuecomment-427614964
 
 
   Merged as it had been approved by Alex. Thanks.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 152014)
Time Spent: 1.5h  (was: 1h 20m)
Remaining Estimate: 70.5h  (was: 70h 40m)

> Dataflow FnApi Worker overrides some of Coders due to coder ID generation 
> collision.
> 
>
> Key: BEAM-5653
> URL: https://issues.apache.org/jira/browse/BEAM-5653
> Project: Beam
>  Issue Type: Test
>  Components: java-fn-execution
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Blocker
> Fix For: 2.8.0
>
>   Original Estimate: 72h
>  Time Spent: 1.5h
>  Remaining Estimate: 70.5h
>
> Due to one of latest refactorings, we got a bug in Java FnApi Worker that it 
> overrides Coders in ProcessBundleDescriptor sent to SDK Harness that causes 
> jobs to fail.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Merge pull request #6578 from Ardagan/b111881785

[pabloem]

This is an automated email from the ASF dual-hosted git repository.

pabloem pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit b2ad757876ee05e73a3b4ce805530a02baa238f9
Merge: 324f0b3 d0f44ed
Author: Pablo 
AuthorDate: Sat Oct 6 17:01:37 2018 -0700

Merge pull request #6578 from Ardagan/b111881785

 [BEAM-5653] Allow initialize SdkComponents with existing Components list.

 .../runners/core/construction/SdkComponents.java   | 48 +++---
 1 file changed, 34 insertions(+), 14 deletions(-)

[beam] branch master updated (324f0b3 -> b2ad757)

[pabloem]

This is an automated email from the ASF dual-hosted git repository.

pabloem pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 324f0b3  Merge pull request #6564 from udim/pubsub-0-35-4
 add d0f44ed  [BEAM-5653] Allow initialize SdkComponents with existing 
Components list.
 new b2ad757  Merge pull request #6578 from Ardagan/b111881785

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../runners/core/construction/SdkComponents.java   | 48 +++---
 1 file changed, 34 insertions(+), 14 deletions(-)

[jira] [Work logged] (BEAM-5653) Dataflow FnApi Worker overrides some of Coders due to coder ID generation collision.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5653?focusedWorklogId=152013=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-152013
 ]

ASF GitHub Bot logged work on BEAM-5653:


Author: ASF GitHub Bot
Created on: 07/Oct/18 00:01
Start Date: 07/Oct/18 00:01
Worklog Time Spent: 10m 
  Work Description: pabloem closed pull request #6578:  [BEAM-5653] Allow 
initialize SdkComponents with existing Components list.
URL: https://github.com/apache/beam/pull/6578
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/SdkComponents.java
 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/SdkComponents.java
index 4f73a7b243a..65c398ce78e 100644
--- 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/SdkComponents.java
+++ 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/SdkComponents.java
@@ -26,6 +26,7 @@
 import com.google.common.collect.HashBiMap;
 import com.google.common.collect.Iterables;
 import java.io.IOException;
+import java.util.HashSet;
 import java.util.List;
 import java.util.Set;
 import org.apache.beam.model.pipeline.v1.RunnerApi;
@@ -43,22 +44,34 @@
 
 /** SDK objects that will be represented at some later point within a {@link 
Components} object. */
 public class SdkComponents {
-  private final RunnerApi.Components.Builder componentsBuilder;
+  private final RunnerApi.Components.Builder componentsBuilder = 
RunnerApi.Components.newBuilder();
 
-  private final BiMap, String> transformIds;
-  private final BiMap, String> pCollectionIds;
-  private final BiMap, String> windowingStrategyIds;
+  private final BiMap, String> transformIds = 
HashBiMap.create();
+  private final BiMap, String> pCollectionIds = 
HashBiMap.create();
+  private final BiMap, String> windowingStrategyIds = 
HashBiMap.create();
 
   /** A map of Coder to IDs. Coders are stored here with identity equivalence. 
*/
-  private final BiMap>, String> 
coderIds;
+  private final BiMap>, String> 
coderIds =
+  HashBiMap.create();
 
-  private final BiMap environmentIds;
+  private final BiMap environmentIds = HashBiMap.create();
+
+  private final Set reservedIds = new HashSet<>();
 
   /** Create a new {@link SdkComponents} with no components. */
   public static SdkComponents create() {
 return new SdkComponents();
   }
 
+  /**
+   * Create new {@link SdkComponents} importing all items from provided {@link 
Components} object.
+   *
+   * WARNING: This action might cause some of duplicate items created.
+   */
+  public static SdkComponents create(RunnerApi.Components components) {
+return new SdkComponents(components);
+  }
+
   public static SdkComponents create(PipelineOptions options) {
 SdkComponents sdkComponents = new SdkComponents();
 PortablePipelineOptions portablePipelineOptions = 
options.as(PortablePipelineOptions.class);
@@ -69,13 +82,20 @@ public static SdkComponents create(PipelineOptions options) 
{
 return sdkComponents;
   }
 
-  private SdkComponents() {
-this.componentsBuilder = RunnerApi.Components.newBuilder();
-this.transformIds = HashBiMap.create();
-this.pCollectionIds = HashBiMap.create();
-this.windowingStrategyIds = HashBiMap.create();
-this.coderIds = HashBiMap.create();
-this.environmentIds = HashBiMap.create();
+  private SdkComponents() {}
+
+  private SdkComponents(RunnerApi.Components components) {
+if (components == null) {
+  return;
+}
+
+reservedIds.addAll(components.getTransformsMap().keySet());
+reservedIds.addAll(components.getPcollectionsMap().keySet());
+reservedIds.addAll(components.getWindowingStrategiesMap().keySet());
+reservedIds.addAll(components.getCodersMap().keySet());
+reservedIds.addAll(components.getEnvironmentsMap().keySet());
+
+componentsBuilder.mergeFrom(components);
   }
 
   /**
@@ -222,7 +242,7 @@ public String getOnlyEnvironmentId() {
   private String uniqify(String baseName, Set existing) {
 String name = baseName;
 int increment = 1;
-while (existing.contains(name)) {
+while (existing.contains(name) || reservedIds.contains(name)) {
   name = baseName + Integer.toString(increment);
   increment++;
 }


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:

Build failed in Jenkins: beam_PreCommit_Website_Stage_GCS_Cron #10

[Apache Jenkins Server]

See 


--
[...truncated 6.95 KB...]
:buildSrc:jar (Thread[Task worker for ':buildSrc' Thread 4,5,main]) started.

> Task :buildSrc:jar
Build cache key for task ':buildSrc:jar' is 7445e5c45b21f8a690f2f547fcb49594
Caching disabled for task ':buildSrc:jar': Caching has not been enabled for the 
task
Task ':buildSrc:jar' is not up-to-date because:
  No history is available.
:buildSrc:jar (Thread[Task worker for ':buildSrc' Thread 4,5,main]) completed. 
Took 0.102 secs.
:buildSrc:assemble (Thread[Task worker for ':buildSrc' Thread 4,5,main]) 
started.

> Task :buildSrc:assemble
Skipping task ':buildSrc:assemble' as it has no actions.
:buildSrc:assemble (Thread[Task worker for ':buildSrc' Thread 4,5,main]) 
completed. Took 0.0 secs.
:buildSrc:spotlessGroovy (Thread[Task worker for ':buildSrc' Thread 4,5,main]) 
started.

> Task :buildSrc:spotlessGroovy
file or directory 
'
 not found
file or directory 
'
 not found
file or directory 
'
 not found
Caching disabled for task ':buildSrc:spotlessGroovy': Caching has not been 
enabled for the task
Task ':buildSrc:spotlessGroovy' is not up-to-date because:
  No history is available.
All input files are considered out-of-date for incremental task 
':buildSrc:spotlessGroovy'.
file or directory 
'
 not found
:buildSrc:spotlessGroovy (Thread[Task worker for ':buildSrc' Thread 4,5,main]) 
completed. Took 1.276 secs.
:buildSrc:spotlessGroovyCheck (Thread[Task worker for ':buildSrc' Thread 
4,5,main]) started.

> Task :buildSrc:spotlessGroovyCheck
Skipping task ':buildSrc:spotlessGroovyCheck' as it has no actions.
:buildSrc:spotlessGroovyCheck (Thread[Task worker for ':buildSrc' Thread 
4,5,main]) completed. Took 0.001 secs.
:buildSrc:spotlessGroovyGradle (Thread[Task worker for ':buildSrc' Thread 
4,5,main]) started.

> Task :buildSrc:spotlessGroovyGradle
Caching disabled for task ':buildSrc:spotlessGroovyGradle': Caching has not 
been enabled for the task
Task ':buildSrc:spotlessGroovyGradle' is not up-to-date because:
  No history is available.
All input files are considered out-of-date for incremental task 
':buildSrc:spotlessGroovyGradle'.
:buildSrc:spotlessGroovyGradle (Thread[Task worker for ':buildSrc' Thread 
4,5,main]) completed. Took 0.029 secs.
:buildSrc:spotlessGroovyGradleCheck (Thread[Task worker for ':buildSrc' Thread 
4,5,main]) started.

> Task :buildSrc:spotlessGroovyGradleCheck
Skipping task ':buildSrc:spotlessGroovyGradleCheck' as it has no actions.
:buildSrc:spotlessGroovyGradleCheck (Thread[Task worker for ':buildSrc' Thread 
4,5,main]) completed. Took 0.0 secs.
:buildSrc:spotlessCheck (Thread[Task worker for ':buildSrc' Thread 4,5,main]) 
started.

> Task :buildSrc:spotlessCheck
Skipping task ':buildSrc:spotlessCheck' as it has no actions.
:buildSrc:spotlessCheck (Thread[Task worker for ':buildSrc' Thread 4,5,main]) 
completed. Took 0.0 secs.
:buildSrc:compileTestJava (Thread[Task worker for ':buildSrc' Thread 4,5,main]) 
started.

> Task :buildSrc:compileTestJava NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:compileTestJava' as it has no source files and no 
previous output files.
:buildSrc:compileTestJava (Thread[Task worker for ':buildSrc' Thread 4,5,main]) 
completed. Took 0.002 secs.
:buildSrc:compileTestGroovy (Thread[Task worker for ':buildSrc' Thread 
4,5,main]) started.

> Task :buildSrc:compileTestGroovy NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:compileTestGroovy' as it has no source files and no 
previous output files.
:buildSrc:compileTestGroovy (Thread[Task worker for ':buildSrc' Thread 
4,5,main]) completed. Took 0.003 secs.
:buildSrc:processTestResources (Thread[Task worker for ':buildSrc' Thread 
4,5,main]) started.

> Task :buildSrc:processTestResources NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:processTestResources' as it has no source files and no 
previous output files.
:buildSrc:processTestResources (Thread[Task worker for ':buildSrc' Thread 
4,5,main]) completed. Took 0.001 secs.
:buildSrc:testClasses (Thread[Task worker for ':buildSrc' Thread 4,5,main]) 
started.

> Task :buildSrc:testClasses UP-TO-DATE
Skipping task 

[jira] [Work logged] (BEAM-4049) Improve write throughput of CassandraIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4049?focusedWorklogId=152011=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-152011
 ]

ASF GitHub Bot logged work on BEAM-4049:


Author: ASF GitHub Bot
Created on: 06/Oct/18 22:47
Start Date: 06/Oct/18 22:47
Worklog Time Spent: 10m 
  Work Description: asethia commented on issue #5112: [BEAM-4049] Improve 
CassandraIO write throughput by performing async queries
URL: https://github.com/apache/beam/pull/5112#issuecomment-427611649
 
 
   any update .. I am also getting similar error with following libaries: 
   
   1. beam-sdks-java-io-cassandra - 2.5.0 
   2. google-cloud-dataflow-java-sdk-all - 2.5.0
   
   Exception in thread "main" 
org.apache.beam.sdk.Pipeline$PipelineExecutionException: 
java.lang.NoSuchMethodError: 
com.datastax.driver.mapping.Mapper.saveAsync(Ljava/lang/Object;)Lorg/apache/beam/repackaged/beam_sdks_java_io_cassandra/com/google/common/util/concurrent/ListenableFuture;
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 152011)
Time Spent: 7h 40m  (was: 7.5h)

> Improve write throughput of CassandraIO
> ---
>
> Key: BEAM-4049
> URL: https://issues.apache.org/jira/browse/BEAM-4049
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-cassandra
>Affects Versions: 2.4.0
>Reporter: Alexander Dejanovski
>Assignee: Alexander Dejanovski
>Priority: Major
>  Labels: performance
> Fix For: 2.5.0
>
>  Time Spent: 7h 40m
>  Remaining Estimate: 0h
>
> The CassandraIO currently uses the mapper to perform writes in a synchronous 
> fashion. 
> This implies that writes are serialized and is a very suboptimal way of 
> writing to Cassandra.
> The IO should use the saveAsync() method instead of save() and should wait 
> for completion each time 100 queries are in flight, in order to avoid 
> overwhelming clusters.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-5620) Some tests use assertItemsEqual method, not available in Python 3

[Matthias Feys (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias Feys reassigned BEAM-5620:
---

Assignee: Matthias Feys

> Some tests use assertItemsEqual method, not available in Python 3
> -
>
> Key: BEAM-5620
> URL: https://issues.apache.org/jira/browse/BEAM-5620
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Matthias Feys
>Priority: Major
>
> See: 
> https://github.com/apache/beam/search?q=assertItemsEqual_q=assertItemsEqual



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5638) Add exception handling to single message transforms in Java SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5638?focusedWorklogId=152006=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-152006
 ]

ASF GitHub Bot logged work on BEAM-5638:


Author: ASF GitHub Bot
Created on: 06/Oct/18 21:23
Start Date: 06/Oct/18 21:23
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on a change in pull request #6586: 
[BEAM-5638] Exception handling for Java single message transforms
URL: https://github.com/apache/beam/pull/6586#discussion_r223194112
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Failure.java
 ##
 @@ -0,0 +1,101 @@
+package org.apache.beam.sdk.transforms;
+
+import com.google.auto.value.AutoValue;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ObjectArrays;
+import java.io.Serializable;
+import java.util.List;
+import org.apache.beam.sdk.coders.SerializableCoder;
+import org.apache.beam.sdk.transforms.DoFn.MultiOutputReceiver;
+import org.apache.beam.sdk.values.PCollectionTuple;
+import org.apache.beam.sdk.values.TupleTag;
+import org.apache.beam.sdk.values.TupleTagList;
+import org.apache.beam.sdk.values.TypeDescriptor;
+
+/**
+ * Wraps an exception along with an input value; this is the element type of 
failure collections
+ * returned by single message transforms configured to catch exceptions.
+ *
+ * @param  type of the wrapped input value that caused an exception to be 
raised
+ */
+@AutoValue
+public abstract class Failure implements Serializable {
+  public static  Failure of(Exception exception, T value) {
+return new AutoValue_Failure<>(exception, value);
+  }
+
+  public abstract Exception exception();
+
+  public abstract T value();
+
+  /**
+   * Internal class for collecting tuple tags associated with collections of 
{@link Exception}
+   * classes that should route to them. Also contains helper methods to 
simplify implementation of
+   * the {@code WithFailures} nested classes of {@link MapElements}, {@link 
FlatMapElements}, etc.
+   */
+  @AutoValue
+  abstract static class TaggedExceptionsList implements Serializable {
+abstract ImmutableList>> tags();
+
+abstract ImmutableList>> exceptionLists();
+
+static  TaggedExceptionsList empty() {
+  return new AutoValue_Failure_TaggedExceptionsList<>(ImmutableList.of(), 
ImmutableList.of());
+}
+
+/**
+ * Return a new {@link TaggedExceptionsList} that has all the tags and 
exceptions of this {@link
+ * TaggedExceptionsList} plus a new element representing the arguments 
passed in here.
+ */
+TaggedExceptionsList and(
+TupleTag> tag, Class exceptionToCatch, Class[] 
additionalExceptions) {
+  final ImmutableList>> newTags =
+  
ImmutableList.>>builder().addAll(tags()).add(tag).build();
+  final ImmutableList>> newExceptionLists =
+  ImmutableList.>>builder()
+  .addAll(exceptionLists())
+  .add(
+  ImmutableList.copyOf(ObjectArrays.concat(exceptionToCatch, 
additionalExceptions)))
+  .build();
+  return new AutoValue_Failure_TaggedExceptionsList<>(newTags, 
newExceptionLists);
+}
+
+/** Return the internal typed list of tags as an untyped {@link 
TupleTagList}. */
+TupleTagList tupleTagList() {
+  TupleTagList l = TupleTagList.empty();
+  for (TupleTag tag : tags()) {
+l = l.and(tag);
+  }
+  return l;
+}
+
+/**
+ * Check the registered exception classes to see if the exception passed 
in here matches. If it
+ * does, wrap the exception and value together in a {@link Failure} and 
send to the output
+ * receiver. If not, rethrow so processing stops on the unexpected failure.
+ */
+void outputOrRethrow(Exception e, T value, MultiOutputReceiver receiver) 
throws Exception {
+  for (int i = 0; i < tags().size(); i++) {
+for (Class cls : exceptionLists().get(i)) {
+  if (cls.isInstance(e)) {
+receiver.get(tags().get(i)).output(Failure.of(e, value));
+return;
+  }
+}
+  }
+  throw e;
+}
+
+/**
+ * Set appropriate coders on all the failure collections in the given 
{@link PCollectionTuple}.
+ */
+PCollectionTuple applyFailureCoders(PCollectionTuple pcs) {
+  final SerializableCoder> failureCoder =
+  SerializableCoder.of(new TypeDescriptor>() {});
 
 Review comment:
   I think it's somewhat common for T elements to not be serializable actually. 
The T coder you can get from the PCollection in expand; you will need to change 
the WithFailures input type to be PCollection instead of PCollection. This still leave us with SerializableCoder for Exception, 
which is not ideal but I can't think of a better option.


This is an automated message from 

[jira] [Work logged] (BEAM-5653) Dataflow FnApi Worker overrides some of Coders due to coder ID generation collision.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5653?focusedWorklogId=152001=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-152001
 ]

ASF GitHub Bot logged work on BEAM-5653:


Author: ASF GitHub Bot
Created on: 06/Oct/18 19:48
Start Date: 06/Oct/18 19:48
Worklog Time Spent: 10m 
  Work Description: Ardagan commented on issue #6578:  [BEAM-5653] Allow 
initialize SdkComponents with existing Components list.
URL: https://github.com/apache/beam/pull/6578#issuecomment-427601422
 
 
   R: @pabloem (committer)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 152001)
Time Spent: 1h 10m  (was: 1h)
Remaining Estimate: 70h 50m  (was: 71h)

> Dataflow FnApi Worker overrides some of Coders due to coder ID generation 
> collision.
> 
>
> Key: BEAM-5653
> URL: https://issues.apache.org/jira/browse/BEAM-5653
> Project: Beam
>  Issue Type: Test
>  Components: java-fn-execution
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Blocker
> Fix For: 2.8.0
>
>   Original Estimate: 72h
>  Time Spent: 1h 10m
>  Remaining Estimate: 70h 50m
>
> Due to one of latest refactorings, we got a bug in Java FnApi Worker that it 
> overrides Coders in ProcessBundleDescriptor sent to SDK Harness that causes 
> jobs to fail.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4049) Improve write throughput of CassandraIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4049?focusedWorklogId=152000=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-152000
 ]

ASF GitHub Bot logged work on BEAM-4049:


Author: ASF GitHub Bot
Created on: 06/Oct/18 18:58
Start Date: 06/Oct/18 18:58
Worklog Time Spent: 10m 
  Work Description: Arqu edited a comment on issue #5112: [BEAM-4049] 
Improve CassandraIO write throughput by performing async queries
URL: https://github.com/apache/beam/pull/5112#issuecomment-427596222
 
 
   @script3r I'm having the same issue right now. Any progress on it?
   I've removed all relevant local jars but still get the same issue. Currently 
running on 2.5.0. Didn't happen on 2.3.0.
   
   Edit: In hindsight, it's obvious it worked on 2.3 as we didn't use this 
implementation.
   
   Edit2: For what it's worth, I've double checked, and the correct version is 
in the path and actually has the function. Not sure what it maps to at runtime 
and why it fails.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 152000)
Time Spent: 7.5h  (was: 7h 20m)

> Improve write throughput of CassandraIO
> ---
>
> Key: BEAM-4049
> URL: https://issues.apache.org/jira/browse/BEAM-4049
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-cassandra
>Affects Versions: 2.4.0
>Reporter: Alexander Dejanovski
>Assignee: Alexander Dejanovski
>Priority: Major
>  Labels: performance
> Fix For: 2.5.0
>
>  Time Spent: 7.5h
>  Remaining Estimate: 0h
>
> The CassandraIO currently uses the mapper to perform writes in a synchronous 
> fashion. 
> This implies that writes are serialized and is a very suboptimal way of 
> writing to Cassandra.
> The IO should use the saveAsync() method instead of save() and should wait 
> for completion each time 100 queries are in flight, in order to avoid 
> overwhelming clusters.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PerformanceTests_ParquetIOIT #571

[Apache Jenkins Server]

See 

[jira] [Assigned] (BEAM-5627) Several IO tests fail in Python 3 when accessing a temporary file with TypeError: a bytes-like object is required, not 'str'

[Rakesh Kumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rakesh Kumar reassigned BEAM-5627:
--

Assignee: Rakesh Kumar

> Several IO tests fail in Python 3  when accessing a temporary file with  
> TypeError: a bytes-like object is required, not 'str'
> --
>
> Key: BEAM-5627
> URL: https://issues.apache.org/jira/browse/BEAM-5627
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Rakesh Kumar
>Priority: Major
>
> ERROR: test_split_at_fraction_exhaustive 
> (apache_beam.io.source_test_utils_test.SourceTestUtilsTest)
>  --
>  Traceback (most recent call last):
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils_test.py",
>  line 120, in test_split_at_fraction_exhaustive
>  source = self._create_source(data)
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils_test.py",
>  line 43, in _create_source
>  source = LineSource(self._create_file_with_data(data))
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils_test.py",
>  line 35, in _create_file_with_data
>  f.write(line + '\n')
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/target/.tox/py3/lib/python3.5/tempfile.py",
>  line 622, in func_wrapper
>  return func(*args, **kwargs)
> TypeError: a bytes-like object is required, not 'str'
> Also similar:
> ==
>  ERROR: test_file_sink_writing 
> (apache_beam.io.filebasedsink_test.TestFileBasedSink)
> --
> Traceback (most recent call last):
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink_test.py", line 121, in 
> test_file_sink_writing
>   init_token, writer_results = self._common_init(sink)
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink_test.py", line 103, in _common_init
>   writer1 = sink.open_writer(init_token, '1')
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/options/value_provider.py", line 133, in _f
>   return fnc(self, *args, **kwargs)
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink.py", line 185, in open_writer
> return FileBasedSinkWriter(self, os.path.join(init_result, uid) + suffix)
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink.py", line 385, in __init__
>   self.temp_handle = self.sink.open(temp_shard_path)
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink_test.py", line 82, in open
>   file_handle.write('[start]')
>   TypeError: a bytes-like object is required, not 'str'



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4049) Improve write throughput of CassandraIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4049?focusedWorklogId=151999=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151999
 ]

ASF GitHub Bot logged work on BEAM-4049:


Author: ASF GitHub Bot
Created on: 06/Oct/18 18:31
Start Date: 06/Oct/18 18:31
Worklog Time Spent: 10m 
  Work Description: Arqu edited a comment on issue #5112: [BEAM-4049] 
Improve CassandraIO write throughput by performing async queries
URL: https://github.com/apache/beam/pull/5112#issuecomment-427596222
 
 
   @script3r I'm having the same issue right now. Any progress on it?
   I've removed all relevant local jars but still get the same issue. Currently 
running on 2.5.0. Didn't happen on 2.3.0.
   
   Edit: In hindsight, it's obvious it worked on 2.3 as we didn't use this 
implementation.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151999)
Time Spent: 7h 20m  (was: 7h 10m)

> Improve write throughput of CassandraIO
> ---
>
> Key: BEAM-4049
> URL: https://issues.apache.org/jira/browse/BEAM-4049
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-cassandra
>Affects Versions: 2.4.0
>Reporter: Alexander Dejanovski
>Assignee: Alexander Dejanovski
>Priority: Major
>  Labels: performance
> Fix For: 2.5.0
>
>  Time Spent: 7h 20m
>  Remaining Estimate: 0h
>
> The CassandraIO currently uses the mapper to perform writes in a synchronous 
> fashion. 
> This implies that writes are serialized and is a very suboptimal way of 
> writing to Cassandra.
> The IO should use the saveAsync() method instead of save() and should wait 
> for completion each time 100 queries are in flight, in order to avoid 
> overwhelming clusters.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5626) Several IO tests fail in Python 3 with RuntimeError('dictionary changed size during iteration',)}

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5626?focusedWorklogId=151998=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151998
 ]

ASF GitHub Bot logged work on BEAM-5626:


Author: ASF GitHub Bot
Created on: 06/Oct/18 18:30
Start Date: 06/Oct/18 18:30
Worklog Time Spent: 10m 
  Work Description: HuangLED commented on a change in pull request #6587: 
[BEAM-5626] Fix hadoop filesystem test for py3.
URL: https://github.com/apache/beam/pull/6587#discussion_r223157210
 
 

 ##
 File path: sdks/python/apache_beam/io/hadoopfilesystem_test.py
 ##
 @@ -214,6 +214,11 @@ def setUp(self):
   url = self.fs.join(self.tmpdir, filename)
   self.fs.create(url).close()
 
+try:# Python 2
 
 Review comment:
   @aaltay, sure.  just to clarify.   Did you mean move this to top of this 
Setup() function? 
   
   @tvalentyn I might be wrong, but I think relying on sys.version_info[0] is 
more indirect way of verifying things, thus less preferred to me.  And in test 
file a bit code redundancy for self-containing makes sense to me. just my 2 
cents. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151998)
Time Spent: 2.5h  (was: 2h 20m)

> Several IO tests fail in Python 3 with RuntimeError('dictionary changed size 
> during iteration',)}
> -
>
> Key: BEAM-5626
> URL: https://issues.apache.org/jira/browse/BEAM-5626
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Ruoyun Huang
>Priority: Major
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
>  ERROR: test_delete_dir 
> (apache_beam.io.hadoopfilesystem_test.HadoopFileSystemTest)
> --
> Traceback (most recent call last):
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/hadoopfilesystem_test.py",
>  line 506, in test_delete_dir
>  self.fs.delete([url_t1])
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/hadoopfilesystem.py",
>  line 370, in delete
>  raise BeamIOError("Delete operation failed", exceptions)
>  apache_beam.io.filesystem.BeamIOError: Delete operation failed with 
> exceptions {'hdfs://test_dir/new_dir1': RuntimeError('dictionary changed size 
> during iteration',   )}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5638) Add exception handling to single message transforms in Java SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5638?focusedWorklogId=151997=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151997
 ]

ASF GitHub Bot logged work on BEAM-5638:


Author: ASF GitHub Bot
Created on: 06/Oct/18 18:29
Start Date: 06/Oct/18 18:29
Worklog Time Spent: 10m 
  Work Description: tims commented on a change in pull request #6586: 
[BEAM-5638] Exception handling for Java single message transforms
URL: https://github.com/apache/beam/pull/6586#discussion_r223189848
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Failure.java
 ##
 @@ -0,0 +1,101 @@
+package org.apache.beam.sdk.transforms;
+
+import com.google.auto.value.AutoValue;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ObjectArrays;
+import java.io.Serializable;
+import java.util.List;
+import org.apache.beam.sdk.coders.SerializableCoder;
+import org.apache.beam.sdk.transforms.DoFn.MultiOutputReceiver;
+import org.apache.beam.sdk.values.PCollectionTuple;
+import org.apache.beam.sdk.values.TupleTag;
+import org.apache.beam.sdk.values.TupleTagList;
+import org.apache.beam.sdk.values.TypeDescriptor;
+
+/**
+ * Wraps an exception along with an input value; this is the element type of 
failure collections
+ * returned by single message transforms configured to catch exceptions.
+ *
+ * @param  type of the wrapped input value that caused an exception to be 
raised
+ */
+@AutoValue
+public abstract class Failure implements Serializable {
+  public static  Failure of(Exception exception, T value) {
+return new AutoValue_Failure<>(exception, value);
+  }
+
+  public abstract Exception exception();
+
+  public abstract T value();
+
+  /**
+   * Internal class for collecting tuple tags associated with collections of 
{@link Exception}
+   * classes that should route to them. Also contains helper methods to 
simplify implementation of
+   * the {@code WithFailures} nested classes of {@link MapElements}, {@link 
FlatMapElements}, etc.
+   */
+  @AutoValue
+  abstract static class TaggedExceptionsList implements Serializable {
+abstract ImmutableList>> tags();
+
+abstract ImmutableList>> exceptionLists();
+
+static  TaggedExceptionsList empty() {
+  return new AutoValue_Failure_TaggedExceptionsList<>(ImmutableList.of(), 
ImmutableList.of());
+}
+
+/**
+ * Return a new {@link TaggedExceptionsList} that has all the tags and 
exceptions of this {@link
+ * TaggedExceptionsList} plus a new element representing the arguments 
passed in here.
+ */
+TaggedExceptionsList and(
+TupleTag> tag, Class exceptionToCatch, Class[] 
additionalExceptions) {
+  final ImmutableList>> newTags =
+  
ImmutableList.>>builder().addAll(tags()).add(tag).build();
+  final ImmutableList>> newExceptionLists =
+  ImmutableList.>>builder()
+  .addAll(exceptionLists())
+  .add(
+  ImmutableList.copyOf(ObjectArrays.concat(exceptionToCatch, 
additionalExceptions)))
+  .build();
+  return new AutoValue_Failure_TaggedExceptionsList<>(newTags, 
newExceptionLists);
+}
+
+/** Return the internal typed list of tags as an untyped {@link 
TupleTagList}. */
+TupleTagList tupleTagList() {
+  TupleTagList l = TupleTagList.empty();
+  for (TupleTag tag : tags()) {
+l = l.and(tag);
+  }
+  return l;
+}
+
+/**
+ * Check the registered exception classes to see if the exception passed 
in here matches. If it
+ * does, wrap the exception and value together in a {@link Failure} and 
send to the output
+ * receiver. If not, rethrow so processing stops on the unexpected failure.
+ */
+void outputOrRethrow(Exception e, T value, MultiOutputReceiver receiver) 
throws Exception {
+  for (int i = 0; i < tags().size(); i++) {
+for (Class cls : exceptionLists().get(i)) {
+  if (cls.isInstance(e)) {
+receiver.get(tags().get(i)).output(Failure.of(e, value));
+return;
+  }
+}
+  }
+  throw e;
+}
+
+/**
+ * Set appropriate coders on all the failure collections in the given 
{@link PCollectionTuple}.
+ */
+PCollectionTuple applyFailureCoders(PCollectionTuple pcs) {
+  final SerializableCoder> failureCoder =
+  SerializableCoder.of(new TypeDescriptor>() {});
 
 Review comment:
   I implemented less general exception handling recently just as a base class 
for my DoFns, I ended up having a proto that contains the stack trace and other 
info just as strings. Also because I was worried about the SerializableCoder 
efficiency.
   
   One option might be for the failure output type be a generic? And we provide 
a function for handling the exception instead?
   
   eg withFailureHandler(TupleTag failureTag, 

[jira] [Work logged] (BEAM-4049) Improve write throughput of CassandraIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4049?focusedWorklogId=151996=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151996
 ]

ASF GitHub Bot logged work on BEAM-4049:


Author: ASF GitHub Bot
Created on: 06/Oct/18 18:29
Start Date: 06/Oct/18 18:29
Worklog Time Spent: 10m 
  Work Description: Arqu commented on issue #5112: [BEAM-4049] Improve 
CassandraIO write throughput by performing async queries
URL: https://github.com/apache/beam/pull/5112#issuecomment-427596222
 
 
   @script3r I'm having the same issue right now. Any progress on it?
   I've removed all relevant local jars but still get the same issue. Currently 
running on 2.5.0. Didn't happen on 2.3.0.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151996)
Time Spent: 7h 10m  (was: 7h)

> Improve write throughput of CassandraIO
> ---
>
> Key: BEAM-4049
> URL: https://issues.apache.org/jira/browse/BEAM-4049
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-cassandra
>Affects Versions: 2.4.0
>Reporter: Alexander Dejanovski
>Assignee: Alexander Dejanovski
>Priority: Major
>  Labels: performance
> Fix For: 2.5.0
>
>  Time Spent: 7h 10m
>  Remaining Estimate: 0h
>
> The CassandraIO currently uses the mapper to perform writes in a synchronous 
> fashion. 
> This implies that writes are serialized and is a very suboptimal way of 
> writing to Cassandra.
> The IO should use the saveAsync() method instead of save() and should wait 
> for completion each time 100 queries are in flight, in order to avoid 
> overwhelming clusters.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5638) Add exception handling to single message transforms in Java SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5638?focusedWorklogId=151995=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151995
 ]

ASF GitHub Bot logged work on BEAM-5638:


Author: ASF GitHub Bot
Created on: 06/Oct/18 18:27
Start Date: 06/Oct/18 18:27
Worklog Time Spent: 10m 
  Work Description: tims commented on a change in pull request #6586: 
[BEAM-5638] Exception handling for Java single message transforms
URL: https://github.com/apache/beam/pull/6586#discussion_r223189848
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Failure.java
 ##
 @@ -0,0 +1,101 @@
+package org.apache.beam.sdk.transforms;
+
+import com.google.auto.value.AutoValue;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ObjectArrays;
+import java.io.Serializable;
+import java.util.List;
+import org.apache.beam.sdk.coders.SerializableCoder;
+import org.apache.beam.sdk.transforms.DoFn.MultiOutputReceiver;
+import org.apache.beam.sdk.values.PCollectionTuple;
+import org.apache.beam.sdk.values.TupleTag;
+import org.apache.beam.sdk.values.TupleTagList;
+import org.apache.beam.sdk.values.TypeDescriptor;
+
+/**
+ * Wraps an exception along with an input value; this is the element type of 
failure collections
+ * returned by single message transforms configured to catch exceptions.
+ *
+ * @param  type of the wrapped input value that caused an exception to be 
raised
+ */
+@AutoValue
+public abstract class Failure implements Serializable {
+  public static  Failure of(Exception exception, T value) {
+return new AutoValue_Failure<>(exception, value);
+  }
+
+  public abstract Exception exception();
+
+  public abstract T value();
+
+  /**
+   * Internal class for collecting tuple tags associated with collections of 
{@link Exception}
+   * classes that should route to them. Also contains helper methods to 
simplify implementation of
+   * the {@code WithFailures} nested classes of {@link MapElements}, {@link 
FlatMapElements}, etc.
+   */
+  @AutoValue
+  abstract static class TaggedExceptionsList implements Serializable {
+abstract ImmutableList>> tags();
+
+abstract ImmutableList>> exceptionLists();
+
+static  TaggedExceptionsList empty() {
+  return new AutoValue_Failure_TaggedExceptionsList<>(ImmutableList.of(), 
ImmutableList.of());
+}
+
+/**
+ * Return a new {@link TaggedExceptionsList} that has all the tags and 
exceptions of this {@link
+ * TaggedExceptionsList} plus a new element representing the arguments 
passed in here.
+ */
+TaggedExceptionsList and(
+TupleTag> tag, Class exceptionToCatch, Class[] 
additionalExceptions) {
+  final ImmutableList>> newTags =
+  
ImmutableList.>>builder().addAll(tags()).add(tag).build();
+  final ImmutableList>> newExceptionLists =
+  ImmutableList.>>builder()
+  .addAll(exceptionLists())
+  .add(
+  ImmutableList.copyOf(ObjectArrays.concat(exceptionToCatch, 
additionalExceptions)))
+  .build();
+  return new AutoValue_Failure_TaggedExceptionsList<>(newTags, 
newExceptionLists);
+}
+
+/** Return the internal typed list of tags as an untyped {@link 
TupleTagList}. */
+TupleTagList tupleTagList() {
+  TupleTagList l = TupleTagList.empty();
+  for (TupleTag tag : tags()) {
+l = l.and(tag);
+  }
+  return l;
+}
+
+/**
+ * Check the registered exception classes to see if the exception passed 
in here matches. If it
+ * does, wrap the exception and value together in a {@link Failure} and 
send to the output
+ * receiver. If not, rethrow so processing stops on the unexpected failure.
+ */
+void outputOrRethrow(Exception e, T value, MultiOutputReceiver receiver) 
throws Exception {
+  for (int i = 0; i < tags().size(); i++) {
+for (Class cls : exceptionLists().get(i)) {
+  if (cls.isInstance(e)) {
+receiver.get(tags().get(i)).output(Failure.of(e, value));
+return;
+  }
+}
+  }
+  throw e;
+}
+
+/**
+ * Set appropriate coders on all the failure collections in the given 
{@link PCollectionTuple}.
+ */
+PCollectionTuple applyFailureCoders(PCollectionTuple pcs) {
+  final SerializableCoder> failureCoder =
+  SerializableCoder.of(new TypeDescriptor>() {});
 
 Review comment:
   I implemented less general exception handling recently just as a base class 
for my DoFns, I ended up having a proto that contains the stack trace and other 
info just as strings. Also because I was worried about the SerializableCoder 
efficiency.
   
   One option might be for the failure output type be a generic? And we provide 
a function for handling the exception instead?
   
   eg withFailureHandler(TupleTag failureTag, 

[jira] [Work logged] (BEAM-5638) Add exception handling to single message transforms in Java SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5638?focusedWorklogId=151994=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151994
 ]

ASF GitHub Bot logged work on BEAM-5638:


Author: ASF GitHub Bot
Created on: 06/Oct/18 18:26
Start Date: 06/Oct/18 18:26
Worklog Time Spent: 10m 
  Work Description: tims commented on a change in pull request #6586: 
[BEAM-5638] Exception handling for Java single message transforms
URL: https://github.com/apache/beam/pull/6586#discussion_r223189848
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Failure.java
 ##
 @@ -0,0 +1,101 @@
+package org.apache.beam.sdk.transforms;
+
+import com.google.auto.value.AutoValue;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ObjectArrays;
+import java.io.Serializable;
+import java.util.List;
+import org.apache.beam.sdk.coders.SerializableCoder;
+import org.apache.beam.sdk.transforms.DoFn.MultiOutputReceiver;
+import org.apache.beam.sdk.values.PCollectionTuple;
+import org.apache.beam.sdk.values.TupleTag;
+import org.apache.beam.sdk.values.TupleTagList;
+import org.apache.beam.sdk.values.TypeDescriptor;
+
+/**
+ * Wraps an exception along with an input value; this is the element type of 
failure collections
+ * returned by single message transforms configured to catch exceptions.
+ *
+ * @param  type of the wrapped input value that caused an exception to be 
raised
+ */
+@AutoValue
+public abstract class Failure implements Serializable {
+  public static  Failure of(Exception exception, T value) {
+return new AutoValue_Failure<>(exception, value);
+  }
+
+  public abstract Exception exception();
+
+  public abstract T value();
+
+  /**
+   * Internal class for collecting tuple tags associated with collections of 
{@link Exception}
+   * classes that should route to them. Also contains helper methods to 
simplify implementation of
+   * the {@code WithFailures} nested classes of {@link MapElements}, {@link 
FlatMapElements}, etc.
+   */
+  @AutoValue
+  abstract static class TaggedExceptionsList implements Serializable {
+abstract ImmutableList>> tags();
+
+abstract ImmutableList>> exceptionLists();
+
+static  TaggedExceptionsList empty() {
+  return new AutoValue_Failure_TaggedExceptionsList<>(ImmutableList.of(), 
ImmutableList.of());
+}
+
+/**
+ * Return a new {@link TaggedExceptionsList} that has all the tags and 
exceptions of this {@link
+ * TaggedExceptionsList} plus a new element representing the arguments 
passed in here.
+ */
+TaggedExceptionsList and(
+TupleTag> tag, Class exceptionToCatch, Class[] 
additionalExceptions) {
+  final ImmutableList>> newTags =
+  
ImmutableList.>>builder().addAll(tags()).add(tag).build();
+  final ImmutableList>> newExceptionLists =
+  ImmutableList.>>builder()
+  .addAll(exceptionLists())
+  .add(
+  ImmutableList.copyOf(ObjectArrays.concat(exceptionToCatch, 
additionalExceptions)))
+  .build();
+  return new AutoValue_Failure_TaggedExceptionsList<>(newTags, 
newExceptionLists);
+}
+
+/** Return the internal typed list of tags as an untyped {@link 
TupleTagList}. */
+TupleTagList tupleTagList() {
+  TupleTagList l = TupleTagList.empty();
+  for (TupleTag tag : tags()) {
+l = l.and(tag);
+  }
+  return l;
+}
+
+/**
+ * Check the registered exception classes to see if the exception passed 
in here matches. If it
+ * does, wrap the exception and value together in a {@link Failure} and 
send to the output
+ * receiver. If not, rethrow so processing stops on the unexpected failure.
+ */
+void outputOrRethrow(Exception e, T value, MultiOutputReceiver receiver) 
throws Exception {
+  for (int i = 0; i < tags().size(); i++) {
+for (Class cls : exceptionLists().get(i)) {
+  if (cls.isInstance(e)) {
+receiver.get(tags().get(i)).output(Failure.of(e, value));
+return;
+  }
+}
+  }
+  throw e;
+}
+
+/**
+ * Set appropriate coders on all the failure collections in the given 
{@link PCollectionTuple}.
+ */
+PCollectionTuple applyFailureCoders(PCollectionTuple pcs) {
+  final SerializableCoder> failureCoder =
+  SerializableCoder.of(new TypeDescriptor>() {});
 
 Review comment:
   I implemented less general exception handling recently just as a base class 
for my DoFns, I ended up having a proto that contains the stack trace and other 
info just as strings. Also because I was worried about the SerializableCoder 
efficiency.
   
   One option might be for the failure output type be a generic? And we provide 
a function for handling the exception instead?
   
   eg withExceptionHandler(TupleTag failureTag, 

[jira] [Work logged] (BEAM-5638) Add exception handling to single message transforms in Java SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5638?focusedWorklogId=151992=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151992
 ]

ASF GitHub Bot logged work on BEAM-5638:


Author: ASF GitHub Bot
Created on: 06/Oct/18 18:25
Start Date: 06/Oct/18 18:25
Worklog Time Spent: 10m 
  Work Description: tims commented on a change in pull request #6586: 
[BEAM-5638] Exception handling for Java single message transforms
URL: https://github.com/apache/beam/pull/6586#discussion_r223189848
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Failure.java
 ##
 @@ -0,0 +1,101 @@
+package org.apache.beam.sdk.transforms;
+
+import com.google.auto.value.AutoValue;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ObjectArrays;
+import java.io.Serializable;
+import java.util.List;
+import org.apache.beam.sdk.coders.SerializableCoder;
+import org.apache.beam.sdk.transforms.DoFn.MultiOutputReceiver;
+import org.apache.beam.sdk.values.PCollectionTuple;
+import org.apache.beam.sdk.values.TupleTag;
+import org.apache.beam.sdk.values.TupleTagList;
+import org.apache.beam.sdk.values.TypeDescriptor;
+
+/**
+ * Wraps an exception along with an input value; this is the element type of 
failure collections
+ * returned by single message transforms configured to catch exceptions.
+ *
+ * @param  type of the wrapped input value that caused an exception to be 
raised
+ */
+@AutoValue
+public abstract class Failure implements Serializable {
+  public static  Failure of(Exception exception, T value) {
+return new AutoValue_Failure<>(exception, value);
+  }
+
+  public abstract Exception exception();
+
+  public abstract T value();
+
+  /**
+   * Internal class for collecting tuple tags associated with collections of 
{@link Exception}
+   * classes that should route to them. Also contains helper methods to 
simplify implementation of
+   * the {@code WithFailures} nested classes of {@link MapElements}, {@link 
FlatMapElements}, etc.
+   */
+  @AutoValue
+  abstract static class TaggedExceptionsList implements Serializable {
+abstract ImmutableList>> tags();
+
+abstract ImmutableList>> exceptionLists();
+
+static  TaggedExceptionsList empty() {
+  return new AutoValue_Failure_TaggedExceptionsList<>(ImmutableList.of(), 
ImmutableList.of());
+}
+
+/**
+ * Return a new {@link TaggedExceptionsList} that has all the tags and 
exceptions of this {@link
+ * TaggedExceptionsList} plus a new element representing the arguments 
passed in here.
+ */
+TaggedExceptionsList and(
+TupleTag> tag, Class exceptionToCatch, Class[] 
additionalExceptions) {
+  final ImmutableList>> newTags =
+  
ImmutableList.>>builder().addAll(tags()).add(tag).build();
+  final ImmutableList>> newExceptionLists =
+  ImmutableList.>>builder()
+  .addAll(exceptionLists())
+  .add(
+  ImmutableList.copyOf(ObjectArrays.concat(exceptionToCatch, 
additionalExceptions)))
+  .build();
+  return new AutoValue_Failure_TaggedExceptionsList<>(newTags, 
newExceptionLists);
+}
+
+/** Return the internal typed list of tags as an untyped {@link 
TupleTagList}. */
+TupleTagList tupleTagList() {
+  TupleTagList l = TupleTagList.empty();
+  for (TupleTag tag : tags()) {
+l = l.and(tag);
+  }
+  return l;
+}
+
+/**
+ * Check the registered exception classes to see if the exception passed 
in here matches. If it
+ * does, wrap the exception and value together in a {@link Failure} and 
send to the output
+ * receiver. If not, rethrow so processing stops on the unexpected failure.
+ */
+void outputOrRethrow(Exception e, T value, MultiOutputReceiver receiver) 
throws Exception {
+  for (int i = 0; i < tags().size(); i++) {
+for (Class cls : exceptionLists().get(i)) {
+  if (cls.isInstance(e)) {
+receiver.get(tags().get(i)).output(Failure.of(e, value));
+return;
+  }
+}
+  }
+  throw e;
+}
+
+/**
+ * Set appropriate coders on all the failure collections in the given 
{@link PCollectionTuple}.
+ */
+PCollectionTuple applyFailureCoders(PCollectionTuple pcs) {
+  final SerializableCoder> failureCoder =
+  SerializableCoder.of(new TypeDescriptor>() {});
 
 Review comment:
   I implemented less general exception handling recently just as a base class 
for my DoFns, I ended up having a proto that contains the stack trace and other 
info just as strings. Also because I was worried about the SerializableCoder 
efficiency.
   
   One option might be for the failure output type be a generic? And we provide 
a function for handling the exception instead?
   
   eg withExceptionHandler(TupleTag failureTag, 

[jira] [Work logged] (BEAM-5638) Add exception handling to single message transforms in Java SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5638?focusedWorklogId=151991=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151991
 ]

ASF GitHub Bot logged work on BEAM-5638:


Author: ASF GitHub Bot
Created on: 06/Oct/18 18:22
Start Date: 06/Oct/18 18:22
Worklog Time Spent: 10m 
  Work Description: tims commented on a change in pull request #6586: 
[BEAM-5638] Exception handling for Java single message transforms
URL: https://github.com/apache/beam/pull/6586#discussion_r223189848
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Failure.java
 ##
 @@ -0,0 +1,101 @@
+package org.apache.beam.sdk.transforms;
+
+import com.google.auto.value.AutoValue;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ObjectArrays;
+import java.io.Serializable;
+import java.util.List;
+import org.apache.beam.sdk.coders.SerializableCoder;
+import org.apache.beam.sdk.transforms.DoFn.MultiOutputReceiver;
+import org.apache.beam.sdk.values.PCollectionTuple;
+import org.apache.beam.sdk.values.TupleTag;
+import org.apache.beam.sdk.values.TupleTagList;
+import org.apache.beam.sdk.values.TypeDescriptor;
+
+/**
+ * Wraps an exception along with an input value; this is the element type of 
failure collections
+ * returned by single message transforms configured to catch exceptions.
+ *
+ * @param  type of the wrapped input value that caused an exception to be 
raised
+ */
+@AutoValue
+public abstract class Failure implements Serializable {
+  public static  Failure of(Exception exception, T value) {
+return new AutoValue_Failure<>(exception, value);
+  }
+
+  public abstract Exception exception();
+
+  public abstract T value();
+
+  /**
+   * Internal class for collecting tuple tags associated with collections of 
{@link Exception}
+   * classes that should route to them. Also contains helper methods to 
simplify implementation of
+   * the {@code WithFailures} nested classes of {@link MapElements}, {@link 
FlatMapElements}, etc.
+   */
+  @AutoValue
+  abstract static class TaggedExceptionsList implements Serializable {
+abstract ImmutableList>> tags();
+
+abstract ImmutableList>> exceptionLists();
+
+static  TaggedExceptionsList empty() {
+  return new AutoValue_Failure_TaggedExceptionsList<>(ImmutableList.of(), 
ImmutableList.of());
+}
+
+/**
+ * Return a new {@link TaggedExceptionsList} that has all the tags and 
exceptions of this {@link
+ * TaggedExceptionsList} plus a new element representing the arguments 
passed in here.
+ */
+TaggedExceptionsList and(
+TupleTag> tag, Class exceptionToCatch, Class[] 
additionalExceptions) {
+  final ImmutableList>> newTags =
+  
ImmutableList.>>builder().addAll(tags()).add(tag).build();
+  final ImmutableList>> newExceptionLists =
+  ImmutableList.>>builder()
+  .addAll(exceptionLists())
+  .add(
+  ImmutableList.copyOf(ObjectArrays.concat(exceptionToCatch, 
additionalExceptions)))
+  .build();
+  return new AutoValue_Failure_TaggedExceptionsList<>(newTags, 
newExceptionLists);
+}
+
+/** Return the internal typed list of tags as an untyped {@link 
TupleTagList}. */
+TupleTagList tupleTagList() {
+  TupleTagList l = TupleTagList.empty();
+  for (TupleTag tag : tags()) {
+l = l.and(tag);
+  }
+  return l;
+}
+
+/**
+ * Check the registered exception classes to see if the exception passed 
in here matches. If it
+ * does, wrap the exception and value together in a {@link Failure} and 
send to the output
+ * receiver. If not, rethrow so processing stops on the unexpected failure.
+ */
+void outputOrRethrow(Exception e, T value, MultiOutputReceiver receiver) 
throws Exception {
+  for (int i = 0; i < tags().size(); i++) {
+for (Class cls : exceptionLists().get(i)) {
+  if (cls.isInstance(e)) {
+receiver.get(tags().get(i)).output(Failure.of(e, value));
+return;
+  }
+}
+  }
+  throw e;
+}
+
+/**
+ * Set appropriate coders on all the failure collections in the given 
{@link PCollectionTuple}.
+ */
+PCollectionTuple applyFailureCoders(PCollectionTuple pcs) {
+  final SerializableCoder> failureCoder =
+  SerializableCoder.of(new TypeDescriptor>() {});
 
 Review comment:
   I implemented less general exception handling recently just as a base class 
for my DoFns, I ended up having a proto that contains the stack trace and other 
info just as strings. Also because I was worried about the SerializableCoder 
efficiency.
   
   One option might be for the failure output type be a generic? And we provide 
a function for handling the exception instead?


This 

[beam] branch asf-site updated: Publishing website 2018/10/06 18:00:46 at commit 324f0b3

[git-site-role]

This is an automated email from the ASF dual-hosted git repository.

git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/asf-site by this push:
 new e2d5f41  Publishing website 2018/10/06 18:00:46 at commit 324f0b3
e2d5f41 is described below

commit e2d5f4166af41626117315808c0f27fb9a077217
Author: jenkins 
AuthorDate: Sat Oct 6 18:00:46 2018 +

Publishing website 2018/10/06 18:00:46 at commit 324f0b3

Build failed in Jenkins: beam_PreCommit_Website_Cron #147

[Apache Jenkins Server]

See 


--
[...truncated 166.67 KB...]
  *  External link http://images/logos/runners/spark.png failed: response code 
0 means something's wrong.
 It's possible libcurl couldn't connect to the server or perhaps 
the request timed out.
 Sometimes, making too many requests at once also breaks things.
 Either way, the return message (if any) from the server is: 
Couldn't resolve host name
  *  External link http://images/logos/sdks/go.png failed: response code 0 
means something's wrong.
 It's possible libcurl couldn't connect to the server or perhaps 
the request timed out.
 Sometimes, making too many requests at once also breaks things.
 Either way, the return message (if any) from the server is: 
Couldn't resolve host name
  *  External link http://images/logos/sdks/java.png failed: response code 0 
means something's wrong.
 It's possible libcurl couldn't connect to the server or perhaps 
the request timed out.
 Sometimes, making too many requests at once also breaks things.
 Either way, the return message (if any) from the server is: 
Couldn't resolve host name
  *  External link http://images/logos/sdks/python.png failed: response code 0 
means something's wrong.
 It's possible libcurl couldn't connect to the server or perhaps 
the request timed out.
 Sometimes, making too many requests at once also breaks things.
 Either way, the return message (if any) from the server is: 
Couldn't resolve host name
  *  External link http://images/logos/sdks/scala.png failed: response code 0 
means something's wrong.
 It's possible libcurl couldn't connect to the server or perhaps 
the request timed out.
 Sometimes, making too many requests at once also breaks things.
 Either way, the return message (if any) from the server is: 
Couldn't resolve host name
  *  http:// is an invalid URL (line 77)
 

  
- ./generated-content/get-started/downloads/index.html
  *  http:// is an invalid URL (line 77)
 

  
- ./generated-content/get-started/index.html
  *  External link http://get-started/beam-overview failed: response code 0 
means something's wrong.
 It's possible libcurl couldn't connect to the server or perhaps 
the request timed out.
 Sometimes, making too many requests at once also breaks things.
 Either way, the return message (if any) from the server is: 
Couldn't resolve host name
  *  External link http://get-started/mobile-gaming-example failed: response 
code 0 means something's wrong.
 It's possible libcurl couldn't connect to the server or perhaps 
the request timed out.
 Sometimes, making too many requests at once also breaks things.
 Either way, the return message (if any) from the server is: 
Couldn't resolve host name
  *  http:// is an invalid URL (line 77)
 

  
- ./generated-content/get-started/mobile-gaming-example/index.html
  *  External link http://documentation/programming-guide/ failed: response 
code 0 means something's wrong.
 It's possible libcurl couldn't connect to the server or perhaps 
the request timed out.
 Sometimes, making too many requests at once also breaks things.
 Either way, the return message (if any) from the server is: 
Couldn't resolve host name
  *  External link http://documentation/programming-guide/ failed: response 
code 0 means something's wrong.
 It's possible libcurl couldn't connect to the server or perhaps 
the request timed out.
 Sometimes, making too many requests at once also breaks things.
 Either way, the return message (if any) from the server is: 
Couldn't resolve host name
  *  External link http://images/gaming-example-basic.png failed: response code 
0 means something's wrong.
 It's possible libcurl couldn't connect to the server or perhaps 
the request timed out.
 Sometimes, making too many requests at once also breaks things.
 Either way, the return message (if any) from the server is: 
Couldn't resolve host name
  *  External link http://images/gaming-example-event-time-narrow.gif failed: 
response code 0 means something's wrong.
 It's possible libcurl couldn't connect to the server or perhaps 
the request timed out.
 Sometimes, making too many requests at once also breaks things.
 Either way, the return message (if any) from the server is: 
Couldn't resolve host name
  *  External link http://images/gaming-example-proc-time-narrow.gif failed: 
response code 0 means something's wrong.
 It's possible libcurl couldn't connect to the server or perhaps 
the request timed out.
 

Build failed in Jenkins: beam_PreCommit_Website_Stage_GCS_Cron #9

[Apache Jenkins Server]

See 


--
[...truncated 6.86 KB...]
Skipping task ':buildSrc:classes' as it has no actions.
:buildSrc:classes (Thread[Task worker for ':buildSrc' Thread 4,5,main]) 
completed. Took 0.0 secs.
:buildSrc:jar (Thread[Task worker for ':buildSrc' Thread 4,5,main]) started.

> Task :buildSrc:jar
Build cache key for task ':buildSrc:jar' is 7445e5c45b21f8a690f2f547fcb49594
Caching disabled for task ':buildSrc:jar': Caching has not been enabled for the 
task
Task ':buildSrc:jar' is not up-to-date because:
  No history is available.
:buildSrc:jar (Thread[Task worker for ':buildSrc' Thread 4,5,main]) completed. 
Took 0.099 secs.
:buildSrc:assemble (Thread[Task worker for ':buildSrc' Thread 4,5,main]) 
started.

> Task :buildSrc:assemble
Skipping task ':buildSrc:assemble' as it has no actions.
:buildSrc:assemble (Thread[Task worker for ':buildSrc' Thread 4,5,main]) 
completed. Took 0.0 secs.
:buildSrc:spotlessGroovy (Thread[Task worker for ':buildSrc' Thread 4,5,main]) 
started.

> Task :buildSrc:spotlessGroovy
file or directory 
'
 not found
file or directory 
'
 not found
file or directory 
'
 not found
Caching disabled for task ':buildSrc:spotlessGroovy': Caching has not been 
enabled for the task
Task ':buildSrc:spotlessGroovy' is not up-to-date because:
  No history is available.
All input files are considered out-of-date for incremental task 
':buildSrc:spotlessGroovy'.
file or directory 
'
 not found
:buildSrc:spotlessGroovy (Thread[Task worker for ':buildSrc' Thread 4,5,main]) 
completed. Took 1.199 secs.
:buildSrc:spotlessGroovyCheck (Thread[Task worker for ':buildSrc' Thread 
4,5,main]) started.

> Task :buildSrc:spotlessGroovyCheck
Skipping task ':buildSrc:spotlessGroovyCheck' as it has no actions.
:buildSrc:spotlessGroovyCheck (Thread[Task worker for ':buildSrc' Thread 
4,5,main]) completed. Took 0.001 secs.
:buildSrc:spotlessGroovyGradle (Thread[Task worker for ':buildSrc' Thread 
4,5,main]) started.

> Task :buildSrc:spotlessGroovyGradle
Caching disabled for task ':buildSrc:spotlessGroovyGradle': Caching has not 
been enabled for the task
Task ':buildSrc:spotlessGroovyGradle' is not up-to-date because:
  No history is available.
All input files are considered out-of-date for incremental task 
':buildSrc:spotlessGroovyGradle'.
:buildSrc:spotlessGroovyGradle (Thread[Task worker for ':buildSrc' Thread 
4,5,main]) completed. Took 0.025 secs.
:buildSrc:spotlessGroovyGradleCheck (Thread[Task worker for ':buildSrc' Thread 
4,5,main]) started.

> Task :buildSrc:spotlessGroovyGradleCheck
Skipping task ':buildSrc:spotlessGroovyGradleCheck' as it has no actions.
:buildSrc:spotlessGroovyGradleCheck (Thread[Task worker for ':buildSrc' Thread 
4,5,main]) completed. Took 0.0 secs.
:buildSrc:spotlessCheck (Thread[Task worker for ':buildSrc' Thread 4,5,main]) 
started.

> Task :buildSrc:spotlessCheck
Skipping task ':buildSrc:spotlessCheck' as it has no actions.
:buildSrc:spotlessCheck (Thread[Task worker for ':buildSrc' Thread 4,5,main]) 
completed. Took 0.0 secs.
:buildSrc:compileTestJava (Thread[Task worker for ':buildSrc' Thread 4,5,main]) 
started.

> Task :buildSrc:compileTestJava NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:compileTestJava' as it has no source files and no 
previous output files.
:buildSrc:compileTestJava (Thread[Task worker for ':buildSrc' Thread 4,5,main]) 
completed. Took 0.002 secs.
:buildSrc:compileTestGroovy (Thread[Task worker for ':buildSrc' Thread 
4,5,main]) started.

> Task :buildSrc:compileTestGroovy NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:compileTestGroovy' as it has no source files and no 
previous output files.
:buildSrc:compileTestGroovy (Thread[Task worker for ':buildSrc' Thread 
4,5,main]) completed. Took 0.002 secs.
:buildSrc:processTestResources (Thread[Task worker for ':buildSrc' Thread 
4,5,main]) started.

> Task :buildSrc:processTestResources NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:processTestResources' as it has no source files and no 
previous output files.
:buildSrc:processTestResources (Thread[Task worker for ':buildSrc' Thread 
4,5,main]) completed. Took 0.001 

[jira] [Commented] (BEAM-5627) Several IO tests fail in Python 3 when accessing a temporary file with TypeError: a bytes-like object is required, not 'str'

[Rakesh Kumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640777#comment-16640777
 ] 

Rakesh Kumar commented on BEAM-5627:


[~tvalentyn] can you also assign this Jira ticket to me. Currently, I am not 
able to assign this to me.

> Several IO tests fail in Python 3  when accessing a temporary file with  
> TypeError: a bytes-like object is required, not 'str'
> --
>
> Key: BEAM-5627
> URL: https://issues.apache.org/jira/browse/BEAM-5627
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Priority: Major
>
> ERROR: test_split_at_fraction_exhaustive 
> (apache_beam.io.source_test_utils_test.SourceTestUtilsTest)
>  --
>  Traceback (most recent call last):
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils_test.py",
>  line 120, in test_split_at_fraction_exhaustive
>  source = self._create_source(data)
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils_test.py",
>  line 43, in _create_source
>  source = LineSource(self._create_file_with_data(data))
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils_test.py",
>  line 35, in _create_file_with_data
>  f.write(line + '\n')
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/target/.tox/py3/lib/python3.5/tempfile.py",
>  line 622, in func_wrapper
>  return func(*args, **kwargs)
> TypeError: a bytes-like object is required, not 'str'
> Also similar:
> ==
>  ERROR: test_file_sink_writing 
> (apache_beam.io.filebasedsink_test.TestFileBasedSink)
> --
> Traceback (most recent call last):
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink_test.py", line 121, in 
> test_file_sink_writing
>   init_token, writer_results = self._common_init(sink)
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink_test.py", line 103, in _common_init
>   writer1 = sink.open_writer(init_token, '1')
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/options/value_provider.py", line 133, in _f
>   return fnc(self, *args, **kwargs)
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink.py", line 185, in open_writer
> return FileBasedSinkWriter(self, os.path.join(init_result, uid) + suffix)
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink.py", line 385, in __init__
>   self.temp_handle = self.sink.open(temp_shard_path)
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink_test.py", line 82, in open
>   file_handle.write('[start]')
>   TypeError: a bytes-like object is required, not 'str'



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5626) Several IO tests fail in Python 3 with RuntimeError('dictionary changed size during iteration',)}

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5626?focusedWorklogId=151989=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151989
 ]

ASF GitHub Bot logged work on BEAM-5626:


Author: ASF GitHub Bot
Created on: 06/Oct/18 16:39
Start Date: 06/Oct/18 16:39
Worklog Time Spent: 10m 
  Work Description: rakeshcusat commented on a change in pull request 
#6587: [BEAM-5626] Fix hadoop filesystem test for py3.
URL: https://github.com/apache/beam/pull/6587#discussion_r223187329
 
 

 ##
 File path: sdks/python/apache_beam/io/hadoopfilesystem_test.py
 ##
 @@ -153,7 +153,7 @@ def delete(self, path, recursive=True):
 
 _ = self.status(path)
 
-for filepath in self.files.keys():  # pylint: 
disable=consider-iterating-dictionary
+for filepath in list(self.files):
 
 Review comment:
   This could be also efficient in terms of memory.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151989)
Time Spent: 2h 20m  (was: 2h 10m)

> Several IO tests fail in Python 3 with RuntimeError('dictionary changed size 
> during iteration',)}
> -
>
> Key: BEAM-5626
> URL: https://issues.apache.org/jira/browse/BEAM-5626
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Ruoyun Huang
>Priority: Major
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
>  ERROR: test_delete_dir 
> (apache_beam.io.hadoopfilesystem_test.HadoopFileSystemTest)
> --
> Traceback (most recent call last):
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/hadoopfilesystem_test.py",
>  line 506, in test_delete_dir
>  self.fs.delete([url_t1])
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/hadoopfilesystem.py",
>  line 370, in delete
>  raise BeamIOError("Delete operation failed", exceptions)
>  apache_beam.io.filesystem.BeamIOError: Delete operation failed with 
> exceptions {'hdfs://test_dir/new_dir1': RuntimeError('dictionary changed size 
> during iteration',   )}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5626) Several IO tests fail in Python 3 with RuntimeError('dictionary changed size during iteration',)}

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5626?focusedWorklogId=151988=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151988
 ]

ASF GitHub Bot logged work on BEAM-5626:


Author: ASF GitHub Bot
Created on: 06/Oct/18 16:29
Start Date: 06/Oct/18 16:29
Worklog Time Spent: 10m 
  Work Description: rakeshcusat commented on a change in pull request 
#6587: [BEAM-5626] Fix hadoop filesystem test for py3.
URL: https://github.com/apache/beam/pull/6587#discussion_r223187094
 
 

 ##
 File path: sdks/python/apache_beam/io/hadoopfilesystem_test.py
 ##
 @@ -153,7 +153,7 @@ def delete(self, path, recursive=True):
 
 _ = self.status(path)
 
-for filepath in self.files.keys():  # pylint: 
disable=consider-iterating-dictionary
+for filepath in list(self.files):
 
 Review comment:
   Any specific reason for not using `six` library for porting? I see our 
dependencies are using six. 
   In case we are using six this could be something like this:
   ```
import six
   ..
   ..
   for filepath in six.iterkeys(self.files):
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151988)
Time Spent: 2h 10m  (was: 2h)

> Several IO tests fail in Python 3 with RuntimeError('dictionary changed size 
> during iteration',)}
> -
>
> Key: BEAM-5626
> URL: https://issues.apache.org/jira/browse/BEAM-5626
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Ruoyun Huang
>Priority: Major
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
>  ERROR: test_delete_dir 
> (apache_beam.io.hadoopfilesystem_test.HadoopFileSystemTest)
> --
> Traceback (most recent call last):
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/hadoopfilesystem_test.py",
>  line 506, in test_delete_dir
>  self.fs.delete([url_t1])
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/hadoopfilesystem.py",
>  line 370, in delete
>  raise BeamIOError("Delete operation failed", exceptions)
>  apache_beam.io.filesystem.BeamIOError: Delete operation failed with 
> exceptions {'hdfs://test_dir/new_dir1': RuntimeError('dictionary changed size 
> during iteration',   )}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=151980=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151980
 ]

ASF GitHub Bot logged work on BEAM-5315:


Author: ASF GitHub Bot
Created on: 06/Oct/18 13:27
Start Date: 06/Oct/18 13:27
Worklog Time Spent: 10m 
  Work Description: splovyt commented on issue #6590: [BEAM-5315] Partially 
port io
URL: https://github.com/apache/beam/pull/6590#issuecomment-427573770
 
 
   I ran across an additional error that I am trying to resolve. I will notify 
you when I squashed commits and we're ready to merge. Thanks


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151980)
Time Spent: 1h 20m  (was: 1h 10m)

> Finish Python 3 porting for io module
> -
>
> Key: BEAM-5315
> URL: https://issues.apache.org/jira/browse/BEAM-5315
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Simon
>Priority: Major
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5097) Increment counter for "small words" in go SDK example

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5097?focusedWorklogId=151979=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151979
 ]

ASF GitHub Bot logged work on BEAM-5097:


Author: ASF GitHub Bot
Created on: 06/Oct/18 13:16
Start Date: 06/Oct/18 13:16
Worklog Time Spent: 10m 
  Work Description: stale[bot] commented on issue #6157: [BEAM-5097][WIP] 
Add counter to combine example in go sdk
URL: https://github.com/apache/beam/pull/6157#issuecomment-427573014
 
 
   This pull request has been marked as stale due to 60 days of inactivity. It 
will be closed in 1 week if no further activity occurs. If you think that’s 
incorrect or this pull request requires a review, please simply write any 
comment. If closed, you can revive the PR at any time and @mention a reviewer 
or discuss it on the d...@beam.apache.org list. Thank you for your 
contributions.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151979)
Time Spent: 1h 20m  (was: 1h 10m)

> Increment counter for "small words" in go SDK example
> -
>
> Key: BEAM-5097
> URL: https://issues.apache.org/jira/browse/BEAM-5097
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: holdenk
>Assignee: holdenk
>Priority: Trivial
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Increment counter for "small words" in go SDK example



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=151978=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151978
 ]

ASF GitHub Bot logged work on BEAM-5315:


Author: ASF GitHub Bot
Created on: 06/Oct/18 13:11
Start Date: 06/Oct/18 13:11
Worklog Time Spent: 10m 
  Work Description: splovyt commented on a change in pull request #6590: 
[BEAM-5315] Partially port io
URL: https://github.com/apache/beam/pull/6590#discussion_r223180087
 
 

 ##
 File path: sdks/python/setup.py
 ##
 @@ -136,11 +136,17 @@ def get_version():
 'pyhamcrest>=1.9,<2.0',
 ]
 
+GCP_REQUIREMENTS_PY2_ONLY = [
+'googledatastore==7.0.1'
+]
+
+GCP_REQUIREMENTS_PY3_ONLY = [
+]
+
 GCP_REQUIREMENTS = [
 # oauth2client >=4 only works with google-apitools>=0.5.18.
 'google-apitools>=0.5.18,<=0.5.20',
 'proto-google-cloud-datastore-v1>=0.90.0,<=0.90.4',
-'googledatastore==7.0.1',
 
 Review comment:
   I have updated my setup.py to the latest version in the master for testing 
purposes and have added a version qualifier instead. Thanks for the feedback
   
   Currently working on fixing the py27-gcp environment cause there seemed to 
be an issue


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151978)
Time Spent: 1h 10m  (was: 1h)

> Finish Python 3 porting for io module
> -
>
> Key: BEAM-5315
> URL: https://issues.apache.org/jira/browse/BEAM-5315
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Simon
>Priority: Major
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=151976=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151976
 ]

ASF GitHub Bot logged work on BEAM-5315:


Author: ASF GitHub Bot
Created on: 06/Oct/18 12:46
Start Date: 06/Oct/18 12:46
Worklog Time Spent: 10m 
  Work Description: splovyt commented on a change in pull request #6590: 
[BEAM-5315] Partially port io
URL: https://github.com/apache/beam/pull/6590#discussion_r223180087
 
 

 ##
 File path: sdks/python/setup.py
 ##
 @@ -136,11 +136,17 @@ def get_version():
 'pyhamcrest>=1.9,<2.0',
 ]
 
+GCP_REQUIREMENTS_PY2_ONLY = [
+'googledatastore==7.0.1'
+]
+
+GCP_REQUIREMENTS_PY3_ONLY = [
+]
+
 GCP_REQUIREMENTS = [
 # oauth2client >=4 only works with google-apitools>=0.5.18.
 'google-apitools>=0.5.18,<=0.5.20',
 'proto-google-cloud-datastore-v1>=0.90.0,<=0.90.4',
-'googledatastore==7.0.1',
 
 Review comment:
   I have updated my setup.py to the latest version in the master for testing 
purposes and have added a version qualifier instead. Thanks for the feedback
   
   Currently working on fixing the py27-gcp environment


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151976)
Time Spent: 1h  (was: 50m)

> Finish Python 3 porting for io module
> -
>
> Key: BEAM-5315
> URL: https://issues.apache.org/jira/browse/BEAM-5315
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Simon
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=151972=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151972
 ]

ASF GitHub Bot logged work on BEAM-5315:


Author: ASF GitHub Bot
Created on: 06/Oct/18 12:38
Start Date: 06/Oct/18 12:38
Worklog Time Spent: 10m 
  Work Description: splovyt commented on a change in pull request #6590: 
[BEAM-5315] Partially port io
URL: https://github.com/apache/beam/pull/6590#discussion_r223180087
 
 

 ##
 File path: sdks/python/setup.py
 ##
 @@ -136,11 +136,17 @@ def get_version():
 'pyhamcrest>=1.9,<2.0',
 ]
 
+GCP_REQUIREMENTS_PY2_ONLY = [
+'googledatastore==7.0.1'
+]
+
+GCP_REQUIREMENTS_PY3_ONLY = [
+]
+
 GCP_REQUIREMENTS = [
 # oauth2client >=4 only works with google-apitools>=0.5.18.
 'google-apitools>=0.5.18,<=0.5.20',
 'proto-google-cloud-datastore-v1>=0.90.0,<=0.90.4',
-'googledatastore==7.0.1',
 
 Review comment:
   I have updated my setup.py to the latest version in the master for testing 
purposes and have added a version qualifier instead. Thanks for the feedback


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151972)
Time Spent: 50m  (was: 40m)

> Finish Python 3 porting for io module
> -
>
> Key: BEAM-5315
> URL: https://issues.apache.org/jira/browse/BEAM-5315
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Simon
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_ParquetIOIT #570

[Apache Jenkins Server]

See 


--
[...truncated 229.92 KB...]
INFO: Adding Read matched files/ParDo(ToReadableFile) as step s33
Oct 06, 2018 12:28:31 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read parquet files/ParDo(Read) as step s34
Oct 06, 2018 12:28:31 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Map records to strings/Map as step s35
Oct 06, 2018 12:28:31 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Calculate hashcode/WithKeys/AddKeys/Map as step s36
Oct 06, 2018 12:28:31 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Calculate hashcode/Combine.perKey(Hashing)/GroupByKey as step 
s37
Oct 06, 2018 12:28:31 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Calculate 
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues as step s38
Oct 06, 2018 12:28:31 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Calculate hashcode/Values/Values/Map as step s39
Oct 06, 2018 12:28:31 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Calculate 
hashcode/View.AsIterable/ParDo(ToIsmRecordForGlobalWindow) as step s40
Oct 06, 2018 12:28:31 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Calculate hashcode/View.AsIterable/CreateDataflowView as step 
s41
Oct 06, 2018 12:28:31 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Calculate hashcode/CreateVoid/Read(CreateSource) as step s42
Oct 06, 2018 12:28:31 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Calculate hashcode/ProduceDefault as step s43
Oct 06, 2018 12:28:31 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Calculate hashcode/Flatten.PCollections as step s44
Oct 06, 2018 12:28:31 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$0/CreateActual/FilterActuals/Window.Assign as step s45
Oct 06, 2018 12:28:31 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$0/CreateActual/GatherPanes/Reify.Window/ParDo(Anonymous) as step s46
Oct 06, 2018 12:28:31 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$0/CreateActual/GatherPanes/WithKeys/AddKeys/Map as 
step s47
Oct 06, 2018 12:28:31 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$0/CreateActual/GatherPanes/Window.Into()/Window.Assign 
as step s48
Oct 06, 2018 12:28:31 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$0/CreateActual/GatherPanes/GroupByKey as step s49
Oct 06, 2018 12:28:31 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$0/CreateActual/GatherPanes/Values/Values/Map as step 
s50
Oct 06, 2018 12:28:31 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$0/CreateActual/ExtractPane/Map as step s51
Oct 06, 2018 12:28:31 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$0/CreateActual/Flatten.Iterables/FlattenIterables/FlatMap as step s52
Oct 06, 2018 12:28:31 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$0/CreateActual/RewindowActuals/Window.Assign as step 
s53
Oct 06, 2018 12:28:31 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$0/CreateActual/ParDo(Anonymous) as step s54
Oct 06, 2018 12:28:31 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$0/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map
 as step s55
Oct 06, 2018 12:28:31 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$0/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey
 as step s56
Oct 06, 2018 12:28:31 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$0/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues
 as step 

Build failed in Jenkins: beam_PreCommit_Website_Cron #146

[Apache Jenkins Server]

See 


--
[...truncated 166.43 KB...]
  *  External link http://images/logos/runners/spark.png failed: response code 
0 means something's wrong.
 It's possible libcurl couldn't connect to the server or perhaps 
the request timed out.
 Sometimes, making too many requests at once also breaks things.
 Either way, the return message (if any) from the server is: 
Couldn't resolve host name
  *  External link http://images/logos/sdks/go.png failed: response code 0 
means something's wrong.
 It's possible libcurl couldn't connect to the server or perhaps 
the request timed out.
 Sometimes, making too many requests at once also breaks things.
 Either way, the return message (if any) from the server is: 
Couldn't resolve host name
  *  External link http://images/logos/sdks/java.png failed: response code 0 
means something's wrong.
 It's possible libcurl couldn't connect to the server or perhaps 
the request timed out.
 Sometimes, making too many requests at once also breaks things.
 Either way, the return message (if any) from the server is: 
Couldn't resolve host name
  *  External link http://images/logos/sdks/python.png failed: response code 0 
means something's wrong.
 It's possible libcurl couldn't connect to the server or perhaps 
the request timed out.
 Sometimes, making too many requests at once also breaks things.
 Either way, the return message (if any) from the server is: 
Couldn't resolve host name
  *  External link http://images/logos/sdks/scala.png failed: response code 0 
means something's wrong.
 It's possible libcurl couldn't connect to the server or perhaps 
the request timed out.
 Sometimes, making too many requests at once also breaks things.
 Either way, the return message (if any) from the server is: 
Couldn't resolve host name
  *  http:// is an invalid URL (line 77)
 

  
- ./generated-content/get-started/downloads/index.html
  *  http:// is an invalid URL (line 77)
 

  
- ./generated-content/get-started/index.html
  *  External link http://get-started/beam-overview failed: response code 0 
means something's wrong.
 It's possible libcurl couldn't connect to the server or perhaps 
the request timed out.
 Sometimes, making too many requests at once also breaks things.
 Either way, the return message (if any) from the server is: 
Couldn't resolve host name
  *  External link http://get-started/mobile-gaming-example failed: response 
code 0 means something's wrong.
 It's possible libcurl couldn't connect to the server or perhaps 
the request timed out.
 Sometimes, making too many requests at once also breaks things.
 Either way, the return message (if any) from the server is: 
Couldn't resolve host name
  *  http:// is an invalid URL (line 77)
 

  
- ./generated-content/get-started/mobile-gaming-example/index.html
  *  External link http://documentation/programming-guide/ failed: response 
code 0 means something's wrong.
 It's possible libcurl couldn't connect to the server or perhaps 
the request timed out.
 Sometimes, making too many requests at once also breaks things.
 Either way, the return message (if any) from the server is: 
Couldn't resolve host name
  *  External link http://documentation/programming-guide/ failed: response 
code 0 means something's wrong.
 It's possible libcurl couldn't connect to the server or perhaps 
the request timed out.
 Sometimes, making too many requests at once also breaks things.
 Either way, the return message (if any) from the server is: 
Couldn't resolve host name
  *  External link http://images/gaming-example-basic.png failed: response code 
0 means something's wrong.
 It's possible libcurl couldn't connect to the server or perhaps 
the request timed out.
 Sometimes, making too many requests at once also breaks things.
 Either way, the return message (if any) from the server is: 
Couldn't resolve host name
  *  External link http://images/gaming-example-event-time-narrow.gif failed: 
response code 0 means something's wrong.
 It's possible libcurl couldn't connect to the server or perhaps 
the request timed out.
 Sometimes, making too many requests at once also breaks things.
 Either way, the return message (if any) from the server is: 
Couldn't resolve host name
  *  External link http://images/gaming-example-proc-time-narrow.gif failed: 
response code 0 means something's wrong.
 It's possible libcurl couldn't connect to the server or perhaps 
the request timed out.
 

Build failed in Jenkins: beam_PreCommit_Website_Stage_GCS_Cron #8

[Apache Jenkins Server]

See 


--
[...truncated 6.86 KB...]
Skipping task ':buildSrc:classes' as it has no actions.
:buildSrc:classes (Thread[Task worker for ':buildSrc' Thread 3,5,main]) 
completed. Took 0.0 secs.
:buildSrc:jar (Thread[Task worker for ':buildSrc' Thread 3,5,main]) started.

> Task :buildSrc:jar
Build cache key for task ':buildSrc:jar' is 7445e5c45b21f8a690f2f547fcb49594
Caching disabled for task ':buildSrc:jar': Caching has not been enabled for the 
task
Task ':buildSrc:jar' is not up-to-date because:
  No history is available.
:buildSrc:jar (Thread[Task worker for ':buildSrc' Thread 3,5,main]) completed. 
Took 0.106 secs.
:buildSrc:assemble (Thread[Task worker for ':buildSrc' Thread 3,5,main]) 
started.

> Task :buildSrc:assemble
Skipping task ':buildSrc:assemble' as it has no actions.
:buildSrc:assemble (Thread[Task worker for ':buildSrc' Thread 3,5,main]) 
completed. Took 0.0 secs.
:buildSrc:spotlessGroovy (Thread[Task worker for ':buildSrc' Thread 3,5,main]) 
started.

> Task :buildSrc:spotlessGroovy
file or directory 
'
 not found
file or directory 
'
 not found
file or directory 
'
 not found
Caching disabled for task ':buildSrc:spotlessGroovy': Caching has not been 
enabled for the task
Task ':buildSrc:spotlessGroovy' is not up-to-date because:
  No history is available.
All input files are considered out-of-date for incremental task 
':buildSrc:spotlessGroovy'.
file or directory 
'
 not found
:buildSrc:spotlessGroovy (Thread[Task worker for ':buildSrc' Thread 3,5,main]) 
completed. Took 1.217 secs.
:buildSrc:spotlessGroovyCheck (Thread[Task worker for ':buildSrc' Thread 
3,5,main]) started.

> Task :buildSrc:spotlessGroovyCheck
Skipping task ':buildSrc:spotlessGroovyCheck' as it has no actions.
:buildSrc:spotlessGroovyCheck (Thread[Task worker for ':buildSrc' Thread 
3,5,main]) completed. Took 0.001 secs.
:buildSrc:spotlessGroovyGradle (Thread[Task worker for ':buildSrc' Thread 
3,5,main]) started.

> Task :buildSrc:spotlessGroovyGradle
Caching disabled for task ':buildSrc:spotlessGroovyGradle': Caching has not 
been enabled for the task
Task ':buildSrc:spotlessGroovyGradle' is not up-to-date because:
  No history is available.
All input files are considered out-of-date for incremental task 
':buildSrc:spotlessGroovyGradle'.
:buildSrc:spotlessGroovyGradle (Thread[Task worker for ':buildSrc' Thread 
3,5,main]) completed. Took 0.027 secs.
:buildSrc:spotlessGroovyGradleCheck (Thread[Task worker for ':buildSrc' Thread 
3,5,main]) started.

> Task :buildSrc:spotlessGroovyGradleCheck
Skipping task ':buildSrc:spotlessGroovyGradleCheck' as it has no actions.
:buildSrc:spotlessGroovyGradleCheck (Thread[Task worker for ':buildSrc' Thread 
3,5,main]) completed. Took 0.0 secs.
:buildSrc:spotlessCheck (Thread[Task worker for ':buildSrc' Thread 3,5,main]) 
started.

> Task :buildSrc:spotlessCheck
Skipping task ':buildSrc:spotlessCheck' as it has no actions.
:buildSrc:spotlessCheck (Thread[Task worker for ':buildSrc' Thread 3,5,main]) 
completed. Took 0.0 secs.
:buildSrc:compileTestJava (Thread[Task worker for ':buildSrc' Thread 3,5,main]) 
started.

> Task :buildSrc:compileTestJava NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:compileTestJava' as it has no source files and no 
previous output files.
:buildSrc:compileTestJava (Thread[Task worker for ':buildSrc' Thread 3,5,main]) 
completed. Took 0.002 secs.
:buildSrc:compileTestGroovy (Thread[Task worker for ':buildSrc' Thread 
3,5,main]) started.

> Task :buildSrc:compileTestGroovy NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:compileTestGroovy' as it has no source files and no 
previous output files.
:buildSrc:compileTestGroovy (Thread[Task worker for ':buildSrc' Thread 
3,5,main]) completed. Took 0.003 secs.
:buildSrc:processTestResources (Thread[Task worker for ':buildSrc' Thread 
3,5,main]) started.

> Task :buildSrc:processTestResources NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:processTestResources' as it has no source files and no 
previous output files.
:buildSrc:processTestResources (Thread[Task worker for ':buildSrc' Thread 
3,5,main]) completed. Took 0.002 

[beam] branch asf-site updated: Publishing website 2018/10/06 12:00:44 at commit 324f0b3

[git-site-role]

This is an automated email from the ASF dual-hosted git repository.

git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/asf-site by this push:
 new 1006a8e  Publishing website 2018/10/06 12:00:44 at commit 324f0b3
1006a8e is described below

commit 1006a8eaf8d9d0228799236021cfb891f4bd1e0e
Author: jenkins 
AuthorDate: Sat Oct 6 12:00:44 2018 +

Publishing website 2018/10/06 12:00:44 at commit 324f0b3

[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=151969=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151969
 ]

ASF GitHub Bot logged work on BEAM-5315:


Author: ASF GitHub Bot
Created on: 06/Oct/18 11:47
Start Date: 06/Oct/18 11:47
Worklog Time Spent: 10m 
  Work Description: splovyt commented on a change in pull request #6590: 
[BEAM-5315] Partially port io
URL: https://github.com/apache/beam/pull/6590#discussion_r223180087
 
 

 ##
 File path: sdks/python/setup.py
 ##
 @@ -136,11 +136,17 @@ def get_version():
 'pyhamcrest>=1.9,<2.0',
 ]
 
+GCP_REQUIREMENTS_PY2_ONLY = [
+'googledatastore==7.0.1'
+]
+
+GCP_REQUIREMENTS_PY3_ONLY = [
+]
+
 GCP_REQUIREMENTS = [
 # oauth2client >=4 only works with google-apitools>=0.5.18.
 'google-apitools>=0.5.18,<=0.5.20',
 'proto-google-cloud-datastore-v1>=0.90.0,<=0.90.4',
-'googledatastore==7.0.1',
 
 Review comment:
   I have updated my setup.py to the latest version in the master for testing 
purposes and have added a version qualifier instead. Thanks for the feedback. 
PTAL at the next commit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151969)
Time Spent: 40m  (was: 0.5h)

> Finish Python 3 porting for io module
> -
>
> Key: BEAM-5315
> URL: https://issues.apache.org/jira/browse/BEAM-5315
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Simon
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-5668) [beam_PostCommit_Python_Verify] [:pythonPostCommit] Many tests failing

[Maximilian Michels (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maximilian Michels resolved BEAM-5668.
--
   Resolution: Fixed
Fix Version/s: Not applicable

PostCommit is back to normal.

> [beam_PostCommit_Python_Verify] [:pythonPostCommit] Many tests failing
> --
>
> Key: BEAM-5668
> URL: https://issues.apache.org/jira/browse/BEAM-5668
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Scott Wegner
>Assignee: Maximilian Michels
>Priority: Major
>  Labels: currently-failing
> Fix For: Not applicable
>
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> _Use this form to file an issue for test failure:_
>  * [Jenkins 
> Job|https://builds.apache.org/job/beam_PostCommit_Python_Verify/6181/]
>  * [Gradle Build 
> Scan|https://scans.gradle.com/s/fcpta7s7wcsnm/console-log?task=:beam-sdks-python:postCommitITTests]
>  * [Test source 
> code|https://github.com/apache/beam/blob/4c693e7aa1ba71fa83370dd383f4bcebe4690b86/sdks/python/apache_beam/io/gcp/pubsub_integration_test.py#L171]
>  * [Dataflow 
> job|https://pantheon.corp.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-10-05_07_00_20-5526009939236014896?project=apache-beam-testing]
> Initial investigation:
> Lots of different looking test failures from the Gradle build scan.
> 
> _After you've filled out the above details, please [assign the issue to an 
> individual|https://beam.apache.org/contribute/postcommits-guides/index.html#find_specialist].
>  Assignee should [treat test failures as 
> high-priority|https://beam.apache.org/contribute/postcommits-policies/#assigned-failing-test],
>  helping to fix the issue or find a more appropriate owner. See [Apache Beam 
> Post-Commit 
> Policies|https://beam.apache.org/contribute/postcommits-policies]._



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5668) [beam_PostCommit_Python_Verify] [:pythonPostCommit] Many tests failing

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5668?focusedWorklogId=151966=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151966
 ]

ASF GitHub Bot logged work on BEAM-5668:


Author: ASF GitHub Bot
Created on: 06/Oct/18 11:18
Start Date: 06/Oct/18 11:18
Worklog Time Spent: 10m 
  Work Description: mxm closed pull request #6588: [BEAM-5668] Revert 
"[BEAM-5442] Pass unknown SDK pipeline options to Runner (#6557)"
URL: https://github.com/apache/beam/pull/6588
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/python/apache_beam/options/pipeline_options.py 
b/sdks/python/apache_beam/options/pipeline_options.py
index bc8c962bcd4..a0059dbb381 100644
--- a/sdks/python/apache_beam/options/pipeline_options.py
+++ b/sdks/python/apache_beam/options/pipeline_options.py
@@ -20,7 +20,6 @@
 from __future__ import absolute_import
 
 import argparse
-import logging
 from builtins import list
 from builtins import object
 
@@ -214,19 +213,7 @@ def get_all_options(self, drop_default=False):
   subset[str(cls)] = cls
 for cls in subset.values():
   cls._add_argparse_args(parser)  # pylint: disable=protected-access
-known_args, unknown_args = parser.parse_known_args(self._flags)
-# Parse args which are not known at this point but might be recognized
-# at a later point in time, i.e. by the actual Runner.
-if unknown_args and unknown_args[0] != '':
-  logging.info("Parsing unknown args: %s", unknown_args)
-  for arg in unknown_args:
-if arg.startswith('--'):
-  parser.add_argument(arg.split('=', 1)[0], nargs='?')
-  # repeat parsing with unknown options added
-  known_args, unknown_args = parser.parse_known_args(self._flags)
-  if unknown_args:
-logging.warn("Discarding unparseable args: %s", unknown_args)
-
+known_args, _ = parser.parse_known_args(self._flags)
 result = vars(known_args)
 
 # Apply the overrides if any
diff --git a/sdks/python/apache_beam/options/pipeline_options_test.py 
b/sdks/python/apache_beam/options/pipeline_options_test.py
index 651e7336b39..9c14c25668e 100644
--- a/sdks/python/apache_beam/options/pipeline_options_test.py
+++ b/sdks/python/apache_beam/options/pipeline_options_test.py
@@ -45,24 +45,16 @@ def tearDown(self):
'display_data': [DisplayDataItemMatcher('num_workers', 5)]},
   {
   'flags': [
-  '--profile_cpu', '--profile_location', 'gs://bucket/',
-  'ignored', '-invalid=arg', '--unknown_arg', 'unknown_value',
-  '--unknown_flag'
-  ],
+  '--profile_cpu', '--profile_location', 'gs://bucket/', 
'ignored'],
   'expected': {
   'profile_cpu': True, 'profile_location': 'gs://bucket/',
   'mock_flag': False, 'mock_option': None,
-  'mock_multi_option': None,
-  'unknown_arg': 'unknown_value',
-  'unknown_flag': None},
+  'mock_multi_option': None},
   'display_data': [
   DisplayDataItemMatcher('profile_cpu',
  True),
   DisplayDataItemMatcher('profile_location',
- 'gs://bucket/'),
-  DisplayDataItemMatcher('unknown_arg',
- 'unknown_value')
-  ]
+ 'gs://bucket/')]
   },
   {'flags': ['--num_workers', '5', '--mock_flag'],
'expected': {'num_workers': 5,
diff --git a/sdks/python/apache_beam/runners/worker/sdk_worker_main_test.py 
b/sdks/python/apache_beam/runners/worker/sdk_worker_main_test.py
index e9b584a339f..6b5972e4ac4 100644
--- a/sdks/python/apache_beam/runners/worker/sdk_worker_main_test.py
+++ b/sdks/python/apache_beam/runners/worker/sdk_worker_main_test.py
@@ -56,8 +56,8 @@ def wrapped_method_for_test():
   def test_work_count_default_value(self):
 self._check_worker_count('{}', 12)
 
-  def test_parse_pipeline_options(self):
-expected_options = PipelineOptions([])
+  def test_parse_pipeine_options(self):
+expected_options = PipelineOptions()
 expected_options.view_as(
 SdkWorkerMainTest.MockOptions).m_m_option = [
 'worker_threads=1', 'beam_fn_api'


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151966)
Time 

[jira] [Work logged] (BEAM-5668) [beam_PostCommit_Python_Verify] [:pythonPostCommit] Many tests failing

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5668?focusedWorklogId=151965=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151965
 ]

ASF GitHub Bot logged work on BEAM-5668:


Author: ASF GitHub Bot
Created on: 06/Oct/18 11:18
Start Date: 06/Oct/18 11:18
Worklog Time Spent: 10m 
  Work Description: mxm commented on issue #6588: [BEAM-5668] Revert 
"[BEAM-5442] Pass unknown SDK pipeline options to Runner (#6557)"
URL: https://github.com/apache/beam/pull/6588#issuecomment-427565981
 
 
   Thanks @swegner for spotting this so quickly. We have fixed PostCommit via 
#6589. Will address the root cause in a follow-up. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151965)
Time Spent: 1h 50m  (was: 1h 40m)

> [beam_PostCommit_Python_Verify] [:pythonPostCommit] Many tests failing
> --
>
> Key: BEAM-5668
> URL: https://issues.apache.org/jira/browse/BEAM-5668
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Scott Wegner
>Assignee: Maximilian Michels
>Priority: Major
>  Labels: currently-failing
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> _Use this form to file an issue for test failure:_
>  * [Jenkins 
> Job|https://builds.apache.org/job/beam_PostCommit_Python_Verify/6181/]
>  * [Gradle Build 
> Scan|https://scans.gradle.com/s/fcpta7s7wcsnm/console-log?task=:beam-sdks-python:postCommitITTests]
>  * [Test source 
> code|https://github.com/apache/beam/blob/4c693e7aa1ba71fa83370dd383f4bcebe4690b86/sdks/python/apache_beam/io/gcp/pubsub_integration_test.py#L171]
>  * [Dataflow 
> job|https://pantheon.corp.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-10-05_07_00_20-5526009939236014896?project=apache-beam-testing]
> Initial investigation:
> Lots of different looking test failures from the Gradle build scan.
> 
> _After you've filled out the above details, please [assign the issue to an 
> individual|https://beam.apache.org/contribute/postcommits-guides/index.html#find_specialist].
>  Assignee should [treat test failures as 
> high-priority|https://beam.apache.org/contribute/postcommits-policies/#assigned-failing-test],
>  helping to fix the issue or find a more appropriate owner. See [Apache Beam 
> Post-Commit 
> Policies|https://beam.apache.org/contribute/postcommits-policies]._



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5442) PortableRunner swallows custom options for Runner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5442?focusedWorklogId=151964=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151964
 ]

ASF GitHub Bot logged work on BEAM-5442:


Author: ASF GitHub Bot
Created on: 06/Oct/18 11:16
Start Date: 06/Oct/18 11:16
Worklog Time Spent: 10m 
  Work Description: mxm commented on a change in pull request #6589: 
[BEAM-5442] Attempt to fix postcommit breakage.
URL: https://github.com/apache/beam/pull/6589#discussion_r223179379
 
 

 ##
 File path: sdks/python/apache_beam/options/pipeline_options.py
 ##
 @@ -220,12 +220,13 @@ def get_all_options(self, drop_default=False):
 if unknown_args and unknown_args[0] != '':
   logging.info("Parsing unknown args: %s", unknown_args)
   for arg in unknown_args:
-if arg.startswith('--'):
+# https://issues.apache.org/jira/browse/BEAM-5442
+if arg.startswith('--') and not arg.startswith('--beam_plugins'):
 
 Review comment:
   Will do a follow-up because this just cures a symptom. We need an extra 
check to avoid registering the same option twice.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151964)
Time Spent: 6h 20m  (was: 6h 10m)

> PortableRunner swallows custom options for Runner
> -
>
> Key: BEAM-5442
> URL: https://issues.apache.org/jira/browse/BEAM-5442
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core, sdk-py-core
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Major
>  Labels: portability, portability-flink
> Fix For: 2.8.0
>
>  Time Spent: 6h 20m
>  Remaining Estimate: 0h
>
> The PortableRunner doesn't pass custom PipelineOptions to the executing 
> Runner.
> Example: {{--parallelism=4}} won't be forwarded to the FlinkRunner.
> (The option is just removed during proto translation without any warning)
> We should allow some form of customization through the options, even for the 
> PortableRunner. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5442) PortableRunner swallows custom options for Runner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5442?focusedWorklogId=151963=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151963
 ]

ASF GitHub Bot logged work on BEAM-5442:


Author: ASF GitHub Bot
Created on: 06/Oct/18 11:15
Start Date: 06/Oct/18 11:15
Worklog Time Spent: 10m 
  Work Description: mxm commented on issue #6589: [BEAM-5442] Attempt to 
fix postcommit breakage.
URL: https://github.com/apache/beam/pull/6589#issuecomment-427565801
 
 
   Thanks for taking care of this @tweise.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151963)
Time Spent: 6h 10m  (was: 6h)

> PortableRunner swallows custom options for Runner
> -
>
> Key: BEAM-5442
> URL: https://issues.apache.org/jira/browse/BEAM-5442
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core, sdk-py-core
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Major
>  Labels: portability, portability-flink
> Fix For: 2.8.0
>
>  Time Spent: 6h 10m
>  Remaining Estimate: 0h
>
> The PortableRunner doesn't pass custom PipelineOptions to the executing 
> Runner.
> Example: {{--parallelism=4}} won't be forwarded to the FlinkRunner.
> (The option is just removed during proto translation without any warning)
> We should allow some form of customization through the options, even for the 
> PortableRunner. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5668) [beam_PostCommit_Python_Verify] [:pythonPostCommit] Many tests failing

[Maximilian Michels (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640685#comment-16640685
 ] 

Maximilian Michels commented on BEAM-5668:
--

Thanks [~swegner]. Yes, the error you posted is related to the changes. The 
same option is registered twice. Will make sure to run PostCommit for such 
changes (only ran Portable Runner tests).

> [beam_PostCommit_Python_Verify] [:pythonPostCommit] Many tests failing
> --
>
> Key: BEAM-5668
> URL: https://issues.apache.org/jira/browse/BEAM-5668
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Scott Wegner
>Assignee: Maximilian Michels
>Priority: Major
>  Labels: currently-failing
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> _Use this form to file an issue for test failure:_
>  * [Jenkins 
> Job|https://builds.apache.org/job/beam_PostCommit_Python_Verify/6181/]
>  * [Gradle Build 
> Scan|https://scans.gradle.com/s/fcpta7s7wcsnm/console-log?task=:beam-sdks-python:postCommitITTests]
>  * [Test source 
> code|https://github.com/apache/beam/blob/4c693e7aa1ba71fa83370dd383f4bcebe4690b86/sdks/python/apache_beam/io/gcp/pubsub_integration_test.py#L171]
>  * [Dataflow 
> job|https://pantheon.corp.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-10-05_07_00_20-5526009939236014896?project=apache-beam-testing]
> Initial investigation:
> Lots of different looking test failures from the Gradle build scan.
> 
> _After you've filled out the above details, please [assign the issue to an 
> individual|https://beam.apache.org/contribute/postcommits-guides/index.html#find_specialist].
>  Assignee should [treat test failures as 
> high-priority|https://beam.apache.org/contribute/postcommits-policies/#assigned-failing-test],
>  helping to fix the issue or find a more appropriate owner. See [Apache Beam 
> Post-Commit 
> Policies|https://beam.apache.org/contribute/postcommits-policies]._



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5668) [beam_PostCommit_Python_Verify] [:pythonPostCommit] Many tests failing

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5668?focusedWorklogId=151961=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151961
 ]

ASF GitHub Bot logged work on BEAM-5668:


Author: ASF GitHub Bot
Created on: 06/Oct/18 11:11
Start Date: 06/Oct/18 11:11
Worklog Time Spent: 10m 
  Work Description: mxm commented on a change in pull request #6588: 
[BEAM-5668] Revert "[BEAM-5442] Pass unknown SDK pipeline options to Runner 
(#6557)"
URL: https://github.com/apache/beam/pull/6588#discussion_r223179237
 
 

 ##
 File path: sdks/python/apache_beam/options/pipeline_options.py
 ##
 @@ -214,19 +213,7 @@ def get_all_options(self, drop_default=False):
   subset[str(cls)] = cls
 for cls in subset.values():
   cls._add_argparse_args(parser)  # pylint: disable=protected-access
-known_args, unknown_args = parser.parse_known_args(self._flags)
-# Parse args which are not known at this point but might be recognized
-# at a later point in time, i.e. by the actual Runner.
-if unknown_args and unknown_args[0] != '':
-  logging.info("Parsing unknown args: %s", unknown_args)
-  for arg in unknown_args:
-if arg.startswith('--'):
-  parser.add_argument(arg.split('=', 1)[0], nargs='?')
 
 Review comment:
   This works for `--myargname=myargvalue` or `--myargname myargvalue` syntax. 
The split will always return `--myargname`. Note that in the second case, the 
actual arguments passed are `--option` and `value`.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151961)
Time Spent: 1.5h  (was: 1h 20m)

> [beam_PostCommit_Python_Verify] [:pythonPostCommit] Many tests failing
> --
>
> Key: BEAM-5668
> URL: https://issues.apache.org/jira/browse/BEAM-5668
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Scott Wegner
>Assignee: Maximilian Michels
>Priority: Major
>  Labels: currently-failing
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> _Use this form to file an issue for test failure:_
>  * [Jenkins 
> Job|https://builds.apache.org/job/beam_PostCommit_Python_Verify/6181/]
>  * [Gradle Build 
> Scan|https://scans.gradle.com/s/fcpta7s7wcsnm/console-log?task=:beam-sdks-python:postCommitITTests]
>  * [Test source 
> code|https://github.com/apache/beam/blob/4c693e7aa1ba71fa83370dd383f4bcebe4690b86/sdks/python/apache_beam/io/gcp/pubsub_integration_test.py#L171]
>  * [Dataflow 
> job|https://pantheon.corp.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-10-05_07_00_20-5526009939236014896?project=apache-beam-testing]
> Initial investigation:
> Lots of different looking test failures from the Gradle build scan.
> 
> _After you've filled out the above details, please [assign the issue to an 
> individual|https://beam.apache.org/contribute/postcommits-guides/index.html#find_specialist].
>  Assignee should [treat test failures as 
> high-priority|https://beam.apache.org/contribute/postcommits-policies/#assigned-failing-test],
>  helping to fix the issue or find a more appropriate owner. See [Apache Beam 
> Post-Commit 
> Policies|https://beam.apache.org/contribute/postcommits-policies]._



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5668) [beam_PostCommit_Python_Verify] [:pythonPostCommit] Many tests failing

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5668?focusedWorklogId=151962=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151962
 ]

ASF GitHub Bot logged work on BEAM-5668:


Author: ASF GitHub Bot
Created on: 06/Oct/18 11:11
Start Date: 06/Oct/18 11:11
Worklog Time Spent: 10m 
  Work Description: mxm commented on a change in pull request #6588: 
[BEAM-5668] Revert "[BEAM-5442] Pass unknown SDK pipeline options to Runner 
(#6557)"
URL: https://github.com/apache/beam/pull/6588#discussion_r223179237
 
 

 ##
 File path: sdks/python/apache_beam/options/pipeline_options.py
 ##
 @@ -214,19 +213,7 @@ def get_all_options(self, drop_default=False):
   subset[str(cls)] = cls
 for cls in subset.values():
   cls._add_argparse_args(parser)  # pylint: disable=protected-access
-known_args, unknown_args = parser.parse_known_args(self._flags)
-# Parse args which are not known at this point but might be recognized
-# at a later point in time, i.e. by the actual Runner.
-if unknown_args and unknown_args[0] != '':
-  logging.info("Parsing unknown args: %s", unknown_args)
-  for arg in unknown_args:
-if arg.startswith('--'):
-  parser.add_argument(arg.split('=', 1)[0], nargs='?')
 
 Review comment:
   This works for `--myargname=myargvalue` or `--myargname myargvalue` syntax. 
The split will always return `--myargname`. Note that in the second case, the 
actual arguments passed are `--myargname` and `myargvalue`.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151962)
Time Spent: 1h 40m  (was: 1.5h)

> [beam_PostCommit_Python_Verify] [:pythonPostCommit] Many tests failing
> --
>
> Key: BEAM-5668
> URL: https://issues.apache.org/jira/browse/BEAM-5668
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Scott Wegner
>Assignee: Maximilian Michels
>Priority: Major
>  Labels: currently-failing
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> _Use this form to file an issue for test failure:_
>  * [Jenkins 
> Job|https://builds.apache.org/job/beam_PostCommit_Python_Verify/6181/]
>  * [Gradle Build 
> Scan|https://scans.gradle.com/s/fcpta7s7wcsnm/console-log?task=:beam-sdks-python:postCommitITTests]
>  * [Test source 
> code|https://github.com/apache/beam/blob/4c693e7aa1ba71fa83370dd383f4bcebe4690b86/sdks/python/apache_beam/io/gcp/pubsub_integration_test.py#L171]
>  * [Dataflow 
> job|https://pantheon.corp.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-10-05_07_00_20-5526009939236014896?project=apache-beam-testing]
> Initial investigation:
> Lots of different looking test failures from the Gradle build scan.
> 
> _After you've filled out the above details, please [assign the issue to an 
> individual|https://beam.apache.org/contribute/postcommits-guides/index.html#find_specialist].
>  Assignee should [treat test failures as 
> high-priority|https://beam.apache.org/contribute/postcommits-policies/#assigned-failing-test],
>  helping to fix the issue or find a more appropriate owner. See [Apache Beam 
> Post-Commit 
> Policies|https://beam.apache.org/contribute/postcommits-policies]._



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2887) Python SDK support for portable pipelines

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2887?focusedWorklogId=151952=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151952
 ]

ASF GitHub Bot logged work on BEAM-2887:


Author: ASF GitHub Bot
Created on: 06/Oct/18 08:27
Start Date: 06/Oct/18 08:27
Worklog Time Spent: 10m 
  Work Description: robertwb commented on issue #6504: [BEAM-2887] Remove 
special FnApi version of wordcount.
URL: https://github.com/apache/beam/pull/6504#issuecomment-427556457
 
 
   Run Python Postcommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151952)
Time Spent: 2h 10m  (was: 2h)

> Python SDK support for portable pipelines
> -
>
> Key: BEAM-2887
> URL: https://issues.apache.org/jira/browse/BEAM-2887
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Henning Rohde
>Assignee: Ahmet Altay
>Priority: Major
>  Labels: portability
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5644) make Planner configurable

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5644?focusedWorklogId=151947=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151947
 ]

ASF GitHub Bot logged work on BEAM-5644:


Author: ASF GitHub Bot
Created on: 06/Oct/18 06:33
Start Date: 06/Oct/18 06:33
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on issue #6598: [BEAM-5644] make 
planner impl configurable
URL: https://github.com/apache/beam/pull/6598#issuecomment-427550613
 
 
   Added options registrar and test cases.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151947)
Time Spent: 1h  (was: 50m)

> make Planner configurable 
> --
>
> Key: BEAM-5644
> URL: https://issues.apache.org/jira/browse/BEAM-5644
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> We can make planner configurable here: 
> [BeamQueryPlanner.java#L145|https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamQueryPlanner.java#L145]
>  
> By doing so, we can have different planner implementation to support 
> different SQL dialect.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5644) make Planner configurable

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5644?focusedWorklogId=151946=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151946
 ]

ASF GitHub Bot logged work on BEAM-5644:


Author: ASF GitHub Bot
Created on: 06/Oct/18 06:32
Start Date: 06/Oct/18 06:32
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on a change in pull request #6598: 
[BEAM-5644] make planner impl configurable
URL: https://github.com/apache/beam/pull/6598#discussion_r223173520
 
 

 ##
 File path: 
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamQueryPlanner.java
 ##
 @@ -97,6 +102,12 @@
 .typeSystem(connection.getTypeFactory().getTypeSystem())
 .operatorTable(ChainedSqlOperatorTable.of(opTab0, catalogReader))
 .build();
+
+options = PipelineOptionsFactory.create();
+  }
+
+  public void setOptions(PipelineOptions options) {
 
 Review comment:
   I updated this PR and keep/pass in `Constructor` to `BeamQueryPlanner`.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151946)
Time Spent: 50m  (was: 40m)

> make Planner configurable 
> --
>
> Key: BEAM-5644
> URL: https://issues.apache.org/jira/browse/BEAM-5644
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> We can make planner configurable here: 
> [BeamQueryPlanner.java#L145|https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamQueryPlanner.java#L145]
>  
> By doing so, we can have different planner implementation to support 
> different SQL dialect.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (BEAM-5627) Several IO tests fail in Python 3 when accessing a temporary file with TypeError: a bytes-like object is required, not 'str'

[Rakesh Kumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640578#comment-16640578
 ] 

Rakesh Kumar edited comment on BEAM-5627 at 10/6/18 6:11 AM:
-

[~tvalentyn] can you also add the command and tips to reproduce it?


was (Author: rakeshkumar):
[~tvalentyn] can you also add the command to reproduce it?

> Several IO tests fail in Python 3  when accessing a temporary file with  
> TypeError: a bytes-like object is required, not 'str'
> --
>
> Key: BEAM-5627
> URL: https://issues.apache.org/jira/browse/BEAM-5627
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Priority: Major
>
> ERROR: test_split_at_fraction_exhaustive 
> (apache_beam.io.source_test_utils_test.SourceTestUtilsTest)
>  --
>  Traceback (most recent call last):
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils_test.py",
>  line 120, in test_split_at_fraction_exhaustive
>  source = self._create_source(data)
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils_test.py",
>  line 43, in _create_source
>  source = LineSource(self._create_file_with_data(data))
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils_test.py",
>  line 35, in _create_file_with_data
>  f.write(line + '\n')
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/target/.tox/py3/lib/python3.5/tempfile.py",
>  line 622, in func_wrapper
>  return func(*args, **kwargs)
> TypeError: a bytes-like object is required, not 'str'
> Also similar:
> ==
>  ERROR: test_file_sink_writing 
> (apache_beam.io.filebasedsink_test.TestFileBasedSink)
> --
> Traceback (most recent call last):
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink_test.py", line 121, in 
> test_file_sink_writing
>   init_token, writer_results = self._common_init(sink)
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink_test.py", line 103, in _common_init
>   writer1 = sink.open_writer(init_token, '1')
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/options/value_provider.py", line 133, in _f
>   return fnc(self, *args, **kwargs)
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink.py", line 185, in open_writer
> return FileBasedSinkWriter(self, os.path.join(init_result, uid) + suffix)
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink.py", line 385, in __init__
>   self.temp_handle = self.sink.open(temp_shard_path)
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink_test.py", line 82, in open
>   file_handle.write('[start]')
>   TypeError: a bytes-like object is required, not 'str'



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5627) Several IO tests fail in Python 3 when accessing a temporary file with TypeError: a bytes-like object is required, not 'str'

[Rakesh Kumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640578#comment-16640578
 ] 

Rakesh Kumar commented on BEAM-5627:


[~tvalentyn] can you also add the command to reproduce it?

> Several IO tests fail in Python 3  when accessing a temporary file with  
> TypeError: a bytes-like object is required, not 'str'
> --
>
> Key: BEAM-5627
> URL: https://issues.apache.org/jira/browse/BEAM-5627
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Priority: Major
>
> ERROR: test_split_at_fraction_exhaustive 
> (apache_beam.io.source_test_utils_test.SourceTestUtilsTest)
>  --
>  Traceback (most recent call last):
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils_test.py",
>  line 120, in test_split_at_fraction_exhaustive
>  source = self._create_source(data)
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils_test.py",
>  line 43, in _create_source
>  source = LineSource(self._create_file_with_data(data))
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils_test.py",
>  line 35, in _create_file_with_data
>  f.write(line + '\n')
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/target/.tox/py3/lib/python3.5/tempfile.py",
>  line 622, in func_wrapper
>  return func(*args, **kwargs)
> TypeError: a bytes-like object is required, not 'str'
> Also similar:
> ==
>  ERROR: test_file_sink_writing 
> (apache_beam.io.filebasedsink_test.TestFileBasedSink)
> --
> Traceback (most recent call last):
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink_test.py", line 121, in 
> test_file_sink_writing
>   init_token, writer_results = self._common_init(sink)
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink_test.py", line 103, in _common_init
>   writer1 = sink.open_writer(init_token, '1')
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/options/value_provider.py", line 133, in _f
>   return fnc(self, *args, **kwargs)
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink.py", line 185, in open_writer
> return FileBasedSinkWriter(self, os.path.join(init_result, uid) + suffix)
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink.py", line 385, in __init__
>   self.temp_handle = self.sink.open(temp_shard_path)
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink_test.py", line 82, in open
>   file_handle.write('[start]')
>   TypeError: a bytes-like object is required, not 'str'



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Python_VR_Flink #258

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PreCommit_Website_Cron #145

[Apache Jenkins Server]

See 


--
[...truncated 166.36 KB...]
  *  External link http://images/logos/runners/spark.png failed: response code 
0 means something's wrong.
 It's possible libcurl couldn't connect to the server or perhaps 
the request timed out.
 Sometimes, making too many requests at once also breaks things.
 Either way, the return message (if any) from the server is: 
Couldn't resolve host name
  *  External link http://images/logos/sdks/go.png failed: response code 0 
means something's wrong.
 It's possible libcurl couldn't connect to the server or perhaps 
the request timed out.
 Sometimes, making too many requests at once also breaks things.
 Either way, the return message (if any) from the server is: 
Couldn't resolve host name
  *  External link http://images/logos/sdks/java.png failed: response code 0 
means something's wrong.
 It's possible libcurl couldn't connect to the server or perhaps 
the request timed out.
 Sometimes, making too many requests at once also breaks things.
 Either way, the return message (if any) from the server is: 
Couldn't resolve host name
  *  External link http://images/logos/sdks/python.png failed: response code 0 
means something's wrong.
 It's possible libcurl couldn't connect to the server or perhaps 
the request timed out.
 Sometimes, making too many requests at once also breaks things.
 Either way, the return message (if any) from the server is: 
Couldn't resolve host name
  *  External link http://images/logos/sdks/scala.png failed: response code 0 
means something's wrong.
 It's possible libcurl couldn't connect to the server or perhaps 
the request timed out.
 Sometimes, making too many requests at once also breaks things.
 Either way, the return message (if any) from the server is: 
Couldn't resolve host name
  *  http:// is an invalid URL (line 77)
 

  
- ./generated-content/get-started/downloads/index.html
  *  http:// is an invalid URL (line 77)
 

  
- ./generated-content/get-started/index.html
  *  External link http://get-started/beam-overview failed: response code 0 
means something's wrong.
 It's possible libcurl couldn't connect to the server or perhaps 
the request timed out.
 Sometimes, making too many requests at once also breaks things.
 Either way, the return message (if any) from the server is: 
Couldn't resolve host name
  *  External link http://get-started/mobile-gaming-example failed: response 
code 0 means something's wrong.
 It's possible libcurl couldn't connect to the server or perhaps 
the request timed out.
 Sometimes, making too many requests at once also breaks things.
 Either way, the return message (if any) from the server is: 
Couldn't resolve host name
  *  http:// is an invalid URL (line 77)
 

  
- ./generated-content/get-started/mobile-gaming-example/index.html
  *  External link http://documentation/programming-guide/ failed: response 
code 0 means something's wrong.
 It's possible libcurl couldn't connect to the server or perhaps 
the request timed out.
 Sometimes, making too many requests at once also breaks things.
 Either way, the return message (if any) from the server is: 
Couldn't resolve host name
  *  External link http://documentation/programming-guide/ failed: response 
code 0 means something's wrong.
 It's possible libcurl couldn't connect to the server or perhaps 
the request timed out.
 Sometimes, making too many requests at once also breaks things.
 Either way, the return message (if any) from the server is: 
Couldn't resolve host name
  *  External link http://images/gaming-example-basic.png failed: response code 
0 means something's wrong.
 It's possible libcurl couldn't connect to the server or perhaps 
the request timed out.
 Sometimes, making too many requests at once also breaks things.
 Either way, the return message (if any) from the server is: 
Couldn't resolve host name
  *  External link http://images/gaming-example-event-time-narrow.gif failed: 
response code 0 means something's wrong.
 It's possible libcurl couldn't connect to the server or perhaps 
the request timed out.
 Sometimes, making too many requests at once also breaks things.
 Either way, the return message (if any) from the server is: 
Couldn't resolve host name
  *  External link http://images/gaming-example-proc-time-narrow.gif failed: 
response code 0 means something's wrong.
 It's possible libcurl couldn't connect to the server or perhaps 
the request timed out.
 

[jira] [Comment Edited] (BEAM-5618) Several tests fail on Python 3 with: unsupported operand type(s) for +: 'int' and 'EmptySideInput'

[Rakesh Kumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640573#comment-16640573
 ] 

Rakesh Kumar edited comment on BEAM-5618 at 10/6/18 6:00 AM:
-

[~tvalentyn]: can you share the test command to reproduce this error?


was (Author: rakeshkumar):
[~tvalentyn]: can you share the command to reproduce this error?

> Several tests fail on Python 3 with: unsupported operand type(s) for +: 'int' 
> and 'EmptySideInput'
> --
>
> Key: BEAM-5618
> URL: https://issues.apache.org/jira/browse/BEAM-5618
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Priority: Major
>
> ERROR: test_do_with_side_input_as_arg 
> (apache_beam.transforms.ptransform_test.PTransformTest)
> --
> Traceback (most recent call last):
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/common.py",
>  line 677, in process
> self.do_fn_invoker.invoke_process(windowed_value)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/common.py",
>  line 529, in invoke_process
> windowed_value, additional_args, additional_kwargs, output_processor)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/common.py",
>  line 598, in _invoke_per_window
> windowed_value, self.process_method(*args_for_process))
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/ptransform_test.py",
>  line 135, in 
> lambda x, addon: [x + addon], pvalue.AsSingleton(side))
> TypeError: unsupported operand type(s) for +: 'int' and 'EmptySideInput'



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PreCommit_Website_Stage_GCS_Cron #7

[Apache Jenkins Server]

See 


--
[...truncated 6.80 KB...]
Skipping task ':buildSrc:classes' as it has no actions.
:buildSrc:classes (Thread[Task worker for ':buildSrc',5,main]) completed. Took 
0.0 secs.
:buildSrc:jar (Thread[Task worker for ':buildSrc',5,main]) started.

> Task :buildSrc:jar
Build cache key for task ':buildSrc:jar' is 7445e5c45b21f8a690f2f547fcb49594
Caching disabled for task ':buildSrc:jar': Caching has not been enabled for the 
task
Task ':buildSrc:jar' is not up-to-date because:
  No history is available.
:buildSrc:jar (Thread[Task worker for ':buildSrc',5,main]) completed. Took 
0.101 secs.
:buildSrc:assemble (Thread[Task worker for ':buildSrc',5,main]) started.

> Task :buildSrc:assemble
Skipping task ':buildSrc:assemble' as it has no actions.
:buildSrc:assemble (Thread[Task worker for ':buildSrc',5,main]) completed. Took 
0.0 secs.
:buildSrc:spotlessGroovy (Thread[Task worker for ':buildSrc',5,main]) started.

> Task :buildSrc:spotlessGroovy
file or directory 
'
 not found
file or directory 
'
 not found
file or directory 
'
 not found
Caching disabled for task ':buildSrc:spotlessGroovy': Caching has not been 
enabled for the task
Task ':buildSrc:spotlessGroovy' is not up-to-date because:
  No history is available.
All input files are considered out-of-date for incremental task 
':buildSrc:spotlessGroovy'.
file or directory 
'
 not found
:buildSrc:spotlessGroovy (Thread[Task worker for ':buildSrc',5,main]) 
completed. Took 1.211 secs.
:buildSrc:spotlessGroovyCheck (Thread[Task worker for ':buildSrc',5,main]) 
started.

> Task :buildSrc:spotlessGroovyCheck
Skipping task ':buildSrc:spotlessGroovyCheck' as it has no actions.
:buildSrc:spotlessGroovyCheck (Thread[Task worker for ':buildSrc',5,main]) 
completed. Took 0.001 secs.
:buildSrc:spotlessGroovyGradle (Thread[Task worker for ':buildSrc',5,main]) 
started.

> Task :buildSrc:spotlessGroovyGradle
Caching disabled for task ':buildSrc:spotlessGroovyGradle': Caching has not 
been enabled for the task
Task ':buildSrc:spotlessGroovyGradle' is not up-to-date because:
  No history is available.
All input files are considered out-of-date for incremental task 
':buildSrc:spotlessGroovyGradle'.
:buildSrc:spotlessGroovyGradle (Thread[Task worker for ':buildSrc',5,main]) 
completed. Took 0.027 secs.
:buildSrc:spotlessGroovyGradleCheck (Thread[Task worker for 
':buildSrc',5,main]) started.

> Task :buildSrc:spotlessGroovyGradleCheck
Skipping task ':buildSrc:spotlessGroovyGradleCheck' as it has no actions.
:buildSrc:spotlessGroovyGradleCheck (Thread[Task worker for 
':buildSrc',5,main]) completed. Took 0.0 secs.
:buildSrc:spotlessCheck (Thread[Task worker for ':buildSrc',5,main]) started.

> Task :buildSrc:spotlessCheck
Skipping task ':buildSrc:spotlessCheck' as it has no actions.
:buildSrc:spotlessCheck (Thread[Task worker for ':buildSrc',5,main]) completed. 
Took 0.0 secs.
:buildSrc:compileTestJava (Thread[Task worker for ':buildSrc',5,main]) started.

> Task :buildSrc:compileTestJava NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:compileTestJava' as it has no source files and no 
previous output files.
:buildSrc:compileTestJava (Thread[Task worker for ':buildSrc',5,main]) 
completed. Took 0.002 secs.
:buildSrc:compileTestGroovy (Thread[Task worker for ':buildSrc',5,main]) 
started.

> Task :buildSrc:compileTestGroovy NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:compileTestGroovy' as it has no source files and no 
previous output files.
:buildSrc:compileTestGroovy (Thread[Task worker for ':buildSrc',5,main]) 
completed. Took 0.002 secs.
:buildSrc:processTestResources (Thread[Task worker for ':buildSrc',5,main]) 
started.

> Task :buildSrc:processTestResources NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:processTestResources' as it has no source files and no 
previous output files.
:buildSrc:processTestResources (Thread[Task worker for ':buildSrc',5,main]) 
completed. Took 0.001 secs.
:buildSrc:testClasses (Thread[Task worker for ':buildSrc',5,main]) started.

> Task :buildSrc:testClasses UP-TO-DATE
Skipping task ':buildSrc:testClasses' as it has no actions.

[beam] branch asf-site updated: Publishing website 2018/10/06 06:00:46 at commit 324f0b3

[git-site-role]

This is an automated email from the ASF dual-hosted git repository.

git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/asf-site by this push:
 new 5791c01  Publishing website 2018/10/06 06:00:46 at commit 324f0b3
5791c01 is described below

commit 5791c010eac56b6fb82a01d3e3027301b5da87d9
Author: jenkins 
AuthorDate: Sat Oct 6 06:00:47 2018 +

Publishing website 2018/10/06 06:00:46 at commit 324f0b3

[jira] [Commented] (BEAM-5618) Several tests fail on Python 3 with: unsupported operand type(s) for +: 'int' and 'EmptySideInput'

[Rakesh Kumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640573#comment-16640573
 ] 

Rakesh Kumar commented on BEAM-5618:


[~tvalentyn]: can you share the command to reproduce this error?

> Several tests fail on Python 3 with: unsupported operand type(s) for +: 'int' 
> and 'EmptySideInput'
> --
>
> Key: BEAM-5618
> URL: https://issues.apache.org/jira/browse/BEAM-5618
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Priority: Major
>
> ERROR: test_do_with_side_input_as_arg 
> (apache_beam.transforms.ptransform_test.PTransformTest)
> --
> Traceback (most recent call last):
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/common.py",
>  line 677, in process
> self.do_fn_invoker.invoke_process(windowed_value)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/common.py",
>  line 529, in invoke_process
> windowed_value, additional_args, additional_kwargs, output_processor)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/common.py",
>  line 598, in _invoke_per_window
> windowed_value, self.process_method(*args_for_process))
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/ptransform_test.py",
>  line 135, in 
> lambda x, addon: [x + addon], pvalue.AsSingleton(side))
> TypeError: unsupported operand type(s) for +: 'int' and 'EmptySideInput'



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)