date:20180308

Build failed in Jenkins: beam_PerformanceTests_Spark #1446

2018-03-08 Thread Apache Jenkins Server

See 


Changes:

[apilloud] [Nexmark] Sickbay query 6

--
[...truncated 89.85 KB...]
'apache-beam-testing:bqjob_r57176e520b8cd5b_01620969b439_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-03-09 06:19:07,285 81f836b1 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-09 06:19:30,534 81f836b1 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-09 06:19:34,004 81f836b1 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  WallTime:0:03.44s,  CPU:0.36s,  MaxMemory:25524kb 
STDOUT: Upload complete.
Waiting on bqjob_r4e1affe3858f755b_0162096a1b5d_1 ... (0s) Current status: 
RUNNING 
 Waiting on bqjob_r4e1affe3858f755b_0162096a1b5d_1 ... (0s) 
Current status: DONE   
BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r4e1affe3858f755b_0162096a1b5d_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-03-09 06:19:34,005 81f836b1 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-09 06:19:51,074 81f836b1 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-09 06:19:53,873 81f836b1 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  WallTime:0:02.77s,  CPU:0.37s,  MaxMemory:25576kb 
STDOUT: Upload complete.
Waiting on bqjob_re1a2f0955ae6920_0162096a6a96_1 ... (0s) Current status: 
RUNNING 
Waiting on bqjob_re1a2f0955ae6920_0162096a6a96_1 ... (0s) 
Current status: DONE   
BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_re1a2f0955ae6920_0162096a6a96_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-03-09 06:19:53,873 81f836b1 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-09 06:20:16,934 81f836b1 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-09 06:20:20,323 81f836b1 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  WallTime:0:03.37s,  CPU:0.40s,  MaxMemory:25504kb 
STDOUT: Upload complete.
Waiting on bqjob_r4de5bcfee37f2378_0162096ad11a_1 ... (0s) Current status: 
RUNNING 
 Waiting on bqjob_r4de5bcfee37f2378_0162096ad11a_1 ... (0s) 
Current status: DONE   
BigQuery error in load operation: Error

Jenkins build is back to normal : beam_PerformanceTests_JDBC #308

2018-03-08 Thread Apache Jenkins Server

See

Build failed in Jenkins: beam_PerformanceTests_Python #1002

2018-03-08 Thread Apache Jenkins Server

See 


Changes:

[apilloud] [Nexmark] Sickbay query 6

--
[...truncated 1.11 KB...]
 > git rev-list --no-walk b3468b694fcef8fbed9cba669bbb56a5cc475a30 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins7846104817723222631.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins9020036577849177403.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins934742180121407879.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4000857054087463720.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in ./.env/lib/python2.7/site-packages
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5011738305493732179.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4176905410216264015.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Collecting numpy==1.13.3 (from -r PerfKitBenchmarker/requirements.txt (line 22))
:318:
 SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name 
Indication) extension to TLS is not available on this platform. This may cause 
the server to present an incorrect TLS certificate, which can cause validation 
failures. You can upgrade to a newer version of Python to solve this. For more 
information, see 
https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
:122:
 InsecurePlatformWarning: A true SSLContext object is not available. This 
prevents urllib3 from configuring SSL appropriately and may cause certain SSL 
connections to fail. You can upgrade to a newer version of Python to solve 
this. For more information, see 
https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
  Using cached numpy-1.13.3-cp27-cp27mu-manylinux1_x86_64.whl
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe>=0.23 in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenc

[jira] [Work logged] (BEAM-1251) Python 3 Support

2018-03-08 Thread ASF GitHub Bot (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-1251?focusedWorklogId=78807&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-78807
 ]

ASF GitHub Bot logged work on BEAM-1251:


Author: ASF GitHub Bot
Created on: 09/Mar/18 06:08
Start Date: 09/Mar/18 06:08
Worklog Time Spent: 10m 
  Work Description: cclauss commented on issue #4834: [BEAM-1251] Fix 
basestring for Python 3 - again
URL: https://github.com/apache/beam/pull/4834#issuecomment-371722689
 
 
   Retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 78807)
Time Spent: 4h 50m  (was: 4h 40m)

> Python 3 Support
> 
>
> Key: BEAM-1251
> URL: https://issues.apache.org/jira/browse/BEAM-1251
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Eyad Sibai
>Priority: Trivial
>  Time Spent: 4h 50m
>  Remaining Estimate: 0h
>
> I have been trying to use google datalab with python3. As I see there are 
> several packages that does not support python3 yet which google datalab 
> depends on. This is one of them.
> https://github.com/GoogleCloudPlatform/DataflowPythonSDK/issues/6



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-1251) Python 3 Support

2018-03-08 Thread ASF GitHub Bot (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-1251?focusedWorklogId=78806&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-78806
 ]

ASF GitHub Bot logged work on BEAM-1251:


Author: ASF GitHub Bot
Created on: 09/Mar/18 06:08
Start Date: 09/Mar/18 06:08
Worklog Time Spent: 10m 
  Work Description: cclauss commented on issue #4834: [BEAM-1251] Fix 
basestring for Python 3 - again
URL: https://github.com/apache/beam/pull/4834#issuecomment-371722689
 
 
   Retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 78806)
Time Spent: 4h 40m  (was: 4.5h)

> Python 3 Support
> 
>
> Key: BEAM-1251
> URL: https://issues.apache.org/jira/browse/BEAM-1251
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Eyad Sibai
>Priority: Trivial
>  Time Spent: 4h 40m
>  Remaining Estimate: 0h
>
> I have been trying to use google datalab with python3. As I see there are 
> several packages that does not support python3 yet which google datalab 
> depends on. This is one of them.
> https://github.com/GoogleCloudPlatform/DataflowPythonSDK/issues/6



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-1251) Python 3 Support

2018-03-08 Thread ASF GitHub Bot (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-1251?focusedWorklogId=78805&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-78805
 ]

ASF GitHub Bot logged work on BEAM-1251:


Author: ASF GitHub Bot
Created on: 09/Mar/18 06:07
Start Date: 09/Mar/18 06:07
Worklog Time Spent: 10m 
  Work Description: cclauss commented on issue #4834: [BEAM-1251] Fix 
basestring for Python 3 - again
URL: https://github.com/apache/beam/pull/4834#issuecomment-371722618
 
 
   Retest this


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 78805)
Time Spent: 4.5h  (was: 4h 20m)

> Python 3 Support
> 
>
> Key: BEAM-1251
> URL: https://issues.apache.org/jira/browse/BEAM-1251
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Eyad Sibai
>Priority: Trivial
>  Time Spent: 4.5h
>  Remaining Estimate: 0h
>
> I have been trying to use google datalab with python3. As I see there are 
> several packages that does not support python3 yet which google datalab 
> depends on. This is one of them.
> https://github.com/GoogleCloudPlatform/DataflowPythonSDK/issues/6



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-1251) Python 3 Support

2018-03-08 Thread ASF GitHub Bot (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-1251?focusedWorklogId=78804&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-78804
 ]

ASF GitHub Bot logged work on BEAM-1251:


Author: ASF GitHub Bot
Created on: 09/Mar/18 06:07
Start Date: 09/Mar/18 06:07
Worklog Time Spent: 10m 
  Work Description: cclauss commented on issue #4834: [BEAM-1251] Fix 
basestring for Python 3 - again
URL: https://github.com/apache/beam/pull/4834#issuecomment-371722618
 
 
   Retest this


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 78804)
Time Spent: 4h 20m  (was: 4h 10m)

> Python 3 Support
> 
>
> Key: BEAM-1251
> URL: https://issues.apache.org/jira/browse/BEAM-1251
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Eyad Sibai
>Priority: Trivial
>  Time Spent: 4h 20m
>  Remaining Estimate: 0h
>
> I have been trying to use google datalab with python3. As I see there are 
> several packages that does not support python3 yet which google datalab 
> depends on. This is one of them.
> https://github.com/GoogleCloudPlatform/DataflowPythonSDK/issues/6



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-1251) Python 3 Support

2018-03-08 Thread ASF GitHub Bot (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-1251?focusedWorklogId=78803&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-78803
 ]

ASF GitHub Bot logged work on BEAM-1251:


Author: ASF GitHub Bot
Created on: 09/Mar/18 06:07
Start Date: 09/Mar/18 06:07
Worklog Time Spent: 10m 
  Work Description: cclauss commented on issue #4834: [BEAM-1251] Fix 
basestring for Python 3 - again
URL: https://github.com/apache/beam/pull/4834#issuecomment-371722557
 
 
   Retest


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 78803)
Time Spent: 4h 10m  (was: 4h)

> Python 3 Support
> 
>
> Key: BEAM-1251
> URL: https://issues.apache.org/jira/browse/BEAM-1251
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Eyad Sibai
>Priority: Trivial
>  Time Spent: 4h 10m
>  Remaining Estimate: 0h
>
> I have been trying to use google datalab with python3. As I see there are 
> several packages that does not support python3 yet which google datalab 
> depends on. This is one of them.
> https://github.com/GoogleCloudPlatform/DataflowPythonSDK/issues/6



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-1251) Python 3 Support

2018-03-08 Thread ASF GitHub Bot (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-1251?focusedWorklogId=78801&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-78801
 ]

ASF GitHub Bot logged work on BEAM-1251:


Author: ASF GitHub Bot
Created on: 09/Mar/18 06:07
Start Date: 09/Mar/18 06:07
Worklog Time Spent: 10m 
  Work Description: cclauss commented on issue #4834: [BEAM-1251] Fix 
basestring for Python 3 - again
URL: https://github.com/apache/beam/pull/4834#issuecomment-371722557
 
 
   Retest


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 78801)
Time Spent: 4h  (was: 3h 50m)

> Python 3 Support
> 
>
> Key: BEAM-1251
> URL: https://issues.apache.org/jira/browse/BEAM-1251
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Eyad Sibai
>Priority: Trivial
>  Time Spent: 4h
>  Remaining Estimate: 0h
>
> I have been trying to use google datalab with python3. As I see there are 
> several packages that does not support python3 yet which google datalab 
> depends on. This is one of them.
> https://github.com/GoogleCloudPlatform/DataflowPythonSDK/issues/6



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-1251) Python 3 Support

2018-03-08 Thread ASF GitHub Bot (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-1251?focusedWorklogId=78800&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-78800
 ]

ASF GitHub Bot logged work on BEAM-1251:


Author: ASF GitHub Bot
Created on: 09/Mar/18 06:06
Start Date: 09/Mar/18 06:06
Worklog Time Spent: 10m 
  Work Description: cclauss commented on issue #4834: [BEAM-1251] Fix 
basestring for Python 3 - again
URL: https://github.com/apache/beam/pull/4834#issuecomment-371722492
 
 
   Please retest this


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 78800)
Time Spent: 3h 50m  (was: 3h 40m)

> Python 3 Support
> 
>
> Key: BEAM-1251
> URL: https://issues.apache.org/jira/browse/BEAM-1251
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Eyad Sibai
>Priority: Trivial
>  Time Spent: 3h 50m
>  Remaining Estimate: 0h
>
> I have been trying to use google datalab with python3. As I see there are 
> several packages that does not support python3 yet which google datalab 
> depends on. This is one of them.
> https://github.com/GoogleCloudPlatform/DataflowPythonSDK/issues/6



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PerformanceTests_TextIOIT #247

2018-03-08 Thread Apache Jenkins Server

See

[jira] [Work logged] (BEAM-1251) Python 3 Support

2018-03-08 Thread ASF GitHub Bot (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-1251?focusedWorklogId=78799&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-78799
 ]

ASF GitHub Bot logged work on BEAM-1251:


Author: ASF GitHub Bot
Created on: 09/Mar/18 06:06
Start Date: 09/Mar/18 06:06
Worklog Time Spent: 10m 
  Work Description: cclauss commented on issue #4834: [BEAM-1251] Fix 
basestring for Python 3 - again
URL: https://github.com/apache/beam/pull/4834#issuecomment-371722492
 
 
   Please retest this


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 78799)
Time Spent: 3h 40m  (was: 3.5h)

> Python 3 Support
> 
>
> Key: BEAM-1251
> URL: https://issues.apache.org/jira/browse/BEAM-1251
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Eyad Sibai
>Priority: Trivial
>  Time Spent: 3h 40m
>  Remaining Estimate: 0h
>
> I have been trying to use google datalab with python3. As I see there are 
> several packages that does not support python3 yet which google datalab 
> depends on. This is one of them.
> https://github.com/GoogleCloudPlatform/DataflowPythonSDK/issues/6



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Dataflow #5117

2018-03-08 Thread Apache Jenkins Server

See

[jira] [Work logged] (BEAM-3500) JdbcIO: Improve connection management

2018-03-08 Thread ASF GitHub Bot (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-3500?focusedWorklogId=78790&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-78790
 ]

ASF GitHub Bot logged work on BEAM-3500:


Author: ASF GitHub Bot
Created on: 09/Mar/18 05:19
Start Date: 09/Mar/18 05:19
Worklog Time Spent: 10m 
  Work Description: jbonofre commented on a change in pull request #4461: 
[BEAM-3500] "Attach" JDBC connection to the bundle and add DataSourceFactory 
allowing full control of the way the DataSource is created
URL: https://github.com/apache/beam/pull/4461#discussion_r173367376
 
 

 ##
 File path: 
sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java
 ##
 @@ -297,6 +364,39 @@ public DataSourceConfiguration withConnectionProperties(
   return builder().setConnectionProperties(connectionProperties).build();
 }
 
+/**
+ * Allows to tweak the {@link PoolingDataSource} created internally by 
{@link JdbcIO}.
+ */
+public DataSourceConfiguration withPoolConfiguration(int maxTotal,
 
 Review comment:
   OK, I wanted to give complete control to users. I think, only the 
validationQuery is required IMHO.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 78790)
Time Spent: 2.5h  (was: 2h 20m)

> JdbcIO: Improve connection management
> -
>
> Key: BEAM-3500
> URL: https://issues.apache.org/jira/browse/BEAM-3500
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-jdbc
>Affects Versions: 2.2.0
>Reporter: Pawel Bartoszek
>Assignee: Jean-Baptiste Onofré
>Priority: Major
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> JdbcIO write DoFn acquires connection in {{@Setup}} and release it in 
> {{@Teardown}} methods, which means that connection might stay opened for days 
> in streaming job case. Keeping single connection open for so long might be 
> very risky as it's exposed to database, network etc issues.
> *Taking connection from the pool when it is actually needed*
> I suggest that connection would be taken from the connection pool in 
> {{executeBatch}} method and released when the batch is flushed. This will 
> allow the pool to take care of any returned unhealthy connections etc.
> *Make JdbcIO accept data source factory*
>  It would be nice if JdbcIO accepted DataSourceFactory rather than DataSource 
> itself. I am saying that because sink checks if DataSource implements 
> `Serializable` interface, which make it impossible to pass 
> BasicDataSource(used internally by sink) as it doesn’t implement this 
> interface. Something like:
> {code:java}
> interface DataSourceFactory extends Serializable{
>  DataSource create();
> }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3500) JdbcIO: Improve connection management

2018-03-08 Thread ASF GitHub Bot (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-3500?focusedWorklogId=78788&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-78788
 ]

ASF GitHub Bot logged work on BEAM-3500:


Author: ASF GitHub Bot
Created on: 09/Mar/18 05:18
Start Date: 09/Mar/18 05:18
Worklog Time Spent: 10m 
  Work Description: jbonofre commented on a change in pull request #4461: 
[BEAM-3500] "Attach" JDBC connection to the bundle and add DataSourceFactory 
allowing full control of the way the DataSource is created
URL: https://github.com/apache/beam/pull/4461#discussion_r173367286
 
 

 ##
 File path: 
sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java
 ##
 @@ -327,12 +428,47 @@ DataSource buildDatasource() throws Exception{
 if (getConnectionProperties() != null && 
getConnectionProperties().get() != null) {
   
basicDataSource.setConnectionProperties(getConnectionProperties().get());
 }
-return basicDataSource;
+current = basicDataSource;
   }
+
+  // wrapping the datasource as a pooling datasource
+  DataSourceConnectionFactory connectionFactory = new 
DataSourceConnectionFactory(current);
+  PoolableConnectionFactory poolableConnectionFactory =
+  new PoolableConnectionFactory(connectionFactory, null);
+  GenericObjectPoolConfig poolConfig = new GenericObjectPoolConfig();
+  poolConfig.setMaxTotal(getPoolMaxTotal());
+  poolConfig.setBlockWhenExhausted(getPoolBlockWhenExhausted());
+  poolConfig.setMaxWaitMillis(getPoolMaxWaitMillis());
+  poolConfig.setMaxIdle(getPoolMaxIdle());
+  poolConfig.setMinIdle(getPoolMinIdle());
+  poolConfig.setTestOnBorrow(getPoolTestOnBorrow());
+  poolConfig.setTestOnReturn(getPoolTestOnReturn());
+  poolConfig.setNumTestsPerEvictionRun(getPoolNumTestsPerEvictionRun());
+  
poolConfig.setMinEvictableIdleTimeMillis(getPoolMinEvictableIdleTimeMillis());
+  poolConfig.setTestWhileIdle(getPoolTestWhileIdle());
+  
poolConfig.setSoftMinEvictableIdleTimeMillis(getPoolSoftMinEvictableIdleTimeMillis());
+  poolConfig.setLifo(getPoolLifo());
+  GenericObjectPool connectionPool =
+  new GenericObjectPool(poolableConnectionFactory, poolConfig);
+  poolableConnectionFactory.setPool(connectionPool);
+  poolableConnectionFactory.setValidationQuery("SELECT 1 FROM DUAL");
 
 Review comment:
   My bad, I forgot to update with the provided value. By default, the 
`validationQuery` should be null and the user can define it depending of his 
database.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 78788)
Time Spent: 2h 20m  (was: 2h 10m)

> JdbcIO: Improve connection management
> -
>
> Key: BEAM-3500
> URL: https://issues.apache.org/jira/browse/BEAM-3500
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-jdbc
>Affects Versions: 2.2.0
>Reporter: Pawel Bartoszek
>Assignee: Jean-Baptiste Onofré
>Priority: Major
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> JdbcIO write DoFn acquires connection in {{@Setup}} and release it in 
> {{@Teardown}} methods, which means that connection might stay opened for days 
> in streaming job case. Keeping single connection open for so long might be 
> very risky as it's exposed to database, network etc issues.
> *Taking connection from the pool when it is actually needed*
> I suggest that connection would be taken from the connection pool in 
> {{executeBatch}} method and released when the batch is flushed. This will 
> allow the pool to take care of any returned unhealthy connections etc.
> *Make JdbcIO accept data source factory*
>  It would be nice if JdbcIO accepted DataSourceFactory rather than DataSource 
> itself. I am saying that because sink checks if DataSource implements 
> `Serializable` interface, which make it impossible to pass 
> BasicDataSource(used internally by sink) as it doesn’t implement this 
> interface. Something like:
> {code:java}
> interface DataSourceFactory extends Serializable{
>  DataSource create();
> }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_Verify #4393

2018-03-08 Thread Apache Jenkins Server

See 


Changes:

[apilloud] [Nexmark] Sickbay query 6

--
[...truncated 1.02 MB...]
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py 
-> apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/evaluation_context.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/executor.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/helper_transforms.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/transform_evaluator.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/util.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/watermark_manager.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/experimental/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/experimental
copying apache_beam/runners/experimental/python_rpc_direct/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying 
apache_beam/runners/experimental/python_rpc_direct/python_rpc_direct_runner.py 
-> apache-beam-2.5.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/experimental/python_rpc_direct/server.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/job/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/manager.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/utils.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/job
copying apache_beam/runners/portability/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/fn_api_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/fn_api_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/maptask_executor_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/maptask_executor_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner_main.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/test/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/test
copying apache_beam/runners/worker/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/bundle_processor.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/data_plane.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/data_plane_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/log_handler.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/log_handler_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/logger.pxd -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/logger.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
cop

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1076

2018-03-08 Thread Apache Jenkins Server

See 


Changes:

[apilloud] [Nexmark] Sickbay query 6

--
[...truncated 118.50 KB...]
context.transforms.get_by_id(root_transform_id)]
  File 
"
 line 69, in get_by_id
self._id_to_proto[id], self._pipeline_context)
  File 
"
 line 842, in from_runner_api
part = context.transforms.get_by_id(transform_id)
  File 
"
 line 69, in get_by_id
self._id_to_proto[id], self._pipeline_context)
  File 
"
 line 842, in from_runner_api
part = context.transforms.get_by_id(transform_id)
  File 
"
 line 69, in get_by_id
self._id_to_proto[id], self._pipeline_context)
  File 
"
 line 833, in from_runner_api
transform=ptransform.PTransform.from_runner_api(proto.spec, context),
  File 
"
 line 555, in from_runner_api
context)
  File 
"
 line 888, in from_runner_api_parameter
result = ParDo(fn, *args, **kwargs)
  File 
"
 line 783, in __init__
super(ParDo, self).__init__(fn, *args, **kwargs)
  File 
"
 line 627, in __init__
self.fn = pickler.loads(pickler.dumps(self.fn))
  File 
"
 line 221, in loads
return dill.loads(s)
  File 
"
 line 277, in loads
return load(file)
  File 
"
 line 266, in load
obj = pik.load()
  File "/usr/lib/python2.7/pickle.py", line 858, in load
dispatch[key](self)
  File "/usr/lib/python2.7/pickle.py", line 1090, in load_global
klass = self.find_class(module, name)
  File 
"
 line 423, in find_class
return StockUnpickler.find_class(self, module, name)
  File "/usr/lib/python2.7/pickle.py", line 1126, in find_class
klass = getattr(mod, name)
  File 
"
 line 276, in signalhandler
raise TimedOutException()
TimedOutException: 'test_flattened_side_input 
(apache_beam.transforms.sideinputs_test.SideInputsTest)'

==
ERROR: test_iterable_side_input 
(apache_beam.transforms.sideinputs_test.SideInputsTest)
--
Traceback (most recent call last):
  File 
"
 line 812, in run
test(orig)
  File 
"
 line 45, in __call__
return self.run(*arg, **kwarg)
  File 
"
 line 133, in run
self.runTest(result)
  File 
"
 line 151, in runTest
test(result)
  F

Build failed in Jenkins: beam_PostCommit_Python_Verify #4392

2018-03-08 Thread Apache Jenkins Server

See 


--
[...truncated 1.02 MB...]
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py 
-> apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/evaluation_context.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/executor.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/helper_transforms.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/transform_evaluator.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/util.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/watermark_manager.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/experimental/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/experimental
copying apache_beam/runners/experimental/python_rpc_direct/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying 
apache_beam/runners/experimental/python_rpc_direct/python_rpc_direct_runner.py 
-> apache-beam-2.5.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/experimental/python_rpc_direct/server.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/job/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/manager.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/utils.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/job
copying apache_beam/runners/portability/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/fn_api_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/fn_api_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/maptask_executor_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/maptask_executor_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner_main.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/test/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/test
copying apache_beam/runners/worker/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/bundle_processor.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/data_plane.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/data_plane_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/log_handler.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/log_handler_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/logger.pxd -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/logger.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/logger_test.py -> 
apache-bea

[beam] branch master updated (b3468b6 -> 8844771)

2018-03-08 Thread kenn

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from b3468b6  Merge pull request #4745: [BEAM-3182] Ensure nexmark model 
tests produce output
 add ad06b0f  [Nexmark] Sickbay query 6
 new 8844771  Merge pull request #4837: [BEAM-3816] [Nexmark] Sickbay query 
6

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../src/test/java/org/apache/beam/sdk/nexmark/queries/QueryTest.java   | 3 +++
 1 file changed, 3 insertions(+)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[jira] [Work logged] (BEAM-3182) [Nexmark][SQL] Implement supported queries

2018-03-08 Thread ASF GitHub Bot (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-3182?focusedWorklogId=78780&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-78780
 ]

ASF GitHub Bot logged work on BEAM-3182:


Author: ASF GitHub Bot
Created on: 09/Mar/18 03:28
Start Date: 09/Mar/18 03:28
Worklog Time Spent: 10m 
  Work Description: kennknowles closed pull request #4836: Revert 
"[BEAM-3182] Ensure nexmark model tests produce output"
URL: https://github.com/apache/beam/pull/4836
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/sdks/java/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/NexmarkQueryModel.java
 
b/sdks/java/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/NexmarkQueryModel.java
index 5d7346be781..2efab3e8f59 100644
--- 
a/sdks/java/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/NexmarkQueryModel.java
+++ 
b/sdks/java/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/NexmarkQueryModel.java
@@ -102,7 +102,6 @@ static Instant windowStart(Duration size, Duration period, 
Instant timestamp) {
   /** Return assertion to use on results of pipeline for this query. */
   public SerializableFunction>, Void> 
assertionFor() {
 final Collection expectedStrings = 
toCollection(simulator().results());
-Assert.assertFalse(expectedStrings.isEmpty());
 
 return new SerializableFunction>, 
Void>() {
   @Override
diff --git 
a/sdks/java/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/Query6Model.java
 
b/sdks/java/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/Query6Model.java
index 0f625a2a91f..b5152d8a0bb 100644
--- 
a/sdks/java/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/Query6Model.java
+++ 
b/sdks/java/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/Query6Model.java
@@ -21,8 +21,6 @@
 import java.util.Collection;
 import java.util.Iterator;
 import java.util.Map;
-import java.util.PriorityQueue;
-import java.util.Queue;
 import java.util.TreeMap;
 import org.apache.beam.sdk.nexmark.NexmarkConfiguration;
 import org.apache.beam.sdk.nexmark.NexmarkUtils;
@@ -44,17 +42,17 @@
* Simulator for query 6.
*/
   private static class Simulator extends AbstractSimulator {
-/** The last 10 winning bids ordered by age, indexed by seller id. */
-private final Map> winningBidsPerSeller;
+/** The cumulative count of winning bids, indexed by seller id. */
+private final Map numWinningBidsPerSeller;
 
-/** The cumulative total of last 10 winning bid prices, indexed by seller 
id. */
+/** The cumulative total of winning bid prices, indexed by seller id. */
 private final Map totalWinningBidPricesPerSeller;
 
 private Instant lastTimestamp;
 
 public Simulator(NexmarkConfiguration configuration) {
   super(new WinningBidsSimulator(configuration).results());
-  winningBidsPerSeller = new TreeMap<>();
+  numWinningBidsPerSeller = new TreeMap<>();
   totalWinningBidPricesPerSeller = new TreeMap<>();
   lastTimestamp = BoundedWindow.TIMESTAMP_MIN_VALUE;
 }
@@ -64,24 +62,19 @@ public Simulator(NexmarkConfiguration configuration) {
  */
 private void captureWinningBid(Auction auction, Bid bid, Instant 
timestamp) {
   NexmarkUtils.info("winning auction, bid: %s, %s", auction, bid);
-  Queue queue = winningBidsPerSeller.get(auction.seller);
-  if (queue == null) {
-queue = new PriorityQueue(10,
-(Bid b1, Bid b2) -> Long.compare(b1.dateTime, b2.dateTime));
+  Long count = numWinningBidsPerSeller.get(auction.seller);
+  if (count == null) {
+count = 1L;
+  } else {
+count += 1;
   }
+  numWinningBidsPerSeller.put(auction.seller, count);
   Long total = totalWinningBidPricesPerSeller.get(auction.seller);
   if (total == null) {
-total = 0L;
-  }
-  int count = queue.size();
-  if (count == 10) {
-total -= queue.remove().price;
+total = bid.price;
   } else {
-count += 1;
+total += bid.price;
   }
-  queue.add(bid);
-  total += bid.price;
-  winningBidsPerSeller.put(auction.seller, queue);
   totalWinningBidPricesPerSeller.put(auction.seller, total);
   TimestampedValue intermediateResult = TimestampedValue.of(
   new SellerPrice(auction.seller, Math.round((double) total / count)), 
timestamp);
@@ -93,9 +86,9 @@ private void captureWinningBid(Auction auction, Bid bid, 
Instant timestamp) {
 protected void run() {
   TimestampedValue timestampedWinningBid = nextInput();
   if (timestampedWinningBid == null) {
-for (Map.Entry> entry : 
winningBidsPerSeller.entrySet

[beam] 01/01: Merge pull request #4837: [BEAM-3816] [Nexmark] Sickbay query 6

2018-03-08 Thread kenn

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 8844771d1fa8e70ffa8a15d260e967b73c794ada
Merge: b3468b6 ad06b0f
Author: Kenn Knowles 
AuthorDate: Thu Mar 8 19:28:51 2018 -0800

Merge pull request #4837: [BEAM-3816] [Nexmark] Sickbay query 6

 .../src/test/java/org/apache/beam/sdk/nexmark/queries/QueryTest.java   | 3 +++
 1 file changed, 3 insertions(+)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[jira] [Work logged] (BEAM-3816) [nexmark] Something is slightly off with Query 6

2018-03-08 Thread ASF GitHub Bot (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-3816?focusedWorklogId=78779&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-78779
 ]

ASF GitHub Bot logged work on BEAM-3816:


Author: ASF GitHub Bot
Created on: 09/Mar/18 03:28
Start Date: 09/Mar/18 03:28
Worklog Time Spent: 10m 
  Work Description: kennknowles closed pull request #4837: [BEAM-3816] 
[Nexmark] Sickbay query 6
URL: https://github.com/apache/beam/pull/4837
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/sdks/java/nexmark/src/test/java/org/apache/beam/sdk/nexmark/queries/QueryTest.java
 
b/sdks/java/nexmark/src/test/java/org/apache/beam/sdk/nexmark/queries/QueryTest.java
index 937a727819c..07e68f18bcb 100644
--- 
a/sdks/java/nexmark/src/test/java/org/apache/beam/sdk/nexmark/queries/QueryTest.java
+++ 
b/sdks/java/nexmark/src/test/java/org/apache/beam/sdk/nexmark/queries/QueryTest.java
@@ -34,6 +34,7 @@
 import org.apache.beam.sdk.testing.UsesTimersInParDo;
 import org.apache.beam.sdk.values.PCollection;
 import org.apache.beam.sdk.values.TimestampedValue;
+import org.junit.Ignore;
 import org.junit.Rule;
 import org.junit.Test;
 import org.junit.experimental.categories.Category;
@@ -205,12 +206,14 @@ public void sqlQuery5MatchesModelStreaming() {
 new Query5Model(CONFIG), true);
   }
 
+  @Ignore("https://issues.apache.org/jira/browse/BEAM-3816";)
   @Test
   @Category(NeedsRunner.class)
   public void query6MatchesModelBatch() {
 queryMatchesModel("Query6TestBatch", new Query6(CONFIG), new 
Query6Model(CONFIG), false);
   }
 
+  @Ignore("https://issues.apache.org/jira/browse/BEAM-3816";)
   @Test
   @Category(NeedsRunner.class)
   public void query6MatchesModelStreaming() {


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 78779)
Time Spent: 50m  (was: 40m)

> [nexmark] Something is slightly off with Query 6
> 
>
> Key: BEAM-3816
> URL: https://issues.apache.org/jira/browse/BEAM-3816
> Project: Beam
>  Issue Type: Bug
>  Components: examples-nexmark
>Reporter: Andrew Pilloud
>Priority: Major
>  Labels: easyfix, newbie, test
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> java.lang.AssertionError: Query6/Query6.Stamp/ParMultiDo(Anonymous).output: 
> wrong pipeline output Expected: <[\{"seller":1048,"price":83609648}, 
> \{"seller":1052,"price":61788353}, \{"seller":1086,"price":33744823}, 
> \{"seller":1078,"price":19876735}, \{"seller":1058,"price":50692833}, 
> \{"seller":1044,"price":6719489}, \{"seller":1096,"price":31287415}, 
> \{"seller":1095,"price":37004879}, \{"seller":1082,"price":22528654}, 
> \{"seller":1006,"price":57288736}, \{"seller":1051,"price":3967261}, 
> \{"seller":1084,"price":6394160}, \{"seller":1020,"price":3871757}, 
> \{"seller":1007,"price":185293}, \{"seller":1031,"price":11840889}, 
> \{"seller":1080,"price":26896442}, \{"seller":1030,"price":294928}, 
> \{"seller":1066,"price":26839191}, \{"seller":1000,"price":28257749}, 
> \{"seller":1055,"price":17087173}, \{"seller":1072,"price":45662210}, 
> \{"seller":1057,"price":4568399}, \{"seller":1025,"price":29008970}, 
> \{"seller":1064,"price":85810641}, \{"seller":1040,"price":99819658}, 
> \{"seller":1014,"price":11256690}, \{"seller":1098,"price":97259323}, 
> \{"seller":1011,"price":20447800}, \{"seller":1092,"price":77520938}, 
> \{"seller":1010,"price":53323687}, \{"seller":1060,"price":70032044}, 
> \{"seller":1062,"price":29076960}, \{"seller":1075,"price":19451464}, 
> \{"seller":1087,"price":27669185}, \{"seller":1009,"price":22951354}, 
> \{"seller":1065,"price":71875611}, \{"seller":1063,"price":87596779}, 
> \{"seller":1021,"price":62918185}, \{"seller":1034,"price":18472448}, 
> \{"seller":1028,"price":68556008}, \{"seller":1070,"price":92550447}]> but: 
> was <[\{"seller":1048,"price":83609648}, \{"seller":1052,"price":61788353}, 
> \{"seller":1086,"price":33744823}, \{"seller":1078,"price":19876735}, 
> \{"seller":1058,"price":50692833}, \{"seller":1044,"price":6719489}, 
> \{"seller":1096,"price":31287415}, \{"seller":1095,"price":37004879}, 
> \{"seller":1082,"price":22528654}, \{"seller":1006,"price":57288736}, 
> \{"seller":1051,"price":3967261}, \{"seller":1084,"price":6394160}, 
> \{"se

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1075

2018-03-08 Thread Apache Jenkins Server

See 


--
[...truncated 119.69 KB...]
return self.run(*args, **kwds)
  File "/usr/lib/python2.7/unittest/case.py", line 331, in run
testMethod()
  File 
"
 line 178, in test_iterable_side_input
pipeline.run()
  File 
"
 line 102, in run
result = super(TestPipeline, self).run()
  File 
"
 line 369, in run
self.to_runner_api(), self.runner, self._options).run(False)
  File 
"
 line 382, in run
return self.runner.run_pipeline(self)
  File 
"
 line 285, in run_pipeline
return_context=True)
  File 
"
 line 580, in to_runner_api
root_transform_id = context.transforms.get_id(self._root_transform())
  File 
"
 line 60, in get_id
self._id_to_proto[id] = obj.to_runner_api(self._pipeline_context)
  File 
"
 line 810, in to_runner_api
for part in self.parts],
  File 
"
 line 60, in get_id
self._id_to_proto[id] = obj.to_runner_api(self._pipeline_context)
  File 
"
 line 808, in to_runner_api
spec=transform_to_runner_api(self.transform, context),
  File 
"
 line 805, in transform_to_runner_api
return transform.to_runner_api(context)
  File 
"
 line 542, in to_runner_api
urn, typed_param = self.to_runner_api_parameter(context)
  File 
"
 line 876, in to_runner_api_parameter
for ix, si in enumerate(self.side_inputs)}))
  File 
"
 line 876, in 
for ix, si in enumerate(self.side_inputs)}))
  File 
"
 line 319, in to_runner_api
return self._side_input_data().to_runner_api(context)
  File 
"
 line 362, in to_runner_api
payload=pickler.dumps((self.view_fn, self.coder,
  File 
"
 line 193, in dumps
s = dill.dumps(o)
  File 
"
 line 259, in dumps
dump(obj, file, protocol, byref, fmode, recurse)#, strictio)
  File 
"
 line 252, in dump
pik.dump(obj)
  File "/usr/lib/python2.7/pickle.py", line 224, in dump
self.save(obj)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 548, in save_tuple
save(element)
  File "/usr/lib/python2.7/pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python2.7/pickle.py", line 419, in save_reduce
save(state)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File 
"

Build failed in Jenkins: beam_PostCommit_Python_Verify #4391

2018-03-08 Thread Apache Jenkins Server

See 


Changes:

[markliu] [BEAM-3750] Remove TestPipeline.covnertToArgs from integration tests

[markliu] Fix integration tests that use NumberedSharedFiles

[apilloud] [Nexmark] Use queue for Query 6 model

[apilloud] [Nexmark] Ensure enough data to produce output

--
[...truncated 1.02 MB...]
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py 
-> apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/evaluation_context.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/executor.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/helper_transforms.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/transform_evaluator.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/util.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/watermark_manager.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/experimental/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/experimental
copying apache_beam/runners/experimental/python_rpc_direct/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying 
apache_beam/runners/experimental/python_rpc_direct/python_rpc_direct_runner.py 
-> apache-beam-2.5.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/experimental/python_rpc_direct/server.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/job/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/manager.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/utils.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/job
copying apache_beam/runners/portability/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/fn_api_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/fn_api_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/maptask_executor_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/maptask_executor_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner_main.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/test/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/test
copying apache_beam/runners/worker/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/bundle_processor.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/data_plane.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/data_plane_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/log_handler.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/log_handler_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runn

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Dataflow #5116

2018-03-08 Thread Apache Jenkins Server

See 


Changes:

[markliu] [BEAM-3750] Remove TestPipeline.covnertToArgs from integration tests

[markliu] Fix integration tests that use NumberedSharedFiles

[herohde] [BEAM-3793] Validate provision response and add beamctl support

[apilloud] [Nexmark] Use queue for Query 6 model

[apilloud] [Nexmark] Ensure enough data to produce output

[altay] Fix topic URIs.

--
[...truncated 25.12 MB...]
[INFO] 2018-03-09T02:25:21.098Z: (367e26e6a0bd7fa): Executing operation 
Create.Values/Read(CreateSource)+ToString.Elements/MapElements/Map+PAssert$378/GroupGlobally/Window.Into()/Window.Assign+PAssert$378/GroupGlobally/GatherAllOutputs/Reify.Window/ParDo(Anonymous)+PAssert$378/GroupGlobally/GatherAllOutputs/WithKeys/AddKeys/Map+PAssert$378/GroupGlobally/GatherAllOutputs/Window.Into()/Window.Assign+PAssert$378/GroupGlobally/GatherAllOutputs/GroupByKey/Reify+PAssert$378/GroupGlobally/GatherAllOutputs/GroupByKey/Write
[INFO] 2018-03-09T02:25:22.386Z: (969345bcb52be922): Executing operation 
View.AsMap/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Close
[INFO] 2018-03-09T02:25:22.453Z: (969345bcb52be1eb): Executing operation 
View.AsMap/GBKaSVForKeys/Create
[INFO] 2018-03-09T02:25:22.485Z: (5d9068b74029f0d1): Executing operation 
View.AsMap/GBKaSVForSize/Create
[INFO] 2018-03-09T02:25:22.628Z: (969345bcb52be63e): Executing operation 
View.AsMap/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read+View.AsMap/ParMultiDo(ToIsmRecordForMapLike)+View.AsMap/GBKaSVForKeys/Write+View.AsMap/GBKaSVForSize/Write
[INFO] 2018-03-09T02:25:24.922Z: (e7be950b9f86733d): Workers have started 
successfully.
[INFO] 2018-03-09T02:25:25.377Z: (7eb71115d0a5e2a6): Workers have started 
successfully.
[INFO] 2018-03-09T02:25:26.475Z: (4fa1debe93a2e9e9): Autoscaling: Raised the 
number of workers to 0 based on the rate of progress in the currently running 
step(s).
[INFO] 2018-03-09T02:25:28.003Z: (5aa29130025c9177): Autoscaling: Raised the 
number of workers to 1 based on the rate of progress in the currently running 
step(s).
[INFO] Job 2018-03-08_18_20_54-5488195075092750080 finished with status DONE.
[INFO] Success result for Dataflow job 2018-03-08_18_20_54-5488195075092750080. 
Found 1 success, 0 failures out of 1 expected assertions.
[INFO] 2018-03-09T02:25:28.128Z: (b60aa05f580eb8ef): Autoscaling: Raised the 
number of workers to 0 based on the rate of progress in the currently running 
step(s).
[INFO] 2018-03-09T02:25:30.541Z: (24c578588b9f564c): Autoscaling: Resized 
worker pool from 1 to 0.
[INFO] 2018-03-09T02:25:30.562Z: (24c578588b9f5776): Autoscaling: Would further 
reduce the number of workers but reached the minimum number allowed for the job.
[INFO] 2018-03-09T02:25:32.972Z: (969345bcb52be7d0): Executing operation 
View.AsMap/GBKaSVForKeys/Close
[INFO] 2018-03-09T02:25:32.995Z: (5d9068b74029f5bd): Executing operation 
View.AsMap/GBKaSVForSize/Close
[INFO] 2018-03-09T02:25:33.033Z: (1677046f32e79462): Executing operation 
View.AsMap/GBKaSVForKeys/Read+View.AsMap/ParDo(ToIsmMetadataRecordForKey)
[INFO] 2018-03-09T02:25:33.066Z: (5d9068b74029fc9c): Executing operation 
View.AsMap/GBKaSVForSize/Read+View.AsMap/ParDo(ToIsmMetadataRecordForSize)
[INFO] Job 2018-03-08_18_21_11-12597411693115008135 finished with status DONE.
[INFO] Success result for Dataflow job 
2018-03-08_18_21_11-12597411693115008135. Found 1 success, 0 failures out of 1 
expected assertions.
[INFO] 2018-03-09T02:25:40.193Z: (ed73051423d3db94): Workers have started 
successfully.
[INFO] 2018-03-09T02:25:42.293Z: (4fa1debe93a2ec24): Autoscaling: Raised the 
number of workers to 1 based on the rate of progress in the currently running 
step(s).
[INFO] 2018-03-09T02:25:43.898Z: (b60aa05f580eb5c0): Autoscaling: Raised the 
number of workers to 1 based on the rate of progress in the currently running 
step(s).
[INFO] 2018-03-09T02:25:43.801Z: (1677046f32e79508): Executing operation 
View.AsMap/Flatten.PCollections
[INFO] 2018-03-09T02:25:43.965Z: (1677046f32e793ac): Executing operation 
View.AsMap/CreateDataflowView
[INFO] 2018-03-09T02:25:44.112Z: (1677046f32e794a5): Executing operation 
PAssert$370/GroupGlobally/GatherAllOutputs/GroupByKey/Create
[INFO] 2018-03-09T02:25:44.255Z: (969345bcb52bef47): Executing operation 
CreateMainInput/Create.Values/Read(CreateSource)+CreateMainInput/ParDo(ConvertTimestamps)+MainWindowInto/Window.Assign+OutputSideInputs+PAssert$370/GroupGlobally/Window.Into()/Window.Assign+PAssert$370/GroupGlobally/GatherAllOutputs/Reify.Window/ParDo(Anonymous)+PAssert$370/GroupGlobally/GatherAllOutputs/WithKeys/AddKeys/Map+PAssert$370/GroupGlobally/GatherAllOutputs/Window.Into()/Window.Assign+PAssert$370/GroupGlobally/GatherAllOutputs/GroupByKey/Reify+PAssert$370/GroupGlobally/GatherAllOutputs/GroupByKey/Write
[INFO] 2018-03-09T02:25:43.819Z: (e4920002087c678d): Workers have

[jira] [Created] (BEAM-3817) Incompatible input encoding running Tornadoes example on dataflow

2018-03-08 Thread Braden Bassingthwaite (JIRA)

Braden Bassingthwaite created BEAM-3817:
---

 Summary: Incompatible input encoding running Tornadoes example on 
dataflow
 Key: BEAM-3817
 URL: https://issues.apache.org/jira/browse/BEAM-3817
 Project: Beam
  Issue Type: Bug
  Components: sdk-go
Reporter: Braden Bassingthwaite
Assignee: Henning Rohde


Trying to run:

go run tornadoes.go --output=:bbass.tornadoes --project  
--runner dataflow --staging_location=gs://bbass/tornadoes 
--worker_harness_container_image=gcr.io//beam/go

Found here:

[https://github.com/apache/beam/blob/master/sdks/go/examples/cookbook/tornadoes/tornadoes.go]

I can run it locally but I get the error on Dataflow:

(8fa522c2bb03a769): Workflow failed. Causes: (8fa522c2bb03ab04): Incompatible 
input encoding. 

 

I built the worker_harness_container_image using:

mvn clean install -DskipTests -Pbuild-containers 
-Ddocker-repository-root=gcr.io//beam

 

Thanks!

 

Very excited to start using the golang beam sdk! great work!

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3327) Add abstractions to manage Environment Instance lifecycles.

2018-03-08 Thread ASF GitHub Bot (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-3327?focusedWorklogId=78768&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-78768
 ]

ASF GitHub Bot logged work on BEAM-3327:


Author: ASF GitHub Bot
Created on: 09/Mar/18 02:18
Start Date: 09/Mar/18 02:18
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on a change in pull request #4751: 
[BEAM-3327] Implement simple Docker container manager
URL: https://github.com/apache/beam/pull/4751#discussion_r173350125
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/SingletonDockerEnvironmentManager.java
 ##
 @@ -0,0 +1,107 @@
+package org.apache.beam.runners.fnexecution.environment;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.Arrays;
+import java.util.List;
+import java.util.concurrent.TimeoutException;
+import javax.annotation.concurrent.GuardedBy;
+import org.apache.beam.model.pipeline.v1.RunnerApi.Environment;
+import org.apache.beam.runners.fnexecution.GrpcFnServer;
+import org.apache.beam.runners.fnexecution.artifact.ArtifactRetrievalService;
+import 
org.apache.beam.runners.fnexecution.control.SdkHarnessClientControlService;
+import org.apache.beam.runners.fnexecution.logging.GrpcLoggingService;
+import 
org.apache.beam.runners.fnexecution.provisioning.StaticGrpcProvisionService;
+
+/** An {@link EnvironmentManager} that manages a single docker container. */
+public class SingletonDockerEnvironmentManager implements EnvironmentManager {
+
+  public static SingletonDockerEnvironmentManager forServices(
+  DockerWrapper docker,
+  GrpcFnServer controlServiceServer,
+  GrpcFnServer loggingServiceServer,
+  GrpcFnServer retrievalServiceServer,
+  GrpcFnServer provisioningServiceServer) {
+return new SingletonDockerEnvironmentManager(docker, controlServiceServer, 
loggingServiceServer,
+retrievalServiceServer, provisioningServiceServer);
+  }
+
+  private final Object lock = new Object();
+  private final DockerWrapper docker;
+  private final GrpcFnServer 
controlServiceServer;
+  private final GrpcFnServer loggingServiceServer;
+  private final GrpcFnServer retrievalServiceServer;
+  private final GrpcFnServer 
provisioningServiceServer;
+
+  @GuardedBy("lock")
+  private RemoteEnvironment dockerEnvironment = null;
+
+  private SingletonDockerEnvironmentManager(
+  DockerWrapper docker,
+  GrpcFnServer controlServiceServer,
+  GrpcFnServer loggingServiceServer,
+  GrpcFnServer retrievalServiceServer,
+  GrpcFnServer provisioningServiceServer) {
+this.docker = docker;
+this.controlServiceServer = controlServiceServer;
+this.loggingServiceServer = loggingServiceServer;
+this.retrievalServiceServer = retrievalServiceServer;
+this.provisioningServiceServer = provisioningServiceServer;
+  }
+
+  /**
+   * Retrieve a handle for the given environment. The same environment must be 
requested every time.
+   * The same remote handle is returned to every caller, so the environment 
cannot be used once
+   * closed.
+   */
+  @Override
+  public RemoteEnvironment getEnvironment(Environment environment) throws 
Exception {
+synchronized (lock) {
+  if (dockerEnvironment == null) {
+dockerEnvironment = createDockerEnv(environment);
+  } else {
+checkArgument(
+
environment.getUrl().equals(dockerEnvironment.getEnvironment().getUrl()),
+"A %s must only be queried for a single %s. Existing %s, Argument 
%s",
+SingletonDockerEnvironmentManager.class.getSimpleName(),
+Environment.class.getSimpleName(),
+dockerEnvironment.getEnvironment().getUrl(),
+environment.getUrl());
+  }
+  return dockerEnvironment;
+}
+  }
+
+  private DockerContainerEnvironment createDockerEnv(Environment environment)
+  throws IOException, TimeoutException, InterruptedException {
+// TODO: Generate environment id correctly.
+String environmentId = Long.toString(-123);
+Path workerPersistentDirectory = 
Files.createTempDirectory("worker_persistent_directory");
+Path semiPersistentDirectory = 
Files.createTempDirectory("semi_persistent_dir");
+String containerImage = environment.getUrl();
+// TODO: The default service address will not work for Docker for Mac.
+String loggingEndpoint = 
loggingServiceServer.getApiServiceDescriptor().getUrl();
+String artifactEndpoint = 
retrievalServiceServer.getApiServiceDescriptor().getUrl();
+String provisionEndpoint = 
provisioningServiceServer.getApiServiceDescriptor().getUrl();
+String controlEndpoint = 
controlServiceServer.getApiServiceDescriptor().getUrl();
+List args = Arrays.asList(
+"-v",
+

[jira] [Work logged] (BEAM-3734) Add performance tests for XmlIO

2018-03-08 Thread ASF GitHub Bot (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-3734?focusedWorklogId=78765&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-78765
 ]

ASF GitHub Bot logged work on BEAM-3734:


Author: ASF GitHub Bot
Created on: 09/Mar/18 01:55
Start Date: 09/Mar/18 01:55
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on issue #4747: [BEAM-3734] 
Performance tests for XmlIO 
URL: https://github.com/apache/beam/pull/4747#issuecomment-371687696
 
 
   Retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 78765)
Time Spent: 8h  (was: 7h 50m)

> Add performance tests for XmlIO
> ---
>
> Key: BEAM-3734
> URL: https://issues.apache.org/jira/browse/BEAM-3734
> Project: Beam
>  Issue Type: Test
>  Components: testing
>Reporter: Łukasz Gajowy
>Assignee: Łukasz Gajowy
>Priority: Major
>  Time Spent: 8h
>  Remaining Estimate: 0h
>
> We should add a large scale performance test for XmlIO as described in 
> documentation [1].
> [1] 
> https://beam.apache.org/documentation/io/testing/#i-o-transform-integration-tests



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1074

2018-03-08 Thread Apache Jenkins Server

See 


Changes:

[markliu] [BEAM-3750] Remove TestPipeline.covnertToArgs from integration tests

[markliu] Fix integration tests that use NumberedSharedFiles

[apilloud] [Nexmark] Use queue for Query 6 model

[apilloud] [Nexmark] Ensure enough data to produce output

--
[...truncated 124.85 KB...]
return m(transform, input)
  File 
"
 line 180, in apply_PTransform
return transform.expand(input)
  File 
"
 line 147, in expand
| "Match" >> Map(matcher))
  File 
"
 line 820, in __ror__
return self.transform.__ror__(pvalueish, self.label)
  File 
"
 line 488, in __ror__
result = p.apply(self, pvalueish, label)
  File 
"
 line 443, in apply
return self.apply(transform, pvalueish)
  File 
"
 line 479, in apply
pvalueish_result = self.runner.apply(transform, pvalueish)
  File 
"
 line 174, in apply
return m(transform, input)
  File 
"
 line 180, in apply_PTransform
return transform.expand(input)
  File 
"
 line 165, in expand
| Map(_merge_tagged_vals_under_key, result_ctor, result_ctor_arg))
  File 
"
 line 990, in Map
pardo = FlatMap(wrapper, *args, **kwargs)
  File 
"
 line 941, in FlatMap
pardo = ParDo(CallableWrapperDoFn(fn), *args, **kwargs)
  File 
"
 line 783, in __init__
super(ParDo, self).__init__(fn, *args, **kwargs)
  File 
"
 line 627, in __init__
self.fn = pickler.loads(pickler.dumps(self.fn))
  File 
"
 line 193, in dumps
s = dill.dumps(o)
  File 
"
 line 259, in dumps
dump(obj, file, protocol, byref, fmode, recurse)#, strictio)
  File 
"
 line 252, in dump
pik.dump(obj)
  File "/usr/lib/python2.7/pickle.py", line 224, in dump
self.save(obj)
  File "/usr/lib/python2.7/pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python2.7/pickle.py", line 419, in save_reduce
save(state)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File 
"
 line 165, in new_save_module_dict
return old_save_module_dict(pickler, obj)
  File 
"
 line 841, in save_module_dict
StockPickler.save_dict(pickler, obj)
  File "/usr/lib/python2.7/pickle.py", line 649, in save_dict
self._batch_setitems(obj.iteritems())
  File "/usr/lib/python2.7/pickle.py", line 681, in _batch_setitems
save(v)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File 
"

[jira] [Updated] (BEAM-3815) 2.4.0 RC2 uses java worker version beam-master-20180228, should be 2.4.0 or beam-2.4.0

2018-03-08 Thread Robert Bradshaw (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-3815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Bradshaw updated BEAM-3815:
--
Fix Version/s: 2.4.0

> 2.4.0 RC2 uses java worker version beam-master-20180228, should be 2.4.0 or 
> beam-2.4.0
> --
>
> Key: BEAM-3815
> URL: https://issues.apache.org/jira/browse/BEAM-3815
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Affects Versions: 2.4.0
>Reporter: Alan Myrvold
>Assignee: Robert Bradshaw
>Priority: Blocker
> Fix For: 2.4.0
>
>
> Running dataflow quickstart with the 2.4.0 RC2 uses a container named 
> beam-master-20180228, should be 2.4.0 or beam-2.4.0, as in previous releases.
> For 2.3.0, this version was set on the release branch with 
> [https://github.com/apache/beam/commit/245cb4cbec9c1dd7d4f9ff8eb3ab38667c5472e9#diff-0032670e9006c6eba52569423c64e6d2]
> Perhaps the release guide needs to be updated?
>  wget 
> https://repository.apache.org/content/repositories/orgapachebeam-1030/org/apache/beam/beam-runners-google-cloud-dataflow-java/2.4.0/beam-runners-google-cloud-dataflow-java-2.4.0.jar
> jar xf beam-runners-google-cloud-dataflow-java-2.4.0.jar 
> org/apache/beam/runners/dataflow/dataflow.properties
> grep container.version org/apache/beam/runners/dataflow/dataflow.properties



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch release-2.4.0 updated: Update dataflow container version.

2018-03-08 Thread robertwb

This is an automated email from the ASF dual-hosted git repository.

robertwb pushed a commit to branch release-2.4.0
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/release-2.4.0 by this push:
 new 237130c  Update dataflow container version.
237130c is described below

commit 237130cb718c7e939b80e47f2ed60a3018302d5e
Author: Robert Bradshaw 
AuthorDate: Thu Mar 8 16:59:59 2018 -0800

Update dataflow container version.
---
 runners/google-cloud-dataflow-java/pom.xml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/runners/google-cloud-dataflow-java/pom.xml 
b/runners/google-cloud-dataflow-java/pom.xml
index ab66211..fbcf129 100644
--- a/runners/google-cloud-dataflow-java/pom.xml
+++ b/runners/google-cloud-dataflow-java/pom.xml
@@ -33,7 +33,7 @@
   jar
 
   
-
beam-master-20180228
+beam-2.4.0
 
1
 
7
   

-- 
To stop receiving notification emails like this one, please contact
rober...@apache.org.

[jira] [Work logged] (BEAM-3816) [nexmark] Something is slightly off with Query 6

2018-03-08 Thread ASF GitHub Bot (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-3816?focusedWorklogId=78760&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-78760
 ]

ASF GitHub Bot logged work on BEAM-3816:


Author: ASF GitHub Bot
Created on: 09/Mar/18 00:45
Start Date: 09/Mar/18 00:45
Worklog Time Spent: 10m 
  Work Description: apilloud commented on a change in pull request #4837: 
[BEAM-3816] [Nexmark] Sickbay query 6
URL: https://github.com/apache/beam/pull/4837#discussion_r173337436
 
 

 ##
 File path: 
sdks/java/nexmark/src/test/java/org/apache/beam/sdk/nexmark/queries/QueryTest.java
 ##
 @@ -205,12 +206,14 @@ public void sqlQuery5MatchesModelStreaming() {
 new Query5Model(CONFIG), true);
   }
 
+  @Ignore("TODO: BEAM-3816")
 
 Review comment:
   Done.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 78760)
Time Spent: 40m  (was: 0.5h)

> [nexmark] Something is slightly off with Query 6
> 
>
> Key: BEAM-3816
> URL: https://issues.apache.org/jira/browse/BEAM-3816
> Project: Beam
>  Issue Type: Bug
>  Components: examples-nexmark
>Reporter: Andrew Pilloud
>Priority: Major
>  Labels: easyfix, newbie, test
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> java.lang.AssertionError: Query6/Query6.Stamp/ParMultiDo(Anonymous).output: 
> wrong pipeline output Expected: <[\{"seller":1048,"price":83609648}, 
> \{"seller":1052,"price":61788353}, \{"seller":1086,"price":33744823}, 
> \{"seller":1078,"price":19876735}, \{"seller":1058,"price":50692833}, 
> \{"seller":1044,"price":6719489}, \{"seller":1096,"price":31287415}, 
> \{"seller":1095,"price":37004879}, \{"seller":1082,"price":22528654}, 
> \{"seller":1006,"price":57288736}, \{"seller":1051,"price":3967261}, 
> \{"seller":1084,"price":6394160}, \{"seller":1020,"price":3871757}, 
> \{"seller":1007,"price":185293}, \{"seller":1031,"price":11840889}, 
> \{"seller":1080,"price":26896442}, \{"seller":1030,"price":294928}, 
> \{"seller":1066,"price":26839191}, \{"seller":1000,"price":28257749}, 
> \{"seller":1055,"price":17087173}, \{"seller":1072,"price":45662210}, 
> \{"seller":1057,"price":4568399}, \{"seller":1025,"price":29008970}, 
> \{"seller":1064,"price":85810641}, \{"seller":1040,"price":99819658}, 
> \{"seller":1014,"price":11256690}, \{"seller":1098,"price":97259323}, 
> \{"seller":1011,"price":20447800}, \{"seller":1092,"price":77520938}, 
> \{"seller":1010,"price":53323687}, \{"seller":1060,"price":70032044}, 
> \{"seller":1062,"price":29076960}, \{"seller":1075,"price":19451464}, 
> \{"seller":1087,"price":27669185}, \{"seller":1009,"price":22951354}, 
> \{"seller":1065,"price":71875611}, \{"seller":1063,"price":87596779}, 
> \{"seller":1021,"price":62918185}, \{"seller":1034,"price":18472448}, 
> \{"seller":1028,"price":68556008}, \{"seller":1070,"price":92550447}]> but: 
> was <[\{"seller":1048,"price":83609648}, \{"seller":1052,"price":61788353}, 
> \{"seller":1086,"price":33744823}, \{"seller":1078,"price":19876735}, 
> \{"seller":1058,"price":50692833}, \{"seller":1044,"price":6719489}, 
> \{"seller":1096,"price":31287415}, \{"seller":1095,"price":37004879}, 
> \{"seller":1082,"price":22528654}, \{"seller":1006,"price":57288736}, 
> \{"seller":1051,"price":3967261}, \{"seller":1084,"price":6394160}, 
> \{"seller":1000,"price":34395558}, \{"seller":1020,"price":3871757}, 
> \{"seller":1007,"price":185293}, \{"seller":1031,"price":11840889}, 
> \{"seller":1080,"price":26896442}, \{"seller":1030,"price":294928}, 
> \{"seller":1066,"price":26839191}, \{"seller":1055,"price":17087173}, 
> \{"seller":1072,"price":45662210}, \{"seller":1057,"price":4568399}, 
> \{"seller":1025,"price":29008970}, \{"seller":1064,"price":85810641}, 
> \{"seller":1040,"price":99819658}, \{"seller":1014,"price":11256690}, 
> \{"seller":1098,"price":97259323}, \{"seller":1011,"price":20447800}, 
> \{"seller":1092,"price":77520938}, \{"seller":1010,"price":53323687}, 
> \{"seller":1060,"price":70032044}, \{"seller":1062,"price":29076960}, 
> \{"seller":1075,"price":19451464}, \{"seller":1087,"price":27669185}, 
> \{"seller":1009,"price":22951354}, \{"seller":1065,"price":71875611}, 
> \{"seller":1063,"price":87596779}, \{"seller":1021,"price":62918185}, 
> \{"seller":1034,"price":18472448}, \{"seller":1028,"price":68556008}, 
> \{"seller":1070,"price":92550447}]>



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_Spark #1445

2018-03-08 Thread Apache Jenkins Server

See 


Changes:

[markliu] [BEAM-3750] Remove TestPipeline.covnertToArgs from integration tests

[markliu] Fix integration tests that use NumberedSharedFiles

[apilloud] [Nexmark] Use queue for Query 6 model

[apilloud] [Nexmark] Ensure enough data to produce output

--
[...truncated 92.15 KB...]
'apache-beam-testing:bqjob_r726b057d014a8cd7_016208206011_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-03-09 00:19:24,290 18468a6d MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-09 00:19:44,186 18468a6d MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-09 00:19:46,447 18468a6d MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  WallTime:0:02.25s,  CPU:0.28s,  MaxMemory:25544kb 
STDOUT: Upload complete.
Waiting on bqjob_r75fbd209342dcca_01620820b6de_1 ... (0s) Current status: 
RUNNING 
Waiting on bqjob_r75fbd209342dcca_01620820b6de_1 ... (0s) 
Current status: DONE   
BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r75fbd209342dcca_01620820b6de_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-03-09 00:19:46,448 18468a6d MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-09 00:20:12,249 18468a6d MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-09 00:20:17,879 18468a6d MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  WallTime:0:05.62s,  CPU:0.30s,  MaxMemory:27776kb 
STDOUT: Upload complete.
Waiting on bqjob_r6f01b160bf2360e_016208212481_1 ... (0s) Current status: 
RUNNING 
Waiting on bqjob_r6f01b160bf2360e_016208212481_1 ... (1s) 
Current status: RUNNING 
Waiting on 
bqjob_r6f01b160bf2360e_016208212481_1 ... (2s) Current status: RUNNING  
   
Waiting on bqjob_r6f01b160bf2360e_016208212481_1 ... (3s) Current status: 
RUNNING 
Waiting on bqjob_r6f01b160bf2360e_016208212481_1 ... (3s) 
Current status: DONE   
BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r6f01b160bf2360e_016208212481_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-03-09 00:20:17,879 18468a6d MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-09 00:20:40,013 18468a6d MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results

Build failed in Jenkins: beam_PerformanceTests_JDBC #307

2018-03-08 Thread Apache Jenkins Server

See 


Changes:

[markliu] [BEAM-3750] Remove TestPipeline.covnertToArgs from integration tests

[markliu] Fix integration tests that use NumberedSharedFiles

[apilloud] [Nexmark] Use queue for Query 6 model

[apilloud] [Nexmark] Ensure enough data to produce output

--
[...truncated 50.31 KB...]

2018-03-09 00:10:14,871 5f7681d1 MainThread beam_integration_benchmark(1/1) 
INFO Retrying exception running IssueRetryableCommand: Command returned a 
non-zero exit code.

2018-03-09 00:10:29,966 5f7681d1 MainThread beam_integration_benchmark(1/1) 
INFO Running: kubectl --kubeconfig=/home/jenkins/.kube/config create -f 

2018-03-09 00:10:30,169 5f7681d1 MainThread beam_integration_benchmark(1/1) 
INFO Ran: {kubectl --kubeconfig=/home/jenkins/.kube/config create -f 

  ReturnCode:1,  WallTime:0:00.18s,  CPU:0.16s,  MaxMemory:45864kb 
STDOUT: 
STDERR: Error from server (AlreadyExists): error when creating 
":
 services "postgres-for-dev" already exists

2018-03-09 00:10:30,169 5f7681d1 MainThread beam_integration_benchmark(1/1) 
INFO Retrying exception running IssueRetryableCommand: Command returned a 
non-zero exit code.

2018-03-09 00:10:47,644 5f7681d1 MainThread beam_integration_benchmark(1/1) 
INFO Running: kubectl --kubeconfig=/home/jenkins/.kube/config create -f 

2018-03-09 00:10:47,827 5f7681d1 MainThread beam_integration_benchmark(1/1) 
INFO Ran: {kubectl --kubeconfig=/home/jenkins/.kube/config create -f 

  ReturnCode:1,  WallTime:0:00.17s,  CPU:0.14s,  MaxMemory:48728kb 
STDOUT: 
STDERR: Error from server (AlreadyExists): error when creating 
":
 services "postgres-for-dev" already exists

2018-03-09 00:10:47,827 5f7681d1 MainThread beam_integration_benchmark(1/1) 
INFO Retrying exception running IssueRetryableCommand: Command returned a 
non-zero exit code.

2018-03-09 00:11:05,640 5f7681d1 MainThread beam_integration_benchmark(1/1) 
INFO Running: kubectl --kubeconfig=/home/jenkins/.kube/config create -f 

2018-03-09 00:11:06,063 5f7681d1 MainThread beam_integration_benchmark(1/1) 
INFO Ran: {kubectl --kubeconfig=/home/jenkins/.kube/config create -f 

  ReturnCode:1,  WallTime:0:00.41s,  CPU:0.16s,  MaxMemory:46684kb 
STDOUT: 
STDERR: Error from server (AlreadyExists): error when creating 
":
 services "postgres-for-dev" already exists

2018-03-09 00:11:06,064 5f7681d1 MainThread beam_integration_benchmark(1/1) 
INFO Retrying exception running IssueRetryableCommand: Command returned a 
non-zero exit code.

2018-03-09 00:11:26,721 5f7681d1 MainThread beam_integration_benchmark(1/1) 
INFO Running: kubectl --kubeconfig=/home/jenkins/.kube/config create -f 

2018-03-09 00:11:26,917 5f7681d1 MainThread beam_integration_benchmark(1/1) 
INFO Ran: {kubectl --kubeconfig=/home/jenkins/.kube/config create -f 

  ReturnCode:1,  WallTime:0:00.18s,  CPU:0.16s,  MaxMemory:47696kb 
STDOUT: 
STDERR: Error from server (AlreadyExists): error when creating 
":
 services "postgres-for-dev" already exists

2018-03-09 00:11:26,918 5f7681d1 MainThread beam_integration_benchmark(1/1) 
INFO Retrying exception running IssueRetryableCommand: Command returned a 
non-zero exit code.

2018-03-09 00:11:48,518 5f7681d1 MainThread beam_integration_benchmark(1/1) 
INFO Running: kubectl

Build failed in Jenkins: beam_PerformanceTests_Python #1001

2018-03-08 Thread Apache Jenkins Server

See 


Changes:

[markliu] [BEAM-3750] Remove TestPipeline.covnertToArgs from integration tests

[markliu] Fix integration tests that use NumberedSharedFiles

[apilloud] [Nexmark] Use queue for Query 6 model

[apilloud] [Nexmark] Ensure enough data to produce output

--
[...truncated 1.74 KB...]
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins7690726898259677022.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5682255312206703484.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3479974723707802714.sh
+ .env/bin/pip install --upgrade setuptools pip
Downloading/unpacking setuptools from 
https://pypi.python.org/packages/ad/dc/fcced9ec3f2561c0cbe8eb6527eef7cf4f4919a2b3a07891a36e846635af/setuptools-38.5.2-py2.py3-none-any.whl#md5=abd3307cdce6fb543b5a4d0e3e98bdb6
Downloading/unpacking pip from 
https://pypi.python.org/packages/b6/ac/7015eb97dc749283ffdec1c3a88ddb8ae03b8fad0f0e611408f196358da3/pip-9.0.1-py2.py3-none-any.whl#md5=297dbd16ef53bcef0447d245815f5144
Installing collected packages: setuptools, pip
  Found existing installation: setuptools 2.2
Uninstalling setuptools:
  Successfully uninstalled setuptools
  Found existing installation: pip 1.5.4
Uninstalling pip:
  Successfully uninstalled pip
Successfully installed setuptools pip
Cleaning up...
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins101895607782627884.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins6006084503002747069.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Collecting numpy==1.13.3 (from -r PerfKitBenchmarker/requirements.txt (line 22))
:318:
 SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name 
Indication) extension to TLS is not available on this platform. This may cause 
the server to present an incorrect TLS certificate, which can cause validation 
failures. You can upgrade to a newer version of Python to solve this. For more 
information, see 
https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
:122:
 InsecurePlatformWarning: A true SSLContext object is not available. This 
prevents urllib3 from configuring SSL appropriately and may cause certain SSL 
connections to fail. You can upgrade to a newer version of Python to solve 
this. For more information, see 
https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
  Using cached numpy-1.13.3-cp27-cp27mu-manylinux1_x86_64.whl
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.

[jira] [Assigned] (BEAM-3816) [nexmark] Something is slightly off with Query 6

2018-03-08 Thread Kenneth Knowles (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-3816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-3816:
-

Assignee: (was: Kenneth Knowles)

> [nexmark] Something is slightly off with Query 6
> 
>
> Key: BEAM-3816
> URL: https://issues.apache.org/jira/browse/BEAM-3816
> Project: Beam
>  Issue Type: Bug
>  Components: examples-nexmark
>Reporter: Andrew Pilloud
>Priority: Major
>
> java.lang.AssertionError: Query6/Query6.Stamp/ParMultiDo(Anonymous).output: 
> wrong pipeline output Expected: <[\{"seller":1048,"price":83609648}, 
> \{"seller":1052,"price":61788353}, \{"seller":1086,"price":33744823}, 
> \{"seller":1078,"price":19876735}, \{"seller":1058,"price":50692833}, 
> \{"seller":1044,"price":6719489}, \{"seller":1096,"price":31287415}, 
> \{"seller":1095,"price":37004879}, \{"seller":1082,"price":22528654}, 
> \{"seller":1006,"price":57288736}, \{"seller":1051,"price":3967261}, 
> \{"seller":1084,"price":6394160}, \{"seller":1020,"price":3871757}, 
> \{"seller":1007,"price":185293}, \{"seller":1031,"price":11840889}, 
> \{"seller":1080,"price":26896442}, \{"seller":1030,"price":294928}, 
> \{"seller":1066,"price":26839191}, \{"seller":1000,"price":28257749}, 
> \{"seller":1055,"price":17087173}, \{"seller":1072,"price":45662210}, 
> \{"seller":1057,"price":4568399}, \{"seller":1025,"price":29008970}, 
> \{"seller":1064,"price":85810641}, \{"seller":1040,"price":99819658}, 
> \{"seller":1014,"price":11256690}, \{"seller":1098,"price":97259323}, 
> \{"seller":1011,"price":20447800}, \{"seller":1092,"price":77520938}, 
> \{"seller":1010,"price":53323687}, \{"seller":1060,"price":70032044}, 
> \{"seller":1062,"price":29076960}, \{"seller":1075,"price":19451464}, 
> \{"seller":1087,"price":27669185}, \{"seller":1009,"price":22951354}, 
> \{"seller":1065,"price":71875611}, \{"seller":1063,"price":87596779}, 
> \{"seller":1021,"price":62918185}, \{"seller":1034,"price":18472448}, 
> \{"seller":1028,"price":68556008}, \{"seller":1070,"price":92550447}]> but: 
> was <[\{"seller":1048,"price":83609648}, \{"seller":1052,"price":61788353}, 
> \{"seller":1086,"price":33744823}, \{"seller":1078,"price":19876735}, 
> \{"seller":1058,"price":50692833}, \{"seller":1044,"price":6719489}, 
> \{"seller":1096,"price":31287415}, \{"seller":1095,"price":37004879}, 
> \{"seller":1082,"price":22528654}, \{"seller":1006,"price":57288736}, 
> \{"seller":1051,"price":3967261}, \{"seller":1084,"price":6394160}, 
> \{"seller":1000,"price":34395558}, \{"seller":1020,"price":3871757}, 
> \{"seller":1007,"price":185293}, \{"seller":1031,"price":11840889}, 
> \{"seller":1080,"price":26896442}, \{"seller":1030,"price":294928}, 
> \{"seller":1066,"price":26839191}, \{"seller":1055,"price":17087173}, 
> \{"seller":1072,"price":45662210}, \{"seller":1057,"price":4568399}, 
> \{"seller":1025,"price":29008970}, \{"seller":1064,"price":85810641}, 
> \{"seller":1040,"price":99819658}, \{"seller":1014,"price":11256690}, 
> \{"seller":1098,"price":97259323}, \{"seller":1011,"price":20447800}, 
> \{"seller":1092,"price":77520938}, \{"seller":1010,"price":53323687}, 
> \{"seller":1060,"price":70032044}, \{"seller":1062,"price":29076960}, 
> \{"seller":1075,"price":19451464}, \{"seller":1087,"price":27669185}, 
> \{"seller":1009,"price":22951354}, \{"seller":1065,"price":71875611}, 
> \{"seller":1063,"price":87596779}, \{"seller":1021,"price":62918185}, 
> \{"seller":1034,"price":18472448}, \{"seller":1028,"price":68556008}, 
> \{"seller":1070,"price":92550447}]>



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_TextIOIT #246

2018-03-08 Thread Apache Jenkins Server

See 


Changes:

[markliu] [BEAM-3750] Remove TestPipeline.covnertToArgs from integration tests

[markliu] Fix integration tests that use NumberedSharedFiles

[apilloud] [Nexmark] Use queue for Query 6 model

[apilloud] [Nexmark] Ensure enough data to produce output

--
[...truncated 31.67 KB...]
[INFO] Excluding 
com.google.apis:google-api-services-storage:jar:v1-rev71-1.22.0 from the shaded 
jar.
[INFO] Excluding com.google.auth:google-auth-library-credentials:jar:0.7.1 from 
the shaded jar.
[INFO] Excluding com.google.auth:google-auth-library-oauth2-http:jar:0.7.1 from 
the shaded jar.
[INFO] Excluding com.google.cloud.bigdataoss:util:jar:1.4.5 from the shaded jar.
[INFO] Excluding com.google.api-client:google-api-client-java6:jar:1.22.0 from 
the shaded jar.
[INFO] Excluding com.google.api-client:google-api-client-jackson2:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.oauth-client:google-oauth-client-java6:jar:1.22.0 
from the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0-SNAPSHOT from 
the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-sdks-java-extensions-protobuf:jar:2.5.0-SNAPSHOT from the 
shaded jar.
[INFO] Excluding io.grpc:grpc-core:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.errorprone:error_prone_annotations:jar:2.0.15 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-context:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.instrumentation:instrumentation-api:jar:0.3.0 from 
the shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-bigquery:jar:v2-rev374-1.22.0 from the 
shaded jar.
[INFO] Excluding com.google.api:gax-grpc:jar:0.20.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.api:api-common:jar:1.0.0-rc2 from the shaded jar.
[INFO] Excluding com.google.auto.value:auto-value:jar:1.5.3 from the shaded jar.
[INFO] Excluding com.google.api:gax:jar:1.3.1 from the shaded jar.
[INFO] Excluding org.threeten:threetenbp:jar:1.3.3 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core-grpc:jar:1.2.0 from the 
shaded jar.
[INFO] Excluding com.google.protobuf:protobuf-java-util:jar:3.2.0 from the 
shaded jar.
[INFO] Excluding com.google.code.gson:gson:jar:2.7 from the shaded jar.
[INFO] Excluding com.google.apis:google-api-services-pubsub:jar:v1-rev10-1.22.0 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-iam-v1:jar:0.1.18 from the 
shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-proto-client:jar:1.4.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-protobuf:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-jackson:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-protos:jar:1.3.0 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-common-protos:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-auth:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-netty:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http2:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler-proxy:jar:4.1.8.Final from the shaded 
jar.
[INFO] Excluding io.netty:netty-codec-socks:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-buffer:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-common:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-transport:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-resolver:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.grpc:grpc-stub:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-all:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-okhttp:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.squareup.okhttp:okhttp:jar:2.5.0 from the shaded jar.
[INFO] Excluding com.squareup.okio:okio:jar:1.6.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf-lite:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf-nano:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.protobuf.nano:protobuf-javanano:jar:3.0.0-alpha-5 
from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core:jar:1.0.2 from the shaded 
jar.
[INFO] Excluding org.json:json:jar:20160810 from the shaded j

[jira] [Updated] (BEAM-3816) [nexmark] Something is slightly off with Query 6

2018-03-08 Thread Andrew Pilloud (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-3816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Pilloud updated BEAM-3816:
-
Labels: easyfix newbie test  (was: )

> [nexmark] Something is slightly off with Query 6
> 
>
> Key: BEAM-3816
> URL: https://issues.apache.org/jira/browse/BEAM-3816
> Project: Beam
>  Issue Type: Bug
>  Components: examples-nexmark
>Reporter: Andrew Pilloud
>Priority: Major
>  Labels: easyfix, newbie, test
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> java.lang.AssertionError: Query6/Query6.Stamp/ParMultiDo(Anonymous).output: 
> wrong pipeline output Expected: <[\{"seller":1048,"price":83609648}, 
> \{"seller":1052,"price":61788353}, \{"seller":1086,"price":33744823}, 
> \{"seller":1078,"price":19876735}, \{"seller":1058,"price":50692833}, 
> \{"seller":1044,"price":6719489}, \{"seller":1096,"price":31287415}, 
> \{"seller":1095,"price":37004879}, \{"seller":1082,"price":22528654}, 
> \{"seller":1006,"price":57288736}, \{"seller":1051,"price":3967261}, 
> \{"seller":1084,"price":6394160}, \{"seller":1020,"price":3871757}, 
> \{"seller":1007,"price":185293}, \{"seller":1031,"price":11840889}, 
> \{"seller":1080,"price":26896442}, \{"seller":1030,"price":294928}, 
> \{"seller":1066,"price":26839191}, \{"seller":1000,"price":28257749}, 
> \{"seller":1055,"price":17087173}, \{"seller":1072,"price":45662210}, 
> \{"seller":1057,"price":4568399}, \{"seller":1025,"price":29008970}, 
> \{"seller":1064,"price":85810641}, \{"seller":1040,"price":99819658}, 
> \{"seller":1014,"price":11256690}, \{"seller":1098,"price":97259323}, 
> \{"seller":1011,"price":20447800}, \{"seller":1092,"price":77520938}, 
> \{"seller":1010,"price":53323687}, \{"seller":1060,"price":70032044}, 
> \{"seller":1062,"price":29076960}, \{"seller":1075,"price":19451464}, 
> \{"seller":1087,"price":27669185}, \{"seller":1009,"price":22951354}, 
> \{"seller":1065,"price":71875611}, \{"seller":1063,"price":87596779}, 
> \{"seller":1021,"price":62918185}, \{"seller":1034,"price":18472448}, 
> \{"seller":1028,"price":68556008}, \{"seller":1070,"price":92550447}]> but: 
> was <[\{"seller":1048,"price":83609648}, \{"seller":1052,"price":61788353}, 
> \{"seller":1086,"price":33744823}, \{"seller":1078,"price":19876735}, 
> \{"seller":1058,"price":50692833}, \{"seller":1044,"price":6719489}, 
> \{"seller":1096,"price":31287415}, \{"seller":1095,"price":37004879}, 
> \{"seller":1082,"price":22528654}, \{"seller":1006,"price":57288736}, 
> \{"seller":1051,"price":3967261}, \{"seller":1084,"price":6394160}, 
> \{"seller":1000,"price":34395558}, \{"seller":1020,"price":3871757}, 
> \{"seller":1007,"price":185293}, \{"seller":1031,"price":11840889}, 
> \{"seller":1080,"price":26896442}, \{"seller":1030,"price":294928}, 
> \{"seller":1066,"price":26839191}, \{"seller":1055,"price":17087173}, 
> \{"seller":1072,"price":45662210}, \{"seller":1057,"price":4568399}, 
> \{"seller":1025,"price":29008970}, \{"seller":1064,"price":85810641}, 
> \{"seller":1040,"price":99819658}, \{"seller":1014,"price":11256690}, 
> \{"seller":1098,"price":97259323}, \{"seller":1011,"price":20447800}, 
> \{"seller":1092,"price":77520938}, \{"seller":1010,"price":53323687}, 
> \{"seller":1060,"price":70032044}, \{"seller":1062,"price":29076960}, 
> \{"seller":1075,"price":19451464}, \{"seller":1087,"price":27669185}, 
> \{"seller":1009,"price":22951354}, \{"seller":1065,"price":71875611}, 
> \{"seller":1063,"price":87596779}, \{"seller":1021,"price":62918185}, 
> \{"seller":1034,"price":18472448}, \{"seller":1028,"price":68556008}, 
> \{"seller":1070,"price":92550447}]>



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3816) [nexmark] Something is slightly off with Query 6

2018-03-08 Thread Andrew Pilloud (JIRA)

Andrew Pilloud created BEAM-3816:


 Summary: [nexmark] Something is slightly off with Query 6
 Key: BEAM-3816
 URL: https://issues.apache.org/jira/browse/BEAM-3816
 Project: Beam
  Issue Type: Bug
  Components: examples-nexmark
Reporter: Andrew Pilloud
Assignee: Kenneth Knowles


java.lang.AssertionError: Query6/Query6.Stamp/ParMultiDo(Anonymous).output: 
wrong pipeline output Expected: <[\{"seller":1048,"price":83609648}, 
\{"seller":1052,"price":61788353}, \{"seller":1086,"price":33744823}, 
\{"seller":1078,"price":19876735}, \{"seller":1058,"price":50692833}, 
\{"seller":1044,"price":6719489}, \{"seller":1096,"price":31287415}, 
\{"seller":1095,"price":37004879}, \{"seller":1082,"price":22528654}, 
\{"seller":1006,"price":57288736}, \{"seller":1051,"price":3967261}, 
\{"seller":1084,"price":6394160}, \{"seller":1020,"price":3871757}, 
\{"seller":1007,"price":185293}, \{"seller":1031,"price":11840889}, 
\{"seller":1080,"price":26896442}, \{"seller":1030,"price":294928}, 
\{"seller":1066,"price":26839191}, \{"seller":1000,"price":28257749}, 
\{"seller":1055,"price":17087173}, \{"seller":1072,"price":45662210}, 
\{"seller":1057,"price":4568399}, \{"seller":1025,"price":29008970}, 
\{"seller":1064,"price":85810641}, \{"seller":1040,"price":99819658}, 
\{"seller":1014,"price":11256690}, \{"seller":1098,"price":97259323}, 
\{"seller":1011,"price":20447800}, \{"seller":1092,"price":77520938}, 
\{"seller":1010,"price":53323687}, \{"seller":1060,"price":70032044}, 
\{"seller":1062,"price":29076960}, \{"seller":1075,"price":19451464}, 
\{"seller":1087,"price":27669185}, \{"seller":1009,"price":22951354}, 
\{"seller":1065,"price":71875611}, \{"seller":1063,"price":87596779}, 
\{"seller":1021,"price":62918185}, \{"seller":1034,"price":18472448}, 
\{"seller":1028,"price":68556008}, \{"seller":1070,"price":92550447}]> but: was 
<[\{"seller":1048,"price":83609648}, \{"seller":1052,"price":61788353}, 
\{"seller":1086,"price":33744823}, \{"seller":1078,"price":19876735}, 
\{"seller":1058,"price":50692833}, \{"seller":1044,"price":6719489}, 
\{"seller":1096,"price":31287415}, \{"seller":1095,"price":37004879}, 
\{"seller":1082,"price":22528654}, \{"seller":1006,"price":57288736}, 
\{"seller":1051,"price":3967261}, \{"seller":1084,"price":6394160}, 
\{"seller":1000,"price":34395558}, \{"seller":1020,"price":3871757}, 
\{"seller":1007,"price":185293}, \{"seller":1031,"price":11840889}, 
\{"seller":1080,"price":26896442}, \{"seller":1030,"price":294928}, 
\{"seller":1066,"price":26839191}, \{"seller":1055,"price":17087173}, 
\{"seller":1072,"price":45662210}, \{"seller":1057,"price":4568399}, 
\{"seller":1025,"price":29008970}, \{"seller":1064,"price":85810641}, 
\{"seller":1040,"price":99819658}, \{"seller":1014,"price":11256690}, 
\{"seller":1098,"price":97259323}, \{"seller":1011,"price":20447800}, 
\{"seller":1092,"price":77520938}, \{"seller":1010,"price":53323687}, 
\{"seller":1060,"price":70032044}, \{"seller":1062,"price":29076960}, 
\{"seller":1075,"price":19451464}, \{"seller":1087,"price":27669185}, 
\{"seller":1009,"price":22951354}, \{"seller":1065,"price":71875611}, 
\{"seller":1063,"price":87596779}, \{"seller":1021,"price":62918185}, 
\{"seller":1034,"price":18472448}, \{"seller":1028,"price":68556008}, 
\{"seller":1070,"price":92550447}]>



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3815) 2.4.0 RC2 uses java worker version beam-master-20180228, should be 2.4.0 or beam-2.4.0

2018-03-08 Thread Alan Myrvold (JIRA)

Alan Myrvold created BEAM-3815:
--

 Summary: 2.4.0 RC2 uses java worker version beam-master-20180228, 
should be 2.4.0 or beam-2.4.0
 Key: BEAM-3815
 URL: https://issues.apache.org/jira/browse/BEAM-3815
 Project: Beam
  Issue Type: Bug
  Components: runner-dataflow
Affects Versions: 2.4.0
Reporter: Alan Myrvold
Assignee: Robert Bradshaw


Running dataflow quickstart with the 2.4.0 RC2 uses a container named 
beam-master-20180228, should be 2.4.0 or beam-2.4.0, as in previous releases.

For 2.3.0, this version was set on the release branch with 
[https://github.com/apache/beam/commit/245cb4cbec9c1dd7d4f9ff8eb3ab38667c5472e9#diff-0032670e9006c6eba52569423c64e6d2]

Perhaps the release guide needs to be updated?

 wget 
https://repository.apache.org/content/repositories/orgapachebeam-1030/org/apache/beam/beam-runners-google-cloud-dataflow-java/2.4.0/beam-runners-google-cloud-dataflow-java-2.4.0.jar

jar xf beam-runners-google-cloud-dataflow-java-2.4.0.jar 
org/apache/beam/runners/dataflow/dataflow.properties

grep container.version org/apache/beam/runners/dataflow/dataflow.properties



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (BEAM-3339) Create post-release testing of the nightly snapshots

2018-03-08 Thread Luke Cwik (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-3339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16392088#comment-16392088
 ] 

Luke Cwik edited comment on BEAM-3339 at 3/8/18 11:32 PM:
--

I tried running the beam_PostRelease_NightlySnapshot by specifying the 
repo/version and noticed that the wrong version was fetched. The console 
contains "[INFO] Archetype repository not defined. Using the one from 
[org.apache.beam:beam-sdks-java-maven-archetypes-examples:2.3.0] found in 
catalog remote" which means that we didn't use the 2.4.0 snapshot.

 

My run:

https://builds.apache.org/view/A-D/view/Beam/job/beam_PostRelease_NightlySnapshot/118

 

The build parameters:

https://builds.apache.org/view/A-D/view/Beam/job/beam_PostRelease_NightlySnapshot/118/parameters/


was (Author: lcwik):
I tried running the beam_PostRelease_NightlySnapshot by specifying the 
repo/version and noticed that the wrong version was fetched. The console 
contains "[INFO] Archetype repository not defined. Using the one from 
[org.apache.beam:beam-sdks-java-maven-archetypes-examples:2.3.0] found in 
catalog remote" which means that we didn't use the 2.4.0 snapshot.

 

My run:

[https://builds.apache.org/view/A-D/view/Beam/job/beam_PostRelease_NightlySnapshot/118]

 

The build parameters:

https://builds.apache.org/view/A-D/view/Beam/job/beam_PostRelease_NightlySnapshot/118/parameters/

> Create post-release testing of the nightly snapshots
> 
>
> Key: BEAM-3339
> URL: https://issues.apache.org/jira/browse/BEAM-3339
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Alan Myrvold
>Assignee: Jason Kuster
>Priority: Major
>  Time Spent: 51h 20m
>  Remaining Estimate: 0h
>
> The nightly java snapshots in 
> https://repository.apache.org/content/groups/snapshots/org/apache/beam should 
> be verified by following the 
> https://beam.apache.org/get-started/quickstart-java/ instructions, to verify 
> that the release is usable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3339) Create post-release testing of the nightly snapshots

2018-03-08 Thread Luke Cwik (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-3339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16392088#comment-16392088
 ] 

Luke Cwik commented on BEAM-3339:
-

I tried running the beam_PostRelease_NightlySnapshot by specifying the 
repo/version and noticed that the wrong version was fetched. The console 
contains "[INFO] Archetype repository not defined. Using the one from 
[org.apache.beam:beam-sdks-java-maven-archetypes-examples:2.3.0] found in 
catalog remote" which means that we didn't use the 2.4.0 snapshot.

 

My run:

[https://builds.apache.org/view/A-D/view/Beam/job/beam_PostRelease_NightlySnapshot/118]

 

The build parameters:

https://builds.apache.org/view/A-D/view/Beam/job/beam_PostRelease_NightlySnapshot/118/parameters/

> Create post-release testing of the nightly snapshots
> 
>
> Key: BEAM-3339
> URL: https://issues.apache.org/jira/browse/BEAM-3339
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Alan Myrvold
>Assignee: Jason Kuster
>Priority: Major
>  Time Spent: 51h 20m
>  Remaining Estimate: 0h
>
> The nightly java snapshots in 
> https://repository.apache.org/content/groups/snapshots/org/apache/beam should 
> be verified by following the 
> https://beam.apache.org/get-started/quickstart-java/ instructions, to verify 
> that the release is usable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostRelease_NightlySnapshot #117

2018-03-08 Thread Apache Jenkins Server

See

[jira] [Assigned] (BEAM-3814) MapReduce runner page and capability matrix should clearly call out that it is on a feature branch

2018-03-08 Thread Kenneth Knowles (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-3814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-3814:
-

Assignee: (was: Melissa Pashniak)

> MapReduce runner page and capability matrix should clearly call out that it 
> is on a feature branch
> --
>
> Key: BEAM-3814
> URL: https://issues.apache.org/jira/browse/BEAM-3814
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Kenneth Knowles
>Priority: Major
>
> The page is here: https://beam.apache.org/documentation/runners/mapreduce/
> It does not call out that it is on a feature branch, and it even demonstrates 
> setting up a maven dependency for it, implying that it has releases, which it 
> does not.
> Similarly, the capability matrix should (until we have a larger revamp) make 
> very clear which runners are not actually available to users in the form of 
> released artifacts.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3814) MapReduce runner page and capability matrix should clearly call out that it is on a feature branch

2018-03-08 Thread Kenneth Knowles (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-3814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-3814:
-

Assignee: Melissa Pashniak

> MapReduce runner page and capability matrix should clearly call out that it 
> is on a feature branch
> --
>
> Key: BEAM-3814
> URL: https://issues.apache.org/jira/browse/BEAM-3814
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Kenneth Knowles
>Assignee: Melissa Pashniak
>Priority: Major
>
> The page is here: https://beam.apache.org/documentation/runners/mapreduce/
> It does not call out that it is on a feature branch, and it even demonstrates 
> setting up a maven dependency for it, implying that it has releases, which it 
> does not.
> Similarly, the capability matrix should (until we have a larger revamp) make 
> very clear which runners are not actually available to users in the form of 
> released artifacts.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3814) MapReduce runner page and capability matrix should clearly call out that it is on a feature branch

2018-03-08 Thread Kenneth Knowles (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-3814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-3814:
-

Assignee: (was: Reuven Lax)

> MapReduce runner page and capability matrix should clearly call out that it 
> is on a feature branch
> --
>
> Key: BEAM-3814
> URL: https://issues.apache.org/jira/browse/BEAM-3814
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Kenneth Knowles
>Priority: Major
>
> The page is here: https://beam.apache.org/documentation/runners/mapreduce/
> It does not call out that it is on a feature branch, and it even demonstrates 
> setting up a maven dependency for it, implying that it has releases, which it 
> does not.
> Similarly, the capability matrix should (until we have a larger revamp) make 
> very clear which runners are not actually available to users in the form of 
> released artifacts.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3814) MapReduce runner page and capability matrix should clearly call out that it is on a feature branch

2018-03-08 Thread Kenneth Knowles (JIRA)

Kenneth Knowles created BEAM-3814:
-

 Summary: MapReduce runner page and capability matrix should 
clearly call out that it is on a feature branch
 Key: BEAM-3814
 URL: https://issues.apache.org/jira/browse/BEAM-3814
 Project: Beam
  Issue Type: Bug
  Components: website
Reporter: Kenneth Knowles
Assignee: Reuven Lax


The page is here: https://beam.apache.org/documentation/runners/mapreduce/

It does not call out that it is on a feature branch, and it even demonstrates 
setting up a maven dependency for it, implying that it has releases, which it 
does not.

Similarly, the capability matrix should (until we have a larger revamp) make 
very clear which runners are not actually available to users in the form of 
released artifacts.




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3813) Support encryption for S3FileSystem (SSE-C and SSE-KMS)

2018-03-08 Thread Luke Cwik (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-3813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391989#comment-16391989
 ] 

Luke Cwik commented on BEAM-3813:
-

Do you plan to integrate/build a pluggable KMS solution for Apache Beam or are 
you specifically extending how the AWSCredentialsProvider can be configured via 
PipelineOptions inside of S3Options?

If its the former then I was curious as to what you were planning to do.

> Support encryption for S3FileSystem (SSE-C and SSE-KMS)
> ---
>
> Key: BEAM-3813
> URL: https://issues.apache.org/jira/browse/BEAM-3813
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-aws
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>Priority: Minor
>
> We should enable Amazon AWS S3 users to provide encryption keys or use a key 
> management system (KMS) to automatically configure encryption to read/write 
> objects stored via Beam.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3813) Support encryption for S3FileSystem (SSE-C and SSE-KMS)

2018-03-08 Thread JIRA

Ismaël Mejía created BEAM-3813:
--

 Summary: Support encryption for S3FileSystem (SSE-C and SSE-KMS)
 Key: BEAM-3813
 URL: https://issues.apache.org/jira/browse/BEAM-3813
 Project: Beam
  Issue Type: Improvement
  Components: io-java-aws
Reporter: Ismaël Mejía
Assignee: Ismaël Mejía


We should enable Amazon AWS S3 users to provide encryption keys or use a key 
management system (KMS) to automatically configure encryption to read/write 
objects stored via Beam.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3812) Avoid pickling PTransforms in proto representation

2018-03-08 Thread Robert Bradshaw (JIRA)

Robert Bradshaw created BEAM-3812:
-

 Summary: Avoid pickling PTransforms in proto representation
 Key: BEAM-3812
 URL: https://issues.apache.org/jira/browse/BEAM-3812
 Project: Beam
  Issue Type: Bug
  Components: sdk-py-core
Reporter: Robert Bradshaw
Assignee: Ahmet Altay


Any transform that requires passing information through the runner protos 
should have an explicit urn and payload. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Merge pull request #4745: [BEAM-3182] Ensure nexmark model tests produce output

2018-03-08 Thread kenn

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit b3468b694fcef8fbed9cba669bbb56a5cc475a30
Merge: 86b899a 0eed8bc
Author: Kenn Knowles 
AuthorDate: Thu Mar 8 12:14:31 2018 -0800

Merge pull request #4745: [BEAM-3182] Ensure nexmark model tests produce 
output

 .../sdk/nexmark/queries/NexmarkQueryModel.java |  1 +
 .../beam/sdk/nexmark/queries/Query6Model.java  | 35 +-
 .../apache/beam/sdk/nexmark/queries/QueryTest.java |  2 +-
 3 files changed, 23 insertions(+), 15 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[beam] branch master updated (86b899a -> b3468b6)

2018-03-08 Thread kenn

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 86b899a  [BEAM-3750] Remove TestPipeline.covnertToArgs from 
integration tests
 add 6b160e2  [Nexmark] Use queue for Query 6 model
 add 0eed8bc  [Nexmark] Ensure enough data to produce output
 new b3468b6  Merge pull request #4745: [BEAM-3182] Ensure nexmark model 
tests produce output

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../sdk/nexmark/queries/NexmarkQueryModel.java |  1 +
 .../beam/sdk/nexmark/queries/Query6Model.java  | 35 +-
 .../apache/beam/sdk/nexmark/queries/QueryTest.java |  2 +-
 3 files changed, 23 insertions(+), 15 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

Build failed in Jenkins: beam_PostRelease_NightlySnapshot #116

2018-03-08 Thread Apache Jenkins Server

See 


--
[...truncated 1.58 MB...]
"reason" : "notFound"
  } ],
  "message" : "Not found: Table 
apache-beam-testing:beam_postrelease_mobile_gaming.leaderboard_DataflowRunner_team",
  "status" : "NOT_FOUND"
}

com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)

com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)

com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)

com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:321)
com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1065)

com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)

com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)

com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)

org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.lambda$insertAll$0(BigQueryServicesImpl.java:714)
java.util.concurrent.FutureTask.run(FutureTask.java:266)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
Mar 08, 2018 8:03:39 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
SEVERE: 2018-03-08T20:03:38.339Z: (2a239357445d7c8e): 
java.lang.RuntimeException: 
com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not Found
{
  "code" : 404,
  "errors" : [ {
"domain" : "global",
"message" : "Not found: Table 
apache-beam-testing:beam_postrelease_mobile_gaming.leaderboard_DataflowRunner_team",
"reason" : "notFound"
  } ],
  "message" : "Not found: Table 
apache-beam-testing:beam_postrelease_mobile_gaming.leaderboard_DataflowRunner_team",
  "status" : "NOT_FOUND"
}

org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.insertAll(BigQueryServicesImpl.java:767)

org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.insertAll(BigQueryServicesImpl.java:802)

org.apache.beam.sdk.io.gcp.bigquery.StreamingWriteFn.flushRows(StreamingWriteFn.java:121)

org.apache.beam.sdk.io.gcp.bigquery.StreamingWriteFn.finishBundle(StreamingWriteFn.java:96)
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 
404 Not Found
{
  "code" : 404,
  "errors" : [ {
"domain" : "global",
"message" : "Not found: Table 
apache-beam-testing:beam_postrelease_mobile_gaming.leaderboard_DataflowRunner_team",
"reason" : "notFound"
  } ],
  "message" : "Not found: Table 
apache-beam-testing:beam_postrelease_mobile_gaming.leaderboard_DataflowRunner_team",
  "status" : "NOT_FOUND"
}

com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)

com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)

com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)

com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:321)
com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1065)

com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)

com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)

com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)

org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.lambda$insertAll$0(BigQueryServicesImpl.java:714)
java.util.concurrent.FutureTask.run(FutureTask.java:266)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
java.lang.RuntimeException: 
com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not Found
{
  "code" : 404,
  "errors" : [ {
"domain" : "global",
"message" : "Not found: Table 
apache-beam-testing:beam_postrelea

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Dataflow #5115

2018-03-08 Thread Apache Jenkins Server

See

[jira] [Created] (BEAM-3811) Add a CancellableQueue to make close methods safer

2018-03-08 Thread Thomas Groh (JIRA)

Thomas Groh created BEAM-3811:
-

 Summary: Add a CancellableQueue to make close methods safer
 Key: BEAM-3811
 URL: https://issues.apache.org/jira/browse/BEAM-3811
 Project: Beam
  Issue Type: Bug
  Components: runner-core
Reporter: Thomas Groh


There are multiple locations in which, on a call to {{close}} within some 
portability service, we wish to close all of our outstanding clients. However, 
the call to {{close}} can be interleaved with calls to the method which creates 
a new client. Without having an explicit thread-safe signal from the collection 
of pending clients, there's no way to ensure that all of the clients for a 
service are closed when that service is closed. 

 

As a result, some clients are only terminated by the forceful termination of 
the server, killing the connection.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_JDBC #306

2018-03-08 Thread Apache Jenkins Server

See 


Changes:

[herohde] [BEAM-3793] Validate provision response and add beamctl support

[cclauss] [BEAM-1251] Fix basestring, file(), reduce(), and xrange() for Python 
3

[altay] Fix topic URIs.

--
[...truncated 124.79 KB...]
Error from server (NotFound): error when stopping 
":
 replicationcontrollers "postgres" not found

2018-03-08 19:27:32,039 0f5c00f9 MainThread beam_integration_benchmark(1/1) 
INFO Retrying exception running IssueRetryableCommand: Command returned a 
non-zero exit code.

2018-03-08 19:27:53,319 0f5c00f9 MainThread beam_integration_benchmark(1/1) 
INFO Running: kubectl --kubeconfig=/home/jenkins/.kube/config delete -f 

2018-03-08 19:27:53,514 0f5c00f9 MainThread beam_integration_benchmark(1/1) 
INFO Ran: {kubectl --kubeconfig=/home/jenkins/.kube/config delete -f 

  ReturnCode:1,  WallTime:0:00.18s,  CPU:0.12s,  MaxMemory:38072kb 
STDOUT: 
STDERR: Error from server (NotFound): error when deleting 
":
 services "postgres" not found
Error from server (NotFound): error when stopping 
":
 replicationcontrollers "postgres" not found

2018-03-08 19:27:53,515 0f5c00f9 MainThread beam_integration_benchmark(1/1) 
INFO Retrying exception running IssueRetryableCommand: Command returned a 
non-zero exit code.

2018-03-08 19:28:14,180 0f5c00f9 MainThread beam_integration_benchmark(1/1) 
INFO Running: kubectl --kubeconfig=/home/jenkins/.kube/config delete -f 

2018-03-08 19:28:14,527 0f5c00f9 MainThread beam_integration_benchmark(1/1) 
INFO Ran: {kubectl --kubeconfig=/home/jenkins/.kube/config delete -f 

  ReturnCode:1,  WallTime:0:00.34s,  CPU:0.12s,  MaxMemory:37652kb 
STDOUT: 
STDERR: Error from server (NotFound): error when deleting 
":
 services "postgres" not found
Error from server (NotFound): error when stopping 
":
 replicationcontrollers "postgres" not found

2018-03-08 19:28:14,527 0f5c00f9 MainThread beam_integration_benchmark(1/1) 
INFO Retrying exception running IssueRetryableCommand: Command returned a 
non-zero exit code.

2018-03-08 19:28:40,441 0f5c00f9 MainThread beam_integration_benchmark(1/1) 
INFO Running: kubectl --kubeconfig=/home/jenkins/.kube/config delete -f 

2018-03-08 19:28:40,581 0f5c00f9 MainThread beam_integration_benchmark(1/1) 
INFO Ran: {kubectl --kubeconfig=/home/jenkins/.kube/config delete -f 

  ReturnCode:1,  WallTime:0:00.11s,  CPU:0.11s,  MaxMemory:37584kb 
STDOUT: 
STDERR: Error from server (NotFound): error when deleting 
":
 services "postgres" not found
Error from server (NotFound): error when stopping 
":
 replicationcontrollers "postgres" not found

2018-03-08 19:28:40,582 0f5c00f9 MainThread beam_integration_benchmark(1/1) 
INFO Retrying exception running IssueRetryableCommand: Command returned a 
non-zero exit code.

2018-03-08 19:28:56,527 0f5c00f9 MainThread beam_integration_benchmark(1/1) 
INFO Running: kubectl --kubeconfig=/home/jenkins/.kube/config delete -f 

2018-03-08 19:28:56,658 0f5c00f9 MainThread beam_integration_benchmark(1/1) 
INFO Ran: {kubectl --kubeconfig=/home/jenkins/.kube/config delete -f 

  ReturnCode:1,  WallTime:0:00.12s,  CPU:0.12s,  MaxMemory:37228kb 
STDOUT: 
STDERR: Error from server (NotFound): error

[jira] [Created] (BEAM-3810) Add new matcher to check output file content for WordCountIT and WindowedWordCountIT

2018-03-08 Thread Mark Liu (JIRA)

Mark Liu created BEAM-3810:
--

 Summary: Add new matcher to check output file content for 
WordCountIT and WindowedWordCountIT
 Key: BEAM-3810
 URL: https://issues.apache.org/jira/browse/BEAM-3810
 Project: Beam
  Issue Type: Test
  Components: testing
Reporter: Mark Liu
Assignee: Mark Liu


Test use case like 
[https://github.com/apache/beam/pull/4755#discussion_r173254182] shows that 
verify lines of output for integration test makes the validation and debugging 
easier than simple verify checksum. We'd like to add a new matcher (like 
FileContentMatcher) to very lines given expected files.

This matcher can be applied to ITs that already use FileChecksumMatcher (etc. 
WordCountIT and WindowedWordCountIT).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (BEAM-3809) beam_PerformanceTests_Python failing

2018-03-08 Thread Mark Liu (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-3809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391749#comment-16391749
 ] 

Mark Liu edited comment on BEAM-3809 at 3/8/18 7:11 PM:


The command pkb use to execute pipeline is:
{code}
python setup.py nosetests --tests=apache_beam.examples.wordcount_it_test 
--attr=IT --test-pipeline-options="--project=apache-beam-testing" 
"--staging_location=gs://temp-storage-for-end-to-end-tests/staging-it" 
"--temp_location=gs://temp-storage-for-end-to-end-tests/temp-it" 
"--output=gs://temp-storage-for-end-to-end-tests/py-it-cloud/output" 
--runner=TestDataflowRunner 
--sdk_location=/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/runs/9e009d45/beam/sdks/python/target/apache-beam-2.5.0.dev0.tar.gz
{code}
which looks good to me. Probably need more investigate on pkb side.

 


was (Author: markflyhigh):
The command pkb use to execute pipeline is:
python setup.py nosetests --tests=apache_beam.examples.wordcount_it_test 
--attr=IT --test-pipeline-options="--project=apache-beam-testing" 
"--staging_location=gs://temp-storage-for-end-to-end-tests/staging-it" 
"--temp_location=gs://temp-storage-for-end-to-end-tests/temp-it" 
"--output=gs://temp-storage-for-end-to-end-tests/py-it-cloud/output" 
--runner=TestDataflowRunner 
--sdk_location=/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/runs/9e009d45/beam/sdks/python/target/apache-beam-2.5.0.dev0.tar.gz
which looks good to me. Probably need more investigate on pkb side.

 

> beam_PerformanceTests_Python failing
> 
>
> Key: BEAM-3809
> URL: https://issues.apache.org/jira/browse/BEAM-3809
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Ahmet Altay
>Assignee: Mark Liu
>Priority: Major
>
> Looks like a misconfiguration based on the error.
>  
> Example test:
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PerformanceTests_Python/1000/console]
>  
> Error I see:
> error: option --test-pipeline-options not recognized
> 2018-03-08 18:48:08,202 9e009d45 MainThread beam_integration_benchmark(1/1) 
> ERRORError during benchmark beam_integration_benchmark
> Traceback (most recent call last):
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",
>  line 622, in RunBenchmark
> DoRunPhase(spec, collector, detailed_timer)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",
>  line 525, in DoRunPhase
> samples = spec.BenchmarkRun(spec)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/PerfKitBenchmarker/perfkitbenchmarker/linux_benchmarks/beam_integration_benchmark.py",
>  line 159, in Run
> job_type=job_type)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/PerfKitBenchmarker/perfkitbenchmarker/providers/gcp/gcp_dpb_dataflow.py",
>  line 90, in SubmitJob
> assert retcode == 0, "Integration Test Failed."



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3809) beam_PerformanceTests_Python failing

2018-03-08 Thread Mark Liu (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-3809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391749#comment-16391749
 ] 

Mark Liu commented on BEAM-3809:


The command pkb use to execute pipeline is:
python setup.py nosetests --tests=apache_beam.examples.wordcount_it_test 
--attr=IT --test-pipeline-options="--project=apache-beam-testing" 
"--staging_location=gs://temp-storage-for-end-to-end-tests/staging-it" 
"--temp_location=gs://temp-storage-for-end-to-end-tests/temp-it" 
"--output=gs://temp-storage-for-end-to-end-tests/py-it-cloud/output" 
--runner=TestDataflowRunner 
--sdk_location=/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/runs/9e009d45/beam/sdks/python/target/apache-beam-2.5.0.dev0.tar.gz
which looks good to me. Probably need more investigate on pkb side.

 

> beam_PerformanceTests_Python failing
> 
>
> Key: BEAM-3809
> URL: https://issues.apache.org/jira/browse/BEAM-3809
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Ahmet Altay
>Assignee: Mark Liu
>Priority: Major
>
> Looks like a misconfiguration based on the error.
>  
> Example test:
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PerformanceTests_Python/1000/console]
>  
> Error I see:
> error: option --test-pipeline-options not recognized
> 2018-03-08 18:48:08,202 9e009d45 MainThread beam_integration_benchmark(1/1) 
> ERRORError during benchmark beam_integration_benchmark
> Traceback (most recent call last):
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",
>  line 622, in RunBenchmark
> DoRunPhase(spec, collector, detailed_timer)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",
>  line 525, in DoRunPhase
> samples = spec.BenchmarkRun(spec)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/PerfKitBenchmarker/perfkitbenchmarker/linux_benchmarks/beam_integration_benchmark.py",
>  line 159, in Run
> job_type=job_type)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/PerfKitBenchmarker/perfkitbenchmarker/providers/gcp/gcp_dpb_dataflow.py",
>  line 90, in SubmitJob
> assert retcode == 0, "Integration Test Failed."



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_Spark #1444

2018-03-08 Thread Apache Jenkins Server

See 


Changes:

[herohde] [BEAM-3793] Validate provision response and add beamctl support

[cclauss] [BEAM-1251] Fix basestring, file(), reduce(), and xrange() for Python 
3

[altay] Fix topic URIs.

--
[...truncated 66.67 KB...]
2018-03-08 18:58:53,887 81a693f6 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-08 18:59:19,529 81a693f6 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-08 18:59:22,950 81a693f6 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  WallTime:0:03.41s,  CPU:0.38s,  MaxMemory:31632kb 
STDOUT: 

BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r2a2f100b2c1225a7_016206fb5f06_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: Upload complete.Waiting on bqjob_r2a2f100b2c1225a7_016206fb5f06_1 
... (0s) Current status: RUNNING
  Waiting on 
bqjob_r2a2f100b2c1225a7_016206fb5f06_1 ... (0s) Current status: DONE   
2018-03-08 18:59:22,950 81a693f6 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-08 18:59:42,128 81a693f6 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-08 18:59:45,581 81a693f6 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  WallTime:0:03.44s,  CPU:0.40s,  MaxMemory:31500kb 
STDOUT: 

BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r57a72e7f02f0a86a_016206fbb753_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: Upload complete.Waiting on bqjob_r57a72e7f02f0a86a_016206fbb753_1 
... (0s) Current status: RUNNING
  Waiting on 
bqjob_r57a72e7f02f0a86a_016206fbb753_1 ... (0s) Current status: DONE   
2018-03-08 18:59:45,581 81a693f6 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-08 19:00:12,852 81a693f6 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-08 19:00:16,321 81a693f6 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  WallTime:0:03.46s,  CPU:0.42s,  MaxMemory:31520kb 
STDOUT: 

BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r4cfcd43cd8c0d7e0_016206fc2f54_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: Upload complete.Waiting on bqjob_r4cfcd43cd8c0d7e0_016206fc2f54_1 
... (0s) Current status: RUNNING
  Waiting on 
bqjob_r4cfcd43cd8c0d7e0_016206fc2f54_1 ... (0s) Current status: DONE   
2018-03-08 19:00:16,322 81a693f6 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-08 19:00:42,377 81a693f6 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-08 19:00:46,258 81a693f6 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  WallTime:0:03.87s,  CPU:0.40s,  MaxMemory:31632kb 
STDOUT: 

BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r11b0dc30f1e1841d_016206fca2b2_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: Upload complete.Waiting on b

[jira] [Updated] (BEAM-3750) Integration test will fail when use TestPipeline.convertToArgs in some special cases

2018-03-08 Thread Luke Cwik (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-3750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik updated BEAM-3750:

Priority: Minor  (was: Major)

> Integration test will fail when use TestPipeline.convertToArgs in some 
> special cases
> 
>
> Key: BEAM-3750
> URL: https://issues.apache.org/jira/browse/BEAM-3750
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Minor
> Fix For: Not applicable
>
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> ITs(Integration Tests) are using {{TestPipeline.convertToArgs}} to convert 
> well-defined options to list of pipeline-recognizable arguments and pass them 
> to pipeline main function. A example use case is 
> {{WordCountIT}}([https://github.com/apache/beam/blob/master/examples/java/src/test/java/org/apache/beam/examples/WordCountIT.java#L70]).
> However, incorrect format will be generated by this function if the option 
> that we want to convert has attribute type {{List}}.
> Ideally, we want to get rid of this complicated conversion step in testing 
> which can avoid more problematic cases, but also make integration test better 
> understandable.
> Some discussions happened in [https://github.com/apache/beam/pull/4346.] One 
> possible approach is to extract part of pipeline code so that integration 
> test can reference directly. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-3750) Integration test will fail when use TestPipeline.convertToArgs in some special cases

2018-03-08 Thread Luke Cwik (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-3750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-3750.
-
   Resolution: Fixed
Fix Version/s: Not applicable

> Integration test will fail when use TestPipeline.convertToArgs in some 
> special cases
> 
>
> Key: BEAM-3750
> URL: https://issues.apache.org/jira/browse/BEAM-3750
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> ITs(Integration Tests) are using {{TestPipeline.convertToArgs}} to convert 
> well-defined options to list of pipeline-recognizable arguments and pass them 
> to pipeline main function. A example use case is 
> {{WordCountIT}}([https://github.com/apache/beam/blob/master/examples/java/src/test/java/org/apache/beam/examples/WordCountIT.java#L70]).
> However, incorrect format will be generated by this function if the option 
> that we want to convert has attribute type {{List}}.
> Ideally, we want to get rid of this complicated conversion step in testing 
> which can avoid more problematic cases, but also make integration test better 
> understandable.
> Some discussions happened in [https://github.com/apache/beam/pull/4346.] One 
> possible approach is to extract part of pipeline code so that integration 
> test can reference directly. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: [BEAM-3750] Remove TestPipeline.covnertToArgs from integration tests

2018-03-08 Thread lcwik

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 86b899afeb4cbc106a57d89cc55080798dfaaf9f
Merge: 9eca13c 6bc20eb
Author: Lukasz Cwik 
AuthorDate: Thu Mar 8 11:03:06 2018 -0800

[BEAM-3750] Remove TestPipeline.covnertToArgs from integration tests

 .../apache/beam/examples/DebuggingWordCount.java   | 11 ++--
 .../apache/beam/examples/WindowedWordCount.java|  9 --
 .../java/org/apache/beam/examples/WordCount.java   | 11 ++--
 .../beam/examples/cookbook/BigQueryTornadoes.java  | 10 +--
 .../beam/examples/DebuggingWordCountTest.java  |  2 +-
 .../apache/beam/examples/WindowedWordCountIT.java  | 17 ++--
 .../java/org/apache/beam/examples/WordCountIT.java |  4 +--
 .../examples/cookbook/BigQueryTornadoesIT.java |  2 +-
 .../beam/runners/apex/examples/WordCountTest.java  | 13 ++---
 .../org/apache/beam/sdk/testing/TestPipeline.java  | 32 --
 .../apache/beam/sdk/testing/TestPipelineTest.java  | 17 
 .../beam/sdk/io/cassandra/CassandraIOIT.java   |  2 --
 12 files changed, 52 insertions(+), 78 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

[jira] [Created] (BEAM-3809) beam_PerformanceTests_Python failing

2018-03-08 Thread Ahmet Altay (JIRA)

Ahmet Altay created BEAM-3809:
-

 Summary: beam_PerformanceTests_Python failing
 Key: BEAM-3809
 URL: https://issues.apache.org/jira/browse/BEAM-3809
 Project: Beam
  Issue Type: Bug
  Components: testing
Reporter: Ahmet Altay
Assignee: Mark Liu


Looks like a misconfiguration based on the error.

 

Example test:

[https://builds.apache.org/view/A-D/view/Beam/job/beam_PerformanceTests_Python/1000/console]

 

Error I see:
error: option --test-pipeline-options not recognized

2018-03-08 18:48:08,202 9e009d45 MainThread beam_integration_benchmark(1/1) 
ERRORError during benchmark beam_integration_benchmark
Traceback (most recent call last):
  File 
"/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",
 line 622, in RunBenchmark
DoRunPhase(spec, collector, detailed_timer)
  File 
"/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",
 line 525, in DoRunPhase
samples = spec.BenchmarkRun(spec)
  File 
"/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/PerfKitBenchmarker/perfkitbenchmarker/linux_benchmarks/beam_integration_benchmark.py",
 line 159, in Run
job_type=job_type)
  File 
"/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/PerfKitBenchmarker/perfkitbenchmarker/providers/gcp/gcp_dpb_dataflow.py",
 line 90, in SubmitJob
assert retcode == 0, "Integration Test Failed."



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (9eca13c -> 86b899a)

2018-03-08 Thread lcwik

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 9eca13c  This closes #4250
 add 09a4b69  [BEAM-3750] Remove TestPipeline.covnertToArgs from 
integration tests
 add 6bc20eb  Fix integration tests that use NumberedSharedFiles
 new 86b899a  [BEAM-3750] Remove TestPipeline.covnertToArgs from 
integration tests

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../apache/beam/examples/DebuggingWordCount.java   | 11 ++--
 .../apache/beam/examples/WindowedWordCount.java|  9 --
 .../java/org/apache/beam/examples/WordCount.java   | 11 ++--
 .../beam/examples/cookbook/BigQueryTornadoes.java  | 10 +--
 .../beam/examples/DebuggingWordCountTest.java  |  2 +-
 .../apache/beam/examples/WindowedWordCountIT.java  | 17 ++--
 .../java/org/apache/beam/examples/WordCountIT.java |  4 +--
 .../examples/cookbook/BigQueryTornadoesIT.java |  2 +-
 .../beam/runners/apex/examples/WordCountTest.java  | 13 ++---
 .../org/apache/beam/sdk/testing/TestPipeline.java  | 32 --
 .../apache/beam/sdk/testing/TestPipelineTest.java  | 17 
 .../beam/sdk/io/cassandra/CassandraIOIT.java   |  2 --
 12 files changed, 52 insertions(+), 78 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

Build failed in Jenkins: beam_PerformanceTests_Python #1000

2018-03-08 Thread Apache Jenkins Server

See 


Changes:

[herohde] [BEAM-3793] Validate provision response and add beamctl support

[cclauss] [BEAM-1251] Fix basestring, file(), reduce(), and xrange() for Python 
3

[altay] Fix topic URIs.

--
[...truncated 1.68 KB...]
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins558057978556174012.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins828986483513223582.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2559577499239004872.sh
+ .env/bin/pip install --upgrade setuptools pip
Downloading/unpacking setuptools from 
https://pypi.python.org/packages/ad/dc/fcced9ec3f2561c0cbe8eb6527eef7cf4f4919a2b3a07891a36e846635af/setuptools-38.5.2-py2.py3-none-any.whl#md5=abd3307cdce6fb543b5a4d0e3e98bdb6
Downloading/unpacking pip from 
https://pypi.python.org/packages/b6/ac/7015eb97dc749283ffdec1c3a88ddb8ae03b8fad0f0e611408f196358da3/pip-9.0.1-py2.py3-none-any.whl#md5=297dbd16ef53bcef0447d245815f5144
Installing collected packages: setuptools, pip
  Found existing installation: setuptools 2.2
Uninstalling setuptools:
  Successfully uninstalled setuptools
  Found existing installation: pip 1.5.4
Uninstalling pip:
  Successfully uninstalled pip
Successfully installed setuptools pip
Cleaning up...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4778643268549889773.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3566056854404155290.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Collecting numpy==1.13.3 (from -r PerfKitBenchmarker/requirements.txt (line 22))
:318:
 SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name 
Indication) extension to TLS is not available on this platform. This may cause 
the server to present an incorrect TLS certificate, which can cause validation 
failures. You can upgrade to a newer version of Python to solve this. For more 
information, see 
https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
:122:
 InsecurePlatformWarning: A true SSLContext object is not available. This 
prevents urllib3 from configuring SSL appropriately and may cause certain SSL 
connections to fail. You can upgrade to a newer version of Python to solve 
this. For more information, see 
https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
  Using cached numpy-1.13.3-cp27-cp27mu-manylinux1_x86_64.whl
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe>=0.23 in 
/u

Build failed in Jenkins: beam_PostCommit_Python_Verify #4390

2018-03-08 Thread Apache Jenkins Server

See 


Changes:

[herohde] [BEAM-3793] Validate provision response and add beamctl support

[cclauss] [BEAM-1251] Fix basestring, file(), reduce(), and xrange() for Python 
3

[altay] Fix topic URIs.

--
[...truncated 1.02 MB...]
copying apache_beam/runners/worker/data_plane_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/log_handler.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/log_handler_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/logger.pxd -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/logger.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/logger_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/opcounters.pxd -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/opcounters.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/opcounters_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/operation_specs.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/operations.pxd -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/operations.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sdk_worker.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sdk_worker_main.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sdk_worker_main_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sdk_worker_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sideinputs.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sideinputs_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/statesampler.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/statesampler_fast.pyx -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/statesampler_slow.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/statesampler_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/testing/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/testing
copying apache_beam/testing/pipeline_verifiers.py -> 
apache-beam-2.5.0.dev0/apache_beam/testing
copying apache_beam/testing/pipeline_verifiers_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/testing
copying apache_beam/testing/test_pipeline.py -> 
apache-beam-2.5.0.dev0/apache_beam/testing
copying apache_beam/testing/test_pipeline_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/testing
copying apache_beam/testing/test_stream.py -> 
apache-beam-2.5.0.dev0/apache_beam/testing
copying apache_beam/testing/test_stream_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/testing
copying apache_beam/testing/test_utils.py -> 
apache-beam-2.5.0.dev0/apache_beam/testing
copying apache_beam/testing/test_utils_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/testing
copying apache_beam/testing/util.py -> 
apache-beam-2.5.0.dev0/apache_beam/testing
copying apache_beam/testing/util_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/testing
copying apache_beam/testing/data/standard_coders.yaml -> 
apache-beam-2.5.0.dev0/apache_beam/testing/data
copying apache_beam/testing/data/trigger_transcripts.yaml -> 
apache-beam-2.5.0.dev0/apache_beam/testing/data
copying apache_beam/transforms/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/transforms
copying apache_beam/transforms/combiners.py -> 
apache-beam-2.5.0.dev0/apache_beam/transforms
copying apache_beam/transforms/combiners_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/transforms
copying apache_beam/transforms/core.py -> 
apache-beam-2.5.0.dev0/apache_beam/transforms
copying apache_beam/transforms/create_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/transforms
copying apache_beam/transforms/cy_combiners.pxd -> 
apache-beam-2.5.0.dev0/apache_beam/transforms
copying apache_beam/transforms/cy_combiners.py -> 
apache-beam-2.5.0.dev0/apache_beam/transforms
copying apache_beam/transforms/cy_combiners_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/transforms
copying apache_beam/transforms/display.py -> 
apache-beam-2.5.0.dev0/apache_beam/transforms
copying apache_beam/transforms/display_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/transforms
copying apache_beam/transforms/ptransform.py -> 
apache-beam-2.5.0.dev0/apache_beam/transforms
copying apache_bea

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1073

2018-03-08 Thread Apache Jenkins Server

See 


Changes:

[herohde] [BEAM-3793] Validate provision response and add beamctl support

[cclauss] [BEAM-1251] Fix basestring, file(), reduce(), and xrange() for Python 
3

[altay] Fix topic URIs.

--
[...truncated 126.43 KB...]
  File 
"
 line 180, in apply_PTransform
return transform.expand(input)
  File 
"
 line 147, in expand
| "Match" >> Map(matcher))
  File 
"
 line 820, in __ror__
return self.transform.__ror__(pvalueish, self.label)
  File 
"
 line 488, in __ror__
result = p.apply(self, pvalueish, label)
  File 
"
 line 443, in apply
return self.apply(transform, pvalueish)
  File 
"
 line 479, in apply
pvalueish_result = self.runner.apply(transform, pvalueish)
  File 
"
 line 174, in apply
return m(transform, input)
  File 
"
 line 180, in apply_PTransform
return transform.expand(input)
  File 
"
 line 165, in expand
| Map(_merge_tagged_vals_under_key, result_ctor, result_ctor_arg))
  File 
"
 line 109, in __or__
return self.pipeline.apply(ptransform, self)
  File 
"
 line 433, in apply
label or transform.label)
  File 
"
 line 443, in apply
return self.apply(transform, pvalueish)
  File 
"
 line 493, in apply
self._infer_result_type(transform, inputs, result)
  File 
"
 line 513, in _infer_result_type
type_options = self._options.view_as(TypeOptions)
  File 
"
 line 227, in view_as
view = cls(self._flags)
  File 
"
 line 150, in __init__
parser = _BeamArgumentParser()
  File "/usr/lib/python2.7/argparse.py", line 1586, in __init__
self._positionals = add_group(_('positional arguments'))
  File "/usr/lib/python2.7/argparse.py", line 1300, in add_argument_group
group = _ArgumentGroup(self, *args, **kwargs)
  File "/usr/lib/python2.7/argparse.py", line 1482, in __init__
super_init(description=description, **kwargs)
  File "/usr/lib/python2.7/argparse.py", line 1214, in __init__
self._negative_number_matcher = _re.compile(r'^-\d+$|^-\d*\.\d+$')
  File 
"
 line 190, in compile
return _compile(pattern, flags)
  File 
"
 line 231, in _compile
cachekey = (type(key[0]),) + key
  File 
"
 line 276, in signalhandler
raise TimedOutException()
TimedOutException: 'test_flattened_side_input 
(apache_beam.transforms.sideinputs_test.SideInputsTest)'

==
ERROR: test_ite

[beam] branch master updated: Fix topic URIs.

2018-03-08 Thread altay

This is an automated email from the ASF dual-hosted git repository.

altay pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new 3898ba4  Fix topic URIs.
 new 9eca13c  This closes #4250
3898ba4 is described below

commit 3898ba45a2ffe059ee80271040baedac4ba3aaa0
Author: postmasters 
AuthorDate: Tue Dec 12 13:56:50 2017 -0800

Fix topic URIs.
---
 sdks/python/apache_beam/examples/streaming_wordcap.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/sdks/python/apache_beam/examples/streaming_wordcap.py 
b/sdks/python/apache_beam/examples/streaming_wordcap.py
index 19f9e5f..01e0df2 100644
--- a/sdks/python/apache_beam/examples/streaming_wordcap.py
+++ b/sdks/python/apache_beam/examples/streaming_wordcap.py
@@ -35,10 +35,10 @@ def run(argv=None):
   parser = argparse.ArgumentParser()
   parser.add_argument(
   '--input_topic', dest='input_topic', required=True,
-  help='Input PubSub topic of the form "/topics//".')
+  help='Input PubSub topic with form "projects//topics/".')
   parser.add_argument(
   '--output_topic', dest='output_topic', required=True,
-  help='Output PubSub topic of the form "/topics//".')
+  help='Output PubSub topic with form 
"projects//topics/".')
   known_args, pipeline_args = parser.parse_known_args(argv)
 
   with beam.Pipeline(argv=pipeline_args) as p:

-- 
To stop receiving notification emails like this one, please contact
al...@apache.org.

[beam] branch master updated (cc57d93 -> 09f6dec)

2018-03-08 Thread altay

This is an automated email from the ASF dual-hosted git repository.

altay pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from cc57d93  Merge pull request #4804 from 
cclauss/basestring-to-six.string_types
 add 952f571  [BEAM-3793] Validate provision response and add beamctl 
support
 new 09f6dec  Merge pull request #4827 from herohde/provisioning

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../query.go => cmd/beamctl/cmd/provision.go}  | 47 ++
 sdks/go/cmd/beamctl/cmd/root.go|  2 +-
 .../beam/provision/{provison.go => provision.go}   |  6 +++
 3 files changed, 37 insertions(+), 18 deletions(-)
 copy sdks/go/{pkg/beam/core/util/protox/query.go => 
cmd/beamctl/cmd/provision.go} (57%)
 rename sdks/go/pkg/beam/provision/{provison.go => provision.go} (95%)

-- 
To stop receiving notification emails like this one, please contact
al...@apache.org.

[beam] 01/01: Merge pull request #4827 from herohde/provisioning

2018-03-08 Thread altay

This is an automated email from the ASF dual-hosted git repository.

altay pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 09f6decb395fab1298a7e40972a5a61f2f1c3779
Merge: cc57d93 952f571
Author: Ahmet Altay 
AuthorDate: Thu Mar 8 09:31:28 2018 -0800

Merge pull request #4827 from herohde/provisioning

[BEAM-3793] Validate provisioning info response

 sdks/go/cmd/beamctl/cmd/provision.go   | 58 ++
 sdks/go/cmd/beamctl/cmd/root.go|  2 +-
 .../beam/provision/{provison.go => provision.go}   |  6 +++
 3 files changed, 65 insertions(+), 1 deletion(-)

-- 
To stop receiving notification emails like this one, please contact
al...@apache.org.

[beam-site] branch asf-site updated: Update site to mention the 2 Python lint envs

2018-03-08 Thread altay

This is an automated email from the ASF dual-hosted git repository.

altay pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git


The following commit(s) were added to refs/heads/asf-site by this push:
 new 770fe8f  Update site to mention the 2 Python lint envs
 new afb4164  This closes #396
770fe8f is described below

commit 770fe8fd07419feb49e207d7ef16de64a3dcccac
Author: Holden Karau 
AuthorDate: Thu Mar 1 18:37:10 2018 -0800

Update site to mention the 2 Python lint envs
---
 content/contribute/contribution-guide/index.html | 2 +-
 src/contribute/contribution-guide.md | 3 ++-
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/content/contribute/contribution-guide/index.html 
b/content/contribute/contribution-guide/index.html
index ec6addf..fa8ebe8 100644
--- a/content/contribute/contribution-guide/index.html
+++ b/content/contribute/contribution-guide/index.html
@@ -435,7 +435,7 @@ environment before testing your code.
 command.
 
 $ pip install tox
-$ tox -e lint
+$ tox -e lint_py2,lint_py3
 
 
 
diff --git a/src/contribute/contribution-guide.md 
b/src/contribute/contribution-guide.md
index 5a7f0b9..fcf7502 100644
--- a/src/contribute/contribution-guide.md
+++ b/src/contribute/contribution-guide.md
@@ -248,7 +248,8 @@ To Check for lint errors locally, install "tox" package and 
run following
 command.
 
 $ pip install tox
-$ tox -e lint
+$ tox -e lint_py2,lint_py3
+
 
 Beam supports running Python SDK tests using Maven. For this, navigate to root
 directory of your Apache Beam clone and execute following command. Currently

-- 
To stop receiving notification emails like this one, please contact
al...@apache.org.

[beam] 01/01: Merge pull request #4804 from cclauss/basestring-to-six.string_types

2018-03-08 Thread altay

This is an automated email from the ASF dual-hosted git repository.

altay pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit cc57d93336c2a9533d759bfdfbe5f7a1928e2e4d
Merge: 99de7ca e35dc23
Author: Ahmet Altay 
AuthorDate: Thu Mar 8 09:27:14 2018 -0800

Merge pull request #4804 from cclauss/basestring-to-six.string_types

[BEAM-1251] Fix basestring, file(), reduce(), and xrange() for Python 3

 sdks/python/apache_beam/examples/snippets/snippets_test.py |  2 +-
 sdks/python/apache_beam/io/filebasedsink.py| 10 ++
 sdks/python/apache_beam/io/filebasedsource.py  |  5 +++--
 sdks/python/apache_beam/io/filesystem.py   |  3 ++-
 sdks/python/apache_beam/io/gcp/bigquery.py |  6 --
 sdks/python/apache_beam/io/vcfio.py|  4 +++-
 sdks/python/apache_beam/pvalue.py  |  4 +++-
 sdks/python/apache_beam/runners/common.py  |  4 ++--
 sdks/python/apache_beam/testing/util.py|  9 +
 sdks/python/apache_beam/transforms/core.py |  4 +++-
 sdks/python/apache_beam/transforms/util.py |  2 +-
 11 files changed, 33 insertions(+), 20 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
al...@apache.org.

[beam] branch master updated (99de7ca -> cc57d93)

2018-03-08 Thread altay

This is an automated email from the ASF dual-hosted git repository.

altay pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 99de7ca  Merge pull request #4718 from apilloud/nexmark-sql-models
 add e35dc23  [BEAM-1251] Fix basestring, file(), reduce(), and xrange() 
for Python 3
 new cc57d93  Merge pull request #4804 from 
cclauss/basestring-to-six.string_types

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/python/apache_beam/examples/snippets/snippets_test.py |  2 +-
 sdks/python/apache_beam/io/filebasedsink.py| 10 ++
 sdks/python/apache_beam/io/filebasedsource.py  |  5 +++--
 sdks/python/apache_beam/io/filesystem.py   |  3 ++-
 sdks/python/apache_beam/io/gcp/bigquery.py |  6 --
 sdks/python/apache_beam/io/vcfio.py|  4 +++-
 sdks/python/apache_beam/pvalue.py  |  4 +++-
 sdks/python/apache_beam/runners/common.py  |  4 ++--
 sdks/python/apache_beam/testing/util.py|  9 +
 sdks/python/apache_beam/transforms/core.py |  4 +++-
 sdks/python/apache_beam/transforms/util.py |  2 +-
 11 files changed, 33 insertions(+), 20 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
al...@apache.org.

[jira] [Commented] (BEAM-3808) Dataflow Jobs meet Cloud Label Spec

2018-03-08 Thread Ahmet Altay (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-3808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391523#comment-16391523
 ] 

Ahmet Altay commented on BEAM-3808:
---

As a note: the solution needs to be applied to dataflow runners in all 
languages. Python currently truncates default job names to 63 characters by 
truncating the user name where necessary and preserving date as much as 
possible. We can converge on a strategy to keep most useful pieces in places.

> Dataflow Jobs meet Cloud Label Spec
> ---
>
> Key: BEAM-3808
> URL: https://issues.apache.org/jira/browse/BEAM-3808
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Reporter: Andrea Foegler
>Assignee: Thomas Groh
>Priority: Major
>
> Dataflow Job Names should meet Cloud Label Specifications:
> [https://cloud.google.com/resource-manager/docs/creating-managing-labels]
> This will allow Dataflow users to organize resources according to the 
> Dataflow Job using those resources.  The Dataflow Service will be able to 
> automatically associate accessed and consumed resources and associated 
> metrics to support streamlined monitoring and analysis of these resources, in 
> particular GCE instance usage metrics.
> Potential Issues:  Some current users appear to encode a lot of data in their 
> job names.  Need to identify if applying this restriction breaks any 
> necessary use cases and verify an alternative is available before proceeding.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3808) Dataflow Jobs meet Cloud Label Spec

2018-03-08 Thread Andrea Foegler (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-3808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391421#comment-16391421
 ] 

Andrea Foegler commented on BEAM-3808:
--

Note, the current format for auto-generating names in Beam often fails to meet 
the 63-char limit requirement.  Might be worth considering removing the user 
name and date from the auto-generated name to bring this closer to compliance 
and provide a simpler, easier to parse name.

> Dataflow Jobs meet Cloud Label Spec
> ---
>
> Key: BEAM-3808
> URL: https://issues.apache.org/jira/browse/BEAM-3808
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Reporter: Andrea Foegler
>Assignee: Thomas Groh
>Priority: Major
>
> Dataflow Job Names should meet Cloud Label Specifications:
> [https://cloud.google.com/resource-manager/docs/creating-managing-labels]
> This will allow Dataflow users to organize resources according to the 
> Dataflow Job using those resources.  The Dataflow Service will be able to 
> automatically associate accessed and consumed resources and associated 
> metrics to support streamlined monitoring and analysis of these resources, in 
> particular GCE instance usage metrics.
> Potential Issues:  Some current users appear to encode a lot of data in their 
> job names.  Need to identify if applying this restriction breaks any 
> necessary use cases and verify an alternative is available before proceeding.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-2831) Pipeline crashes due to Beam encoder breaking Flink memory management

2018-03-08 Thread Guillaume Balaine (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-2831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391419#comment-16391419
 ] 

Guillaume Balaine commented on BEAM-2831:
-

I fixed that in every Coder, including the KryoAtomicCoder (I am using Scio) 
and it now works properly on Flink, thanks.

> Pipeline crashes due to Beam encoder breaking Flink memory management
> -
>
> Key: BEAM-2831
> URL: https://issues.apache.org/jira/browse/BEAM-2831
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 2.0.0, 2.1.0
> Environment: Flink 1.2.1 and 1.3.0, Java HotSpot and OpenJDK 8, macOS 
> 10.12.6 and unknown Linux
>Reporter: Reinier Kip
>Assignee: Aljoscha Krettek
>Priority: Major
>
> I’ve been running a Beam pipeline on Flink. Depending on the dataset size and 
> the heap memory configuration of the jobmanager and taskmanager, I may run 
> into an EOFException, which causes the job to fail.
> As [discussed on Flink's 
> mailinglist|http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/EOFException-related-to-memory-segments-during-run-of-Beam-pipeline-on-Flink-td15255.html]
>  (stacktrace enclosed), Flink catches these EOFExceptions and activates disk 
> spillover. Because Beam wraps these exceptions, this mechanism fails, the 
> exception travels up the stack, and the job aborts.
> Hopefully this is enough information and this is something that can be 
> adjusted for in Beam. I'd be glad to provide more information where needed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3808) Dataflow Jobs meet Cloud Label Spec

2018-03-08 Thread Andrea Foegler (JIRA)

Andrea Foegler created BEAM-3808:


 Summary: Dataflow Jobs meet Cloud Label Spec
 Key: BEAM-3808
 URL: https://issues.apache.org/jira/browse/BEAM-3808
 Project: Beam
  Issue Type: Improvement
  Components: runner-dataflow
Reporter: Andrea Foegler
Assignee: Thomas Groh


Dataflow Job Names should meet Cloud Label Specifications:

[https://cloud.google.com/resource-manager/docs/creating-managing-labels]

This will allow Dataflow users to organize resources according to the Dataflow 
Job using those resources.  The Dataflow Service will be able to automatically 
associate accessed and consumed resources and associated metrics to support 
streamlined monitoring and analysis of these resources, in particular GCE 
instance usage metrics.

Potential Issues:  Some current users appear to encode a lot of data in their 
job names.  Need to identify if applying this restriction breaks any necessary 
use cases and verify an alternative is available before proceeding.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_Verify #4389

2018-03-08 Thread Apache Jenkins Server

See 


--
[...truncated 1.02 MB...]
copying apache_beam/runners/dataflow/native_io/iobase_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py 
-> apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/evaluation_context.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/executor.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/helper_transforms.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/transform_evaluator.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/util.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/watermark_manager.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/experimental/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/experimental
copying apache_beam/runners/experimental/python_rpc_direct/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying 
apache_beam/runners/experimental/python_rpc_direct/python_rpc_direct_runner.py 
-> apache-beam-2.5.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/experimental/python_rpc_direct/server.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/job/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/manager.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/utils.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/job
copying apache_beam/runners/portability/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/fn_api_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/fn_api_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/maptask_executor_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/maptask_executor_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner_main.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/test/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/test
copying apache_beam/runners/worker/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/bundle_processor.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/data_plane.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/data_plane_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/log_handler.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/log_handler_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/logger.pxd -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1072

2018-03-08 Thread Apache Jenkins Server

See 


--
[...truncated 118.68 KB...]
  File 
"
 line 178, in test_iterable_side_input
pipeline.run()
  File 
"
 line 102, in run
result = super(TestPipeline, self).run()
  File 
"
 line 369, in run
self.to_runner_api(), self.runner, self._options).run(False)
  File 
"
 line 597, in from_runner_api
context.transforms.get_by_id(root_transform_id)]
  File 
"
 line 69, in get_by_id
self._id_to_proto[id], self._pipeline_context)
  File 
"
 line 842, in from_runner_api
part = context.transforms.get_by_id(transform_id)
  File 
"
 line 69, in get_by_id
self._id_to_proto[id], self._pipeline_context)
  File 
"
 line 842, in from_runner_api
part = context.transforms.get_by_id(transform_id)
  File 
"
 line 69, in get_by_id
self._id_to_proto[id], self._pipeline_context)
  File 
"
 line 833, in from_runner_api
transform=ptransform.PTransform.from_runner_api(proto.spec, context),
  File 
"
 line 555, in from_runner_api
context)
  File 
"
 line 886, in from_runner_api_parameter
result = ParDo(fn, *args, **kwargs)
  File 
"
 line 781, in __init__
super(ParDo, self).__init__(fn, *args, **kwargs)
  File 
"
 line 627, in __init__
self.fn = pickler.loads(pickler.dumps(self.fn))
  File 
"
 line 193, in dumps
s = dill.dumps(o)
  File 
"
 line 259, in dumps
dump(obj, file, protocol, byref, fmode, recurse)#, strictio)
  File 
"
 line 252, in dump
pik.dump(obj)
  File "/usr/lib/python2.7/pickle.py", line 224, in dump
self.save(obj)
  File "/usr/lib/python2.7/pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python2.7/pickle.py", line 419, in save_reduce
save(state)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File 
"
 line 165, in new_save_module_dict
return old_save_module_dict(pickler, obj)
  File 
"
 line 841, in save_module_dict
StockPickler.save_dict(pickler, obj)
  File "/usr/lib/python2.7/pickle.py", line 649, in save_dict
self._batch_setitems(obj.iteritems())
  File "/usr/lib/python2.7/pickle.py", line 681, in _batch_setitems
save(v)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File 
"

[jira] [Commented] (BEAM-3796) Implement TypedWrite extending WriteFilesResult for XmlIO

2018-03-08 Thread JIRA


[ 
https://issues.apache.org/jira/browse/BEAM-3796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391289#comment-16391289
 ] 

Łukasz Gajowy commented on BEAM-3796:
-

Ok, given what you said and all the "@Deprecated" annotations I should have 
noticed earlier, now I totally agree this shouldn't be done this way. Thanks.

> Implement TypedWrite extending WriteFilesResult for XmlIO
> -
>
> Key: BEAM-3796
> URL: https://issues.apache.org/jira/browse/BEAM-3796
> Project: Beam
>  Issue Type: Improvement
>  Components: io-ideas
>Reporter: Łukasz Gajowy
>Assignee: Eugene Kirpichov
>Priority: Major
>
> I mean similar implementation that is there in TextIO or AvroIO. Users could 
> leverage things like getPerDestinationFilenames() directly from the IO 
> without the need to use FileIO explicitly. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #5114

2018-03-08 Thread Apache Jenkins Server

See

Build failed in Jenkins: beam_PerformanceTests_Spark #1443

2018-03-08 Thread Apache Jenkins Server

See 


Changes:

[apilloud] [Nexmark][SQL] Convert SQL Rows to Java models

[apilloud] [Nexmark][SQL] Output java pojos

[apilloud] [Nexmark][SQL] Add model tests for sql

--
[...truncated 95.28 KB...]
'apache-beam-testing:bqjob_r74be67a740e1b237_0162058cdbcf_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-03-08 12:19:02,020 253161b5 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-08 12:19:28,423 253161b5 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-08 12:19:30,573 253161b5 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  WallTime:0:02.14s,  CPU:0.24s,  MaxMemory:25392kb 
STDOUT: Upload complete.
Waiting on bqjob_r9a2e0cd7d555d65_0162058d4b29_1 ... (0s) Current status: 
RUNNING 
Waiting on bqjob_r9a2e0cd7d555d65_0162058d4b29_1 ... (0s) 
Current status: DONE   
BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r9a2e0cd7d555d65_0162058d4b29_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-03-08 12:19:30,573 253161b5 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-08 12:19:52,138 253161b5 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-08 12:19:54,176 253161b5 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  WallTime:0:02.03s,  CPU:0.25s,  MaxMemory:25476kb 
STDOUT: Upload complete.
Waiting on bqjob_r2fdbc0494aab78d7_0162058da7d4_1 ... (0s) Current status: 
RUNNING 
 Waiting on bqjob_r2fdbc0494aab78d7_0162058da7d4_1 ... (0s) 
Current status: DONE   
BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r2fdbc0494aab78d7_0162058da7d4_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-03-08 12:19:54,177 253161b5 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-08 12:20:16,301 253161b5 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-08 12:20:18,320 253161b5 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  WallTime:0:02.01s,  CPU:0.24s,  MaxMemory:25408kb 
STDOUT: Upload complete.
Waiting on bqjob_r66aba995f79f1037_0162058e062b_1 ... (0s) Current status: 
RUNNING 
 Waiting on

Build failed in Jenkins: beam_PerformanceTests_Python #999

2018-03-08 Thread Apache Jenkins Server

See 


Changes:

[apilloud] [Nexmark][SQL] Convert SQL Rows to Java models

[apilloud] [Nexmark][SQL] Output java pojos

[apilloud] [Nexmark][SQL] Add model tests for sql

--
[...truncated 1.11 KB...]
 > git rev-list --no-walk e8db6a81e9b03b5a0bbffc9117f3128ea23ff184 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins668791650396620.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2734067950861874035.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8497311161632315190.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins9105229944995690793.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in ./.env/lib/python2.7/site-packages
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1013009739023722551.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1719582288683258191.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Collecting numpy==1.13.3 (from -r PerfKitBenchmarker/requirements.txt (line 22))
:318:
 SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name 
Indication) extension to TLS is not available on this platform. This may cause 
the server to present an incorrect TLS certificate, which can cause validation 
failures. You can upgrade to a newer version of Python to solve this. For more 
information, see 
https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
:122:
 InsecurePlatformWarning: A true SSLContext object is not available. This 
prevents urllib3 from configuring SSL appropriately and may cause certain SSL 
connections to fail. You can upgrade to a newer version of Python to solve 
this. For more information, see 
https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
  Using cached numpy-1.13.3-cp27-cp27mu-manylinux1_x86_64.whl
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Require

Jenkins build is back to normal : beam_PerformanceTests_TextIOIT #244

2018-03-08 Thread Apache Jenkins Server

See

Jenkins build is back to normal : beam_PostRelease_NightlySnapshot #115

2018-03-08 Thread Apache Jenkins Server

See

Build failed in Jenkins: beam_PostCommit_Python_Verify #4388

2018-03-08 Thread Apache Jenkins Server

See 


--
[...truncated 1.02 MB...]
copying apache_beam/runners/dataflow/native_io/iobase_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py 
-> apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/evaluation_context.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/executor.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/helper_transforms.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/transform_evaluator.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/util.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/watermark_manager.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/experimental/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/experimental
copying apache_beam/runners/experimental/python_rpc_direct/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying 
apache_beam/runners/experimental/python_rpc_direct/python_rpc_direct_runner.py 
-> apache-beam-2.5.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/experimental/python_rpc_direct/server.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/job/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/manager.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/utils.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/job
copying apache_beam/runners/portability/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/fn_api_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/fn_api_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/maptask_executor_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/maptask_executor_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner_main.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/test/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/test
copying apache_beam/runners/worker/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/bundle_processor.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/data_plane.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/data_plane_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/log_handler.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/log_handler_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/logger.pxd -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1071

2018-03-08 Thread Apache Jenkins Server

See 


Changes:

[apilloud] [Nexmark][SQL] Convert SQL Rows to Java models

[apilloud] [Nexmark][SQL] Output java pojos

[apilloud] [Nexmark][SQL] Add model tests for sql

--
[...truncated 137.74 KB...]
==
ERROR: test_iterable_side_input 
(apache_beam.transforms.sideinputs_test.SideInputsTest)
--
Traceback (most recent call last):
  File 
"
 line 812, in run
test(orig)
  File 
"
 line 45, in __call__
return self.run(*arg, **kwarg)
  File 
"
 line 133, in run
self.runTest(result)
  File 
"
 line 151, in runTest
test(result)
  File "/usr/lib/python2.7/unittest/case.py", line 395, in __call__
return self.run(*args, **kwds)
  File "/usr/lib/python2.7/unittest/case.py", line 331, in run
testMethod()
  File 
"
 line 178, in test_iterable_side_input
pipeline.run()
  File 
"
 line 102, in run
result = super(TestPipeline, self).run()
  File 
"
 line 369, in run
self.to_runner_api(), self.runner, self._options).run(False)
  File 
"
 line 382, in run
return self.runner.run_pipeline(self)
  File 
"
 line 285, in run_pipeline
return_context=True)
  File 
"
 line 580, in to_runner_api
root_transform_id = context.transforms.get_id(self._root_transform())
  File 
"
 line 60, in get_id
self._id_to_proto[id] = obj.to_runner_api(self._pipeline_context)
  File 
"
 line 810, in to_runner_api
for part in self.parts],
  File 
"
 line 60, in get_id
self._id_to_proto[id] = obj.to_runner_api(self._pipeline_context)
  File 
"
 line 810, in to_runner_api
for part in self.parts],
  File 
"
 line 60, in get_id
self._id_to_proto[id] = obj.to_runner_api(self._pipeline_context)
  File 
"
 line 808, in to_runner_api
spec=transform_to_runner_api(self.transform, context),
  File 
"
 line 805, in transform_to_runner_api
return transform.to_runner_api(context)
  File 
"
 line 542, in to_runner_api
urn, typed_param = self.to_runner_api_parameter(context)
  File 
"
 line 863, in to_runner_api_parameter
source=self.source.to_runner_api(context),
  File 
"

Build failed in Jenkins: beam_PostCommit_Python_Verify #4387

2018-03-08 Thread Apache Jenkins Server

See 


Changes:

[apilloud] [Nexmark][SQL] Convert SQL Rows to Java models

[apilloud] [Nexmark][SQL] Output java pojos

[apilloud] [Nexmark][SQL] Add model tests for sql

--
[...truncated 1.02 MB...]
copying apache_beam/runners/dataflow/native_io/iobase_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py 
-> apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/evaluation_context.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/executor.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/helper_transforms.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/transform_evaluator.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/util.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/watermark_manager.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/experimental/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/experimental
copying apache_beam/runners/experimental/python_rpc_direct/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying 
apache_beam/runners/experimental/python_rpc_direct/python_rpc_direct_runner.py 
-> apache-beam-2.5.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/experimental/python_rpc_direct/server.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/job/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/manager.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/utils.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/job
copying apache_beam/runners/portability/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/fn_api_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/fn_api_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/maptask_executor_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/maptask_executor_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner_main.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/test/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/test
copying apache_beam/runners/worker/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/bundle_processor.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/data_plane.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/data_plane_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/log_handler.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/log_handler_test.py -> 
apache-

[beam] branch master updated (e8db6a8 -> 99de7ca)

2018-03-08 Thread echauchot

This is an automated email from the ASF dual-hosted git repository.

echauchot pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from e8db6a8  Merge pull request #4817: Fallback to byte equality in 
MutaitonDetection
 add 96865ec  [Nexmark][SQL] Convert SQL Rows to Java models
 add 1a489ac  [Nexmark][SQL] Output java pojos
 add c693db4  [Nexmark][SQL] Add model tests for sql
 new 99de7ca  Merge pull request #4718 from apilloud/nexmark-sql-models

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../beam/sdk/nexmark/model/AuctionCount.java   |  36 +--
 .../beam/sdk/nexmark/model/AuctionPrice.java   |  24 -
 .../org/apache/beam/sdk/nexmark/model/Bid.java |  23 +
 .../beam/sdk/nexmark/model/NameCityStateId.java|  30 +-
 .../model/sql/adapter/ModelAdaptersMapping.java| 114 +
 .../model/sql/adapter/ModelFieldsAdapter.java  |  17 ++-
 .../sdk/nexmark/queries/sql/NexmarkSqlQuery.java   |  14 +--
 .../beam/sdk/nexmark/queries/sql/SqlQuery0.java|  18 +++-
 .../beam/sdk/nexmark/queries/sql/SqlQuery1.java|  17 ++-
 .../beam/sdk/nexmark/queries/sql/SqlQuery2.java|  18 +++-
 .../beam/sdk/nexmark/queries/sql/SqlQuery3.java|  22 ++--
 .../beam/sdk/nexmark/queries/sql/SqlQuery5.java|  23 +++--
 .../beam/sdk/nexmark/queries/sql/SqlQuery7.java|  26 +++--
 .../apache/beam/sdk/nexmark/queries/QueryTest.java |  86 
 .../sdk/nexmark/queries/sql/SqlQuery0Test.java |  15 +--
 .../sdk/nexmark/queries/sql/SqlQuery1Test.java |  15 +--
 .../sdk/nexmark/queries/sql/SqlQuery2Test.java |  30 +++---
 .../sdk/nexmark/queries/sql/SqlQuery3Test.java |  40 ++--
 .../sdk/nexmark/queries/sql/SqlQuery5Test.java |  34 ++
 .../sdk/nexmark/queries/sql/SqlQuery7Test.java |  40 +---
 20 files changed, 445 insertions(+), 197 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
echauc...@apache.org.

[beam] 01/01: Merge pull request #4718 from apilloud/nexmark-sql-models

2018-03-08 Thread echauchot

This is an automated email from the ASF dual-hosted git repository.

echauchot pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 99de7cae296444d80f10b5e087b3ec39e00135a2
Merge: e8db6a8 c693db4
Author: Etienne Chauchot 
AuthorDate: Thu Mar 8 09:38:48 2018 +0100

Merge pull request #4718 from apilloud/nexmark-sql-models

Merge pull request #4718: [BEAM-3182] Hookup Nexmark query models to SQL

 .../beam/sdk/nexmark/model/AuctionCount.java   |  36 +--
 .../beam/sdk/nexmark/model/AuctionPrice.java   |  24 -
 .../org/apache/beam/sdk/nexmark/model/Bid.java |  23 +
 .../beam/sdk/nexmark/model/NameCityStateId.java|  30 +-
 .../model/sql/adapter/ModelAdaptersMapping.java| 114 +
 .../model/sql/adapter/ModelFieldsAdapter.java  |  17 ++-
 .../sdk/nexmark/queries/sql/NexmarkSqlQuery.java   |  14 +--
 .../beam/sdk/nexmark/queries/sql/SqlQuery0.java|  18 +++-
 .../beam/sdk/nexmark/queries/sql/SqlQuery1.java|  17 ++-
 .../beam/sdk/nexmark/queries/sql/SqlQuery2.java|  18 +++-
 .../beam/sdk/nexmark/queries/sql/SqlQuery3.java|  22 ++--
 .../beam/sdk/nexmark/queries/sql/SqlQuery5.java|  23 +++--
 .../beam/sdk/nexmark/queries/sql/SqlQuery7.java|  26 +++--
 .../apache/beam/sdk/nexmark/queries/QueryTest.java |  86 
 .../sdk/nexmark/queries/sql/SqlQuery0Test.java |  15 +--
 .../sdk/nexmark/queries/sql/SqlQuery1Test.java |  15 +--
 .../sdk/nexmark/queries/sql/SqlQuery2Test.java |  30 +++---
 .../sdk/nexmark/queries/sql/SqlQuery3Test.java |  40 ++--
 .../sdk/nexmark/queries/sql/SqlQuery5Test.java |  34 ++
 .../sdk/nexmark/queries/sql/SqlQuery7Test.java |  40 +---
 20 files changed, 445 insertions(+), 197 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
echauc...@apache.org.

91 matches

Mail list logo