[jira] [Work logged] (BEAM-4774) Intergrate Nexmark SQL with Perfkit
[
https://issues.apache.org/jira/browse/BEAM-4774?focusedWorklogId=124021=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-124021
]
ASF GitHub Bot logged work on BEAM-4774:
Author: ASF GitHub Bot
Created on: 17/Jul/18 09:01
Start Date: 17/Jul/18 09:01
Worklog Time Spent: 10m
Work Description: echauchot closed pull request #5947: [BEAM-4774]
Nexmark SQL Postcommits
URL: https://github.com/apache/beam/pull/5947
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/.test-infra/jenkins/job_PostCommit_Java_Nexmark_Direct.groovy
b/.test-infra/jenkins/job_PostCommit_Java_Nexmark_Direct.groovy
index 70e5bc6b47c..ccf101fc1b2 100644
--- a/.test-infra/jenkins/job_PostCommit_Java_Nexmark_Direct.groovy
+++ b/.test-infra/jenkins/job_PostCommit_Java_Nexmark_Direct.groovy
@@ -65,5 +65,39 @@
NoPhraseTriggeringPostCommitBuilder.postCommitJob('beam_PostCommit_Java_Nexmark_
'--enforceEncodability=true',
'--enforceImmutability=true"'].join(' '))
}
+shell('echo *** RUN NEXMARK IN SQL BATCH MODE USING DIRECT RUNNER ***')
+gradle {
+ rootBuildScriptDir(commonJobProperties.checkoutDir)
+ tasks(':beam-sdks-java-nexmark:run')
+ commonJobProperties.setGradleSwitches(delegate)
+ switches('-Pnexmark.runner=":beam-runners-direct-java"' +
+ ' -Pnexmark.args="' +
+ [NexmarkBigqueryProperties.nexmarkBigQueryArgs,
+ '--runner=DirectRunner',
+ '--queryLanguage=sql',
+ '--streaming=false',
+ '--suite=SMOKE',
+ '--manageResources=false',
+ '--monitorJobs=true',
+ '--enforceEncodability=true',
+ '--enforceImmutability=true"'].join(' '))
+}
+shell('echo *** RUN NEXMARK IN SQL STREAMING MODE USING DIRECT RUNNER ***')
+gradle {
+ rootBuildScriptDir(commonJobProperties.checkoutDir)
+ tasks(':beam-sdks-java-nexmark:run')
+ commonJobProperties.setGradleSwitches(delegate)
+ switches('-Pnexmark.runner=":beam-runners-direct-java"' +
+ ' -Pnexmark.args="' +
+ [NexmarkBigqueryProperties.nexmarkBigQueryArgs,
+ '--runner=DirectRunner',
+ '--queryLanguage=sql',
+ '--streaming=true',
+ '--suite=SMOKE',
+ '--manageResources=false',
+ '--monitorJobs=true',
+ '--enforceEncodability=true',
+ '--enforceImmutability=true"'].join(' '))
+}
}
}
diff --git a/.test-infra/jenkins/job_PostCommit_Java_Nexmark_Flink.groovy
b/.test-infra/jenkins/job_PostCommit_Java_Nexmark_Flink.groovy
index 40c3f0e8f68..736c7d8b11f 100644
--- a/.test-infra/jenkins/job_PostCommit_Java_Nexmark_Flink.groovy
+++ b/.test-infra/jenkins/job_PostCommit_Java_Nexmark_Flink.groovy
@@ -65,5 +65,39 @@
NoPhraseTriggeringPostCommitBuilder.postCommitJob('beam_PostCommit_Java_Nexmark_
'--monitorJobs=true',
'--flinkMaster=local"'].join(' '))
}
+shell('echo *** RUN NEXMARK IN SQL BATCH MODE USING FLINK RUNNER ***')
+gradle {
+ rootBuildScriptDir(commonJobProperties.checkoutDir)
+ tasks(':beam-sdks-java-nexmark:run')
+ commonJobProperties.setGradleSwitches(delegate)
+ switches('-Pnexmark.runner=":beam-runners-flink_2.11"' +
+ ' -Pnexmark.args="' +
+ [NexmarkBigqueryProperties.nexmarkBigQueryArgs,
+ '--runner=FlinkRunner',
+ '--queryLanguage=sql',
+ '--streaming=false',
+ '--suite=SMOKE',
+ '--streamTimeout=60' ,
+ '--manageResources=false',
+ '--monitorJobs=true',
+ '--flinkMaster=local"'].join(' '))
+}
+shell('echo *** RUN NEXMARK IN SQL STREAMING MODE USING FLINK RUNNER ***')
+gradle {
+ rootBuildScriptDir(commonJobProperties.checkoutDir)
+ tasks(':beam-sdks-java-nexmark:run')
+ commonJobProperties.setGradleSwitches(delegate)
+ switches('-Pnexmark.runner=":beam-runners-flink_2.11"' +
+ ' -Pnexmark.args="' +
+ [NexmarkBigqueryProperties.nexmarkBigQueryArgs,
+ '--runner=FlinkRunner',
+ '--queryLanguage=sql',
+ '--streaming=true',
+ '--suite=SMOKE',
+ '--streamTimeout=60' ,
+ '--manageResources=false',
+ '--monitorJobs=true',
+ '--flinkMaster=local"'].join(' '))
+}
}
}
diff --git a/.test-infra/jenkins/job_PostCommit_Java_Nexmark_Spark.groovy
b/.test-infra/jenkins/job_PostCommit_Java_Nexmark_Spark.groovy
index b27e00a3fc4..e4c5363994f
[jira] [Work logged] (BEAM-4774) Intergrate Nexmark SQL with Perfkit
[ https://issues.apache.org/jira/browse/BEAM-4774?focusedWorklogId=123666=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123666 ] ASF GitHub Bot logged work on BEAM-4774: Author: ASF GitHub Bot Created on: 16/Jul/18 16:02 Start Date: 16/Jul/18 16:02 Worklog Time Spent: 10m Work Description: apilloud commented on issue #5947: [BEAM-4774] Nexmark SQL Postcommits URL: https://github.com/apache/beam/pull/5947#issuecomment-405297763 I've squashed and renamed the commit. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 123666) Time Spent: 1h 40m (was: 1.5h) > Intergrate Nexmark SQL with Perfkit > --- > > Key: BEAM-4774 > URL: https://issues.apache.org/jira/browse/BEAM-4774 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Andrew Pilloud >Assignee: Andrew Pilloud >Priority: Major > Time Spent: 1h 40m > Remaining Estimate: 0h > > We should publish to a dashboard like pure JAVA does. See BEAM-4283. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4774) Intergrate Nexmark SQL with Perfkit
[ https://issues.apache.org/jira/browse/BEAM-4774?focusedWorklogId=123548=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123548 ] ASF GitHub Bot logged work on BEAM-4774: Author: ASF GitHub Bot Created on: 16/Jul/18 09:21 Start Date: 16/Jul/18 09:21 Worklog Time Spent: 10m Work Description: echauchot commented on issue #5947: [BEAM-4774] Nexmark SQL Postcommits URL: https://github.com/apache/beam/pull/5947#issuecomment-405190904 @apilloud can you squash the 2 commits into one and rename it with the ticket number? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 123548) Time Spent: 1.5h (was: 1h 20m) > Intergrate Nexmark SQL with Perfkit > --- > > Key: BEAM-4774 > URL: https://issues.apache.org/jira/browse/BEAM-4774 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Andrew Pilloud >Assignee: Andrew Pilloud >Priority: Major > Time Spent: 1.5h > Remaining Estimate: 0h > > We should publish to a dashboard like pure JAVA does. See BEAM-4283. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4774) Intergrate Nexmark SQL with Perfkit
[ https://issues.apache.org/jira/browse/BEAM-4774?focusedWorklogId=123522=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123522 ] ASF GitHub Bot logged work on BEAM-4774: Author: ASF GitHub Bot Created on: 16/Jul/18 08:05 Start Date: 16/Jul/18 08:05 Worklog Time Spent: 10m Work Description: echauchot commented on issue #5947: [BEAM-4774] Nexmark SQL Postcommits URL: https://github.com/apache/beam/pull/5947#issuecomment-405174150 @apilloud yes there was some side-effect flakiness introduced in ElasticsearchIOTest lately. I need to take a look This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 123522) Time Spent: 1h 20m (was: 1h 10m) > Intergrate Nexmark SQL with Perfkit > --- > > Key: BEAM-4774 > URL: https://issues.apache.org/jira/browse/BEAM-4774 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Andrew Pilloud >Assignee: Andrew Pilloud >Priority: Major > Time Spent: 1h 20m > Remaining Estimate: 0h > > We should publish to a dashboard like pure JAVA does. See BEAM-4283. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4774) Intergrate Nexmark SQL with Perfkit
[ https://issues.apache.org/jira/browse/BEAM-4774?focusedWorklogId=123066=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123066 ] ASF GitHub Bot logged work on BEAM-4774: Author: ASF GitHub Bot Created on: 13/Jul/18 22:18 Start Date: 13/Jul/18 22:18 Worklog Time Spent: 10m Work Description: apilloud commented on issue #5947: [BEAM-4774] Nexmark SQL Postcommits URL: https://github.com/apache/beam/pull/5947#issuecomment-404967884 run java precommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 123066) Time Spent: 1h 10m (was: 1h) > Intergrate Nexmark SQL with Perfkit > --- > > Key: BEAM-4774 > URL: https://issues.apache.org/jira/browse/BEAM-4774 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Andrew Pilloud >Assignee: Andrew Pilloud >Priority: Major > Time Spent: 1h 10m > Remaining Estimate: 0h > > We should publish to a dashboard like pure JAVA does. See BEAM-4283. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4774) Intergrate Nexmark SQL with Perfkit
[ https://issues.apache.org/jira/browse/BEAM-4774?focusedWorklogId=123065=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123065 ] ASF GitHub Bot logged work on BEAM-4774: Author: ASF GitHub Bot Created on: 13/Jul/18 22:18 Start Date: 13/Jul/18 22:18 Worklog Time Spent: 10m Work Description: apilloud commented on issue #5947: [BEAM-4774] Nexmark SQL Postcommits URL: https://github.com/apache/beam/pull/5947#issuecomment-404967862 `ElasticsearchIOTest` again. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 123065) Time Spent: 1h (was: 50m) > Intergrate Nexmark SQL with Perfkit > --- > > Key: BEAM-4774 > URL: https://issues.apache.org/jira/browse/BEAM-4774 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Andrew Pilloud >Assignee: Andrew Pilloud >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > We should publish to a dashboard like pure JAVA does. See BEAM-4283. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4774) Intergrate Nexmark SQL with Perfkit
[ https://issues.apache.org/jira/browse/BEAM-4774?focusedWorklogId=123037=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123037 ] ASF GitHub Bot logged work on BEAM-4774: Author: ASF GitHub Bot Created on: 13/Jul/18 21:04 Start Date: 13/Jul/18 21:04 Worklog Time Spent: 10m Work Description: apilloud commented on issue #5947: [BEAM-4774] Nexmark SQL Postcommits URL: https://github.com/apache/beam/pull/5947#issuecomment-404953812 run java precommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 123037) Time Spent: 50m (was: 40m) > Intergrate Nexmark SQL with Perfkit > --- > > Key: BEAM-4774 > URL: https://issues.apache.org/jira/browse/BEAM-4774 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Andrew Pilloud >Assignee: Andrew Pilloud >Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > > We should publish to a dashboard like pure JAVA does. See BEAM-4283. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4774) Intergrate Nexmark SQL with Perfkit
[ https://issues.apache.org/jira/browse/BEAM-4774?focusedWorklogId=123036=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123036 ] ASF GitHub Bot logged work on BEAM-4774: Author: ASF GitHub Bot Created on: 13/Jul/18 21:04 Start Date: 13/Jul/18 21:04 Worklog Time Spent: 10m Work Description: apilloud commented on issue #5947: [BEAM-4774] Nexmark SQL Postcommits URL: https://github.com/apache/beam/pull/5947#issuecomment-404953788 Hit flaps in `ElasticsearchIOTest`. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 123036) Time Spent: 40m (was: 0.5h) > Intergrate Nexmark SQL with Perfkit > --- > > Key: BEAM-4774 > URL: https://issues.apache.org/jira/browse/BEAM-4774 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Andrew Pilloud >Assignee: Andrew Pilloud >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > We should publish to a dashboard like pure JAVA does. See BEAM-4283. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4774) Intergrate Nexmark SQL with Perfkit
[ https://issues.apache.org/jira/browse/BEAM-4774?focusedWorklogId=123015=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123015 ] ASF GitHub Bot logged work on BEAM-4774: Author: ASF GitHub Bot Created on: 13/Jul/18 19:52 Start Date: 13/Jul/18 19:52 Worklog Time Spent: 10m Work Description: akedin commented on issue #5947: [BEAM-4774] Nexmark SQL Postcommits URL: https://github.com/apache/beam/pull/5947#issuecomment-404936494 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 123015) Time Spent: 0.5h (was: 20m) > Intergrate Nexmark SQL with Perfkit > --- > > Key: BEAM-4774 > URL: https://issues.apache.org/jira/browse/BEAM-4774 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Andrew Pilloud >Assignee: Andrew Pilloud >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > We should publish to a dashboard like pure JAVA does. See BEAM-4283. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4774) Intergrate Nexmark SQL with Perfkit
[ https://issues.apache.org/jira/browse/BEAM-4774?focusedWorklogId=123006=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123006 ] ASF GitHub Bot logged work on BEAM-4774: Author: ASF GitHub Bot Created on: 13/Jul/18 19:15 Start Date: 13/Jul/18 19:15 Worklog Time Spent: 10m Work Description: apilloud commented on issue #5947: [BEAM-4774] Nexmark SQL Postcommits URL: https://github.com/apache/beam/pull/5947#issuecomment-404927609 R: @echauchot cc: @akedin @amaliujia This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 123006) Time Spent: 20m (was: 10m) > Intergrate Nexmark SQL with Perfkit > --- > > Key: BEAM-4774 > URL: https://issues.apache.org/jira/browse/BEAM-4774 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Andrew Pilloud >Assignee: Andrew Pilloud >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > We should publish to a dashboard like pure JAVA does. See BEAM-4283. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4774) Intergrate Nexmark SQL with Perfkit
[ https://issues.apache.org/jira/browse/BEAM-4774?focusedWorklogId=123005=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123005 ] ASF GitHub Bot logged work on BEAM-4774: Author: ASF GitHub Bot Created on: 13/Jul/18 19:14 Start Date: 13/Jul/18 19:14 Worklog Time Spent: 10m Work Description: apilloud opened a new pull request #5947: [BEAM-4774] Nexmark SQL Postcommits URL: https://github.com/apache/beam/pull/5947 This adds the Nexmark SQL jobs to the post commits. Follow this checklist to help us incorporate your contribution quickly and easily: - [X] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). It will help us expedite review of your Pull Request if you tag someone (e.g. `@username`) to look at it. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | --- | --- | --- | --- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 123005) Time Spent: 10m Remaining Estimate: 0h > Intergrate Nexmark SQL with Perfkit > --- > > Key: BEAM-4774 > URL: https://issues.apache.org/jira/browse/BEAM-4774 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Andrew Pilloud >Assignee: Andrew Pilloud >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > We should publish to a dashboard like pure JAVA does. See BEAM-4283. -- This message was sent by
