[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[ https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=122737&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-122737 ] ASF GitHub Bot logged work on BEAM-3995: Author: ASF GitHub Bot Created on: 13/Jul/18 10:09 Start Date: 13/Jul/18 10:09 Worklog Time Spent: 10m Work Description: vectorijk commented on issue #5051: [BEAM-3995] Add basic Gradle run task to Nexmark URL: https://github.com/apache/beam/pull/5051#issuecomment-404789589 I am proposing a new `tpc` module. It needs this feature. This `run` gradle task looks good. Eventually, we could figure out fat jar solution. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 122737) Time Spent: 10h 50m (was: 10h 40m) > Launch Nexmark suites from gradle and update web page docs > -- > > Key: BEAM-3995 > URL: https://issues.apache.org/jira/browse/BEAM-3995 > Project: Beam > Issue Type: Sub-task > Components: examples-nexmark, website >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Fix For: 2.5.0 > > Time Spent: 10h 50m > Remaining Estimate: 0h > > Currently our instructions for running Nexmark benchmarks on various runners > is pretty tightly tied to Maven. We need a good story for running them with > gradle (or just building an executable with gradle and running that > standalone). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[ https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=95346&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-95346 ] ASF GitHub Bot logged work on BEAM-3995: Author: ASF GitHub Bot Created on: 26/Apr/18 05:04 Start Date: 26/Apr/18 05:04 Worklog Time Spent: 10m Work Description: melap commented on issue #418: [BEAM-3995] Update Nexmark launch instructions for Gradle URL: https://github.com/apache/beam-site/pull/418#issuecomment-384513986 the robot likes me right now @kennknowles ;) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 95346) Time Spent: 10h 40m (was: 10.5h) > Launch Nexmark suites from gradle and update web page docs > -- > > Key: BEAM-3995 > URL: https://issues.apache.org/jira/browse/BEAM-3995 > Project: Beam > Issue Type: Sub-task > Components: examples-nexmark, website >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 10h 40m > Remaining Estimate: 0h > > Currently our instructions for running Nexmark benchmarks on various runners > is pretty tightly tied to Maven. We need a good story for running them with > gradle (or just building an executable with gradle and running that > standalone). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[ https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=95345&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-95345 ] ASF GitHub Bot logged work on BEAM-3995: Author: ASF GitHub Bot Created on: 26/Apr/18 05:03 Start Date: 26/Apr/18 05:03 Worklog Time Spent: 10m Work Description: asfgit closed pull request #418: [BEAM-3995] Update Nexmark launch instructions for Gradle URL: https://github.com/apache/beam-site/pull/418 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/src/documentation/sdks/nexmark.md b/src/documentation/sdks/nexmark.md index 5e86cd782..d73943a4f 100644 --- a/src/documentation/sdks/nexmark.md +++ b/src/documentation/sdks/nexmark.md @@ -9,12 +9,14 @@ permalink: /documentation/sdks/java/nexmark/ ## What it is Nexmark is a suite of pipelines inspired by the 'continuous data stream' -queries in [Nexmark research paper](http://datalab.cs.pdx.edu/niagaraST/NEXMark/) +queries in [Nexmark research +paper](http://datalab.cs.pdx.edu/niagaraST/NEXMark/) -These are multiple queries over a three entities model representing on online auction system: +These are multiple queries over a three entities model representing on online +auction system: - - **Person** represents a person submitting an item for auction and/or making a bid -on an auction. + - **Person** represents a person submitting an item for auction and/or making + a bid on an auction. - **Auction** represents an item under auction. - **Bid** represents a bid for an item under auction. @@ -62,11 +64,14 @@ We have augmented the original queries with five more: queries. ## Benchmark workload configuration -Here are some of the knobs of the benchmark workload (see [NexmarkConfiguration.java](https://github.com/apache/beam/blob/master/sdks/java/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java)). + +Here are some of the knobs of the benchmark workload (see +[NexmarkConfiguration.java](https://github.com/apache/beam/blob/master/sdks/java/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java)). These configuration items can be passed to the launch command line. ### Events generation (defaults) + * 100 000 events generated * 100 generator threads * Event rate in SIN curve @@ -76,21 +81,26 @@ These configuration items can be passed to the launch command line. * 1000 concurrent persons bidding / creating auctions ### Windows (defaults) + * size 10s * sliding period 5s * watermark hold for 0s ### Events Proportions (defaults) + * Hot Auctions = ½ * Hot Bidders =¼ * Hot Sellers=¼ ### Technical + * Artificial CPU load * Artificial IO load ## Nexmark output -Here is an example output of the Nexmark benchmark run in streaming mode with the SMOKE suite on the (local) direct runner: + +Here is an example output of the Nexmark benchmark run in streaming mode with +the SMOKE suite on the (local) direct runner: Performance: @@ -112,17 +122,19 @@ Performance: ## Benchmark launch configuration -We can specify the Beam runner to use with maven profiles, available profiles are: - -direct-runner -spark-runner -flink-runner -apex-runner +The Nexmark launcher accepts the `--runner` argument as usual for programs that +use Beam PipelineOptions to manage their command line arguments. In addition +to this, the necessary dependencies must be configured. -The runner must also be specified like in any other Beam pipeline using: +When running via Gradle, the following two parameters control the execution: ---runner +-P nexmark.args +The command line to pass to the Nexmark main program. +-P nexmark.runner + The Gradle project name of the runner, such as ":beam-runners-direct-java" or + ":beam-runners-flink. The project names can be found in the root +`settings.gradle`. Test data is deterministically synthesized on demand. The test data may be synthesized in the same pipeline as the query itself, @@ -417,124 +429,192 @@ Yet to come ### Running SMOKE suite on the DirectRunner (local) +The DirectRunner is default, so it is not required to pass `-Pnexmark.runner`. +Here we do it for maximum clarity. + +The direct runner does not have separate batch and streaming modes, but the +Nexmark launch does. + +These parameters leave on many of the DirectRunner's extra safety checks so the +SMOKE suite can make sure there is nothing broken in the Nexmark suite. + Batch Mode: -mvn exec:java -Dexec.mainClass=org.apache.beam.sdk.nexmark.Main -Pdirect-runner -Dexec.args="--runner=DirectRunner --suite=SMOKE --streaming=fal
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[ https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=95343&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-95343 ] ASF GitHub Bot logged work on BEAM-3995: Author: ASF GitHub Bot Created on: 26/Apr/18 05:00 Start Date: 26/Apr/18 05:00 Worklog Time Spent: 10m Work Description: melap commented on issue #418: [BEAM-3995] Update Nexmark launch instructions for Gradle URL: https://github.com/apache/beam-site/pull/418#issuecomment-384513438 @asfgit merge This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 95343) Time Spent: 10h 20m (was: 10h 10m) > Launch Nexmark suites from gradle and update web page docs > -- > > Key: BEAM-3995 > URL: https://issues.apache.org/jira/browse/BEAM-3995 > Project: Beam > Issue Type: Sub-task > Components: examples-nexmark, website >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 10h 20m > Remaining Estimate: 0h > > Currently our instructions for running Nexmark benchmarks on various runners > is pretty tightly tied to Maven. We need a good story for running them with > gradle (or just building an executable with gradle and running that > standalone). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[ https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=95342&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-95342 ] ASF GitHub Bot logged work on BEAM-3995: Author: ASF GitHub Bot Created on: 26/Apr/18 04:59 Start Date: 26/Apr/18 04:59 Worklog Time Spent: 10m Work Description: asfgit commented on issue #418: [BEAM-3995] Update Nexmark launch instructions for Gradle URL: https://github.com/apache/beam-site/pull/418#issuecomment-384513328 Error: PR failed in verification; check the Jenkins job for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 95342) Time Spent: 10h 10m (was: 10h) > Launch Nexmark suites from gradle and update web page docs > -- > > Key: BEAM-3995 > URL: https://issues.apache.org/jira/browse/BEAM-3995 > Project: Beam > Issue Type: Sub-task > Components: examples-nexmark, website >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 10h 10m > Remaining Estimate: 0h > > Currently our instructions for running Nexmark benchmarks on various runners > is pretty tightly tied to Maven. We need a good story for running them with > gradle (or just building an executable with gradle and running that > standalone). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[ https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=95341&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-95341 ] ASF GitHub Bot logged work on BEAM-3995: Author: ASF GitHub Bot Created on: 26/Apr/18 04:55 Start Date: 26/Apr/18 04:55 Worklog Time Spent: 10m Work Description: kennknowles commented on issue #418: [BEAM-3995] Update Nexmark launch instructions for Gradle URL: https://github.com/apache/beam-site/pull/418#issuecomment-384512907 @asfgit merge This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 95341) Time Spent: 10h (was: 9h 50m) > Launch Nexmark suites from gradle and update web page docs > -- > > Key: BEAM-3995 > URL: https://issues.apache.org/jira/browse/BEAM-3995 > Project: Beam > Issue Type: Sub-task > Components: examples-nexmark, website >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 10h > Remaining Estimate: 0h > > Currently our instructions for running Nexmark benchmarks on various runners > is pretty tightly tied to Maven. We need a good story for running them with > gradle (or just building an executable with gradle and running that > standalone). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[ https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=95338&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-95338 ] ASF GitHub Bot logged work on BEAM-3995: Author: ASF GitHub Bot Created on: 26/Apr/18 04:45 Start Date: 26/Apr/18 04:45 Worklog Time Spent: 10m Work Description: melap commented on issue #418: [BEAM-3995] Update Nexmark launch instructions for Gradle URL: https://github.com/apache/beam-site/pull/418#issuecomment-384511690 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 95338) Time Spent: 9h 50m (was: 9h 40m) > Launch Nexmark suites from gradle and update web page docs > -- > > Key: BEAM-3995 > URL: https://issues.apache.org/jira/browse/BEAM-3995 > Project: Beam > Issue Type: Sub-task > Components: examples-nexmark, website >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 9h 50m > Remaining Estimate: 0h > > Currently our instructions for running Nexmark benchmarks on various runners > is pretty tightly tied to Maven. We need a good story for running them with > gradle (or just building an executable with gradle and running that > standalone). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[ https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=92568&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-92568 ] ASF GitHub Bot logged work on BEAM-3995: Author: ASF GitHub Bot Created on: 19/Apr/18 13:53 Start Date: 19/Apr/18 13:53 Worklog Time Spent: 10m Work Description: asfgit commented on issue #418: [BEAM-3995] Update Nexmark launch instructions for Gradle URL: https://github.com/apache/beam-site/pull/418#issuecomment-382744330 Error: PR failed in verification; check the Jenkins job for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 92568) Time Spent: 9h 40m (was: 9.5h) > Launch Nexmark suites from gradle and update web page docs > -- > > Key: BEAM-3995 > URL: https://issues.apache.org/jira/browse/BEAM-3995 > Project: Beam > Issue Type: Sub-task > Components: examples-nexmark, website >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 9h 40m > Remaining Estimate: 0h > > Currently our instructions for running Nexmark benchmarks on various runners > is pretty tightly tied to Maven. We need a good story for running them with > gradle (or just building an executable with gradle and running that > standalone). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[ https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=92567&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-92567 ] ASF GitHub Bot logged work on BEAM-3995: Author: ASF GitHub Bot Created on: 19/Apr/18 13:49 Start Date: 19/Apr/18 13:49 Worklog Time Spent: 10m Work Description: asfgit commented on issue #418: [BEAM-3995] Update Nexmark launch instructions for Gradle URL: https://github.com/apache/beam-site/pull/418#issuecomment-382742827 Error: PR failed in verification; check the Jenkins job for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 92567) Time Spent: 9.5h (was: 9h 20m) > Launch Nexmark suites from gradle and update web page docs > -- > > Key: BEAM-3995 > URL: https://issues.apache.org/jira/browse/BEAM-3995 > Project: Beam > Issue Type: Sub-task > Components: examples-nexmark, website >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 9.5h > Remaining Estimate: 0h > > Currently our instructions for running Nexmark benchmarks on various runners > is pretty tightly tied to Maven. We need a good story for running them with > gradle (or just building an executable with gradle and running that > standalone). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[ https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=92560&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-92560 ] ASF GitHub Bot logged work on BEAM-3995: Author: ASF GitHub Bot Created on: 19/Apr/18 13:44 Start Date: 19/Apr/18 13:44 Worklog Time Spent: 10m Work Description: kennknowles commented on a change in pull request #418: [BEAM-3995] Update Nexmark launch instructions for Gradle URL: https://github.com/apache/beam-site/pull/418#discussion_r182749979 ## File path: src/documentation/sdks/nexmark.md ## @@ -112,17 +122,19 @@ Performance: ## Benchmark launch configuration -We can specify the Beam runner to use with maven profiles, available profiles are: - -direct-runner -spark-runner -flink-runner -apex-runner +The Nexmark launcher accepts the `--runner` argument as usual for programs that +use Beam PipelineOptions to manage their command line arguments. In addition +to this, the necessary dependencies must be configured. -The runner must also be specified like in any other Beam pipeline using: +When running via Gradle, the following two parameters control the execution: ---runner +-P nexmark.args +The command line to pass to the Nexmark main program. +-P nexmark.runner Review comment: OK. Filed https://issues.apache.org/jira/browse/BEAM-4139 in the gradle improvements to go back to the repo and change it. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 92560) Time Spent: 9h 10m (was: 9h) > Launch Nexmark suites from gradle and update web page docs > -- > > Key: BEAM-3995 > URL: https://issues.apache.org/jira/browse/BEAM-3995 > Project: Beam > Issue Type: Sub-task > Components: examples-nexmark, website >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 9h 10m > Remaining Estimate: 0h > > Currently our instructions for running Nexmark benchmarks on various runners > is pretty tightly tied to Maven. We need a good story for running them with > gradle (or just building an executable with gradle and running that > standalone). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[ https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=92561&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-92561 ] ASF GitHub Bot logged work on BEAM-3995: Author: ASF GitHub Bot Created on: 19/Apr/18 13:44 Start Date: 19/Apr/18 13:44 Worklog Time Spent: 10m Work Description: kennknowles commented on issue #418: [BEAM-3995] Update Nexmark launch instructions for Gradle URL: https://github.com/apache/beam-site/pull/418#issuecomment-382741167 @asfgit merge This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 92561) Time Spent: 9h 20m (was: 9h 10m) > Launch Nexmark suites from gradle and update web page docs > -- > > Key: BEAM-3995 > URL: https://issues.apache.org/jira/browse/BEAM-3995 > Project: Beam > Issue Type: Sub-task > Components: examples-nexmark, website >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 9h 20m > Remaining Estimate: 0h > > Currently our instructions for running Nexmark benchmarks on various runners > is pretty tightly tied to Maven. We need a good story for running them with > gradle (or just building an executable with gradle and running that > standalone). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[ https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=91189&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-91189 ] ASF GitHub Bot logged work on BEAM-3995: Author: ASF GitHub Bot Created on: 16/Apr/18 07:43 Start Date: 16/Apr/18 07:43 Worklog Time Spent: 10m Work Description: echauchot commented on issue #5051: [BEAM-3995] Add basic Gradle run task to Nexmark URL: https://github.com/apache/beam/pull/5051#issuecomment-381508674 @kennknowles if you want to schedule spark streaming run you need to skip queries 3 and 7. The only way to do this right now is to use --query=x --query=y --query=z skipping 3 and 7 or create a custom suite for spark (needs an update of the nexmark code) ... I'm not entirely sure this command line works. But the main problem is the termination part in spark streaming pipelines that I was mentioning above (see related https://issues.apache.org/jira/browse/BEAM-2847). Also, we can schedule Apex runs for which we need to skip query 3 in both batch and streaming modes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 91189) Time Spent: 9h (was: 8h 50m) > Launch Nexmark suites from gradle and update web page docs > -- > > Key: BEAM-3995 > URL: https://issues.apache.org/jira/browse/BEAM-3995 > Project: Beam > Issue Type: Sub-task > Components: examples-nexmark, website >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 9h > Remaining Estimate: 0h > > Currently our instructions for running Nexmark benchmarks on various runners > is pretty tightly tied to Maven. We need a good story for running them with > gradle (or just building an executable with gradle and running that > standalone). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[ https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=91102&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-91102 ] ASF GitHub Bot logged work on BEAM-3995: Author: ASF GitHub Bot Created on: 14/Apr/18 23:27 Start Date: 14/Apr/18 23:27 Worklog Time Spent: 10m Work Description: kennknowles commented on issue #5051: [BEAM-3995] Add basic Gradle run task to Nexmark URL: https://github.com/apache/beam/pull/5051#issuecomment-381367922 @echauchot yes that sounds like a good idea. Let's set things up so we can run things that are not 100% supported by every runner. I think Spark streaming probably supports some of the suites, just not the joins? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 91102) Time Spent: 8h 50m (was: 8h 40m) > Launch Nexmark suites from gradle and update web page docs > -- > > Key: BEAM-3995 > URL: https://issues.apache.org/jira/browse/BEAM-3995 > Project: Beam > Issue Type: Sub-task > Components: examples-nexmark, website >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 8h 50m > Remaining Estimate: 0h > > Currently our instructions for running Nexmark benchmarks on various runners > is pretty tightly tied to Maven. We need a good story for running them with > gradle (or just building an executable with gradle and running that > standalone). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[ https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=90736&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90736 ] ASF GitHub Bot logged work on BEAM-3995: Author: ASF GitHub Bot Created on: 13/Apr/18 08:31 Start Date: 13/Apr/18 08:31 Worklog Time Spent: 10m Work Description: echauchot commented on a change in pull request #418: [BEAM-3995] Update Nexmark launch instructions for Gradle URL: https://github.com/apache/beam-site/pull/418#discussion_r181319081 ## File path: src/documentation/sdks/nexmark.md ## @@ -112,17 +122,19 @@ Performance: ## Benchmark launch configuration -We can specify the Beam runner to use with maven profiles, available profiles are: - -direct-runner -spark-runner -flink-runner -apex-runner +The Nexmark launcher accepts the `--runner` argument as usual for programs that +use Beam PipelineOptions to manage their command line arguments. In addition +to this, the necessary dependencies must be configured. -The runner must also be specified like in any other Beam pipeline using: +When running via Gradle, the following two parameters control the execution: ---runner +-P nexmark.args +The command line to pass to the Nexmark main program. +-P nexmark.runner Review comment: Yes I think I prefer this explicit version rather than the magic one. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 90736) Time Spent: 8h 40m (was: 8.5h) > Launch Nexmark suites from gradle and update web page docs > -- > > Key: BEAM-3995 > URL: https://issues.apache.org/jira/browse/BEAM-3995 > Project: Beam > Issue Type: Sub-task > Components: examples-nexmark, website >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 8h 40m > Remaining Estimate: 0h > > Currently our instructions for running Nexmark benchmarks on various runners > is pretty tightly tied to Maven. We need a good story for running them with > gradle (or just building an executable with gradle and running that > standalone). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[ https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=90568&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90568 ] ASF GitHub Bot logged work on BEAM-3995: Author: ASF GitHub Bot Created on: 12/Apr/18 20:37 Start Date: 12/Apr/18 20:37 Worklog Time Spent: 10m Work Description: kennknowles commented on a change in pull request #418: [BEAM-3995] Update Nexmark launch instructions for Gradle URL: https://github.com/apache/beam-site/pull/418#discussion_r181214223 ## File path: src/documentation/sdks/nexmark.md ## @@ -112,17 +122,19 @@ Performance: ## Benchmark launch configuration -We can specify the Beam runner to use with maven profiles, available profiles are: - -direct-runner -spark-runner -flink-runner -apex-runner +The Nexmark launcher accepts the `--runner` argument as usual for programs that +use Beam PipelineOptions to manage their command line arguments. In addition +to this, the necessary dependencies must be configured. -The runner must also be specified like in any other Beam pipeline using: +When running via Gradle, the following two parameters control the execution: ---runner +-P nexmark.args +The command line to pass to the Nexmark main program. +-P nexmark.runner Review comment: The alternative which I mean is that we could just make it like before where you choose a profile and `-P nexmark.runner` is a choice from an enum, not a magic Gradle subproject name. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 90568) Time Spent: 8.5h (was: 8h 20m) > Launch Nexmark suites from gradle and update web page docs > -- > > Key: BEAM-3995 > URL: https://issues.apache.org/jira/browse/BEAM-3995 > Project: Beam > Issue Type: Sub-task > Components: examples-nexmark, website >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 8.5h > Remaining Estimate: 0h > > Currently our instructions for running Nexmark benchmarks on various runners > is pretty tightly tied to Maven. We need a good story for running them with > gradle (or just building an executable with gradle and running that > standalone). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[ https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=90565&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90565 ] ASF GitHub Bot logged work on BEAM-3995: Author: ASF GitHub Bot Created on: 12/Apr/18 20:36 Start Date: 12/Apr/18 20:36 Worklog Time Spent: 10m Work Description: kennknowles commented on a change in pull request #418: [BEAM-3995] Update Nexmark launch instructions for Gradle URL: https://github.com/apache/beam-site/pull/418#discussion_r181213684 ## File path: src/documentation/sdks/nexmark.md ## @@ -417,124 +429,192 @@ Yet to come ### Running SMOKE suite on the DirectRunner (local) +The DirectRunner is default, so it is not required to pass `-Pnexmark.runner`. +Here we do it for maximum clarity. + +The direct runner does not have separate batch and streaming modes, but the +Nexmark launch does. + +These parameters leave on many of the DirectRunner's extra safety checks so the +SMOKE suite can make sure there is nothing broken in the Nexmark suite. + Batch Mode: -mvn exec:java -Dexec.mainClass=org.apache.beam.sdk.nexmark.Main -Pdirect-runner -Dexec.args="--runner=DirectRunner --suite=SMOKE --streaming=false --manageResources=false --monitorJobs=true --enforceEncodability=true --enforceImmutability=true" +./gradlew :beam-sdks-java-nexmark:run \ +-Pnexmark.runner=":beam-runners-direct-java" \ +-Pnexmark.args=" +--runner=DirectRunner +--streaming=false +--suite=SMOKE +--manageResources=false +--monitorJobs=true +--enforceEncodability=true +--enforceImmutability=true" Streaming Mode: -mvn exec:java -Dexec.mainClass=org.apache.beam.sdk.nexmark.Main -Pdirect-runner -Dexec.args="--runner=DirectRunner --suite=SMOKE --streaming=true --manageResources=false --monitorJobs=true --enforceEncodability=true --enforceImmutability=true" - +./gradlew :beam-sdks-java-nexmark:run \ +-Pnexmark.runner=":beam-runners-direct-java" \ +-Pnexmark.args=" +--runner=DirectRunner +--streaming=true +--suite=SMOKE +--manageResources=false +--monitorJobs=true +--enforceEncodability=true +--enforceImmutability=true" ### Running SMOKE suite on the SparkRunner (local) +The SparkRunner is special-cased in the Nexmark gradle launch. The task will +provide the version of Spark that the SparkRunner is built against, and +configure logging. Review comment: This is a comment about the hardcoding above. The nexmark build tweaks the dependencies based on special needs of the SparkRunner instead of being just a generic suite of pipelines. I will delete the comment here since it "just works" for users copying and pasting from here. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 90565) Time Spent: 8h 10m (was: 8h) > Launch Nexmark suites from gradle and update web page docs > -- > > Key: BEAM-3995 > URL: https://issues.apache.org/jira/browse/BEAM-3995 > Project: Beam > Issue Type: Sub-task > Components: examples-nexmark, website >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 8h 10m > Remaining Estimate: 0h > > Currently our instructions for running Nexmark benchmarks on various runners > is pretty tightly tied to Maven. We need a good story for running them with > gradle (or just building an executable with gradle and running that > standalone). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[ https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=90566&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90566 ] ASF GitHub Bot logged work on BEAM-3995: Author: ASF GitHub Bot Created on: 12/Apr/18 20:36 Start Date: 12/Apr/18 20:36 Worklog Time Spent: 10m Work Description: kennknowles commented on a change in pull request #418: [BEAM-3995] Update Nexmark launch instructions for Gradle URL: https://github.com/apache/beam-site/pull/418#discussion_r181212848 ## File path: src/documentation/sdks/nexmark.md ## @@ -112,17 +122,19 @@ Performance: ## Benchmark launch configuration -We can specify the Beam runner to use with maven profiles, available profiles are: - -direct-runner -spark-runner -flink-runner -apex-runner +The Nexmark launcher accepts the `--runner` argument as usual for programs that +use Beam PipelineOptions to manage their command line arguments. In addition +to this, the necessary dependencies must be configured. -The runner must also be specified like in any other Beam pipeline using: +When running via Gradle, the following two parameters control the execution: ---runner +-P nexmark.args +The command line to pass to the Nexmark main program. +-P nexmark.runner Review comment: Yes, Gradle puts them on the classpath. The one exception is the Spark runner. I used to have a `-P nexmark.provideSpark` parameter to keep it a little decoupled, but since it is pretty much still "magic" knowledge, I followed @lukecwik's suggestion and just hardcoded knowledge of the dependency, so I did that [here](https://github.com/apache/beam/blob/master/sdks/java/nexmark/build.gradle#L35). I think when I add the fat jar capability it may require a separate fat jar for each known runner config. At that point, pretending that the runner dependency can be anything (versus a selection of a hardcoded profile) may not be useful. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 90566) Time Spent: 8h 20m (was: 8h 10m) > Launch Nexmark suites from gradle and update web page docs > -- > > Key: BEAM-3995 > URL: https://issues.apache.org/jira/browse/BEAM-3995 > Project: Beam > Issue Type: Sub-task > Components: examples-nexmark, website >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 8h 20m > Remaining Estimate: 0h > > Currently our instructions for running Nexmark benchmarks on various runners > is pretty tightly tied to Maven. We need a good story for running them with > gradle (or just building an executable with gradle and running that > standalone). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[ https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=90357&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90357 ] ASF GitHub Bot logged work on BEAM-3995: Author: ASF GitHub Bot Created on: 12/Apr/18 09:58 Start Date: 12/Apr/18 09:58 Worklog Time Spent: 10m Work Description: echauchot commented on a change in pull request #418: [BEAM-3995] Update Nexmark launch instructions for Gradle URL: https://github.com/apache/beam-site/pull/418#discussion_r181024910 ## File path: src/documentation/sdks/nexmark.md ## @@ -112,17 +122,19 @@ Performance: ## Benchmark launch configuration -We can specify the Beam runner to use with maven profiles, available profiles are: - -direct-runner -spark-runner -flink-runner -apex-runner +The Nexmark launcher accepts the `--runner` argument as usual for programs that +use Beam PipelineOptions to manage their command line arguments. In addition +to this, the necessary dependencies must be configured. -The runner must also be specified like in any other Beam pipeline using: +When running via Gradle, the following two parameters control the execution: ---runner +-P nexmark.args +The command line to pass to the Nexmark main program. +-P nexmark.runner Review comment: no more profiles for the runners then? The needed dependencies are all included in the runner projet specified by the user? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 90357) Time Spent: 8h (was: 7h 50m) > Launch Nexmark suites from gradle and update web page docs > -- > > Key: BEAM-3995 > URL: https://issues.apache.org/jira/browse/BEAM-3995 > Project: Beam > Issue Type: Sub-task > Components: examples-nexmark, website >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 8h > Remaining Estimate: 0h > > Currently our instructions for running Nexmark benchmarks on various runners > is pretty tightly tied to Maven. We need a good story for running them with > gradle (or just building an executable with gradle and running that > standalone). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[ https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=90356&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90356 ] ASF GitHub Bot logged work on BEAM-3995: Author: ASF GitHub Bot Created on: 12/Apr/18 09:58 Start Date: 12/Apr/18 09:58 Worklog Time Spent: 10m Work Description: echauchot commented on a change in pull request #418: [BEAM-3995] Update Nexmark launch instructions for Gradle URL: https://github.com/apache/beam-site/pull/418#discussion_r181026806 ## File path: src/documentation/sdks/nexmark.md ## @@ -417,124 +429,192 @@ Yet to come ### Running SMOKE suite on the DirectRunner (local) +The DirectRunner is default, so it is not required to pass `-Pnexmark.runner`. +Here we do it for maximum clarity. + +The direct runner does not have separate batch and streaming modes, but the +Nexmark launch does. + +These parameters leave on many of the DirectRunner's extra safety checks so the +SMOKE suite can make sure there is nothing broken in the Nexmark suite. + Batch Mode: -mvn exec:java -Dexec.mainClass=org.apache.beam.sdk.nexmark.Main -Pdirect-runner -Dexec.args="--runner=DirectRunner --suite=SMOKE --streaming=false --manageResources=false --monitorJobs=true --enforceEncodability=true --enforceImmutability=true" +./gradlew :beam-sdks-java-nexmark:run \ +-Pnexmark.runner=":beam-runners-direct-java" \ +-Pnexmark.args=" +--runner=DirectRunner +--streaming=false +--suite=SMOKE +--manageResources=false +--monitorJobs=true +--enforceEncodability=true +--enforceImmutability=true" Streaming Mode: -mvn exec:java -Dexec.mainClass=org.apache.beam.sdk.nexmark.Main -Pdirect-runner -Dexec.args="--runner=DirectRunner --suite=SMOKE --streaming=true --manageResources=false --monitorJobs=true --enforceEncodability=true --enforceImmutability=true" - +./gradlew :beam-sdks-java-nexmark:run \ +-Pnexmark.runner=":beam-runners-direct-java" \ +-Pnexmark.args=" +--runner=DirectRunner +--streaming=true +--suite=SMOKE +--manageResources=false +--monitorJobs=true +--enforceEncodability=true +--enforceImmutability=true" ### Running SMOKE suite on the SparkRunner (local) +The SparkRunner is special-cased in the Nexmark gradle launch. The task will +provide the version of Spark that the SparkRunner is built against, and +configure logging. Review comment: Sorry but I don't understand this comment: owing to what's bellow, the version is automatically provided and no parameter needs to be set by the user. So why not just remove this comment if it is not targeted to the user of nexmark? Is there something I missed? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 90356) Time Spent: 7h 50m (was: 7h 40m) > Launch Nexmark suites from gradle and update web page docs > -- > > Key: BEAM-3995 > URL: https://issues.apache.org/jira/browse/BEAM-3995 > Project: Beam > Issue Type: Sub-task > Components: examples-nexmark, website >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 7h 50m > Remaining Estimate: 0h > > Currently our instructions for running Nexmark benchmarks on various runners > is pretty tightly tied to Maven. We need a good story for running them with > gradle (or just building an executable with gradle and running that > standalone). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[ https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=90354&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90354 ] ASF GitHub Bot logged work on BEAM-3995: Author: ASF GitHub Bot Created on: 12/Apr/18 09:53 Start Date: 12/Apr/18 09:53 Worklog Time Spent: 10m Work Description: echauchot commented on issue #5051: [BEAM-3995] Add basic Gradle run task to Nexmark URL: https://github.com/apache/beam/pull/5051#issuecomment-380745803 @kennknowles Do you agree to schedule the list of runs in my above comment? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 90354) Time Spent: 7h 40m (was: 7.5h) > Launch Nexmark suites from gradle and update web page docs > -- > > Key: BEAM-3995 > URL: https://issues.apache.org/jira/browse/BEAM-3995 > Project: Beam > Issue Type: Sub-task > Components: examples-nexmark, website >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 7h 40m > Remaining Estimate: 0h > > Currently our instructions for running Nexmark benchmarks on various runners > is pretty tightly tied to Maven. We need a good story for running them with > gradle (or just building an executable with gradle and running that > standalone). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[ https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=90126&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90126 ] ASF GitHub Bot logged work on BEAM-3995: Author: ASF GitHub Bot Created on: 11/Apr/18 19:01 Start Date: 11/Apr/18 19:01 Worklog Time Spent: 10m Work Description: kennknowles commented on issue #418: [BEAM-3995] Update Nexmark launch instructions for Gradle URL: https://github.com/apache/beam-site/pull/418#issuecomment-380560925 @jbonofre @echauchot @akedin @apilloud This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 90126) Time Spent: 7.5h (was: 7h 20m) > Launch Nexmark suites from gradle and update web page docs > -- > > Key: BEAM-3995 > URL: https://issues.apache.org/jira/browse/BEAM-3995 > Project: Beam > Issue Type: Sub-task > Components: examples-nexmark, website >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 7.5h > Remaining Estimate: 0h > > Currently our instructions for running Nexmark benchmarks on various runners > is pretty tightly tied to Maven. We need a good story for running them with > gradle (or just building an executable with gradle and running that > standalone). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[ https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=90125&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90125 ] ASF GitHub Bot logged work on BEAM-3995: Author: ASF GitHub Bot Created on: 11/Apr/18 19:00 Start Date: 11/Apr/18 19:00 Worklog Time Spent: 10m Work Description: kennknowles opened a new pull request #418: [BEAM-3995] Update Nexmark launch instructions for Gradle URL: https://github.com/apache/beam-site/pull/418 This ports the instructions to Gradle. There are a couple of known issues, but I thought a PR updating the docs was a great place to make sure the new invocations work. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 90125) Time Spent: 7h 20m (was: 7h 10m) > Launch Nexmark suites from gradle and update web page docs > -- > > Key: BEAM-3995 > URL: https://issues.apache.org/jira/browse/BEAM-3995 > Project: Beam > Issue Type: Sub-task > Components: examples-nexmark, website >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 7h 20m > Remaining Estimate: 0h > > Currently our instructions for running Nexmark benchmarks on various runners > is pretty tightly tied to Maven. We need a good story for running them with > gradle (or just building an executable with gradle and running that > standalone). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[ https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=90085&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90085 ] ASF GitHub Bot logged work on BEAM-3995: Author: ASF GitHub Bot Created on: 11/Apr/18 17:56 Start Date: 11/Apr/18 17:56 Worklog Time Spent: 10m Work Description: kennknowles commented on issue #5051: [BEAM-3995] Add basic Gradle run task to Nexmark URL: https://github.com/apache/beam/pull/5051#issuecomment-380535731 I deliberately considered and rejected the idea that Nexmark should have such explicit intimate knowledge of a runner's implementation details. TBH I'm also unhappy about the fact that it has to know as much as it does about how to provide Spark, or have a special parameter just to do so. Even the presence of the parameter is a major smell. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 90085) Time Spent: 7h 10m (was: 7h) > Launch Nexmark suites from gradle and update web page docs > -- > > Key: BEAM-3995 > URL: https://issues.apache.org/jira/browse/BEAM-3995 > Project: Beam > Issue Type: Sub-task > Components: examples-nexmark, website >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 7h 10m > Remaining Estimate: 0h > > Currently our instructions for running Nexmark benchmarks on various runners > is pretty tightly tied to Maven. We need a good story for running them with > gradle (or just building an executable with gradle and running that > standalone). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[ https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=90084&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90084 ] ASF GitHub Bot logged work on BEAM-3995: Author: ASF GitHub Bot Created on: 11/Apr/18 17:55 Start Date: 11/Apr/18 17:55 Worklog Time Spent: 10m Work Description: kennknowles commented on issue #5051: [BEAM-3995] Add basic Gradle run task to Nexmark URL: https://github.com/apache/beam/pull/5051#issuecomment-380537905 I want this done more than I want to haggle about getting it very high quality in a way we both agree on. I will merge your suggestion to take the LGTM and move on. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 90084) Time Spent: 7h (was: 6h 50m) > Launch Nexmark suites from gradle and update web page docs > -- > > Key: BEAM-3995 > URL: https://issues.apache.org/jira/browse/BEAM-3995 > Project: Beam > Issue Type: Sub-task > Components: examples-nexmark, website >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 7h > Remaining Estimate: 0h > > Currently our instructions for running Nexmark benchmarks on various runners > is pretty tightly tied to Maven. We need a good story for running them with > gradle (or just building an executable with gradle and running that > standalone). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[
https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=90079&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90079
]
ASF GitHub Bot logged work on BEAM-3995:
Author: ASF GitHub Bot
Created on: 11/Apr/18 17:50
Start Date: 11/Apr/18 17:50
Worklog Time Spent: 10m
Work Description: kennknowles closed pull request #5051: [BEAM-3995] Add
basic Gradle run task to Nexmark
URL: https://github.com/apache/beam/pull/5051
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/sdks/java/nexmark/build.gradle b/sdks/java/nexmark/build.gradle
index 83a3d05f99d..8fcba5fa91b 100644
--- a/sdks/java/nexmark/build.gradle
+++ b/sdks/java/nexmark/build.gradle
@@ -21,6 +21,26 @@ applyJavaNature()
description = "Apache Beam :: SDKs :: Java :: Nexmark"
+// When running via Gradle, this property can be used to pass commandline
arguments
+// to the nexmark launch
+def nexmarkArgsProperty = "nexmark.args"
+def nexmarkArgs = project.hasProperty(nexmarkArgsProperty) ?
+project.getProperty(nexmarkArgsProperty).split() : []
+
+// When running via Gradle, this property sets the runner dependency
+def nexmarkRunnerProperty = "nexmark.runner"
+def nexmarkRunnerDependency = (project.hasProperty(nexmarkRunnerProperty)
+? project.getProperty(nexmarkRunnerProperty)
+: ":beam-runners-direct-java")
+def shouldProvideSpark = ":beam-runners-spark".equals(nexmarkRunnerDependency)
+
+configurations {
+ // A configuration for running the Nexmark launcher directly from Gradle,
which
+ // uses Gradle to put the appropriate dependencies on the Classpath rather
than
+ // bundling them into a fat jar
+ gradleRun
+}
+
dependencies {
compile library.java.guava
shadow project(path: ":beam-sdks-java-core", configuration: "shadow")
@@ -43,8 +63,43 @@ dependencies {
shadow project(path: ":beam-runners-direct-java", configuration: "shadow")
shadow library.java.slf4j_jdk14
testCompile library.java.hamcrest_core
+
+ gradleRun project(path: project.path, configuration: "shadow")
+ gradleRun project(path: nexmarkRunnerDependency, configuration: "shadow")
+
+ // The Spark runner requires the user to provide a Spark dependency. For
self-contained
+ // runs with the Spark runner, we can provide such a dependency. This is
deliberately phrased
+ // to not hardcode any runner other than :beam-runners-direct-java
+ if (shouldProvideSpark) {
+gradleRun library.java.spark_streaming
+gradleRun library.java.spark_core, {
+ exclude group:"org.slf4j", module:"jul-to-slf4j"
+}
+ }
+}
+
+if (shouldProvideSpark) {
+ configurations.gradleRun {
+// Using Spark runner causes a StackOverflowError if slf4j-jdk14 is on the
classpath
+exclude group: "org.slf4j", module: "slf4j-jdk14"
+ }
}
test {
jvmArgs "-da"
}
+
+// Execute the Nexmark queries or suites via Gradle.
+//
+// Parameters:
+// -Pnexmark.runner
+// Specify a runner subproject, such as ":beam-runners-spark" or
":beam-runners-flink"
+// Defaults to ":beam-runners-direct-java"
+//
+// -Pnexmark.args
+// Specify the command line for invoking org.apache.beam.sdk.nexmark.Main
+task run(type: JavaExec) {
+ main = "org.apache.beam.sdk.nexmark.Main"
+ classpath = configurations.gradleRun
+ args nexmarkArgs
+}
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 90079)
Time Spent: 6h 50m (was: 6h 40m)
> Launch Nexmark suites from gradle and update web page docs
> --
>
> Key: BEAM-3995
> URL: https://issues.apache.org/jira/browse/BEAM-3995
> Project: Beam
> Issue Type: Sub-task
> Components: examples-nexmark, website
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
> Time Spent: 6h 50m
> Remaining Estimate: 0h
>
> Currently our instructions for running Nexmark benchmarks on various runners
> is pretty tightly tied to Maven. We need a good story for running them with
> gradle (or just building an executable with gradle and running that
> standalone).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[ https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=90076&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90076 ] ASF GitHub Bot logged work on BEAM-3995: Author: ASF GitHub Bot Created on: 11/Apr/18 17:46 Start Date: 11/Apr/18 17:46 Worklog Time Spent: 10m Work Description: kennknowles commented on issue #5051: [BEAM-3995] Add basic Gradle run task to Nexmark URL: https://github.com/apache/beam/pull/5051#issuecomment-380537905 Unfortunately, I want this done more than I want to haggle about getting it to my quality bar. I will merge your suggestion to take the LGTM and move on. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 90076) Time Spent: 6h 40m (was: 6.5h) > Launch Nexmark suites from gradle and update web page docs > -- > > Key: BEAM-3995 > URL: https://issues.apache.org/jira/browse/BEAM-3995 > Project: Beam > Issue Type: Sub-task > Components: examples-nexmark, website >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 6h 40m > Remaining Estimate: 0h > > Currently our instructions for running Nexmark benchmarks on various runners > is pretty tightly tied to Maven. We need a good story for running them with > gradle (or just building an executable with gradle and running that > standalone). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[ https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=90074&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90074 ] ASF GitHub Bot logged work on BEAM-3995: Author: ASF GitHub Bot Created on: 11/Apr/18 17:44 Start Date: 11/Apr/18 17:44 Worklog Time Spent: 10m Work Description: kennknowles commented on issue #5051: [BEAM-3995] Add basic Gradle run task to Nexmark URL: https://github.com/apache/beam/pull/5051#issuecomment-380537355 An alternative would be to make the `nexmark.runner` parameter an enum rather than a free-form dependency, making this _only_ special cases like with Maven profiles. That is also internally consistent. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 90074) Time Spent: 6.5h (was: 6h 20m) > Launch Nexmark suites from gradle and update web page docs > -- > > Key: BEAM-3995 > URL: https://issues.apache.org/jira/browse/BEAM-3995 > Project: Beam > Issue Type: Sub-task > Components: examples-nexmark, website >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 6.5h > Remaining Estimate: 0h > > Currently our instructions for running Nexmark benchmarks on various runners > is pretty tightly tied to Maven. We need a good story for running them with > gradle (or just building an executable with gradle and running that > standalone). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[ https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=90070&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90070 ] ASF GitHub Bot logged work on BEAM-3995: Author: ASF GitHub Bot Created on: 11/Apr/18 17:41 Start Date: 11/Apr/18 17:41 Worklog Time Spent: 10m Work Description: kennknowles commented on issue #5051: [BEAM-3995] Add basic Gradle run task to Nexmark URL: https://github.com/apache/beam/pull/5051#issuecomment-380535731 I deliberately considered and rejected the idea that Nexmark should have such explicit intimate knowledge of a runner's implementation details. TBH I'm also unhappy about the fact that it has to know as much as it does, but I don't want to make it worse. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 90070) Time Spent: 6h 20m (was: 6h 10m) > Launch Nexmark suites from gradle and update web page docs > -- > > Key: BEAM-3995 > URL: https://issues.apache.org/jira/browse/BEAM-3995 > Project: Beam > Issue Type: Sub-task > Components: examples-nexmark, website >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 6h 20m > Remaining Estimate: 0h > > Currently our instructions for running Nexmark benchmarks on various runners > is pretty tightly tied to Maven. We need a good story for running them with > gradle (or just building an executable with gradle and running that > standalone). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[ https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=90066&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90066 ] ASF GitHub Bot logged work on BEAM-3995: Author: ASF GitHub Bot Created on: 11/Apr/18 17:40 Start Date: 11/Apr/18 17:40 Worklog Time Spent: 10m Work Description: kennknowles commented on issue #5051: [BEAM-3995] Add basic Gradle run task to Nexmark URL: https://github.com/apache/beam/pull/5051#issuecomment-380535731 I deliberately considered and rejected the idea that Nexmark should have intimate knowledge of a runner's implementation details. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 90066) Time Spent: 6h (was: 5h 50m) > Launch Nexmark suites from gradle and update web page docs > -- > > Key: BEAM-3995 > URL: https://issues.apache.org/jira/browse/BEAM-3995 > Project: Beam > Issue Type: Sub-task > Components: examples-nexmark, website >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 6h > Remaining Estimate: 0h > > Currently our instructions for running Nexmark benchmarks on various runners > is pretty tightly tied to Maven. We need a good story for running them with > gradle (or just building an executable with gradle and running that > standalone). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[ https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=90067&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90067 ] ASF GitHub Bot logged work on BEAM-3995: Author: ASF GitHub Bot Created on: 11/Apr/18 17:40 Start Date: 11/Apr/18 17:40 Worklog Time Spent: 10m Work Description: kennknowles commented on issue #5051: [BEAM-3995] Add basic Gradle run task to Nexmark URL: https://github.com/apache/beam/pull/5051#issuecomment-380535731 I deliberately considered and rejected the idea that Nexmark should have such explicit intimate knowledge of a runner's implementation details. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 90067) Time Spent: 6h 10m (was: 6h) > Launch Nexmark suites from gradle and update web page docs > -- > > Key: BEAM-3995 > URL: https://issues.apache.org/jira/browse/BEAM-3995 > Project: Beam > Issue Type: Sub-task > Components: examples-nexmark, website >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 6h 10m > Remaining Estimate: 0h > > Currently our instructions for running Nexmark benchmarks on various runners > is pretty tightly tied to Maven. We need a good story for running them with > gradle (or just building an executable with gradle and running that > standalone). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[ https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=90065&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90065 ] ASF GitHub Bot logged work on BEAM-3995: Author: ASF GitHub Bot Created on: 11/Apr/18 17:39 Start Date: 11/Apr/18 17:39 Worklog Time Spent: 10m Work Description: kennknowles commented on issue #5051: [BEAM-3995] Add basic Gradle run task to Nexmark URL: https://github.com/apache/beam/pull/5051#issuecomment-380535731 I deliberately considered and rejected the idea that we should explicitly special case based on magic strings or knowledge of a particular runner. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 90065) Time Spent: 5h 50m (was: 5h 40m) > Launch Nexmark suites from gradle and update web page docs > -- > > Key: BEAM-3995 > URL: https://issues.apache.org/jira/browse/BEAM-3995 > Project: Beam > Issue Type: Sub-task > Components: examples-nexmark, website >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 5h 50m > Remaining Estimate: 0h > > Currently our instructions for running Nexmark benchmarks on various runners > is pretty tightly tied to Maven. We need a good story for running them with > gradle (or just building an executable with gradle and running that > standalone). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[ https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=90062&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90062 ] ASF GitHub Bot logged work on BEAM-3995: Author: ASF GitHub Bot Created on: 11/Apr/18 17:38 Start Date: 11/Apr/18 17:38 Worklog Time Spent: 10m Work Description: kennknowles commented on issue #5051: [BEAM-3995] Add basic Gradle run task to Nexmark URL: https://github.com/apache/beam/pull/5051#issuecomment-380535731 I deliberately considered and rejected the idea that we should explicitly special case based on magic strings. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 90062) Time Spent: 5h 40m (was: 5.5h) > Launch Nexmark suites from gradle and update web page docs > -- > > Key: BEAM-3995 > URL: https://issues.apache.org/jira/browse/BEAM-3995 > Project: Beam > Issue Type: Sub-task > Components: examples-nexmark, website >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 5h 40m > Remaining Estimate: 0h > > Currently our instructions for running Nexmark benchmarks on various runners > is pretty tightly tied to Maven. We need a good story for running them with > gradle (or just building an executable with gradle and running that > standalone). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[ https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=90060&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90060 ] ASF GitHub Bot logged work on BEAM-3995: Author: ASF GitHub Bot Created on: 11/Apr/18 17:38 Start Date: 11/Apr/18 17:38 Worklog Time Spent: 10m Work Description: kennknowles commented on issue #5051: [BEAM-3995] Add basic Gradle run task to Nexmark URL: https://github.com/apache/beam/pull/5051#issuecomment-380535492 Your change LGTM. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 90060) Time Spent: 5.5h (was: 5h 20m) > Launch Nexmark suites from gradle and update web page docs > -- > > Key: BEAM-3995 > URL: https://issues.apache.org/jira/browse/BEAM-3995 > Project: Beam > Issue Type: Sub-task > Components: examples-nexmark, website >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 5.5h > Remaining Estimate: 0h > > Currently our instructions for running Nexmark benchmarks on various runners > is pretty tightly tied to Maven. We need a good story for running them with > gradle (or just building an executable with gradle and running that > standalone). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[ https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=90059&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90059 ] ASF GitHub Bot logged work on BEAM-3995: Author: ASF GitHub Bot Created on: 11/Apr/18 17:37 Start Date: 11/Apr/18 17:37 Worklog Time Spent: 10m Work Description: kennknowles commented on issue #5051: [BEAM-3995] Add basic Gradle run task to Nexmark URL: https://github.com/apache/beam/pull/5051#issuecomment-380535433 It is not credible that the naming revert was technical required. Easy to believe it was the shortest path to getting things out the door, though. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 90059) Time Spent: 5h 20m (was: 5h 10m) > Launch Nexmark suites from gradle and update web page docs > -- > > Key: BEAM-3995 > URL: https://issues.apache.org/jira/browse/BEAM-3995 > Project: Beam > Issue Type: Sub-task > Components: examples-nexmark, website >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 5h 20m > Remaining Estimate: 0h > > Currently our instructions for running Nexmark benchmarks on various runners > is pretty tightly tied to Maven. We need a good story for running them with > gradle (or just building an executable with gradle and running that > standalone). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[ https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=90036&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90036 ] ASF GitHub Bot logged work on BEAM-3995: Author: ASF GitHub Bot Created on: 11/Apr/18 16:45 Start Date: 11/Apr/18 16:45 Worklog Time Spent: 10m Work Description: akedin commented on issue #5051: [BEAM-3995] Add basic Gradle run task to Nexmark URL: https://github.com/apache/beam/pull/5051#issuecomment-380519633 lgtm This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 90036) Time Spent: 5h 10m (was: 5h) > Launch Nexmark suites from gradle and update web page docs > -- > > Key: BEAM-3995 > URL: https://issues.apache.org/jira/browse/BEAM-3995 > Project: Beam > Issue Type: Sub-task > Components: examples-nexmark, website >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 5h 10m > Remaining Estimate: 0h > > Currently our instructions for running Nexmark benchmarks on various runners > is pretty tightly tied to Maven. We need a good story for running them with > gradle (or just building an executable with gradle and running that > standalone). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[
https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=89963&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-89963
]
ASF GitHub Bot logged work on BEAM-3995:
Author: ASF GitHub Bot
Created on: 11/Apr/18 13:52
Start Date: 11/Apr/18 13:52
Worklog Time Spent: 10m
Work Description: lukecwik commented on a change in pull request #5051:
[BEAM-3995] Add basic Gradle run task to Nexmark
URL: https://github.com/apache/beam/pull/5051#discussion_r180760912
##
File path: sdks/java/nexmark/build.gradle
##
@@ -43,8 +68,53 @@ dependencies {
shadow project(path: ":beam-runners-direct-java", configuration: "shadow")
shadow library.java.slf4j_jdk14
testCompile library.java.hamcrest_core
+
+ gradleRun project(path: project.path, configuration: "shadow")
+ gradleRun project(path: nexmarkRunnerDependency, configuration: "shadow")
+
+ // The Spark runner requires the user to provide a Spark dependency. For
self-contained
+ // runs with the Spark runner, we can provide such a dependency. This is
deliberately phrased
+ // to not hardcode any runner other than :runners:direct-java
+ if (shouldProvideSpark) {
+gradleRun library.java.spark_streaming
+gradleRun library.java.spark_core, {
+ exclude group:"org.slf4j", module:"jul-to-slf4j"
+}
+
+// TODO: pre-Gradle this was excluded in the SparkRunner overall, in
Gradle it is
+// only excluded in the testRuntimeClasspath.
+gradleRun project(path: ":beam-sdks-java-core", configuration: "shadow"), {
Review comment:
You don't need to specify the project, you can just exclude the values
configuration wide which your doing below.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 89963)
Time Spent: 5h (was: 4h 50m)
> Launch Nexmark suites from gradle and update web page docs
> --
>
> Key: BEAM-3995
> URL: https://issues.apache.org/jira/browse/BEAM-3995
> Project: Beam
> Issue Type: Sub-task
> Components: examples-nexmark, website
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
> Time Spent: 5h
> Remaining Estimate: 0h
>
> Currently our instructions for running Nexmark benchmarks on various runners
> is pretty tightly tied to Maven. We need a good story for running them with
> gradle (or just building an executable with gradle and running that
> standalone).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[
https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=89962&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-89962
]
ASF GitHub Bot logged work on BEAM-3995:
Author: ASF GitHub Bot
Created on: 11/Apr/18 13:51
Start Date: 11/Apr/18 13:51
Worklog Time Spent: 10m
Work Description: lukecwik commented on a change in pull request #5051:
[BEAM-3995] Add basic Gradle run task to Nexmark
URL: https://github.com/apache/beam/pull/5051#discussion_r180760912
##
File path: sdks/java/nexmark/build.gradle
##
@@ -43,8 +68,53 @@ dependencies {
shadow project(path: ":beam-runners-direct-java", configuration: "shadow")
shadow library.java.slf4j_jdk14
testCompile library.java.hamcrest_core
+
+ gradleRun project(path: project.path, configuration: "shadow")
+ gradleRun project(path: nexmarkRunnerDependency, configuration: "shadow")
+
+ // The Spark runner requires the user to provide a Spark dependency. For
self-contained
+ // runs with the Spark runner, we can provide such a dependency. This is
deliberately phrased
+ // to not hardcode any runner other than :runners:direct-java
+ if (shouldProvideSpark) {
+gradleRun library.java.spark_streaming
+gradleRun library.java.spark_core, {
+ exclude group:"org.slf4j", module:"jul-to-slf4j"
+}
+
+// TODO: pre-Gradle this was excluded in the SparkRunner overall, in
Gradle it is
+// only excluded in the testRuntimeClasspath.
+gradleRun project(path: ":beam-sdks-java-core", configuration: "shadow"), {
Review comment:
You don't need to specify the project, you can just exclude the values
configuration wide.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 89962)
Time Spent: 4h 50m (was: 4h 40m)
> Launch Nexmark suites from gradle and update web page docs
> --
>
> Key: BEAM-3995
> URL: https://issues.apache.org/jira/browse/BEAM-3995
> Project: Beam
> Issue Type: Sub-task
> Components: examples-nexmark, website
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
> Time Spent: 4h 50m
> Remaining Estimate: 0h
>
> Currently our instructions for running Nexmark benchmarks on various runners
> is pretty tightly tied to Maven. We need a good story for running them with
> gradle (or just building an executable with gradle and running that
> standalone).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[ https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=89838&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-89838 ] ASF GitHub Bot logged work on BEAM-3995: Author: ASF GitHub Bot Created on: 11/Apr/18 08:04 Start Date: 11/Apr/18 08:04 Worklog Time Spent: 10m Work Description: echauchot commented on issue #5051: [BEAM-3995] Add basic Gradle run task to Nexmark URL: https://github.com/apache/beam/pull/5051#issuecomment-380357483 I have recently updated the nexmark support matrix on the website after running all the test suites. Runners and modes that support 100% of the queries are - on direct runner in batch mode - on direct runner in streaming mode - on flink runner in batch mode - on flink runner in streaming mode - on spark runner in batch mode => I think we should schedule all these runs It is a pity that we can not run spark in streaming mode but we still have major issues with it in streaming mode: - state support - timer support - pcollectionView support - spark does not comply to the user given timeout in `waitUntilFinish` probably because it forks a lot of micro-batches and, to gracefull stop the pipeline, the underlining spark engine needs to wait for all of them leading to almost never-ending benchmark suites. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 89838) Time Spent: 4h 40m (was: 4.5h) > Launch Nexmark suites from gradle and update web page docs > -- > > Key: BEAM-3995 > URL: https://issues.apache.org/jira/browse/BEAM-3995 > Project: Beam > Issue Type: Sub-task > Components: examples-nexmark, website >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 4h 40m > Remaining Estimate: 0h > > Currently our instructions for running Nexmark benchmarks on various runners > is pretty tightly tied to Maven. We need a good story for running them with > gradle (or just building an executable with gradle and running that > standalone). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[
https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=89837&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-89837
]
ASF GitHub Bot logged work on BEAM-3995:
Author: ASF GitHub Bot
Created on: 11/Apr/18 08:01
Start Date: 11/Apr/18 08:01
Worklog Time Spent: 10m
Work Description: echauchot commented on a change in pull request #5051:
[BEAM-3995] Add basic Gradle run task to Nexmark
URL: https://github.com/apache/beam/pull/5051#discussion_r180658821
##
File path: sdks/java/nexmark/build.gradle
##
@@ -46,3 +46,14 @@ dependencies {
test {
jvmArgs "-da"
}
+
+def nexmarkArgsProperty = "nexmark.args"
+
+task run (type: JavaExec) {
Review comment:
yes you just run Main and it runs all the queries that are specified on the
benchmark suite (e.g. SMOKE)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 89837)
Time Spent: 4.5h (was: 4h 20m)
> Launch Nexmark suites from gradle and update web page docs
> --
>
> Key: BEAM-3995
> URL: https://issues.apache.org/jira/browse/BEAM-3995
> Project: Beam
> Issue Type: Sub-task
> Components: examples-nexmark, website
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
> Currently our instructions for running Nexmark benchmarks on various runners
> is pretty tightly tied to Maven. We need a good story for running them with
> gradle (or just building an executable with gradle and running that
> standalone).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[ https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=89834&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-89834 ] ASF GitHub Bot logged work on BEAM-3995: Author: ASF GitHub Bot Created on: 11/Apr/18 07:57 Start Date: 11/Apr/18 07:57 Worklog Time Spent: 10m Work Description: echauchot commented on issue #5051: [BEAM-3995] Add basic Gradle run task to Nexmark URL: https://github.com/apache/beam/pull/5051#issuecomment-380356889 @kennknowles I had an ongoing PR (https://github.com/apache/beam/pull/4976) for doing exactly that on maven. I was about to convert it to gradle, you were quicker :) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 89834) Time Spent: 4h 20m (was: 4h 10m) > Launch Nexmark suites from gradle and update web page docs > -- > > Key: BEAM-3995 > URL: https://issues.apache.org/jira/browse/BEAM-3995 > Project: Beam > Issue Type: Sub-task > Components: examples-nexmark, website >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 4h 20m > Remaining Estimate: 0h > > Currently our instructions for running Nexmark benchmarks on various runners > is pretty tightly tied to Maven. We need a good story for running them with > gradle (or just building an executable with gradle and running that > standalone). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[ https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=89829&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-89829 ] ASF GitHub Bot logged work on BEAM-3995: Author: ASF GitHub Bot Created on: 11/Apr/18 07:40 Start Date: 11/Apr/18 07:40 Worklog Time Spent: 10m Work Description: echauchot commented on issue #5051: [BEAM-3995] Add basic Gradle run task to Nexmark URL: https://github.com/apache/beam/pull/5051#issuecomment-380356889 @kennknowles I had an ongoing PR for doing exactly that on maven. I was about to convert it to gradle, you were quicker :) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 89829) Time Spent: 4h 10m (was: 4h) > Launch Nexmark suites from gradle and update web page docs > -- > > Key: BEAM-3995 > URL: https://issues.apache.org/jira/browse/BEAM-3995 > Project: Beam > Issue Type: Sub-task > Components: examples-nexmark, website >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 4h 10m > Remaining Estimate: 0h > > Currently our instructions for running Nexmark benchmarks on various runners > is pretty tightly tied to Maven. We need a good story for running them with > gradle (or just building an executable with gradle and running that > standalone). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[ https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=89826&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-89826 ] ASF GitHub Bot logged work on BEAM-3995: Author: ASF GitHub Bot Created on: 11/Apr/18 07:37 Start Date: 11/Apr/18 07:37 Worklog Time Spent: 10m Work Description: echauchot commented on issue #5051: [BEAM-3995] Add basic Gradle run task to Nexmark URL: https://github.com/apache/beam/pull/5051#issuecomment-380357483 I have recently updated the nexmark support matrix on the website. Runners and modes that support 100% of the queries are - on direct runner in batch mode - on direct runner in streaming mode - on flink runner in batch mode - on flink runner in streaming mode - on spark runner in batch mode I think we should schedule all these runs This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 89826) Time Spent: 4h (was: 3h 50m) > Launch Nexmark suites from gradle and update web page docs > -- > > Key: BEAM-3995 > URL: https://issues.apache.org/jira/browse/BEAM-3995 > Project: Beam > Issue Type: Sub-task > Components: examples-nexmark, website >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 4h > Remaining Estimate: 0h > > Currently our instructions for running Nexmark benchmarks on various runners > is pretty tightly tied to Maven. We need a good story for running them with > gradle (or just building an executable with gradle and running that > standalone). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[ https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=89825&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-89825 ] ASF GitHub Bot logged work on BEAM-3995: Author: ASF GitHub Bot Created on: 11/Apr/18 07:35 Start Date: 11/Apr/18 07:35 Worklog Time Spent: 10m Work Description: echauchot commented on issue #5051: [BEAM-3995] Add basic Gradle run task to Nexmark URL: https://github.com/apache/beam/pull/5051#issuecomment-380356889 @kennknowles I had an ongoing PR for doing exactly that on maven. I was about to convert it to gradle. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 89825) Time Spent: 3h 50m (was: 3h 40m) > Launch Nexmark suites from gradle and update web page docs > -- > > Key: BEAM-3995 > URL: https://issues.apache.org/jira/browse/BEAM-3995 > Project: Beam > Issue Type: Sub-task > Components: examples-nexmark, website >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 3h 50m > Remaining Estimate: 0h > > Currently our instructions for running Nexmark benchmarks on various runners > is pretty tightly tied to Maven. We need a good story for running them with > gradle (or just building an executable with gradle and running that > standalone). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[
https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=89824&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-89824
]
ASF GitHub Bot logged work on BEAM-3995:
Author: ASF GitHub Bot
Created on: 11/Apr/18 07:32
Start Date: 11/Apr/18 07:32
Worklog Time Spent: 10m
Work Description: echauchot commented on a change in pull request #5051:
[BEAM-3995] Add basic Gradle run task to Nexmark
URL: https://github.com/apache/beam/pull/5051#discussion_r180658821
##
File path: sdks/java/nexmark/build.gradle
##
@@ -46,3 +46,14 @@ dependencies {
test {
jvmArgs "-da"
}
+
+def nexmarkArgsProperty = "nexmark.args"
+
+task run (type: JavaExec) {
Review comment:
yes you just run Main and it runs all the queries that are specified on the
benchmark profile (e.g. SMOKE)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 89824)
Time Spent: 3h 40m (was: 3.5h)
> Launch Nexmark suites from gradle and update web page docs
> --
>
> Key: BEAM-3995
> URL: https://issues.apache.org/jira/browse/BEAM-3995
> Project: Beam
> Issue Type: Sub-task
> Components: examples-nexmark, website
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> Currently our instructions for running Nexmark benchmarks on various runners
> is pretty tightly tied to Maven. We need a good story for running them with
> gradle (or just building an executable with gradle and running that
> standalone).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[
https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=89823&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-89823
]
ASF GitHub Bot logged work on BEAM-3995:
Author: ASF GitHub Bot
Created on: 11/Apr/18 07:31
Start Date: 11/Apr/18 07:31
Worklog Time Spent: 10m
Work Description: echauchot commented on a change in pull request #5051:
[BEAM-3995] Add basic Gradle run task to Nexmark
URL: https://github.com/apache/beam/pull/5051#discussion_r180658821
##
File path: sdks/java/nexmark/build.gradle
##
@@ -46,3 +46,14 @@ dependencies {
test {
jvmArgs "-da"
}
+
+def nexmarkArgsProperty = "nexmark.args"
+
+task run (type: JavaExec) {
Review comment:
yes you just run Main
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 89823)
Time Spent: 3.5h (was: 3h 20m)
> Launch Nexmark suites from gradle and update web page docs
> --
>
> Key: BEAM-3995
> URL: https://issues.apache.org/jira/browse/BEAM-3995
> Project: Beam
> Issue Type: Sub-task
> Components: examples-nexmark, website
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> Currently our instructions for running Nexmark benchmarks on various runners
> is pretty tightly tied to Maven. We need a good story for running them with
> gradle (or just building an executable with gradle and running that
> standalone).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[ https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=89750&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-89750 ] ASF GitHub Bot logged work on BEAM-3995: Author: ASF GitHub Bot Created on: 11/Apr/18 02:11 Start Date: 11/Apr/18 02:11 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #5051: [BEAM-3995] Add basic Gradle run task to Nexmark URL: https://github.com/apache/beam/pull/5051#issuecomment-380303801 Aligning the naming for projects was needed to generate the correct dependency names between projects. Note that you can always use `-p path/to/project taskWithinProject` to execute using directory names instead of `:beam-project-name:taskWithinProject` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 89750) Time Spent: 3h 20m (was: 3h 10m) > Launch Nexmark suites from gradle and update web page docs > -- > > Key: BEAM-3995 > URL: https://issues.apache.org/jira/browse/BEAM-3995 > Project: Beam > Issue Type: Sub-task > Components: examples-nexmark, website >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 3h 20m > Remaining Estimate: 0h > > Currently our instructions for running Nexmark benchmarks on various runners > is pretty tightly tied to Maven. We need a good story for running them with > gradle (or just building an executable with gradle and running that > standalone). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[ https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=89654&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-89654 ] ASF GitHub Bot logged work on BEAM-3995: Author: ASF GitHub Bot Created on: 10/Apr/18 22:23 Start Date: 10/Apr/18 22:23 Worklog Time Spent: 10m Work Description: kennknowles commented on issue #5051: [BEAM-3995] Add basic Gradle run task to Nexmark URL: https://github.com/apache/beam/pull/5051#issuecomment-380266575 New commands (will port web page after merge) DirectRunner SMOKE (default): ``` ./gradlew :beam-sdks-java-nexmark:run -Pnexmark.args=" --runner=DirectRunner --suite=SMOKE --streaming=false --manageResources=false --monitorJobs=true --enforceEncodability=true --enforceImmutability=true" ``` DirectRunner SMOKE (explicit): ``` ./gradlew :beam-sdks-java-nexmark:run \ -Pnexmark.runner=":beam-runners-direct-java" \ -Pnexmark.args=" --runner=DirectRunner --suite=SMOKE --streaming=false --manageResources=false --monitorJobs=true --enforceEncodability=true --enforceImmutability=true" ``` Spark (special provided dep demonstration): ``` ./gradlew :beam-sdks-java-nexmark:run \ -Pnexmark.runner=":beam-runners-spark" \ -Pnexmark.provideSpark=true \ -Pnexmark.args=" --runner=SparkRunner --suite=SMOKE --streamTimeout=60 --streaming=false --manageResources=false --monitorJobs=true" ``` Flink: ``` ./gradlew :beam-sdks-java-nexmark:run \ -Pnexmark.runner=":beam-runners-flink_2.11" \ -Pnexmark.args=" --runner=FlinkRunner --suite=SMOKE --streamTimeout=60 --streaming=false --manageResources=false --monitorJobs=true --flinkMaster=local" ``` CC @akedin @apilloud This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 89654) Time Spent: 3h 10m (was: 3h) > Launch Nexmark suites from gradle and update web page docs > -- > > Key: BEAM-3995 > URL: https://issues.apache.org/jira/browse/BEAM-3995 > Project: Beam > Issue Type: Sub-task > Components: examples-nexmark, website >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 3h 10m > Remaining Estimate: 0h > > Currently our instructions for running Nexmark benchmarks on various runners > is pretty tightly tied to Maven. We need a good story for running them with > gradle (or just building an executable with gradle and running that > standalone). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[ https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=89652&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-89652 ] ASF GitHub Bot logged work on BEAM-3995: Author: ASF GitHub Bot Created on: 10/Apr/18 22:10 Start Date: 10/Apr/18 22:10 Worklog Time Spent: 10m Work Description: kennknowles commented on issue #5051: [BEAM-3995] Add basic Gradle run task to Nexmark URL: https://github.com/apache/beam/pull/5051#issuecomment-380263705 OK I've regressed this PR to match. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 89652) Time Spent: 3h (was: 2h 50m) > Launch Nexmark suites from gradle and update web page docs > -- > > Key: BEAM-3995 > URL: https://issues.apache.org/jira/browse/BEAM-3995 > Project: Beam > Issue Type: Sub-task > Components: examples-nexmark, website >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 3h > Remaining Estimate: 0h > > Currently our instructions for running Nexmark benchmarks on various runners > is pretty tightly tied to Maven. We need a good story for running them with > gradle (or just building an executable with gradle and running that > standalone). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[ https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=89636&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-89636 ] ASF GitHub Bot logged work on BEAM-3995: Author: ASF GitHub Bot Created on: 10/Apr/18 21:42 Start Date: 10/Apr/18 21:42 Worklog Time Spent: 10m Work Description: kennknowles commented on issue #5051: [BEAM-3995] Add basic Gradle run task to Nexmark URL: https://github.com/apache/beam/pull/5051#issuecomment-380256699 If the renames are mandatory for some obscure reason, I'll get to it in a bit. Otherwise we should revert and make the project structure simple and have explicit artifact ids. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 89636) Time Spent: 2h 50m (was: 2h 40m) > Launch Nexmark suites from gradle and update web page docs > -- > > Key: BEAM-3995 > URL: https://issues.apache.org/jira/browse/BEAM-3995 > Project: Beam > Issue Type: Sub-task > Components: examples-nexmark, website >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 2h 50m > Remaining Estimate: 0h > > Currently our instructions for running Nexmark benchmarks on various runners > is pretty tightly tied to Maven. We need a good story for running them with > gradle (or just building an executable with gradle and running that > standalone). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[ https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=89635&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-89635 ] ASF GitHub Bot logged work on BEAM-3995: Author: ASF GitHub Bot Created on: 10/Apr/18 21:36 Start Date: 10/Apr/18 21:36 Worklog Time Spent: 10m Work Description: kennknowles commented on issue #5051: [BEAM-3995] Add basic Gradle run task to Nexmark URL: https://github.com/apache/beam/pull/5051#issuecomment-380255022 It looks like the automatic correspondence between directories and projects was regressed? Is this a maven compatibility thing? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 89635) Time Spent: 2h 40m (was: 2.5h) > Launch Nexmark suites from gradle and update web page docs > -- > > Key: BEAM-3995 > URL: https://issues.apache.org/jira/browse/BEAM-3995 > Project: Beam > Issue Type: Sub-task > Components: examples-nexmark, website >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 2h 40m > Remaining Estimate: 0h > > Currently our instructions for running Nexmark benchmarks on various runners > is pretty tightly tied to Maven. We need a good story for running them with > gradle (or just building an executable with gradle and running that > standalone). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[ https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=89634&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-89634 ] ASF GitHub Bot logged work on BEAM-3995: Author: ASF GitHub Bot Created on: 10/Apr/18 21:34 Start Date: 10/Apr/18 21:34 Worklog Time Spent: 10m Work Description: kennknowles commented on issue #5051: [BEAM-3995] Add basic Gradle run task to Nexmark URL: https://github.com/apache/beam/pull/5051#issuecomment-380254396 Confirmed via: DirectRunner SMOKE: ``` // Defaults to direct runner ./gradlew :sdks:java:nexmark:run -Pnexmark.args="--runner=DirectRunner --suite=SMOKE --streaming=false --manageResources=false --monitorJobs=true --enforceEncodability=true --enforceImmutability=true" ./gradlew :sdks:java:nexmark:run -Pnexmark.runner=":runners:direct-java" -Pnexmark.args="--runner=DirectRunner --suite=SMOKE --streaming=false --manageResources=false --monitorJobs=true --enforceEncodability=true --enforceImmutability=true" ``` FlinkRunner SMOKE: ``` ./gradlew :sdks:java:nexmark:run -Pnexmark.runner=":runners:flink" -Pnexmark.args="--runner=FlinkRunner --suite=SMOKE --streamTimeout=60 --streaming=false --manageResources=false --monitorJobs=true --flinkMaster=local" ``` SparkRunner SMOKE: ``` ./gradlew :sdks:java:nexmark:run -Pnexmark.runner=":runners:spark" -Pnexmark.provideSpark=true -Pnexmark.args="--runner=SparkRunner --suite=SMOKE --streamTimeout=60 --streaming=false --manageResources=false --monitorJobs=true" ``` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 89634) Time Spent: 2.5h (was: 2h 20m) > Launch Nexmark suites from gradle and update web page docs > -- > > Key: BEAM-3995 > URL: https://issues.apache.org/jira/browse/BEAM-3995 > Project: Beam > Issue Type: Sub-task > Components: examples-nexmark, website >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 2.5h > Remaining Estimate: 0h > > Currently our instructions for running Nexmark benchmarks on various runners > is pretty tightly tied to Maven. We need a good story for running them with > gradle (or just building an executable with gradle and running that > standalone). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[
https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=89631&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-89631
]
ASF GitHub Bot logged work on BEAM-3995:
Author: ASF GitHub Bot
Created on: 10/Apr/18 21:32
Start Date: 10/Apr/18 21:32
Worklog Time Spent: 10m
Work Description: kennknowles commented on a change in pull request
#5051: [BEAM-3995] Add basic DirectRunner Gradle run task to Nexmark
URL: https://github.com/apache/beam/pull/5051#discussion_r180561808
##
File path: sdks/java/nexmark/build.gradle
##
@@ -46,3 +46,14 @@ dependencies {
test {
jvmArgs "-da"
}
+
+def nexmarkArgsProperty = "nexmark.args"
+
+task run (type: JavaExec) {
Review comment:
Nope, it is just one executable, configured by command line.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 89631)
Time Spent: 2h 10m (was: 2h)
> Launch Nexmark suites from gradle and update web page docs
> --
>
> Key: BEAM-3995
> URL: https://issues.apache.org/jira/browse/BEAM-3995
> Project: Beam
> Issue Type: Sub-task
> Components: examples-nexmark, website
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> Currently our instructions for running Nexmark benchmarks on various runners
> is pretty tightly tied to Maven. We need a good story for running them with
> gradle (or just building an executable with gradle and running that
> standalone).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[
https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=89632&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-89632
]
ASF GitHub Bot logged work on BEAM-3995:
Author: ASF GitHub Bot
Created on: 10/Apr/18 21:32
Start Date: 10/Apr/18 21:32
Worklog Time Spent: 10m
Work Description: kennknowles commented on a change in pull request
#5051: [BEAM-3995] Add basic DirectRunner Gradle run task to Nexmark
URL: https://github.com/apache/beam/pull/5051#discussion_r180567338
##
File path: sdks/java/nexmark/build.gradle
##
@@ -46,3 +46,14 @@ dependencies {
test {
jvmArgs "-da"
}
+
+def nexmarkArgsProperty = "nexmark.args"
+
+task run (type: JavaExec) {
+ main = "org.apache.beam.sdk.nexmark.Main"
+ classpath = sourceSets.main.runtimeClasspath
Review comment:
Nice idea. Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 89632)
Time Spent: 2h 20m (was: 2h 10m)
> Launch Nexmark suites from gradle and update web page docs
> --
>
> Key: BEAM-3995
> URL: https://issues.apache.org/jira/browse/BEAM-3995
> Project: Beam
> Issue Type: Sub-task
> Components: examples-nexmark, website
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> Currently our instructions for running Nexmark benchmarks on various runners
> is pretty tightly tied to Maven. We need a good story for running them with
> gradle (or just building an executable with gradle and running that
> standalone).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[
https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=89113&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-89113
]
ASF GitHub Bot logged work on BEAM-3995:
Author: ASF GitHub Bot
Created on: 09/Apr/18 20:31
Start Date: 09/Apr/18 20:31
Worklog Time Spent: 10m
Work Description: lukecwik commented on a change in pull request #5051:
[BEAM-3995] Add basic DirectRunner Gradle run task to Nexmark
URL: https://github.com/apache/beam/pull/5051#discussion_r180218636
##
File path: sdks/java/nexmark/build.gradle
##
@@ -46,3 +46,14 @@ dependencies {
test {
jvmArgs "-da"
}
+
+def nexmarkArgsProperty = "nexmark.args"
+
+task run (type: JavaExec) {
+ main = "org.apache.beam.sdk.nexmark.Main"
+ classpath = sourceSets.main.runtimeClasspath
Review comment:
The reason why I suggested to run using the shaded artifacts is that it
would match what users would be running if they depended on the Apache Beam
nexmark artifact distributed via Maven central.
Also, if you want to integrate nexmark into the perf/integration testing it
would seem like you would want to run it using gradle and not have to build a
shell script to first build the bundled jar and then execute it using java.
Having a separate configuration per runner makes sense if a runner needs
additional overrides (typical of spark) to get it to work with Nexmark because
of a dependency conflict.
Configuration per runner with dynamic classifier?
```
configuration {
nexmarkRun
}
dependencies {
...
nexmarkRun project(name: project.path, configuration: 'shadow')
nexmarkRun project(name: runnerToUse, configuration: 'shadow')
}
task run(type: JavaExec) {
...
classpath = configurations.nexmarkRun
}
```
Run via `./gradlew -p sdks/java/nexmark run
-PrunnerToUse=':beam-runners-direct-java'`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 89113)
Time Spent: 2h (was: 1h 50m)
> Launch Nexmark suites from gradle and update web page docs
> --
>
> Key: BEAM-3995
> URL: https://issues.apache.org/jira/browse/BEAM-3995
> Project: Beam
> Issue Type: Sub-task
> Components: examples-nexmark, website
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Currently our instructions for running Nexmark benchmarks on various runners
> is pretty tightly tied to Maven. We need a good story for running them with
> gradle (or just building an executable with gradle and running that
> standalone).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[
https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=89112&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-89112
]
ASF GitHub Bot logged work on BEAM-3995:
Author: ASF GitHub Bot
Created on: 09/Apr/18 20:26
Start Date: 09/Apr/18 20:26
Worklog Time Spent: 10m
Work Description: lukecwik commented on a change in pull request #5051:
[BEAM-3995] Add basic DirectRunner Gradle run task to Nexmark
URL: https://github.com/apache/beam/pull/5051#discussion_r180219450
##
File path: sdks/java/nexmark/build.gradle
##
@@ -46,3 +46,14 @@ dependencies {
test {
jvmArgs "-da"
}
+
+def nexmarkArgsProperty = "nexmark.args"
+
+task run (type: JavaExec) {
Review comment:
I was under the impression that each query was its own executable.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 89112)
Time Spent: 1h 50m (was: 1h 40m)
> Launch Nexmark suites from gradle and update web page docs
> --
>
> Key: BEAM-3995
> URL: https://issues.apache.org/jira/browse/BEAM-3995
> Project: Beam
> Issue Type: Sub-task
> Components: examples-nexmark, website
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> Currently our instructions for running Nexmark benchmarks on various runners
> is pretty tightly tied to Maven. We need a good story for running them with
> gradle (or just building an executable with gradle and running that
> standalone).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[
https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=89108&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-89108
]
ASF GitHub Bot logged work on BEAM-3995:
Author: ASF GitHub Bot
Created on: 09/Apr/18 20:23
Start Date: 09/Apr/18 20:23
Worklog Time Spent: 10m
Work Description: lukecwik commented on a change in pull request #5051:
[BEAM-3995] Add basic DirectRunner Gradle run task to Nexmark
URL: https://github.com/apache/beam/pull/5051#discussion_r180218636
##
File path: sdks/java/nexmark/build.gradle
##
@@ -46,3 +46,14 @@ dependencies {
test {
jvmArgs "-da"
}
+
+def nexmarkArgsProperty = "nexmark.args"
+
+task run (type: JavaExec) {
+ main = "org.apache.beam.sdk.nexmark.Main"
+ classpath = sourceSets.main.runtimeClasspath
Review comment:
The reason why I suggested to run using the shaded artifacts is that it
would match what users would be running if they depended on the Apache Beam
nexmark artifact distributed via Maven central.
Having a separate configuration per runner makes sense if a runner needs
additional overrides (typical of spark) to get it to work with Nexmark because
of a dependency conflict.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 89108)
Time Spent: 1h 40m (was: 1.5h)
> Launch Nexmark suites from gradle and update web page docs
> --
>
> Key: BEAM-3995
> URL: https://issues.apache.org/jira/browse/BEAM-3995
> Project: Beam
> Issue Type: Sub-task
> Components: examples-nexmark, website
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> Currently our instructions for running Nexmark benchmarks on various runners
> is pretty tightly tied to Maven. We need a good story for running them with
> gradle (or just building an executable with gradle and running that
> standalone).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[
https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=89097&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-89097
]
ASF GitHub Bot logged work on BEAM-3995:
Author: ASF GitHub Bot
Created on: 09/Apr/18 20:10
Start Date: 09/Apr/18 20:10
Worklog Time Spent: 10m
Work Description: kennknowles commented on a change in pull request
#5051: [BEAM-3995] Add basic DirectRunner Gradle run task to Nexmark
URL: https://github.com/apache/beam/pull/5051#discussion_r180214420
##
File path: sdks/java/nexmark/build.gradle
##
@@ -46,3 +46,14 @@ dependencies {
test {
jvmArgs "-da"
}
+
+def nexmarkArgsProperty = "nexmark.args"
+
+task run (type: JavaExec) {
+ main = "org.apache.beam.sdk.nexmark.Main"
+ classpath = sourceSets.main.runtimeClasspath
Review comment:
Nexmark is not a test suite but a main application. Not sure if that changes
your comment.
I wanted to build a bundled artifact that included all transitive deps and
was standalone. Needs one per runner, which is unfortunate but a natural
consequence. The shaded configuration currently fails to include things like
joda time when I switch it to `compile` so I abandoned the effort. I will
resume it, actually, as I do like being able to run this way. Configuration per
runner with dynamic classifier?
Separately, what I am doing here is just using JavaExec to launch the main
class. It isn't a library but an application so shading should be a noop when
executed via JavaExec, is that not so? So I figure the non-bundled run should
be entirely unshaded to save needless build.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 89097)
Time Spent: 1h 20m (was: 1h 10m)
> Launch Nexmark suites from gradle and update web page docs
> --
>
> Key: BEAM-3995
> URL: https://issues.apache.org/jira/browse/BEAM-3995
> Project: Beam
> Issue Type: Sub-task
> Components: examples-nexmark, website
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Currently our instructions for running Nexmark benchmarks on various runners
> is pretty tightly tied to Maven. We need a good story for running them with
> gradle (or just building an executable with gradle and running that
> standalone).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[
https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=89098&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-89098
]
ASF GitHub Bot logged work on BEAM-3995:
Author: ASF GitHub Bot
Created on: 09/Apr/18 20:10
Start Date: 09/Apr/18 20:10
Worklog Time Spent: 10m
Work Description: kennknowles commented on a change in pull request
#5051: [BEAM-3995] Add basic DirectRunner Gradle run task to Nexmark
URL: https://github.com/apache/beam/pull/5051#discussion_r180213366
##
File path: sdks/java/nexmark/build.gradle
##
@@ -46,3 +46,14 @@ dependencies {
test {
jvmArgs "-da"
}
+
+def nexmarkArgsProperty = "nexmark.args"
+
+task run (type: JavaExec) {
Review comment:
Well, the reason I chose it is because `run` because Nexmark is an
application with one specific entry point. It is the same task that that
`application` plugin uses, and we are doing the same thing here. I decided that
plugin might not be the right fit, but if you feel differently I'll try it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 89098)
Time Spent: 1.5h (was: 1h 20m)
> Launch Nexmark suites from gradle and update web page docs
> --
>
> Key: BEAM-3995
> URL: https://issues.apache.org/jira/browse/BEAM-3995
> Project: Beam
> Issue Type: Sub-task
> Components: examples-nexmark, website
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Currently our instructions for running Nexmark benchmarks on various runners
> is pretty tightly tied to Maven. We need a good story for running them with
> gradle (or just building an executable with gradle and running that
> standalone).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[ https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=88919&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88919 ] ASF GitHub Bot logged work on BEAM-3995: Author: ASF GitHub Bot Created on: 09/Apr/18 13:11 Start Date: 09/Apr/18 13:11 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #5051: [BEAM-3995] Add basic DirectRunner Gradle run task to Nexmark URL: https://github.com/apache/beam/pull/5051#issuecomment-379746410 You can make the exec dynamic by making a new configuration which takes a project property that chooses which runner to include with the default being the direct runner. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 88919) Time Spent: 1h 10m (was: 1h) > Launch Nexmark suites from gradle and update web page docs > -- > > Key: BEAM-3995 > URL: https://issues.apache.org/jira/browse/BEAM-3995 > Project: Beam > Issue Type: Sub-task > Components: examples-nexmark, website >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 1h 10m > Remaining Estimate: 0h > > Currently our instructions for running Nexmark benchmarks on various runners > is pretty tightly tied to Maven. We need a good story for running them with > gradle (or just building an executable with gradle and running that > standalone). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[
https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=88916&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88916
]
ASF GitHub Bot logged work on BEAM-3995:
Author: ASF GitHub Bot
Created on: 09/Apr/18 13:09
Start Date: 09/Apr/18 13:09
Worklog Time Spent: 10m
Work Description: lukecwik commented on a change in pull request #5051:
[BEAM-3995] Add basic DirectRunner Gradle run task to Nexmark
URL: https://github.com/apache/beam/pull/5051#discussion_r180087769
##
File path: sdks/java/nexmark/build.gradle
##
@@ -46,3 +46,14 @@ dependencies {
test {
jvmArgs "-da"
}
+
+def nexmarkArgsProperty = "nexmark.args"
+
+task run (type: JavaExec) {
+ main = "org.apache.beam.sdk.nexmark.Main"
+ classpath = sourceSets.main.runtimeClasspath
Review comment:
This will run on the preshaded artifacts.
Create another configuration similar to the Flink validates runner which
contains nexmark `shadow` configuration.
https://github.com/apache/beam/blob/c2d674448611d0490804533b96c41ba05e071572/runners/flink/build.gradle#L81
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 88916)
Time Spent: 1h (was: 50m)
> Launch Nexmark suites from gradle and update web page docs
> --
>
> Key: BEAM-3995
> URL: https://issues.apache.org/jira/browse/BEAM-3995
> Project: Beam
> Issue Type: Sub-task
> Components: examples-nexmark, website
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Currently our instructions for running Nexmark benchmarks on various runners
> is pretty tightly tied to Maven. We need a good story for running them with
> gradle (or just building an executable with gradle and running that
> standalone).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[
https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=88917&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88917
]
ASF GitHub Bot logged work on BEAM-3995:
Author: ASF GitHub Bot
Created on: 09/Apr/18 13:09
Start Date: 09/Apr/18 13:09
Worklog Time Spent: 10m
Work Description: lukecwik commented on a change in pull request #5051:
[BEAM-3995] Add basic DirectRunner Gradle run task to Nexmark
URL: https://github.com/apache/beam/pull/5051#discussion_r180088173
##
File path: sdks/java/nexmark/build.gradle
##
@@ -46,3 +46,14 @@ dependencies {
test {
jvmArgs "-da"
}
+
+def nexmarkArgsProperty = "nexmark.args"
+
+task run (type: JavaExec) {
Review comment:
Is there a more meaningful name you could provide instead of `run`?
nit: `run (` -> `run(` to follow the same style used in all the other build
files and Java
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 88917)
Time Spent: 1h (was: 50m)
> Launch Nexmark suites from gradle and update web page docs
> --
>
> Key: BEAM-3995
> URL: https://issues.apache.org/jira/browse/BEAM-3995
> Project: Beam
> Issue Type: Sub-task
> Components: examples-nexmark, website
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Currently our instructions for running Nexmark benchmarks on various runners
> is pretty tightly tied to Maven. We need a good story for running them with
> gradle (or just building an executable with gradle and running that
> standalone).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[ https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=88649&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88649 ] ASF GitHub Bot logged work on BEAM-3995: Author: ASF GitHub Bot Created on: 06/Apr/18 23:48 Start Date: 06/Apr/18 23:48 Worklog Time Spent: 10m Work Description: kennknowles commented on issue #5051: [BEAM-3995] Add basic DirectRunner Gradle run task to Nexmark URL: https://github.com/apache/beam/pull/5051#issuecomment-379413203 I've gone through a few iterations. First, hardcoded some configs. Second, building a fat jar - I still like it best but something about our shadow config is already screwed up ;_;. Last, using JavaExec to do like Maven `exec:java`. You have to plumb the args as a string then split but otherwise the same. Not as bad as I was worried about. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 88649) Time Spent: 50m (was: 40m) > Launch Nexmark suites from gradle and update web page docs > -- > > Key: BEAM-3995 > URL: https://issues.apache.org/jira/browse/BEAM-3995 > Project: Beam > Issue Type: Sub-task > Components: examples-nexmark, website >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > > Currently our instructions for running Nexmark benchmarks on various runners > is pretty tightly tied to Maven. We need a good story for running them with > gradle (or just building an executable with gradle and running that > standalone). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[
https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=87813&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-87813
]
ASF GitHub Bot logged work on BEAM-3995:
Author: ASF GitHub Bot
Created on: 04/Apr/18 23:18
Start Date: 04/Apr/18 23:18
Worklog Time Spent: 10m
Work Description: kennknowles closed pull request #5026: [BEAM-3995]
Build self-contained Nexmark jar
URL: https://github.com/apache/beam/pull/5026
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/sdks/java/nexmark/build.gradle b/sdks/java/nexmark/build.gradle
index 078365df375..b2f8c3e4552 100644
--- a/sdks/java/nexmark/build.gradle
+++ b/sdks/java/nexmark/build.gradle
@@ -23,26 +23,32 @@ description = "Apache Beam :: SDKs :: Java :: Nexmark"
dependencies {
compile library.java.guava
- shadow project(path: ":sdks:java:core", configuration: "shadow")
- shadow project(path: ":sdks:java:io:google-cloud-platform", configuration:
"shadow")
- shadow project(path: ":sdks:java:extensions:google-cloud-platform-core",
configuration: "shadow")
- shadow project(path: ":sdks:java:extensions:sql", configuration: "shadow")
- shadow library.java.google_api_services_bigquery
- shadow library.java.jackson_core
- shadow library.java.jackson_annotations
- shadow library.java.jackson_databind
- shadow library.java.avro
- shadow library.java.joda_time
- shadow library.java.slf4j_api
- shadow library.java.findbugs_jsr305
- shadow library.java.junit
- shadow library.java.hamcrest_core
- shadow library.java.commons_lang3
- shadow project(path: ":runners:direct-java", configuration: "shadow")
- shadow library.java.slf4j_jdk14
+ compile project(path: ":sdks:java:core")
+ compile project(path: ":sdks:java:io:google-cloud-platform")
+ compile project(path: ":sdks:java:extensions:google-cloud-platform-core")
+ compile project(path: ":sdks:java:extensions:sql")
+ compile library.java.google_api_services_bigquery
+ compile library.java.jackson_core
+ compile library.java.jackson_annotations
+ compile library.java.jackson_databind
+ compile library.java.avro
+ compile library.java.joda_time
+ compile library.java.slf4j_api
+ compile library.java.findbugs_jsr305
+ compile library.java.junit
+ compile library.java.hamcrest_core
+ compile library.java.commons_lang3
+ compile project(path: ":runners:direct-java")
+ runtime library.java.slf4j_jdk14
testCompile library.java.hamcrest_core
}
test {
jvmArgs "-da"
}
+
+jar {
+ manifest {
+attributes 'Main-Class': 'org.apache.beam.sdk.nexmark.Main'
+ }
+}
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 87813)
Time Spent: 40m (was: 0.5h)
> Launch Nexmark suites from gradle and update web page docs
> --
>
> Key: BEAM-3995
> URL: https://issues.apache.org/jira/browse/BEAM-3995
> Project: Beam
> Issue Type: Sub-task
> Components: examples-nexmark, website
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Currently our instructions for running Nexmark benchmarks on various runners
> is pretty tightly tied to Maven. We need a good story for running them with
> gradle (or just building an executable with gradle and running that
> standalone).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[ https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=87812&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-87812 ] ASF GitHub Bot logged work on BEAM-3995: Author: ASF GitHub Bot Created on: 04/Apr/18 23:18 Start Date: 04/Apr/18 23:18 Worklog Time Spent: 10m Work Description: kennknowles commented on issue #5026: [BEAM-3995] Build self-contained Nexmark jar URL: https://github.com/apache/beam/pull/5026#issuecomment-378774775 Ah, I think this is really not going to be workable. String munging to plumb cmdline args may still be what we need to do, in 2018. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 87812) Time Spent: 0.5h (was: 20m) > Launch Nexmark suites from gradle and update web page docs > -- > > Key: BEAM-3995 > URL: https://issues.apache.org/jira/browse/BEAM-3995 > Project: Beam > Issue Type: Sub-task > Components: examples-nexmark, website >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > Currently our instructions for running Nexmark benchmarks on various runners > is pretty tightly tied to Maven. We need a good story for running them with > gradle (or just building an executable with gradle and running that > standalone). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[ https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=87810&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-87810 ] ASF GitHub Bot logged work on BEAM-3995: Author: ASF GitHub Bot Created on: 04/Apr/18 23:14 Start Date: 04/Apr/18 23:14 Worklog Time Spent: 10m Work Description: kennknowles commented on issue #5026: [BEAM-3995] Build self-contained Nexmark jar URL: https://github.com/apache/beam/pull/5026#issuecomment-378774094 It is a big hammer; too big. We'll actually need to have a way to get the different runners on the classpath without bundling. But Gradle has nothing that I would care to use equivalent to `mvn exec:java`. Still poking around but the solutions I've found are not reasonable. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 87810) Time Spent: 20m (was: 10m) > Launch Nexmark suites from gradle and update web page docs > -- > > Key: BEAM-3995 > URL: https://issues.apache.org/jira/browse/BEAM-3995 > Project: Beam > Issue Type: Sub-task > Components: examples-nexmark, website >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > Currently our instructions for running Nexmark benchmarks on various runners > is pretty tightly tied to Maven. We need a good story for running them with > gradle (or just building an executable with gradle and running that > standalone). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs
[ https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=87805&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-87805 ] ASF GitHub Bot logged work on BEAM-3995: Author: ASF GitHub Bot Created on: 04/Apr/18 23:12 Start Date: 04/Apr/18 23:12 Worklog Time Spent: 10m Work Description: kennknowles opened a new pull request #5026: [BEAM-3995] Build self-contained Nexmark jar URL: https://github.com/apache/beam/pull/5026 To run the Nexmark benchmarks, we will separate the build from the run step. The build step is owned by Gradle, while the run step is simply running a program. Nexmark is already designed this way. WIP: the expected full set of dependencies is _not_ contained in the shadow jar. I don't know why this is yet, but expect it has to do with how we manage the base setup. TBD. Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Make sure there is a [JIRA issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the change (usually before you start working on it). Trivial changes like typos do not require a JIRA issue. Your pull request should address just this issue, without pulling in other changes. - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue. - [ ] Write a pull request description that is detailed enough to understand: - [ ] What the pull request does - [ ] Why it does it - [ ] How it does it - [ ] Why this approach - [ ] Each commit in the pull request should have a meaningful subject line and body. - [ ] Run `mvn clean verify` to make sure basic checks pass. A more thorough check will be performed on your pull request automatically. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 87805) Time Spent: 10m Remaining Estimate: 0h > Launch Nexmark suites from gradle and update web page docs > -- > > Key: BEAM-3995 > URL: https://issues.apache.org/jira/browse/BEAM-3995 > Project: Beam > Issue Type: Sub-task > Components: examples-nexmark, website >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Currently our instructions for running Nexmark benchmarks on various runners > is pretty tightly tied to Maven. We need a good story for running them with > gradle (or just building an executable with gradle and running that > standalone). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
