[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/552#issuecomment-42520751 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14808/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/552#issuecomment-42520752 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14809/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/552#issuecomment-42411316 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14771/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/552#issuecomment-42518042 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/552#discussion_r12388585
--- Diff:
examples/src/main/scala/org/apache/spark/examples/graphx/LiveJournalPageRank.scala
---
@@ -44,6 +44,6 @@ object LiveJournalPageRank {
System.exit(-1)
--- End diff --
Ok, it's just inconsistent...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/552#issuecomment-42518033 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/552#discussion_r12389059
--- Diff: graphx/src/main/scala/org/apache/spark/graphx/lib/Analytics.scala
---
@@ -27,10 +27,9 @@ import org.apache.spark.graphx.PartitionStrategy._
object Analytics extends Logging {
def main(args: Array[String]): Unit = {
--- End diff --
I think we should still, though not copy and pasting, because all the
options after the file name are specific to the application (e.g. page rank).
This still leaves the first argument undocumented. I would do something like
```
Usage: Analytics [other options]
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/552#issuecomment-42412487 Build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/552#issuecomment-42401923 Build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/552#issuecomment-42417263 Build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/552#issuecomment-42417264 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14772/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/552#issuecomment-42576140 This LGTM. Thanks @ScrapCodes for all the effort! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/552#discussion_r12389167
--- Diff:
examples/src/main/java/org/apache/spark/examples/JavaLogQuery.java ---
@@ -98,15 +99,11 @@ public static Stats extractStats(String line) {
}
public static void main(String[] args) {
-if (args.length == 0) {
- System.err.println("Usage: JavaLogQuery [logFile]");
--- End diff --
Great, thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/552#issuecomment-42520747 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/552#issuecomment-42578185 Thanks @ScrapCodes - sorry you had to up-merge this... good stuff :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/552#issuecomment-42412500 Build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/552#issuecomment-42402362 Build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/552#issuecomment-42517527 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user ScrapCodes commented on a diff in the pull request:
https://github.com/apache/spark/pull/552#discussion_r12370101
--- Diff:
examples/src/main/java/org/apache/spark/examples/JavaLogQuery.java ---
@@ -98,15 +99,11 @@ public static Stats extractStats(String line) {
}
public static void main(String[] args) {
-if (args.length == 0) {
- System.err.println("Usage: JavaLogQuery [logFile]");
--- End diff --
added a comment since it was just optional
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/552#issuecomment-42450961 @ScrapCodes you'll need to merge this with master - unfortunately there was another patch that renamed/moved some of the example files. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/552 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user ScrapCodes commented on the pull request: https://github.com/apache/spark/pull/552#issuecomment-42527532 @pwendell Done ! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/552#issuecomment-42517525 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user ScrapCodes commented on a diff in the pull request:
https://github.com/apache/spark/pull/552#discussion_r12416819
--- Diff: graphx/src/main/scala/org/apache/spark/graphx/lib/Analytics.scala
---
@@ -27,10 +27,9 @@ import org.apache.spark.graphx.PartitionStrategy._
object Analytics extends Logging {
def main(args: Array[String]): Unit = {
--- End diff --
Great done !
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/552#issuecomment-42520748 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/552#issuecomment-42401917 Build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/552#issuecomment-42405191 Build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/552#issuecomment-42402364 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14765/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/552#issuecomment-42405184 Build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/552#issuecomment-42411313 Build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user ScrapCodes commented on a diff in the pull request:
https://github.com/apache/spark/pull/552#discussion_r12365591
--- Diff: graphx/src/main/scala/org/apache/spark/graphx/lib/Analytics.scala
---
@@ -27,10 +27,9 @@ import org.apache.spark.graphx.PartitionStrategy._
object Analytics extends Logging {
def main(args: Array[String]): Unit = {
--- End diff --
It is supposed to be *only* used by `LiveJournalPageRank`. Which has the
right usage, copy pasting it here as well did not feel useful ? Should I still
do it ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/552#issuecomment-42373182 Hi @ScrapCodes, I left more comments. The biggest thing is that the usage seems to be deleted from many files, but we still want them around at least in the comments as you have done in a few places. (I stopped pointing this out in my comments half way through but you should verify this for all the files you changed). The other thing is that I'm not sure what `[]` vs `[slices]` mean. If they both mean optional arguments, then I think it's more consistent to just use the latter. This isn't introduced in this PR, but it would be good if you could fix this. Thanks again for doing all this tedious but important work. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/552#discussion_r12355586
--- Diff:
examples/src/main/scala/org/apache/spark/examples/graphx/LiveJournalPageRank.scala
---
@@ -44,6 +44,6 @@ object LiveJournalPageRank {
System.exit(-1)
--- End diff --
@andrewor14 I think any non zero value here is fine
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/552#discussion_r12355513
--- Diff: graphx/src/main/scala/org/apache/spark/graphx/lib/Analytics.scala
---
@@ -27,10 +27,9 @@ import org.apache.spark.graphx.PartitionStrategy._
object Analytics extends Logging {
def main(args: Array[String]): Unit = {
--- End diff --
Not your changes, but could you add the usage here? It looks like it was
already missing before your patch.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/552#discussion_r12355467
--- Diff:
examples/src/main/scala/org/apache/spark/streaming/examples/StatefulNetworkWordCount.scala
---
@@ -17,29 +17,28 @@
package org.apache.spark.streaming.examples
+import org.apache.spark.SparkConf
import org.apache.spark.streaming._
import org.apache.spark.streaming.StreamingContext._
-// scalastyle:off
+
/**
* Counts words cumulatively in UTF8 encoded, '\n' delimited text received
from the network every
* second.
- * Usage: StatefulNetworkWordCount
- *is the Spark master URL. In local mode, should be
'local[n]' with n > 1.
+ * Usage: StatefulNetworkWordCount
*and describe the TCP server that Spark Streaming
would connect to receive
* data.
*
* To run this on your local machine, you need to first run a Netcat server
*`$ nc -lk `
* and then run the example
- *`$ ./bin/run-example
org.apache.spark.streaming.examples.StatefulNetworkWordCount local[2] localhost
`
+ *`$ ./bin/spark-submit examples.jar
+ *--class org.apache.spark.streaming.examples.StatefulNetworkWordCount
localhost `
*/
-// scalastyle:on
object StatefulNetworkWordCount {
def main(args: Array[String]) {
-if (args.length < 3) {
- System.err.println("Usage: StatefulNetworkWordCount
\n" +
-"In local mode, should be 'local[n]' with n > 1")
- System.exit(1)
+if (args.length < 2) {
+ System.err.println("Usage: StatefulNetworkWordCount
")
+System.exit(1)
--- End diff --
nit: indentation is off
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/552#discussion_r12355400
--- Diff:
examples/src/main/scala/org/apache/spark/streaming/examples/NetworkWordCount.scala
---
@@ -17,41 +17,39 @@
package org.apache.spark.streaming.examples
+import org.apache.spark.SparkConf
import org.apache.spark.streaming.{Seconds, StreamingContext}
import org.apache.spark.streaming.StreamingContext._
import org.apache.spark.storage.StorageLevel
-// scalastyle:off
/**
* Counts words in text encoded with UTF8 received from the network every
second.
*
- * Usage: NetworkWordCount
- *is the Spark master URL. In local mode, should be
'local[n]' with n > 1.
- *and describe the TCP server that Spark Streaming
would connect to receive data.
+ * Usage: NetworkWordCount
+ *and describe the TCP server that Spark Streaming
would connect to receive
+ *data.
--- End diff --
nit: move data back on previous line? I don't think this exceeds the 100
char limit
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/552#discussion_r12355330
--- Diff:
examples/src/main/scala/org/apache/spark/examples/bagel/WikipediaPageRank.scala
---
@@ -32,22 +32,22 @@ import scala.xml.{XML,NodeSeq}
*/
object WikipediaPageRank {
def main(args: Array[String]) {
-if (args.length < 5) {
+if (args.length < 4) {
System.err.println(
-"Usage: WikipediaPageRank
")
+"Usage: WikipediaPageRank
")
System.exit(-1)
--- End diff --
Not your change, but this should probably be `System.exit(1)` to be
consistent with other files
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/552#discussion_r12355312
--- Diff:
examples/src/main/scala/org/apache/spark/examples/graphx/LiveJournalPageRank.scala
---
@@ -44,6 +44,6 @@ object LiveJournalPageRank {
System.exit(-1)
--- End diff --
`System.exit(1)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/552#discussion_r12355319
--- Diff:
examples/src/main/scala/org/apache/spark/examples/bagel/WikipediaPageRankStandalone.scala
---
@@ -30,22 +30,20 @@ import org.apache.spark.rdd.RDD
object WikipediaPageRankStandalone {
def main(args: Array[String]) {
-if (args.length < 5) {
+if (args.length < 4) {
System.err.println("Usage: WikipediaPageRankStandalone
" +
-" ")
+" ")
System.exit(-1)
--- End diff --
`System.exit(1)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/552#discussion_r12355253
--- Diff:
examples/src/main/scala/org/apache/spark/examples/bagel/WikipediaPageRank.scala
---
@@ -32,22 +32,22 @@ import scala.xml.{XML,NodeSeq}
*/
object WikipediaPageRank {
def main(args: Array[String]) {
-if (args.length < 5) {
+if (args.length < 4) {
System.err.println(
-"Usage: WikipediaPageRank
")
+"Usage: WikipediaPageRank
")
System.exit(-1)
--- End diff --
not your change, but this should probably be `System.exit(1)` to be
consistent with the rest
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/552#discussion_r12354679
--- Diff: examples/src/main/scala/org/apache/spark/examples/SparkLR.scala
---
@@ -47,13 +47,9 @@ object SparkLR {
}
def main(args: Array[String]) {
-if (args.length == 0) {
- System.err.println("Usage: SparkLR []")
--- End diff --
add back usage
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/552#discussion_r12354655
--- Diff:
examples/src/main/scala/org/apache/spark/examples/SparkHdfsLR.scala ---
@@ -49,20 +49,16 @@ object SparkHdfsLR {
}
def main(args: Array[String]) {
-if (args.length < 3) {
- System.err.println("Usage: SparkHdfsLR ")
--- End diff --
same here, add back usage
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/552#discussion_r12354668
--- Diff:
examples/src/main/scala/org/apache/spark/examples/SparkKMeans.scala ---
@@ -53,15 +53,15 @@ object SparkKMeans {
def main(args: Array[String]) {
if (args.length < 4) {
--- End diff --
this is no longer 4 right? Should be 3?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/552#discussion_r12354631
--- Diff: examples/src/main/scala/org/apache/spark/examples/SparkALS.scala
---
@@ -88,32 +88,24 @@ object SparkALS {
}
def main(args: Array[String]) {
-if (args.length == 0) {
- System.err.println("Usage: SparkALS [
]")
- System.exit(1)
-}
-
-var host = ""
var slices = 0
-val options = (0 to 5).map(i => if (i < args.length) Some(args(i))
else None)
+val options = (0 to 4).map(i => if (i < args.length) Some(args(i))
else None)
options.toArray match {
- case Array(host_, m, u, f, iters, slices_) =>
-host = host_.get
+ case Array(m, u, f, iters, slices_) =>
M = m.getOrElse("100").toInt
U = u.getOrElse("500").toInt
F = f.getOrElse("10").toInt
ITERATIONS = iters.getOrElse("5").toInt
slices = slices_.getOrElse("2").toInt
case _ =>
-System.err.println("Usage: SparkALS [
]")
+System.err.println("Usage: Please specify [
]")
--- End diff --
I would just do `Usage: SparkALS [M] [U] [F] ...`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/552#discussion_r12354583
--- Diff:
examples/src/main/scala/org/apache/spark/examples/MultiBroadcastTest.scala ---
@@ -18,20 +18,19 @@
package org.apache.spark.examples
import org.apache.spark.rdd.RDD
-import org.apache.spark.SparkContext
+import org.apache.spark.{SparkConf, SparkContext}
+/**
+ * Usage: MultiBroadcastTest [] [numElem]
--- End diff --
`[]` -> `[slices]`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/552#discussion_r12354566
--- Diff: examples/src/main/scala/org/apache/spark/examples/LogQuery.scala
---
@@ -40,16 +40,12 @@ object LogQuery {
)
def main(args: Array[String]) {
-if (args.length == 0) {
- System.err.println("Usage: LogQuery [logFile]")
--- End diff --
Same here, need doc usage
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/552#discussion_r12354495
--- Diff:
examples/src/main/java/org/apache/spark/streaming/examples/JavaQueueStream.java
---
@@ -17,8 +17,15 @@
package org.apache.spark.streaming.examples;
-import com.google.common.collect.Lists;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Queue;
--- End diff --
nit: you could just group them `import java.util.{LinkedList, List, Queue}`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/552#discussion_r12354345
--- Diff: examples/src/main/java/org/apache/spark/examples/JavaTC.java ---
@@ -61,14 +62,9 @@
}
public static void main(String[] args) {
-if (args.length == 0) {
- System.err.println("Usage: JavaTC []");
--- End diff --
Same here
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/552#discussion_r12354371
--- Diff: examples/src/main/java/org/apache/spark/examples/JavaTC.java ---
@@ -61,14 +62,9 @@
}
public static void main(String[] args) {
-if (args.length == 0) {
- System.err.println("Usage: JavaTC []");
--- End diff --
Also what does `[]` mean? If it means it's optional, I think it's
better if it's just `[slices]`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/552#discussion_r12354332
--- Diff: examples/src/main/java/org/apache/spark/examples/JavaSparkPi.java
---
@@ -29,15 +30,10 @@
public final class JavaSparkPi {
public static void main(String[] args) throws Exception {
-if (args.length == 0) {
- System.err.println("Usage: JavaSparkPi [slices]");
--- End diff --
Same here, add back usage as a doc comment.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/552#discussion_r12354302 --- Diff: examples/src/main/java/org/apache/spark/examples/JavaPageRank.java --- @@ -18,6 +18,7 @@ package org.apache.spark.examples; +import org.apache.spark.SparkConf; --- End diff -- nit: move this down --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/552#discussion_r12354289
--- Diff:
examples/src/main/java/org/apache/spark/examples/JavaLogQuery.java ---
@@ -98,15 +99,11 @@ public static Stats extractStats(String line) {
}
public static void main(String[] args) {
-if (args.length == 0) {
- System.err.println("Usage: JavaLogQuery [logFile]");
--- End diff --
You deleted the usage here. Could you add it back as a comment somewhere
else? Right now there's no way to tell what the first parameter should be
unless we dig into the code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/552#discussion_r12353063
--- Diff: examples/src/main/java/org/apache/spark/examples/JavaTC.java ---
@@ -61,14 +62,9 @@
}
public static void main(String[] args) {
-if (args.length == 0) {
- System.err.println("Usage: JavaTC []");
- System.exit(1);
-}
-
-JavaSparkContext sc = new JavaSparkContext(args[0], "JavaTC",
-System.getenv("SPARK_HOME"),
JavaSparkContext.jarOfClass(JavaTC.class));
-Integer slices = (args.length > 1) ? Integer.parseInt(args[1]): 2;
+SparkConf sparkConf = new
org.apache.spark.SparkConf().setAppName("JavaHdfsLR");
+JavaSparkContext sc = new JavaSparkContext(sparkConf);
+Integer slices = (args.length > 0) ? Integer.parseInt(args[0]): 2;
--- End diff --
Yeah, in scala we have no space before colon, but this is a common pattern
in Java (e.g. `bool ? 1 : 2` is short-hand for `if (bool) { 1 } else { 2 }`).
We do this elsewhere in other examples, actually.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user mateiz commented on a diff in the pull request:
https://github.com/apache/spark/pull/552#discussion_r12352647
--- Diff: examples/src/main/java/org/apache/spark/examples/JavaTC.java ---
@@ -61,14 +62,9 @@
}
public static void main(String[] args) {
-if (args.length == 0) {
- System.err.println("Usage: JavaTC []");
- System.exit(1);
-}
-
-JavaSparkContext sc = new JavaSparkContext(args[0], "JavaTC",
-System.getenv("SPARK_HOME"),
JavaSparkContext.jarOfClass(JavaTC.class));
-Integer slices = (args.length > 1) ? Integer.parseInt(args[1]): 2;
+SparkConf sparkConf = new
org.apache.spark.SparkConf().setAppName("JavaHdfsLR");
+JavaSparkContext sc = new JavaSparkContext(sparkConf);
+Integer slices = (args.length > 0) ? Integer.parseInt(args[0]): 2;
--- End diff --
Also, don't put the full package name (org.apache.spark.SparkConf) here
since you imported it above
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/552#issuecomment-42308493 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14717/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/552#issuecomment-42308490 Build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/552#issuecomment-42306396 Build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/552#issuecomment-42306397 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14716/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/552#issuecomment-42303614 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14715/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/552#issuecomment-42303612 Build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/552#issuecomment-42300383 Build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/552#issuecomment-42300392 Build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/552#issuecomment-42298879 Build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/552#issuecomment-42298871 Build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/552#issuecomment-42297055 Build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user ScrapCodes commented on the pull request: https://github.com/apache/spark/pull/552#issuecomment-42297072 @andrewor14 Thanks for taking a look ! I have fixed those, and at some places those args were optional arguments for which it may be not logical to have args.length check. So I have just documented them. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/552#issuecomment-42297067 Build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user ScrapCodes commented on a diff in the pull request:
https://github.com/apache/spark/pull/552#discussion_r12320940
--- Diff:
examples/src/main/scala/org/apache/spark/examples/MultiBroadcastTest.scala ---
@@ -18,20 +18,16 @@
package org.apache.spark.examples
import org.apache.spark.rdd.RDD
-import org.apache.spark.SparkContext
+import org.apache.spark.{SparkConf, SparkContext}
object MultiBroadcastTest {
def main(args: Array[String]) {
-if (args.length == 0) {
- System.err.println("Usage: MultiBroadcastTest []
[numElem]")
--- End diff --
May be I could just document it ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user ScrapCodes commented on a diff in the pull request:
https://github.com/apache/spark/pull/552#discussion_r12320653
--- Diff:
examples/src/main/scala/org/apache/spark/examples/GroupByTest.scala ---
@@ -19,24 +19,18 @@ package org.apache.spark.examples
import java.util.Random
-import org.apache.spark.SparkContext
+import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.SparkContext._
object GroupByTest {
def main(args: Array[String]) {
-if (args.length == 0) {
- System.err.println(
-"Usage: GroupByTest [numMappers] [numKVPairs] [KeySize]
[numReducers]")
- System.exit(1)
-}
-
-var numMappers = if (args.length > 1) args(1).toInt else 2
-var numKVPairs = if (args.length > 2) args(2).toInt else 1000
-var valSize = if (args.length > 3) args(3).toInt else 1000
-var numReducers = if (args.length > 4) args(4).toInt else numMappers
-
-val sc = new SparkContext(args(0), "GroupBy Test",
- System.getenv("SPARK_HOME"),
SparkContext.jarOfClass(this.getClass).toSeq)
+val sparkConf = new SparkConf().setAppName("GroupBy Test")
+var numMappers = if (args.length > 0) args(0).toInt else 2
--- End diff --
Here as well !
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user ScrapCodes commented on a diff in the pull request:
https://github.com/apache/spark/pull/552#discussion_r12320627
--- Diff:
examples/src/main/scala/org/apache/spark/examples/BroadcastTest.scala ---
@@ -27,18 +27,18 @@ object BroadcastTest {
System.exit(1)
}
-val bcName = if (args.length > 3) args(3) else "Http"
-val blockSize = if (args.length > 4) args(4) else "4096"
+val bcName = if (args.length > 2) args(2) else "Http"
--- End diff --
Yes, and then all arguments will be optional. So may be args.length check
can be completely removed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user ScrapCodes commented on a diff in the pull request:
https://github.com/apache/spark/pull/552#discussion_r12320290
--- Diff: examples/src/main/java/org/apache/spark/examples/JavaTC.java ---
@@ -61,14 +62,9 @@
}
public static void main(String[] args) {
-if (args.length == 0) {
- System.err.println("Usage: JavaTC []");
- System.exit(1);
-}
-
-JavaSparkContext sc = new JavaSparkContext(args[0], "JavaTC",
-System.getenv("SPARK_HOME"),
JavaSparkContext.jarOfClass(JavaTC.class));
-Integer slices = (args.length > 1) ? Integer.parseInt(args[1]): 2;
+SparkConf sparkConf = new
org.apache.spark.SparkConf().setAppName("JavaHdfsLR");
+JavaSparkContext sc = new JavaSparkContext(sparkConf);
+Integer slices = (args.length > 0) ? Integer.parseInt(args[0]): 2;
--- End diff --
Do we leave space before colon ? I think the convention was nospace before
colon and single space after it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/552#issuecomment-42252475 @ScrapCodes I left a bunch of small comments. In general, we want to keep the usage around, but update it to exclude any mention of `` or ``. Also, in a few places you removed the check for `args.length`, and directly access `args(n)` for some value of `n`. This will cause IndexOutOfBoundsException, which we try to avoid. Can you fix these? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/552#discussion_r12305092
--- Diff:
examples/src/main/scala/org/apache/spark/examples/MultiBroadcastTest.scala ---
@@ -18,20 +18,16 @@
package org.apache.spark.examples
import org.apache.spark.rdd.RDD
-import org.apache.spark.SparkContext
+import org.apache.spark.{SparkConf, SparkContext}
object MultiBroadcastTest {
def main(args: Array[String]) {
-if (args.length == 0) {
- System.err.println("Usage: MultiBroadcastTest []
[numElem]")
--- End diff --
You should really keep the usage here; now this is not documented at all
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/552#discussion_r12305049
--- Diff: examples/src/main/scala/org/apache/spark/examples/LocalALS.scala
---
@@ -116,7 +116,7 @@ object LocalALS {
ITERATIONS = iters.toInt
}
case _ => {
-System.err.println("Usage: LocalALS")
+System.err.println("Please specify")
--- End diff --
Why this change? The old one is more descriptive...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/552#discussion_r12305036
--- Diff: examples/src/main/scala/org/apache/spark/examples/HBaseTest.scala
---
@@ -26,8 +26,9 @@ import org.apache.spark.rdd.NewHadoopRDD
object HBaseTest {
def main(args: Array[String]) {
-val sc = new SparkContext(args(0), "HBaseTest",
- System.getenv("SPARK_HOME"),
SparkContext.jarOfClass(this.getClass).toSeq)
+val sparkConf = new SparkConf().setAppName("HBaseTest")
+
--- End diff --
nit: no need for new line here, and in L32
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/552#discussion_r12305015
--- Diff:
examples/src/main/scala/org/apache/spark/examples/GroupByTest.scala ---
@@ -19,24 +19,18 @@ package org.apache.spark.examples
import java.util.Random
-import org.apache.spark.SparkContext
+import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.SparkContext._
object GroupByTest {
def main(args: Array[String]) {
-if (args.length == 0) {
- System.err.println(
-"Usage: GroupByTest [numMappers] [numKVPairs] [KeySize]
[numReducers]")
- System.exit(1)
-}
-
-var numMappers = if (args.length > 1) args(1).toInt else 2
-var numKVPairs = if (args.length > 2) args(2).toInt else 1000
-var valSize = if (args.length > 3) args(3).toInt else 1000
-var numReducers = if (args.length > 4) args(4).toInt else numMappers
-
-val sc = new SparkContext(args(0), "GroupBy Test",
- System.getenv("SPARK_HOME"),
SparkContext.jarOfClass(this.getClass).toSeq)
+val sparkConf = new SparkConf().setAppName("GroupBy Test")
+var numMappers = if (args.length > 0) args(0).toInt else 2
--- End diff --
You should keep the `args.length` check, as in other tests
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/552#discussion_r12304982
--- Diff:
examples/src/main/scala/org/apache/spark/examples/CassandraTest.scala ---
@@ -54,9 +54,9 @@ import org.apache.spark.SparkContext._
object CassandraTest {
def main(args: Array[String]) {
-
+val sparkConf = new SparkConf().setAppName("casDemo")
--- End diff --
Same in L41 and L44
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/552#discussion_r12304965
--- Diff:
examples/src/main/scala/org/apache/spark/examples/CassandraCQLTest.scala ---
@@ -72,12 +72,11 @@ import org.apache.spark.SparkContext._
object CassandraCQLTest {
def main(args: Array[String]) {
-val sc = new SparkContext(args(0),
--- End diff --
Same here in L68 and L69; the `` and `local[2]` shouldn't be
there
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/552#discussion_r12304931
--- Diff:
examples/src/main/scala/org/apache/spark/examples/BroadcastTest.scala ---
@@ -27,18 +27,18 @@ object BroadcastTest {
System.exit(1)
}
-val bcName = if (args.length > 3) args(3) else "Http"
-val blockSize = if (args.length > 4) args(4) else "4096"
+val bcName = if (args.length > 2) args(2) else "Http"
--- End diff --
Did you forget to update the usage in L25? (Github wouldn't let me comment
up there) The argument indices don't match up; right now, args(2) is "numElem",
not "broadcastAlgo"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/552#discussion_r12304842 --- Diff: examples/src/main/java/org/apache/spark/streaming/examples/JavaQueueStream.java --- @@ -18,6 +18,7 @@ package org.apache.spark.streaming.examples; import com.google.common.collect.Lists; +import org.apache.spark.SparkConf; --- End diff -- nit: group with other `org.apache.spark.*` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/552#discussion_r12304796
--- Diff:
examples/src/main/java/org/apache/spark/streaming/examples/JavaKafkaWordCount.java
---
@@ -54,26 +56,24 @@ private JavaKafkaWordCount() {
}
public static void main(String[] args) {
-if (args.length < 5) {
- System.err.println("Usage: KafkaWordCount
");
+if (args.length < 3) {
+ System.err.println("Usage: JavaKafkaWordCount
");
--- End diff --
miscount? We have 4 arguments
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/552#discussion_r12304762 --- Diff: examples/src/main/java/org/apache/spark/streaming/examples/JavaKafkaWordCount.java --- @@ -21,6 +21,9 @@ import java.util.HashMap; import java.util.regex.Pattern; +import org.apache.spark.SparkConf; --- End diff -- nit: group this with other `org.apache.spark.*` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/552#discussion_r12304730
--- Diff:
examples/src/main/java/org/apache/spark/mllib/examples/JavaLR.java ---
@@ -51,17 +52,12 @@ public LabeledPoint call(String line) {
}
public static void main(String[] args) {
-if (args.length != 4) {
- System.err.println("Usage: JavaLR
");
- System.exit(1);
-}
-
-JavaSparkContext sc = new JavaSparkContext(args[0], "JavaLR",
-System.getenv("SPARK_HOME"),
JavaSparkContext.jarOfClass(JavaLR.class));
-JavaRDD lines = sc.textFile(args[1]);
+SparkConf sparkConf = new SparkConf().setAppName("JavaLR");
+JavaSparkContext sc = new JavaSparkContext(sparkConf);
+JavaRDD lines = sc.textFile(args[0]);
--- End diff --
same here
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/552#discussion_r12304704
--- Diff:
examples/src/main/java/org/apache/spark/mllib/examples/JavaKMeans.java ---
@@ -49,23 +50,16 @@ public Vector call(String line) {
public static void main(String[] args) {
-if (args.length < 4) {
- System.err.println(
- "Usage: JavaKMeans
[]");
- System.exit(1);
-}
-
-String inputFile = args[1];
-int k = Integer.parseInt(args[2]);
-int iterations = Integer.parseInt(args[3]);
+String inputFile = args[0];
--- End diff --
Same here, need args.length check
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/552#discussion_r12304690
--- Diff:
examples/src/main/java/org/apache/spark/mllib/examples/JavaALS.java ---
@@ -57,23 +58,17 @@ public String call(Tuple2 element) {
public static void main(String[] args) {
-if (args.length != 5 && args.length != 6) {
- System.err.println(
- "Usage: JavaALS
[]");
- System.exit(1);
-}
-
-int rank = Integer.parseInt(args[2]);
-int iterations = Integer.parseInt(args[3]);
-String outputDir = args[4];
+SparkConf sparkConf = new SparkConf().setAppName("JavaALS");
+int rank = Integer.parseInt(args[1]);
--- End diff --
We should keep the checks for `args.length` above, otherwise we may get an
obscure IndexOutOfBoundsException
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/552#discussion_r12304621
--- Diff: examples/src/main/java/org/apache/spark/examples/JavaTC.java ---
@@ -61,14 +62,9 @@
}
public static void main(String[] args) {
-if (args.length == 0) {
- System.err.println("Usage: JavaTC []");
- System.exit(1);
-}
-
-JavaSparkContext sc = new JavaSparkContext(args[0], "JavaTC",
-System.getenv("SPARK_HOME"),
JavaSparkContext.jarOfClass(JavaTC.class));
-Integer slices = (args.length > 1) ? Integer.parseInt(args[1]): 2;
+SparkConf sparkConf = new
org.apache.spark.SparkConf().setAppName("JavaHdfsLR");
+JavaSparkContext sc = new JavaSparkContext(sparkConf);
+Integer slices = (args.length > 0) ? Integer.parseInt(args[0]): 2;
--- End diff --
nit: space before colon
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/552#issuecomment-42036153 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14623/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/552#issuecomment-42036152 Build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/552#issuecomment-42028445 Build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/552#issuecomment-42028463 Build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/552#issuecomment-42028159 Build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/552#issuecomment-42028160 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14622/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/552#issuecomment-42027968 Build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/552#issuecomment-42027982 Build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/552#issuecomment-41963491 @ScrapCodes yes let's update the streaming examples too. If people run it in driver mode, this will be exactly the same as the current examples. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/552#issuecomment-41665951 Build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/552#issuecomment-41665952 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14565/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user ScrapCodes commented on the pull request: https://github.com/apache/spark/pull/552#issuecomment-41662663 Do you want this change to go for streaming as well !, because for some things it may not make sense. And then how do people stop it ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/552#issuecomment-41661200 Build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1565, update examples to be used with sp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/552#issuecomment-41661207 Build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
