[GitHub] spark issue #22575: [SPARK-24630][SS] Support SQLStreaming in Spark
Github user shijinkui commented on the issue: https://github.com/apache/spark/pull/22575 @cloud-fan Hi, Wenchen. Is it ready for merge in? This PR is very useful and is what I want to develop and need. Once Spark support StreamSQL, it will be easier for developping streaming job. Thanks. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12953][Examples]RDDRelation writer set ...
Github user shijinkui commented on the pull request: https://github.com/apache/spark/pull/10864#issuecomment-175477733 @JoshRosen fix checkstyle failure, please retest it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12953][Examples]RDDRelation writer set ...
Github user shijinkui commented on the pull request: https://github.com/apache/spark/pull/10864#issuecomment-175317509 test it --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12953][Examples]RDDRelation writer set ...
GitHub user shijinkui reopened a pull request: https://github.com/apache/spark/pull/10864 [SPARK-12953][Examples]RDDRelation writer set overwrite mode https://issues.apache.org/jira/browse/SPARK-12953 fix error when run RDDRelation.main(): "path file:/Users/sjk/pair.parquet already exists" Set DataFrameWriter's mode to SaveMode.Overwrite You can merge this pull request into a Git repository by running: $ git pull https://github.com/shijinkui/spark set_mode Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/10864.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #10864 commit 958a419877e36ad0d3987e83e56b6007937334e8 Author: shijinkui <shijinkui...@163.com> Date: 2016-01-21T08:56:26Z fix error when run RDDRelation.main():"path file:/Users/sjk/pair.parquet already exists" Setting DataFrameWriter's mode to `SaveMode.Overwrite` commit 2de63ecdfa8c7b0241587c0f9bb7d3f2edd4f7ce Author: shijinkui <shijinkui...@163.com> Date: 2016-01-25T16:13:21Z import order --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12953][Examples]RDDRelation writer set ...
Github user shijinkui commented on the pull request: https://github.com/apache/spark/pull/10864#issuecomment-174562740 hi, @rxin, I have reopened, please trigger the test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12953][Examples]RDDRelation writer set ...
Github user shijinkui commented on the pull request: https://github.com/apache/spark/pull/10864#issuecomment-174383417 @rxin This case throw exception at the second time of running. Unit test is import to project, example is useful for spark user. :) this problem is not important and critical, if can be fixed by the way, maybe better for user :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12953][Examples]RDDRelation writer set ...
Github user shijinkui closed the pull request at: https://github.com/apache/spark/pull/10864 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: fix error when run RDDRelation.main():"path fi...
GitHub user shijinkui opened a pull request: https://github.com/apache/spark/pull/10864 fix error when run RDDRelation.main():"path file:/Users/sjk/pair.parq⦠https://issues.apache.org/jira/browse/SPARK-12953 fix error when run RDDRelation.main(): "path file:/Users/sjk/pair.parquet already exists" Set DataFrameWriter's mode to SaveMode.Overwrite You can merge this pull request into a Git repository by running: $ git pull https://github.com/shijinkui/spark set_mode Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/10864.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #10864 commit 958a419877e36ad0d3987e83e56b6007937334e8 Author: shijinkui <shijinkui...@163.com> Date: 2016-01-21T08:56:26Z fix error when run RDDRelation.main():"path file:/Users/sjk/pair.parquet already exists" Setting DataFrameWriter's mode to `SaveMode.Overwrite` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9782] [YARN] Support YARN application t...
Github user shijinkui commented on a diff in the pull request:
https://github.com/apache/spark/pull/8072#discussion_r46532508
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -163,6 +163,23 @@ private[spark] class Client(
appContext.setQueue(args.amQueue)
appContext.setAMContainerSpec(containerContext)
appContext.setApplicationType("SPARK")
+sparkConf.getOption(CONF_SPARK_YARN_APPLICATION_TAGS)
+ .map(StringUtils.getTrimmedStringCollection(_))
+ .filter(!_.isEmpty())
+ .foreach { tagCollection =>
+try {
+ // The setApplicationTags method was only introduced in Hadoop
2.4+, so we need to use
+ // reflection to set it, printing a warning if a tag was
specified but the YARN version
+ // doesn't support it.
+ val method = appContext.getClass().getMethod(
+"setApplicationTags", classOf[java.util.Set[String]])
+ method.invoke(appContext, new
java.util.HashSet[String](tagCollection))
+} catch {
+ case e: NoSuchMethodException =>
+logWarning(s"Ignoring $CONF_SPARK_YARN_APPLICATION_TAGS
because this version of " +
+ "YARN does not support it")
+}
+ }
--- End diff --
using reflect make the code reading more difficult. generally not be
recommanded.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5062][Graphx] replace mapReduceTriplets...
Github user shijinkui closed the pull request at: https://github.com/apache/spark/pull/3883 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-7984][Graphx]Complex Operators between ...
Github user shijinkui commented on the pull request: https://github.com/apache/spark/pull/6685#issuecomment-114531864 hi @andrewor14 , what can i do for this error ``` [error] running /home/jenkins/workspace/SparkPullRequestBuilder/dev/../dev/mima ; received return code 255 Archiving unit tests logs... No log files found. Attempting to post to Github... Post successful. Build step 'Execute shell' marked build as failure Archiving artifacts Recording test results ERROR: Publisher 'Publish JUnit test result report' failed: No test report files were found. Configuration error? Finished: FAILURE ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5062][Graphx] replace mapReduceTriplets...
Github user shijinkui commented on the pull request: https://github.com/apache/spark/pull/3883#issuecomment-114688822 cc @andrewor14 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5036][Graphx]Better support sending par...
Github user shijinkui commented on the pull request: https://github.com/apache/spark/pull/3866#issuecomment-114688694 cc @andrewor14 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-7984][Graphx]Complex Operators between ...
Github user shijinkui commented on the pull request: https://github.com/apache/spark/pull/6685#issuecomment-113561706 what does this building error mean? is there more detail error log for `Binary compatibility check failed `? ``` [info] [SUCCESSFUL ] org.apache.spark#spark-mllib_2.10;1.4.0-rc4!spark-mllib_2.10.jar (1085ms) [info] spark-mllib: found 0 potential binary incompatibilities (filtered 5) [info] downloading https://repository.apache.org/content/repositories/orgapachespark-1112/org/apache/spark/spark-streaming-zeromq_2.10/1.4.0-rc4/spark-streaming-zeromq_2.10-1.4.0-rc4.jar ... [info] [SUCCESSFUL ] org.apache.spark#spark-streaming-zeromq_2.10;1.4.0-rc4!spark-streaming-zeromq_2.10.jar (635ms) [info] spark-streaming-zeromq: found 0 potential binary incompatibilities (filtered 1) java.lang.RuntimeException: spark-graphx: Binary compatibility check failed! at scala.sys.package$.error(package.scala:27) at com.typesafe.tools.mima.plugin.SbtMima$.reportErrors(SbtMima.scala:64) at com.typesafe.tools.mima.plugin.MimaPlugin$$anonfun$mimaReportSettings$3.apply(MimaPlugin.scala:23) at com.typesafe.tools.mima.plugin.MimaPlugin$$anonfun$mimaReportSettings$3.apply(MimaPlugin.scala:23) at scala.Function5$$anonfun$tupled$1.apply(Function5.scala:35) at scala.Function5$$anonfun$tupled$1.apply(Function5.scala:34) at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47) at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:40) at sbt.std.Transform$$anon$4.work(System.scala:63) at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226) at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226) at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17) at sbt.Execute.work(Execute.scala:235) at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226) at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226) at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159) at sbt.CompletionService$$anon$2.call(CompletionService.scala:28) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) [error] (graphx/*:mimaReportBinaryIssues) spark-graphx: Binary compatibility check failed! [error] Total time: 181 s, completed Jun 19, 2015 5:22:38 AM [error] running ['/home/jenkins/workspace/SparkPullRequestBuilder/dev/../dev/mima'] ; received return code 1 Archiving unit tests logs... No log files found. Attempting to post to Github... Post successful. Build step 'Execute shell' marked build as failure Archiving artifacts Recording test results ERROR: Publisher 'Publish JUnit test result report' failed: No test report files were found. Configuration error? Finished: FAILURE ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-7893][Graphx]Complex Operators between ...
GitHub user shijinkui opened a pull request: https://github.com/apache/spark/pull/6685 [SPARK-7893][Graphx]Complex Operators between Graphs: Union Union of Graphs ( G ⪠H ) You can merge this pull request into a Git repository by running: $ git pull https://github.com/shijinkui/spark graph_union Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/6685.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #6685 commit 5481d5e9e5fd7e02dd7a218dc87375aaf710073e Author: çç jinkui@alibaba-inc.com Date: 2015-05-31T23:43:05Z scala plugin tips err commit d7c6b783419718e2255eb5d71f0ba9cfda282064 Author: çç jinkui@alibaba-inc.com Date: 2015-06-01T09:33:07Z add edge full out join fn commit 96509ef0db1e12ee9dae3448873531965d0230e3 Author: çç jinkui@alibaba-inc.com Date: 2015-06-02T07:46:39Z vertex full out join; todo edgePartition merge vertex attr commit f8229416cce7ee0858e3c87de41bafd9736f383c Author: çç jinkui@alibaba-inc.com Date: 2015-06-02T09:48:01Z annotation modify commit 55132bc1437040386d666b5faaee77ee63af164f Author: çç jinkui@alibaba-inc.com Date: 2015-06-06T15:32:13Z add vertex merge function --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6494][Core]rdd polymorphic method zipPa...
Github user shijinkui commented on the pull request: https://github.com/apache/spark/pull/5164#issuecomment-86017000 @sryza @rxin understand, thx --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6494][Core]rdd polymorphic method zipPa...
Github user shijinkui closed the pull request at: https://github.com/apache/spark/pull/5164 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6494][Core]rdd polymorphic method zipPa...
Github user shijinkui commented on the pull request: https://github.com/apache/spark/pull/5164#issuecomment-85744268 @srowen thanks for viewing the trivial changes. the `zipPartitions` maybe have litter value. when readding code, i fount these trivial place, modify it by the way. i'm not bored:) just like you said marginal value, but when reading code in IDE, you can see so many yellow tips below screen shot. bad smell. if not clean the bad smell code, it will be everywhere. if not aggree, i'll close it. Thanks  --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6494][Core]rdd polymorphic method zipPa...
Github user shijinkui commented on a diff in the pull request:
https://github.com/apache/spark/pull/5164#discussion_r27084276
--- Diff: core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala ---
@@ -185,8 +181,10 @@ class HadoopRDD[K, V](
// done in each local process.
val newInputFormat =
ReflectionUtils.newInstance(inputFormatClass.asInstanceOf[Class[_]], conf)
.asInstanceOf[InputFormat[K, V]]
-if (newInputFormat.isInstanceOf[Configurable]) {
- newInputFormat.asInstanceOf[Configurable].setConf(conf)
+newInputFormat match {
--- End diff --
maybe.
i think too much nest if statement not looking better.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6494][Core]rdd polymorphic method zipPa...
Github user shijinkui commented on the pull request:
https://github.com/apache/spark/pull/5164#issuecomment-85805787
@rxin , curry function's first parameter will use default value if you miss
one value. it will be compatible ï¼ï¼
test code like this:
```
object Test11 {
def main(args: Array[String]): Unit = {
a1(1)(3)
a1(1, 2)(3)
}
def a1(a: Int, b: Int = 0)(c: Int) = {
println(a + , + b + , + c)
}
}
```
result:
1,0,3
1,2,3
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6494][Core]rdd polymorphic method zipPa...
Github user shijinkui commented on the pull request: https://github.com/apache/spark/pull/5164#issuecomment-85769334 @srowen @rxin please review zipPartitions refactor only. thanks --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6494][Core]rdd polymorphic method zipPa...
Github user shijinkui commented on a diff in the pull request: https://github.com/apache/spark/pull/5164#discussion_r27085539 --- Diff: core/src/main/scala/org/apache/spark/rdd/ZippedPartitionsRDD.scala --- @@ -60,7 +60,7 @@ private[spark] abstract class ZippedPartitionsBaseRDD[V: ClassTag]( val prefs = rdds.map(rdd = rdd.preferredLocations(rdd.partitions(i))) // Check whether there are any hosts that match all RDDs; otherwise return the union val exactMatchLocations = prefs.reduce((x, y) = x.intersect(y)) - val locs = if (!exactMatchLocations.isEmpty) exactMatchLocations else prefs.flatten.distinct + val locs = if (exactMatchLocations.nonEmpty) exactMatchLocations else prefs.flatten.distinct --- End diff -- `def isEmpty: Boolean = lengthCompare(0) == 0` `def nonEmpty: Boolean = !isEmpty` `!exactMatchLocations.isEmpty` equal `exactMatchLocations.nonEmpty` nonEmpty maybe more clear semantically --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: rdd polymorphic method zipPartitions refactor
GitHub user shijinkui opened a pull request: https://github.com/apache/spark/pull/5164 rdd polymorphic method zipPartitions refactor no need so many polymorphic method, only add default value instead. modify partition.size instead of partition.length, partitions is Array object. You can merge this pull request into a Git repository by running: $ git pull https://github.com/shijinkui/spark common Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/5164.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5164 commit 602d64788b47e16454792c0cbf5c7c42e498cb50 Author: çç jinkui@alibaba-inc.com Date: 2015-03-24T08:25:18Z reflact zipPartitions of RDD commit cca1703da824ada40c543d777a0a9ce8b777932e Author: çç jinkui@alibaba-inc.com Date: 2015-03-24T08:44:32Z partition.size change to partition.length --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5036][Graphx]Better support sending par...
Github user shijinkui commented on the pull request: https://github.com/apache/spark/pull/3866#issuecomment-69871618 @rxin @ankurdave have any other problem? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5036][Graphx]Better support sending par...
Github user shijinkui commented on the pull request: https://github.com/apache/spark/pull/3866#issuecomment-68969847 @ankurdave thanks for reviewing :) 1. Activeness is from `innerJoin`'s result by `vprog` the message of pre iteration.`isAttrChanged` is from solution1. It can run, but the key problem is after 100 or more iteration, it's progressive increase one second every iteration. So, I consider the solution 2. Solution 2 is surprised on both performance and code logic is more clear. 2. take k-core algorithm for example. The coreness from 4 to 3 need iterating 68 times, from 3 to 2 need hundreds. But the sum coreness of 4, 2, 1 in the final result is not important, or user can ignore the effect of lacking computing coreness `4, 3, 2,1`. So they want't to finish the loop ahead of time. `isTerminal ` provide a custom condition of terminal by current iterate id `curIter` and `messageCount`. Default `isTerminal` is true. 3. making `initialMsg` an `Option`, as sometimes the initalMsg is empty. when `initalMsg` is empty, aggregate message on every Vertex directly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5062][Graphx] replace mapReduceTriplets...
Github user shijinkui commented on the pull request: https://github.com/apache/spark/pull/3883#issuecomment-68973132 hi, @ankurdave , I remembered that the PR should be as soon as smaller for easy testing and reviewing. #3866 is more complexï¼let it independent maybe better. How do u think about? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5036][Graphx]Better support sending par...
Github user shijinkui commented on the pull request: https://github.com/apache/spark/pull/3866#issuecomment-68974660 @ankurdave Solution 1 is better. It have to add `joinVertices ` for reset `isAttrChanged` to default value. It have two problem having no idea for solving: 1. Progressive increase one second every iteration. We can't accept one second progressive increased, because after 600th iteration, the extra cost will become 600s. It like this: iteration | increace | cost | - | 100 | 1 | 150 101 | 1 | 151 102 | 1 | 152 103 | 1 | 153 104 | 1 | 154 .. | 1 | .. 700 | 1 | 850 701 | 1 | 851 703 | 1 | 852 704 | 1 | 853 2 `joinVertices` `joinVertices` will own half cost of the whole `Pregel-liked` cost. The more iteration the bigger of `joinVertices`. It's a linear increase. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5036][Graphx]Better support sending par...
Github user shijinkui commented on the pull request: https://github.com/apache/spark/pull/3866#issuecomment-68710366 @ankurdave @rxin --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5062][Graphx] replace mapReduceTriplets...
GitHub user shijinkui opened a pull request: https://github.com/apache/spark/pull/3883 [SPARK-5062][Graphx] replace mapReduceTriplets with aggregateMessage in Pregel Api since spark 1.2 introduce aggregateMessage instead of mapReduceTriplets, it improve the performance indeed. it's time to replace mapReduceTriplets with aggregateMessage in Pregel. i provide a deprecated method thinking about compatibility -- i have draw a graph of aggregateMessage to show why it can improve the performance.  dfgdfgd You can merge this pull request into a Git repository by running: $ git pull https://github.com/shijinkui/spark pregel_agg Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/3883.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3883 commit 93ae74bc5c9011719775e9862f257c2e81a9 Author: çç jinkui@alibaba-inc.com Date: 2015-01-01T02:43:27Z change mapReduceTriplets to aggregateMessages of Pregel API commit d2519e235c53c8ee53c5f127cf680585f139eb0c Author: çç jinkui@alibaba-inc.com Date: 2015-01-01T03:21:30Z change mapReduceTriplets to aggregateMessages of Pregel API --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3896] Pool#checkSpeculatableTasks fask ...
Github user shijinkui commented on the pull request: https://github.com/apache/spark/pull/2751#issuecomment-59157814 the Pool#checkSpeculatableTasks method named ambiguity. TaskSetManager.checkSpeculatableTasks:Check for tasks to be speculated and return true if there are any it put the specula table task to Map: speculatableTasks bad smell design --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3896] Pool#checkSpeculatableTasks fask ...
Github user shijinkui closed the pull request at: https://github.com/apache/spark/pull/2751 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3897]format example code and explicit v...
Github user shijinkui closed the pull request at: https://github.com/apache/spark/pull/2754 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3781] code format and little improvemen...
Github user shijinkui closed the pull request at: https://github.com/apache/spark/pull/2734 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3896] checkSpeculatableTasks fask quit ...
GitHub user shijinkui opened a pull request: https://github.com/apache/spark/pull/2751 [SPARK-3896] checkSpeculatableTasks fask quit loop; checkSpeculatableTasks is expensive 1. invoking checkSpeculatableTasks is expensive, once true then return true 2. import section formatting You can merge this pull request into a Git repository by running: $ git pull https://github.com/shijinkui/spark fast_quit_loop Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2751.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2751 commit 63f60d5ac25b6ad37bfbb63bd93b5072f1f955a1 Author: çç jinkui@alibaba-inc.com Date: 2014-10-10T06:20:51Z checkSpeculatableTasks fask quit loop; checkSpeculatableTasks is expensive --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3897]format example code and explicit v...
GitHub user shijinkui opened a pull request: https://github.com/apache/spark/pull/2754 [SPARK-3897]format example code and explicit val instead of var You can merge this pull request into a Git repository by running: $ git pull https://github.com/shijinkui/spark example_improve Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2754.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2754 commit b2009bc04ed750b2ef03e76b5fdae2bffd958967 Author: çç jinkui@alibaba-inc.com Date: 2014-10-10T07:19:37Z format example code and explicit val instead of var --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3897]format example code and explicit v...
Github user shijinkui commented on a diff in the pull request:
https://github.com/apache/spark/pull/2754#discussion_r18693582
--- Diff: examples/src/main/scala/org/apache/spark/examples/LocalALS.scala
---
@@ -31,9 +31,12 @@ import cern.jet.math._
*/
object LocalALS {
// Parameters set through command line arguments
- var M = 0 // Number of movies
- var U = 0 // Number of users
- var F = 0 // Number of features
+ var M = 0
+ // Number of movies
--- End diff --
standard usage is two line.
code separate with annotation as fa as possible, as scala can omit
semicolon, not likely java
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3897]format example code and explicit v...
Github user shijinkui commented on a diff in the pull request:
https://github.com/apache/spark/pull/2754#discussion_r18693711
--- Diff:
examples/src/main/scala/org/apache/spark/examples/streaming/ActorWordCount.scala
---
@@ -127,12 +127,12 @@ object FeederActor {
* A sample word count program demonstrating the use of plugging in
* Actor as Receiver
* Usage: ActorWordCount hostname port
- * hostname and port describe the AkkaSystem that Spark Sample
feeder is running on.
+ * hostname and port describe the AkkaSystem that Spark Sample feeder
is running on.
--- End diff --
agree.
the problem is i cannot find where to set two space which line start with
in intellij IDEA
i think we can tolerate this less readability place
:)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3897]format example code and explicit v...
Github user shijinkui commented on the pull request: https://github.com/apache/spark/pull/2754#issuecomment-58628292 this format only in example module, cannot effect the other module our coding habit is formatting after modify by the way. if there are several difference after formatting, it will effect the current modify. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3897]format example code and explicit v...
Github user shijinkui commented on the pull request: https://github.com/apache/spark/pull/2754#issuecomment-58628485 @srowen can u review the other PR #2751 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: code style format and some litter improvment
GitHub user shijinkui opened a pull request: https://github.com/apache/spark/pull/2733 code style format and some litter improvment You can merge this pull request into a Git repository by running: $ git pull https://github.com/shijinkui/spark styleFormat Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2733.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2733 commit bd103b78adebf933d3e33d4628c2394a845dbaa2 Author: shijinkui shijinkui...@163.com Date: 2014-09-29T05:34:02Z code style format commit 0c9d1754f9c8bf36236f915868eb6f62863fea25 Author: shijinkui shijinkui...@163.com Date: 2014-10-03T14:21:44Z code format commit 237bacc3d1c911e6475da4ad08dcd4d0031883ec Author: shijinkui shijinkui...@163.com Date: 2014-10-08T03:05:01Z resolve conflic commit 78f69b9526ea546251dac5eddf4da9c9eb6e20ad Author: shijinkui shijinkui...@163.com Date: 2014-10-08T03:31:24Z code format commit 725eec51fb7d29d0df99b92df3ef62fcec301d90 Author: çç jinkui@alibaba-inc.com Date: 2014-10-08T14:42:29Z resolve test fail commit e54344b33b4c8cb4c1ff0dfb18a08188de464cfc Author: çç jinkui@alibaba-inc.com Date: 2014-10-09T08:55:18Z code format --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3781] code Style format and little impr...
GitHub user shijinkui opened a pull request: https://github.com/apache/spark/pull/2734 [SPARK-3781] code Style format and little improvement 1. use scala recommended usage 2. method body's left bracket 3. parameter list format 4. explicit mutable collection, such as new mutable.HashMap 5. others You can merge this pull request into a Git repository by running: $ git pull https://github.com/shijinkui/spark master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2734.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2734 commit bd103b78adebf933d3e33d4628c2394a845dbaa2 Author: shijinkui shijinkui...@163.com Date: 2014-09-29T05:34:02Z code style format commit 0c9d1754f9c8bf36236f915868eb6f62863fea25 Author: shijinkui shijinkui...@163.com Date: 2014-10-03T14:21:44Z code format commit 237bacc3d1c911e6475da4ad08dcd4d0031883ec Author: shijinkui shijinkui...@163.com Date: 2014-10-08T03:05:01Z resolve conflic commit 78f69b9526ea546251dac5eddf4da9c9eb6e20ad Author: shijinkui shijinkui...@163.com Date: 2014-10-08T03:31:24Z code format commit 725eec51fb7d29d0df99b92df3ef62fcec301d90 Author: çç jinkui@alibaba-inc.com Date: 2014-10-08T14:42:29Z resolve test fail commit e54344b33b4c8cb4c1ff0dfb18a08188de464cfc Author: çç jinkui@alibaba-inc.com Date: 2014-10-09T08:55:18Z code format --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: code style format and some litter improvment
Github user shijinkui closed the pull request at: https://github.com/apache/spark/pull/2733 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3781] code format and little improvemen...
Github user shijinkui commented on the pull request: https://github.com/apache/spark/pull/2734#issuecomment-58612450 have add three sub tasks #SPARK-3849 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3781] code Style format
Github user shijinkui closed the pull request at: https://github.com/apache/spark/pull/2704 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3781] code Style format
GitHub user shijinkui opened a pull request: https://github.com/apache/spark/pull/2714 [SPARK-3781] code Style format code Style format, modify the third time :( You can merge this pull request into a Git repository by running: $ git pull https://github.com/shijinkui/spark master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2714.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2714 commit e00f13a554fb4a95996250e6cd65403fcc20b6e2 Author: shijinkui shijinkui...@163.com Date: 2014-09-29T05:34:02Z code style format commit 271e2a4a60ec2412f32966b65448c8ca25dc377f Author: shijinkui shijinkui...@163.com Date: 2014-10-03T14:21:44Z code format commit 033af6a1bd2011b847633934ba719716d459aca8 Author: shijinkui shijinkui...@163.com Date: 2014-10-03T15:03:24Z code style format commit 5732fae2d1ad94c820cc0a2fbc246aeee3142c00 Author: shijinkui shijinkui...@163.com Date: 2014-10-08T01:30:07Z resolve conflic commit 1f96a878815fd65035e11f5755426e6c9fda3ece Author: shijinkui shijinkui...@163.com Date: 2014-10-08T03:05:01Z resolve conflic commit 4a685cc2ea323f0fc54d6593e0ee568d829ebb02 Author: shijinkui shijinkui...@163.com Date: 2014-10-08T03:31:24Z code format commit 5be64c970c2f5d7fb895910863eb6aa5d1e923bf Author: çç jinkui@alibaba-inc.com Date: 2014-10-08T14:41:58Z Merge branch 'master' of git://git.apache.org/spark commit 42a9d81773d198c9fab7f77e0f2673d16c71 Author: çç jinkui@alibaba-inc.com Date: 2014-10-08T14:42:29Z resolve test fail --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3781] code Style format
Github user shijinkui commented on the pull request: https://github.com/apache/spark/pull/2704#issuecomment-58369964 @AmplabJenkins @andrewor14 review and run test again please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3781] code Style format
Github user shijinkui commented on the pull request: https://github.com/apache/spark/pull/2714#issuecomment-58370144 @AmplabJenkins @andrewor14 review and run test again please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3781] code Style format
Github user shijinkui closed the pull request at: https://github.com/apache/spark/pull/2714 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: code style format
Github user shijinkui commented on the pull request: https://github.com/apache/spark/pull/2643#issuecomment-58187162 in the intellij IDEA, too much yellow tips to fix. after changing, the code looks better. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3781] code Style format
Github user shijinkui commented on a diff in the pull request:
https://github.com/apache/spark/pull/2644#discussion_r18560510
--- Diff: core/src/main/scala/org/apache/spark/Aggregator.scala ---
@@ -40,10 +40,9 @@ case class Aggregator[K, V, C] (
def combineValuesByKey(iter: Iterator[_ : Product2[K, V]]):
Iterator[(K, C)] =
combineValuesByKey(iter, null)
- def combineValuesByKey(iter: Iterator[_ : Product2[K, V]],
- context: TaskContext): Iterator[(K, C)] = {
+ def combineValuesByKey(iter: Iterator[_ : Product2[K, V]], context:
TaskContext): Iterator[(K, C)] = {
--- End diff --
i think method parameter should be in one line possibly, there is no
necessary each param one line
:)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3781] code Style format
Github user shijinkui commented on a diff in the pull request:
https://github.com/apache/spark/pull/2644#discussion_r18561868
--- Diff:
core/src/main/scala/org/apache/spark/broadcast/BroadcastManager.scala ---
@@ -59,7 +59,7 @@ private[spark] class BroadcastManager(
private val nextBroadcastId = new AtomicLong(0)
def newBroadcast[T: ClassTag](value_ : T, isLocal: Boolean) = {
-broadcastFactory.newBroadcast[T](value_, isLocal,
nextBroadcastId.getAndIncrement())
+broadcastFactory.newBroadcast[T](value_, isLocal,
nextBroadcastId.getAndIncrement)
--- End diff --
val i: AtomicLong = new AtomicLong(0)
i.incrementAndGet()
Code:
0: new #16 // class
java/util/concurrent/atomic/AtomicLong
3: dup
4: lconst_0
5: invokespecial #19 // Method
java/util/concurrent/atomic/AtomicLong.init:(J)V
8: astore_2
9: aload_2
10: invokevirtual #23 // Method
java/util/concurrent/atomic/AtomicLong.incrementAndGet:()J
13: pop2
14: return
val i: AtomicLong = new AtomicLong(0)
i.incrementAndGet
Code:
0: new #16 // class
java/util/concurrent/atomic/AtomicLong
3: dup
4: lconst_0
5: invokespecial #19 // Method
java/util/concurrent/atomic/AtomicLong.init:(J)V
8: astore_2
9: aload_2
10: invokevirtual #23 // Method
java/util/concurrent/atomic/AtomicLong.incrementAndGet:()J
13: pop2
14: return
they are same byte code.
scala recommend having no () if having no param
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3781] code Style format
Github user shijinkui commented on a diff in the pull request:
https://github.com/apache/spark/pull/2644#discussion_r18562022
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/SchedulerBackend.scala ---
@@ -38,5 +42,4 @@ private[spark] trait SchedulerBackend {
* @return The application ID, or None if the backend does not provide
an ID.
*/
def applicationId(): Option[String] = None
-
-}
+}
--- End diff --
OK
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3781] code Style format
Github user shijinkui commented on a diff in the pull request:
https://github.com/apache/spark/pull/2644#discussion_r18562132
--- Diff: core/src/main/scala/org/apache/spark/ui/UIWorkloadGenerator.scala
---
@@ -17,11 +17,11 @@
package org.apache.spark.ui
-import scala.util.Random
-
-import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.SparkContext._
import org.apache.spark.scheduler.SchedulingMode
+import org.apache.spark.{SparkConf, SparkContext}
+
+import scala.util.Random
--- End diff --
http://google-styleguide.googlecode.com/svn/trunk/javaguide.html#s3.3-import-statements
in ASCII sort order
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3781] code Style format
Github user shijinkui commented on a diff in the pull request:
https://github.com/apache/spark/pull/2644#discussion_r18562140
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -265,15 +265,15 @@ private[spark] object Utils extends Logging {
/** Copy all data from an InputStream to an OutputStream */
def copyStream(in: InputStream,
- out: OutputStream,
- closeStreams: Boolean = false): Long =
+out: OutputStream,
+closeStreams: Boolean = false): Long =
--- End diff --
yes
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3781] code Style format
Github user shijinkui commented on the pull request: https://github.com/apache/spark/pull/2644#issuecomment-58302944 repush at #2704 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3781] code Style format
GitHub user shijinkui opened a pull request: https://github.com/apache/spark/pull/2704 [SPARK-3781] code Style format 1. code formater 2. scala normal usage about method and param You can merge this pull request into a Git repository by running: $ git pull https://github.com/shijinkui/spark master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2704.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2704 commit e00f13a554fb4a95996250e6cd65403fcc20b6e2 Author: shijinkui shijinkui...@163.com Date: 2014-09-29T05:34:02Z code style format commit 271e2a4a60ec2412f32966b65448c8ca25dc377f Author: shijinkui shijinkui...@163.com Date: 2014-10-03T14:21:44Z code format commit 033af6a1bd2011b847633934ba719716d459aca8 Author: shijinkui shijinkui...@163.com Date: 2014-10-03T15:03:24Z code style format commit 5732fae2d1ad94c820cc0a2fbc246aeee3142c00 Author: shijinkui shijinkui...@163.com Date: 2014-10-08T01:30:07Z resolve conflic commit 1f96a878815fd65035e11f5755426e6c9fda3ece Author: shijinkui shijinkui...@163.com Date: 2014-10-08T03:05:01Z resolve conflic commit 4a685cc2ea323f0fc54d6593e0ee568d829ebb02 Author: shijinkui shijinkui...@163.com Date: 2014-10-08T03:31:24Z code format --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3781] code Style format
Github user shijinkui closed the pull request at: https://github.com/apache/spark/pull/2644 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: code style format
GitHub user shijinkui opened a pull request: https://github.com/apache/spark/pull/2643 code style format code style format You can merge this pull request into a Git repository by running: $ git pull https://github.com/shijinkui/spark master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2643.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2643 commit e00f13a554fb4a95996250e6cd65403fcc20b6e2 Author: shijinkui shijinkui...@163.com Date: 2014-09-29T05:34:02Z code style format --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: code style format
Github user shijinkui closed the pull request at: https://github.com/apache/spark/pull/2643 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3781] code Style format
GitHub user shijinkui opened a pull request: https://github.com/apache/spark/pull/2644 [SPARK-3781] code Style format You can merge this pull request into a Git repository by running: $ git pull https://github.com/shijinkui/spark styleFormat Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2644.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2644 commit e00f13a554fb4a95996250e6cd65403fcc20b6e2 Author: shijinkui shijinkui...@163.com Date: 2014-09-29T05:34:02Z code style format commit 271e2a4a60ec2412f32966b65448c8ca25dc377f Author: shijinkui shijinkui...@163.com Date: 2014-10-03T14:21:44Z code format commit 033af6a1bd2011b847633934ba719716d459aca8 Author: shijinkui shijinkui...@163.com Date: 2014-10-03T15:03:24Z code style format --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
