Github user berngp commented on the pull request:
https://github.com/apache/incubator-spark/pull/584#issuecomment-34955400
Thanks @pwendell
Github user bijaybisht commented on the pull request:
https://github.com/apache/incubator-spark/pull/568#issuecomment-34955181
Yes it is getting set via SparkConf now.
Github user pwendell commented on the pull request:
https://github.com/apache/incubator-spark/pull/568#issuecomment-34955056
@aarondav pick this into master as well when you merge it.
Github user pwendell commented on the pull request:
https://github.com/apache/incubator-spark/pull/568#issuecomment-34955035
LGTM
Github user pwendell commented on the pull request:
https://github.com/apache/incubator-spark/pull/568#issuecomment-34955031
@ScrapCodes ah I see - this is changing the system properties iterator to a
spark conf iterator. Makes sense.
Github user pwendell commented on the pull request:
https://github.com/apache/incubator-spark/pull/584#issuecomment-34954968
Thanks I merged this into master and 0.9.
Github user rxin commented on the pull request:
https://github.com/apache/incubator-spark/pull/593#issuecomment-34954956
Ok I merged this using the new script!
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/incubator-spark/pull/584#issuecomment-34954878
All automated tests passed.
Refer to this link for build results:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12703/
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/incubator-spark/pull/584#issuecomment-34954877
Merged build finished.
Github user ash211 closed the pull request at:
https://github.com/apache/incubator-spark/pull/574
Github user ScrapCodes commented on the pull request:
https://github.com/apache/incubator-spark/pull/568#issuecomment-34954162
Not if they have set it via SparkConf. (I guess)
Github user rxin closed the pull request at:
https://github.com/apache/incubator-spark/pull/592
Github user pwendell commented on the pull request:
https://github.com/apache/incubator-spark/pull/574#issuecomment-34954401
Merged (guess how!). Thanks Andrew
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/incubator-spark/pull/593#issuecomment-34954238
Merged build finished.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/incubator-spark/pull/593#issuecomment-34954239
All automated tests passed.
Refer to this link for build results:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12702/
Github user srowen commented on the pull request:
https://github.com/apache/incubator-spark/pull/575#issuecomment-34954740
My $0.02 to the discussion:
1. Within whatever operations mllib provides, serialization can be
considered an implementation detail. But external serializa
Github user pwendell commented on the pull request:
https://github.com/apache/incubator-spark/pull/593#issuecomment-34954387
LGTM
Github user pwendell commented on the pull request:
https://github.com/apache/incubator-spark/pull/568#issuecomment-34953913
@bijaybisht Could you explain somewhere what the bug is that this is
fixing? The jira references "spark.driver.host" but in theory those should have
worked unde
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/incubator-spark/pull/584#issuecomment-34953722
Merged build triggered.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/incubator-spark/pull/584#issuecomment-34953723
Merged build started.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/incubator-spark/pull/574#issuecomment-34953630
All automated tests passed.
Refer to this link for build results:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12701/
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/incubator-spark/pull/574#issuecomment-34953629
Merged build finished.
Github user pwendell commented on the pull request:
https://github.com/apache/incubator-spark/pull/584#issuecomment-34953539
Jenkins, test this please.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/incubator-spark/pull/593#issuecomment-34953101
Merged build triggered.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/incubator-spark/pull/593#issuecomment-34953102
Merged build started.
Github user rxin commented on the pull request:
https://github.com/apache/incubator-spark/pull/590#issuecomment-34953037
Closing this one since it has been merged.
Github user rxin closed the pull request at:
https://github.com/apache/incubator-spark/pull/590
Github user pwendell commented on the pull request:
https://github.com/apache/incubator-spark/pull/592#issuecomment-34952990
@aarondav @rxin we should modify Jenkins now so if this script is present,
it calls it :)
GitHub user rxin opened a pull request:
https://github.com/apache/incubator-spark/pull/593
SPARK-1088: Create a script for running tests so we can have version
specific testing on Jenkins (branch-0.9)
This is for branch-0.9.
#592 is for master branch (1.0).
You can merge t
Github user pwendell commented on the pull request:
https://github.com/apache/incubator-spark/pull/574#issuecomment-34952676
Andrew - looks great. Thanks for contributing this. I think the test error
was unrelated... pending successful tests LGTM.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/incubator-spark/pull/574#issuecomment-34952512
Merged build started.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/incubator-spark/pull/574#issuecomment-34952511
Merged build triggered.
Github user pwendell commented on the pull request:
https://github.com/apache/incubator-spark/pull/574#issuecomment-34952393
Jenkins, retest this please.
Github user aarondav commented on the pull request:
https://github.com/apache/incubator-spark/pull/592#issuecomment-34952267
Thanks, merged into master.
Github user aarondav commented on the pull request:
https://github.com/apache/incubator-spark/pull/592#issuecomment-34952134
Gotcha, well, this is great, we can tell people that they can run it
manually before submitting a PR to catch style violations and such. Easier than
telling the
Github user rxin commented on the pull request:
https://github.com/apache/incubator-spark/pull/592#issuecomment-34952038
Yea. It just ran a script similar to this, except it is not a "script",
just a sequence of commands defined in jenkins.
Github user aarondav commented on the pull request:
https://github.com/apache/incubator-spark/pull/592#issuecomment-34951992
What is jenkins currently running? It is a script similar to this, that
simply wasn't previously in our repo? I'm wondering, for instance, how
scalastyle was ad
Github user ash211 commented on the pull request:
https://github.com/apache/incubator-spark/pull/574#issuecomment-34949509
Those errors look like unrelated pyspark issues. Those weren't caused by
this change were they?
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullR
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/incubator-spark/pull/592#issuecomment-34949471
Merged build finished.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/incubator-spark/pull/592#issuecomment-34949472
All automated tests passed.
Refer to this link for build results:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12700/
Github user rxin commented on the pull request:
https://github.com/apache/incubator-spark/pull/592#issuecomment-34949375
As suggested in #590.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/incubator-spark/pull/574#issuecomment-34949294
Merged build finished.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/incubator-spark/pull/574#issuecomment-34949295
One or more automated tests failed
Refer to this link for build results:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12699/
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/incubator-spark/pull/592#issuecomment-34948424
Merged build triggered.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/incubator-spark/pull/592#issuecomment-34948425
Merged build started.
Github user rxin commented on the pull request:
https://github.com/apache/incubator-spark/pull/592#issuecomment-34948410
I will submit another one for branch-0.9 if this looks good.
GitHub user rxin opened a pull request:
https://github.com/apache/incubator-spark/pull/592
SPARK-1088: Create a script for running tests so we can have version
specific testing on Jenkins.
@pwendell
You can merge this pull request into a Git repository by running:
$ git pull h
Github user shivaram commented on the pull request:
https://github.com/apache/incubator-spark/pull/575#issuecomment-34948280
Sorry I missed this thread, but I'd like to understand a bit more about the
scope of what we require in terms of library support before taking a decision.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/incubator-spark/pull/574#issuecomment-34948232
Merged build triggered.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/incubator-spark/pull/574#issuecomment-34948233
Merged build started.
Github user rxin commented on the pull request:
https://github.com/apache/incubator-spark/pull/590#issuecomment-34947355
That's a good idea. I will submit a PR.
Github user pwendell commented on the pull request:
https://github.com/apache/incubator-spark/pull/574#issuecomment-34946324
Hey @ash211 this is an improvement! Let's just remove the "Merge pull
request..." as it is now redundant anyways with other information. Instead this
content ca
Github user pwendell commented on the pull request:
https://github.com/apache/incubator-spark/pull/590#issuecomment-34945973
btw - I think the long term solution here is to have a script inside of the
spark repo that jenkins calls to run tests (./dev/run-tests) that way as we
evolve t
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/incubator-spark/pull/586#issuecomment-34941773
All automated tests passed.
Refer to this link for build results:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12698/
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/incubator-spark/pull/586#issuecomment-34941772
Merged build finished.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/incubator-spark/pull/586#issuecomment-34940302
Merged build started.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/incubator-spark/pull/586#issuecomment-34940301
Merged build triggered.
Github user srowen commented on the pull request:
https://github.com/apache/incubator-spark/pull/586#issuecomment-34940308
Done, I believe. Have another glance at it.
Github user srowen commented on the pull request:
https://github.com/apache/incubator-spark/pull/586#issuecomment-34935722
I have a new commit ready to go that addresses the comments, but before I
pull the trigger, see replies inline with some questions about how you'd
like to p
Github user mengxr closed the pull request at:
https://github.com/apache/incubator-spark/pull/591
Github user rxin commented on the pull request:
https://github.com/apache/incubator-spark/pull/591#issuecomment-34935532
Ok I'm merging this one. Thanks.
Sorry, missed this email reply =)
Thanks much, looks good to me, will update the wiki as needed to help
add more links to ASF resources.
- Henry
On Sun, Feb 9, 2014 at 10:13 AM, Patrick Wendell wrote:
> Done, thanks. Feel free to edit it directly as well :)
>
> On Sat, Feb 8, 2014 at 11:28 PM,
Github user willb commented on the pull request:
https://github.com/apache/incubator-spark/pull/582#issuecomment-34931526
Thanks for the review, Aaron! After others have weighed in, I'll amend my
branch to catch the right exception (and correct my email address).
Usually inlined, not always. From the infamous Coda Hale rant:
4. Always use private[this]. Doing so avoids turning simple field access into an
> invokevirtual on generated getters and setters. Generally HotSpot would end up
> inlining these, but inside our inner serialization loop this made a hu
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/incubator-spark/pull/591#issuecomment-34927810
Merged build finished.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/incubator-spark/pull/591#issuecomment-34927811
All automated tests passed.
Refer to this link for build results:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12697/
I believe the performance implications are negligible due to the JIT. If
that getter is not inlined at runtime, I will eat a shoe.
I think the main question is still whether we want to avoid fields secretly
springing into existence or we want the significantly more concise syntax
of unannotated pa
There is at least potential for performance difference with the extra level
of indirection of `private val` compared to `private[this] val`; so from
that perspective, `private[this]` or closing over an unannotated
constructor parameter is preferable to using a `private val` parameter. On
the other
Github user aarondav commented on the pull request:
https://github.com/apache/incubator-spark/pull/590#issuecomment-34926093
Thanks, merged!
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/incubator-spark/pull/590#issuecomment-34925843
All automated tests passed.
Refer to this link for build results:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12696/
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/incubator-spark/pull/590#issuecomment-34925842
Merged build finished.
Github user markhamstra commented on the pull request:
https://github.com/apache/incubator-spark/pull/421#issuecomment-34925691
@rxin Any thoughts on how to do something like:
```scala
val futureAction = rdd.toIterator.nextAsync
futureAction.cancel()
```
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/incubator-spark/pull/591#issuecomment-34925086
Merged build triggered.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/incubator-spark/pull/591#issuecomment-34925087
Merged build started.
GitHub user mengxr opened a pull request:
https://github.com/apache/incubator-spark/pull/591
SPARK-1076: [Fix #578] add @transient to some vals
I'll try to be more careful next time.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com
Thanks for the clarification, Mark. So "private val" generates the Scala
getter whereas "private[this] val" does not, and the field-ified
constructor parameter mimics "private[this] val".
However, the distinction between those two seems less important than the
distinction between a constructor par
It's actually a little more complicated than that, mostly due to the
difference between private and private[this]. Allow me to demonstrate:
package dummy
class Foo1(a: Int, b: Int) {
private val c = a + b
}
class Foo2(a: Int, b: Int) {
private[this] val c = a + b
}
class Foo3(a: Int, b: In
Github user rxin commented on the pull request:
https://github.com/apache/incubator-spark/pull/584#issuecomment-34923271
We can test this once #590 is in.
Github user marmbrus commented on the pull request:
https://github.com/apache/incubator-spark/pull/590#issuecomment-34923209
LGTM if jenkins likes it :)
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/incubator-spark/pull/590#issuecomment-34923008
Merged build triggered.
Github user rxin commented on the pull request:
https://github.com/apache/incubator-spark/pull/590#issuecomment-34923019
@aarondav @marmbrus
i'm no sbt expert
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/incubator-spark/pull/590#issuecomment-34923009
Merged build started.
GitHub user rxin opened a pull request:
https://github.com/apache/incubator-spark/pull/590
SPARK-1085: Fix Jenkins pull request builder for branch-0.9 (scalastyle
command not found)
Added a dummy scalastyle task to sbt.
https://spark-project.atlassian.net/browse/SPARK-1085
Github user aarondav commented on the pull request:
https://github.com/apache/incubator-spark/pull/582#issuecomment-34922638
This looks good to me, only significant change is correcting the thrown
exception. I am somewhat underwhelmed by json4s's documentation. For instance,
this file
Regarding styling: as we all know, constructor parameters in Scala are
automatically upgraded to "private val" fields if they're referenced
outside of the constructor. For instance:
class Foo(a: Int, b: Int) {
def getB = b
}
In the above case, 'b' is actually a "private val" field of Foo, wherea
Github user aarondav commented on the pull request:
https://github.com/apache/incubator-spark/pull/586#issuecomment-34918907
Thanks for this! Did a quick pass and left some comments.
Github user rxin commented on the pull request:
https://github.com/apache/incubator-spark/pull/587#issuecomment-34916300
Thanks for submitting this. Just curious, what is the advantage of this
over rdd.cartesian(rdd), i.e. just use cartesian to join itself?
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/incubator-spark/pull/572#issuecomment-34915737
Merged build finished.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/incubator-spark/pull/572#issuecomment-34915739
All automated tests passed.
Refer to this link for build results:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12695/
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/incubator-spark/pull/572#issuecomment-34912554
Merged build triggered.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/incubator-spark/pull/572#issuecomment-34912556
Merged build started.
As far as I understand, json4s attempts to do this by providing a single
interface for users where the underlying JSON parser is pluggable.
Currently, there are lift-json (called "native") and jackson plugins, and
we are using the latter in the current PR.
On Wed, Feb 12, 2014 at 11:36 AM, Pascal
Yup; you're right:
https://github.com/json4s/json4s/blob/3.2.6_2.10/project/Dependencies.scala
The older deps are only in use in examples/benchmarking. All good.
—
p...@mult.ifario.us | Multifarious, Inc. | http://mult.ifario.us/
On Wed, Feb 12, 2014 at 11:35 AM, Aaron Davidson wrote:
> Th
Github user mengxr closed the pull request at:
https://github.com/apache/incubator-spark/pull/589
Github user aarondav commented on the pull request:
https://github.com/apache/incubator-spark/pull/589#issuecomment-34908063
Timely merges may not have been the problem -- the title of this PR only
mentions the change you had already made :)
Please signal if your PR is not rea
I have one question : isn't it possible to abstract a bit and not depend on
a given json implementation as this is still a moving target?
Regards
Pascal
Le 12 févr. 2014 20:30, "Paul Brown" a écrit :
> Hi, Aaron --
>
> I can't speak to issues relevant to Spark, but it looks like json4s is
> curr
The version of json4s we're using (3.2.6 in the 2.10 branch) does seem to
depend on Jackson 2.3.0 and Scala 2.10.0:
http://mvnrepository.com/artifact/org.json4s/json4s-jackson_2.10/3.2.6
On Wed, Feb 12, 2014 at 11:29 AM, Paul Brown wrote:
> Hi, Aaron --
>
> I can't speak to issues relevant to S
Github user sryza commented on the pull request:
https://github.com/apache/incubator-spark/pull/553#issuecomment-34906646
@tgravescs I should have tried this - it looks like it actually works fine
when SPARK_YARN_APP_JAR isn't specified. The client must be serving the jar in
the typi
Hi, Aaron --
I can't speak to issues relevant to Spark, but it looks like json4s is
currently using the Jackson Scala module 2.1.3 and Scala 2.9.2. There have
been quite a few significant changes to the Scala module and underpinnings
between the 2.1.x and 2.3.x series, but I can't speak to how th
Github user mengxr commented on the pull request:
https://github.com/apache/incubator-spark/pull/589#issuecomment-34906057
I will make another PR for the second commit. Next time we should leave the
PR open for a day or half before merge.
1 - 100 of 159 matches
Mail list logo
