Curious, what was the issue ?
- Mridul
On Sun, Feb 23, 2014 at 11:41 PM, Nan Zhu wrote:
> OK, I know where I was wrong
>
>
> Best,
>
> --
> Nan Zhu
> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>
>
> On Sunday, February 23, 2014 at 12:50 PM, Nan Zhu wrote:
>
>> String, it should be ge
e (sparkbank)? Curious to know how you'd decide what
> should go where.
>
> Amandeep
>
> > On Feb 22, 2014, at 10:06 PM, Mridul Muralidharan
> wrote:
> >
> > Hi,
> >
> > Over the past few months, I have seen a bunch of pull requests which
> have
Hi,
Over the past few months, I have seen a bunch of pull requests which have
extended spark api ... most commonly RDD itself.
Most of them are either relatively niche case of specialization (which
might not be useful for most cases) or idioms which can be expressed
(sometimes with minor perf p
I am not sure if this is resolved now - but maven was better at
building the assembly jars compared to sbt.
To the point where I stopped using sbt due to unpredictable order in
which it unjars the dependencies to create the assembled jar (we do
have quite a lot of conflicting classes in our depende
Wonderful news ! Congrats all :-)
Regards,
Mridul
On Feb 20, 2014 10:07 PM, "Andy Konwinski" wrote:
> Congrats Spark community! I think this means we are officially now a TLP!
> -- Forwarded message --
> From: "Brett Porter"
> Date: Feb 19, 2014 11:26 PM
> Subject: ASF Board Mee
ideline. It is invoking a function createFoo, not
> the
> > constructor of Foo.
> >
> >
> >
> >
> > On Wed, Feb 19, 2014 at 10:39 AM, Mridul Muralidharan > >wrote:
> >
> > > Without bikeshedding this too much ... It is likely incorrect (no
e createFoo example?
>
> def myFunc = createFoo
>
> is disallowed in my guideline. It is invoking a function createFoo, not the
> constructor of Foo.
>
>
>
>
> On Wed, Feb 19, 2014 at 10:39 AM, Mridul Muralidharan >wrote:
>
> > Without bikeshedding this too much
t up is not a matter of readability or style. If it
> returns a different type, it should be declared (otherwise it is just
> wrong).
>
>
> On Wed, Feb 19, 2014 at 12:17 AM, Mridul Muralidharan >wrote:
>
> > You are right.
> > A degenerate case would be :
&g
. It was implicit in my email, but the return type
>> should be the same as the class itself.
>>
>>
>>
>>
>> On Tue, Feb 18, 2014 at 10:37 PM, Mridul Muralidharan > >wrote:
>>
>> > Case 3 can be a potential issue.
>> > Curre
Case 3 can be a potential issue.
Current implementation might be returning a concrete class which we
might want to change later - making it a type change.
The intention might be to return an RDD (for example), but the
inferred type might be a subclass of RDD - and future changes will
cause signatur
I had not resolved it in time for 0.9 - but IIRC there was a recent PR
which fixed bugs in spill [1] : are you able to reproduce this with
spark master ?
Regards,
Mridul
[1] https://github.com/apache/incubator-spark/pull/533
On Wed, Feb 19, 2014 at 9:58 AM, Andrew Ash wrote:
> I confirmed also
There is nothing wrong with 9k partitions - I actually use much higher :-) [1]
I have not really seen this interesting issue you mentioned - should
investigate more, thanks for the note !
Regards,
Mridul
[1] I do use insanely high frame size anyway - and my workers/master
run with 8g; maybe why
+1 !
- Mridul
On Tue, Feb 11, 2014 at 9:57 AM, Chris Mattmann wrote:
> Hi Everyone,
>
> This is a new VOTE to decide if Apache Spark should graduate
> from the Incubator. Please VOTE on the resolution pasted below
> the ballot. I'll leave this VOTE open for at least 72 hours.
>
> Thanks!
>
> [ ]
scala/org/apache/spark/scheduler/DAGScheduler.scala#L610
>> > > >>> does
>> > > >> not seem to actually cancel the jobs, only mark them as failed. So
>> it
>> > > >> may be a simple addition.
>> > > >>
>> > > >
This is neat, thanks Reynold !
Regards,
Mridul
On Fri, Feb 7, 2014 at 6:20 AM, Reynold Xin wrote:
> You can do
>
> sbt/sbt assemble-deps
>
>
> and then just run
>
> sbt/sbt package
>
> each time.
>
>
> You can even do
>
> sbt/sbt ~package
>
> for automatic incremental compilation.
>
>
>
> On Thu
f-deleted temp dirs scattered across the cluster.
>
> How feasible do you think it'd be to interrupt the other threads?
>
>
> On Thu, Feb 6, 2014 at 10:54 AM, Mridul Muralidharan wrote:
>
>> Looks like a pathological corner case here - where the the delete
>> thread is
+1
Would be great if the JIRA tag was 'clickable' to go to the actual JIRA :-)
Regards,
Mridul
On Fri, Feb 7, 2014 at 5:35 AM, Patrick Wendell wrote:
> As a break out from the other thread. I'd like to propose two
> guidelines for pull requests. These guidelines are to make things
> easier to
han the cleanup threads can delete from disk.
>
> What do you think of that theory?
>
>
> Andrew
>
>
>
> On Thu, Feb 6, 2014 at 2:30 AM, Mridul Muralidharan
> wrote:
>>
>> shutdown hooks should not take 15 mins are you mentioned !
>> On the other hand, how bus
shutdown hooks should not take 15 mins are you mentioned !
On the other hand, how busy was your disk when this was happening ?
(either due to spark or something else ?)
It might just be that there was a lot of stuff to remove ?
Regards,
Mridul
On Thu, Feb 6, 2014 at 3:50 PM, Andrew Ash wrote:
The reason I explicitly mentioned about binary compatibility was
because it was sort of hand waved in the proposal as good to have.
My understanding is that scala does make it painful to ensure binary
compatibility - but stability of interfaces is vital to ensure
dependable platforms.
Recompilation
Before we move to 1.0, we need to address two things :
a) backward compatibility not just at api level, but also at binary
level (not forcing recompile).
b) minimize external dependencies - some of them would go away/not be
actively maintained.
Regards,
Mridul
On Thu, Feb 6, 2014 at 11:50 AM,
Great news !
+1
Regards,
Mridul
On Fri, Jan 24, 2014 at 4:15 AM, Matei Zaharia wrote:
> Hi folks,
>
> We’ve been working on the transition to Apache for a while, and our last
> shepherd’s report says the following:
>
>
> Spark
>
> Alan Cabrera (acabrera):
>
> Seems like
are included since you raised the issues in prev RC?
>
> Thanks,
>
> Henry
>
> On Sun, Jan 19, 2014 at 10:33 AM, Mridul Muralidharan
> wrote:
>> Oh great, just saw the PR from Matei ... for some odd reason, the dev
>> mails are coming to be horribly delayed.
&g
ptions which had a.b and a.b.c... they
> should all work in the new RC.
>
> On Sun, Jan 19, 2014 at 4:56 AM, Mridul Muralidharan wrote:
>> Chanced upon spill related config which exhibit same pattern ...
>>
>> - Mridul
>>
>> On Sun, Jan 19, 2014 at 1:10 AM,
tei Zaharia
> wrote:
>
>> This is definitely an important issue to fix. Instead of renaming
>> properties, one solution would be to replace Typesafe Config with just
>> reading Java system properties, and disable config files for this release.
>> I kind of like that over
this release.
>> I kind of like that over renaming.
>>
>> Matei
>>
>> On Jan 18, 2014, at 11:30 AM, Mridul Muralidharan
>> wrote:
>>
>> > Hi,
>> >
>> > Speculation was an example, there are others in spark which are
>> &g
tch?
>
> maybe we can restrict that all properties in Spark should be "three levels"
>
>
> On Sat, Jan 18, 2014 at 2:10 PM, Mridul Muralidharan wrote:
>
>> Hi,
>>
>> Unless I am mistaken, the change to using typesafe ConfigFactory has
>> broken s
I would vote -1 for this release until we resolve config property
issue [1] : if there is a known resolution for this (which I could not
find unfortunately, apologies if it exists !), then will change my
vote.
Thanks,
Mridul
[1]
http://apache-spark-developers-list.1001551.n3.nabble.com/Config-p
Hi,
Unless I am mistaken, the change to using typesafe ConfigFactory has
broken some of the system properties we use in spark.
For example: if we have both
-Dspark.speculation=true -Dspark.speculation.multiplier=0.95
set, then the spark.speculation property is dropped.
The rules of parseProper
hadoop2, in this context, is use of spark on a hadoop cluster without
yarn but with hadoop2 interfaces.
hadoop2-yarn uses yarn RM to launch a spark job (and obviously uses
hadoop2 interfaces).
Regards,
Mridul
On Wed, Aug 21, 2013 at 11:52 PM, Konstantin Boudnik wrote:
> For what it worth guys -
Hi Matei,
Not sure if it is already planned, but the write related Connection race
condition Patrick reported/fixed might need to go to 0.8 also ...
Regards
Mridul
On Aug 9, 2013 11:12 PM, "Matei Zaharia" wrote:
> Hi folks,
>
> In order to make the 0.8 release soon, I've created a new branch
31 matches
Mail list logo
