at 11:46 Sean Owen wrote:
> I don't think anyone's tried it. I think we'd first have to agree to drop
> Java 7 support before that could be seriously considered. The 8-9
> difference is a bit more of a breaking change.
>
> On Tue, Feb 7, 2017 at 11:44 AM Pete Robbins
Is anyone working on support for running Spark on Java 9? Is this in a
roadmap anywhere?
Cheers,
I have gmail filters to add labels and skip inbox for anything sent to
dev@spark user@spark etc but still get the occasional message marked as spam
On Wed, 2 Nov 2016 at 08:18 Sean Owen wrote:
> I couldn't figure out why I was missing a lot of dev@ announcements, and
> have just realized hundre
We see a regression since 1.6.2. I think this PR needs to be backported
https://github.com/apache/spark/pull/13784 which resolves SPARK-16078. The
PR that causes the issue (for SPARK-15613) was reverted just before 1.6.2
release then re-applied afterwards but this fix was only backported to 2.0.
T
I maintainable? E.g. Is it OK to just expose coda
> hale metrics in the API? Do we need to worry about dependency conflicts?
> Should we wrap it?
>
> 2. Is the existing API sufficiently general (to cover use cases)? What
> about security related setup?
>
>
>
>
> On Fri,
APIs are sufficiently expressive and
> maintainable is not a great way to design APIs in general.
>
> On Friday, October 7, 2016, Pete Robbins wrote:
>
> I brought this up last year and there was a Jira raised:
> https://issues.apache.org/jira/browse/SPARK-14151
>
> For no
I brought this up last year and there was a Jira raised:
https://issues.apache.org/jira/browse/SPARK-14151
For now I just have my SInk and Source in an o.a.s package name which is
not ideal but the only way round this.
On Fri, 7 Oct 2016 at 08:30 Reynold Xin wrote:
> They have always been priva
/news/spark-2.0.0-preview.html
>
> On Thu, Jun 9, 2016 at 7:33 AM, Pete Robbins wrote:
> > It would be nice to have a "what's new in 2.0.0" equivalent to
> > https://spark.apache.org/releases/spark-release-1-6-0.html available or
> am I
> > just miss
It looks like the vote on 2.0-rc2 will not pass so there will be a new RC
from the 2.0 branch. With a project management hat on I would expect to see
only fixes to the remaining blocker issues or high priority bug fixes going
into the 2.0 branch as defect burn down. However, I see several new
funct
Ok, thanks. I'll await it appearing.
On Thu, 30 Jun 2016 at 14:51 Sean Owen wrote:
> TD has literally just merged the fix.
>
> On Thu, Jun 30, 2016 at 2:37 PM, Pete Robbins wrote:
> > Our build on branch-2.0 is failing after the PR for updating kafka to
> 0.10.
> >
Our build on branch-2.0 is failing after the PR for updating kafka to 0.10.
The new kafka pom.xml files are naming the parent version as 2.0.0-SNAPSHOT
but the branch 2.0 poms have been updated to 2.0.1-SNAPSHOT after the rc1
cut. Shouldn't the pom versions remain as 2.0.0-SNAPSHOT until a 2.0.0 ha
I'm also seeing some of these same failures:
- spilling with compression *** FAILED ***
I have seen this occassionaly
- to UTC timestamp *** FAILED ***
This was fixed yesterday in branch-2.0 (
https://issues.apache.org/jira/browse/SPARK-16078)
- offset recovery *** FAILED ***
Haven't seen this f
This has failed on our 1.6 stream builds regularly. (
https://issues.apache.org/jira/browse/SPARK-6005) looks fixed in 2.0?
On Wed, 22 Jun 2016 at 11:15 Sean Owen wrote:
> Oops, one more in the "does anybody else see this" department:
>
> - offset recovery *** FAILED ***
> recoveredOffsetRange
e event
> loops, so similar issues are still there even after applying this patch.
> Hence, I don't think it's a blocker for 1.6.2.
>
> On Tue, Jun 21, 2016 at 2:57 AM, Pete Robbins wrote:
>
>> The PR (https://github.com/apache/spark/pull/13055) to fix
>> https:
The PR (https://github.com/apache/spark/pull/13055) to fix
https://issues.apache.org/jira/browse/SPARK-15262 was applied to 1.6.2
however this fix caused another issue
https://issues.apache.org/jira/browse/SPARK-15606 the fix for which (
https://github.com/apache/spark/pull/13355) has not been back
It would be nice to have a "what's new in 2.0.0" equivalent to
https://spark.apache.org/releases/spark-release-1-6-0.html available or am
I just missing it?
On Wed, 8 Jun 2016 at 13:15 Sean Owen wrote:
> OK, this is done:
>
> http://spark.apache.org/documentation.html
> http://spark.apache.org/d
I just raised https://issues.apache.org/jira/browse/SPARK-15822 for a
similar looking issue. Analyzing the core dump from the segv with Memory
Analyzer it looks very much like a UTF8String is very corrupt.
Cheers,
On Fri, 27 May 2016 at 21:00 Koert Kuipers wrote:
> hello all,
> after getting ou
https://issues.apache.org/jira/browse/SPARK-13745
is really a defect and a blocker unless it is the decision to drop support
for Big Endian platforms. The PR has been reviewed and tested and I
strongly believe this needs to be targeted for 2.0.
On Mon, May 2, 2016 at 12:00 AM Reynold Xin wrote:
Is there a list of Jiras to be considered for 2.0? I would really like to
get https://issues.apache.org/jira/browse/SPARK-13745 in so that Big Endian
platforms are not broken.
Cheers,
On Wed, 13 Apr 2016 at 08:51 Reynold Xin wrote:
> I think the main things are API things that we need to get ri
grate these into core Spark. Opening up the Sink/Source interfaces
would at least allow these to exist somewhere such as spark-packages
without having to pollute the o.a.s namespace
On Sat, 19 Mar 2016 at 13:05 Gerard Maas wrote:
> +1
> On Mar 19, 2016 08:33, "Pete Robbins" wrot
This seems to me to be unnecessarily restrictive. These are very useful
extension points for adding 3rd party sources and sinks.
I intend to make an Elasticsearch sink available on spark-packages but this
will require a single class, the sink, to be in the org.apache.spark
package tree. I could su
So the answer to my previous question is NO.
It looks like I could use SparkEnv.get.conf but
* * NOTE: This is not intended for external use. This is exposed for Shark
and may be made private * in a future release. */
On Wed, 16 Mar 2016 at 08:22 Pete Robbins wrote:
> OK thanks. Does t
OK thanks. Does that work in an executor?
On Wed, 16 Mar 2016 at 07:58 Reynold Xin wrote:
> SparkConf is not a singleton.
>
> However, SparkContext in almost all cases are. So you can use
> SparkContext.getOrCreate().getConf
>
> On Wed, Mar 16, 2016 at 12:38 AM, Pete Robbins
I'm writing a metrics sink and reporter to push metrics to Elasticsearch.
An example format of a metric in JSON:
{
"timestamp": "2016-03-15T16:11:19.314+",
"hostName": "10.192.0.87"
"applicationName": "My application",
"applicationId": "app-20160315093931-0003",
"executorId": "17",
"exec
Is the SparkConf effectively a singleton? Could there be a Utils method to
return a clone of the SparkConf?
Cheers
On Tue, 15 Mar 2016 at 16:49 Marcelo Vanzin wrote:
> Oh, my bad. I think I left that from a previous part of the patch and
> forgot to revert it. Will fix.
>
> On Tue, Mar 15, 2016
Yiannis,
I'm interested in what you've done here as I was looking for ways to allow
the Spark UI to display custom metrics in a pluggable way without having to
modify the Spark source code. It would be good to see if we could have
modify your code to add extension points into the UI so we could co
see if this
was already implemented before continuing.
On 15 January 2016 at 09:18, Nick Pentreath
wrote:
> I haven't come across anything, but could you provide more detail on what
> issues you're encountering?
>
>
>
> On Fri, Jan 15, 2016 at 11:09 AM, Pete Robbins
&
Has anyone tried pushing Spark metrics into elasticsearch? We have other
metrics, eg some runtime information, going into ES and would like to be
able to combine this with the Spark metrics for visualization with Kibana.
I experimented with a new sink using ES's ElasticsearchReporter for the
Coda
so forcing the ShuffleMemoryManager to assume 32 cores and therefore
calculate a pagesize of 1MB passes the tests.
How can we determine the correct value to use in getPageSize rather than
Runtime.getRuntime.availableProcessors()?
On 16 September 2015 at 10:17, Pete Robbins wrote:
> I see w
. If a task is launched before
> others and hogs a lot of memory quickly, the other tasks that are launched
> after it might not be able to get enough memory allocation, and thus will
> fail. This is not super ideal, but probably fine because tasks can be
> retried, and can succeed in retr
-- but it turned out to be a bad approximation in tests
> because it is set to 32 to increase concurrency.
>
>
> On Tue, Sep 15, 2015 at 10:47 PM, Pete Robbins
> wrote:
>
>> Oops... I meant to say "The page size calculation is NOT the issue here"
>>
>> On 16
Oops... I meant to say "The page size calculation is NOT the issue here"
On 16 September 2015 at 06:46, Pete Robbins wrote:
> The page size calculation is the issue here as there is plenty of free
> memory, although there is maybe a fair bit of wasted space in some pages.
>
to use
> SparkContext.defaultParallelism if it is local mode.
>
>
> On Tue, Sep 15, 2015 at 10:28 AM, Pete Robbins
> wrote:
>
>> Yes and at least there is an override by setting spark.sql.test.master
>> to local[8] , in fact local[16] worked on my 8 core box.
>>
&
14, 2015 at 11:22 PM, Reynold Xin wrote:
> > Yea I think this is where the heuristics is failing -- it uses 8 cores to
> > approximate the number of active tasks, but the tests somehow is using 32
> > (maybe because it explicitly sets it to that, or you set it yourself? I'm
>
) so maybe that should be changed to limit
threads to num cores?
Cheers,
On 15 September 2015 at 08:50, Pete Robbins wrote:
> Ok so it looks like the max number of active tasks reaches 30. I'm not
> setting anything as it is a clean environment with clean spark code
> checkout. I
is where the heuristics is failing -- it uses 8 cores to
> approximate the number of active tasks, but the tests somehow is using 32
> (maybe because it explicitly sets it to that, or you set it yourself? I'm
> not sure which one)
>
> On Mon, Sep 14, 2015 at 11:06 PM, Pete Robbi
r/core/src/main/scala/org/apache/spark/shuffle/ShuffleMemoryManager.scala#L174
>>
>> Maybe there is a place that in the maven tests that we explicitly set the
>> page size (spark.buffer.pageSize) to 4MB? If yes, we need to find it and
>> just remove it.
>>
>>
>&
I keep hitting errors running the tests on 1.5 such as
- join31 *** FAILED ***
Failed to execute query using catalyst:
Error: Job aborted due to stage failure: Task 9 in stage 3653.0 failed 1
times, most recent failure: Lost task 9.0 in stage 3653.0 (TID 123363,
localhost): java.io.IOExceptio
raised https://issues.apache.org/jira/browse/SPARK-10454 and PR
On 4 September 2015 at 21:24, Pete Robbins wrote:
> I've also just hit this and was about to raise a JIRA for this if there
> isn't one already. I have a simple fix.
>
> On 4 September 2015 at 19:09, Cheolso
I've also just hit this and was about to raise a JIRA for this if there
isn't one already. I have a simple fix.
On 4 September 2015 at 19:09, Cheolsoo Park wrote:
> Hi devs,
>
> I noticed this test case fails intermittently in Jenkins.
>
> For eg, see the following builds-
> https://amplab.cs.be
That would be me then ;-)
I'm working on a patch.
Cheers,
On 14 August 2015 at 23:43, Reynold Xin wrote:
> I pinged the IBM team to submit a patch that would work on IBM JVM.
>
>
> On Fri, Aug 14, 2015 at 11:27 AM, Pete Robbins
> wrote:
>
>> ref: https://issues
ref: https://issues.apache.org/jira/browse/SPARK-9370
The code to handle BigInteger types in
org.apache.spark.sql.catalyst.expressions.UnsafeRowWriters.java
and
org.apache.spark.unsafe.Platform.java
is dependant on the implementation of java.math.BigInteger
eg:
try {
signumOffs
42 matches
Mail list logo
