Re: Java 9

at 11:46 Sean Owen <so...@cloudera.com> wrote: > I don't think anyone's tried it. I think we'd first have to agree to drop > Java 7 support before that could be seriously considered. The 8-9 > difference is a bit more of a breaking change. > > On Tue, Feb 7, 2017 at 11:44 AM

Java 9

Is anyone working on support for running Spark on Java 9? Is this in a roadmap anywhere? Cheers,

Re: Anyone seeing a lot of Spark emails go to Gmail spam?

I have gmail filters to add labels and skip inbox for anything sent to dev@spark user@spark etc but still get the occasional message marked as spam On Wed, 2 Nov 2016 at 08:18 Sean Owen wrote: > I couldn't figure out why I was missing a lot of dev@ announcements, and > have

Re: [VOTE] Release Apache Spark 1.6.3 (RC1)

We see a regression since 1.6.2. I think this PR needs to be backported https://github.com/apache/spark/pull/13784 which resolves SPARK-16078. The PR that causes the issue (for SPARK-15613) was reverted just before 1.6.2 release then re-applied afterwards but this fix was only backported to 2.0.

Re: Monitoring system extensibility

he existing API maintainable? E.g. Is it OK to just expose coda > hale metrics in the API? Do we need to worry about dependency conflicts? > Should we wrap it? > > 2. Is the existing API sufficiently general (to cover use cases)? What > about security related setup? > > >

Re: Monitoring system extensibility

thinking about whether those APIs are sufficiently expressive and > maintainable is not a great way to design APIs in general. > > On Friday, October 7, 2016, Pete Robbins <robbin...@gmail.com> wrote: > > I brought this up last year and there was a Jira raised: > https://issues.apa

Re: Monitoring system extensibility

I brought this up last year and there was a Jira raised: https://issues.apache.org/jira/browse/SPARK-14151 For now I just have my SInk and Source in an o.a.s package name which is not ideal but the only way round this. On Fri, 7 Oct 2016 at 08:30 Reynold Xin wrote: > They

Re: Spark 2.0.0 preview docs uploaded

park.apache.org/news/spark-2.0.0-preview.html > > On Thu, Jun 9, 2016 at 7:33 AM, Pete Robbins <robbin...@gmail.com> wrote: > > It would be nice to have a "what's new in 2.0.0" equivalent to > > https://spark.apache.org/releases/spark-release-1-6-0.html available or > am

Stability of branch-2.0

It looks like the vote on 2.0-rc2 will not pass so there will be a new RC from the 2.0 branch. With a project management hat on I would expect to see only fixes to the remaining blocker issues or high priority bug fixes going into the 2.0 branch as defect burn down. However, I see several new

Re: branch-2.0 build failure

Ok, thanks. I'll await it appearing. On Thu, 30 Jun 2016 at 14:51 Sean Owen <so...@cloudera.com> wrote: > TD has literally just merged the fix. > > On Thu, Jun 30, 2016 at 2:37 PM, Pete Robbins <robbin...@gmail.com> wrote: > > Our build on branch-2.0 is failing afte

branch-2.0 build failure

Our build on branch-2.0 is failing after the PR for updating kafka to 0.10. The new kafka pom.xml files are naming the parent version as 2.0.0-SNAPSHOT but the branch 2.0 poms have been updated to 2.0.1-SNAPSHOT after the rc1 cut. Shouldn't the pom versions remain as 2.0.0-SNAPSHOT until a 2.0.0

Re: [VOTE] Release Apache Spark 2.0.0 (RC1)

I'm also seeing some of these same failures: - spilling with compression *** FAILED *** I have seen this occassionaly - to UTC timestamp *** FAILED *** This was fixed yesterday in branch-2.0 ( https://issues.apache.org/jira/browse/SPARK-16078) - offset recovery *** FAILED *** Haven't seen this

Re: [VOTE] Release Apache Spark 1.6.2 (RC2)

This has failed on our 1.6 stream builds regularly. ( https://issues.apache.org/jira/browse/SPARK-6005) looks fixed in 2.0? On Wed, 22 Jun 2016 at 11:15 Sean Owen wrote: > Oops, one more in the "does anybody else see this" department: > > - offset recovery *** FAILED *** >

Re: [VOTE] Release Apache Spark 1.6.2 (RC2)

g blocking methods in the event > loops, so similar issues are still there even after applying this patch. > Hence, I don't think it's a blocker for 1.6.2. > > On Tue, Jun 21, 2016 at 2:57 AM, Pete Robbins <robbin...@gmail.com> wrote: > >> The PR (https://github.com/apache/spark/pu

Re: [VOTE] Release Apache Spark 1.6.2 (RC2)

The PR (https://github.com/apache/spark/pull/13055) to fix https://issues.apache.org/jira/browse/SPARK-15262 was applied to 1.6.2 however this fix caused another issue https://issues.apache.org/jira/browse/SPARK-15606 the fix for which ( https://github.com/apache/spark/pull/13355) has not been

Re: Spark 2.0.0 preview docs uploaded

It would be nice to have a "what's new in 2.0.0" equivalent to https://spark.apache.org/releases/spark-release-1-6-0.html available or am I just missing it? On Wed, 8 Jun 2016 at 13:15 Sean Owen wrote: > OK, this is done: > > http://spark.apache.org/documentation.html >

Re: NegativeArraySizeException / segfault

I just raised https://issues.apache.org/jira/browse/SPARK-15822 for a similar looking issue. Analyzing the core dump from the segv with Memory Analyzer it looks very much like a UTF8String is very corrupt. Cheers, On Fri, 27 May 2016 at 21:00 Koert Kuipers wrote: > hello

Re: [ANNOUNCE] Spark branch-2.0

https://issues.apache.org/jira/browse/SPARK-13745 is really a defect and a blocker unless it is the decision to drop support for Big Endian platforms. The PR has been reviewed and tested and I strongly believe this needs to be targeted for 2.0. On Mon, May 2, 2016 at 12:00 AM Reynold Xin

Re: Code freeze?

Is there a list of Jiras to be considered for 2.0? I would really like to get https://issues.apache.org/jira/browse/SPARK-13745 in so that Big Endian platforms are not broken. Cheers, On Wed, 13 Apr 2016 at 08:51 Reynold Xin wrote: > I think the main things are API things

Re: Can we remove private[spark] from Metrics Source and SInk traits?

these into core Spark. Opening up the Sink/Source interfaces would at least allow these to exist somewhere such as spark-packages without having to pollute the o.a.s namespace On Sat, 19 Mar 2016 at 13:05 Gerard Maas <gerard.m...@gmail.com> wrote: > +1 > On Mar 19, 2016 08:33, &

Can we remove private[spark] from Metrics Source and SInk traits?

This seems to me to be unnecessarily restrictive. These are very useful extension points for adding 3rd party sources and sinks. I intend to make an Elasticsearch sink available on spark-packages but this will require a single class, the sink, to be in the org.apache.spark package tree. I could

Re: Accessing SparkConf in metrics sink

So the answer to my previous question is NO. It looks like I could use SparkEnv.get.conf but * * NOTE: This is not intended for external use. This is exposed for Shark and may be made private * in a future release. */ On Wed, 16 Mar 2016 at 08:22 Pete Robbins <robbin...@gmail.com>

Re: Accessing SparkConf in metrics sink

OK thanks. Does that work in an executor? On Wed, 16 Mar 2016 at 07:58 Reynold Xin <r...@databricks.com> wrote: > SparkConf is not a singleton. > > However, SparkContext in almost all cases are. So you can use > SparkContext.getOrCreate().getConf > > On Wed, Mar 16

Accessing SparkConf in metrics sink

I'm writing a metrics sink and reporter to push metrics to Elasticsearch. An example format of a metric in JSON: { "timestamp": "2016-03-15T16:11:19.314+", "hostName": "10.192.0.87" "applicationName": "My application", "applicationId": "app-20160315093931-0003", "executorId": "17",

Re: SparkConf constructor now private

Is the SparkConf effectively a singleton? Could there be a Utils method to return a clone of the SparkConf? Cheers On Tue, 15 Mar 2016 at 16:49 Marcelo Vanzin wrote: > Oh, my bad. I think I left that from a previous part of the patch and > forgot to revert it. Will fix. >

Re: SparkOscope: Enabling Spark Optimization through Cross-stack Monitoring and Visualization

Yiannis, I'm interested in what you've done here as I was looking for ways to allow the Spark UI to display custom metrics in a pluggable way without having to modify the Spark source code. It would be good to see if we could have modify your code to add extension points into the UI so we could

Fwd: Elasticsearch sink for metrics

ing to see if this was already implemented before continuing. On 15 January 2016 at 09:18, Nick Pentreath <nick.pentre...@gmail.com> wrote: > I haven't come across anything, but could you provide more detail on what > issues you're encountering? > > > > On Fri, Jan 15, 2016 at 11:09 AM,

Elasticsearch sink for metrics

Has anyone tried pushing Spark metrics into elasticsearch? We have other metrics, eg some runtime information, going into ES and would like to be able to combine this with the Spark metrics for visualization with Kibana. I experimented with a new sink using ES's ElasticsearchReporter for the Coda

Re: Unable to acquire memory errors in HiveCompatibilitySuite

f tasks -- but it turned out to be a bad approximation in tests > because it is set to 32 to increase concurrency. > > > On Tue, Sep 15, 2015 at 10:47 PM, Pete Robbins <robbin...@gmail.com> > wrote: > >> Oops... I meant to say "The page size calculation is NOT the is

Re: Unable to acquire memory errors in HiveCompatibilitySuite

he number of active tasks. If a task is launched before > others and hogs a lot of memory quickly, the other tasks that are launched > after it might not be able to get enough memory allocation, and thus will > fail. This is not super ideal, but probably fine because tasks can be > retried, and

Re: Unable to acquire memory errors in HiveCompatibilitySuite

so forcing the ShuffleMemoryManager to assume 32 cores and therefore calculate a pagesize of 1MB passes the tests. How can we determine the correct value to use in getPageSize rather than Runtime.getRuntime.availableProcessors()? On 16 September 2015 at 10:17, Pete Robbins <robbin...@gmail.

Re: Unable to acquire memory errors in HiveCompatibilitySuite

Oops... I meant to say "The page size calculation is NOT the issue here" On 16 September 2015 at 06:46, Pete Robbins <robbin...@gmail.com> wrote: > The page size calculation is the issue here as there is plenty of free > memory, although there is maybe a fair bit of wast

Re: Unable to acquire memory errors in HiveCompatibilitySuite

euristics in memory calculation to use > SparkContext.defaultParallelism if it is local mode. > > > On Tue, Sep 15, 2015 at 10:28 AM, Pete Robbins <robbin...@gmail.com> > wrote: > >> Yes and at least there is an override by setting spark.sql.test.master >> to local[8

Re: Unable to acquire memory errors in HiveCompatibilitySuite

than 4MB. >> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/shuffle/ShuffleMemoryManager.scala#L174 >> >> Maybe there is a place that in the maven tests that we explicitly set the >> page size (spark.buffer.pageSize) to 4MB? If yes, we need to find it and &

Re: Unable to acquire memory errors in HiveCompatibilitySuite

gt; Yea I think this is where the heuristics is failing -- it uses 8 cores to > approximate the number of active tasks, but the tests somehow is using 32 > (maybe because it explicitly sets it to that, or you set it yourself? I'm > not sure which one) > > On Mon, Sep 14, 2015 at 11:06

Re: Unable to acquire memory errors in HiveCompatibilitySuite

xplicitly sets it to that, or you set it yourself? I'm > > not sure which one) > > > > On Mon, Sep 14, 2015 at 11:06 PM, Pete Robbins <robbin...@gmail.com> > wrote: > >> > >> Reynold, thanks for replying. > >> > >> getPageSize parameters: maxMemo

Re: Unable to acquire memory errors in HiveCompatibilitySuite

) so maybe that should be changed to limit threads to num cores? Cheers, On 15 September 2015 at 08:50, Pete Robbins <robbin...@gmail.com> wrote: > Ok so it looks like the max number of active tasks reaches 30. I'm not > setting anything as it is a clean environment with clean spark

Unable to acquire memory errors in HiveCompatibilitySuite

I keep hitting errors running the tests on 1.5 such as - join31 *** FAILED *** Failed to execute query using catalyst: Error: Job aborted due to stage failure: Task 9 in stage 3653.0 failed 1 times, most recent failure: Lost task 9.0 in stage 3653.0 (TID 123363, localhost):

Re: Flaky test in DAGSchedulerSuite?

raised https://issues.apache.org/jira/browse/SPARK-10454 and PR On 4 September 2015 at 21:24, Pete Robbins <robbin...@gmail.com> wrote: > I've also just hit this and was about to raise a JIRA for this if there > isn't one already. I have a simple fix. > > On 4 September 2015

Re: Flaky test in DAGSchedulerSuite?

I've also just hit this and was about to raise a JIRA for this if there isn't one already. I have a simple fix. On 4 September 2015 at 19:09, Cheolsoo Park wrote: > Hi devs, > > I noticed this test case fails intermittently in Jenkins. > > For eg, see the following builds-

Re: Reliance on java.math.BigInteger implementation

That would be me then ;-) I'm working on a patch. Cheers, On 14 August 2015 at 23:43, Reynold Xin r...@databricks.com wrote: I pinged the IBM team to submit a patch that would work on IBM JVM. On Fri, Aug 14, 2015 at 11:27 AM, Pete Robbins robbin...@gmail.com wrote: ref: https

Reliance on java.math.BigInteger implementation

ref: https://issues.apache.org/jira/browse/SPARK-9370 The code to handle BigInteger types in org.apache.spark.sql.catalyst.expressions.UnsafeRowWriters.java and org.apache.spark.unsafe.Platform.java is dependant on the implementation of java.math.BigInteger eg: try {