[darktable-dev] Re: Online documentation at version 2.4

Apologies, I just saw another thread already raising this. Teaching to look for "documentation" rather than "manual" Cheers On Fri, 1 Mar 2019, 14:20 Pete Robbins, wrote: > Hi, I've just started using darktable 2.6 (impressed so far). > > I noticed

[darktable-dev] Online documentation at version 2.4

Hi, I've just started using darktable 2.6 (impressed so far). I noticed the online manual here: https://www.darktable.org/usermanual/en/ is at version 2.4. Should this be updated to 2.6? I believe the download pdf is v2.6 Cheers, Pete

[jira] [Created] (SPARK-19710) Test Failures in SQLQueryTests on big endian platforms

Pete Robbins created SPARK-19710: Summary: Test Failures in SQLQueryTests on big endian platforms Key: SPARK-19710 URL: https://issues.apache.org/jira/browse/SPARK-19710 Project: Spark Issue

Re: Java 9

at 11:46 Sean Owen <so...@cloudera.com> wrote: > I don't think anyone's tried it. I think we'd first have to agree to drop > Java 7 support before that could be seriously considered. The 8-9 > difference is a bit more of a breaking change. > > On Tue, Feb 7, 2017 at 11:44 AM

Java 9

Is anyone working on support for running Spark on Java 9? Is this in a roadmap anywhere? Cheers,

[jira] [Created] (SPARK-18963) Test Failuire on big endian; o.a.s.unsafe.types.UTF8StringSuite.writeToOutputStreamIntArray

Pete Robbins created SPARK-18963: Summary: Test Failuire on big endian; o.a.s.unsafe.types.UTF8StringSuite.writeToOutputStreamIntArray Key: SPARK-18963 URL: https://issues.apache.org/jira/browse/SPARK-18963

Re: Anyone seeing a lot of Spark emails go to Gmail spam?

I have gmail filters to add labels and skip inbox for anything sent to dev@spark user@spark etc but still get the occasional message marked as spam On Wed, 2 Nov 2016 at 08:18 Sean Owen wrote: > I couldn't figure out why I was missing a lot of dev@ announcements, and > have

Re: [VOTE] Release Apache Spark 1.6.3 (RC1)

We see a regression since 1.6.2. I think this PR needs to be backported https://github.com/apache/spark/pull/13784 which resolves SPARK-16078. The PR that causes the issue (for SPARK-15613) was reverted just before 1.6.2 release then re-applied afterwards but this fix was only backported to 2.0.

[jira] [Commented] (SPARK-17827) StatisticsColumnSuite failures on big endian platforms

[ https://issues.apache.org/jira/browse/SPARK-17827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15571890#comment-15571890 ] Pete Robbins commented on SPARK-17827: -- I have a PR ready which I will submit as soon as I have run

[jira] [Commented] (SPARK-17827) StatisticsColumnSuite failures on big endian platforms

[ https://issues.apache.org/jira/browse/SPARK-17827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15571546#comment-15571546 ] Pete Robbins commented on SPARK-17827: -- right so in these two cases maxLength

[jira] [Comment Edited] (SPARK-17827) StatisticsColumnSuite failures on big endian platforms

[ https://issues.apache.org/jira/browse/SPARK-17827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15571348#comment-15571348 ] Pete Robbins edited comment on SPARK-17827 at 10/13/16 9:11 AM

[jira] [Comment Edited] (SPARK-17827) StatisticsColumnSuite failures on big endian platforms

[ https://issues.apache.org/jira/browse/SPARK-17827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15571348#comment-15571348 ] Pete Robbins edited comment on SPARK-17827 at 10/13/16 9:10 AM

[jira] [Commented] (SPARK-17827) StatisticsColumnSuite failures on big endian platforms

[ https://issues.apache.org/jira/browse/SPARK-17827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15571348#comment-15571348 ] Pete Robbins commented on SPARK-17827: -- In Statistics,scala case class StringColumnStat(statRow

[jira] [Commented] (SPARK-17827) StatisticsColumnSuite failures on big endian platforms

[ https://issues.apache.org/jira/browse/SPARK-17827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15569048#comment-15569048 ] Pete Robbins commented on SPARK-17827: -- So this looks like the max field is being written as an Int

[jira] [Commented] (SPARK-17827) StatisticsColumnSuite failures on big endian platforms

[ https://issues.apache.org/jira/browse/SPARK-17827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15561849#comment-15561849 ] Pete Robbins commented on SPARK-17827: -- [~ZenWzh] Any ideas what code introduced that could cause

Re: Monitoring system extensibility

he existing API maintainable? E.g. Is it OK to just expose coda > hale metrics in the API? Do we need to worry about dependency conflicts? > Should we wrap it? > > 2. Is the existing API sufficiently general (to cover use cases)? What > about security related setup? > > >

[jira] [Commented] (SPARK-17827) StatisticsColumnSuite failures on big endian platforms

[ https://issues.apache.org/jira/browse/SPARK-17827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15554825#comment-15554825 ] Pete Robbins commented on SPARK-17827: -- I'm investigating this > StatisticsColumnSuite failu

[jira] [Created] (SPARK-17827) StatisticsColumnSuite failures on big endian platforms

Pete Robbins created SPARK-17827: Summary: StatisticsColumnSuite failures on big endian platforms Key: SPARK-17827 URL: https://issues.apache.org/jira/browse/SPARK-17827 Project: Spark Issue

Re: Monitoring system extensibility

thinking about whether those APIs are sufficiently expressive and > maintainable is not a great way to design APIs in general. > > On Friday, October 7, 2016, Pete Robbins <robbin...@gmail.com> wrote: > > I brought this up last year and there was a Jira raised: > https://issues.apa

Re: Monitoring system extensibility

I brought this up last year and there was a Jira raised: https://issues.apache.org/jira/browse/SPARK-14151 For now I just have my SInk and Source in an o.a.s package name which is not ideal but the only way round this. On Fri, 7 Oct 2016 at 08:30 Reynold Xin wrote: > They

Re: Spark 2.0.0 preview docs uploaded

park.apache.org/news/spark-2.0.0-preview.html > > On Thu, Jun 9, 2016 at 7:33 AM, Pete Robbins <robbin...@gmail.com> wrote: > > It would be nice to have a "what's new in 2.0.0" equivalent to > > https://spark.apache.org/releases/spark-release-1-6-0.html available or > am

Stability of branch-2.0

It looks like the vote on 2.0-rc2 will not pass so there will be a new RC from the 2.0 branch. With a project management hat on I would expect to see only fixes to the remaining blocker issues or high priority bug fixes going into the 2.0 branch as defect burn down. However, I see several new

Re: branch-2.0 build failure

Ok, thanks. I'll await it appearing. On Thu, 30 Jun 2016 at 14:51 Sean Owen <so...@cloudera.com> wrote: > TD has literally just merged the fix. > > On Thu, Jun 30, 2016 at 2:37 PM, Pete Robbins <robbin...@gmail.com> wrote: > > Our build on branch-2.0 is failing afte

branch-2.0 build failure

Our build on branch-2.0 is failing after the PR for updating kafka to 0.10. The new kafka pom.xml files are naming the parent version as 2.0.0-SNAPSHOT but the branch 2.0 poms have been updated to 2.0.1-SNAPSHOT after the rc1 cut. Shouldn't the pom versions remain as 2.0.0-SNAPSHOT until a 2.0.0

Re: [VOTE] Release Apache Spark 2.0.0 (RC1)

I'm also seeing some of these same failures: - spilling with compression *** FAILED *** I have seen this occassionaly - to UTC timestamp *** FAILED *** This was fixed yesterday in branch-2.0 ( https://issues.apache.org/jira/browse/SPARK-16078) - offset recovery *** FAILED *** Haven't seen this

Re: [VOTE] Release Apache Spark 1.6.2 (RC2)

This has failed on our 1.6 stream builds regularly. ( https://issues.apache.org/jira/browse/SPARK-6005) looks fixed in 2.0? On Wed, 22 Jun 2016 at 11:15 Sean Owen wrote: > Oops, one more in the "does anybody else see this" department: > > - offset recovery *** FAILED *** >

Re: [VOTE] Release Apache Spark 1.6.2 (RC2)

g blocking methods in the event > loops, so similar issues are still there even after applying this patch. > Hence, I don't think it's a blocker for 1.6.2. > > On Tue, Jun 21, 2016 at 2:57 AM, Pete Robbins <robbin...@gmail.com> wrote: > >> The PR (https://github.com/apache/spark/pu

Re: [VOTE] Release Apache Spark 1.6.2 (RC2)

The PR (https://github.com/apache/spark/pull/13055) to fix https://issues.apache.org/jira/browse/SPARK-15262 was applied to 1.6.2 however this fix caused another issue https://issues.apache.org/jira/browse/SPARK-15606 the fix for which ( https://github.com/apache/spark/pull/13355) has not been

[jira] [Updated] (SPARK-15822) segmentation violation in o.a.s.unsafe.types.UTF8String

[ https://issues.apache.org/jira/browse/SPARK-15822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pete Robbins updated SPARK-15822: - Component/s: SQL > segmentation violation in o.a.s.unsafe.types.UTF8Str

[jira] [Commented] (SPARK-15822) segmentation violation in o.a.s.unsafe.types.UTF8String

[ https://issues.apache.org/jira/browse/SPARK-15822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15333594#comment-15333594 ] Pete Robbins commented on SPARK-15822: -- Tracking where the memory is being allocated

[jira] [Comment Edited] (SPARK-15822) segmentation violation in o.a.s.unsafe.types.UTF8String

[ https://issues.apache.org/jira/browse/SPARK-15822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15333594#comment-15333594 ] Pete Robbins edited comment on SPARK-15822 at 6/16/16 11:23 AM: Tracking

[jira] [Comment Edited] (SPARK-15822) segmentation violation in o.a.s.unsafe.types.UTF8String

[ https://issues.apache.org/jira/browse/SPARK-15822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15333586#comment-15333586 ] Pete Robbins edited comment on SPARK-15822 at 6/16/16 11:18 AM: OK so I

[jira] [Commented] (SPARK-15822) segmentation violation in o.a.s.unsafe.types.UTF8String

[ https://issues.apache.org/jira/browse/SPARK-15822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15333586#comment-15333586 ] Pete Robbins commented on SPARK-15822: -- OK so I think I know what is happening! In the following

[jira] [Commented] (SPARK-15822) segmentation violation in o.a.s.unsafe.types.UTF8String

[ https://issues.apache.org/jira/browse/SPARK-15822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15333462#comment-15333462 ] Pete Robbins commented on SPARK-15822: -- Tracing through off heap memory allocation this looks like

[jira] [Commented] (SPARK-15822) segmentation violation in o.a.s.unsafe.types.UTF8String

[ https://issues.apache.org/jira/browse/SPARK-15822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15331580#comment-15331580 ] Pete Robbins commented on SPARK-15822: -- I can also recreate this issue on Oracle JDK 1.8: {noformat

[jira] [Commented] (SPARK-15822) segmentation violation in o.a.s.unsafe.types.UTF8String

[ https://issues.apache.org/jira/browse/SPARK-15822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15331375#comment-15331375 ] Pete Robbins commented on SPARK-15822: -- and the plan: {noformat} == Parsed Logical Plan == 'Project

[jira] [Commented] (SPARK-15822) segmentation violation in o.a.s.unsafe.types.UTF8String

[ https://issues.apache.org/jira/browse/SPARK-15822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15331370#comment-15331370 ] Pete Robbins commented on SPARK-15822: -- Chatting with [~hvanhovell] here is the current state. I can

[jira] [Commented] (SPARK-15822) segmentation violation in o.a.s.unsafe.types.UTF8String

[ https://issues.apache.org/jira/browse/SPARK-15822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15331371#comment-15331371 ] Pete Robbins commented on SPARK-15822: -- The generated code is: {code} Top Arrival Carrier

[jira] [Commented] (SPARK-15822) segmentation violation in o.a.s.unsafe.types.UTF8String

[ https://issues.apache.org/jira/browse/SPARK-15822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15329474#comment-15329474 ] Pete Robbins commented on SPARK-15822: -- modified app to remove .cache()'s and still get a segv

[jira] [Commented] (SPARK-15822) segmentation violation in o.a.s.unsafe.types.UTF8String

[ https://issues.apache.org/jira/browse/SPARK-15822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15327992#comment-15327992 ] Pete Robbins commented on SPARK-15822: -- So this does seem to cause the NPE or SEGV intermittently

[jira] [Commented] (SPARK-15822) segmentation violation in o.a.s.unsafe.types.UTF8String

[ https://issues.apache.org/jira/browse/SPARK-15822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15325761#comment-15325761 ] Pete Robbins commented on SPARK-15822: -- has failed on latest Branch 2.0 and master. Currently using

[jira] [Commented] (SPARK-15822) segmentation violation in o.a.s.unsafe.types.UTF8String

[ https://issues.apache.org/jira/browse/SPARK-15822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15325759#comment-15325759 ] Pete Robbins commented on SPARK-15822: -- The stack trace is taken earlier when I detect

[jira] [Commented] (SPARK-15822) segmentation violation in o.a.s.unsafe.types.UTF8String

[ https://issues.apache.org/jira/browse/SPARK-15822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15325159#comment-15325159 ] Pete Robbins commented on SPARK-15822: -- I am forcing a system dump when I detect a corrupt

[jira] [Comment Edited] (SPARK-15822) segmentation violation in o.a.s.unsafe.types.UTF8String

[ https://issues.apache.org/jira/browse/SPARK-15822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15325093#comment-15325093 ] Pete Robbins edited comment on SPARK-15822 at 6/10/16 7:17 PM: --- How do I

[jira] [Commented] (SPARK-15822) segmentation violation in o.a.s.unsafe.types.UTF8String

[ https://issues.apache.org/jira/browse/SPARK-15822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15325093#comment-15325093 ] Pete Robbins commented on SPARK-15822: -- How do I disable whole-stage codegen? > segmentat

[jira] [Commented] (SPARK-15822) segmentation violation in o.a.s.unsafe.types.UTF8String

[ https://issues.apache.org/jira/browse/SPARK-15822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15324506#comment-15324506 ] Pete Robbins commented on SPARK-15822: -- generated SMJ code from the stack: {code} public Object

[jira] [Updated] (SPARK-15822) segmentation violation in o.a.s.unsafe.types.UTF8String

[ https://issues.apache.org/jira/browse/SPARK-15822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pete Robbins updated SPARK-15822: - Description: Executors fail with segmentation violation while running application

[jira] [Updated] (SPARK-15822) segmentation violation in o.a.s.unsafe.types.UTF8String

[ https://issues.apache.org/jira/browse/SPARK-15822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pete Robbins updated SPARK-15822: - Summary: segmentation violation in o.a.s.unsafe.types.UTF8String (was: segmentation violation

[jira] [Commented] (SPARK-15822) segmentation violation in o.a.s.unsafe.types.UTF8String with spark.memory.offHeap.enabled=true

[ https://issues.apache.org/jira/browse/SPARK-15822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15324331#comment-15324331 ] Pete Robbins commented on SPARK-15822: -- I'm still looking into this tracing back through the code

Re: Spark 2.0.0 preview docs uploaded

It would be nice to have a "what's new in 2.0.0" equivalent to https://spark.apache.org/releases/spark-release-1-6-0.html available or am I just missing it? On Wed, 8 Jun 2016 at 13:15 Sean Owen wrote: > OK, this is done: > > http://spark.apache.org/documentation.html >

Re: NegativeArraySizeException / segfault

I just raised https://issues.apache.org/jira/browse/SPARK-15822 for a similar looking issue. Analyzing the core dump from the segv with Memory Analyzer it looks very much like a UTF8String is very corrupt. Cheers, On Fri, 27 May 2016 at 21:00 Koert Kuipers wrote: > hello

[jira] [Commented] (SPARK-15822) segmentation violation in o.a.s.unsafe.types.UTF8String with spark.memory.offHeap.enabled=true

[ https://issues.apache.org/jira/browse/SPARK-15822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15320380#comment-15320380 ] Pete Robbins commented on SPARK-15822: -- I'm investigating this and will attach the app and config

[jira] [Created] (SPARK-15822) segmentation violation in o.a.s.unsafe.types.UTF8String with spark.memory.offHeap.enabled=true

Pete Robbins created SPARK-15822: Summary: segmentation violation in o.a.s.unsafe.types.UTF8String with spark.memory.offHeap.enabled=true Key: SPARK-15822 URL: https://issues.apache.org/jira/browse/SPARK-15822

[jira] [Commented] (SPARK-15065) HiveSparkSubmitSuite's "set spark.sql.warehouse.dir" is flaky

[ https://issues.apache.org/jira/browse/SPARK-15065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15318492#comment-15318492 ] Pete Robbins commented on SPARK-15065: -- I think this may be related to https://issues.apache.org

[jira] [Created] (SPARK-15606) Driver hang in o.a.s.DistributedSuite on 2 core machine

Pete Robbins created SPARK-15606: Summary: Driver hang in o.a.s.DistributedSuite on 2 core machine Key: SPARK-15606 URL: https://issues.apache.org/jira/browse/SPARK-15606 Project: Spark

[jira] [Updated] (SPARK-15154) LongHashedRelation test fails on Big Endian platform

[ https://issues.apache.org/jira/browse/SPARK-15154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pete Robbins updated SPARK-15154: - Priority: Minor (was: Major) Summary: LongHashedRelation test fails on Big Endian platform

[jira] [Commented] (SPARK-15154) LongHashedRelation fails on Big Endian platform

[ https://issues.apache.org/jira/browse/SPARK-15154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15276543#comment-15276543 ] Pete Robbins commented on SPARK-15154: -- I'm convinced the test is invalid. The creation

[jira] [Comment Edited] (SPARK-15154) LongHashedRelation fails on Big Endian platform

[ https://issues.apache.org/jira/browse/SPARK-15154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15273885#comment-15273885 ] Pete Robbins edited comment on SPARK-15154 at 5/6/16 11:20 AM: --- [~davies

[jira] [Comment Edited] (SPARK-15154) LongHashedRelation fails on Big Endian platform

[ https://issues.apache.org/jira/browse/SPARK-15154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15273885#comment-15273885 ] Pete Robbins edited comment on SPARK-15154 at 5/6/16 11:14 AM: --- [~davies

[jira] [Comment Edited] (SPARK-15154) LongHashedRelation fails on Big Endian platform

[ https://issues.apache.org/jira/browse/SPARK-15154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15273897#comment-15273897 ] Pete Robbins edited comment on SPARK-15154 at 5/6/16 11:13 AM

[jira] [Commented] (SPARK-15154) LongHashedRelation fails on Big Endian platform

[ https://issues.apache.org/jira/browse/SPARK-15154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15273897#comment-15273897 ] Pete Robbins commented on SPARK-15154: -- Is this just a testcase issue where in HashedRelationSuite

[jira] [Comment Edited] (SPARK-15154) LongHashedRelation fails on Big Endian platform

[ https://issues.apache.org/jira/browse/SPARK-15154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15273885#comment-15273885 ] Pete Robbins edited comment on SPARK-15154 at 5/6/16 10:28 AM: --- [~davies

[jira] [Comment Edited] (SPARK-15154) LongHashedRelation fails on Big Endian platform

[ https://issues.apache.org/jira/browse/SPARK-15154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15273885#comment-15273885 ] Pete Robbins edited comment on SPARK-15154 at 5/6/16 10:27 AM: --- [~davies

[jira] [Comment Edited] (SPARK-15154) LongHashedRelation fails on Big Endian platform

[ https://issues.apache.org/jira/browse/SPARK-15154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15273885#comment-15273885 ] Pete Robbins edited comment on SPARK-15154 at 5/6/16 10:24 AM: --- [~davies

[jira] [Comment Edited] (SPARK-15154) LongHashedRelation fails on Big Endian platform

[ https://issues.apache.org/jira/browse/SPARK-15154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15273885#comment-15273885 ] Pete Robbins edited comment on SPARK-15154 at 5/6/16 10:25 AM: --- [~davies

[jira] [Commented] (SPARK-15154) LongHashedRelation fails on Big Endian platform

[ https://issues.apache.org/jira/browse/SPARK-15154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15273885#comment-15273885 ] Pete Robbins commented on SPARK-15154: -- [~davies] as you are the author of this code can you comment

[jira] [Updated] (SPARK-15154) LongHashedRelation fails on Big Endian platform

[ https://issues.apache.org/jira/browse/SPARK-15154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pete Robbins updated SPARK-15154: - Summary: LongHashedRelation fails on Big Endian platform (was: HashedRelation fails on Big

[jira] [Updated] (SPARK-15154) HashedRelation fails on Big Endian platform

[ https://issues.apache.org/jira/browse/SPARK-15154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pete Robbins updated SPARK-15154: - Labels: big-endian (was: ) > HashedRelation fails on Big Endian platf

[jira] [Created] (SPARK-15154) HashedRelation fails on Big Endian platform

Pete Robbins created SPARK-15154: Summary: HashedRelation fails on Big Endian platform Key: SPARK-15154 URL: https://issues.apache.org/jira/browse/SPARK-15154 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-15070) Data corruption when using Dataset.groupBy[K : Encoder](func: T => K) when data loaded from JSON file.

[ https://issues.apache.org/jira/browse/SPARK-15070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15268500#comment-15268500 ] Pete Robbins commented on SPARK-15070: -- could this be related to https://issues.apache.org/jira

[jira] [Commented] (SPARK-13552) Incorrect data for Long.minValue in SQLQuerySuite on IBM Java

[ https://issues.apache.org/jira/browse/SPARK-13552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15268232#comment-15268232 ] Pete Robbins commented on SPARK-13552: -- [~aroberts] This Jira can be closed as this is not a Spark

Re: [ANNOUNCE] Spark branch-2.0

https://issues.apache.org/jira/browse/SPARK-13745 is really a defect and a blocker unless it is the decision to drop support for Big Endian platforms. The PR has been reviewed and tested and I strongly believe this needs to be targeted for 2.0. On Mon, May 2, 2016 at 12:00 AM Reynold Xin

[jira] [Commented] (SPARK-13552) Incorrect data for Long.minValue in SQLQuerySuite on IBM Java

[ https://issues.apache.org/jira/browse/SPARK-13552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15262482#comment-15262482 ] Pete Robbins commented on SPARK-13552: -- This is looking like an issue with the IBM implementation

[jira] [Updated] (SPARK-14848) DatasetSuite - Java encoder fails on Big Endian platforms

[ https://issues.apache.org/jira/browse/SPARK-14848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pete Robbins updated SPARK-14848: - Description: Since this PR https://github.com/apache/spark/pull/10703 for https

[jira] [Commented] (SPARK-14848) DatasetSuite - Java encoder fails on Big Endian platforms

[ https://issues.apache.org/jira/browse/SPARK-14848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15253816#comment-15253816 ] Pete Robbins commented on SPARK-14848: -- changing the Java encoder test to use toSet and compare

[jira] [Created] (SPARK-14848) DatasetSuite - Java encoder fails on Big Endian platforms

Pete Robbins created SPARK-14848: Summary: DatasetSuite - Java encoder fails on Big Endian platforms Key: SPARK-14848 URL: https://issues.apache.org/jira/browse/SPARK-14848 Project: Spark

Re: Code freeze?

Is there a list of Jiras to be considered for 2.0? I would really like to get https://issues.apache.org/jira/browse/SPARK-13745 in so that Big Endian platforms are not broken. Cheers, On Wed, 13 Apr 2016 at 08:51 Reynold Xin wrote: > I think the main things are API things

[jira] [Commented] (SPARK-14151) Propose to refactor and expose Metrics Sink and Source interface

[ https://issues.apache.org/jira/browse/SPARK-14151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15211638#comment-15211638 ] Pete Robbins commented on SPARK-14151: -- Agreed that is the way to go. I was also working

[jira] [Commented] (SPARK-14151) Propose to expose Metrics Sink and Source interface

[ https://issues.apache.org/jira/browse/SPARK-14151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15211560#comment-15211560 ] Pete Robbins commented on SPARK-14151: -- In addition the constructor used by MetricsSystem for Sinks

Re: Can we remove private[spark] from Metrics Source and SInk traits?

these into core Spark. Opening up the Sink/Source interfaces would at least allow these to exist somewhere such as spark-packages without having to pollute the o.a.s namespace On Sat, 19 Mar 2016 at 13:05 Gerard Maas <gerard.m...@gmail.com> wrote: > +1 > On Mar 19, 2016 08:33, &

Can we remove private[spark] from Metrics Source and SInk traits?

This seems to me to be unnecessarily restrictive. These are very useful extension points for adding 3rd party sources and sinks. I intend to make an Elasticsearch sink available on spark-packages but this will require a single class, the sink, to be in the org.apache.spark package tree. I could

Re: Accessing SparkConf in metrics sink

So the answer to my previous question is NO. It looks like I could use SparkEnv.get.conf but * * NOTE: This is not intended for external use. This is exposed for Shark and may be made private * in a future release. */ On Wed, 16 Mar 2016 at 08:22 Pete Robbins <robbin...@gmail.com>

Re: Accessing SparkConf in metrics sink

OK thanks. Does that work in an executor? On Wed, 16 Mar 2016 at 07:58 Reynold Xin <r...@databricks.com> wrote: > SparkConf is not a singleton. > > However, SparkContext in almost all cases are. So you can use > SparkContext.getOrCreate().getConf > > On Wed, Mar 16

Accessing SparkConf in metrics sink

I'm writing a metrics sink and reporter to push metrics to Elasticsearch. An example format of a metric in JSON: { "timestamp": "2016-03-15T16:11:19.314+", "hostName": "10.192.0.87" "applicationName": "My application", "applicationId": "app-20160315093931-0003", "executorId": "17",

Re: SparkConf constructor now private

Is the SparkConf effectively a singleton? Could there be a Utils method to return a clone of the SparkConf? Cheers On Tue, 15 Mar 2016 at 16:49 Marcelo Vanzin wrote: > Oh, my bad. I think I left that from a previous part of the patch and > forgot to revert it. Will fix. >

[jira] [Comment Edited] (SPARK-10610) Using AppName instead of AppId in the name of all metrics

[ https://issues.apache.org/jira/browse/SPARK-10610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15179507#comment-15179507 ] Pete Robbins edited comment on SPARK-10610 at 3/4/16 8:01 AM: -- I think

[jira] [Commented] (SPARK-10610) Using AppName instead of AppId in the name of all metrics

[ https://issues.apache.org/jira/browse/SPARK-10610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15179507#comment-15179507 ] Pete Robbins commented on SPARK-10610: -- I think the appId is an important piece of information when

Re: SparkOscope: Enabling Spark Optimization through Cross-stack Monitoring and Visualization

Yiannis, I'm interested in what you've done here as I was looking for ways to allow the Spark UI to display custom metrics in a pluggable way without having to modify the Spark source code. It would be good to see if we could have modify your code to add extension points into the UI so we could

Fwd: Elasticsearch sink for metrics

ing to see if this was already implemented before continuing. On 15 January 2016 at 09:18, Nick Pentreath <nick.pentre...@gmail.com> wrote: > I haven't come across anything, but could you provide more detail on what > issues you're encountering? > > > > On Fri, Jan 15, 2016 at 11:09 AM,

Elasticsearch sink for metrics

Has anyone tried pushing Spark metrics into elasticsearch? We have other metrics, eg some runtime information, going into ES and would like to be able to combine this with the Spark metrics for visualization with Kibana. I experimented with a new sink using ES's ElasticsearchReporter for the Coda

[jira] [Commented] (SPARK-12647) 1.6 branch test failure o.a.s.sql.execution.ExchangeCoordinatorSuite.determining the number of reducers: aggregate operator

[ https://issues.apache.org/jira/browse/SPARK-12647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15082879#comment-15082879 ] Pete Robbins commented on SPARK-12647: -- @sowen should I close this and move the PR? > 1.6 bra

[jira] [Created] (SPARK-12647) 1.6 branch test failure o.a.s.sql.execution.ExchangeCoordinatorSuite.determining the number of reducers: aggregate operator

Pete Robbins created SPARK-12647: Summary: 1.6 branch test failure o.a.s.sql.execution.ExchangeCoordinatorSuite.determining the number of reducers: aggregate operator Key: SPARK-12647 URL: https

[jira] [Comment Edited] (SPARK-12647) 1.6 branch test failure o.a.s.sql.execution.ExchangeCoordinatorSuite.determining the number of reducers: aggregate operator

[ https://issues.apache.org/jira/browse/SPARK-12647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15082879#comment-15082879 ] Pete Robbins edited comment on SPARK-12647 at 1/5/16 11:30 AM: --- [~sowen

[jira] [Commented] (SPARK-12470) Incorrect calculation of row size in o.a.s.sql.catalyst.expressions.codegen.GenerateUnsafeRowJoiner

[ https://issues.apache.org/jira/browse/SPARK-12470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15068693#comment-15068693 ] Pete Robbins commented on SPARK-12470: -- I'm fairly sure the code in my PR is correct

[jira] [Comment Edited] (SPARK-12470) Incorrect calculation of row size in o.a.s.sql.catalyst.expressions.codegen.GenerateUnsafeRowJoiner

[ https://issues.apache.org/jira/browse/SPARK-12470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15068693#comment-15068693 ] Pete Robbins edited comment on SPARK-12470 at 12/22/15 9:47 PM: I'm

[jira] [Updated] (SPARK-12470) Incorrect calculation of row size in o.a.s.sql.catalyst.expressions.codegen.GenerateUnsafeRowJoiner

[ https://issues.apache.org/jira/browse/SPARK-12470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pete Robbins updated SPARK-12470: - Component/s: SQL Summary: Incorrect calculation of row size

[jira] [Created] (SPARK-12470) Incorrect calculation of row size in o.a.s.catalyst.expressions.codegen.GenerateUnsafeRowJoiner

Pete Robbins created SPARK-12470: Summary: Incorrect calculation of row size in o.a.s.catalyst.expressions.codegen.GenerateUnsafeRowJoiner Key: SPARK-12470 URL: https://issues.apache.org/jira/browse/SPARK-12470

[jira] [Commented] (SPARK-6873) Some Hive-Catalyst comparison tests fail due to unimportant order of some printed elements

[ https://issues.apache.org/jira/browse/SPARK-6873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14907841#comment-14907841 ] Pete Robbins commented on SPARK-6873: - I no longer see these errors in my 1.5 branch Java 8 build. Did

[jira] [Commented] (SPARK-9710) RPackageUtilsSuite fails if R is not installed

[ https://issues.apache.org/jira/browse/SPARK-9710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904485#comment-14904485 ] Pete Robbins commented on SPARK-9710: - The Fix Version for this says 1.5.0 but the PR

Re: Unable to acquire memory errors in HiveCompatibilitySuite

f tasks -- but it turned out to be a bad approximation in tests > because it is set to 32 to increase concurrency. > > > On Tue, Sep 15, 2015 at 10:47 PM, Pete Robbins <robbin...@gmail.com> > wrote: > >> Oops... I meant to say "The page size calculation is NOT the is

  1   2   3   4   5   6   7   8   9   10   >