[darktable-dev] Re: Online documentation at version 2.4

[Pete Robbins]

Apologies, I just saw another thread already raising this. Teaching to look
for "documentation" rather than "manual"

Cheers

On Fri, 1 Mar 2019, 14:20 Pete Robbins,  wrote:

> Hi, I've just started using darktable 2.6 (impressed so far).
>
> I noticed the online manual here: https://www.darktable.org/usermanual/en/
> is at version 2.4. Should this be updated to 2.6?
>
> I believe the download pdf is v2.6
>
> Cheers,
>
> Pete
>

___
darktable developer mailing list
to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org

[darktable-dev] Online documentation at version 2.4

[Pete Robbins]

Hi, I've just started using darktable 2.6 (impressed so far).

I noticed the online manual here: https://www.darktable.org/usermanual/en/
is at version 2.4. Should this be updated to 2.6?

I believe the download pdf is v2.6

Cheers,

Pete

___
darktable developer mailing list
to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org

[jira] [Created] (SPARK-19710) Test Failures in SQLQueryTests on big endian platforms

[Pete Robbins (JIRA)]

Pete Robbins created SPARK-19710:


 Summary: Test Failures in SQLQueryTests on big endian platforms
 Key: SPARK-19710
 URL: https://issues.apache.org/jira/browse/SPARK-19710
 Project: Spark
  Issue Type: Bug
  Components: SQL
Affects Versions: 2.2.0
Reporter: Pete Robbins
Priority: Minor


Some of the new test queries introduced by 
https://issues.apache.org/jira/browse/SPARK-18871 fail when run on zLinux (big 
endian)

The order of the return rows is different to the results file, hence the 
failures, but the results are valid for the queries as insufficient ordering is 
specified to give absolute results.

The failing tests are in o.a.s.SQLQuerTestSuite
in-joins.sql
not-in-joins.sql
in-set-operations.sql

These can be fixed by adding to the ORDER BY clauses to determine the resulting 
row order.

PR on it's way



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Re: Java 9

[Pete Robbins]

Yes, I agree but it may be worthwhile starting to look at this. I was just
trying a build and it trips over some of the now defunct/inaccessible
sun.misc classes.

I was just interested in hearing if anyone has already gone through this to
save me duplicating effort.

Cheers,

On Tue, 7 Feb 2017 at 11:46 Sean Owen <so...@cloudera.com> wrote:

> I don't think anyone's tried it. I think we'd first have to agree to drop
> Java 7 support before that could be seriously considered. The 8-9
> difference is a bit more of a breaking change.
>
> On Tue, Feb 7, 2017 at 11:44 AM Pete Robbins <robbin...@gmail.com> wrote:
>
> Is anyone working on support for running Spark on Java 9? Is this in a
> roadmap anywhere?
>
>
> Cheers,
>
>

Java 9

[Pete Robbins]

Is anyone working on support for running Spark on Java 9? Is this in a
roadmap anywhere?


Cheers,

[jira] [Created] (SPARK-18963) Test Failuire on big endian; o.a.s.unsafe.types.UTF8StringSuite.writeToOutputStreamIntArray

[Pete Robbins (JIRA)]

Pete Robbins created SPARK-18963:


 Summary: Test Failuire on big endian; 
o.a.s.unsafe.types.UTF8StringSuite.writeToOutputStreamIntArray
 Key: SPARK-18963
 URL: https://issues.apache.org/jira/browse/SPARK-18963
 Project: Spark
  Issue Type: Bug
  Components: Tests
Affects Versions: 2.1.1
Reporter: Pete Robbins
Priority: Minor


SPARK-18658 introduced a new test which is flipping a ByteBuffer into little 
endian order. This is not necessary on a big endian platform and results in:

writeToOutputStreamIntArray(org.apache.spark.unsafe.types.UTF8StringSuite)  
Time elapsed: 0.01 sec  <<< FAILURE!
org.junit.ComparisonFailure: expected:<[大千世界]> but was:<[姤�䃍���]>
at 
org.apache.spark.unsafe.types.UTF8StringSuite.writeToOutputStreamIntArray(UTF8StringSuite.java:609)


PR on it's way



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Re: Anyone seeing a lot of Spark emails go to Gmail spam?

[Pete Robbins]

I have gmail filters to add labels and skip inbox for anything sent to
dev@spark user@spark etc but still get the occasional message marked as spam


On Wed, 2 Nov 2016 at 08:18 Sean Owen  wrote:

> I couldn't figure out why I was missing a lot of dev@ announcements, and
> have just realized hundreds of messages to dev@ over the past month or so
> have been marked as spam for me by Gmail. I have no idea why but it's
> usually messages from Michael and Reynold, but not all of them. I'll see
> replies to the messages but not the original. Who knows. I can make a
> filter. I just wanted to give a heads up in case anyone else has been
> silently missing a lot of messages.
>

Re: [VOTE] Release Apache Spark 1.6.3 (RC1)

[Pete Robbins]

We see a regression since 1.6.2. I think this PR needs to be backported
https://github.com/apache/spark/pull/13784 which resolves SPARK-16078. The
PR that causes the issue (for SPARK-15613) was reverted just before 1.6.2
release then re-applied afterwards but this fix was only backported to 2.0.

Test failure: org.apache.spark.sql.catalyst.util.DateTimeUtilsSuite.to UTC
timestamp

On Tue, 18 Oct 2016 at 01:19 Reynold Xin  wrote:

> Please vote on releasing the following candidate as Apache Spark version
> 1.6.3. The vote is open until Thursday, Oct 20, 2016 at 18:00 PDT and
> passes if a majority of at least 3+1 PMC votes are cast.
>
> [ ] +1 Release this package as Apache Spark 1.6.3
> [ ] -1 Do not release this package because ...
>
>
> The tag to be voted on is v1.6.3-rc1
> (7375bb0c825408ea010dcef31c0759cf94ffe5c2)
>
> This release candidate addresses 50 JIRA tickets:
> https://s.apache.org/spark-1.6.3-jira
>
> The release files, including signatures, digests, etc. can be found at:
> http://people.apache.org/~pwendell/spark-releases/spark-1.6.3-rc1-bin/
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/pwendell.asc
>
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapachespark-1205/
>
> The documentation corresponding to this release can be found at:
> http://people.apache.org/~pwendell/spark-releases/spark-1.6.3-rc1-docs/
>
>
> ===
> == How can I help test this release?
> ===
> If you are a Spark user, you can help us test this release by taking an
> existing Spark workload and running on this release candidate, then
> reporting any regressions from 1.6.2.
>
> 
> == What justifies a -1 vote for this release?
> 
> This is a maintenance release in the 1.6.x series.  Bugs already present
> in 1.6.2, missing features, or bugs related to new features will not
> necessarily block this release.
>
>

[jira] [Commented] (SPARK-17827) StatisticsColumnSuite failures on big endian platforms

[Pete Robbins (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-17827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15571890#comment-15571890
 ] 

Pete Robbins commented on SPARK-17827:
--

I have a PR ready which I will submit as soon as I have run the tests on both 
Big and Little Endian

> StatisticsColumnSuite failures on big endian platforms
> --
>
> Key: SPARK-17827
> URL: https://issues.apache.org/jira/browse/SPARK-17827
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.1.0
> Environment: big endian
>    Reporter: Pete Robbins
>  Labels: big-endian
>
> https://issues.apache.org/jira/browse/SPARK-17073
> introduces new tests/function that fails on big endian platforms
> Failing tests:
>  org.apache.spark.sql.StatisticsColumnSuite.column-level statistics for 
> string column
>  org.apache.spark.sql.StatisticsColumnSuite.column-level statistics for 
> binary column
>  org.apache.spark.sql.StatisticsColumnSuite.column-level statistics for 
> columns with different types
>  org.apache.spark.sql.hive.StatisticsSuite.generate column-level statistics 
> and load them from hive metastore
> all fail in checkColStat eg: 
> java.lang.AssertionError: assertion failed
>   at scala.Predef$.assert(Predef.scala:156)
>   at 
> org.apache.spark.sql.StatisticsTest$.checkColStat(StatisticsTest.scala:92)
>   at 
> org.apache.spark.sql.StatisticsTest$$anonfun$checkColStats$1$$anonfun$apply$mcV$sp$1.apply(StatisticsTest.scala:43)
>   at 
> org.apache.spark.sql.StatisticsTest$$anonfun$checkColStats$1$$anonfun$apply$mcV$sp$1.apply(StatisticsTest.scala:40)
>   at scala.collection.immutable.List.foreach(List.scala:381)
>   at 
> org.apache.spark.sql.StatisticsTest$$anonfun$checkColStats$1.apply$mcV$sp(StatisticsTest.scala:40)
>   at 
> org.apache.spark.sql.test.SQLTestUtils$class.withTable(SQLTestUtils.scala:168)
>   at 
> org.apache.spark.sql.StatisticsColumnSuite.withTable(StatisticsColumnSuite.scala:30)
>   at 
> org.apache.spark.sql.StatisticsTest$class.checkColStats(StatisticsTest.scala:33)
>   at 
> org.apache.spark.sql.StatisticsColumnSuite.checkColStats(StatisticsColumnSuite.scala:30)
>   at 
> org.apache.spark.sql.StatisticsColumnSuite$$anonfun$7.apply$mcV$sp(StatisticsColumnSuite.scala:171)
>   at 
> org.apache.spark.sql.StatisticsColumnSuite$$anonfun$7.apply(StatisticsColumnSuite.scala:160)
>   at 
> org.apache.spark.sql.StatisticsColumnSuite$$anonfun$7.apply(StatisticsColumnSuite.scala:160)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-17827) StatisticsColumnSuite failures on big endian platforms

[Pete Robbins (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-17827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15571546#comment-15571546
 ] 

Pete Robbins commented on SPARK-17827:
--

right so in these two cases maxLength  in  AnalyzeColumnCommand is returning an 
Int type and I guess in other cases it could be Long??

> StatisticsColumnSuite failures on big endian platforms
> --
>
> Key: SPARK-17827
> URL: https://issues.apache.org/jira/browse/SPARK-17827
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.1.0
> Environment: big endian
>    Reporter: Pete Robbins
>  Labels: big-endian
>
> https://issues.apache.org/jira/browse/SPARK-17073
> introduces new tests/function that fails on big endian platforms
> Failing tests:
>  org.apache.spark.sql.StatisticsColumnSuite.column-level statistics for 
> string column
>  org.apache.spark.sql.StatisticsColumnSuite.column-level statistics for 
> binary column
>  org.apache.spark.sql.StatisticsColumnSuite.column-level statistics for 
> columns with different types
>  org.apache.spark.sql.hive.StatisticsSuite.generate column-level statistics 
> and load them from hive metastore
> all fail in checkColStat eg: 
> java.lang.AssertionError: assertion failed
>   at scala.Predef$.assert(Predef.scala:156)
>   at 
> org.apache.spark.sql.StatisticsTest$.checkColStat(StatisticsTest.scala:92)
>   at 
> org.apache.spark.sql.StatisticsTest$$anonfun$checkColStats$1$$anonfun$apply$mcV$sp$1.apply(StatisticsTest.scala:43)
>   at 
> org.apache.spark.sql.StatisticsTest$$anonfun$checkColStats$1$$anonfun$apply$mcV$sp$1.apply(StatisticsTest.scala:40)
>   at scala.collection.immutable.List.foreach(List.scala:381)
>   at 
> org.apache.spark.sql.StatisticsTest$$anonfun$checkColStats$1.apply$mcV$sp(StatisticsTest.scala:40)
>   at 
> org.apache.spark.sql.test.SQLTestUtils$class.withTable(SQLTestUtils.scala:168)
>   at 
> org.apache.spark.sql.StatisticsColumnSuite.withTable(StatisticsColumnSuite.scala:30)
>   at 
> org.apache.spark.sql.StatisticsTest$class.checkColStats(StatisticsTest.scala:33)
>   at 
> org.apache.spark.sql.StatisticsColumnSuite.checkColStats(StatisticsColumnSuite.scala:30)
>   at 
> org.apache.spark.sql.StatisticsColumnSuite$$anonfun$7.apply$mcV$sp(StatisticsColumnSuite.scala:171)
>   at 
> org.apache.spark.sql.StatisticsColumnSuite$$anonfun$7.apply(StatisticsColumnSuite.scala:160)
>   at 
> org.apache.spark.sql.StatisticsColumnSuite$$anonfun$7.apply(StatisticsColumnSuite.scala:160)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Comment Edited] (SPARK-17827) StatisticsColumnSuite failures on big endian platforms

[Pete Robbins (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-17827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15571348#comment-15571348
 ] 

Pete Robbins edited comment on SPARK-17827 at 10/13/16 9:11 AM:


In Statistics.scala

{code}
case class StringColumnStat(statRow: InternalRow) {
  println("StringColumnStat: " + statRow)
  // The indices here must be consistent with 
`ColumnStatStruct.stringColumnStat`.
  val numNulls: Long = statRow.getLong(0)
  val avgColLen: Double = statRow.getDouble(1)
  val maxColLen: Long = statRow.getLong(2)   <<<<<< Actual type in 
statRow is Int
  val ndv: Long = statRow.getLong(3)
}

case class BinaryColumnStat(statRow: InternalRow) {
  // The indices here must be consistent with 
`ColumnStatStruct.binaryColumnStat`.
  val numNulls: Long = statRow.getLong(0)
  val avgColLen: Double = statRow.getDouble(1)
  val maxColLen: Long = statRow.getLong(2)<<<<<< Actual type in 
statRow is Int
}

{code}

So either the code above should be using getInt for the maxColLen or the code 
generating the row should be creating a Long


was (Author: robbinspg):
In Statistics.scala

case class StringColumnStat(statRow: InternalRow) {
  println("StringColumnStat: " + statRow)
  // The indices here must be consistent with 
`ColumnStatStruct.stringColumnStat`.
  val numNulls: Long = statRow.getLong(0)
  val avgColLen: Double = statRow.getDouble(1)
  val maxColLen: Long = statRow.getLong(2)   <<<<<< Actual type in 
statRow is Int
  val ndv: Long = statRow.getLong(3)
}

case class BinaryColumnStat(statRow: InternalRow) {
  // The indices here must be consistent with 
`ColumnStatStruct.binaryColumnStat`.
  val numNulls: Long = statRow.getLong(0)
  val avgColLen: Double = statRow.getDouble(1)
  val maxColLen: Long = statRow.getLong(2)<<<<<< Actual type in 
statRow is Int
}

So either the code above should be using getInt for the maxColLen or the code 
generating the row should be creating a Long

> StatisticsColumnSuite failures on big endian platforms
> --
>
> Key: SPARK-17827
> URL: https://issues.apache.org/jira/browse/SPARK-17827
> Project: Spark
>  Issue Type: Bug
>      Components: SQL
>Affects Versions: 2.1.0
> Environment: big endian
>Reporter: Pete Robbins
>  Labels: big-endian
>
> https://issues.apache.org/jira/browse/SPARK-17073
> introduces new tests/function that fails on big endian platforms
> Failing tests:
>  org.apache.spark.sql.StatisticsColumnSuite.column-level statistics for 
> string column
>  org.apache.spark.sql.StatisticsColumnSuite.column-level statistics for 
> binary column
>  org.apache.spark.sql.StatisticsColumnSuite.column-level statistics for 
> columns with different types
>  org.apache.spark.sql.hive.StatisticsSuite.generate column-level statistics 
> and load them from hive metastore
> all fail in checkColStat eg: 
> java.lang.AssertionError: assertion failed
>   at scala.Predef$.assert(Predef.scala:156)
>   at 
> org.apache.spark.sql.StatisticsTest$.checkColStat(StatisticsTest.scala:92)
>   at 
> org.apache.spark.sql.StatisticsTest$$anonfun$checkColStats$1$$anonfun$apply$mcV$sp$1.apply(StatisticsTest.scala:43)
>   at 
> org.apache.spark.sql.StatisticsTest$$anonfun$checkColStats$1$$anonfun$apply$mcV$sp$1.apply(StatisticsTest.scala:40)
>   at scala.collection.immutable.List.foreach(List.scala:381)
>   at 
> org.apache.spark.sql.StatisticsTest$$anonfun$checkColStats$1.apply$mcV$sp(StatisticsTest.scala:40)
>   at 
> org.apache.spark.sql.test.SQLTestUtils$class.withTable(SQLTestUtils.scala:168)
>   at 
> org.apache.spark.sql.StatisticsColumnSuite.withTable(StatisticsColumnSuite.scala:30)
>   at 
> org.apache.spark.sql.StatisticsTest$class.checkColStats(StatisticsTest.scala:33)
>   at 
> org.apache.spark.sql.StatisticsColumnSuite.checkColStats(StatisticsColumnSuite.scala:30)
>   at 
> org.apache.spark.sql.StatisticsColumnSuite$$anonfun$7.apply$mcV$sp(StatisticsColumnSuite.scala:171)
>   at 
> org.apache.spark.sql.StatisticsColumnSuite$$anonfun$7.apply(StatisticsColumnSuite.scala:160)
>   at 
> org.apache.spark.sql.StatisticsColumnSuite$$anonfun$7.apply(StatisticsColumnSuite.scala:160)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Comment Edited] (SPARK-17827) StatisticsColumnSuite failures on big endian platforms

[Pete Robbins (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-17827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15571348#comment-15571348
 ] 

Pete Robbins edited comment on SPARK-17827 at 10/13/16 9:10 AM:


In Statistics.scala

case class StringColumnStat(statRow: InternalRow) {
  println("StringColumnStat: " + statRow)
  // The indices here must be consistent with 
`ColumnStatStruct.stringColumnStat`.
  val numNulls: Long = statRow.getLong(0)
  val avgColLen: Double = statRow.getDouble(1)
  val maxColLen: Long = statRow.getLong(2)   <<<<<< Actual type in 
statRow is Int
  val ndv: Long = statRow.getLong(3)
}

case class BinaryColumnStat(statRow: InternalRow) {
  // The indices here must be consistent with 
`ColumnStatStruct.binaryColumnStat`.
  val numNulls: Long = statRow.getLong(0)
  val avgColLen: Double = statRow.getDouble(1)
  val maxColLen: Long = statRow.getLong(2)<<<<<< Actual type in 
statRow is Int
}

So either the code above should be using getInt for the maxColLen or the code 
generating the row should be creating a Long


was (Author: robbinspg):
In Statistics,scala

case class StringColumnStat(statRow: InternalRow) {
  println("StringColumnStat: " + statRow)
  // The indices here must be consistent with 
`ColumnStatStruct.stringColumnStat`.
  val numNulls: Long = statRow.getLong(0)
  val avgColLen: Double = statRow.getDouble(1)
  val maxColLen: Long = statRow.getLong(2)   <<<<<< Actual type in 
statRow is Int
  val ndv: Long = statRow.getLong(3)
}

case class BinaryColumnStat(statRow: InternalRow) {
  // The indices here must be consistent with 
`ColumnStatStruct.binaryColumnStat`.
  val numNulls: Long = statRow.getLong(0)
  val avgColLen: Double = statRow.getDouble(1)
  val maxColLen: Long = statRow.getLong(2)<<<<<< Actual type in 
statRow is Int
}

So either the code above should be using getInt for the maxColLen or the code 
generating the row should be creating a Long

> StatisticsColumnSuite failures on big endian platforms
> --
>
> Key: SPARK-17827
> URL: https://issues.apache.org/jira/browse/SPARK-17827
> Project: Spark
>  Issue Type: Bug
>      Components: SQL
>Affects Versions: 2.1.0
> Environment: big endian
>Reporter: Pete Robbins
>  Labels: big-endian
>
> https://issues.apache.org/jira/browse/SPARK-17073
> introduces new tests/function that fails on big endian platforms
> Failing tests:
>  org.apache.spark.sql.StatisticsColumnSuite.column-level statistics for 
> string column
>  org.apache.spark.sql.StatisticsColumnSuite.column-level statistics for 
> binary column
>  org.apache.spark.sql.StatisticsColumnSuite.column-level statistics for 
> columns with different types
>  org.apache.spark.sql.hive.StatisticsSuite.generate column-level statistics 
> and load them from hive metastore
> all fail in checkColStat eg: 
> java.lang.AssertionError: assertion failed
>   at scala.Predef$.assert(Predef.scala:156)
>   at 
> org.apache.spark.sql.StatisticsTest$.checkColStat(StatisticsTest.scala:92)
>   at 
> org.apache.spark.sql.StatisticsTest$$anonfun$checkColStats$1$$anonfun$apply$mcV$sp$1.apply(StatisticsTest.scala:43)
>   at 
> org.apache.spark.sql.StatisticsTest$$anonfun$checkColStats$1$$anonfun$apply$mcV$sp$1.apply(StatisticsTest.scala:40)
>   at scala.collection.immutable.List.foreach(List.scala:381)
>   at 
> org.apache.spark.sql.StatisticsTest$$anonfun$checkColStats$1.apply$mcV$sp(StatisticsTest.scala:40)
>   at 
> org.apache.spark.sql.test.SQLTestUtils$class.withTable(SQLTestUtils.scala:168)
>   at 
> org.apache.spark.sql.StatisticsColumnSuite.withTable(StatisticsColumnSuite.scala:30)
>   at 
> org.apache.spark.sql.StatisticsTest$class.checkColStats(StatisticsTest.scala:33)
>   at 
> org.apache.spark.sql.StatisticsColumnSuite.checkColStats(StatisticsColumnSuite.scala:30)
>   at 
> org.apache.spark.sql.StatisticsColumnSuite$$anonfun$7.apply$mcV$sp(StatisticsColumnSuite.scala:171)
>   at 
> org.apache.spark.sql.StatisticsColumnSuite$$anonfun$7.apply(StatisticsColumnSuite.scala:160)
>   at 
> org.apache.spark.sql.StatisticsColumnSuite$$anonfun$7.apply(StatisticsColumnSuite.scala:160)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-17827) StatisticsColumnSuite failures on big endian platforms

[Pete Robbins (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-17827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15571348#comment-15571348
 ] 

Pete Robbins commented on SPARK-17827:
--

In Statistics,scala

case class StringColumnStat(statRow: InternalRow) {
  println("StringColumnStat: " + statRow)
  // The indices here must be consistent with 
`ColumnStatStruct.stringColumnStat`.
  val numNulls: Long = statRow.getLong(0)
  val avgColLen: Double = statRow.getDouble(1)
  val maxColLen: Long = statRow.getLong(2)   <<<<<< Actual type in 
statRow is Int
  val ndv: Long = statRow.getLong(3)
}

case class BinaryColumnStat(statRow: InternalRow) {
  // The indices here must be consistent with 
`ColumnStatStruct.binaryColumnStat`.
  val numNulls: Long = statRow.getLong(0)
  val avgColLen: Double = statRow.getDouble(1)
  val maxColLen: Long = statRow.getLong(2)<<<<<< Actual type in 
statRow is Int
}

So either the code above should be using getInt for the maxColLen or the code 
generating the row should be creating a Long

> StatisticsColumnSuite failures on big endian platforms
> --
>
> Key: SPARK-17827
> URL: https://issues.apache.org/jira/browse/SPARK-17827
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.1.0
> Environment: big endian
>Reporter: Pete Robbins
>  Labels: big-endian
>
> https://issues.apache.org/jira/browse/SPARK-17073
> introduces new tests/function that fails on big endian platforms
> Failing tests:
>  org.apache.spark.sql.StatisticsColumnSuite.column-level statistics for 
> string column
>  org.apache.spark.sql.StatisticsColumnSuite.column-level statistics for 
> binary column
>  org.apache.spark.sql.StatisticsColumnSuite.column-level statistics for 
> columns with different types
>  org.apache.spark.sql.hive.StatisticsSuite.generate column-level statistics 
> and load them from hive metastore
> all fail in checkColStat eg: 
> java.lang.AssertionError: assertion failed
>   at scala.Predef$.assert(Predef.scala:156)
>   at 
> org.apache.spark.sql.StatisticsTest$.checkColStat(StatisticsTest.scala:92)
>   at 
> org.apache.spark.sql.StatisticsTest$$anonfun$checkColStats$1$$anonfun$apply$mcV$sp$1.apply(StatisticsTest.scala:43)
>   at 
> org.apache.spark.sql.StatisticsTest$$anonfun$checkColStats$1$$anonfun$apply$mcV$sp$1.apply(StatisticsTest.scala:40)
>   at scala.collection.immutable.List.foreach(List.scala:381)
>   at 
> org.apache.spark.sql.StatisticsTest$$anonfun$checkColStats$1.apply$mcV$sp(StatisticsTest.scala:40)
>   at 
> org.apache.spark.sql.test.SQLTestUtils$class.withTable(SQLTestUtils.scala:168)
>   at 
> org.apache.spark.sql.StatisticsColumnSuite.withTable(StatisticsColumnSuite.scala:30)
>   at 
> org.apache.spark.sql.StatisticsTest$class.checkColStats(StatisticsTest.scala:33)
>   at 
> org.apache.spark.sql.StatisticsColumnSuite.checkColStats(StatisticsColumnSuite.scala:30)
>   at 
> org.apache.spark.sql.StatisticsColumnSuite$$anonfun$7.apply$mcV$sp(StatisticsColumnSuite.scala:171)
>   at 
> org.apache.spark.sql.StatisticsColumnSuite$$anonfun$7.apply(StatisticsColumnSuite.scala:160)
>   at 
> org.apache.spark.sql.StatisticsColumnSuite$$anonfun$7.apply(StatisticsColumnSuite.scala:160)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-17827) StatisticsColumnSuite failures on big endian platforms

[Pete Robbins (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-17827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15569048#comment-15569048
 ] 

Pete Robbins commented on SPARK-17827:
--

So this looks like the max field is being written as an Int into the UnsafeRow 
but is later read as a Long. Code stack to the write:

java.lang.Thread.dumpStack(Thread.java:462)
at 
org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter.write(UnsafeRowWriter.java:149)
at 
org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown
 Source)
at 
org.apache.spark.sql.execution.aggregate.AggregationIterator$$anonfun$generateResultProjection$1.apply(AggregationIterator.scala:232)
at 
org.apache.spark.sql.execution.aggregate.AggregationIterator$$anonfun$generateResultProjection$1.apply(AggregationIterator.scala:221)
at 
org.apache.spark.sql.execution.aggregate.TungstenAggregationIterator.next(TungstenAggregationIterator.scala:392)
at 
org.apache.spark.sql.execution.aggregate.TungstenAggregationIterator.next(TungstenAggregationIterator.scala:79)
at scala.collection.Iterator$class.foreach(Iterator.scala:893)
at 
org.apache.spark.sql.execution.aggregate.AggregationIterator.foreach(AggregationIterator.scala:35)
at 
scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59)
at 
scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:104)
at 
scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:48)
at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:310)
at 
org.apache.spark.sql.execution.aggregate.AggregationIterator.to(AggregationIterator.scala:35)
at 
scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:302)
at 
org.apache.spark.sql.execution.aggregate.AggregationIterator.toBuffer(AggregationIterator.scala:35)
at 
scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:289)
at 
org.apache.spark.sql.execution.aggregate.AggregationIterator.toArray(AggregationIterator.scala:35)
at 
org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$13.apply(RDD.scala:912)
at 
org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$13.apply(RDD.scala:912)
at 
org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1927)
at 
org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1927)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1153)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.lang.Thread.run(Thread.java:785)

> StatisticsColumnSuite failures on big endian platforms
> --
>
> Key: SPARK-17827
> URL: https://issues.apache.org/jira/browse/SPARK-17827
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.1.0
> Environment: big endian
>    Reporter: Pete Robbins
>  Labels: big-endian
>
> https://issues.apache.org/jira/browse/SPARK-17073
> introduces new tests/function that fails on big endian platforms
> Failing tests:
>  org.apache.spark.sql.StatisticsColumnSuite.column-level statistics for 
> string column
>  org.apache.spark.sql.StatisticsColumnSuite.column-level statistics for 
> binary column
>  org.apache.spark.sql.StatisticsColumnSuite.column-level statistics for 
> columns with different types
>  org.apache.spark.sql.hive.StatisticsSuite.generate column-level statistics 
> and load them from hive metastore
> all fail in checkColStat eg: 
> java.lang.AssertionError: assertion failed
>   at scala.Predef$.assert(Predef.scala:156)
>   at 
> org.apache.spark.sql.StatisticsTest$.checkColStat(StatisticsTest.scala:92)
>   at 
> org.apache.spark.sql.StatisticsTest$$anonfun$checkColStats$1$$anonfun$apply$mcV$sp$1.apply(StatisticsTest.scala:43)
>   at 
> org.apache.spark.sql.StatisticsTest$$anonfun$checkColStats$1$$anonfun$apply$mcV$sp$1.apply(StatisticsTest.scala:40)
>   at scala.collection.immutable.List.foreach(List.scala:381)
>   at 
> org.apache.spark.sql.StatisticsTest$$anonfun$checkColStats$1.apply$mcV$sp(StatisticsTest.scala:40)
>   at 
> org.apache.spark.sql.test.SQLTestUtils$class.withTable(SQLTestUtils.scala:168)
>   at 
> org.apache.spark.sql.StatisticsColumnSuite.withTable(Stati

[jira] [Commented] (SPARK-17827) StatisticsColumnSuite failures on big endian platforms

[Pete Robbins (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-17827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15561849#comment-15561849
 ] 

Pete Robbins commented on SPARK-17827:
--

[~ZenWzh] Any ideas what code introduced that could cause endian issues? This 
is usually something like writing a field as one type but reading it as another 
eg putLong but then readInt.

> StatisticsColumnSuite failures on big endian platforms
> --
>
> Key: SPARK-17827
> URL: https://issues.apache.org/jira/browse/SPARK-17827
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.1.0
> Environment: big endian
>    Reporter: Pete Robbins
>  Labels: big-endian
>
> https://issues.apache.org/jira/browse/SPARK-17073
> introduces new tests/function that fails on big endian platforms
> Failing tests:
>  org.apache.spark.sql.StatisticsColumnSuite.column-level statistics for 
> string column
>  org.apache.spark.sql.StatisticsColumnSuite.column-level statistics for 
> binary column
>  org.apache.spark.sql.StatisticsColumnSuite.column-level statistics for 
> columns with different types
>  org.apache.spark.sql.hive.StatisticsSuite.generate column-level statistics 
> and load them from hive metastore
> all fail in checkColStat eg: 
> java.lang.AssertionError: assertion failed
>   at scala.Predef$.assert(Predef.scala:156)
>   at 
> org.apache.spark.sql.StatisticsTest$.checkColStat(StatisticsTest.scala:92)
>   at 
> org.apache.spark.sql.StatisticsTest$$anonfun$checkColStats$1$$anonfun$apply$mcV$sp$1.apply(StatisticsTest.scala:43)
>   at 
> org.apache.spark.sql.StatisticsTest$$anonfun$checkColStats$1$$anonfun$apply$mcV$sp$1.apply(StatisticsTest.scala:40)
>   at scala.collection.immutable.List.foreach(List.scala:381)
>   at 
> org.apache.spark.sql.StatisticsTest$$anonfun$checkColStats$1.apply$mcV$sp(StatisticsTest.scala:40)
>   at 
> org.apache.spark.sql.test.SQLTestUtils$class.withTable(SQLTestUtils.scala:168)
>   at 
> org.apache.spark.sql.StatisticsColumnSuite.withTable(StatisticsColumnSuite.scala:30)
>   at 
> org.apache.spark.sql.StatisticsTest$class.checkColStats(StatisticsTest.scala:33)
>   at 
> org.apache.spark.sql.StatisticsColumnSuite.checkColStats(StatisticsColumnSuite.scala:30)
>   at 
> org.apache.spark.sql.StatisticsColumnSuite$$anonfun$7.apply$mcV$sp(StatisticsColumnSuite.scala:171)
>   at 
> org.apache.spark.sql.StatisticsColumnSuite$$anonfun$7.apply(StatisticsColumnSuite.scala:160)
>   at 
> org.apache.spark.sql.StatisticsColumnSuite$$anonfun$7.apply(StatisticsColumnSuite.scala:160)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Re: Monitoring system extensibility

[Pete Robbins]

Yes I agree. I'm not sure how important this is anyway. It's a little
annoying but easy to work around.

On Mon, 10 Oct 2016 at 09:01 Reynold Xin <r...@databricks.com> wrote:

> I just took a quick look and set a target version on the JIRA. But Pete I
> think the primary problem with the JIRA and pull request is that it really
> just argues (or implements) opening up a private API, which is a valid
> point but there are a lot more that needs to be done before making some
> private API public.
>
> At the very least, we need to answer the following:
>
> 1. Is the existing API maintainable? E.g. Is it OK to just expose coda
> hale metrics in the API? Do we need to worry about dependency conflicts?
> Should we wrap it?
>
> 2. Is the existing API sufficiently general (to cover use cases)? What
> about security related setup?
>
>
>
>
> On Fri, Oct 7, 2016 at 2:03 AM, Pete Robbins <robbin...@gmail.com> wrote:
>
> Which has happened. The last comment being in August with someone saying
> it was important to them. They PR has been around since March and despite a
> request to be reviewed has not got any committer's attention. Without that,
> it is going nowhere. The historic Jiras requesting other sinks such as
> Kafka, OpenTSBD etc have also been ignored.
>
> So for now we continue creating classes in o.a.s package.
>
> On Fri, 7 Oct 2016 at 09:50 Reynold Xin <r...@databricks.com> wrote:
>
> So to be constructive and in order to actually open up these APIs, it
> would be useful for users to comment on the JIRA ticket on their use cases
> (rather than "I want this to be public"), and then we can design an API
> that would address those use cases. In some cases the solution is to just
> make the existing internal API public. But turning some internal API public
> without thinking about whether those APIs are sufficiently expressive and
> maintainable is not a great way to design APIs in general.
>
> On Friday, October 7, 2016, Pete Robbins <robbin...@gmail.com> wrote:
>
> I brought this up last year and there was a Jira raised:
> https://issues.apache.org/jira/browse/SPARK-14151
>
> For now I just have my SInk and Source in an o.a.s package name which is
> not ideal but the only way round this.
>
> On Fri, 7 Oct 2016 at 08:30 Reynold Xin <r...@databricks.com> wrote:
>
> They have always been private, haven't they?
>
>
> https://github.com/apache/spark/blob/branch-1.6/core/src/main/scala/org/apache/spark/metrics/source/Source.scala
>
>
>
> On Thu, Oct 6, 2016 at 7:38 AM, Alexander Oleynikov <oleyniko...@gmail.com
> > wrote:
>
> Hi.
>
> As of v2.0.1, the traits `org.apache.spark.metrics.source.Source` and
> `org.apache.spark.metrics.sink.Sink` are defined as private to ‘spark’
> package, so it becomes troublesome to create a new implementation in the
> user’s code (but still possible in a hacky way).
> This seems to be the only missing piece to allow extension of the metrics
> system and I wonder whether is was conscious design decision to limit the
> visibility. Is it possible to broaden the visibility scope for these traits
> in the future versions?
>
> Thanks,
> Alexander
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
>
>
>

[jira] [Commented] (SPARK-17827) StatisticsColumnSuite failures on big endian platforms

[Pete Robbins (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-17827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15554825#comment-15554825
 ] 

Pete Robbins commented on SPARK-17827:
--

I'm investigating this

> StatisticsColumnSuite failures on big endian platforms
> --
>
> Key: SPARK-17827
> URL: https://issues.apache.org/jira/browse/SPARK-17827
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.1.0
> Environment: big endian
>    Reporter: Pete Robbins
>  Labels: big-endian
>
> https://issues.apache.org/jira/browse/SPARK-17073
> introduces new tests/function that fails on big endian platforms
> Failing tests:
>  org.apache.spark.sql.StatisticsColumnSuite.column-level statistics for 
> string column
>  org.apache.spark.sql.StatisticsColumnSuite.column-level statistics for 
> binary column
>  org.apache.spark.sql.StatisticsColumnSuite.column-level statistics for 
> columns with different types
>  org.apache.spark.sql.hive.StatisticsSuite.generate column-level statistics 
> and load them from hive metastore
> all fail in checkColStat eg: 
> java.lang.AssertionError: assertion failed
>   at scala.Predef$.assert(Predef.scala:156)
>   at 
> org.apache.spark.sql.StatisticsTest$.checkColStat(StatisticsTest.scala:92)
>   at 
> org.apache.spark.sql.StatisticsTest$$anonfun$checkColStats$1$$anonfun$apply$mcV$sp$1.apply(StatisticsTest.scala:43)
>   at 
> org.apache.spark.sql.StatisticsTest$$anonfun$checkColStats$1$$anonfun$apply$mcV$sp$1.apply(StatisticsTest.scala:40)
>   at scala.collection.immutable.List.foreach(List.scala:381)
>   at 
> org.apache.spark.sql.StatisticsTest$$anonfun$checkColStats$1.apply$mcV$sp(StatisticsTest.scala:40)
>   at 
> org.apache.spark.sql.test.SQLTestUtils$class.withTable(SQLTestUtils.scala:168)
>   at 
> org.apache.spark.sql.StatisticsColumnSuite.withTable(StatisticsColumnSuite.scala:30)
>   at 
> org.apache.spark.sql.StatisticsTest$class.checkColStats(StatisticsTest.scala:33)
>   at 
> org.apache.spark.sql.StatisticsColumnSuite.checkColStats(StatisticsColumnSuite.scala:30)
>   at 
> org.apache.spark.sql.StatisticsColumnSuite$$anonfun$7.apply$mcV$sp(StatisticsColumnSuite.scala:171)
>   at 
> org.apache.spark.sql.StatisticsColumnSuite$$anonfun$7.apply(StatisticsColumnSuite.scala:160)
>   at 
> org.apache.spark.sql.StatisticsColumnSuite$$anonfun$7.apply(StatisticsColumnSuite.scala:160)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Created] (SPARK-17827) StatisticsColumnSuite failures on big endian platforms

[Pete Robbins (JIRA)]

Pete Robbins created SPARK-17827:


 Summary: StatisticsColumnSuite failures on big endian platforms
 Key: SPARK-17827
 URL: https://issues.apache.org/jira/browse/SPARK-17827
 Project: Spark
  Issue Type: Bug
  Components: SQL
Affects Versions: 2.1.0
 Environment: big endian
Reporter: Pete Robbins


https://issues.apache.org/jira/browse/SPARK-17073

introduces new tests/function that fails on big endian platforms

Failing tests:

 org.apache.spark.sql.StatisticsColumnSuite.column-level statistics for string 
column
 org.apache.spark.sql.StatisticsColumnSuite.column-level statistics for binary 
column
 org.apache.spark.sql.StatisticsColumnSuite.column-level statistics for columns 
with different types
 org.apache.spark.sql.hive.StatisticsSuite.generate column-level statistics and 
load them from hive metastore

all fail in checkColStat eg: 
java.lang.AssertionError: assertion failed
  at scala.Predef$.assert(Predef.scala:156)
  at 
org.apache.spark.sql.StatisticsTest$.checkColStat(StatisticsTest.scala:92)
  at 
org.apache.spark.sql.StatisticsTest$$anonfun$checkColStats$1$$anonfun$apply$mcV$sp$1.apply(StatisticsTest.scala:43)
  at 
org.apache.spark.sql.StatisticsTest$$anonfun$checkColStats$1$$anonfun$apply$mcV$sp$1.apply(StatisticsTest.scala:40)
  at scala.collection.immutable.List.foreach(List.scala:381)
  at 
org.apache.spark.sql.StatisticsTest$$anonfun$checkColStats$1.apply$mcV$sp(StatisticsTest.scala:40)
  at 
org.apache.spark.sql.test.SQLTestUtils$class.withTable(SQLTestUtils.scala:168)
  at 
org.apache.spark.sql.StatisticsColumnSuite.withTable(StatisticsColumnSuite.scala:30)
  at 
org.apache.spark.sql.StatisticsTest$class.checkColStats(StatisticsTest.scala:33)
  at 
org.apache.spark.sql.StatisticsColumnSuite.checkColStats(StatisticsColumnSuite.scala:30)
  at 
org.apache.spark.sql.StatisticsColumnSuite$$anonfun$7.apply$mcV$sp(StatisticsColumnSuite.scala:171)
  at 
org.apache.spark.sql.StatisticsColumnSuite$$anonfun$7.apply(StatisticsColumnSuite.scala:160)
  at 
org.apache.spark.sql.StatisticsColumnSuite$$anonfun$7.apply(StatisticsColumnSuite.scala:160)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Re: Monitoring system extensibility

[Pete Robbins]

Which has happened. The last comment being in August with someone saying it
was important to them. They PR has been around since March and despite a
request to be reviewed has not got any committer's attention. Without that,
it is going nowhere. The historic Jiras requesting other sinks such as
Kafka, OpenTSBD etc have also been ignored.

So for now we continue creating classes in o.a.s package.

On Fri, 7 Oct 2016 at 09:50 Reynold Xin <r...@databricks.com> wrote:

> So to be constructive and in order to actually open up these APIs, it
> would be useful for users to comment on the JIRA ticket on their use cases
> (rather than "I want this to be public"), and then we can design an API
> that would address those use cases. In some cases the solution is to just
> make the existing internal API public. But turning some internal API public
> without thinking about whether those APIs are sufficiently expressive and
> maintainable is not a great way to design APIs in general.
>
> On Friday, October 7, 2016, Pete Robbins <robbin...@gmail.com> wrote:
>
> I brought this up last year and there was a Jira raised:
> https://issues.apache.org/jira/browse/SPARK-14151
>
> For now I just have my SInk and Source in an o.a.s package name which is
> not ideal but the only way round this.
>
> On Fri, 7 Oct 2016 at 08:30 Reynold Xin <r...@databricks.com> wrote:
>
> They have always been private, haven't they?
>
>
> https://github.com/apache/spark/blob/branch-1.6/core/src/main/scala/org/apache/spark/metrics/source/Source.scala
>
>
>
> On Thu, Oct 6, 2016 at 7:38 AM, Alexander Oleynikov <oleyniko...@gmail.com
> > wrote:
>
> Hi.
>
> As of v2.0.1, the traits `org.apache.spark.metrics.source.Source` and
> `org.apache.spark.metrics.sink.Sink` are defined as private to ‘spark’
> package, so it becomes troublesome to create a new implementation in the
> user’s code (but still possible in a hacky way).
> This seems to be the only missing piece to allow extension of the metrics
> system and I wonder whether is was conscious design decision to limit the
> visibility. Is it possible to broaden the visibility scope for these traits
> in the future versions?
>
> Thanks,
> Alexander
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
>
>

Re: Monitoring system extensibility

[Pete Robbins]

I brought this up last year and there was a Jira raised:
https://issues.apache.org/jira/browse/SPARK-14151

For now I just have my SInk and Source in an o.a.s package name which is
not ideal but the only way round this.

On Fri, 7 Oct 2016 at 08:30 Reynold Xin  wrote:

> They have always been private, haven't they?
>
>
> https://github.com/apache/spark/blob/branch-1.6/core/src/main/scala/org/apache/spark/metrics/source/Source.scala
>
>
>
> On Thu, Oct 6, 2016 at 7:38 AM, Alexander Oleynikov  > wrote:
>
> Hi.
>
> As of v2.0.1, the traits `org.apache.spark.metrics.source.Source` and
> `org.apache.spark.metrics.sink.Sink` are defined as private to ‘spark’
> package, so it becomes troublesome to create a new implementation in the
> user’s code (but still possible in a hacky way).
> This seems to be the only missing piece to allow extension of the metrics
> system and I wonder whether is was conscious design decision to limit the
> visibility. Is it possible to broaden the visibility scope for these traits
> in the future versions?
>
> Thanks,
> Alexander
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
>
>

Re: Spark 2.0.0 preview docs uploaded

[Pete Robbins]

Are there any 'work in progress' release notes for 2.0.0 yet? I don't see
anything in the rc docs like "what's new" or "migration guide"?

On Thu, 9 Jun 2016 at 10:06 Sean Owen <so...@cloudera.com> wrote:

> Available but mostly as JIRA output:
> https://spark.apache.org/news/spark-2.0.0-preview.html
>
> On Thu, Jun 9, 2016 at 7:33 AM, Pete Robbins <robbin...@gmail.com> wrote:
> > It would be nice to have a "what's new in 2.0.0" equivalent to
> > https://spark.apache.org/releases/spark-release-1-6-0.html available or
> am I
> > just missing it?
> >
> > On Wed, 8 Jun 2016 at 13:15 Sean Owen <so...@cloudera.com> wrote:
> >>
> >> OK, this is done:
> >>
> >> http://spark.apache.org/documentation.html
> >> http://spark.apache.org/docs/2.0.0-preview/
> >> http://spark.apache.org/docs/preview/
> >>
> >> On Tue, Jun 7, 2016 at 4:59 PM, Shivaram Venkataraman
> >> <shiva...@eecs.berkeley.edu> wrote:
> >> > As far as I know the process is just to copy docs/_site from the build
> >> > to the appropriate location in the SVN repo (i.e.
> >> > site/docs/2.0.0-preview).
> >> >
> >> > Thanks
> >> > Shivaram
> >> >
> >> > On Tue, Jun 7, 2016 at 8:14 AM, Sean Owen <so...@cloudera.com> wrote:
> >> >> As a stop-gap, I can edit that page to have a small section about
> >> >> preview releases and point to the nightly docs.
> >> >>
> >> >> Not sure who has the power to push 2.0.0-preview to site/docs, but,
> if
> >> >> that's done then we can symlink "preview" in that dir to it and be
> >> >> done, and update this section about preview docs accordingly.
> >> >>
> >>
> >> -
> >> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
> >> For additional commands, e-mail: dev-h...@spark.apache.org
> >>
> >
>

Stability of branch-2.0

[Pete Robbins]

It looks like the vote on 2.0-rc2 will not pass so there will be a new RC
from the 2.0 branch. With a project management hat on I would expect to see
only fixes to the remaining blocker issues or high priority bug fixes going
into the 2.0 branch as defect burn down. However, I see several new
functional PRs which were originally targeted at 2.1 being merged into
branch-2.0 (eg children of https://issues.apache.org/jira/browse/SPARK-16275
) and these will now be in the upcoming 2.0-RC3.

I assume these are zero risk changes that will not further delay a 2.0
release.

Cheers,

Re: branch-2.0 build failure

[Pete Robbins]

Ok, thanks. I'll await it appearing.

On Thu, 30 Jun 2016 at 14:51 Sean Owen <so...@cloudera.com> wrote:

> TD has literally just merged the fix.
>
> On Thu, Jun 30, 2016 at 2:37 PM, Pete Robbins <robbin...@gmail.com> wrote:
> > Our build on branch-2.0 is failing after the PR for updating kafka to
> 0.10.
> > The new kafka pom.xml files are naming the parent version as
> 2.0.0-SNAPSHOT
> > but the branch 2.0 poms have been updated to 2.0.1-SNAPSHOT after the rc1
> > cut. Shouldn't the pom versions remain as 2.0.0-SNAPSHOT until a 2.0.0
> has
> > been released?
>

branch-2.0 build failure

[Pete Robbins]

Our build on branch-2.0 is failing after the PR for updating kafka to 0.10.
The new kafka pom.xml files are naming the parent version as 2.0.0-SNAPSHOT
but the branch 2.0 poms have been updated to 2.0.1-SNAPSHOT after the rc1
cut. Shouldn't the pom versions remain as 2.0.0-SNAPSHOT until a 2.0.0 has
been released?

Re: [VOTE] Release Apache Spark 2.0.0 (RC1)

[Pete Robbins]

I'm also seeing some of these same failures:

- spilling with compression *** FAILED ***
I have seen this occassionaly

- to UTC timestamp *** FAILED ***
This was fixed yesterday in branch-2.0 (
https://issues.apache.org/jira/browse/SPARK-16078)

- offset recovery *** FAILED ***
Haven't seen this for a while and thought the flaky test was fixed but it
popped up again in one of our builds.

StateStoreSuite:
- maintenance *** FAILED ***
Just seen this has been failing for last 2 days on one build machine (linux
amd64)

On 23 June 2016 at 08:51, Sean Owen  wrote:

> First pass of feedback on the RC: all the sigs, hashes, etc are fine.
> Licensing is up to date to the best of my knowledge.
>
> I'm hitting test failures, some of which may be spurious. Just putting
> them out there to see if they ring bells. This is Java 8 on Ubuntu 16.
>
>
> - spilling with compression *** FAILED ***
>   java.lang.Exception: Test failed with compression using codec
> org.apache.spark.io.SnappyCompressionCodec:
> assertion failed: expected cogroup to spill, but did not
>   at scala.Predef$.assert(Predef.scala:170)
>   at org.apache.spark.TestUtils$.assertSpilled(TestUtils.scala:170)
>   at org.apache.spark.util.collection.ExternalAppendOnlyMapSuite.org
> $apache$spark$util$collection$ExternalAppendOnlyMapSuite$$testSimpleSpilling(ExternalAppendOnlyMapSuite.scala:263)
> ...
>
> I feel like I've seen this before, and see some possibly relevant
> fixes, but they're in 2.0.0 already:
> https://github.com/apache/spark/pull/10990
> Is this something where a native library needs to be installed or
> something?
>
>
> - to UTC timestamp *** FAILED ***
>   "2016-03-13 [02]:00:00.0" did not equal "2016-03-13 [10]:00:00.0"
> (DateTimeUtilsSuite.scala:506)
>
> I know, we talked about this for the 1.6.2 RC, but I reproduced this
> locally too. I will investigate, could still be spurious.
>
>
> StateStoreSuite:
> - maintenance *** FAILED ***
>   The code passed to eventually never returned normally. Attempted 627
> times over 10.000180116 seconds. Last failure message:
> StateStoreSuite.this.fileExists(provider, 1L, false) was true earliest
> file not deleted. (StateStoreSuite.scala:395)
>
> No idea.
>
>
> - offset recovery *** FAILED ***
>   The code passed to eventually never returned normally. Attempted 197
> times over 10.040864806 seconds. Last failure message:
> strings.forall({
> ((x$1: Any) => DirectKafkaStreamSuite.collectedData.contains(x$1))
>   }) was false. (DirectKafkaStreamSuite.scala:250)
>
> Also something that was possibly fixed already for 2.0.0 and that I
> just back-ported into 1.6. Could be just a very similar failure.
>
> On Wed, Jun 22, 2016 at 2:26 AM, Reynold Xin  wrote:
> > Please vote on releasing the following candidate as Apache Spark version
> > 2.0.0. The vote is open until Friday, June 24, 2016 at 19:00 PDT and
> passes
> > if a majority of at least 3+1 PMC votes are cast.
> >
> > [ ] +1 Release this package as Apache Spark 2.0.0
> > [ ] -1 Do not release this package because ...
> >
> >
> > The tag to be voted on is v2.0.0-rc1
> > (0c66ca41afade6db73c9aeddd5aed6e5dcea90df).
> >
> > This release candidate resolves ~2400 issues:
> > https://s.apache.org/spark-2.0.0-rc1-jira
> >
> > The release files, including signatures, digests, etc. can be found at:
> > http://people.apache.org/~pwendell/spark-releases/spark-2.0.0-rc1-bin/
> >
> > Release artifacts are signed with the following key:
> > https://people.apache.org/keys/committer/pwendell.asc
> >
> > The staging repository for this release can be found at:
> > https://repository.apache.org/content/repositories/orgapachespark-1187/
> >
> > The documentation corresponding to this release can be found at:
> > http://people.apache.org/~pwendell/spark-releases/spark-2.0.0-rc1-docs/
> >
> >
> > ===
> > == How can I help test this release? ==
> > ===
> > If you are a Spark user, you can help us test this release by taking an
> > existing Spark workload and running on this release candidate, then
> > reporting any regressions from 1.x.
> >
> > 
> > == What justifies a -1 vote for this release? ==
> > 
> > Critical bugs impacting major functionalities.
> >
> > Bugs already present in 1.x, missing features, or bugs related to new
> > features will not necessarily block this release. Note that historically
> > Spark documentation has been published on the website separately from the
> > main release so we do not need to block the release due to documentation
> > errors either.
> >
> >
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
> For additional commands, e-mail: dev-h...@spark.apache.org
>
>

Re: [VOTE] Release Apache Spark 1.6.2 (RC2)

[Pete Robbins]

This has failed on our 1.6 stream builds regularly. (
https://issues.apache.org/jira/browse/SPARK-6005) looks fixed in 2.0?

On Wed, 22 Jun 2016 at 11:15 Sean Owen  wrote:

> Oops, one more in the "does anybody else see this" department:
>
> - offset recovery *** FAILED ***
>   recoveredOffsetRanges.forall(((or: (org.apache.spark.streaming.Time,
> Array[org.apache.spark.streaming.kafka.OffsetRange])) =>
>
> earlierOffsetRangesAsSets.contains(scala.Tuple2.apply[org.apache.spark.streaming.Time,
>
> scala.collection.immutable.Set[org.apache.spark.streaming.kafka.OffsetRange]](or._1,
>
> scala.this.Predef.refArrayOps[org.apache.spark.streaming.kafka.OffsetRange](or._2).toSet[org.apache.spark.streaming.kafka.OffsetRange]
> was false Recovered ranges are not the same as the ones generated
> (DirectKafkaStreamSuite.scala:301)
>
> This actually fails consistently for me too in the Kafka integration
> code. Not timezone related, I think.
>
> On Wed, Jun 22, 2016 at 9:02 AM, Sean Owen  wrote:
> > I'm fairly convinced this error and others that appear timestamp
> > related are an environment problem. This test and method have been
> > present for several Spark versions, without change. I reviewed the
> > logic and it seems sound, explicitly setting the time zone correctly.
> > I am not sure why it behaves differently on this machine.
> >
> > I'd give a +1 to this release if nobody else is seeing errors like
> > this. The sigs, hashes, other tests pass for me.
> >
> > On Tue, Jun 21, 2016 at 6:49 PM, Sean Owen  wrote:
> >> UIUtilsSuite:
> >> - formatBatchTime *** FAILED ***
> >>   "2015/05/14 [14]:04:40" did not equal "2015/05/14 [21]:04:40"
> >> (UIUtilsSuite.scala:73)
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
> For additional commands, e-mail: dev-h...@spark.apache.org
>
>

Re: [VOTE] Release Apache Spark 1.6.2 (RC2)

[Pete Robbins]

It breaks Spark running on machines with less than 3 cores/threads, which
may be rare, and it is maybe an edge case.

Personally, I like to fix known bugs and the fact there are other blocking
methods in event loops actually makes it worse not to fix ones that you
know about.

Probably not a blocker to release though but that's your call.

Cheers,

On Tue, Jun 21, 2016 at 6:40 PM Shixiong(Ryan) Zhu <shixi...@databricks.com>
wrote:

> Hey Pete,
>
> I didn't backport it to 1.6 because it just affects tests in most cases.
> I'm sure we also have other places calling blocking methods in the event
> loops, so similar issues are still there even after applying this patch.
> Hence, I don't think it's a blocker for 1.6.2.
>
> On Tue, Jun 21, 2016 at 2:57 AM, Pete Robbins <robbin...@gmail.com> wrote:
>
>> The PR (https://github.com/apache/spark/pull/13055) to fix
>> https://issues.apache.org/jira/browse/SPARK-15262 was applied to 1.6.2
>> however this fix caused another issue
>> https://issues.apache.org/jira/browse/SPARK-15606 the fix for which (
>> https://github.com/apache/spark/pull/13355) has not been backported to
>> the 1.6 branch so I'm now seeing the same failure in 1.6.2
>>
>> Cheers,
>>
>> On Mon, 20 Jun 2016 at 05:25 Reynold Xin <r...@databricks.com> wrote:
>>
>>> Please vote on releasing the following candidate as Apache Spark version
>>> 1.6.2. The vote is open until Wednesday, June 22, 2016 at 22:00 PDT and
>>> passes if a majority of at least 3+1 PMC votes are cast.
>>>
>>> [ ] +1 Release this package as Apache Spark 1.6.2
>>> [ ] -1 Do not release this package because ...
>>>
>>>
>>> The tag to be voted on is v1.6.2-rc2
>>> (54b1121f351f056d6b67d2bb4efe0d553c0f7482)
>>>
>>> The release files, including signatures, digests, etc. can be found at:
>>> http://people.apache.org/~pwendell/spark-releases/spark-1.6.2-rc2-bin/
>>>
>>> Release artifacts are signed with the following key:
>>> https://people.apache.org/keys/committer/pwendell.asc
>>>
>>> The staging repository for this release can be found at:
>>> https://repository.apache.org/content/repositories/orgapachespark-1186/
>>>
>>> The documentation corresponding to this release can be found at:
>>> http://people.apache.org/~pwendell/spark-releases/spark-1.6.2-rc2-docs/
>>>
>>>
>>> ===
>>> == How can I help test this release? ==
>>> ===
>>> If you are a Spark user, you can help us test this release by taking an
>>> existing Spark workload and running on this release candidate, then
>>> reporting any regressions from 1.6.1.
>>>
>>> 
>>> == What justifies a -1 vote for this release? ==
>>> 
>>> This is a maintenance release in the 1.6.x series.  Bugs already present
>>> in 1.6.1, missing features, or bugs related to new features will not
>>> necessarily block this release.
>>>
>>>
>>>
>>>
>

Re: [VOTE] Release Apache Spark 1.6.2 (RC2)

[Pete Robbins]

The PR (https://github.com/apache/spark/pull/13055) to fix
https://issues.apache.org/jira/browse/SPARK-15262 was applied to 1.6.2
however this fix caused another issue
https://issues.apache.org/jira/browse/SPARK-15606 the fix for which (
https://github.com/apache/spark/pull/13355) has not been backported to the
1.6 branch so I'm now seeing the same failure in 1.6.2

Cheers,

On Mon, 20 Jun 2016 at 05:25 Reynold Xin  wrote:

> Please vote on releasing the following candidate as Apache Spark version
> 1.6.2. The vote is open until Wednesday, June 22, 2016 at 22:00 PDT and
> passes if a majority of at least 3+1 PMC votes are cast.
>
> [ ] +1 Release this package as Apache Spark 1.6.2
> [ ] -1 Do not release this package because ...
>
>
> The tag to be voted on is v1.6.2-rc2
> (54b1121f351f056d6b67d2bb4efe0d553c0f7482)
>
> The release files, including signatures, digests, etc. can be found at:
> http://people.apache.org/~pwendell/spark-releases/spark-1.6.2-rc2-bin/
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/pwendell.asc
>
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapachespark-1186/
>
> The documentation corresponding to this release can be found at:
> http://people.apache.org/~pwendell/spark-releases/spark-1.6.2-rc2-docs/
>
>
> ===
> == How can I help test this release? ==
> ===
> If you are a Spark user, you can help us test this release by taking an
> existing Spark workload and running on this release candidate, then
> reporting any regressions from 1.6.1.
>
> 
> == What justifies a -1 vote for this release? ==
> 
> This is a maintenance release in the 1.6.x series.  Bugs already present
> in 1.6.1, missing features, or bugs related to new features will not
> necessarily block this release.
>
>
>
>

[jira] [Updated] (SPARK-15822) segmentation violation in o.a.s.unsafe.types.UTF8String

[Pete Robbins (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-15822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pete Robbins updated SPARK-15822:
-
Component/s: SQL

> segmentation violation in o.a.s.unsafe.types.UTF8String 
> 
>
> Key: SPARK-15822
> URL: https://issues.apache.org/jira/browse/SPARK-15822
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.0.0
> Environment: linux amd64
> openjdk version "1.8.0_91"
> OpenJDK Runtime Environment (build 1.8.0_91-b14)
> OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode)
>Reporter: Pete Robbins
>Assignee: Herman van Hovell
>Priority: Blocker
>
> Executors fail with segmentation violation while running application with
> spark.memory.offHeap.enabled true
> spark.memory.offHeap.size 512m
> Also now reproduced with 
> spark.memory.offHeap.enabled false
> {noformat}
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  SIGSEGV (0xb) at pc=0x7f4559b4d4bd, pid=14182, tid=139935319750400
> #
> # JRE version: OpenJDK Runtime Environment (8.0_91-b14) (build 1.8.0_91-b14)
> # Java VM: OpenJDK 64-Bit Server VM (25.91-b14 mixed mode linux-amd64 
> compressed oops)
> # Problematic frame:
> # J 4816 C2 
> org.apache.spark.unsafe.types.UTF8String.compareTo(Lorg/apache/spark/unsafe/types/UTF8String;)I
>  (64 bytes) @ 0x7f4559b4d4bd [0x7f4559b4d460+0x5d]
> {noformat}
> We initially saw this on IBM java on PowerPC box but is recreatable on linux 
> with OpenJDK. On linux with IBM Java 8 we see a null pointer exception at the 
> same code point:
> {noformat}
> 16/06/08 11:14:58 ERROR Executor: Exception in task 1.0 in stage 5.0 (TID 48)
> java.lang.NullPointerException
>   at 
> org.apache.spark.unsafe.types.UTF8String.compareTo(UTF8String.java:831)
>   at org.apache.spark.unsafe.types.UTF8String.compare(UTF8String.java:844)
>   at 
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.findNextInnerJoinRows$(Unknown
>  Source)
>   at 
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown
>  Source)
>   at 
> org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
>   at 
> org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$doExecute$2$$anon$2.hasNext(WholeStageCodegenExec.scala:377)
>   at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
>   at 
> scala.collection.convert.Wrappers$IteratorWrapper.hasNext(Wrappers.scala:30)
>   at org.spark_project.guava.collect.Ordering.leastOf(Ordering.java:664)
>   at org.apache.spark.util.collection.Utils$.takeOrdered(Utils.scala:37)
>   at 
> org.apache.spark.rdd.RDD$$anonfun$takeOrdered$1$$anonfun$30.apply(RDD.scala:1365)
>   at 
> org.apache.spark.rdd.RDD$$anonfun$takeOrdered$1$$anonfun$30.apply(RDD.scala:1362)
>   at 
> org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:757)
>   at 
> org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:757)
>   at 
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
>   at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:318)
>   at org.apache.spark.rdd.RDD.iterator(RDD.scala:282)
>   at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)
>   at org.apache.spark.scheduler.Task.run(Task.scala:85)
>   at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1153)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>   at java.lang.Thread.run(Thread.java:785)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-15822) segmentation violation in o.a.s.unsafe.types.UTF8String

[Pete Robbins (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-15822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15333594#comment-15333594
 ] 

Pete Robbins commented on SPARK-15822:
--

Tracking where the memory is being allocated it is interesting that for the 
Executor that crashes (Executor task launch worker-3) the Unsafe off-heap 
memory is allocated in a much different address range than the other executors. 
I think the generated code is incorrect anyway but may be accidentally passing 
as sometimes the memory in the freed page is still accessible?

{noformat}
org.apache.spark.unsafe.memory.UnsafeMemoryAllocator@7bf8aaef Executor task 
launch worker-3 UnsafeMemoryAllocator.allocated: 262144 at *28900976*
org.apache.spark.unsafe.memory.UnsafeMemoryAllocator@7bf8aaef Executor task 
launch worker-1 UnsafeMemoryAllocator.allocated: 262144 at 140689774260048
org.apache.spark.unsafe.memory.UnsafeMemoryAllocator@7bf8aaef Executor task 
launch worker-4 UnsafeMemoryAllocator.allocated: 262144 at 140689572734864
org.apache.spark.unsafe.memory.UnsafeMemoryAllocator@7bf8aaef Executor task 
launch worker-7 UnsafeMemoryAllocator.allocated: 262144 at 140689908250560
org.apache.spark.unsafe.memory.UnsafeMemoryAllocator@7bf8aaef Executor task 
launch worker-8 UnsafeMemoryAllocator.allocated: 262144 at 140690243746416
org.apache.spark.unsafe.memory.UnsafeMemoryAllocator@7bf8aaef Executor task 
launch worker-0 UnsafeMemoryAllocator.allocated: 262144 at 140690176924448
org.apache.spark.unsafe.memory.UnsafeMemoryAllocator@7bf8aaef Executor task 
launch worker-5 UnsafeMemoryAllocator.allocated: 262144 at 140689773997888
org.apache.spark.unsafe.memory.UnsafeMemoryAllocator@7bf8aaef Executor task 
launch worker-6 UnsafeMemoryAllocator.allocated: 262144 at 140689707058576
{noformat}

> segmentation violation in o.a.s.unsafe.types.UTF8String 
> 
>
> Key: SPARK-15822
> URL: https://issues.apache.org/jira/browse/SPARK-15822
> Project: Spark
>  Issue Type: Bug
>Affects Versions: 2.0.0
> Environment: linux amd64
> openjdk version "1.8.0_91"
> OpenJDK Runtime Environment (build 1.8.0_91-b14)
> OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode)
>Reporter: Pete Robbins
>Assignee: Herman van Hovell
>Priority: Blocker
>
> Executors fail with segmentation violation while running application with
> spark.memory.offHeap.enabled true
> spark.memory.offHeap.size 512m
> Also now reproduced with 
> spark.memory.offHeap.enabled false
> {noformat}
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  SIGSEGV (0xb) at pc=0x7f4559b4d4bd, pid=14182, tid=139935319750400
> #
> # JRE version: OpenJDK Runtime Environment (8.0_91-b14) (build 1.8.0_91-b14)
> # Java VM: OpenJDK 64-Bit Server VM (25.91-b14 mixed mode linux-amd64 
> compressed oops)
> # Problematic frame:
> # J 4816 C2 
> org.apache.spark.unsafe.types.UTF8String.compareTo(Lorg/apache/spark/unsafe/types/UTF8String;)I
>  (64 bytes) @ 0x7f4559b4d4bd [0x7f4559b4d460+0x5d]
> {noformat}
> We initially saw this on IBM java on PowerPC box but is recreatable on linux 
> with OpenJDK. On linux with IBM Java 8 we see a null pointer exception at the 
> same code point:
> {noformat}
> 16/06/08 11:14:58 ERROR Executor: Exception in task 1.0 in stage 5.0 (TID 48)
> java.lang.NullPointerException
>   at 
> org.apache.spark.unsafe.types.UTF8String.compareTo(UTF8String.java:831)
>   at org.apache.spark.unsafe.types.UTF8String.compare(UTF8String.java:844)
>   at 
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.findNextInnerJoinRows$(Unknown
>  Source)
>   at 
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown
>  Source)
>   at 
> org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
>   at 
> org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$doExecute$2$$anon$2.hasNext(WholeStageCodegenExec.scala:377)
>   at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
>   at 
> scala.collection.convert.Wrappers$IteratorWrapper.hasNext(Wrappers.scala:30)
>   at org.spark_project.guava.collect.Ordering.leastOf(Ordering.java:664)
>   at org.apache.spark.util.collection.Utils$.takeOrdered(Utils.scala:37)
>   at 
> org.apache.spark.rdd.RDD$$anonfun$takeOrdered$1$$anonfun$30.apply(RDD.scala:1365)
>   at 
> org.apache.spark.rdd.RDD$$anonfun$takeOrdered$1$$anonfun$30.apply(RDD.scala:1362)
>   at 
> org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.appl

[jira] [Comment Edited] (SPARK-15822) segmentation violation in o.a.s.unsafe.types.UTF8String

[Pete Robbins (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-15822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15333594#comment-15333594
 ] 

Pete Robbins edited comment on SPARK-15822 at 6/16/16 11:23 AM:


Tracking where the memory is being allocated it is interesting that for the 
Executor that crashes (Executor task launch worker-3) the Unsafe off-heap 
memory is allocated in a much different address range than the other executors. 
I think the generated code is incorrect anyway but may be accidentally passing 
as sometimes the memory in the freed page is still accessible?

org.apache.spark.unsafe.memory.UnsafeMemoryAllocator@7bf8aaef Executor task 
launch worker-3 UnsafeMemoryAllocator.allocated: 262144 at *28900976*
org.apache.spark.unsafe.memory.UnsafeMemoryAllocator@7bf8aaef Executor task 
launch worker-1 UnsafeMemoryAllocator.allocated: 262144 at 140689774260048
org.apache.spark.unsafe.memory.UnsafeMemoryAllocator@7bf8aaef Executor task 
launch worker-4 UnsafeMemoryAllocator.allocated: 262144 at 140689572734864
org.apache.spark.unsafe.memory.UnsafeMemoryAllocator@7bf8aaef Executor task 
launch worker-7 UnsafeMemoryAllocator.allocated: 262144 at 140689908250560
org.apache.spark.unsafe.memory.UnsafeMemoryAllocator@7bf8aaef Executor task 
launch worker-8 UnsafeMemoryAllocator.allocated: 262144 at 140690243746416
org.apache.spark.unsafe.memory.UnsafeMemoryAllocator@7bf8aaef Executor task 
launch worker-0 UnsafeMemoryAllocator.allocated: 262144 at 140690176924448
org.apache.spark.unsafe.memory.UnsafeMemoryAllocator@7bf8aaef Executor task 
launch worker-5 UnsafeMemoryAllocator.allocated: 262144 at 140689773997888
org.apache.spark.unsafe.memory.UnsafeMemoryAllocator@7bf8aaef Executor task 
launch worker-6 UnsafeMemoryAllocator.allocated: 262144 at 140689707058576



was (Author: robbinspg):
Tracking where the memory is being allocated it is interesting that for the 
Executor that crashes (Executor task launch worker-3) the Unsafe off-heap 
memory is allocated in a much different address range than the other executors. 
I think the generated code is incorrect anyway but may be accidentally passing 
as sometimes the memory in the freed page is still accessible?

{noformat}
org.apache.spark.unsafe.memory.UnsafeMemoryAllocator@7bf8aaef Executor task 
launch worker-3 UnsafeMemoryAllocator.allocated: 262144 at *28900976*
org.apache.spark.unsafe.memory.UnsafeMemoryAllocator@7bf8aaef Executor task 
launch worker-1 UnsafeMemoryAllocator.allocated: 262144 at 140689774260048
org.apache.spark.unsafe.memory.UnsafeMemoryAllocator@7bf8aaef Executor task 
launch worker-4 UnsafeMemoryAllocator.allocated: 262144 at 140689572734864
org.apache.spark.unsafe.memory.UnsafeMemoryAllocator@7bf8aaef Executor task 
launch worker-7 UnsafeMemoryAllocator.allocated: 262144 at 140689908250560
org.apache.spark.unsafe.memory.UnsafeMemoryAllocator@7bf8aaef Executor task 
launch worker-8 UnsafeMemoryAllocator.allocated: 262144 at 140690243746416
org.apache.spark.unsafe.memory.UnsafeMemoryAllocator@7bf8aaef Executor task 
launch worker-0 UnsafeMemoryAllocator.allocated: 262144 at 140690176924448
org.apache.spark.unsafe.memory.UnsafeMemoryAllocator@7bf8aaef Executor task 
launch worker-5 UnsafeMemoryAllocator.allocated: 262144 at 140689773997888
org.apache.spark.unsafe.memory.UnsafeMemoryAllocator@7bf8aaef Executor task 
launch worker-6 UnsafeMemoryAllocator.allocated: 262144 at 140689707058576
{noformat}

> segmentation violation in o.a.s.unsafe.types.UTF8String 
> 
>
> Key: SPARK-15822
> URL: https://issues.apache.org/jira/browse/SPARK-15822
> Project: Spark
>  Issue Type: Bug
>Affects Versions: 2.0.0
> Environment: linux amd64
> openjdk version "1.8.0_91"
> OpenJDK Runtime Environment (build 1.8.0_91-b14)
> OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode)
>Reporter: Pete Robbins
>Assignee: Herman van Hovell
>Priority: Blocker
>
> Executors fail with segmentation violation while running application with
> spark.memory.offHeap.enabled true
> spark.memory.offHeap.size 512m
> Also now reproduced with 
> spark.memory.offHeap.enabled false
> {noformat}
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  SIGSEGV (0xb) at pc=0x7f4559b4d4bd, pid=14182, tid=139935319750400
> #
> # JRE version: OpenJDK Runtime Environment (8.0_91-b14) (build 1.8.0_91-b14)
> # Java VM: OpenJDK 64-Bit Server VM (25.91-b14 mixed mode linux-amd64 
> compressed oops)
> # Problematic frame:
> # J 4816 C2 
> org.apache.spark.unsafe.types.UTF8String.compareTo(Lorg/apache/spark/unsafe/types/UTF8String;)I
>  (64 bytes) @ 0x7f4559b4d4bd [0x7f4559b4d460+0x5d]
> {noformat}
> We 

[jira] [Comment Edited] (SPARK-15822) segmentation violation in o.a.s.unsafe.types.UTF8String

[Pete Robbins (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-15822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15333586#comment-15333586
 ] 

Pete Robbins edited comment on SPARK-15822 at 6/16/16 11:18 AM:


OK so I think I know what is happening! 

In the following generate SMJ code on line 058 the call to leftIter.next() will 
return either a Row constructed by pointing into a page OR for the final Row in 
the iterator it returns a copy of the row and the underlying memory pages are 
freed in cleanupResources. This now means that any Rows previously returned 
from the iterator are invalid as they are addressing freed memory. In the case 
where I see the segv the Row assigned to smj_value6 is pointing into freed 
memory and this causes the  segmentation fault.


{code}
/* 001 */ public Object generate(Object[] references) {
/* 002 */   return new GeneratedIterator(references);
/* 003 */ }
/* 004 */
/* 005 */ final class GeneratedIterator extends 
org.apache.spark.sql.execution.BufferedRowIterator {
/* 006 */   private Object[] references;
/* 007 */   private scala.collection.Iterator smj_leftInput;
/* 008 */   private scala.collection.Iterator smj_rightInput;
/* 009 */   private InternalRow smj_leftRow;
/* 010 */   private InternalRow smj_rightRow;
/* 011 */   private UTF8String smj_value4;
/* 012 */   private UTF8String smj_value5;
/* 013 */   private java.util.ArrayList smj_matches;
/* 014 */   private UTF8String smj_value6;
/* 015 */   private UTF8String smj_value7;
/* 016 */   private UTF8String smj_value8;
/* 017 */   private boolean smj_isNull4;
/* 018 */   private UTF8String smj_value9;
/* 019 */   private boolean smj_isNull5;
/* 020 */   private long smj_value10;
/* 021 */   private org.apache.spark.sql.execution.metric.SQLMetric 
smj_numOutputRows;
/* 022 */   private UnsafeRow smj_result;
/* 023 */   private 
org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder smj_holder;
/* 024 */   private 
org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter smj_rowWriter;
/* 025 */   private UnsafeRow project_result;
/* 026 */   private 
org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder project_holder;
/* 027 */   private 
org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter 
project_rowWriter;
/* 028 */
/* 029 */   public GeneratedIterator(Object[] references) {
/* 030 */ this.references = references;
/* 031 */   }
/* 032 */
/* 033 */   public void init(int index, scala.collection.Iterator inputs[]) {
/* 034 */ partitionIndex = index;
/* 035 */ smj_leftInput = inputs[0];
/* 036 */ smj_rightInput = inputs[1];
/* 037 */
/* 038 */ smj_rightRow = null;
/* 039 */
/* 040 */ smj_matches = new java.util.ArrayList();
/* 041 */
/* 042 */ this.smj_numOutputRows = 
(org.apache.spark.sql.execution.metric.SQLMetric) references[0];
/* 043 */ smj_result = new UnsafeRow(6);
/* 044 */ this.smj_holder = new 
org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder(smj_result, 128);
/* 045 */ this.smj_rowWriter = new 
org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(smj_holder, 
6);
/* 046 */ project_result = new UnsafeRow(3);
/* 047 */ this.project_holder = new 
org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder(project_result, 
64);
/* 048 */ this.project_rowWriter = new 
org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(project_holder,
 3);
/* 049 */   }
/* 050 */
/* 051 */   private boolean findNextInnerJoinRows(
/* 052 */ scala.collection.Iterator leftIter,
/* 053 */ scala.collection.Iterator rightIter) {
/* 054 */ smj_leftRow = null;
/* 055 */ int comp = 0;
/* 056 */ while (smj_leftRow == null) {
/* 057 */   if (!leftIter.hasNext()) return false;
/* 058 */   smj_leftRow = (InternalRow) leftIter.next();
/* 059 */
/* 060 */   boolean smj_isNull = smj_leftRow.isNullAt(0);
/* 061 */   UTF8String smj_value = smj_isNull ? null : 
(smj_leftRow.getUTF8String(0));
/* 062 */
/* 063 */   boolean smj_isNull1 = smj_leftRow.isNullAt(1);
/* 064 */   UTF8String smj_value1 = smj_isNull1 ? null : 
(smj_leftRow.getUTF8String(1));
/* 065 */   if (smj_isNull || smj_isNull1) {
/* 066 */ smj_leftRow = null;
/* 067 */ continue;
/* 068 */   }
/* 069 */   if (!smj_matches.isEmpty()) {
/* 070 */ comp = 0;
/* 071 */ if (comp == 0) {
/* 072 */   comp = smj_value.compare(smj_value6);
/* 073 */ }
/* 074 */ if (comp == 0) {
/* 075 */   comp = smj_value1.compare(smj_value7);
/* 076 */ }
/* 077 */
/* 078 */ if (comp == 0) {
/* 079 */   return true;
/* 080 */ }
/* 081 */ smj_matches.clear();
/* 082 */   }
/* 083 */
/* 084 */   do {
/* 085 */ if (smj_rightRow == null) {
/* 086 */   if (!rightIter.hasNext()) {
/* 087 */ smj_value6

[jira] [Commented] (SPARK-15822) segmentation violation in o.a.s.unsafe.types.UTF8String

[Pete Robbins (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-15822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15333586#comment-15333586
 ] 

Pete Robbins commented on SPARK-15822:
--

OK so I think I know what is happening! 

In the following generate SMJ code on line 058 the call to leftIter.next() will 
return either a Row constructed by pointing into a page OR for the final Row in 
the iterator it returns a copy of the row and the memory pages are freed in 
cleanupResources. This now means that any Rows previously returned from the 
iterator are invalid as they are addressing freed memory. In the case where I 
see the segv the Row assigned to smj_value6 is pointing into freed memory and 
this causes the  segmentation fault.


{code}
/* 001 */ public Object generate(Object[] references) {
/* 002 */   return new GeneratedIterator(references);
/* 003 */ }
/* 004 */
/* 005 */ final class GeneratedIterator extends 
org.apache.spark.sql.execution.BufferedRowIterator {
/* 006 */   private Object[] references;
/* 007 */   private scala.collection.Iterator smj_leftInput;
/* 008 */   private scala.collection.Iterator smj_rightInput;
/* 009 */   private InternalRow smj_leftRow;
/* 010 */   private InternalRow smj_rightRow;
/* 011 */   private UTF8String smj_value4;
/* 012 */   private UTF8String smj_value5;
/* 013 */   private java.util.ArrayList smj_matches;
/* 014 */   private UTF8String smj_value6;
/* 015 */   private UTF8String smj_value7;
/* 016 */   private UTF8String smj_value8;
/* 017 */   private boolean smj_isNull4;
/* 018 */   private UTF8String smj_value9;
/* 019 */   private boolean smj_isNull5;
/* 020 */   private long smj_value10;
/* 021 */   private org.apache.spark.sql.execution.metric.SQLMetric 
smj_numOutputRows;
/* 022 */   private UnsafeRow smj_result;
/* 023 */   private 
org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder smj_holder;
/* 024 */   private 
org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter smj_rowWriter;
/* 025 */   private UnsafeRow project_result;
/* 026 */   private 
org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder project_holder;
/* 027 */   private 
org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter 
project_rowWriter;
/* 028 */
/* 029 */   public GeneratedIterator(Object[] references) {
/* 030 */ this.references = references;
/* 031 */   }
/* 032 */
/* 033 */   public void init(int index, scala.collection.Iterator inputs[]) {
/* 034 */ partitionIndex = index;
/* 035 */ smj_leftInput = inputs[0];
/* 036 */ smj_rightInput = inputs[1];
/* 037 */
/* 038 */ smj_rightRow = null;
/* 039 */
/* 040 */ smj_matches = new java.util.ArrayList();
/* 041 */
/* 042 */ this.smj_numOutputRows = 
(org.apache.spark.sql.execution.metric.SQLMetric) references[0];
/* 043 */ smj_result = new UnsafeRow(6);
/* 044 */ this.smj_holder = new 
org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder(smj_result, 128);
/* 045 */ this.smj_rowWriter = new 
org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(smj_holder, 
6);
/* 046 */ project_result = new UnsafeRow(3);
/* 047 */ this.project_holder = new 
org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder(project_result, 
64);
/* 048 */ this.project_rowWriter = new 
org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(project_holder,
 3);
/* 049 */   }
/* 050 */
/* 051 */   private boolean findNextInnerJoinRows(
/* 052 */ scala.collection.Iterator leftIter,
/* 053 */ scala.collection.Iterator rightIter) {
/* 054 */ smj_leftRow = null;
/* 055 */ int comp = 0;
/* 056 */ while (smj_leftRow == null) {
/* 057 */   if (!leftIter.hasNext()) return false;
/* 058 */   smj_leftRow = (InternalRow) leftIter.next();
/* 059 */
/* 060 */   boolean smj_isNull = smj_leftRow.isNullAt(0);
/* 061 */   UTF8String smj_value = smj_isNull ? null : 
(smj_leftRow.getUTF8String(0));
/* 062 */
/* 063 */   boolean smj_isNull1 = smj_leftRow.isNullAt(1);
/* 064 */   UTF8String smj_value1 = smj_isNull1 ? null : 
(smj_leftRow.getUTF8String(1));
/* 065 */   if (smj_isNull || smj_isNull1) {
/* 066 */ smj_leftRow = null;
/* 067 */ continue;
/* 068 */   }
/* 069 */   if (!smj_matches.isEmpty()) {
/* 070 */ comp = 0;
/* 071 */ if (comp == 0) {
/* 072 */   comp = smj_value.compare(smj_value6);
/* 073 */ }
/* 074 */ if (comp == 0) {
/* 075 */   comp = smj_value1.compare(smj_value7);
/* 076 */ }
/* 077 */
/* 078 */ if (comp == 0) {
/* 079 */   return true;
/* 080 */ }
/* 081 */ smj_matches.clear();
/* 082 */   }
/* 083 */
/* 084 */   do {
/* 085 */ if (smj_rightRow == null) {
/* 086 */   if (!rightIter.hasNext()) {
/* 087 */ smj_value6 = smj_value;
/* 088 */
/* 089 */ smj_value7

[jira] [Commented] (SPARK-15822) segmentation violation in o.a.s.unsafe.types.UTF8String

[Pete Robbins (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-15822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15333462#comment-15333462
 ] 

Pete Robbins commented on SPARK-15822:
--

Tracing through  off heap memory allocation this looks like the segv is caused 
by the UTf8String base+offset still addressing a page that has recently been 
freed.

> segmentation violation in o.a.s.unsafe.types.UTF8String 
> 
>
> Key: SPARK-15822
> URL: https://issues.apache.org/jira/browse/SPARK-15822
> Project: Spark
>  Issue Type: Bug
>Affects Versions: 2.0.0
> Environment: linux amd64
> openjdk version "1.8.0_91"
> OpenJDK Runtime Environment (build 1.8.0_91-b14)
> OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode)
>Reporter: Pete Robbins
>Assignee: Herman van Hovell
>Priority: Blocker
>
> Executors fail with segmentation violation while running application with
> spark.memory.offHeap.enabled true
> spark.memory.offHeap.size 512m
> Also now reproduced with 
> spark.memory.offHeap.enabled false
> {noformat}
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  SIGSEGV (0xb) at pc=0x7f4559b4d4bd, pid=14182, tid=139935319750400
> #
> # JRE version: OpenJDK Runtime Environment (8.0_91-b14) (build 1.8.0_91-b14)
> # Java VM: OpenJDK 64-Bit Server VM (25.91-b14 mixed mode linux-amd64 
> compressed oops)
> # Problematic frame:
> # J 4816 C2 
> org.apache.spark.unsafe.types.UTF8String.compareTo(Lorg/apache/spark/unsafe/types/UTF8String;)I
>  (64 bytes) @ 0x7f4559b4d4bd [0x7f4559b4d460+0x5d]
> {noformat}
> We initially saw this on IBM java on PowerPC box but is recreatable on linux 
> with OpenJDK. On linux with IBM Java 8 we see a null pointer exception at the 
> same code point:
> {noformat}
> 16/06/08 11:14:58 ERROR Executor: Exception in task 1.0 in stage 5.0 (TID 48)
> java.lang.NullPointerException
>   at 
> org.apache.spark.unsafe.types.UTF8String.compareTo(UTF8String.java:831)
>   at org.apache.spark.unsafe.types.UTF8String.compare(UTF8String.java:844)
>   at 
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.findNextInnerJoinRows$(Unknown
>  Source)
>   at 
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown
>  Source)
>   at 
> org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
>   at 
> org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$doExecute$2$$anon$2.hasNext(WholeStageCodegenExec.scala:377)
>   at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
>   at 
> scala.collection.convert.Wrappers$IteratorWrapper.hasNext(Wrappers.scala:30)
>   at org.spark_project.guava.collect.Ordering.leastOf(Ordering.java:664)
>   at org.apache.spark.util.collection.Utils$.takeOrdered(Utils.scala:37)
>   at 
> org.apache.spark.rdd.RDD$$anonfun$takeOrdered$1$$anonfun$30.apply(RDD.scala:1365)
>   at 
> org.apache.spark.rdd.RDD$$anonfun$takeOrdered$1$$anonfun$30.apply(RDD.scala:1362)
>   at 
> org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:757)
>   at 
> org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:757)
>   at 
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
>   at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:318)
>   at org.apache.spark.rdd.RDD.iterator(RDD.scala:282)
>   at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)
>   at org.apache.spark.scheduler.Task.run(Task.scala:85)
>   at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1153)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>   at java.lang.Thread.run(Thread.java:785)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-15822) segmentation violation in o.a.s.unsafe.types.UTF8String

[Pete Robbins (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-15822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15331580#comment-15331580
 ] 

Pete Robbins commented on SPARK-15822:
--

I can also recreate this issue on Oracle JDK 1.8:

{noformat}
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x7f0c65d06aec, pid=7521, tid=0x7f0b69ffd700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_92-b14) (build 1.8.0_92-b14)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.92-b14 mixed mode linux-amd64 
compressed oops)
# Problematic frame:
# J 7453 C1 org.apache.spark.unsafe.Platform.getByte(Ljava/lang/Object;J)B (9 
bytes) @ 0x7f0c65d06aec [0x7f0c65d06ae0+0xc]
#
# Failed to write core dump. Core dumps have been disabled. To enable core 
dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#

---  T H R E A D  ---

Current thread (0x7f0bf4008800):  JavaThread "Executor task launch 
worker-3" daemon [_thread_in_Java, id=7662, 
stack(0x7f0b69efd000,0x7f0b69ffe000)]

siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 
0x02868e54

Registers:
RAX=0x7f0c461abb38, RBX=0x7f0c461abb38, RCX=0x7f0c213547c8, 
RDX=0x02868e54
RSP=0x7f0b69ffba40, RBP=0x7f0b69ffbae0, RSI=0x, 
RDI=0x0001008254d8
R8 =0x200bd0a6, R9 =0xd9fa2650, R10=0x7f0c79d39020, 
R11=0x7f0c65d06ae0
R12=0x, R13=0x7f0b69ffba88, R14=0x7f0b69ffbaf8, 
R15=0x7f0bf4008800
RIP=0x7f0c65d06aec, EFLAGS=0x00010202, CSGSFS=0x0033, 
ERR=0x0004
  TRAPNO=0x000e

Top of Stack: (sp=0x7f0b69ffba40)
0x7f0b69ffba40:   7f0b684b4a70 
0x7f0b69ffba50:   7f0b69ffbb10 7f0c65e96d4c
0x7f0b69ffba60:   7f0c65008040 d9fa2628
0x7f0b69ffba70:   7f0b69ffbae0 7f0c650079c0
0x7f0b69ffba80:   7f0c650079c0 02868e54
0x7f0b69ffba90:   0030 
0x7f0b69ffbaa0:   7f0b69ffbaa0 7f0c21351403
0x7f0b69ffbab0:   7f0b69ffbaf8 7f0c213547c8
0x7f0b69ffbac0:    7f0c21351428
0x7f0b69ffbad0:   7f0b69ffba88 7f0b69ffbaf0
0x7f0b69ffbae0:   7f0b69ffbb48 7f0c650079c0
0x7f0b69ffbaf0:    d9f57cf0
0x7f0b69ffbb00:   004c 7f0b69ffbb08
0x7f0b69ffbb10:   7f0c21353726 7f0b69ffbb78
0x7f0b69ffbb20:   7f0c213547c8 
0x7f0b69ffbb30:   7f0c213537a0 7f0b69ffbaf0
0x7f0b69ffbb40:   7f0b69ffbb70 7f0b69ffbbc0
0x7f0b69ffbb50:   7f0c65007d00 
0x7f0b69ffbb60:    0003
0x7f0b69ffbb70:   d9f57cf0 d9fa33b0
0x7f0b69ffbb80:   7f0b69ffbb80 7f0c2135385a
0x7f0b69ffbb90:   7f0b69ffbbd8 7f0c213547c8
0x7f0b69ffbba0:    7f0c21353880
0x7f0b69ffbbb0:   7f0b69ffbb70 7f0b69ffbbd0
0x7f0b69ffbbc0:   7f0b69ffbc20 7f0c65007d00
0x7f0b69ffbbd0:   d9f57cf0 d9fa33b0
0x7f0b69ffbbe0:   7f0b69ffbbe0 7f0b684a24e5
0x7f0b69ffbbf0:   7f0b69ffbc88 7f0b684a2950
0x7f0b69ffbc00:    7f0b684a2618
0x7f0b69ffbc10:   7f0b69ffbbd0 7f0b69ffbc78
0x7f0b69ffbc20:   7f0b69ffbcd0 7f0c65007a90
0x7f0b69ffbc30:     

Instructions: (pc=0x7f0c65d06aec)
0x7f0c65d06acc:   0a 80 11 64 01 f8 12 fe 06 90 0c 64 01 f8 12 fe
0x7f0c65d06adc:   06 90 0c 64 89 84 24 00 c0 fe ff 55 48 83 ec 30
0x7f0c65d06aec:   0f be 04 16 c1 e0 18 c1 f8 18 48 83 c4 30 5d 85
0x7f0c65d06afc:   05 ff f5 28 14 c3 90 90 49 8b 87 a8 02 00 00 49 

Register to memory mapping:

RAX={method} {0x7f0c461abb38} 'getByte' '(Ljava/lang/Object;J)B' in 
'org/apache/spark/unsafe/Platform'
RBX={method} {0x7f0c461abb38} 'getByte' '(Ljava/lang/Object;J)B' in 
'org/apache/spark/unsafe/Platform'
RCX=0x7f0c213547c8 is pointing into metadata
RDX=0x02868e54 is an unknown value
RSP=0x7f0b69ffba40 is pointing into the stack for thread: 0x7f0bf4008800
RBP=0x7f0b69ffbae0 is pointing into the stack for thread: 0x7f0bf4008800
RSI=0x is an unknown value
RDI=0x0001008254d8 is pointing into metadata
R8 =0x200bd0a6 is an unknown value
R9 =0xd9fa2650 is an oop
[B 
 - klass: {type array byte}
 - length: 48
R10=0x7f0c79d39020:  in 
/home/robbins/sdks/jdk1.8.0_92/jre/lib/amd64/server/libjvm.so at 
0x7f0c78d7d000
R11=0x7f0c65d06ae0 is at entry_point+0 in (nmethod*)0x7f0c65d06990
R12=0x is an unknown value
R13=0x7f0b

[jira] [Commented] (SPARK-15822) segmentation violation in o.a.s.unsafe.types.UTF8String

[Pete Robbins (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-15822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15331375#comment-15331375
 ] 

Pete Robbins commented on SPARK-15822:
--

and the plan:

{noformat}
== Parsed Logical Plan ==
'Project [unresolvedalias('Origin, None), unresolvedalias('UniqueCarrier, 
None), 'round((('count * 100) / 'total), 2) AS rank#173]
+- Project [Origin#16, UniqueCarrier#8, count#134L, total#97L]
   +- Join Inner, ((Origin#16 = Origin#155) && (UniqueCarrier#8 = 
UniqueCarrier#147))
  :- Aggregate [Origin#16, UniqueCarrier#8], [Origin#16, UniqueCarrier#8, 
count(1) AS count#134L]
  :  +- Filter (NOT (Cancelled#21 = 0) && (CancellationCode#22 = A))
  : +- Filter (Dest#17 = ORD)
  :+- 
Relation[Year#0,Month#1,DayofMonth#2,DayOfWeek#3,DepTime#4,CRSDepTime#5,ArrTime#6,CRSArrTime#7,UniqueCarrier#8,FlightNum#9,TailNum#10,ActualElapsedTime#11,CRSElapsedTime#12,AirTime#13,ArrDelay#14,DepDelay#15,Origin#16,Dest#17,Distance#18,TaxiIn#19,TaxiOut#20,Cancelled#21,CancellationCode#22,Diverted#23,...
 5 more fields] csv
  +- Project [Origin#155, UniqueCarrier#147, count#92L AS total#97L]
 +- Aggregate [Origin#155, UniqueCarrier#147], [Origin#155, 
UniqueCarrier#147, count(1) AS count#92L]
+- Filter (Dest#156 = ORD)
   +- 
Relation[Year#139,Month#140,DayofMonth#141,DayOfWeek#142,DepTime#143,CRSDepTime#144,ArrTime#145,CRSArrTime#146,UniqueCarrier#147,FlightNum#148,TailNum#149,ActualElapsedTime#150,CRSElapsedTime#151,AirTime#152,ArrDelay#153,DepDelay#154,Origin#155,Dest#156,Distance#157,TaxiIn#158,TaxiOut#159,Cancelled#160,CancellationCode#161,Diverted#162,...
 5 more fields] csv

== Analyzed Logical Plan ==
Origin: string, UniqueCarrier: string, rank: double
Project [Origin#16, UniqueCarrier#8, round((cast((count#134L * cast(100 as 
bigint)) as double) / cast(total#97L as double)), 2) AS rank#173]
+- Project [Origin#16, UniqueCarrier#8, count#134L, total#97L]
   +- Join Inner, ((Origin#16 = Origin#155) && (UniqueCarrier#8 = 
UniqueCarrier#147))
  :- Aggregate [Origin#16, UniqueCarrier#8], [Origin#16, UniqueCarrier#8, 
count(1) AS count#134L]
  :  +- Filter (NOT (Cancelled#21 = 0) && (CancellationCode#22 = A))
  : +- Filter (Dest#17 = ORD)
  :+- 
Relation[Year#0,Month#1,DayofMonth#2,DayOfWeek#3,DepTime#4,CRSDepTime#5,ArrTime#6,CRSArrTime#7,UniqueCarrier#8,FlightNum#9,TailNum#10,ActualElapsedTime#11,CRSElapsedTime#12,AirTime#13,ArrDelay#14,DepDelay#15,Origin#16,Dest#17,Distance#18,TaxiIn#19,TaxiOut#20,Cancelled#21,CancellationCode#22,Diverted#23,...
 5 more fields] csv
  +- Project [Origin#155, UniqueCarrier#147, count#92L AS total#97L]
 +- Aggregate [Origin#155, UniqueCarrier#147], [Origin#155, 
UniqueCarrier#147, count(1) AS count#92L]
+- Filter (Dest#156 = ORD)
   +- 
Relation[Year#139,Month#140,DayofMonth#141,DayOfWeek#142,DepTime#143,CRSDepTime#144,ArrTime#145,CRSArrTime#146,UniqueCarrier#147,FlightNum#148,TailNum#149,ActualElapsedTime#150,CRSElapsedTime#151,AirTime#152,ArrDelay#153,DepDelay#154,Origin#155,Dest#156,Distance#157,TaxiIn#158,TaxiOut#159,Cancelled#160,CancellationCode#161,Diverted#162,...
 5 more fields] csv

== Optimized Logical Plan ==
Project [Origin#16, UniqueCarrier#8, round((cast((count#134L * 100) as double) 
/ cast(total#97L as double)), 2) AS rank#173]
+- Join Inner, ((Origin#16 = Origin#155) && (UniqueCarrier#8 = 
UniqueCarrier#147))
   :- Aggregate [Origin#16, UniqueCarrier#8], [Origin#16, UniqueCarrier#8, 
count(1) AS count#134L]
   :  +- Project [UniqueCarrier#8, Origin#16]
   : +- Filter (((isnotnull(Origin#16) && isnotnull(UniqueCarrier#8)) 
&& isnotnull(Cancelled#21)) && isnotnull(CancellationCode#22)) && NOT 
(Cancelled#21 = 0)) && (CancellationCode#22 = A)) && isnotnull(Dest#17)) && 
(Dest#17 = ORD))
   :+- 
Relation[Year#0,Month#1,DayofMonth#2,DayOfWeek#3,DepTime#4,CRSDepTime#5,ArrTime#6,CRSArrTime#7,UniqueCarrier#8,FlightNum#9,TailNum#10,ActualElapsedTime#11,CRSElapsedTime#12,AirTime#13,ArrDelay#14,DepDelay#15,Origin#16,Dest#17,Distance#18,TaxiIn#19,TaxiOut#20,Cancelled#21,CancellationCode#22,Diverted#23,...
 5 more fields] csv
   +- Aggregate [Origin#155, UniqueCarrier#147], [Origin#155, 
UniqueCarrier#147, count(1) AS total#97L]
  +- Project [UniqueCarrier#147, Origin#155]
 +- Filter (((isnotnull(UniqueCarrier#147) && isnotnull(Origin#155)) && 
isnotnull(Dest#156)) && (Dest#156 = ORD))
+- 
Relation[Year#139,Month#140,DayofMonth#141,DayOfWeek#142,DepTime#143,CRSDepTime#144,ArrTime#145,CRSArrTime#146,UniqueCarrier#147,FlightNum#148,TailNum#149,ActualElapsedTime#150,CRSElapsedTime#151,AirTime#152,ArrDelay#153,DepDelay#154,Origin#155,Dest#156,Distance#157,TaxiIn#158,TaxiOut#159,Cancelled#160,CancellationCode#161,Diver

[jira] [Commented] (SPARK-15822) segmentation violation in o.a.s.unsafe.types.UTF8String

[Pete Robbins (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-15822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15331370#comment-15331370
 ] 

Pete Robbins commented on SPARK-15822:
--

Chatting with [~hvanhovell] here is the current state. I can reproduce a segv 
using local[8] on an 8 core machine. It is intermittent but  many many runs 
with eg local[2] produce no issues. The segv info is:

{noformat}
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x7fe8c118ca58, pid=3558, tid=140633451779840
#
# JRE version: OpenJDK Runtime Environment (8.0_91-b14) (build 1.8.0_91-b14)
# Java VM: OpenJDK 64-Bit Server VM (25.91-b14 mixed mode linux-amd64 
compressed oops)
# Problematic frame:
# J 7467 C1 org.apache.spark.unsafe.Platform.getByte(Ljava/lang/Object;J)B (9 
bytes) @ 0x7fe8c118ca58 [0x7fe8c118ca20+0x38]
#
# Failed to write core dump. Core dumps have been disabled. To enable core 
dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#

---  T H R E A D  ---

Current thread (0x7fe858018800):  JavaThread "Executor task launch 
worker-3" daemon [_thread_in_Java, id=3698, 
stack(0x7fe7c6dfd000,0x7fe7c6efe000)]

siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 
0x00a09cf4

Registers:
RAX=0x7fe884ce5828, RBX=0x7fe884ce5828, RCX=0x7fe81e0a5360, 
RDX=0x00a09cf4
RSP=0x7fe7c6efb9e0, RBP=0x7fe7c6efba80, RSI=0x, 
RDI=0x3848
R8 =0x200b94c8, R9 =0xeef66bf0, R10=0x7fe8d87a2f00, 
R11=0x7fe8c118ca20
R12=0x, R13=0x7fe7c6efba28, R14=0x7fe7c6efba98, 
R15=0x7fe858018800
RIP=0x7fe8c118ca58, EFLAGS=0x00010206, CSGSFS=0x0033, 
ERR=0x0004
  TRAPNO=0x000e

Top of Stack: (sp=0x7fe7c6efb9e0)
0x7fe7c6efb9e0:   7fe7c56941e8 
0x7fe7c6efb9f0:   7fe7c6efbab0 7fe8c140c38c
0x7fe7c6efba00:   7fe8c1007d80 eef66bc8
0x7fe7c6efba10:   7fe7c6efba80 7fe8c1007700
0x7fe7c6efba20:   7fe8c1007700 00a09cf4
0x7fe7c6efba30:   0030 
0x7fe7c6efba40:   7fe7c6efba40 7fe81e0a1f9b
0x7fe7c6efba50:   7fe7c6efba98 7fe81e0a5360
0x7fe7c6efba60:    7fe81e0a1fc0
0x7fe7c6efba70:   7fe7c6efba28 7fe7c6efba90
0x7fe7c6efba80:   7fe7c6efbae8 7fe8c1007700
0x7fe7c6efba90:    ee4f4898
0x7fe7c6efbaa0:   004d 7fe7c6efbaa8
0x7fe7c6efbab0:   7fe81e0a42be 7fe7c6efbb18
0x7fe7c6efbac0:   7fe81e0a5360 
0x7fe7c6efbad0:   7fe81e0a4338 7fe7c6efba90
0x7fe7c6efbae0:   7fe7c6efbb10 7fe7c6efbb60
0x7fe7c6efbaf0:   7fe8c1007a40 
0x7fe7c6efbb00:    0003
0x7fe7c6efbb10:   ee4f4898 eef67950
0x7fe7c6efbb20:   7fe7c6efbb20 7fe81e0a43f2
0x7fe7c6efbb30:   7fe7c6efbb78 7fe81e0a5360
0x7fe7c6efbb40:    7fe81e0a4418
0x7fe7c6efbb50:   7fe7c6efbb10 7fe7c6efbb70
0x7fe7c6efbb60:   7fe7c6efbbc0 7fe8c1007a40
0x7fe7c6efbb70:   ee4f4898 eef67950
0x7fe7c6efbb80:   7fe7c6efbb80 7fe7c56844e5
0x7fe7c6efbb90:   7fe7c6efbc28 7fe7c5684950
0x7fe7c6efbba0:    7fe7c5684618
0x7fe7c6efbbb0:   7fe7c6efbb70 7fe7c6efbc18
0x7fe7c6efbbc0:   7fe7c6efbc70 7fe8c10077d0
0x7fe7c6efbbd0:     

Instructions: (pc=0x7fe8c118ca58)
0x7fe8c118ca38:   08 83 c7 08 89 78 08 48 b8 28 58 ce 84 e8 7f 00
0x7fe8c118ca48:   00 81 e7 f8 3f 00 00 83 ff 00 0f 84 16 00 00 00
0x7fe8c118ca58:   0f be 04 16 c1 e0 18 c1 f8 18 48 83 c4 30 5d 85
0x7fe8c118ca68:   05 93 c6 85 17 c3 48 89 44 24 08 48 c7 04 24 ff 

Register to memory mapping:

RAX={method} {0x7fe884ce5828} 'getByte' '(Ljava/lang/Object;J)B' in 
'org/apache/spark/unsafe/Platform'
RBX={method} {0x7fe884ce5828} 'getByte' '(Ljava/lang/Object;J)B' in 
'org/apache/spark/unsafe/Platform'
RCX=0x7fe81e0a5360 is pointing into metadata
RDX=0x00a09cf4 is an unknown value
RSP=0x7fe7c6efb9e0 is pointing into the stack for thread: 0x7fe858018800
RBP=0x7fe7c6efba80 is pointing into the stack for thread: 0x7fe858018800
RSI=0x is an unknown value
RDI=0x3848 is an unknown value
R8 =0x200b94c8 is an unknown value
R9 =0xeef66bf0 is an oop
[B 
 - klass: {type array byte}
 - length: 48
R10=0x7fe8d87a2f00:  in 
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.91-0.b14.el6_7.x86_64/jre/lib/am

[jira] [Commented] (SPARK-15822) segmentation violation in o.a.s.unsafe.types.UTF8String

[Pete Robbins (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-15822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15331371#comment-15331371
 ] 

Pete Robbins commented on SPARK-15822:
--

The generated code is:

{code}
Top Arrival Carrier Cancellations:
Found 5 WholeStageCodegen subtrees.
== Subtree 1 / 5 ==
*HashAggregate(key=[Origin#16,UniqueCarrier#8], functions=[partial_count(1)], 
output=[Origin#16,UniqueCarrier#8,count#296L])
+- *Project [UniqueCarrier#8, Origin#16]
   +- *Filter (((isnotnull(Origin#16) && isnotnull(UniqueCarrier#8)) && 
isnotnull(Cancelled#21)) && isnotnull(CancellationCode#22)) && NOT 
(Cancelled#21 = 0)) && (CancellationCode#22 = A)) && isnotnull(Dest#17)) && 
(Dest#17 = ORD))
  +- *Scan csv 
[UniqueCarrier#8,Origin#16,Dest#17,Cancelled#21,CancellationCode#22] Format: 
CSV, InputPaths: file:/home/robbins/brandberry/2008.csv, PushedFilters: 
[IsNotNull(Origin), IsNotNull(UniqueCarrier), IsNotNull(Cancelled), 
IsNotNull(CancellationCode), ..., ReadSchema: 
struct<UniqueCarrier:string,Origin:string,Dest:string,Cancelled:int,CancellationCode:string>

Generated code:
/* 001 */ public Object generate(Object[] references) {
/* 002 */   return new GeneratedIterator(references);
/* 003 */ }
/* 004 */
/* 005 */ final class GeneratedIterator extends 
org.apache.spark.sql.execution.BufferedRowIterator {
/* 006 */   private Object[] references;
/* 007 */   private boolean agg_initAgg;
/* 008 */   private boolean agg_bufIsNull;
/* 009 */   private long agg_bufValue;
/* 010 */   private agg_VectorizedHashMap agg_vectorizedHashMap;
/* 011 */   private 
java.util.Iterator 
agg_vectorizedHashMapIter;
/* 012 */   private org.apache.spark.sql.execution.aggregate.HashAggregateExec 
agg_plan;
/* 013 */   private 
org.apache.spark.sql.execution.UnsafeFixedWidthAggregationMap agg_hashMap;
/* 014 */   private org.apache.spark.sql.execution.UnsafeKVExternalSorter 
agg_sorter;
/* 015 */   private org.apache.spark.unsafe.KVIterator agg_mapIter;
/* 016 */   private org.apache.spark.sql.execution.metric.SQLMetric 
agg_peakMemory;
/* 017 */   private org.apache.spark.sql.execution.metric.SQLMetric 
agg_spillSize;
/* 018 */   private org.apache.spark.sql.execution.metric.SQLMetric 
scan_numOutputRows;
/* 019 */   private scala.collection.Iterator scan_input;
/* 020 */   private org.apache.spark.sql.execution.metric.SQLMetric 
filter_numOutputRows;
/* 021 */   private UnsafeRow filter_result;
/* 022 */   private 
org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder filter_holder;
/* 023 */   private 
org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter 
filter_rowWriter;
/* 024 */   private UnsafeRow project_result;
/* 025 */   private 
org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder project_holder;
/* 026 */   private 
org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter 
project_rowWriter;
/* 027 */   private UnsafeRow agg_result2;
/* 028 */   private 
org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder agg_holder;
/* 029 */   private 
org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter agg_rowWriter;
/* 030 */   private 
org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowJoiner 
agg_unsafeRowJoiner;
/* 031 */   private org.apache.spark.sql.execution.metric.SQLMetric 
wholestagecodegen_numOutputRows;
/* 032 */   private org.apache.spark.sql.execution.metric.SQLMetric 
wholestagecodegen_aggTime;
/* 033 */   private UnsafeRow wholestagecodegen_result;
/* 034 */   private 
org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder 
wholestagecodegen_holder;
/* 035 */   private 
org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter 
wholestagecodegen_rowWriter;
/* 036 */
/* 037 */   public GeneratedIterator(Object[] references) {
/* 038 */ this.references = references;
/* 039 */   }
/* 040 */
/* 041 */   public void init(int index, scala.collection.Iterator inputs[]) {
/* 042 */ partitionIndex = index;
/* 043 */ agg_initAgg = false;
/* 044 */
/* 045 */ agg_vectorizedHashMap = new agg_VectorizedHashMap();
/* 046 */
/* 047 */ this.agg_plan = 
(org.apache.spark.sql.execution.aggregate.HashAggregateExec) references[0];
/* 048 */
/* 049 */ this.agg_peakMemory = 
(org.apache.spark.sql.execution.metric.SQLMetric) references[1];
/* 050 */ this.agg_spillSize = 
(org.apache.spark.sql.execution.metric.SQLMetric) references[2];
/* 051 */ this.scan_numOutputRows = 
(org.apache.spark.sql.execution.metric.SQLMetric) references[3];
/* 052 */ scan_input = inputs[0];
/* 053 */ this.filter_numOutputRows = 
(org.apache.spark.sql.execution.metric.SQLMetric) references[4];
/* 054 */ filter_result = new UnsafeRow(5);
/* 055 */ this.filter_holder = new 
org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder(f

[jira] [Commented] (SPARK-15822) segmentation violation in o.a.s.unsafe.types.UTF8String

[Pete Robbins (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-15822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15329474#comment-15329474
 ] 

Pete Robbins commented on SPARK-15822:
--

modified app to remove .cache()'s and still get a segv on open jdk 8. 

I may have been mistaken about it failing with 'spark.sql.codegen.wholeStage 
false' as I can not reproduce it with that set.

> segmentation violation in o.a.s.unsafe.types.UTF8String 
> 
>
> Key: SPARK-15822
> URL: https://issues.apache.org/jira/browse/SPARK-15822
> Project: Spark
>  Issue Type: Bug
>Affects Versions: 2.0.0
> Environment: linux amd64
> openjdk version "1.8.0_91"
> OpenJDK Runtime Environment (build 1.8.0_91-b14)
> OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode)
>Reporter: Pete Robbins
>Assignee: Herman van Hovell
>Priority: Blocker
>
> Executors fail with segmentation violation while running application with
> spark.memory.offHeap.enabled true
> spark.memory.offHeap.size 512m
> Also now reproduced with 
> spark.memory.offHeap.enabled false
> {noformat}
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  SIGSEGV (0xb) at pc=0x7f4559b4d4bd, pid=14182, tid=139935319750400
> #
> # JRE version: OpenJDK Runtime Environment (8.0_91-b14) (build 1.8.0_91-b14)
> # Java VM: OpenJDK 64-Bit Server VM (25.91-b14 mixed mode linux-amd64 
> compressed oops)
> # Problematic frame:
> # J 4816 C2 
> org.apache.spark.unsafe.types.UTF8String.compareTo(Lorg/apache/spark/unsafe/types/UTF8String;)I
>  (64 bytes) @ 0x7f4559b4d4bd [0x7f4559b4d460+0x5d]
> {noformat}
> We initially saw this on IBM java on PowerPC box but is recreatable on linux 
> with OpenJDK. On linux with IBM Java 8 we see a null pointer exception at the 
> same code point:
> {noformat}
> 16/06/08 11:14:58 ERROR Executor: Exception in task 1.0 in stage 5.0 (TID 48)
> java.lang.NullPointerException
>   at 
> org.apache.spark.unsafe.types.UTF8String.compareTo(UTF8String.java:831)
>   at org.apache.spark.unsafe.types.UTF8String.compare(UTF8String.java:844)
>   at 
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.findNextInnerJoinRows$(Unknown
>  Source)
>   at 
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown
>  Source)
>   at 
> org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
>   at 
> org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$doExecute$2$$anon$2.hasNext(WholeStageCodegenExec.scala:377)
>   at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
>   at 
> scala.collection.convert.Wrappers$IteratorWrapper.hasNext(Wrappers.scala:30)
>   at org.spark_project.guava.collect.Ordering.leastOf(Ordering.java:664)
>   at org.apache.spark.util.collection.Utils$.takeOrdered(Utils.scala:37)
>   at 
> org.apache.spark.rdd.RDD$$anonfun$takeOrdered$1$$anonfun$30.apply(RDD.scala:1365)
>   at 
> org.apache.spark.rdd.RDD$$anonfun$takeOrdered$1$$anonfun$30.apply(RDD.scala:1362)
>   at 
> org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:757)
>   at 
> org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:757)
>   at 
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
>   at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:318)
>   at org.apache.spark.rdd.RDD.iterator(RDD.scala:282)
>   at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)
>   at org.apache.spark.scheduler.Task.run(Task.scala:85)
>   at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1153)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>   at java.lang.Thread.run(Thread.java:785)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-15822) segmentation violation in o.a.s.unsafe.types.UTF8String

[Pete Robbins (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-15822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15327992#comment-15327992
 ] 

Pete Robbins commented on SPARK-15822:
--

So this does seem to cause the NPE or SEGV intermittently, ie I get some clean 
runs. However, I added some tracing to detect when the UnsafeRow looks corrupt 
(baseobject = null, offset=massive) and I see these in every run so I suspect 
there is always corruption but that doesn't always lead to a visible failure. 
The app usually gives the appearance of success as Spark re-submits the lost 
tasks and restarts failing executors. Here is what I think is the plan 
associated with one of the failing jobs:

== Parsed Logical Plan ==
'Project [unresolvedalias('Origin, None), unresolvedalias('UniqueCarrier, 
None), 'round((('count * 100) / 'total), 2) AS rank#927]
+- Project [Origin#16, UniqueCarrier#8, count#888L, total#851L]
   +- Join Inner, ((Origin#16 = Origin#909) && (UniqueCarrier#8 = 
UniqueCarrier#901))
  :- Aggregate [Origin#16, UniqueCarrier#8], [Origin#16, UniqueCarrier#8, 
count(1) AS count#888L]
  :  +- Filter (NOT (Cancelled#21 = 0) && (CancellationCode#22 = A))
  : +- Filter (Dest#17 = ORD)
  :+- 
Relation[Year#0,Month#1,DayofMonth#2,DayOfWeek#3,DepTime#4,CRSDepTime#5,ArrTime#6,CRSArrTime#7,UniqueCarrier#8,FlightNum#9,TailNum#10,ActualElapsedTime#11,CRSElapsedTime#12,AirTime#13,ArrDelay#14,DepDelay#15,Origin#16,Dest#17,Distance#18,TaxiIn#19,TaxiOut#20,Cancelled#21,CancellationCode#22,Diverted#23,CarrierDelay#24,WeatherDelay#25,NASDelay#26,SecurityDelay#27,LateAircraftDelay#28]
 csv
  +- Project [Origin#909, UniqueCarrier#901, count#846L AS total#851L]
 +- Aggregate [Origin#909, UniqueCarrier#901], [Origin#909, 
UniqueCarrier#901, count(1) AS count#846L]
+- Filter (Dest#910 = ORD)
   +- 
Relation[Year#893,Month#894,DayofMonth#895,DayOfWeek#896,DepTime#897,CRSDepTime#898,ArrTime#899,CRSArrTime#900,UniqueCarrier#901,FlightNum#902,TailNum#903,ActualElapsedTime#904,CRSElapsedTime#905,AirTime#906,ArrDelay#907,DepDelay#908,Origin#909,Dest#910,Distance#911,TaxiIn#912,TaxiOut#913,Cancelled#914,CancellationCode#915,Diverted#916,CarrierDelay#917,WeatherDelay#918,NASDelay#919,SecurityDelay#920,LateAircraftDelay#921]
 csv

== Analyzed Logical Plan ==
Origin: string, UniqueCarrier: string, rank: double
Project [Origin#16, UniqueCarrier#8, round((cast((count#888L * cast(100 as 
bigint)) as double) / cast(total#851L as double)), 2) AS rank#927]
+- Project [Origin#16, UniqueCarrier#8, count#888L, total#851L]
   +- Join Inner, ((Origin#16 = Origin#909) && (UniqueCarrier#8 = 
UniqueCarrier#901))
  :- Aggregate [Origin#16, UniqueCarrier#8], [Origin#16, UniqueCarrier#8, 
count(1) AS count#888L]
  :  +- Filter (NOT (Cancelled#21 = 0) && (CancellationCode#22 = A))
  : +- Filter (Dest#17 = ORD)
  :+- 
Relation[Year#0,Month#1,DayofMonth#2,DayOfWeek#3,DepTime#4,CRSDepTime#5,ArrTime#6,CRSArrTime#7,UniqueCarrier#8,FlightNum#9,TailNum#10,ActualElapsedTime#11,CRSElapsedTime#12,AirTime#13,ArrDelay#14,DepDelay#15,Origin#16,Dest#17,Distance#18,TaxiIn#19,TaxiOut#20,Cancelled#21,CancellationCode#22,Diverted#23,CarrierDelay#24,WeatherDelay#25,NASDelay#26,SecurityDelay#27,LateAircraftDelay#28]
 csv
  +- Project [Origin#909, UniqueCarrier#901, count#846L AS total#851L]
 +- Aggregate [Origin#909, UniqueCarrier#901], [Origin#909, 
UniqueCarrier#901, count(1) AS count#846L]
+- Filter (Dest#910 = ORD)
   +- 
Relation[Year#893,Month#894,DayofMonth#895,DayOfWeek#896,DepTime#897,CRSDepTime#898,ArrTime#899,CRSArrTime#900,UniqueCarrier#901,FlightNum#902,TailNum#903,ActualElapsedTime#904,CRSElapsedTime#905,AirTime#906,ArrDelay#907,DepDelay#908,Origin#909,Dest#910,Distance#911,TaxiIn#912,TaxiOut#913,Cancelled#914,CancellationCode#915,Diverted#916,CarrierDelay#917,WeatherDelay#918,NASDelay#919,SecurityDelay#920,LateAircraftDelay#921]
 csv

== Optimized Logical Plan ==
Project [Origin#16, UniqueCarrier#8, round((cast((count#888L * 100) as double) 
/ cast(total#851L as double)), 2) AS rank#927]
+- Join Inner, ((Origin#16 = Origin#909) && (UniqueCarrier#8 = 
UniqueCarrier#901))
   :- Aggregate [Origin#16, UniqueCarrier#8], [Origin#16, UniqueCarrier#8, 
count(1) AS count#888L]
   :  +- Project [UniqueCarrier#8, Origin#16]
   : +- Filter (isnotnull(UniqueCarrier#8) && isnotnull(Origin#16)) && 
isnotnull(Cancelled#21)) && isnotnull(CancellationCode#22)) && NOT 
(Cancelled#21 = 0)) && (CancellationCode#22 = A))
   :+- InMemoryRelation [Year#0, Month#1, DayofMonth#2, DayOfWeek#3, 
DepTime#4, CRSDepTime#5, ArrTime#6, CRSArrTime#7, UniqueCarrier#8, FlightNum#9, 
TailNum#10, ActualElapsedTime#11, CRSElapsedTime#12, AirTime#13, ArrDelay#14, 
DepDelay#15, Origin#16, Dest#17, Distance#18, TaxiIn#

[jira] [Commented] (SPARK-15822) segmentation violation in o.a.s.unsafe.types.UTF8String

[Pete Robbins (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-15822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15325761#comment-15325761
 ] 

Pete Robbins commented on SPARK-15822:
--

has failed on latest Branch 2.0 and master. Currently using Branch-2.0

commit a790ac5793e1988895341fa878f947b09b275926
Author: yinxusen <yinxu...@gmail.com>
Date:   Wed Jun 8 09:18:04 2016 +0100



> segmentation violation in o.a.s.unsafe.types.UTF8String 
> 
>
> Key: SPARK-15822
> URL: https://issues.apache.org/jira/browse/SPARK-15822
> Project: Spark
>  Issue Type: Bug
>Affects Versions: 2.0.0
> Environment: linux amd64
> openjdk version "1.8.0_91"
> OpenJDK Runtime Environment (build 1.8.0_91-b14)
> OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode)
>Reporter: Pete Robbins
>Assignee: Herman van Hovell
>Priority: Blocker
>
> Executors fail with segmentation violation while running application with
> spark.memory.offHeap.enabled true
> spark.memory.offHeap.size 512m
> Also now reproduced with 
> spark.memory.offHeap.enabled false
> {noformat}
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  SIGSEGV (0xb) at pc=0x7f4559b4d4bd, pid=14182, tid=139935319750400
> #
> # JRE version: OpenJDK Runtime Environment (8.0_91-b14) (build 1.8.0_91-b14)
> # Java VM: OpenJDK 64-Bit Server VM (25.91-b14 mixed mode linux-amd64 
> compressed oops)
> # Problematic frame:
> # J 4816 C2 
> org.apache.spark.unsafe.types.UTF8String.compareTo(Lorg/apache/spark/unsafe/types/UTF8String;)I
>  (64 bytes) @ 0x7f4559b4d4bd [0x7f4559b4d460+0x5d]
> {noformat}
> We initially saw this on IBM java on PowerPC box but is recreatable on linux 
> with OpenJDK. On linux with IBM Java 8 we see a null pointer exception at the 
> same code point:
> {noformat}
> 16/06/08 11:14:58 ERROR Executor: Exception in task 1.0 in stage 5.0 (TID 48)
> java.lang.NullPointerException
>   at 
> org.apache.spark.unsafe.types.UTF8String.compareTo(UTF8String.java:831)
>   at org.apache.spark.unsafe.types.UTF8String.compare(UTF8String.java:844)
>   at 
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.findNextInnerJoinRows$(Unknown
>  Source)
>   at 
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown
>  Source)
>   at 
> org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
>   at 
> org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$doExecute$2$$anon$2.hasNext(WholeStageCodegenExec.scala:377)
>   at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
>   at 
> scala.collection.convert.Wrappers$IteratorWrapper.hasNext(Wrappers.scala:30)
>   at org.spark_project.guava.collect.Ordering.leastOf(Ordering.java:664)
>   at org.apache.spark.util.collection.Utils$.takeOrdered(Utils.scala:37)
>   at 
> org.apache.spark.rdd.RDD$$anonfun$takeOrdered$1$$anonfun$30.apply(RDD.scala:1365)
>   at 
> org.apache.spark.rdd.RDD$$anonfun$takeOrdered$1$$anonfun$30.apply(RDD.scala:1362)
>   at 
> org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:757)
>   at 
> org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:757)
>   at 
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
>   at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:318)
>   at org.apache.spark.rdd.RDD.iterator(RDD.scala:282)
>   at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)
>   at org.apache.spark.scheduler.Task.run(Task.scala:85)
>   at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1153)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>   at java.lang.Thread.run(Thread.java:785)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-15822) segmentation violation in o.a.s.unsafe.types.UTF8String

[Pete Robbins (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-15822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15325759#comment-15325759
 ] 

Pete Robbins commented on SPARK-15822:
--

The stack trace is taken earlier when I detect that the UTF8String created from 
the corrupt UnsafeRow is created as I'm trying to backtrack to the point of 
corruption. The earlier stacktrace is the npe which occurs later on trying to 
use the corrupt UTF8String.

Dumb question but how do I post the plan?

> segmentation violation in o.a.s.unsafe.types.UTF8String 
> 
>
> Key: SPARK-15822
> URL: https://issues.apache.org/jira/browse/SPARK-15822
> Project: Spark
>  Issue Type: Bug
>Affects Versions: 2.0.0
> Environment: linux amd64
> openjdk version "1.8.0_91"
> OpenJDK Runtime Environment (build 1.8.0_91-b14)
> OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode)
>Reporter: Pete Robbins
>Assignee: Herman van Hovell
>Priority: Blocker
>
> Executors fail with segmentation violation while running application with
> spark.memory.offHeap.enabled true
> spark.memory.offHeap.size 512m
> Also now reproduced with 
> spark.memory.offHeap.enabled false
> {noformat}
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  SIGSEGV (0xb) at pc=0x7f4559b4d4bd, pid=14182, tid=139935319750400
> #
> # JRE version: OpenJDK Runtime Environment (8.0_91-b14) (build 1.8.0_91-b14)
> # Java VM: OpenJDK 64-Bit Server VM (25.91-b14 mixed mode linux-amd64 
> compressed oops)
> # Problematic frame:
> # J 4816 C2 
> org.apache.spark.unsafe.types.UTF8String.compareTo(Lorg/apache/spark/unsafe/types/UTF8String;)I
>  (64 bytes) @ 0x7f4559b4d4bd [0x7f4559b4d460+0x5d]
> {noformat}
> We initially saw this on IBM java on PowerPC box but is recreatable on linux 
> with OpenJDK. On linux with IBM Java 8 we see a null pointer exception at the 
> same code point:
> {noformat}
> 16/06/08 11:14:58 ERROR Executor: Exception in task 1.0 in stage 5.0 (TID 48)
> java.lang.NullPointerException
>   at 
> org.apache.spark.unsafe.types.UTF8String.compareTo(UTF8String.java:831)
>   at org.apache.spark.unsafe.types.UTF8String.compare(UTF8String.java:844)
>   at 
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.findNextInnerJoinRows$(Unknown
>  Source)
>   at 
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown
>  Source)
>   at 
> org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
>   at 
> org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$doExecute$2$$anon$2.hasNext(WholeStageCodegenExec.scala:377)
>   at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
>   at 
> scala.collection.convert.Wrappers$IteratorWrapper.hasNext(Wrappers.scala:30)
>   at org.spark_project.guava.collect.Ordering.leastOf(Ordering.java:664)
>   at org.apache.spark.util.collection.Utils$.takeOrdered(Utils.scala:37)
>   at 
> org.apache.spark.rdd.RDD$$anonfun$takeOrdered$1$$anonfun$30.apply(RDD.scala:1365)
>   at 
> org.apache.spark.rdd.RDD$$anonfun$takeOrdered$1$$anonfun$30.apply(RDD.scala:1362)
>   at 
> org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:757)
>   at 
> org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:757)
>   at 
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
>   at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:318)
>   at org.apache.spark.rdd.RDD.iterator(RDD.scala:282)
>   at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)
>   at org.apache.spark.scheduler.Task.run(Task.scala:85)
>   at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1153)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>   at java.lang.Thread.run(Thread.java:785)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-15822) segmentation violation in o.a.s.unsafe.types.UTF8String

[Pete Robbins (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-15822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15325159#comment-15325159
 ] 

Pete Robbins commented on SPARK-15822:
--

I am forcing a system dump when I detect a corrupt UTF8String is being created. 
This is using IBM JVM because I can analyse the dump and see the stacks and 
object contents using Eclipse Memory Analyzer.

So... with whole stage codegen enabled we get a stack of:
java.lang.Thread @ 0x835f9838  
|- at com.ibm.jvm.Dump.SystemDumpImpl()I (Native Method)

  
|- at com.ibm.jvm.Dump.SystemDump()V (Dump.java:139)

  
|- at org.apache.spark.unsafe.types.UTF8String.(Ljava/lang/Object;JI)V 
(UTF8String.java:125(Compiled Code))


|- at 
org.apache.spark.unsafe.types.UTF8String.fromAddress(Ljava/lang/Object;JI)Lorg/apache/spark/unsafe/types/UTF8String;
 (UTF8String.java:102(Compiled Code))   

|- at 
org.apache.spark.sql.catalyst.expressions.UnsafeRow.getUTF8String(I)Lorg/apache/spark/unsafe/types/UTF8String;
 (UnsafeRow.java:414(Compiled Code))
  
|- at 
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.sort_addToSorter$(Lorg/apache/spark/sql/catalyst/expressions/GeneratedClass$GeneratedIterator;)V
 (null)  
|- at 
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext()V
 (null) 
 
|- at org.apache.spark.sql.execution.BufferedRowIterator.hasNext()Z 
(BufferedRowIterator.java:43(Compiled Code))


|- at 
org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext()Z
 (WholeStageCodegenExec.scala:361(Compiled Code))   

|- at 
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.findNextInnerJoinRows$(Lorg/apache/spark/sql/catalyst/expressions/GeneratedClass$GeneratedIterator;Lscala/collection/Iterator;Lscala/collection/Iterator;)Z
 (null)
|- at 
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext()V
 (null) 
  
|- at org.apache.spark.sql.execution.BufferedRowIterator.hasNext()Z 
(BufferedRowIterator.java:43(Compiled Code))

 
|- at 
org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$doExecute$2$$anon$2.hasNext()Z
 (WholeStageCodegenExec.scala:377)  

|- at scala.collection.Iterator$$anon$11.hasNext()Z 
(Iterator.scala:408(Compiled Code)) 


|- at scala.collection.convert.Wrappers$IteratorWrapper.hasNext()Z 
(Wrappers.scala:30) 

  
|- at 
org.spark_project.guava.collect.Ordering.leastOf(Ljava/util/Iterator;I)Ljava/util/List;
 (Ordering.java:628)
 
|- at 
org.apache.spark.util.collection.Utils$.takeOrdered(Lscala/collection/Iterator;ILscala/math/Ordering;)Lscala/collection/Iterator;
 (Utils.scala:37)   

|- at 
org.apache.spark.rdd.RDD$$anonfun$takeOrdered$1$$anonfun$30.apply(Lscala/collection/Iterator;)Lscala/collection/Iterator;
 (RDD.scala:1365)   

|- at 
org.apache.spark.rdd.RDD$$anonfun$takeOrdered$1$$anonfun

[jira] [Comment Edited] (SPARK-15822) segmentation violation in o.a.s.unsafe.types.UTF8String

[Pete Robbins (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-15822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15325093#comment-15325093
 ] 

Pete Robbins edited comment on SPARK-15822 at 6/10/16 7:17 PM:
---

How do I disable whole-stage codegen?

found it 

spark.sql.codegen.wholeStage false


was (Author: robbinspg):
How do I disable whole-stage codegen?

> segmentation violation in o.a.s.unsafe.types.UTF8String 
> 
>
> Key: SPARK-15822
> URL: https://issues.apache.org/jira/browse/SPARK-15822
> Project: Spark
>  Issue Type: Bug
>Affects Versions: 2.0.0
> Environment: linux amd64
> openjdk version "1.8.0_91"
> OpenJDK Runtime Environment (build 1.8.0_91-b14)
> OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode)
>Reporter: Pete Robbins
>Assignee: Herman van Hovell
>Priority: Blocker
>
> Executors fail with segmentation violation while running application with
> spark.memory.offHeap.enabled true
> spark.memory.offHeap.size 512m
> Also now reproduced with 
> spark.memory.offHeap.enabled false
> {noformat}
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  SIGSEGV (0xb) at pc=0x7f4559b4d4bd, pid=14182, tid=139935319750400
> #
> # JRE version: OpenJDK Runtime Environment (8.0_91-b14) (build 1.8.0_91-b14)
> # Java VM: OpenJDK 64-Bit Server VM (25.91-b14 mixed mode linux-amd64 
> compressed oops)
> # Problematic frame:
> # J 4816 C2 
> org.apache.spark.unsafe.types.UTF8String.compareTo(Lorg/apache/spark/unsafe/types/UTF8String;)I
>  (64 bytes) @ 0x7f4559b4d4bd [0x7f4559b4d460+0x5d]
> {noformat}
> We initially saw this on IBM java on PowerPC box but is recreatable on linux 
> with OpenJDK. On linux with IBM Java 8 we see a null pointer exception at the 
> same code point:
> {noformat}
> 16/06/08 11:14:58 ERROR Executor: Exception in task 1.0 in stage 5.0 (TID 48)
> java.lang.NullPointerException
>   at 
> org.apache.spark.unsafe.types.UTF8String.compareTo(UTF8String.java:831)
>   at org.apache.spark.unsafe.types.UTF8String.compare(UTF8String.java:844)
>   at 
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.findNextInnerJoinRows$(Unknown
>  Source)
>   at 
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown
>  Source)
>   at 
> org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
>   at 
> org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$doExecute$2$$anon$2.hasNext(WholeStageCodegenExec.scala:377)
>   at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
>   at 
> scala.collection.convert.Wrappers$IteratorWrapper.hasNext(Wrappers.scala:30)
>   at org.spark_project.guava.collect.Ordering.leastOf(Ordering.java:664)
>   at org.apache.spark.util.collection.Utils$.takeOrdered(Utils.scala:37)
>   at 
> org.apache.spark.rdd.RDD$$anonfun$takeOrdered$1$$anonfun$30.apply(RDD.scala:1365)
>   at 
> org.apache.spark.rdd.RDD$$anonfun$takeOrdered$1$$anonfun$30.apply(RDD.scala:1362)
>   at 
> org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:757)
>   at 
> org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:757)
>   at 
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
>   at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:318)
>   at org.apache.spark.rdd.RDD.iterator(RDD.scala:282)
>   at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)
>   at org.apache.spark.scheduler.Task.run(Task.scala:85)
>   at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1153)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>   at java.lang.Thread.run(Thread.java:785)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-15822) segmentation violation in o.a.s.unsafe.types.UTF8String

[Pete Robbins (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-15822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15325093#comment-15325093
 ] 

Pete Robbins commented on SPARK-15822:
--

How do I disable whole-stage codegen?

> segmentation violation in o.a.s.unsafe.types.UTF8String 
> 
>
> Key: SPARK-15822
> URL: https://issues.apache.org/jira/browse/SPARK-15822
> Project: Spark
>  Issue Type: Bug
>Affects Versions: 2.0.0
> Environment: linux amd64
> openjdk version "1.8.0_91"
> OpenJDK Runtime Environment (build 1.8.0_91-b14)
> OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode)
>Reporter: Pete Robbins
>Assignee: Herman van Hovell
>Priority: Blocker
>
> Executors fail with segmentation violation while running application with
> spark.memory.offHeap.enabled true
> spark.memory.offHeap.size 512m
> Also now reproduced with 
> spark.memory.offHeap.enabled false
> {noformat}
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  SIGSEGV (0xb) at pc=0x7f4559b4d4bd, pid=14182, tid=139935319750400
> #
> # JRE version: OpenJDK Runtime Environment (8.0_91-b14) (build 1.8.0_91-b14)
> # Java VM: OpenJDK 64-Bit Server VM (25.91-b14 mixed mode linux-amd64 
> compressed oops)
> # Problematic frame:
> # J 4816 C2 
> org.apache.spark.unsafe.types.UTF8String.compareTo(Lorg/apache/spark/unsafe/types/UTF8String;)I
>  (64 bytes) @ 0x7f4559b4d4bd [0x7f4559b4d460+0x5d]
> {noformat}
> We initially saw this on IBM java on PowerPC box but is recreatable on linux 
> with OpenJDK. On linux with IBM Java 8 we see a null pointer exception at the 
> same code point:
> {noformat}
> 16/06/08 11:14:58 ERROR Executor: Exception in task 1.0 in stage 5.0 (TID 48)
> java.lang.NullPointerException
>   at 
> org.apache.spark.unsafe.types.UTF8String.compareTo(UTF8String.java:831)
>   at org.apache.spark.unsafe.types.UTF8String.compare(UTF8String.java:844)
>   at 
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.findNextInnerJoinRows$(Unknown
>  Source)
>   at 
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown
>  Source)
>   at 
> org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
>   at 
> org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$doExecute$2$$anon$2.hasNext(WholeStageCodegenExec.scala:377)
>   at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
>   at 
> scala.collection.convert.Wrappers$IteratorWrapper.hasNext(Wrappers.scala:30)
>   at org.spark_project.guava.collect.Ordering.leastOf(Ordering.java:664)
>   at org.apache.spark.util.collection.Utils$.takeOrdered(Utils.scala:37)
>   at 
> org.apache.spark.rdd.RDD$$anonfun$takeOrdered$1$$anonfun$30.apply(RDD.scala:1365)
>   at 
> org.apache.spark.rdd.RDD$$anonfun$takeOrdered$1$$anonfun$30.apply(RDD.scala:1362)
>   at 
> org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:757)
>   at 
> org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:757)
>   at 
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
>   at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:318)
>   at org.apache.spark.rdd.RDD.iterator(RDD.scala:282)
>   at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)
>   at org.apache.spark.scheduler.Task.run(Task.scala:85)
>   at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1153)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>   at java.lang.Thread.run(Thread.java:785)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-15822) segmentation violation in o.a.s.unsafe.types.UTF8String

[Pete Robbins (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-15822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15324506#comment-15324506
 ] 

Pete Robbins commented on SPARK-15822:
--

generated SMJ code from the stack:

{code}
public Object generate(Object[] references) {
return new GeneratedIterator(references);
}

/*wholestagecodegen_c1*/
final class GeneratedIterator extends 
org.apache.spark.sql.execution.BufferedRowIterator {
private Object[] references;
private scala.collection.Iterator smj_leftInput;
private scala.collection.Iterator smj_rightInput;
private InternalRow smj_leftRow;
private InternalRow smj_rightRow;
private UTF8String smj_value4;
private UTF8String smj_value5;
private java.util.ArrayList smj_matches;
private UTF8String smj_value6;
private UTF8String smj_value7;
private UTF8String smj_value8;
private boolean smj_isNull4;
private UTF8String smj_value9;
private boolean smj_isNull5;
private long smj_value10;
private org.apache.spark.sql.execution.metric.SQLMetric smj_numOutputRows;
private UnsafeRow smj_result;
private org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder 
smj_holder;
private org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter 
smj_rowWriter;
private UnsafeRow project_result;
private org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder 
project_holder;
private org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter 
project_rowWriter;

public GeneratedIterator(Object[] references) {
this.references = references;
}

public void init(int index, scala.collection.Iterator inputs[]) {
partitionIndex = index;
smj_leftInput = inputs[0];
smj_rightInput = inputs[1];

smj_rightRow = null;

smj_matches = new java.util.ArrayList();

this.smj_numOutputRows = (org.apache.spark.sql.execution.metric.SQLMetric) 
references[0];
smj_result = new UnsafeRow(6);
this.smj_holder = new 
org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder(smj_result, 128);
this.smj_rowWriter = new 
org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(smj_holder, 
6);
project_result = new UnsafeRow(3);
this.project_holder = new 
org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder(project_result, 
64);
this.project_rowWriter = new 
org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(project_holder,
 3);
}

private boolean findNextInnerJoinRows(
scala.collection.Iterator leftIter,
scala.collection.Iterator rightIter) {
smj_leftRow = null;
int comp = 0;
while (smj_leftRow == null) {
if (!leftIter.hasNext()) return false;
smj_leftRow = (InternalRow) leftIter.next();
/*smj_c1*/
boolean smj_isNull = smj_leftRow.isNullAt(0);
UTF8String smj_value = smj_isNull ? null : (smj_leftRow.getUTF8String(0));
/*smj_c2*/
boolean smj_isNull1 = smj_leftRow.isNullAt(1);
UTF8String smj_value1 = smj_isNull1 ? null : (smj_leftRow.getUTF8String(1));
if (smj_isNull || smj_isNull1) {
smj_leftRow = null;
continue;
}
if (!smj_matches.isEmpty()) {
comp = 0;
if (comp == 0) {
comp = smj_value.compare(smj_value6);
}
if (comp == 0) {
comp = smj_value1.compare(smj_value7);
}

if (comp == 0) {
return true;
}
smj_matches.clear();
}

do {
if (smj_rightRow == null) {
if (!rightIter.hasNext()) {
smj_value6 = smj_value;

smj_value7 = smj_value1;

return !smj_matches.isEmpty();
}
smj_rightRow = (InternalRow) rightIter.next();
/*smj_c3*/
boolean smj_isNull2 = smj_rightRow.isNullAt(0);
UTF8String smj_value2 = smj_isNull2 ? null : (smj_rightRow.getUTF8String(0));
/*smj_c4*/
boolean smj_isNull3 = smj_rightRow.isNullAt(1);
UTF8String smj_value3 = smj_isNull3 ? null : (smj_rightRow.getUTF8String(1));
if (smj_isNull2 || smj_isNull3) {
smj_rightRow = null;
continue;
}

smj_value4 = smj_value2;

smj_value5 = smj_value3;

}

comp = 0;
if (comp == 0) {
comp = smj_value.compare(smj_value4);
}
if (comp == 0) {
comp = smj_value1.compare(smj_value5);
}

if (comp > 0) {
smj_rightRow = null;
} else if (comp < 0) {
if (!smj_matches.isEmpty()) {
smj_value6 = smj_value;

smj_value7 = smj_value1;

return true;
}
smj_leftRow = null;
} else {
smj_matches.add(smj_rightRow.copy());
smj_rightRow = null;;
}
} while (smj_leftRow != null);
}
return false; // unreachable
}

protected void processNext() throws java.io.IOException {
/*project_c*/
/*smj_c*/
while (findNextInnerJoinRows(smj_leftInput, smj_rightInput)) {
int smj_size = smj_matches.size();
smj_isNull4 = smj_leftRow.isNullAt(0);
smj_value8 = smj_isNull4 ? null : (smj_leftRow.getUTF8String(0));
smj_isNull5 = smj_leftRow.isNullAt(1);
smj_value9 = smj_isNull5 ? null : (smj_leftRow.getUTF8String(1));
smj_value10 = smj_leftRow.getLong(2);
for (int smj_i = 0; smj_i < smj_size; smj_i ++) {
InternalRow smj_rightRow1 = (InternalRow) smj_matches.get(smj_i);

smj_numOutputRows.add(1);

/*project_c1*/
/*wholestagecodegen_c*/
/*project_c7*/
/*smj_c7*/
long smj_value13 = smj_rightRow1.getLong(2);
boolean project_isNull8 = false;
double project_value8 = -1.0;
if (!false) {
proje

[jira] [Updated] (SPARK-15822) segmentation violation in o.a.s.unsafe.types.UTF8String

[Pete Robbins (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-15822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pete Robbins updated SPARK-15822:
-
Description: 
Executors fail with segmentation violation while running application with
spark.memory.offHeap.enabled true
spark.memory.offHeap.size 512m

Also now reproduced with 
spark.memory.offHeap.enabled false

{noformat}
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x7f4559b4d4bd, pid=14182, tid=139935319750400
#
# JRE version: OpenJDK Runtime Environment (8.0_91-b14) (build 1.8.0_91-b14)
# Java VM: OpenJDK 64-Bit Server VM (25.91-b14 mixed mode linux-amd64 
compressed oops)
# Problematic frame:
# J 4816 C2 
org.apache.spark.unsafe.types.UTF8String.compareTo(Lorg/apache/spark/unsafe/types/UTF8String;)I
 (64 bytes) @ 0x7f4559b4d4bd [0x7f4559b4d460+0x5d]
{noformat}
We initially saw this on IBM java on PowerPC box but is recreatable on linux 
with OpenJDK. On linux with IBM Java 8 we see a null pointer exception at the 
same code point:
{noformat}
16/06/08 11:14:58 ERROR Executor: Exception in task 1.0 in stage 5.0 (TID 48)
java.lang.NullPointerException
at 
org.apache.spark.unsafe.types.UTF8String.compareTo(UTF8String.java:831)
at org.apache.spark.unsafe.types.UTF8String.compare(UTF8String.java:844)
at 
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.findNextInnerJoinRows$(Unknown
 Source)
at 
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown
 Source)
at 
org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at 
org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$doExecute$2$$anon$2.hasNext(WholeStageCodegenExec.scala:377)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
at 
scala.collection.convert.Wrappers$IteratorWrapper.hasNext(Wrappers.scala:30)
at org.spark_project.guava.collect.Ordering.leastOf(Ordering.java:664)
at org.apache.spark.util.collection.Utils$.takeOrdered(Utils.scala:37)
at 
org.apache.spark.rdd.RDD$$anonfun$takeOrdered$1$$anonfun$30.apply(RDD.scala:1365)
at 
org.apache.spark.rdd.RDD$$anonfun$takeOrdered$1$$anonfun$30.apply(RDD.scala:1362)
at 
org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:757)
at 
org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:757)
at 
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:318)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:282)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)
at org.apache.spark.scheduler.Task.run(Task.scala:85)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1153)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.lang.Thread.run(Thread.java:785)
{noformat}

  was:
Executors fail with segmentation violation while running application with
spark.memory.offHeap.enabled true
spark.memory.offHeap.size 512m
{noformat}
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x7f4559b4d4bd, pid=14182, tid=139935319750400
#
# JRE version: OpenJDK Runtime Environment (8.0_91-b14) (build 1.8.0_91-b14)
# Java VM: OpenJDK 64-Bit Server VM (25.91-b14 mixed mode linux-amd64 
compressed oops)
# Problematic frame:
# J 4816 C2 
org.apache.spark.unsafe.types.UTF8String.compareTo(Lorg/apache/spark/unsafe/types/UTF8String;)I
 (64 bytes) @ 0x7f4559b4d4bd [0x7f4559b4d460+0x5d]
{noformat}
We initially saw this on IBM java on PowerPC box but is recreatable on linux 
with OpenJDK. On linux with IBM Java 8 we see a null pointer exception at the 
same code point:
{noformat}
16/06/08 11:14:58 ERROR Executor: Exception in task 1.0 in stage 5.0 (TID 48)
java.lang.NullPointerException
at 
org.apache.spark.unsafe.types.UTF8String.compareTo(UTF8String.java:831)
at org.apache.spark.unsafe.types.UTF8String.compare(UTF8String.java:844)
at 
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.findNextInnerJoinRows$(Unknown
 Source)
at 
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown
 Source)
at 
org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at 
org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$doExecute$2$$anon$2.hasNext(WholeStageCodegenExec.scala:377)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408

[jira] [Updated] (SPARK-15822) segmentation violation in o.a.s.unsafe.types.UTF8String

[Pete Robbins (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-15822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pete Robbins updated SPARK-15822:
-
Summary: segmentation violation in o.a.s.unsafe.types.UTF8String   (was: 
segmentation violation in o.a.s.unsafe.types.UTF8String with 
spark.memory.offHeap.enabled=true)

> segmentation violation in o.a.s.unsafe.types.UTF8String 
> 
>
> Key: SPARK-15822
> URL: https://issues.apache.org/jira/browse/SPARK-15822
> Project: Spark
>  Issue Type: Bug
>Affects Versions: 2.0.0
> Environment: linux amd64
> openjdk version "1.8.0_91"
> OpenJDK Runtime Environment (build 1.8.0_91-b14)
> OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode)
>Reporter: Pete Robbins
>Assignee: Herman van Hovell
>Priority: Blocker
>
> Executors fail with segmentation violation while running application with
> spark.memory.offHeap.enabled true
> spark.memory.offHeap.size 512m
> {noformat}
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  SIGSEGV (0xb) at pc=0x7f4559b4d4bd, pid=14182, tid=139935319750400
> #
> # JRE version: OpenJDK Runtime Environment (8.0_91-b14) (build 1.8.0_91-b14)
> # Java VM: OpenJDK 64-Bit Server VM (25.91-b14 mixed mode linux-amd64 
> compressed oops)
> # Problematic frame:
> # J 4816 C2 
> org.apache.spark.unsafe.types.UTF8String.compareTo(Lorg/apache/spark/unsafe/types/UTF8String;)I
>  (64 bytes) @ 0x7f4559b4d4bd [0x7f4559b4d460+0x5d]
> {noformat}
> We initially saw this on IBM java on PowerPC box but is recreatable on linux 
> with OpenJDK. On linux with IBM Java 8 we see a null pointer exception at the 
> same code point:
> {noformat}
> 16/06/08 11:14:58 ERROR Executor: Exception in task 1.0 in stage 5.0 (TID 48)
> java.lang.NullPointerException
>   at 
> org.apache.spark.unsafe.types.UTF8String.compareTo(UTF8String.java:831)
>   at org.apache.spark.unsafe.types.UTF8String.compare(UTF8String.java:844)
>   at 
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.findNextInnerJoinRows$(Unknown
>  Source)
>   at 
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown
>  Source)
>   at 
> org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
>   at 
> org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$doExecute$2$$anon$2.hasNext(WholeStageCodegenExec.scala:377)
>   at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
>   at 
> scala.collection.convert.Wrappers$IteratorWrapper.hasNext(Wrappers.scala:30)
>   at org.spark_project.guava.collect.Ordering.leastOf(Ordering.java:664)
>   at org.apache.spark.util.collection.Utils$.takeOrdered(Utils.scala:37)
>   at 
> org.apache.spark.rdd.RDD$$anonfun$takeOrdered$1$$anonfun$30.apply(RDD.scala:1365)
>   at 
> org.apache.spark.rdd.RDD$$anonfun$takeOrdered$1$$anonfun$30.apply(RDD.scala:1362)
>   at 
> org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:757)
>   at 
> org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:757)
>   at 
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
>   at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:318)
>   at org.apache.spark.rdd.RDD.iterator(RDD.scala:282)
>   at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)
>   at org.apache.spark.scheduler.Task.run(Task.scala:85)
>   at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1153)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>   at java.lang.Thread.run(Thread.java:785)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-15822) segmentation violation in o.a.s.unsafe.types.UTF8String with spark.memory.offHeap.enabled=true

[Pete Robbins (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-15822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15324331#comment-15324331
 ] 

Pete Robbins commented on SPARK-15822:
--

I'm still looking into this tracing back through the code using Memory Analyzer 
on the core dumps.

Currently on the stack we have the following generated code

{code}
public Object generate(Object[] references) {
return new GeneratedIterator(references);
}

final class GeneratedIterator extends 
org.apache.spark.sql.execution.BufferedRowIterator {
private Object[] references;
private boolean sort_needToSort;
private org.apache.spark.sql.execution.SortExec sort_plan;
private org.apache.spark.sql.execution.UnsafeExternalRowSorter sort_sorter;
private org.apache.spark.executor.TaskMetrics sort_metrics;
private scala.collection.Iterator sort_sortedIter;
private boolean agg_initAgg;
private boolean agg_bufIsNull;
private long agg_bufValue;
private org.apache.spark.sql.execution.aggregate.HashAggregateExec agg_plan;
private org.apache.spark.sql.execution.UnsafeFixedWidthAggregationMap 
agg_hashMap;
private org.apache.spark.sql.execution.UnsafeKVExternalSorter agg_sorter;
private org.apache.spark.unsafe.KVIterator agg_mapIter;
private org.apache.spark.sql.execution.metric.SQLMetric agg_peakMemory;
private org.apache.spark.sql.execution.metric.SQLMetric agg_spillSize;
private scala.collection.Iterator inputadapter_input;
private UnsafeRow agg_result;
private org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder 
agg_holder;
private org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter 
agg_rowWriter;
private UnsafeRow agg_result1;
private org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder 
agg_holder1;
private org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter 
agg_rowWriter1;
private org.apache.spark.sql.execution.metric.SQLMetric sort_numOutputRows;
private org.apache.spark.sql.execution.metric.SQLMetric sort_aggTime;
private org.apache.spark.sql.execution.metric.SQLMetric sort_peakMemory;
private org.apache.spark.sql.execution.metric.SQLMetric sort_spillSize;
private org.apache.spark.sql.execution.metric.SQLMetric sort_sortTime;

public GeneratedIterator(Object[] references) {
this.references = references;
}

public void init(int index, scala.collection.Iterator inputs[]) {
partitionIndex = index;
sort_needToSort = true;
this.sort_plan = (org.apache.spark.sql.execution.SortExec) references[0];
sort_sorter = sort_plan.createSorter();
sort_metrics = org.apache.spark.TaskContext.get().taskMetrics();

agg_initAgg = false;

this.agg_plan = (org.apache.spark.sql.execution.aggregate.HashAggregateExec) 
references[1];

this.agg_peakMemory = (org.apache.spark.sql.execution.metric.SQLMetric) 
references[2];
this.agg_spillSize = (org.apache.spark.sql.execution.metric.SQLMetric) 
references[3];
inputadapter_input = inputs[0];
agg_result = new UnsafeRow(2);
this.agg_holder = new 
org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder(agg_result, 64);
this.agg_rowWriter = new 
org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(agg_holder, 
2);
agg_result1 = new UnsafeRow(3);
this.agg_holder1 = new 
org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder(agg_result1, 64);
this.agg_rowWriter1 = new 
org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(agg_holder1, 
3);
this.sort_numOutputRows = (org.apache.spark.sql.execution.metric.SQLMetric) 
references[4];
this.sort_aggTime = (org.apache.spark.sql.execution.metric.SQLMetric) 
references[5];
this.sort_peakMemory = (org.apache.spark.sql.execution.metric.SQLMetric) 
references[6];
this.sort_spillSize = (org.apache.spark.sql.execution.metric.SQLMetric) 
references[7];
this.sort_sortTime = (org.apache.spark.sql.execution.metric.SQLMetric) 
references[8];
}

private void agg_doAggregateWithKeys() throws java.io.IOException {
agg_hashMap = agg_plan.createHashMap();

while (inputadapter_input.hasNext()) {
InternalRow inputadapter_row = (InternalRow) inputadapter_input.next();
boolean inputadapter_isNull = inputadapter_row.isNullAt(0);
UTF8String inputadapter_value = inputadapter_isNull ? null : 
(inputadapter_row.getUTF8String(0));
boolean inputadapter_isNull1 = inputadapter_row.isNullAt(1);
UTF8String inputadapter_value1 = inputadapter_isNull1 ? null : 
(inputadapter_row.getUTF8String(1));
long inputadapter_value2 = inputadapter_row.getLong(2);

UnsafeRow agg_unsafeRowAggBuffer = null;
org.apache.spark.sql.execution.vectorized.ColumnarBatch.Row 
agg_vectorizedAggBuffer = null;

if (agg_vectorizedAggBuffer == null) {
// generate grouping key
agg_holder.reset();

agg_rowWriter.zeroOutNullBytes();

if (inputadapter_isNull) {
agg_rowWriter.setNullAt(0);
} else {
agg_rowWriter.write(0, inputadapter_value);
}

if (inputadapter_isNull1) {
agg_rowWriter.setNullAt(1);
} else {
agg_rowWriter.write(1, inputadapter_value1);
}
agg_result.setTotalSize

Re: Spark 2.0.0 preview docs uploaded

[Pete Robbins]

It would be nice to have a "what's new in 2.0.0" equivalent to
https://spark.apache.org/releases/spark-release-1-6-0.html available or am
I just missing it?

On Wed, 8 Jun 2016 at 13:15 Sean Owen  wrote:

> OK, this is done:
>
> http://spark.apache.org/documentation.html
> http://spark.apache.org/docs/2.0.0-preview/
> http://spark.apache.org/docs/preview/
>
> On Tue, Jun 7, 2016 at 4:59 PM, Shivaram Venkataraman
>  wrote:
> > As far as I know the process is just to copy docs/_site from the build
> > to the appropriate location in the SVN repo (i.e.
> > site/docs/2.0.0-preview).
> >
> > Thanks
> > Shivaram
> >
> > On Tue, Jun 7, 2016 at 8:14 AM, Sean Owen  wrote:
> >> As a stop-gap, I can edit that page to have a small section about
> >> preview releases and point to the nightly docs.
> >>
> >> Not sure who has the power to push 2.0.0-preview to site/docs, but, if
> >> that's done then we can symlink "preview" in that dir to it and be
> >> done, and update this section about preview docs accordingly.
> >>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
> For additional commands, e-mail: dev-h...@spark.apache.org
>
>

Re: NegativeArraySizeException / segfault

[Pete Robbins]

I just raised https://issues.apache.org/jira/browse/SPARK-15822 for a
similar looking issue. Analyzing the core dump from the segv with Memory
Analyzer it looks very much like a UTF8String is very corrupt.

Cheers,

On Fri, 27 May 2016 at 21:00 Koert Kuipers  wrote:

> hello all,
> after getting our unit tests to pass on spark 2.0.0-SNAPSHOT we are now
> trying to run some algorithms at scale on our cluster.
> unfortunately this means that when i see errors i am having a harder time
> boiling it down to a small reproducible example.
>
> today we are running an iterative algo using the dataset api and we are
> seeing tasks fail with errors which seem to related to unsafe operations.
> the same tasks succeed without issues in our unit tests.
>
> i see either:
>
> 16/05/27 12:54:46 ERROR executor.Executor: Exception in task 31.0 in stage
> 21.0 (TID 1073)
> java.lang.NegativeArraySizeException
> at
> org.apache.spark.unsafe.types.UTF8String.getBytes(UTF8String.java:229)
> at
> org.apache.spark.unsafe.types.UTF8String.toString(UTF8String.java:821)
> at
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificSafeProjection.apply(Unknown
> Source)
> at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
> at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
> at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
> at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434)
> at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
> at
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.sort_addToSorter$(Unknown
> Source)
> at
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown
> Source)
> at
> org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
> at
> org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$7$$anon$1.hasNext(WholeStageCodegenExec.scala:359)
> at
> org.apache.spark.sql.execution.aggregate.SortBasedAggregateExec$$anonfun$doExecute$1$$anonfun$3.apply(SortBasedAggregateExec.scala:74)
> at
> org.apache.spark.sql.execution.aggregate.SortBasedAggregateExec$$anonfun$doExecute$1$$anonfun$3.apply(SortBasedAggregateExec.scala:71)
> at
> org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:775)
> at
> org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:775)
> at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:318)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:282)
> at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:318)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:282)
> at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79)
> at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47)
> at org.apache.spark.scheduler.Task.run(Task.scala:85)
> at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>
> or alternatively:
>
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  SIGSEGV (0xb) at pc=0x7fe571041cba, pid=2450, tid=140622965913344
> #
> # JRE version: Java(TM) SE Runtime Environment (7.0_75-b13) (build
> 1.7.0_75-b13)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (24.75-b04 mixed mode
> linux-amd64 compressed oops)
> # Problematic frame:
> # v  ~StubRoutines::jbyte_disjoint_arraycopy
>
> i assume the best thing would be to try to get it to print out the
> generated code that is causing this?
> what switch do i need to use again to do so?
> thanks,
> koert
>

[jira] [Commented] (SPARK-15822) segmentation violation in o.a.s.unsafe.types.UTF8String with spark.memory.offHeap.enabled=true

[Pete Robbins (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-15822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15320380#comment-15320380
 ] 

Pete Robbins commented on SPARK-15822:
--

I'm investigating this and will attach the app and config later

> segmentation violation in o.a.s.unsafe.types.UTF8String with 
> spark.memory.offHeap.enabled=true
> --
>
> Key: SPARK-15822
> URL: https://issues.apache.org/jira/browse/SPARK-15822
> Project: Spark
>  Issue Type: Bug
>Affects Versions: 2.0.0
> Environment: linux amd64
> openjdk version "1.8.0_91"
> OpenJDK Runtime Environment (build 1.8.0_91-b14)
> OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode)
>Reporter: Pete Robbins
>Priority: Critical
>
> Executors fail with segmentation violation while running application with
> spark.memory.offHeap.enabled true
> spark.memory.offHeap.size 512m
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  SIGSEGV (0xb) at pc=0x7f4559b4d4bd, pid=14182, tid=139935319750400
> #
> # JRE version: OpenJDK Runtime Environment (8.0_91-b14) (build 1.8.0_91-b14)
> # Java VM: OpenJDK 64-Bit Server VM (25.91-b14 mixed mode linux-amd64 
> compressed oops)
> # Problematic frame:
> # J 4816 C2 
> org.apache.spark.unsafe.types.UTF8String.compareTo(Lorg/apache/spark/unsafe/types/UTF8String;)I
>  (64 bytes) @ 0x7f4559b4d4bd [0x7f4559b4d460+0x5d]
> We initially saw this on IBM java on PowerPC box but is recreatable on linux 
> with OpenJDK. On linux with IBM Java 8 we see a null pointer exception at the 
> same code point:
> 16/06/08 11:14:58 ERROR Executor: Exception in task 1.0 in stage 5.0 (TID 48)
> java.lang.NullPointerException
>   at 
> org.apache.spark.unsafe.types.UTF8String.compareTo(UTF8String.java:831)
>   at org.apache.spark.unsafe.types.UTF8String.compare(UTF8String.java:844)
>   at 
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.findNextInnerJoinRows$(Unknown
>  Source)
>   at 
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown
>  Source)
>   at 
> org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
>   at 
> org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$doExecute$2$$anon$2.hasNext(WholeStageCodegenExec.scala:377)
>   at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
>   at 
> scala.collection.convert.Wrappers$IteratorWrapper.hasNext(Wrappers.scala:30)
>   at org.spark_project.guava.collect.Ordering.leastOf(Ordering.java:664)
>   at org.apache.spark.util.collection.Utils$.takeOrdered(Utils.scala:37)
>   at 
> org.apache.spark.rdd.RDD$$anonfun$takeOrdered$1$$anonfun$30.apply(RDD.scala:1365)
>   at 
> org.apache.spark.rdd.RDD$$anonfun$takeOrdered$1$$anonfun$30.apply(RDD.scala:1362)
>   at 
> org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:757)
>   at 
> org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:757)
>   at 
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
>   at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:318)
>   at org.apache.spark.rdd.RDD.iterator(RDD.scala:282)
>   at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)
>   at org.apache.spark.scheduler.Task.run(Task.scala:85)
>   at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1153)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>   at java.lang.Thread.run(Thread.java:785)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Created] (SPARK-15822) segmentation violation in o.a.s.unsafe.types.UTF8String with spark.memory.offHeap.enabled=true

[Pete Robbins (JIRA)]

Pete Robbins created SPARK-15822:


 Summary: segmentation violation in o.a.s.unsafe.types.UTF8String 
with spark.memory.offHeap.enabled=true
 Key: SPARK-15822
 URL: https://issues.apache.org/jira/browse/SPARK-15822
 Project: Spark
  Issue Type: Bug
Affects Versions: 2.0.0
 Environment: linux amd64

openjdk version "1.8.0_91"
OpenJDK Runtime Environment (build 1.8.0_91-b14)
OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode)

Reporter: Pete Robbins
Priority: Critical


Executors fail with segmentation violation while running application with
spark.memory.offHeap.enabled true
spark.memory.offHeap.size 512m

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x7f4559b4d4bd, pid=14182, tid=139935319750400
#
# JRE version: OpenJDK Runtime Environment (8.0_91-b14) (build 1.8.0_91-b14)
# Java VM: OpenJDK 64-Bit Server VM (25.91-b14 mixed mode linux-amd64 
compressed oops)
# Problematic frame:
# J 4816 C2 
org.apache.spark.unsafe.types.UTF8String.compareTo(Lorg/apache/spark/unsafe/types/UTF8String;)I
 (64 bytes) @ 0x7f4559b4d4bd [0x7f4559b4d460+0x5d]

We initially saw this on IBM java on PowerPC box but is recreatable on linux 
with OpenJDK. On linux with IBM Java 8 we see a null pointer exception at the 
same code point:

16/06/08 11:14:58 ERROR Executor: Exception in task 1.0 in stage 5.0 (TID 48)
java.lang.NullPointerException
at 
org.apache.spark.unsafe.types.UTF8String.compareTo(UTF8String.java:831)
at org.apache.spark.unsafe.types.UTF8String.compare(UTF8String.java:844)
at 
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.findNextInnerJoinRows$(Unknown
 Source)
at 
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown
 Source)
at 
org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at 
org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$doExecute$2$$anon$2.hasNext(WholeStageCodegenExec.scala:377)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
at 
scala.collection.convert.Wrappers$IteratorWrapper.hasNext(Wrappers.scala:30)
at org.spark_project.guava.collect.Ordering.leastOf(Ordering.java:664)
at org.apache.spark.util.collection.Utils$.takeOrdered(Utils.scala:37)
at 
org.apache.spark.rdd.RDD$$anonfun$takeOrdered$1$$anonfun$30.apply(RDD.scala:1365)
at 
org.apache.spark.rdd.RDD$$anonfun$takeOrdered$1$$anonfun$30.apply(RDD.scala:1362)
at 
org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:757)
at 
org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:757)
at 
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:318)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:282)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)
at org.apache.spark.scheduler.Task.run(Task.scala:85)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1153)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.lang.Thread.run(Thread.java:785)




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-15065) HiveSparkSubmitSuite's "set spark.sql.warehouse.dir" is flaky

[Pete Robbins (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-15065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15318492#comment-15318492
 ] 

Pete Robbins commented on SPARK-15065:
--

I think this may be related to 
https://issues.apache.org/jira/browse/SPARK-15606 where there is a deadlock in 
executor shutdown. This test was consistently failing on our machine with only 
2 cores but since my fix to SPARK-15606 it has passed all the time.

> HiveSparkSubmitSuite's "set spark.sql.warehouse.dir" is flaky
> -
>
> Key: SPARK-15065
> URL: https://issues.apache.org/jira/browse/SPARK-15065
> Project: Spark
>  Issue Type: Bug
>  Components: SQL, Tests
>Reporter: Yin Huai
>Priority: Critical
> Attachments: log.txt
>
>
> https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-sbt-hadoop-2.4/861/testReport/junit/org.apache.spark.sql.hive/HiveSparkSubmitSuite/dir/
> There are several WARN messages like {{16/05/02 00:51:06 WARN Master: Got 
> status update for unknown executor app-20160502005054-/3}}, which are 
> suspicious. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Created] (SPARK-15606) Driver hang in o.a.s.DistributedSuite on 2 core machine

[Pete Robbins (JIRA)]

Pete Robbins created SPARK-15606:


 Summary: Driver hang in o.a.s.DistributedSuite on 2 core machine
 Key: SPARK-15606
 URL: https://issues.apache.org/jira/browse/SPARK-15606
 Project: Spark
  Issue Type: Bug
  Components: Spark Core
Affects Versions: 2.0.0
 Environment: AMD64 box with only 2 cores
Reporter: Pete Robbins


repeatedly failing task that crashes JVM *** FAILED ***
  The code passed to failAfter did not complete within 10 milliseconds. 
(DistributedSuite.scala:128)

This test started failing and DistrbutedSuite hanging following 
https://github.com/apache/spark/pull/13055

It looks like the extra message to remove the BlockManager deadlocks as there 
are only 2 message processing loop threads. Related to 
https://issues.apache.org/jira/browse/SPARK-13906

{code}
  /** Thread pool used for dispatching messages. */
  private val threadpool: ThreadPoolExecutor = {
val numThreads = 
nettyEnv.conf.getInt("spark.rpc.netty.dispatcher.numThreads",
  math.max(2, Runtime.getRuntime.availableProcessors()))
val pool = ThreadUtils.newDaemonFixedThreadPool(numThreads, 
"dispatcher-event-loop")
for (i <- 0 until numThreads) {
  pool.execute(new MessageLoop)
}
pool
  }

{code} 

Setting a minimum of 3 threads alleviates this issue but I'm not sure there 
isn't another underlying problem.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-15154) LongHashedRelation test fails on Big Endian platform

[Pete Robbins (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-15154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pete Robbins updated SPARK-15154:
-
Priority: Minor  (was: Major)
 Summary: LongHashedRelation test fails on Big Endian platform  (was: 
LongHashedRelation fails on Big Endian platform)

> LongHashedRelation test fails on Big Endian platform
> 
>
> Key: SPARK-15154
> URL: https://issues.apache.org/jira/browse/SPARK-15154
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.0.0
>Reporter: Pete Robbins
>Priority: Minor
>  Labels: big-endian
>
> NPE in 
> org.apache.spark.sql.execution.joins.HashedRelationSuite.LongToUnsafeRowMap
> Error Message
> java.lang.NullPointerException was thrown.
> Stacktrace
>   java.lang.NullPointerException
>   at 
> org.apache.spark.sql.execution.joins.HashedRelationSuite$$anonfun$3$$anonfun$apply$mcV$sp$1.apply$mcVI$sp(HashedRelationSuite.scala:121)
>   at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:160)
>   at 
> org.apache.spark.sql.execution.joins.HashedRelationSuite$$anonfun$3.apply$mcV$sp(HashedRelationSuite.scala:119)
>   at 
> org.apache.spark.sql.execution.joins.HashedRelationSuite$$anonfun$3.apply(HashedRelationSuite.scala:112)
>   at 
> org.apache.spark.sql.execution.joins.HashedRelationSuite$$anonfun$3.apply(HashedRelationSuite.scala:112)
>   at 
> org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
>   at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
>   at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
>   at org.scalatest.Transformer.apply(Transformer.scala:22)
>   at org.scalatest.Transformer.apply(Transformer.scala:20)
>   at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:166)
>   at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:57)
>   at 
> org.scalatest.FunSuiteLike$class.invokeWithFixture$1(FunSuiteLike.scala:163)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)
>   at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
>   at org.scalatest.FunSuiteLike$class.runTest(FunSuiteLike.scala:175)
>   at org.scalatest.FunSuite.runTest(FunSuite.scala:1555)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:208)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:208)
>   at 
> org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:413)
>   at 
> org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:401)
>   at scala.collection.immutable.List.foreach(List.scala:381)
>   at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
>   at 
> org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:396)
>   at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:483)
>   at org.scalatest.FunSuiteLike$class.runTests(FunSuiteLike.scala:208)
>   at org.scalatest.FunSuite.runTests(FunSuite.scala:1555)
>   at org.scalatest.Suite$class.run(Suite.scala:1424)
>   at 
> org.scalatest.FunSuite.org$scalatest$FunSuiteLike$$super$run(FunSuite.scala:1555)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:212)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:212)
>   at org.scalatest.SuperEngine.runImpl(Engine.scala:545)
>   at org.scalatest.FunSuiteLike$class.run(FunSuiteLike.scala:212)
>   at 
> org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:29)
>   at 
> org.scalatest.BeforeAndAfterAll$class.liftedTree1$1(BeforeAndAfterAll.scala:257)
>   at 
> org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:256)
>   at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:29)
>   at org.scalatest.Suite$class.callExecuteOnSuite$1(Suite.scala:1492)
>   at 
> org.scalatest.Suite$$anonfun$runNestedSuites$1.apply(Suite.scala:1528)
>   at 
> org.scalatest.Suite$$anonfun$runNestedSuites$1.apply(Suite.scala:1526)
>   at 
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
>   at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
>   at org.scalatest.Suite$class.runNestedSuites(Suite.scala:1526)
>   at 
> org.scalatest

[jira] [Commented] (SPARK-15154) LongHashedRelation fails on Big Endian platform

[Pete Robbins (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-15154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15276543#comment-15276543
 ] 

Pete Robbins commented on SPARK-15154:
--

I'm convinced the test is invalid. The creation of LongHashedRelation is 
guarded by

{code}
   if (key.length == 1 && key.head.dataType == LongType) {
  LongHashedRelation(input, key, sizeEstimate, mm)
}
{code}

In this failing test the key dataType is IntegerType

I'll submit a PR to fix the tests

> LongHashedRelation fails on Big Endian platform
> ---
>
> Key: SPARK-15154
> URL: https://issues.apache.org/jira/browse/SPARK-15154
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.0.0
>Reporter: Pete Robbins
>  Labels: big-endian
>
> NPE in 
> org.apache.spark.sql.execution.joins.HashedRelationSuite.LongToUnsafeRowMap
> Error Message
> java.lang.NullPointerException was thrown.
> Stacktrace
>   java.lang.NullPointerException
>   at 
> org.apache.spark.sql.execution.joins.HashedRelationSuite$$anonfun$3$$anonfun$apply$mcV$sp$1.apply$mcVI$sp(HashedRelationSuite.scala:121)
>   at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:160)
>   at 
> org.apache.spark.sql.execution.joins.HashedRelationSuite$$anonfun$3.apply$mcV$sp(HashedRelationSuite.scala:119)
>   at 
> org.apache.spark.sql.execution.joins.HashedRelationSuite$$anonfun$3.apply(HashedRelationSuite.scala:112)
>   at 
> org.apache.spark.sql.execution.joins.HashedRelationSuite$$anonfun$3.apply(HashedRelationSuite.scala:112)
>   at 
> org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
>   at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
>   at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
>   at org.scalatest.Transformer.apply(Transformer.scala:22)
>   at org.scalatest.Transformer.apply(Transformer.scala:20)
>   at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:166)
>   at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:57)
>   at 
> org.scalatest.FunSuiteLike$class.invokeWithFixture$1(FunSuiteLike.scala:163)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)
>   at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
>   at org.scalatest.FunSuiteLike$class.runTest(FunSuiteLike.scala:175)
>   at org.scalatest.FunSuite.runTest(FunSuite.scala:1555)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:208)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:208)
>   at 
> org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:413)
>   at 
> org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:401)
>   at scala.collection.immutable.List.foreach(List.scala:381)
>   at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
>   at 
> org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:396)
>   at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:483)
>   at org.scalatest.FunSuiteLike$class.runTests(FunSuiteLike.scala:208)
>   at org.scalatest.FunSuite.runTests(FunSuite.scala:1555)
>   at org.scalatest.Suite$class.run(Suite.scala:1424)
>   at 
> org.scalatest.FunSuite.org$scalatest$FunSuiteLike$$super$run(FunSuite.scala:1555)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:212)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:212)
>   at org.scalatest.SuperEngine.runImpl(Engine.scala:545)
>   at org.scalatest.FunSuiteLike$class.run(FunSuiteLike.scala:212)
>   at 
> org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:29)
>   at 
> org.scalatest.BeforeAndAfterAll$class.liftedTree1$1(BeforeAndAfterAll.scala:257)
>   at 
> org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:256)
>   at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:29)
>   at org.scalatest.Suite$class.callExecuteOnSuite$1(Suite.scala:1492)
>   at 
> org.scalatest.Suite$$anonfun$runNestedSuites$1.apply(Suite.scala:1528)
>   at 
> org.scalatest.Suite$$anonfun$runNestedSuites$1.apply(Suite.scala:1526)
>   at 
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
>   at scala.collec

[jira] [Comment Edited] (SPARK-15154) LongHashedRelation fails on Big Endian platform

[Pete Robbins (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-15154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15273885#comment-15273885
 ] 

Pete Robbins edited comment on SPARK-15154 at 5/6/16 11:20 AM:
---

[~davies] as you are the author of this code can you comment on my findings?

So the issue here is that the keyGenerator returns an UnsafeRow containing Int 
values but the code below from LongHashedRelation.apply retrieves the key from 
this as a Long. The bytes in the row are

on Little Endian: 01 00 00 00 00 00 00 00 
on Big Endian:   00 00 00 01 00 00 00 00

By chance getInt and getLong  will both return "1" on Little Endian because the 
following 4 bytes happen to be 0, whereas on Big Endian getInt returns "1" but 
get Long will return "268435456"

{code}
val keyGenerator = UnsafeProjection.create(key)

// Create a mapping of key -> rows
var numFields = 0
while (input.hasNext) {
  val unsafeRow = input.next().asInstanceOf[UnsafeRow]
  numFields = unsafeRow.numFields()
  val rowKey = keyGenerator(unsafeRow)
  if (!rowKey.isNullAt(0)) {
val key = rowKey.getLong(0) // <<< Values in rowKey are Ints not 
Longs
map.append(key, unsafeRow)
  }
}
{code}



was (Author: robbinspg):
[~davies] as you are the author of this code can you comment on my findings?

So the issue here is that the keyGenerator returns an UnsafeRow containing Int 
values but the code below from LongHashedRelation.apply retrieves the key from 
this as a Long. The bytes in the row are

on Little Endian: 01 00 00 00 00 00 00 00 
on Big Endian:   00 00 00 01 00 00 00 00

By chance getInt and getLong  will both return "1" on Little Endian because the 
following 4 bytes happen to be 0, whereas on Big Endian getInt returns "1" but 
get Long will return "268435456"

{code}
val keyGenerator = UnsafeProjection.create(key)

// Create a mapping of key -> rows
var numFields = 0
while (input.hasNext) {
  val unsafeRow = input.next().asInstanceOf[UnsafeRow]
  numFields = unsafeRow.numFields()
  val rowKey = keyGenerator(unsafeRow)
  if (!rowKey.isNullAt(0)) {
val key = rowKey.getLong(0) // <<< Values in rowKey are Intsnot 
Longs
map.append(key, unsafeRow)
  }
}
{code}


> LongHashedRelation fails on Big Endian platform
> ---
>
> Key: SPARK-15154
> URL: https://issues.apache.org/jira/browse/SPARK-15154
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.0.0
>Reporter: Pete Robbins
>  Labels: big-endian
>
> NPE in 
> org.apache.spark.sql.execution.joins.HashedRelationSuite.LongToUnsafeRowMap
> Error Message
> java.lang.NullPointerException was thrown.
> Stacktrace
>   java.lang.NullPointerException
>   at 
> org.apache.spark.sql.execution.joins.HashedRelationSuite$$anonfun$3$$anonfun$apply$mcV$sp$1.apply$mcVI$sp(HashedRelationSuite.scala:121)
>   at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:160)
>   at 
> org.apache.spark.sql.execution.joins.HashedRelationSuite$$anonfun$3.apply$mcV$sp(HashedRelationSuite.scala:119)
>   at 
> org.apache.spark.sql.execution.joins.HashedRelationSuite$$anonfun$3.apply(HashedRelationSuite.scala:112)
>   at 
> org.apache.spark.sql.execution.joins.HashedRelationSuite$$anonfun$3.apply(HashedRelationSuite.scala:112)
>   at 
> org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
>   at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
>   at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
>   at org.scalatest.Transformer.apply(Transformer.scala:22)
>   at org.scalatest.Transformer.apply(Transformer.scala:20)
>   at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:166)
>   at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:57)
>   at 
> org.scalatest.FunSuiteLike$class.invokeWithFixture$1(FunSuiteLike.scala:163)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)
>   at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
>   at org.scalatest.FunSuiteLike$class.runTest(FunSuiteLike.scala:175)
>   at org.scalatest.FunSuite.runTest(FunSuite.scala:1555)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:208)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:208)
>   at 
> org.s

[jira] [Comment Edited] (SPARK-15154) LongHashedRelation fails on Big Endian platform

[Pete Robbins (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-15154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15273885#comment-15273885
 ] 

Pete Robbins edited comment on SPARK-15154 at 5/6/16 11:14 AM:
---

[~davies] as you are the author of this code can you comment on my findings?

So the issue here is that the keyGenerator returns an UnsafeRow containing Int 
values but the code below from LongHashedRelation.apply retrieves the key from 
this as a Long. The bytes in the row are

on Little Endian: 01 00 00 00 00 00 00 00 
on Big Endian:   00 00 00 01 00 00 00 00

By chance getInt and getLong  will both return "1" on Little Endian because the 
following 4 bytes happen to be 0, whereas on Big Endian getInt returns "1" but 
get Long will return "268435456"

{code}
val keyGenerator = UnsafeProjection.create(key)

// Create a mapping of key -> rows
var numFields = 0
while (input.hasNext) {
  val unsafeRow = input.next().asInstanceOf[UnsafeRow]
  numFields = unsafeRow.numFields()
  val rowKey = keyGenerator(unsafeRow)
  if (!rowKey.isNullAt(0)) {
val key = rowKey.getLong(0) // <<< Values in rowKey are Intsnot 
Longs
map.append(key, unsafeRow)
  }
}
{code}



was (Author: robbinspg):
[~davies] as you are the author of this code can you comment on my findings?

So the issue here is that the keyGenerator returns an UnsafeRow containing Int 
values but the code below from LongHashedRelation.apply retrieves the key from 
this as a Long. The bytes in the row are

on Little Endian: 01 00 00 00 00 00 00 00 
on Big Endian:   00 00 00 01 00 00 00 00

By chance getInt and getLong  will both return "1" on Little Endian whereas on 
Big Endian getInt returns "1" but get Long will return "268435456"

{code}
val keyGenerator = UnsafeProjection.create(key)

// Create a mapping of key -> rows
var numFields = 0
while (input.hasNext) {
  val unsafeRow = input.next().asInstanceOf[UnsafeRow]
  numFields = unsafeRow.numFields()
  val rowKey = keyGenerator(unsafeRow)
  if (!rowKey.isNullAt(0)) {
val key = rowKey.getLong(0) // <<< Values in rowKey are Intsnot 
Longs
map.append(key, unsafeRow)
  }
}
{code}


> LongHashedRelation fails on Big Endian platform
> ---
>
> Key: SPARK-15154
> URL: https://issues.apache.org/jira/browse/SPARK-15154
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.0.0
>Reporter: Pete Robbins
>  Labels: big-endian
>
> NPE in 
> org.apache.spark.sql.execution.joins.HashedRelationSuite.LongToUnsafeRowMap
> Error Message
> java.lang.NullPointerException was thrown.
> Stacktrace
>   java.lang.NullPointerException
>   at 
> org.apache.spark.sql.execution.joins.HashedRelationSuite$$anonfun$3$$anonfun$apply$mcV$sp$1.apply$mcVI$sp(HashedRelationSuite.scala:121)
>   at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:160)
>   at 
> org.apache.spark.sql.execution.joins.HashedRelationSuite$$anonfun$3.apply$mcV$sp(HashedRelationSuite.scala:119)
>   at 
> org.apache.spark.sql.execution.joins.HashedRelationSuite$$anonfun$3.apply(HashedRelationSuite.scala:112)
>   at 
> org.apache.spark.sql.execution.joins.HashedRelationSuite$$anonfun$3.apply(HashedRelationSuite.scala:112)
>   at 
> org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
>   at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
>   at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
>   at org.scalatest.Transformer.apply(Transformer.scala:22)
>   at org.scalatest.Transformer.apply(Transformer.scala:20)
>   at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:166)
>   at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:57)
>   at 
> org.scalatest.FunSuiteLike$class.invokeWithFixture$1(FunSuiteLike.scala:163)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)
>   at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
>   at org.scalatest.FunSuiteLike$class.runTest(FunSuiteLike.scala:175)
>   at org.scalatest.FunSuite.runTest(FunSuite.scala:1555)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:208)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:208)
>   at 
> org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:

[jira] [Comment Edited] (SPARK-15154) LongHashedRelation fails on Big Endian platform

[Pete Robbins (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-15154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15273897#comment-15273897
 ] 

Pete Robbins edited comment on SPARK-15154 at 5/6/16 11:13 AM:
---

Is this just a testcase issue where in HashedRelationSuite


{code}
val key = Seq(BoundReference(0, IntegerType, false))
{code}

should be

{code}
val key = Seq(BoundReference(0, LongType, false))
{code}


Ans: No, still fails with that change.


was (Author: robbinspg):
Is this just a testcase issue where in HashedRelationSuite


{code}
val key = Seq(BoundReference(0, IntegerType, false))
{code}

should be

{code}
val key = Seq(BoundReference(0, LongType, false))
{code}

> LongHashedRelation fails on Big Endian platform
> ---
>
> Key: SPARK-15154
> URL: https://issues.apache.org/jira/browse/SPARK-15154
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.0.0
>Reporter: Pete Robbins
>  Labels: big-endian
>
> NPE in 
> org.apache.spark.sql.execution.joins.HashedRelationSuite.LongToUnsafeRowMap
> Error Message
> java.lang.NullPointerException was thrown.
> Stacktrace
>   java.lang.NullPointerException
>   at 
> org.apache.spark.sql.execution.joins.HashedRelationSuite$$anonfun$3$$anonfun$apply$mcV$sp$1.apply$mcVI$sp(HashedRelationSuite.scala:121)
>   at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:160)
>   at 
> org.apache.spark.sql.execution.joins.HashedRelationSuite$$anonfun$3.apply$mcV$sp(HashedRelationSuite.scala:119)
>   at 
> org.apache.spark.sql.execution.joins.HashedRelationSuite$$anonfun$3.apply(HashedRelationSuite.scala:112)
>   at 
> org.apache.spark.sql.execution.joins.HashedRelationSuite$$anonfun$3.apply(HashedRelationSuite.scala:112)
>   at 
> org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
>   at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
>   at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
>   at org.scalatest.Transformer.apply(Transformer.scala:22)
>   at org.scalatest.Transformer.apply(Transformer.scala:20)
>   at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:166)
>   at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:57)
>   at 
> org.scalatest.FunSuiteLike$class.invokeWithFixture$1(FunSuiteLike.scala:163)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)
>   at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
>   at org.scalatest.FunSuiteLike$class.runTest(FunSuiteLike.scala:175)
>   at org.scalatest.FunSuite.runTest(FunSuite.scala:1555)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:208)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:208)
>   at 
> org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:413)
>   at 
> org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:401)
>   at scala.collection.immutable.List.foreach(List.scala:381)
>   at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
>   at 
> org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:396)
>   at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:483)
>   at org.scalatest.FunSuiteLike$class.runTests(FunSuiteLike.scala:208)
>   at org.scalatest.FunSuite.runTests(FunSuite.scala:1555)
>   at org.scalatest.Suite$class.run(Suite.scala:1424)
>   at 
> org.scalatest.FunSuite.org$scalatest$FunSuiteLike$$super$run(FunSuite.scala:1555)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:212)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:212)
>   at org.scalatest.SuperEngine.runImpl(Engine.scala:545)
>   at org.scalatest.FunSuiteLike$class.run(FunSuiteLike.scala:212)
>   at 
> org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:29)
>   at 
> org.scalatest.BeforeAndAfterAll$class.liftedTree1$1(BeforeAndAfterAll.scala:257)
>   at 
> org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:256)
>   at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:29)
>   at org.scalatest.Suite$class.callExecuteOnSuite$1(Suite.scala:1492)
>   at 
> org.scalatest.Suite$$anonfun$runNestedSuites$1.apply(Suite.sc

[jira] [Commented] (SPARK-15154) LongHashedRelation fails on Big Endian platform

[Pete Robbins (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-15154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15273897#comment-15273897
 ] 

Pete Robbins commented on SPARK-15154:
--

Is this just a testcase issue where in HashedRelationSuite


{code}
val key = Seq(BoundReference(0, IntegerType, false))
{code}

should be

{code}
val key = Seq(BoundReference(0, LongType, false))
{code}

> LongHashedRelation fails on Big Endian platform
> ---
>
> Key: SPARK-15154
> URL: https://issues.apache.org/jira/browse/SPARK-15154
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.0.0
>Reporter: Pete Robbins
>  Labels: big-endian
>
> NPE in 
> org.apache.spark.sql.execution.joins.HashedRelationSuite.LongToUnsafeRowMap
> Error Message
> java.lang.NullPointerException was thrown.
> Stacktrace
>   java.lang.NullPointerException
>   at 
> org.apache.spark.sql.execution.joins.HashedRelationSuite$$anonfun$3$$anonfun$apply$mcV$sp$1.apply$mcVI$sp(HashedRelationSuite.scala:121)
>   at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:160)
>   at 
> org.apache.spark.sql.execution.joins.HashedRelationSuite$$anonfun$3.apply$mcV$sp(HashedRelationSuite.scala:119)
>   at 
> org.apache.spark.sql.execution.joins.HashedRelationSuite$$anonfun$3.apply(HashedRelationSuite.scala:112)
>   at 
> org.apache.spark.sql.execution.joins.HashedRelationSuite$$anonfun$3.apply(HashedRelationSuite.scala:112)
>   at 
> org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
>   at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
>   at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
>   at org.scalatest.Transformer.apply(Transformer.scala:22)
>   at org.scalatest.Transformer.apply(Transformer.scala:20)
>   at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:166)
>   at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:57)
>   at 
> org.scalatest.FunSuiteLike$class.invokeWithFixture$1(FunSuiteLike.scala:163)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)
>   at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
>   at org.scalatest.FunSuiteLike$class.runTest(FunSuiteLike.scala:175)
>   at org.scalatest.FunSuite.runTest(FunSuite.scala:1555)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:208)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:208)
>   at 
> org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:413)
>   at 
> org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:401)
>   at scala.collection.immutable.List.foreach(List.scala:381)
>   at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
>   at 
> org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:396)
>   at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:483)
>   at org.scalatest.FunSuiteLike$class.runTests(FunSuiteLike.scala:208)
>   at org.scalatest.FunSuite.runTests(FunSuite.scala:1555)
>   at org.scalatest.Suite$class.run(Suite.scala:1424)
>   at 
> org.scalatest.FunSuite.org$scalatest$FunSuiteLike$$super$run(FunSuite.scala:1555)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:212)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:212)
>   at org.scalatest.SuperEngine.runImpl(Engine.scala:545)
>   at org.scalatest.FunSuiteLike$class.run(FunSuiteLike.scala:212)
>   at 
> org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:29)
>   at 
> org.scalatest.BeforeAndAfterAll$class.liftedTree1$1(BeforeAndAfterAll.scala:257)
>   at 
> org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:256)
>   at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:29)
>   at org.scalatest.Suite$class.callExecuteOnSuite$1(Suite.scala:1492)
>   at 
> org.scalatest.Suite$$anonfun$runNestedSuites$1.apply(Suite.scala:1528)
>   at 
> org.scalatest.Suite$$anonfun$runNestedSuites$1.apply(Suite.scala:1526)
>   at 
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
>   at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
>   at org.scalatest.Suite$class.runN

[jira] [Comment Edited] (SPARK-15154) LongHashedRelation fails on Big Endian platform

[Pete Robbins (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-15154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15273885#comment-15273885
 ] 

Pete Robbins edited comment on SPARK-15154 at 5/6/16 10:28 AM:
---

[~davies] as you are the author of this code can you comment on my findings?

So the issue here is that the keyGenerator returns an UnsafeRow containing Int 
values but the code below from LongHashedRelation.apply retrieves the key from 
this as a Long. The bytes in the row are

on Little Endian: 01 00 00 00 00 00 00 00 
on Big Endian:   00 00 00 01 00 00 00 00

By chance getInt and getLong  will both return "1" on Little Endian whereas on 
Big Endian getInt returns "1" but get Long will return "268435456"

{code}
val keyGenerator = UnsafeProjection.create(key)

// Create a mapping of key -> rows
var numFields = 0
while (input.hasNext) {
  val unsafeRow = input.next().asInstanceOf[UnsafeRow]
  numFields = unsafeRow.numFields()
  val rowKey = keyGenerator(unsafeRow)
  if (!rowKey.isNullAt(0)) {
val key = rowKey.getLong(0) // <<< Values in rowKey are Intsnot 
Longs
map.append(key, unsafeRow)
  }
}
{code}



was (Author: robbinspg):
[~davies] as you are the author of this code can you comment on my findings?

So the issue here is that the keyGenerator returns an UnsafeRow containing Int 
values but the code below from LongHashedRelation.apply retrieves the key from 
this as a Long. The bytes in the row are

on Little Endian: 01 00 00 00 00 00 00 00 
on Big Endian:   00 00 00 01 00 00 00 00

By chance getInt and getLong  will both return "1" on Little Endian whereas on 
Big Endian getInt returns "1" but get Long will return "268435456"

{quote}
val keyGenerator = UnsafeProjection.create(key)

// Create a mapping of key -> rows
var numFields = 0
while (input.hasNext) {
  val unsafeRow = input.next().asInstanceOf[UnsafeRow]
  numFields = unsafeRow.numFields()
  val rowKey = keyGenerator(unsafeRow)
  if (!rowKey.isNullAt(0)) {
val key = rowKey.getLong(0) // <<< Values in rowKey are Int not Long

map.append(key, unsafeRow)
  }
}
{quote}


> LongHashedRelation fails on Big Endian platform
> ---
>
> Key: SPARK-15154
> URL: https://issues.apache.org/jira/browse/SPARK-15154
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.0.0
>Reporter: Pete Robbins
>  Labels: big-endian
>
> NPE in 
> org.apache.spark.sql.execution.joins.HashedRelationSuite.LongToUnsafeRowMap
> Error Message
> java.lang.NullPointerException was thrown.
> Stacktrace
>   java.lang.NullPointerException
>   at 
> org.apache.spark.sql.execution.joins.HashedRelationSuite$$anonfun$3$$anonfun$apply$mcV$sp$1.apply$mcVI$sp(HashedRelationSuite.scala:121)
>   at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:160)
>   at 
> org.apache.spark.sql.execution.joins.HashedRelationSuite$$anonfun$3.apply$mcV$sp(HashedRelationSuite.scala:119)
>   at 
> org.apache.spark.sql.execution.joins.HashedRelationSuite$$anonfun$3.apply(HashedRelationSuite.scala:112)
>   at 
> org.apache.spark.sql.execution.joins.HashedRelationSuite$$anonfun$3.apply(HashedRelationSuite.scala:112)
>   at 
> org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
>   at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
>   at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
>   at org.scalatest.Transformer.apply(Transformer.scala:22)
>   at org.scalatest.Transformer.apply(Transformer.scala:20)
>   at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:166)
>   at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:57)
>   at 
> org.scalatest.FunSuiteLike$class.invokeWithFixture$1(FunSuiteLike.scala:163)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)
>   at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
>   at org.scalatest.FunSuiteLike$class.runTest(FunSuiteLike.scala:175)
>   at org.scalatest.FunSuite.runTest(FunSuite.scala:1555)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:208)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:208)
>   at 
> org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:413)

[jira] [Comment Edited] (SPARK-15154) LongHashedRelation fails on Big Endian platform

[Pete Robbins (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-15154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15273885#comment-15273885
 ] 

Pete Robbins edited comment on SPARK-15154 at 5/6/16 10:27 AM:
---

[~davies] as you are the author of this code can you comment on my findings?

So the issue here is that the keyGenerator returns an UnsafeRow containing Int 
values but the code below from LongHashedRelation.apply retrieves the key from 
this as a Long. The bytes in the row are

on Little Endian: 01 00 00 00 00 00 00 00 
on Big Endian:   00 00 00 01 00 00 00 00

By chance getInt and getLong  will both return "1" on Little Endian whereas on 
Big Endian getInt returns "1" but get Long will return "268435456"

{quote}
val keyGenerator = UnsafeProjection.create(key)

// Create a mapping of key -> rows
var numFields = 0
while (input.hasNext) {
  val unsafeRow = input.next().asInstanceOf[UnsafeRow]
  numFields = unsafeRow.numFields()
  val rowKey = keyGenerator(unsafeRow)
  if (!rowKey.isNullAt(0)) {
val key = rowKey.getLong(0) // <<< Values in rowKey are Int not Long

map.append(key, unsafeRow)
  }
}
{quote}



was (Author: robbinspg):
[~davies] as you are the author of this code can you comment on my findings?

So the issue here is that the keyGenerator returns an UnsafeRow containing Int 
values but the code below from LongHashedRelation.apply retrieves the key from 
this as a Long. The bytes in the row are

on Little Endian: 01 00 00 00 00 00 00 00 
on Big Endian:   00 00 00 01 00 00 00 00

By chance getInt and getLong  will both return "1" on Little Endian whereas on 
Big Endian getInt returns "1" but get Long will return "268435456"


```
val keyGenerator = UnsafeProjection.create(key)

// Create a mapping of key -> rows
var numFields = 0
while (input.hasNext) {
  val unsafeRow = input.next().asInstanceOf[UnsafeRow]
  numFields = unsafeRow.numFields()
  val rowKey = keyGenerator(unsafeRow)
  if (!rowKey.isNullAt(0)) {
val key = rowKey.getLong(0) // <<< Values in rowKey are Int not Long

map.append(key, unsafeRow)
  }
}
```


> LongHashedRelation fails on Big Endian platform
> ---
>
> Key: SPARK-15154
> URL: https://issues.apache.org/jira/browse/SPARK-15154
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.0.0
>Reporter: Pete Robbins
>  Labels: big-endian
>
> NPE in 
> org.apache.spark.sql.execution.joins.HashedRelationSuite.LongToUnsafeRowMap
> Error Message
> java.lang.NullPointerException was thrown.
> Stacktrace
>   java.lang.NullPointerException
>   at 
> org.apache.spark.sql.execution.joins.HashedRelationSuite$$anonfun$3$$anonfun$apply$mcV$sp$1.apply$mcVI$sp(HashedRelationSuite.scala:121)
>   at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:160)
>   at 
> org.apache.spark.sql.execution.joins.HashedRelationSuite$$anonfun$3.apply$mcV$sp(HashedRelationSuite.scala:119)
>   at 
> org.apache.spark.sql.execution.joins.HashedRelationSuite$$anonfun$3.apply(HashedRelationSuite.scala:112)
>   at 
> org.apache.spark.sql.execution.joins.HashedRelationSuite$$anonfun$3.apply(HashedRelationSuite.scala:112)
>   at 
> org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
>   at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
>   at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
>   at org.scalatest.Transformer.apply(Transformer.scala:22)
>   at org.scalatest.Transformer.apply(Transformer.scala:20)
>   at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:166)
>   at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:57)
>   at 
> org.scalatest.FunSuiteLike$class.invokeWithFixture$1(FunSuiteLike.scala:163)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)
>   at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
>   at org.scalatest.FunSuiteLike$class.runTest(FunSuiteLike.scala:175)
>   at org.scalatest.FunSuite.runTest(FunSuite.scala:1555)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:208)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:208)
>   at 
> org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:413)
>   at 
> org.scalatest.SuperEngin

[jira] [Comment Edited] (SPARK-15154) LongHashedRelation fails on Big Endian platform

[Pete Robbins (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-15154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15273885#comment-15273885
 ] 

Pete Robbins edited comment on SPARK-15154 at 5/6/16 10:24 AM:
---

[~davies] as you are the author of this code can you comment on my findings?

So the issue here is that the keyGenerator returns an UnsafeRow containing Int 
values but the code below from LongHashedRelation.apply retrieves the key from 
this as a Long. The bytes in the row are

on Little Endian: 01 00 00 00 00 00 00 00 
on Big Endian:   00 00 00 01 00 00 00 00

By chance getInt and getLong  will both return "1" on Little Endian whereas on 
Big Endian getInt returns "1" but get Long will return "268435456"



val keyGenerator = UnsafeProjection.create(key)

// Create a mapping of key -> rows
var numFields = 0
while (input.hasNext) {
  val unsafeRow = input.next().asInstanceOf[UnsafeRow]
  numFields = unsafeRow.numFields()
  val rowKey = keyGenerator(unsafeRow)
  if (!rowKey.isNullAt(0)) {
val key = rowKey.getLong(0) // <<< Values in rowKey are Int not Long

map.append(key, unsafeRow)
  }
}




was (Author: robbinspg):
[~davies] as you are the author of this code can you comment on my findings?

So the issue here is that the keyGenerator returns an UnsafeRow containing Int 
values but the code below from LongHashedRelation.apply retrieves the key from 
this as a Long. The bytes in the row are

on Little Endian: 01 00 00 00 00 00 00 00 
on Big Endian:   00 00 00 01 00 00 00 00

By chance getInt and getLong  will both return "1" on Little Endian whereas on 
Big Endian getInt returns "1" but get Long will return "268435456"



val keyGenerator = UnsafeProjection.create(key)

// Create a mapping of key -> rows
var numFields = 0
while (input.hasNext) {
  val unsafeRow = input.next().asInstanceOf[UnsafeRow]
  numFields = unsafeRow.numFields()
  val rowKey = keyGenerator(unsafeRow)
  if (!rowKey.isNullAt(0)) {
val key = rowKey.getLong(0) // <<< Values in rowKey are Int not Long
map.append(key, unsafeRow)
  }
}



> LongHashedRelation fails on Big Endian platform
> ---
>
> Key: SPARK-15154
> URL: https://issues.apache.org/jira/browse/SPARK-15154
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.0.0
>Reporter: Pete Robbins
>  Labels: big-endian
>
> NPE in 
> org.apache.spark.sql.execution.joins.HashedRelationSuite.LongToUnsafeRowMap
> Error Message
> java.lang.NullPointerException was thrown.
> Stacktrace
>   java.lang.NullPointerException
>   at 
> org.apache.spark.sql.execution.joins.HashedRelationSuite$$anonfun$3$$anonfun$apply$mcV$sp$1.apply$mcVI$sp(HashedRelationSuite.scala:121)
>   at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:160)
>   at 
> org.apache.spark.sql.execution.joins.HashedRelationSuite$$anonfun$3.apply$mcV$sp(HashedRelationSuite.scala:119)
>   at 
> org.apache.spark.sql.execution.joins.HashedRelationSuite$$anonfun$3.apply(HashedRelationSuite.scala:112)
>   at 
> org.apache.spark.sql.execution.joins.HashedRelationSuite$$anonfun$3.apply(HashedRelationSuite.scala:112)
>   at 
> org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
>   at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
>   at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
>   at org.scalatest.Transformer.apply(Transformer.scala:22)
>   at org.scalatest.Transformer.apply(Transformer.scala:20)
>   at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:166)
>   at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:57)
>   at 
> org.scalatest.FunSuiteLike$class.invokeWithFixture$1(FunSuiteLike.scala:163)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)
>   at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
>   at org.scalatest.FunSuiteLike$class.runTest(FunSuiteLike.scala:175)
>   at org.scalatest.FunSuite.runTest(FunSuite.scala:1555)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:208)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:208)
>   at 
> org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:413)
>   at 
> org.scalatest.SuperEngine$$anonfun$traverseSub

[jira] [Comment Edited] (SPARK-15154) LongHashedRelation fails on Big Endian platform

[Pete Robbins (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-15154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15273885#comment-15273885
 ] 

Pete Robbins edited comment on SPARK-15154 at 5/6/16 10:25 AM:
---

[~davies] as you are the author of this code can you comment on my findings?

So the issue here is that the keyGenerator returns an UnsafeRow containing Int 
values but the code below from LongHashedRelation.apply retrieves the key from 
this as a Long. The bytes in the row are

on Little Endian: 01 00 00 00 00 00 00 00 
on Big Endian:   00 00 00 01 00 00 00 00

By chance getInt and getLong  will both return "1" on Little Endian whereas on 
Big Endian getInt returns "1" but get Long will return "268435456"


```
val keyGenerator = UnsafeProjection.create(key)

// Create a mapping of key -> rows
var numFields = 0
while (input.hasNext) {
  val unsafeRow = input.next().asInstanceOf[UnsafeRow]
  numFields = unsafeRow.numFields()
  val rowKey = keyGenerator(unsafeRow)
  if (!rowKey.isNullAt(0)) {
val key = rowKey.getLong(0) // <<< Values in rowKey are Int not Long

map.append(key, unsafeRow)
  }
}
```



was (Author: robbinspg):
[~davies] as you are the author of this code can you comment on my findings?

So the issue here is that the keyGenerator returns an UnsafeRow containing Int 
values but the code below from LongHashedRelation.apply retrieves the key from 
this as a Long. The bytes in the row are

on Little Endian: 01 00 00 00 00 00 00 00 
on Big Endian:   00 00 00 01 00 00 00 00

By chance getInt and getLong  will both return "1" on Little Endian whereas on 
Big Endian getInt returns "1" but get Long will return "268435456"



val keyGenerator = UnsafeProjection.create(key)

// Create a mapping of key -> rows
var numFields = 0
while (input.hasNext) {
  val unsafeRow = input.next().asInstanceOf[UnsafeRow]
  numFields = unsafeRow.numFields()
  val rowKey = keyGenerator(unsafeRow)
  if (!rowKey.isNullAt(0)) {
val key = rowKey.getLong(0) // <<< Values in rowKey are Int not Long

map.append(key, unsafeRow)
  }
}



> LongHashedRelation fails on Big Endian platform
> ---
>
> Key: SPARK-15154
> URL: https://issues.apache.org/jira/browse/SPARK-15154
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.0.0
>Reporter: Pete Robbins
>  Labels: big-endian
>
> NPE in 
> org.apache.spark.sql.execution.joins.HashedRelationSuite.LongToUnsafeRowMap
> Error Message
> java.lang.NullPointerException was thrown.
> Stacktrace
>   java.lang.NullPointerException
>   at 
> org.apache.spark.sql.execution.joins.HashedRelationSuite$$anonfun$3$$anonfun$apply$mcV$sp$1.apply$mcVI$sp(HashedRelationSuite.scala:121)
>   at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:160)
>   at 
> org.apache.spark.sql.execution.joins.HashedRelationSuite$$anonfun$3.apply$mcV$sp(HashedRelationSuite.scala:119)
>   at 
> org.apache.spark.sql.execution.joins.HashedRelationSuite$$anonfun$3.apply(HashedRelationSuite.scala:112)
>   at 
> org.apache.spark.sql.execution.joins.HashedRelationSuite$$anonfun$3.apply(HashedRelationSuite.scala:112)
>   at 
> org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
>   at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
>   at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
>   at org.scalatest.Transformer.apply(Transformer.scala:22)
>   at org.scalatest.Transformer.apply(Transformer.scala:20)
>   at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:166)
>   at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:57)
>   at 
> org.scalatest.FunSuiteLike$class.invokeWithFixture$1(FunSuiteLike.scala:163)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)
>   at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
>   at org.scalatest.FunSuiteLike$class.runTest(FunSuiteLike.scala:175)
>   at org.scalatest.FunSuite.runTest(FunSuite.scala:1555)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:208)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:208)
>   at 
> org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:413)
>   at 
> org.scalatest.SuperEngine$$anonfun

[jira] [Commented] (SPARK-15154) LongHashedRelation fails on Big Endian platform

[Pete Robbins (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-15154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15273885#comment-15273885
 ] 

Pete Robbins commented on SPARK-15154:
--

[~davies] as you are the author of this code can you comment on my findings?

So the issue here is that the keyGenerator returns an UnsafeRow containing Int 
values but the code below from LongHashedRelation.apply retrieves the key from 
this as a Long. The bytes in the row are

on Little Endian: 01 00 00 00 00 00 00 00 
on Big Endian:   00 00 00 01 00 00 00 00

By chance getInt and getLong  will both return "1" on Little Endian whereas on 
Big Endian getInt returns "1" but get Long will return "268435456"



val keyGenerator = UnsafeProjection.create(key)

// Create a mapping of key -> rows
var numFields = 0
while (input.hasNext) {
  val unsafeRow = input.next().asInstanceOf[UnsafeRow]
  numFields = unsafeRow.numFields()
  val rowKey = keyGenerator(unsafeRow)
  if (!rowKey.isNullAt(0)) {
val key = rowKey.getLong(0) // <<< Values in rowKey are Int not Long
map.append(key, unsafeRow)
  }
}



> LongHashedRelation fails on Big Endian platform
> ---
>
> Key: SPARK-15154
> URL: https://issues.apache.org/jira/browse/SPARK-15154
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.0.0
>Reporter: Pete Robbins
>  Labels: big-endian
>
> NPE in 
> org.apache.spark.sql.execution.joins.HashedRelationSuite.LongToUnsafeRowMap
> Error Message
> java.lang.NullPointerException was thrown.
> Stacktrace
>   java.lang.NullPointerException
>   at 
> org.apache.spark.sql.execution.joins.HashedRelationSuite$$anonfun$3$$anonfun$apply$mcV$sp$1.apply$mcVI$sp(HashedRelationSuite.scala:121)
>   at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:160)
>   at 
> org.apache.spark.sql.execution.joins.HashedRelationSuite$$anonfun$3.apply$mcV$sp(HashedRelationSuite.scala:119)
>   at 
> org.apache.spark.sql.execution.joins.HashedRelationSuite$$anonfun$3.apply(HashedRelationSuite.scala:112)
>   at 
> org.apache.spark.sql.execution.joins.HashedRelationSuite$$anonfun$3.apply(HashedRelationSuite.scala:112)
>   at 
> org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
>   at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
>   at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
>   at org.scalatest.Transformer.apply(Transformer.scala:22)
>   at org.scalatest.Transformer.apply(Transformer.scala:20)
>   at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:166)
>   at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:57)
>   at 
> org.scalatest.FunSuiteLike$class.invokeWithFixture$1(FunSuiteLike.scala:163)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)
>   at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
>   at org.scalatest.FunSuiteLike$class.runTest(FunSuiteLike.scala:175)
>   at org.scalatest.FunSuite.runTest(FunSuite.scala:1555)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:208)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:208)
>   at 
> org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:413)
>   at 
> org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:401)
>   at scala.collection.immutable.List.foreach(List.scala:381)
>   at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
>   at 
> org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:396)
>   at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:483)
>   at org.scalatest.FunSuiteLike$class.runTests(FunSuiteLike.scala:208)
>   at org.scalatest.FunSuite.runTests(FunSuite.scala:1555)
>   at org.scalatest.Suite$class.run(Suite.scala:1424)
>   at 
> org.scalatest.FunSuite.org$scalatest$FunSuiteLike$$super$run(FunSuite.scala:1555)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:212)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:212)
>   at org.scalatest.SuperEngine.runImpl(Engine.scala:545)
>   at org.scalatest.FunSuiteLike$class.run(FunSuiteLike.scala:212)
>   at 
> org.apache.spark.SparkFu

[jira] [Updated] (SPARK-15154) LongHashedRelation fails on Big Endian platform

[Pete Robbins (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-15154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pete Robbins updated SPARK-15154:
-
Summary: LongHashedRelation fails on Big Endian platform  (was: 
HashedRelation fails on Big Endian platform)

> LongHashedRelation fails on Big Endian platform
> ---
>
> Key: SPARK-15154
> URL: https://issues.apache.org/jira/browse/SPARK-15154
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.0.0
>Reporter: Pete Robbins
>  Labels: big-endian
>
> NPE in 
> org.apache.spark.sql.execution.joins.HashedRelationSuite.LongToUnsafeRowMap
> Error Message
> java.lang.NullPointerException was thrown.
> Stacktrace
>   java.lang.NullPointerException
>   at 
> org.apache.spark.sql.execution.joins.HashedRelationSuite$$anonfun$3$$anonfun$apply$mcV$sp$1.apply$mcVI$sp(HashedRelationSuite.scala:121)
>   at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:160)
>   at 
> org.apache.spark.sql.execution.joins.HashedRelationSuite$$anonfun$3.apply$mcV$sp(HashedRelationSuite.scala:119)
>   at 
> org.apache.spark.sql.execution.joins.HashedRelationSuite$$anonfun$3.apply(HashedRelationSuite.scala:112)
>   at 
> org.apache.spark.sql.execution.joins.HashedRelationSuite$$anonfun$3.apply(HashedRelationSuite.scala:112)
>   at 
> org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
>   at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
>   at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
>   at org.scalatest.Transformer.apply(Transformer.scala:22)
>   at org.scalatest.Transformer.apply(Transformer.scala:20)
>   at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:166)
>   at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:57)
>   at 
> org.scalatest.FunSuiteLike$class.invokeWithFixture$1(FunSuiteLike.scala:163)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)
>   at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
>   at org.scalatest.FunSuiteLike$class.runTest(FunSuiteLike.scala:175)
>   at org.scalatest.FunSuite.runTest(FunSuite.scala:1555)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:208)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:208)
>   at 
> org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:413)
>   at 
> org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:401)
>   at scala.collection.immutable.List.foreach(List.scala:381)
>   at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
>   at 
> org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:396)
>   at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:483)
>   at org.scalatest.FunSuiteLike$class.runTests(FunSuiteLike.scala:208)
>   at org.scalatest.FunSuite.runTests(FunSuite.scala:1555)
>   at org.scalatest.Suite$class.run(Suite.scala:1424)
>   at 
> org.scalatest.FunSuite.org$scalatest$FunSuiteLike$$super$run(FunSuite.scala:1555)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:212)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:212)
>   at org.scalatest.SuperEngine.runImpl(Engine.scala:545)
>   at org.scalatest.FunSuiteLike$class.run(FunSuiteLike.scala:212)
>   at 
> org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:29)
>   at 
> org.scalatest.BeforeAndAfterAll$class.liftedTree1$1(BeforeAndAfterAll.scala:257)
>   at 
> org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:256)
>   at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:29)
>   at org.scalatest.Suite$class.callExecuteOnSuite$1(Suite.scala:1492)
>   at 
> org.scalatest.Suite$$anonfun$runNestedSuites$1.apply(Suite.scala:1528)
>   at 
> org.scalatest.Suite$$anonfun$runNestedSuites$1.apply(Suite.scala:1526)
>   at 
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
>   at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
>   at org.scalatest.Suite$class.runNestedSuites(Suite.scala:1526)
>   at 
> org.scalatest.tools.DiscoverySuite.runNestedSuites(DiscoverySuite.scala:29)
>   at org.scalat

[jira] [Updated] (SPARK-15154) HashedRelation fails on Big Endian platform

[Pete Robbins (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-15154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pete Robbins updated SPARK-15154:
-
Labels: big-endian  (was: )

> HashedRelation fails on Big Endian platform
> ---
>
> Key: SPARK-15154
> URL: https://issues.apache.org/jira/browse/SPARK-15154
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.0.0
>Reporter: Pete Robbins
>  Labels: big-endian
>
> NPE in 
> org.apache.spark.sql.execution.joins.HashedRelationSuite.LongToUnsafeRowMap
> Error Message
> java.lang.NullPointerException was thrown.
> Stacktrace
>   java.lang.NullPointerException
>   at 
> org.apache.spark.sql.execution.joins.HashedRelationSuite$$anonfun$3$$anonfun$apply$mcV$sp$1.apply$mcVI$sp(HashedRelationSuite.scala:121)
>   at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:160)
>   at 
> org.apache.spark.sql.execution.joins.HashedRelationSuite$$anonfun$3.apply$mcV$sp(HashedRelationSuite.scala:119)
>   at 
> org.apache.spark.sql.execution.joins.HashedRelationSuite$$anonfun$3.apply(HashedRelationSuite.scala:112)
>   at 
> org.apache.spark.sql.execution.joins.HashedRelationSuite$$anonfun$3.apply(HashedRelationSuite.scala:112)
>   at 
> org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
>   at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
>   at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
>   at org.scalatest.Transformer.apply(Transformer.scala:22)
>   at org.scalatest.Transformer.apply(Transformer.scala:20)
>   at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:166)
>   at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:57)
>   at 
> org.scalatest.FunSuiteLike$class.invokeWithFixture$1(FunSuiteLike.scala:163)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)
>   at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
>   at org.scalatest.FunSuiteLike$class.runTest(FunSuiteLike.scala:175)
>   at org.scalatest.FunSuite.runTest(FunSuite.scala:1555)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:208)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:208)
>   at 
> org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:413)
>   at 
> org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:401)
>   at scala.collection.immutable.List.foreach(List.scala:381)
>   at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
>   at 
> org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:396)
>   at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:483)
>   at org.scalatest.FunSuiteLike$class.runTests(FunSuiteLike.scala:208)
>   at org.scalatest.FunSuite.runTests(FunSuite.scala:1555)
>   at org.scalatest.Suite$class.run(Suite.scala:1424)
>   at 
> org.scalatest.FunSuite.org$scalatest$FunSuiteLike$$super$run(FunSuite.scala:1555)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:212)
>   at 
> org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:212)
>   at org.scalatest.SuperEngine.runImpl(Engine.scala:545)
>   at org.scalatest.FunSuiteLike$class.run(FunSuiteLike.scala:212)
>   at 
> org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:29)
>   at 
> org.scalatest.BeforeAndAfterAll$class.liftedTree1$1(BeforeAndAfterAll.scala:257)
>   at 
> org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:256)
>   at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:29)
>   at org.scalatest.Suite$class.callExecuteOnSuite$1(Suite.scala:1492)
>   at 
> org.scalatest.Suite$$anonfun$runNestedSuites$1.apply(Suite.scala:1528)
>   at 
> org.scalatest.Suite$$anonfun$runNestedSuites$1.apply(Suite.scala:1526)
>   at 
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
>   at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
>   at org.scalatest.Suite$class.runNestedSuites(Suite.scala:1526)
>   at 
> org.scalatest.tools.DiscoverySuite.runNestedSuites(DiscoverySuite.scala:29)
>   at org.scalatest.Suite$class.run(Suite.scala:1421)
>   at org.scalatest.too

[jira] [Created] (SPARK-15154) HashedRelation fails on Big Endian platform

[Pete Robbins (JIRA)]

Pete Robbins created SPARK-15154:


 Summary: HashedRelation fails on Big Endian platform
 Key: SPARK-15154
 URL: https://issues.apache.org/jira/browse/SPARK-15154
 Project: Spark
  Issue Type: Bug
  Components: SQL
Affects Versions: 2.0.0
Reporter: Pete Robbins


NPE in 
org.apache.spark.sql.execution.joins.HashedRelationSuite.LongToUnsafeRowMap

Error Message

java.lang.NullPointerException was thrown.

Stacktrace

  java.lang.NullPointerException
  at 
org.apache.spark.sql.execution.joins.HashedRelationSuite$$anonfun$3$$anonfun$apply$mcV$sp$1.apply$mcVI$sp(HashedRelationSuite.scala:121)
  at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:160)
  at 
org.apache.spark.sql.execution.joins.HashedRelationSuite$$anonfun$3.apply$mcV$sp(HashedRelationSuite.scala:119)
  at 
org.apache.spark.sql.execution.joins.HashedRelationSuite$$anonfun$3.apply(HashedRelationSuite.scala:112)
  at 
org.apache.spark.sql.execution.joins.HashedRelationSuite$$anonfun$3.apply(HashedRelationSuite.scala:112)
  at 
org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
  at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
  at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
  at org.scalatest.Transformer.apply(Transformer.scala:22)
  at org.scalatest.Transformer.apply(Transformer.scala:20)
  at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:166)
  at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:57)
  at 
org.scalatest.FunSuiteLike$class.invokeWithFixture$1(FunSuiteLike.scala:163)
  at 
org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)
  at 
org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)
  at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
  at org.scalatest.FunSuiteLike$class.runTest(FunSuiteLike.scala:175)
  at org.scalatest.FunSuite.runTest(FunSuite.scala:1555)
  at 
org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:208)
  at 
org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:208)
  at 
org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:413)
  at 
org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:401)
  at scala.collection.immutable.List.foreach(List.scala:381)
  at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
  at 
org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:396)
  at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:483)
  at org.scalatest.FunSuiteLike$class.runTests(FunSuiteLike.scala:208)
  at org.scalatest.FunSuite.runTests(FunSuite.scala:1555)
  at org.scalatest.Suite$class.run(Suite.scala:1424)
  at 
org.scalatest.FunSuite.org$scalatest$FunSuiteLike$$super$run(FunSuite.scala:1555)
  at org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:212)
  at org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:212)
  at org.scalatest.SuperEngine.runImpl(Engine.scala:545)
  at org.scalatest.FunSuiteLike$class.run(FunSuiteLike.scala:212)
  at 
org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:29)
  at 
org.scalatest.BeforeAndAfterAll$class.liftedTree1$1(BeforeAndAfterAll.scala:257)
  at org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:256)
  at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:29)
  at org.scalatest.Suite$class.callExecuteOnSuite$1(Suite.scala:1492)
  at org.scalatest.Suite$$anonfun$runNestedSuites$1.apply(Suite.scala:1528)
  at org.scalatest.Suite$$anonfun$runNestedSuites$1.apply(Suite.scala:1526)
  at 
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
  at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
  at org.scalatest.Suite$class.runNestedSuites(Suite.scala:1526)
  at 
org.scalatest.tools.DiscoverySuite.runNestedSuites(DiscoverySuite.scala:29)
  at org.scalatest.Suite$class.run(Suite.scala:1421)
  at org.scalatest.tools.DiscoverySuite.run(DiscoverySuite.scala:29)
  at org.scalatest.tools.SuiteRunner.run(SuiteRunner.scala:55)
  at 
org.scalatest.tools.Runner$$anonfun$doRunRunRunDaDoRunRun$3.apply(Runner.scala:2563)
  at 
org.scalatest.tools.Runner$$anonfun$doRunRunRunDaDoRunRun$3.apply(Runner.scala:2557)
  at scala.collection.immutable.List.foreach(List.scala:381)
  at org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:2557)
  at 
org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:1044)
  at 
org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:1043

[jira] [Commented] (SPARK-15070) Data corruption when using Dataset.groupBy[K : Encoder](func: T => K) when data loaded from JSON file.

[Pete Robbins (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-15070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15268500#comment-15268500
 ] 

Pete Robbins commented on SPARK-15070:
--

could this be related to https://issues.apache.org/jira/browse/SPARK-12555 ?

> Data corruption when using Dataset.groupBy[K : Encoder](func: T => K) when 
> data loaded from JSON file.
> --
>
> Key: SPARK-15070
> URL: https://issues.apache.org/jira/browse/SPARK-15070
> Project: Spark
>  Issue Type: Bug
>  Components: Input/Output, SQL
>Affects Versions: 1.6.1
> Environment: produced on Mac OS X 10.11.4 in local mode
>Reporter: Eric Wasserman
>
> full running case at: https://github.com/ewasserman/spark-bug.git
> Bug.scala
> ==
> package bug
> import org.apache.spark.sql.functions._
> import org.apache.spark.sql.SQLContext
> import org.apache.spark.{SparkContext, SparkConf}
> case class BugRecord(m: String, elapsed_time: java.lang.Double)
> object Bug {
>   def main(args: Array[String]): Unit = {
> val c = new SparkConf().setMaster("local[2]").setAppName("BugTest")
> val sc = new SparkContext(c)
> val sqlc = new SQLContext(sc)
> import sqlc.implicits._
> val logs = sqlc.read.json("bug-data.json").as[BugRecord]
> logs.groupBy(r => "FOO").agg(avg($"elapsed_time").as[Double]).show(20, 
> truncate = false)
> 
> sc.stop()
>   }
> }
> bug-data.json
> ==
> {"m":"POST","elapsed_time":0.123456789012345678,"source_time":"abcdefghijk"}
> -
> Expected Output:
> +---+---+
> |_1 |_2 |
> +---+---+
> |FOO |0.12345678901234568|
> +---+---+
> Observed Output:
> +---+---+
> |_1 |_2 |
> +---+---+
> |POSTabc|0.12345726584950388|
> +---+---+
> The grouping key has been corrupted (it is *not* the product of the groupBy 
> function) and is a combination of bytes from the actual key column and an 
> extra attribute in the JSON not present in the case class. The aggregated 
> value is also corrupted.
> NOTE:
> The problem does not manifest when using an alternate form of groupBy:
> logs.groupBy($"m").agg(avg($"elapsed_time").as[Double])
> The corrupted key problem does not manifest when there is not an additional 
> field in the JSON. Ie. if the data file is this:
> {"m":"POST","elapsed_time":0.123456789012345678}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-13552) Incorrect data for Long.minValue in SQLQuerySuite on IBM Java

[Pete Robbins (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-13552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15268232#comment-15268232
 ] 

Pete Robbins commented on SPARK-13552:
--

[~aroberts] This Jira can be closed as this is not a Spark issue

> Incorrect data for Long.minValue in SQLQuerySuite on IBM Java
> -
>
> Key: SPARK-13552
> URL: https://issues.apache.org/jira/browse/SPARK-13552
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.0.0
> Environment: IBM Java only, all platforms
>Reporter: Adam Roberts
>Priority: Minor
> Attachments: DefectBadMinValueLongResized.jpg
>
>
> The Long.minValue test fails on IBM Java 8, we get the following incorrect 
> answer with the slightly simplified test case:
> {code:SQL}
> val tester = sql(s"SELECT ${Long.MinValue} FROM testData")
> {code}
> result is
> _-9,223,372,041,149,743,104_ instead of _-9,223,372,036,854,775,808_ (there's 
> only one bit difference if we convert to binary representation).
> Here's the full test output:
> {code}
> Results do not match for query:
> == Parsed Logical Plan ==
> 'GlobalLimit 1
> +- 'LocalLimit 1
>+- 'Sort ['key ASC], true
>   +- 'Project [unresolvedalias(-9223372036854775808, None)]
>  +- 'UnresolvedRelation `testData`, None
> == Analyzed Logical Plan ==
> (-9223372036854775808): decimal(19,0)
> GlobalLimit 1
> +- LocalLimit 1
>+- Project [(-9223372036854775808)#4391]
>   +- Sort [key#101 ASC], true
>  +- Project [-9223372036854775808 AS 
> (-9223372036854775808)#4391,key#101]
> +- SubqueryAlias testData
>+- LogicalRDD [key#101,value#102], MapPartitionsRDD[3] at 
> beforeAll at BeforeAndAfterAll.scala:187
> == Optimized Logical Plan ==
> GlobalLimit 1
> +- LocalLimit 1
>+- Project [(-9223372036854775808)#4391]
>   +- Sort [key#101 ASC], true
>  +- Project [-9223372036854775808 AS 
> (-9223372036854775808)#4391,key#101]
> +- LogicalRDD [key#101,value#102], MapPartitionsRDD[3] at 
> beforeAll at BeforeAndAfterAll.scala:187
> == Physical Plan ==
> TakeOrderedAndProject(limit=1, orderBy=[key#101 ASC], 
> output=[(-9223372036854775808)#4391])
> +- WholeStageCodegen
>:  +- Project [-9223372036854775808 AS (-9223372036854775808)#4391,key#101]
>: +- INPUT
>+- Scan ExistingRDD[key#101,value#102]
> == Results ==
> == Results ==
> !== Correct Answer - 1 ==   == Spark Answer - 1 ==
> ![-9223372036854775808] [-9223372041149743104]
> {code}
> Debugging in Intellij shows the query seems to be parsed OK and we eventually 
> have a schema with the correct data in the struct field but the BigDecimal's 
> BigInteger is incorrect when we have a GenericRowWithSchema.
> I've identified that the problem started when SPARK-12575 was implemented and 
> suspect the following paragraph is important:
> "Hive and the SQL Parser treat decimal literals differently. Hive will turn 
> any decimal into a Double whereas the SQL Parser would convert a 
> non-scientific decimal into a BigDecimal, and would turn a scientific decimal 
> into a Double. We follow Hive's behavior here. The new parser supports a big 
> decimal literal, for instance: 81923801.42BD, which can be used when a big 
> decimal is needed."
> Done, both "value" and "row" return the correct result for both Java 
> implementations: -9223372036854775808
> FWIW, I know the first time we can see the incorrect row values is in the 
> {code}withCallback[T]{code} method in DataFrame.scala, the specific line of 
> code is
> {code}
> val result = action(df)
> {code}
> Stepping into this doesn't clearly indicate how the resulting rows are being 
> produced though (could be that I'm debugging with the wrong thread in 
> Intellij - the first time I see a value for "result" is when it's too late - 
> when we're seeing the incorrect values).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Re: [ANNOUNCE] Spark branch-2.0

[Pete Robbins]

https://issues.apache.org/jira/browse/SPARK-13745

is really a defect and a blocker unless it is the decision to drop support
for Big Endian platforms. The PR has been reviewed and tested and I
strongly believe this needs to be targeted for 2.0.

On Mon, May 2, 2016 at 12:00 AM Reynold Xin  wrote:

> Hi devs,
>
> Three weeks ago I mentioned on the dev list creating branch-2.0
> (effectively "feature freeze") in 2 - 3 weeks. I've just created Spark's
> branch-2.0 to form the basis of the 2.0 release. We have closed ~ 1700
> issues. That's huge progress, and we should celebrate that.
>
> Compared with past releases when we cut the release branch, we have way
> fewer open issues. In the past we usually have 200 - 400 open issues when
> we cut the release branch. As of today we have less than 100 open issues
> for 2.0.0, and among these 14 critical and 2 blocker (Jersey dependency
> upgrade and some remaining issues in separating out local linear algebra
> library).
>
> What does this mean for committers?
>
> 0. For patches that should go into Spark 2.0.0, make sure you also merge
> them into not just master, but also branch-2.0.
>
> 1. In the next couple of days, sheppard some of the more important,
> straggler pull requests in.
>
> 2. Switch the focus from new feature development to bug fixes, stability
> improvements, finalizing API tweaks, and documentation.
>
> 3. Experimental features (e.g. R, structured streaming) can continue to be
> developed, provided that the changes don't impact the non-experimental
> features.
>
> 4. We should become increasingly conservative as time goes on, even for
> experimental features.
>
> 5. Please un-target or re-target issues if they don't make sense for 2.0.
> We should burn # issues down to ~ 0 by the time we have a release candidate.
>
> 7. If possible, reach out to users and start testing branch-2.0 to find
> bugs. The more testing we can do on real workloads before the release, the
> less bugs we will find in the actual Spark 2.0 release.
>
>
>
>
>

[jira] [Commented] (SPARK-13552) Incorrect data for Long.minValue in SQLQuerySuite on IBM Java

[Pete Robbins (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-13552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15262482#comment-15262482
 ] 

Pete Robbins commented on SPARK-13552:
--

This is looking like an issue with the IBM implementation of 
java.math.BigInteger. I'm still investigating and we can close this jira if my 
theory is correct.



> Incorrect data for Long.minValue in SQLQuerySuite on IBM Java
> -
>
> Key: SPARK-13552
> URL: https://issues.apache.org/jira/browse/SPARK-13552
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.0.0
> Environment: IBM Java only, all platforms
>Reporter: Adam Roberts
>Priority: Minor
> Attachments: DefectBadMinValueLongResized.jpg
>
>
> The Long.minValue test fails on IBM Java 8, we get the following incorrect 
> answer with the slightly simplified test case:
> {code:SQL}
> val tester = sql(s"SELECT ${Long.MinValue} FROM testData")
> {code}
> result is
> _-9,223,372,041,149,743,104_ instead of _-9,223,372,036,854,775,808_ (there's 
> only one bit difference if we convert to binary representation).
> Here's the full test output:
> {code}
> Results do not match for query:
> == Parsed Logical Plan ==
> 'GlobalLimit 1
> +- 'LocalLimit 1
>+- 'Sort ['key ASC], true
>   +- 'Project [unresolvedalias(-9223372036854775808, None)]
>  +- 'UnresolvedRelation `testData`, None
> == Analyzed Logical Plan ==
> (-9223372036854775808): decimal(19,0)
> GlobalLimit 1
> +- LocalLimit 1
>+- Project [(-9223372036854775808)#4391]
>   +- Sort [key#101 ASC], true
>  +- Project [-9223372036854775808 AS 
> (-9223372036854775808)#4391,key#101]
> +- SubqueryAlias testData
>+- LogicalRDD [key#101,value#102], MapPartitionsRDD[3] at 
> beforeAll at BeforeAndAfterAll.scala:187
> == Optimized Logical Plan ==
> GlobalLimit 1
> +- LocalLimit 1
>+- Project [(-9223372036854775808)#4391]
>   +- Sort [key#101 ASC], true
>  +- Project [-9223372036854775808 AS 
> (-9223372036854775808)#4391,key#101]
> +- LogicalRDD [key#101,value#102], MapPartitionsRDD[3] at 
> beforeAll at BeforeAndAfterAll.scala:187
> == Physical Plan ==
> TakeOrderedAndProject(limit=1, orderBy=[key#101 ASC], 
> output=[(-9223372036854775808)#4391])
> +- WholeStageCodegen
>:  +- Project [-9223372036854775808 AS (-9223372036854775808)#4391,key#101]
>: +- INPUT
>+- Scan ExistingRDD[key#101,value#102]
> == Results ==
> == Results ==
> !== Correct Answer - 1 ==   == Spark Answer - 1 ==
> ![-9223372036854775808] [-9223372041149743104]
> {code}
> Debugging in Intellij shows the query seems to be parsed OK and we eventually 
> have a schema with the correct data in the struct field but the BigDecimal's 
> BigInteger is incorrect when we have a GenericRowWithSchema.
> I've identified that the problem started when SPARK-12575 was implemented and 
> suspect the following paragraph is important:
> "Hive and the SQL Parser treat decimal literals differently. Hive will turn 
> any decimal into a Double whereas the SQL Parser would convert a 
> non-scientific decimal into a BigDecimal, and would turn a scientific decimal 
> into a Double. We follow Hive's behavior here. The new parser supports a big 
> decimal literal, for instance: 81923801.42BD, which can be used when a big 
> decimal is needed."
> Done, both "value" and "row" return the correct result for both Java 
> implementations: -9223372036854775808
> FWIW, I know the first time we can see the incorrect row values is in the 
> {code}withCallback[T]{code} method in DataFrame.scala, the specific line of 
> code is
> {code}
> val result = action(df)
> {code}
> Stepping into this doesn't clearly indicate how the resulting rows are being 
> produced though (could be that I'm debugging with the wrong thread in 
> Intellij - the first time I see a value for "result" is when it's too late - 
> when we're seeing the incorrect values).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-14848) DatasetSuite - Java encoder fails on Big Endian platforms

[Pete Robbins (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-14848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pete Robbins updated SPARK-14848:
-
Description: 
Since this PR https://github.com/apache/spark/pull/10703 for 
https://issues.apache.org/jira/browse/SPARK-12756 the "Java encoder" test in 
DatasetSuite has been failing on big endian platforms:

- Java encoder *** FAILED ***
  Array((JavaData(2),1), (JavaData(1),1)) did not equal List((JavaData(1),1), 
(JavaData(2),1)) (DatasetSuite.scala:478)

I note that the code for the "Kryo encoder" test was changed in the PR to use 
toSet and compare results against a Set to stop it failing in the same way 
whereas the Java encoder test still uses toSeq. 

Is it that the order is not guaranteed (but happens to be in the expected order 
on little endian) and this is a test issue?

  was:
Since this PR https://github.com/apache/spark/pull/10703 for 
https://issues.apache.org/jira/browse/SPARK-12756 the "Java encoder" test in 
DatasetSuite has been failing on big endian platforms:

- Java encoder *** FAILED ***
  Array((JavaData(2),1), (JavaData(1),1)) did not equal List((JavaData(1),1), 
(JavaData(2),1)) (DatasetSuite.scala:478)

I note that the code for the "Kyro encoder" test was changed in the PR to use 
toSet and compare results against a Set to stop it failing in the same way 
whereas the Java encoder test still uses toSeq. 

Is it that the order is not guaranteed (but happens to be in the expected order 
on little endian) and this is a test issue?


> DatasetSuite - Java encoder fails on Big Endian platforms
> -
>
> Key: SPARK-14848
> URL: https://issues.apache.org/jira/browse/SPARK-14848
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.0.0
>Reporter: Pete Robbins
>
> Since this PR https://github.com/apache/spark/pull/10703 for 
> https://issues.apache.org/jira/browse/SPARK-12756 the "Java encoder" test in 
> DatasetSuite has been failing on big endian platforms:
> - Java encoder *** FAILED ***
>   Array((JavaData(2),1), (JavaData(1),1)) did not equal List((JavaData(1),1), 
> (JavaData(2),1)) (DatasetSuite.scala:478)
> I note that the code for the "Kryo encoder" test was changed in the PR to use 
> toSet and compare results against a Set to stop it failing in the same way 
> whereas the Java encoder test still uses toSeq. 
> Is it that the order is not guaranteed (but happens to be in the expected 
> order on little endian) and this is a test issue?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-14848) DatasetSuite - Java encoder fails on Big Endian platforms

[Pete Robbins (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-14848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15253816#comment-15253816
 ] 

Pete Robbins commented on SPARK-14848:
--

changing the Java encoder test to use toSet and compare against Set(...) makes 
the test pass on both little endian and big endian platforms.

I will submit a PR.

[~cloud_fan] can you confirm my thoughts?

> DatasetSuite - Java encoder fails on Big Endian platforms
> -
>
> Key: SPARK-14848
> URL: https://issues.apache.org/jira/browse/SPARK-14848
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.0.0
>Reporter: Pete Robbins
>
> Since this PR https://github.com/apache/spark/pull/10703 for 
> https://issues.apache.org/jira/browse/SPARK-12756 the "Java encoder" test in 
> DatasetSuite has been failing on big endian platforms:
> - Java encoder *** FAILED ***
>   Array((JavaData(2),1), (JavaData(1),1)) did not equal List((JavaData(1),1), 
> (JavaData(2),1)) (DatasetSuite.scala:478)
> I note that the code for the "Kyro encoder" test was changed in the PR to use 
> toSet and compare results against a Set to stop it failing in the same way 
> whereas the Java encoder test still uses toSeq. 
> Is it that the order is not guaranteed (but happens to be in the expected 
> order on little endian) and this is a test issue?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Created] (SPARK-14848) DatasetSuite - Java encoder fails on Big Endian platforms

[Pete Robbins (JIRA)]

Pete Robbins created SPARK-14848:


 Summary: DatasetSuite - Java encoder fails on Big Endian platforms
 Key: SPARK-14848
 URL: https://issues.apache.org/jira/browse/SPARK-14848
 Project: Spark
  Issue Type: Bug
  Components: SQL
Affects Versions: 2.0.0
Reporter: Pete Robbins


Since this PR https://github.com/apache/spark/pull/10703 for 
https://issues.apache.org/jira/browse/SPARK-12756 the "Java encoder" test in 
DatasetSuite has been failing on big endian platforms:

- Java encoder *** FAILED ***
  Array((JavaData(2),1), (JavaData(1),1)) did not equal List((JavaData(1),1), 
(JavaData(2),1)) (DatasetSuite.scala:478)

I note that the code for the "Kyro encoder" test was changed in the PR to use 
toSet and compare results against a Set to stop it failing in the same way 
whereas the Java encoder test still uses toSeq. 

Is it that the order is not guaranteed (but happens to be in the expected order 
on little endian) and this is a test issue?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Re: Code freeze?

[Pete Robbins]

Is there a list of Jiras to be considered for 2.0? I would really like to
get https://issues.apache.org/jira/browse/SPARK-13745 in so that Big Endian
platforms are not broken.

Cheers,

On Wed, 13 Apr 2016 at 08:51 Reynold Xin  wrote:

> I think the main things are API things that we need to get right.
>
> - Implement essential DDLs
> https://issues.apache.org/jira/browse/SPARK-14118  this blocks the next
> one
>
> - Merge HiveContext and SQLContext and create SparkSession
> https://issues.apache.org/jira/browse/SPARK-13485
>
> - Separate out local linear algebra as a standalone module without Spark
> dependency https://issues.apache.org/jira/browse/SPARK-13944
>
> - Run Spark without assembly jars (mostly done?)
>
>
> Probably realistic to have it in ~ 2 weeks.
>
>
>
> On Wed, Apr 13, 2016 at 12:45 AM, Sean Owen  wrote:
>
>> I've heard several people refer to a code freeze for 2.0. Unless I missed
>> it, nobody has discussed a particular date for this:
>> https://cwiki.apache.org/confluence/display/SPARK/Wiki+Homepage
>>
>> I'd like to start with a review of JIRAs before anyone decides a freeze
>> is appropriate. There are hundreds of issues, some blockers, still targeted
>> for 2.0. Probably best for everyone to review and retarget non essentials
>> and then see where we are at?
>>
>
>

[jira] [Commented] (SPARK-14151) Propose to refactor and expose Metrics Sink and Source interface

[Pete Robbins (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-14151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15211638#comment-15211638
 ] 

Pete Robbins commented on SPARK-14151:
--

Agreed that is the way to go. I was also working on it but will leave it to 
you. In addition we will need to document the interfaces and use.

> Propose to refactor and expose Metrics Sink and Source interface
> 
>
> Key: SPARK-14151
> URL: https://issues.apache.org/jira/browse/SPARK-14151
> Project: Spark
>  Issue Type: Improvement
>  Components: Spark Core
>Reporter: Saisai Shao
>Priority: Minor
>
> MetricsSystem is designed for plug-in different sources and sinks, user could 
> write their own sources and sinks and configured through metrics.properties, 
> MetricsSystem will register it through reflection. But current Source and 
> Sink interface is private, which means user cannot create their own sources 
> and sinks unless using the same package.
> So here propose to expose source and sink interface, this will let user build 
> and maintain their own source and sink, alleviate the maintenance overhead of 
> spark codebase. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-14151) Propose to expose Metrics Sink and Source interface

[Pete Robbins (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-14151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15211560#comment-15211560
 ] 

Pete Robbins commented on SPARK-14151:
--

In addition the constructor used by MetricsSystem for Sinks passes the 
SecurityManager which is also marked as private[spark]. Currently only the 
MetricsServlet sink uses this.

We could either a) remove private[spark] from SecurityManager or b) add 
additional logic in MetricsSystem to look for a Sink constructor which does not 
have the SecurityManager as a parameter if the one with SecurityManager is not 
found

> Propose to expose Metrics Sink and Source interface
> ---
>
> Key: SPARK-14151
> URL: https://issues.apache.org/jira/browse/SPARK-14151
> Project: Spark
>  Issue Type: Improvement
>  Components: Spark Core
>Reporter: Saisai Shao
>Priority: Minor
>
> MetricsSystem is designed for plug-in different sources and sinks, user could 
> write their own sources and sinks and configured through metrics.properties, 
> MetricsSystem will register it through reflection. But current Source and 
> Sink interface is private, which means user cannot create their own sources 
> and sinks unless using the same package.
> So here propose to expose source and sink interface, this will let user build 
> and maintain their own source and sink, alleviate the maintenance overhead of 
> spark codebase. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Re: Can we remove private[spark] from Metrics Source and SInk traits?

[Pete Robbins]

There are several open Jiras to add new Sinks

OpenTSDB https://issues.apache.org/jira/browse/SPARK-12194
StatsD https://issues.apache.org/jira/browse/SPARK-11574
Kafka https://issues.apache.org/jira/browse/SPARK-13392

Some have PRs from 2015 so I'm assuming there is not the desire to
integrate these into core Spark. Opening up the Sink/Source interfaces
would at least allow these to exist somewhere such as spark-packages
without having to pollute the o.a.s namespace


On Sat, 19 Mar 2016 at 13:05 Gerard Maas <gerard.m...@gmail.com> wrote:

> +1
> On Mar 19, 2016 08:33, "Pete Robbins" <robbin...@gmail.com> wrote:
>
>> This seems to me to be unnecessarily restrictive. These are very useful
>> extension points for adding 3rd party sources and sinks.
>>
>> I intend to make an Elasticsearch sink available on spark-packages but
>> this will require a single class, the sink, to be in the org.apache.spark
>> package tree. I could submit the package as a PR to the Spark codebase, and
>> I'd be happy to do that but it could be a completely separate add-on.
>>
>> There are similar issues with writing a 3rd party metrics source which
>> may not be of interest to the community at large so would probably not
>> warrant inclusion in the Spark codebase.
>>
>> Any thoughts?
>>
>

Can we remove private[spark] from Metrics Source and SInk traits?

[Pete Robbins]

This seems to me to be unnecessarily restrictive. These are very useful
extension points for adding 3rd party sources and sinks.

I intend to make an Elasticsearch sink available on spark-packages but this
will require a single class, the sink, to be in the org.apache.spark
package tree. I could submit the package as a PR to the Spark codebase, and
I'd be happy to do that but it could be a completely separate add-on.

There are similar issues with writing a 3rd party metrics source which may
not be of interest to the community at large so would probably not warrant
inclusion in the Spark codebase.

Any thoughts?

Re: Accessing SparkConf in metrics sink

[Pete Robbins]

So the answer to my previous question is NO.

It looks like I could use SparkEnv.get.conf but

* * NOTE: This is not intended for external use. This is exposed for Shark
and may be made private * in a future release. */



On Wed, 16 Mar 2016 at 08:22 Pete Robbins <robbin...@gmail.com> wrote:

> OK thanks. Does that work in an executor?
>
> On Wed, 16 Mar 2016 at 07:58 Reynold Xin <r...@databricks.com> wrote:
>
>> SparkConf is not a singleton.
>>
>> However, SparkContext in almost all cases are. So you can use
>> SparkContext.getOrCreate().getConf
>>
>> On Wed, Mar 16, 2016 at 12:38 AM, Pete Robbins <robbin...@gmail.com>
>> wrote:
>>
>>> I'm writing a metrics sink and reporter to push metrics to
>>> Elasticsearch. An example format of a metric in JSON:
>>>
>>> {
>>>  "timestamp": "2016-03-15T16:11:19.314+",
>>>  "hostName": "10.192.0.87"
>>>  "applicationName": "My application",
>>>  "applicationId": "app-20160315093931-0003",
>>>  "executorId": "17",
>>>  "executor_threadpool_completeTasks": 20
>>> }
>>>
>>> For correlating the metrics I want the timestamp, hostname,
>>> applicationId, executorId and applicationName.
>>>
>>> Currently I am extracting the applicationId and executor Id from the
>>> metric name as MetricsSystem prepends these to the name. As the sink is
>>> instantiated without the SparkConf I can not determine the applicationName.
>>>
>>> Another proposed change in
>>> https://issues.apache.org/jira/browse/SPARK-10610 would also make me
>>> require access to the SparkConf to get the applicationId/executorId.
>>>
>>> So, Is the SparkConf a singleton and can there be a Utils method for
>>> accessing it? Instantiating a SparkConf myself will not pick up the appName
>>> etc as these are set via methods on the conf.
>>>
>>> I'm trying to write this without modifying any Spark code by just using
>>> a definition in the metrics properties to load my sink.
>>>
>>> Cheers,
>>>
>>
>>

Re: Accessing SparkConf in metrics sink

[Pete Robbins]

OK thanks. Does that work in an executor?

On Wed, 16 Mar 2016 at 07:58 Reynold Xin <r...@databricks.com> wrote:

> SparkConf is not a singleton.
>
> However, SparkContext in almost all cases are. So you can use
> SparkContext.getOrCreate().getConf
>
> On Wed, Mar 16, 2016 at 12:38 AM, Pete Robbins <robbin...@gmail.com>
> wrote:
>
>> I'm writing a metrics sink and reporter to push metrics to Elasticsearch.
>> An example format of a metric in JSON:
>>
>> {
>>  "timestamp": "2016-03-15T16:11:19.314+",
>>  "hostName": "10.192.0.87"
>>  "applicationName": "My application",
>>  "applicationId": "app-20160315093931-0003",
>>  "executorId": "17",
>>  "executor_threadpool_completeTasks": 20
>> }
>>
>> For correlating the metrics I want the timestamp, hostname,
>> applicationId, executorId and applicationName.
>>
>> Currently I am extracting the applicationId and executor Id from the
>> metric name as MetricsSystem prepends these to the name. As the sink is
>> instantiated without the SparkConf I can not determine the applicationName.
>>
>> Another proposed change in
>> https://issues.apache.org/jira/browse/SPARK-10610 would also make me
>> require access to the SparkConf to get the applicationId/executorId.
>>
>> So, Is the SparkConf a singleton and can there be a Utils method for
>> accessing it? Instantiating a SparkConf myself will not pick up the appName
>> etc as these are set via methods on the conf.
>>
>> I'm trying to write this without modifying any Spark code by just using a
>> definition in the metrics properties to load my sink.
>>
>> Cheers,
>>
>
>

Accessing SparkConf in metrics sink

[Pete Robbins]

I'm writing a metrics sink and reporter to push metrics to Elasticsearch.
An example format of a metric in JSON:

{
 "timestamp": "2016-03-15T16:11:19.314+",
 "hostName": "10.192.0.87"
 "applicationName": "My application",
 "applicationId": "app-20160315093931-0003",
 "executorId": "17",
 "executor_threadpool_completeTasks": 20
}

For correlating the metrics I want the timestamp, hostname, applicationId,
executorId and applicationName.

Currently I am extracting the applicationId and executor Id from the metric
name as MetricsSystem prepends these to the name. As the sink is
instantiated without the SparkConf I can not determine the applicationName.

Another proposed change in https://issues.apache.org/jira/browse/SPARK-10610
would also make me require access to the SparkConf to get the
applicationId/executorId.

So, Is the SparkConf a singleton and can there be a Utils method for
accessing it? Instantiating a SparkConf myself will not pick up the appName
etc as these are set via methods on the conf.

I'm trying to write this without modifying any Spark code by just using a
definition in the metrics properties to load my sink.

Cheers,

Re: SparkConf constructor now private

[Pete Robbins]

Is the SparkConf effectively a singleton? Could there be a Utils method to
return a clone of the SparkConf?

Cheers

On Tue, 15 Mar 2016 at 16:49 Marcelo Vanzin  wrote:

> Oh, my bad. I think I left that from a previous part of the patch and
> forgot to revert it. Will fix.
>
> On Tue, Mar 15, 2016 at 7:37 AM, Koert Kuipers  wrote:
> > in this commit
> >
> > 8301fadd8d269da11e72870b7a889596e3337839
> > Author: Marcelo Vanzin 
> > Date:   Mon Mar 14 14:27:33 2016 -0700
> > [SPARK-13626][CORE] Avoid duplicate config deprecation warnings.
> >
> > the following change was made
> >
> > -class SparkConf(loadDefaults: Boolean) extends Cloneable with Logging {
> > +class SparkConf private[spark] (loadDefaults: Boolean) extends Cloneable
> > with Logging {
> >
> > i use the constructor new SparkConf(false) to build a SparkConf for our
> > in-house unit tests (where i do not want system properties to change
> meddle
> > with things).
> >
> > is this API change on purpose?
>
>
>
> --
> Marcelo
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
> For additional commands, e-mail: dev-h...@spark.apache.org
>
>

[jira] [Comment Edited] (SPARK-10610) Using AppName instead of AppId in the name of all metrics

[Pete Robbins (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-10610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15179507#comment-15179507
 ] 

Pete Robbins edited comment on SPARK-10610 at 3/4/16 8:01 AM:
--

I think the appId is an important piece of information when visualizing the 
metrics along with hostname, executorId etc. I'm writing a sink and reporter to 
push the metrics to Elasticsearch and I include these in the metrics types for 
better correlation. eg

{
"timestamp": "2016-03-03T15:58:31.903+",
"hostName": "9.20.187.127"
"applicationId": "app-20160303155742-0005",
"executorId": "driver",
"BlockManager_memory_maxMem_MB": 3933
  }

The appId and executorId I extract form the metric name. When the sink is 
instantiated I don't believe I have access to any Utils to obtain the current 
appId and executorId so I'm kind of relying on these being in the metric name 
for the moment.

Is it possible to make appId, applicationName, executorId avaiable to me via 
some Utils function that I have access to in a metrics Sink?

I guess I'm asking: How can I get hold of the SparkConf if I've not been passed 
it?


was (Author: robbinspg):
I think the appId is an important piece of information when visualizing the 
metrics along with hostname, executorId etc. I'm writing a sink and reporter to 
push the metrics to Elasticsearch and I include these in the metrics types for 
better correlation. eg

{
"timestamp": "2016-03-03T15:58:31.903+",
"hostName": "9.20.187.127"
"applicationId": "app-20160303155742-0005",
"executorId": "driver",
"BlockManager_memory_maxMem_MB": 3933
  }

The appId and executorId I extract form the metric name. When the sink is 
instantiated I don't believe I have access to any Utils to obtain the current 
appId and executorId so I'm kind of relying on these being in the metric name 
for the moment.

Is it possible to make appId, applicationName, executorId avaiable to me via 
some Utils function that I have access to in a metrics Sink?

> Using AppName instead of AppId in the name of all metrics
> -
>
> Key: SPARK-10610
> URL: https://issues.apache.org/jira/browse/SPARK-10610
> Project: Spark
>  Issue Type: New Feature
>  Components: Spark Core
>Affects Versions: 1.5.0
>Reporter: Yi Tian
>Priority: Minor
>
> When we using {{JMX}} to monitor spark system,  We have to configure the name 
> of target metrics in the monitor system. But the current name of metrics is 
> {{appId}} + {{executorId}} + {{source}} . So when the spark program 
> restarted, we have to update the name of metrics in the monitor system.
> We should add an optional configuration property to control whether using the 
> appName instead of appId in spark metrics system.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-10610) Using AppName instead of AppId in the name of all metrics

[Pete Robbins (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-10610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15179507#comment-15179507
 ] 

Pete Robbins commented on SPARK-10610:
--

I think the appId is an important piece of information when visualizing the 
metrics along with hostname, executorId etc. I'm writing a sink and reporter to 
push the metrics to Elasticsearch and I include these in the metrics types for 
better correlation. eg

{
"timestamp": "2016-03-03T15:58:31.903+",
"hostName": "9.20.187.127"
"applicationId": "app-20160303155742-0005",
"executorId": "driver",
"BlockManager_memory_maxMem_MB": 3933
  }

The appId and executorId I extract form the metric name. When the sink is 
instantiated I don't believe I have access to any Utils to obtain the current 
appId and executorId so I'm kind of relying on these being in the metric name 
for the moment.

Is it possible to make appId, applicationName, executorId avaiable to me via 
some Utils function that I have access to in a metrics Sink?

> Using AppName instead of AppId in the name of all metrics
> -
>
> Key: SPARK-10610
> URL: https://issues.apache.org/jira/browse/SPARK-10610
> Project: Spark
>  Issue Type: New Feature
>  Components: Spark Core
>Affects Versions: 1.5.0
>Reporter: Yi Tian
>Priority: Minor
>
> When we using {{JMX}} to monitor spark system,  We have to configure the name 
> of target metrics in the monitor system. But the current name of metrics is 
> {{appId}} + {{executorId}} + {{source}} . So when the spark program 
> restarted, we have to update the name of metrics in the monitor system.
> We should add an optional configuration property to control whether using the 
> appName instead of appId in spark metrics system.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Re: SparkOscope: Enabling Spark Optimization through Cross-stack Monitoring and Visualization

[Pete Robbins]

Yiannis,

I'm interested in what you've done here as I was looking for ways to allow
the Spark UI to display custom metrics in a pluggable way without having to
modify the Spark source code. It would be good to see if we could have
modify your code to add extension points into the UI so we could configure
sources of the additional metrics. So for instance rather than creating
events from your HDFS files I would like to have a module that is pulling
in system/jvm metrics that are in eg Elasticsearch.

Do any of the Spark committers have any thoughts on this?

Cheers,


On 3 February 2016 at 15:26, Yiannis Gkoufas  wrote:

> Hi all,
>
> I just wanted to introduce some of my recent work in IBM Research around
> Spark and especially its Metric System and Web UI.
> As a quick overview of our contributions:
> We have a created a new type of Sink for the metrics ( HDFSSink ) which
> captures the metrics into HDFS,
> We have extended the metrics reported by the Executors to include OS-level
> metrics regarding CPU, RAM, Disk IO, Network IO utilizing the Hyperic Sigar
> library
> We have extended the Web UI for the completed applications to visualize
> any of the above metrics the user wants to.
> The above functionalities can be configured in the metrics.properties and
> spark-defaults.conf files.
> We have recorded a small demo that shows those capabilities which you can
> find here : https://ibm.app.box.com/s/vyaedlyb444a4zna1215c7puhxliqxdg
> There is a blog post which gives more details on the functionality here:
> *www.spark.tc/sparkoscope-enabling-spark-optimization-through-cross-stack-monitoring-and-visualization-2/*
> 
> and also there is a public repo where anyone can try it:
> *https://github.com/ibm-research-ireland/sparkoscope*
> 
>
> I would really appreciate any feedback or advice regarding this work.
> Especially if you think it's worth it to upstream to the official Spark
> repository.
>
> Thanks a lot!
>

Fwd: Elasticsearch sink for metrics

[Pete Robbins]

The issue I had was with the ElasticsearchReporter and how it maps eg a
Gauge in JSON. The "value" was typed to whatever the first Guage was, eg
int, which caused issues with some of my other guages which were double.

As I say I've just started looking at this and was wanting to see if this
was already implemented before continuing.

On 15 January 2016 at 09:18, Nick Pentreath <nick.pentre...@gmail.com>
wrote:

> I haven't come across anything, but could you provide more detail on what
> issues you're encountering?
>
>
>
> On Fri, Jan 15, 2016 at 11:09 AM, Pete Robbins <robbin...@gmail.com>
> wrote:
>
>> Has anyone tried pushing Spark metrics into elasticsearch? We have other
>> metrics, eg some runtime information, going into ES and would like to be
>> able to combine this with the Spark metrics for visualization with Kibana.
>>
>> I experimented with a new sink using ES's ElasticsearchReporter for the
>> Coda Hale metrics but have a few issues with default mappings.
>>
>> Has anyone already implemented this before I start to dig deeper?
>>
>> Cheers,
>>
>>
>>
>

Elasticsearch sink for metrics

[Pete Robbins]

Has anyone tried pushing Spark metrics into elasticsearch? We have other
metrics, eg some runtime information, going into ES and would like to be
able to combine this with the Spark metrics for visualization with Kibana.

I experimented with a new sink using ES's ElasticsearchReporter for the
Coda Hale metrics but have a few issues with default mappings.

Has anyone already implemented this before I start to dig deeper?

Cheers,

[jira] [Commented] (SPARK-12647) 1.6 branch test failure o.a.s.sql.execution.ExchangeCoordinatorSuite.determining the number of reducers: aggregate operator

[Pete Robbins (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-12647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15082879#comment-15082879
 ] 

Pete Robbins commented on SPARK-12647:
--

@sowen should I close this and move the PR?


> 1.6 branch test failure 
> o.a.s.sql.execution.ExchangeCoordinatorSuite.determining the number of 
> reducers: aggregate operator
> ---
>
> Key: SPARK-12647
> URL: https://issues.apache.org/jira/browse/SPARK-12647
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 1.6.0
>Reporter: Pete Robbins
>Priority: Minor
>
> All 1.6 branch builds failing eg 
> https://amplab.cs.berkeley.edu/jenkins/job/spark-branch-1.6-test-maven-pre-yarn-2.0.0-mr1-cdh4.1.2/lastCompletedBuild/testReport/org.apache.spark.sql.execution/ExchangeCoordinatorSuite/determining_the_number_of_reducers__aggregate_operator/
> 3 did not equal 2
> PR for SPARK-12470 causes change in partition size so test needs updating



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Created] (SPARK-12647) 1.6 branch test failure o.a.s.sql.execution.ExchangeCoordinatorSuite.determining the number of reducers: aggregate operator

[Pete Robbins (JIRA)]

Pete Robbins created SPARK-12647:


 Summary: 1.6 branch test failure 
o.a.s.sql.execution.ExchangeCoordinatorSuite.determining the number of 
reducers: aggregate operator
 Key: SPARK-12647
 URL: https://issues.apache.org/jira/browse/SPARK-12647
 Project: Spark
  Issue Type: Bug
  Components: SQL
Affects Versions: 1.6.0
Reporter: Pete Robbins
Priority: Minor


All 1.6 branch builds failing eg 
https://amplab.cs.berkeley.edu/jenkins/job/spark-branch-1.6-test-maven-pre-yarn-2.0.0-mr1-cdh4.1.2/lastCompletedBuild/testReport/org.apache.spark.sql.execution/ExchangeCoordinatorSuite/determining_the_number_of_reducers__aggregate_operator/

3 did not equal 2

PR for SPARK-12470 causes change in partition size so test needs updating



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Comment Edited] (SPARK-12647) 1.6 branch test failure o.a.s.sql.execution.ExchangeCoordinatorSuite.determining the number of reducers: aggregate operator

[Pete Robbins (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-12647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15082879#comment-15082879
 ] 

Pete Robbins edited comment on SPARK-12647 at 1/5/16 11:30 AM:
---

[~sowen] should I close this and move the PR?



was (Author: robbinspg):
@sowen should I close this and move the PR?


> 1.6 branch test failure 
> o.a.s.sql.execution.ExchangeCoordinatorSuite.determining the number of 
> reducers: aggregate operator
> ---
>
> Key: SPARK-12647
> URL: https://issues.apache.org/jira/browse/SPARK-12647
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 1.6.0
>Reporter: Pete Robbins
>Priority: Minor
>
> All 1.6 branch builds failing eg 
> https://amplab.cs.berkeley.edu/jenkins/job/spark-branch-1.6-test-maven-pre-yarn-2.0.0-mr1-cdh4.1.2/lastCompletedBuild/testReport/org.apache.spark.sql.execution/ExchangeCoordinatorSuite/determining_the_number_of_reducers__aggregate_operator/
> 3 did not equal 2
> PR for SPARK-12470 causes change in partition size so test needs updating



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-12470) Incorrect calculation of row size in o.a.s.sql.catalyst.expressions.codegen.GenerateUnsafeRowJoiner

[Pete Robbins (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-12470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15068693#comment-15068693
 ] 

Pete Robbins commented on SPARK-12470:
--

I'm fairly sure the code in my PR is correct but it is causing an 
ExchangeCoordinatorSuite test to fail. I'm struggling to see why this test is 
failing with the change I made. The failure is:

 determining the number of reducers: aggregate operator *** FAILED ***
 3 did not equal 2 (ExchangeCoordinatorSuite.scala:316)

putting some debug into the test I see that before my change the pre-shuffle 
partition sizes are 600, 600, 600, 600, 600 an after my change are 800. 800. 
800. 800. 720 but I have no idea why. I'd really appreciate anyone with 
knowledge of this area a) checking my PR and b) helping explain the failing 
test.

> Incorrect calculation of row size in 
> o.a.s.sql.catalyst.expressions.codegen.GenerateUnsafeRowJoiner
> ---
>
> Key: SPARK-12470
> URL: https://issues.apache.org/jira/browse/SPARK-12470
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 1.5.2
>    Reporter: Pete Robbins
>Priority: Minor
>
> While looking into https://issues.apache.org/jira/browse/SPARK-12319 I 
> noticed that the row size is incorrectly calculated.
> The "sizeReduction" value is calculated in words:
>// The number of words we can reduce when we concat two rows together.
> // The only reduction comes from merging the bitset portion of the two 
> rows, saving 1 word.
> val sizeReduction = bitset1Words + bitset2Words - outputBitsetWords
> but then it is subtracted from the size of the row in bytes:
>|out.pointTo(buf, ${schema1.size + schema2.size}, sizeInBytes - 
> $sizeReduction);
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Comment Edited] (SPARK-12470) Incorrect calculation of row size in o.a.s.sql.catalyst.expressions.codegen.GenerateUnsafeRowJoiner

[Pete Robbins (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-12470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15068693#comment-15068693
 ] 

Pete Robbins edited comment on SPARK-12470 at 12/22/15 9:47 PM:


I'm fairly sure the code in my PR is correct but it is causing an 
ExchangeCoordinatorSuite test to fail. I'm struggling to see why this test is 
failing with the change I made. The failure is:

 determining the number of reducers: aggregate operator *** FAILED ***
 3 did not equal 2 (ExchangeCoordinatorSuite.scala:316)

putting some debug into the test I see that before my change the pre-shuffle 
partition sizes are 600, 600, 600, 600, 600 an after my change are 800. 800. 
800. 800. 720 but I have no idea why. I'd really appreciate anyone with 
knowledge of this area a) checking my PR and b) helping explain the failing 
test.

EDIT Please ignore. Merged with latest head including changes for SPARK-12388 
now passes all tests


was (Author: robbinspg):
I'm fairly sure the code in my PR is correct but it is causing an 
ExchangeCoordinatorSuite test to fail. I'm struggling to see why this test is 
failing with the change I made. The failure is:

 determining the number of reducers: aggregate operator *** FAILED ***
 3 did not equal 2 (ExchangeCoordinatorSuite.scala:316)

putting some debug into the test I see that before my change the pre-shuffle 
partition sizes are 600, 600, 600, 600, 600 an after my change are 800. 800. 
800. 800. 720 but I have no idea why. I'd really appreciate anyone with 
knowledge of this area a) checking my PR and b) helping explain the failing 
test.

> Incorrect calculation of row size in 
> o.a.s.sql.catalyst.expressions.codegen.GenerateUnsafeRowJoiner
> ---
>
> Key: SPARK-12470
> URL: https://issues.apache.org/jira/browse/SPARK-12470
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 1.5.2
>    Reporter: Pete Robbins
>Priority: Minor
>
> While looking into https://issues.apache.org/jira/browse/SPARK-12319 I 
> noticed that the row size is incorrectly calculated.
> The "sizeReduction" value is calculated in words:
>// The number of words we can reduce when we concat two rows together.
> // The only reduction comes from merging the bitset portion of the two 
> rows, saving 1 word.
> val sizeReduction = bitset1Words + bitset2Words - outputBitsetWords
> but then it is subtracted from the size of the row in bytes:
>|out.pointTo(buf, ${schema1.size + schema2.size}, sizeInBytes - 
> $sizeReduction);
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-12470) Incorrect calculation of row size in o.a.s.sql.catalyst.expressions.codegen.GenerateUnsafeRowJoiner

[Pete Robbins (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-12470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pete Robbins updated SPARK-12470:
-
Component/s: SQL
Summary: Incorrect calculation of row size in 
o.a.s.sql.catalyst.expressions.codegen.GenerateUnsafeRowJoiner  (was: Incorrect 
calculation of row size in 
o.a.s.catalyst.expressions.codegen.GenerateUnsafeRowJoiner)

> Incorrect calculation of row size in 
> o.a.s.sql.catalyst.expressions.codegen.GenerateUnsafeRowJoiner
> ---
>
> Key: SPARK-12470
> URL: https://issues.apache.org/jira/browse/SPARK-12470
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 1.5.2
>    Reporter: Pete Robbins
>Priority: Minor
>
> While looking into https://issues.apache.org/jira/browse/SPARK-12319 I 
> noticed that the row size is incorrectly calculated.
> The "sizeReduction" value is calculated in words:
>// The number of words we can reduce when we concat two rows together.
> // The only reduction comes from merging the bitset portion of the two 
> rows, saving 1 word.
> val sizeReduction = bitset1Words + bitset2Words - outputBitsetWords
> but then it is subtracted from the size of the row in bytes:
>|out.pointTo(buf, ${schema1.size + schema2.size}, sizeInBytes - 
> $sizeReduction);
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Created] (SPARK-12470) Incorrect calculation of row size in o.a.s.catalyst.expressions.codegen.GenerateUnsafeRowJoiner

[Pete Robbins (JIRA)]

Pete Robbins created SPARK-12470:


 Summary: Incorrect calculation of row size in 
o.a.s.catalyst.expressions.codegen.GenerateUnsafeRowJoiner
 Key: SPARK-12470
 URL: https://issues.apache.org/jira/browse/SPARK-12470
 Project: Spark
  Issue Type: Bug
Affects Versions: 1.5.2
Reporter: Pete Robbins
Priority: Minor


While looking into https://issues.apache.org/jira/browse/SPARK-12319 I noticed 
that the row size is incorrectly calculated.

The "sizeReduction" value is calculated in words:

   // The number of words we can reduce when we concat two rows together.
// The only reduction comes from merging the bitset portion of the two 
rows, saving 1 word.
val sizeReduction = bitset1Words + bitset2Words - outputBitsetWords

but then it is subtracted from the size of the row in bytes:

   |out.pointTo(buf, ${schema1.size + schema2.size}, sizeInBytes - 
$sizeReduction);
 





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-6873) Some Hive-Catalyst comparison tests fail due to unimportant order of some printed elements

[Pete Robbins (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-6873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14907841#comment-14907841
 ] 

Pete Robbins commented on SPARK-6873:
-

I no longer see these errors in my 1.5 branch Java 8 build. Did someone fix 
them, remove the tests or is it just chance?

> Some Hive-Catalyst comparison tests fail due to unimportant order of some 
> printed elements
> --
>
> Key: SPARK-6873
> URL: https://issues.apache.org/jira/browse/SPARK-6873
> Project: Spark
>  Issue Type: Bug
>  Components: SQL, Tests
>Affects Versions: 1.3.1
>Reporter: Sean Owen
>Assignee: Cheng Lian
>Priority: Minor
>
> As I mentioned, I've been seeing 4 test failures in Hive tests for a while, 
> and actually it still affects master. I think it's a superficial problem that 
> only turns up when running on Java 8, but still, would probably be an easy 
> fix and good to fix.
> Specifically, here are four tests and the bit that fails the comparison, 
> below. I tried to diagnose this but had trouble even finding where some of 
> this occurs, like the list of synonyms?
> {code}
> - show_tblproperties *** FAILED ***
>   Results do not match for show_tblproperties:
> ...
>   !== HIVE - 2 row(s) ==   == CATALYST - 2 row(s) ==
>   !tmptruebar bar value
>   !barbar value   tmp true (HiveComparisonTest.scala:391)
> {code}
> {code}
> - show_create_table_serde *** FAILED ***
>   Results do not match for show_create_table_serde:
> ...
>WITH SERDEPROPERTIES (  WITH 
> SERDEPROPERTIES ( 
>   !  'serialization.format'='$', 
> 'field.delim'=',', 
>   !  'field.delim'=',')  
> 'serialization.format'='$')
> {code}
> {code}
> - udf_std *** FAILED ***
>   Results do not match for udf_std:
> ...
>   !== HIVE - 2 row(s) == == CATALYST 
> - 2 row(s) ==
>std(x) - Returns the standard deviation of a set of numbers   std(x) - 
> Returns the standard deviation of a set of numbers
>   !Synonyms: stddev_pop, stddev  Synonyms: 
> stddev, stddev_pop (HiveComparisonTest.scala:391)
> {code}
> {code}
> - udf_stddev *** FAILED ***
>   Results do not match for udf_stddev:
> ...
>   !== HIVE - 2 row(s) ==== 
> CATALYST - 2 row(s) ==
>stddev(x) - Returns the standard deviation of a set of numbers   stddev(x) 
> - Returns the standard deviation of a set of numbers
>   !Synonyms: stddev_pop, stdSynonyms: 
> std, stddev_pop (HiveComparisonTest.scala:391)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-9710) RPackageUtilsSuite fails if R is not installed

[Pete Robbins (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-9710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904485#comment-14904485
 ] 

Pete Robbins commented on SPARK-9710:
-

The Fix Version for this says 1.5.0 but the PR is not in the 1.5 branch as far 
as I can see and my 1.5 branch build is failing with this issue

> RPackageUtilsSuite fails if R is not installed
> --
>
> Key: SPARK-9710
> URL: https://issues.apache.org/jira/browse/SPARK-9710
> Project: Spark
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 1.5.0
>Reporter: Marcelo Vanzin
>Assignee: Marcelo Vanzin
> Fix For: 1.5.0
>
>
> That's because there's a bug in RUtils.scala. PR soon.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Re: Unable to acquire memory errors in HiveCompatibilitySuite

[Pete Robbins]

ok so let me try again ;-)

I don't think that the page size calculation matters apart from hitting the
allocation limit earlier if the page size is too large.

If a task is going to need X bytes, it is going to need X bytes. In this
case, for at least one of the tasks, X > maxmemory/no_active_tasks at some
point during execution. A smaller page size may use the memory more
efficiently but would not necessarily avoid this issue.

The next question would be: Is the memory limit per task of
max_memory/no_active_tasks reasonable? It seems fair but if this limit is
reached currently an exception is thrown, maybe the task could wait for
no_active_tasks to decrease?

I think what causes my test issue is that the 32 tasks don't execute as
quickly on my 8 core box so more are active at any one time.

I will experiment with the page size calculation to see what effect it has.

Cheers,



On 16 September 2015 at 06:53, Reynold Xin <r...@databricks.com> wrote:

> It is exactly the issue here, isn't it?
>
> We are using memory / N, where N should be the maximum number of active
> tasks. In the current master, we use the number of cores to approximate the
> number of tasks -- but it turned out to be a bad approximation in tests
> because it is set to 32 to increase concurrency.
>
>
> On Tue, Sep 15, 2015 at 10:47 PM, Pete Robbins <robbin...@gmail.com>
> wrote:
>
>> Oops... I meant to say "The page size calculation is NOT the issue here"
>>
>> On 16 September 2015 at 06:46, Pete Robbins <robbin...@gmail.com> wrote:
>>
>>> The page size calculation is the issue here as there is plenty of free
>>> memory, although there is maybe a fair bit of wasted space in some pages.
>>> It is that when we have a lot of tasks each is only allowed to reach 1/n of
>>> the available memory and several of the tasks bump in to that limit. With
>>> tasks 4 times the number of cores there will be some contention and so they
>>> remain active for longer.
>>>
>>> So I think this is a test case issue configuring the number of executors
>>> too high.
>>>
>>> On 15 September 2015 at 18:54, Reynold Xin <r...@databricks.com> wrote:
>>>
>>>> Maybe we can change the heuristics in memory calculation to use
>>>> SparkContext.defaultParallelism if it is local mode.
>>>>
>>>>
>>>> On Tue, Sep 15, 2015 at 10:28 AM, Pete Robbins <robbin...@gmail.com>
>>>> wrote:
>>>>
>>>>> Yes and at least there is an override by setting
>>>>> spark.sql.test.master to local[8] , in fact local[16] worked on my 8 core
>>>>> box.
>>>>>
>>>>> I'm happy to use this as a workaround but the 32 hard-coded will fail
>>>>> running build/tests on a clean checkout if you only have 8 cores.
>>>>>
>>>>> On 15 September 2015 at 17:40, Marcelo Vanzin <van...@cloudera.com>
>>>>> wrote:
>>>>>
>>>>>> That test explicitly sets the number of executor cores to 32.
>>>>>>
>>>>>> object TestHive
>>>>>>   extends TestHiveContext(
>>>>>> new SparkContext(
>>>>>>   System.getProperty("spark.sql.test.master", "local[32]"),
>>>>>>
>>>>>>
>>>>>> On Mon, Sep 14, 2015 at 11:22 PM, Reynold Xin <r...@databricks.com>
>>>>>> wrote:
>>>>>> > Yea I think this is where the heuristics is failing -- it uses 8
>>>>>> cores to
>>>>>> > approximate the number of active tasks, but the tests somehow is
>>>>>> using 32
>>>>>> > (maybe because it explicitly sets it to that, or you set it
>>>>>> yourself? I'm
>>>>>> > not sure which one)
>>>>>> >
>>>>>> > On Mon, Sep 14, 2015 at 11:06 PM, Pete Robbins <robbin...@gmail.com>
>>>>>> wrote:
>>>>>> >>
>>>>>> >> Reynold, thanks for replying.
>>>>>> >>
>>>>>> >> getPageSize parameters: maxMemory=515396075, numCores=0
>>>>>> >> Calculated values: cores=8, default=4194304
>>>>>> >>
>>>>>> >> So am I getting a large page size as I only have 8 cores?
>>>>>> >>
>>>>>> >> On 15 September 2015 at 00:40, Reynold Xin <r...@databricks.com>
>>>>>> wrote:
>>>>>> >>>
>>>