[jira] [Commented] (SPARK-16233) test_sparkSQL.R is failing
[
https://issues.apache.org/jira/browse/SPARK-16233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15360299#comment-15360299
]
Dongjoon Hyun commented on SPARK-16233:
---
Sorry for not being clear!
> test_sparkSQL.R is failing
> --
>
> Key: SPARK-16233
> URL: https://issues.apache.org/jira/browse/SPARK-16233
> Project: Spark
> Issue Type: Bug
> Components: SparkR, Tests
>Affects Versions: 2.0.0
>Reporter: Xin Ren
>Assignee: Dongjoon Hyun
>Priority: Minor
> Fix For: 2.0.0
>
>
> By running
> {code}
> ./R/run-tests.sh
> {code}
> Getting error:
> {code}
> xin:spark xr$ ./R/run-tests.sh
> Warning: Ignoring non-spark config property: SPARK_SCALA_VERSION=2.11
> Loading required package: methods
> Attaching package: ‘SparkR’
> The following object is masked from ‘package:testthat’:
> describe
> The following objects are masked from ‘package:stats’:
> cov, filter, lag, na.omit, predict, sd, var, window
> The following objects are masked from ‘package:base’:
> as.data.frame, colnames, colnames<-, drop, endsWith, intersect,
> rank, rbind, sample, startsWith, subset, summary, transform, union
> binary functions: ...
> functions on binary files:
> broadcast variables: ..
> functions in client.R: .
> test functions in sparkR.R: .Re-using existing Spark Context. Call
> sparkR.session.stop() or restart R to create a new Spark Context
> Re-using existing Spark Context. Call sparkR.session.stop() or restart R
> to create a new Spark Context
> ...
> include an external JAR in SparkContext: Warning: Ignoring non-spark config
> property: SPARK_SCALA_VERSION=2.11
> ..
> include R packages:
> MLlib functions: .SLF4J: Failed to load class
> "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: Defaulting to no-operation (NOP) logger implementation
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
> details.
> .27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.codec.CodecConfig:
> Compression: SNAPPY
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet block size to 134217728
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet page size to 1048576
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet dictionary page size to 1048576
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Dictionary is on
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Validation is off
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Writer version is: PARQUET_1_0
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Maximum row group padding size is 0 bytes
> 27-Jun-2016 1:51:25 PM INFO:
> org.apache.parquet.hadoop.InternalParquetRecordWriter: Flushing mem
> columnStore to file. allocated memory: 65,622
> 27-Jun-2016 1:51:25 PM INFO:
> org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written 70B for [label]
> BINARY: 1 values, 21B raw, 23B comp, 1 pages, encodings: [PLAIN, RLE,
> BIT_PACKED]
> 27-Jun-2016 1:51:25 PM INFO:
> org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written 87B for [terms,
> list, element, list, element] BINARY: 2 values, 42B raw, 43B comp, 1 pages,
> encodings: [PLAIN, RLE]
> 27-Jun-2016 1:51:25 PM INFO:
> org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written 30B for
> [hasIntercept] BOOLEAN: 1 values, 1B raw, 3B comp, 1 pages, encodings:
> [PLAIN, BIT_PACKED]
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.codec.CodecConfig:
> Compression: SNAPPY
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet block size to 134217728
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet page size to 1048576
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet dictionary page size to 1048576
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Dictionary is on
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Validation is off
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Writer version is: PARQUET_1_0
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Maximum row group padding size is 0 bytes
> 27-Jun-2016 1:51:26 PM INFO:
> org.apache.parquet.hadoop.InternalParquetRecordWriter: Flushing mem
> columnStore to file. allocated memory: 49
> 27-Jun-2016 1:51:26 PM INFO:
> org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written 90B for [labels,
> list, element] BINARY: 3 values, 50B raw, 50B comp, 1 pages, encodings:
> [PLAIN
[jira] [Commented] (SPARK-16233) test_sparkSQL.R is failing
[
https://issues.apache.org/jira/browse/SPARK-16233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15360058#comment-15360058
]
Xin Ren commented on SPARK-16233:
-
Oh sorry I thought you guys would take over so I stopped working on this one.
Thanks a lot resolving this (y)
> test_sparkSQL.R is failing
> --
>
> Key: SPARK-16233
> URL: https://issues.apache.org/jira/browse/SPARK-16233
> Project: Spark
> Issue Type: Bug
> Components: SparkR, Tests
>Affects Versions: 2.0.0
>Reporter: Xin Ren
>Assignee: Dongjoon Hyun
>Priority: Minor
> Fix For: 2.0.0
>
>
> By running
> {code}
> ./R/run-tests.sh
> {code}
> Getting error:
> {code}
> xin:spark xr$ ./R/run-tests.sh
> Warning: Ignoring non-spark config property: SPARK_SCALA_VERSION=2.11
> Loading required package: methods
> Attaching package: ‘SparkR’
> The following object is masked from ‘package:testthat’:
> describe
> The following objects are masked from ‘package:stats’:
> cov, filter, lag, na.omit, predict, sd, var, window
> The following objects are masked from ‘package:base’:
> as.data.frame, colnames, colnames<-, drop, endsWith, intersect,
> rank, rbind, sample, startsWith, subset, summary, transform, union
> binary functions: ...
> functions on binary files:
> broadcast variables: ..
> functions in client.R: .
> test functions in sparkR.R: .Re-using existing Spark Context. Call
> sparkR.session.stop() or restart R to create a new Spark Context
> Re-using existing Spark Context. Call sparkR.session.stop() or restart R
> to create a new Spark Context
> ...
> include an external JAR in SparkContext: Warning: Ignoring non-spark config
> property: SPARK_SCALA_VERSION=2.11
> ..
> include R packages:
> MLlib functions: .SLF4J: Failed to load class
> "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: Defaulting to no-operation (NOP) logger implementation
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
> details.
> .27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.codec.CodecConfig:
> Compression: SNAPPY
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet block size to 134217728
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet page size to 1048576
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet dictionary page size to 1048576
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Dictionary is on
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Validation is off
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Writer version is: PARQUET_1_0
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Maximum row group padding size is 0 bytes
> 27-Jun-2016 1:51:25 PM INFO:
> org.apache.parquet.hadoop.InternalParquetRecordWriter: Flushing mem
> columnStore to file. allocated memory: 65,622
> 27-Jun-2016 1:51:25 PM INFO:
> org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written 70B for [label]
> BINARY: 1 values, 21B raw, 23B comp, 1 pages, encodings: [PLAIN, RLE,
> BIT_PACKED]
> 27-Jun-2016 1:51:25 PM INFO:
> org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written 87B for [terms,
> list, element, list, element] BINARY: 2 values, 42B raw, 43B comp, 1 pages,
> encodings: [PLAIN, RLE]
> 27-Jun-2016 1:51:25 PM INFO:
> org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written 30B for
> [hasIntercept] BOOLEAN: 1 values, 1B raw, 3B comp, 1 pages, encodings:
> [PLAIN, BIT_PACKED]
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.codec.CodecConfig:
> Compression: SNAPPY
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet block size to 134217728
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet page size to 1048576
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet dictionary page size to 1048576
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Dictionary is on
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Validation is off
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Writer version is: PARQUET_1_0
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Maximum row group padding size is 0 bytes
> 27-Jun-2016 1:51:26 PM INFO:
> org.apache.parquet.hadoop.InternalParquetRecordWriter: Flushing mem
> columnStore to file. allocated memory: 49
> 27-Jun-2016 1:51:26 PM INFO:
> org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written 90B for [labels,
> list, ele
[jira] [Commented] (SPARK-16233) test_sparkSQL.R is failing
[
https://issues.apache.org/jira/browse/SPARK-16233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15359726#comment-15359726
]
Dongjoon Hyun commented on SPARK-16233:
---
https://github.com/apache/spark/pull/14019
> test_sparkSQL.R is failing
> --
>
> Key: SPARK-16233
> URL: https://issues.apache.org/jira/browse/SPARK-16233
> Project: Spark
> Issue Type: Bug
> Components: SparkR, Tests
>Affects Versions: 2.0.0
>Reporter: Xin Ren
>Assignee: Apache Spark
>Priority: Minor
>
> By running
> {code}
> ./R/run-tests.sh
> {code}
> Getting error:
> {code}
> xin:spark xr$ ./R/run-tests.sh
> Warning: Ignoring non-spark config property: SPARK_SCALA_VERSION=2.11
> Loading required package: methods
> Attaching package: ‘SparkR’
> The following object is masked from ‘package:testthat’:
> describe
> The following objects are masked from ‘package:stats’:
> cov, filter, lag, na.omit, predict, sd, var, window
> The following objects are masked from ‘package:base’:
> as.data.frame, colnames, colnames<-, drop, endsWith, intersect,
> rank, rbind, sample, startsWith, subset, summary, transform, union
> binary functions: ...
> functions on binary files:
> broadcast variables: ..
> functions in client.R: .
> test functions in sparkR.R: .Re-using existing Spark Context. Call
> sparkR.session.stop() or restart R to create a new Spark Context
> Re-using existing Spark Context. Call sparkR.session.stop() or restart R
> to create a new Spark Context
> ...
> include an external JAR in SparkContext: Warning: Ignoring non-spark config
> property: SPARK_SCALA_VERSION=2.11
> ..
> include R packages:
> MLlib functions: .SLF4J: Failed to load class
> "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: Defaulting to no-operation (NOP) logger implementation
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
> details.
> .27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.codec.CodecConfig:
> Compression: SNAPPY
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet block size to 134217728
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet page size to 1048576
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet dictionary page size to 1048576
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Dictionary is on
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Validation is off
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Writer version is: PARQUET_1_0
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Maximum row group padding size is 0 bytes
> 27-Jun-2016 1:51:25 PM INFO:
> org.apache.parquet.hadoop.InternalParquetRecordWriter: Flushing mem
> columnStore to file. allocated memory: 65,622
> 27-Jun-2016 1:51:25 PM INFO:
> org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written 70B for [label]
> BINARY: 1 values, 21B raw, 23B comp, 1 pages, encodings: [PLAIN, RLE,
> BIT_PACKED]
> 27-Jun-2016 1:51:25 PM INFO:
> org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written 87B for [terms,
> list, element, list, element] BINARY: 2 values, 42B raw, 43B comp, 1 pages,
> encodings: [PLAIN, RLE]
> 27-Jun-2016 1:51:25 PM INFO:
> org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written 30B for
> [hasIntercept] BOOLEAN: 1 values, 1B raw, 3B comp, 1 pages, encodings:
> [PLAIN, BIT_PACKED]
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.codec.CodecConfig:
> Compression: SNAPPY
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet block size to 134217728
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet page size to 1048576
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet dictionary page size to 1048576
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Dictionary is on
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Validation is off
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Writer version is: PARQUET_1_0
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Maximum row group padding size is 0 bytes
> 27-Jun-2016 1:51:26 PM INFO:
> org.apache.parquet.hadoop.InternalParquetRecordWriter: Flushing mem
> columnStore to file. allocated memory: 49
> 27-Jun-2016 1:51:26 PM INFO:
> org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written 90B for [labels,
> list, element] BINARY: 3 values, 50B raw, 50B comp, 1 pages, encodings:
> [PLAIN, RLE]
> 27-Jun-
[jira] [Commented] (SPARK-16233) test_sparkSQL.R is failing
[
https://issues.apache.org/jira/browse/SPARK-16233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15359725#comment-15359725
]
Apache Spark commented on SPARK-16233:
--
User 'dongjoon-hyun' has created a pull request for this issue:
https://github.com/apache/spark/pull/14019
> test_sparkSQL.R is failing
> --
>
> Key: SPARK-16233
> URL: https://issues.apache.org/jira/browse/SPARK-16233
> Project: Spark
> Issue Type: Bug
> Components: SparkR, Tests
>Affects Versions: 2.0.0
>Reporter: Xin Ren
>Priority: Minor
>
> By running
> {code}
> ./R/run-tests.sh
> {code}
> Getting error:
> {code}
> xin:spark xr$ ./R/run-tests.sh
> Warning: Ignoring non-spark config property: SPARK_SCALA_VERSION=2.11
> Loading required package: methods
> Attaching package: ‘SparkR’
> The following object is masked from ‘package:testthat’:
> describe
> The following objects are masked from ‘package:stats’:
> cov, filter, lag, na.omit, predict, sd, var, window
> The following objects are masked from ‘package:base’:
> as.data.frame, colnames, colnames<-, drop, endsWith, intersect,
> rank, rbind, sample, startsWith, subset, summary, transform, union
> binary functions: ...
> functions on binary files:
> broadcast variables: ..
> functions in client.R: .
> test functions in sparkR.R: .Re-using existing Spark Context. Call
> sparkR.session.stop() or restart R to create a new Spark Context
> Re-using existing Spark Context. Call sparkR.session.stop() or restart R
> to create a new Spark Context
> ...
> include an external JAR in SparkContext: Warning: Ignoring non-spark config
> property: SPARK_SCALA_VERSION=2.11
> ..
> include R packages:
> MLlib functions: .SLF4J: Failed to load class
> "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: Defaulting to no-operation (NOP) logger implementation
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
> details.
> .27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.codec.CodecConfig:
> Compression: SNAPPY
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet block size to 134217728
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet page size to 1048576
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet dictionary page size to 1048576
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Dictionary is on
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Validation is off
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Writer version is: PARQUET_1_0
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Maximum row group padding size is 0 bytes
> 27-Jun-2016 1:51:25 PM INFO:
> org.apache.parquet.hadoop.InternalParquetRecordWriter: Flushing mem
> columnStore to file. allocated memory: 65,622
> 27-Jun-2016 1:51:25 PM INFO:
> org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written 70B for [label]
> BINARY: 1 values, 21B raw, 23B comp, 1 pages, encodings: [PLAIN, RLE,
> BIT_PACKED]
> 27-Jun-2016 1:51:25 PM INFO:
> org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written 87B for [terms,
> list, element, list, element] BINARY: 2 values, 42B raw, 43B comp, 1 pages,
> encodings: [PLAIN, RLE]
> 27-Jun-2016 1:51:25 PM INFO:
> org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written 30B for
> [hasIntercept] BOOLEAN: 1 values, 1B raw, 3B comp, 1 pages, encodings:
> [PLAIN, BIT_PACKED]
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.codec.CodecConfig:
> Compression: SNAPPY
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet block size to 134217728
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet page size to 1048576
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet dictionary page size to 1048576
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Dictionary is on
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Validation is off
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Writer version is: PARQUET_1_0
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Maximum row group padding size is 0 bytes
> 27-Jun-2016 1:51:26 PM INFO:
> org.apache.parquet.hadoop.InternalParquetRecordWriter: Flushing mem
> columnStore to file. allocated memory: 49
> 27-Jun-2016 1:51:26 PM INFO:
> org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written 90B for [labels,
> list, element] BINARY: 3 values, 50B raw, 50B comp, 1 pages, encodings:
[jira] [Commented] (SPARK-16233) test_sparkSQL.R is failing
[
https://issues.apache.org/jira/browse/SPARK-16233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15359717#comment-15359717
]
Dongjoon Hyun commented on SPARK-16233:
---
hi, [~shivaram].
I made a PR for this.
> test_sparkSQL.R is failing
> --
>
> Key: SPARK-16233
> URL: https://issues.apache.org/jira/browse/SPARK-16233
> Project: Spark
> Issue Type: Bug
> Components: SparkR, Tests
>Affects Versions: 2.0.0
>Reporter: Xin Ren
>Priority: Minor
>
> By running
> {code}
> ./R/run-tests.sh
> {code}
> Getting error:
> {code}
> xin:spark xr$ ./R/run-tests.sh
> Warning: Ignoring non-spark config property: SPARK_SCALA_VERSION=2.11
> Loading required package: methods
> Attaching package: ‘SparkR’
> The following object is masked from ‘package:testthat’:
> describe
> The following objects are masked from ‘package:stats’:
> cov, filter, lag, na.omit, predict, sd, var, window
> The following objects are masked from ‘package:base’:
> as.data.frame, colnames, colnames<-, drop, endsWith, intersect,
> rank, rbind, sample, startsWith, subset, summary, transform, union
> binary functions: ...
> functions on binary files:
> broadcast variables: ..
> functions in client.R: .
> test functions in sparkR.R: .Re-using existing Spark Context. Call
> sparkR.session.stop() or restart R to create a new Spark Context
> Re-using existing Spark Context. Call sparkR.session.stop() or restart R
> to create a new Spark Context
> ...
> include an external JAR in SparkContext: Warning: Ignoring non-spark config
> property: SPARK_SCALA_VERSION=2.11
> ..
> include R packages:
> MLlib functions: .SLF4J: Failed to load class
> "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: Defaulting to no-operation (NOP) logger implementation
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
> details.
> .27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.codec.CodecConfig:
> Compression: SNAPPY
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet block size to 134217728
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet page size to 1048576
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet dictionary page size to 1048576
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Dictionary is on
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Validation is off
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Writer version is: PARQUET_1_0
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Maximum row group padding size is 0 bytes
> 27-Jun-2016 1:51:25 PM INFO:
> org.apache.parquet.hadoop.InternalParquetRecordWriter: Flushing mem
> columnStore to file. allocated memory: 65,622
> 27-Jun-2016 1:51:25 PM INFO:
> org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written 70B for [label]
> BINARY: 1 values, 21B raw, 23B comp, 1 pages, encodings: [PLAIN, RLE,
> BIT_PACKED]
> 27-Jun-2016 1:51:25 PM INFO:
> org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written 87B for [terms,
> list, element, list, element] BINARY: 2 values, 42B raw, 43B comp, 1 pages,
> encodings: [PLAIN, RLE]
> 27-Jun-2016 1:51:25 PM INFO:
> org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written 30B for
> [hasIntercept] BOOLEAN: 1 values, 1B raw, 3B comp, 1 pages, encodings:
> [PLAIN, BIT_PACKED]
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.codec.CodecConfig:
> Compression: SNAPPY
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet block size to 134217728
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet page size to 1048576
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet dictionary page size to 1048576
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Dictionary is on
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Validation is off
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Writer version is: PARQUET_1_0
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Maximum row group padding size is 0 bytes
> 27-Jun-2016 1:51:26 PM INFO:
> org.apache.parquet.hadoop.InternalParquetRecordWriter: Flushing mem
> columnStore to file. allocated memory: 49
> 27-Jun-2016 1:51:26 PM INFO:
> org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written 90B for [labels,
> list, element] BINARY: 3 values, 50B raw, 50B comp, 1 pages, encodings:
> [PLAIN, RLE]
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet
[jira] [Commented] (SPARK-16233) test_sparkSQL.R is failing
[
https://issues.apache.org/jira/browse/SPARK-16233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15359683#comment-15359683
]
Dongjoon Hyun commented on SPARK-16233:
---
Oh, Sure.
I though [~iamshrek] was woking on this issue.
I'll make a PR today.
> test_sparkSQL.R is failing
> --
>
> Key: SPARK-16233
> URL: https://issues.apache.org/jira/browse/SPARK-16233
> Project: Spark
> Issue Type: Bug
> Components: SparkR, Tests
>Affects Versions: 2.0.0
>Reporter: Xin Ren
>Priority: Minor
>
> By running
> {code}
> ./R/run-tests.sh
> {code}
> Getting error:
> {code}
> xin:spark xr$ ./R/run-tests.sh
> Warning: Ignoring non-spark config property: SPARK_SCALA_VERSION=2.11
> Loading required package: methods
> Attaching package: ‘SparkR’
> The following object is masked from ‘package:testthat’:
> describe
> The following objects are masked from ‘package:stats’:
> cov, filter, lag, na.omit, predict, sd, var, window
> The following objects are masked from ‘package:base’:
> as.data.frame, colnames, colnames<-, drop, endsWith, intersect,
> rank, rbind, sample, startsWith, subset, summary, transform, union
> binary functions: ...
> functions on binary files:
> broadcast variables: ..
> functions in client.R: .
> test functions in sparkR.R: .Re-using existing Spark Context. Call
> sparkR.session.stop() or restart R to create a new Spark Context
> Re-using existing Spark Context. Call sparkR.session.stop() or restart R
> to create a new Spark Context
> ...
> include an external JAR in SparkContext: Warning: Ignoring non-spark config
> property: SPARK_SCALA_VERSION=2.11
> ..
> include R packages:
> MLlib functions: .SLF4J: Failed to load class
> "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: Defaulting to no-operation (NOP) logger implementation
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
> details.
> .27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.codec.CodecConfig:
> Compression: SNAPPY
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet block size to 134217728
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet page size to 1048576
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet dictionary page size to 1048576
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Dictionary is on
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Validation is off
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Writer version is: PARQUET_1_0
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Maximum row group padding size is 0 bytes
> 27-Jun-2016 1:51:25 PM INFO:
> org.apache.parquet.hadoop.InternalParquetRecordWriter: Flushing mem
> columnStore to file. allocated memory: 65,622
> 27-Jun-2016 1:51:25 PM INFO:
> org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written 70B for [label]
> BINARY: 1 values, 21B raw, 23B comp, 1 pages, encodings: [PLAIN, RLE,
> BIT_PACKED]
> 27-Jun-2016 1:51:25 PM INFO:
> org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written 87B for [terms,
> list, element, list, element] BINARY: 2 values, 42B raw, 43B comp, 1 pages,
> encodings: [PLAIN, RLE]
> 27-Jun-2016 1:51:25 PM INFO:
> org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written 30B for
> [hasIntercept] BOOLEAN: 1 values, 1B raw, 3B comp, 1 pages, encodings:
> [PLAIN, BIT_PACKED]
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.codec.CodecConfig:
> Compression: SNAPPY
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet block size to 134217728
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet page size to 1048576
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet dictionary page size to 1048576
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Dictionary is on
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Validation is off
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Writer version is: PARQUET_1_0
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Maximum row group padding size is 0 bytes
> 27-Jun-2016 1:51:26 PM INFO:
> org.apache.parquet.hadoop.InternalParquetRecordWriter: Flushing mem
> columnStore to file. allocated memory: 49
> 27-Jun-2016 1:51:26 PM INFO:
> org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written 90B for [labels,
> list, element] BINARY: 3 values, 50B raw, 50B comp, 1 pages, encodings:
> [PLAIN, RLE]
> 27-Jun-
[jira] [Commented] (SPARK-16233) test_sparkSQL.R is failing
[
https://issues.apache.org/jira/browse/SPARK-16233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15359661#comment-15359661
]
Shivaram Venkataraman commented on SPARK-16233:
---
[~dongjoon] Thanks for taking a look. Can you send a PR to ignore this test for
now ?
> test_sparkSQL.R is failing
> --
>
> Key: SPARK-16233
> URL: https://issues.apache.org/jira/browse/SPARK-16233
> Project: Spark
> Issue Type: Bug
> Components: SparkR, Tests
>Affects Versions: 2.0.0
>Reporter: Xin Ren
>Priority: Minor
>
> By running
> {code}
> ./R/run-tests.sh
> {code}
> Getting error:
> {code}
> xin:spark xr$ ./R/run-tests.sh
> Warning: Ignoring non-spark config property: SPARK_SCALA_VERSION=2.11
> Loading required package: methods
> Attaching package: ‘SparkR’
> The following object is masked from ‘package:testthat’:
> describe
> The following objects are masked from ‘package:stats’:
> cov, filter, lag, na.omit, predict, sd, var, window
> The following objects are masked from ‘package:base’:
> as.data.frame, colnames, colnames<-, drop, endsWith, intersect,
> rank, rbind, sample, startsWith, subset, summary, transform, union
> binary functions: ...
> functions on binary files:
> broadcast variables: ..
> functions in client.R: .
> test functions in sparkR.R: .Re-using existing Spark Context. Call
> sparkR.session.stop() or restart R to create a new Spark Context
> Re-using existing Spark Context. Call sparkR.session.stop() or restart R
> to create a new Spark Context
> ...
> include an external JAR in SparkContext: Warning: Ignoring non-spark config
> property: SPARK_SCALA_VERSION=2.11
> ..
> include R packages:
> MLlib functions: .SLF4J: Failed to load class
> "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: Defaulting to no-operation (NOP) logger implementation
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
> details.
> .27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.codec.CodecConfig:
> Compression: SNAPPY
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet block size to 134217728
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet page size to 1048576
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet dictionary page size to 1048576
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Dictionary is on
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Validation is off
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Writer version is: PARQUET_1_0
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Maximum row group padding size is 0 bytes
> 27-Jun-2016 1:51:25 PM INFO:
> org.apache.parquet.hadoop.InternalParquetRecordWriter: Flushing mem
> columnStore to file. allocated memory: 65,622
> 27-Jun-2016 1:51:25 PM INFO:
> org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written 70B for [label]
> BINARY: 1 values, 21B raw, 23B comp, 1 pages, encodings: [PLAIN, RLE,
> BIT_PACKED]
> 27-Jun-2016 1:51:25 PM INFO:
> org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written 87B for [terms,
> list, element, list, element] BINARY: 2 values, 42B raw, 43B comp, 1 pages,
> encodings: [PLAIN, RLE]
> 27-Jun-2016 1:51:25 PM INFO:
> org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written 30B for
> [hasIntercept] BOOLEAN: 1 values, 1B raw, 3B comp, 1 pages, encodings:
> [PLAIN, BIT_PACKED]
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.codec.CodecConfig:
> Compression: SNAPPY
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet block size to 134217728
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet page size to 1048576
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet dictionary page size to 1048576
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Dictionary is on
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Validation is off
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Writer version is: PARQUET_1_0
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Maximum row group padding size is 0 bytes
> 27-Jun-2016 1:51:26 PM INFO:
> org.apache.parquet.hadoop.InternalParquetRecordWriter: Flushing mem
> columnStore to file. allocated memory: 49
> 27-Jun-2016 1:51:26 PM INFO:
> org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written 90B for [labels,
> list, element] BINARY: 3 values, 50B raw, 50B comp, 1 pages, encodings:
[jira] [Commented] (SPARK-16233) test_sparkSQL.R is failing
[
https://issues.apache.org/jira/browse/SPARK-16233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15353405#comment-15353405
]
Dongjoon Hyun commented on SPARK-16233:
---
Hi, [~shivaram].
Actually, in `DataFrameReaderWriterSuite.scala`, the ORC test is also marked
`ignored`.
https://github.com/apache/spark/blob/master/sql/core/src/test/scala/org/apache/spark/sql/test/DataFrameReaderWriterSuite.scala#L353
All the other ORC tests are in `spark/sql/hive` packages. I think we can make
that test just as `ignored` like Scala testsuite.
> test_sparkSQL.R is failing
> --
>
> Key: SPARK-16233
> URL: https://issues.apache.org/jira/browse/SPARK-16233
> Project: Spark
> Issue Type: Bug
> Components: SparkR, Tests
>Affects Versions: 2.0.0
>Reporter: Xin Ren
>Priority: Minor
>
> By running
> {code}
> ./R/run-tests.sh
> {code}
> Getting error:
> {code}
> xin:spark xr$ ./R/run-tests.sh
> Warning: Ignoring non-spark config property: SPARK_SCALA_VERSION=2.11
> Loading required package: methods
> Attaching package: ‘SparkR’
> The following object is masked from ‘package:testthat’:
> describe
> The following objects are masked from ‘package:stats’:
> cov, filter, lag, na.omit, predict, sd, var, window
> The following objects are masked from ‘package:base’:
> as.data.frame, colnames, colnames<-, drop, endsWith, intersect,
> rank, rbind, sample, startsWith, subset, summary, transform, union
> binary functions: ...
> functions on binary files:
> broadcast variables: ..
> functions in client.R: .
> test functions in sparkR.R: .Re-using existing Spark Context. Call
> sparkR.session.stop() or restart R to create a new Spark Context
> Re-using existing Spark Context. Call sparkR.session.stop() or restart R
> to create a new Spark Context
> ...
> include an external JAR in SparkContext: Warning: Ignoring non-spark config
> property: SPARK_SCALA_VERSION=2.11
> ..
> include R packages:
> MLlib functions: .SLF4J: Failed to load class
> "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: Defaulting to no-operation (NOP) logger implementation
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
> details.
> .27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.codec.CodecConfig:
> Compression: SNAPPY
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet block size to 134217728
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet page size to 1048576
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet dictionary page size to 1048576
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Dictionary is on
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Validation is off
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Writer version is: PARQUET_1_0
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Maximum row group padding size is 0 bytes
> 27-Jun-2016 1:51:25 PM INFO:
> org.apache.parquet.hadoop.InternalParquetRecordWriter: Flushing mem
> columnStore to file. allocated memory: 65,622
> 27-Jun-2016 1:51:25 PM INFO:
> org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written 70B for [label]
> BINARY: 1 values, 21B raw, 23B comp, 1 pages, encodings: [PLAIN, RLE,
> BIT_PACKED]
> 27-Jun-2016 1:51:25 PM INFO:
> org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written 87B for [terms,
> list, element, list, element] BINARY: 2 values, 42B raw, 43B comp, 1 pages,
> encodings: [PLAIN, RLE]
> 27-Jun-2016 1:51:25 PM INFO:
> org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written 30B for
> [hasIntercept] BOOLEAN: 1 values, 1B raw, 3B comp, 1 pages, encodings:
> [PLAIN, BIT_PACKED]
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.codec.CodecConfig:
> Compression: SNAPPY
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet block size to 134217728
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet page size to 1048576
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet dictionary page size to 1048576
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Dictionary is on
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Validation is off
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Writer version is: PARQUET_1_0
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Maximum row group padding size is 0 bytes
> 27-Jun-2016 1:51:26 PM INFO:
> org.apache.parquet.hadoop.InternalP
[jira] [Commented] (SPARK-16233) test_sparkSQL.R is failing
[
https://issues.apache.org/jira/browse/SPARK-16233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15353388#comment-15353388
]
Dongjoon Hyun commented on SPARK-16233:
---
Oh, thank you for pinging me. For this issue, correctly, your concern is right.
My bads. That will be the root cause.
> test_sparkSQL.R is failing
> --
>
> Key: SPARK-16233
> URL: https://issues.apache.org/jira/browse/SPARK-16233
> Project: Spark
> Issue Type: Bug
> Components: SparkR, Tests
>Affects Versions: 2.0.0
>Reporter: Xin Ren
>Priority: Minor
>
> By running
> {code}
> ./R/run-tests.sh
> {code}
> Getting error:
> {code}
> xin:spark xr$ ./R/run-tests.sh
> Warning: Ignoring non-spark config property: SPARK_SCALA_VERSION=2.11
> Loading required package: methods
> Attaching package: ‘SparkR’
> The following object is masked from ‘package:testthat’:
> describe
> The following objects are masked from ‘package:stats’:
> cov, filter, lag, na.omit, predict, sd, var, window
> The following objects are masked from ‘package:base’:
> as.data.frame, colnames, colnames<-, drop, endsWith, intersect,
> rank, rbind, sample, startsWith, subset, summary, transform, union
> binary functions: ...
> functions on binary files:
> broadcast variables: ..
> functions in client.R: .
> test functions in sparkR.R: .Re-using existing Spark Context. Call
> sparkR.session.stop() or restart R to create a new Spark Context
> Re-using existing Spark Context. Call sparkR.session.stop() or restart R
> to create a new Spark Context
> ...
> include an external JAR in SparkContext: Warning: Ignoring non-spark config
> property: SPARK_SCALA_VERSION=2.11
> ..
> include R packages:
> MLlib functions: .SLF4J: Failed to load class
> "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: Defaulting to no-operation (NOP) logger implementation
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
> details.
> .27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.codec.CodecConfig:
> Compression: SNAPPY
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet block size to 134217728
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet page size to 1048576
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet dictionary page size to 1048576
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Dictionary is on
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Validation is off
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Writer version is: PARQUET_1_0
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Maximum row group padding size is 0 bytes
> 27-Jun-2016 1:51:25 PM INFO:
> org.apache.parquet.hadoop.InternalParquetRecordWriter: Flushing mem
> columnStore to file. allocated memory: 65,622
> 27-Jun-2016 1:51:25 PM INFO:
> org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written 70B for [label]
> BINARY: 1 values, 21B raw, 23B comp, 1 pages, encodings: [PLAIN, RLE,
> BIT_PACKED]
> 27-Jun-2016 1:51:25 PM INFO:
> org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written 87B for [terms,
> list, element, list, element] BINARY: 2 values, 42B raw, 43B comp, 1 pages,
> encodings: [PLAIN, RLE]
> 27-Jun-2016 1:51:25 PM INFO:
> org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written 30B for
> [hasIntercept] BOOLEAN: 1 values, 1B raw, 3B comp, 1 pages, encodings:
> [PLAIN, BIT_PACKED]
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.codec.CodecConfig:
> Compression: SNAPPY
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet block size to 134217728
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet page size to 1048576
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet dictionary page size to 1048576
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Dictionary is on
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Validation is off
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Writer version is: PARQUET_1_0
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Maximum row group padding size is 0 bytes
> 27-Jun-2016 1:51:26 PM INFO:
> org.apache.parquet.hadoop.InternalParquetRecordWriter: Flushing mem
> columnStore to file. allocated memory: 49
> 27-Jun-2016 1:51:26 PM INFO:
> org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written 90B for [labels,
> list, element] BINARY: 3 values, 50B raw, 50B comp, 1 page
[jira] [Commented] (SPARK-16233) test_sparkSQL.R is failing
[
https://issues.apache.org/jira/browse/SPARK-16233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15353352#comment-15353352
]
Xin Ren commented on SPARK-16233:
-
Actually I was just following the docs here
https://github.com/keypointt/spark/tree/master/R#examples-unit-tests
Maybe we should update the docs here to point it out that "-Phive" could be
needed?
{code}
build/mvn -DskipTests -Psparkr package
{code}
{code}
You can also run the unit tests for SparkR by running. You need to install the
testthat package first:
R -e 'install.packages("testthat", repos="http://cran.us.r-project.org";)'
./R/run-tests.sh
{code}
> test_sparkSQL.R is failing
> --
>
> Key: SPARK-16233
> URL: https://issues.apache.org/jira/browse/SPARK-16233
> Project: Spark
> Issue Type: Bug
> Components: SparkR, Tests
>Affects Versions: 2.0.0
>Reporter: Xin Ren
>Priority: Minor
>
> By running
> {code}
> ./R/run-tests.sh
> {code}
> Getting error:
> {code}
> xin:spark xr$ ./R/run-tests.sh
> Warning: Ignoring non-spark config property: SPARK_SCALA_VERSION=2.11
> Loading required package: methods
> Attaching package: ‘SparkR’
> The following object is masked from ‘package:testthat’:
> describe
> The following objects are masked from ‘package:stats’:
> cov, filter, lag, na.omit, predict, sd, var, window
> The following objects are masked from ‘package:base’:
> as.data.frame, colnames, colnames<-, drop, endsWith, intersect,
> rank, rbind, sample, startsWith, subset, summary, transform, union
> binary functions: ...
> functions on binary files:
> broadcast variables: ..
> functions in client.R: .
> test functions in sparkR.R: .Re-using existing Spark Context. Call
> sparkR.session.stop() or restart R to create a new Spark Context
> Re-using existing Spark Context. Call sparkR.session.stop() or restart R
> to create a new Spark Context
> ...
> include an external JAR in SparkContext: Warning: Ignoring non-spark config
> property: SPARK_SCALA_VERSION=2.11
> ..
> include R packages:
> MLlib functions: .SLF4J: Failed to load class
> "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: Defaulting to no-operation (NOP) logger implementation
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
> details.
> .27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.codec.CodecConfig:
> Compression: SNAPPY
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet block size to 134217728
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet page size to 1048576
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet dictionary page size to 1048576
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Dictionary is on
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Validation is off
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Writer version is: PARQUET_1_0
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Maximum row group padding size is 0 bytes
> 27-Jun-2016 1:51:25 PM INFO:
> org.apache.parquet.hadoop.InternalParquetRecordWriter: Flushing mem
> columnStore to file. allocated memory: 65,622
> 27-Jun-2016 1:51:25 PM INFO:
> org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written 70B for [label]
> BINARY: 1 values, 21B raw, 23B comp, 1 pages, encodings: [PLAIN, RLE,
> BIT_PACKED]
> 27-Jun-2016 1:51:25 PM INFO:
> org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written 87B for [terms,
> list, element, list, element] BINARY: 2 values, 42B raw, 43B comp, 1 pages,
> encodings: [PLAIN, RLE]
> 27-Jun-2016 1:51:25 PM INFO:
> org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written 30B for
> [hasIntercept] BOOLEAN: 1 values, 1B raw, 3B comp, 1 pages, encodings:
> [PLAIN, BIT_PACKED]
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.codec.CodecConfig:
> Compression: SNAPPY
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet block size to 134217728
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet page size to 1048576
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet dictionary page size to 1048576
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Dictionary is on
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Validation is off
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Writer version is: PARQUET_1_0
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Maximum row group padding s
[jira] [Commented] (SPARK-16233) test_sparkSQL.R is failing
[
https://issues.apache.org/jira/browse/SPARK-16233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15353344#comment-15353344
]
Shivaram Venkataraman commented on SPARK-16233:
---
Ideally we should skip the tests which depend on Hive if SparkR was built
without Hive. Was there some recent change in how ORC files are handled ?
cc [~dongjoon] who added read.orc, write.orc support
> test_sparkSQL.R is failing
> --
>
> Key: SPARK-16233
> URL: https://issues.apache.org/jira/browse/SPARK-16233
> Project: Spark
> Issue Type: Bug
> Components: SparkR, Tests
>Affects Versions: 2.0.0
>Reporter: Xin Ren
>Priority: Minor
>
> By running
> {code}
> ./R/run-tests.sh
> {code}
> Getting error:
> {code}
> xin:spark xr$ ./R/run-tests.sh
> Warning: Ignoring non-spark config property: SPARK_SCALA_VERSION=2.11
> Loading required package: methods
> Attaching package: ‘SparkR’
> The following object is masked from ‘package:testthat’:
> describe
> The following objects are masked from ‘package:stats’:
> cov, filter, lag, na.omit, predict, sd, var, window
> The following objects are masked from ‘package:base’:
> as.data.frame, colnames, colnames<-, drop, endsWith, intersect,
> rank, rbind, sample, startsWith, subset, summary, transform, union
> binary functions: ...
> functions on binary files:
> broadcast variables: ..
> functions in client.R: .
> test functions in sparkR.R: .Re-using existing Spark Context. Call
> sparkR.session.stop() or restart R to create a new Spark Context
> Re-using existing Spark Context. Call sparkR.session.stop() or restart R
> to create a new Spark Context
> ...
> include an external JAR in SparkContext: Warning: Ignoring non-spark config
> property: SPARK_SCALA_VERSION=2.11
> ..
> include R packages:
> MLlib functions: .SLF4J: Failed to load class
> "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: Defaulting to no-operation (NOP) logger implementation
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
> details.
> .27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.codec.CodecConfig:
> Compression: SNAPPY
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet block size to 134217728
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet page size to 1048576
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet dictionary page size to 1048576
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Dictionary is on
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Validation is off
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Writer version is: PARQUET_1_0
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Maximum row group padding size is 0 bytes
> 27-Jun-2016 1:51:25 PM INFO:
> org.apache.parquet.hadoop.InternalParquetRecordWriter: Flushing mem
> columnStore to file. allocated memory: 65,622
> 27-Jun-2016 1:51:25 PM INFO:
> org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written 70B for [label]
> BINARY: 1 values, 21B raw, 23B comp, 1 pages, encodings: [PLAIN, RLE,
> BIT_PACKED]
> 27-Jun-2016 1:51:25 PM INFO:
> org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written 87B for [terms,
> list, element, list, element] BINARY: 2 values, 42B raw, 43B comp, 1 pages,
> encodings: [PLAIN, RLE]
> 27-Jun-2016 1:51:25 PM INFO:
> org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written 30B for
> [hasIntercept] BOOLEAN: 1 values, 1B raw, 3B comp, 1 pages, encodings:
> [PLAIN, BIT_PACKED]
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.codec.CodecConfig:
> Compression: SNAPPY
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet block size to 134217728
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet page size to 1048576
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet dictionary page size to 1048576
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Dictionary is on
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Validation is off
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Writer version is: PARQUET_1_0
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Maximum row group padding size is 0 bytes
> 27-Jun-2016 1:51:26 PM INFO:
> org.apache.parquet.hadoop.InternalParquetRecordWriter: Flushing mem
> columnStore to file. allocated memory: 49
> 27-Jun-2016 1:51:26 PM INFO:
> org.apache.parquet.hadoop.ColumnChunk
[jira] [Commented] (SPARK-16233) test_sparkSQL.R is failing
[
https://issues.apache.org/jira/browse/SPARK-16233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15353340#comment-15353340
]
Xin Ren commented on SPARK-16233:
-
this is what I used to build sparkR, should I add "-Phive"? sorry I'm new to
this part.
{code}
build/mvn -DskipTests -Psparkr package
{code}
> test_sparkSQL.R is failing
> --
>
> Key: SPARK-16233
> URL: https://issues.apache.org/jira/browse/SPARK-16233
> Project: Spark
> Issue Type: Bug
> Components: SparkR, Tests
>Affects Versions: 2.0.0
>Reporter: Xin Ren
>Priority: Minor
>
> By running
> {code}
> ./R/run-tests.sh
> {code}
> Getting error:
> {code}
> xin:spark xr$ ./R/run-tests.sh
> Warning: Ignoring non-spark config property: SPARK_SCALA_VERSION=2.11
> Loading required package: methods
> Attaching package: ‘SparkR’
> The following object is masked from ‘package:testthat’:
> describe
> The following objects are masked from ‘package:stats’:
> cov, filter, lag, na.omit, predict, sd, var, window
> The following objects are masked from ‘package:base’:
> as.data.frame, colnames, colnames<-, drop, endsWith, intersect,
> rank, rbind, sample, startsWith, subset, summary, transform, union
> binary functions: ...
> functions on binary files:
> broadcast variables: ..
> functions in client.R: .
> test functions in sparkR.R: .Re-using existing Spark Context. Call
> sparkR.session.stop() or restart R to create a new Spark Context
> Re-using existing Spark Context. Call sparkR.session.stop() or restart R
> to create a new Spark Context
> ...
> include an external JAR in SparkContext: Warning: Ignoring non-spark config
> property: SPARK_SCALA_VERSION=2.11
> ..
> include R packages:
> MLlib functions: .SLF4J: Failed to load class
> "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: Defaulting to no-operation (NOP) logger implementation
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
> details.
> .27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.codec.CodecConfig:
> Compression: SNAPPY
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet block size to 134217728
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet page size to 1048576
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet dictionary page size to 1048576
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Dictionary is on
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Validation is off
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Writer version is: PARQUET_1_0
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Maximum row group padding size is 0 bytes
> 27-Jun-2016 1:51:25 PM INFO:
> org.apache.parquet.hadoop.InternalParquetRecordWriter: Flushing mem
> columnStore to file. allocated memory: 65,622
> 27-Jun-2016 1:51:25 PM INFO:
> org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written 70B for [label]
> BINARY: 1 values, 21B raw, 23B comp, 1 pages, encodings: [PLAIN, RLE,
> BIT_PACKED]
> 27-Jun-2016 1:51:25 PM INFO:
> org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written 87B for [terms,
> list, element, list, element] BINARY: 2 values, 42B raw, 43B comp, 1 pages,
> encodings: [PLAIN, RLE]
> 27-Jun-2016 1:51:25 PM INFO:
> org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written 30B for
> [hasIntercept] BOOLEAN: 1 values, 1B raw, 3B comp, 1 pages, encodings:
> [PLAIN, BIT_PACKED]
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.codec.CodecConfig:
> Compression: SNAPPY
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet block size to 134217728
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet page size to 1048576
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet dictionary page size to 1048576
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Dictionary is on
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Validation is off
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Writer version is: PARQUET_1_0
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Maximum row group padding size is 0 bytes
> 27-Jun-2016 1:51:26 PM INFO:
> org.apache.parquet.hadoop.InternalParquetRecordWriter: Flushing mem
> columnStore to file. allocated memory: 49
> 27-Jun-2016 1:51:26 PM INFO:
> org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written 90B for [labels,
> list, element] BINARY: 3 values, 50B raw, 50B
[jira] [Commented] (SPARK-16233) test_sparkSQL.R is failing
[
https://issues.apache.org/jira/browse/SPARK-16233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15352885#comment-15352885
]
Prashant Sharma commented on SPARK-16233:
-
Did you build spark with -Phive ?
> test_sparkSQL.R is failing
> --
>
> Key: SPARK-16233
> URL: https://issues.apache.org/jira/browse/SPARK-16233
> Project: Spark
> Issue Type: Bug
> Components: SparkR, Tests
>Affects Versions: 2.0.0
>Reporter: Xin Ren
>Priority: Minor
>
> By running
> {code}
> ./R/run-tests.sh
> {code}
> Getting error:
> {code}
> xin:spark xr$ ./R/run-tests.sh
> Warning: Ignoring non-spark config property: SPARK_SCALA_VERSION=2.11
> Loading required package: methods
> Attaching package: ‘SparkR’
> The following object is masked from ‘package:testthat’:
> describe
> The following objects are masked from ‘package:stats’:
> cov, filter, lag, na.omit, predict, sd, var, window
> The following objects are masked from ‘package:base’:
> as.data.frame, colnames, colnames<-, drop, endsWith, intersect,
> rank, rbind, sample, startsWith, subset, summary, transform, union
> binary functions: ...
> functions on binary files:
> broadcast variables: ..
> functions in client.R: .
> test functions in sparkR.R: .Re-using existing Spark Context. Call
> sparkR.session.stop() or restart R to create a new Spark Context
> Re-using existing Spark Context. Call sparkR.session.stop() or restart R
> to create a new Spark Context
> ...
> include an external JAR in SparkContext: Warning: Ignoring non-spark config
> property: SPARK_SCALA_VERSION=2.11
> ..
> include R packages:
> MLlib functions: .SLF4J: Failed to load class
> "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: Defaulting to no-operation (NOP) logger implementation
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
> details.
> .27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.codec.CodecConfig:
> Compression: SNAPPY
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet block size to 134217728
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet page size to 1048576
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet dictionary page size to 1048576
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Dictionary is on
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Validation is off
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Writer version is: PARQUET_1_0
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Maximum row group padding size is 0 bytes
> 27-Jun-2016 1:51:25 PM INFO:
> org.apache.parquet.hadoop.InternalParquetRecordWriter: Flushing mem
> columnStore to file. allocated memory: 65,622
> 27-Jun-2016 1:51:25 PM INFO:
> org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written 70B for [label]
> BINARY: 1 values, 21B raw, 23B comp, 1 pages, encodings: [PLAIN, RLE,
> BIT_PACKED]
> 27-Jun-2016 1:51:25 PM INFO:
> org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written 87B for [terms,
> list, element, list, element] BINARY: 2 values, 42B raw, 43B comp, 1 pages,
> encodings: [PLAIN, RLE]
> 27-Jun-2016 1:51:25 PM INFO:
> org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written 30B for
> [hasIntercept] BOOLEAN: 1 values, 1B raw, 3B comp, 1 pages, encodings:
> [PLAIN, BIT_PACKED]
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.codec.CodecConfig:
> Compression: SNAPPY
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet block size to 134217728
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet page size to 1048576
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Parquet dictionary page size to 1048576
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Dictionary is on
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Validation is off
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Writer version is: PARQUET_1_0
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat:
> Maximum row group padding size is 0 bytes
> 27-Jun-2016 1:51:26 PM INFO:
> org.apache.parquet.hadoop.InternalParquetRecordWriter: Flushing mem
> columnStore to file. allocated memory: 49
> 27-Jun-2016 1:51:26 PM INFO:
> org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written 90B for [labels,
> list, element] BINARY: 3 values, 50B raw, 50B comp, 1 pages, encodings:
> [PLAIN, RLE]
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.
[jira] [Commented] (SPARK-16233) test_sparkSQL.R is failing
[
https://issues.apache.org/jira/browse/SPARK-16233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15351703#comment-15351703
]
Xin Ren commented on SPARK-16233:
-
I'm working on this
> test_sparkSQL.R is failing
> --
>
> Key: SPARK-16233
> URL: https://issues.apache.org/jira/browse/SPARK-16233
> Project: Spark
> Issue Type: Bug
> Components: SparkR, Tests
>Affects Versions: 2.0.0
>Reporter: Xin Ren
>Priority: Minor
>
> By running
> {code}
> ./R/run-tests.sh
> {code}
> Getting error:
> {code}
> 15. Error: create DataFrame from list or data.frame (@test_sparkSQL.R#277)
> -
> java.lang.NoClassDefFoundorg/apache/spark/sql/execution/datasources/PreInsertCastAndRename$
> at
> org.apache.spark.sql.hive.HiveSessionState$$anon$1.(HiveSessionState.scala:69)
> at
> org.apache.spark.sql.hive.HiveSessionState.analyzer$lzycompute(HiveSessionState.scala:63)
> at
> org.apache.spark.sql.hive.HiveSessionState.analyzer(HiveSessionState.scala:62)
> at
> org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:49)
> at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:64)
> at
> org.apache.spark.sql.SparkSession.createDataFrame(SparkSession.scala:533)
> at
> org.apache.spark.sql.SparkSession.createDataFrame(SparkSession.scala:293)
> at org.apache.spark.sql.api.r.SQLUtils$.createDF(SQLUtils.scala:135)
> at org.apache.spark.sql.api.r.SQLUtils.createDF(SQLUtils.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at
> org.apache.spark.api.r.RBackendHandler.handleMethodCall(RBackendHandler.scala:141)
> at
> org.apache.spark.api.r.RBackendHandler.channelRead0(RBackendHandler.scala:86)
> at
> org.apache.spark.api.r.RBackendHandler.channelRead0(RBackendHandler.scala:38)
> at
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
> at
> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:244)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
> at
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846)
> at
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
> at
> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
> at java.lang.Thread.run(Thread.java:745)
> 1: createDataFrame(l, c("a", "b")) at
> /Users/quickmobile/workspace/spark/R/lib/SparkR/tests/testthat/test_sparkSQL.R:277
> 2: dispatchFunc("createDataFrame(data, schema = NULL, samplingRatio = 1.0)",
> x, ...)
> 3: f(x, ...)
> 4: callJStatic("org.apache.spark.sql.api.r.SQLUtils", "createDF", srdd,
> schema$jobj,
>sparkSession)
> 5: invokeJava(isStatic = TRUE, className, methodName, ...)
> 6: stop(readString(conn))
> DONE
> ===
> Execution halted
> {code}
> Cause: most probably these tests are using 'createDataFrame(sqlContext...)'
> which is deprecated. Should update tests method invocations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.a
