[ https://issues.apache.org/jira/browse/SPARK-16437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15372210#comment-15372210 ]
Xin Ren edited comment on SPARK-16437 at 7/12/16 6:04 PM: ---------------------------------------------------------- I worked on this for couple days, and I found it's not caused by Spark, but the parquet library "parquet-mr/parquet-hadoop". I've debug by step, and found this error is from here: https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala#L820 and after digging into "parquet-hadoop", it's mostly probably because this library is missing the slf4j binder: https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java#L231 But it's technically not a bug, since Spark is using latest version of slf4j and parquet {code} <slf4j.version>1.7.16</slf4j.version> <parquet.version>1.8.1</parquet.version> {code} and since 1.6 SLF4J is defaulting to no-operation (NOP) logger implementation, so should be ok. was (Author: iamshrek): I worked on this for couple days, and I found it's not caused by Spark, but the parquet library "parquet-mr/parquet-hadoop". I've debug by step, and found this error is from here: https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala#L820 and after digging into "parquet-hadoop", it's mostly probably because this library is missing the slf4j binder: https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java#L231 But it's technically not a bug, since Spark is using {code}<slf4j.version>1.7.16</slf4j.version>{code}, and since 1.6 SLF4J is defaulting to no-operation (NOP) logger implementation, so should be ok. > SparkR read.df() from parquet got error: SLF4J: Failed to load class > "org.slf4j.impl.StaticLoggerBinder" > -------------------------------------------------------------------------------------------------------- > > Key: SPARK-16437 > URL: https://issues.apache.org/jira/browse/SPARK-16437 > Project: Spark > Issue Type: Bug > Components: SQL > Reporter: Xin Ren > Priority: Minor > > build SparkR with command > {code} > build/mvn -DskipTests -Psparkr package > {code} > start SparkR console > {code} > ./bin/sparkR > {code} > then get error > {code} > Welcome to > ____ __ > / __/__ ___ _____/ /__ > _\ \/ _ \/ _ `/ __/ '_/ > /___/ .__/\_,_/_/ /_/\_\ version 2.0.0-SNAPSHOT > /_/ > SparkSession available as 'spark'. > > > > > > library(SparkR) > > > > df <- read.df("examples/src/main/resources/users.parquet") > SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". > SLF4J: Defaulting to no-operation (NOP) logger implementation > SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further > details. > > > > > > head(df) > 16/07/07 23:20:54 WARN ParquetRecordReader: Can not initialize counter due to > context is not a instance of TaskInputOutputContext, but is > org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl > name favorite_color favorite_numbers > 1 Alyssa <NA> 3, 9, 15, 20 > 2 Ben red NULL > {code} > Reference > * seems need to add a lib from slf4j to point to older version > http://stackoverflow.com/questions/7421612/slf4j-failed-to-load-class-org-slf4j-impl-staticloggerbinder > * on slf4j official site: http://www.slf4j.org/codes.html#StaticLoggerBinder -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org