[ https://issues.apache.org/jira/browse/SPARK-16924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Xin Wu updated SPARK-16924: --------------------------- Issue Type: Improvement (was: Bug) > DataStreamReader can not support option("inferSchema", true/false) for csv > and json file source > ----------------------------------------------------------------------------------------------- > > Key: SPARK-16924 > URL: https://issues.apache.org/jira/browse/SPARK-16924 > Project: Spark > Issue Type: Improvement > Components: SQL > Affects Versions: 2.0.0 > Reporter: Xin Wu > > Currently DataStreamReader can not support option("inferSchema", true|false) > for csv and json file source. It only takes SQLConf setting > "spark.sql.streaming.schemaInference", which needs to be set at session > level. > For example: > {code} > scala> val in = spark.readStream.format("json").option("inferSchema", > true).load("/Users/xinwu/spark-test/data/json/t1") > java.lang.IllegalArgumentException: Schema must be specified when creating a > streaming source DataFrame. If some files already exist in the directory, > then depending on the file format you may be able to create a static > DataFrame on that directory with 'spark.read.load(directory)' and infer > schema from it. > at > org.apache.spark.sql.execution.datasources.DataSource.sourceSchema(DataSource.scala:223) > at > org.apache.spark.sql.execution.datasources.DataSource.sourceInfo$lzycompute(DataSource.scala:80) > at > org.apache.spark.sql.execution.datasources.DataSource.sourceInfo(DataSource.scala:80) > at > org.apache.spark.sql.execution.streaming.StreamingRelation$.apply(StreamingRelation.scala:30) > at > org.apache.spark.sql.streaming.DataStreamReader.load(DataStreamReader.scala:142) > at > org.apache.spark.sql.streaming.DataStreamReader.load(DataStreamReader.scala:153) > ... 48 elided > scala> val in = spark.readStream.format("csv").option("inferSchema", > true).load("/Users/xinwu/spark-test/data/csv") > java.lang.IllegalArgumentException: Schema must be specified when creating a > streaming source DataFrame. If some files already exist in the directory, > then depending on the file format you may be able to create a static > DataFrame on that directory with 'spark.read.load(directory)' and infer > schema from it. > at > org.apache.spark.sql.execution.datasources.DataSource.sourceSchema(DataSource.scala:223) > at > org.apache.spark.sql.execution.datasources.DataSource.sourceInfo$lzycompute(DataSource.scala:80) > at > org.apache.spark.sql.execution.datasources.DataSource.sourceInfo(DataSource.scala:80) > at > org.apache.spark.sql.execution.streaming.StreamingRelation$.apply(StreamingRelation.scala:30) > at > org.apache.spark.sql.streaming.DataStreamReader.load(DataStreamReader.scala:142) > at > org.apache.spark.sql.streaming.DataStreamReader.load(DataStreamReader.scala:153) > ... 48 elided > {code} > In the example, even though users specify the option("inferSchema", true), it > does not take it. But for batch data, DataFrameReader can take it: > {code} > scala> val in = spark.read.format("csv").option("header", > true).option("inferSchema", true).load("/Users/xinwu/spark-test/data/csv1") > in: org.apache.spark.sql.DataFrame = [signal: string, flash: int] > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org