Yes, it contain one line On Wed, Aug 26, 2015 at 8:20 PM, Yin Huai-2 [via Apache Spark Developers List] <ml-node+s1001551n13852...@n3.nabble.com> wrote:
> The JSON support in Spark SQL handles a file with one JSON object per line > or one JSON array of objects per line. What is the format your file? Does > it only contain a single line? > > On Wed, Aug 26, 2015 at 6:47 AM, gsvic <[hidden email] > <http:///user/SendEmail.jtp?type=node&node=13852&i=0>> wrote: > >> Hi, >> >> I have the following issue. I am trying to load a 2.5G JSON file from a >> 10-node Hadoop Cluster. Actually, I am trying to create a DataFrame, using >> sqlContext.read.json("hdfs://master:9000/path/file.json"). >> >> The JSON file contains a parsed table(relation) from the TPCH benchmark. >> >> After finishing some tasks, the job fails by throwing several >> java.io.IOExceptions. For smaller files (eg 700M it works fine). I am >> posting a part of the log and the whole stack trace below: >> >> 15/08/26 16:31:44 INFO TaskSetManager: Starting task 10.1 in stage 1.0 >> (TID >> 47, 192.168.5.146, ANY, 1416 bytes) >> 15/08/26 16:31:44 INFO TaskSetManager: Starting task 11.1 in stage 1.0 >> (TID >> 48, 192.168.5.150, ANY, 1416 bytes) >> 15/08/26 16:31:44 INFO TaskSetManager: Starting task 4.1 in stage 1.0 (TID >> 49, 192.168.5.149, ANY, 1416 bytes) >> 15/08/26 16:31:44 INFO TaskSetManager: Starting task 8.1 in stage 1.0 (TID >> 50, 192.168.5.246, ANY, 1416 bytes) >> 15/08/26 16:31:53 INFO TaskSetManager: Finished task 10.0 in stage 1.0 >> (TID >> 17) in 104681 ms on 192.168.5.243 (27/35) >> 15/08/26 16:31:53 INFO TaskSetManager: Finished task 8.0 in stage 1.0 (TID >> 15) in 105541 ms on 192.168.5.193 (28/35) >> 15/08/26 16:31:55 INFO TaskSetManager: Finished task 11.0 in stage 1.0 >> (TID >> 18) in 107122 ms on 192.168.5.167 (29/35) >> 15/08/26 16:31:57 INFO TaskSetManager: Finished task 5.0 in stage 1.0 (TID >> 12) in 109583 ms on 192.168.5.245 (30/35) >> 15/08/26 16:32:08 INFO TaskSetManager: Finished task 4.1 in stage 1.0 (TID >> 49) in 24135 ms on 192.168.5.149 (31/35) >> 15/08/26 16:32:13 WARN TaskSetManager: Lost task 2.0 in stage 1.0 (TID 9, >> 192.168.5.246): java.io.IOException: Too many bytes before newline: >> 2147483648 >> at >> org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:249) >> at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174) >> at >> >> org.apache.hadoop.mapred.LineRecordReader.<init>(LineRecordReader.java:134) >> at >> >> org.apache.hadoop.mapred.TextInputFormat.getRecordReader(TextInputFormat.java:67) >> at >> org.apache.spark.rdd.HadoopRDD$$anon$1.<init>(HadoopRDD.scala:239) >> at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:216) >> at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:101) >> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244) >> at >> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35) >> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244) >> at >> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35) >> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244) >> at >> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35) >> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244) >> at >> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35) >> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244) >> at >> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:70) >> at >> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) >> at org.apache.spark.scheduler.Task.run(Task.scala:70) >> at >> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213) >> at >> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> at >> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> at java.lang.Thread.run(Thread.java:745) >> >> >> >> -- >> View this message in context: >> http://apache-spark-developers-list.1001551.n3.nabble.com/SQLContext-read-json-path-throws-java-io-IOException-tp13841.html >> Sent from the Apache Spark Developers List mailing list archive at >> Nabble.com. >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [hidden email] >> <http:///user/SendEmail.jtp?type=node&node=13852&i=1> >> For additional commands, e-mail: [hidden email] >> <http:///user/SendEmail.jtp?type=node&node=13852&i=2> >> >> > > > ------------------------------ > If you reply to this email, your message will be added to the discussion > below: > > http://apache-spark-developers-list.1001551.n3.nabble.com/SQLContext-read-json-path-throws-java-io-IOException-tp13841p13852.html > To unsubscribe from SQLContext.read.json("path") throws > java.io.IOException, click here > <http://apache-spark-developers-list.1001551.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=13841&code=dmljdG9yYXNnc0BnbWFpbC5jb218MTM4NDF8MzQ0MzYwNzA=> > . > NAML > <http://apache-spark-developers-list.1001551.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> > -- Victor Giannakouris - Salalidis Researcher Computing Systems Laboratory (CSLab), National Technical University of Athens Software Engineer isMOOD Data Technology Services LinkedIn: http://gr.linkedin.com/pub/victor-giannakouris-salalidis/69/585/b23/ -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/SQLContext-read-json-path-throws-java-io-IOException-tp13841p13854.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
