Hi, try this conf
val sc = new SparkContext(conf) sc.hadoopConfiguration.setBoolean("parquet.enable.summary-metadata", false) Regards, Sai Ganesh On Thu, Sep 15, 2016 at 11:34 PM, gaurav24 [via Apache Spark User List] < ml-node+s1001560n27738...@n3.nabble.com> wrote: > Hi Rok, > > facing similar issue with streaming where I append to parquet data every > hour. Writing seems to be slowing down it time it writes. It has gone from > 17 mins to 40 mins in a month > > ------------------------------ > If you reply to this email, your message will be added to the discussion > below: > http://apache-spark-user-list.1001560.n3.nabble.com/very- > slow-parquet-file-write-tp25295p27738.html > To start a new topic under Apache Spark User List, email > ml-node+s1001560n1...@n3.nabble.com > To unsubscribe from Apache Spark User List, click here > <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=dG9zYWlnYW5lc2hAZ21haWwuY29tfDF8LTExOTE1Nzk1MTU=> > . > NAML > <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> > ----- Sai Ganesh -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/very-slow-parquet-file-write-tp25295p27739.html Sent from the Apache Spark User List mailing list archive at Nabble.com.