Hi,
try this conf
val sc = new SparkContext(conf)
sc.hadoopConfiguration.setBoolean("parquet.enable.summary-metadata", false)
Regards,
Sai Ganesh
On Thu, Sep 15, 2016 at 11:34 PM, gaurav24 [via Apache Spark User List] <
ml-node+s1001560n27738...@n3.nabble.com> wrote:
> Hi Rok,
>
> facing similar issue with streaming where I append to parquet data every
> hour. Writing seems to be slowing down it time it writes. It has gone from
> 17 mins to 40 mins in a month
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-spark-user-list.1001560.n3.nabble.com/very-
> slow-parquet-file-write-tp25295p27738.html
> To start a new topic under Apache Spark User List, email
> ml-node+s1001560n1...@n3.nabble.com
> To unsubscribe from Apache Spark User List, click here
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=dG9zYWlnYW5lc2hAZ21haWwuY29tfDF8LTExOTE1Nzk1MTU=>
> .
> NAML
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
-----
Sai Ganesh
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/very-slow-parquet-file-write-tp25295p27739.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
