In case anyone else has the same problem and finds this - in my case it was
fixed by increasing spark.sql.broadcastTimeout (I used 9000).
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Concurrent-Spark-jobs-tp26011p26648.html
Sent from the Apache Spark
yet, so changing how it executes the jobs on
the queue won't help.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Concurrent-Spark-jobs-tp26011p26062.html
Sent from the Apache Spark User List mailing list
rk.redshift").save()}.execute()
> f1.get()
> f2.get()
>
> In the hope that the first write would no longer block the following steps,
> but instead it fails with a TimeoutException (see stack trace in previous
> message). Is there a way to
e). Is there a way to start the different writes concurrently, or is
that not possible in Spark?
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Concurrent-Spark-jobs-tp26011p26030.html
Sent from
Redshift(RedshiftWriter.scala:374)
> at
>
> com.databricks.spark.redshift.DefaultSource.createRelation(DefaultSource.scala:106)
> at
>
> org.apache.spark.sql.execution.datasources.ResolvedDataSource$.a
aultSource.scala:106)
at
org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:222)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:148)
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Concurrent-Spark-jobs-tp26011.html
Sent from the Apache Spark Us
map
over them in a high level way, but as this is not supported afaik).
Thanks!
Staffan
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/How-to-efficiently-control-concurrent-Spark-jobs-tp21800.html
Sent from the Apache Spark User List mailing list
://apache-spark-user-list.1001560.n3.nabble.com/How-to-efficiently-control-concurrent-Spark-jobs-tp21800.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
-
To unsubscribe, e-mail: user-unsubscr
