Hi Rico,
using SQL saves a lot of time, effort, and budget over the long term. But I
guess that there are certain joys in solving self induced complexities.
Thanks for sharing your findings.
Regards,
Gourav Sengupta
On Fri, Feb 18, 2022 at 7:26 AM Rico Bergmann wrote:
> I found the reason
I have been able to partially fix this issue by creating a static final
field (i.e. a constant) for Encoders.STRING(). This removes the
bottleneck associated with instantiating this Encoder. However, this
moved the performance issue only to these two methods:
Addendum: I have tried to replace localIterator with a forEach() call on
the dataset directly, but this hasn't improved the performance.
If the forEach call is the issue, there probably isn't much that can be
done to further improve things, other than perhaps trying to batch the
prediction
That doesn't make a lot of sense. Are you profiling the driver, rather than
executors where the work occurs?
Is your data set quite small such that small overheads look big?
Do you even need Spark if your data is not distributed - coming from the
driver anyway?
The fact that a static final field
Unsubscribe
Could someone recommend a Scala/Spark kernel for Jupyter/JupyterHub that
support the latest Spark version? Thanks!
-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org
Hi,
I need to arrange a class for members using GCP with Dataproc or GCP with
Kubernetes I think 樂
Ok it is a good practice to create namespace spark for this purpose rather
than using default namespace
kubectl create namespace spark
Tell me exactly what you are trying to do? Are you running
Hi Mich
I'm running spark from GCP Platform and this is the error.
Exception in thread "main"
io.fabric8.kubernetes.client.KubernetesClientException: Operation: [create]
for kind: [Pod] with name: [null] in namespace: [default] failed.
Thanks
GK
On Fri, Feb 18, 2022 at 12:37 AM Mich
These kinds of static analysis have limited value to send around. It's not
clear whether any of the CVEs actually affect Spark's usage of the library.
jackson -- generally, yes could theoretically affect Spark apps.
I can't really read this output, but seems like the affected versions are
