You may want to read about the JVM and have some degree of understanding
what you're talking about, and then you'd know that those options have
different meanings. You can view both at the same time, for example.
On Thu, Apr 16, 2020, 2:13 AM jane thorpe
wrote:
>
>
> I want to know if Spark is headed in my direction.
>
You are implying Spark could be.
What direction are you headed in, exactly? I don't feel as if anything were
implied when you were asked for use cases or what problem you are solving.
You were asked to identify some use cases, of which
We are trying to use spark event logging with s3a as a destination for event
data.
We added these settings to the spark submits:
spark.eventLog.dir s3a://ourbucket/sparkHistoryServer/eventLogs
spark.eventLog.enabled true
Everything works fine with smaller jobs, and we can see the history data
I’m attempting to use Spark 2.3.1 (spark-2.3.1-bin-hadoop2.7.tgz) in cluster
mode and running into some issues. This is a cluster where we've had success
using Spark 2.2.0 (spark-2.2.0-bin-hadoop2.7.tgz), and I'm simply upgrading our
nodes with the new Spark 2.3.1 package and testing it out.
