Hi all
I see UGI credentials (ex sparkCookie) shared from driver to executor is being
lost on driver side in yarn mode. Below is the analysis on start of
thriftserver,
Step 1. SparkSubmit create submit env which does a loginUserFromKeytab
"main@1" prio=5 tid=0x1 nid=NA runnable
java.lang.
I am trying to add my custom jar in spark job using "spark.jars" property.
Although I can read the info in logs of jar getting added but when I check
the jars that are added to class path, I doesn't find it.Below are the
functions that I also have tried it out.
1)spark.jars
2)spark.driver.extraLibr
awesome thanks Joseph
2018-03-20 14:51 GMT-07:00 Joseph Bradley :
> The promised roadmap JIRA: https://issues.apache.
> org/jira/browse/SPARK-23758
>
> Note it doesn't have much explicitly listed yet, but committers can add
> items as they agree to shepherd them. (Committers, make sure to check
The promised roadmap JIRA: https://issues.apache.org/jira/browse/SPARK-23758
Note it doesn't have much explicitly listed yet, but committers can add
items as they agree to shepherd them. (Committers, make sure to check what
you're currently listed as shepherding!) The links for searching can be
To clarify what's going on here: dfs.blocksize and dfs.block.size set the
HDFS block size (the spark.hadoop. prefix adds this to the Hadoop
configuration). The Parquet "block size" is more accurately called the "row
group size", but is set using the unfortunately-named property
parquet.block.size.
Hi!
I can't speak for the other tasks, but SPARK-23444 I'd expect to be pretty
complicated. It's not obvious what the right strategy is, and there's a
bunch of minor stuff that needs to be cleaned up (e.g. tasks shouldn't
print cancellation warnings when cancellation is expected).
If you're inter
Good time of day,
I'd like to contribute to Spark development, but find it difficult to get into
the process. I'm somewhat overwhelmed by Spark's Jira as it's hard for me to
figure out the complexity of tasks and choose an appropriate one.
I've surfed Jira for some time and have selected a few i
