Hello Spark dev community, I took this the following jira item (https://github.com/apache/spark/pull/15848) and am looking for some general pointers, it seems that I am running into issues where things work successfully doing local development on my macbook pro but fail on jenkins for a multitiude of reasons and errors, here's an example, if you see this build output report: https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69297/ you will see the DataFrameStatSuite, now locally I am running these individual tests with this command: ./build/mvn test -P... -DwildcardSuites=none -Dtest=org.apache.spark.sql.DataFrameStatSuite. It seems that I need to emulate a jenkins like environment locally, this seems sort of like an untenable hurdle, granted that my changes involve changing the total number of workers in the sparkcontext and if so should I be testing my changes in an environment that more closely resembles jenkins. I really want to work on/complete this PR but I keep getting hamstrung by a dev environment that is not equivalent to our CI environment.
I'm guessing/hoping I'm not the first one to run into this so some insights. pointers to get past this would be very appreciated , would love to keep contributing and hoping this is a hurdle that's overcomeable with some tweaks to my dev environment. Thanks in advance.
