That seems like a fair solution. I noticed there’s also a spark-submit command that’s required when running this pipeline:worke <https://github.com/apache/incubator-spot/blob/9f63ab724e075fc3da458bd8ff86a777ddb94d68/spot-ingest/pipelines/proxy/worker.py> r.py . We should make the update here as well since I believe this also requries Spark 2.1+.
On Fri, Mar 20, 2020 at 4:24 PM Tadd Wood <taddw...@apache.org> wrote: > I notice there’s a dependency on Spark 2.1+ that didn’t get resolved per > this JIRA (https://issues.apache.org/jira/projects/SPOT/issues/SPOT-266). > I > propose adding a check in this script to verify the user’s version of Spark > and if it's not at least 2.1 the script will exit and indicate that the > user should upgrade their version of Spark. We also need to replace > spark-submit with spark-submit2. Any thoughts for/against updating this > script? > > Also would anyone be willing to update the documentation on the Spot > website to reflect the correct required version of Spark to run Spot? > > Thank you, > Tadd Wood > -- if( bool halfWayThere == true); echo "WOOOOOOOOOOOOOOOOAH! LIVIN ON A PRAYER";
