Hi I just joined a project that runs on spark-1.6.1 and I have no prior spark experience.
The project build is quite fragile when it comes to runtime dependencies. Often the project builds fine but after deployment we end up with ClassNotFoundException's or NoSuchMethodError's when submitting a job. To catch these issues early, I'm trying like to setup integrations tests with maven. In the pre-integration phase I would like to startup a master and a worker (using process-exec-maven-plugin in the pre-integration-test phase). I have managed to get it working for spark 1.6.1 (against a downloaded spark distribution), but would prefer to be able to download all the required jars as maven dependencies. Is there a relatively simple way to get all the required dependencies? It is ok if its only for 2.x since we are planning to migrate. I would prefer to do this without docker. Has anyone done something similar already or is there a simpler way? Best regards Jens
