branch-2.0 is now 2.0.1-SNAPSHOT?

is that correct? where do i get the latest 2.0.0-SNAPSHOT? thanks, koert

Re: Code Style Formatting

There isn't one pre-made, but the default works out OK. The main thing you'd need to update are spacing changes for function argument indentation and import ordering. On Fri, Jul 1, 2016 at 4:11 AM, Anton Okolnychyi wrote: > Hi, all. > > I've read the Spark code

Re: [build system] quick jenkins restart

aand we're back. On Fri, Jul 1, 2016 at 10:10 AM, shane knapp wrote: > i put jenkins in quiet mode as i noticed we have almost no builds > queued. one of our students needed rust installed on the workers, and > i need to update the PATH on all of the workers. > > we

[build system] quick jenkins restart

i put jenkins in quiet mode as i noticed we have almost no builds queued. one of our students needed rust installed on the workers, and i need to update the PATH on all of the workers. we should be back up and building within 30 minutes. thanks! shane

Re: Jenkins networking / port contention

Multiple instances of test runs are usually running in parallel, so they would need to bind to different ports. On Friday, July 1, 2016, Cody Koeninger wrote: > Thanks for the response. I'm talking about test code that starts up > embedded network services for integration

Re: Jenkins networking / port contention

gotcha... adding @joshrosen directly who might be of more assistance... :) On Fri, Jul 1, 2016 at 9:38 AM, Cody Koeninger wrote: > Thanks for the response. I'm talking about test code that starts up > embedded network services for integration testing. > > KafkaTestUtils in

Re: Jenkins networking / port contention

Thanks for the response. I'm talking about test code that starts up embedded network services for integration testing. KafkaTestUtils in particular always attempts to start a kafka broker on the standard port, 9092. Util.startServiceInPort is intended to pick a higher port if the starting one

Re: Jenkins networking / port contention

i assume you're talking about zinc ports? the tests are designed to run one at a time on randomized ports -- no containerization. we're on bare metal. the test launch code executes this for each build: # Generate random point for Zinc export ZINC_PORT ZINC_PORT=$(python -S -c "import random;

Jetty 9.3 CVE to be avoided...

To anyone contemplating an upgrade of the Jetty component in use with Apache Spark, please be aware of CVE-2016-4800 , and ensure that you are attempting to only integrate a version of the Jetty 9.3 stream that is *9.3.9* /or later/.

Jenkins networking / port contention

Can someone familiar with amplab's jenkins setup clarify whether all tests running at a given time are competing for network ports, or whether there's some sort of containerization being done? Based on the use of Utils.startServiceOnPort in the tests, I'd assume the former.

Deploying ML Pipeline Model

Hi All, I am looking for ways to deploy a ML Pipeline model in production . Spark has already proved to be a one of the best framework for model training and creation, but once the ml pipeline model is ready how can I deploy it outside spark context ? MLlib model has toPMML method but today

Code Style Formatting

Hi, all. I've read the Spark code style guide. I am wondering if there is an easy way to configure the code formatting in IntelliJ IDEA to match the existing code base style. IntelliJ IDEA highlights all failed checks from scalastyle-config.xml. However, I did not find any predefined

Re: MinMaxScaler With features include category variables

You can combine the columns which are need to be normalized into a vector by VectorAssembler and do normalization on it. Do another assembling for columns should not be normalized. At last, you can assemble the two vector into one vector as the feature column and feed it into model training.

Re: Debugging Spark itself in standalone cluster mode

Thanks for the guidance! Setting the --driver-java-options in spark-shell instead of SPARK_MASTER_OPTS made the debugger connect to the right JVM. My breakpoints get hit now. nirandap [via Apache Spark Developers List] < ml-node+s1001551n18145...@n3.nabble.com> schrieb am Fr., 1. Juli 2016 um