NoSuchMethodError in general means that your runtime and compile-time environments are different. I think you need to first make sure you don't have mismatching versions of Spark.
On Wed, Mar 25, 2015 at 11:00 AM, <zapletal-mar...@email.cz> wrote: > Hi, > > I have started implementing a machine learning pipeline using Spark 1.3.0 > and the new pipelining API and DataFrames. I got to a point where I have my > training data set prepared using a sequence of Transformers, but I am > struggling to actually train a model and use it for predictions. > > I am getting a java.lang.NoSuchMethodException: > org.apache.spark.ml.regression.LinearRegression.myFeaturesColumnName() > exception thrown at checkInputColumn method in Params trait when using a > Predictor (LinearRegression in my case, but that should not matter). This > looks like a bug - the exception is thrown when executing getParam(colName) > when the require(actualDataType.equals(datatype), ...) requirement is not > met so the expected requirement failed exception is not thrown and is hidden > by the unexpected NoSuchMethodException instead. I can raise a bug if this > really is an issue and I am not using something incorrectly. > > The problem I am facing however is that the Predictor expects features to > have VectorUDT type as defined in Predictor class (protected def > featuresDataType: DataType = new VectorUDT). But since this type is > private[spark] my Transformer can not prepare features with this type which > then correctly results in the exception above when I use a different type. > > Is there a way to define a custom Pipeline that would be able to use the > existing Predictors without having to bypass the access modifiers or > reimplement something or is the pipelining API not yet expected to be used > in this way? > > Thanks, > Martin > > --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
