How many partitions do you have? It might be that one partition is too large, and there is Integer overflow. Could you double your number of partitions?
Burak On Fri, Jul 3, 2015 at 4:41 AM, Danny <kont...@dannylinden.de> wrote: > hi, > > i want to run a multiclass classification with 390 classes on120k label > points(tf-idf vectors). but i get the following exception. If i reduce the > number of classes to ~20 everythings work fine. How can i fix this? > > i use the LogisticRegressionWithLBFGS class for my classification on a 8 > Node Cluster with > > > total-executor-cores = 30 > > executor-memory = 20g > > My Exception: > > 15/07/02 15:55:00 INFO DAGScheduler: Job 11 finished: count at > LBFGS.scala:170, took 0,521823 s > 15/07/02 15:55:02 INFO MemoryStore: ensureFreeSpace(-1069858488) called > with > curMem=308280107, maxMem=11113699737 > 15/07/02 15:55:02 INFO MemoryStore: Block broadcast_22 stored as values in > memory (estimated size -1069858488.0 B, free 11.1 GB) > Exception in thread "main" java.lang.reflect.InvocationTargetException > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:58) > at > org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala) > Caused by: java.lang.IllegalArgumentException: requirement failed: > sizeInBytes was negative: -1069858488 > at scala.Predef$.require(Predef.scala:233) > at org.apache.spark.storage.BlockInfo.markReady(BlockInfo.scala:55) > at > org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:812) > at > org.apache.spark.storage.BlockManager.putIterator(BlockManager.scala:635) > at > org.apache.spark.storage.BlockManager.putSingle(BlockManager.scala:993) > at > > org.apache.spark.broadcast.TorrentBroadcast.writeBlocks(TorrentBroadcast.scala:99) > at > > org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:85) > at > > org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34) > at > > org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:62) > at org.apache.spark.SparkContext.broadcast(SparkContext.scala:1289) > at > > org.apache.spark.mllib.optimization.LBFGS$CostFun.calculate(LBFGS.scala:215) > at > > org.apache.spark.mllib.optimization.LBFGS$CostFun.calculate(LBFGS.scala:204) > at > breeze.optimize.CachedDiffFunction.calculate(CachedDiffFunction.scala:23) > at > > breeze.optimize.FirstOrderMinimizer.calculateObjective(FirstOrderMinimizer.scala:108) > at > > breeze.optimize.FirstOrderMinimizer.initialState(FirstOrderMinimizer.scala:101) > at > > breeze.optimize.FirstOrderMinimizer.iterations(FirstOrderMinimizer.scala:146) > at > org.apache.spark.mllib.optimization.LBFGS$.runLBFGS(LBFGS.scala:178) > at > org.apache.spark.mllib.optimization.LBFGS.optimize(LBFGS.scala:117) > at > > org.apache.spark.mllib.regression.GeneralizedLinearAlgorithm.run(GeneralizedLinearAlgorithm.scala:282) > at > > org.apache.spark.mllib.regression.GeneralizedLinearAlgorithm.run(GeneralizedLinearAlgorithm.scala:205) > at > > com.test.spark.SVMSimpleAppEC2$.createNaiveBayesModel(SVMSimpleAppEC2.scala:150) > at com.test.spark.SVMSimpleAppEC2$.main(SVMSimpleAppEC2.scala:48) > at com.test.spark.SVMSimpleAppEC2.main(SVMSimpleAppEC2.scala) > ... 6 more > 15/07/02 15:55:02 INFO SparkContext: Invoking stop() from shutdown hook > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-4-MLLib-Bug-Multiclass-Classification-requirement-failed-sizeInBytes-was-negative-tp23610.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
