Re: [DISCUSS] More meaningful error when running on Spark 2.0
+1 On Tue, Nov 15, 2016 at 9:23 AM, Dmitriy Lyubimov wrote: > +1 on version checking. > And, there's a little bug as well. this error is technically generated by > something like > > dense(Set.empty[Vector]), > > i.e., it cannot form a matrix out of an empty collection of vectors. While > this is true, i suppose it needs a `require(...)` insert there to generate > a more meaningful response instead of allowing Scala complaining about > empty collection. > > -d > > > On Mon, Nov 14, 2016 at 7:32 AM, Andrew Palumbo > wrote: > > > +1 > > > > > > > > Sent from my Verizon Wireless 4G LTE smartphone > > > > > > Original message > > From: Trevor Grant > > Date: 11/14/2016 6:49 AM (GMT-08:00) > > To: dev@mahout.apache.org > > Subject: [DISCUSS] More meaningful error when running on Spark 2.0 > > > > Hi, > > > > currently when running on Spark 2.0 the user will hit some sort of error, > > one such error is: > > > > java.util.NoSuchElementException: next on empty iterator > > at scala.collection.Iterator$$anon$2.next(Iterator.scala:39) > > at scala.collection.Iterator$$anon$2.next(Iterator.scala:37) > > at scala.collection.IndexedSeqLike$Elements.next( > IndexedSeqLike.scala:63) > > at scala.collection.IterableLike$class.head(IterableLike.scala:107) > > at scala.collection.mutable.ArrayOps$ofRef.scala$collection$Ind > > exedSeqOptimized$$super$head(ArrayOps.scala:186) > > at scala.collection.IndexedSeqOptimized$class.head(IndexedSeqOp > > timized.scala:126) > > at scala.collection.mutable.ArrayOps$ofRef.head(ArrayOps.scala:186) > > at org.apache.mahout.math.scalabindings.package$$anonfun$1.appl > > y(package.scala:155) > > at org.apache.mahout.math.scalabindings.package$$anonfun$1.appl > > y(package.scala:133) > > at scala.collection.TraversableLike$$anonfun$map$1.apply(Traver > > sableLike.scala:234) > > at scala.collection.TraversableLike$$anonfun$map$1.apply(Traver > > sableLike.scala:234) > > at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSe > > qOptimized.scala:33) > > at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35) > > at scala.collection.TraversableLike$class.map(TraversableLike.scala:234) > > at scala.collection.AbstractTraversable.map(Traversable.scala:104) > > at org.apache.mahout.math.scalabindings.package$.dense( > package.scala:133) > > at org.apache.mahout.sparkbindings.SparkEngine$.drmSampleKRows( > > SparkEngine.scala:289) > > at org.apache.mahout.math.drm.package$.drmSampleKRows(package.scala:149) > > at org.apache.mahout.math.drm.package$.drmSampleToTSV(package.scala:165) > > ... 58 elided > > > > With the recent Zeppelin-Mahout integration, there are going to be a lot > of > > users unknowingly attempting to run on Mahout on Spark 2.0. I think it > > would be simple to implement yet save a lot of time on the Zeppelin and > > Mahout mailing lists to do something like: > > > > if sc.version > 1.6.2 then: > >error("Spark versions ${sc.verion} isn't supported. Please see > > MAHOUT-... (appropriate jira info)") > > > > I'd like to put something together and, depending on how many issues > people > > have on Zeppelin list, be prepared to do a hotfix on 0.12.2 if it becomes > > prudent. Everyone always complaining that Zeppelin doesn't work because > of > > some mystical error, is bad pr. It DOES say in the notebook and > elsewhere > > that we're not 2.0 compliant, however one of the advantages/drawbacks of > > Zeppelin is that without having to really know what you're doing you can > > get a functional local cluster of Flink, Spark, etc. all going. > > > > So we easily could have a space where someone read none of the docs, and > is > > whining. Surely few if any would ever do such a thing, but still I > think a > > prudent fix to have in the back pocket. > > > > tg > > > > Trevor Grant > > Data Scientist > > https://github.com/rawkintrevo > > http://stackexchange.com/users/3002022/rawkintrevo > > http://trevorgrant.org > > > > *"Fortunate is he, who is able to know the causes of things." -Virgil* > > >
Re: [DISCUSS] More meaningful error when running on Spark 2.0
+1 on version checking. And, there's a little bug as well. this error is technically generated by something like dense(Set.empty[Vector]), i.e., it cannot form a matrix out of an empty collection of vectors. While this is true, i suppose it needs a `require(...)` insert there to generate a more meaningful response instead of allowing Scala complaining about empty collection. -d On Mon, Nov 14, 2016 at 7:32 AM, Andrew Palumbo wrote: > +1 > > > > Sent from my Verizon Wireless 4G LTE smartphone > > > Original message > From: Trevor Grant > Date: 11/14/2016 6:49 AM (GMT-08:00) > To: dev@mahout.apache.org > Subject: [DISCUSS] More meaningful error when running on Spark 2.0 > > Hi, > > currently when running on Spark 2.0 the user will hit some sort of error, > one such error is: > > java.util.NoSuchElementException: next on empty iterator > at scala.collection.Iterator$$anon$2.next(Iterator.scala:39) > at scala.collection.Iterator$$anon$2.next(Iterator.scala:37) > at scala.collection.IndexedSeqLike$Elements.next(IndexedSeqLike.scala:63) > at scala.collection.IterableLike$class.head(IterableLike.scala:107) > at scala.collection.mutable.ArrayOps$ofRef.scala$collection$Ind > exedSeqOptimized$$super$head(ArrayOps.scala:186) > at scala.collection.IndexedSeqOptimized$class.head(IndexedSeqOp > timized.scala:126) > at scala.collection.mutable.ArrayOps$ofRef.head(ArrayOps.scala:186) > at org.apache.mahout.math.scalabindings.package$$anonfun$1.appl > y(package.scala:155) > at org.apache.mahout.math.scalabindings.package$$anonfun$1.appl > y(package.scala:133) > at scala.collection.TraversableLike$$anonfun$map$1.apply(Traver > sableLike.scala:234) > at scala.collection.TraversableLike$$anonfun$map$1.apply(Traver > sableLike.scala:234) > at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSe > qOptimized.scala:33) > at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35) > at scala.collection.TraversableLike$class.map(TraversableLike.scala:234) > at scala.collection.AbstractTraversable.map(Traversable.scala:104) > at org.apache.mahout.math.scalabindings.package$.dense(package.scala:133) > at org.apache.mahout.sparkbindings.SparkEngine$.drmSampleKRows( > SparkEngine.scala:289) > at org.apache.mahout.math.drm.package$.drmSampleKRows(package.scala:149) > at org.apache.mahout.math.drm.package$.drmSampleToTSV(package.scala:165) > ... 58 elided > > With the recent Zeppelin-Mahout integration, there are going to be a lot of > users unknowingly attempting to run on Mahout on Spark 2.0. I think it > would be simple to implement yet save a lot of time on the Zeppelin and > Mahout mailing lists to do something like: > > if sc.version > 1.6.2 then: >error("Spark versions ${sc.verion} isn't supported. Please see > MAHOUT-... (appropriate jira info)") > > I'd like to put something together and, depending on how many issues people > have on Zeppelin list, be prepared to do a hotfix on 0.12.2 if it becomes > prudent. Everyone always complaining that Zeppelin doesn't work because of > some mystical error, is bad pr. It DOES say in the notebook and elsewhere > that we're not 2.0 compliant, however one of the advantages/drawbacks of > Zeppelin is that without having to really know what you're doing you can > get a functional local cluster of Flink, Spark, etc. all going. > > So we easily could have a space where someone read none of the docs, and is > whining. Surely few if any would ever do such a thing, but still I think a > prudent fix to have in the back pocket. > > tg > > Trevor Grant > Data Scientist > https://github.com/rawkintrevo > http://stackexchange.com/users/3002022/rawkintrevo > http://trevorgrant.org > > *"Fortunate is he, who is able to know the causes of things." -Virgil* >
RE: [DISCUSS] More meaningful error when running on Spark 2.0
+1 Sent from my Verizon Wireless 4G LTE smartphone Original message From: Trevor Grant Date: 11/14/2016 6:49 AM (GMT-08:00) To: dev@mahout.apache.org Subject: [DISCUSS] More meaningful error when running on Spark 2.0 Hi, currently when running on Spark 2.0 the user will hit some sort of error, one such error is: java.util.NoSuchElementException: next on empty iterator at scala.collection.Iterator$$anon$2.next(Iterator.scala:39) at scala.collection.Iterator$$anon$2.next(Iterator.scala:37) at scala.collection.IndexedSeqLike$Elements.next(IndexedSeqLike.scala:63) at scala.collection.IterableLike$class.head(IterableLike.scala:107) at scala.collection.mutable.ArrayOps$ofRef.scala$collection$Ind exedSeqOptimized$$super$head(ArrayOps.scala:186) at scala.collection.IndexedSeqOptimized$class.head(IndexedSeqOp timized.scala:126) at scala.collection.mutable.ArrayOps$ofRef.head(ArrayOps.scala:186) at org.apache.mahout.math.scalabindings.package$$anonfun$1.appl y(package.scala:155) at org.apache.mahout.math.scalabindings.package$$anonfun$1.appl y(package.scala:133) at scala.collection.TraversableLike$$anonfun$map$1.apply(Traver sableLike.scala:234) at scala.collection.TraversableLike$$anonfun$map$1.apply(Traver sableLike.scala:234) at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSe qOptimized.scala:33) at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35) at scala.collection.TraversableLike$class.map(TraversableLike.scala:234) at scala.collection.AbstractTraversable.map(Traversable.scala:104) at org.apache.mahout.math.scalabindings.package$.dense(package.scala:133) at org.apache.mahout.sparkbindings.SparkEngine$.drmSampleKRows( SparkEngine.scala:289) at org.apache.mahout.math.drm.package$.drmSampleKRows(package.scala:149) at org.apache.mahout.math.drm.package$.drmSampleToTSV(package.scala:165) ... 58 elided With the recent Zeppelin-Mahout integration, there are going to be a lot of users unknowingly attempting to run on Mahout on Spark 2.0. I think it would be simple to implement yet save a lot of time on the Zeppelin and Mahout mailing lists to do something like: if sc.version > 1.6.2 then: error("Spark versions ${sc.verion} isn't supported. Please see MAHOUT-... (appropriate jira info)") I'd like to put something together and, depending on how many issues people have on Zeppelin list, be prepared to do a hotfix on 0.12.2 if it becomes prudent. Everyone always complaining that Zeppelin doesn't work because of some mystical error, is bad pr. It DOES say in the notebook and elsewhere that we're not 2.0 compliant, however one of the advantages/drawbacks of Zeppelin is that without having to really know what you're doing you can get a functional local cluster of Flink, Spark, etc. all going. So we easily could have a space where someone read none of the docs, and is whining. Surely few if any would ever do such a thing, but still I think a prudent fix to have in the back pocket. tg Trevor Grant Data Scientist https://github.com/rawkintrevo http://stackexchange.com/users/3002022/rawkintrevo http://trevorgrant.org *"Fortunate is he, who is able to know the causes of things." -Virgil*
[DISCUSS] More meaningful error when running on Spark 2.0
Hi, currently when running on Spark 2.0 the user will hit some sort of error, one such error is: java.util.NoSuchElementException: next on empty iterator at scala.collection.Iterator$$anon$2.next(Iterator.scala:39) at scala.collection.Iterator$$anon$2.next(Iterator.scala:37) at scala.collection.IndexedSeqLike$Elements.next(IndexedSeqLike.scala:63) at scala.collection.IterableLike$class.head(IterableLike.scala:107) at scala.collection.mutable.ArrayOps$ofRef.scala$collection$Ind exedSeqOptimized$$super$head(ArrayOps.scala:186) at scala.collection.IndexedSeqOptimized$class.head(IndexedSeqOp timized.scala:126) at scala.collection.mutable.ArrayOps$ofRef.head(ArrayOps.scala:186) at org.apache.mahout.math.scalabindings.package$$anonfun$1.appl y(package.scala:155) at org.apache.mahout.math.scalabindings.package$$anonfun$1.appl y(package.scala:133) at scala.collection.TraversableLike$$anonfun$map$1.apply(Traver sableLike.scala:234) at scala.collection.TraversableLike$$anonfun$map$1.apply(Traver sableLike.scala:234) at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSe qOptimized.scala:33) at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35) at scala.collection.TraversableLike$class.map(TraversableLike.scala:234) at scala.collection.AbstractTraversable.map(Traversable.scala:104) at org.apache.mahout.math.scalabindings.package$.dense(package.scala:133) at org.apache.mahout.sparkbindings.SparkEngine$.drmSampleKRows( SparkEngine.scala:289) at org.apache.mahout.math.drm.package$.drmSampleKRows(package.scala:149) at org.apache.mahout.math.drm.package$.drmSampleToTSV(package.scala:165) ... 58 elided With the recent Zeppelin-Mahout integration, there are going to be a lot of users unknowingly attempting to run on Mahout on Spark 2.0. I think it would be simple to implement yet save a lot of time on the Zeppelin and Mahout mailing lists to do something like: if sc.version > 1.6.2 then: error("Spark versions ${sc.verion} isn't supported. Please see MAHOUT-... (appropriate jira info)") I'd like to put something together and, depending on how many issues people have on Zeppelin list, be prepared to do a hotfix on 0.12.2 if it becomes prudent. Everyone always complaining that Zeppelin doesn't work because of some mystical error, is bad pr. It DOES say in the notebook and elsewhere that we're not 2.0 compliant, however one of the advantages/drawbacks of Zeppelin is that without having to really know what you're doing you can get a functional local cluster of Flink, Spark, etc. all going. So we easily could have a space where someone read none of the docs, and is whining. Surely few if any would ever do such a thing, but still I think a prudent fix to have in the back pocket. tg Trevor Grant Data Scientist https://github.com/rawkintrevo http://stackexchange.com/users/3002022/rawkintrevo http://trevorgrant.org *"Fortunate is he, who is able to know the causes of things." -Virgil*