Beta3 work with Spark SQL 1.0 and 1.1
Spark SQL 1.2 was released after that and broke binary backwards compatibility however his has been fixed in master/dev
version [1]
Note that Spark SQL 1.3 was released as well and again, broke backwards compatibility this time significant hence why
there are now two versions [2] - make sure to use the one appropriate for your Spark version.
Cheers,
[1]
http://www.elastic.co/guide/en/elasticsearch/hadoop/master/install.html#download-dev
[2]
http://www.elastic.co/guide/en/elasticsearch/hadoop/master/spark.html#spark-sql-versions
On 4/20/15 1:17 PM, michele crudele wrote:
I wrote this simple notebook in scala using Elasticsearch Spark adapter:
%AddJar
file:///tools/elasticsearch-hadoop-2.1.0.Beta3/dist/elasticsearch-spark_2.10-2.1.0.BUILD-SNAPSHOT.jar
%AddJar
file:///tools/elasticsearch-hadoop-2.1.0.Beta3/dist/elasticsearch-hadoop-2.1.0.BUILD-SNAPSHOT.jar
// Write something to spark/docs index
//
import org.elasticsearch.spark._
val michelangelo = Map(artist - Michelangelo, bio - Painter, sculptor,
architect born in Florence in... )
val leonardo = Map(artist - Leonardo, bio - Absolute genius; painter,
inventor born in the little village of
Vinci...)
sc.makeRDD(Seq(michelangelo, leonardo)).saveToEs(spark/docs)
// Search for painters through spark/docs
//
val painters = sc.esRDD(spark/docs, ?q=painter)
println(Number of painters in spark/docs: + painters.count())
painters.collect().foreach(x = println(ID: + x._1))
// Try to read using SparkSQL
//
import org.apache.spark.sql.SQLContext
import org.elasticsearch.spark._
import org.elasticsearch.spark.sql._
val sql = new SQLContext(sc)
// Here is where I get an exception
//
val docsSql = sql.esRDD(spark/docs)
Name: java.lang.NoSuchMethodError
Message:
org.apache.spark.sql.catalyst.types.StructField.init(Ljava/lang/String;Lorg/apache/spark/sql/catalyst/types/DataType;Z)V
StackTrace:
org.elasticsearch.spark.sql.MappingUtils$.org$elasticsearch$spark$sql$MappingUtils$$convertField(MappingUtils.scala:75)
org.elasticsearch.spark.sql.MappingUtils$$anonfun$convertToStruct$1.apply(MappingUtils.scala:54)
org.elasticsearch.spark.sql.MappingUtils$$anonfun$convertToStruct$1.apply(MappingUtils.scala:54)
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108)
org.elasticsearch.spark.sql.MappingUtils$.convertToStruct(MappingUtils.scala:54)
org.elasticsearch.spark.sql.MappingUtils$.discoverMapping(MappingUtils.scala:47)
org.elasticsearch.spark.sql.EsSparkSQL$.esRDD(EsSparkSQL.scala:27)
org.elasticsearch.spark.sql.EsSparkSQL$.esRDD(EsSparkSQL.scala:23)
org.elasticsearch.spark.sql.package$SQLContextFunctions.esRDD(package.scala:16)
$line460.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$c57ec8bf9b0d5f6161b97741d596ff0wC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.init(console:111)
$line460.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.init(console:116)
$line460.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.init(console:118)
$line460.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$
What am I doing wrong? Thanks
- Michele
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to
elasticsearch+unsubscr...@googlegroups.com
mailto:elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/49a5c624-bda0-4835-98bc-915ec80d4fa3%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/49a5c624-bda0-4835-98bc-915ec80d4fa3%40googlegroups.com?utm_medium=emailutm_source=footer.
For more options, visit https://groups.google.com/d/optout.
--
Costin
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
