Beta3 work with Spark SQL 1.0 and 1.1
Spark SQL 1.2 was released after that and broke binary backwards compatibility however his has been fixed in master/dev
version [1]
Note that Spark SQL 1.3 was released as well and again, broke backwards compatibility this time significant hence why
there are now two versions [2] - make sure to use the one appropriate for your Spark version.
Cheers,
[1]
http://www.elastic.co/guide/en/elasticsearch/hadoop/master/install.html#download-dev
[2]
http://www.elastic.co/guide/en/elasticsearch/hadoop/master/spark.html#spark-sql-versions
On 4/20/15 1:17 PM, michele crudele wrote:
I wrote this simple notebook in scala using Elasticsearch Spark adapter:
%AddJar
file:///tools/elasticsearch-hadoop-2.1.0.Beta3/dist/elasticsearch-spark_2.10-2.1.0.BUILD-SNAPSHOT.jar
%AddJar
file:///tools/elasticsearch-hadoop-2.1.0.Beta3/dist/elasticsearch-hadoop-2.1.0.BUILD-SNAPSHOT.jar
// Write something to spark/docs index
//
import org.elasticsearch.spark._
val michelangelo = Map("artist" -> "Michelangelo", "bio" -> "Painter, sculptor,
architect born in Florence in... ")
val leonardo = Map("artist" -> "Leonardo", "bio" -> "Absolute genius; painter,
inventor born in the little village of
Vinci...")
sc.makeRDD(Seq(michelangelo, leonardo)).saveToEs("spark/docs")
// Search for painters through spark/docs
//
val painters = sc.esRDD("spark/docs", "?q=painter")
println("Number of painters in spark/docs: " + painters.count())
painters.collect().foreach(x => println("ID: " + x._1))
// Try to read using SparkSQL
//
import org.apache.spark.sql.SQLContext
import org.elasticsearch.spark._
import org.elasticsearch.spark.sql._
val sql = new SQLContext(sc)
// Here is where I get an exception
//
val docsSql = sql.esRDD("spark/docs")
Name: java.lang.NoSuchMethodError
Message:
org.apache.spark.sql.catalyst.types.StructField.<init>(Ljava/lang/String;Lorg/apache/spark/sql/catalyst/types/DataType;Z)V
StackTrace:
org.elasticsearch.spark.sql.MappingUtils$.org$elasticsearch$spark$sql$MappingUtils$$convertField(MappingUtils.scala:75)
org.elasticsearch.spark.sql.MappingUtils$$anonfun$convertToStruct$1.apply(MappingUtils.scala:54)
org.elasticsearch.spark.sql.MappingUtils$$anonfun$convertToStruct$1.apply(MappingUtils.scala:54)
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108)
org.elasticsearch.spark.sql.MappingUtils$.convertToStruct(MappingUtils.scala:54)
org.elasticsearch.spark.sql.MappingUtils$.discoverMapping(MappingUtils.scala:47)
org.elasticsearch.spark.sql.EsSparkSQL$.esRDD(EsSparkSQL.scala:27)
org.elasticsearch.spark.sql.EsSparkSQL$.esRDD(EsSparkSQL.scala:23)
org.elasticsearch.spark.sql.package$SQLContextFunctions.esRDD(package.scala:16)
$line460.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$$$$$c57ec8bf9b0d5f6161b97741d596ff0$$$$wC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:111)
$line460.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:116)
$line460.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:118)
$line460.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$
What am I doing wrong? Thanks
- Michele
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to
elasticsearch+unsubscr...@googlegroups.com
<mailto:elasticsearch+unsubscr...@googlegroups.com>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/49a5c624-bda0-4835-98bc-915ec80d4fa3%40googlegroups.com
<https://groups.google.com/d/msgid/elasticsearch/49a5c624-bda0-4835-98bc-915ec80d4fa3%40googlegroups.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.
--
Costin
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/5534D710.3020402%40gmail.com.
For more options, visit https://groups.google.com/d/optout.
