Igniters,
Lets discuss the IgniteRDD type mapping features.
You can see the problem discovery at the issue:
https://issues.apache.org/jira/browse/IGNITE-3175
* Description:
The current implementation IgniteRDD supports only types from
conjunction of the H2 types
wrappers & spark type set.
I've created the test to illustrate the problem with type mapping. The
test try to select the non-primitive field from the IgniteRDD
(please take a look the pull request:
https://github.com/apache/ignite/pull/784).
The exception scala.MatchError is thrown at the
IgniteRDDTypeMappingTest.java:111
scala.MatchError :
org.apache.ignite.spark.IgniteRDDTypeMappingTest$StructObj@68afa21 (of
class org.apache.ignite.spark.IgniteRDDTypeMappingTest$StructObj)
* Background:
Please take a look at the H2 type mapping & the Spark type mapping specs:
http://www.h2database.com/html/datatypes.html
https://spark.apache.org/docs/1.3.0/sql-programming-guide.html#data-types
* Root cause:
The information about class of the TestObj.structVal field is lost in
Ignite. IgniteRDD uses the
QueryCursor.fieldsMeta to discover type of the query fields. Ignite
returns java.lang.Object in according with
H2 type mapping. But AFAIK Spark doesn't support plain java objects.
--
Taras Ledkov
Mail-To: tled...@gridgain.com
