applySchema() only accept RDD of Row/list/tuple, it does not work with
numpy.array.
After applySchema(), the Python RDD will be pickled and unpickled in
JVM, so you will not have any benefit by using numpy.array.
It will work if you convert ndarray into list:
schemaRDD = sqlContext.applySchema(rdd.map(list), schema)
On Mon, Dec 1, 2014 at 6:33 PM, Joseph Winston josephwins...@me.com wrote:
This works as expected in the 1.1 branch:
from pyspark.sql import *
rdd = sc.parallelize([range(0, 10), range(10,20), range(20, 30)]
# define the schema
schemaString = value1 value2 value3 value4 value5 value6 value7 value8
value9 value10
fields = [StructField(field_name, IntegerType(), True) for field_name in
schemaString.split()]
schema = StructType(fields)
# Apply the schema to the RDD.
schemaRDD = sqlContext.applySchema(rdd, schema)
# Register the table
schemaRDD.registerTempTable(slice)
# SQL can be run over SchemaRDDs that have been registered as a table.
results = sqlContext.sql(SELECT value1 FROM slice)
# The results of SQL queries are RDDs and support all the normal RDD
operations.
print results.collect()
However changing the rdd to use a numpy array fails:
import np as np
rdd = sc.parallelize(np.arange(20).reshape(2, 10))
# define the schema
schemaString = value1 value2 value3 value4 value5 value6 value7 value8
value9 value10
fields = [StructField(field_name, np.ndarray, True) for field_name in
schemaString.split()]
schema = StructType(fields)
# Apply the schema to the RDD.
schemaRDD = sqlContext.applySchema(rdd, schema)
The error is:
Traceback (most recent call last):
File stdin, line 2, in module
File /Users/jbw/src/Remote/GIT/spark/python/pyspark/sql.py, line 1119, in
applySchema
_verify_type(row, schema)
File /Users/jbw/src/Remote/GIT/spark/python/pyspark/sql.py, line 735, in
_verify_type
% (dataType, type(obj)))
TypeError: StructType(List(StructField(value1,type
'numpy.ndarray',true),StructField(value2,type
'numpy.ndarray',true),StructField(value3,type
'numpy.ndarray',true),StructField(value4,type
'numpy.ndarray',true),StructField(value5,type
'numpy.ndarray',true),StructField(value6,type
'numpy.ndarray',true),StructField(value7,type
'numpy.ndarray',true),StructField(value8,type
'numpy.ndarray',true),StructField(value9,type
'numpy.ndarray',true),StructField(value10,type 'numpy.ndarray',true))) can
not accept abject in type type 'numpy.ndarray'
I’ve tried np.int_ and np.int32 and they fail too. What type should I use to
make a numpy arrays work?
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
