Re: SparkSQL not support CharType

[Jörn Franke]

Or bytetype depending on the use case 

> On 23. Nov 2017, at 10:18, Herman van Hövell tot Westerflier 
>  wrote:
> 
> You need to use a StringType. The CharType and VarCharType are there to 
> ensure compatibility with Hive and ORC; they should not be used anywhere else.
> 
>> On Thu, Nov 23, 2017 at 4:09 AM, 163  wrote:
>> Hi,
>>  when I use Dataframe with table schema, It goes wrong:
>> 
>> val test_schema = StructType(Array(
>>   StructField("id", IntegerType, false),
>>   StructField("flag", CharType(1), false),
>>   StructField("time", DateType, false)));
>> 
>> val df = spark.read.format("com.databricks.spark.csv")
>>   .schema(test_schema)
>>   .option("header", "false")
>>   .option("inferSchema", "false")
>>   .option("delimiter", ",")
>>   .load("file:///Users/name/b")
>> 
>> The log is below:
>> Exception in thread "main" scala.MatchError: CharType(1) (of class 
>> org.apache.spark.sql.types.CharType)
>>  at 
>> org.apache.spark.sql.catalyst.encoders.RowEncoder$.org$apache$spark$sql$catalyst$encoders$RowEncoder$$serializerFor(RowEncoder.scala:73)
>>  at 
>> org.apache.spark.sql.catalyst.encoders.RowEncoder$$anonfun$2.apply(RowEncoder.scala:158)
>>  at 
>> org.apache.spark.sql.catalyst.encoders.RowEncoder$$anonfun$2.apply(RowEncoder.scala:157)
>> 
>> Why? Is this a bug?
>> 
>>  But I found spark will translate char type to string when using create 
>> table command:
>> 
>>   create table test(flag char(1));
>>  desc test:flag string;
>> 
>> 
>> 
>> 
>> Regards
>> Wendy He
> 
> 
> 
> -- 
> Herman van Hövell
> Software Engineer
> Databricks Inc.
> hvanhov...@databricks.com
> +31 6 420 590 27
> databricks.com
> 
> 
> 
> 

Re: SparkSQL not support CharType

[Herman van Hövell tot Westerflier]

You need to use a StringType. The CharType and VarCharType are there to
ensure compatibility with Hive and ORC; they should not be used anywhere
else.

On Thu, Nov 23, 2017 at 4:09 AM, 163  wrote:

> Hi,
>  when I use Dataframe with table schema, It goes wrong:
>
> val test_schema = StructType(Array(
>
>   StructField("id", IntegerType, false),
>   StructField("flag", CharType(1), false),
>   StructField("time", DateType, false)));
>
> val df = spark.read.format("com.databricks.spark.csv")
>   .schema(test_schema)
>   .option("header", "false")
>   .option("inferSchema", "false")
>   .option("delimiter", ",")
>   .load("file:///Users/name/b")
>
>
> The log is below:
> Exception in thread "main" scala.MatchError: CharType(1) (of class
> org.apache.spark.sql.types.CharType)
> at org.apache.spark.sql.catalyst.encoders.RowEncoder$.org$
> apache$spark$sql$catalyst$encoders$RowEncoder$$serializerFor(RowEncoder.
> scala:73)
> at org.apache.spark.sql.catalyst.encoders.RowEncoder$$anonfun$
> 2.apply(RowEncoder.scala:158)
> at org.apache.spark.sql.catalyst.encoders.RowEncoder$$anonfun$
> 2.apply(RowEncoder.scala:157)
>
> Why? Is this a bug?
>
> But I found spark will translate char type to string when using create
> table command:
>
>   create table test(flag char(1));
>   desc test:flag string;
>
>
>
>
> Regards
> Wendy He
>



-- 

Herman van Hövell

Software Engineer

Databricks Inc.

hvanhov...@databricks.com

+31 6 420 590 27

databricks.com

[image: http://databricks.com]