Can you share your code and a sample of your data? WIthout seeing it, I
can't give a definitive answer. I can offer some hints. If you have a
column of strings you should either be able to create a new column casted
to Integer. This can be accomplished two ways:
df.withColumn("newColumn", df.currentColumn.cast(IntegerType))
or
val df = df.select("cast(CurretColumn as int) newColum")
Without seeing your json, I really can't offer assistance.
On Thu, Jan 25, 2018 at 11:39 PM, kant kodali <kanth...@gmail.com> wrote:
> It seems like its hard to construct a DataType given its String literal
> representation.
>
> dataframe.types() return column names and its corresponding Types. for
> example say I have an integer column named "sum" doing dataframe.dtypes()
> would return "sum" and "IntegerType" but this string representation
> "IntegerType" doesnt seem to be very useful because I cannot do
> DataType.fromJson("IntegerType") This will throw an error. so I am not
> quite sure how to construct a DataType given its String representation ?
>
> On Thu, Jan 25, 2018 at 4:22 PM, kant kodali <kanth...@gmail.com> wrote:
>
>> Hi All,
>>
>> I have a datatype "IntegerType" represented as a String and now I want to
>> create DataType object out of that. I couldn't find in the DataType or
>> DataTypes api on how to do that?
>>
>> Thanks!
>>
>
>
