You meant for fields which are nullable.
Can you pastebin the complete stack trace ?
Try 1.6.1 when you have a chance.
Thanks
On Wed, Apr 20, 2016 at 10:20 PM, Charles Nnamdi Akalugwu <
cprenzb...@gmail.com> wrote:
> I get the same error for fields which are not null unfortunately.
>
> Can't translate null value for field
> StructField(density,DecimalType(4,2),true)
> On Apr 21, 2016 1:37 AM, "Ted Yu" <yuzhih...@gmail.com> wrote:
>
>> The weight field is not nullable.
>>
>> Looks like your source table had null value for this field.
>>
>> On Wed, Apr 20, 2016 at 4:11 PM, Charles Nnamdi Akalugwu <
>> cprenzb...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I am using spark 1.4.1 and trying to copy all rows from a table in one
>>> MySQL Database to a Amazon RDS table using spark SQL.
>>>
>>> Some columns in the source table are defined as DECIMAL type and are
>>> nullable. Others are not. When I run my spark job,
>>>
>>> val writeData = sqlContext.read.format("jdbc").option("url",
>>>>> sourceUrl).option("driver", "com.mysql.jdbc.Driver").option("dbtable",
>>>>> table).option("user", sourceUsername).option("password",
>>>>> sourcePassword).load()
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> writeData.write.format("com.databricks.spark.redshift").option("url",
>>>>> String.format(targetUrl, targetUsername,
>>>>> targetPassword)).option("dbtable",
>>>>> table).option("tempdir", redshiftTempDir+table).mode("append").save()
>>>>
>>>>
>>> it fails with the following exception
>>>
>>> Can't translate null value for field
>>>> StructField(weight,DecimalType(5,2),false)
>>>
>>>
>>> Any insights about this exception would be very helpful.
>>>
>>
>>
