Hi All,
In CarbonDataFrameWriter, there is an option to load using CSV file.
if (options.tempCSV) {
loadTempCSV(options)
} else {
loadDataFrame(options)
}
Why is this choice required? Is there any issue if we load it directly
without using CSV?
I have many dimension table with comma in string columns, and so always use
.option("tempCSV", "false"). In CarbonOption can we set the default value
as "false" as below
def tempCSV: Boolean = options.getOrElse("tempCSV", "false").toBoolean
Thanks,
Sanoj
On Thu, Mar 30, 2017 at 12:14 PM, Sanoj MG (JIRA) <j...@apache.org> wrote:
> Sanoj MG created CARBONDATA-836:
> -----------------------------------
>
> Summary: Error in load using dataframe - columns containing
> comma
> Key: CARBONDATA-836
> URL: https://issues.apache.org/jira/browse/CARBONDATA-836
> Project: CarbonData
> Issue Type: Bug
> Components: spark-integration
> Affects Versions: 1.1.0-incubating
> Environment: HDP sandbox 2.5, Spark 1.6.2
> Reporter: Sanoj MG
> Priority: Minor
> Fix For: NONE
>
>
> While trying to load data into Carabondata table using dataframe, the
> columns containing commas are not properly loaded.
>
> Eg:
> scala> df.show(false)
> +-------+------+-----------+----------------+---------+------+
> |Country|Branch|Name |Address |ShortName|Status|
> +-------+------+-----------+----------------+---------+------+
> |2 |1 |Main Branch|XXXX, Dubai, UAE|UHO |256 |
> +-------+------+-----------+----------------+---------+------+
>
>
> scala> df.write.format("carbondata").option("tableName",
> "Branch1").option("compress", "true").mode(SaveMode.Overwrite).save()
>
>
> scala> cc.sql("select * from branch1").show(false)
>
> +-------+------+-----------+-------+---------+------+
> |country|branch|name |address|shortname|status|
> +-------+------+-----------+-------+---------+------+
> |2 |1 |Main Branch|XXXX | Dubai |null |
> +-------+------+-----------+-------+---------+------+
>
>
>
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v6.3.15#6346)
>
