[
https://issues.apache.org/jira/browse/CARBONDATA-1168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16053380#comment-16053380
]
Kingsley Chen commented on CARBONDATA-1168:
-------------------------------------------
I tested it in the version of spark1.6+CarbonData-1.0.0-incubating, it can
process 100,000+ in one batch. So is that a bug of version 1.1.0?? Is there
anyone can help me???
> Driver Delete data operation is failed due to failure in creating delete
> delta file for segment
> -----------------------------------------------------------------------------------------------
>
> Key: CARBONDATA-1168
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1168
> Project: CarbonData
> Issue Type: Bug
> Components: sql
> Affects Versions: 1.1.0
> Environment: spark1.6+carbon1.1.0
> we have a 20 nodes cluster and 32GB ram each
> Reporter: Kingsley Chen
> Labels: features
> Fix For: NONE
>
>
> We use spark code to delete data from table as below
> ------------------spark code----------------------
> val deleteSql = s"DELETE FROM $tableName WHERE $rowkeyName IN
> (${rowKeyVals.mkString(",")})"
> cc.sql(deleteSql).show()
> ------------------spark code----------------------
> when the array size of rowKeyVals is larger than 200, the delete operation
> will be failed, and print the log:
> Delete data request has been received for default.item
> Delete data operation is failed for default.item
> Driver Delete data operation is failed due to failure in creating delete
> delta file for segment : null block : null
> ++
> ||
> ++
> ++
> That is to say, it only delete success maximun at 200 a batch,and took about
> 1min which is
> too slow. So my question is how to tuning the performance to make the batch
> larger
> and delete faster
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
