[
https://issues.apache.org/jira/browse/HDFS-12283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16131642#comment-16131642
]
Anu Engineer edited comment on HDFS-12283 at 8/18/17 2:52 AM:
--------------------------------------------------------------
bq. But this also means TXID is non-unique, this might cause problems at some
places such as auditing ?
[~cheersyang] I did not think about it, thanks for pointing it out. I agree
let us keep the LastTXID in a separate key. This is similar to what HDFS does,
where we write this down in a separate file.
bq. Do you really want to keep retrying without a limit ?
At least for the short term, yes. Let us say we take the option of going with
-1, what happens when a delete fails, I don't think we have come up with a good
strategy to handle this case yet. We can only fail once we define what happens
after the failure of delete.
Or make this retry really big for time being, say 4096 retries with 5 min
intervals in between, that is a 2 weeks window before we fail.
was (Author: anu):
bq. But this also means TXID is non-unique, this might cause problems at some
places such as auditing ?
[~cheersyang] I did not think about it, thanks for pointing it out. I agree
let us keep the LastTXID in a separate key. This is similar to what HDFS does,
where we write this down in a separate file.
bq. Do you really want to keep retrying without a limit ?
At least for the short term, yes. Let us say we take the option of going with
-1, what happens when a delete fails, I don't think we have come up with a good
strategy to handle this case yet. We can only fail once we define what happens
after the failure of delete.
> Ozone: DeleteKey-5: Implement SCM DeletedBlockLog
> -------------------------------------------------
>
> Key: HDFS-12283
> URL: https://issues.apache.org/jira/browse/HDFS-12283
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone, scm
> Reporter: Weiwei Yang
> Assignee: Yuanbo Liu
> Attachments: HDFS-12283.001.patch, HDFS-12283-HDFS-7240.001.patch,
> HDFS-12283-HDFS-7240.002.patch, HDFS-12283-HDFS-7240.003.patch
>
>
> The DeletedBlockLog is a persisted log in SCM to keep tracking container
> blocks which are under deletion. It maintains info about under-deletion
> container blocks that notified by KSM, and the state how it is processed. We
> can use RocksDB to implement the 1st version of the log, the schema looks like
> ||TxID||ContainerName||Block List||ProcessedCount||
> |0|c1|b1,b2,b3|0|
> |1|c2|b1|3|
> |2|c2|b2, b3|-1|
> Some explanations
> # TxID is an incremental long value transaction ID for ONE container and
> multiple blocks
> # Container name is the name of the container
> # Block list is a list of block IDs
> # ProcessedCount is the number of times SCM has sent this record to datanode,
> it represents the "state" of the transaction, it is in range of \[-1, 5\], -1
> means the transaction eventually failed after some retries, 5 is the max
> number times of retries.
> We need to define {{DeletedBlockLog}} as an interface and implement this with
> RocksDB {{MetadataStore}} as the first version.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
