Thanks so much to Rich for comments/questions related to API design. They are
quite helpful. Please see my embedded response below.
> On Mar 20, 2020, at 1:23 AM, Richard Ellis <ricel...@uk.ibm.com> wrote:
>
> This is a nice proposal, but I have a few comments/questions regarding the
> proposed API:
>
> ## API
>
> 1) `DELETE /{db}`
>
> I wonder if there should be some consideration of the status code here.
> Currently for a db delete a 200/202 is returned depending if quorum is
> met. Do FDB transactional semantics mean that is likely to change at all?
> I can't really decide if a 202 always makes more sense for a soft-deletion
> (in the sense that the server has accepted the request to delete the
> database, but won't really do it for some time). On the other hand as soon
> as the database is in a soft-deleted state the {db} paths will return 404
> because the database is not operable at that path, so perhaps there isn't
> really any change.
With the soft-deletion, the semantic for `DELETE /{db}` in CouchDB 4.0 context
is not changed. Especially, it will not be long-run action because just a few
key/values are changed.
>
> 2) `GET /{db}/_deleted_dbs_info`
>
> It isn't shown in the example, but I'm assuming as this is a view-like
> response that the database name is the `key` for the row similarly to
> _dbs_info endpoint. Is that correct?
Different from `_dbs_info`, the `GET /{db}/_deleted_dbs_info` only returns the
information for specified database. So I didn’t add database name in returned
response before. If it is helpful and keeps consistent, I can add “key” in
every deleted instances.
>
> Would there be any advantage to using a standard timestamp format
> (specifically ISO 8601, which I think should be used for all new internet
> date/time representations). Compared to the examples shown this would
> probably be more user friendly for people to identify the actual time
> associated with instances and would have improved interoperability with
> other systems interacting with the API that might want to display the
> deleted date/time. Having said that I think the additional complexity of
> ISO8601 would be less useful as the identifier for restore operations,
> which would make me lean towards the UUID proposed by Nick for that part.
`20200318.071532` is based on original internal name for couch file. Yes, for
external usage, we can change it with ISO 8601 format. I am not sure to replace
timestamp with UUID because timestamp is meaningful and our operation team ever
used it to identify which database to restore.
Based on existing discussion, the modified response looks like:
{
"total_rows": 2,
"rows": [{
"key": "db01",
"timestamp": "2020-03-20T12:18:07Z",
"info": {
"update_seq":
"0000019100b5992700000000",
"doc_del_count": 0,
"doc_count": 3,
"sizes": {
"external": 287,
"views":
0
}
}
}, {
"key": "db01",
"timestamp": “2020-03-20T13:15:30Z",
"info": {
"update_seq":
"0000019105f0e29900000000",
"doc_del_count": 0,
"doc_count": 2,
"sizes": {
"external": 200,
"views":
0
}
}
}]
}
>
> 3) `PUT /{db}/_restore/{deletedTS}`
>
> It seems that this is basically identical in structure to the create
> operation. Are we expecting down the line a more complex set of options,
> different response structure, or different permission to a normal db
> create that warrants having a new endpoint? Given there is precedent for
> using a `partitioned=true` parameter to create a database in a "special"
> way, is there a good reason not to follow the same pattern here?
> e.g. PUT /{db}?restore=timestamp/uuid
>
> That would maintain the symmetry of the PUT/DELETE operations on the /{db}
> path. What I mean by this is that other `PUT /{db}/foo/bar` path
> operations always create sub-resources of a database (i.e. docs,
> design_docs, attachments etc) and 404 in the cases that the database does
> not exist. It seems strange to me to use a path structure similar to those
> that has different semantics because it doesn't 404 when the {db} part
> doesn't exist. Consider an equivalent could be the expectation that the
> `PUT /{db}/{doc_id}` operation would create the database if it doesn't
> exist instead of returning 404. Additionally usually the PUT verb is used
> for creating resources at the path of the request, but that is not the
> outcome of this call.
Good call. I ever have another alternative with `PUT
/{db}/_restore?deleteTS={deletedTS}`. But your suggestion with `PUT
/{db}?restore=timestamp/uuid` looks to be better. I don’t see other complex set
of options.
>
>
> I also wonder a little bit about the use of "restore" here. I guess there
> maybe should be a little consideration of whether there might be some
> future backup/restore functionality and whether it be able to restore via
> the same method or not. If not then calling this "restore" might cause
> confusion with any future backup/restore API, say, compared to calling
> this something like "undelete”.
Okay, to reduce confusion with backup/restore, we can select other words. How
about using “recover” instead of “undelete”? If so, endpoint will change to
`PUT /{db}?recover=timestamp/uuid`
>
> Thanks,
> Rich
>
>
>
> From: jiangph <jiangpeng...@hotmail.com>
> To: dev@couchdb.apache.org
> Date: 18/03/2020 12:05
> Subject: [EXTERNAL] [DISCUSS] soft-deletion
>
>
>
> Hi Everyone,
>
> Instead of automatically and immediately removing data and index in
> database after a delete operation, soft-deletion allows to restore the
> deleted data back to original state due to a “fat finger”or undesired
> delete operation, up to defined periods, such as 48 hours.
>
> In CouchDB 3.0, soft-deletion of database is implemented in [1]. The
> .couch file is renamed with the .<timestamp>.deleted.couch file after
> soft-deletion is enabled, and such file can be changed back to .couch for
> the purpose of restore. If restore is not needed and some specified period
> passed, the .<timestamp>.deleted.couch file can be deleted to achieve
> deletion of database permanently.
>
> In CouchDB 4.0, with the introduction of FoundationDB, the data model and
> storage is changed. In order to support soft-deletion, we propose below
> solution and then implement them.
>
> ## Proposed Data model change
>
> In CouchDB 4.0, directories and indirection access in FoundationDB are
> already used to better build data model. One key/value pair is used to
> build reference from Dbkey to DbPrefix. All other key/value pairs are
> based on DbPrefix instead of DbKey. This decouples the direct relationship
> between DBName and data in this database. The current implementation for
> `DBKey -> DBPrefix` is in [2]. So you can see below information in
> FoundationDB using fdbcli, etc.
>
> ```
> {?ALL_DBS, DbName} -> {?DBS, DbName}
> {?DBS, DbName, other part of key} -> <value>
> ```
>
> To support soft-deletion, especially allowing one database to be
> deleted/re-created multiple time, we need to use different DbPrefix for
> the same DbKey/DBName. The proposed change is to use a unique value
> allocated via High Contention Allocator(HCA) algorithm in [3].
>
>
> ```
> DbPrefixAllocator = erlfdb_hca:create(?ERLFDB_EXTEND(DbId,
> <<"hca">>)),,
> DbPrefix = erlfdb_hca:allocate(DBPrefixAllocator, Tx),
> erlfdb:set(Tx, DbKey, DbPrefix),
> ```
>
> The data in FoundationDB looks like:
>
> ```
> {?ALL_DBS, DbName} -> <unique key allocated by hca>
> {<unique key allocated by hca>, other part of key} -> <value>
> ```
>
> Using HCA algorithm, it can acquire one unique key quickly while avoiding
> conflicting. The more important, it is shorter enough to save space
> because `DBPrefix` exists in almost every key/value pair for database.
>
> ## Soft-deletion, restore and permanent-deletion
>
> Once database is soft-deleted, the only action is to change `DBKey ->
> DBPrefix` pair. All other data for this data is not changed. In order to
> give clear namespace management, the proposal is to move DBkey from
> `?ALL_DBS` to `?DELETED_DBS`. The timestamp when database was deleted is
> added to `DBKey` so that we can know when the data in this database can be
> permantenly restored. The `DBKey -> DBPrefix` pair is changed to
>
> ```
> {?DELETED_DBS, DbName, TimeStamp} -> <unique key allocated by hca>
> ```
>
> There is a background task to clear the ranges eventually. Depending on
> setting on how long the soft-deleted database will be kept, such as 48
> hours, the background task will check `DELETED_DBS` namespace, and find
> eligible key/value pairs, and delete data associated with this `DBPrefix`
> and then delete DbKey/DbPrefix pair finally.
>
> Overtime, it is possible that database can be deleted several times. The
> `_deleted_dbs_info` endpoint is proposed to list information about all
> deleted instances for the specified database, including deletion
> timestamp, document counts and disk size, etc. This allows users to
> identify which one to be restored. Also it also provides information for
> billing. In given period, such as 48 hours, the deletion times of same
> database is most likely limited, the design API is to list all instances
> in one time using GET method with query parameter.
>
> After deciding which instance to be restored, users can use the `_restore`
> endpoint with `deletedTS` to restore database. The underlying logic is to
> change `DBKey -> DBPrefix` back to
>
> ```
> {?ALL_DBS, DbName} -> <unique key allocated by hca>
> ```
>
> Considering the sensitive actions, the `_deleted_dbs_info` and `_restore`
> endpoints are supposed to be an admin-only endpoints only to allow granted
> user to restore the database.
>
> ##view index and search index
>
> Although view index and search index is based on the `DBPrefix`, the
> change of value of `DBPrefix` doesn't have impact on storage and search of
> view index and search index because `DBPrefix` is one opaque value. If
> database is soft-deleted, the `DBkey -> DBPrefix` pair is changed so that
> any access to view index and search index will be blocked with `"Database
> does not exist."` error. This is expected.
>
> The only thing we need to care is to stop all indexing or pending requests
> for soft-deleted database.
>
>
> ## API
>
> 1) `DELETE /{db}`
>
> There is no change on this endpoint [4] to send DELETE against one
> database. The soft-deletion is triggered once
> [couchdb][enable_database_recovery] is set to true in configuration file.
>
>
> 2) `GET /{db}/_deleted_dbs_info`
>
> returning basic information of all deleted instances for the specified
> database, including when the instance was deleted.
> Parameters:
>
> db –Database name
>
> Request Headers:
>
> Content-Type –application/json
>
> Response Headers:
>
>
> Content-Type –
> application/json
>
> Status Codes:
>
> 200 OK –Request completed successfully
> 404 Not Found –Requested database not found
>
> Request:
>
> GET /db/_deleted_dbs_info HTTP/1.1
> Accept: application/json
> Host: localhost:5984
>
> Response:
>
> HTTP/1.1 200 OK
> Cache-Control: must-revalidate
> Content-Type: application/json
> {
> "total_rows": 2,
> "rows": [{
> "deleted_when": "20200318.071532",
> "info": {
> "update_seq":
> "0000019100b5992700000000",
> "doc_del_count": 0,
> "doc_count": 3,
> "sizes": {
> "external": 287,
> "views":
> 0
> }
> }
> }, {
> "deleted_when": "20200318.071703",
> "info": {
> "update_seq":
> "0000019105f0e29900000000",
> "doc_del_count": 0,
> "doc_count": 2,
> "sizes": {
> "external": 200,
> "views":
> 0
> }
> }
> }]
> }
>
>
> 3) `PUT /{db}/_restore/{deletedTS}`
>
> Restore a deleted database.
> Parameters:
>
> db –Database name
> deletedTS - timestamp when database was deleted
>
> Request Headers:
>
>
> Accept –
> application/json
> text/plain
>
> Response Headers:
>
>
> Content-Type –
> application/json
> text/plain; charset=utf-8
>
> Response JSON Object:
>
>
> ok (boolean) –Operation status. Available in case of success
> error (string) –Error type. Available if response code is 4xx
> reason (string) –Error description. Available if response code is
> 4xx
>
> Status Codes:
>
> 200 Restored –Database restored successfully
> 400 Bad Request –Invalid database name or deleted timestamp
> 401 Unauthorized –CouchDB Server Administrator privileges required
> 412 Precondition Failed –Database already exists
>
>
> What do you think of that? Any questions or thoughts on this? Once again a
> big acknowledgment to Nick and Paul who helped with initial design and
> provide consultation on this.
>
>
> Cheers
> Peng Hui
>
>
> [1]
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_couchdb_blob_master_src_couch_src_couch-5Ffile.erl-23L251&d=DwIFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=CDCq0vbFWjQXx1sCFm2-iYoMZQ4i0QQj2XmPZmLvZp0&m=Axkxn5LYrVZw0h2AT61RjMT83ZlljDikPgUfCzAql5E&s=91sTDh27SxPlA-ejydi3-mlMFm5qA-makaWpFPhqrlM&e=
>
>
> [2]
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_couchdb_blob_prototype_fdb-2Dlayer_src_fabric_src_fabric2-5Ffdb.erl-23L182-2DL184&d=DwIFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=CDCq0vbFWjQXx1sCFm2-iYoMZQ4i0QQj2XmPZmLvZp0&m=Axkxn5LYrVZw0h2AT61RjMT83ZlljDikPgUfCzAql5E&s=44Jf80PJJPAsgg2LKZXYelRa8sWTxlm2CCtJN1JWOMg&e=
>
>
> [3]
> https://urldefense.proofpoint.com/v2/url?u=https-3A__activesphere.com_blog_2018_08_05_high-2Dcontention-2Dallocator&d=DwIFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=CDCq0vbFWjQXx1sCFm2-iYoMZQ4i0QQj2XmPZmLvZp0&m=Axkxn5LYrVZw0h2AT61RjMT83ZlljDikPgUfCzAql5E&s=5jLD7_hzgxzAhJP-NRqIbQuw2e8Q1vNeQQwAqXMsPBw&e=
>
> <
> https://urldefense.proofpoint.com/v2/url?u=https-3A__activesphere.com_blog_2018_08_05_high-2Dcontention-2Dallocator&d=DwIFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=CDCq0vbFWjQXx1sCFm2-iYoMZQ4i0QQj2XmPZmLvZp0&m=Axkxn5LYrVZw0h2AT61RjMT83ZlljDikPgUfCzAql5E&s=5jLD7_hzgxzAhJP-NRqIbQuw2e8Q1vNeQQwAqXMsPBw&e=
>
>>
> [4]
> https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.couchdb.org_en_stable_api_database_common.html-23delete-2D-2Ddb&d=DwIFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=CDCq0vbFWjQXx1sCFm2-iYoMZQ4i0QQj2XmPZmLvZp0&m=Axkxn5LYrVZw0h2AT61RjMT83ZlljDikPgUfCzAql5E&s=WkBi398mDF995FmYkRw0DeqwQSMtCD4xYLmPwntrRkI&e=
>
>
>
>
>
>
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>
