[couchdb] branch main updated (5f43148 -> ec4b213)
This is an automated email from the ASF dual-hosted git repository. davisp pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/couchdb.git. from 5f43148 Merge pull request #3365 from apache/active-tasks-process-status-main add 79b64ea Fix typo add 73875b5 Add failing cases for ebtree:lookup_multi/3 bug add ec4b213 Fix ebtree:lookup_multi/3 No new revisions were added by this update. Summary of changes: src/couch_views/test/couch_views_indexer_test.erl | 42 ++- src/ebtree/src/ebtree.erl | 14 2 files changed, 41 insertions(+), 15 deletions(-)
[couchdb] branch fix-view-row-bug updated (2a66a03 -> f8507e7)
This is an automated email from the ASF dual-hosted git repository. davisp pushed a change to branch fix-view-row-bug in repository https://gitbox.apache.org/repos/asf/couchdb.git. discard 2a66a03 Fix ebtree:lookup_multi/3 discard dc7382f Add failing cases for ebtree:lookup_multi/3 bug add 5a97edc Add failing cases for ebtree:lookup_multi/3 bug add f8507e7 Fix ebtree:lookup_multi/3 This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (2a66a03) \ N -- N -- N refs/heads/fix-view-row-bug (f8507e7) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: src/couch_views/test/couch_views_indexer_test.erl | 1 - 1 file changed, 1 deletion(-)
[couchdb] 02/03: Add failing cases for ebtree:lookup_multi/3 bug
This is an automated email from the ASF dual-hosted git repository.
davisp pushed a commit to branch fix-view-row-bug
in repository https://gitbox.apache.org/repos/asf/couchdb.git
commit dc7382fa11a64fc4b63d19cc4ba95d313340d8a7
Author: Paul J. Davis
AuthorDate: Fri Feb 26 13:08:42 2021 -0600
Add failing cases for ebtree:lookup_multi/3 bug
These two test cases expose the subtle bug in ebtree:lookup_multi/3
where a key that doesn't exist in the tree can prevent a subsequent
lookup key from matching in the same KV node.
---
src/couch_views/test/couch_views_indexer_test.erl | 27 +++
src/ebtree/src/ebtree.erl | 3 ++-
2 files changed, 29 insertions(+), 1 deletion(-)
diff --git a/src/couch_views/test/couch_views_indexer_test.erl
b/src/couch_views/test/couch_views_indexer_test.erl
index e18c007..09493db 100644
--- a/src/couch_views/test/couch_views_indexer_test.erl
+++ b/src/couch_views/test/couch_views_indexer_test.erl
@@ -47,6 +47,7 @@ indexer_test_() ->
?TDEF_FE(multiple_identical_keys_from_same_doc),
?TDEF_FE(fewer_multiple_identical_keys_from_same_doc),
?TDEF_FE(multiple_design_docs),
+?TDEF_FE(multiple_doc_update_with_existing_rows),
?TDEF_FE(handle_size_key_limits),
?TDEF_FE(handle_size_value_limits),
?TDEF_FE(index_autoupdater_callback),
@@ -424,6 +425,32 @@ multiple_design_docs(Db) ->
?assertError({ddoc_deleted, _}, run_query(Db, DDoc2, ?MAP_FUN1)).
+multiple_doc_update_with_existing_rows(Db) ->
+couch_log:error("START TEST", []),
+DDoc = create_ddoc(),
+Doc0 = doc(0),
+Doc1 = doc(1),
+
+{ok, _} = fabric2_db:update_doc(Db, DDoc, []),
+{ok, {Pos, Rev}} = fabric2_db:update_doc(Db, Doc1, []),
+
+{ok, Out1} = run_query(Db, DDoc, ?MAP_FUN1),
+
+?assertEqual([row(<<"1">>, 1, 1)], Out1),
+
+Doc2 = Doc1#doc{
+revs = {Pos, [Rev]},
+body = {[{<<"val">>, 2}]}
+},
+{ok, _} = fabric2_db:update_docs(Db, [Doc0, Doc2], []),
+
+{ok, Out2} = run_query(Db, DDoc, ?MAP_FUN1),
+
+?assertEqual([
+row(<<"0">>, 0, 0),
+row(<<"1">>, 2, 2)
+], Out2).
+
handle_db_recreated_when_running(Db) ->
DbName = fabric2_db:name(Db),
diff --git a/src/ebtree/src/ebtree.erl b/src/ebtree/src/ebtree.erl
index 1a65c9a..bbb0ca8 100644
--- a/src/ebtree/src/ebtree.erl
+++ b/src/ebtree/src/ebtree.erl
@@ -1394,7 +1394,8 @@ lookup_multi_test() ->
validate_tree(Db, Tree),
?assertEqual([{1, 2}], lookup_multi(Db, Tree, [1])),
?assertEqual([{15, 16}, {2, 3}], lookup_multi(Db, Tree, [2, 15])),
-?assertEqual([{15, 16}, {4, 5}, {2, 3}], lookup_multi(Db, Tree, [2, 101,
15, 4, -3])).
+?assertEqual([{15, 16}, {4, 5}, {2, 3}], lookup_multi(Db, Tree, [2, 101,
15, 4, -3])),
+?assertEqual([{2, 3}], lookup_multi(Db, Tree, [1.5, 2])).
insert_multi_test() ->
[couchdb] branch fix-view-row-bug created (now 2a66a03)
This is an automated email from the ASF dual-hosted git repository. davisp pushed a change to branch fix-view-row-bug in repository https://gitbox.apache.org/repos/asf/couchdb.git. at 2a66a03 Fix ebtree:lookup_multi/3 This branch includes the following new commits: new c9440e6 Fix typo new dc7382f Add failing cases for ebtree:lookup_multi/3 bug new 2a66a03 Fix ebtree:lookup_multi/3 The 3 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[couchdb] 03/03: Fix ebtree:lookup_multi/3
This is an automated email from the ASF dual-hosted git repository.
davisp pushed a commit to branch fix-view-row-bug
in repository https://gitbox.apache.org/repos/asf/couchdb.git
commit 2a66a03eb55f88286f3a3c513271daeb34946b30
Author: Paul J. Davis
AuthorDate: Fri Feb 26 12:37:04 2021 -0600
Fix ebtree:lookup_multi/3
If one of the provided lookup keys doesn't exist in the ebtree, it can
inadvertently prevent a second lookup key from being found if it the
first key greater than the missing lookup key is equal to the second
lookup key.
---
src/ebtree/src/ebtree.erl | 11 +--
1 file changed, 5 insertions(+), 6 deletions(-)
diff --git a/src/ebtree/src/ebtree.erl b/src/ebtree/src/ebtree.erl
index bbb0ca8..8c921d1 100644
--- a/src/ebtree/src/ebtree.erl
+++ b/src/ebtree/src/ebtree.erl
@@ -173,18 +173,17 @@ lookup_multi_fold(_, {_, [], _} = Acc) ->
{stop, Acc};
lookup_multi_fold({visit, Key1, Value}, {Tree, [Key2 | Rest], Acc}) ->
-{NewKeys, NewAcc} = case collate(Tree, Key1, Key2) of
+case collate(Tree, Key1, Key2) of
lt ->
% Still looking for the next user key
-{[Key2 | Rest], Acc};
+{ok, {Tree, [Key2 | Rest], Acc}};
eq ->
% Found a requested key
-{Rest, [{Key2, Value} | Acc]};
+{ok, {Tree, Rest, [{Key2, Value} | Acc]}};
gt ->
% The user key wasn't found so we drop it
-{Rest, Acc}
-end,
-{ok, {Tree, NewKeys, NewAcc}};
+lookup_multi_fold({visit, Key1, Value}, {Tree, Rest, Acc})
+end;
lookup_multi_fold({traverse, FKey, LKey, R}, {Tree, [UKey | Rest], Acc}) ->
case collate(Tree, FKey, UKey, [gt]) of
[couchdb] 01/03: Fix typo
This is an automated email from the ASF dual-hosted git repository.
davisp pushed a commit to branch fix-view-row-bug
in repository https://gitbox.apache.org/repos/asf/couchdb.git
commit c9440e67ab57c8d3e583af46f143394d3221936e
Author: Paul J. Davis
AuthorDate: Fri Feb 26 12:45:19 2021 -0600
Fix typo
---
src/couch_views/test/couch_views_indexer_test.erl | 16
1 file changed, 8 insertions(+), 8 deletions(-)
diff --git a/src/couch_views/test/couch_views_indexer_test.erl
b/src/couch_views/test/couch_views_indexer_test.erl
index a0890da..e18c007 100644
--- a/src/couch_views/test/couch_views_indexer_test.erl
+++ b/src/couch_views/test/couch_views_indexer_test.erl
@@ -42,10 +42,10 @@ indexer_test_() ->
?TDEF_FE(updated_docs_without_changes_are_reindexed),
?TDEF_FE(deleted_docs_not_indexed),
?TDEF_FE(deleted_docs_are_unindexed),
-?TDEF_FE(multipe_docs_with_same_key),
-?TDEF_FE(multipe_keys_from_same_doc),
-?TDEF_FE(multipe_identical_keys_from_same_doc),
-?TDEF_FE(fewer_multipe_identical_keys_from_same_doc),
+?TDEF_FE(multiple_docs_with_same_key),
+?TDEF_FE(multiple_keys_from_same_doc),
+?TDEF_FE(multiple_identical_keys_from_same_doc),
+?TDEF_FE(fewer_multiple_identical_keys_from_same_doc),
?TDEF_FE(multiple_design_docs),
?TDEF_FE(handle_size_key_limits),
?TDEF_FE(handle_size_value_limits),
@@ -216,7 +216,7 @@ deleted_docs_are_unindexed(Db) ->
end).
-multipe_docs_with_same_key(Db) ->
+multiple_docs_with_same_key(Db) ->
DDoc = create_ddoc(),
Doc1 = doc(0, 1),
Doc2 = doc(1, 1),
@@ -232,7 +232,7 @@ multipe_docs_with_same_key(Db) ->
], Out).
-multipe_keys_from_same_doc(Db) ->
+multiple_keys_from_same_doc(Db) ->
DDoc = create_ddoc(multi_emit_different),
Doc = doc(0, 1),
@@ -247,7 +247,7 @@ multipe_keys_from_same_doc(Db) ->
], Out).
-multipe_identical_keys_from_same_doc(Db) ->
+multiple_identical_keys_from_same_doc(Db) ->
DDoc = create_ddoc(multi_emit_same),
Doc = doc(0, 1),
@@ -262,7 +262,7 @@ multipe_identical_keys_from_same_doc(Db) ->
], Out).
-fewer_multipe_identical_keys_from_same_doc(Db) ->
+fewer_multiple_identical_keys_from_same_doc(Db) ->
DDoc = create_ddoc(multi_emit_same),
Doc0 = #doc{
id = <<"0">>,
[couchdb] branch 3.x-devnode-boot-script updated (e123adf -> 4638b49)
This is an automated email from the ASF dual-hosted git repository. davisp pushed a change to branch 3.x-devnode-boot-script in repository https://gitbox.apache.org/repos/asf/couchdb.git. from e123adf Rename boot_node to monitor_parent add 4638b49 Add formatting suggestions from python-black No new revisions were added by this update. Summary of changes: dev/run | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
[couchdb-erlfdb] tag v1.2.4 created (now efef20c)
This is an automated email from the ASF dual-hosted git repository. davisp pushed a change to tag v1.2.4 in repository https://gitbox.apache.org/repos/asf/couchdb-erlfdb.git. at efef20c (commit) No new revisions were added by this update.
[couchdb-erlfdb] branch main updated: Avoid sending future ready messages when flushed
This is an automated email from the ASF dual-hosted git repository.
davisp pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/couchdb-erlfdb.git
The following commit(s) were added to refs/heads/main by this push:
new efef20c Avoid sending future ready messages when flushed
efef20c is described below
commit efef20c439daa6c24227e840482f82945e5c2437
Author: Paul J. Davis
AuthorDate: Mon Dec 14 15:21:22 2020 -0600
Avoid sending future ready messages when flushed
There's a small race condition in `erlfdb:flush_future_message` if the
`is_ready` returns true before the future's callback is invoked to send
the ready message. This adds an API to silence the message from a future
to close this race condition.
Fixes apache/couchdb#3294
---
c_src/main.c | 39 +++
c_src/resources.h | 1 +
src/erlfdb.erl | 1 +
src/erlfdb_nif.erl | 7 +++
4 files changed, 48 insertions(+)
diff --git a/c_src/main.c b/c_src/main.c
index b1db949..4f669f5 100644
--- a/c_src/main.c
+++ b/c_src/main.c
@@ -84,11 +84,15 @@ erlfdb_future_cb(FDBFuture* fdb_future, void* data)
caller = future->pid_env;
}
+enif_mutex_lock(future->lock);
+
if(!future->cancelled) {
msg = T2(future->msg_env, future->msg_ref, ATOM_ready);
enif_send(caller, &(future->pid), future->msg_env, msg);
}
+enif_mutex_unlock(future->lock);
+
// We're now done with this future which means we need
// to release our handle to it. See erlfdb_create_future
// for more on why this happens here.
@@ -114,6 +118,7 @@ erlfdb_create_future(ErlNifEnv* env, FDBFuture* future,
ErlFDBFutureType ftype)
f->pid_env = env;
f->msg_env = enif_alloc_env();
f->msg_ref = enif_make_copy(f->msg_env, ref);
+f->lock = enif_mutex_create("fdb:future_lock");
f->cancelled = false;
// This resource reference counting dance is a bit
@@ -579,14 +584,47 @@ erlfdb_future_cancel(ErlNifEnv* env, int argc, const
ERL_NIF_TERM argv[])
}
future = (ErlFDBFuture*) res;
+enif_mutex_lock(future->lock);
+
future->cancelled = true;
fdb_future_cancel(future->future);
+enif_mutex_unlock(future->lock);
+
return ATOM_ok;
}
static ERL_NIF_TERM
+erlfdb_future_silence(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[])
+{
+ErlFDBSt* st = (ErlFDBSt*) enif_priv_data(env);
+ErlFDBFuture* future;
+void* res;
+
+if(st->lib_state != ErlFDB_CONNECTED) {
+return enif_make_badarg(env);
+}
+
+if(argc != 1) {
+return enif_make_badarg(env);
+}
+
+if(!enif_get_resource(env, argv[0], ErlFDBFutureRes, )) {
+return enif_make_badarg(env);
+}
+future = (ErlFDBFuture*) res;
+
+enif_mutex_lock(future->lock);
+
+future->cancelled = true;
+
+enif_mutex_unlock(future->lock);
+
+return ATOM_ok;
+}
+
+static ERL_NIF_TERM
erlfdb_future_is_ready(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[])
{
ErlFDBSt* st = (ErlFDBSt*) enif_priv_data(env);
@@ -2158,6 +2196,7 @@ static ErlNifFunc funcs[] =
NIF_FUNC(erlfdb_setup_network, 0),
NIF_FUNC(erlfdb_future_cancel, 1),
+NIF_FUNC(erlfdb_future_silence, 1),
NIF_FUNC(erlfdb_future_is_ready, 1),
NIF_FUNC(erlfdb_future_get_error, 1),
NIF_FUNC(erlfdb_future_get, 1),
diff --git a/c_src/resources.h b/c_src/resources.h
index 6694a99..b25a43f 100644
--- a/c_src/resources.h
+++ b/c_src/resources.h
@@ -44,6 +44,7 @@ typedef struct _ErlFDBFuture
ErlNifEnv* pid_env;
ErlNifEnv* msg_env;
ERL_NIF_TERM msg_ref;
+ErlNifMutex* lock;
bool cancelled;
} ErlFDBFuture;
diff --git a/src/erlfdb.erl b/src/erlfdb.erl
index d2ddb1f..506b4b0 100644
--- a/src/erlfdb.erl
+++ b/src/erlfdb.erl
@@ -789,6 +789,7 @@ options_to_fold_st(StartKey, EndKey, Options) ->
flush_future_message(?IS_FUTURE = Future) ->
+erlfdb_nif:future_silence(Future),
{erlfdb_future, MsgRef, _Res} = Future,
receive
{MsgRef, ready} -> ok
diff --git a/src/erlfdb_nif.erl b/src/erlfdb_nif.erl
index 7ec8e52..ddb7d0f 100644
--- a/src/erlfdb_nif.erl
+++ b/src/erlfdb_nif.erl
@@ -21,6 +21,7 @@
get_max_api_version/0,
future_cancel/1,
+future_silence/1,
future_is_ready/1,
future_get_error/1,
future_get/1,
@@ -195,6 +196,11 @@ future_cancel({erlfdb_future, _Ref, Ft}) ->
erlfdb_future_cancel(Ft).
+-spec future_silence(future()) -> ok.
+future_silence({erlfdb_future, _Ref, Ft}) ->
+erlfdb_future_silence(Ft).
+
+
-spec future_is_ready(future()) -> boolean().
future_is_ready({erlfdb_future, _Ref, Ft}) ->
erlfdb_future_is_ready(Ft).
@@ -527,6 +533,7 @@ erlfdb_setup_network() -> ?NOT_LOADED.
% Futures
erlfdb_future_cancel(_Future) -> ?NOT_LOADED.
+erlfdb_future_silence(_Future) -&g
[couchdb-erlfdb] branch add-silence-future-api created (now 50e2c9b)
This is an automated email from the ASF dual-hosted git repository. davisp pushed a change to branch add-silence-future-api in repository https://gitbox.apache.org/repos/asf/couchdb-erlfdb.git. at 50e2c9b Avoid sending future ready messages when flushed This branch includes the following new commits: new 50e2c9b Avoid sending future ready messages when flushed The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[couchdb-erlfdb] 01/01: Avoid sending future ready messages when flushed
This is an automated email from the ASF dual-hosted git repository.
davisp pushed a commit to branch add-silence-future-api
in repository https://gitbox.apache.org/repos/asf/couchdb-erlfdb.git
commit 50e2c9b07e56b669a9068308d2f95c6517aa75a4
Author: Paul J. Davis
AuthorDate: Mon Dec 14 15:21:22 2020 -0600
Avoid sending future ready messages when flushed
There's a small race condition in `erlfdb:flush_future_message` if the
`is_ready` returns true before the future's callback is invoked to send
the ready message. This adds an API to silence the message from a future
to close this race condition.
Fixes apache/couchdb#3294
---
c_src/main.c | 39 +++
c_src/resources.h | 1 +
src/erlfdb.erl | 1 +
src/erlfdb_nif.erl | 7 +++
4 files changed, 48 insertions(+)
diff --git a/c_src/main.c b/c_src/main.c
index b1db949..4f669f5 100644
--- a/c_src/main.c
+++ b/c_src/main.c
@@ -84,11 +84,15 @@ erlfdb_future_cb(FDBFuture* fdb_future, void* data)
caller = future->pid_env;
}
+enif_mutex_lock(future->lock);
+
if(!future->cancelled) {
msg = T2(future->msg_env, future->msg_ref, ATOM_ready);
enif_send(caller, &(future->pid), future->msg_env, msg);
}
+enif_mutex_unlock(future->lock);
+
// We're now done with this future which means we need
// to release our handle to it. See erlfdb_create_future
// for more on why this happens here.
@@ -114,6 +118,7 @@ erlfdb_create_future(ErlNifEnv* env, FDBFuture* future,
ErlFDBFutureType ftype)
f->pid_env = env;
f->msg_env = enif_alloc_env();
f->msg_ref = enif_make_copy(f->msg_env, ref);
+f->lock = enif_mutex_create("fdb:future_lock");
f->cancelled = false;
// This resource reference counting dance is a bit
@@ -579,14 +584,47 @@ erlfdb_future_cancel(ErlNifEnv* env, int argc, const
ERL_NIF_TERM argv[])
}
future = (ErlFDBFuture*) res;
+enif_mutex_lock(future->lock);
+
future->cancelled = true;
fdb_future_cancel(future->future);
+enif_mutex_unlock(future->lock);
+
return ATOM_ok;
}
static ERL_NIF_TERM
+erlfdb_future_silence(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[])
+{
+ErlFDBSt* st = (ErlFDBSt*) enif_priv_data(env);
+ErlFDBFuture* future;
+void* res;
+
+if(st->lib_state != ErlFDB_CONNECTED) {
+return enif_make_badarg(env);
+}
+
+if(argc != 1) {
+return enif_make_badarg(env);
+}
+
+if(!enif_get_resource(env, argv[0], ErlFDBFutureRes, )) {
+return enif_make_badarg(env);
+}
+future = (ErlFDBFuture*) res;
+
+enif_mutex_lock(future->lock);
+
+future->cancelled = true;
+
+enif_mutex_unlock(future->lock);
+
+return ATOM_ok;
+}
+
+static ERL_NIF_TERM
erlfdb_future_is_ready(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[])
{
ErlFDBSt* st = (ErlFDBSt*) enif_priv_data(env);
@@ -2158,6 +2196,7 @@ static ErlNifFunc funcs[] =
NIF_FUNC(erlfdb_setup_network, 0),
NIF_FUNC(erlfdb_future_cancel, 1),
+NIF_FUNC(erlfdb_future_silence, 1),
NIF_FUNC(erlfdb_future_is_ready, 1),
NIF_FUNC(erlfdb_future_get_error, 1),
NIF_FUNC(erlfdb_future_get, 1),
diff --git a/c_src/resources.h b/c_src/resources.h
index 6694a99..b25a43f 100644
--- a/c_src/resources.h
+++ b/c_src/resources.h
@@ -44,6 +44,7 @@ typedef struct _ErlFDBFuture
ErlNifEnv* pid_env;
ErlNifEnv* msg_env;
ERL_NIF_TERM msg_ref;
+ErlNifMutex* lock;
bool cancelled;
} ErlFDBFuture;
diff --git a/src/erlfdb.erl b/src/erlfdb.erl
index d2ddb1f..506b4b0 100644
--- a/src/erlfdb.erl
+++ b/src/erlfdb.erl
@@ -789,6 +789,7 @@ options_to_fold_st(StartKey, EndKey, Options) ->
flush_future_message(?IS_FUTURE = Future) ->
+erlfdb_nif:future_silence(Future),
{erlfdb_future, MsgRef, _Res} = Future,
receive
{MsgRef, ready} -> ok
diff --git a/src/erlfdb_nif.erl b/src/erlfdb_nif.erl
index 7ec8e52..ddb7d0f 100644
--- a/src/erlfdb_nif.erl
+++ b/src/erlfdb_nif.erl
@@ -21,6 +21,7 @@
get_max_api_version/0,
future_cancel/1,
+future_silence/1,
future_is_ready/1,
future_get_error/1,
future_get/1,
@@ -195,6 +196,11 @@ future_cancel({erlfdb_future, _Ref, Ft}) ->
erlfdb_future_cancel(Ft).
+-spec future_silence(future()) -> ok.
+future_silence({erlfdb_future, _Ref, Ft}) ->
+erlfdb_future_silence(Ft).
+
+
-spec future_is_ready(future()) -> boolean().
future_is_ready({erlfdb_future, _Ref, Ft}) ->
erlfdb_future_is_ready(Ft).
@@ -527,6 +533,7 @@ erlfdb_setup_network() -> ?NOT_LOADED.
% Futures
erlfdb_future_cancel(_Future) -> ?NOT_LOADED.
+erlfdb_future_silence(_Future) -> ?NOT_LOADED.
erlfdb_future_is_ready(_Future) -> ?NOT_LOADED.
erlfdb_future_get_error(_Future) -> ?NOT_LOADED.
erlfdb_future_get(_Future) -> ?NOT_LOADED.
[couchdb-infra-cm] branch main updated: Ensure root account is locked
This is an automated email from the ASF dual-hosted git repository. davisp pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/couchdb-infra-cm.git The following commit(s) were added to refs/heads/main by this push: new 24bea20 Ensure root account is locked 24bea20 is described below commit 24bea20e9a098cf4a462a4b5d9dbabbea84f2ec6 Author: Paul J. Davis AuthorDate: Mon Dec 14 15:11:50 2020 -0600 Ensure root account is locked Its locked by default but we might as well add the extra protection just in case. --- roles/common/tasks/main.yml | 5 + 1 file changed, 5 insertions(+) diff --git a/roles/common/tasks/main.yml b/roles/common/tasks/main.yml index 814e1e8..639 100644 --- a/roles/common/tasks/main.yml +++ b/roles/common/tasks/main.yml @@ -9,6 +9,11 @@ - https://github.com/nickva.keys - https://github.com/wohali.keys +- name: Lock root account + user: +name: root +password_lock: true + - name: Disable password authentication lineinfile: dest: /etc/ssh/sshd_config
[couchdb-infra-cm] branch main updated: Harden SSH settings on CI nodes
This is an automated email from the ASF dual-hosted git repository.
davisp pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/couchdb-infra-cm.git
The following commit(s) were added to refs/heads/main by this push:
new dc63a8c Harden SSH settings on CI nodes
dc63a8c is described below
commit dc63a8cd29401edb0b79761fcf88ac32fe3797d2
Author: Paul J. Davis
AuthorDate: Mon Dec 14 14:41:59 2020 -0600
Harden SSH settings on CI nodes
---
roles/common/handlers/main.yml | 4
roles/common/tasks/main.yml| 22 ++
2 files changed, 26 insertions(+)
diff --git a/roles/common/handlers/main.yml b/roles/common/handlers/main.yml
new file mode 100644
index 000..660b2b7
--- /dev/null
+++ b/roles/common/handlers/main.yml
@@ -0,0 +1,4 @@
+- name: restart sshd
+ service:
+name: sshd
+state: restarted
\ No newline at end of file
diff --git a/roles/common/tasks/main.yml b/roles/common/tasks/main.yml
index a449f36..814e1e8 100644
--- a/roles/common/tasks/main.yml
+++ b/roles/common/tasks/main.yml
@@ -9,6 +9,28 @@
- https://github.com/nickva.keys
- https://github.com/wohali.keys
+- name: Disable password authentication
+ lineinfile:
+dest: /etc/ssh/sshd_config
+regexp: '^(#\s*)?PasswordAuthentication'
+line: "PasswordAuthentication no"
+state: present
+ notify: restart sshd
+
+- name: Disable empty password login
+ lineinfile:
+dest: /etc/ssh/sshd_config
+regexp: '^(#\s*)?PermitEmptyPasswords'
+line: 'PermitEmptyPasswords no'
+ notify: restart sshd
+
+- name: Disable challenge response authentication
+ lineinfile:
+dest: /etc/ssh/sshd_config
+regexp: '^(#\s*)?ChallengeResponseAuthentication'
+line: 'ChallengeResponseAuthentication no'
+ notify: restart sshd
+
- name: Install basic ubiquitous packages
apt:
name: "{{ packages }}"
[couchdb-erlfdb] branch add-stats created (now 187b585)
This is an automated email from the ASF dual-hosted git repository. davisp pushed a change to branch add-stats in repository https://gitbox.apache.org/repos/asf/couchdb-erlfdb.git. at 187b585 Add metrics for tracking futures This branch includes the following new commits: new 187b585 Add metrics for tracking futures The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[couchdb-erlfdb] 01/01: Add metrics for tracking futures
This is an automated email from the ASF dual-hosted git repository.
davisp pushed a commit to branch add-stats
in repository https://gitbox.apache.org/repos/asf/couchdb-erlfdb.git
commit 187b585b0bc7c2bcc27fd875f3d59e54a9c028ac
Author: Paul J. Davis
AuthorDate: Thu Dec 10 13:14:22 2020 -0600
Add metrics for tracking futures
---
c_src/atom_names.h | 6 +
c_src/main.c | 38 +++-
c_src/metrics.cc | 74 ++
c_src/metrics.h| 28 +
c_src/resources.c | 3 +++
rebar.config | 4 +--
src/erlfdb.erl | 7 +-
src/erlfdb_nif.erl | 8 +-
8 files changed, 163 insertions(+), 5 deletions(-)
diff --git a/c_src/atom_names.h b/c_src/atom_names.h
index 98b5c4f..377df1f 100644
--- a/c_src/atom_names.h
+++ b/c_src/atom_names.h
@@ -143,3 +143,9 @@ ATOM_MAP(write);
ATOM_MAP(retryable);
ATOM_MAP(maybe_committed);
ATOM_MAP(retryable_not_committed);
+
+// Metrics
+ATOM_MAP(futures_created);
+ATOM_MAP(futures_destroyed);
+ATOM_MAP(futures_fired);
+ATOM_MAP(futures_read);
\ No newline at end of file
diff --git a/c_src/main.c b/c_src/main.c
index b1db949..e7ac014 100644
--- a/c_src/main.c
+++ b/c_src/main.c
@@ -18,6 +18,7 @@
#include "fdb.h"
#include "atoms.h"
+#include "metrics.h"
#include "resources.h"
#include "util.h"
@@ -95,6 +96,8 @@ erlfdb_future_cb(FDBFuture* fdb_future, void* data)
enif_release_resource(future);
+erlfdb_future_fired();
+
return;
}
@@ -159,6 +162,8 @@ erlfdb_create_future(ErlNifEnv* env, FDBFuture* future,
ErlFDBFutureType ftype)
// thread has a reference. If its 1 then only
// Erlang has a reference.
+erlfdb_future_created();
+
return T3(env, ATOM_erlfdb_future, ref, ret);
}
@@ -173,6 +178,8 @@ erlfdb_future_get_void(ErlNifEnv* env, ErlFDBFuture* f)
return erlfdb_erlang_error(env, err);
}
+erlfdb_future_read();
+
return ATOM_ok;
}
@@ -191,6 +198,8 @@ erlfdb_future_get_int64(ErlNifEnv* env, ErlFDBFuture* f)
nif_res = fdb_res;
+erlfdb_future_read();
+
return enif_make_int64(env, nif_res);
}
@@ -212,6 +221,8 @@ erlfdb_future_get_key(ErlNifEnv* env, ErlFDBFuture* f)
buf = enif_make_new_binary(env, len, );
memcpy(buf, key, len);
+erlfdb_future_read();
+
return ret;
}
@@ -238,6 +249,8 @@ erlfdb_future_get_value(ErlNifEnv* env, ErlFDBFuture* f)
buf = enif_make_new_binary(env, len, );
memcpy(buf, val, len);
+erlfdb_future_read();
+
return ret;
}
@@ -266,6 +279,8 @@ erlfdb_future_get_string_array(ErlNifEnv* env,
ErlFDBFuture* f)
ret = enif_make_list_cell(env, bin, ret);
}
+erlfdb_future_read();
+
return ret;
}
@@ -298,6 +313,8 @@ erlfdb_future_get_keyvalue_array(ErlNifEnv* env,
ErlFDBFuture* f)
ret = enif_make_list_cell(env, T2(env, key, val), ret);
}
+erlfdb_future_read();
+
if(more) {
return T3(env, ret, enif_make_int(env, count), ATOM_true);
} else {
@@ -2145,6 +2162,23 @@ erlfdb_error_predicate(ErlNifEnv* env, int argc, const
ERL_NIF_TERM argv[])
}
}
+static ERL_NIF_TERM
+erlfdb_get_metrics(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[])
+{
+ERL_NIF_TERM metrics[4];
+
+ERL_NIF_TERM created = enif_make_uint64(env, erlfdb_num_futures_created());
+ERL_NIF_TERM destroyed = enif_make_uint64(env,
erlfdb_num_futures_destroyed());
+ERL_NIF_TERM fired = enif_make_uint64(env, erlfdb_num_futures_fired());
+ERL_NIF_TERM read = enif_make_uint64(env, erlfdb_num_futures_read());
+
+metrics[0] = T2(env, ATOM_futures_created, created);
+metrics[1] = T2(env, ATOM_futures_destroyed, destroyed);
+metrics[2] = T2(env, ATOM_futures_fired, fired);
+metrics[3] = T2(env, ATOM_futures_read, read);
+
+return enif_make_list_from_array(env, metrics, 4);
+}
#define NIF_FUNC(name, arity) {#name, arity, name}
static ErlNifFunc funcs[] =
@@ -2192,7 +2226,9 @@ static ErlNifFunc funcs[] =
NIF_FUNC(erlfdb_transaction_get_writes_allowed, 1),
NIF_FUNC(erlfdb_get_error, 1),
-NIF_FUNC(erlfdb_error_predicate, 2)
+NIF_FUNC(erlfdb_error_predicate, 2),
+
+NIF_FUNC(erlfdb_get_metrics, 0)
};
#undef NIF_FUNC
diff --git a/c_src/metrics.cc b/c_src/metrics.cc
new file mode 100644
index 000..35f73b4
--- /dev/null
+++ b/c_src/metrics.cc
@@ -0,0 +1,74 @@
+// Licensed under the Apache License, Version 2.0 (the "License"); you may not
+// use this file except in compliance with the License. You may obtain a copy
of
+// the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+// License for the specific
[couchdb-infra-cm] branch main updated: Update inventory after rebuilding the farm
This is an automated email from the ASF dual-hosted git repository. davisp pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/couchdb-infra-cm.git The following commit(s) were added to refs/heads/main by this push: new 8ebaeb3 Update inventory after rebuilding the farm 8ebaeb3 is described below commit 8ebaeb315aa8a68b190099610f5b98705be4de16 Author: Paul J. Davis AuthorDate: Fri Dec 4 12:12:59 2020 -0600 Update inventory after rebuilding the farm IBM needed me to recreate the CI farm because they had block storage instances that needed updating to a never version. --- production | 92 -- ssh.cfg| 36 ++-- 2 files changed, 38 insertions(+), 90 deletions(-) diff --git a/production b/production index 5670102..da6f32d 100644 --- a/production +++ b/production @@ -2,17 +2,17 @@ all: children: bastions: hosts: -couchdb-bastion-x86-64-debian-dal-1-1: +couchdb-bastion-x86-64-debian-dal-1-01: instance: -created_at: '2020-01-07T18:38:33Z' -id: 0717_5ecb1169-95ac-465b-a505-d172093972d1 -name: couchdb-bastion-x86-64-debian-dal-1-1 +created_at: '2020-12-04T17:51:31Z' +id: 0717_27810451-1a4b-4f12-8fcb-47375063cf28 +name: couchdb-bastion-x86-64-debian-dal-1-01 profile: cx2-2x4 subnet: couchdb-ci-farm-dal-1 vpc: couchdb-ci-farm-vpc zone: us-south-1 ip_addrs: -private: 10.240.0.11 +private: 10.240.0.17 public: 169.48.153.153 system: arch: amd64 @@ -20,44 +20,10 @@ all: ram: 4 ci_agents: hosts: -couchdb-worker-ppc64le-ubuntu-dal-1-01: - instance: -created_at: '2020-01-10T18:59:17Z' -id: 0717_00d6f517-25e3-417c-b9d5-0d4fa8cde5ac -name: couchdb-worker-ppc64le-ubuntu-dal-1-01 -profile: cp2-4x8 -subnet: couchdb-ci-farm-dal-1 -vpc: couchdb-ci-farm-vpc -zone: us-south-1 - ip_addrs: -bastion: 169.48.153.153 -private: 10.240.0.20 -public: null - system: -arch: power -num_cpus: 4 -ram: 8 -couchdb-worker-ppc64le-ubuntu-dal-1-02: - instance: -created_at: '2020-01-10T19:04:46Z' -id: 0717_a0c8bd5a-5ffe-44da-8058-b2a87180ce9b -name: couchdb-worker-ppc64le-ubuntu-dal-1-02 -profile: cp2-4x8 -subnet: couchdb-ci-farm-dal-1 -vpc: couchdb-ci-farm-vpc -zone: us-south-1 - ip_addrs: -bastion: 169.48.153.153 -private: 10.240.0.21 -public: null - system: -arch: power -num_cpus: 4 -ram: 8 couchdb-worker-x86-64-debian-dal-1-01: instance: -created_at: '2019-12-11T16:50:33Z' -id: 0717_4d64226a-ffad-4523-b5b3-78769a1d0bbe +created_at: '2020-12-04T17:37:42Z' +id: 0717_0380c8b9-988a-4938-aa9d-95ebf50ff9a7 name: couchdb-worker-x86-64-debian-dal-1-01 profile: cx2-4x8 subnet: couchdb-ci-farm-dal-1 @@ -65,7 +31,7 @@ all: zone: us-south-1 ip_addrs: bastion: 169.48.153.153 -private: 10.240.0.4 +private: 10.240.0.16 public: null system: arch: amd64 @@ -73,8 +39,8 @@ all: ram: 8 couchdb-worker-x86-64-debian-dal-1-02: instance: -created_at: '2019-12-11T16:51:02Z' -id: 0717_d97c67df-1f04-41f8-9461-9b1d5721e408 +created_at: '2020-12-04T17:37:20Z' +id: 0717_20746912-9251-417a-8b2e-75de411c37a4 name: couchdb-worker-x86-64-debian-dal-1-02 profile: cx2-4x8 subnet: couchdb-ci-farm-dal-1 @@ -82,7 +48,7 @@ all: zone: us-south-1 ip_addrs: bastion: 169.48.153.153 -private: 10.240.0.5 +private: 10.240.0.15 public: null system: arch: amd64 @@ -90,8 +56,8 @@ all: ram: 8 couchdb-worker-x86-64-debian-dal-1-03: instance: -created_at: '2019-12-11T16:51:21Z' -id: 0717_04df61d7-fb30-4251-9f59-7566c93c8a92 +created_at: '2020-12-04T17:36:58Z' +id: 0717_c158fd8e-69d9-486d-86a9-094421b5c923 name: couchdb-worker-x86-64-debian-dal-1-03 profile: cx2-4x8 subnet: couchdb-ci-farm-dal-1 @@ -99,7 +65,7 @@ all: zone: us-south-1 ip_addrs: bastion: 169.48.153.153 -private: 10.240.0.6 +private: 10.240.0.14 public: null
[couchdb] 02/04: Allow specifying an end_key for fold_changes
This is an automated email from the ASF dual-hosted git repository.
davisp pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/couchdb.git
commit 0e91f2f608d4a920b6c22f1fe1eb9e0ea6982d5a
Author: Paul J. Davis
AuthorDate: Fri Nov 13 15:13:07 2020 -0600
Allow specifying an end_key for fold_changes
This is useful so that read conflicts on the changes feed will
eventually be resolved. Without an end key specified a reader could end
up in an infinite conflict retry loop if there are clients updating
documents in the database.
---
src/fabric/src/fabric2_db.erl | 18 +++---
src/fabric/test/fabric2_changes_fold_tests.erl | 11 +++
2 files changed, 22 insertions(+), 7 deletions(-)
diff --git a/src/fabric/src/fabric2_db.erl b/src/fabric/src/fabric2_db.erl
index b3e510b..a310470 100644
--- a/src/fabric/src/fabric2_db.erl
+++ b/src/fabric/src/fabric2_db.erl
@@ -1093,14 +1093,18 @@ fold_changes(Db, SinceSeq, UserFun, UserAcc, Options) ->
end,
StartKey = get_since_seq(TxDb, Dir, SinceSeq),
-EndKey = case Dir of
-rev -> fabric2_util:seq_zero_vs();
-_ -> fabric2_util:seq_max_vs()
+EndKey = case fabric2_util:get_value(end_key, Options) of
+undefined when Dir == rev ->
+fabric2_util:seq_zero_vs();
+undefined ->
+fabric2_util:seq_max_vs();
+EK when is_binary(EK) ->
+fabric2_fdb:seq_to_vs(EK);
+EK when is_tuple(EK), element(1, EK) == versionstamp ->
+EK
end,
-FoldOpts = [
-{start_key, StartKey},
-{end_key, EndKey}
-] ++ RestartTx ++ Options,
+BaseOpts = [{start_key, StartKey}] ++ RestartTx ++ Options,
+FoldOpts = lists:keystore(end_key, 1, BaseOpts, {end_key, EndKey}),
{ok, fabric2_fdb:fold_range(TxDb, Prefix, fun({K, V}, Acc) ->
{SeqVS} = erlfdb_tuple:unpack(K, Prefix),
diff --git a/src/fabric/test/fabric2_changes_fold_tests.erl
b/src/fabric/test/fabric2_changes_fold_tests.erl
index 8541d97..fa79f25 100644
--- a/src/fabric/test/fabric2_changes_fold_tests.erl
+++ b/src/fabric/test/fabric2_changes_fold_tests.erl
@@ -40,6 +40,7 @@ changes_fold_test_() ->
?TDEF_FE(fold_changes_basic_rev),
?TDEF_FE(fold_changes_since_now_rev),
?TDEF_FE(fold_changes_since_seq_rev),
+?TDEF_FE(fold_changes_with_end_key),
?TDEF_FE(fold_changes_basic_tx_too_old),
?TDEF_FE(fold_changes_reverse_tx_too_old),
?TDEF_FE(fold_changes_tx_too_old_with_single_row_emits),
@@ -124,6 +125,16 @@ fold_changes_since_seq_rev({Db, DocRows}) ->
fold_changes_since_seq_rev({Db, RestRows}).
+fold_changes_with_end_key({Db, DocRows}) ->
+lists:foldl(fun(DocRow, Acc) ->
+EndSeq = maps:get(sequence, DocRow),
+Changes = changes(Db, 0, [{end_key, EndSeq}]),
+NewAcc = [DocRow | Acc],
+?assertEqual(Changes, NewAcc),
+NewAcc
+end, [], DocRows).
+
+
fold_changes_basic_tx_too_old({Db, DocRows0}) ->
DocRows = lists:reverse(DocRows0),
[couchdb] 03/04: Minimize conflicts while building views
This is an automated email from the ASF dual-hosted git repository.
davisp pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/couchdb.git
commit 27e8e332cf974bff69b86a8d3c5e2f5a6de4125f
Author: Paul J. Davis
AuthorDate: Fri Nov 13 14:11:00 2020 -0600
Minimize conflicts while building views
This flips the view indexer to grab the database update_seq outside of
the update transaction. Previously we would cosntantly refresh the
db_seq value on every retry of the transactional loop.
We use a snapshot to get the update_seq so that we don't trigger
spurious read conflicts with any clients that might be updating the
database.
---
src/couch_views/src/couch_views_indexer.erl | 54 +++
src/couch_views/test/couch_views_indexer_test.erl | 45 +++
2 files changed, 91 insertions(+), 8 deletions(-)
diff --git a/src/couch_views/src/couch_views_indexer.erl
b/src/couch_views/src/couch_views_indexer.erl
index 2735f66..83ccb2c 100644
--- a/src/couch_views/src/couch_views_indexer.erl
+++ b/src/couch_views/src/couch_views_indexer.erl
@@ -86,15 +86,22 @@ init() ->
fail_job(Job, Data, sig_changed, "Design document was modified")
end,
+DbSeq = fabric2_fdb:transactional(Db, fun(TxDb) ->
+fabric2_fdb:with_snapshot(TxDb, fun(SSDb) ->
+fabric2_db:get_update_seq(SSDb)
+end)
+end),
+
State = #{
tx_db => undefined,
db_uuid => DbUUID,
-db_seq => undefined,
+db_seq => DbSeq,
view_seq => undefined,
last_seq => undefined,
view_vs => undefined,
job => Job,
job_data => Data,
+rows_processed => 0,
count => 0,
changes_done => 0,
doc_acc => [],
@@ -206,8 +213,6 @@ do_update(Db, Mrst0, State0) ->
{ok, State2} = fold_changes(State1),
#{
-count := Count,
-limit := Limit,
doc_acc := DocAcc,
last_seq := LastSeq,
view_vs := ViewVS,
@@ -228,7 +233,7 @@ do_update(Db, Mrst0, State0) ->
total_kvs => TotalKVs
},
-case Count < Limit of
+case is_update_finished(State2) of
true ->
maybe_set_build_status(TxDb, Mrst2, ViewVS,
?INDEX_READY),
@@ -249,6 +254,20 @@ do_update(Db, Mrst0, State0) ->
end).
+is_update_finished(State) ->
+#{
+db_seq := DbSeq,
+last_seq := LastSeq,
+view_vs := ViewVs
+} = State,
+AtDbSeq = LastSeq == DbSeq,
+AtViewVs = case ViewVs of
+not_found -> false;
+_ -> LastSeq == fabric2_fdb:vs_to_seq(ViewVs)
+end,
+AtDbSeq orelse AtViewVs.
+
+
maybe_set_build_status(_TxDb, _Mrst1, not_found, _State) ->
ok;
@@ -258,7 +277,7 @@ maybe_set_build_status(TxDb, Mrst1, _ViewVS, State) ->
% In the first iteration of update we need
% to populate our db and view sequences
-get_update_start_state(TxDb, Mrst, #{db_seq := undefined} = State) ->
+get_update_start_state(TxDb, Mrst, #{view_seq := undefined} = State) ->
#{
view_vs := ViewVS,
view_seq := ViewSeq
@@ -266,7 +285,6 @@ get_update_start_state(TxDb, Mrst, #{db_seq := undefined} =
State) ->
State#{
tx_db := TxDb,
-db_seq := fabric2_db:get_update_seq(TxDb),
view_vs := ViewVS,
view_seq := ViewSeq,
last_seq := ViewSeq
@@ -281,18 +299,36 @@ get_update_start_state(TxDb, _Idx, State) ->
fold_changes(State) ->
#{
view_seq := SinceSeq,
+db_seq := DbSeq,
limit := Limit,
tx_db := TxDb
} = State,
+FoldState = State#{
+rows_processed := 0
+},
+
Fun = fun process_changes/2,
-Opts = [{limit, Limit}, {restart_tx, false}],
-fabric2_db:fold_changes(TxDb, SinceSeq, Fun, State, Opts).
+Opts = [
+{end_key, fabric2_fdb:seq_to_vs(DbSeq)},
+{limit, Limit},
+{restart_tx, false}
+],
+case fabric2_db:fold_changes(TxDb, SinceSeq, Fun, FoldState, Opts) of
+{ok, #{rows_processed := 0} = FinalState} when Limit > 0 ->
+% If we read zero rows with a non-zero limit
+% it means we've caught up to the DbSeq as our
+% last_seq.
+{ok, FinalState#{last_seq := DbSeq}};
+Result ->
+Result
+end.
process_changes(Change, Acc) ->
#{
doc_acc := DocAcc,
+rows_processed := RowsProcessed,
count := Count,
design_opts := DesignOpts,
view_vs := ViewVS
@@ -308,12 +344,14 @@ process_changes(Change, Acc) ->
Acc1 = case {Id, IncludeDesign} of
{<>, false} ->
maps:merge(Acc, #{
+rows_processed => RowsProcessed + 1,
[couchdb] 01/04: Allow snapshots through fabric2_fdb:transactional
This is an automated email from the ASF dual-hosted git repository.
davisp pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/couchdb.git
commit 1b30d0810ac0d985676c90dc1b6532f9fb84284c
Author: Paul J. Davis
AuthorDate: Fri Nov 13 14:08:52 2020 -0600
Allow snapshots through fabric2_fdb:transactional
---
src/fabric/src/fabric2_fdb.erl | 17
src/fabric/test/fabric2_snapshot_tests.erl | 134 +
2 files changed, 151 insertions(+)
diff --git a/src/fabric/src/fabric2_fdb.erl b/src/fabric/src/fabric2_fdb.erl
index aa2bade..1690f2f 100644
--- a/src/fabric/src/fabric2_fdb.erl
+++ b/src/fabric/src/fabric2_fdb.erl
@@ -18,6 +18,8 @@
transactional/3,
transactional/2,
+with_snapshot/2,
+
create/2,
open/2,
ensure_current/1,
@@ -150,6 +152,11 @@ transactional(#{tx := undefined} = Db, Fun) ->
transactional(Db#{reopen => true}, Fun)
end)
end;
+transactional(#{tx := {erlfdb_snapshot, _}} = Db, Fun) ->
+DbName = maps:get(name, Db, undefined),
+with_span(Fun, #{'db.name' => DbName}, fun() ->
+Fun(Db)
+end);
transactional(#{tx := {erlfdb_transaction, _}} = Db, Fun) ->
DbName = maps:get(name, Db, undefined),
@@ -183,6 +190,14 @@ do_transaction(Fun, LayerPrefix) when is_function(Fun, 1)
->
end.
+with_snapshot(#{tx := {erlfdb_transaction, _} = Tx} = TxDb, Fun) ->
+SSDb = TxDb#{tx := erlfdb:snapshot(Tx)},
+Fun(SSDb);
+
+with_snapshot(#{tx := {erlfdb_snapshot, _}} = SSDb, Fun) ->
+Fun(SSDb).
+
+
create(#{} = Db0, Options) ->
#{
name := DbName,
@@ -1865,6 +1880,8 @@ get_db_handle() ->
end.
+require_transaction(#{tx := {erlfdb_snapshot, _}} = _Db) ->
+ok;
require_transaction(#{tx := {erlfdb_transaction, _}} = _Db) ->
ok;
require_transaction(#{} = _Db) ->
diff --git a/src/fabric/test/fabric2_snapshot_tests.erl
b/src/fabric/test/fabric2_snapshot_tests.erl
new file mode 100644
index 000..37ba626
--- /dev/null
+++ b/src/fabric/test/fabric2_snapshot_tests.erl
@@ -0,0 +1,134 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+% http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(fabric2_snapshot_tests).
+
+
+-include_lib("couch/include/couch_db.hrl").
+-include_lib("couch/include/couch_eunit.hrl").
+-include_lib("eunit/include/eunit.hrl").
+-include("fabric2.hrl").
+-include("fabric2_test.hrl").
+
+
+fdb_ss_test_() ->
+{
+"Test snapshot usage",
+setup,
+fun setup/0,
+fun cleanup/1,
+with([
+?TDEF(retry_without_snapshot),
+?TDEF(no_retry_with_snapshot)
+])
+}.
+
+
+setup() ->
+Ctx = test_util:start_couch([fabric]),
+{ok, Db} = fabric2_db:create(?tempdb(), [{user_ctx, ?ADMIN_USER}]),
+{Db, Ctx}.
+
+
+cleanup({Db, Ctx}) ->
+ok = fabric2_db:delete(fabric2_db:name(Db), []),
+test_util:stop_couch(Ctx).
+
+
+retry_without_snapshot({Db, _}) ->
+DbName = fabric2_db:name(Db),
+put(retry_count, 0),
+erase(conflict_pid),
+InitDbSeq = fabric2_db:get_update_seq(Db),
+DbSeq = fabric2_fdb:transactional(Db, fun(TxDb) ->
+put(retry_count, get(retry_count) + 1),
+
+% Fetch the update_seq
+Seq = fabric2_db:get_update_seq(TxDb),
+
+% Generate a no-op write so that we don't hit the
+% optimization to skip commits on read-only
+% transactions
+bump_view_size(TxDb),
+
+% Generate a conflicting transaction while
+% we're not yet committed
+case get(conflict_pid) of
+undefined ->
+{Pid, Ref} = spawn_monitor(fun() -> generate_conflict(DbName)
end),
+receive {'DOWN', Ref, _, _, normal} -> ok end,
+put(conflict_pid, Pid);
+Pid when is_pid(Pid) ->
+ok
+end,
+
+Seq
+end),
+
+?assertEqual(2, get(retry_count)),
+?assertNotEqual(InitDbSeq, DbSeq).
+
+
+no_retry_with_snapshot({Db, _}) ->
+DbName = fabric2_db:name(Db),
+put(retry_count, 0),
+erase(conflict_pid),
+InitDbSeq = fabric2_db:get_update_seq(Db),
+DbSeq = fabric2_fdb:transactional(Db, fun(TxDb) ->
+put(retry_count, get(retry_count) + 1),
+
+% Fetch the update_seq
+Seq = fabric2_fdb:with_snapshot(TxDb, fun(SSDb) ->
+
[couchdb] branch main updated (0b98b3b -> 3168a4d)
This is an automated email from the ASF dual-hosted git repository. davisp pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/couchdb.git. from 0b98b3b Merge pull request #3274 from cloudant/fix-mango-tests new 1b30d08 Allow snapshots through fabric2_fdb:transactional new 0e91f2f Allow specifying an end_key for fold_changes new 27e8e33 Minimize conflicts while building views new 3168a4d Remove unnecessary line wraps The 4 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: src/couch_views/src/couch_views_indexer.erl | 60 -- src/couch_views/test/couch_views_indexer_test.erl | 45 src/fabric/src/fabric2_db.erl | 18 +-- src/fabric/src/fabric2_fdb.erl| 17 +++ src/fabric/test/fabric2_changes_fold_tests.erl| 11 ++ src/fabric/test/fabric2_snapshot_tests.erl| 134 ++ 6 files changed, 266 insertions(+), 19 deletions(-) create mode 100644 src/fabric/test/fabric2_snapshot_tests.erl
[couchdb] 04/04: Remove unnecessary line wraps
This is an automated email from the ASF dual-hosted git repository.
davisp pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/couchdb.git
commit 3168a4dbcef7181b67a989e1d723dc6a93e76280
Author: Paul J. Davis
AuthorDate: Tue Nov 17 11:23:36 2020 -0600
Remove unnecessary line wraps
---
src/couch_views/src/couch_views_indexer.erl | 6 ++
1 file changed, 2 insertions(+), 4 deletions(-)
diff --git a/src/couch_views/src/couch_views_indexer.erl
b/src/couch_views/src/couch_views_indexer.erl
index 83ccb2c..e3b2ad5 100644
--- a/src/couch_views/src/couch_views_indexer.erl
+++ b/src/couch_views/src/couch_views_indexer.erl
@@ -235,10 +235,8 @@ do_update(Db, Mrst0, State0) ->
case is_update_finished(State2) of
true ->
-maybe_set_build_status(TxDb, Mrst2, ViewVS,
-?INDEX_READY),
-report_progress(State2#{changes_done := ChangesDone},
-finished),
+maybe_set_build_status(TxDb, Mrst2, ViewVS, ?INDEX_READY),
+report_progress(State2#{changes_done := ChangesDone},
finished),
{Mrst2, finished};
false ->
State3 = report_progress(State2, update),
[couchdb] 02/04: Allow specifying an end_key for fold_changes
This is an automated email from the ASF dual-hosted git repository.
davisp pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/couchdb.git
commit 0e91f2f608d4a920b6c22f1fe1eb9e0ea6982d5a
Author: Paul J. Davis
AuthorDate: Fri Nov 13 15:13:07 2020 -0600
Allow specifying an end_key for fold_changes
This is useful so that read conflicts on the changes feed will
eventually be resolved. Without an end key specified a reader could end
up in an infinite conflict retry loop if there are clients updating
documents in the database.
---
src/fabric/src/fabric2_db.erl | 18 +++---
src/fabric/test/fabric2_changes_fold_tests.erl | 11 +++
2 files changed, 22 insertions(+), 7 deletions(-)
diff --git a/src/fabric/src/fabric2_db.erl b/src/fabric/src/fabric2_db.erl
index b3e510b..a310470 100644
--- a/src/fabric/src/fabric2_db.erl
+++ b/src/fabric/src/fabric2_db.erl
@@ -1093,14 +1093,18 @@ fold_changes(Db, SinceSeq, UserFun, UserAcc, Options) ->
end,
StartKey = get_since_seq(TxDb, Dir, SinceSeq),
-EndKey = case Dir of
-rev -> fabric2_util:seq_zero_vs();
-_ -> fabric2_util:seq_max_vs()
+EndKey = case fabric2_util:get_value(end_key, Options) of
+undefined when Dir == rev ->
+fabric2_util:seq_zero_vs();
+undefined ->
+fabric2_util:seq_max_vs();
+EK when is_binary(EK) ->
+fabric2_fdb:seq_to_vs(EK);
+EK when is_tuple(EK), element(1, EK) == versionstamp ->
+EK
end,
-FoldOpts = [
-{start_key, StartKey},
-{end_key, EndKey}
-] ++ RestartTx ++ Options,
+BaseOpts = [{start_key, StartKey}] ++ RestartTx ++ Options,
+FoldOpts = lists:keystore(end_key, 1, BaseOpts, {end_key, EndKey}),
{ok, fabric2_fdb:fold_range(TxDb, Prefix, fun({K, V}, Acc) ->
{SeqVS} = erlfdb_tuple:unpack(K, Prefix),
diff --git a/src/fabric/test/fabric2_changes_fold_tests.erl
b/src/fabric/test/fabric2_changes_fold_tests.erl
index 8541d97..fa79f25 100644
--- a/src/fabric/test/fabric2_changes_fold_tests.erl
+++ b/src/fabric/test/fabric2_changes_fold_tests.erl
@@ -40,6 +40,7 @@ changes_fold_test_() ->
?TDEF_FE(fold_changes_basic_rev),
?TDEF_FE(fold_changes_since_now_rev),
?TDEF_FE(fold_changes_since_seq_rev),
+?TDEF_FE(fold_changes_with_end_key),
?TDEF_FE(fold_changes_basic_tx_too_old),
?TDEF_FE(fold_changes_reverse_tx_too_old),
?TDEF_FE(fold_changes_tx_too_old_with_single_row_emits),
@@ -124,6 +125,16 @@ fold_changes_since_seq_rev({Db, DocRows}) ->
fold_changes_since_seq_rev({Db, RestRows}).
+fold_changes_with_end_key({Db, DocRows}) ->
+lists:foldl(fun(DocRow, Acc) ->
+EndSeq = maps:get(sequence, DocRow),
+Changes = changes(Db, 0, [{end_key, EndSeq}]),
+NewAcc = [DocRow | Acc],
+?assertEqual(Changes, NewAcc),
+NewAcc
+end, [], DocRows).
+
+
fold_changes_basic_tx_too_old({Db, DocRows0}) ->
DocRows = lists:reverse(DocRows0),
[couchdb] 03/04: Minimize conflicts while building views
This is an automated email from the ASF dual-hosted git repository.
davisp pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/couchdb.git
commit 27e8e332cf974bff69b86a8d3c5e2f5a6de4125f
Author: Paul J. Davis
AuthorDate: Fri Nov 13 14:11:00 2020 -0600
Minimize conflicts while building views
This flips the view indexer to grab the database update_seq outside of
the update transaction. Previously we would cosntantly refresh the
db_seq value on every retry of the transactional loop.
We use a snapshot to get the update_seq so that we don't trigger
spurious read conflicts with any clients that might be updating the
database.
---
src/couch_views/src/couch_views_indexer.erl | 54 +++
src/couch_views/test/couch_views_indexer_test.erl | 45 +++
2 files changed, 91 insertions(+), 8 deletions(-)
diff --git a/src/couch_views/src/couch_views_indexer.erl
b/src/couch_views/src/couch_views_indexer.erl
index 2735f66..83ccb2c 100644
--- a/src/couch_views/src/couch_views_indexer.erl
+++ b/src/couch_views/src/couch_views_indexer.erl
@@ -86,15 +86,22 @@ init() ->
fail_job(Job, Data, sig_changed, "Design document was modified")
end,
+DbSeq = fabric2_fdb:transactional(Db, fun(TxDb) ->
+fabric2_fdb:with_snapshot(TxDb, fun(SSDb) ->
+fabric2_db:get_update_seq(SSDb)
+end)
+end),
+
State = #{
tx_db => undefined,
db_uuid => DbUUID,
-db_seq => undefined,
+db_seq => DbSeq,
view_seq => undefined,
last_seq => undefined,
view_vs => undefined,
job => Job,
job_data => Data,
+rows_processed => 0,
count => 0,
changes_done => 0,
doc_acc => [],
@@ -206,8 +213,6 @@ do_update(Db, Mrst0, State0) ->
{ok, State2} = fold_changes(State1),
#{
-count := Count,
-limit := Limit,
doc_acc := DocAcc,
last_seq := LastSeq,
view_vs := ViewVS,
@@ -228,7 +233,7 @@ do_update(Db, Mrst0, State0) ->
total_kvs => TotalKVs
},
-case Count < Limit of
+case is_update_finished(State2) of
true ->
maybe_set_build_status(TxDb, Mrst2, ViewVS,
?INDEX_READY),
@@ -249,6 +254,20 @@ do_update(Db, Mrst0, State0) ->
end).
+is_update_finished(State) ->
+#{
+db_seq := DbSeq,
+last_seq := LastSeq,
+view_vs := ViewVs
+} = State,
+AtDbSeq = LastSeq == DbSeq,
+AtViewVs = case ViewVs of
+not_found -> false;
+_ -> LastSeq == fabric2_fdb:vs_to_seq(ViewVs)
+end,
+AtDbSeq orelse AtViewVs.
+
+
maybe_set_build_status(_TxDb, _Mrst1, not_found, _State) ->
ok;
@@ -258,7 +277,7 @@ maybe_set_build_status(TxDb, Mrst1, _ViewVS, State) ->
% In the first iteration of update we need
% to populate our db and view sequences
-get_update_start_state(TxDb, Mrst, #{db_seq := undefined} = State) ->
+get_update_start_state(TxDb, Mrst, #{view_seq := undefined} = State) ->
#{
view_vs := ViewVS,
view_seq := ViewSeq
@@ -266,7 +285,6 @@ get_update_start_state(TxDb, Mrst, #{db_seq := undefined} =
State) ->
State#{
tx_db := TxDb,
-db_seq := fabric2_db:get_update_seq(TxDb),
view_vs := ViewVS,
view_seq := ViewSeq,
last_seq := ViewSeq
@@ -281,18 +299,36 @@ get_update_start_state(TxDb, _Idx, State) ->
fold_changes(State) ->
#{
view_seq := SinceSeq,
+db_seq := DbSeq,
limit := Limit,
tx_db := TxDb
} = State,
+FoldState = State#{
+rows_processed := 0
+},
+
Fun = fun process_changes/2,
-Opts = [{limit, Limit}, {restart_tx, false}],
-fabric2_db:fold_changes(TxDb, SinceSeq, Fun, State, Opts).
+Opts = [
+{end_key, fabric2_fdb:seq_to_vs(DbSeq)},
+{limit, Limit},
+{restart_tx, false}
+],
+case fabric2_db:fold_changes(TxDb, SinceSeq, Fun, FoldState, Opts) of
+{ok, #{rows_processed := 0} = FinalState} when Limit > 0 ->
+% If we read zero rows with a non-zero limit
+% it means we've caught up to the DbSeq as our
+% last_seq.
+{ok, FinalState#{last_seq := DbSeq}};
+Result ->
+Result
+end.
process_changes(Change, Acc) ->
#{
doc_acc := DocAcc,
+rows_processed := RowsProcessed,
count := Count,
design_opts := DesignOpts,
view_vs := ViewVS
@@ -308,12 +344,14 @@ process_changes(Change, Acc) ->
Acc1 = case {Id, IncludeDesign} of
{<>, false} ->
maps:merge(Acc, #{
+rows_processed => RowsProcessed + 1,
[couchdb] 04/04: Remove unnecessary line wraps
This is an automated email from the ASF dual-hosted git repository.
davisp pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/couchdb.git
commit 3168a4dbcef7181b67a989e1d723dc6a93e76280
Author: Paul J. Davis
AuthorDate: Tue Nov 17 11:23:36 2020 -0600
Remove unnecessary line wraps
---
src/couch_views/src/couch_views_indexer.erl | 6 ++
1 file changed, 2 insertions(+), 4 deletions(-)
diff --git a/src/couch_views/src/couch_views_indexer.erl
b/src/couch_views/src/couch_views_indexer.erl
index 83ccb2c..e3b2ad5 100644
--- a/src/couch_views/src/couch_views_indexer.erl
+++ b/src/couch_views/src/couch_views_indexer.erl
@@ -235,10 +235,8 @@ do_update(Db, Mrst0, State0) ->
case is_update_finished(State2) of
true ->
-maybe_set_build_status(TxDb, Mrst2, ViewVS,
-?INDEX_READY),
-report_progress(State2#{changes_done := ChangesDone},
-finished),
+maybe_set_build_status(TxDb, Mrst2, ViewVS, ?INDEX_READY),
+report_progress(State2#{changes_done := ChangesDone},
finished),
{Mrst2, finished};
false ->
State3 = report_progress(State2, update),
[couchdb] 01/04: Allow snapshots through fabric2_fdb:transactional
This is an automated email from the ASF dual-hosted git repository.
davisp pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/couchdb.git
commit 1b30d0810ac0d985676c90dc1b6532f9fb84284c
Author: Paul J. Davis
AuthorDate: Fri Nov 13 14:08:52 2020 -0600
Allow snapshots through fabric2_fdb:transactional
---
src/fabric/src/fabric2_fdb.erl | 17
src/fabric/test/fabric2_snapshot_tests.erl | 134 +
2 files changed, 151 insertions(+)
diff --git a/src/fabric/src/fabric2_fdb.erl b/src/fabric/src/fabric2_fdb.erl
index aa2bade..1690f2f 100644
--- a/src/fabric/src/fabric2_fdb.erl
+++ b/src/fabric/src/fabric2_fdb.erl
@@ -18,6 +18,8 @@
transactional/3,
transactional/2,
+with_snapshot/2,
+
create/2,
open/2,
ensure_current/1,
@@ -150,6 +152,11 @@ transactional(#{tx := undefined} = Db, Fun) ->
transactional(Db#{reopen => true}, Fun)
end)
end;
+transactional(#{tx := {erlfdb_snapshot, _}} = Db, Fun) ->
+DbName = maps:get(name, Db, undefined),
+with_span(Fun, #{'db.name' => DbName}, fun() ->
+Fun(Db)
+end);
transactional(#{tx := {erlfdb_transaction, _}} = Db, Fun) ->
DbName = maps:get(name, Db, undefined),
@@ -183,6 +190,14 @@ do_transaction(Fun, LayerPrefix) when is_function(Fun, 1)
->
end.
+with_snapshot(#{tx := {erlfdb_transaction, _} = Tx} = TxDb, Fun) ->
+SSDb = TxDb#{tx := erlfdb:snapshot(Tx)},
+Fun(SSDb);
+
+with_snapshot(#{tx := {erlfdb_snapshot, _}} = SSDb, Fun) ->
+Fun(SSDb).
+
+
create(#{} = Db0, Options) ->
#{
name := DbName,
@@ -1865,6 +1880,8 @@ get_db_handle() ->
end.
+require_transaction(#{tx := {erlfdb_snapshot, _}} = _Db) ->
+ok;
require_transaction(#{tx := {erlfdb_transaction, _}} = _Db) ->
ok;
require_transaction(#{} = _Db) ->
diff --git a/src/fabric/test/fabric2_snapshot_tests.erl
b/src/fabric/test/fabric2_snapshot_tests.erl
new file mode 100644
index 000..37ba626
--- /dev/null
+++ b/src/fabric/test/fabric2_snapshot_tests.erl
@@ -0,0 +1,134 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+% http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(fabric2_snapshot_tests).
+
+
+-include_lib("couch/include/couch_db.hrl").
+-include_lib("couch/include/couch_eunit.hrl").
+-include_lib("eunit/include/eunit.hrl").
+-include("fabric2.hrl").
+-include("fabric2_test.hrl").
+
+
+fdb_ss_test_() ->
+{
+"Test snapshot usage",
+setup,
+fun setup/0,
+fun cleanup/1,
+with([
+?TDEF(retry_without_snapshot),
+?TDEF(no_retry_with_snapshot)
+])
+}.
+
+
+setup() ->
+Ctx = test_util:start_couch([fabric]),
+{ok, Db} = fabric2_db:create(?tempdb(), [{user_ctx, ?ADMIN_USER}]),
+{Db, Ctx}.
+
+
+cleanup({Db, Ctx}) ->
+ok = fabric2_db:delete(fabric2_db:name(Db), []),
+test_util:stop_couch(Ctx).
+
+
+retry_without_snapshot({Db, _}) ->
+DbName = fabric2_db:name(Db),
+put(retry_count, 0),
+erase(conflict_pid),
+InitDbSeq = fabric2_db:get_update_seq(Db),
+DbSeq = fabric2_fdb:transactional(Db, fun(TxDb) ->
+put(retry_count, get(retry_count) + 1),
+
+% Fetch the update_seq
+Seq = fabric2_db:get_update_seq(TxDb),
+
+% Generate a no-op write so that we don't hit the
+% optimization to skip commits on read-only
+% transactions
+bump_view_size(TxDb),
+
+% Generate a conflicting transaction while
+% we're not yet committed
+case get(conflict_pid) of
+undefined ->
+{Pid, Ref} = spawn_monitor(fun() -> generate_conflict(DbName)
end),
+receive {'DOWN', Ref, _, _, normal} -> ok end,
+put(conflict_pid, Pid);
+Pid when is_pid(Pid) ->
+ok
+end,
+
+Seq
+end),
+
+?assertEqual(2, get(retry_count)),
+?assertNotEqual(InitDbSeq, DbSeq).
+
+
+no_retry_with_snapshot({Db, _}) ->
+DbName = fabric2_db:name(Db),
+put(retry_count, 0),
+erase(conflict_pid),
+InitDbSeq = fabric2_db:get_update_seq(Db),
+DbSeq = fabric2_fdb:transactional(Db, fun(TxDb) ->
+put(retry_count, get(retry_count) + 1),
+
+% Fetch the update_seq
+Seq = fabric2_fdb:with_snapshot(TxDb, fun(SSDb) ->
+
[couchdb] branch main updated (0b98b3b -> 3168a4d)
This is an automated email from the ASF dual-hosted git repository. davisp pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/couchdb.git. from 0b98b3b Merge pull request #3274 from cloudant/fix-mango-tests new 1b30d08 Allow snapshots through fabric2_fdb:transactional new 0e91f2f Allow specifying an end_key for fold_changes new 27e8e33 Minimize conflicts while building views new 3168a4d Remove unnecessary line wraps The 4 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: src/couch_views/src/couch_views_indexer.erl | 60 -- src/couch_views/test/couch_views_indexer_test.erl | 45 src/fabric/src/fabric2_db.erl | 18 +-- src/fabric/src/fabric2_fdb.erl| 17 +++ src/fabric/test/fabric2_changes_fold_tests.erl| 11 ++ src/fabric/test/fabric2_snapshot_tests.erl| 134 ++ 6 files changed, 266 insertions(+), 19 deletions(-) create mode 100644 src/fabric/test/fabric2_snapshot_tests.erl
[couchdb] 03/04: Minimize conflicts while building views
This is an automated email from the ASF dual-hosted git repository.
davisp pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/couchdb.git
commit 27e8e332cf974bff69b86a8d3c5e2f5a6de4125f
Author: Paul J. Davis
AuthorDate: Fri Nov 13 14:11:00 2020 -0600
Minimize conflicts while building views
This flips the view indexer to grab the database update_seq outside of
the update transaction. Previously we would cosntantly refresh the
db_seq value on every retry of the transactional loop.
We use a snapshot to get the update_seq so that we don't trigger
spurious read conflicts with any clients that might be updating the
database.
---
src/couch_views/src/couch_views_indexer.erl | 54 +++
src/couch_views/test/couch_views_indexer_test.erl | 45 +++
2 files changed, 91 insertions(+), 8 deletions(-)
diff --git a/src/couch_views/src/couch_views_indexer.erl
b/src/couch_views/src/couch_views_indexer.erl
index 2735f66..83ccb2c 100644
--- a/src/couch_views/src/couch_views_indexer.erl
+++ b/src/couch_views/src/couch_views_indexer.erl
@@ -86,15 +86,22 @@ init() ->
fail_job(Job, Data, sig_changed, "Design document was modified")
end,
+DbSeq = fabric2_fdb:transactional(Db, fun(TxDb) ->
+fabric2_fdb:with_snapshot(TxDb, fun(SSDb) ->
+fabric2_db:get_update_seq(SSDb)
+end)
+end),
+
State = #{
tx_db => undefined,
db_uuid => DbUUID,
-db_seq => undefined,
+db_seq => DbSeq,
view_seq => undefined,
last_seq => undefined,
view_vs => undefined,
job => Job,
job_data => Data,
+rows_processed => 0,
count => 0,
changes_done => 0,
doc_acc => [],
@@ -206,8 +213,6 @@ do_update(Db, Mrst0, State0) ->
{ok, State2} = fold_changes(State1),
#{
-count := Count,
-limit := Limit,
doc_acc := DocAcc,
last_seq := LastSeq,
view_vs := ViewVS,
@@ -228,7 +233,7 @@ do_update(Db, Mrst0, State0) ->
total_kvs => TotalKVs
},
-case Count < Limit of
+case is_update_finished(State2) of
true ->
maybe_set_build_status(TxDb, Mrst2, ViewVS,
?INDEX_READY),
@@ -249,6 +254,20 @@ do_update(Db, Mrst0, State0) ->
end).
+is_update_finished(State) ->
+#{
+db_seq := DbSeq,
+last_seq := LastSeq,
+view_vs := ViewVs
+} = State,
+AtDbSeq = LastSeq == DbSeq,
+AtViewVs = case ViewVs of
+not_found -> false;
+_ -> LastSeq == fabric2_fdb:vs_to_seq(ViewVs)
+end,
+AtDbSeq orelse AtViewVs.
+
+
maybe_set_build_status(_TxDb, _Mrst1, not_found, _State) ->
ok;
@@ -258,7 +277,7 @@ maybe_set_build_status(TxDb, Mrst1, _ViewVS, State) ->
% In the first iteration of update we need
% to populate our db and view sequences
-get_update_start_state(TxDb, Mrst, #{db_seq := undefined} = State) ->
+get_update_start_state(TxDb, Mrst, #{view_seq := undefined} = State) ->
#{
view_vs := ViewVS,
view_seq := ViewSeq
@@ -266,7 +285,6 @@ get_update_start_state(TxDb, Mrst, #{db_seq := undefined} =
State) ->
State#{
tx_db := TxDb,
-db_seq := fabric2_db:get_update_seq(TxDb),
view_vs := ViewVS,
view_seq := ViewSeq,
last_seq := ViewSeq
@@ -281,18 +299,36 @@ get_update_start_state(TxDb, _Idx, State) ->
fold_changes(State) ->
#{
view_seq := SinceSeq,
+db_seq := DbSeq,
limit := Limit,
tx_db := TxDb
} = State,
+FoldState = State#{
+rows_processed := 0
+},
+
Fun = fun process_changes/2,
-Opts = [{limit, Limit}, {restart_tx, false}],
-fabric2_db:fold_changes(TxDb, SinceSeq, Fun, State, Opts).
+Opts = [
+{end_key, fabric2_fdb:seq_to_vs(DbSeq)},
+{limit, Limit},
+{restart_tx, false}
+],
+case fabric2_db:fold_changes(TxDb, SinceSeq, Fun, FoldState, Opts) of
+{ok, #{rows_processed := 0} = FinalState} when Limit > 0 ->
+% If we read zero rows with a non-zero limit
+% it means we've caught up to the DbSeq as our
+% last_seq.
+{ok, FinalState#{last_seq := DbSeq}};
+Result ->
+Result
+end.
process_changes(Change, Acc) ->
#{
doc_acc := DocAcc,
+rows_processed := RowsProcessed,
count := Count,
design_opts := DesignOpts,
view_vs := ViewVS
@@ -308,12 +344,14 @@ process_changes(Change, Acc) ->
Acc1 = case {Id, IncludeDesign} of
{<>, false} ->
maps:merge(Acc, #{
+rows_processed => RowsProcessed + 1,
[couchdb] 01/04: Allow snapshots through fabric2_fdb:transactional
This is an automated email from the ASF dual-hosted git repository.
davisp pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/couchdb.git
commit 1b30d0810ac0d985676c90dc1b6532f9fb84284c
Author: Paul J. Davis
AuthorDate: Fri Nov 13 14:08:52 2020 -0600
Allow snapshots through fabric2_fdb:transactional
---
src/fabric/src/fabric2_fdb.erl | 17
src/fabric/test/fabric2_snapshot_tests.erl | 134 +
2 files changed, 151 insertions(+)
diff --git a/src/fabric/src/fabric2_fdb.erl b/src/fabric/src/fabric2_fdb.erl
index aa2bade..1690f2f 100644
--- a/src/fabric/src/fabric2_fdb.erl
+++ b/src/fabric/src/fabric2_fdb.erl
@@ -18,6 +18,8 @@
transactional/3,
transactional/2,
+with_snapshot/2,
+
create/2,
open/2,
ensure_current/1,
@@ -150,6 +152,11 @@ transactional(#{tx := undefined} = Db, Fun) ->
transactional(Db#{reopen => true}, Fun)
end)
end;
+transactional(#{tx := {erlfdb_snapshot, _}} = Db, Fun) ->
+DbName = maps:get(name, Db, undefined),
+with_span(Fun, #{'db.name' => DbName}, fun() ->
+Fun(Db)
+end);
transactional(#{tx := {erlfdb_transaction, _}} = Db, Fun) ->
DbName = maps:get(name, Db, undefined),
@@ -183,6 +190,14 @@ do_transaction(Fun, LayerPrefix) when is_function(Fun, 1)
->
end.
+with_snapshot(#{tx := {erlfdb_transaction, _} = Tx} = TxDb, Fun) ->
+SSDb = TxDb#{tx := erlfdb:snapshot(Tx)},
+Fun(SSDb);
+
+with_snapshot(#{tx := {erlfdb_snapshot, _}} = SSDb, Fun) ->
+Fun(SSDb).
+
+
create(#{} = Db0, Options) ->
#{
name := DbName,
@@ -1865,6 +1880,8 @@ get_db_handle() ->
end.
+require_transaction(#{tx := {erlfdb_snapshot, _}} = _Db) ->
+ok;
require_transaction(#{tx := {erlfdb_transaction, _}} = _Db) ->
ok;
require_transaction(#{} = _Db) ->
diff --git a/src/fabric/test/fabric2_snapshot_tests.erl
b/src/fabric/test/fabric2_snapshot_tests.erl
new file mode 100644
index 000..37ba626
--- /dev/null
+++ b/src/fabric/test/fabric2_snapshot_tests.erl
@@ -0,0 +1,134 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+% http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(fabric2_snapshot_tests).
+
+
+-include_lib("couch/include/couch_db.hrl").
+-include_lib("couch/include/couch_eunit.hrl").
+-include_lib("eunit/include/eunit.hrl").
+-include("fabric2.hrl").
+-include("fabric2_test.hrl").
+
+
+fdb_ss_test_() ->
+{
+"Test snapshot usage",
+setup,
+fun setup/0,
+fun cleanup/1,
+with([
+?TDEF(retry_without_snapshot),
+?TDEF(no_retry_with_snapshot)
+])
+}.
+
+
+setup() ->
+Ctx = test_util:start_couch([fabric]),
+{ok, Db} = fabric2_db:create(?tempdb(), [{user_ctx, ?ADMIN_USER}]),
+{Db, Ctx}.
+
+
+cleanup({Db, Ctx}) ->
+ok = fabric2_db:delete(fabric2_db:name(Db), []),
+test_util:stop_couch(Ctx).
+
+
+retry_without_snapshot({Db, _}) ->
+DbName = fabric2_db:name(Db),
+put(retry_count, 0),
+erase(conflict_pid),
+InitDbSeq = fabric2_db:get_update_seq(Db),
+DbSeq = fabric2_fdb:transactional(Db, fun(TxDb) ->
+put(retry_count, get(retry_count) + 1),
+
+% Fetch the update_seq
+Seq = fabric2_db:get_update_seq(TxDb),
+
+% Generate a no-op write so that we don't hit the
+% optimization to skip commits on read-only
+% transactions
+bump_view_size(TxDb),
+
+% Generate a conflicting transaction while
+% we're not yet committed
+case get(conflict_pid) of
+undefined ->
+{Pid, Ref} = spawn_monitor(fun() -> generate_conflict(DbName)
end),
+receive {'DOWN', Ref, _, _, normal} -> ok end,
+put(conflict_pid, Pid);
+Pid when is_pid(Pid) ->
+ok
+end,
+
+Seq
+end),
+
+?assertEqual(2, get(retry_count)),
+?assertNotEqual(InitDbSeq, DbSeq).
+
+
+no_retry_with_snapshot({Db, _}) ->
+DbName = fabric2_db:name(Db),
+put(retry_count, 0),
+erase(conflict_pid),
+InitDbSeq = fabric2_db:get_update_seq(Db),
+DbSeq = fabric2_fdb:transactional(Db, fun(TxDb) ->
+put(retry_count, get(retry_count) + 1),
+
+% Fetch the update_seq
+Seq = fabric2_fdb:with_snapshot(TxDb, fun(SSDb) ->
+
[couchdb] 02/04: Allow specifying an end_key for fold_changes
This is an automated email from the ASF dual-hosted git repository.
davisp pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/couchdb.git
commit 0e91f2f608d4a920b6c22f1fe1eb9e0ea6982d5a
Author: Paul J. Davis
AuthorDate: Fri Nov 13 15:13:07 2020 -0600
Allow specifying an end_key for fold_changes
This is useful so that read conflicts on the changes feed will
eventually be resolved. Without an end key specified a reader could end
up in an infinite conflict retry loop if there are clients updating
documents in the database.
---
src/fabric/src/fabric2_db.erl | 18 +++---
src/fabric/test/fabric2_changes_fold_tests.erl | 11 +++
2 files changed, 22 insertions(+), 7 deletions(-)
diff --git a/src/fabric/src/fabric2_db.erl b/src/fabric/src/fabric2_db.erl
index b3e510b..a310470 100644
--- a/src/fabric/src/fabric2_db.erl
+++ b/src/fabric/src/fabric2_db.erl
@@ -1093,14 +1093,18 @@ fold_changes(Db, SinceSeq, UserFun, UserAcc, Options) ->
end,
StartKey = get_since_seq(TxDb, Dir, SinceSeq),
-EndKey = case Dir of
-rev -> fabric2_util:seq_zero_vs();
-_ -> fabric2_util:seq_max_vs()
+EndKey = case fabric2_util:get_value(end_key, Options) of
+undefined when Dir == rev ->
+fabric2_util:seq_zero_vs();
+undefined ->
+fabric2_util:seq_max_vs();
+EK when is_binary(EK) ->
+fabric2_fdb:seq_to_vs(EK);
+EK when is_tuple(EK), element(1, EK) == versionstamp ->
+EK
end,
-FoldOpts = [
-{start_key, StartKey},
-{end_key, EndKey}
-] ++ RestartTx ++ Options,
+BaseOpts = [{start_key, StartKey}] ++ RestartTx ++ Options,
+FoldOpts = lists:keystore(end_key, 1, BaseOpts, {end_key, EndKey}),
{ok, fabric2_fdb:fold_range(TxDb, Prefix, fun({K, V}, Acc) ->
{SeqVS} = erlfdb_tuple:unpack(K, Prefix),
diff --git a/src/fabric/test/fabric2_changes_fold_tests.erl
b/src/fabric/test/fabric2_changes_fold_tests.erl
index 8541d97..fa79f25 100644
--- a/src/fabric/test/fabric2_changes_fold_tests.erl
+++ b/src/fabric/test/fabric2_changes_fold_tests.erl
@@ -40,6 +40,7 @@ changes_fold_test_() ->
?TDEF_FE(fold_changes_basic_rev),
?TDEF_FE(fold_changes_since_now_rev),
?TDEF_FE(fold_changes_since_seq_rev),
+?TDEF_FE(fold_changes_with_end_key),
?TDEF_FE(fold_changes_basic_tx_too_old),
?TDEF_FE(fold_changes_reverse_tx_too_old),
?TDEF_FE(fold_changes_tx_too_old_with_single_row_emits),
@@ -124,6 +125,16 @@ fold_changes_since_seq_rev({Db, DocRows}) ->
fold_changes_since_seq_rev({Db, RestRows}).
+fold_changes_with_end_key({Db, DocRows}) ->
+lists:foldl(fun(DocRow, Acc) ->
+EndSeq = maps:get(sequence, DocRow),
+Changes = changes(Db, 0, [{end_key, EndSeq}]),
+NewAcc = [DocRow | Acc],
+?assertEqual(Changes, NewAcc),
+NewAcc
+end, [], DocRows).
+
+
fold_changes_basic_tx_too_old({Db, DocRows0}) ->
DocRows = lists:reverse(DocRows0),
[couchdb] 04/04: Remove unnecessary line wraps
This is an automated email from the ASF dual-hosted git repository.
davisp pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/couchdb.git
commit 3168a4dbcef7181b67a989e1d723dc6a93e76280
Author: Paul J. Davis
AuthorDate: Tue Nov 17 11:23:36 2020 -0600
Remove unnecessary line wraps
---
src/couch_views/src/couch_views_indexer.erl | 6 ++
1 file changed, 2 insertions(+), 4 deletions(-)
diff --git a/src/couch_views/src/couch_views_indexer.erl
b/src/couch_views/src/couch_views_indexer.erl
index 83ccb2c..e3b2ad5 100644
--- a/src/couch_views/src/couch_views_indexer.erl
+++ b/src/couch_views/src/couch_views_indexer.erl
@@ -235,10 +235,8 @@ do_update(Db, Mrst0, State0) ->
case is_update_finished(State2) of
true ->
-maybe_set_build_status(TxDb, Mrst2, ViewVS,
-?INDEX_READY),
-report_progress(State2#{changes_done := ChangesDone},
-finished),
+maybe_set_build_status(TxDb, Mrst2, ViewVS, ?INDEX_READY),
+report_progress(State2#{changes_done := ChangesDone},
finished),
{Mrst2, finished};
false ->
State3 = report_progress(State2, update),
[couchdb] branch main updated (0b98b3b -> 3168a4d)
This is an automated email from the ASF dual-hosted git repository. davisp pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/couchdb.git. from 0b98b3b Merge pull request #3274 from cloudant/fix-mango-tests new 1b30d08 Allow snapshots through fabric2_fdb:transactional new 0e91f2f Allow specifying an end_key for fold_changes new 27e8e33 Minimize conflicts while building views new 3168a4d Remove unnecessary line wraps The 4 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: src/couch_views/src/couch_views_indexer.erl | 60 -- src/couch_views/test/couch_views_indexer_test.erl | 45 src/fabric/src/fabric2_db.erl | 18 +-- src/fabric/src/fabric2_fdb.erl| 17 +++ src/fabric/test/fabric2_changes_fold_tests.erl| 11 ++ src/fabric/test/fabric2_snapshot_tests.erl| 134 ++ 6 files changed, 266 insertions(+), 19 deletions(-) create mode 100644 src/fabric/test/fabric2_snapshot_tests.erl
[couchdb] 02/04: Allow specifying an end_key for fold_changes
This is an automated email from the ASF dual-hosted git repository.
davisp pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/couchdb.git
commit 0e91f2f608d4a920b6c22f1fe1eb9e0ea6982d5a
Author: Paul J. Davis
AuthorDate: Fri Nov 13 15:13:07 2020 -0600
Allow specifying an end_key for fold_changes
This is useful so that read conflicts on the changes feed will
eventually be resolved. Without an end key specified a reader could end
up in an infinite conflict retry loop if there are clients updating
documents in the database.
---
src/fabric/src/fabric2_db.erl | 18 +++---
src/fabric/test/fabric2_changes_fold_tests.erl | 11 +++
2 files changed, 22 insertions(+), 7 deletions(-)
diff --git a/src/fabric/src/fabric2_db.erl b/src/fabric/src/fabric2_db.erl
index b3e510b..a310470 100644
--- a/src/fabric/src/fabric2_db.erl
+++ b/src/fabric/src/fabric2_db.erl
@@ -1093,14 +1093,18 @@ fold_changes(Db, SinceSeq, UserFun, UserAcc, Options) ->
end,
StartKey = get_since_seq(TxDb, Dir, SinceSeq),
-EndKey = case Dir of
-rev -> fabric2_util:seq_zero_vs();
-_ -> fabric2_util:seq_max_vs()
+EndKey = case fabric2_util:get_value(end_key, Options) of
+undefined when Dir == rev ->
+fabric2_util:seq_zero_vs();
+undefined ->
+fabric2_util:seq_max_vs();
+EK when is_binary(EK) ->
+fabric2_fdb:seq_to_vs(EK);
+EK when is_tuple(EK), element(1, EK) == versionstamp ->
+EK
end,
-FoldOpts = [
-{start_key, StartKey},
-{end_key, EndKey}
-] ++ RestartTx ++ Options,
+BaseOpts = [{start_key, StartKey}] ++ RestartTx ++ Options,
+FoldOpts = lists:keystore(end_key, 1, BaseOpts, {end_key, EndKey}),
{ok, fabric2_fdb:fold_range(TxDb, Prefix, fun({K, V}, Acc) ->
{SeqVS} = erlfdb_tuple:unpack(K, Prefix),
diff --git a/src/fabric/test/fabric2_changes_fold_tests.erl
b/src/fabric/test/fabric2_changes_fold_tests.erl
index 8541d97..fa79f25 100644
--- a/src/fabric/test/fabric2_changes_fold_tests.erl
+++ b/src/fabric/test/fabric2_changes_fold_tests.erl
@@ -40,6 +40,7 @@ changes_fold_test_() ->
?TDEF_FE(fold_changes_basic_rev),
?TDEF_FE(fold_changes_since_now_rev),
?TDEF_FE(fold_changes_since_seq_rev),
+?TDEF_FE(fold_changes_with_end_key),
?TDEF_FE(fold_changes_basic_tx_too_old),
?TDEF_FE(fold_changes_reverse_tx_too_old),
?TDEF_FE(fold_changes_tx_too_old_with_single_row_emits),
@@ -124,6 +125,16 @@ fold_changes_since_seq_rev({Db, DocRows}) ->
fold_changes_since_seq_rev({Db, RestRows}).
+fold_changes_with_end_key({Db, DocRows}) ->
+lists:foldl(fun(DocRow, Acc) ->
+EndSeq = maps:get(sequence, DocRow),
+Changes = changes(Db, 0, [{end_key, EndSeq}]),
+NewAcc = [DocRow | Acc],
+?assertEqual(Changes, NewAcc),
+NewAcc
+end, [], DocRows).
+
+
fold_changes_basic_tx_too_old({Db, DocRows0}) ->
DocRows = lists:reverse(DocRows0),
[couchdb] 04/04: Remove unnecessary line wraps
This is an automated email from the ASF dual-hosted git repository.
davisp pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/couchdb.git
commit 3168a4dbcef7181b67a989e1d723dc6a93e76280
Author: Paul J. Davis
AuthorDate: Tue Nov 17 11:23:36 2020 -0600
Remove unnecessary line wraps
---
src/couch_views/src/couch_views_indexer.erl | 6 ++
1 file changed, 2 insertions(+), 4 deletions(-)
diff --git a/src/couch_views/src/couch_views_indexer.erl
b/src/couch_views/src/couch_views_indexer.erl
index 83ccb2c..e3b2ad5 100644
--- a/src/couch_views/src/couch_views_indexer.erl
+++ b/src/couch_views/src/couch_views_indexer.erl
@@ -235,10 +235,8 @@ do_update(Db, Mrst0, State0) ->
case is_update_finished(State2) of
true ->
-maybe_set_build_status(TxDb, Mrst2, ViewVS,
-?INDEX_READY),
-report_progress(State2#{changes_done := ChangesDone},
-finished),
+maybe_set_build_status(TxDb, Mrst2, ViewVS, ?INDEX_READY),
+report_progress(State2#{changes_done := ChangesDone},
finished),
{Mrst2, finished};
false ->
State3 = report_progress(State2, update),
[couchdb] 03/04: Minimize conflicts while building views
This is an automated email from the ASF dual-hosted git repository.
davisp pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/couchdb.git
commit 27e8e332cf974bff69b86a8d3c5e2f5a6de4125f
Author: Paul J. Davis
AuthorDate: Fri Nov 13 14:11:00 2020 -0600
Minimize conflicts while building views
This flips the view indexer to grab the database update_seq outside of
the update transaction. Previously we would cosntantly refresh the
db_seq value on every retry of the transactional loop.
We use a snapshot to get the update_seq so that we don't trigger
spurious read conflicts with any clients that might be updating the
database.
---
src/couch_views/src/couch_views_indexer.erl | 54 +++
src/couch_views/test/couch_views_indexer_test.erl | 45 +++
2 files changed, 91 insertions(+), 8 deletions(-)
diff --git a/src/couch_views/src/couch_views_indexer.erl
b/src/couch_views/src/couch_views_indexer.erl
index 2735f66..83ccb2c 100644
--- a/src/couch_views/src/couch_views_indexer.erl
+++ b/src/couch_views/src/couch_views_indexer.erl
@@ -86,15 +86,22 @@ init() ->
fail_job(Job, Data, sig_changed, "Design document was modified")
end,
+DbSeq = fabric2_fdb:transactional(Db, fun(TxDb) ->
+fabric2_fdb:with_snapshot(TxDb, fun(SSDb) ->
+fabric2_db:get_update_seq(SSDb)
+end)
+end),
+
State = #{
tx_db => undefined,
db_uuid => DbUUID,
-db_seq => undefined,
+db_seq => DbSeq,
view_seq => undefined,
last_seq => undefined,
view_vs => undefined,
job => Job,
job_data => Data,
+rows_processed => 0,
count => 0,
changes_done => 0,
doc_acc => [],
@@ -206,8 +213,6 @@ do_update(Db, Mrst0, State0) ->
{ok, State2} = fold_changes(State1),
#{
-count := Count,
-limit := Limit,
doc_acc := DocAcc,
last_seq := LastSeq,
view_vs := ViewVS,
@@ -228,7 +233,7 @@ do_update(Db, Mrst0, State0) ->
total_kvs => TotalKVs
},
-case Count < Limit of
+case is_update_finished(State2) of
true ->
maybe_set_build_status(TxDb, Mrst2, ViewVS,
?INDEX_READY),
@@ -249,6 +254,20 @@ do_update(Db, Mrst0, State0) ->
end).
+is_update_finished(State) ->
+#{
+db_seq := DbSeq,
+last_seq := LastSeq,
+view_vs := ViewVs
+} = State,
+AtDbSeq = LastSeq == DbSeq,
+AtViewVs = case ViewVs of
+not_found -> false;
+_ -> LastSeq == fabric2_fdb:vs_to_seq(ViewVs)
+end,
+AtDbSeq orelse AtViewVs.
+
+
maybe_set_build_status(_TxDb, _Mrst1, not_found, _State) ->
ok;
@@ -258,7 +277,7 @@ maybe_set_build_status(TxDb, Mrst1, _ViewVS, State) ->
% In the first iteration of update we need
% to populate our db and view sequences
-get_update_start_state(TxDb, Mrst, #{db_seq := undefined} = State) ->
+get_update_start_state(TxDb, Mrst, #{view_seq := undefined} = State) ->
#{
view_vs := ViewVS,
view_seq := ViewSeq
@@ -266,7 +285,6 @@ get_update_start_state(TxDb, Mrst, #{db_seq := undefined} =
State) ->
State#{
tx_db := TxDb,
-db_seq := fabric2_db:get_update_seq(TxDb),
view_vs := ViewVS,
view_seq := ViewSeq,
last_seq := ViewSeq
@@ -281,18 +299,36 @@ get_update_start_state(TxDb, _Idx, State) ->
fold_changes(State) ->
#{
view_seq := SinceSeq,
+db_seq := DbSeq,
limit := Limit,
tx_db := TxDb
} = State,
+FoldState = State#{
+rows_processed := 0
+},
+
Fun = fun process_changes/2,
-Opts = [{limit, Limit}, {restart_tx, false}],
-fabric2_db:fold_changes(TxDb, SinceSeq, Fun, State, Opts).
+Opts = [
+{end_key, fabric2_fdb:seq_to_vs(DbSeq)},
+{limit, Limit},
+{restart_tx, false}
+],
+case fabric2_db:fold_changes(TxDb, SinceSeq, Fun, FoldState, Opts) of
+{ok, #{rows_processed := 0} = FinalState} when Limit > 0 ->
+% If we read zero rows with a non-zero limit
+% it means we've caught up to the DbSeq as our
+% last_seq.
+{ok, FinalState#{last_seq := DbSeq}};
+Result ->
+Result
+end.
process_changes(Change, Acc) ->
#{
doc_acc := DocAcc,
+rows_processed := RowsProcessed,
count := Count,
design_opts := DesignOpts,
view_vs := ViewVS
@@ -308,12 +344,14 @@ process_changes(Change, Acc) ->
Acc1 = case {Id, IncludeDesign} of
{<>, false} ->
maps:merge(Acc, #{
+rows_processed => RowsProcessed + 1,
[couchdb] branch main updated (0b98b3b -> 3168a4d)
This is an automated email from the ASF dual-hosted git repository. davisp pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/couchdb.git. from 0b98b3b Merge pull request #3274 from cloudant/fix-mango-tests new 1b30d08 Allow snapshots through fabric2_fdb:transactional new 0e91f2f Allow specifying an end_key for fold_changes new 27e8e33 Minimize conflicts while building views new 3168a4d Remove unnecessary line wraps The 4 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: src/couch_views/src/couch_views_indexer.erl | 60 -- src/couch_views/test/couch_views_indexer_test.erl | 45 src/fabric/src/fabric2_db.erl | 18 +-- src/fabric/src/fabric2_fdb.erl| 17 +++ src/fabric/test/fabric2_changes_fold_tests.erl| 11 ++ src/fabric/test/fabric2_snapshot_tests.erl| 134 ++ 6 files changed, 266 insertions(+), 19 deletions(-) create mode 100644 src/fabric/test/fabric2_snapshot_tests.erl
[couchdb] 01/04: Allow snapshots through fabric2_fdb:transactional
This is an automated email from the ASF dual-hosted git repository.
davisp pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/couchdb.git
commit 1b30d0810ac0d985676c90dc1b6532f9fb84284c
Author: Paul J. Davis
AuthorDate: Fri Nov 13 14:08:52 2020 -0600
Allow snapshots through fabric2_fdb:transactional
---
src/fabric/src/fabric2_fdb.erl | 17
src/fabric/test/fabric2_snapshot_tests.erl | 134 +
2 files changed, 151 insertions(+)
diff --git a/src/fabric/src/fabric2_fdb.erl b/src/fabric/src/fabric2_fdb.erl
index aa2bade..1690f2f 100644
--- a/src/fabric/src/fabric2_fdb.erl
+++ b/src/fabric/src/fabric2_fdb.erl
@@ -18,6 +18,8 @@
transactional/3,
transactional/2,
+with_snapshot/2,
+
create/2,
open/2,
ensure_current/1,
@@ -150,6 +152,11 @@ transactional(#{tx := undefined} = Db, Fun) ->
transactional(Db#{reopen => true}, Fun)
end)
end;
+transactional(#{tx := {erlfdb_snapshot, _}} = Db, Fun) ->
+DbName = maps:get(name, Db, undefined),
+with_span(Fun, #{'db.name' => DbName}, fun() ->
+Fun(Db)
+end);
transactional(#{tx := {erlfdb_transaction, _}} = Db, Fun) ->
DbName = maps:get(name, Db, undefined),
@@ -183,6 +190,14 @@ do_transaction(Fun, LayerPrefix) when is_function(Fun, 1)
->
end.
+with_snapshot(#{tx := {erlfdb_transaction, _} = Tx} = TxDb, Fun) ->
+SSDb = TxDb#{tx := erlfdb:snapshot(Tx)},
+Fun(SSDb);
+
+with_snapshot(#{tx := {erlfdb_snapshot, _}} = SSDb, Fun) ->
+Fun(SSDb).
+
+
create(#{} = Db0, Options) ->
#{
name := DbName,
@@ -1865,6 +1880,8 @@ get_db_handle() ->
end.
+require_transaction(#{tx := {erlfdb_snapshot, _}} = _Db) ->
+ok;
require_transaction(#{tx := {erlfdb_transaction, _}} = _Db) ->
ok;
require_transaction(#{} = _Db) ->
diff --git a/src/fabric/test/fabric2_snapshot_tests.erl
b/src/fabric/test/fabric2_snapshot_tests.erl
new file mode 100644
index 000..37ba626
--- /dev/null
+++ b/src/fabric/test/fabric2_snapshot_tests.erl
@@ -0,0 +1,134 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+% http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(fabric2_snapshot_tests).
+
+
+-include_lib("couch/include/couch_db.hrl").
+-include_lib("couch/include/couch_eunit.hrl").
+-include_lib("eunit/include/eunit.hrl").
+-include("fabric2.hrl").
+-include("fabric2_test.hrl").
+
+
+fdb_ss_test_() ->
+{
+"Test snapshot usage",
+setup,
+fun setup/0,
+fun cleanup/1,
+with([
+?TDEF(retry_without_snapshot),
+?TDEF(no_retry_with_snapshot)
+])
+}.
+
+
+setup() ->
+Ctx = test_util:start_couch([fabric]),
+{ok, Db} = fabric2_db:create(?tempdb(), [{user_ctx, ?ADMIN_USER}]),
+{Db, Ctx}.
+
+
+cleanup({Db, Ctx}) ->
+ok = fabric2_db:delete(fabric2_db:name(Db), []),
+test_util:stop_couch(Ctx).
+
+
+retry_without_snapshot({Db, _}) ->
+DbName = fabric2_db:name(Db),
+put(retry_count, 0),
+erase(conflict_pid),
+InitDbSeq = fabric2_db:get_update_seq(Db),
+DbSeq = fabric2_fdb:transactional(Db, fun(TxDb) ->
+put(retry_count, get(retry_count) + 1),
+
+% Fetch the update_seq
+Seq = fabric2_db:get_update_seq(TxDb),
+
+% Generate a no-op write so that we don't hit the
+% optimization to skip commits on read-only
+% transactions
+bump_view_size(TxDb),
+
+% Generate a conflicting transaction while
+% we're not yet committed
+case get(conflict_pid) of
+undefined ->
+{Pid, Ref} = spawn_monitor(fun() -> generate_conflict(DbName)
end),
+receive {'DOWN', Ref, _, _, normal} -> ok end,
+put(conflict_pid, Pid);
+Pid when is_pid(Pid) ->
+ok
+end,
+
+Seq
+end),
+
+?assertEqual(2, get(retry_count)),
+?assertNotEqual(InitDbSeq, DbSeq).
+
+
+no_retry_with_snapshot({Db, _}) ->
+DbName = fabric2_db:name(Db),
+put(retry_count, 0),
+erase(conflict_pid),
+InitDbSeq = fabric2_db:get_update_seq(Db),
+DbSeq = fabric2_fdb:transactional(Db, fun(TxDb) ->
+put(retry_count, get(retry_count) + 1),
+
+% Fetch the update_seq
+Seq = fabric2_fdb:with_snapshot(TxDb, fun(SSDb) ->
+
[couchdb] 02/04: Allow specifying an end_key for fold_changes
This is an automated email from the ASF dual-hosted git repository.
davisp pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/couchdb.git
commit 0e91f2f608d4a920b6c22f1fe1eb9e0ea6982d5a
Author: Paul J. Davis
AuthorDate: Fri Nov 13 15:13:07 2020 -0600
Allow specifying an end_key for fold_changes
This is useful so that read conflicts on the changes feed will
eventually be resolved. Without an end key specified a reader could end
up in an infinite conflict retry loop if there are clients updating
documents in the database.
---
src/fabric/src/fabric2_db.erl | 18 +++---
src/fabric/test/fabric2_changes_fold_tests.erl | 11 +++
2 files changed, 22 insertions(+), 7 deletions(-)
diff --git a/src/fabric/src/fabric2_db.erl b/src/fabric/src/fabric2_db.erl
index b3e510b..a310470 100644
--- a/src/fabric/src/fabric2_db.erl
+++ b/src/fabric/src/fabric2_db.erl
@@ -1093,14 +1093,18 @@ fold_changes(Db, SinceSeq, UserFun, UserAcc, Options) ->
end,
StartKey = get_since_seq(TxDb, Dir, SinceSeq),
-EndKey = case Dir of
-rev -> fabric2_util:seq_zero_vs();
-_ -> fabric2_util:seq_max_vs()
+EndKey = case fabric2_util:get_value(end_key, Options) of
+undefined when Dir == rev ->
+fabric2_util:seq_zero_vs();
+undefined ->
+fabric2_util:seq_max_vs();
+EK when is_binary(EK) ->
+fabric2_fdb:seq_to_vs(EK);
+EK when is_tuple(EK), element(1, EK) == versionstamp ->
+EK
end,
-FoldOpts = [
-{start_key, StartKey},
-{end_key, EndKey}
-] ++ RestartTx ++ Options,
+BaseOpts = [{start_key, StartKey}] ++ RestartTx ++ Options,
+FoldOpts = lists:keystore(end_key, 1, BaseOpts, {end_key, EndKey}),
{ok, fabric2_fdb:fold_range(TxDb, Prefix, fun({K, V}, Acc) ->
{SeqVS} = erlfdb_tuple:unpack(K, Prefix),
diff --git a/src/fabric/test/fabric2_changes_fold_tests.erl
b/src/fabric/test/fabric2_changes_fold_tests.erl
index 8541d97..fa79f25 100644
--- a/src/fabric/test/fabric2_changes_fold_tests.erl
+++ b/src/fabric/test/fabric2_changes_fold_tests.erl
@@ -40,6 +40,7 @@ changes_fold_test_() ->
?TDEF_FE(fold_changes_basic_rev),
?TDEF_FE(fold_changes_since_now_rev),
?TDEF_FE(fold_changes_since_seq_rev),
+?TDEF_FE(fold_changes_with_end_key),
?TDEF_FE(fold_changes_basic_tx_too_old),
?TDEF_FE(fold_changes_reverse_tx_too_old),
?TDEF_FE(fold_changes_tx_too_old_with_single_row_emits),
@@ -124,6 +125,16 @@ fold_changes_since_seq_rev({Db, DocRows}) ->
fold_changes_since_seq_rev({Db, RestRows}).
+fold_changes_with_end_key({Db, DocRows}) ->
+lists:foldl(fun(DocRow, Acc) ->
+EndSeq = maps:get(sequence, DocRow),
+Changes = changes(Db, 0, [{end_key, EndSeq}]),
+NewAcc = [DocRow | Acc],
+?assertEqual(Changes, NewAcc),
+NewAcc
+end, [], DocRows).
+
+
fold_changes_basic_tx_too_old({Db, DocRows0}) ->
DocRows = lists:reverse(DocRows0),
[couchdb] 04/04: Remove unnecessary line wraps
This is an automated email from the ASF dual-hosted git repository.
davisp pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/couchdb.git
commit 3168a4dbcef7181b67a989e1d723dc6a93e76280
Author: Paul J. Davis
AuthorDate: Tue Nov 17 11:23:36 2020 -0600
Remove unnecessary line wraps
---
src/couch_views/src/couch_views_indexer.erl | 6 ++
1 file changed, 2 insertions(+), 4 deletions(-)
diff --git a/src/couch_views/src/couch_views_indexer.erl
b/src/couch_views/src/couch_views_indexer.erl
index 83ccb2c..e3b2ad5 100644
--- a/src/couch_views/src/couch_views_indexer.erl
+++ b/src/couch_views/src/couch_views_indexer.erl
@@ -235,10 +235,8 @@ do_update(Db, Mrst0, State0) ->
case is_update_finished(State2) of
true ->
-maybe_set_build_status(TxDb, Mrst2, ViewVS,
-?INDEX_READY),
-report_progress(State2#{changes_done := ChangesDone},
-finished),
+maybe_set_build_status(TxDb, Mrst2, ViewVS, ?INDEX_READY),
+report_progress(State2#{changes_done := ChangesDone},
finished),
{Mrst2, finished};
false ->
State3 = report_progress(State2, update),
[couchdb] 01/04: Allow snapshots through fabric2_fdb:transactional
This is an automated email from the ASF dual-hosted git repository.
davisp pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/couchdb.git
commit 1b30d0810ac0d985676c90dc1b6532f9fb84284c
Author: Paul J. Davis
AuthorDate: Fri Nov 13 14:08:52 2020 -0600
Allow snapshots through fabric2_fdb:transactional
---
src/fabric/src/fabric2_fdb.erl | 17
src/fabric/test/fabric2_snapshot_tests.erl | 134 +
2 files changed, 151 insertions(+)
diff --git a/src/fabric/src/fabric2_fdb.erl b/src/fabric/src/fabric2_fdb.erl
index aa2bade..1690f2f 100644
--- a/src/fabric/src/fabric2_fdb.erl
+++ b/src/fabric/src/fabric2_fdb.erl
@@ -18,6 +18,8 @@
transactional/3,
transactional/2,
+with_snapshot/2,
+
create/2,
open/2,
ensure_current/1,
@@ -150,6 +152,11 @@ transactional(#{tx := undefined} = Db, Fun) ->
transactional(Db#{reopen => true}, Fun)
end)
end;
+transactional(#{tx := {erlfdb_snapshot, _}} = Db, Fun) ->
+DbName = maps:get(name, Db, undefined),
+with_span(Fun, #{'db.name' => DbName}, fun() ->
+Fun(Db)
+end);
transactional(#{tx := {erlfdb_transaction, _}} = Db, Fun) ->
DbName = maps:get(name, Db, undefined),
@@ -183,6 +190,14 @@ do_transaction(Fun, LayerPrefix) when is_function(Fun, 1)
->
end.
+with_snapshot(#{tx := {erlfdb_transaction, _} = Tx} = TxDb, Fun) ->
+SSDb = TxDb#{tx := erlfdb:snapshot(Tx)},
+Fun(SSDb);
+
+with_snapshot(#{tx := {erlfdb_snapshot, _}} = SSDb, Fun) ->
+Fun(SSDb).
+
+
create(#{} = Db0, Options) ->
#{
name := DbName,
@@ -1865,6 +1880,8 @@ get_db_handle() ->
end.
+require_transaction(#{tx := {erlfdb_snapshot, _}} = _Db) ->
+ok;
require_transaction(#{tx := {erlfdb_transaction, _}} = _Db) ->
ok;
require_transaction(#{} = _Db) ->
diff --git a/src/fabric/test/fabric2_snapshot_tests.erl
b/src/fabric/test/fabric2_snapshot_tests.erl
new file mode 100644
index 000..37ba626
--- /dev/null
+++ b/src/fabric/test/fabric2_snapshot_tests.erl
@@ -0,0 +1,134 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+% http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(fabric2_snapshot_tests).
+
+
+-include_lib("couch/include/couch_db.hrl").
+-include_lib("couch/include/couch_eunit.hrl").
+-include_lib("eunit/include/eunit.hrl").
+-include("fabric2.hrl").
+-include("fabric2_test.hrl").
+
+
+fdb_ss_test_() ->
+{
+"Test snapshot usage",
+setup,
+fun setup/0,
+fun cleanup/1,
+with([
+?TDEF(retry_without_snapshot),
+?TDEF(no_retry_with_snapshot)
+])
+}.
+
+
+setup() ->
+Ctx = test_util:start_couch([fabric]),
+{ok, Db} = fabric2_db:create(?tempdb(), [{user_ctx, ?ADMIN_USER}]),
+{Db, Ctx}.
+
+
+cleanup({Db, Ctx}) ->
+ok = fabric2_db:delete(fabric2_db:name(Db), []),
+test_util:stop_couch(Ctx).
+
+
+retry_without_snapshot({Db, _}) ->
+DbName = fabric2_db:name(Db),
+put(retry_count, 0),
+erase(conflict_pid),
+InitDbSeq = fabric2_db:get_update_seq(Db),
+DbSeq = fabric2_fdb:transactional(Db, fun(TxDb) ->
+put(retry_count, get(retry_count) + 1),
+
+% Fetch the update_seq
+Seq = fabric2_db:get_update_seq(TxDb),
+
+% Generate a no-op write so that we don't hit the
+% optimization to skip commits on read-only
+% transactions
+bump_view_size(TxDb),
+
+% Generate a conflicting transaction while
+% we're not yet committed
+case get(conflict_pid) of
+undefined ->
+{Pid, Ref} = spawn_monitor(fun() -> generate_conflict(DbName)
end),
+receive {'DOWN', Ref, _, _, normal} -> ok end,
+put(conflict_pid, Pid);
+Pid when is_pid(Pid) ->
+ok
+end,
+
+Seq
+end),
+
+?assertEqual(2, get(retry_count)),
+?assertNotEqual(InitDbSeq, DbSeq).
+
+
+no_retry_with_snapshot({Db, _}) ->
+DbName = fabric2_db:name(Db),
+put(retry_count, 0),
+erase(conflict_pid),
+InitDbSeq = fabric2_db:get_update_seq(Db),
+DbSeq = fabric2_fdb:transactional(Db, fun(TxDb) ->
+put(retry_count, get(retry_count) + 1),
+
+% Fetch the update_seq
+Seq = fabric2_fdb:with_snapshot(TxDb, fun(SSDb) ->
+
[couchdb] 03/04: Minimize conflicts while building views
This is an automated email from the ASF dual-hosted git repository.
davisp pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/couchdb.git
commit 27e8e332cf974bff69b86a8d3c5e2f5a6de4125f
Author: Paul J. Davis
AuthorDate: Fri Nov 13 14:11:00 2020 -0600
Minimize conflicts while building views
This flips the view indexer to grab the database update_seq outside of
the update transaction. Previously we would cosntantly refresh the
db_seq value on every retry of the transactional loop.
We use a snapshot to get the update_seq so that we don't trigger
spurious read conflicts with any clients that might be updating the
database.
---
src/couch_views/src/couch_views_indexer.erl | 54 +++
src/couch_views/test/couch_views_indexer_test.erl | 45 +++
2 files changed, 91 insertions(+), 8 deletions(-)
diff --git a/src/couch_views/src/couch_views_indexer.erl
b/src/couch_views/src/couch_views_indexer.erl
index 2735f66..83ccb2c 100644
--- a/src/couch_views/src/couch_views_indexer.erl
+++ b/src/couch_views/src/couch_views_indexer.erl
@@ -86,15 +86,22 @@ init() ->
fail_job(Job, Data, sig_changed, "Design document was modified")
end,
+DbSeq = fabric2_fdb:transactional(Db, fun(TxDb) ->
+fabric2_fdb:with_snapshot(TxDb, fun(SSDb) ->
+fabric2_db:get_update_seq(SSDb)
+end)
+end),
+
State = #{
tx_db => undefined,
db_uuid => DbUUID,
-db_seq => undefined,
+db_seq => DbSeq,
view_seq => undefined,
last_seq => undefined,
view_vs => undefined,
job => Job,
job_data => Data,
+rows_processed => 0,
count => 0,
changes_done => 0,
doc_acc => [],
@@ -206,8 +213,6 @@ do_update(Db, Mrst0, State0) ->
{ok, State2} = fold_changes(State1),
#{
-count := Count,
-limit := Limit,
doc_acc := DocAcc,
last_seq := LastSeq,
view_vs := ViewVS,
@@ -228,7 +233,7 @@ do_update(Db, Mrst0, State0) ->
total_kvs => TotalKVs
},
-case Count < Limit of
+case is_update_finished(State2) of
true ->
maybe_set_build_status(TxDb, Mrst2, ViewVS,
?INDEX_READY),
@@ -249,6 +254,20 @@ do_update(Db, Mrst0, State0) ->
end).
+is_update_finished(State) ->
+#{
+db_seq := DbSeq,
+last_seq := LastSeq,
+view_vs := ViewVs
+} = State,
+AtDbSeq = LastSeq == DbSeq,
+AtViewVs = case ViewVs of
+not_found -> false;
+_ -> LastSeq == fabric2_fdb:vs_to_seq(ViewVs)
+end,
+AtDbSeq orelse AtViewVs.
+
+
maybe_set_build_status(_TxDb, _Mrst1, not_found, _State) ->
ok;
@@ -258,7 +277,7 @@ maybe_set_build_status(TxDb, Mrst1, _ViewVS, State) ->
% In the first iteration of update we need
% to populate our db and view sequences
-get_update_start_state(TxDb, Mrst, #{db_seq := undefined} = State) ->
+get_update_start_state(TxDb, Mrst, #{view_seq := undefined} = State) ->
#{
view_vs := ViewVS,
view_seq := ViewSeq
@@ -266,7 +285,6 @@ get_update_start_state(TxDb, Mrst, #{db_seq := undefined} =
State) ->
State#{
tx_db := TxDb,
-db_seq := fabric2_db:get_update_seq(TxDb),
view_vs := ViewVS,
view_seq := ViewSeq,
last_seq := ViewSeq
@@ -281,18 +299,36 @@ get_update_start_state(TxDb, _Idx, State) ->
fold_changes(State) ->
#{
view_seq := SinceSeq,
+db_seq := DbSeq,
limit := Limit,
tx_db := TxDb
} = State,
+FoldState = State#{
+rows_processed := 0
+},
+
Fun = fun process_changes/2,
-Opts = [{limit, Limit}, {restart_tx, false}],
-fabric2_db:fold_changes(TxDb, SinceSeq, Fun, State, Opts).
+Opts = [
+{end_key, fabric2_fdb:seq_to_vs(DbSeq)},
+{limit, Limit},
+{restart_tx, false}
+],
+case fabric2_db:fold_changes(TxDb, SinceSeq, Fun, FoldState, Opts) of
+{ok, #{rows_processed := 0} = FinalState} when Limit > 0 ->
+% If we read zero rows with a non-zero limit
+% it means we've caught up to the DbSeq as our
+% last_seq.
+{ok, FinalState#{last_seq := DbSeq}};
+Result ->
+Result
+end.
process_changes(Change, Acc) ->
#{
doc_acc := DocAcc,
+rows_processed := RowsProcessed,
count := Count,
design_opts := DesignOpts,
view_vs := ViewVS
@@ -308,12 +344,14 @@ process_changes(Change, Acc) ->
Acc1 = case {Id, IncludeDesign} of
{<>, false} ->
maps:merge(Acc, #{
+rows_processed => RowsProcessed + 1,
[couchdb] branch main updated (0b98b3b -> 3168a4d)
This is an automated email from the ASF dual-hosted git repository. davisp pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/couchdb.git. from 0b98b3b Merge pull request #3274 from cloudant/fix-mango-tests new 1b30d08 Allow snapshots through fabric2_fdb:transactional new 0e91f2f Allow specifying an end_key for fold_changes new 27e8e33 Minimize conflicts while building views new 3168a4d Remove unnecessary line wraps The 4 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: src/couch_views/src/couch_views_indexer.erl | 60 -- src/couch_views/test/couch_views_indexer_test.erl | 45 src/fabric/src/fabric2_db.erl | 18 +-- src/fabric/src/fabric2_fdb.erl| 17 +++ src/fabric/test/fabric2_changes_fold_tests.erl| 11 ++ src/fabric/test/fabric2_snapshot_tests.erl| 134 ++ 6 files changed, 266 insertions(+), 19 deletions(-) create mode 100644 src/fabric/test/fabric2_snapshot_tests.erl
[couchdb] branch ensure-view-build-progress updated (08352a8 -> 652876a)
This is an automated email from the ASF dual-hosted git repository. davisp pushed a change to branch ensure-view-build-progress in repository https://gitbox.apache.org/repos/asf/couchdb.git. discard 08352a8 Remove unnecessary line wraps discard 351d13c Minimize conflicts while building views discard 0f7f2a6 Allow specifying an end_key for fold_changes discard fc9e7cf Allow snapshots through fabric2_fdb:transactional add b9bf00a Allow snapshots through fabric2_fdb:transactional add 5e28a46 Allow specifying an end_key for fold_changes add 9819cb8 Minimize conflicts while building views add 652876a Remove unnecessary line wraps This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (08352a8) \ N -- N -- N refs/heads/ensure-view-build-progress (652876a) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: src/couch_views/test/couch_views_indexer_test.erl | 45 +++ src/fabric/test/fabric2_snapshot_tests.erl| 5 +++ 2 files changed, 50 insertions(+)
[couchdb] branch ensure-view-build-progress updated (7995675 -> 08352a8)
This is an automated email from the ASF dual-hosted git repository. davisp pushed a change to branch ensure-view-build-progress in repository https://gitbox.apache.org/repos/asf/couchdb.git. discard 7995675 Remove unnecessary line wraps discard feb66d5 Minimize conflicts while building views discard b04cdc4 Allow specifying an end_key for fold_changes discard 3844f93 Allow snapshots through fabric2_fdb:transactional add fc9e7cf Allow snapshots through fabric2_fdb:transactional add 0f7f2a6 Allow specifying an end_key for fold_changes add 351d13c Minimize conflicts while building views add 08352a8 Remove unnecessary line wraps This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (7995675) \ N -- N -- N refs/heads/ensure-view-build-progress (08352a8) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: src/fabric/src/fabric2_db.erl | 4 +- src/fabric/test/fabric2_changes_fold_tests.erl | 11 +++ src/fabric/test/fabric2_snapshot_tests.erl | 129 + 3 files changed, 143 insertions(+), 1 deletion(-) create mode 100644 src/fabric/test/fabric2_snapshot_tests.erl
[couchdb] branch ensure-view-build-progress updated (ab5d87e -> 7995675)
This is an automated email from the ASF dual-hosted git repository. davisp pushed a change to branch ensure-view-build-progress in repository https://gitbox.apache.org/repos/asf/couchdb.git. discard ab5d87e Minimize conflicts while building views discard 82ff3e0 Allow specifying an end_key for fold_changes omit 7aa1806 Allow snapshots through fabric2_fdb:transactional add 1be2312 Use `req_body` field if present add 5cfa4a8 Merge pull request #3268 from cloudant/reuse-json-body add 8c21567 Bump erlfdb to v1.2.3 add 3844f93 Allow snapshots through fabric2_fdb:transactional add b04cdc4 Allow specifying an end_key for fold_changes add feb66d5 Minimize conflicts while building views add 7995675 Remove unnecessary line wraps This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (ab5d87e) \ N -- N -- N refs/heads/ensure-view-build-progress (7995675) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: rebar.config.script | 2 +- src/couch/src/couch_httpd.erl | 7 -- src/couch_views/src/couch_views_indexer.erl | 38 + 3 files changed, 34 insertions(+), 13 deletions(-)
[couchdb] branch ensure-view-build-progress updated (6baf574 -> ab5d87e)
This is an automated email from the ASF dual-hosted git repository. davisp pushed a change to branch ensure-view-build-progress in repository https://gitbox.apache.org/repos/asf/couchdb.git. omit 6baf574 Minimize conflicts while building views add 82ff3e0 Allow specifying an end_key for fold_changes add ab5d87e Minimize conflicts while building views This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (6baf574) \ N -- N -- N refs/heads/ensure-view-build-progress (ab5d87e) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: src/couch_views/src/couch_views_indexer.erl | 61 ++--- src/fabric/src/fabric2_db.erl | 16 2 files changed, 38 insertions(+), 39 deletions(-)
[couchdb] 02/02: Minimize conflicts while building views
This is an automated email from the ASF dual-hosted git repository.
davisp pushed a commit to branch ensure-view-build-progress
in repository https://gitbox.apache.org/repos/asf/couchdb.git
commit 6baf57452e2ccb9869c50da93880f0742a11a97b
Author: Paul J. Davis
AuthorDate: Fri Nov 13 14:11:00 2020 -0600
Minimize conflicts while building views
If a client is continuously updating a database the view engine would
fail to make progress because the last update sequence is continuously
changing and the unbounded changes read would create a conflict to the
maximum versionstamp value used to delineate the end of the changes key
space.
This commits makes two changes to avoid this. First, the current
update_seq is read via snapshot so that it doesn't become a read
conflict if there's an update. This part is fairly straightforward.
The second change reads the changes feed in a snapshot and then manually
adds the read conflict range for the keys that were actually read. This
solves the issue for large view builds while a database is being
updated. However, this does not address the issue when its a single
document being updated repeatedly. Perhaps if we keep track of the last
sequence read and apply that as a limit when updating the view on a
conflict retry?
---
src/couch_views/src/couch_views_indexer.erl | 37 ++---
1 file changed, 29 insertions(+), 8 deletions(-)
diff --git a/src/couch_views/src/couch_views_indexer.erl
b/src/couch_views/src/couch_views_indexer.erl
index 2735f66..9fdbdb2 100644
--- a/src/couch_views/src/couch_views_indexer.erl
+++ b/src/couch_views/src/couch_views_indexer.erl
@@ -264,13 +264,15 @@ get_update_start_state(TxDb, Mrst, #{db_seq := undefined}
= State) ->
view_seq := ViewSeq
} = couch_views_fdb:get_view_state(TxDb, Mrst),
-State#{
-tx_db := TxDb,
-db_seq := fabric2_db:get_update_seq(TxDb),
-view_vs := ViewVS,
-view_seq := ViewSeq,
-last_seq := ViewSeq
-};
+fabric2_fdb:with_snapshot(TxDb, fun(SSDb) ->
+State#{
+tx_db := TxDb,
+db_seq := fabric2_db:get_update_seq(SSDb),
+view_vs := ViewVS,
+view_seq := ViewSeq,
+last_seq := ViewSeq
+}
+end);
get_update_start_state(TxDb, _Idx, State) ->
State#{
@@ -287,7 +289,16 @@ fold_changes(State) ->
Fun = fun process_changes/2,
Opts = [{limit, Limit}, {restart_tx, false}],
-fabric2_db:fold_changes(TxDb, SinceSeq, Fun, State, Opts).
+Result = fabric2_fdb:with_snapshot(TxDb, fun(SSDb) ->
+fabric2_db:fold_changes(SSDb, SinceSeq, Fun, State, Opts)
+end),
+case Result of
+{ok, #{last_seq := LastSeq}} ->
+set_changes_conflict(TxDb, SinceSeq,
fabric2_fdb:seq_to_vs(LastSeq));
+_ ->
+ok
+end,
+Result.
process_changes(Change, Acc) ->
@@ -597,6 +608,16 @@ fail_job(Job, Data, Error, Reason) ->
exit(normal).
+set_changes_conflict(TxDb, StartVS, EndVS) ->
+#{
+tx := Tx,
+db_prefix := DbPrefix
+} = TxDb,
+StartKey = erlfdb_tuple:pack({?DB_CHANGES, StartVS}, DbPrefix),
+EndKey = erlfdb_tuple:pack({?DB_CHANGES, EndVS}, DbPrefix),
+erlfdb:add_read_conflict_range(Tx, StartKey, EndKey).
+
+
retry_limit() ->
config:get_integer("couch_views", "retry_limit", 3).
[couchdb] 01/02: Allow snapshots through fabric2_fdb:transactional
This is an automated email from the ASF dual-hosted git repository.
davisp pushed a commit to branch ensure-view-build-progress
in repository https://gitbox.apache.org/repos/asf/couchdb.git
commit 7aa1806fc052c9e770e00596fa53f25cc6dbfb43
Author: Paul J. Davis
AuthorDate: Fri Nov 13 14:08:52 2020 -0600
Allow snapshots through fabric2_fdb:transactional
---
src/fabric/src/fabric2_fdb.erl | 17 +
1 file changed, 17 insertions(+)
diff --git a/src/fabric/src/fabric2_fdb.erl b/src/fabric/src/fabric2_fdb.erl
index aa2bade..1690f2f 100644
--- a/src/fabric/src/fabric2_fdb.erl
+++ b/src/fabric/src/fabric2_fdb.erl
@@ -18,6 +18,8 @@
transactional/3,
transactional/2,
+with_snapshot/2,
+
create/2,
open/2,
ensure_current/1,
@@ -150,6 +152,11 @@ transactional(#{tx := undefined} = Db, Fun) ->
transactional(Db#{reopen => true}, Fun)
end)
end;
+transactional(#{tx := {erlfdb_snapshot, _}} = Db, Fun) ->
+DbName = maps:get(name, Db, undefined),
+with_span(Fun, #{'db.name' => DbName}, fun() ->
+Fun(Db)
+end);
transactional(#{tx := {erlfdb_transaction, _}} = Db, Fun) ->
DbName = maps:get(name, Db, undefined),
@@ -183,6 +190,14 @@ do_transaction(Fun, LayerPrefix) when is_function(Fun, 1)
->
end.
+with_snapshot(#{tx := {erlfdb_transaction, _} = Tx} = TxDb, Fun) ->
+SSDb = TxDb#{tx := erlfdb:snapshot(Tx)},
+Fun(SSDb);
+
+with_snapshot(#{tx := {erlfdb_snapshot, _}} = SSDb, Fun) ->
+Fun(SSDb).
+
+
create(#{} = Db0, Options) ->
#{
name := DbName,
@@ -1865,6 +1880,8 @@ get_db_handle() ->
end.
+require_transaction(#{tx := {erlfdb_snapshot, _}} = _Db) ->
+ok;
require_transaction(#{tx := {erlfdb_transaction, _}} = _Db) ->
ok;
require_transaction(#{} = _Db) ->
[couchdb] branch ensure-view-build-progress created (now 6baf574)
This is an automated email from the ASF dual-hosted git repository. davisp pushed a change to branch ensure-view-build-progress in repository https://gitbox.apache.org/repos/asf/couchdb.git. at 6baf574 Minimize conflicts while building views This branch includes the following new commits: new 7aa1806 Allow snapshots through fabric2_fdb:transactional new 6baf574 Minimize conflicts while building views The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[couchdb] branch fix-ebtree-range-clears updated (3828f92 -> e11ab46)
This is an automated email from the ASF dual-hosted git repository. davisp pushed a change to branch fix-ebtree-range-clears in repository https://gitbox.apache.org/repos/asf/couchdb.git. discard 3828f92 Avoid deleting UUID keys that start with zeros new e11ab46 Avoid deleting UUID keys that start with zeros This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (3828f92) \ N -- N -- N refs/heads/fix-ebtree-range-clears (e11ab46) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes:
[couchdb] 01/01: Avoid deleting UUID keys that start with zeros
This is an automated email from the ASF dual-hosted git repository. davisp pushed a commit to branch fix-ebtree-range-clears in repository https://gitbox.apache.org/repos/asf/couchdb.git commit e11ab46a16a2fd9ecef3bc813e2d73a4ff84888d Author: Paul J. Davis AuthorDate: Thu Nov 12 16:57:44 2020 -0600 Avoid deleting UUID keys that start with zeros Any ebtree that uses chunked key encoding will accidentally wipe out any nodes that have a UUID with more than one leading zero byte. --- src/ebtree/src/ebtree.erl | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/ebtree/src/ebtree.erl b/src/ebtree/src/ebtree.erl index 31e1fc8..8bfb2ea 100644 --- a/src/ebtree/src/ebtree.erl +++ b/src/ebtree/src/ebtree.erl @@ -56,7 +56,7 @@ -define(META_ORDER, 0). -define(NODE, 1). --define(NODE_ROOT_ID, <<0>>). +-define(NODE_ROOT_ID, <<0:128>>). -define(underflow(Tree, Node), Tree#tree.min > length(Node#node.members)). -define(at_min(Tree, Node), Tree#tree.min == length(Node#node.members)).
[couchdb] 01/01: Avoid deleting UUID keys that start with zeros
This is an automated email from the ASF dual-hosted git repository. davisp pushed a commit to branch fix-ebtree-range-clears in repository https://gitbox.apache.org/repos/asf/couchdb.git commit 3828f924459899b98d3414e697877839f0df7f75 Author: Paul J. Davis AuthorDate: Thu Nov 12 16:57:44 2020 -0600 Avoid deleting UUID keys that start with zeros --- src/ebtree/src/ebtree.erl | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/ebtree/src/ebtree.erl b/src/ebtree/src/ebtree.erl index 31e1fc8..8bfb2ea 100644 --- a/src/ebtree/src/ebtree.erl +++ b/src/ebtree/src/ebtree.erl @@ -56,7 +56,7 @@ -define(META_ORDER, 0). -define(NODE, 1). --define(NODE_ROOT_ID, <<0>>). +-define(NODE_ROOT_ID, <<0:128>>). -define(underflow(Tree, Node), Tree#tree.min > length(Node#node.members)). -define(at_min(Tree, Node), Tree#tree.min == length(Node#node.members)).
[couchdb] branch fix-ebtree-range-clears created (now 3828f92)
This is an automated email from the ASF dual-hosted git repository. davisp pushed a change to branch fix-ebtree-range-clears in repository https://gitbox.apache.org/repos/asf/couchdb.git. at 3828f92 Avoid deleting UUID keys that start with zeros This branch includes the following new commits: new 3828f92 Avoid deleting UUID keys that start with zeros The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[couchdb] 03/03: Fix flaky test
This is an automated email from the ASF dual-hosted git repository.
davisp pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/couchdb.git
commit 1a138804f692d0e578a7d4a19f9f13099a59a1cd
Author: Paul J. Davis
AuthorDate: Tue Nov 10 12:46:07 2020 -0600
Fix flaky test
Waiting for the timeout option to be set means we could still sneak in
and grab the old FDB database handle before fabric2_server updated it in
the application environment.
This new approach just waits until the handle has been updated by
watching the value in the application environment directly.
---
src/fabric/test/fabric2_tx_options_tests.erl | 10 --
1 file changed, 8 insertions(+), 2 deletions(-)
diff --git a/src/fabric/test/fabric2_tx_options_tests.erl
b/src/fabric/test/fabric2_tx_options_tests.erl
index b93cc3d..3232922 100644
--- a/src/fabric/test/fabric2_tx_options_tests.erl
+++ b/src/fabric/test/fabric2_tx_options_tests.erl
@@ -84,9 +84,15 @@ can_configure_options_at_runtime(_) ->
meck:reset(erlfdb),
+% Wait until after fabric2_server has updated the new fdb handle
+OldDbHandle = get(?PDICT_DB_KEY),
config:delete("fdb_tx_options", "size_limit", false),
-% Assert that we get a new handle and are setting our default values
-meck:wait(erlfdb, set_option, ['_', timeout, '_'], 4000),
+test_util:wait(fun() ->
+case application:get_env(fabric, db) of
+{ok, OldDbHandle} -> wait;
+{ok, _} -> ok
+end
+end),
erase(?PDICT_DB_KEY),
{ok, Db1} = fabric2_db:open(DbName, [?ADMIN_CTX]),
[couchdb] 01/03: Remove ebtree caching
This is an automated email from the ASF dual-hosted git repository.
davisp pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/couchdb.git
commit 8267950dc8055b883e599cd993e688de1cec3ab4
Author: Paul J. Davis
AuthorDate: Tue Nov 10 11:44:53 2020 -0600
Remove ebtree caching
The ebtree caching layer does not work correctly in conjunction with
FoundationDB transaction retry semantics. If we incorrectly cache nodes
that are not actually read from FoundationDB, a retried transaction will
rely on incorrectly cached state and corrupt the ebtree persisted in
FoundationDB.
---
src/ebtree/src/ebtree.erl | 130 +++---
1 file changed, 18 insertions(+), 112 deletions(-)
diff --git a/src/ebtree/src/ebtree.erl b/src/ebtree/src/ebtree.erl
index 97a8203..31e1fc8 100644
--- a/src/ebtree/src/ebtree.erl
+++ b/src/ebtree/src/ebtree.erl
@@ -49,8 +49,7 @@
collate_fun,
reduce_fun,
encode_fun,
-persist_fun,
-cache_fun
+persist_fun
}).
-define(META, 0).
@@ -93,15 +92,13 @@ open(Db, Prefix, Order, Options) when is_binary(Prefix),
is_integer(Order), Orde
CollateFun = proplists:get_value(collate_fun, Options, fun collate_raw/2),
EncodeFun = proplists:get_value(encode_fun, Options, fun encode_erlang/3),
PersistFun = proplists:get_value(persist_fun, Options, fun
simple_persist/3),
-CacheFun = proplists:get_value(cache_fun, Options, fun cache_noop/2),
Tree = #tree{
prefix = Prefix,
reduce_fun = ReduceFun,
collate_fun = CollateFun,
encode_fun = EncodeFun,
-persist_fun = PersistFun,
-cache_fun = CacheFun
+persist_fun = PersistFun
},
erlfdb:transactional(Db, fun(Tx) ->
@@ -607,10 +604,9 @@ split_child(Tx, #tree{} = Tree, #node{} = Parent0, #node{}
= Child) ->
umerge_members(Tree, Parent0#node.level, [{FirstRightKey,
LastRightKey, RightId, RightReduction}],
lists:keydelete(Child#node.id, 3, Parent0#node.members)))
},
-Parent2 = new_node_id_if_cacheable(Tx, Tree, Parent0, Parent1),
clear_node(Tx, Tree, Child),
-set_nodes(Tx, Tree, [LeftChild, RightChild, Parent2]),
-{Parent2, LeftChild, RightChild}.
+set_nodes(Tx, Tree, [LeftChild, RightChild, Parent1]),
+{Parent1, LeftChild, RightChild}.
update_prev_neighbour(_Tx, #tree{} = _Tree, #node{prev = undefined} = _Node) ->
@@ -634,7 +630,7 @@ insert_nonfull(Tx, #tree{} = Tree, #node{level = 0} =
Node0, Key, Value) ->
members = umerge_members(Tree, 0, [{Key, Value}], Node0#node.members)
},
set_node(Tx, Tree, Node0, Node1),
-{Node1#node.id, reduce_node(Tree, Node1)};
+reduce_node(Tree, Node1);
insert_nonfull(Tx, #tree{} = Tree, #node{} = Node0, Key, Value) ->
ChildId0 = find_child_id(Tree, Node0, Key),
@@ -654,17 +650,16 @@ insert_nonfull(Tx, #tree{} = Tree, #node{} = Node0, Key,
Value) ->
{Node0, Child0}
end,
ChildId1 = Child1#node.id,
-{ChildId2, NewReduction} = insert_nonfull(Tx, Tree, Child1, Key, Value),
+NewReduction = insert_nonfull(Tx, Tree, Child1, Key, Value),
{CurrentFirstKey, CurrentLastKey, ChildId1, _OldReduction} =
lists:keyfind(ChildId1, 3, Node1#node.members),
[NewFirstKey, _] = sort_keys(Tree, [Key, CurrentFirstKey]),
[_, NewLastKey] = sort_keys(Tree, [Key, CurrentLastKey]),
Node2 = Node1#node{
members = lists:keyreplace(ChildId1, 3, Node1#node.members,
-{NewFirstKey, NewLastKey, ChildId2, NewReduction})
+{NewFirstKey, NewLastKey, ChildId1, NewReduction})
},
-Node3 = new_node_id_if_cacheable(Tx, Tree, Node0, Node2),
-set_node(Tx, Tree, Node0, Node3),
-{Node3#node.id, reduce_node(Tree, Node2)}.
+set_node(Tx, Tree, Node0, Node2),
+reduce_node(Tree, Node2).
%% @doc Inserts or updates multiple values in the ebtree
@@ -724,14 +719,8 @@ split_node_multi(Tx, Tree, Node) ->
true when Node#node.id == ?NODE_ROOT_ID ->
Node#node.members;
true ->
-NewNode = case node_is_cacheable(Node) of
-true ->
-Node#node{id = new_node_id()};
-false ->
-Node
-end,
-set_node(Tx, Tree, NewNode),
-[to_member(Tree, NewNode)];
+set_node(Tx, Tree, Node),
+[to_member(Tree, Node)];
false ->
clear_node(Tx, Tree, Node),
Nodes0 = create_nodes(Tx, Tree, Node),
@@ -883,18 +872,16 @@ delete(Tx, #tree{} = Tree, #node{} = Parent0, Key) ->
Parent1 = Parent0#node{
members = Members3
},
-Parent2 = new_node_id_if_cacheable(Tx, Tree, Parent0, Parent1),
clear_nodes(Tx, Tree, [Child0, Sibling]),
set_nodes(Tx, Tree, NewNodes),
-Parent
[couchdb] 02/03: Remove use of caching
This is an automated email from the ASF dual-hosted git repository.
davisp pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/couchdb.git
commit 7d3caa97b18a8bd953c96b39c9a986a4a11e5f34
Author: Paul J. Davis
AuthorDate: Tue Nov 10 11:52:17 2020 -0600
Remove use of caching
Turns out that ebtree caching wasn't quite correct so removing it for
now.
---
src/couch_views/src/couch_views_trees.erl | 28 +---
1 file changed, 1 insertion(+), 27 deletions(-)
diff --git a/src/couch_views/src/couch_views_trees.erl
b/src/couch_views/src/couch_views_trees.erl
index 236e89f..51c1e46 100644
--- a/src/couch_views/src/couch_views_trees.erl
+++ b/src/couch_views/src/couch_views_trees.erl
@@ -229,7 +229,6 @@ open_id_tree(TxDb, Sig) ->
Prefix = id_tree_prefix(DbPrefix, Sig),
TreeOpts = [
{persist_fun, fun couch_views_fdb:persist_chunks/3},
-{cache_fun, create_cache_fun(id_tree)},
{encode_fun, create_encode_fun(TxDb)}
],
ebtree:open(Tx, Prefix, get_order(id_btree), TreeOpts).
@@ -255,8 +254,7 @@ open_view_tree(TxDb, Sig, Lang, View, Options) ->
[{reduce_fun, RedFun}];
false ->
[
-{reduce_fun, make_reduce_fun(Lang, View)},
-{cache_fun, create_cache_fun({view, ViewId})}
+{reduce_fun, make_reduce_fun(Lang, View)}
]
end,
TreeOpts = BaseOpts ++ ExtraOpts,
@@ -333,30 +331,6 @@ make_reduce_fun(Lang, #mrview{} = View) ->
end.
-create_cache_fun(TreeId) ->
-CacheTid = case get(TreeId) of
-undefined ->
-Tid = ets:new(?MODULE, [protected, set]),
-put(TreeId, {ebtree_cache, Tid}),
-Tid;
-{ebtree_cache, Tid} ->
-Tid
-end,
-fun
-(set, [Id, Node]) ->
-true = ets:insert_new(CacheTid, {Id, Node}),
-ok;
-(clear, Id) ->
-ets:delete(CacheTid, Id),
-ok;
-(get, Id) ->
-case ets:lookup(CacheTid, Id) of
-[{Id, Node}] -> Node;
-[] -> undefined
-end
-end.
-
-
create_encode_fun(TxDb) ->
fun
(encode, Key, Term) ->
[couchdb] branch main updated (647fd16 -> 1a13880)
This is an automated email from the ASF dual-hosted git repository. davisp pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/couchdb.git. from 647fd16 Merge pull request #3244 from apache/aegis_db_info new 8267950 Remove ebtree caching new 7d3caa9 Remove use of caching new 1a13880 Fix flaky test The 3 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: src/couch_views/src/couch_views_trees.erl| 28 +- src/ebtree/src/ebtree.erl| 130 --- src/fabric/test/fabric2_tx_options_tests.erl | 10 ++- 3 files changed, 27 insertions(+), 141 deletions(-)
[couchdb] branch remove-ebtree-caching updated (7bcb944 -> 2a8b920)
This is an automated email from the ASF dual-hosted git repository. davisp pushed a change to branch remove-ebtree-caching in repository https://gitbox.apache.org/repos/asf/couchdb.git. discard 7bcb944 Fix flaky test add 2a8b920 Fix flaky test This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (7bcb944) \ N -- N -- N refs/heads/remove-ebtree-caching (2a8b920) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: src/fabric/test/fabric2_tx_options_tests.erl | 11 +-- 1 file changed, 9 insertions(+), 2 deletions(-)
[couchdb] branch remove-ebtree-caching updated (107cdeb -> 7bcb944)
This is an automated email from the ASF dual-hosted git repository. davisp pushed a change to branch remove-ebtree-caching in repository https://gitbox.apache.org/repos/asf/couchdb.git. from 107cdeb Remove use of caching add 7bcb944 Fix flaky test No new revisions were added by this update. Summary of changes: src/fabric/test/fabric2_tx_options_tests.erl | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[couchdb] 01/02: Remove ebtree caching
This is an automated email from the ASF dual-hosted git repository.
davisp pushed a commit to branch remove-ebtree-caching
in repository https://gitbox.apache.org/repos/asf/couchdb.git
commit 78400b900f971219295b759d0f5b36d4efca75fc
Author: Paul J. Davis
AuthorDate: Tue Nov 10 11:44:53 2020 -0600
Remove ebtree caching
The ebtree caching layer does not work correctly in conjunction with
FoundationDB transaction retry semantics. If we incorrectly cache nodes
that are not actually read from FoundationDB, a retried transaction will
rely on incorrectly cached state and corrupt the ebtree persisted in
FoundationDB.
---
src/ebtree/src/ebtree.erl | 130 +++---
1 file changed, 18 insertions(+), 112 deletions(-)
diff --git a/src/ebtree/src/ebtree.erl b/src/ebtree/src/ebtree.erl
index 97a8203..31e1fc8 100644
--- a/src/ebtree/src/ebtree.erl
+++ b/src/ebtree/src/ebtree.erl
@@ -49,8 +49,7 @@
collate_fun,
reduce_fun,
encode_fun,
-persist_fun,
-cache_fun
+persist_fun
}).
-define(META, 0).
@@ -93,15 +92,13 @@ open(Db, Prefix, Order, Options) when is_binary(Prefix),
is_integer(Order), Orde
CollateFun = proplists:get_value(collate_fun, Options, fun collate_raw/2),
EncodeFun = proplists:get_value(encode_fun, Options, fun encode_erlang/3),
PersistFun = proplists:get_value(persist_fun, Options, fun
simple_persist/3),
-CacheFun = proplists:get_value(cache_fun, Options, fun cache_noop/2),
Tree = #tree{
prefix = Prefix,
reduce_fun = ReduceFun,
collate_fun = CollateFun,
encode_fun = EncodeFun,
-persist_fun = PersistFun,
-cache_fun = CacheFun
+persist_fun = PersistFun
},
erlfdb:transactional(Db, fun(Tx) ->
@@ -607,10 +604,9 @@ split_child(Tx, #tree{} = Tree, #node{} = Parent0, #node{}
= Child) ->
umerge_members(Tree, Parent0#node.level, [{FirstRightKey,
LastRightKey, RightId, RightReduction}],
lists:keydelete(Child#node.id, 3, Parent0#node.members)))
},
-Parent2 = new_node_id_if_cacheable(Tx, Tree, Parent0, Parent1),
clear_node(Tx, Tree, Child),
-set_nodes(Tx, Tree, [LeftChild, RightChild, Parent2]),
-{Parent2, LeftChild, RightChild}.
+set_nodes(Tx, Tree, [LeftChild, RightChild, Parent1]),
+{Parent1, LeftChild, RightChild}.
update_prev_neighbour(_Tx, #tree{} = _Tree, #node{prev = undefined} = _Node) ->
@@ -634,7 +630,7 @@ insert_nonfull(Tx, #tree{} = Tree, #node{level = 0} =
Node0, Key, Value) ->
members = umerge_members(Tree, 0, [{Key, Value}], Node0#node.members)
},
set_node(Tx, Tree, Node0, Node1),
-{Node1#node.id, reduce_node(Tree, Node1)};
+reduce_node(Tree, Node1);
insert_nonfull(Tx, #tree{} = Tree, #node{} = Node0, Key, Value) ->
ChildId0 = find_child_id(Tree, Node0, Key),
@@ -654,17 +650,16 @@ insert_nonfull(Tx, #tree{} = Tree, #node{} = Node0, Key,
Value) ->
{Node0, Child0}
end,
ChildId1 = Child1#node.id,
-{ChildId2, NewReduction} = insert_nonfull(Tx, Tree, Child1, Key, Value),
+NewReduction = insert_nonfull(Tx, Tree, Child1, Key, Value),
{CurrentFirstKey, CurrentLastKey, ChildId1, _OldReduction} =
lists:keyfind(ChildId1, 3, Node1#node.members),
[NewFirstKey, _] = sort_keys(Tree, [Key, CurrentFirstKey]),
[_, NewLastKey] = sort_keys(Tree, [Key, CurrentLastKey]),
Node2 = Node1#node{
members = lists:keyreplace(ChildId1, 3, Node1#node.members,
-{NewFirstKey, NewLastKey, ChildId2, NewReduction})
+{NewFirstKey, NewLastKey, ChildId1, NewReduction})
},
-Node3 = new_node_id_if_cacheable(Tx, Tree, Node0, Node2),
-set_node(Tx, Tree, Node0, Node3),
-{Node3#node.id, reduce_node(Tree, Node2)}.
+set_node(Tx, Tree, Node0, Node2),
+reduce_node(Tree, Node2).
%% @doc Inserts or updates multiple values in the ebtree
@@ -724,14 +719,8 @@ split_node_multi(Tx, Tree, Node) ->
true when Node#node.id == ?NODE_ROOT_ID ->
Node#node.members;
true ->
-NewNode = case node_is_cacheable(Node) of
-true ->
-Node#node{id = new_node_id()};
-false ->
-Node
-end,
-set_node(Tx, Tree, NewNode),
-[to_member(Tree, NewNode)];
+set_node(Tx, Tree, Node),
+[to_member(Tree, Node)];
false ->
clear_node(Tx, Tree, Node),
Nodes0 = create_nodes(Tx, Tree, Node),
@@ -883,18 +872,16 @@ delete(Tx, #tree{} = Tree, #node{} = Parent0, Key) ->
Parent1 = Parent0#node{
members = Members3
},
-Parent2 = new_node_id_if_cacheable(Tx, Tree, Parent0, Parent1),
clear_nodes(Tx, Tree, [Child0, Sibling]),
set_nodes(Tx, Tree, NewNodes),
-
[couchdb] branch remove-ebtree-caching created (now 107cdeb)
This is an automated email from the ASF dual-hosted git repository. davisp pushed a change to branch remove-ebtree-caching in repository https://gitbox.apache.org/repos/asf/couchdb.git. at 107cdeb Remove use of caching This branch includes the following new commits: new 78400b9 Remove ebtree caching new 107cdeb Remove use of caching The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[couchdb] 02/02: Remove use of caching
This is an automated email from the ASF dual-hosted git repository.
davisp pushed a commit to branch remove-ebtree-caching
in repository https://gitbox.apache.org/repos/asf/couchdb.git
commit 107cdebd550a3d60d65a741d16a857285127a1ee
Author: Paul J. Davis
AuthorDate: Tue Nov 10 11:52:17 2020 -0600
Remove use of caching
Turns out that ebtree caching wasn't quite correct so removing it for
now.
---
src/couch_views/src/couch_views_trees.erl | 28 +---
1 file changed, 1 insertion(+), 27 deletions(-)
diff --git a/src/couch_views/src/couch_views_trees.erl
b/src/couch_views/src/couch_views_trees.erl
index 236e89f..51c1e46 100644
--- a/src/couch_views/src/couch_views_trees.erl
+++ b/src/couch_views/src/couch_views_trees.erl
@@ -229,7 +229,6 @@ open_id_tree(TxDb, Sig) ->
Prefix = id_tree_prefix(DbPrefix, Sig),
TreeOpts = [
{persist_fun, fun couch_views_fdb:persist_chunks/3},
-{cache_fun, create_cache_fun(id_tree)},
{encode_fun, create_encode_fun(TxDb)}
],
ebtree:open(Tx, Prefix, get_order(id_btree), TreeOpts).
@@ -255,8 +254,7 @@ open_view_tree(TxDb, Sig, Lang, View, Options) ->
[{reduce_fun, RedFun}];
false ->
[
-{reduce_fun, make_reduce_fun(Lang, View)},
-{cache_fun, create_cache_fun({view, ViewId})}
+{reduce_fun, make_reduce_fun(Lang, View)}
]
end,
TreeOpts = BaseOpts ++ ExtraOpts,
@@ -333,30 +331,6 @@ make_reduce_fun(Lang, #mrview{} = View) ->
end.
-create_cache_fun(TreeId) ->
-CacheTid = case get(TreeId) of
-undefined ->
-Tid = ets:new(?MODULE, [protected, set]),
-put(TreeId, {ebtree_cache, Tid}),
-Tid;
-{ebtree_cache, Tid} ->
-Tid
-end,
-fun
-(set, [Id, Node]) ->
-true = ets:insert_new(CacheTid, {Id, Node}),
-ok;
-(clear, Id) ->
-ets:delete(CacheTid, Id),
-ok;
-(get, Id) ->
-case ets:lookup(CacheTid, Id) of
-[{Id, Node}] -> Node;
-[] -> undefined
-end
-end.
-
-
create_encode_fun(TxDb) ->
fun
(encode, Key, Term) ->
[couchdb] branch changes_filter_all_docs_oom_main updated (ab01683 -> 2a8ddbd)
This is an automated email from the ASF dual-hosted git repository. davisp pushed a change to branch changes_filter_all_docs_oom_main in repository https://gitbox.apache.org/repos/asf/couchdb.git. from ab01683 Retry filter_docs sequentially if the patch exceeds couchjs stack add 2a8ddbd Add test for splitting filter batches No new revisions were added by this update. Summary of changes: src/couch/test/eunit/couch_query_servers_tests.erl | 55 ++ 1 file changed, 55 insertions(+)
[couchdb] branch main updated: Allow administrators to optionally encrypt views
This is an automated email from the ASF dual-hosted git repository.
davisp pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/couchdb.git
The following commit(s) were added to refs/heads/main by this push:
new 4b10750 Allow administrators to optionally encrypt views
4b10750 is described below
commit 4b107505e9862af35fe616e12892fdf6cfffb12d
Author: Paul J. Davis
AuthorDate: Tue Nov 3 12:36:50 2020 -0600
Allow administrators to optionally encrypt views
---
src/couch_views/src/couch_views_trees.erl | 17 +++--
1 file changed, 15 insertions(+), 2 deletions(-)
diff --git a/src/couch_views/src/couch_views_trees.erl
b/src/couch_views/src/couch_views_trees.erl
index ad3660e..236e89f 100644
--- a/src/couch_views/src/couch_views_trees.erl
+++ b/src/couch_views/src/couch_views_trees.erl
@@ -229,7 +229,8 @@ open_id_tree(TxDb, Sig) ->
Prefix = id_tree_prefix(DbPrefix, Sig),
TreeOpts = [
{persist_fun, fun couch_views_fdb:persist_chunks/3},
-{cache_fun, create_cache_fun(id_tree)}
+{cache_fun, create_cache_fun(id_tree)},
+{encode_fun, create_encode_fun(TxDb)}
],
ebtree:open(Tx, Prefix, get_order(id_btree), TreeOpts).
@@ -245,7 +246,8 @@ open_view_tree(TxDb, Sig, Lang, View, Options) ->
Prefix = view_tree_prefix(DbPrefix, Sig, ViewId),
BaseOpts = [
{collate_fun, couch_views_util:collate_fun(View)},
-{persist_fun, fun couch_views_fdb:persist_chunks/3}
+{persist_fun, fun couch_views_fdb:persist_chunks/3},
+{encode_fun, create_encode_fun(TxDb)}
],
ExtraOpts = case lists:keyfind(read_only, 1, Options) of
{read_only, Idx} ->
@@ -355,6 +357,17 @@ create_cache_fun(TreeId) ->
end.
+create_encode_fun(TxDb) ->
+fun
+(encode, Key, Term) ->
+Bin = term_to_binary(Term, [compressed, {minor_version, 2}]),
+aegis:encrypt(TxDb, Key, Bin);
+(decode, Key, Ciphertext) ->
+Bin = aegis:decrypt(TxDb, Key, Ciphertext),
+binary_to_term(Bin, [safe])
+end.
+
+
to_map_opts(Options) ->
Dir = case lists:keyfind(dir, 1, Options) of
{dir, D} -> D;
[couchdb] branch feat-aegis-for-views created (now ee335d4)
This is an automated email from the ASF dual-hosted git repository. davisp pushed a change to branch feat-aegis-for-views in repository https://gitbox.apache.org/repos/asf/couchdb.git. at ee335d4 Allow administrators to optionally encrypt views No new revisions were added by this update.
[couchdb] 01/02: Fix empty reduce output to match 3.x behavior
This is an automated email from the ASF dual-hosted git repository.
davisp pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/couchdb.git
commit ea80fe379d3548fd4b0e98827c17190fabcfd03b
Author: Paul J. Davis
AuthorDate: Wed Oct 21 14:42:43 2020 -0500
Fix empty reduce output to match 3.x behavior
Before this change a reduce call that contained no rows would end up
returning the "default" value of the given reduce function which is
whatever it would return when given an empty array as input. This
changes the behavior to return `{"rows": []"}` when there are no
rows in the requested range.
---
src/couch_views/src/couch_views_trees.erl | 25 +
src/couch_views/test/couch_views_red_test.erl | 4 ++--
2 files changed, 7 insertions(+), 22 deletions(-)
diff --git a/src/couch_views/src/couch_views_trees.erl
b/src/couch_views/src/couch_views_trees.erl
index d9340ad..6c32f97 100644
--- a/src/couch_views/src/couch_views_trees.erl
+++ b/src/couch_views/src/couch_views_trees.erl
@@ -144,15 +144,8 @@ fold_red_idx(TxDb, View, Idx, Options, Callback, Acc0) ->
Callback(GroupKey, RedValue, WAcc)
end,
-case {GroupKeyFun, Dir} of
-{group_all, fwd} ->
-EBtreeOpts = [
-{dir, fwd},
-{inclusive_end, InclusiveEnd}
-],
-Reduction = ebtree:reduce(Tx, Btree, StartKey, EndKey, EBtreeOpts),
-Wrapper({null, Reduction}, Acc0);
-{F, fwd} when is_function(F) ->
+case Dir of
+fwd ->
EBtreeOpts = [
{dir, fwd},
{inclusive_end, InclusiveEnd}
@@ -167,16 +160,7 @@ fold_red_idx(TxDb, View, Idx, Options, Callback, Acc0) ->
Acc0,
EBtreeOpts
);
-{group_all, rev} ->
-% Start/End keys swapped on purpose because ebtree. Also
-% inclusive_start for same reason.
-EBtreeOpts = [
-{dir, rev},
-{inclusive_start, InclusiveEnd}
-],
-Reduction = ebtree:reduce(Tx, Btree, EndKey, StartKey, EBtreeOpts),
-Wrapper({null, Reduction}, Acc0);
-{F, rev} when is_function(F) ->
+rev ->
% Start/End keys swapped on purpose because ebtree. Also
% inclusive_start for same reason.
EBtreeOpts = [
@@ -404,8 +388,9 @@ to_red_opts(Options) ->
{Dir, StartKey, EndKey, InclusiveEnd} = to_map_opts(Options),
GroupKeyFun = case lists:keyfind(group_key_fun, 1, Options) of
+{group_key_fun, group_all} -> fun({_Key, _DocId}) -> null end;
{group_key_fun, GKF} -> GKF;
-false -> fun({_Key, _DocId}) -> global_group end
+false -> fun({_Key, _DocId}) -> null end
end,
{Dir, StartKey, EndKey, InclusiveEnd, GroupKeyFun}.
diff --git a/src/couch_views/test/couch_views_red_test.erl
b/src/couch_views/test/couch_views_red_test.erl
index 707611f..84c6473 100644
--- a/src/couch_views/test/couch_views_red_test.erl
+++ b/src/couch_views/test/couch_views_red_test.erl
@@ -213,7 +213,7 @@ should_reduce_empty_range({Db, _}) ->
end_key => 11
},
Result = run_query(Db, <<"baz_count">>, Args),
-Expect = {ok, [row(null, 0)]},
+Expect = {ok, []},
?assertEqual(Expect, Result).
@@ -224,7 +224,7 @@ should_reduce_empty_range_rev({Db, _}) ->
end_key => 10
},
Result = run_query(Db, <<"baz_count">>, Args),
-Expect = {ok, [row(null, 0)]},
+Expect = {ok, []},
?assertEqual(Expect, Result).
[couchdb] branch main updated (2428f0f -> 49399ff)
This is an automated email from the ASF dual-hosted git repository. davisp pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/couchdb.git. from 2428f0f update to latest fauxton 1.2.6 (#3134) new ea80fe3 Fix empty reduce output to match 3.x behavior new 49399ff Remove debug logging The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: src/couch_views/src/couch_views_trees.erl | 28 +-- src/couch_views/test/couch_views_red_test.erl | 4 ++-- 2 files changed, 7 insertions(+), 25 deletions(-)
[couchdb] 02/02: Remove debug logging
This is an automated email from the ASF dual-hosted git repository.
davisp pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/couchdb.git
commit 49399ff4510449793ae3241a09cc546a3f9cb925
Author: Paul J. Davis
AuthorDate: Wed Oct 21 14:44:47 2020 -0500
Remove debug logging
---
src/couch_views/src/couch_views_trees.erl | 3 ---
1 file changed, 3 deletions(-)
diff --git a/src/couch_views/src/couch_views_trees.erl
b/src/couch_views/src/couch_views_trees.erl
index 6c32f97..ad3660e 100644
--- a/src/couch_views/src/couch_views_trees.erl
+++ b/src/couch_views/src/couch_views_trees.erl
@@ -279,9 +279,6 @@ min_order(V) ->
make_read_only_reduce_fun(Lang, View, NthRed) ->
RedFuns = [Src || {_, Src} <- View#mrview.reduce_funs],
-if RedFuns /= [] -> ok; true ->
-io:format(standard_error, "~p~n", [process_info(self(),
current_stacktrace)])
-end,
LPad = lists:duplicate(NthRed - 1, []),
RPad = lists:duplicate(length(RedFuns) - NthRed, []),
FunSrc = lists:nth(NthRed, RedFuns),
[couchdb] branch fix-empty-reduce-output updated (04736c0 -> 9e91891)
This is an automated email from the ASF dual-hosted git repository. davisp pushed a change to branch fix-empty-reduce-output in repository https://gitbox.apache.org/repos/asf/couchdb.git. discard 04736c0 Remove debug logging discard 4d28a6e Fix empty reduce output to match 3.x behavior new 69f1c60 Fix empty reduce output to match 3.x behavior new 9e91891 Remove debug logging This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (04736c0) \ N -- N -- N refs/heads/fix-empty-reduce-output (9e91891) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: src/couch_views/test/couch_views_red_test.erl | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
[couchdb] 02/02: Remove debug logging
This is an automated email from the ASF dual-hosted git repository.
davisp pushed a commit to branch fix-empty-reduce-output
in repository https://gitbox.apache.org/repos/asf/couchdb.git
commit 9e918919a40265f2790946f37df453df4ba01d32
Author: Paul J. Davis
AuthorDate: Wed Oct 21 14:44:47 2020 -0500
Remove debug logging
---
src/couch_views/src/couch_views_trees.erl | 3 ---
1 file changed, 3 deletions(-)
diff --git a/src/couch_views/src/couch_views_trees.erl
b/src/couch_views/src/couch_views_trees.erl
index 6c32f97..ad3660e 100644
--- a/src/couch_views/src/couch_views_trees.erl
+++ b/src/couch_views/src/couch_views_trees.erl
@@ -279,9 +279,6 @@ min_order(V) ->
make_read_only_reduce_fun(Lang, View, NthRed) ->
RedFuns = [Src || {_, Src} <- View#mrview.reduce_funs],
-if RedFuns /= [] -> ok; true ->
-io:format(standard_error, "~p~n", [process_info(self(),
current_stacktrace)])
-end,
LPad = lists:duplicate(NthRed - 1, []),
RPad = lists:duplicate(length(RedFuns) - NthRed, []),
FunSrc = lists:nth(NthRed, RedFuns),
[couchdb] 01/02: Fix empty reduce output to match 3.x behavior
This is an automated email from the ASF dual-hosted git repository.
davisp pushed a commit to branch fix-empty-reduce-output
in repository https://gitbox.apache.org/repos/asf/couchdb.git
commit 69f1c6076fd29a7ef925dbeeb7fef2527ad8b68d
Author: Paul J. Davis
AuthorDate: Wed Oct 21 14:42:43 2020 -0500
Fix empty reduce output to match 3.x behavior
Before this change a reduce call that contained no rows would end up
returning the "default" value of the given reduce function which is
whatever it would return when given an empty array as input. This
changes the behavior to return `{"rows": []"}` when there are no
rows in the requested range.
---
src/couch_views/src/couch_views_trees.erl | 25 +
src/couch_views/test/couch_views_red_test.erl | 4 ++--
2 files changed, 7 insertions(+), 22 deletions(-)
diff --git a/src/couch_views/src/couch_views_trees.erl
b/src/couch_views/src/couch_views_trees.erl
index d9340ad..6c32f97 100644
--- a/src/couch_views/src/couch_views_trees.erl
+++ b/src/couch_views/src/couch_views_trees.erl
@@ -144,15 +144,8 @@ fold_red_idx(TxDb, View, Idx, Options, Callback, Acc0) ->
Callback(GroupKey, RedValue, WAcc)
end,
-case {GroupKeyFun, Dir} of
-{group_all, fwd} ->
-EBtreeOpts = [
-{dir, fwd},
-{inclusive_end, InclusiveEnd}
-],
-Reduction = ebtree:reduce(Tx, Btree, StartKey, EndKey, EBtreeOpts),
-Wrapper({null, Reduction}, Acc0);
-{F, fwd} when is_function(F) ->
+case Dir of
+fwd ->
EBtreeOpts = [
{dir, fwd},
{inclusive_end, InclusiveEnd}
@@ -167,16 +160,7 @@ fold_red_idx(TxDb, View, Idx, Options, Callback, Acc0) ->
Acc0,
EBtreeOpts
);
-{group_all, rev} ->
-% Start/End keys swapped on purpose because ebtree. Also
-% inclusive_start for same reason.
-EBtreeOpts = [
-{dir, rev},
-{inclusive_start, InclusiveEnd}
-],
-Reduction = ebtree:reduce(Tx, Btree, EndKey, StartKey, EBtreeOpts),
-Wrapper({null, Reduction}, Acc0);
-{F, rev} when is_function(F) ->
+rev ->
% Start/End keys swapped on purpose because ebtree. Also
% inclusive_start for same reason.
EBtreeOpts = [
@@ -404,8 +388,9 @@ to_red_opts(Options) ->
{Dir, StartKey, EndKey, InclusiveEnd} = to_map_opts(Options),
GroupKeyFun = case lists:keyfind(group_key_fun, 1, Options) of
+{group_key_fun, group_all} -> fun({_Key, _DocId}) -> null end;
{group_key_fun, GKF} -> GKF;
-false -> fun({_Key, _DocId}) -> global_group end
+false -> fun({_Key, _DocId}) -> null end
end,
{Dir, StartKey, EndKey, InclusiveEnd, GroupKeyFun}.
diff --git a/src/couch_views/test/couch_views_red_test.erl
b/src/couch_views/test/couch_views_red_test.erl
index 707611f..84c6473 100644
--- a/src/couch_views/test/couch_views_red_test.erl
+++ b/src/couch_views/test/couch_views_red_test.erl
@@ -213,7 +213,7 @@ should_reduce_empty_range({Db, _}) ->
end_key => 11
},
Result = run_query(Db, <<"baz_count">>, Args),
-Expect = {ok, [row(null, 0)]},
+Expect = {ok, []},
?assertEqual(Expect, Result).
@@ -224,7 +224,7 @@ should_reduce_empty_range_rev({Db, _}) ->
end_key => 10
},
Result = run_query(Db, <<"baz_count">>, Args),
-Expect = {ok, [row(null, 0)]},
+Expect = {ok, []},
?assertEqual(Expect, Result).
[couchdb] 02/02: Remove debug logging
This is an automated email from the ASF dual-hosted git repository.
davisp pushed a commit to branch fix-empty-reduce-output
in repository https://gitbox.apache.org/repos/asf/couchdb.git
commit 04736c03033eb14b337edc4bcaba78af078a9083
Author: Paul J. Davis
AuthorDate: Wed Oct 21 14:44:47 2020 -0500
Remove debug logging
---
src/couch_views/src/couch_views_trees.erl | 3 ---
1 file changed, 3 deletions(-)
diff --git a/src/couch_views/src/couch_views_trees.erl
b/src/couch_views/src/couch_views_trees.erl
index 6c32f97..ad3660e 100644
--- a/src/couch_views/src/couch_views_trees.erl
+++ b/src/couch_views/src/couch_views_trees.erl
@@ -279,9 +279,6 @@ min_order(V) ->
make_read_only_reduce_fun(Lang, View, NthRed) ->
RedFuns = [Src || {_, Src} <- View#mrview.reduce_funs],
-if RedFuns /= [] -> ok; true ->
-io:format(standard_error, "~p~n", [process_info(self(),
current_stacktrace)])
-end,
LPad = lists:duplicate(NthRed - 1, []),
RPad = lists:duplicate(length(RedFuns) - NthRed, []),
FunSrc = lists:nth(NthRed, RedFuns),
[couchdb] 01/02: Fix empty reduce output to match 3.x behavior
This is an automated email from the ASF dual-hosted git repository.
davisp pushed a commit to branch fix-empty-reduce-output
in repository https://gitbox.apache.org/repos/asf/couchdb.git
commit 4d28a6e9c6c3502c8566265abe761eb1937cb026
Author: Paul J. Davis
AuthorDate: Wed Oct 21 14:42:43 2020 -0500
Fix empty reduce output to match 3.x behavior
Before this change a reduce call that contained no rows would end up
returning the "default" value of the given reduce function which is
whatever it would return when given an empty array as input. This
changes the behavior to return `{"rows": []"}` when there are no
rows in the requested range.
---
src/couch_views/src/couch_views_trees.erl | 25 +
1 file changed, 5 insertions(+), 20 deletions(-)
diff --git a/src/couch_views/src/couch_views_trees.erl
b/src/couch_views/src/couch_views_trees.erl
index d9340ad..6c32f97 100644
--- a/src/couch_views/src/couch_views_trees.erl
+++ b/src/couch_views/src/couch_views_trees.erl
@@ -144,15 +144,8 @@ fold_red_idx(TxDb, View, Idx, Options, Callback, Acc0) ->
Callback(GroupKey, RedValue, WAcc)
end,
-case {GroupKeyFun, Dir} of
-{group_all, fwd} ->
-EBtreeOpts = [
-{dir, fwd},
-{inclusive_end, InclusiveEnd}
-],
-Reduction = ebtree:reduce(Tx, Btree, StartKey, EndKey, EBtreeOpts),
-Wrapper({null, Reduction}, Acc0);
-{F, fwd} when is_function(F) ->
+case Dir of
+fwd ->
EBtreeOpts = [
{dir, fwd},
{inclusive_end, InclusiveEnd}
@@ -167,16 +160,7 @@ fold_red_idx(TxDb, View, Idx, Options, Callback, Acc0) ->
Acc0,
EBtreeOpts
);
-{group_all, rev} ->
-% Start/End keys swapped on purpose because ebtree. Also
-% inclusive_start for same reason.
-EBtreeOpts = [
-{dir, rev},
-{inclusive_start, InclusiveEnd}
-],
-Reduction = ebtree:reduce(Tx, Btree, EndKey, StartKey, EBtreeOpts),
-Wrapper({null, Reduction}, Acc0);
-{F, rev} when is_function(F) ->
+rev ->
% Start/End keys swapped on purpose because ebtree. Also
% inclusive_start for same reason.
EBtreeOpts = [
@@ -404,8 +388,9 @@ to_red_opts(Options) ->
{Dir, StartKey, EndKey, InclusiveEnd} = to_map_opts(Options),
GroupKeyFun = case lists:keyfind(group_key_fun, 1, Options) of
+{group_key_fun, group_all} -> fun({_Key, _DocId}) -> null end;
{group_key_fun, GKF} -> GKF;
-false -> fun({_Key, _DocId}) -> global_group end
+false -> fun({_Key, _DocId}) -> null end
end,
{Dir, StartKey, EndKey, InclusiveEnd, GroupKeyFun}.
[couchdb] branch fix-empty-reduce-output created (now 04736c0)
This is an automated email from the ASF dual-hosted git repository. davisp pushed a change to branch fix-empty-reduce-output in repository https://gitbox.apache.org/repos/asf/couchdb.git. at 04736c0 Remove debug logging This branch includes the following new commits: new 4d28a6e Fix empty reduce output to match 3.x behavior new 04736c0 Remove debug logging The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[couchdb] branch main updated: Allow for disabling custom reduce functions
This is an automated email from the ASF dual-hosted git repository.
davisp pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/couchdb.git
The following commit(s) were added to refs/heads/main by this push:
new bfda163 Allow for disabling custom reduce functions
bfda163 is described below
commit bfda16348f5c61e274ad5087e4a6407fa8306e0d
Author: Paul J. Davis
AuthorDate: Mon Oct 12 14:08:01 2020 -0500
Allow for disabling custom reduce functions
This changes allows administators to disable custom JavaScript reduce
functions. Custom reduce functions are notoriously difficult to write
correctly and most of the time when written correctly would be more
efficiently done using an builtin reduce.
---
rel/overlay/etc/default.ini| 3 +
src/chttpd/src/chttpd.erl | 2 +
src/couch_views/src/couch_views_reader.erl | 10 +-
src/couch_views/src/couch_views_trees.erl | 2 +-
src/couch_views/src/couch_views_util.erl | 34 +++-
.../test/couch_views_custom_red_test.erl | 194 +
6 files changed, 240 insertions(+), 5 deletions(-)
diff --git a/rel/overlay/etc/default.ini b/rel/overlay/etc/default.ini
index 8970be5..ac4e2fa 100644
--- a/rel/overlay/etc/default.ini
+++ b/rel/overlay/etc/default.ini
@@ -323,6 +323,9 @@ iterations = 10 ; iterations for password hashing
; Settings for view indexing
[couch_views]
+; Enable custom reduce functions
+;custom_reduce_enabled = true
+
; Maximum acceptors waiting to accept view indexing jobs
;max_acceptors = 5
;
diff --git a/src/chttpd/src/chttpd.erl b/src/chttpd/src/chttpd.erl
index 1a9b19b..3100694 100644
--- a/src/chttpd/src/chttpd.erl
+++ b/src/chttpd/src/chttpd.erl
@@ -1017,6 +1017,8 @@ error_info(all_workers_died) ->
"request due to overloading or maintenance mode.">>};
error_info(not_implemented) ->
{501, <<"not_implemented">>, <<"this feature is not yet implemented">>};
+error_info({disabled, Reason}) ->
+{501, <<"disabled">>, Reason};
error_info(timeout) ->
{500, <<"timeout">>, <<"The request could not be processed in a reasonable"
" amount of time.">>};
diff --git a/src/couch_views/src/couch_views_reader.erl
b/src/couch_views/src/couch_views_reader.erl
index 3c58627..35ee8a0 100644
--- a/src/couch_views/src/couch_views_reader.erl
+++ b/src/couch_views/src/couch_views_reader.erl
@@ -245,11 +245,19 @@ get_map_view(Lang, Args, ViewName, Views) ->
get_red_view(Lang, Args, ViewName, Views) ->
case couch_mrview_util:extract_view(Lang, Args, ViewName, Views) of
-{red, {Idx, Lang, View}, _} -> {Idx, Lang, View};
+{red, {Idx, Lang, View}, _} -> check_red_enabled({Idx, Lang, View});
_ -> throw({not_found, missing_named_view})
end.
+check_red_enabled({Idx, _Lang, View} = Resp) ->
+case lists:nth(Idx, View#mrview.reduce_funs) of
+{_, disabled} ->
+throw({disabled, <<"Custom reduce functions are disabled.">>});
+_ ->
+Resp
+end.
+
expand_keys_args(#mrargs{keys = undefined} = Args) ->
[Args];
diff --git a/src/couch_views/src/couch_views_trees.erl
b/src/couch_views/src/couch_views_trees.erl
index b45750b..d9340ad 100644
--- a/src/couch_views/src/couch_views_trees.erl
+++ b/src/couch_views/src/couch_views_trees.erl
@@ -323,7 +323,7 @@ make_read_only_reduce_fun(Lang, View, NthRed) ->
make_reduce_fun(Lang, #mrview{} = View) ->
-RedFuns = [Src || {_, Src} <- View#mrview.reduce_funs],
+RedFuns = [Src || {_, Src} <- View#mrview.reduce_funs, Src /= disabled],
fun
(KVs0, _ReReduce = false) ->
KVs1 = expand_dupes(KVs0),
diff --git a/src/couch_views/src/couch_views_util.erl
b/src/couch_views/src/couch_views_util.erl
index 1e3e4be..7040020 100644
--- a/src/couch_views/src/couch_views_util.erl
+++ b/src/couch_views/src/couch_views_util.erl
@@ -65,7 +65,8 @@ ddoc_to_mrst(DbName, #doc{id=Id, body={Fields}}) ->
NumViews = fun({_, View}, N) ->
{View#mrview{id_num = N}, N+1}
end,
-{Views, _} = lists:mapfoldl(NumViews, 0, lists:sort(dict:to_list(BySrc))),
+{Views0, _} = lists:mapfoldl(NumViews, 0, lists:sort(dict:to_list(BySrc))),
+Views1 = maybe_disable_custom_reduce_funs(Views0),
Language = couch_util:get_value(<<"language">>, Fields, <<"javascript">>),
Lib = couch_util:get_value(<<"lib">>, RawViews, {[]}),
@@ -74,12 +75,12 @@ ddoc_to_mrst(DbName, #doc{id=Id, body={Fields}}) ->
db_name=DbName,
idx_name=Id,
lib=Lib,
-views=Views,
+views=Views1,
language=Language,
desig
[couchdb] branch feat-disable-custom-reduce-functions updated (be2f732 -> dc70576)
This is an automated email from the ASF dual-hosted git repository. davisp pushed a change to branch feat-disable-custom-reduce-functions in repository https://gitbox.apache.org/repos/asf/couchdb.git. discard be2f732 Disable custom reduce functions by default add dc70576 Allow for disabling custom reduce functions This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (be2f732) \ N -- N -- N refs/heads/feat-disable-custom-reduce-functions (dc70576) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: rel/overlay/etc/default.ini | 2 +- src/couch_views/src/couch_views_util.erl | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
[couchdb] branch feat-disable-custom-reduce-functions updated (403bec9 -> be2f732)
This is an automated email from the ASF dual-hosted git repository. davisp pushed a change to branch feat-disable-custom-reduce-functions in repository https://gitbox.apache.org/repos/asf/couchdb.git. discard 403bec9 Disable custom reduce functions by default add be2f732 Disable custom reduce functions by default This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (403bec9) \ N -- N -- N refs/heads/feat-disable-custom-reduce-functions (be2f732) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: src/couch_views/test/couch_views_custom_red_test.erl | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[couchdb] branch feat-disable-custom-reduce-functions updated (403bec9 -> be2f732)
This is an automated email from the ASF dual-hosted git repository. davisp pushed a change to branch feat-disable-custom-reduce-functions in repository https://gitbox.apache.org/repos/asf/couchdb.git. discard 403bec9 Disable custom reduce functions by default add be2f732 Disable custom reduce functions by default This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (403bec9) \ N -- N -- N refs/heads/feat-disable-custom-reduce-functions (be2f732) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: src/couch_views/test/couch_views_custom_red_test.erl | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[couchdb] branch feat-disable-custom-reduce-functions updated (9345a71 -> 403bec9)
This is an automated email from the ASF dual-hosted git repository. davisp pushed a change to branch feat-disable-custom-reduce-functions in repository https://gitbox.apache.org/repos/asf/couchdb.git. discard 9345a71 Disable custom reduce functions by default add 403bec9 Disable custom reduce functions by default This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (9345a71) \ N -- N -- N refs/heads/feat-disable-custom-reduce-functions (403bec9) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: src/couch_views/src/couch_views_util.erl | 8 1 file changed, 4 insertions(+), 4 deletions(-)
[couchdb] branch feat-disable-custom-reduce-functions updated (9345a71 -> 403bec9)
This is an automated email from the ASF dual-hosted git repository. davisp pushed a change to branch feat-disable-custom-reduce-functions in repository https://gitbox.apache.org/repos/asf/couchdb.git. discard 9345a71 Disable custom reduce functions by default add 403bec9 Disable custom reduce functions by default This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (9345a71) \ N -- N -- N refs/heads/feat-disable-custom-reduce-functions (403bec9) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: src/couch_views/src/couch_views_util.erl | 8 1 file changed, 4 insertions(+), 4 deletions(-)
[couchdb] 01/01: Disable custom reduce functions by default
This is an automated email from the ASF dual-hosted git repository.
davisp pushed a commit to branch feat-disable-custom-reduce-functions
in repository https://gitbox.apache.org/repos/asf/couchdb.git
commit 9345a71c59e3c9e4b27cc37b8979a097031c6b8c
Author: Paul J. Davis
AuthorDate: Mon Oct 12 14:08:01 2020 -0500
Disable custom reduce functions by default
This prevents users from creating custom reduce functions. Custom reduce
functions are notoriously difficult to write correctly. This change
disables them by default in the eventual 4.0 release.
---
rel/overlay/etc/default.ini| 3 +
src/chttpd/src/chttpd.erl | 2 +
src/couch_views/src/couch_views_reader.erl | 10 +-
src/couch_views/src/couch_views_trees.erl | 2 +-
src/couch_views/src/couch_views_util.erl | 34 +++-
.../test/couch_views_custom_red_test.erl | 194 +
6 files changed, 240 insertions(+), 5 deletions(-)
diff --git a/rel/overlay/etc/default.ini b/rel/overlay/etc/default.ini
index 3a377c7..745a1c9 100644
--- a/rel/overlay/etc/default.ini
+++ b/rel/overlay/etc/default.ini
@@ -325,6 +325,9 @@ iterations = 10 ; iterations for password hashing
; Settings for view indexing
[couch_views]
+; Enable custom reduce functions
+;custom_reduce_enabled = false
+
; Maximum acceptors waiting to accept view indexing jobs
;max_acceptors = 5
;
diff --git a/src/chttpd/src/chttpd.erl b/src/chttpd/src/chttpd.erl
index 1a9b19b..3100694 100644
--- a/src/chttpd/src/chttpd.erl
+++ b/src/chttpd/src/chttpd.erl
@@ -1017,6 +1017,8 @@ error_info(all_workers_died) ->
"request due to overloading or maintenance mode.">>};
error_info(not_implemented) ->
{501, <<"not_implemented">>, <<"this feature is not yet implemented">>};
+error_info({disabled, Reason}) ->
+{501, <<"disabled">>, Reason};
error_info(timeout) ->
{500, <<"timeout">>, <<"The request could not be processed in a reasonable"
" amount of time.">>};
diff --git a/src/couch_views/src/couch_views_reader.erl
b/src/couch_views/src/couch_views_reader.erl
index 3c58627..35ee8a0 100644
--- a/src/couch_views/src/couch_views_reader.erl
+++ b/src/couch_views/src/couch_views_reader.erl
@@ -245,11 +245,19 @@ get_map_view(Lang, Args, ViewName, Views) ->
get_red_view(Lang, Args, ViewName, Views) ->
case couch_mrview_util:extract_view(Lang, Args, ViewName, Views) of
-{red, {Idx, Lang, View}, _} -> {Idx, Lang, View};
+{red, {Idx, Lang, View}, _} -> check_red_enabled({Idx, Lang, View});
_ -> throw({not_found, missing_named_view})
end.
+check_red_enabled({Idx, _Lang, View} = Resp) ->
+case lists:nth(Idx, View#mrview.reduce_funs) of
+{_, disabled} ->
+throw({disabled, <<"Custom reduce functions are disabled.">>});
+_ ->
+Resp
+end.
+
expand_keys_args(#mrargs{keys = undefined} = Args) ->
[Args];
diff --git a/src/couch_views/src/couch_views_trees.erl
b/src/couch_views/src/couch_views_trees.erl
index b45750b..d9340ad 100644
--- a/src/couch_views/src/couch_views_trees.erl
+++ b/src/couch_views/src/couch_views_trees.erl
@@ -323,7 +323,7 @@ make_read_only_reduce_fun(Lang, View, NthRed) ->
make_reduce_fun(Lang, #mrview{} = View) ->
-RedFuns = [Src || {_, Src} <- View#mrview.reduce_funs],
+RedFuns = [Src || {_, Src} <- View#mrview.reduce_funs, Src /= disabled],
fun
(KVs0, _ReReduce = false) ->
KVs1 = expand_dupes(KVs0),
diff --git a/src/couch_views/src/couch_views_util.erl
b/src/couch_views/src/couch_views_util.erl
index 1e3e4be..c4130f4 100644
--- a/src/couch_views/src/couch_views_util.erl
+++ b/src/couch_views/src/couch_views_util.erl
@@ -65,7 +65,8 @@ ddoc_to_mrst(DbName, #doc{id=Id, body={Fields}}) ->
NumViews = fun({_, View}, N) ->
{View#mrview{id_num = N}, N+1}
end,
-{Views, _} = lists:mapfoldl(NumViews, 0, lists:sort(dict:to_list(BySrc))),
+{Views0, _} = lists:mapfoldl(NumViews, 0, lists:sort(dict:to_list(BySrc))),
+Views1 = maybe_filter_custom_reduce_funs(Views0),
Language = couch_util:get_value(<<"language">>, Fields, <<"javascript">>),
Lib = couch_util:get_value(<<"lib">>, RawViews, {[]}),
@@ -74,12 +75,12 @@ ddoc_to_mrst(DbName, #doc{id=Id, body={Fields}}) ->
db_name=DbName,
idx_name=Id,
lib=Lib,
-views=Views,
+views=Views1,
language=Language,
design_opts=DesignOpts,
partitioned=Partitioned
},
-SigInfo = {Views, Language, DesignOpts, couch_index_util:sort_lib(Lib)},
+SigInfo = {Views1, Language, DesignOpt
[couchdb] branch feat-disable-custom-reduce-functions created (now 9345a71)
This is an automated email from the ASF dual-hosted git repository. davisp pushed a change to branch feat-disable-custom-reduce-functions in repository https://gitbox.apache.org/repos/asf/couchdb.git. at 9345a71 Disable custom reduce functions by default This branch includes the following new commits: new 9345a71 Disable custom reduce functions by default The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[couchdb] 03/08: Reimplement db wide view size tracking
This is an automated email from the ASF dual-hosted git repository.
davisp pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/couchdb.git
commit a8cebfa6d3ddbfb6419c83ca7b3b607da330de1a
Author: Paul J. Davis
AuthorDate: Thu Sep 3 12:05:45 2020 -0500
Reimplement db wide view size tracking
---
src/couch_views/src/couch_views_fdb.erl| 30 +-
src/couch_views/src/couch_views_trees.erl | 27 +-
src/couch_views/test/couch_views_size_test.erl | 829 +
3 files changed, 357 insertions(+), 529 deletions(-)
diff --git a/src/couch_views/src/couch_views_fdb.erl
b/src/couch_views/src/couch_views_fdb.erl
index e813f2b..f238a8f 100644
--- a/src/couch_views/src/couch_views_fdb.erl
+++ b/src/couch_views/src/couch_views_fdb.erl
@@ -25,7 +25,8 @@
list_signatures/1,
clear_index/2,
-persist_chunks/3
+persist_chunks/3,
+update_kv_size/4
]).
-ifdef(TEST).
@@ -135,6 +136,11 @@ clear_index(Db, Signature) ->
db_prefix := DbPrefix
} = Db,
+% Get view size to remove from global counter
+SizeTuple = {?DB_VIEWS, ?VIEW_INFO, ?VIEW_KV_SIZE, Signature},
+SizeKey = erlfdb_tuple:pack(SizeTuple, DbPrefix),
+ViewSize = ?bin2uint(erlfdb:wait(erlfdb:get(Tx, SizeKey))),
+
% Clear index info keys
Keys = [
{?DB_VIEWS, ?VIEW_INFO, ?VIEW_UPDATE_SEQ, Signature},
@@ -154,7 +160,12 @@ clear_index(Db, Signature) ->
% Clear tree data
TreeTuple = {?DB_VIEWS, ?VIEW_TREES, Signature},
TreePrefix = erlfdb_tuple:pack(TreeTuple, DbPrefix),
-erlfdb:clear_range_startswith(Tx, TreePrefix).
+erlfdb:clear_range_startswith(Tx, TreePrefix),
+
+% Decrement db wide view size counter
+DbSizeTuple = {?DB_STATS, <<"sizes">>, <<"views">>},
+DbSizeKey = erlfdb_tuple:pack(DbSizeTuple, DbPrefix),
+erlfdb:add(Tx, DbSizeKey, -ViewSize).
persist_chunks(Tx, set, [Key, Value]) ->
@@ -181,6 +192,21 @@ persist_chunks(Tx, clear, Key) ->
erlfdb:clear_range_startswith(Tx, Key).
+update_kv_size(TxDb, Sig, OldSize, NewSize) ->
+#{
+tx := Tx,
+db_prefix := DbPrefix
+} = TxDb,
+
+ViewTuple = {?DB_VIEWS, ?VIEW_INFO, ?VIEW_KV_SIZE, Sig},
+ViewKey = erlfdb_tuple:pack(ViewTuple, DbPrefix),
+erlfdb:set(Tx, ViewKey, ?uint2bin(NewSize)),
+
+DbTuple = {?DB_STATS, <<"sizes">>, <<"views">>},
+DbKey = erlfdb_tuple:pack(DbTuple, DbPrefix),
+erlfdb:add(Tx, DbKey, NewSize - OldSize).
+
+
seq_key(DbPrefix, Sig) ->
Key = {?DB_VIEWS, ?VIEW_INFO, ?VIEW_UPDATE_SEQ, Sig},
erlfdb_tuple:pack(Key, DbPrefix).
diff --git a/src/couch_views/src/couch_views_trees.erl
b/src/couch_views/src/couch_views_trees.erl
index 0f680a6..7ce3505 100644
--- a/src/couch_views/src/couch_views_trees.erl
+++ b/src/couch_views/src/couch_views_trees.erl
@@ -127,6 +127,12 @@ update_views(TxDb, Mrst, Docs) ->
tx := Tx
} = TxDb,
+% Get initial KV size
+OldKVSize = lists:foldl(fun(View, SizeAcc) ->
+{_, Size} = ebtree:full_reduce(Tx, View#mrview.btree),
+SizeAcc + Size
+end, 0, Mrst#mrst.views),
+
% Collect update information
#{
ids := IdMap,
@@ -146,7 +152,15 @@ update_views(TxDb, Mrst, Docs) ->
ViewMap = maps:get(ViewId, ViewMaps, #{}),
update_btree(Tx, BTree, ViewMap, DeleteRef)
-end, Mrst#mrst.views).
+end, Mrst#mrst.views),
+
+% Get new KV size after update
+NewKVSize = lists:foldl(fun(View, SizeAcc) ->
+{_, Size} = ebtree:full_reduce(Tx, View#mrview.btree),
+SizeAcc + Size
+end, 0, Mrst#mrst.views),
+
+couch_views_fdb:update_kv_size(TxDb, Mrst#mrst.sig, OldKVSize, NewKVSize).
open_id_tree(TxDb, Sig) ->
@@ -201,15 +215,16 @@ make_reduce_fun(_Lang, #mrview{}) ->
(KVs, _ReReduce = false) ->
TotalSize = lists:foldl(fun({{K, _DocId}, V}, Acc) ->
KSize = couch_ejson_size:encoded_size(K),
-VSize = case V of
+Acc + case V of
{dups, Dups} ->
lists:foldl(fun(D, DAcc) ->
-DAcc + couch_ejson_size:encoded_size(D)
+VSize = couch_ejson_size:encoded_size(D),
+DAcc + KSize + VSize
end, 0, Dups);
_ ->
-couch_ejson_size:encoded_size(V)
-end,
-KSize + VSize + Acc
+VSize = couch_ejson_size:encoded_size(V),
+KSize + VSize
+end
end, 0, KVs),
{length(KVs), TotalSize};
(KRs, _ReReduce = true) ->
diff --git a/src/couch_views/test/couch_views_size_test.erl
b/src/couch_views/test/couch_views_size_test.erl
index cc2fe39..e69b5b2 10064
[couchdb] 02/08: Views on ebtree
This is an automated email from the ASF dual-hosted git repository.
davisp pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/couchdb.git
commit b91f193563c9b3dadd4f8de4c49de9cbf4304837
Author: Paul J. Davis
AuthorDate: Fri Jul 24 10:59:05 2020 -0500
Views on ebtree
---
rel/overlay/etc/default.ini | 6 +
src/couch_views/include/couch_views.hrl | 5 +
src/couch_views/src/couch_views.erl | 55 +--
src/couch_views/src/couch_views_fdb.erl | 331 ++---
src/couch_views/src/couch_views_indexer.erl | 54 +--
src/couch_views/src/couch_views_reader.erl| 115 +++---
src/couch_views/src/couch_views_trees.erl | 429 ++
src/couch_views/src/couch_views_updater.erl | 13 +-
src/couch_views/src/couch_views_util.erl | 35 ++
src/couch_views/test/couch_views_cleanup_test.erl | 2 +-
src/couch_views/test/couch_views_indexer_test.erl | 64 ++--
src/couch_views/test/couch_views_size_test.erl| 25 +-
src/couch_views/test/couch_views_updater_test.erl | 4 +-
src/mango/src/mango_cursor_view.erl | 14 +-
src/mango/src/mango_idx_view.erl | 7 +-
src/mango/src/mango_idx_view.hrl | 13 +
16 files changed, 687 insertions(+), 485 deletions(-)
diff --git a/rel/overlay/etc/default.ini b/rel/overlay/etc/default.ini
index abcf0bd..3a377c7 100644
--- a/rel/overlay/etc/default.ini
+++ b/rel/overlay/etc/default.ini
@@ -337,6 +337,12 @@ iterations = 10 ; iterations for password hashing
; The maximum allowed value size emitted from a view for a document (in bytes)
;value_size_limit = 64000
;
+; The maximum size of B+Tree nodes used by the id btree
+;id_btree_node_size = 100
+;
+; The maximum size of B+Tree nodes used by view btrees
+;view_btree_node_size = 100
+;
; Batch size sensing parameters
; batch_initial_size = 100 ; Initial batch size in number of documents
; batch_search_increment = 500 ; Size change when searching for the threshold
diff --git a/src/couch_views/include/couch_views.hrl
b/src/couch_views/include/couch_views.hrl
index 3d0110f..3882191 100644
--- a/src/couch_views/include/couch_views.hrl
+++ b/src/couch_views/include/couch_views.hrl
@@ -13,6 +13,7 @@
% Index info/data subspaces
-define(VIEW_INFO, 0).
-define(VIEW_DATA, 1).
+-define(VIEW_TREES, 3).
% Index info keys
-define(VIEW_UPDATE_SEQ, 0).
@@ -25,6 +26,10 @@
-define(VIEW_ID_RANGE, 0).
-define(VIEW_MAP_RANGE, 1).
+% Tree keys
+-define(VIEW_ID_TREE, 0).
+-define(VIEW_ROW_TREES, 1).
+
% jobs api
-define(INDEX_JOB_TYPE, <<"views">>).
diff --git a/src/couch_views/src/couch_views.erl
b/src/couch_views/src/couch_views.erl
index d9ba0c1..da8a142 100644
--- a/src/couch_views/src/couch_views.erl
+++ b/src/couch_views/src/couch_views.erl
@@ -48,11 +48,7 @@ query(Db, DDoc, ViewName, Callback, Acc0, Args0) ->
Args1 = to_mrargs(Args0),
Args2 = couch_mrview_util:set_view_type(Args1, ViewName, Views),
Args3 = couch_mrview_util:validate_args(Args2),
-ok = check_range(Args3),
-case is_reduce_view(Args3) of
-true -> throw(not_implemented);
-false -> ok
-end,
+ok = check_range(Mrst, ViewName, Args3),
try
fabric2_fdb:transactional(Db, fun(TxDb) ->
@@ -100,9 +96,10 @@ get_info(Db, DDoc) ->
{ok, Mrst} = couch_views_util:ddoc_to_mrst(DbName, DDoc),
Sig = fabric2_util:to_hex(Mrst#mrst.sig),
{UpdateSeq, DataSize, Status} = fabric2_fdb:transactional(Db, fun(TxDb) ->
-Seq = couch_views_fdb:get_update_seq(TxDb, Mrst),
-DataSize = get_total_view_size(TxDb, Mrst),
-JobStatus = case couch_views_jobs:job_state(TxDb, Mrst) of
+Mrst1 = couch_views_trees:open(TxDb, Mrst),
+Seq = couch_views_fdb:get_update_seq(TxDb, Mrst1),
+DataSize = get_total_view_size(TxDb, Mrst1),
+JobStatus = case couch_views_jobs:job_state(TxDb, Mrst1) of
{ok, pending} -> true;
{ok, running} -> true;
{ok, finished} -> false;
@@ -124,10 +121,9 @@ get_info(Db, DDoc) ->
get_total_view_size(TxDb, Mrst) ->
-ViewIds = [View#mrview.id_num || View <- Mrst#mrst.views],
-lists:foldl(fun (ViewId, Total) ->
-Total + couch_views_fdb:get_kv_size(TxDb, Mrst, ViewId)
-end, 0, ViewIds).
+lists:foldl(fun(View, Total) ->
+Total + couch_views_trees:get_kv_size(TxDb, View)
+end, 0, Mrst#mrst.views).
read_view(Db, Mrst, ViewName, Callback, Acc0, Args) ->
@@ -185,16 +181,29 @@ to_mrargs(#{} = Args) ->
end, #mrargs{}, Args).
-check_range(#mrargs{start_key = undefined}) ->
+check_range(Mrst, ViewName, Args) ->
+#mrst{
+language = Lang,
+views = Views
+} = Mrst,
+View = case couch_mrview_util:extract_view(Lang, Args, ViewName, Views) of
+{map, V, _} -> V;
+
[couchdb] 05/08: Use ebtree for reduce functions
This is an automated email from the ASF dual-hosted git repository.
davisp pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/couchdb.git
commit 69f0ba1cc0af0c6008f63fb7342a60efa794634b
Author: Paul J. Davis
AuthorDate: Wed Jul 29 10:34:48 2020 -0500
Use ebtree for reduce functions
---
src/couch_views/src/couch_views.erl| 6 -
src/couch_views/src/couch_views_fdb.erl| 1 -
src/couch_views/src/couch_views_reader.erl | 159 ++-
src/couch_views/src/couch_views_trees.erl | 199 +
4 files changed, 327 insertions(+), 38 deletions(-)
diff --git a/src/couch_views/src/couch_views.erl
b/src/couch_views/src/couch_views.erl
index da8a142..2d91631 100644
--- a/src/couch_views/src/couch_views.erl
+++ b/src/couch_views/src/couch_views.erl
@@ -161,12 +161,6 @@ maybe_update_view(TxDb, Mrst, false, _Args) ->
end.
-is_reduce_view(#mrargs{view_type = ViewType}) ->
-ViewType =:= red;
-is_reduce_view({Reduce, _, _}) ->
-Reduce =:= red.
-
-
to_mrargs(#mrargs{} = Args) ->
Args;
diff --git a/src/couch_views/src/couch_views_fdb.erl
b/src/couch_views/src/couch_views_fdb.erl
index 28a60b8..b0fb82e 100644
--- a/src/couch_views/src/couch_views_fdb.erl
+++ b/src/couch_views/src/couch_views_fdb.erl
@@ -299,7 +299,6 @@ reset_interactive_index(Db, Sig, _St) ->
{VS, ?INDEX_BUILDING}.
-
version_key(Db, Sig) ->
#{
db_prefix := DbPrefix
diff --git a/src/couch_views/src/couch_views_reader.erl
b/src/couch_views/src/couch_views_reader.erl
index a785c7b..3c58627 100644
--- a/src/couch_views/src/couch_views_reader.erl
+++ b/src/couch_views/src/couch_views_reader.erl
@@ -23,7 +23,15 @@
-include_lib("fabric/include/fabric2.hrl").
-read(Db, Mrst0, ViewName, UserCallback, UserAcc0, Args) ->
+read(Db, Mrst, ViewName, UserCallback, UserAcc, Args) ->
+ReadFun = case Args of
+#mrargs{view_type = map} -> fun read_map_view/6;
+#mrargs{view_type = red} -> fun read_red_view/6
+end,
+ReadFun(Db, Mrst, ViewName, UserCallback, UserAcc, Args).
+
+
+read_map_view(Db, Mrst0, ViewName, UserCallback, UserAcc0, Args) ->
try
fabric2_fdb:transactional(Db, fun(TxDb) ->
#mrst{
@@ -68,6 +76,79 @@ read(Db, Mrst0, ViewName, UserCallback, UserAcc0, Args) ->
end.
+read_red_view(Db, Mrst0, ViewName, UserCallback, UserAcc0, Args) ->
+#mrst{
+language = Lang,
+views = Views
+} = Mrst0,
+{Idx, Lang, View0} = get_red_view(Lang, Args, ViewName, Views),
+Mrst1 = Mrst0#mrst{views = [View0]},
+ReadOpts = [{read_only, Idx}],
+try
+fabric2_fdb:transactional(Db, fun(TxDb) ->
+#mrst{
+language = Lang,
+views = [View1]
+} = Mrst = couch_views_trees:open(TxDb, Mrst1, ReadOpts),
+
+#mrargs{
+extra = Extra
+} = Args,
+
+Fun = fun handle_red_row/3,
+
+Meta = get_red_meta(TxDb, Mrst, View1, Args),
+UserAcc1 = maybe_stop(UserCallback(Meta, UserAcc0)),
+
+Finalizer = case couch_util:get_value(finalizer, Extra) of
+undefined ->
+{_, FunSrc} = lists:nth(Idx, View1#mrview.reduce_funs),
+FunSrc;
+CustomFun->
+CustomFun
+end,
+
+Acc0 = #{
+db => TxDb,
+skip => Args#mrargs.skip,
+limit => Args#mrargs.limit,
+mrargs => undefined,
+finalizer => Finalizer,
+red_idx => Idx,
+language => Lang,
+callback => UserCallback,
+acc => UserAcc1
+},
+
+Acc1 = lists:foldl(fun(KeyArgs, KeyAcc0) ->
+Opts = mrargs_to_fdb_options(KeyArgs),
+KeyAcc1 = KeyAcc0#{
+mrargs := KeyArgs
+},
+couch_views_trees:fold_red_idx(
+TxDb,
+View1,
+Idx,
+Opts,
+Fun,
+KeyAcc1
+)
+end, Acc0, expand_keys_args(Args)),
+
+#{
+acc := UserAcc2
+} = Acc1,
+{ok, maybe_stop(UserCallback(complete, UserAcc2))}
+end)
+catch
+throw:{complete, Out} ->
+{_, Final} = UserCallback(complete, Out),
+{ok, Final};
+throw:{done, Out} ->
+{ok, Out}
+end.
+
+
get_map_meta(TxDb, Mrst, View, #mrargs{update_seq = true}) ->
TotalRows = couch_views_trees:get_row_count(TxDb, View),
ViewSeq = couch_views_fdb:get_update_seq(TxDb, Mrst),
@@ -78,6 +159,14 @@ get_map_meta(TxDb, _Mrst, View, #mrargs{}
[couchdb] 07/08: Add elixir tests for builtin reduce group levels
This is an automated email from the ASF dual-hosted git repository.
davisp pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/couchdb.git
commit 811741d9b6dce3403b0330d9bdb860518764e1a9
Author: Garren Smith
AuthorDate: Thu Sep 24 12:10:29 2020 -0500
Add elixir tests for builtin reduce group levels
---
.../test/reduce_builtin_group_level_tests.exs | 549 +
1 file changed, 549 insertions(+)
diff --git a/test/elixir/test/reduce_builtin_group_level_tests.exs
b/test/elixir/test/reduce_builtin_group_level_tests.exs
new file mode 100644
index 000..2a18349
--- /dev/null
+++ b/test/elixir/test/reduce_builtin_group_level_tests.exs
@@ -0,0 +1,549 @@
+defmodule ReduceBuiltinGroupLevelTests do
+ use CouchTestCase
+
+ setup do
+db_name = random_db_name()
+{:ok, _} = create_db(db_name)
+on_exit(fn -> delete_db(db_name) end)
+
+docs = create_docs()
+ddoc = create_ddoc()
+
+body = %{
+ docs: [ddoc | docs]
+}
+
+resp = Couch.post("/#{db_name}/_bulk_docs", body: body)
+Enum.each(resp.body, (&1["ok"]))
+
+%{
+ :db_name => db_name,
+ :ddoc => ddoc
+}
+ end
+
+ test "group_level=0 reduce startkey/endkey", context do
+args = %{
+ reduce: true,
+ group_level: 0,
+ start_key: [2018, 3, 2],
+ end_key: [2019, 5, 1]
+}
+
+correct = [
+ %{"key" => :null, "value" => 31}
+]
+
+run_query(context, args, "dates_sum", correct)
+ end
+
+ test "group_level=0 reduce", context do
+args = %{
+ reduce: true,
+ group_level: 0
+}
+
+correct = [
+ %{"key" => :null, "value" => 68}
+]
+
+run_query(context, args, "dates_sum", correct)
+ end
+
+ test "group_level=1 reduce", context do
+args = %{
+ reduce: true,
+ group_level: 1
+}
+
+correct = [
+ %{"key" => [2017], "value" => 31},
+ %{"key" => [2018], "value" => 20},
+ %{"key" => [2019], "value" => 17}
+]
+
+run_query(context, args, "dates_sum", correct)
+ end
+
+ test "group_level=1 reduce with startkey/endkey", context do
+args = %{
+ reduce: true,
+ group_level: 1,
+ start_key: [2017, 4, 1],
+ end_key: [2018, 3, 1]
+}
+
+correct = [
+ %{"key" => [2017], "value" => 22},
+ %{"key" => [2018], "value" => 6}
+]
+
+run_query(context, args, "dates_sum", correct)
+ end
+
+ test "group_level=1 reduce with startkey/endkey take 2", context do
+args = %{
+ reduce: true,
+ group_level: 1,
+ start_key: [2017, 4, 1],
+ end_key: [2019, 3, 2]
+}
+
+correct = [
+ %{"key" => [2017], "value" => 22},
+ %{"key" => [2018], "value" => 20},
+ %{"key" => [2019], "value" => 4}
+]
+
+run_query(context, args, "dates_sum", correct)
+ end
+
+ test "group_level=1 reduce with startkey/endkey take 3", context do
+args = %{
+ reduce: true,
+ group_level: 1,
+ start_key: [2017, 4, 1],
+ end_key: [2019, 05, 1]
+}
+
+correct = [
+ %{"key" => [2017], "value" => 22},
+ %{"key" => [2018], "value" => 20},
+ %{"key" => [2019], "value" => 17}
+]
+
+run_query(context, args, "dates_sum", correct)
+ end
+
+ test "group_level=1 reduce with startkey", context do
+args = %{
+ reduce: true,
+ group_level: 1,
+ start_key: [2017, 4, 1]
+}
+
+correct = [
+ %{"key" => [2017], "value" => 22},
+ %{"key" => [2018], "value" => 20},
+ %{"key" => [2019], "value" => 17}
+]
+
+run_query(context, args, "dates_sum", correct)
+ end
+
+ test "group_level=1 reduce with endkey", context do
+args = %{
+ reduce: true,
+ group_level: 1,
+ end_key: [2018, 5, 2]
+}
+
+correct = [
+ %{"key" => [2017], "value" => 31},
+ %{"key" => [2018], "value" => 20}
+]
+
+run_query(context, args, "dates_sum", correct)
+ end
+
+ test "group=true reduce with startkey/endkey", context do
+args = %{
+ reduce: true,
+ group: true,
+ start_key: [2018, 5, 1],
+ end_key: [2019, 04, 1]
+}
+
+correct = [
+ %{"key" => [2018, 5, 1], "value" => 7},
+ %{"key" => [2019, 3, 1], "value" => 4},
+ %{"key" => [20
[couchdb] branch main updated (f190677 -> 4a4515e)
This is an automated email from the ASF dual-hosted git repository.
davisp pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/couchdb.git.
from f190677 Workaround dirty schedulers in run_queue stats (#3168)
new ddae72a Export fabric2_fdb:chunkify_binary/1,2
new b91f193 Views on ebtree
new a8cebfa Reimplement db wide view size tracking
new ef6b60b Upgrade legacy views
new 69f0ba1 Use ebtree for reduce functions
new 2f1a456 Add test suite for reduce views
new 811741d Add elixir tests for builtin reduce group levels
new 4a4515e Optimizations for reading reduce views
The 8 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
rel/overlay/etc/default.ini| 6 +
src/couch/src/couch_query_servers.erl | 7 +-
src/couch_views/include/couch_views.hrl| 9 +
src/couch_views/src/couch_views.erl| 61 +-
src/couch_views/src/couch_views_fdb.erl| 508 +
src/couch_views/src/couch_views_indexer.erl| 60 +-
src/couch_views/src/couch_views_reader.erl | 254 +--
src/couch_views/src/couch_views_trees.erl | 587 +++
src/couch_views/src/couch_views_updater.erl| 13 +-
src/couch_views/src/couch_views_util.erl | 35 +
src/couch_views/test/couch_views_cleanup_test.erl | 2 +-
src/couch_views/test/couch_views_indexer_test.erl | 64 +-
src/couch_views/test/couch_views_red_test.erl | 745 +++
src/couch_views/test/couch_views_size_test.erl | 824 -
src/couch_views/test/couch_views_updater_test.erl | 4 +-
src/couch_views/test/couch_views_upgrade_test.erl | 400 ++
src/fabric/src/fabric2_fdb.erl | 33 +-
src/mango/src/mango_cursor_view.erl| 14 +-
src/mango/src/mango_idx_view.erl | 7 +-
src/mango/src/{mango.hrl => mango_idx_view.hrl}| 2 +-
.../test/reduce_builtin_group_level_tests.exs | 549 ++
21 files changed, 3151 insertions(+), 1033 deletions(-)
create mode 100644 src/couch_views/src/couch_views_trees.erl
create mode 100644 src/couch_views/test/couch_views_red_test.erl
create mode 100644 src/couch_views/test/couch_views_upgrade_test.erl
copy src/mango/src/{mango.hrl => mango_idx_view.hrl} (90%)
create mode 100644 test/elixir/test/reduce_builtin_group_level_tests.exs
[couchdb] 04/08: Upgrade legacy views
This is an automated email from the ASF dual-hosted git repository.
davisp pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/couchdb.git
commit ef6b60b760a79542df31df9c7ad64d737b860c92
Author: Paul J. Davis
AuthorDate: Fri Sep 18 11:05:00 2020 -0500
Upgrade legacy views
---
src/couch_views/include/couch_views.hrl | 4 +
src/couch_views/src/couch_views_fdb.erl | 180 --
src/couch_views/src/couch_views_indexer.erl | 8 +-
src/couch_views/test/couch_views_upgrade_test.erl | 400 ++
4 files changed, 551 insertions(+), 41 deletions(-)
diff --git a/src/couch_views/include/couch_views.hrl
b/src/couch_views/include/couch_views.hrl
index 3882191..92b8f46 100644
--- a/src/couch_views/include/couch_views.hrl
+++ b/src/couch_views/include/couch_views.hrl
@@ -10,6 +10,9 @@
% License for the specific language governing permissions and limitations under
% the License.
+% Current implementation version
+-define(CURRENT_VIEW_IMPL_VERSION, 1).
+
% Index info/data subspaces
-define(VIEW_INFO, 0).
-define(VIEW_DATA, 1).
@@ -21,6 +24,7 @@
-define(VIEW_KV_SIZE, 2).
-define(VIEW_BUILD_STATUS, 3).
-define(VIEW_CREATION_VS, 4).
+-define(VIEW_IMPL_VERSION, 5).
% Data keys
-define(VIEW_ID_RANGE, 0).
diff --git a/src/couch_views/src/couch_views_fdb.erl
b/src/couch_views/src/couch_views_fdb.erl
index f238a8f..28a60b8 100644
--- a/src/couch_views/src/couch_views_fdb.erl
+++ b/src/couch_views/src/couch_views_fdb.erl
@@ -13,6 +13,8 @@
-module(couch_views_fdb).
-export([
+get_view_state/2,
+
new_interactive_index/3,
new_creation_vs/3,
get_creation_vs/2,
@@ -40,52 +42,89 @@
-include_lib("fabric/include/fabric2.hrl").
-new_interactive_index(Db, Mrst, VS) ->
-couch_views_fdb:new_creation_vs(Db, Mrst, VS),
-couch_views_fdb:set_build_status(Db, Mrst, ?INDEX_BUILDING).
+get_view_state(Db, #mrst{} = Mrst) ->
+get_view_state(Db, Mrst#mrst.sig);
+
+get_view_state(Db, Sig) when is_binary(Sig) ->
+#{
+tx := Tx
+} = Db,
+
+VersionF = erlfdb:get(Tx, version_key(Db, Sig)),
+ViewSeqF = erlfdb:get(Tx, seq_key(Db, Sig)),
+ViewVSF = erlfdb:get(Tx, creation_vs_key(Db, Sig)),
+BuildStatusF = erlfdb:get(Tx, build_status_key(Db, Sig)),
+
+Version = case erlfdb:wait(VersionF) of
+not_found -> not_found;
+VsnVal -> element(1, erlfdb_tuple:unpack(VsnVal))
+end,
+
+ViewSeq = case erlfdb:wait(ViewSeqF) of
+not_found -> <<>>;
+SeqVal -> SeqVal
+end,
+
+ViewVS = case erlfdb:wait(ViewVSF) of
+not_found -> not_found;
+VSVal -> element(1, erlfdb_tuple:unpack(VSVal))
+end,
+
+State = #{
+version => Version,
+view_seq => ViewSeq,
+view_vs => ViewVS,
+build_status => erlfdb:wait(BuildStatusF)
+},
+
+maybe_upgrade_view(Db, Sig, State).
+
+
+new_interactive_index(Db, #mrst{} = Mrst, VS) ->
+new_interactive_index(Db, Mrst#mrst.sig, VS);
+
+new_interactive_index(Db, Sig, VS) ->
+set_version(Db, Sig),
+new_creation_vs(Db, Sig, VS),
+set_build_status(Db, Sig, ?INDEX_BUILDING).
%Interactive View Creation Versionstamp
%(, ?DB_VIEWS, ?VIEW_INFO, ?VIEW_CREATION_VS, Sig) = VS
new_creation_vs(TxDb, #mrst{} = Mrst, VS) ->
+new_creation_vs(TxDb, Mrst#mrst.sig, VS);
+
+new_creation_vs(TxDb, Sig, VS) ->
#{
tx := Tx
} = TxDb,
-Key = creation_vs_key(TxDb, Mrst#mrst.sig),
+Key = creation_vs_key(TxDb, Sig),
Value = erlfdb_tuple:pack_vs({VS}),
ok = erlfdb:set_versionstamped_value(Tx, Key, Value).
-get_creation_vs(TxDb, #mrst{} = Mrst) ->
-get_creation_vs(TxDb, Mrst#mrst.sig);
-
-get_creation_vs(TxDb, Sig) ->
+get_creation_vs(TxDb, MrstOrSig) ->
#{
-tx := Tx
-} = TxDb,
-Key = creation_vs_key(TxDb, Sig),
-case erlfdb:wait(erlfdb:get(Tx, Key)) of
-not_found ->
-not_found;
-EK ->
-{VS} = erlfdb_tuple:unpack(EK),
-VS
-end.
+view_vs := ViewVS
+} = get_view_state(TxDb, MrstOrSig),
+ViewVS.
%Interactive View Build Status
%(, ?DB_VIEWS, ?VIEW_INFO, ?VIEW_BUILD_STATUS, Sig) = INDEX_BUILDING |
INDEX_READY
-get_build_status(TxDb, #mrst{sig = Sig}) ->
+get_build_status(TxDb, MrstOrSig) ->
#{
-tx := Tx
-} = TxDb,
-Key = build_status_key(TxDb, Sig),
-erlfdb:wait(erlfdb:get(Tx, Key)).
+build_status := BuildStatus
+} = get_view_state(TxDb, MrstOrSig),
+BuildStatus.
-set_build_status(TxDb, #mrst{sig = Sig}, State) ->
+set_build_status(TxDb, #mrst{} = Mrst, State) ->
+set_build_status(TxDb, Mrst#mrst.sig, State);
+
+set_build_status(TxDb, Sig, State) ->
#{
tx := Tx
} = TxDb,
@@ -98,24 +137,18 @@ set_build_status(TxDb, #mrst{sig = Sig}, State) ->
% (, ?DB_VI
[couchdb] 01/08: Export fabric2_fdb:chunkify_binary/1,2
This is an automated email from the ASF dual-hosted git repository.
davisp pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/couchdb.git
commit ddae72a707ce51c229da9188275f18664302cba7
Author: Paul J. Davis
AuthorDate: Thu Aug 6 12:34:29 2020 -0500
Export fabric2_fdb:chunkify_binary/1,2
---
src/fabric/src/fabric2_fdb.erl | 33 ++---
1 file changed, 18 insertions(+), 15 deletions(-)
diff --git a/src/fabric/src/fabric2_fdb.erl b/src/fabric/src/fabric2_fdb.erl
index 52303ce..36fa451 100644
--- a/src/fabric/src/fabric2_fdb.erl
+++ b/src/fabric/src/fabric2_fdb.erl
@@ -77,6 +77,9 @@
get_approximate_tx_size/1,
+chunkify_binary/1,
+chunkify_binary/2,
+
debug_cluster/0,
debug_cluster/2
]).
@@ -1176,6 +1179,21 @@ get_approximate_tx_size(#{} = TxDb) ->
erlfdb:wait(erlfdb:get_approximate_size(Tx)).
+chunkify_binary(Data) ->
+chunkify_binary(Data, binary_chunk_size()).
+
+
+chunkify_binary(Data, Size) ->
+case Data of
+<<>> ->
+[];
+<> ->
+[Head | chunkify_binary(Rest, Size)];
+<<_/binary>> when size(Data) < Size ->
+[Data]
+end.
+
+
debug_cluster() ->
debug_cluster(<<>>, <<16#FE, 16#FF, 16#FF>>).
@@ -1677,21 +1695,6 @@ sum_rem_rev_sizes(RevInfos) ->
end, 0, RevInfos).
-chunkify_binary(Data) ->
-chunkify_data(Data, binary_chunk_size()).
-
-
-chunkify_data(Data, Size) ->
-case Data of
-<<>> ->
-[];
-<> ->
-[Head | chunkify_data(Rest, Size)];
-<<_/binary>> when size(Data) < Size ->
-[Data]
-end.
-
-
get_fold_acc(Db, RangePrefix, UserCallback, UserAcc, Options)
when is_map(Db) orelse Db =:= undefined ->
[couchdb] 08/08: Optimizations for reading reduce views
This is an automated email from the ASF dual-hosted git repository.
davisp pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/couchdb.git
commit 4a4515e01d82a6f88ecd3a6b61b5beeab6127424
Author: Paul J. Davis
AuthorDate: Tue Sep 29 15:23:32 2020 -0500
Optimizations for reading reduce views
These are a few micro optimizations to avoid unnecessary work when
reading from a single reduce function during a view read.
---
src/couch/src/couch_query_servers.erl | 7 ++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/src/couch/src/couch_query_servers.erl
b/src/couch/src/couch_query_servers.erl
index 447daea..8eb07ab 100644
--- a/src/couch/src/couch_query_servers.erl
+++ b/src/couch/src/couch_query_servers.erl
@@ -111,6 +111,8 @@ rereduce(Lang, RedSrcs, ReducedValues) ->
reduce(_Lang, [], _KVs) ->
{ok, []};
+reduce(Lang, [<<"_", _/binary>>] = RedSrcs, KVs) ->
+builtin_reduce(reduce, RedSrcs, KVs, []);
reduce(Lang, RedSrcs, KVs) ->
{OsRedSrcs, BuiltinReds} = lists:partition(fun
(<<"_", _/binary>>) -> false;
@@ -171,7 +173,10 @@ builtin_reduce(_Re, [], _KVs, Acc) ->
{ok, lists:reverse(Acc)};
builtin_reduce(Re, [<<"_sum",_/binary>>|BuiltinReds], KVs, Acc) ->
Sum = builtin_sum_rows(KVs, 0),
-Red = check_sum_overflow(?term_size(KVs), ?term_size(Sum), Sum),
+Red = case is_number(Sum) of
+true -> Sum;
+false -> check_sum_overflow(?term_size(KVs), ?term_size(Sum), Sum)
+end,
builtin_reduce(Re, BuiltinReds, KVs, [Red|Acc]);
builtin_reduce(reduce, [<<"_count",_/binary>>|BuiltinReds], KVs, Acc) ->
Count = length(KVs),
[couchdb] 06/08: Add test suite for reduce views
This is an automated email from the ASF dual-hosted git repository.
davisp pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/couchdb.git
commit 2f1a4562e98a423d3d93dab06682bd1be9b39435
Author: Paul J. Davis
AuthorDate: Wed Aug 5 12:47:37 2020 -0500
Add test suite for reduce views
---
src/couch_views/test/couch_views_red_test.erl | 745 ++
1 file changed, 745 insertions(+)
diff --git a/src/couch_views/test/couch_views_red_test.erl
b/src/couch_views/test/couch_views_red_test.erl
new file mode 100644
index 000..707611f
--- /dev/null
+++ b/src/couch_views/test/couch_views_red_test.erl
@@ -0,0 +1,745 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+% http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_views_red_test).
+
+-include_lib("couch/include/couch_eunit.hrl").
+-include_lib("couch/include/couch_db.hrl").
+-include_lib("fabric/test/fabric2_test.hrl").
+-include("couch_views.hrl").
+
+
+-define(NUM_DOCS, 2000).
+
+
+reduce_views_shraed_db_test_() ->
+{
+"Reduce views",
+{
+setup,
+fun setup_db_with_docs/0,
+fun teardown_db/1,
+with([
+?TDEF(should_reduce),
+?TDEF(should_reduce_rev),
+?TDEF(should_reduce_start_key),
+?TDEF(should_reduce_start_key_rev),
+?TDEF(should_reduce_end_key),
+?TDEF(should_reduce_end_key_rev),
+?TDEF(should_reduce_inclusive_end_false),
+?TDEF(should_reduce_inclusive_end_false_rev),
+?TDEF(should_reduce_start_and_end_key),
+?TDEF(should_reduce_start_and_end_key_rev),
+?TDEF(should_reduce_empty_range),
+?TDEF(should_reduce_empty_range_rev),
+?TDEF(should_reduce_grouped),
+?TDEF(should_reduce_grouped_rev),
+?TDEF(should_reduce_grouped_start_key),
+?TDEF(should_reduce_grouped_start_key_rev),
+?TDEF(should_reduce_grouped_end_key),
+?TDEF(should_reduce_grouped_end_key_rev),
+?TDEF(should_reduce_grouped_inclusive_end_false),
+?TDEF(should_reduce_grouped_inclusive_end_false_rev),
+?TDEF(should_reduce_grouped_start_and_end_key),
+?TDEF(should_reduce_grouped_start_and_end_key_rev),
+?TDEF(should_reduce_grouped_empty_range),
+?TDEF(should_reduce_grouped_empty_range_rev),
+?TDEF(should_reduce_array_keys),
+?TDEF(should_reduce_grouped_array_keys),
+?TDEF(should_reduce_group_1_array_keys),
+?TDEF(should_reduce_group_1_array_keys_start_key),
+?TDEF(should_reduce_group_1_array_keys_start_key_rev),
+?TDEF(should_reduce_group_1_array_keys_end_key),
+?TDEF(should_reduce_group_1_array_keys_end_key_rev),
+?TDEF(should_reduce_group_1_array_keys_inclusive_end_false),
+
?TDEF(should_reduce_group_1_array_keys_inclusive_end_false_rev),
+?TDEF(should_reduce_group_1_array_keys_start_and_end_key),
+?TDEF(should_reduce_group_1_array_keys_start_and_end_key_rev),
+?TDEF(should_reduce_group_1_array_keys_sub_array_select),
+?TDEF(should_reduce_group_1_array_keys_sub_array_select_rev),
+
?TDEF(should_reduce_group_1_array_keys_sub_array_inclusive_end),
+?TDEF(should_reduce_group_1_array_keys_empty_range),
+?TDEF(should_reduce_group_1_array_keys_empty_range_rev)
+])
+}
+}.
+
+
+reduce_views_collation_test_() ->
+{
+"Reduce collation test",
+{
+setup,
+fun setup_db/0,
+fun teardown_db/1,
+with([
+?TDEF(should_collate_group_keys)
+])
+}
+}.
+
+
+setup_db() ->
+Ctx = test_util:start_couch([
+fabric,
+couch_jobs,
+couch_js,
+couch_views
+]),
+{ok, Db} = fabric2_db:create(?tempdb(), [{user_ctx, ?ADMIN_USER}]),
+{Db, Ctx}.
+
+
+setup_db_with_docs() ->
+{Db, Ctx} = setup_db(),
+fabric2_db:update_docs(Db, [create_ddoc()]),
+make_docs(Db, ?NUM_DOCS),
+run_query(Db, <<"baz">>, #{l
[couchdb] 01/01: Use JSON collation
This is an automated email from the ASF dual-hosted git repository.
davisp pushed a commit to branch fdb-btree-reduce-davisp
in repository https://gitbox.apache.org/repos/asf/couchdb.git
commit ecb067ba1a2b32a1d6a5a210d496664adedd55ce
Author: Paul J. Davis
AuthorDate: Tue Sep 29 09:56:19 2020 -0500
Use JSON collation
---
src/couch_views/src/couch_views_reduce_fdb.erl | 7 +++
1 file changed, 7 insertions(+)
diff --git a/src/couch_views/src/couch_views_reduce_fdb.erl
b/src/couch_views/src/couch_views_reduce_fdb.erl
index a5cb905..8ea2061 100644
--- a/src/couch_views/src/couch_views_reduce_fdb.erl
+++ b/src/couch_views/src/couch_views_reduce_fdb.erl
@@ -193,11 +193,18 @@ open_tree(TxDb, Sig, ViewId, ViewReduceFun) ->
TreeOpts = [
{reduce_fun, get_reducer(ViewReduceFun)},
+{collate_fun, fun less_json_rows/2},
{encode_fun, EncodeFun}
],
ebtree:open(Tx, ReduceIdxPrefix, btree_order_size(), TreeOpts).
+less_json_rows({_, _} = A, {_, _} = B) ->
+couch_ejson_compare:less_json_ids(A, B);
+less_json_rows(A, B) ->
+couch_ejson_compare:less_json(A, B).
+
+
delete_keys(Tx, Tree, DocId, Keys) ->
lists:foreach(fun (Key) ->
EK = create_key(Key, DocId),
[couchdb] branch fdb-btree-reduce-davisp created (now ecb067b)
This is an automated email from the ASF dual-hosted git repository. davisp pushed a change to branch fdb-btree-reduce-davisp in repository https://gitbox.apache.org/repos/asf/couchdb.git. at ecb067b Use JSON collation This branch includes the following new commits: new ecb067b Use JSON collation The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[couchdb] branch feature-ebtree-views updated (fc5b4cf -> a57d73b)
This is an automated email from the ASF dual-hosted git repository. davisp pushed a change to branch feature-ebtree-views in repository https://gitbox.apache.org/repos/asf/couchdb.git. discard fc5b4cf Add elixir tests for builtin reduce group levels discard 91c0e20 Add test suite for reduce views discard 4029451 Use ebtree for reduce functions add 99d9951 Use ebtree for reduce functions add 234a295 Add test suite for reduce views add 1460d4f Add elixir tests for builtin reduce group levels add a57d73b Optimizations for reading reduce views This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (fc5b4cf) \ N -- N -- N refs/heads/feature-ebtree-views (a57d73b) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: src/couch/src/couch_query_servers.erl | 7 +++- src/couch_views/src/couch_views_reader.erl | 18 ++ src/couch_views/src/couch_views_trees.erl | 55 ++ 3 files changed, 67 insertions(+), 13 deletions(-)
[couchdb] branch feature-ebtree-views updated (e079a87 -> fc5b4cf)
This is an automated email from the ASF dual-hosted git repository. davisp pushed a change to branch feature-ebtree-views in repository https://gitbox.apache.org/repos/asf/couchdb.git. omit e079a87 Add elixir tests for builtin reduce group levels omit cb28503 Add test suite for reduce views omit 54e9bcf Use ebtree for reduce functions omit d035d8d Upgrade legacy views omit bd9486b Reimplement db wide view size tracking omit 4e32f82 Views on ebtree omit 3b508e0 Export fabric2_fdb:chunkify_binary/1,2 omit fa3c3c8 Calculate external JSON size of a view row add aa21588 Export fabric2_fdb:chunkify_binary/1,2 add 40238da Views on ebtree add f47c066 Reimplement db wide view size tracking add 1f83890 Upgrade legacy views add 4029451 Use ebtree for reduce functions add 91c0e20 Add test suite for reduce views add fc5b4cf Add elixir tests for builtin reduce group levels This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (e079a87) \ N -- N -- N refs/heads/feature-ebtree-views (fc5b4cf) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: src/couch/src/couch_ejson_size.erl | 5 --- src/couch_views/src/couch_views_trees.erl | 2 +- src/couch_views/test/couch_views_size_test.erl | 60 +- 3 files changed, 31 insertions(+), 36 deletions(-)
[couchdb] branch re-enable-most-elixir-tests created (now 2e763ae)
This is an automated email from the ASF dual-hosted git repository. davisp pushed a change to branch re-enable-most-elixir-tests in repository https://gitbox.apache.org/repos/asf/couchdb.git. at 2e763ae Re-enable most elixir tests This branch includes the following new commits: new 7e861d8 Remove quorum tests new 2e763ae Re-enable most elixir tests The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[couchdb] 01/02: Remove quorum tests
This is an automated email from the ASF dual-hosted git repository. davisp pushed a commit to branch re-enable-most-elixir-tests in repository https://gitbox.apache.org/repos/asf/couchdb.git commit 7e861d87fa9af46d153bb5233f66799b55291edf Author: Paul J. Davis AuthorDate: Thu Sep 24 13:36:10 2020 -0500 Remove quorum tests Quorum is no longer a thing now that we rely on FoundationDB. --- Makefile | 18 +- Makefile.win | 16 +- test/elixir/test/cluster_with_quorum_test.exs | 186 - test/elixir/test/cluster_without_quorum_test.exs | 185 .../tests-cluster/with-quorum/attachments.js | 37 .../with-quorum/attachments_delete.js | 33 .../attachments_delete_overridden_quorum.js| 38 - .../with-quorum/attachments_overridden_quorum.js | 42 - .../tests-cluster/with-quorum/db_creation.js | 28 .../with-quorum/db_creation_overridden_quorum.js | 30 .../tests-cluster/with-quorum/db_deletion.js | 31 .../with-quorum/db_deletion_overridden_quorum.js | 24 --- .../tests-cluster/with-quorum/doc_bulk.js | 26 --- .../with-quorum/doc_bulk_overridden_quorum.js | 26 --- .../tests-cluster/with-quorum/doc_copy.js | 28 .../with-quorum/doc_copy_overridden_quorum.js | 32 .../tests-cluster/with-quorum/doc_crud.js | 32 .../with-quorum/doc_crud_overridden_quorum.js | 32 .../tests-cluster/without-quorum/attachments.js| 40 - .../without-quorum/attachments_delete.js | 39 - .../attachments_delete_overridden_quorum.js| 38 - .../attachments_overridden_quorum.js | 43 - .../tests-cluster/without-quorum/db_creation.js| 28 .../db_creation_overridden_quorum.js | 32 .../tests-cluster/without-quorum/db_deletion.js| 31 .../db_deletion_overridden_quorum.js | 26 --- .../tests-cluster/without-quorum/doc_bulk.js | 29 .../without-quorum/doc_bulk_overridden_quorum.js | 29 .../tests-cluster/without-quorum/doc_copy.js | 31 .../without-quorum/doc_copy_overridden_quorum.js | 35 .../tests-cluster/without-quorum/doc_crud.js | 36 .../without-quorum/doc_crud_overridden_quorum.js | 35 32 files changed, 3 insertions(+), 1313 deletions(-) diff --git a/Makefile b/Makefile index 5bef9c7..b1d9d71 100644 --- a/Makefile +++ b/Makefile @@ -244,7 +244,7 @@ elixir: elixir-init elixir-check-formatted elixir-credo devclean --enable-erlang-views \ --locald-config test/elixir/test/config/test-config.ini \ --erlang-config rel/files/eunit.config \ - --no-eval 'mix test --trace --exclude without_quorum_test --exclude with_quorum_test $(EXUNIT_OPTS)' + --no-eval 'mix test --trace $(EXUNIT_OPTS)' .PHONY: elixir-only elixir-only: devclean @@ -254,27 +254,13 @@ elixir-only: devclean --enable-erlang-views \ --locald-config test/elixir/test/config/test-config.ini \ --erlang-config rel/files/eunit.config \ - --no-eval 'mix test --trace --exclude without_quorum_test --exclude with_quorum_test $(EXUNIT_OPTS)' + --no-eval 'mix test --trace $(EXUNIT_OPTS)' .PHONY: elixir-init elixir-init: MIX_ENV=test elixir-init: config.erl @mix local.rebar --force && mix local.hex --force && mix deps.get -.PHONY: elixir-cluster-without-quorum -elixir-cluster-without-quorum: export MIX_ENV=integration -elixir-cluster-without-quorum: elixir-init elixir-check-formatted elixir-credo devclean - @dev/run -n 3 -q -a adm:pass \ - --degrade-cluster 2 \ - --no-eval 'mix test --trace --only without_quorum_test $(EXUNIT_OPTS)' - -.PHONY: elixir-cluster-with-quorum -elixir-cluster-with-quorum: export MIX_ENV=integration -elixir-cluster-with-quorum: elixir-init elixir-check-formatted elixir-credo devclean - @dev/run -n 3 -q -a adm:pass \ - --degrade-cluster 1 \ - --no-eval 'mix test --trace --only with_quorum_test $(EXUNIT_OPTS)' - .PHONY: elixir-check-formatted elixir-check-formatted: elixir-init @mix format --check-formatted diff --git a/Makefile.win b/Makefile.win index edfc1bf..6b254e2 100644 --- a/Makefile.win +++ b/Makefile.win @@ -207,27 +207,13 @@ elixir: export COUCHDB_TEST_ADMIN_PARTY_OVERRIDE=1 elixir: elixir-init elixir-check-formatted elixir-credo devclean @dev\run $(TEST_OPTS) -a adm:pass -n 1 --enable-erlang-views \ --locald-config test/elixir/test/config/test-config.ini \ - --no-eval 'mix test --trace --exclude without_quorum_test --exclude with_quorum_test $(EXUNIT_OPTS)' + --no-eval 'mix test --trace $(EXUNIT_OPTS)' .PH
[couchdb] 02/02: Re-enable most elixir tests
This is an automated email from the ASF dual-hosted git repository. davisp pushed a commit to branch re-enable-most-elixir-tests in repository https://gitbox.apache.org/repos/asf/couchdb.git commit 2e763aea3dcdede5e69cac1a4aa8d3859b6159ed Author: Paul J. Davis AuthorDate: Thu Sep 24 15:06:13 2020 -0500 Re-enable most elixir tests This change tags all tests that are currently failing against main with a `pending` tag that is then excluded. --- Makefile | 4 ++-- Makefile.win | 2 +- test/elixir/test/attachments_multipart_test.exs| 1 + test/elixir/test/attachments_test.exs | 2 ++ test/elixir/test/changes_async_test.exs| 9 + test/elixir/test/changes_test.exs | 1 + test/elixir/test/coffee_test.exs | 1 + test/elixir/test/config_test.exs | 1 + test/elixir/test/cookie_auth_test.exs | 1 + test/elixir/test/design_docs_test.exs | 4 test/elixir/test/design_options_test.exs | 1 + test/elixir/test/design_paths_test.exs | 3 +++ test/elixir/test/erlang_views_test.exs | 3 +++ test/elixir/test/http_test.exs | 2 ++ test/elixir/test/proxyauth_test.exs| 2 ++ test/elixir/test/purge_test.exs| 2 ++ test/elixir/test/reader_acl_test.exs | 5 + test/elixir/test/replicator_db_bad_rep_id_test.exs | 1 + test/elixir/test/replicator_db_by_doc_id_test.exs | 1 + test/elixir/test/rev_stemming_test.exs | 1 + test/elixir/test/rewrite_test.exs | 1 + test/elixir/test/update_documents_test.exs | 9 + test/elixir/test/users_db_test.exs | 1 + test/elixir/test/view_compaction_test.exs | 5 +++-- test/elixir/test/view_errors_test.exs | 3 +++ test/elixir/test/view_include_docs_test.exs| 1 + test/elixir/test/view_offsets_test.exs | 2 ++ test/elixir/test/view_pagination_test.exs | 6 ++ test/elixir/test/view_update_seq_test.exs | 3 +++ 29 files changed, 73 insertions(+), 5 deletions(-) diff --git a/Makefile b/Makefile index b1d9d71..714b9b1 100644 --- a/Makefile +++ b/Makefile @@ -244,7 +244,7 @@ elixir: elixir-init elixir-check-formatted elixir-credo devclean --enable-erlang-views \ --locald-config test/elixir/test/config/test-config.ini \ --erlang-config rel/files/eunit.config \ - --no-eval 'mix test --trace $(EXUNIT_OPTS)' + --no-eval 'mix test --trace --exclude pending $(EXUNIT_OPTS)' .PHONY: elixir-only elixir-only: devclean @@ -254,7 +254,7 @@ elixir-only: devclean --enable-erlang-views \ --locald-config test/elixir/test/config/test-config.ini \ --erlang-config rel/files/eunit.config \ - --no-eval 'mix test --trace $(EXUNIT_OPTS)' + --no-eval 'mix test --trace --exclude pending $(EXUNIT_OPTS)' .PHONY: elixir-init elixir-init: MIX_ENV=test diff --git a/Makefile.win b/Makefile.win index 6b254e2..84a933c 100644 --- a/Makefile.win +++ b/Makefile.win @@ -207,7 +207,7 @@ elixir: export COUCHDB_TEST_ADMIN_PARTY_OVERRIDE=1 elixir: elixir-init elixir-check-formatted elixir-credo devclean @dev\run $(TEST_OPTS) -a adm:pass -n 1 --enable-erlang-views \ --locald-config test/elixir/test/config/test-config.ini \ - --no-eval 'mix test --trace $(EXUNIT_OPTS)' + --no-eval 'mix test --trace --exclude pending $(EXUNIT_OPTS)' .PHONY: elixir-init elixir-init: MIX_ENV=test diff --git a/test/elixir/test/attachments_multipart_test.exs b/test/elixir/test/attachments_multipart_test.exs index f635377..5f387cf 100644 --- a/test/elixir/test/attachments_multipart_test.exs +++ b/test/elixir/test/attachments_multipart_test.exs @@ -9,6 +9,7 @@ defmodule AttachmentMultipartTest do This is a port of the attachments_multipart.js suite """ + @tag :pending # HTTP 500 @tag :with_db test "manages attachments multipart requests successfully", context do db_name = context[:db_name] diff --git a/test/elixir/test/attachments_test.exs b/test/elixir/test/attachments_test.exs index 8e7f7d3..d608847 100644 --- a/test/elixir/test/attachments_test.exs +++ b/test/elixir/test/attachments_test.exs @@ -115,6 +115,7 @@ defmodule AttachmentsTest do assert resp.headers["location"] == nil end + @tag :pending # Wrong Content-Type @tag :with_db test "saves binary", context do db_name = context[:db_name] @@ -346,6 +347,7 @@ defmodule AttachmentsTest do assert resp.status_code == 400 end + @tag :pending # HTTP 500 @tag :with_db test "COUCHDB-809 - stubs should only require the 'stub' field", context do db_name = context[:db_nam
[couchdb] branch feature-ebtree-views updated (726bde4 -> e079a87)
This is an automated email from the ASF dual-hosted git repository. davisp pushed a change to branch feature-ebtree-views in repository https://gitbox.apache.org/repos/asf/couchdb.git. discard 726bde4 Add elixir tests for builtin reduce group levels discard 0f35508 Add test suite for reduce views discard 4d7dc40 Use ebtree for reduce functions discard ec3efaf Upgrade legacy views discard 7368a72 Reimplement db wide view size tracking discard 1f773fb Views on ebtree add 4e32f82 Views on ebtree add bd9486b Reimplement db wide view size tracking add d035d8d Upgrade legacy views add 54e9bcf Use ebtree for reduce functions add cb28503 Add test suite for reduce views add e079a87 Add elixir tests for builtin reduce group levels This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (726bde4) \ N -- N -- N refs/heads/feature-ebtree-views (e079a87) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: src/couch_views/src/couch_views_indexer.erl | 2 -- 1 file changed, 2 deletions(-)
[couchdb] branch feature-ebtree-views updated (f712e80 -> 726bde4)
This is an automated email from the ASF dual-hosted git repository.
davisp pushed a change to branch feature-ebtree-views
in repository https://gitbox.apache.org/repos/asf/couchdb.git.
discard f712e80 Add test suite for reduce views
discard 498a6bb Use ebtree for reduce functions
discard 57dbe74 Update legacy views
discard ba38be9 Reimplement db wide view size tracking
discard 9cd9fda Views on ebtree
discard 63482b9 Export fabric2_fdb:chunkify_binary/1,2
discard ce2dade Calculate external JSON size of a view row
add f190677 Workaround dirty schedulers in run_queue stats (#3168)
add fa3c3c8 Calculate external JSON size of a view row
add 3b508e0 Export fabric2_fdb:chunkify_binary/1,2
add 1f773fb Views on ebtree
add 7368a72 Reimplement db wide view size tracking
add ec3efaf Upgrade legacy views
add 4d7dc40 Use ebtree for reduce functions
add 0f35508 Add test suite for reduce views
add 726bde4 Add elixir tests for builtin reduce group levels
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (f712e80)
\
N -- N -- N refs/heads/feature-ebtree-views (726bde4)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
rel/overlay/etc/default.ini| 4 +-
src/chttpd/src/chttpd_node.erl | 17 +-
src/couch_views/src/couch_views.erl| 4 +-
src/couch_views/src/couch_views_fdb.erl| 594 ++---
src/couch_views/src/couch_views_indexer.erl| 4 +-
src/couch_views/src/couch_views_reader.erl | 12 +-
.../{couch_views_fdb.erl => couch_views_trees.erl} | 431 +++
src/couch_views/src/couch_views_updater.erl| 2 +-
src/couch_views/test/couch_views_indexer_test.erl | 24 +-
src/couch_views/test/couch_views_red_test.erl | 59 +-
src/couch_views/test/couch_views_updater_test.erl | 4 +-
src/couch_views/test/couch_views_upgrade_test.erl | 4 +-
.../test/reduce_builtin_group_level_tests.exs | 549 +++
13 files changed, 724 insertions(+), 984 deletions(-)
copy src/couch_views/src/{couch_views_fdb.erl => couch_views_trees.erl} (56%)
create mode 100644 test/elixir/test/reduce_builtin_group_level_tests.exs
[couchdb] branch feature-ebtree-views-2 created (now 7701f2c)
This is an automated email from the ASF dual-hosted git repository. davisp pushed a change to branch feature-ebtree-views-2 in repository https://gitbox.apache.org/repos/asf/couchdb.git. at 7701f2c Add test suite for reduce views This branch includes the following new commits: new fa3c3c8 Calculate external JSON size of a view row new 3b508e0 Export fabric2_fdb:chunkify_binary/1,2 new 1f773fb Views on ebtree new ba2a426 Reimplement db wide view size tracking new 38ccbe1 Update legacy views new 89c175d Use ebtree for reduce functions new 7701f2c Add test suite for reduce views The 7 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
