[GitHub] [couchdb] nickva commented on a diff in pull request #4718: enhance smoosh to cleanup search indexes when ddocs change
nickva commented on code in PR #4718: URL: https://github.com/apache/couchdb/pull/4718#discussion_r1298530399 ## src/fabric/src/fabric.erl: ## @@ -590,25 +590,38 @@ cleanup_index_files(DbName) -> cleanup_local_indices_and_purge_checkpoints([]) -> ok; cleanup_local_indices_and_purge_checkpoints([_ | _] = Dbs) -> -AllIndices = lists:map(fun couch_mrview_util:get_index_files/1, Dbs), -AllPurges = lists:map(fun couch_mrview_util:get_purge_checkpoints/1, Dbs), -Sigs = couch_mrview_util:get_signatures(hd(Dbs)), -ok = cleanup_purges(Sigs, AllPurges, Dbs), -ok = cleanup_indices(Sigs, AllIndices). +MrViewIndices = lists:map(fun couch_mrview_util:get_index_files/1, Dbs), +MrViewPurges = lists:map(fun couch_mrview_util:get_purge_checkpoints/1, Dbs), +MrViewSigs = couch_mrview_util:get_signatures(hd(Dbs)), +ok = cleanup_mrview_purges(MrViewSigs, MrViewPurges, Dbs), +ok = cleanup_mrview_indices(MrViewSigs, MrViewIndices), -cleanup_purges(Sigs, AllPurges, Dbs) -> +ClouseauSigs = dreyfus_util:active_sigs(hd(Dbs)), +ok = cleanup_clouseau_indices(Dbs, ClouseauSigs), + +NouveauSigs = nouveau_util:active_sigs(hd(Dbs)), +ok = cleanup_nouveau_indices(Dbs, NouveauSigs). + +cleanup_mrview_purges(Sigs, AllPurges, Dbs) -> Fun = fun(DbPurges, Db) -> couch_mrview_cleanup:cleanup_purges(Db, Sigs, DbPurges) end, lists:zipwith(Fun, AllPurges, Dbs), ok. -cleanup_indices(Sigs, AllIndices) -> +cleanup_mrview_indices(Sigs, AllIndices) -> Fun = fun(DbIndices) -> couch_mrview_cleanup:cleanup_indices(Sigs, DbIndices) end, lists:foreach(Fun, AllIndices). +cleanup_clouseau_indices(Dbs, ActiveSigs) -> +Fun = fun(Db) -> clouseau_rpc:cleanup(Db, ActiveSigs) end, +lists:foreach(Fun, Dbs). +cleanup_nouveau_indices(Dbs, ActiveSigs) -> +Fun = fun(Db) -> nouveau_api:delete_path(nouveau_util:index_name(Db), ActiveSigs) end, +lists:foreach(Fun, Dbs). Review Comment: Makes sense. The only worry was that we would throw an exception and prevent other indexes from getting cleaned up. Thanks for double-checking, I think it's fine as is, then. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@couchdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [couchdb] nickva commented on a diff in pull request #4718: enhance smoosh to cleanup search indexes when ddocs change
nickva commented on code in PR #4718: URL: https://github.com/apache/couchdb/pull/4718#discussion_r1297859281 ## src/fabric/src/fabric.erl: ## @@ -590,25 +590,38 @@ cleanup_index_files(DbName) -> cleanup_local_indices_and_purge_checkpoints([]) -> ok; cleanup_local_indices_and_purge_checkpoints([_ | _] = Dbs) -> -AllIndices = lists:map(fun couch_mrview_util:get_index_files/1, Dbs), -AllPurges = lists:map(fun couch_mrview_util:get_purge_checkpoints/1, Dbs), -Sigs = couch_mrview_util:get_signatures(hd(Dbs)), -ok = cleanup_purges(Sigs, AllPurges, Dbs), -ok = cleanup_indices(Sigs, AllIndices). +MrViewIndices = lists:map(fun couch_mrview_util:get_index_files/1, Dbs), +MrViewPurges = lists:map(fun couch_mrview_util:get_purge_checkpoints/1, Dbs), +MrViewSigs = couch_mrview_util:get_signatures(hd(Dbs)), +ok = cleanup_mrview_purges(MrViewSigs, MrViewPurges, Dbs), +ok = cleanup_mrview_indices(MrViewSigs, MrViewIndices), -cleanup_purges(Sigs, AllPurges, Dbs) -> +ClouseauSigs = dreyfus_util:active_sigs(hd(Dbs)), +ok = cleanup_clouseau_indices(Dbs, ClouseauSigs), + +NouveauSigs = nouveau_util:active_sigs(hd(Dbs)), +ok = cleanup_nouveau_indices(Dbs, NouveauSigs). + +cleanup_mrview_purges(Sigs, AllPurges, Dbs) -> Fun = fun(DbPurges, Db) -> couch_mrview_cleanup:cleanup_purges(Db, Sigs, DbPurges) end, lists:zipwith(Fun, AllPurges, Dbs), ok. -cleanup_indices(Sigs, AllIndices) -> +cleanup_mrview_indices(Sigs, AllIndices) -> Fun = fun(DbIndices) -> couch_mrview_cleanup:cleanup_indices(Sigs, DbIndices) end, lists:foreach(Fun, AllIndices). +cleanup_clouseau_indices(Dbs, ActiveSigs) -> +Fun = fun(Db) -> clouseau_rpc:cleanup(Db, ActiveSigs) end, +lists:foreach(Fun, Dbs). +cleanup_nouveau_indices(Dbs, ActiveSigs) -> +Fun = fun(Db) -> nouveau_api:delete_path(nouveau_util:index_name(Db), ActiveSigs) end, +lists:foreach(Fun, Dbs). Review Comment: These shouldn't crash if either of those are not enabled? If there is a chance we could wrap them in a `try ... catch` maybe? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@couchdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [couchdb] nickva commented on a diff in pull request #4718: enhance smoosh to cleanup search indexes when ddocs change
nickva commented on code in PR #4718:
URL: https://github.com/apache/couchdb/pull/4718#discussion_r1286023989
##
src/smoosh/src/smoosh_server.erl:
##
@@ -342,6 +345,8 @@ find_channel(#state{} = State, [Channel | Rest], Object) ->
find_channel(State, Rest, Object)
end.
+stale_enough({?INDEX_CLEANUP, _}) ->
+true;
Review Comment:
We're also triggering these during every shard compaction attempt,
proportionally to the number of shards (so we'd roughly trigger once for each
clustered db). That should be a low enough rate, it may be worth checking what
it looks like a busy cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscr...@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
