Re: [PR] SOLR-17036: Lazy create VersionBucket in the update log to free memory. [solr]

2023-12-05 Thread via GitHub
bruno-roustant merged PR #2021: URL: https://github.com/apache/solr/pull/2021 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.ap

Re: [PR] SOLR-17036: Lazy create VersionBucket in the update log to free memory. [solr]

2023-11-29 Thread via GitHub
bruno-roustant commented on code in PR #2021: URL: https://github.com/apache/solr/pull/2021#discussion_r1408997849 ## solr/core/src/java/org/apache/solr/update/processor/DistributedUpdateProcessor.java: ## @@ -507,29 +504,19 @@ private boolean doVersionAdd(

Re: [PR] SOLR-17036: Lazy create VersionBucket in the update log to free memory. [solr]

2023-11-29 Thread via GitHub
bruno-roustant commented on code in PR #2021: URL: https://github.com/apache/solr/pull/2021#discussion_r1408995357 ## solr/core/src/java/org/apache/solr/update/processor/DistributedUpdateProcessor.java: ## @@ -507,29 +504,19 @@ private boolean doVersionAdd(

Re: [PR] SOLR-17036: Lazy create VersionBucket in the update log to free memory. [solr]

2023-11-29 Thread via GitHub
bruno-roustant commented on code in PR #2021: URL: https://github.com/apache/solr/pull/2021#discussion_r1408995357 ## solr/core/src/java/org/apache/solr/update/processor/DistributedUpdateProcessor.java: ## @@ -507,29 +504,19 @@ private boolean doVersionAdd(

Re: [PR] SOLR-17036: Lazy create VersionBucket in the update log to free memory. [solr]

2023-11-28 Thread via GitHub
dsmiley commented on code in PR #2021: URL: https://github.com/apache/solr/pull/2021#discussion_r1408490861 ## solr/core/src/java/org/apache/solr/update/processor/DistributedUpdateProcessor.java: ## @@ -507,29 +504,19 @@ private boolean doVersionAdd( lastVer

Re: [PR] SOLR-17036: Lazy create VersionBucket in the update log to free memory. [solr]

2023-11-28 Thread via GitHub
bruno-roustant commented on PR #2021: URL: https://github.com/apache/solr/pull/2021#issuecomment-1829889063 I'm going to merge this in 2 days. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] SOLR-17036: Lazy create VersionBucket in the update log to free memory. [solr]

2023-10-22 Thread via GitHub
dsmiley commented on PR #2021: URL: https://github.com/apache/solr/pull/2021#issuecomment-1774196842 Hey way to go Bruno! Thus essentially all we need is a lock per ID (not even necessarily "bucket"-ed); the version was an optimization and doesn't seem to be worth its complexity? @markrmil

Re: [PR] SOLR-17036: Lazy create VersionBucket in the update log to free memory. [solr]

2023-10-20 Thread via GitHub
dsmiley commented on code in PR #2021: URL: https://github.com/apache/solr/pull/2021#discussion_r1366914071 ## solr/core/src/java/org/apache/solr/update/VersionBucket.java: ## @@ -31,14 +31,41 @@ * ignores the lockTimeoutMs. */ public class VersionBucket { - public long hi

Re: [PR] SOLR-17036: Lazy create VersionBucket in the update log to free memory. [solr]

2023-10-20 Thread via GitHub
dsmiley commented on code in PR #2021: URL: https://github.com/apache/solr/pull/2021#discussion_r1366911829 ## solr/core/src/java/org/apache/solr/update/VersionInfo.java: ## @@ -284,11 +297,20 @@ public Long getMaxVersionFromIndex(IndexSearcher searcher) throws IOException {

Re: [PR] SOLR-17036: Lazy create VersionBucket in the update log to free memory. [solr]

2023-10-20 Thread via GitHub
bruno-roustant commented on code in PR #2021: URL: https://github.com/apache/solr/pull/2021#discussion_r1366628537 ## solr/core/src/java/org/apache/solr/update/VersionBucket.java: ## @@ -31,14 +31,41 @@ * ignores the lockTimeoutMs. */ public class VersionBucket { - public

Re: [PR] SOLR-17036: Lazy create VersionBucket in the update log to free memory. [solr]

2023-10-20 Thread via GitHub
bruno-roustant commented on code in PR #2021: URL: https://github.com/apache/solr/pull/2021#discussion_r1366612855 ## solr/core/src/java/org/apache/solr/update/VersionInfo.java: ## @@ -284,11 +297,20 @@ public Long getMaxVersionFromIndex(IndexSearcher searcher) throws IOExcepti

Re: [PR] SOLR-17036: Lazy create VersionBucket in the update log to free memory. [solr]

2023-10-19 Thread via GitHub
dsmiley commented on code in PR #2021: URL: https://github.com/apache/solr/pull/2021#discussion_r1365561237 ## solr/core/src/java/org/apache/solr/update/VersionInfo.java: ## @@ -284,11 +297,20 @@ public Long getMaxVersionFromIndex(IndexSearcher searcher) throws IOException {

Re: [PR] SOLR-17036: Lazy create VersionBucket in the update log to free memory. [solr]

2023-10-19 Thread via GitHub
bruno-roustant commented on PR #2021: URL: https://github.com/apache/solr/pull/2021#issuecomment-1770534588 I removed the Map transition from this PR. The VersionBucket array is created lazily. -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [PR] SOLR-17036: Lazy create VersionBucket in the update log to free memory. [solr]

2023-10-19 Thread via GitHub
bruno-roustant commented on PR #2021: URL: https://github.com/apache/solr/pull/2021#issuecomment-1770500635 I read the code more. The VersionBucket is also used by non-leaders to wait for dependent updates in the case of in-place update. So it's not only a lock for atomic access. -- This

Re: [PR] SOLR-17036: Lazy create VersionBucket in the update log to free memory. [solr]

2023-10-18 Thread via GitHub
dsmiley commented on PR #2021: URL: https://github.com/apache/solr/pull/2021#issuecomment-1769085627 My feedback to Bruno today was overall the this Map upgrade to array (the array we have today) seems kind of pointless; sort of inevitable (assuming there is indexing) so why have this earli

Re: [PR] SOLR-17036: Lazy create VersionBucket in the update log to free memory. [solr]

2023-10-18 Thread via GitHub
bruno-roustant commented on PR #2021: URL: https://github.com/apache/solr/pull/2021#issuecomment-1769002074 Actually, based on an a separate discussion with David, I'll also propose a second PR simplified with the lazy creation of the bucket array, without the map transition. -- This is

Re: [PR] SOLR-17036: Lazy create VersionBucket in the update log to free memory. [solr]

2023-10-18 Thread via GitHub
bruno-roustant commented on code in PR #2021: URL: https://github.com/apache/solr/pull/2021#discussion_r1364255196 ## solr/core/src/java/org/apache/solr/update/VersionInfo.java: ## @@ -284,12 +344,28 @@ public Long getMaxVersionFromIndex(IndexSearcher searcher) throws IOExcepti

Re: [PR] SOLR-17036: Lazy create VersionBucket in the update log to free memory. [solr]

2023-10-18 Thread via GitHub
bruno-roustant commented on code in PR #2021: URL: https://github.com/apache/solr/pull/2021#discussion_r1364249747 ## solr/core/src/java/org/apache/solr/update/VersionInfo.java: ## @@ -205,9 +215,59 @@ public VersionBucket bucket(int hash) { // Make sure high bits are moved

Re: [PR] SOLR-17036: Lazy create VersionBucket in the update log to free memory. [solr]

2023-10-18 Thread via GitHub
stillalex commented on code in PR #2021: URL: https://github.com/apache/solr/pull/2021#discussion_r1364231225 ## solr/core/src/java/org/apache/solr/update/VersionInfo.java: ## @@ -284,12 +344,28 @@ public Long getMaxVersionFromIndex(IndexSearcher searcher) throws IOException {

Re: [PR] SOLR-17036: Lazy create VersionBucket in the update log to free memory. [solr]

2023-10-18 Thread via GitHub
stillalex commented on code in PR #2021: URL: https://github.com/apache/solr/pull/2021#discussion_r1364221046 ## solr/core/src/java/org/apache/solr/update/VersionInfo.java: ## @@ -205,9 +215,59 @@ public VersionBucket bucket(int hash) { // Make sure high bits are moved down

Re: [PR] SOLR-17036: Lazy create VersionBucket in the update log to free memory. [solr]

2023-10-18 Thread via GitHub
stillalex commented on code in PR #2021: URL: https://github.com/apache/solr/pull/2021#discussion_r1364202872 ## solr/core/src/java/org/apache/solr/update/VersionBucket.java: ## @@ -31,14 +31,41 @@ * ignores the lockTimeoutMs. */ public class VersionBucket { - public long

Re: [PR] SOLR-17036: Lazy create VersionBucket in the update log to free memory. [solr]

2023-10-18 Thread via GitHub
stillalex commented on code in PR #2021: URL: https://github.com/apache/solr/pull/2021#discussion_r1364202872 ## solr/core/src/java/org/apache/solr/update/VersionBucket.java: ## @@ -31,14 +31,41 @@ * ignores the lockTimeoutMs. */ public class VersionBucket { - public long

Re: [PR] SOLR-17036: Lazy create VersionBucket in the update log to free memory. [solr]

2023-10-18 Thread via GitHub
bruno-roustant commented on PR #2021: URL: https://github.com/apache/solr/pull/2021#issuecomment-1768027856 Here is my understanding. VersionBucket It serves 2 purposes. First, it is a lock object to lock atomic operations on a doc ID. It is a bucket because it is used to lock o

Re: [PR] SOLR-17036: Lazy create VersionBucket in the update log to free memory. [solr]

2023-10-17 Thread via GitHub
dsmiley commented on code in PR #2021: URL: https://github.com/apache/solr/pull/2021#discussion_r1363159588 ## solr/core/src/java/org/apache/solr/update/VersionInfo.java: ## @@ -103,14 +113,14 @@ public VersionInfo(UpdateLog ulog, int nBuckets) { .get("versionBucket