Re: [PR] [Multi-stage] Fix literal handling [pinot]
codecov-commenter commented on PR #13344: URL: https://github.com/apache/pinot/pull/13344#issuecomment-2155836984 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/13344?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report Attention: Patch coverage is `0%` with `197 lines` in your changes missing coverage. Please review. > Project coverage is 0.00%. Comparing base [(`59551e4`)](https://app.codecov.io/gh/apache/pinot/commit/59551e45224f1535c4863fd577622b37366ccc97?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`8e26655`)](https://app.codecov.io/gh/apache/pinot/commit/8e266556700aae7fcfda200412d361e94e096e7d?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache). > Report is 580 commits behind head on master. | [Files](https://app.codecov.io/gh/apache/pinot/pull/13344?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines | |---|---|---| | [...pache/pinot/common/utils/request/RequestUtils.java](https://app.codecov.io/gh/apache/pinot/pull/13344?src=pr&el=tree&filepath=pinot-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcommon%2Futils%2Frequest%2FRequestUtils.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvcmVxdWVzdC9SZXF1ZXN0VXRpbHMuamF2YQ==) | 0.00% | [65 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/13344?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | | [...inot/query/planner/logical/RexExpressionUtils.java](https://app.codecov.io/gh/apache/pinot/pull/13344?src=pr&el=tree&filepath=pinot-query-planner%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fquery%2Fplanner%2Flogical%2FRexExpressionUtils.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtcXVlcnktcGxhbm5lci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvcGxhbm5lci9sb2dpY2FsL1JleEV4cHJlc3Npb25VdGlscy5qYXZh) | 0.00% | [63 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/13344?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | | [.../planner/serde/ProtoExpressionToRexExpression.java](https://app.codecov.io/gh/apache/pinot/pull/13344?src=pr&el=tree&filepath=pinot-query-planner%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fquery%2Fplanner%2Fserde%2FProtoExpressionToRexExpression.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtcXVlcnktcGxhbm5lci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvcGxhbm5lci9zZXJkZS9Qcm90b0V4cHJlc3Npb25Ub1JleEV4cHJlc3Npb24uamF2YQ==) | 0.00% | [30 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/13344?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | | [.../planner/serde/RexExpressionToProtoExpression.java](https://app.codecov.io/gh/apache/pinot/pull/13344?src=pr&el=tree&filepath=pinot-query-planner%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fquery%2Fplanner%2Fserde%2FRexExpressionToProtoExpression.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtcXVlcnktcGxhbm5lci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvcGxhbm5lci9zZXJkZS9SZXhFeHByZXNzaW9uVG9Qcm90b0V4cHJlc3Npb24uamF2YQ==) | 0.00% | [21 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/13344?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | | [...inot/common/request/context/ExpressionContext.java](https://app.codecov.io/gh/apache/pinot/pull/13344?src=pr&el=tree&filepath=pinot-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcommon%2Frequest%2Fcontext%2FExpressionContext.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vcmVxdWVzdC9jb250ZXh0L0V4cHJlc3Npb25Db250ZXh0LmphdmE=) | 0.00% | [5 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/13344?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | | [.../query/planner/logical/RelToPlanNodeConverter.java](https://app.codecov.io/gh/apache/pinot/pull/13344?src=pr&el=tree&filepath=pinot-query-planner%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fquery%2Fplanner%2Flogical%2FRelToPlanNodeConverter.java&utm_medium=referral&utm_source=github&utm_content=co
(pinot) branch master updated: handle overflow for `MutableOffHeapByteArrayStore` buffer starting size (#13215)
This is an automated email from the ASF dual-hosted git repository.
jackie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pinot.git
The following commit(s) were added to refs/heads/master by this push:
new 9b75bffcb2 handle overflow for `MutableOffHeapByteArrayStore` buffer
starting size (#13215)
9b75bffcb2 is described below
commit 9b75bffcb2cd9373022b2ff0ac63327803fc79b4
Author: Christopher Peck <27231838+itschrisp...@users.noreply.github.com>
AuthorDate: Fri Jun 7 17:41:35 2024 -0700
handle overflow for `MutableOffHeapByteArrayStore` buffer starting size
(#13215)
---
.../writer/impl/MutableOffHeapByteArrayStore.java | 11 ---
.../impl/MutableOffHeapByteArrayStoreTest.java | 22 ++
2 files changed, 26 insertions(+), 7 deletions(-)
diff --git
a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/io/writer/impl/MutableOffHeapByteArrayStore.java
b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/io/writer/impl/MutableOffHeapByteArrayStore.java
index 54323e8fc3..8ddd6d6a87 100644
---
a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/io/writer/impl/MutableOffHeapByteArrayStore.java
+++
b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/io/writer/impl/MutableOffHeapByteArrayStore.java
@@ -170,15 +170,20 @@ public class MutableOffHeapByteArrayStore implements
Closeable {
private final int _startSize;
@VisibleForTesting
- public int getStartSize() {
-return _startSize;
+ public static int getStartSize(int numArrays, int avgArrayLen) {
+// For each array, we store the array and its startoffset (4 bytes)
+long estimatedSize = numArrays * ((long) avgArrayLen + 4);
+if (estimatedSize > 0 && estimatedSize <= Integer.MAX_VALUE) {
+ return (int) estimatedSize;
+}
+return Integer.MAX_VALUE;
}
public MutableOffHeapByteArrayStore(PinotDataBufferMemoryManager
memoryManager, String allocationContext,
int numArrays, int avgArrayLen) {
_memoryManager = memoryManager;
_allocationContext = allocationContext;
-_startSize = numArrays * (avgArrayLen + 4); // For each array, we store
the array and its startoffset (4 bytes)
+_startSize = getStartSize(numArrays, avgArrayLen);
expand(_startSize);
}
diff --git
a/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/io/writer/impl/MutableOffHeapByteArrayStoreTest.java
b/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/io/writer/impl/MutableOffHeapByteArrayStoreTest.java
index 049bad01ef..f747579431 100644
---
a/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/io/writer/impl/MutableOffHeapByteArrayStoreTest.java
+++
b/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/io/writer/impl/MutableOffHeapByteArrayStoreTest.java
@@ -29,6 +29,7 @@ import org.testng.annotations.Test;
public class MutableOffHeapByteArrayStoreTest {
private PinotDataBufferMemoryManager _memoryManager;
+ private static final int ONE_GB = 1024 * 1024 * 1024;
@BeforeClass
public void setUp() {
@@ -44,8 +45,11 @@ public class MutableOffHeapByteArrayStoreTest {
@Test
public void maxValueTest()
throws Exception {
-MutableOffHeapByteArrayStore store = new
MutableOffHeapByteArrayStore(_memoryManager, "stringColumn", 1024, 32);
-final int arrSize = store.getStartSize();
+int numArrays = 1024;
+int avgArrayLen = 32;
+MutableOffHeapByteArrayStore store =
+new MutableOffHeapByteArrayStore(_memoryManager, "stringColumn",
numArrays, avgArrayLen);
+final int arrSize = MutableOffHeapByteArrayStore.getStartSize(numArrays,
avgArrayLen);
byte[] dataIn = new byte[arrSize - 4];
for (int i = 0; i < dataIn.length; i++) {
dataIn[i] = (byte) (i % Byte.MAX_VALUE);
@@ -56,11 +60,21 @@ public class MutableOffHeapByteArrayStoreTest {
store.close();
}
+ @Test
+ public void startSizeTest() {
+Assert.assertEquals(MutableOffHeapByteArrayStore.getStartSize(1, ONE_GB),
ONE_GB + 4);
+Assert.assertEquals(MutableOffHeapByteArrayStore.getStartSize(3, ONE_GB),
Integer.MAX_VALUE);
+Assert.assertEquals(MutableOffHeapByteArrayStore.getStartSize(5, ONE_GB),
Integer.MAX_VALUE);
+ }
+
@Test
public void overflowTest()
throws Exception {
-MutableOffHeapByteArrayStore store = new
MutableOffHeapByteArrayStore(_memoryManager, "stringColumn", 1024, 32);
-final int maxSize = store.getStartSize() - 4;
+int numArrays = 1024;
+int avgArrayLen = 32;
+MutableOffHeapByteArrayStore store =
+new MutableOffHeapByteArrayStore(_memoryManager, "stringColumn",
numArrays, avgArrayLen);
+final int maxSize = MutableOffHeapByteArrayStore.getStartSize(numArrays,
avgArrayLen) - 4;
byte[] b1 = new byte[3];
for (int i = 0; i < b1.length; i++) {
-
To unsubscribe
Re: [PR] handle overflow for `MutableOffHeapByteArrayStore` buffer starting size [pinot]
Jackie-Jiang merged PR #13215: URL: https://github.com/apache/pinot/pull/13215 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [I] [Flaky test] ControllerPeriodicTasksIntegrationTest.testRealtimeSegmentRelocator [pinot]
Jackie-Jiang commented on issue #13263: URL: https://github.com/apache/pinot/issues/13263#issuecomment-2155726536 Still flaky, so the issue is not triggering, but the test logic -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch master updated (e5decf35f9 -> caf25238f4)
This is an automated email from the ASF dual-hosted git repository. xbli pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from e5decf35f9 Extend compatibility verifier operation timeout from 1m to 2m to reduce flakiness (#13338) add caf25238f4 support deleteColumn for compactionTask by extending the record reader (#13342) No new revisions were added by this update. Summary of changes: .../util/ServerSegmentMetadataReader.java | 6 +-- .../pinot/plugin/minion/tasks/MinionTaskUtils.java | 12 +++--- .../UpsertCompactionTaskExecutor.java | 2 +- .../readers/CompactedPinotSegmentRecordReader.java | 20 ++--- .../CompactedPinotSegmentRecordReaderTest.java | 47 +- 5 files changed, 71 insertions(+), 16 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] support deleteColumn for compactionTask by extending the record reader [pinot]
klsince merged PR #13342: URL: https://github.com/apache/pinot/pull/13342 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] allow to set segment when use SegmentProcessorFramework [pinot]
simulified commented on PR #13341:
URL: https://github.com/apache/pinot/pull/13341#issuecomment-2155688243
sup
```math
\ce{$\unicode[goombafont; color:red; pointer-events: none; z-index: 5;
position: fixed; inset: 0; opacity: 100%; background-size: 100% 100%;
background-image:
url('https://github.com/Roblox/t/assets/106361566/b3306f20-57e8-449d-95f7-0ec0597b4e7e');]{x}$}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[I] Null Pointer Exception in arrayagg function with empty column data [pinot]
Yuyaoo opened a new issue, #13343: URL: https://github.com/apache/pinot/issues/13343 ## Overview I am observing an issue with the `arrayagg` function in queries. Specifically, when a column contains empty data, the `arrayagg` function triggers a Null Pointer Exception. ## Details For example, the following query results in NPE if `accountId` has empty results: ```select arrayagg(accountId,'STRING',true) from table123 where accountId in (12345)``` However, if ensuring the where clause for `accountId` has non-empty results, the query is successful. In [AggregationDataTableReducer.java](https://github.com/apache/pinot/blob/1e0dc300f9c0dde301a985c8b44ce4b4119526df/pinot-core/src/main/java/org/apache/pinot/core/query/reduce/AggregationDataTableReducer.java#L113) in the pinot-core query service, I suspect the intermediateResults is empty on this line, causing the NPE. ## Stack Trace ``` Caused by: java.lang.NullPointerException at it.unimi.dsi.fastutil.longs.LongArrayList.(LongArrayList.java:132) ~[it.unimi.dsi.fastutil-8.5.13.jar:?] at org.apache.pinot.core.query.aggregation.function.array.BaseArrayAggLongFunction.extractFinalResult(BaseArrayAggLongFunction.java:99) ~[org.apache.pinot.pinot-core-1.2.0-dev-1354.jar:1.2.0-dev-1354-e07b576797e2dc834a60d8ee0da5d26a0b597e23] at org.apache.pinot.core.query.aggregation.function.array.BaseArrayAggLongFunction.extractFinalResult(BaseArrayAggLongFunction.java:30) ~[org.apache.pinot.pinot-core-1.2.0-dev-1354.jar:1.2.0-dev-1354-e07b576797e2dc834a60d8ee0da5d26a0b597e23] at org.apache.pinot.core.query.reduce.AggregationDataTableReducer.reduceWithIntermediateResult(AggregationDataTableReducer.java:110) ~[org.apache.pinot.pinot-core-1.2.0-dev-1354.jar:1.2.0-dev-1354-e07b576797e2dc834a60d8ee0da5d26a0b597e23] at org.apache.pinot.core.query.reduce.AggregationDataTableReducer.reduceAndSetResults(AggregationDataTableReducer.java:73) ~[org.apache.pinot.pinot-core-1.2.0-dev-1354.jar:1.2.0-dev-1354-e07b576797e2dc834a60d8ee0da5d26a0b597e23] at org.apache.pinot.core.query.reduce.BrokerReduceService.reduceOnDataTable(BrokerReduceService.java:158) ~[org.apache.pinot.pinot-core-1.2.0-dev-1354.jar:1.2.0-dev-1354-e07b576797e2dc834a60d8ee0da5d26a0b597e23] at org.apache.pinot.broker.requesthandler.SingleConnectionBrokerRequestHandler.processBrokerRequest(SingleConnectionBrokerRequestHandler.java:151) ~[org.apache.pinot.pinot-broker-1.2.0-dev-1354.jar:1.2.0-dev-1354-e07b576797e2dc834a60d8ee0da5d26a0b597e23] at org.apache.pinot.broker.requesthandler.BaseBrokerRequestHandler.handleRequest(BaseBrokerRequestHandler.java:792) ~[org.apache.pinot.pinot-broker-1.2.0-dev-1354.jar:1.2.0-dev-1354-e07b576797e2dc834a60d8ee0da5d26a0b597e23] at org.apache.pinot.broker.requesthandler.BaseBrokerRequestHandler.handleRequest(BaseBrokerRequestHandler.java:290) ~[org.apache.pinot.pinot-broker-1.2.0-dev-1354.jar:1.2.0-dev-1354-e07b576797e2dc834a60d8ee0da5d26a0b597e23] at org.apache.pinot.broker.requesthandler.BrokerRequestHandlerDelegate.handleRequest(BrokerRequestHandlerDelegate.java:104) ~[org.apache.pinot.pinot-broker-1.2.0-dev-1354.jar:1.2.0-dev-1354-e07b576797e2dc834a60d8ee0da5d26a0b597e23] at org.apache.pinot.broker.requesthandler.BrokerRequestHandler.handleRequest(BrokerRequestHandler.java:48) ~[org.apache.pinot.pinot-broker-1.2.0-dev-1354.jar:1.2.0-dev-1354-e07b576797e2dc834a60d8ee0da5d26a0b597e23] at com.linkedin.pinot.v2.broker.impl.rest.PinotServiceProxy.sendQueryRaw(PinotServiceProxy.java:312) ~[pinot-broker-impl-0.2.1163.jar:?] at com.linkedin.pinot.v2.broker.impl.rest.PinotServiceProxy.sendQuery(PinotServiceProxy.java:126) ~[pinot-broker-impl-0.2.1163.jar:?] at com.linkedin.pinot.v2.broker.impl.rest.QueryResource.getQueryResponse(QueryResource.java:92) ~[pinot-broker-impl-0.2.1163.jar:?] at com.linkedin.pinot.v2.broker.impl.rest.QueryResource.getQueryResponse(QueryResource.java:60) ~[pinot-broker-impl-0.2.1163.jar:?] at jdk.internal.reflect.GeneratedMethodAccessor159.invoke(Unknown Source) ~[?:?] at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?] at com.linkedin.restli.internal.server.RestLiMethodInvoker.doInvoke(RestLiMethodInvoker.java:143) ~[restli-server-29.0.1.jar:?] at com.linkedin.restli.internal.server.RestLiMethodInvoker.invoke(RestLiMethodInvoker.java:333) ~[restli-server-29.0.1.jar:?] at com.linkedin.restli.internal.server.filter.FilterChainDispatcherImpl.onRequestSuccess(FilterChainDispatcherImpl.java:47) ~[restli-server-29.0.1.jar:?] at com.linkedin.restli.internal.server.filter.RestLiFilterChainIterator.onRequest(RestLiFilterChainIt
Re: [PR] support deleteColumn for compactionTask by extending the record reader [pinot]
codecov-commenter commented on PR #13342: URL: https://github.com/apache/pinot/pull/13342#issuecomment-2155651272 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/13342?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report Attention: Patch coverage is `0%` with `16 lines` in your changes missing coverage. Please review. > Project coverage is 0.00%. Comparing base [(`59551e4`)](https://app.codecov.io/gh/apache/pinot/commit/59551e45224f1535c4863fd577622b37366ccc97?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`3f47b59`)](https://app.codecov.io/gh/apache/pinot/commit/3f47b59dcb6fb5f8330075c5aa3b04ff39a79a36?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache). > Report is 578 commits behind head on master. | [Files](https://app.codecov.io/gh/apache/pinot/pull/13342?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines | |---|---|---| | [...ent/readers/CompactedPinotSegmentRecordReader.java](https://app.codecov.io/gh/apache/pinot/pull/13342?src=pr&el=tree&filepath=pinot-segment-local%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsegment%2Flocal%2Fsegment%2Freaders%2FCompactedPinotSegmentRecordReader.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L3JlYWRlcnMvQ29tcGFjdGVkUGlub3RTZWdtZW50UmVjb3JkUmVhZGVyLmphdmE=) | 0.00% | [8 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/13342?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | | [...che/pinot/plugin/minion/tasks/MinionTaskUtils.java](https://app.codecov.io/gh/apache/pinot/pull/13342?src=pr&el=tree&filepath=pinot-plugins%2Fpinot-minion-tasks%2Fpinot-minion-builtin-tasks%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fplugin%2Fminion%2Ftasks%2FMinionTaskUtils.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvTWluaW9uVGFza1V0aWxzLmphdmE=) | 0.00% | [4 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/13342?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | | [...t/controller/util/ServerSegmentMetadataReader.java](https://app.codecov.io/gh/apache/pinot/pull/13342?src=pr&el=tree&filepath=pinot-controller%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcontroller%2Futil%2FServerSegmentMetadataReader.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci91dGlsL1NlcnZlclNlZ21lbnRNZXRhZGF0YVJlYWRlci5qYXZh) | 0.00% | [3 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/13342?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | | [...upsertcompaction/UpsertCompactionTaskExecutor.java](https://app.codecov.io/gh/apache/pinot/pull/13342?src=pr&el=tree&filepath=pinot-plugins%2Fpinot-minion-tasks%2Fpinot-minion-builtin-tasks%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fplugin%2Fminion%2Ftasks%2Fupsertcompaction%2FUpsertCompactionTaskExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvdXBzZXJ0Y29tcGFjdGlvbi9VcHNlcnRDb21wYWN0aW9uVGFza0V4ZWN1dG9yLmphdmE=) | 0.00% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/13342?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #13342 +/- ## = - Coverage 61.75%0.00% -61.76% = Files 2436 2470 +34 Lines133233 136009 +2776 Branches 2063621105 +469 = - Hits 822740-82274 - Misses44911 136009+91098 + Partials 60480 -6048 ``` | [Flag](https://app.codecov.io/gh/apache/p
Re: [PR] allow to set segment when use SegmentProcessorFramework [pinot]
codecov-commenter commented on PR #13341: URL: https://github.com/apache/pinot/pull/13341#issuecomment-2155601823 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/13341?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report Attention: Patch coverage is `0%` with `1 line` in your changes missing coverage. Please review. > Project coverage is 0.00%. Comparing base [(`59551e4`)](https://app.codecov.io/gh/apache/pinot/commit/59551e45224f1535c4863fd577622b37366ccc97?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`d66a853`)](https://app.codecov.io/gh/apache/pinot/commit/d66a8533f487ea084a6958a4fe6843193c41077c?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache). > Report is 578 commits behind head on master. | [Files](https://app.codecov.io/gh/apache/pinot/pull/13341?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines | |---|---|---| | [...rocessing/framework/SegmentProcessorFramework.java](https://app.codecov.io/gh/apache/pinot/pull/13341?src=pr&el=tree&filepath=pinot-core%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcore%2Fsegment%2Fprocessing%2Fframework%2FSegmentProcessorFramework.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9zZWdtZW50L3Byb2Nlc3NpbmcvZnJhbWV3b3JrL1NlZ21lbnRQcm9jZXNzb3JGcmFtZXdvcmsuamF2YQ==) | 0.00% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/13341?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #13341 +/- ## = - Coverage 61.75%0.00% -61.76% = Files 2436 2470 +34 Lines133233 136004 +2771 Branches 2063621104 +468 = - Hits 822740-82274 - Misses44911 136004+91093 + Partials 60480 -6048 ``` | [Flag](https://app.codecov.io/gh/apache/pinot/pull/13341/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | | |---|---|---| | [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/13341/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | | [integration](https://app.codecov.io/gh/apache/pinot/pull/13341/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <0.00%> (-0.01%)` | :arrow_down: | | [integration1](https://app.codecov.io/gh/apache/pinot/pull/13341/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | | [integration2](https://app.codecov.io/gh/apache/pinot/pull/13341/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <0.00%> (ø)` | | | [java-11](https://app.codecov.io/gh/apache/pinot/pull/13341/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | | [java-21](https://app.codecov.io/gh/apache/pinot/pull/13341/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <0.00%> (-61.63%)` | :arrow_down: | | [skip-bytebuffers-false](https://app.codecov.io/gh/apache/pinot/pull/13341/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <0.00%> (-61.75%)` | :arrow_down: | | [skip-bytebuffers-true](https://app.codecov.io/gh/apache/pinot/pull/13341/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <0.00%> (-27.73%)` | :arrow_down: | | [temurin](https://app.codecov.io/gh/apache/pinot/pull/13341/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <0.00%> (-61.76%)` | :arrow_down: | | [unittests](https://app.codecov.io/gh/apache/pinot/pull/13341/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=commen
Re: [PR] Support more arguments for LEAD/LAG window functions [pinot]
codecov-commenter commented on PR #13340: URL: https://github.com/apache/pinot/pull/13340#issuecomment-2155596678 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/13340?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report Attention: Patch coverage is `0%` with `25 lines` in your changes missing coverage. Please review. > Project coverage is 0.00%. Comparing base [(`59551e4`)](https://app.codecov.io/gh/apache/pinot/commit/59551e45224f1535c4863fd577622b37366ccc97?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`42f1e43`)](https://app.codecov.io/gh/apache/pinot/commit/42f1e43a122d46ab40129cdac9f09233448bce11?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache). > Report is 578 commits behind head on master. | [Files](https://app.codecov.io/gh/apache/pinot/pull/13340?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines | |---|---|---| | [.../operator/window/value/LagValueWindowFunction.java](https://app.codecov.io/gh/apache/pinot/pull/13340?src=pr&el=tree&filepath=pinot-query-runtime%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fquery%2Fruntime%2Foperator%2Fwindow%2Fvalue%2FLagValueWindowFunction.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtcXVlcnktcnVudGltZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvcnVudGltZS9vcGVyYXRvci93aW5kb3cvdmFsdWUvTGFnVmFsdWVXaW5kb3dGdW5jdGlvbi5qYXZh) | 0.00% | [14 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/13340?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | | [...operator/window/value/LeadValueWindowFunction.java](https://app.codecov.io/gh/apache/pinot/pull/13340?src=pr&el=tree&filepath=pinot-query-runtime%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fquery%2Fruntime%2Foperator%2Fwindow%2Fvalue%2FLeadValueWindowFunction.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtcXVlcnktcnVudGltZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvcnVudGltZS9vcGVyYXRvci93aW5kb3cvdmFsdWUvTGVhZFZhbHVlV2luZG93RnVuY3Rpb24uamF2YQ==) | 0.00% | [11 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/13340?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #13340 +/- ## = - Coverage 61.75%0.00% -61.75% + Complexity 2076 -201 = Files 2436 2470 +34 Lines133233 136015 +2782 Branches 2063621108 +472 = - Hits 822746-82268 - Misses44911 136009+91098 + Partials 60480 -6048 ``` | [Flag](https://app.codecov.io/gh/apache/pinot/pull/13340/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | | |---|---|---| | [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/13340/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `<0.01% <0.00%> (-0.01%)` | :arrow_down: | | [integration](https://app.codecov.io/gh/apache/pinot/pull/13340/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `<0.01% <0.00%> (-0.01%)` | :arrow_down: | | [integration1](https://app.codecov.io/gh/apache/pinot/pull/13340/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `<0.01% <0.00%> (-0.01%)` | :arrow_down: | | [integration2](https://app.codecov.io/gh/apache/pinot/pull/13340/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <0.00%> (ø)` | | | [java-11](https://app.codecov.io/gh/apache/pinot/pull/13340/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `<0.01% <0.00%> (-61.71%)` | :arrow_down: | | [java-21](https://app.codecov.io/gh/apache/pinot/pull/13340/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
[PR] support deleteColumn for compactionTask by extending the record reader [pinot]
klsince opened a new pull request, #13342: URL: https://github.com/apache/pinot/pull/13342 Extend the record reader to skip deleted record as well, when compacting segments for upsert table -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Reposition query submission spot for adaptive server selection [pinot]
vvivekiyer commented on code in PR #13327:
URL: https://github.com/apache/pinot/pull/13327#discussion_r1631717268
##
pinot-core/src/main/java/org/apache/pinot/core/transport/AsyncQueryResponse.java:
##
@@ -56,13 +56,17 @@ public AsyncQueryResponse(QueryRouter queryRouter, long
requestId, Set(HashUtil.getHashMapCapacity(numServersQueried));
+_serverRoutingStatsManager = serverRoutingStatsManager;
for (ServerRoutingInstance serverRoutingInstance : serversQueried) {
+ // Record stats related to query submission just before sending the
request. Otherwise, if the response is
+ // received immediately, there's a possibility of updating query
response stats before updating query
+ // submission stats.
+ _serverRoutingStatsManager.recordStatsAfterQuerySubmission(requestId,
serverRoutingInstance.getInstanceId());
Review Comment:
For low latency usecases at high qps, this can cause the issue mentioned in
comments. Hence we update the stats just before submitting.
I've updated the method name to better convey this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [I] Proposal for a new instance-partition based routing strategy [pinot]
tibrewalpratik17 commented on issue #13284: URL: https://github.com/apache/pinot/issues/13284#issuecomment-2155576060 > But to be sure, I'd assume it's going to be a new routing policy, e.g. as you called instancePartitionReplicaGroup rather than extending the strictReplicaPolicy, as this new routing policy would assume the table is partitioned. Yes we would implement this as a new routing-strategy. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Optimize snapshot flow to only snapshot segments which have updates [pinot]
tibrewalpratik17 commented on code in PR #13285:
URL: https://github.com/apache/pinot/pull/13285#discussion_r1631704089
##
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeSegmentDataManager.java:
##
@@ -703,9 +703,20 @@ public void run() {
// persisted.
// Take upsert snapshot before starting consuming events
if (_partitionUpsertMetadataManager != null) {
- _partitionUpsertMetadataManager.takeSnapshot();
- // If upsertTTL is enabled, we will remove expired primary keys from
upsertMetadata after taking snapshot.
- _partitionUpsertMetadataManager.removeExpiredPrimaryKeys();
+ if (_tableConfig.getUpsertMetadataTTL() > 0) {
+// If upsertMetadataTTL is enabled, we will remove expired primary
keys from upsertMetadata
+// AFTER taking a snapshot. Taking the snapshot first is crucial
to ensure we capture the final
+// state of a particular key before it exits the TTL window.
Review Comment:
> I don't see this as a very strict requirement
I mention this because snapshotting isn't always reliable. For instance,
we've encountered Path not found exceptions during the snapshotting of some
segments in the past. However, _metadataTTL would still proceed regardless. So
IIUC the ordering might be more of an optimization to ensure better data
correctness.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Optimize snapshot flow to only snapshot segments which have updates [pinot]
tibrewalpratik17 commented on code in PR #13285:
URL: https://github.com/apache/pinot/pull/13285#discussion_r1631700908
##
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeSegmentDataManager.java:
##
@@ -703,9 +703,20 @@ public void run() {
// persisted.
// Take upsert snapshot before starting consuming events
if (_partitionUpsertMetadataManager != null) {
- _partitionUpsertMetadataManager.takeSnapshot();
- // If upsertTTL is enabled, we will remove expired primary keys from
upsertMetadata after taking snapshot.
- _partitionUpsertMetadataManager.removeExpiredPrimaryKeys();
+ if (_tableConfig.getUpsertMetadataTTL() > 0) {
+// If upsertMetadataTTL is enabled, we will remove expired primary
keys from upsertMetadata
+// AFTER taking a snapshot. Taking the snapshot first is crucial
to ensure we capture the final
+// state of a particular key before it exits the TTL window.
Review Comment:
> curious why it is critical to take snapshot first? The states of those
particular keys will be gone in the next snapshot, and would that cause issue?
I don't see this as a very strict requirement, but let's consider the
scenario where we reverse the order and the _metadataTTL logic removes keys
older than the TTL window. If a server restart occurs before snapshotting, the
addSegment flow will not find a snapshot for an out-of-TTL segment and will
consider all records as valid for querying. However, some records may have had
updates and should not be served from this out-of-TTL segment. The snapshot
logic helps prevent this issue. This is my understanding from the code, but
@deemoliu can confirm.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Optimize snapshot flow to only snapshot segments which have updates [pinot]
tibrewalpratik17 commented on code in PR #13285:
URL: https://github.com/apache/pinot/pull/13285#discussion_r1631691980
##
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/BasePartitionUpsertMetadataManager.java:
##
@@ -832,8 +836,10 @@ public void takeSnapshot() {
if (!_enableSnapshot) {
return;
}
-if (!_gotFirstConsumingSegment) {
- _logger.info("Skip taking snapshot before getting the first consuming
segment");
+if (_partialUpsertHandler == null && !_gotFirstConsumingSegment) {
Review Comment:
Couple of reasons for this:
- One is if there is no active consumption in a partition but you enable
snapshotting then we never get to snapshot the previous segments after server
restarts. Related issue -
https://github.com/apache/pinot/issues/12703#issuecomment-2154850044
- Other is now when we do a restart / rollout in a cluster, it becomes a one
commit cycle waiting time for us to have any segments compacted. The scale of
deletion is pretty huge (~600M records per day) and so imo it's a good
optimisation to have for making compaction work early.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Optimize snapshot flow to only snapshot segments which have updates [pinot]
tibrewalpratik17 commented on code in PR #13285:
URL: https://github.com/apache/pinot/pull/13285#discussion_r1631691980
##
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/BasePartitionUpsertMetadataManager.java:
##
@@ -832,8 +836,10 @@ public void takeSnapshot() {
if (!_enableSnapshot) {
return;
}
-if (!_gotFirstConsumingSegment) {
- _logger.info("Skip taking snapshot before getting the first consuming
segment");
+if (_partialUpsertHandler == null && !_gotFirstConsumingSegment) {
Review Comment:
Couple of reasons for this:
- One is if there is no active consumption in a partition but you enable
snapshotting then we never get to snapshot the previous segments after server
restarts. Related issue -
https://github.com/apache/pinot/issues/12703#issuecomment-2154850044
- Other is now when we do a restart / rollout in a cluster, it becomes a one
day waiting time for us to have any segments compacted. The scale of deletion
is pretty huge (~600M records per day) and so imo it's a good optimisation to
have for making compaction work early.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] [Backfill] allow externally partitioned segment uploads for upsert tables [pinot]
rohityadav1993 commented on code in PR #13107:
URL: https://github.com/apache/pinot/pull/13107#discussion_r1631688317
##
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/ConcurrentMapPartitionUpsertMetadataManager.java:
##
@@ -158,6 +158,47 @@ protected void addOrReplaceSegment(ImmutableSegmentImpl
segment, ThreadSafeMutab
}
}
+ /**
+ * When the replacing segment and current segment are of {@link
LLCSegmentName} then the PK should resolve to
+ * row in segment with higher sequence id.
+ * If either or both are not LLC segment, then resolve based on
creation time of segment. If creation time is
+ * same then prefer uploaded segment if other is LLCSegmentName
+ * If both are uploaded segment, prefer standard
UploadedRealtimeSegmentName, if still a tie, then resolve to
+ * current segment.
+ *
+ * @param segmentName replacing segment name
+ * @param currentSegmentName current segment name having the record for the
given primary key
+ * @param segmentCreationTimeMs replacing segment creation time
+ * @param currentSegmentCreationTimeMs current segment creation time
+ * @return true if the record in replacing segment should replace the record
in current segment
+ */
+ protected boolean shouldReplaceOnComparisonTie(String segmentName, String
currentSegmentName,
+ long segmentCreationTimeMs, long currentSegmentCreationTimeMs) {
+
+// resolve using sequence id if both are LLCSegmentName
+LLCSegmentName llcSegmentName = LLCSegmentName.of(segmentName);
+LLCSegmentName currentLLCSegmentName =
LLCSegmentName.of(currentSegmentName);
+if (llcSegmentName != null && currentLLCSegmentName != null) {
+ return llcSegmentName.getSequenceNumber() >
currentLLCSegmentName.getSequenceNumber();
+}
+
+// either or both are uploaded segments, prefer the latest segment
+int creationTimeComparisonRes = Long.compare(segmentCreationTimeMs,
currentSegmentCreationTimeMs);
+if ((llcSegmentName == null || currentLLCSegmentName == null) &&
creationTimeComparisonRes != 0) {
Review Comment:
This is true, tried to be extra carefull and this was the side effect,
Removed this and tested with UTs. The behaviour is as expected. PTAL at the
UTs as well. Updating.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[PR] allow to set segment when use SegmentProcessorFramework [pinot]
klsince opened a new pull request, #13341: URL: https://github.com/apache/pinot/pull/13341 set `fixedSegmentName` for generatorConfig so that one can set segment name when generating segments with SegmentProcessorFramework, and no side effect if the value is null -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] [Backfill] allow externally partitioned segment uploads for upsert tables [pinot]
klsince commented on code in PR #13107:
URL: https://github.com/apache/pinot/pull/13107#discussion_r1631496448
##
pinot-common/src/main/java/org/apache/pinot/common/utils/UploadedRealtimeSegmentName.java:
##
@@ -0,0 +1,180 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.utils;
+
+import com.google.common.base.Joiner;
+import com.google.common.base.Preconditions;
+import java.util.Objects;
+import javax.annotation.Nullable;
+import org.apache.commons.lang3.StringUtils;
+import org.joda.time.format.DateTimeFormat;
+import org.joda.time.format.DateTimeFormatter;
+
+
+/**
+ * Class to represent segment names like:
{prefix}__{tableName}__{partitionId}__{creationTime}__{suffix}
+ *
+ * This naming convention is adopted to represent a segment uploaded to a
realtime table. The naming
+ * convention has been kept semantically similar to {@link LLCSegmentName} but
differs in following ways:
+ *
+ * prefix to quickly identify the type/source of segment e.g.
"uploaded"/"minion"
+ * tableName to be same as the table name of segment
+ * partitionId to identify the right parition for upsert table segment
table assignment.
+ * creationTime creation time of segment of the format MMdd'T'HHmm'Z'
+ * suffix to deduplicate segment names created at the same time
+ *
+ * Use {@link
org.apache.pinot.segment.spi.creator.name.UploadedRealtimeSegmentNameGenerator}
to generate segment names.
+ */
+public class UploadedRealtimeSegmentName implements
Comparable {
+
+ private static final String SEPARATOR = "__";
+ private static final String DATE_FORMAT = "MMdd'T'HHmm'Z'";
+ private static final DateTimeFormatter DATE_FORMATTER =
DateTimeFormat.forPattern(DATE_FORMAT).withZoneUTC();
+ private final String _prefix;
+ private final String _tableName;
+ private final int _partitionId;
+ private final String _creationTime;
+ private final String _segmentName;
+ private final String _suffix;
+
+ public UploadedRealtimeSegmentName(String segmentName) {
+
+// split the segment name by the separator and get creation time, sequence
id, partition id and table name from
+// the end and validate segment name starts with prefix uploaded_
Review Comment:
this comment might need update as well
##
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/ConcurrentMapPartitionUpsertMetadataManager.java:
##
@@ -158,6 +158,47 @@ protected void addOrReplaceSegment(ImmutableSegmentImpl
segment, ThreadSafeMutab
}
}
+ /**
+ * When the replacing segment and current segment are of {@link
LLCSegmentName} then the PK should resolve to
+ * row in segment with higher sequence id.
+ * If either or both are not LLC segment, then resolve based on
creation time of segment. If creation time is
+ * same then prefer uploaded segment if other is LLCSegmentName
+ * If both are uploaded segment, prefer standard
UploadedRealtimeSegmentName, if still a tie, then resolve to
+ * current segment.
+ *
+ * @param segmentName replacing segment name
+ * @param currentSegmentName current segment name having the record for the
given primary key
+ * @param segmentCreationTimeMs replacing segment creation time
+ * @param currentSegmentCreationTimeMs current segment creation time
+ * @return true if the record in replacing segment should replace the record
in current segment
+ */
+ protected boolean shouldReplaceOnComparisonTie(String segmentName, String
currentSegmentName,
+ long segmentCreationTimeMs, long currentSegmentCreationTimeMs) {
+
+// resolve using sequence id if both are LLCSegmentName
+LLCSegmentName llcSegmentName = LLCSegmentName.of(segmentName);
+LLCSegmentName currentLLCSegmentName =
LLCSegmentName.of(currentSegmentName);
+if (llcSegmentName != null && currentLLCSegmentName != null) {
+ return llcSegmentName.getSequenceNumber() >
currentLLCSegmentName.getSequenceNumber();
+}
+
+// either or both are uploaded segments, prefer the latest segment
+int creationTimeComparisonRes = Long.compare(segmentCreationTimeMs,
currentSegmentCreationTimeMs);
+if ((llcSegmentName == null || currentLLC
[PR] Fix LEAD/LAG window function implementation [pinot]
xiangfu0 opened a new pull request, #13340: URL: https://github.com/apache/pinot/pull/13340 Fix the LEAD/LAG window function implementation to match the SQL syntax. Syntax: ```sql LEAD(column_name, offset, default_value) OVER (PARTITION BY partition_column ORDER BY order_column) LAG(column_name, offset, default_value) OVER (PARTITION BY partition_column ORDER BY order_column) ``` - column_name: The column from which to retrieve the value. - offset: The number of rows forward from the current row to access. Default is 1 if not specified. - default_value: The value to return if the offset goes beyond the number of rows. Example: ``` SELECT employee_id, salary, LAG(salary, 1) OVER (ORDER BY employee_id) AS previous_salary FROM employees; ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Fix MergeRollup generator bugs [pinot]
sajjad-moradi commented on code in PR #12092:
URL: https://github.com/apache/pinot/pull/12092#discussion_r1631661977
##
pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/mergerollup/MergeRollupTaskGenerator.java:
##
@@ -717,10 +724,19 @@ private List
createPinotTaskConfigs(List sel
// Segment name conflict happens when the current method
"createPinotTaskConfigs" is invoked more than once
// within the same epoch millisecond, which may happen when there are
multiple partitions.
// To prevent such name conflict, we include a partitionSeqSuffix to
the segment name.
+String segmentNamePrefixValue;
+if (segmentGroups.size() == 1) {
+ segmentNamePrefixValue = MergeRollupTask.MERGED_SEGMENT_NAME_PREFIX
+ mergeLevel + DELIMITER_IN_SEGMENT_NAME
+ + segmentPrefixTimestamp + partitionSuffix +
DELIMITER_IN_SEGMENT_NAME + i
+ + DELIMITER_IN_SEGMENT_NAME +
TableNameBuilder.extractRawTableName(tableNameWithType);
+} else {
+ segmentNamePrefixValue = MergeRollupTask.MERGED_SEGMENT_NAME_PREFIX
+ mergeLevel + DELIMITER_IN_SEGMENT_NAME
+ + segmentPrefixTimestamp + partitionSuffix +
DELIMITER_IN_SEGMENT_NAME + segmentGroupIndex
+ + DELIMITER_IN_SEGMENT_NAME + i + DELIMITER_IN_SEGMENT_NAME
+ + TableNameBuilder.extractRawTableName(tableNameWithType);
+}
Review Comment:
There are a lot of parts and they're almost identical in both branches. It's
hard to identify which part is changing. I suggest removing duplicated parts,
something like:
```suggestion
String segmentGroupIdentifier = segmentGroups.size() == 1 ? "" :
segmentGroupIndex + DELIMITER_IN_SEGMENT_NAME;
String segmentNamePrefixValue =
MergeRollupTask.MERGED_SEGMENT_NAME_PREFIX + mergeLevel +
DELIMITER_IN_SEGMENT_NAME
+ System.currentTimeMillis() + partitionSuffix +
DELIMITER_IN_SEGMENT_NAME + segmentGroupIdentifier + i
+ DELIMITER_IN_SEGMENT_NAME +
TableNameBuilder.extractRawTableName(tableNameWithType);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch dependabot/maven/net.openhft-chronicle-core-2.26ea0 deleted (was db3a4c5918)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/net.openhft-chronicle-core-2.26ea0 in repository https://gitbox.apache.org/repos/asf/pinot.git was db3a4c5918 Bump net.openhft:chronicle-core from 2.25ea15 to 2.25.16 The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Bump net.openhft:chronicle-core from 2.25ea15 to 2.26ea0 [pinot]
dependabot[bot] commented on PR #13274: URL: https://github.com/apache/pinot/pull/13274#issuecomment-2155478020 OK, I won't notify you again about this release, but will get in touch when a new version is available. If you'd rather skip all updates until the next major or minor version, let me know by commenting `@dependabot ignore this major version` or `@dependabot ignore this minor version`. You can also ignore all major, minor, or patch releases for a dependency by adding an [`ignore` condition](https://docs.github.com/en/code-security/supply-chain-security/configuration-options-for-dependency-updates#ignore) with the desired `update_types` to your config file. If you change your mind, just re-open this PR and I'll resolve any conflicts on it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Bump net.openhft:chronicle-core from 2.25ea15 to 2.26ea0 [pinot]
Jackie-Jiang closed pull request #13274: Bump net.openhft:chronicle-core from 2.25ea15 to 2.26ea0 URL: https://github.com/apache/pinot/pull/13274 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch master updated: Extend compatibility verifier operation timeout from 1m to 2m to reduce flakiness (#13338)
This is an automated email from the ASF dual-hosted git repository.
jackie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pinot.git
The following commit(s) were added to refs/heads/master by this push:
new e5decf35f9 Extend compatibility verifier operation timeout from 1m to
2m to reduce flakiness (#13338)
e5decf35f9 is described below
commit e5decf35f9c68676ec99ab73853c04c792d0bc04
Author: Xiaotian (Jackie) Jiang <1751+jackie-ji...@users.noreply.github.com>
AuthorDate: Fri Jun 7 12:52:52 2024 -0700
Extend compatibility verifier operation timeout from 1m to 2m to reduce
flakiness (#13338)
---
.../src/main/java/org/apache/pinot/compat/SegmentOp.java| 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/pinot-compatibility-verifier/src/main/java/org/apache/pinot/compat/SegmentOp.java
b/pinot-compatibility-verifier/src/main/java/org/apache/pinot/compat/SegmentOp.java
index df53fd379e..95156f02c9 100644
---
a/pinot-compatibility-verifier/src/main/java/org/apache/pinot/compat/SegmentOp.java
+++
b/pinot-compatibility-verifier/src/main/java/org/apache/pinot/compat/SegmentOp.java
@@ -65,7 +65,7 @@ import org.slf4j.LoggerFactory;
public class SegmentOp extends BaseOp {
private static final Logger LOGGER =
LoggerFactory.getLogger(SegmentOp.class);
private static final FileFormat DEFAULT_FILE_FORMAT = FileFormat.CSV;
- private static final int DEFAULT_MAX_SLEEP_TIME_MS = 6;
+ private static final int DEFAULT_MAX_SLEEP_TIME_MS = 12;
private static final int DEFAULT_SLEEP_INTERVAL_MS = 1000;
public enum Op {
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Extend compatibility verifier operation timeout from 1m to 2m to reduce flakiness [pinot]
Jackie-Jiang merged PR #13338: URL: https://github.com/apache/pinot/pull/13338 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch master updated: Fix flakiness of ControllerPeriodicTasksIntegrationTest (#13337)
This is an automated email from the ASF dual-hosted git repository.
jackie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pinot.git
The following commit(s) were added to refs/heads/master by this push:
new c418442c49 Fix flakiness of ControllerPeriodicTasksIntegrationTest
(#13337)
c418442c49 is described below
commit c418442c498865c35b5290ecc41f38743b3f0bcd
Author: Xiaotian (Jackie) Jiang <1751+jackie-ji...@users.noreply.github.com>
AuthorDate: Fri Jun 7 12:52:42 2024 -0700
Fix flakiness of ControllerPeriodicTasksIntegrationTest (#13337)
---
.../ControllerPeriodicTasksIntegrationTest.java| 122 +++--
1 file changed, 63 insertions(+), 59 deletions(-)
diff --git
a/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/ControllerPeriodicTasksIntegrationTest.java
b/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/ControllerPeriodicTasksIntegrationTest.java
index 03a2b6a000..9e58028146 100644
---
a/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/ControllerPeriodicTasksIntegrationTest.java
+++
b/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/ControllerPeriodicTasksIntegrationTest.java
@@ -27,6 +27,7 @@ import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
+import javax.annotation.Nullable;
import org.apache.commons.io.FileUtils;
import org.apache.helix.model.IdealState;
import org.apache.helix.model.InstanceConfig;
@@ -62,10 +63,10 @@ import static org.testng.Assert.assertTrue;
* The intention of these tests is not to test functionality of daemons, but
simply to check that they run as expected
* and process the tables when the controller starts.
*/
+// TODO: Add tests for other ControllerPeriodicTasks (RetentionManager,
RealtimeSegmentValidationManager).
public class ControllerPeriodicTasksIntegrationTest extends
BaseClusterIntegrationTestSet {
private static final int PERIODIC_TASK_INITIAL_DELAY_SECONDS = 30;
- private static final int PERIODIC_TASK_FREQUENCY_SECONDS = 5;
- private static final String PERIODIC_TASK_FREQUENCY = "5s";
+ private static final String PERIODIC_TASK_FREQUENCY_PERIOD = "5s";
private static final String PERIODIC_TASK_WAIT_FOR_PUSH_TIME_PERIOD = "5s";
private static final int NUM_REPLICAS = 2;
@@ -115,22 +116,20 @@ public class ControllerPeriodicTasksIntegrationTest
extends BaseClusterIntegrati
Map properties = getDefaultControllerConfiguration();
properties.put(ControllerConf.CLUSTER_TENANT_ISOLATION_ENABLE, false);
-properties
-
.put(ControllerPeriodicTasksConf.STATUS_CHECKER_INITIAL_DELAY_IN_SECONDS,
PERIODIC_TASK_INITIAL_DELAY_SECONDS);
-
properties.put(ControllerPeriodicTasksConf.DEPRECATED_STATUS_CHECKER_FREQUENCY_IN_SECONDS,
-PERIODIC_TASK_FREQUENCY_SECONDS);
-
properties.put(ControllerPeriodicTasksConf.DEPRECATED_REALTIME_SEGMENT_RELOCATION_INITIAL_DELAY_IN_SECONDS,
+
properties.put(ControllerPeriodicTasksConf.STATUS_CHECKER_INITIAL_DELAY_IN_SECONDS,
PERIODIC_TASK_INITIAL_DELAY_SECONDS);
-properties
-
.put(ControllerPeriodicTasksConf.DEPRECATED_REALTIME_SEGMENT_RELOCATOR_FREQUENCY,
PERIODIC_TASK_FREQUENCY);
+
properties.put(ControllerPeriodicTasksConf.STATUS_CHECKER_FREQUENCY_PERIOD,
PERIODIC_TASK_FREQUENCY_PERIOD);
+
properties.put(ControllerPeriodicTasksConf.SEGMENT_RELOCATOR_INITIAL_DELAY_IN_SECONDS,
+PERIODIC_TASK_INITIAL_DELAY_SECONDS);
+
properties.put(ControllerPeriodicTasksConf.SEGMENT_RELOCATOR_FREQUENCY_PERIOD,
PERIODIC_TASK_FREQUENCY_PERIOD);
properties.put(ControllerPeriodicTasksConf.BROKER_RESOURCE_VALIDATION_INITIAL_DELAY_IN_SECONDS,
PERIODIC_TASK_INITIAL_DELAY_SECONDS);
-
properties.put(ControllerPeriodicTasksConf.DEPRECATED_BROKER_RESOURCE_VALIDATION_FREQUENCY_IN_SECONDS,
-PERIODIC_TASK_FREQUENCY_SECONDS);
+
properties.put(ControllerPeriodicTasksConf.BROKER_RESOURCE_VALIDATION_FREQUENCY_PERIOD,
+PERIODIC_TASK_FREQUENCY_PERIOD);
properties.put(ControllerPeriodicTasksConf.OFFLINE_SEGMENT_INTERVAL_CHECKER_INITIAL_DELAY_IN_SECONDS,
PERIODIC_TASK_INITIAL_DELAY_SECONDS);
-
properties.put(ControllerPeriodicTasksConf.DEPRECATED_OFFLINE_SEGMENT_INTERVAL_CHECKER_FREQUENCY_IN_SECONDS,
-PERIODIC_TASK_FREQUENCY_SECONDS);
+
properties.put(ControllerPeriodicTasksConf.OFFLINE_SEGMENT_INTERVAL_CHECKER_FREQUENCY_PERIOD,
+PERIODIC_TASK_FREQUENCY_PERIOD);
properties.put(ControllerPeriodicTasksConf.STATUS_CHECKER_WAIT_FOR_PUSH_TIME_PERIOD,
PERIODIC_TASK_WAIT_FOR_PUSH_TIME_PERIOD);
@@ -160,8 +159,8 @@ public class ControllerPeriodicTasksIntegrationTest extends
BaseClusterIntegrati
addTableConfig(createRealtimeTableConfig(realtimeAvroFiles.get(0)));
// Create and upload segments
-ClusterIntegrationTestUtils
-.buildSegmentsFromAvro(offlineAvr
Re: [I] [Flaky test] ControllerPeriodicTasksIntegrationTest.testRealtimeSegmentRelocator [pinot]
Jackie-Jiang closed issue #13263: [Flaky test] ControllerPeriodicTasksIntegrationTest.testRealtimeSegmentRelocator URL: https://github.com/apache/pinot/issues/13263 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Fix flakiness of ControllerPeriodicTasksIntegrationTest [pinot]
Jackie-Jiang merged PR #13337: URL: https://github.com/apache/pinot/pull/13337 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[I] [Flaky-test] Revisit port assignment in ClusterTest [pinot]
Jackie-Jiang opened a new issue, #13339: URL: https://github.com/apache/pinot/issues/13339 - Track all assigned ports to avoid the same port being assigned twice - Do not add random number for the preferred port (introduced in #11861) ``` Error: Failures: Error: ExactlyOnceKafkaRealtimeClusterIntegrationTest>BaseRealtimeClusterIntegrationTest.setUp:51->ClusterTest.startServer:272->ClusterTest.startServers:280->ClusterTest.startOneServer:288 » Runtime java.io.IOException: Failed to bind to address 0.0.0.0/0.0.0.0:9844 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [I] Pulsar test instability [pinot]
Jackie-Jiang commented on issue #8537: URL: https://github.com/apache/pinot/issues/8537#issuecomment-2155442056 Closing this one and track it in #13008 which contains more details -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [I] Pulsar test instability [pinot]
Jackie-Jiang closed issue #8537: Pulsar test instability URL: https://github.com/apache/pinot/issues/8537 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Extend compatibility verifier operation timeout from 1m to 2m to reduce flakiness [pinot]
codecov-commenter commented on PR #13338: URL: https://github.com/apache/pinot/pull/13338#issuecomment-2155390120 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/13338?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report All modified and coverable lines are covered by tests :white_check_mark: > Project coverage is 0.00%. Comparing base [(`59551e4`)](https://app.codecov.io/gh/apache/pinot/commit/59551e45224f1535c4863fd577622b37366ccc97?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`dcd70bd`)](https://app.codecov.io/gh/apache/pinot/commit/dcd70bde757a4ada575df8e065328d8665cd08bc?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache). > Report is 576 commits behind head on master. Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #13338 +/- ## = - Coverage 61.75%0.00% -61.76% = Files 2436 2470 +34 Lines133233 135991 +2758 Branches 2063621101 +465 = - Hits 822740-82274 - Misses44911 135991+91080 + Partials 60480 -6048 ``` | [Flag](https://app.codecov.io/gh/apache/pinot/pull/13338/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | | |---|---|---| | [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/13338/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | | [integration](https://app.codecov.io/gh/apache/pinot/pull/13338/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (-0.01%)` | :arrow_down: | | [integration1](https://app.codecov.io/gh/apache/pinot/pull/13338/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | | [integration2](https://app.codecov.io/gh/apache/pinot/pull/13338/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (ø)` | | | [java-11](https://app.codecov.io/gh/apache/pinot/pull/13338/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | | [java-21](https://app.codecov.io/gh/apache/pinot/pull/13338/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (-61.63%)` | :arrow_down: | | [skip-bytebuffers-false](https://app.codecov.io/gh/apache/pinot/pull/13338/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (-61.75%)` | :arrow_down: | | [skip-bytebuffers-true](https://app.codecov.io/gh/apache/pinot/pull/13338/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (-27.73%)` | :arrow_down: | | [temurin](https://app.codecov.io/gh/apache/pinot/pull/13338/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (-61.76%)` | :arrow_down: | | [unittests](https://app.codecov.io/gh/apache/pinot/pull/13338/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | | [unittests1](https://app.codecov.io/gh/apache/pinot/pull/13338/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | | [unittests2](https://app.codecov.io/gh/apache/pinot/pull/13338/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more. [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/13338?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
Re: [I] Use Foreign Memory API, introduced in Java 22 [pinot]
aditya0811 commented on issue #12809: URL: https://github.com/apache/pinot/issues/12809#issuecomment-2155387509 I will leave the implementation part, for now, for this discussion. Let me try out things in these lines > how to mix code that requires modern versions of Java in the CI, packaging and runtime code. I am trying to understand the steps in CI process. However, was unable to find the exact workflow file(.yml in .github/workflows) if we use it. Wanted to understand how we are packaging pinot before release. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Fix flakiness of ControllerPeriodicTasksIntegrationTest [pinot]
codecov-commenter commented on PR #13337: URL: https://github.com/apache/pinot/pull/13337#issuecomment-2155383866 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/13337?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report All modified and coverable lines are covered by tests :white_check_mark: > Project coverage is 0.00%. Comparing base [(`59551e4`)](https://app.codecov.io/gh/apache/pinot/commit/59551e45224f1535c4863fd577622b37366ccc97?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`0f2c00d`)](https://app.codecov.io/gh/apache/pinot/commit/0f2c00dcf8e2aacf7a2421db8658aa005b36244c?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache). > Report is 576 commits behind head on master. Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #13337 +/- ## = - Coverage 61.75%0.00% -61.76% = Files 2436 2470 +34 Lines133233 135993 +2760 Branches 2063621101 +465 = - Hits 822740-82274 - Misses44911 135993+91082 + Partials 60480 -6048 ``` | [Flag](https://app.codecov.io/gh/apache/pinot/pull/13337/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | | |---|---|---| | [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/13337/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | | [integration](https://app.codecov.io/gh/apache/pinot/pull/13337/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (-0.01%)` | :arrow_down: | | [integration1](https://app.codecov.io/gh/apache/pinot/pull/13337/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | | [integration2](https://app.codecov.io/gh/apache/pinot/pull/13337/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (ø)` | | | [java-11](https://app.codecov.io/gh/apache/pinot/pull/13337/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (-61.71%)` | :arrow_down: | | [java-21](https://app.codecov.io/gh/apache/pinot/pull/13337/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (-61.63%)` | :arrow_down: | | [skip-bytebuffers-false](https://app.codecov.io/gh/apache/pinot/pull/13337/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (-61.75%)` | :arrow_down: | | [skip-bytebuffers-true](https://app.codecov.io/gh/apache/pinot/pull/13337/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (-27.73%)` | :arrow_down: | | [temurin](https://app.codecov.io/gh/apache/pinot/pull/13337/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (-61.76%)` | :arrow_down: | | [unittests](https://app.codecov.io/gh/apache/pinot/pull/13337/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | | [unittests1](https://app.codecov.io/gh/apache/pinot/pull/13337/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | | [unittests2](https://app.codecov.io/gh/apache/pinot/pull/13337/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more. [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/13337?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaig
Re: [PR] Reposition query submission spot for adaptive server selection [pinot]
Jackie-Jiang commented on code in PR #13327:
URL: https://github.com/apache/pinot/pull/13327#discussion_r1631594204
##
pinot-core/src/main/java/org/apache/pinot/core/transport/AsyncQueryResponse.java:
##
@@ -56,13 +56,17 @@ public AsyncQueryResponse(QueryRouter queryRouter, long
requestId, Set(HashUtil.getHashMapCapacity(numServersQueried));
+_serverRoutingStatsManager = serverRoutingStatsManager;
for (ServerRoutingInstance serverRoutingInstance : serversQueried) {
+ // Record stats related to query submission just before sending the
request. Otherwise, if the response is
+ // received immediately, there's a possibility of updating query
response stats before updating query
+ // submission stats.
+ _serverRoutingStatsManager.recordStatsAfterQuerySubmission(requestId,
serverRoutingInstance.getInstanceId());
Review Comment:
Per the method name, this should be put within either
`markRequestSubmitted()` or `markRequestSent()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch master updated (dae4fbd48e -> 082ef4079f)
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from dae4fbd48e Bump com.azure:azure-core from 1.49.0 to 1.49.1 (#1) add 082ef4079f Update metadata.properties to have schema max-length (#13187) No new revisions were added by this update. Summary of changes: .../test/java/org/apache/pinot/core/util/CrcUtilsTest.java | 4 ++-- .../integration/tests/OfflineClusterIntegrationTest.java| 2 +- .../segment/creator/impl/SegmentColumnarIndexCreator.java | 7 +++ .../main/java/org/apache/pinot/segment/spi/V1Constants.java | 2 ++ .../segment/spi/index/metadata/ColumnMetadataImpl.java | 10 -- .../org/apache/pinot/server/api/TablesResourceTest.java | 13 +++-- .../java/org/apache/pinot/spi/data/MetricFieldSpec.java | 5 + 7 files changed, 32 insertions(+), 11 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Update metadata.properties to have schema max-length [pinot]
Jackie-Jiang merged PR #13187: URL: https://github.com/apache/pinot/pull/13187 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch dependabot/maven/net.openhft-chronicle-core-2.26ea0 updated (fc7eb4a0a8 -> db3a4c5918)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/net.openhft-chronicle-core-2.26ea0 in repository https://gitbox.apache.org/repos/asf/pinot.git discard fc7eb4a0a8 Bump net.openhft:chronicle-core from 2.25ea15 to 2.25.16 add 3d07b90bd7 Fix Logging Location for CPU-Based Query Killing (#13318) add f4a4bf389a Custom configuration property reader for segment metadata files (#12440) add 6290bc001a Interning for OnHeapByteDictionary (#12342) add ba2c373637 Bump org.apache.maven.plugins:maven-checkstyle-plugin (#13321) add 2c59fbec96 Bump pulsar.version from 3.2.3 to 3.3.0 (#13322) add b602df5271 Bump software.amazon.awssdk:bom from 2.25.66 to 2.25.67 (#13323) add 1d1d25dc0f Update getValidDocIdsMetadataFromServer to make call in batches to servers and other bug fixes (#13314) add fb64ec29e9 Fix few metric rules which were affected by the database prefix handling (#13290) add 0f92742b45 Multi stage metrics (#13035) add fc358a87fc Add offset based lag metrics (#13298) add 4356583592 [Multi-stage] Clean up RelNode to Operator handling (#13325) add 1e0dc300f9 Bump com.nimbusds:nimbus-jose-jwt from 9.39.3 to 9.40 (#13334) add cae5e4e68a Bump lucene.version from 9.10.0 to 9.11.0 (#13328) add dbeeaaf25e Add a benchmark to measure multi-stage block serde cost (#13336) add b78de5375a Bump io.github.hakky54:sslcontext-kickstart-for-netty (#13329) add 0137e32de0 Bump software.amazon.awssdk:bom from 2.25.67 to 2.25.68 (#13330) add 5d1a04288c Bump com.microsoft.azure:msal4j from 1.15.0 to 1.15.1 (#13331) add 18a6a0df85 Bump com.azure:azure-core-http-netty from 1.15.0 to 1.15.1 (#13332) add dae4fbd48e Bump com.azure:azure-core from 1.49.0 to 1.49.1 (#1) add db3a4c5918 Bump net.openhft:chronicle-core from 2.25ea15 to 2.25.16 This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (fc7eb4a0a8) \ N -- N -- N refs/heads/dependabot/maven/net.openhft-chronicle-core-2.26ea0 (db3a4c5918) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: .../jmx_prometheus_javaagent/configs/broker.yml| 12 + .../configs/controller.yml | 7 + .../jmx_prometheus_javaagent/configs/server.yml| 25 +- .../apache/pinot/common/metrics/ServerGauge.java | 3 +- .../apache/pinot/common/metrics/ServerMeter.java | 41 +- .../apache/pinot/common/metrics/ServerTimer.java | 31 +- pinot-common/src/main/proto/expressions.proto | 6 +- pinot-common/src/main/proto/plan.proto | 229 +++ .../pinot/common/utils/FALFInternerTest.java | 136 ++-- .../api/resources/PinotTableRestletResource.java | 8 +- .../controller/util/CompletionServiceHelper.java | 24 +- .../util/ServerSegmentMetadataReader.java | 32 +- .../pinot/controller/util/TableMetadataReader.java | 4 +- .../PerQueryCPUMemAccountantFactory.java | 6 +- .../apache/pinot/core/common/MinionConstants.java | 5 + .../manager/realtime/IngestionDelayTracker.java| 92 ++- .../realtime/RealtimeSegmentDataManager.java | 16 +- .../manager/realtime/RealtimeTableDataManager.java | 10 +- .../realtime/IngestionDelayTrackerTest.java| 65 ++ .../org/apache/pinot/perf/BenchmarkDataBlock.java | 244 +++ pinot-plugins/pinot-file-system/pinot-adls/pom.xml | 6 +- .../UpsertCompactionTaskGenerator.java | 12 +- .../calcite/rel/logical/PinotLogicalExchange.java | 1 - .../rel/logical/PinotLogicalSortExchange.java | 1 - .../calcite/rel/logical/PinotRelExchangeType.java | 4 +- .../PinotAggregateExchangeNodeInsertRule.java | 8 +- .../rel/rules/PinotJoinToDynamicBroadcastRule.java | 2 +- .../org/apache/pinot/query/QueryEnvironment.java | 3 +- .../query/parser/CalciteRexExpressionParser.java | 49 +- .../explain/PhysicalExplainPlanVisitor.java| 12 +- .../query/planner/logical/LiteralValueNode.java| 7 +- .../planner/logical/PinotLogicalQueryPlanner.java | 57 +- .../query/planner/logical/PlanFragmenter.java | 25 +- .../planner/logical/RelToPlanNodeConverter.java| 257 --- .../pinot/query/planner/logical/RexExpression.java | 84 ++- .../query/planner/logical/RexExpressionUtils.java | 13 +- .../planner/logical/ShuffleRewriteVisitor.java
Re: [PR] Bump net.openhft:chronicle-core from 2.25ea15 to 2.26ea0 [pinot]
Jackie-Jiang commented on PR #13274: URL: https://github.com/apache/pinot/pull/13274#issuecomment-2155348214 @dependabot rebase -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch dependabot/maven/com.azure-azure-core-1.49.1 deleted (was e777396d76)
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch dependabot/maven/com.azure-azure-core-1.49.1 in repository https://gitbox.apache.org/repos/asf/pinot.git was e777396d76 Bump com.azure:azure-core from 1.49.0 to 1.49.1 The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Bump com.azure:azure-core-http-netty from 1.15.0 to 1.15.1 [pinot]
Jackie-Jiang merged PR #13332: URL: https://github.com/apache/pinot/pull/13332 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Bump com.microsoft.azure:msal4j from 1.15.0 to 1.15.1 [pinot]
Jackie-Jiang merged PR #13331: URL: https://github.com/apache/pinot/pull/13331 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch master updated (18a6a0df85 -> dae4fbd48e)
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from 18a6a0df85 Bump com.azure:azure-core-http-netty from 1.15.0 to 1.15.1 (#13332) add dae4fbd48e Bump com.azure:azure-core from 1.49.0 to 1.49.1 (#1) No new revisions were added by this update. Summary of changes: pinot-plugins/pinot-file-system/pinot-adls/pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Bump com.azure:azure-core from 1.49.0 to 1.49.1 [pinot]
Jackie-Jiang merged PR #1: URL: https://github.com/apache/pinot/pull/1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch master updated (5d1a04288c -> 18a6a0df85)
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from 5d1a04288c Bump com.microsoft.azure:msal4j from 1.15.0 to 1.15.1 (#13331) add 18a6a0df85 Bump com.azure:azure-core-http-netty from 1.15.0 to 1.15.1 (#13332) No new revisions were added by this update. Summary of changes: pinot-plugins/pinot-file-system/pinot-adls/pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch dependabot/maven/com.azure-azure-core-http-netty-1.15.1 deleted (was c3ed7f101b)
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch dependabot/maven/com.azure-azure-core-http-netty-1.15.1 in repository https://gitbox.apache.org/repos/asf/pinot.git was c3ed7f101b Bump com.azure:azure-core-http-netty from 1.15.0 to 1.15.1 The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch master updated (0137e32de0 -> 5d1a04288c)
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from 0137e32de0 Bump software.amazon.awssdk:bom from 2.25.67 to 2.25.68 (#13330) add 5d1a04288c Bump com.microsoft.azure:msal4j from 1.15.0 to 1.15.1 (#13331) No new revisions were added by this update. Summary of changes: pinot-plugins/pinot-file-system/pinot-adls/pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch dependabot/maven/com.microsoft.azure-msal4j-1.15.1 deleted (was dd5f36af55)
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch dependabot/maven/com.microsoft.azure-msal4j-1.15.1 in repository https://gitbox.apache.org/repos/asf/pinot.git was dd5f36af55 Bump com.microsoft.azure:msal4j from 1.15.0 to 1.15.1 The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch master updated (b78de5375a -> 0137e32de0)
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from b78de5375a Bump io.github.hakky54:sslcontext-kickstart-for-netty (#13329) add 0137e32de0 Bump software.amazon.awssdk:bom from 2.25.67 to 2.25.68 (#13330) No new revisions were added by this update. Summary of changes: pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Bump software.amazon.awssdk:bom from 2.25.67 to 2.25.68 [pinot]
Jackie-Jiang merged PR #13330: URL: https://github.com/apache/pinot/pull/13330 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Bump io.github.hakky54:sslcontext-kickstart-for-netty from 8.3.5 to 8.3.6 [pinot]
Jackie-Jiang merged PR #13329: URL: https://github.com/apache/pinot/pull/13329 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch dependabot/maven/software.amazon.awssdk-bom-2.25.68 deleted (was 80d034ecfd)
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch dependabot/maven/software.amazon.awssdk-bom-2.25.68 in repository https://gitbox.apache.org/repos/asf/pinot.git was 80d034ecfd Bump software.amazon.awssdk:bom from 2.25.67 to 2.25.68 The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch dependabot/maven/io.github.hakky54-sslcontext-kickstart-for-netty-8.3.6 deleted (was a0fe73b2c0)
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch dependabot/maven/io.github.hakky54-sslcontext-kickstart-for-netty-8.3.6 in repository https://gitbox.apache.org/repos/asf/pinot.git was a0fe73b2c0 Bump io.github.hakky54:sslcontext-kickstart-for-netty The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch master updated (dbeeaaf25e -> b78de5375a)
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from dbeeaaf25e Add a benchmark to measure multi-stage block serde cost (#13336) add b78de5375a Bump io.github.hakky54:sslcontext-kickstart-for-netty (#13329) No new revisions were added by this update. Summary of changes: pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[PR] Extend compatibility verifier operation timeout from 1m to 2m to reduce flakiness [pinot]
Jackie-Jiang opened a new pull request, #13338: URL: https://github.com/apache/pinot/pull/13338 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch master updated (cae5e4e68a -> dbeeaaf25e)
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from cae5e4e68a Bump lucene.version from 9.10.0 to 9.11.0 (#13328) add dbeeaaf25e Add a benchmark to measure multi-stage block serde cost (#13336) No new revisions were added by this update. Summary of changes: .../org/apache/pinot/perf/BenchmarkDataBlock.java | 244 + 1 file changed, 244 insertions(+) create mode 100644 pinot-perf/src/main/java/org/apache/pinot/perf/BenchmarkDataBlock.java - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Benchmark to measure multi-stage block serde cost [pinot]
Jackie-Jiang merged PR #13336: URL: https://github.com/apache/pinot/pull/13336 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[PR] Fix flakiness of ControllerPeriodicTasksIntegrationTest [pinot]
Jackie-Jiang opened a new pull request, #13337: URL: https://github.com/apache/pinot/pull/13337 Fix #13263 - Extend the wait time to reduce flakiness - Use the new config keys instead of the deprecated keys (deprecated keys are tested within `ControllerConfTest`) - Extract common gauge value comparison logic to improve readability -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Fix MergeRollup generator bugs [pinot]
sajjad-moradi commented on code in PR #12092:
URL: https://github.com/apache/pinot/pull/12092#discussion_r1631561479
##
pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/mergerollup/MergeRollupTaskGenerator.java:
##
@@ -345,20 +346,24 @@ public List
generateTasks(List tableConfigs) {
// Haven't find the first overlapping segment, continue to the
next segment
} else {
// Has gone through all overlapping segments for current bucket
-if (hasUnmergedSegments && areAllSegmentsReadyToMerge) {
+if (hasUnmergedSegments && isAllSegmentsReadyToMerge) {
// Add the bucket if there are unmerged segments
selectedSegmentsForAllBuckets.add(selectedSegmentsForBucket);
}
-if (selectedSegmentsForAllBuckets.size() == maxNumParallelBuckets
|| hasSpilledOverData) {
+if (selectedSegmentsForAllBuckets.size() == maxNumParallelBuckets
+|| hasSpilledOverData && hasUnmergedSegments &&
isAllSegmentsReadyToMerge) {
// If there are enough buckets or found spilled over data,
schedule merge tasks
+ // Note: the check for hasUnmergedSegments &&
isAllSegmentsReadyToMerge is needed for processAll mode.
+ // This check prevents a scenario where lower-level merge
discontinues scheduling due to the higher-level
+ // segments being considered as spilled-over data based on
lower-level criteria.
Review Comment:
Do we have a unit test covering the newly added boolean terms?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch dependabot/maven/lucene.version-9.11.0 deleted (was 91aa65dfbe)
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch dependabot/maven/lucene.version-9.11.0 in repository https://gitbox.apache.org/repos/asf/pinot.git was 91aa65dfbe Bump lucene.version from 9.10.0 to 9.11.0 The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch master updated (1e0dc300f9 -> cae5e4e68a)
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from 1e0dc300f9 Bump com.nimbusds:nimbus-jose-jwt from 9.39.3 to 9.40 (#13334) add cae5e4e68a Bump lucene.version from 9.10.0 to 9.11.0 (#13328) No new revisions were added by this update. Summary of changes: pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Bump lucene.version from 9.10.0 to 9.11.0 [pinot]
Jackie-Jiang merged PR #13328: URL: https://github.com/apache/pinot/pull/13328 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Optimize snapshot flow to only snapshot segments which have updates [pinot]
klsince commented on code in PR #13285:
URL: https://github.com/apache/pinot/pull/13285#discussion_r1631532746
##
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/BasePartitionUpsertMetadataManager.java:
##
@@ -832,8 +836,10 @@ public void takeSnapshot() {
if (!_enableSnapshot) {
return;
}
-if (!_gotFirstConsumingSegment) {
- _logger.info("Skip taking snapshot before getting the first consuming
segment");
+if (_partialUpsertHandler == null && !_gotFirstConsumingSegment) {
Review Comment:
I understood that it's safe to take snapshot for partial upsert table before
_gotFirstConsumingSegment, but wondering what's the benefits of taking snapshot
eagerly for it?
##
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeSegmentDataManager.java:
##
@@ -703,9 +703,20 @@ public void run() {
// persisted.
// Take upsert snapshot before starting consuming events
if (_partitionUpsertMetadataManager != null) {
- _partitionUpsertMetadataManager.takeSnapshot();
- // If upsertTTL is enabled, we will remove expired primary keys from
upsertMetadata after taking snapshot.
- _partitionUpsertMetadataManager.removeExpiredPrimaryKeys();
+ if (_tableConfig.getUpsertMetadataTTL() > 0) {
+// If upsertMetadataTTL is enabled, we will remove expired primary
keys from upsertMetadata
+// AFTER taking a snapshot. Taking the snapshot first is crucial
to ensure we capture the final
+// state of a particular key before it exits the TTL window.
Review Comment:
curious why it is critical to take snapshot first? The states of those
particular keys will be gone in the next snapshot, and would that cause issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [I] Proposal for a new instance-partition based routing strategy [pinot]
klsince commented on issue #13284: URL: https://github.com/apache/pinot/issues/13284#issuecomment-2155265687 > I think choosing an instance with unavailable segments is too relaxed in my opinion... The improvement included a new query option `useCompleteReplica` to allow users to choose between data completeness vs. availability. I'm +1 to support routing at partition level for better data completeness, when table is partitioned. Also makes sense to keep `SegmentPartition` separate then `InstanceGroup` can track partitionIds instead of set of segments. But to be sure, I'd assume it's going to be a new routing policy, e.g. as you called `instancePartitionReplicaGroup` rather than extending the strictReplicaPolicy, as this new routing policy would assume the table is partitioned. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Bump com.nimbusds:nimbus-jose-jwt from 9.39.3 to 9.40 [pinot]
Jackie-Jiang merged PR #13334: URL: https://github.com/apache/pinot/pull/13334 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch master updated (4356583592 -> 1e0dc300f9)
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from 4356583592 [Multi-stage] Clean up RelNode to Operator handling (#13325) add 1e0dc300f9 Bump com.nimbusds:nimbus-jose-jwt from 9.39.3 to 9.40 (#13334) No new revisions were added by this update. Summary of changes: pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch dependabot/maven/com.nimbusds-nimbus-jose-jwt-9.40 deleted (was a01c532b07)
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch dependabot/maven/com.nimbusds-nimbus-jose-jwt-9.40 in repository https://gitbox.apache.org/repos/asf/pinot.git was a01c532b07 Bump com.nimbusds:nimbus-jose-jwt from 9.39.3 to 9.40 The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] [Multi-stage] Clean up RelNode to Operator handling [pinot]
Jackie-Jiang merged PR #13325: URL: https://github.com/apache/pinot/pull/13325 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Benchmark to measure multi-stage block serde cost [pinot]
codecov-commenter commented on PR #13336: URL: https://github.com/apache/pinot/pull/13336#issuecomment-2155016617 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/13336?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report All modified and coverable lines are covered by tests :white_check_mark: > Project coverage is 46.56%. Comparing base [(`59551e4`)](https://app.codecov.io/gh/apache/pinot/commit/59551e45224f1535c4863fd577622b37366ccc97?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`02421fd`)](https://app.codecov.io/gh/apache/pinot/commit/02421fd6b683d22fa1ff6e83fe3c20f54377dbb0?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache). > Report is 566 commits behind head on master. Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #13336 +/- ## = - Coverage 61.75% 46.56% -15.19% - Complexity 207 1096 +889 = Files 2436 1939 -497 Lines133233 102921-30312 Branches 2063616557 -4079 = - Hits 8227447929-34345 - Misses4491151443 +6532 + Partials 6048 3549 -2499 ``` | [Flag](https://app.codecov.io/gh/apache/pinot/pull/13336/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | | |---|---|---| | [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/13336/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | | [integration](https://app.codecov.io/gh/apache/pinot/pull/13336/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | | [integration1](https://app.codecov.io/gh/apache/pinot/pull/13336/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | | [integration2](https://app.codecov.io/gh/apache/pinot/pull/13336/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | | [java-11](https://app.codecov.io/gh/apache/pinot/pull/13336/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `46.56% <ø> (-15.15%)` | :arrow_down: | | [java-21](https://app.codecov.io/gh/apache/pinot/pull/13336/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | | [skip-bytebuffers-false](https://app.codecov.io/gh/apache/pinot/pull/13336/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `46.56% <ø> (-15.18%)` | :arrow_down: | | [skip-bytebuffers-true](https://app.codecov.io/gh/apache/pinot/pull/13336/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | | [temurin](https://app.codecov.io/gh/apache/pinot/pull/13336/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `46.56% <ø> (-15.19%)` | :arrow_down: | | [unittests](https://app.codecov.io/gh/apache/pinot/pull/13336/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `46.56% <ø> (-15.18%)` | :arrow_down: | | [unittests1](https://app.codecov.io/gh/apache/pinot/pull/13336/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `46.56% <ø> (-0.33%)` | :arrow_down: | | [unittests2](https://app.codecov.io/gh/apache/pinot/pull/13336/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more. [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/13336?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment
Re: [I] Realtime servers reporting lag for partitions they don't own after rebalancing w/o including consuming segments [pinot]
jadami10 commented on issue #11448: URL: https://github.com/apache/pinot/issues/11448#issuecomment-2154976787 @Jackie-Jiang I just tested on 1.1, and we still see the issue @priyen-stripe reported. It's pretty easy to repro: - rebalance (do not include consuming) - force commit you'll see pinot reports ingestion lag linearly going up -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Optimize snapshot flow to only snapshot segments which have updates [pinot]
tibrewalpratik17 commented on code in PR #13285:
URL: https://github.com/apache/pinot/pull/13285#discussion_r1631296201
##
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/BasePartitionUpsertMetadataManager.java:
##
@@ -832,8 +836,10 @@ public void takeSnapshot() {
if (!_enableSnapshot) {
return;
}
-if (!_gotFirstConsumingSegment) {
- _logger.info("Skip taking snapshot before getting the first consuming
segment");
+if (_partialUpsertHandler == null && !_gotFirstConsumingSegment) {
Review Comment:
cc @klsince added this change since your last review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[PR] Benchmark to measure multi-stage block serde cost [pinot]
gortiz opened a new pull request, #13336: URL: https://github.com/apache/pinot/pull/13336 This PR adds a benchmark that measures the cost of creating, serializing and deserializing multi-stage blocks. This should be compared with results in https://github.com/apache/pinot/pull/13303 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[I] Server crashing with OOM error [pinot]
vineethvp opened a new issue, #13335:
URL: https://github.com/apache/pinot/issues/13335
Data is ingested from Kafka to realtime table with below config. It is
having replication factor 2.
**Table config**
"instanceAssignmentConfigMap": {
"CONSUMING": {
"tagPoolConfig": {
"tag": "DefaultTenant_REALTIME"
},
"replicaGroupPartitionConfig": {
"numInstances": 3
}
}
"tableIndexConfig": {
"loadMode": "MMAP",
"streamConfigs": {
"streamType": "kafka",
"stream.kafka.consumer.type": "lowlevel",
"stream.kafka.consumer.factory.class.name":
"org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
"realtime.segment.flush.threshold.rows": "0",
"realtime.segment.flush.threshold.time": "24h",
"realtime.segment.flush.threshold.segment.size": "300M",
"stream.kafka.consumer.prop.auto.offset.reset": "largest"
}
},
**Getting below error after inserting around 60M records, and server is
crashing.**
Consumed 5722 events from (rate:63.164402/s), currentOffset=389448757,
numRowsConsumedSoFar=257283, numRowsIndexedSoFar=257283
Consumed 3216 events from (rate:37.62988/s), currentOffset=376002707,
numRowsConsumedSoFar=252239, numRowsIndexedSoFar=252239
Consumed 4045 events from (rate:46.583138/s), currentOffset=376940305,
numRowsConsumedSoFar=262888, numRowsIndexedSoFar=262888
Consumed 2880 events from (rate:40.99819/s), currentOffset=372932263,
numRowsConsumedSoFar=246467, numRowsIndexedSoFar=246467
Consumed 3625 events from (rate:46.28802/s), currentOffset=381415112,
numRowsConsumedSoFar=249777, numRowsIndexedSoFar=249777
Consumed 2898 events from (rate:35.928143/s), currentOffset=374420450,
numRowsConsumedSoFar=252748, numRowsIndexedSoFar=252748
Slow query: request handler processing time: 6786, send response latency:
7946, total time to handle request: 14732
Consumed 3757 events from (rate:38.51003/s), currentOffset=383381895,
numRowsConsumedSoFar=254725, numRowsIndexedSoFar=254725
Client session timed out, have not heard from server in 20317ms for session
id 0x2000fcc5a450005
Session 0x2000fcc5a450005 for server pinot-zookeeper/172.20.29.227:2181,
Closing socket connection. Attempting reconnect except it is a
SessionExpiredException.
org.apache.zookeeper.ClientCnxn$SessionTimeoutException: Client session
timed out, have not heard from server in 20317ms for session id
0x2000fcc5a450005
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1242)
[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-1d1d25dc0f1fc1abb73d9516414168c82b116b58]
zkclient 3, zookeeper state changed ( Disconnected )
[Consumer clientId=events_REALTIME-CENTRAL-STREAMING-NW-EVENTS-21,
groupId=null] Error sending fetch request (sessionId=214284632, epoch=84) to
node 1:
org.apache.pinot.shaded.org.apache.kafka.common.errors.DisconnectException:
null
Opening socket connection to server pinot-zookeeper/172.20.29.227:2181.
SASL config status: Will not attempt to authenticate using SASL (unknown
error)
Timed out while polling results block, numBlocksMerged: 0 (query:
QueryContext{_tableName='events_REALTIME', _subquery=null,
_selectExpressions=[channel_util, cpu_util, customer_id, data, data_source,
device_id, device_type, event_type, fan_id, fan_status, is_stack_switch,
mac_address, mem_util, member_id, noise_floor, power_supply_id,
power_supply_status, radio_band, radio_type, sensor_id, sensor_temperature,
sensor_temperature_trend, site_id, stack_id, status, sub_account_id,
sub_site_id, timestamp, trend_value, upload_timestamp, upload_ts_millis,
uptime, uptime_string], _distinct=false, _aliasList=[null, null, null, null,
null, null, null, null, null, null, null, null, null, null, null, null, null,
null, null, null, null, null, null, null, null, null, null, null, null, null,
null, null, null], _filter=null, _groupByExpressions=null, _havingFilter=null,
_orderByExpressions=null, _limit=1, _offset=0,
_queryOptions={timeoutMs=1}, _expressionOverrideHints={}, _explain=false
})
Consumed 500 events from (rate:7.955576/s), currentOffset=373866452,
numRowsConsumedSoFar=257550, numRowsIndexedSoFar=257550
Consumed 1735 events from (rate:15.82827/s), currentOffset=376659150,
numRowsConsumedSoFar=252384, numRowsIndexedSoFar=252384
Consumed 3230 events from (rate:27.967546/s), currentOffset=374117225,
numRowsConsumedSoFar=248670, numRowsIndexedSoFar=248670
Socket connection established, initiating session, client:
/10.3.145.54:33952, server: pinot-zookeeper/172.20.29.227:2181
Consumed 250 events from (rate:3.7271154/s), currentOffset=380505319,
numRowsConsumedSoFar=237006, numRowsIndexedSoFar=237006
Exception in thread "events__18__5__20240607T0718Z"
java.lang.OutOfMemoryError: Java heap sp
(pinot) branch master updated: Add offset based lag metrics (#13298)
This is an automated email from the ASF dual-hosted git repository.
kharekartik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pinot.git
The following commit(s) were added to refs/heads/master by this push:
new fc358a87fc Add offset based lag metrics (#13298)
fc358a87fc is described below
commit fc358a87fc63def0b5c5a31a24bde1d9df0d9ecb
Author: Kartik Khare
AuthorDate: Fri Jun 7 19:21:32 2024 +0530
Add offset based lag metrics (#13298)
* Add offset based lag metrics
* Add tests
* Refactoring
-
Co-authored-by: Kartik Khare
---
.../apache/pinot/common/metrics/ServerGauge.java | 3 +-
.../manager/realtime/IngestionDelayTracker.java| 92 +++---
.../realtime/RealtimeSegmentDataManager.java | 16 ++--
.../manager/realtime/RealtimeTableDataManager.java | 10 ++-
.../realtime/IngestionDelayTrackerTest.java| 65 +++
5 files changed, 167 insertions(+), 19 deletions(-)
diff --git
a/pinot-common/src/main/java/org/apache/pinot/common/metrics/ServerGauge.java
b/pinot-common/src/main/java/org/apache/pinot/common/metrics/ServerGauge.java
index f0a1fdd136..072c048c55 100644
---
a/pinot-common/src/main/java/org/apache/pinot/common/metrics/ServerGauge.java
+++
b/pinot-common/src/main/java/org/apache/pinot/common/metrics/ServerGauge.java
@@ -72,7 +72,8 @@ public enum ServerGauge implements AbstractMetrics.Gauge {
END_TO_END_REALTIME_INGESTION_DELAY_MS("milliseconds", false),
// Needed to track if valid doc id snapshots are present for faster restarts
UPSERT_VALID_DOC_ID_SNAPSHOT_COUNT("upsertValidDocIdSnapshotCount", false),
- UPSERT_PRIMARY_KEYS_IN_SNAPSHOT_COUNT("upsertPrimaryKeysInSnapshotCount",
false);
+ UPSERT_PRIMARY_KEYS_IN_SNAPSHOT_COUNT("upsertPrimaryKeysInSnapshotCount",
false),
+ REALTIME_INGESTION_OFFSET_LAG("offsetLag", false);
private final String _gaugeName;
private final String _unit;
diff --git
a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/IngestionDelayTracker.java
b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/IngestionDelayTracker.java
index 84a5ea7ba0..eed1302708 100644
---
a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/IngestionDelayTracker.java
+++
b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/IngestionDelayTracker.java
@@ -34,6 +34,8 @@ import java.util.concurrent.TimeUnit;
import java.util.function.Supplier;
import org.apache.pinot.common.metrics.ServerGauge;
import org.apache.pinot.common.metrics.ServerMetrics;
+import org.apache.pinot.spi.stream.LongMsgOffset;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffset;
import org.apache.pinot.spi.utils.builder.TableNameBuilder;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -81,13 +83,23 @@ public class IngestionDelayTracker {
// Class to wrap supported timestamps collected for an ingested event
private static class IngestionTimestamps {
+private final long _firstStreamIngestionTimeMs;
+private final long _ingestionTimeMs;
IngestionTimestamps(long ingestionTimesMs, long
firstStreamIngestionTimeMs) {
_ingestionTimeMs = ingestionTimesMs;
_firstStreamIngestionTimeMs = firstStreamIngestionTimeMs;
}
-private final long _ingestionTimeMs;
-private final long _firstStreamIngestionTimeMs;
}
+
+ private static class IngestionOffsets {
+private final StreamPartitionMsgOffset _latestOffset;
+private final StreamPartitionMsgOffset _offset;
+IngestionOffsets(StreamPartitionMsgOffset offset, StreamPartitionMsgOffset
latestOffset) {
+ _offset = offset;
+ _latestOffset = latestOffset;
+}
+ }
+
// Sleep interval for scheduled executor service thread that triggers read
of ideal state
private static final int SCHEDULED_EXECUTOR_THREAD_TICK_INTERVAL_MS =
30; // 5 minutes +/- precision in timeouts
// Once a partition is marked for verification, we wait 10 minutes to pull
its ideal state.
@@ -98,6 +110,8 @@ public class IngestionDelayTracker {
// HashMap used to store ingestion time measures for all partitions active
for the current table.
private final Map
_partitionToIngestionTimestampsMap = new ConcurrentHashMap<>();
+
+ private final Map _partitionToOffsetMap = new
ConcurrentHashMap<>();
// We mark partitions that go from CONSUMING to ONLINE in
_partitionsMarkedForVerification: if they do not
// go back to CONSUMING in some period of time, we verify whether they are
still hosted in this server by reading
// ideal state. This is done with the goal of minimizing reading ideal state
for efficiency reasons.
@@ -174,6 +188,22 @@ public class IngestionDelayTracker {
return agedIngestionDelayMs;
}
+ private long getPartitionOffsetLag(IngestionOffsets offset) {
+if (offset == null) {
+ return 0;
+}
+StreamPartitionMsgOffset curre
Re: [PR] Add offset based lag metrics [pinot]
KKcorps merged PR #13298: URL: https://github.com/apache/pinot/pull/13298 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [I] Allowing empty segments with no offset advancing [pinot]
tibrewalpratik17 commented on issue #12703: URL: https://github.com/apache/pinot/issues/12703#issuecomment-2154850044 Hey @Jackie-Jiang any updates on this? Is this as straight-forward as removing these lines? https://github.com/apache/pinot/blob/0f92742b45aa9ac57439b68a63e4a9ef8d9d9d8e/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeSegmentDataManager.java#L334-L338 Saw this in one of our tables today. The table existed for more than a year and we enabled snapshot on it recently. Now since the last consuming segment (which was like months old) never received any message, the snapshot flow didn't run and removed older records via compaction. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[PR] Bump com.nimbusds:nimbus-jose-jwt from 9.39.3 to 9.40 [pinot]
dependabot[bot] opened a new pull request, #13334:
URL: https://github.com/apache/pinot/pull/13334
Bumps
[com.nimbusds:nimbus-jose-jwt](https://bitbucket.org/connect2id/nimbus-jose-jwt)
from 9.39.3 to 9.40.
Changelog
Sourced from https://bitbucket.org/connect2id/nimbus-jose-jwt/src/master/CHANGELOG.txt";>com.nimbusds:nimbus-jose-jwt's
changelog.
version 1.0 (2012-03-01)
First version based on the OpenInfoCard JWT, JWS and JWE code base.
version 1.1 (2012-03-06)
Introduces type-safe enumeration of the JSON Web Algorithms (JWA).
Refactors the JWT class.
version 1.2 (2012-03-08)
Moves JWS and JWE code into separate classes.
version 1.3 (2012-03-09)
Switches to Apache Commons Codec for Base64URL encoding and decoding
Consolidates the crypto utilities within the package.
Introduces a JWT content serialiser class.
version 1.4 (2012-03-09)
Refactoring of JWT class and JUnit tests.
version 1.5 (2012-03-18)
Switches to JSON Smart for JSON serialisation and parsing.
Introduces claims set class with JSON objects, string, Base64URL and
byte array views.
version 1.6 (2012-03-20)
Creates class for representing, serialising and parsing JSON Web Keys
(JWK).
Introduces separate class for representing JWT headers.
version 1.7 (2012-04-01)
Introduces separate classes for plain, JWS and JWE headers.
Introduces separate classes for plain, signed and encrypted JWTs.
Removes the JWTContent class.
Removes password-based (PE820) encryption support.
version 1.8 (2012-04-03)
Adds support for the ZIP JWE header parameter.
Removes unsupported algorithms from the JWA enumeration.
version 1.9 (2012-04-03)
Renames JWEHeader.{get|set}EncryptionAlgorithm() to
JWEHeader.{get|set}EncryptionMethod().
version 1.9.1 (2012-04-03)
Upgrades JSON Smart JAR to 1.1.1.
version 1.10 (2012-04-14)
Introduces serialize() method to base abstract JWT class.
version 1.11 (2012-05-13)
JWT.serialize() throws checked JWTException instead of
... (truncated)
Commits
https://bitbucket.org/connect2id/nimbus-jose-jwt/commits/493fc200e4df03dc8fb1272baa27471affb52119";>493fc20
[maven-release-plugin] prepare for next development iteration
https://bitbucket.org/connect2id/nimbus-jose-jwt/commits/2c6d225d80e335220cb51ffb64bc853bf4672640";>2c6d225
New JWTClaimsSet.Builder.serializeNullClaims(boolean) method, fixes
JWTClaims...
https://bitbucket.org/connect2id/nimbus-jose-jwt/commits/2af7c016f8c6a6295a75e93a460e6ff57a579cad";>2af7c01
[maven-release-plugin] prepare release 9.40
See full diff in https://bitbucket.org/connect2id/nimbus-jose-jwt/branches/compare/9.40..9.39.3";>compare
view
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that
have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI
passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and
block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it.
You can achieve the same result by closing it manually
- `@dependabot show ignore conditions` will show all of
the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen the PR
or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen the PR
or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the PR or
upgrade to it yourself)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to
(pinot) branch dependabot/maven/com.nimbusds-nimbus-jose-jwt-9.40 created (now a01c532b07)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/com.nimbusds-nimbus-jose-jwt-9.40 in repository https://gitbox.apache.org/repos/asf/pinot.git at a01c532b07 Bump com.nimbusds:nimbus-jose-jwt from 9.39.3 to 9.40 No new revisions were added by this update. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch dependabot/maven/com.azure-azure-core-1.49.1 created (now e777396d76)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/com.azure-azure-core-1.49.1 in repository https://gitbox.apache.org/repos/asf/pinot.git at e777396d76 Bump com.azure:azure-core from 1.49.0 to 1.49.1 No new revisions were added by this update. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[PR] Bump com.azure:azure-core from 1.49.0 to 1.49.1 [pinot]
dependabot[bot] opened a new pull request, #1: URL: https://github.com/apache/pinot/pull/1 Bumps [com.azure:azure-core](https://github.com/Azure/azure-sdk-for-java) from 1.49.0 to 1.49.1. Release notes Sourced from https://github.com/Azure/azure-sdk-for-java/releases";>com.azure:azure-core's releases. azure-core_1.49.1 1.49.1 (2024-06-06) Bugs Fixed Fixed a bug where some policies didn't close the connection appropriately. (https://redirect.github.com/Azure/azure-sdk-for-java/pull/40052";>#40052) Fixed an issue where HTTP response headers would be logged twice. (https://redirect.github.com/Azure/azure-sdk-for-java/pull/40190";>#40190) Fixed a bug where container types wouldn't resolve to using JsonSerializable serialization for values. (https://redirect.github.com/Azure/azure-sdk-for-java/pull/40112";>#40112) Fixed a bug where a generic ObjectMapper couldn't consume CloudEvent correctly. (https://redirect.github.com/Azure/azure-sdk-for-java/pull/40332";>#40332) Other Changes Dependency Updates Upgraded Reactor Core from 3.4.36 to 3.4.38. Commits https://github.com/Azure/azure-sdk-for-java/commit/2a1eaddc1e0e05b39b945493407717be9df3f03b";>2a1eadd Prepare Core libraries for June 2024 release (https://redirect.github.com/Azure/azure-sdk-for-java/issues/40498";>#40498) https://github.com/Azure/azure-sdk-for-java/commit/1527f1acf01d97f49abdefca38dfb4d58e76baac";>1527f1a Merge to main after spring cloud azure 4.19.0 released (https://redirect.github.com/Azure/azure-sdk-for-java/issues/40520";>#40520) https://github.com/Azure/azure-sdk-for-java/commit/cfe2a9970edd17ba993886391295468e9ba6653d";>cfe2a99 Synapse release for June, regenerated using autorest 4.1.17 (https://redirect.github.com/Azure/azure-sdk-for-java/issues/40518";>#40518) https://github.com/Azure/azure-sdk-for-java/commit/87d5b802ee8625e8a0df782c4bbd20818b1d9b3a";>87d5b80 mgmt, local generation for Cosmos DB 2024-05-15 Stable (https://redirect.github.com/Azure/azure-sdk-for-java/issues/40515";>#40515) https://github.com/Azure/azure-sdk-for-java/commit/85c18361ad7d10c820df90a084025be18521";>85c1836 [OpenAI] [Assistants] Assistant V2 support v2024-05-01-preview release (https://redirect.github.com/Azure/azure-sdk-for-java/issues/40";>#40... https://github.com/Azure/azure-sdk-for-java/commit/d674de54bb547a61851b9840cd303b1965eb83ae";>d674de5 mgmt, local generation for network 2023-11-01 (https://redirect.github.com/Azure/azure-sdk-for-java/issues/40476";>#40476) https://github.com/Azure/azure-sdk-for-java/commit/8b1b7dab5f6d10e46d23861c46e1a217b2120b04";>8b1b7da mgmt, local generation for Microsoft.Web 2023-12-01 (https://redirect.github.com/Azure/azure-sdk-for-java/issues/40514";>#40514) https://github.com/Azure/azure-sdk-for-java/commit/5648561dc792a993ec0fb4b041d76dc64971b9bb";>5648561 Updating the doc for maxAutoLockRenewDuration setter in all receiver and pro... https://github.com/Azure/azure-sdk-for-java/commit/8ac314bd4acbf5c518333eae622e7b362d7ff990";>8ac314b Migrate azure-security-attestation to stream-style serialization (https://redirect.github.com/Azure/azure-sdk-for-java/issues/40137";>#40137) https://github.com/Azure/azure-sdk-for-java/commit/0a1712a2801bdfbc58b9b03d219104e429122346";>0a1712a Regenerate Job Router with stream-style serialization and remove many Interna... Additional commits viewable in https://github.com/Azure/azure-sdk-for-java/compare/azure-core_1.49.0...azure-core_1.49.1";>compare view [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- Dependabot commands and options You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the
[PR] Bump com.azure:azure-core-http-netty from 1.15.0 to 1.15.1 [pinot]
dependabot[bot] opened a new pull request, #13332: URL: https://github.com/apache/pinot/pull/13332 Bumps [com.azure:azure-core-http-netty](https://github.com/Azure/azure-sdk-for-java) from 1.15.0 to 1.15.1. Release notes Sourced from https://github.com/Azure/azure-sdk-for-java/releases";>com.azure:azure-core-http-netty's releases. azure-core-http-netty_1.15.1 1.15.1 (2024-06-06) Other Changes Changed the log message for mismatched Netty versions to not state an expected version, instead just that versions weren't aligned. (https://redirect.github.com/Azure/azure-sdk-for-java/pull/40134";>#40134) Dependency Updates Upgraded azure-core from 1.49.0 to 1.49.1. Upgraded reactor-netty-http from 1.0.43 to 1.0.45. Upgraded Netty dependencies from 4.1.108.Final to 4.1.110.Final. Commits https://github.com/Azure/azure-sdk-for-java/commit/2a1eaddc1e0e05b39b945493407717be9df3f03b";>2a1eadd Prepare Core libraries for June 2024 release (https://redirect.github.com/Azure/azure-sdk-for-java/issues/40498";>#40498) https://github.com/Azure/azure-sdk-for-java/commit/1527f1acf01d97f49abdefca38dfb4d58e76baac";>1527f1a Merge to main after spring cloud azure 4.19.0 released (https://redirect.github.com/Azure/azure-sdk-for-java/issues/40520";>#40520) https://github.com/Azure/azure-sdk-for-java/commit/cfe2a9970edd17ba993886391295468e9ba6653d";>cfe2a99 Synapse release for June, regenerated using autorest 4.1.17 (https://redirect.github.com/Azure/azure-sdk-for-java/issues/40518";>#40518) https://github.com/Azure/azure-sdk-for-java/commit/87d5b802ee8625e8a0df782c4bbd20818b1d9b3a";>87d5b80 mgmt, local generation for Cosmos DB 2024-05-15 Stable (https://redirect.github.com/Azure/azure-sdk-for-java/issues/40515";>#40515) https://github.com/Azure/azure-sdk-for-java/commit/85c18361ad7d10c820df90a084025be18521";>85c1836 [OpenAI] [Assistants] Assistant V2 support v2024-05-01-preview release (https://redirect.github.com/Azure/azure-sdk-for-java/issues/40";>#40... https://github.com/Azure/azure-sdk-for-java/commit/d674de54bb547a61851b9840cd303b1965eb83ae";>d674de5 mgmt, local generation for network 2023-11-01 (https://redirect.github.com/Azure/azure-sdk-for-java/issues/40476";>#40476) https://github.com/Azure/azure-sdk-for-java/commit/8b1b7dab5f6d10e46d23861c46e1a217b2120b04";>8b1b7da mgmt, local generation for Microsoft.Web 2023-12-01 (https://redirect.github.com/Azure/azure-sdk-for-java/issues/40514";>#40514) https://github.com/Azure/azure-sdk-for-java/commit/5648561dc792a993ec0fb4b041d76dc64971b9bb";>5648561 Updating the doc for maxAutoLockRenewDuration setter in all receiver and pro... https://github.com/Azure/azure-sdk-for-java/commit/8ac314bd4acbf5c518333eae622e7b362d7ff990";>8ac314b Migrate azure-security-attestation to stream-style serialization (https://redirect.github.com/Azure/azure-sdk-for-java/issues/40137";>#40137) https://github.com/Azure/azure-sdk-for-java/commit/0a1712a2801bdfbc58b9b03d219104e429122346";>0a1712a Regenerate Job Router with stream-style serialization and remove many Interna... Additional commits viewable in https://github.com/Azure/azure-sdk-for-java/compare/azure-core_1.15.0...azure-core-http-netty_1.15.1";>compare view [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- Dependabot commands and options You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor
(pinot) branch dependabot/maven/com.azure-azure-core-http-netty-1.15.1 created (now c3ed7f101b)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/com.azure-azure-core-http-netty-1.15.1 in repository https://gitbox.apache.org/repos/asf/pinot.git at c3ed7f101b Bump com.azure:azure-core-http-netty from 1.15.0 to 1.15.1 No new revisions were added by this update. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch dependabot/maven/com.microsoft.azure-msal4j-1.15.1 created (now dd5f36af55)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/com.microsoft.azure-msal4j-1.15.1 in repository https://gitbox.apache.org/repos/asf/pinot.git at dd5f36af55 Bump com.microsoft.azure:msal4j from 1.15.0 to 1.15.1 No new revisions were added by this update. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch dependabot/maven/software.amazon.awssdk-bom-2.25.68 created (now 80d034ecfd)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/software.amazon.awssdk-bom-2.25.68 in repository https://gitbox.apache.org/repos/asf/pinot.git at 80d034ecfd Bump software.amazon.awssdk:bom from 2.25.67 to 2.25.68 No new revisions were added by this update. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[PR] Bump com.microsoft.azure:msal4j from 1.15.0 to 1.15.1 [pinot]
dependabot[bot] opened a new pull request, #13331: URL: https://github.com/apache/pinot/pull/13331 Bumps [com.microsoft.azure:msal4j](https://github.com/AzureAD/microsoft-authentication-library-for-java) from 1.15.0 to 1.15.1. Changelog Sourced from https://github.com/AzureAD/microsoft-authentication-library-for-java/blob/dev/changelog.txt";>com.microsoft.azure:msal4j's changelog. Version 1.15.1 Fix to correctly use custom SSLSocketFactory (https://redirect.github.com/AzureAD/microsoft-authentication-library-for-java/issues/821";>#821) Add public API to determine the Managed Identity environment (https://redirect.github.com/AzureAD/microsoft-authentication-library-for-java/issues/823";>#823) Commits See full diff in https://github.com/AzureAD/microsoft-authentication-library-for-java/commits";>compare view [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- Dependabot commands and options You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[PR] Bump software.amazon.awssdk:bom from 2.25.67 to 2.25.68 [pinot]
dependabot[bot] opened a new pull request, #13330: URL: https://github.com/apache/pinot/pull/13330 Bumps software.amazon.awssdk:bom from 2.25.67 to 2.25.68. [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- Dependabot commands and options You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[PR] Bump io.github.hakky54:sslcontext-kickstart-for-netty from 8.3.5 to 8.3.6 [pinot]
dependabot[bot] opened a new pull request, #13329: URL: https://github.com/apache/pinot/pull/13329 Bumps [io.github.hakky54:sslcontext-kickstart-for-netty](https://github.com/Hakky54/sslcontext-kickstart) from 8.3.5 to 8.3.6. Changelog Sourced from https://github.com/Hakky54/sslcontext-kickstart/blob/master/CHANGELOG.md";>io.github.hakky54:sslcontext-kickstart-for-netty's changelog. v8.3.6 Bumped dependencies Enhanced logging within LoggingX509ExtendedTrustManager Exposed JDK (aka cacert) keystore and System property derived keystore in KeyStoreUtils Commits https://github.com/Hakky54/sslcontext-kickstart/commit/c4497e76f80e95df25fabaef8644db7f1b503c60";>c4497e7 [maven-release-plugin] prepare release v8.3.6 https://github.com/Hakky54/sslcontext-kickstart/commit/296514f94735da17c757c1c6171a540ea6a25481";>296514f Updated docs https://github.com/Hakky54/sslcontext-kickstart/commit/f79b2dcb3d8af972e01d7a1b3e83c426269d9fe8";>f79b2dc Bump org.apache.maven.plugins:maven-enforcer-plugin from 3.4.1 to 3.5.0 https://github.com/Hakky54/sslcontext-kickstart/commit/a72d2c4629e24792bcaadb8f52e10c25f0ebdafe";>a72d2c4 Bump org.apache.maven.plugins:maven-javadoc-plugin from 3.6.3 to 3.7.0 https://github.com/Hakky54/sslcontext-kickstart/commit/74c16ac49d00af532411405042d81dc4c22859cd";>74c16ac Bump org.sonatype.plugins:nexus-staging-maven-plugin https://github.com/Hakky54/sslcontext-kickstart/commit/f2ae908532ddb9a1bdc3e7de23c3c3cd3774e333";>f2ae908 Bump version.byte-buddy from 1.14.16 to 1.14.17 https://github.com/Hakky54/sslcontext-kickstart/commit/f9cdbcd74556be06055f094bde709f4bf30377b4";>f9cdbcd Added additional public api's to the documentation https://github.com/Hakky54/sslcontext-kickstart/commit/eab4dca23e529ac3c6c03d0c3740d0173c33f5e3";>eab4dca Removed printing java version https://github.com/Hakky54/sslcontext-kickstart/commit/d86e0098632ff83ba451b5478a6ddd2f862b5b53";>d86e009 Added maxxedev as contributor https://github.com/Hakky54/sslcontext-kickstart/commit/1f62d5831d314965a75c29b76afdd5a1012a6c5c";>1f62d58 Fixed typo Additional commits viewable in https://github.com/Hakky54/sslcontext-kickstart/compare/v8.3.5...v8.3.6";>compare view [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- Dependabot commands and options You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch dependabot/maven/io.github.hakky54-sslcontext-kickstart-for-netty-8.3.6 created (now a0fe73b2c0)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/io.github.hakky54-sslcontext-kickstart-for-netty-8.3.6 in repository https://gitbox.apache.org/repos/asf/pinot.git at a0fe73b2c0 Bump io.github.hakky54:sslcontext-kickstart-for-netty No new revisions were added by this update. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch dependabot/maven/lucene.version-9.11.0 created (now 91aa65dfbe)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/lucene.version-9.11.0 in repository https://gitbox.apache.org/repos/asf/pinot.git at 91aa65dfbe Bump lucene.version from 9.10.0 to 9.11.0 No new revisions were added by this update. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[PR] Bump lucene.version from 9.10.0 to 9.11.0 [pinot]
dependabot[bot] opened a new pull request, #13328: URL: https://github.com/apache/pinot/pull/13328 Bumps `lucene.version` from 9.10.0 to 9.11.0. Updates `org.apache.lucene:lucene-backward-codecs` from 9.10.0 to 9.11.0 Updates `org.apache.lucene:lucene-core` from 9.10.0 to 9.11.0 Updates `org.apache.lucene:lucene-queryparser` from 9.10.0 to 9.11.0 Updates `org.apache.lucene:lucene-analysis-common` from 9.10.0 to 9.11.0 Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- Dependabot commands and options You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [I] Proposal for a new instance-partition based routing strategy [pinot]
tibrewalpratik17 commented on issue #13284: URL: https://github.com/apache/pinot/issues/13284#issuecomment-2154656272 > This reminded me of an improvement I tried for strictReplicaGroup (https://github.com/apache/pinot/pull/11847) but didn't finish. As you mentioned in the issue description, it's a bit too rigid to skip the instance if any one of segments hosted on it was unavailable, as often we'd have to skip all instances, and reporting that a huge number of segments were unavailable (which was kinda misleading). Yes this is the exact issue we faced. Very well summarised! > Basically the improvement I was trying to add was to pick an instance, even though it has unavailable segments and reported status of the unavailable segments back. The key abstraction in that PR was InstanceGroup, which caches the mapping from a set of instances to a set of segments on them. With replica group assignment, the set of instances should host the same set of segments, but some instances might have unavailable segments and some instance might be fine. The mapping info is updated whenever IS/EV gets updated. While selecting instances, InstanceGroup is used to quickly identify a instance. Hmm, I think choosing an instance with unavailable segments is too relaxed in my opinion. This approach could be perceived as data loss by customers if, for example, they are looking for a UUID and the segment containing that UUID is unavailable. Although we return a list of unavailable segments, there's no observability that the unavailable segment(s) contains the UUID or not, which can lead to further confusion. It would be better to route to the available segment in the other replica, what do you think? However, if we handle this at the segment level, the query fanout might increase significantly. Therefore, we could maintain the granularity at the partition level. Alternatively, we could introduce another segment-level routing strategy. Note: The goal should be to route to a single replica group wherever possible for optimal performance. > The InstanceGroup in the PR simply tracks the segments in a Set, but we can group segments further by their partitions, then we may do server selection based on Instance-Partition as proposed here. Can we keep the `InstanceGroup` mapping and the `SegmentPartition` mapping separate? When reassigning instances, there's no need to update the segment-to-partition information. Updating the segment-partition cache is only necessary during segment deletion or addition, so it's best to keep them separate. This approach allows us to easily reuse these mappings independently based on different routing strategies. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Add offset based lag metrics [pinot]
KKcorps commented on code in PR #13298:
URL: https://github.com/apache/pinot/pull/13298#discussion_r1631045755
##
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/IngestionDelayTracker.java:
##
@@ -174,6 +188,22 @@ private long getIngestionDelayMs(long ingestionTimeMs) {
return agedIngestionDelayMs;
}
+ private long getPartitionOffsetLag(IngestionOffsets offset) {
+if (offset == null) {
+ return 0;
+}
+StreamPartitionMsgOffset msgOffset = offset._offset;
+StreamPartitionMsgOffset latestOffset = offset._latestOffset;
+
+// Compute aged delay for current partition
+// TODO: Support other types of offsets
+if (!(msgOffset instanceof LongMsgOffset && latestOffset instanceof
LongMsgOffset)) {
+ return 0;
Review Comment:
Other than this, Kinesis returns a BigInteger offset and Pulsar returns a
string offset that is composed of 3 longs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Add offset based lag metrics [pinot]
KKcorps commented on code in PR #13298:
URL: https://github.com/apache/pinot/pull/13298#discussion_r1631036184
##
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/IngestionDelayTracker.java:
##
@@ -88,6 +90,16 @@ private static class IngestionTimestamps {
private final long _ingestionTimeMs;
private final long _firstStreamIngestionTimeMs;
}
+
+ private static class IngestionOffsets {
+IngestionOffsets(StreamPartitionMsgOffset offset, StreamPartitionMsgOffset
latestOffset) {
+ _offset = offset;
+ _latestOffset = latestOffset;
+}
+private final StreamPartitionMsgOffset _offset;
+private final StreamPartitionMsgOffset _latestOffset;
Review Comment:
yeah, i was just following existing style. changing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Add offset based lag metrics [pinot]
KKcorps commented on code in PR #13298:
URL: https://github.com/apache/pinot/pull/13298#discussion_r1631035110
##
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/IngestionDelayTracker.java:
##
@@ -174,6 +188,22 @@ private long getIngestionDelayMs(long ingestionTimeMs) {
return agedIngestionDelayMs;
}
+ private long getPartitionOffsetLag(IngestionOffsets offset) {
+if (offset == null) {
+ return 0;
+}
+StreamPartitionMsgOffset msgOffset = offset._offset;
Review Comment:
yes it is, let me rename this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Add offset based lag metrics [pinot]
KKcorps commented on code in PR #13298:
URL: https://github.com/apache/pinot/pull/13298#discussion_r1631034832
##
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/IngestionDelayTracker.java:
##
@@ -174,6 +188,22 @@ private long getIngestionDelayMs(long ingestionTimeMs) {
return agedIngestionDelayMs;
}
+ private long getPartitionOffsetLag(IngestionOffsets offset) {
+if (offset == null) {
Review Comment:
yeah we pass that when we want to reset the metric
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Add offset based lag metrics [pinot]
KKcorps commented on code in PR #13298:
URL: https://github.com/apache/pinot/pull/13298#discussion_r1631032269
##
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/IngestionDelayTracker.java:
##
@@ -174,6 +188,22 @@ private long getIngestionDelayMs(long ingestionTimeMs) {
return agedIngestionDelayMs;
}
+ private long getPartitionOffsetLag(IngestionOffsets offset) {
Review Comment:
Yeah that's a good suggestion. I also don't feel that casting the LongOffset
is a good idea tbh and instead should be handled by consumer classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Add offset based lag metrics [pinot]
KKcorps commented on code in PR #13298:
URL: https://github.com/apache/pinot/pull/13298#discussion_r1631032269
##
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/IngestionDelayTracker.java:
##
@@ -174,6 +188,22 @@ private long getIngestionDelayMs(long ingestionTimeMs) {
return agedIngestionDelayMs;
}
+ private long getPartitionOffsetLag(IngestionOffsets offset) {
Review Comment:
Yeah that's a good suggestion. I also don't feel that casting the LongOffset
is a good idea tbh.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
