Re: [PR] Improve Retention Manager Segment Lineage Clean Up [pinot]
snleee commented on code in PR #13232:
URL: https://github.com/apache/pinot/pull/13232#discussion_r1623143039
##
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/retention/RetentionManager.java:
##
@@ -55,7 +55,7 @@
*/
public class RetentionManager extends ControllerPeriodicTask {
public static final long OLD_LLC_SEGMENTS_RETENTION_IN_MILLIS =
TimeUnit.DAYS.toMillis(5L);
- private static final RetryPolicy DEFAULT_RETRY_POLICY =
RetryPolicies.exponentialBackoffRetryPolicy(5, 1000L, 2.0f);
+ private static final RetryPolicy DEFAULT_RETRY_POLICY =
RetryPolicies.randomDelayRetryPolicy(20, 100L, 200L);
Review Comment:
Main motivation:
- Both retention manager & segment upload were both trying to update
idealstate
- Retention Manager was trying to update the idealstate without lock and
used a simple exponential back-off
- Segment upload side idealstate update is using
`DEFAULT_TABLE_IDEALSTATES_UPDATE_RETRY_POLICY =
RetryPolicies.randomDelayRetryPolicy(20, 100L, 200L);`
- Segment upload side were updating idealstate after holding the lock
Observation
- Retention Manager failed a lot of time after 5 attempts while most of
idealstate updates via segment upload went through.
Goal
- We would like to make ideal state update as robust as the segment upload
path.
I thought about this briefly. I feel that as long as we hold the lock, retry
logic may not matter too much. @Jackie-Jiang how do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Improve Retention Manager Segment Lineage Clean Up [pinot]
snleee commented on code in PR #13232:
URL: https://github.com/apache/pinot/pull/13232#discussion_r1623143039
##
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/retention/RetentionManager.java:
##
@@ -55,7 +55,7 @@
*/
public class RetentionManager extends ControllerPeriodicTask {
public static final long OLD_LLC_SEGMENTS_RETENTION_IN_MILLIS =
TimeUnit.DAYS.toMillis(5L);
- private static final RetryPolicy DEFAULT_RETRY_POLICY =
RetryPolicies.exponentialBackoffRetryPolicy(5, 1000L, 2.0f);
+ private static final RetryPolicy DEFAULT_RETRY_POLICY =
RetryPolicies.randomDelayRetryPolicy(20, 100L, 200L);
Review Comment:
Main motivation:
- Both retention manager & segment upload were both trying to update
idealstate
- Retention Manager was trying to update the idealstate without lock and
used a simple exponential back-off
- Segment upload side idealstate update is using
`DEFAULT_TABLE_IDEALSTATES_UPDATE_RETRY_POLICY =
RetryPolicies.randomDelayRetryPolicy(20, 100L, 200L);`
- Segment upload side were updating idealstate after holding the lock
Observation
- Retention Manager failed a lot of time after 5 attempts while most of
idealstate updates via segment upload went through.
I thought about this briefly. I feel that as long as we hold the lock, retry
logic may not matter too much. @Jackie-Jiang how do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Introduce PinotLogicalAggregate and remove internal hint [pinot]
codecov-commenter commented on PR #13291: URL: https://github.com/apache/pinot/pull/13291#issuecomment-2143218505 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/13291?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report Attention: Patch coverage is `0%` with `40 lines` in your changes are missing coverage. Please review. > Project coverage is 0.00%. Comparing base [(`59551e4`)](https://app.codecov.io/gh/apache/pinot/commit/59551e45224f1535c4863fd577622b37366ccc97?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`3e73992`)](https://app.codecov.io/gh/apache/pinot/commit/3e73992846d9813abd487f4170cdf763f30e7f24?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache). > Report is 535 commits behind head on master. | [Files](https://app.codecov.io/gh/apache/pinot/pull/13291?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines | |---|---|---| | [...not/calcite/rel/logical/PinotLogicalAggregate.java](https://app.codecov.io/gh/apache/pinot/pull/13291?src=pr&el=tree&filepath=pinot-query-planner%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcalcite%2Frel%2Flogical%2FPinotLogicalAggregate.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtcXVlcnktcGxhbm5lci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2FsY2l0ZS9yZWwvbG9naWNhbC9QaW5vdExvZ2ljYWxBZ2dyZWdhdGUuamF2YQ==) | 0.00% | [14 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/13291?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | | [...el/rules/PinotAggregateExchangeNodeInsertRule.java](https://app.codecov.io/gh/apache/pinot/pull/13291?src=pr&el=tree&filepath=pinot-query-planner%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcalcite%2Frel%2Frules%2FPinotAggregateExchangeNodeInsertRule.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtcXVlcnktcGxhbm5lci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2FsY2l0ZS9yZWwvcnVsZXMvUGlub3RBZ2dyZWdhdGVFeGNoYW5nZU5vZGVJbnNlcnRSdWxlLmphdmE=) | 0.00% | [14 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/13291?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | | [...lcite/rel/rules/PinotRelDistributionTraitRule.java](https://app.codecov.io/gh/apache/pinot/pull/13291?src=pr&el=tree&filepath=pinot-query-planner%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcalcite%2Frel%2Frules%2FPinotRelDistributionTraitRule.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtcXVlcnktcGxhbm5lci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2FsY2l0ZS9yZWwvcnVsZXMvUGlub3RSZWxEaXN0cmlidXRpb25UcmFpdFJ1bGUuamF2YQ==) | 0.00% | [7 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/13291?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | | [.../query/planner/logical/RelToPlanNodeConverter.java](https://app.codecov.io/gh/apache/pinot/pull/13291?src=pr&el=tree&filepath=pinot-query-planner%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fquery%2Fplanner%2Flogical%2FRelToPlanNodeConverter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtcXVlcnktcGxhbm5lci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvcGxhbm5lci9sb2dpY2FsL1JlbFRvUGxhbk5vZGVDb252ZXJ0ZXIuamF2YQ==) | 0.00% | [3 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/13291?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | | [...pinot/calcite/rel/hint/PinotHintStrategyTable.java](https://app.codecov.io/gh/apache/pinot/pull/13291?src=pr&el=tree&filepath=pinot-query-planner%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcalcite%2Frel%2Fhint%2FPinotHintStrategyTable.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtcXVlcnktcGxhbm5lci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2FsY2l0ZS9yZWwvaGludC9QaW5vdEhpbnRTdHJhdGVneVRhYmxlLmphdmE=) | 0.00% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/13291?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | | [...he/pinot/query/planner/plannode/AggregateNode.java](https://app.codecov.io/gh/apache/pinot/pull/13291?src=pr&el=tree&filepath=pinot-query-planner%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fquery%2Fplanner%2Fplannode%2FAggregateNode.ja
(pinot) branch master updated (e2c5e73970 -> 26209ed6b0)
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from e2c5e73970 Pass literal within AggregateCall via rexList (#13282) add 26209ed6b0 Updated PDB and reload checksum config annotation for reload. (#13276) No new revisions were added by this update. Summary of changes: helm/index.yaml | 4 ++-- helm/pinot-0.2.9.tgz| Bin 69288 -> 73741 bytes .../pinot/templates/broker/poddisruptionbudget.yaml | 5 +++-- helm/pinot/templates/broker/statefulset.yaml| 6 -- .../templates/controller/poddisruptionbudget.yaml | 5 +++-- helm/pinot/templates/controller/statefulset.yaml| 6 -- helm/pinot/templates/minion/statefulset.yaml| 4 +++- .../pinot/templates/server/poddisruptionbudget.yaml | 5 +++-- helm/pinot/templates/server/statefulset.yaml| 6 -- 9 files changed, 26 insertions(+), 15 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] [HELM]: Bug fix for PDB and checksum config annotation for reload. [pinot]
Jackie-Jiang merged PR #13276: URL: https://github.com/apache/pinot/pull/13276 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[PR] Introduce PinotLogicalAggregate and remove internal hint [pinot]
Jackie-Jiang opened a new pull request, #13291: URL: https://github.com/apache/pinot/pull/13291 - Introduce `PinotLogicalAggregate` to carry the `AggType` along with the aggregate `RelNode` - Remove the internal agg options which is no longer needed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Pass literal within AggregateCall via rexList [pinot]
Jackie-Jiang merged PR #13282: URL: https://github.com/apache/pinot/pull/13282 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch master updated: Pass literal within AggregateCall via rexList (#13282)
This is an automated email from the ASF dual-hosted git repository.
jackie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pinot.git
The following commit(s) were added to refs/heads/master by this push:
new e2c5e73970 Pass literal within AggregateCall via rexList (#13282)
e2c5e73970 is described below
commit e2c5e73970b1e8f64df7c763c5bcac36ff19d2a6
Author: Xiaotian (Jackie) Jiang <1751+jackie-ji...@users.noreply.github.com>
AuthorDate: Fri May 31 18:00:00 2024 -0700
Pass literal within AggregateCall via rexList (#13282)
---
.../pinot/calcite/rel/hint/PinotHintOptions.java | 13 -
.../PinotAggregateExchangeNodeInsertRule.java | 422 ++---
.../rules/PinotAggregateLiteralAttachmentRule.java | 107 --
.../calcite/rel/rules/PinotQueryRuleSets.java | 5 -
.../org/apache/pinot/query/QueryEnvironment.java | 4 -
.../query/parser/CalciteRexExpressionParser.java | 4 +-
.../query/planner/logical/LiteralHintUtils.java| 85 -
.../query/planner/logical/RexExpressionUtils.java | 6 +-
.../apache/pinot/query/QueryCompilationTest.java | 3 +-
.../src/test/resources/queries/GroupByPlans.json | 18 +-
.../src/test/resources/queries/OrderByPlans.json | 4 +-
.../test/resources/queries/PinotHintablePlans.json | 33 +-
.../query/runtime/operator/AggregateOperator.java | 125 ++
.../src/test/resources/queries/QueryHints.json | 8 +-
.../pinot/segment/spi/AggregationFunctionType.java | 7 +-
15 files changed, 256 insertions(+), 588 deletions(-)
diff --git
a/pinot-query-planner/src/main/java/org/apache/pinot/calcite/rel/hint/PinotHintOptions.java
b/pinot-query-planner/src/main/java/org/apache/pinot/calcite/rel/hint/PinotHintOptions.java
index 1d53a3184e..99e07b61df 100644
---
a/pinot-query-planner/src/main/java/org/apache/pinot/calcite/rel/hint/PinotHintOptions.java
+++
b/pinot-query-planner/src/main/java/org/apache/pinot/calcite/rel/hint/PinotHintOptions.java
@@ -20,7 +20,6 @@ package org.apache.pinot.calcite.rel.hint;
import org.apache.calcite.rel.RelNode;
import org.apache.calcite.rel.hint.RelHint;
-import org.apache.pinot.query.planner.logical.LiteralHintUtils;
/**
@@ -47,18 +46,6 @@ public class PinotHintOptions {
public static class InternalAggregateOptions {
public static final String AGG_TYPE = "agg_type";
-/**
- * agg call signature is used to store LITERAL inputs to the Aggregate
Call. which is not supported in Calcite
- * here
- * 1. we store the Map of Pair[aggCallIdx, argListIdx] to RexLiteral to
indicate the RexLiteral being passed into
- * the aggregateCalls[aggCallIdx].operandList[argListIdx] is supposed
to be a RexLiteral.
- * 2. not all RexLiteral types are supported to be part of the input
constant call signature.
- * 3. RexLiteral are encoded as String and decoded as Pinot Literal
objects.
- *
- * see: {@link LiteralHintUtils}.
- * see: https://issues.apache.org/jira/projects/CALCITE/issues/CALCITE-5833
- */
-public static final String AGG_CALL_SIGNATURE = "agg_call_signature";
}
public static class AggregateOptions {
diff --git
a/pinot-query-planner/src/main/java/org/apache/pinot/calcite/rel/rules/PinotAggregateExchangeNodeInsertRule.java
b/pinot-query-planner/src/main/java/org/apache/pinot/calcite/rel/rules/PinotAggregateExchangeNodeInsertRule.java
index ffe0741751..0e6e13b0e7 100644
---
a/pinot-query-planner/src/main/java/org/apache/pinot/calcite/rel/rules/PinotAggregateExchangeNodeInsertRule.java
+++
b/pinot-query-planner/src/main/java/org/apache/pinot/calcite/rel/rules/PinotAggregateExchangeNodeInsertRule.java
@@ -19,20 +19,16 @@
package org.apache.pinot.calcite.rel.rules;
import com.google.common.collect.ImmutableList;
-import com.google.common.collect.ImmutableSet;
import java.util.ArrayList;
-import java.util.Collections;
import java.util.HashMap;
import java.util.List;
-import java.util.Locale;
import java.util.Map;
-import java.util.Set;
import javax.annotation.Nullable;
import org.apache.calcite.plan.RelOptRule;
import org.apache.calcite.plan.RelOptRuleCall;
-import org.apache.calcite.plan.hep.HepRelVertex;
import org.apache.calcite.rel.RelCollation;
import org.apache.calcite.rel.RelCollations;
+import org.apache.calcite.rel.RelDistribution;
import org.apache.calcite.rel.RelDistributions;
import org.apache.calcite.rel.RelFieldCollation;
import org.apache.calcite.rel.RelNode;
@@ -44,16 +40,16 @@ import org.apache.calcite.rel.logical.LogicalAggregate;
import org.apache.calcite.rel.rules.AggregateExtractProjectRule;
import org.apache.calcite.rel.rules.AggregateReduceFunctionsRule;
import org.apache.calcite.rex.RexBuilder;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexLiteral;
import org.apache.calcite.rex.RexNode;
import org.apache.calcite.sql.SqlAggFunction;
-import org.apache.calcite.sql.SqlKind;
import org.apache.calcite.
Re: [I] Proposal for a new instance-partition based routing strategy [pinot]
tibrewalpratik17 commented on issue #13284: URL: https://github.com/apache/pinot/issues/13284#issuecomment-2143181893 > just to confirm, I think you mean that we should select a replica-group for each Kafka partition right? That means if any segment of a kafka partition X are not available in replica-group 0, but are available in replica-group 1, then we will use rg=1 for X. For the same query, if any segment of a Kafka partition Y are not available in replica-group 1, but are available in replica-group 0, then we will use rg=0 for Y. That would mean that a single query can span multiple replica-groups. Yes exactly! We should mark an entire instance as unavailable for all segments in that instance rather just the ones which are a part of the same partition number (X in your case) in that instance. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Address Bug Related to Map fields Containing Multiple Entries in DataTypeTransformer [pinot]
Jackie-Jiang commented on code in PR #13265:
URL: https://github.com/apache/pinot/pull/13265#discussion_r1623070697
##
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/recordtransformer/DataTypeTransformer.java:
##
@@ -151,7 +151,20 @@ static Object standardize(String column, @Nullable Object
value, boolean isSingl
}
List standardizedValues = new ArrayList<>(numValues);
for (Object singleValue : values) {
-Object standardizedValue = standardize(column, singleValue, true);
+Object standardizedValue;
+
+// Check if the value itself is multivalued.
+if (singleValue instanceof Object[] || singleValue instanceof List) {
+ standardizedValue = standardize(column, singleValue, false);
Review Comment:
I don't think this is correct. This can result in nested arrays being
generated, and break the following part.
What you can do is to generate all standardized values for every elements,
and continue when:
- There is no `Object[]`
- There is only one `Object[]`, no other value (all `null`s)
The same logic also applies to collections and map. But before getting into
that, I'd invest why are we getting multiple nested arrays
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch master updated: [bugfix] Handling null value for kafka client id suffix (#13279)
This is an automated email from the ASF dual-hosted git repository.
jackie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pinot.git
The following commit(s) were added to refs/heads/master by this push:
new 3dd653a785 [bugfix] Handling null value for kafka client id suffix
(#13279)
3dd653a785 is described below
commit 3dd653a7859f39698fa99f727cc27bb1a6aa60d1
Author: Pratik Tibrewal
AuthorDate: Sat Jun 1 06:02:04 2024 +0530
[bugfix] Handling null value for kafka client id suffix (#13279)
---
.../core/data/manager/realtime/RealtimeSegmentDataManager.java | 7 ---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git
a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeSegmentDataManager.java
b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeSegmentDataManager.java
index b441f086de..bcc3223141 100644
---
a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeSegmentDataManager.java
+++
b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeSegmentDataManager.java
@@ -1435,11 +1435,12 @@ public class RealtimeSegmentDataManager extends
SegmentDataManager {
_acquiredConsumerSemaphore = new AtomicBoolean(false);
InstanceDataManagerConfig instanceDataManagerConfig =
_indexLoadingConfig.getInstanceDataManagerConfig();
String clientIdSuffix =
-instanceDataManagerConfig != null ?
instanceDataManagerConfig.getConsumerClientIdSuffix() : "";
+instanceDataManagerConfig != null ?
instanceDataManagerConfig.getConsumerClientIdSuffix() : null;
if (StringUtils.isNotBlank(clientIdSuffix)) {
- clientIdSuffix = "-" + clientIdSuffix;
+ _clientId = _tableNameWithType + "-" + streamTopic + "-" +
_partitionGroupId + "-" + clientIdSuffix;
+} else {
+ _clientId = _tableNameWithType + "-" + streamTopic + "-" +
_partitionGroupId;
}
-_clientId = _tableNameWithType + "-" + streamTopic + "-" +
_partitionGroupId + clientIdSuffix;
_segmentLogger =
LoggerFactory.getLogger(RealtimeSegmentDataManager.class.getName() + "_" +
_segmentNameStr);
_tableStreamName = _tableNameWithType + "_" + streamTopic;
if (_indexLoadingConfig.isRealtimeOffHeapAllocation() &&
!_indexLoadingConfig.isDirectRealtimeOffHeapAllocation()) {
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] [bugfix] Handling null value for kafka client id suffix [pinot]
Jackie-Jiang merged PR #13279: URL: https://github.com/apache/pinot/pull/13279 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Add support for raw JSON filter predicates [pinot]
Jackie-Jiang merged PR #13283: URL: https://github.com/apache/pinot/pull/13283 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch master updated (cb5596c4f2 -> 34bab22861)
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from cb5596c4f2 Bump com.nimbusds:nimbus-jose-jwt from 9.39.1 to 9.39.3 (#13289) add 34bab22861 Add support for raw JSON filter predicates (#13283) No new revisions were added by this update. Summary of changes: .../predicate/EqualsPredicateEvaluatorFactory.java | 1 + .../predicate/InPredicateEvaluatorFactory.java | 3 +- .../NotEqualsPredicateEvaluatorFactory.java| 1 + .../predicate/NotInPredicateEvaluatorFactory.java | 3 +- .../NoDictionaryEqualsPredicateEvaluatorsTest.java | 40 .../NoDictionaryInPredicateEvaluatorTest.java | 44 ++ .../predicate/PredicateEvaluatorTestUtils.java | 11 ++ 7 files changed, 101 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch master updated (34bab22861 -> 6c85de5015)
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from 34bab22861 Add support for raw JSON filter predicates (#13283) add 6c85de5015 Bump org.webjars:swagger-ui from 5.17.0 to 5.17.14 (#13286) No new revisions were added by this update. Summary of changes: pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java | 2 +- pom.xml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Bump org.webjars:swagger-ui from 5.17.0 to 5.17.14 [pinot]
Jackie-Jiang merged PR #13286: URL: https://github.com/apache/pinot/pull/13286 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch dependabot/maven/org.webjars-swagger-ui-5.17.14 deleted (was 8dcfa4f2df)
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch dependabot/maven/org.webjars-swagger-ui-5.17.14 in repository https://gitbox.apache.org/repos/asf/pinot.git was 8dcfa4f2df Fix resource path The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [I] Google library dependency management [pinot]
Jackie-Jiang commented on issue #13077: URL: https://github.com/apache/pinot/issues/13077#issuecomment-2143029902 IMO we can delete them. Wondering if we can access these values from the BOM in case we have other dependencies sharing these values? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch master updated (f70ae2d9be -> cb5596c4f2)
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from f70ae2d9be Bump org.apache.maven.plugins:maven-shade-plugin from 3.5.3 to 3.6.0 (#13288) add cb5596c4f2 Bump com.nimbusds:nimbus-jose-jwt from 9.39.1 to 9.39.3 (#13289) No new revisions were added by this update. Summary of changes: pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch dependabot/maven/com.nimbusds-nimbus-jose-jwt-9.39.3 deleted (was 031e5b8b89)
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch dependabot/maven/com.nimbusds-nimbus-jose-jwt-9.39.3 in repository https://gitbox.apache.org/repos/asf/pinot.git was 031e5b8b89 Bump com.nimbusds:nimbus-jose-jwt from 9.39.1 to 9.39.3 The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Bump com.nimbusds:nimbus-jose-jwt from 9.39.1 to 9.39.3 [pinot]
Jackie-Jiang merged PR #13289: URL: https://github.com/apache/pinot/pull/13289 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch master updated (283363af87 -> f70ae2d9be)
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from 283363af87 Make allowDownloadFromServer minion-cluster-level config (#13247) add f70ae2d9be Bump org.apache.maven.plugins:maven-shade-plugin from 3.5.3 to 3.6.0 (#13288) No new revisions were added by this update. Summary of changes: pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch dependabot/maven/org.apache.maven.plugins-maven-shade-plugin-3.6.0 deleted (was 6bb5fae9ae)
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch dependabot/maven/org.apache.maven.plugins-maven-shade-plugin-3.6.0 in repository https://gitbox.apache.org/repos/asf/pinot.git was 6bb5fae9ae Bump org.apache.maven.plugins:maven-shade-plugin from 3.5.3 to 3.6.0 The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Bump org.apache.maven.plugins:maven-shade-plugin from 3.5.3 to 3.6.0 [pinot]
Jackie-Jiang merged PR #13288: URL: https://github.com/apache/pinot/pull/13288 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch master updated (bf84802571 -> 283363af87)
This is an automated email from the ASF dual-hosted git repository. ankitsultana pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from bf84802571 Bump software.amazon.awssdk:bom from 2.25.61 to 2.25.63 (#13287) add 283363af87 Make allowDownloadFromServer minion-cluster-level config (#13247) No new revisions were added by this update. Summary of changes: .../src/main/java/org/apache/pinot/minion/BaseMinionStarter.java | 1 + .../src/main/java/org/apache/pinot/minion/MinionConf.java| 5 + .../src/main/java/org/apache/pinot/minion/MinionContext.java | 9 + .../org/apache/pinot/plugin/minion/tasks/BaseTaskExecutor.java | 3 ++- .../org/apache/pinot/plugin/minion/tasks/MinionTaskUtils.java| 7 --- .../apache/pinot/plugin/minion/tasks/MinionTaskUtilsTest.java| 6 +++--- .../main/java/org/apache/pinot/spi/utils/CommonConstants.java| 2 ++ 7 files changed, 26 insertions(+), 7 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Make allowDownloadFromServer minion-cluster-level config [pinot]
ankitsultana merged PR #13247: URL: https://github.com/apache/pinot/pull/13247 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [I] Google library dependency management [pinot]
abhioncbr commented on issue #13077: URL: https://github.com/apache/pinot/issues/13077#issuecomment-2142883257 I was looking into it, and as I understand, the version mentioned takes effect by importing the sub-projects BOM or including them as a dependency in the parent pom. These properties are useless ```bash 1.23.0 2.10.1 33.1.0-jre 1.44.1 3.25.2 1.61.1 ``` Also, I would like to highlight that overriding the version of a dependency managed by a BOM can lead to compatibility issues, as the BOM's purpose is to ensure that all the libraries and their dependencies work well together. Let me know if I can delete the properties, or if we want to override them, I will add their BOMs in dependency management. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] [Backfill] allow externally partitioned segment uploads for upsert tables [pinot]
rohityadav1993 commented on code in PR #13107:
URL: https://github.com/apache/pinot/pull/13107#discussion_r1622835600
##
pinot-common/src/main/java/org/apache/pinot/common/utils/UploadedRealtimeSegmentName.java:
##
@@ -0,0 +1,157 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.utils;
+
+import com.google.common.base.Joiner;
+import com.google.common.base.Preconditions;
+import java.util.Arrays;
+import java.util.Objects;
+import javax.annotation.Nullable;
+import org.joda.time.format.DateTimeFormat;
+import org.joda.time.format.DateTimeFormatter;
+
+
+/**
+ * Class to represent segment names like:
uploaded_{tableName}_{partitionId}_{sequenceId}_{creationTime}
Review Comment:
The reason to enforce prefix was to quickly identify an uploaded segment.
With optional suffix we would have to fall back to segment status in ZK metadta
to know if it is an uploaded segment(do suggest if there is a better way).
It may even be complex to parse the segment name to figure out partitionId
and segmentId.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [I] Pinot UI load time improvements [pinot]
deepthi912 commented on issue #13278: URL: https://github.com/apache/pinot/issues/13278#issuecomment-2142811974 Looking into it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] [Backfill] allow externally partitioned segment uploads for upsert tables [pinot]
rohityadav1993 commented on code in PR #13107:
URL: https://github.com/apache/pinot/pull/13107#discussion_r1622832355
##
pinot-common/src/main/java/org/apache/pinot/common/utils/Benchmark.java:
##
@@ -0,0 +1,89 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.utils;
+
+import java.util.Random;
+import java.util.regex.Pattern;
+
+
+public class Benchmark {
+ public static final String SEPARATOR = "_";
+ public static final String UPLOADED_PREFIX = "uploaded";
+ public static final String UPLOADED_REALTIME_SEGMENT_NAME_REGEX =
"^uploaded_(.+)_(\\d+)_(\\d+)_(\\d+)$";
+ private static final Pattern NAME_PATTERN =
Pattern.compile(UPLOADED_REALTIME_SEGMENT_NAME_REGEX);
+
+ private Benchmark() {
+ }
+
+ public static boolean isUploadedRealtimeSegmentNameMethod1(String
segmentName) {
+String[] parts = segmentName.split(SEPARATOR);
Review Comment:
Got you, I wasn't aware of the subtelty. This file was also not supposed to
be checked in, removing.
Did some benchmark with 3 methods:
```
Method 1 (String.split()) took: 2266343500 ns
Method 2 (regex match) took: 3760987625 ns
Method 3 (StringUtils.split()) took: 2412549250 ns
Method 3 is 0.939397817474607 times faster than Method 1
Method 3 is 1.5589267763134784 times faster than Method 2
```
```
Method 1 (String.split()) took: 2462188708 ns
Method 2 (regex match) took: 4130721000 ns
Method 3 (StringUtils.split()) took: 2082269750 ns
Method 3 is 1.1824542463818628 times faster than Method 1
Method 3 is 1.983758828557155 times faster than Method 2
```
```
Method 1 (String.split()) took: 2275362125 ns
Method 2 (regex match) took: 4017714167 ns
Method 3 (StringUtils.split()) took: 2003299125 ns
Method 3 is 1.1358074770785915 times faster than Method 1
Method 3 is 2.0055488053986945 times faster than Method 2
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch master updated (61ce07ff50 -> bf84802571)
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from 61ce07ff50 Bump org.apache.maven.plugins:maven-enforcer-plugin from 3.4.1 to 3.5.0 (#13270) add bf84802571 Bump software.amazon.awssdk:bom from 2.25.61 to 2.25.63 (#13287) No new revisions were added by this update. Summary of changes: pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch dependabot/maven/software.amazon.awssdk-bom-2.25.63 deleted (was c9dbb4bf81)
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch dependabot/maven/software.amazon.awssdk-bom-2.25.63 in repository https://gitbox.apache.org/repos/asf/pinot.git was c9dbb4bf81 Bump software.amazon.awssdk:bom from 2.25.61 to 2.25.63 The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Bump software.amazon.awssdk:bom from 2.25.61 to 2.25.63 [pinot]
Jackie-Jiang merged PR #13287: URL: https://github.com/apache/pinot/pull/13287 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Disabling checkstyle plugin and Spotless plugin for Java version > 21.0 [pinot]
Jackie-Jiang commented on PR #13252: URL: https://github.com/apache/pinot/pull/13252#issuecomment-2142792107 We do build java with 21. What failure did you encounter? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch dependabot/maven/org.apache.maven.plugins-maven-enforcer-plugin-3.5.0 deleted (was 01d01f8107)
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch dependabot/maven/org.apache.maven.plugins-maven-enforcer-plugin-3.5.0 in repository https://gitbox.apache.org/repos/asf/pinot.git was 01d01f8107 Bump org.apache.maven.plugins:maven-enforcer-plugin from 3.4.1 to 3.5.0 The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch master updated (a1f9b8997c -> fcd7856412)
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from a1f9b8997c Improve warn logs for requesting validDocID snapshots (#13280) add fcd7856412 Bump com.google.cloud:libraries-bom from 26.39.0 to 26.40.0 (#13272) No new revisions were added by this update. Summary of changes: pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch master updated (fcd7856412 -> 61ce07ff50)
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from fcd7856412 Bump com.google.cloud:libraries-bom from 26.39.0 to 26.40.0 (#13272) add 61ce07ff50 Bump org.apache.maven.plugins:maven-enforcer-plugin from 3.4.1 to 3.5.0 (#13270) No new revisions were added by this update. Summary of changes: pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Bump org.apache.maven.plugins:maven-enforcer-plugin from 3.4.1 to 3.5.0 [pinot]
Jackie-Jiang merged PR #13270: URL: https://github.com/apache/pinot/pull/13270 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch dependabot/maven/com.google.cloud-libraries-bom-26.40.0 deleted (was 26c33bfa35)
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch dependabot/maven/com.google.cloud-libraries-bom-26.40.0 in repository https://gitbox.apache.org/repos/asf/pinot.git was 26c33bfa35 Bump com.google.cloud:libraries-bom from 26.39.0 to 26.40.0 The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Bump com.google.cloud:libraries-bom from 26.39.0 to 26.40.0 [pinot]
Jackie-Jiang merged PR #13272: URL: https://github.com/apache/pinot/pull/13272 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch dependabot/maven/net.openhft-posix-2.26ea0 updated (527f26bfd3 -> 055e1eb8e2)
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch dependabot/maven/net.openhft-posix-2.26ea0 in repository https://gitbox.apache.org/repos/asf/pinot.git discard 527f26bfd3 Bump net.openhft:posix from 2.25ea0 to 2.26ea0 add 055e1eb8e2 Bump net.openhft:posix from 2.25ea0 to 2.25.1 This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (527f26bfd3) \ N -- N -- N refs/heads/dependabot/maven/net.openhft-posix-2.26ea0 (055e1eb8e2) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch dependabot/maven/net.openhft-chronicle-core-2.26ea0 updated (bb51e1751b -> 98b7d50401)
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch dependabot/maven/net.openhft-chronicle-core-2.26ea0 in repository https://gitbox.apache.org/repos/asf/pinot.git discard bb51e1751b Bump net.openhft:chronicle-core from 2.25ea15 to 2.26ea0 add 98b7d50401 Bump net.openhft:chronicle-core from 2.25ea15 to 2.25.16 This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (bb51e1751b) \ N -- N -- N refs/heads/dependabot/maven/net.openhft-chronicle-core-2.26ea0 (98b7d50401) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] [multistage][feature] FunctionRegistry unification [pinot]
walterddr closed pull request #12302: [multistage][feature] FunctionRegistry unification URL: https://github.com/apache/pinot/pull/12302 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch dependabot/maven/org.webjars-swagger-ui-5.17.14 updated (33566e7ca2 -> 8dcfa4f2df)
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch dependabot/maven/org.webjars-swagger-ui-5.17.14 in repository https://gitbox.apache.org/repos/asf/pinot.git from 33566e7ca2 Bump org.webjars:swagger-ui from 5.17.0 to 5.17.14 add 8dcfa4f2df Fix resource path No new revisions were added by this update. Summary of changes: pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Fix few metric rules which were affected by the database prefix handling [pinot]
npawar commented on PR #13290: URL: https://github.com/apache/pinot/pull/13290#issuecomment-2142618653 Please do add testing done and screenshots of validated rules as they appear in jconsole / prometheus explore -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] add metrics for IdeaState update [pinot]
pengding-stripe commented on PR #13266: URL: https://github.com/apache/pinot/pull/13266#issuecomment-2142615457 @Jackie-Jiang how can I merge this PR? looks like it still needs approval to run a workflow -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Improve warn logs for validDocID snapshot flow [pinot]
klsince merged PR #13280: URL: https://github.com/apache/pinot/pull/13280 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch master updated: Improve warn logs for requesting validDocID snapshots (#13280)
This is an automated email from the ASF dual-hosted git repository.
xbli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pinot.git
The following commit(s) were added to refs/heads/master by this push:
new a1f9b8997c Improve warn logs for requesting validDocID snapshots
(#13280)
a1f9b8997c is described below
commit a1f9b8997ccfca00dbd9d472a300250dd7730e2c
Author: Pratik Tibrewal
AuthorDate: Fri May 31 21:35:14 2024 +0530
Improve warn logs for requesting validDocID snapshots (#13280)
* Improve warn logs for requesting validDocID snapshots
* Move verbose logs to debug level and keep some aggregated stats in warn
logs
---
.../pinot/server/api/resources/TablesResource.java | 33 --
1 file changed, 25 insertions(+), 8 deletions(-)
diff --git
a/pinot-server/src/main/java/org/apache/pinot/server/api/resources/TablesResource.java
b/pinot-server/src/main/java/org/apache/pinot/server/api/resources/TablesResource.java
index f497ad68f4..063608235a 100644
---
a/pinot-server/src/main/java/org/apache/pinot/server/api/resources/TablesResource.java
+++
b/pinot-server/src/main/java/org/apache/pinot/server/api/resources/TablesResource.java
@@ -647,6 +647,8 @@ public class TablesResource {
TableDataManager tableDataManager =
ServerResourceUtils.checkGetTableDataManager(_serverInstance,
tableNameWithType);
List missingSegments = new ArrayList<>();
+int nonImmutableSegmentCount = 0;
+int missingValidDocIdSnapshotSegmentCount = 0;
List segmentDataManagers;
if (segments == null) {
segmentDataManagers = tableDataManager.acquireAllSegments();
@@ -668,9 +670,12 @@ public class TablesResource {
}
// Skip the consuming segments
if (!(indexSegment instanceof ImmutableSegmentImpl)) {
- String msg = String.format("Table %s segment %s is not a immutable
segment", tableNameWithType,
- segmentDataManager.getSegmentName());
- LOGGER.warn(msg);
+ if (LOGGER.isDebugEnabled()) {
+String msg = String.format("Table %s segment %s is not a immutable
segment", tableNameWithType,
+segmentDataManager.getSegmentName());
+LOGGER.debug(msg);
+ }
+ nonImmutableSegmentCount++;
continue;
}
@@ -679,11 +684,14 @@ public class TablesResource {
String finalValidDocIdsType =
validDocIdSnapshotPair.getLeft().toString();
MutableRoaringBitmap validDocIdsSnapshot =
validDocIdSnapshotPair.getRight();
if (validDocIdsSnapshot == null) {
- String msg = String.format(
- "Found that validDocIds is missing while processing
validDocIdsMetadata for table %s segment %s while "
- + "reading the validDocIds with validDocIdType %s",
tableNameWithType,
- segmentDataManager.getSegmentName(), validDocIdsType);
- LOGGER.warn(msg);
+ if (LOGGER.isDebugEnabled()) {
+String msg = String.format(
+"Found that validDocIds is missing while processing
validDocIdsMetadata for table %s segment %s while "
++ "reading the validDocIds with validDocIdType %s",
tableNameWithType,
+segmentDataManager.getSegmentName(), validDocIdsType);
+LOGGER.debug(msg);
+ }
+ missingValidDocIdSnapshotSegmentCount++;
continue;
}
@@ -699,6 +707,15 @@ public class TablesResource {
validDocIdsMetadata.put("validDocIdsType", finalValidDocIdsType);
allValidDocIdsMetadata.add(validDocIdsMetadata);
}
+ if (nonImmutableSegmentCount > 0) {
+LOGGER.warn("Table {} has {} non-immutable segments found while
processing validDocIdsMetadata",
+tableNameWithType, nonImmutableSegmentCount);
+ }
+ if (missingValidDocIdSnapshotSegmentCount > 0) {
+LOGGER.warn("Found that validDocIds is missing for {} segments while
processing validDocIdsMetadata "
++ "for table {} while reading the validDocIds with
validDocIdType {}. ",
+missingValidDocIdSnapshotSegmentCount, tableNameWithType,
validDocIdsType);
+ }
return allValidDocIdsMetadata;
} finally {
for (SegmentDataManager segmentDataManager : segmentDataManagers) {
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Fix few metric rules which were affected by the database prefix handling [pinot]
shounakmk219 commented on PR #13290: URL: https://github.com/apache/pinot/pull/13290#issuecomment-2142485592 > How shall we test it out? One possibility is to get the list of available metrics on broker/controller/server using `curl localhost:8080` running 0.8.0, then deploy this image, do the same and compare. This is not trivial to do though. Yeah it's a pain to test these changes. Right now just making sure no random string appear in `table` label. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Fix few metric rules which were affected by the database prefix handling [pinot]
soumitra-st commented on PR #13290: URL: https://github.com/apache/pinot/pull/13290#issuecomment-2142480075 Looks good, deferring to @suddendust to double check. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Fix few metric rules which were affected by the database prefix handling [pinot]
soumitra-st commented on PR #13290: URL: https://github.com/apache/pinot/pull/13290#issuecomment-2142478946 How shall we test it out? One possibility is to get the list of available metrics on broker/controller/server using `curl localhost:8080` running 0.8.0, then deploy this image, do the same and compare. This is not trivial to do though. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Fix few metric rules which were affected by the database prefix handling [pinot]
shounakmk219 commented on PR #13290: URL: https://github.com/apache/pinot/pull/13290#issuecomment-2142284257 > Can you list the broken metrics and how they are fixed in this PR? @soumitra-st Updated the description with metric details. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Fix few metric rules which were affected by the database prefix handling [pinot]
soumitra-st commented on PR #13290: URL: https://github.com/apache/pinot/pull/13290#issuecomment-2142166914 Can you list the broken metrics and how they are fixed in this PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [I] Could not fetch Kafka topic message headers and metadata [pinot]
yagmuratak closed issue #13268: Could not fetch Kafka topic message headers and metadata URL: https://github.com/apache/pinot/issues/13268 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [I] Could not fetch Kafka topic message headers and metadata [pinot]
yagmuratak commented on issue #13268:
URL: https://github.com/apache/pinot/issues/13268#issuecomment-2142084140
I figured it out finally. I set **"stream.kafka.metadata.populate": true**
configuration in **tableIndexConfig.streamConfigs** and
> {
"name": "__metadata$offset",
"dataType": "STRING"
},
{
"name": "__header$channel",
"dataType": "BYTES"
},
{
"name": "channel",
"dataType": "STRING"
}
added these column definitions into **dimensionFieldSpecs**. Header values
are in BYTES type so I converted them to STRING via this code in
**ingestionConfig.transformConfigs**
> {
"columnName": "channel",
"transformFunction": "fromUtf8(__header$channel)"
}
So, **channel** column is visible in Pinot table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Optimize snapshot flow to only snapshot segments which have updates [pinot]
codecov-commenter commented on PR #13285: URL: https://github.com/apache/pinot/pull/13285#issuecomment-2141887032 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/13285?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report Attention: Patch coverage is `0%` with `30 lines` in your changes are missing coverage. Please review. > Project coverage is 0.00%. Comparing base [(`59551e4`)](https://app.codecov.io/gh/apache/pinot/commit/59551e45224f1535c4863fd577622b37366ccc97?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`e30186b`)](https://app.codecov.io/gh/apache/pinot/commit/e30186b4c2b311e841dbbdc183bf9d387b22b3bd?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache). > Report is 523 commits behind head on master. | [Files](https://app.codecov.io/gh/apache/pinot/pull/13285?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines | |---|---|---| | [...cal/upsert/BasePartitionUpsertMetadataManager.java](https://app.codecov.io/gh/apache/pinot/pull/13285?src=pr&el=tree&filepath=pinot-segment-local%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsegment%2Flocal%2Fupsert%2FBasePartitionUpsertMetadataManager.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC91cHNlcnQvQmFzZVBhcnRpdGlvblVwc2VydE1ldGFkYXRhTWFuYWdlci5qYXZh) | 0.00% | [30 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/13285?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #13285 +/- ## = - Coverage 61.75%0.00% -61.76% = Files 2436 2461 +25 Lines133233 135762 +2529 Branches 2063620999 +363 = - Hits 822740-82274 - Misses44911 135762+90851 + Partials 60480 -6048 ``` | [Flag](https://app.codecov.io/gh/apache/pinot/pull/13285/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | | |---|---|---| | [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/13285/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | | [integration](https://app.codecov.io/gh/apache/pinot/pull/13285/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <0.00%> (-0.01%)` | :arrow_down: | | [integration1](https://app.codecov.io/gh/apache/pinot/pull/13285/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | | [integration2](https://app.codecov.io/gh/apache/pinot/pull/13285/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <0.00%> (ø)` | | | [java-11](https://app.codecov.io/gh/apache/pinot/pull/13285/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | | [java-21](https://app.codecov.io/gh/apache/pinot/pull/13285/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <0.00%> (-61.63%)` | :arrow_down: | | [skip-bytebuffers-false](https://app.codecov.io/gh/apache/pinot/pull/13285/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <0.00%> (-61.75%)` | :arrow_down: | | [skip-bytebuffers-true](https://app.codecov.io/gh/apache/pinot/pull/13285/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <0.00%> (-27.73%)` | :arrow_down: | | [temurin](https://app.codecov.io/gh/apache/pinot/pull/13285/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <0.00%> (-61.76%)` | :arrow_down: | | [unittests](https://app.codecov.io/gh/apache/pinot/pull/13285/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_
[PR] Fix few metric rules which were affected by the database prefix handling [pinot]
shounakmk219 opened a new pull request, #13290: URL: https://github.com/apache/pinot/pull/13290 # Description This PR fixes few jmx metric rules that were broken due to the database prefix handling on metrics. # labels `bugfix` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch dependabot/maven/com.nimbusds-nimbus-jose-jwt-9.39.3 created (now 031e5b8b89)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/com.nimbusds-nimbus-jose-jwt-9.39.3 in repository https://gitbox.apache.org/repos/asf/pinot.git at 031e5b8b89 Bump com.nimbusds:nimbus-jose-jwt from 9.39.1 to 9.39.3 No new revisions were added by this update. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Bump com.nimbusds:nimbus-jose-jwt from 9.39.1 to 9.39.2 [pinot]
dependabot[bot] closed pull request #13271: Bump com.nimbusds:nimbus-jose-jwt from 9.39.1 to 9.39.2 URL: https://github.com/apache/pinot/pull/13271 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[PR] Bump com.nimbusds:nimbus-jose-jwt from 9.39.1 to 9.39.3 [pinot]
dependabot[bot] opened a new pull request, #13289:
URL: https://github.com/apache/pinot/pull/13289
Bumps
[com.nimbusds:nimbus-jose-jwt](https://bitbucket.org/connect2id/nimbus-jose-jwt)
from 9.39.1 to 9.39.3.
Changelog
Sourced from https://bitbucket.org/connect2id/nimbus-jose-jwt/src/master/CHANGELOG.txt";>com.nimbusds:nimbus-jose-jwt's
changelog.
version 1.0 (2012-03-01)
First version based on the OpenInfoCard JWT, JWS and JWE code base.
version 1.1 (2012-03-06)
Introduces type-safe enumeration of the JSON Web Algorithms (JWA).
Refactors the JWT class.
version 1.2 (2012-03-08)
Moves JWS and JWE code into separate classes.
version 1.3 (2012-03-09)
Switches to Apache Commons Codec for Base64URL encoding and decoding
Consolidates the crypto utilities within the package.
Introduces a JWT content serialiser class.
version 1.4 (2012-03-09)
Refactoring of JWT class and JUnit tests.
version 1.5 (2012-03-18)
Switches to JSON Smart for JSON serialisation and parsing.
Introduces claims set class with JSON objects, string, Base64URL and
byte array views.
version 1.6 (2012-03-20)
Creates class for representing, serialising and parsing JSON Web Keys
(JWK).
Introduces separate class for representing JWT headers.
version 1.7 (2012-04-01)
Introduces separate classes for plain, JWS and JWE headers.
Introduces separate classes for plain, signed and encrypted JWTs.
Removes the JWTContent class.
Removes password-based (PE820) encryption support.
version 1.8 (2012-04-03)
Adds support for the ZIP JWE header parameter.
Removes unsupported algorithms from the JWA enumeration.
version 1.9 (2012-04-03)
Renames JWEHeader.{get|set}EncryptionAlgorithm() to
JWEHeader.{get|set}EncryptionMethod().
version 1.9.1 (2012-04-03)
Upgrades JSON Smart JAR to 1.1.1.
version 1.10 (2012-04-14)
Introduces serialize() method to base abstract JWT class.
version 1.11 (2012-05-13)
JWT.serialize() throws checked JWTException instead of
... (truncated)
Commits
https://bitbucket.org/connect2id/nimbus-jose-jwt/commits/16d52cd6ce38d46d8e0a3aef01b04f8008a2e260";>16d52cd
[maven-release-plugin] prepare for next development iteration
https://bitbucket.org/connect2id/nimbus-jose-jwt/commits/b530053d09c79e1016faaa5f46776b8c1429bca6";>b530053
Fixes NPE in DefaultJWTClaimsVerifier.verify when checking an exact match
cla...
https://bitbucket.org/connect2id/nimbus-jose-jwt/commits/63be304cb84e16118b3651288bd7e321fd9a64b4";>63be304
Fixes version 9.39.2 date
https://bitbucket.org/connect2id/nimbus-jose-jwt/commits/97183e0d584c4c44f2bb350d2daaf204f11e6c28";>97183e0
[maven-release-plugin] prepare release 9.39.2
https://bitbucket.org/connect2id/nimbus-jose-jwt/commits/2e444edc3ea8fa1b70577be5433af24de1591ea6";>2e444ed
[maven-release-plugin] prepare for next development iteration
https://bitbucket.org/connect2id/nimbus-jose-jwt/commits/26527774ef13ac4246505c0c05b167f4568dfbdd";>2652777
Bumps GSon to 2.11.0, addressing Number parsing performance issue (https://bitbucket.org/connect2id/nimbus-jose-jwt/issues/546";>#546)
https://bitbucket.org/connect2id/nimbus-jose-jwt/commits/7dde63560c2d9efd0f9424d2b5ab8cea77ad8a68";>7dde635
Fixes JSONObjectUtilsTest.testParseFromStringEntity test, ParseException
mess...
https://bitbucket.org/connect2id/nimbus-jose-jwt/commits/6d001d46a3a3095342fa1fc3398c79d5d387b513";>6d001d4
[maven-release-plugin] prepare release 9.39.3
See full diff in https://bitbucket.org/connect2id/nimbus-jose-jwt/branches/compare/9.39.3..9.39.1";>compare
view
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that
have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI
passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and
block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@depend
(pinot) branch dependabot/maven/com.nimbusds-nimbus-jose-jwt-9.39.2 deleted (was cef92c45dc)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/com.nimbusds-nimbus-jose-jwt-9.39.2 in repository https://gitbox.apache.org/repos/asf/pinot.git was cef92c45dc Bump com.nimbusds:nimbus-jose-jwt from 9.39.1 to 9.39.2 The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Bump com.nimbusds:nimbus-jose-jwt from 9.39.1 to 9.39.2 [pinot]
dependabot[bot] commented on PR #13271: URL: https://github.com/apache/pinot/pull/13271#issuecomment-2141817187 Superseded by #13289. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch dependabot/maven/software.amazon.awssdk-bom-2.25.62 deleted (was c703f9505d)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/software.amazon.awssdk-bom-2.25.62 in repository https://gitbox.apache.org/repos/asf/pinot.git was c703f9505d Bump software.amazon.awssdk:bom from 2.25.61 to 2.25.62 The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Bump software.amazon.awssdk:bom from 2.25.61 to 2.25.62 [pinot]
dependabot[bot] commented on PR #13275: URL: https://github.com/apache/pinot/pull/13275#issuecomment-2141811204 Superseded by #13287. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[PR] Bump software.amazon.awssdk:bom from 2.25.61 to 2.25.63 [pinot]
dependabot[bot] opened a new pull request, #13287: URL: https://github.com/apache/pinot/pull/13287 Bumps software.amazon.awssdk:bom from 2.25.61 to 2.25.63. [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- Dependabot commands and options You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Bump software.amazon.awssdk:bom from 2.25.61 to 2.25.62 [pinot]
dependabot[bot] closed pull request #13275: Bump software.amazon.awssdk:bom from 2.25.61 to 2.25.62 URL: https://github.com/apache/pinot/pull/13275 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch dependabot/maven/org.webjars-swagger-ui-5.17.11 deleted (was 5657611090)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/org.webjars-swagger-ui-5.17.11 in repository https://gitbox.apache.org/repos/asf/pinot.git was 5657611090 Bump org.webjars:swagger-ui from 5.17.0 to 5.17.11 The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Bump org.webjars:swagger-ui from 5.17.0 to 5.17.11 [pinot]
dependabot[bot] closed pull request #13210: Bump org.webjars:swagger-ui from 5.17.0 to 5.17.11 URL: https://github.com/apache/pinot/pull/13210 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch dependabot/maven/org.webjars-swagger-ui-5.17.14 created (now 33566e7ca2)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/org.webjars-swagger-ui-5.17.14 in repository https://gitbox.apache.org/repos/asf/pinot.git at 33566e7ca2 Bump org.webjars:swagger-ui from 5.17.0 to 5.17.14 No new revisions were added by this update. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[PR] Bump org.webjars:swagger-ui from 5.17.0 to 5.17.14 [pinot]
dependabot[bot] opened a new pull request, #13286: URL: https://github.com/apache/pinot/pull/13286 Bumps [org.webjars:swagger-ui](https://github.com/swagger-api/swagger-ui) from 5.17.0 to 5.17.14. Release notes Sourced from https://github.com/swagger-api/swagger-ui/releases";>org.webjars:swagger-ui's releases. Swagger UI v5.17.14 Released! https://github.com/swagger-api/swagger-ui/compare/v5.17.13...v5.17.14";>5.17.14 (2024-05-28) Bug Fixes system: remediate component wrapping functionality (https://redirect.github.com/swagger-api/swagger-ui/issues/9978";>#9978) (https://github.com/swagger-api/swagger-ui/commit/8aa529201395224707ab00bb4b6ce71e41d082a4";>8aa5292), closes https://redirect.github.com/swagger-api/swagger-ui/issues/9919";>#9919 Swagger UI v5.17.13 Released! https://github.com/swagger-api/swagger-ui/compare/v5.17.12...v5.17.13";>5.17.13 (2024-05-27) Bug Fixes swagger-ui-react: re-render on spec prop change (https://redirect.github.com/swagger-api/swagger-ui/issues/9966";>#9966) (https://github.com/swagger-api/swagger-ui/commit/df03a8f99c498f5dbb07b6bf5e73b57ab95ea2ef";>df03a8f), closes https://redirect.github.com/swagger-api/swagger-ui/issues/9965";>#9965 Swagger UI v5.17.12 Released! https://github.com/swagger-api/swagger-ui/compare/v5.17.10...v5.17.12";>5.17.12 (2024-05-21) Bug Fixes config: define missing default options and their types (https://redirect.github.com/swagger-api/swagger-ui/issues/9949";>#9949) (https://github.com/swagger-api/swagger-ui/commit/04bbeaa68cf53af82ce633f369d6eb579c44d80d";>04bbeaa), closes https://redirect.github.com/swagger-api/swagger-ui/issues/9945";>#9945 release: fix failed v5.17.11 release (https://github.com/swagger-api/swagger-ui/commit/a35a71eb130283e75a29b977eef7eb4a87d779df";>a35a71e) swagger-client: fix OpenAPI 3.1.0 dereferencing (https://redirect.github.com/swagger-api/swagger-ui/issues/9950";>#9950) (https://github.com/swagger-api/swagger-ui/commit/10e1a5f1ea3abeb8c9ce0e0af8c860a86484ef41";>10e1a5f), closes https://redirect.github.com/swagger-api/swagger-ui/issues/9941";>#9941 Swagger UI v5.17.10 Released! https://github.com/swagger-api/swagger-ui/compare/v5.17.9...v5.17.10";>5.17.10 (2024-05-15) Bug Fixes components: add support for oneOf/anyOf JSON Schema keywords in parameter-row rendering (https://redirect.github.com/swagger-api/swagger-ui/issues/9934";>#9934) (https://github.com/swagger-api/swagger-ui/commit/9037acf508beacac1efcb4f39403c203097d193e";>9037acf), closes https://redirect.github.com/swagger-api/swagger-ui/issues/7912";>#7912 oas31: fix initial rendering of the Webhooks section (https://redirect.github.com/swagger-api/swagger-ui/issues/9938";>#9938) (https://github.com/swagger-api/swagger-ui/commit/b5c84f44c733d004c0c6098c840a68f284b80dd3";>b5c84f4), closes https://redirect.github.com/swagger-api/swagger-ui/issues/9937";>#9937 Swagger UI v5.17.9 Released! https://github.com/swagger-api/swagger-ui/compare/v5.17.8...v5.17.9";>5.17.9 (2024-05-13) Bug Fixes deps: allow to consolidate deps in downstream projects (https://redirect.github.com/swagger-api/swagger-ui/issues/9930";>#9930) (https://github.com/swagger-api/swagger-ui/commit/335372d7a89c3a109469c0abbb977839d341f203";>335372d), closes https://redirect.github.com/swagger-api/swagger-ui/issues/9929";>#9929 Swagger UI v5.17.8 Released! https://github.com/swagger-api/swagger-ui/compare/v5.17.7...v5.17.8";>5.17.8 (2024-05-13) Bug Fixes components: fix rendering for empty examples in responses (https://redirect.github.com/swagger-api/swagger-ui/issues/9926";>#9926) (https://github.com/swagger-api/swagger-ui/commit/94f2d8298a60a6cf29c1be84e622f476c9c1df91";>94f2d82), closes https://redirect.github.com/swagger-api/swagger-ui/issues/9499";>#9499 ... (truncated) Commits https://github.com/swagger-api/swagger-ui/commit/74ed0adebfc9c8dd0de2bf8e81495b022a66c083";>74ed0ad chore(release): cut the v5.17.14 release https://github.com/swagger-api/swagger-ui/commit/8aa529201395224707ab00bb4b6ce71e41d082a4";>8aa5292 fix(system): remediate component wrapping functionality (https://redirect.github.com/swagger-api/swagger-ui/issues/9978";>#9978) https://github.com/swagger-api/swagger-ui/commit/c74fb2c69d4d990752c6044b293cd7fdfd3b87ab";>c74fb2c chore(release): cut the v5.17.13 release https://github.com/swagger-api/swagger-ui/commit/31b35fdca461770251de7f2faddf7f2b3201009f";>31b35fd ci(dependabot): use times that don't conflict with releases (https://redirect.github.com/swagger-api/swagger-ui/issues/9975";>#9975) https://github.com/swagger-api/swagger-ui/commit/df03a8f99c498f5dbb07b6bf5e73b57ab95ea2ef";>df03a8f fix(swagger-ui-react): re-render on spec prop change (https://redirect.github.com/swagger-api/swagger-ui/issues/9966";>#9966) https://github.com/swagger-api/swagger-ui/commit/e
(pinot) branch dependabot/maven/org.apache.maven.plugins-maven-shade-plugin-3.6.0 created (now 6bb5fae9ae)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/org.apache.maven.plugins-maven-shade-plugin-3.6.0 in repository https://gitbox.apache.org/repos/asf/pinot.git at 6bb5fae9ae Bump org.apache.maven.plugins:maven-shade-plugin from 3.5.3 to 3.6.0 No new revisions were added by this update. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[PR] Bump org.apache.maven.plugins:maven-shade-plugin from 3.5.3 to 3.6.0 [pinot]
dependabot[bot] opened a new pull request, #13288: URL: https://github.com/apache/pinot/pull/13288 Bumps [org.apache.maven.plugins:maven-shade-plugin](https://github.com/apache/maven-shade-plugin) from 3.5.3 to 3.6.0. Commits https://github.com/apache/maven-shade-plugin/commit/9a572e232d652a12a2404fe3cbe57fb588a87318";>9a572e2 [maven-release-plugin] prepare release maven-shade-plugin-3.6.0 https://github.com/apache/maven-shade-plugin/commit/ade2e359214c4094876b0100c97b9a55aecc4c2e";>ade2e35 [MSHADE-428] Prevent null value in array of transformers (https://redirect.github.com/apache/maven-shade-plugin/issues/229";>#229) https://github.com/apache/maven-shade-plugin/commit/b573b8c498ea12cd2b84edb45be3036d9853d03e";>b573b8c [MSHADE-478] Extra JARs feature (https://redirect.github.com/apache/maven-shade-plugin/issues/228";>#228) https://github.com/apache/maven-shade-plugin/commit/199ffaecd26a912527173ed4edae366e48a00998";>199ffae Drop the cruft (https://redirect.github.com/apache/maven-shade-plugin/issues/225";>#225) https://github.com/apache/maven-shade-plugin/commit/912a81d95da870e2384359b2836caca4f2d1299f";>912a81d Bump maven-gh-actions-shared to v4 https://github.com/apache/maven-shade-plugin/commit/80e44203f4ed2169b6186a29a7112aa8ed01c7a3";>80e4420 [maven-release-plugin] prepare for next development iteration See full diff in https://github.com/apache/maven-shade-plugin/compare/maven-shade-plugin-3.5.3...maven-shade-plugin-3.6.0";>compare view [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- Dependabot commands and options You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch dependabot/maven/software.amazon.awssdk-bom-2.25.63 created (now c9dbb4bf81)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/software.amazon.awssdk-bom-2.25.63 in repository https://gitbox.apache.org/repos/asf/pinot.git at c9dbb4bf81 Bump software.amazon.awssdk:bom from 2.25.61 to 2.25.63 No new revisions were added by this update. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Bump org.webjars:swagger-ui from 5.17.0 to 5.17.11 [pinot]
dependabot[bot] commented on PR #13210: URL: https://github.com/apache/pinot/pull/13210#issuecomment-214180 Superseded by #13286. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[PR] Optimize snapshot flow to only snapshot segments which have updates [pinot]
tibrewalpratik17 opened a new pull request, #13285: URL: https://github.com/apache/pinot/pull/13285 label: `optimization` `enhancement` This patch enhances the `doTakeSnapshot` flow to not snapshot all segments in a given partition but only the ones which have been updated since last-snapshot taken. This particularly improves scenarios where the number of segments per partition is high. `doTakeSnapshot` workflow runs before new consuming segment starts consumption and directly introduces ingestion lag before starting consumption. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[I] Proposal for a new instance-partition based routing strategy [pinot]
tibrewalpratik17 opened a new issue, #13284: URL: https://github.com/apache/pinot/issues/13284 ### Current scenario All routing strategies are instance-based strategy. In one of recent incidents in our cluster, we saw a lot of query failures because 2 instances were marked as unavailable (unqueryable) and they were serving as 2 common replicas for around 3000 segments. During query, we marked these 3000 segments as unavailable. Why these 2 instances were marked as unqueryable was because 2 different segments were OFFLINE in these 2 instances. Example: There were serverA, serverB and there were 3000 segments which were having both replicas in these 2 servers. Now segmentX was OFFLINE in serverA and segmentY was OFFLINE in serverB. We are using `strictReplicaGroup` strategy and so all 3000 segments became unavailable as serverA, serverB were not treated as eligible serving candidates anymore. ### Possible prevention Both segmentX and segmentY were from different partitions. If we would have accounted serverA to not serve queries only for partitions as that of segmentX and same serverB to not serve queries only for partitions of segmentY, we could have prevented this incident. Note: if each server has only one partition then it makes sense to not allow it to serve queries completely. ### Proposal What we are proposing in this issue is to have a routing strategy like `instancePartitionReplicaGroup` (name can be discussed during implementation). Here, we will maintain a mapping of available instance-partitions and not just instances. If all segments of a partition is ONLINE in a given instance, then we enable it to serve query for that instance-partition. At present, we use Ideal-state to find segments --> instances mapping. In this scenario, we can even fallback to instanceToPartitions info in ZK to fetch this info. Note: this proposal only affects REALTIME tables and not OFFLINE tables. cc @ankitsultana -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Add support for raw JSON filter predicates [pinot]
codecov-commenter commented on PR #13283: URL: https://github.com/apache/pinot/pull/13283#issuecomment-2141635006 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/13283?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report All modified and coverable lines are covered by tests :white_check_mark: > Project coverage is 0.00%. Comparing base [(`59551e4`)](https://app.codecov.io/gh/apache/pinot/commit/59551e45224f1535c4863fd577622b37366ccc97?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`8d7e1e8`)](https://app.codecov.io/gh/apache/pinot/commit/8d7e1e884d28b717445d2faa614ddfe7c1474c46?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache). > Report is 523 commits behind head on master. Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #13283 +/- ## = - Coverage 61.75%0.00% -61.76% = Files 2436 2461 +25 Lines133233 135741 +2508 Branches 2063620994 +358 = - Hits 822740-82274 - Misses44911 135741+90830 + Partials 60480 -6048 ``` | [Flag](https://app.codecov.io/gh/apache/pinot/pull/13283/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | | |---|---|---| | [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/13283/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | | [integration](https://app.codecov.io/gh/apache/pinot/pull/13283/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (-0.01%)` | :arrow_down: | | [integration1](https://app.codecov.io/gh/apache/pinot/pull/13283/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | | [integration2](https://app.codecov.io/gh/apache/pinot/pull/13283/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (ø)` | | | [java-11](https://app.codecov.io/gh/apache/pinot/pull/13283/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | | [java-21](https://app.codecov.io/gh/apache/pinot/pull/13283/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (-61.63%)` | :arrow_down: | | [skip-bytebuffers-false](https://app.codecov.io/gh/apache/pinot/pull/13283/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (-61.75%)` | :arrow_down: | | [skip-bytebuffers-true](https://app.codecov.io/gh/apache/pinot/pull/13283/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (-27.73%)` | :arrow_down: | | [temurin](https://app.codecov.io/gh/apache/pinot/pull/13283/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (-61.76%)` | :arrow_down: | | [unittests](https://app.codecov.io/gh/apache/pinot/pull/13283/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | | [unittests1](https://app.codecov.io/gh/apache/pinot/pull/13283/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | | [unittests2](https://app.codecov.io/gh/apache/pinot/pull/13283/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more. [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/13283?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
[PR] Add support for raw JSON filter predicates [pinot]
yashmayya opened a new pull request, #13283: URL: https://github.com/apache/pinot/pull/13283 - Currently, raw JSON values (non dictionary encoded) can't be queried directly via predicates like `=`, `!=`, `IN`, `NOT IN` and JSON columns are typically queried using the `JSON_MATCH` filter predicate that uses the JSON index. - Attempting to do so results in errors like: ``` QueryExecutionError: org.apache.pinot.spi.exception.BadQueryRequestException: java.lang.IllegalStateException: Unsupported data type: JSON at org.apache.pinot.core.operator.filter.predicate.PredicateEvaluatorProvider.getPredicateEvaluator(PredicateEvaluatorProvider.java:94) at org.apache.pinot.core.operator.filter.predicate.PredicateEvaluatorProvider.getPredicateEvaluator(PredicateEvaluatorProvider.java:100) at org.apache.pinot.core.plan.FilterPlanNode.constructPhysicalOperator(FilterPlanNode.java:310) at org.apache.pinot.core.plan.FilterPlanNode.run(FilterPlanNode.java:93) ... Caused by: java.lang.IllegalStateException: Unsupported data type: JSON at org.apache.pinot.core.operator.filter.predicate.EqualsPredicateEvaluatorFactory.newRawValueBasedEvaluator(EqualsPredicateEvaluatorFactory.java:87) at org.apache.pinot.core.operator.filter.predicate.PredicateEvaluatorProvider.getPredicateEvaluator(PredicateEvaluatorProvider.java:76) ... 24 more ``` - However, for certain simple JSON values, it could be useful to allow direct comparison using predicates like `=`, `!=`, `IN`, `NOT IN` via direct string matching like is done for `STRING` columns since the stored type for JSON columns is also `STRING`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Add config to skip record ingestion on string column length exceeding configured max schema length [pinot]
tibrewalpratik17 commented on code in PR #13103:
URL: https://github.com/apache/pinot/pull/13103#discussion_r1622005593
##
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/recordtransformer/SanitizationTransformer.java:
##
@@ -32,51 +34,197 @@
*
* No {@code null} characters in string values
* String values are within the length limit
- * TODO: add length limit to BYTES values if necessary
*
* NOTE: should put this after the {@link DataTypeTransformer} so that all
values follow the data types in
* {@link FieldSpec}.
+ * This uses the MaxLengthExceedStrategy in the {@link FieldSpec} to decide
what to do when the value exceeds the max.
+ * For TRIM_LENGTH, the value is trimmed to the max length.
+ * For SUBSTITUTE_DEFAULT_VALUE, the value is replaced with the default null
value string.
+ * For ERROR, an exception is thrown and the record is skipped.
+ * For NO_ACTION, the value is kept as is if no NULL_CHARACTER present else
trimmed till NULL.
+ * In the first 2 scenarios, this metric INCOMPLETE_REALTIME_ROWS_CONSUMED can
be tracked to know if a trimmed /
+ * default record was persisted.
+ * In the third scenario, this metric ROWS_WITH_ERRORS can be tracked to know
if a record was skipped.
+ * In the last scenario, this metric INCOMPLETE_REALTIME_ROWS_CONSUMED can be
tracked to know if a record was trimmed
+ * due to having a null character.
*/
public class SanitizationTransformer implements RecordTransformer {
- private final Map _stringColumnMaxLengthMap = new
HashMap<>();
+ private static final String NULL_CHARACTER = "\0";
+ private final Map _columnToColumnInfoMap = new
HashMap<>();
public SanitizationTransformer(Schema schema) {
+FieldSpec.MaxLengthExceedStrategy maxLengthExceedStrategy;
for (FieldSpec fieldSpec : schema.getAllFieldSpecs()) {
- if (!fieldSpec.isVirtualColumn() && fieldSpec.getDataType() ==
DataType.STRING) {
-_stringColumnMaxLengthMap.put(fieldSpec.getName(),
fieldSpec.getMaxLength());
+ if (!fieldSpec.isVirtualColumn()) {
+switch (fieldSpec.getDataType()) {
+ case STRING:
+maxLengthExceedStrategy = fieldSpec.getMaxLengthExceedStrategy()
== null
+? FieldSpec.MaxLengthExceedStrategy.TRIM_LENGTH :
fieldSpec.getMaxLengthExceedStrategy();
+_columnToColumnInfoMap.put(fieldSpec.getName(), new
SanitizedColumnInfo(fieldSpec.getName(),
+fieldSpec.getMaxLength(), maxLengthExceedStrategy,
fieldSpec.getDefaultNullValueString()));
+break;
+ case JSON:
+ case BYTES:
+maxLengthExceedStrategy = fieldSpec.getMaxLengthExceedStrategy()
== null
+? FieldSpec.MaxLengthExceedStrategy.NO_ACTION :
fieldSpec.getMaxLengthExceedStrategy();
+_columnToColumnInfoMap.put(fieldSpec.getName(), new
SanitizedColumnInfo(fieldSpec.getName(),
+fieldSpec.getMaxLength(), maxLengthExceedStrategy,
fieldSpec.getDefaultNullValueString()));
+break;
+ default:
+// Do nothing for other data types
+}
}
}
}
@Override
public boolean isNoOp() {
-return _stringColumnMaxLengthMap.isEmpty();
+return _columnToColumnInfoMap.isEmpty();
}
@Override
public GenericRow transform(GenericRow record) {
-for (Map.Entry entry :
_stringColumnMaxLengthMap.entrySet()) {
- String stringColumn = entry.getKey();
- int maxLength = entry.getValue();
- Object value = record.getValue(stringColumn);
- if (value instanceof String) {
-// Single-valued column
-String stringValue = (String) value;
-String sanitizedValue = StringUtil.sanitizeStringValue(stringValue,
maxLength);
-// NOTE: reference comparison
-//noinspection StringEquality
-if (sanitizedValue != stringValue) {
- record.putValue(stringColumn, sanitizedValue);
+for (Map.Entry entry :
_columnToColumnInfoMap.entrySet()) {
+ String columnName = entry.getKey();
+ Object value = record.getValue(columnName);
+ Pair result;
+ if (value instanceof byte[]) {
+// Single-values BYTES column
+result = sanitizeBytesValue(columnName, (byte[]) value,
entry.getValue());
+record.putValue(columnName, result.getLeft());
+if (result.getRight()) {
+ record.putValue(GenericRow.INCOMPLETE_RECORD_KEY, true);
Review Comment:
It was similar effort to remove this so i added `SANITIZED_RECORD_KEY`
instead
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-uns
Re: [PR] Add config to skip record ingestion on string column length exceeding configured max schema length [pinot]
tibrewalpratik17 commented on code in PR #13103:
URL: https://github.com/apache/pinot/pull/13103#discussion_r1622004889
##
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/recordtransformer/SanitizationTransformer.java:
##
@@ -32,51 +34,197 @@
*
* No {@code null} characters in string values
* String values are within the length limit
- * TODO: add length limit to BYTES values if necessary
*
* NOTE: should put this after the {@link DataTypeTransformer} so that all
values follow the data types in
* {@link FieldSpec}.
+ * This uses the MaxLengthExceedStrategy in the {@link FieldSpec} to decide
what to do when the value exceeds the max.
+ * For TRIM_LENGTH, the value is trimmed to the max length.
+ * For SUBSTITUTE_DEFAULT_VALUE, the value is replaced with the default null
value string.
+ * For ERROR, an exception is thrown and the record is skipped.
+ * For NO_ACTION, the value is kept as is if no NULL_CHARACTER present else
trimmed till NULL.
+ * In the first 2 scenarios, this metric INCOMPLETE_REALTIME_ROWS_CONSUMED can
be tracked to know if a trimmed /
+ * default record was persisted.
+ * In the third scenario, this metric ROWS_WITH_ERRORS can be tracked to know
if a record was skipped.
+ * In the last scenario, this metric INCOMPLETE_REALTIME_ROWS_CONSUMED can be
tracked to know if a record was trimmed
+ * due to having a null character.
*/
public class SanitizationTransformer implements RecordTransformer {
- private final Map _stringColumnMaxLengthMap = new
HashMap<>();
+ private static final String NULL_CHARACTER = "\0";
+ private final Map _columnToColumnInfoMap = new
HashMap<>();
public SanitizationTransformer(Schema schema) {
+FieldSpec.MaxLengthExceedStrategy maxLengthExceedStrategy;
for (FieldSpec fieldSpec : schema.getAllFieldSpecs()) {
- if (!fieldSpec.isVirtualColumn() && fieldSpec.getDataType() ==
DataType.STRING) {
-_stringColumnMaxLengthMap.put(fieldSpec.getName(),
fieldSpec.getMaxLength());
+ if (!fieldSpec.isVirtualColumn()) {
+switch (fieldSpec.getDataType()) {
+ case STRING:
+maxLengthExceedStrategy = fieldSpec.getMaxLengthExceedStrategy()
== null
+? FieldSpec.MaxLengthExceedStrategy.TRIM_LENGTH :
fieldSpec.getMaxLengthExceedStrategy();
+_columnToColumnInfoMap.put(fieldSpec.getName(), new
SanitizedColumnInfo(fieldSpec.getName(),
+fieldSpec.getMaxLength(), maxLengthExceedStrategy,
fieldSpec.getDefaultNullValueString()));
+break;
+ case JSON:
+ case BYTES:
+maxLengthExceedStrategy = fieldSpec.getMaxLengthExceedStrategy()
== null
+? FieldSpec.MaxLengthExceedStrategy.NO_ACTION :
fieldSpec.getMaxLengthExceedStrategy();
+_columnToColumnInfoMap.put(fieldSpec.getName(), new
SanitizedColumnInfo(fieldSpec.getName(),
Review Comment:
In case of string do we want to ingest `NULL_CHARACTER` in case of
`NO_ACTION` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Pass literal within AggregateCall via rexList [pinot]
Jackie-Jiang commented on code in PR #13282:
URL: https://github.com/apache/pinot/pull/13282#discussion_r1621891622
##
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/AggregateOperator.java:
##
@@ -244,78 +215,26 @@ private TransferableBlock consumeAggregation() {
return COUNT_STAR_AGG_FUNCTION;
}
List arguments = new ArrayList<>(numArguments);
-for (int i = 0; i < numArguments; i++) {
- Literal literalArgument = literalArguments.get(i);
- if (literalArgument != null) {
-arguments.add(ExpressionContext.forLiteralContext(literalArgument));
+for (RexExpression operand : operands) {
+ if (operand instanceof RexExpression.InputRef) {
+RexExpression.InputRef inputRef = (RexExpression.InputRef) operand;
+
arguments.add(ExpressionContext.forIdentifier(fromColIdToIdentifier(inputRef.getIndex(;
} else {
-RexExpression operand = operands.get(i);
-switch (operand.getKind()) {
- case INPUT_REF:
-RexExpression.InputRef inputRef = (RexExpression.InputRef) operand;
-
arguments.add(ExpressionContext.forIdentifier(fromColIdToIdentifier(inputRef.getIndex(;
-break;
- case LITERAL:
-RexExpression.Literal literalRexExp = (RexExpression.Literal)
operand;
-
arguments.add(ExpressionContext.forLiteralContext(literalRexExp.getDataType().toDataType(),
-literalRexExp.getValue()));
-break;
- default:
-throw new IllegalStateException("Illegal aggregation function
operand type: " + operand.getKind());
+assert operand instanceof RexExpression.Literal;
Review Comment:
It won't be. We only extract literal as operand, others will remain input
reference.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Add backward compatibility regression test suite for multi-stage query engine [pinot]
yashmayya commented on code in PR #13193:
URL: https://github.com/apache/pinot/pull/13193#discussion_r1621877566
##
pinot-common/src/main/java/org/apache/pinot/common/utils/SqlResultComparator.java:
##
@@ -163,6 +167,97 @@ public static boolean areEqual(JsonNode actual, JsonNode
expected, String query)
return areResultsEqual;
}
+ public static boolean areMultiStageQueriesEqual(JsonNode actual, JsonNode
expected, String query)
+ throws IOException {
+if (hasExceptions(actual)) {
+ return false;
+}
+
+if (areEmpty(actual, expected)) {
+ return true;
+}
+
+if (!areDataSchemaEqual(actual, expected)) {
+ return false;
+}
+
+ArrayNode actualRows = (ArrayNode)
actual.get(FIELD_RESULT_TABLE).get(FIELD_ROWS);
+ArrayNode expectedRows = (ArrayNode)
expected.get(FIELD_RESULT_TABLE).get(FIELD_ROWS);
+ArrayNode columnDataTypes = (ArrayNode)
expected.get(FIELD_RESULT_TABLE).get(FIELD_DATA_SCHEMA).
+get(FIELD_COLUMN_DATA_TYPES);
+
+convertNumbersToString(expectedRows, columnDataTypes);
+convertNumbersToString(actualRows, columnDataTypes);
+
+List actualElementsSerialized = new ArrayList<>();
+List expectedElementsSerialized = new ArrayList<>();
+for (int i = 0; i < actualRows.size(); i++) {
+ actualElementsSerialized.add(actualRows.get(i).toString());
+}
+for (int i = 0; i < expectedRows.size(); i++) {
+ expectedElementsSerialized.add(expectedRows.get(i).toString());
+}
+
+if (!areLengthsEqual(actual, expected)) {
+ return false;
+}
+
+// For now, just directly compare elements in result set
+if (!areNonOrderByQueryElementsEqual(actualElementsSerialized,
expectedElementsSerialized)) {
+ return false;
+}
+
+// Compare stage stats
+ObjectNode actualStageStats = (ObjectNode)
actual.get(FIELD_MULTI_STAGE_STATS);
+ObjectNode expectedStageStats = (ObjectNode)
expected.get(FIELD_MULTI_STAGE_STATS);
+return areMultiStageStatsEqual(actualStageStats, expectedStageStats);
+ }
+
+ private static boolean areMultiStageStatsEqual(ObjectNode actualStageStats,
ObjectNode expectedStageStats) {
+String actualType = actualStageStats.get(FIELD_MULTI_STAGE_STATS_TYPE) !=
null
+? actualStageStats.get(FIELD_MULTI_STAGE_STATS_TYPE).asText()
+: null;
+String expectedType = expectedStageStats.get(FIELD_MULTI_STAGE_STATS_TYPE)
!= null
+? expectedStageStats.get(FIELD_MULTI_STAGE_STATS_TYPE).asText()
+: null;
+
+if (actualType != null && !actualType.equals(expectedType)) {
+ LOGGER.error("Mismatch in stage stats type. Actual: {}, Expected: {}",
actualType, expectedType);
+ return false;
+}
+
+ArrayNode actualChildren = (ArrayNode)
actualStageStats.get(FIELD_MULTI_STAGE_STATS_CHILDREN);
+ArrayNode expectedChildren = (ArrayNode)
expectedStageStats.get(FIELD_MULTI_STAGE_STATS_CHILDREN);
+
+if (actualChildren == null && expectedChildren == null) {
+ return true;
+}
+if (actualChildren == null) {
+ LOGGER.error("No children found in stage stats for type: {}. Expected {}
children.",
+ actualType, expectedChildren.size());
+ return false;
+}
+if (expectedChildren == null) {
+ LOGGER.error("Found unexpected children in stage stats for type: {}.
Expected no children.", actualType);
+ return false;
+}
+if (actualChildren.size() != expectedChildren.size()) {
+ LOGGER.error("Mismatch in number of children for stage stats for type:
{}. Actual: {}, Expected: {}",
+ actualType, actualChildren.size(), expectedChildren.size());
+ return false;
+}
+
+for (int i = 0; i < actualChildren.size(); i++) {
+ if (!areMultiStageStatsEqual((ObjectNode) actualChildren.get(i),
(ObjectNode) expectedChildren.get(i))) {
+return false;
+ }
+}
+
+// TODO: Verify other stats like emittedRows, fanIn, fanOut,
inMemoryMessages, rawMessages etc.?
Review Comment:
Discussed this offline, and we've decided to remove the multi-stage stats
verification altogether since that is still subject to change across versions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
