[GitHub] clintropolis opened a new pull request #6397: Adds bloom filter aggregator to 'druid-bloom-filters' extension
clintropolis opened a new pull request #6397: Adds bloom filter aggregator to
'druid-bloom-filters' extension
URL: https://github.com/apache/incubator-druid/pull/6397
This PR, building on top of the work introduced in #6222, extends
`druid-bloom-filters` with an aggregator, allowing bloom filters to be computed
from query results which can then be used as input to `bloom` filters in
subsequent queries.
example query:
```json
{
"queryType": "timeseries",
"dataSource": "wikiticker",
"intervals": [ "2015-09-12T00:00:00.000/2015-09-13T00:00:00.000" ],
"granularity": "day",
"aggregations": [
{
"type": "bloom",
"name": "userBloom",
"maxNumEntries": 10,
"field": {
"type":"default",
"dimension":"user",
"outputType": "STRING"
}
}
]
}
```
example results:
```json
[{"timestamp":"2015-09-12T00:00:00.000Z","result":{"userBloom":"BAAAJh..."}}]
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] brunodebus opened a new issue #6398: Maven artifacts for druid-0.12.3 missing
brunodebus opened a new issue #6398: Maven artifacts for druid-0.12.3 missing URL: https://github.com/apache/incubator-druid/issues/6398 I can find the maven artifacts for druid 0.12.2 in maven central, but cannot locate the ones from 0.12.3. Is it possible to release them? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] egor-ryashin commented on a change in pull request #6349: maintenance mode for Historical
egor-ryashin commented on a change in pull request #6349: maintenance mode for
Historical
URL: https://github.com/apache/incubator-druid/pull/6349#discussion_r221182166
##
File path:
server/src/test/java/org/apache/druid/server/coordinator/rules/BroadcastDistributionRuleTest.java
##
@@ -253,6 +313,46 @@ public void testBroadcastToSingleDataSource()
assertFalse(holderOfSmallSegment.getPeon().getSegmentsToLoad().contains(smallSegment));
}
+ /**
+ * Servers:
+ * name | segments
+ * -+--
+ * general | large segment
+ * maintenance1 | small segment
+ * maintenance2 | large segment
+ *
+ * After running the rule for the small segment:
+ * general | large & small segments
+ * maintenance1 |
+ * maintenance2 | large segment
+ */
+ @Test
+ public void testBroadcastWithMaintenance()
+ {
+final ForeverBroadcastDistributionRule rule = new
ForeverBroadcastDistributionRule(ImmutableList.of("large_source"));
+
+CoordinatorStats stats = rule.run(
+null,
+DruidCoordinatorRuntimeParams.newBuilder()
+ .withDruidCluster(secondCluster)
+
.withSegmentReplicantLookup(SegmentReplicantLookup.make(secondCluster))
+
.withBalancerReferenceTimestamp(DateTimes.of("2013-01-01"))
+ .withAvailableSegments(Lists.newArrayList(
Review comment:
Sure, but imagine, for example, given a production class with a single
method, I add 3 methods which are used only in tests, the test will look more
succinct, but the class won't. The coder who needs to understand the logic of
the whole component needs to check usages of all 4 methods and scan through
them to confirm that those 3 used only in tests and are not called in
production and shouldn't be followed to understand how the component operates,
waste of time and concentration, I think.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] egor-ryashin commented on a change in pull request #6349: maintenance mode for Historical
egor-ryashin commented on a change in pull request #6349: maintenance mode for
Historical
URL: https://github.com/apache/incubator-druid/pull/6349#discussion_r221182166
##
File path:
server/src/test/java/org/apache/druid/server/coordinator/rules/BroadcastDistributionRuleTest.java
##
@@ -253,6 +313,46 @@ public void testBroadcastToSingleDataSource()
assertFalse(holderOfSmallSegment.getPeon().getSegmentsToLoad().contains(smallSegment));
}
+ /**
+ * Servers:
+ * name | segments
+ * -+--
+ * general | large segment
+ * maintenance1 | small segment
+ * maintenance2 | large segment
+ *
+ * After running the rule for the small segment:
+ * general | large & small segments
+ * maintenance1 |
+ * maintenance2 | large segment
+ */
+ @Test
+ public void testBroadcastWithMaintenance()
+ {
+final ForeverBroadcastDistributionRule rule = new
ForeverBroadcastDistributionRule(ImmutableList.of("large_source"));
+
+CoordinatorStats stats = rule.run(
+null,
+DruidCoordinatorRuntimeParams.newBuilder()
+ .withDruidCluster(secondCluster)
+
.withSegmentReplicantLookup(SegmentReplicantLookup.make(secondCluster))
+
.withBalancerReferenceTimestamp(DateTimes.of("2013-01-01"))
+ .withAvailableSegments(Lists.newArrayList(
Review comment:
Sure, but imagine, for example, given a production class with a single
method, I add 3 methods which are used only in tests, the test will look more
succinct, but the class won't. The coder who needs to understand the logic of
the whole component needs to check usages of all 4 methods and scan through
them to confirm that those 3 used only in tests and are not called in
production and shouldn't be followed to understand how the component operates -
waste of time and concentration, I think.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] egor-ryashin commented on a change in pull request #6349: maintenance mode for Historical
egor-ryashin commented on a change in pull request #6349: maintenance mode for
Historical
URL: https://github.com/apache/incubator-druid/pull/6349#discussion_r221182166
##
File path:
server/src/test/java/org/apache/druid/server/coordinator/rules/BroadcastDistributionRuleTest.java
##
@@ -253,6 +313,46 @@ public void testBroadcastToSingleDataSource()
assertFalse(holderOfSmallSegment.getPeon().getSegmentsToLoad().contains(smallSegment));
}
+ /**
+ * Servers:
+ * name | segments
+ * -+--
+ * general | large segment
+ * maintenance1 | small segment
+ * maintenance2 | large segment
+ *
+ * After running the rule for the small segment:
+ * general | large & small segments
+ * maintenance1 |
+ * maintenance2 | large segment
+ */
+ @Test
+ public void testBroadcastWithMaintenance()
+ {
+final ForeverBroadcastDistributionRule rule = new
ForeverBroadcastDistributionRule(ImmutableList.of("large_source"));
+
+CoordinatorStats stats = rule.run(
+null,
+DruidCoordinatorRuntimeParams.newBuilder()
+ .withDruidCluster(secondCluster)
+
.withSegmentReplicantLookup(SegmentReplicantLookup.make(secondCluster))
+
.withBalancerReferenceTimestamp(DateTimes.of("2013-01-01"))
+ .withAvailableSegments(Lists.newArrayList(
Review comment:
Sure, but imagine, for example, given a production class with a single
method, I add 3 methods which are used only in tests, the test will look more
succinct, but the class won't. The coder who needs to understand the logic of
the whole component needs to check usages of all 4 methods and scan through
them to confirm that those 3 used only in tests and are not called in
production and shouldn't be followed to understand how the component operates -
waste of time and concentration, I think.
Meanwhile it's could be impossible to use a debugger in a production to
understand the workflow, while you can run test in debugger easily if you don't
get some steps.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] egor-ryashin commented on a change in pull request #6349: maintenance mode for Historical
egor-ryashin commented on a change in pull request #6349: maintenance mode for
Historical
URL: https://github.com/apache/incubator-druid/pull/6349#discussion_r221182166
##
File path:
server/src/test/java/org/apache/druid/server/coordinator/rules/BroadcastDistributionRuleTest.java
##
@@ -253,6 +313,46 @@ public void testBroadcastToSingleDataSource()
assertFalse(holderOfSmallSegment.getPeon().getSegmentsToLoad().contains(smallSegment));
}
+ /**
+ * Servers:
+ * name | segments
+ * -+--
+ * general | large segment
+ * maintenance1 | small segment
+ * maintenance2 | large segment
+ *
+ * After running the rule for the small segment:
+ * general | large & small segments
+ * maintenance1 |
+ * maintenance2 | large segment
+ */
+ @Test
+ public void testBroadcastWithMaintenance()
+ {
+final ForeverBroadcastDistributionRule rule = new
ForeverBroadcastDistributionRule(ImmutableList.of("large_source"));
+
+CoordinatorStats stats = rule.run(
+null,
+DruidCoordinatorRuntimeParams.newBuilder()
+ .withDruidCluster(secondCluster)
+
.withSegmentReplicantLookup(SegmentReplicantLookup.make(secondCluster))
+
.withBalancerReferenceTimestamp(DateTimes.of("2013-01-01"))
+ .withAvailableSegments(Lists.newArrayList(
Review comment:
Sure, but imagine, for example, given a production class with a single
method, I add 3 methods which are used only in tests, the test will look more
succinct, but the class won't. The coder who needs to understand the logic of
the whole component needs to check usages of all 4 methods and scan through
them to confirm that those 3 used only in tests and are not called in
production and shouldn't be followed to understand how the component operates -
waste of time and concentration, I think.
Meanwhile it's could be impossible to use a debugger in a production to
understand the workflow, while you can run the test in debugger easily if you
don't get some steps.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] egor-ryashin commented on a change in pull request #6349: maintenance mode for Historical
egor-ryashin commented on a change in pull request #6349: maintenance mode for
Historical
URL: https://github.com/apache/incubator-druid/pull/6349#discussion_r221182166
##
File path:
server/src/test/java/org/apache/druid/server/coordinator/rules/BroadcastDistributionRuleTest.java
##
@@ -253,6 +313,46 @@ public void testBroadcastToSingleDataSource()
assertFalse(holderOfSmallSegment.getPeon().getSegmentsToLoad().contains(smallSegment));
}
+ /**
+ * Servers:
+ * name | segments
+ * -+--
+ * general | large segment
+ * maintenance1 | small segment
+ * maintenance2 | large segment
+ *
+ * After running the rule for the small segment:
+ * general | large & small segments
+ * maintenance1 |
+ * maintenance2 | large segment
+ */
+ @Test
+ public void testBroadcastWithMaintenance()
+ {
+final ForeverBroadcastDistributionRule rule = new
ForeverBroadcastDistributionRule(ImmutableList.of("large_source"));
+
+CoordinatorStats stats = rule.run(
+null,
+DruidCoordinatorRuntimeParams.newBuilder()
+ .withDruidCluster(secondCluster)
+
.withSegmentReplicantLookup(SegmentReplicantLookup.make(secondCluster))
+
.withBalancerReferenceTimestamp(DateTimes.of("2013-01-01"))
+ .withAvailableSegments(Lists.newArrayList(
Review comment:
Sure, but imagine, for example, given a production class with a single
method, I add 3 methods which are used only in tests, the test will look more
succinct, but the class won't. The coder who needs to understand the logic of
the whole component needs to check usages of all 4 methods and scan through
them to confirm that those 3 used only in tests and are not called in
production and shouldn't be followed to understand how the component operates -
waste of time and concentration, I think.
Meanwhile it's could be impossible to use a debugger in a production to
understand the workflow, while you can run the test in debugger easily if you
don't get some steps.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] adursun opened a new pull request #6399: Add missing comma
adursun opened a new pull request #6399: Add missing comma URL: https://github.com/apache/incubator-druid/pull/6399 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] kevinconaway commented on issue #6350: Remove ConvertSegmentTask, HadoopConverterTask, and ConvertSegmentBackwardsCompatibleTask
kevinconaway commented on issue #6350: Remove ConvertSegmentTask, HadoopConverterTask, and ConvertSegmentBackwardsCompatibleTask URL: https://github.com/apache/incubator-druid/issues/6350#issuecomment-425408340 Does this need to be a blocker for 0.13? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] egor-ryashin commented on a change in pull request #6349: maintenance mode for Historical
egor-ryashin commented on a change in pull request #6349: maintenance mode for
Historical
URL: https://github.com/apache/incubator-druid/pull/6349#discussion_r221224655
##
File path:
server/src/main/java/org/apache/druid/server/coordinator/helper/DruidCoordinatorBalancer.java
##
@@ -106,35 +104,76 @@ private void balanceTier(
return;
}
-final List toMoveFrom = Lists.newArrayList(servers);
-final List toMoveTo = Lists.newArrayList(servers);
+Map> partitions = servers.stream()
+
.collect(Collectors.partitioningBy(ServerHolder::isMaintenance));
+final List maintenance = partitions.get(true);
+final List general = partitions.get(false);
+log.info("Found %d servers in maintenance, %d general servers",
maintenance.size(), general.size());
-if (toMoveTo.size() <= 1) {
- log.info("[%s]: One or fewer servers found. Cannot balance.", tier);
+if (general.size() == 0) {
+ log.info("[%s]: 0 general servers found. Cannot balance.", tier);
return;
}
int numSegments = 0;
-for (ServerHolder sourceHolder : toMoveFrom) {
+for (ServerHolder sourceHolder : servers) {
numSegments += sourceHolder.getServer().getSegments().size();
}
-
if (numSegments == 0) {
log.info("No segments found. Cannot balance.");
return;
}
-final BalancerStrategy strategy = params.getBalancerStrategy();
final int maxSegmentsToMove =
Math.min(params.getCoordinatorDynamicConfig().getMaxSegmentsToMove(),
numSegments);
+int priority =
params.getCoordinatorDynamicConfig().getMaintenanceModeSegmentsPriority();
Review comment:
or `maintenancePriority`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] egor-ryashin commented on a change in pull request #6349: maintenance mode for Historical
egor-ryashin commented on a change in pull request #6349: maintenance mode for Historical URL: https://github.com/apache/incubator-druid/pull/6349#discussion_r221225390 ## File path: server/src/main/java/org/apache/druid/server/coordinator/CoordinatorDynamicConfig.java ## @@ -51,6 +51,8 @@ private final boolean emitBalancingStats; private final boolean killAllDataSources; private final Set killDataSourceWhitelist; + private final Set maintenanceList; Review comment: Changed to `historicalMaintenanceNodes` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] egor-ryashin commented on a change in pull request #6349: maintenance mode for Historical
egor-ryashin commented on a change in pull request #6349: maintenance mode for Historical URL: https://github.com/apache/incubator-druid/pull/6349#discussion_r221225390 ## File path: server/src/main/java/org/apache/druid/server/coordinator/CoordinatorDynamicConfig.java ## @@ -51,6 +51,8 @@ private final boolean emitBalancingStats; private final boolean killAllDataSources; private final Set killDataSourceWhitelist; + private final Set maintenanceList; Review comment: Changed to `historicalNodesInMaintenance` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] egor-ryashin commented on a change in pull request #6349: maintenance mode for Historical
egor-ryashin commented on a change in pull request #6349: maintenance mode for
Historical
URL: https://github.com/apache/incubator-druid/pull/6349#discussion_r221226828
##
File path:
server/src/main/java/org/apache/druid/server/coordinator/helper/DruidCoordinatorBalancer.java
##
@@ -106,35 +104,76 @@ private void balanceTier(
return;
}
-final List toMoveFrom = Lists.newArrayList(servers);
-final List toMoveTo = Lists.newArrayList(servers);
+Map> partitions = servers.stream()
+
.collect(Collectors.partitioningBy(ServerHolder::isMaintenance));
+final List maintenance = partitions.get(true);
+final List general = partitions.get(false);
+log.info("Found %d servers in maintenance, %d general servers",
maintenance.size(), general.size());
-if (toMoveTo.size() <= 1) {
- log.info("[%s]: One or fewer servers found. Cannot balance.", tier);
+if (general.size() == 0) {
+ log.info("[%s]: 0 general servers found. Cannot balance.", tier);
return;
}
int numSegments = 0;
-for (ServerHolder sourceHolder : toMoveFrom) {
+for (ServerHolder sourceHolder : servers) {
numSegments += sourceHolder.getServer().getSegments().size();
}
-
if (numSegments == 0) {
log.info("No segments found. Cannot balance.");
return;
}
-final BalancerStrategy strategy = params.getBalancerStrategy();
final int maxSegmentsToMove =
Math.min(params.getCoordinatorDynamicConfig().getMaxSegmentsToMove(),
numSegments);
+int priority =
params.getCoordinatorDynamicConfig().getMaintenanceModeSegmentsPriority();
Review comment:
Changed to `nodesInMaintenancePriority`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] egor-ryashin commented on a change in pull request #6349: maintenance mode for Historical
egor-ryashin commented on a change in pull request #6349: maintenance mode for Historical URL: https://github.com/apache/incubator-druid/pull/6349#discussion_r221227926 ## File path: server/src/main/java/org/apache/druid/server/coordinator/CoordinatorDynamicConfig.java ## @@ -95,6 +100,8 @@ public CoordinatorDynamicConfig( this.killDataSourceWhitelist = parseJsonStringOrArray(killDataSourceWhitelist); this.killPendingSegmentsSkipList = parseJsonStringOrArray(killPendingSegmentsSkipList); this.maxSegmentsInNodeLoadingQueue = maxSegmentsInNodeLoadingQueue; +this.maintenanceList = parseJsonStringOrArray(maintenanceList); +this.maintenanceModeSegmentsPriority = Math.min(10, Math.max(0, maintenanceModeSegmentsPriority)); Review comment: :white_check_mark: This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] egor-ryashin commented on a change in pull request #6349: maintenance mode for Historical
egor-ryashin commented on a change in pull request #6349: maintenance mode for Historical URL: https://github.com/apache/incubator-druid/pull/6349#discussion_r221228324 ## File path: server/src/main/java/org/apache/druid/server/coordinator/ServerHolder.java ## @@ -33,14 +33,22 @@ private static final Logger log = new Logger(ServerHolder.class); private final ImmutableDruidServer server; private final LoadQueuePeon peon; + private final boolean maintenance; Review comment: :white_check_mark: This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] egor-ryashin commented on a change in pull request #6349: maintenance mode for Historical
egor-ryashin commented on a change in pull request #6349: maintenance mode for
Historical
URL: https://github.com/apache/incubator-druid/pull/6349#discussion_r221228346
##
File path:
server/src/main/java/org/apache/druid/server/coordinator/ServerHolder.java
##
@@ -33,14 +33,22 @@
private static final Logger log = new Logger(ServerHolder.class);
private final ImmutableDruidServer server;
private final LoadQueuePeon peon;
+ private final boolean maintenance;
+
+ public ServerHolder(ImmutableDruidServer server, LoadQueuePeon peon)
+ {
+this(server, peon, false);
+ }
public ServerHolder(
Review comment:
:white_check_mark:
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] egor-ryashin commented on a change in pull request #6349: maintenance mode for Historical
egor-ryashin commented on a change in pull request #6349: maintenance mode for
Historical
URL: https://github.com/apache/incubator-druid/pull/6349#discussion_r221228420
##
File path:
server/src/main/java/org/apache/druid/server/coordinator/helper/DruidCoordinatorBalancer.java
##
@@ -106,35 +104,76 @@ private void balanceTier(
return;
}
-final List toMoveFrom = Lists.newArrayList(servers);
-final List toMoveTo = Lists.newArrayList(servers);
+Map> partitions = servers.stream()
+
.collect(Collectors.partitioningBy(ServerHolder::isMaintenance));
+final List maintenance = partitions.get(true);
+final List general = partitions.get(false);
+log.info("Found %d servers in maintenance, %d general servers",
maintenance.size(), general.size());
-if (toMoveTo.size() <= 1) {
- log.info("[%s]: One or fewer servers found. Cannot balance.", tier);
+if (general.size() == 0) {
Review comment:
:white_check_mark:
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] egor-ryashin commented on a change in pull request #6349: maintenance mode for Historical
egor-ryashin commented on a change in pull request #6349: maintenance mode for
Historical
URL: https://github.com/apache/incubator-druid/pull/6349#discussion_r221228625
##
File path:
server/src/main/java/org/apache/druid/server/coordinator/rules/LoadRule.java
##
@@ -144,14 +144,15 @@ private void assign(
private static List getFilteredHolders(
final String tier,
final DruidCluster druidCluster,
- final Predicate predicate
+ Predicate predicate
)
{
final NavigableSet queue =
druidCluster.getHistoricalsByTier(tier);
if (queue == null) {
log.makeAlert("Tier[%s] has no servers! Check your cluster
configuration!", tier).emit();
return Collections.emptyList();
}
+predicate = predicate.and(s -> !s.isMaintenance());
Review comment:
:white_check_mark:
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] egor-ryashin commented on a change in pull request #6349: maintenance mode for Historical
egor-ryashin commented on a change in pull request #6349: maintenance mode for
Historical
URL: https://github.com/apache/incubator-druid/pull/6349#discussion_r221228590
##
File path:
server/src/main/java/org/apache/druid/server/coordinator/rules/BroadcastDistributionRule.java
##
@@ -49,7 +49,8 @@ public CoordinatorStats run(
params.getDruidCluster().getAllServers().forEach(
eachHolder -> {
if (colocatedDataSources.stream()
-.anyMatch(source ->
eachHolder.getServer().getDataSource(source) != null)) {
+.anyMatch(source ->
eachHolder.getServer().getDataSource(source) != null
+&&
!eachHolder.isMaintenance())) {
Review comment:
:white_check_mark:
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] egor-ryashin commented on a change in pull request #6349: maintenance mode for Historical
egor-ryashin commented on a change in pull request #6349: maintenance mode for
Historical
URL: https://github.com/apache/incubator-druid/pull/6349#discussion_r221228486
##
File path:
server/src/main/java/org/apache/druid/server/coordinator/helper/DruidCoordinatorBalancer.java
##
@@ -106,35 +104,76 @@ private void balanceTier(
return;
}
-final List toMoveFrom = Lists.newArrayList(servers);
-final List toMoveTo = Lists.newArrayList(servers);
+Map> partitions = servers.stream()
+
.collect(Collectors.partitioningBy(ServerHolder::isMaintenance));
+final List maintenance = partitions.get(true);
+final List general = partitions.get(false);
+log.info("Found %d servers in maintenance, %d general servers",
maintenance.size(), general.size());
-if (toMoveTo.size() <= 1) {
- log.info("[%s]: One or fewer servers found. Cannot balance.", tier);
+if (general.size() == 0) {
+ log.info("[%s]: 0 general servers found. Cannot balance.", tier);
return;
}
int numSegments = 0;
-for (ServerHolder sourceHolder : toMoveFrom) {
+for (ServerHolder sourceHolder : servers) {
numSegments += sourceHolder.getServer().getSegments().size();
}
-
if (numSegments == 0) {
log.info("No segments found. Cannot balance.");
return;
}
-final BalancerStrategy strategy = params.getBalancerStrategy();
final int maxSegmentsToMove =
Math.min(params.getCoordinatorDynamicConfig().getMaxSegmentsToMove(),
numSegments);
+int priority =
params.getCoordinatorDynamicConfig().getMaintenanceModeSegmentsPriority();
+int maintenanceSegmentsToMove = (int) Math.ceil(maxSegmentsToMove *
priority / 10.0);
+log.info("Balancing %d segments for servers in maintenance mode",
maintenanceSegmentsToMove);
+Pair maintenanceResult = balanceServers(params,
maintenance, general, maintenanceSegmentsToMove);
+int generalSegmentsToMove = maxSegmentsToMove - maintenanceResult.lhs;
+log.info("Balancing %d segments for servers in general mode",
generalSegmentsToMove);
+Pair generalResult = balanceServers(params, general,
general, generalSegmentsToMove);
+Pair result = new Pair(
Review comment:
:white_check_mark:
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] egor-ryashin commented on a change in pull request #6349: maintenance mode for Historical
egor-ryashin commented on a change in pull request #6349: maintenance mode for
Historical
URL: https://github.com/apache/incubator-druid/pull/6349#discussion_r221228554
##
File path:
server/src/main/java/org/apache/druid/server/coordinator/helper/DruidCoordinatorBalancer.java
##
@@ -152,8 +191,9 @@ private void balanceTier(
strategy.findNewSegmentHomeBalancer(segmentToMove,
toMoveToWithLoadQueueCapacityAndNotServingSegment);
if (destinationHolder != null &&
!destinationHolder.getServer().equals(fromServer)) {
-moveSegment(segmentToMoveHolder, destinationHolder.getServer(),
params);
-moved++;
+if (moveSegment(segmentToMoveHolder,
destinationHolder.getServer(), params)) {
Review comment:
:white_check_mark:
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] egor-ryashin commented on a change in pull request #6349: maintenance mode for Historical
egor-ryashin commented on a change in pull request #6349: maintenance mode for
Historical
URL: https://github.com/apache/incubator-druid/pull/6349#discussion_r221228510
##
File path:
server/src/main/java/org/apache/druid/server/coordinator/helper/DruidCoordinatorBalancer.java
##
@@ -106,35 +104,76 @@ private void balanceTier(
return;
}
-final List toMoveFrom = Lists.newArrayList(servers);
-final List toMoveTo = Lists.newArrayList(servers);
+Map> partitions = servers.stream()
+
.collect(Collectors.partitioningBy(ServerHolder::isMaintenance));
+final List maintenance = partitions.get(true);
+final List general = partitions.get(false);
+log.info("Found %d servers in maintenance, %d general servers",
maintenance.size(), general.size());
-if (toMoveTo.size() <= 1) {
- log.info("[%s]: One or fewer servers found. Cannot balance.", tier);
+if (general.size() == 0) {
+ log.info("[%s]: 0 general servers found. Cannot balance.", tier);
return;
}
int numSegments = 0;
-for (ServerHolder sourceHolder : toMoveFrom) {
+for (ServerHolder sourceHolder : servers) {
numSegments += sourceHolder.getServer().getSegments().size();
}
-
if (numSegments == 0) {
log.info("No segments found. Cannot balance.");
return;
}
-final BalancerStrategy strategy = params.getBalancerStrategy();
final int maxSegmentsToMove =
Math.min(params.getCoordinatorDynamicConfig().getMaxSegmentsToMove(),
numSegments);
+int priority =
params.getCoordinatorDynamicConfig().getMaintenanceModeSegmentsPriority();
+int maintenanceSegmentsToMove = (int) Math.ceil(maxSegmentsToMove *
priority / 10.0);
+log.info("Balancing %d segments for servers in maintenance mode",
maintenanceSegmentsToMove);
+Pair maintenanceResult = balanceServers(params,
maintenance, general, maintenanceSegmentsToMove);
+int generalSegmentsToMove = maxSegmentsToMove - maintenanceResult.lhs;
+log.info("Balancing %d segments for servers in general mode",
generalSegmentsToMove);
+Pair generalResult = balanceServers(params, general,
general, generalSegmentsToMove);
+Pair result = new Pair(
+generalResult.lhs + maintenanceResult.lhs,
+generalResult.rhs + maintenanceResult.rhs
+);
+
+int moved = result.lhs;
+int unmoved = result.rhs;
+if (unmoved == maxSegmentsToMove) {
+ // Cluster should be alive and constantly adjusting
+ log.info("No good moves found in tier [%s]", tier);
+}
+stats.addToTieredStat("unmovedCount", tier, unmoved);
+stats.addToTieredStat("movedCount", tier, moved);
+
+if (params.getCoordinatorDynamicConfig().emitBalancingStats()) {
+ final BalancerStrategy strategy = params.getBalancerStrategy();
+ strategy.emitStats(tier, stats, Lists.newArrayList(servers));
+}
+log.info(
Review comment:
:white_check_mark:
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] egor-ryashin commented on a change in pull request #6349: maintenance mode for Historical
egor-ryashin commented on a change in pull request #6349: maintenance mode for
Historical
URL: https://github.com/apache/incubator-druid/pull/6349#discussion_r221228460
##
File path:
server/src/main/java/org/apache/druid/server/coordinator/helper/DruidCoordinatorBalancer.java
##
@@ -106,35 +104,76 @@ private void balanceTier(
return;
}
-final List toMoveFrom = Lists.newArrayList(servers);
-final List toMoveTo = Lists.newArrayList(servers);
+Map> partitions = servers.stream()
+
.collect(Collectors.partitioningBy(ServerHolder::isMaintenance));
+final List maintenance = partitions.get(true);
+final List general = partitions.get(false);
+log.info("Found %d servers in maintenance, %d general servers",
maintenance.size(), general.size());
-if (toMoveTo.size() <= 1) {
- log.info("[%s]: One or fewer servers found. Cannot balance.", tier);
+if (general.size() == 0) {
+ log.info("[%s]: 0 general servers found. Cannot balance.", tier);
return;
}
int numSegments = 0;
-for (ServerHolder sourceHolder : toMoveFrom) {
+for (ServerHolder sourceHolder : servers) {
numSegments += sourceHolder.getServer().getSegments().size();
}
-
if (numSegments == 0) {
log.info("No segments found. Cannot balance.");
return;
}
-final BalancerStrategy strategy = params.getBalancerStrategy();
final int maxSegmentsToMove =
Math.min(params.getCoordinatorDynamicConfig().getMaxSegmentsToMove(),
numSegments);
+int priority =
params.getCoordinatorDynamicConfig().getMaintenanceModeSegmentsPriority();
+int maintenanceSegmentsToMove = (int) Math.ceil(maxSegmentsToMove *
priority / 10.0);
+log.info("Balancing %d segments for servers in maintenance mode",
maintenanceSegmentsToMove);
Review comment:
:white_check_mark:
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] egor-ryashin commented on a change in pull request #6349: maintenance mode for Historical
egor-ryashin commented on a change in pull request #6349: maintenance mode for
Historical
URL: https://github.com/apache/incubator-druid/pull/6349#discussion_r221228647
##
File path:
server/src/test/java/org/apache/druid/server/coordinator/DruidCoordinatorBalancerTest.java
##
@@ -195,6 +201,186 @@ public void testMoveToEmptyServerBalancer()
Assert.assertEquals(2,
params.getCoordinatorStats().getTieredStat("movedCount", "normal"));
}
+ /**
+ * Server 1 has 2 segments.
+ * Server 2 (maintenance) has 2 segments.
+ * Server 3 is empty.
+ * Maintenance has priority 7.
+ * Max segments to move is 3.
+ * 2 (of 2) segments should be moved from Server 2 and 1 (of 2) from Server
1.
+ */
+ @Test
+ public void testMoveMaintenancePriority()
+ {
+mockDruidServer(druidServer1, "1", "normal", 30L, 100L,
ImmutableMap.of(segment1.getIdentifier(), segment1, segment2.getIdentifier(),
segment2));
Review comment:
:white_check_mark:
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] egor-ryashin commented on a change in pull request #6349: maintenance mode for Historical
egor-ryashin commented on a change in pull request #6349: maintenance mode for
Historical
URL: https://github.com/apache/incubator-druid/pull/6349#discussion_r221228691
##
File path:
server/src/test/java/org/apache/druid/server/coordinator/DruidCoordinatorBalancerTest.java
##
@@ -195,6 +201,186 @@ public void testMoveToEmptyServerBalancer()
Assert.assertEquals(2,
params.getCoordinatorStats().getTieredStat("movedCount", "normal"));
}
+ /**
+ * Server 1 has 2 segments.
+ * Server 2 (maintenance) has 2 segments.
+ * Server 3 is empty.
+ * Maintenance has priority 7.
+ * Max segments to move is 3.
+ * 2 (of 2) segments should be moved from Server 2 and 1 (of 2) from Server
1.
+ */
+ @Test
+ public void testMoveMaintenancePriority()
+ {
+mockDruidServer(druidServer1, "1", "normal", 30L, 100L,
ImmutableMap.of(segment1.getIdentifier(), segment1, segment2.getIdentifier(),
segment2));
+mockDruidServer(druidServer2, "2", "normal", 30L, 100L,
ImmutableMap.of(segment3.getIdentifier(), segment3, segment4.getIdentifier(),
segment4));
+mockDruidServer(druidServer3, "3", "normal", 0L, 100L,
Collections.emptyMap());
+
+EasyMock.replay(druidServer4);
+
+mockCoordinator(coordinator);
+
+BalancerStrategy strategy = EasyMock.createMock(BalancerStrategy.class);
+EasyMock.expect(strategy.pickSegmentToMove(ImmutableList.of(new
ServerHolder(druidServer2, peon2, false
+.andReturn(new BalancerSegmentHolder(druidServer2, segment3))
+.andReturn(new BalancerSegmentHolder(druidServer2, segment4));
+EasyMock.expect(strategy.pickSegmentToMove(anyObject()))
+.andReturn(new BalancerSegmentHolder(druidServer1, segment1))
+.andReturn(new BalancerSegmentHolder(druidServer1, segment2));
+
+EasyMock.expect(strategy.findNewSegmentHomeBalancer(anyObject(),
anyObject())).andReturn(new ServerHolder(druidServer3, peon3)).anyTimes();
+replay(strategy);
+
+DruidCoordinatorRuntimeParams params = defaultRuntimeParamsBuilder(
+ImmutableList.of(druidServer1, druidServer2, druidServer3),
+ImmutableList.of(peon1, peon2, peon3),
+ImmutableList.of(false, true, false)
+)
+.withDynamicConfigs(
+CoordinatorDynamicConfig.builder()
Review comment:
:white_check_mark:
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] egor-ryashin commented on a change in pull request #6349: maintenance mode for Historical
egor-ryashin commented on a change in pull request #6349: maintenance mode for
Historical
URL: https://github.com/apache/incubator-druid/pull/6349#discussion_r221228776
##
File path:
server/src/test/java/org/apache/druid/server/coordinator/DruidCoordinatorBalancerTest.java
##
@@ -195,6 +201,186 @@ public void testMoveToEmptyServerBalancer()
Assert.assertEquals(2,
params.getCoordinatorStats().getTieredStat("movedCount", "normal"));
}
+ /**
+ * Server 1 has 2 segments.
+ * Server 2 (maintenance) has 2 segments.
+ * Server 3 is empty.
+ * Maintenance has priority 7.
+ * Max segments to move is 3.
+ * 2 (of 2) segments should be moved from Server 2 and 1 (of 2) from Server
1.
+ */
+ @Test
+ public void testMoveMaintenancePriority()
+ {
+mockDruidServer(druidServer1, "1", "normal", 30L, 100L,
ImmutableMap.of(segment1.getIdentifier(), segment1, segment2.getIdentifier(),
segment2));
+mockDruidServer(druidServer2, "2", "normal", 30L, 100L,
ImmutableMap.of(segment3.getIdentifier(), segment3, segment4.getIdentifier(),
segment4));
+mockDruidServer(druidServer3, "3", "normal", 0L, 100L,
Collections.emptyMap());
+
+EasyMock.replay(druidServer4);
+
+mockCoordinator(coordinator);
+
+BalancerStrategy strategy = EasyMock.createMock(BalancerStrategy.class);
+EasyMock.expect(strategy.pickSegmentToMove(ImmutableList.of(new
ServerHolder(druidServer2, peon2, false
+.andReturn(new BalancerSegmentHolder(druidServer2, segment3))
+.andReturn(new BalancerSegmentHolder(druidServer2, segment4));
+EasyMock.expect(strategy.pickSegmentToMove(anyObject()))
+.andReturn(new BalancerSegmentHolder(druidServer1, segment1))
+.andReturn(new BalancerSegmentHolder(druidServer1, segment2));
+
+EasyMock.expect(strategy.findNewSegmentHomeBalancer(anyObject(),
anyObject())).andReturn(new ServerHolder(druidServer3, peon3)).anyTimes();
+replay(strategy);
+
+DruidCoordinatorRuntimeParams params = defaultRuntimeParamsBuilder(
+ImmutableList.of(druidServer1, druidServer2, druidServer3),
+ImmutableList.of(peon1, peon2, peon3),
+ImmutableList.of(false, true, false)
+)
+.withDynamicConfigs(
+CoordinatorDynamicConfig.builder()
+.withMaxSegmentsToMove(
+3
+
).withMaintenanceModeSegmentsPriority(6).build()// ceil(3 * 0.6) = 2 segments
from servers in maintenance
+)
+.withBalancerStrategy(strategy)
+.build();
+
+params = new DruidCoordinatorBalancerTester(coordinator).run(params);
+Assert.assertEquals(3L,
params.getCoordinatorStats().getTieredStat("movedCount", "normal"));
+Assert.assertThat(peon3.getSegmentsToLoad(),
is(equalTo(ImmutableSet.of(segment1, segment3, segment4;
+ }
+
+ @Test
+ public void testZeroMaintenancePriority()
+ {
+DruidCoordinatorRuntimeParams params =
setupParamsForMaintenancePriority(0);
+params = new DruidCoordinatorBalancerTester(coordinator).run(params);
+Assert.assertEquals(1L,
params.getCoordinatorStats().getTieredStat("movedCount", "normal"));
+Assert.assertThat(peon3.getSegmentsToLoad(),
is(equalTo(ImmutableSet.of(segment1;
+ }
+
+ @Test
+ public void testMaxMaintenancePriority()
+ {
+DruidCoordinatorRuntimeParams params =
setupParamsForMaintenancePriority(10);
+params = new DruidCoordinatorBalancerTester(coordinator).run(params);
+Assert.assertEquals(1L,
params.getCoordinatorStats().getTieredStat("movedCount", "normal"));
+Assert.assertThat(peon3.getSegmentsToLoad(),
is(equalTo(ImmutableSet.of(segment2;
+ }
+
+ /**
+ * Should balance segments as usual (ignoring priority) with empty
maintenanceList.
+ */
+ @Test
+ public void testMoveMaintenancePriorityWithNoMaintenance()
+ {
+mockDruidServer(druidServer1, "1", "normal", 30L, 100L,
ImmutableMap.of(segment1.getIdentifier(), segment1, segment2.getIdentifier(),
segment2));
+mockDruidServer(druidServer2, "2", "normal", 0L, 100L,
ImmutableMap.of(segment3.getIdentifier(), segment3, segment4.getIdentifier(),
segment4));
+mockDruidServer(druidServer3, "3", "normal", 0L, 100L,
Collections.emptyMap());
+
+EasyMock.replay(druidServer4);
+
+mockCoordinator(coordinator);
+
+BalancerStrategy strategy = EasyMock.createMock(BalancerStrategy.class);
+EasyMock.expect(strategy.pickSegmentToMove(anyObject()))
+.andReturn(new BalancerSegmentHolder(druidServer1, segment1))
+.andReturn(new BalancerSegmentHolder(druidServer1, segment2))
+.andReturn(new BalancerSegmentHolder(druidServer2, segment3))
+.andReturn(new BalancerSegmentHolder(druidServer2, segment4));
+
+EasyMock.expect(strategy.findNewSegmentHomeBalancer(anyObject(),
anyObject())).andReturn(new ServerHolder(druidServer3, peon3)).anyTimes();
+replay(strat
[GitHub] egor-ryashin commented on a change in pull request #6349: maintenance mode for Historical
egor-ryashin commented on a change in pull request #6349: maintenance mode for
Historical
URL: https://github.com/apache/incubator-druid/pull/6349#discussion_r221228815
##
File path:
server/src/test/java/org/apache/druid/server/coordinator/DruidCoordinatorBalancerTest.java
##
@@ -432,4 +628,39 @@ public void emitStats(
delegate.emitStats(tier, stats, serverHolderList);
}
}
+
+ private DruidCoordinatorRuntimeParams setupParamsForMaintenancePriority(int
priority)
+ {
+mockDruidServer(druidServer1, "1", "normal", 30L, 100L,
ImmutableMap.of(segment1.getIdentifier(), segment1, segment3.getIdentifier(),
segment3));
+mockDruidServer(druidServer2, "2", "normal", 30L, 100L,
ImmutableMap.of(segment2.getIdentifier(), segment2, segment3.getIdentifier(),
segment3));
+mockDruidServer(druidServer3, "3", "normal", 0L, 100L,
Collections.emptyMap());
+
+EasyMock.replay(druidServer4);
+
+mockCoordinator(coordinator);
+
+// either maintenance servers list or general ones (ie servers list is [2]
or [1, 3])
+BalancerStrategy strategy = EasyMock.createMock(BalancerStrategy.class);
+EasyMock.expect(strategy.pickSegmentToMove(ImmutableList.of(new
ServerHolder(druidServer2, peon2, true
+.andReturn(new BalancerSegmentHolder(druidServer2, segment2));
+EasyMock.expect(strategy.pickSegmentToMove(anyObject()))
+.andReturn(new BalancerSegmentHolder(druidServer1, segment1));
+
+EasyMock.expect(strategy.findNewSegmentHomeBalancer(anyObject(),
anyObject())).andReturn(new ServerHolder(druidServer3, peon3)).anyTimes();
+replay(strategy);
+
+return defaultRuntimeParamsBuilder(
+ImmutableList.of(druidServer1, druidServer2, druidServer3),
+ImmutableList.of(peon1, peon2, peon3),
+ImmutableList.of(false, true, false)
+)
+.withDynamicConfigs(
+CoordinatorDynamicConfig.builder()
Review comment:
:white_check_mark:
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] egor-ryashin commented on a change in pull request #6349: maintenance mode for Historical
egor-ryashin commented on a change in pull request #6349: maintenance mode for
Historical
URL: https://github.com/apache/incubator-druid/pull/6349#discussion_r221228735
##
File path:
server/src/test/java/org/apache/druid/server/coordinator/DruidCoordinatorBalancerTest.java
##
@@ -195,6 +201,186 @@ public void testMoveToEmptyServerBalancer()
Assert.assertEquals(2,
params.getCoordinatorStats().getTieredStat("movedCount", "normal"));
}
+ /**
+ * Server 1 has 2 segments.
+ * Server 2 (maintenance) has 2 segments.
+ * Server 3 is empty.
+ * Maintenance has priority 7.
+ * Max segments to move is 3.
+ * 2 (of 2) segments should be moved from Server 2 and 1 (of 2) from Server
1.
+ */
+ @Test
+ public void testMoveMaintenancePriority()
+ {
+mockDruidServer(druidServer1, "1", "normal", 30L, 100L,
ImmutableMap.of(segment1.getIdentifier(), segment1, segment2.getIdentifier(),
segment2));
+mockDruidServer(druidServer2, "2", "normal", 30L, 100L,
ImmutableMap.of(segment3.getIdentifier(), segment3, segment4.getIdentifier(),
segment4));
+mockDruidServer(druidServer3, "3", "normal", 0L, 100L,
Collections.emptyMap());
+
+EasyMock.replay(druidServer4);
+
+mockCoordinator(coordinator);
+
+BalancerStrategy strategy = EasyMock.createMock(BalancerStrategy.class);
+EasyMock.expect(strategy.pickSegmentToMove(ImmutableList.of(new
ServerHolder(druidServer2, peon2, false
+.andReturn(new BalancerSegmentHolder(druidServer2, segment3))
+.andReturn(new BalancerSegmentHolder(druidServer2, segment4));
+EasyMock.expect(strategy.pickSegmentToMove(anyObject()))
+.andReturn(new BalancerSegmentHolder(druidServer1, segment1))
+.andReturn(new BalancerSegmentHolder(druidServer1, segment2));
+
+EasyMock.expect(strategy.findNewSegmentHomeBalancer(anyObject(),
anyObject())).andReturn(new ServerHolder(druidServer3, peon3)).anyTimes();
+replay(strategy);
+
+DruidCoordinatorRuntimeParams params = defaultRuntimeParamsBuilder(
+ImmutableList.of(druidServer1, druidServer2, druidServer3),
+ImmutableList.of(peon1, peon2, peon3),
+ImmutableList.of(false, true, false)
+)
+.withDynamicConfigs(
+CoordinatorDynamicConfig.builder()
+.withMaxSegmentsToMove(
+3
+
).withMaintenanceModeSegmentsPriority(6).build()// ceil(3 * 0.6) = 2 segments
from servers in maintenance
+)
+.withBalancerStrategy(strategy)
+.build();
+
+params = new DruidCoordinatorBalancerTester(coordinator).run(params);
+Assert.assertEquals(3L,
params.getCoordinatorStats().getTieredStat("movedCount", "normal"));
+Assert.assertThat(peon3.getSegmentsToLoad(),
is(equalTo(ImmutableSet.of(segment1, segment3, segment4;
+ }
+
+ @Test
+ public void testZeroMaintenancePriority()
+ {
+DruidCoordinatorRuntimeParams params =
setupParamsForMaintenancePriority(0);
+params = new DruidCoordinatorBalancerTester(coordinator).run(params);
+Assert.assertEquals(1L,
params.getCoordinatorStats().getTieredStat("movedCount", "normal"));
+Assert.assertThat(peon3.getSegmentsToLoad(),
is(equalTo(ImmutableSet.of(segment1;
+ }
+
+ @Test
+ public void testMaxMaintenancePriority()
+ {
+DruidCoordinatorRuntimeParams params =
setupParamsForMaintenancePriority(10);
+params = new DruidCoordinatorBalancerTester(coordinator).run(params);
+Assert.assertEquals(1L,
params.getCoordinatorStats().getTieredStat("movedCount", "normal"));
+Assert.assertThat(peon3.getSegmentsToLoad(),
is(equalTo(ImmutableSet.of(segment2;
+ }
+
+ /**
+ * Should balance segments as usual (ignoring priority) with empty
maintenanceList.
+ */
+ @Test
+ public void testMoveMaintenancePriorityWithNoMaintenance()
+ {
+mockDruidServer(druidServer1, "1", "normal", 30L, 100L,
ImmutableMap.of(segment1.getIdentifier(), segment1, segment2.getIdentifier(),
segment2));
+mockDruidServer(druidServer2, "2", "normal", 0L, 100L,
ImmutableMap.of(segment3.getIdentifier(), segment3, segment4.getIdentifier(),
segment4));
+mockDruidServer(druidServer3, "3", "normal", 0L, 100L,
Collections.emptyMap());
+
+EasyMock.replay(druidServer4);
+
+mockCoordinator(coordinator);
+
+BalancerStrategy strategy = EasyMock.createMock(BalancerStrategy.class);
+EasyMock.expect(strategy.pickSegmentToMove(anyObject()))
+.andReturn(new BalancerSegmentHolder(druidServer1, segment1))
+.andReturn(new BalancerSegmentHolder(druidServer1, segment2))
+.andReturn(new BalancerSegmentHolder(druidServer2, segment3))
+.andReturn(new BalancerSegmentHolder(druidServer2, segment4));
+
+EasyMock.expect(strategy.findNewSegmentHomeBalancer(anyObject(),
anyObject())).andReturn(new ServerHolder(druidServer3, peon3)).anyTimes();
+replay(strat
[GitHub] egor-ryashin commented on a change in pull request #6349: maintenance mode for Historical
egor-ryashin commented on a change in pull request #6349: maintenance mode for
Historical
URL: https://github.com/apache/incubator-druid/pull/6349#discussion_r221228846
##
File path:
server/src/test/java/org/apache/druid/server/coordinator/DruidCoordinatorBalancerTester.java
##
@@ -19,20 +19,31 @@
package org.apache.druid.server.coordinator;
+import com.google.common.primitives.Doubles;
import org.apache.druid.client.ImmutableDruidServer;
import org.apache.druid.java.util.common.StringUtils;
import org.apache.druid.server.coordinator.helper.DruidCoordinatorBalancer;
import org.apache.druid.timeline.DataSegment;
+import java.util.Comparator;
+
public class DruidCoordinatorBalancerTester extends DruidCoordinatorBalancer
{
+ public static final Comparator percentUsedComparator =
(ServerHolder a, ServerHolder b) -> {
+int c = Doubles.compare(a.getPercentUsed(), b.getPercentUsed());
Review comment:
:white_check_mark:
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] egor-ryashin commented on a change in pull request #6349: maintenance mode for Historical
egor-ryashin commented on a change in pull request #6349: maintenance mode for
Historical
URL: https://github.com/apache/incubator-druid/pull/6349#discussion_r221229577
##
File path:
server/src/test/java/org/apache/druid/server/coordinator/rules/BroadcastDistributionRuleTest.java
##
@@ -253,6 +313,46 @@ public void testBroadcastToSingleDataSource()
assertFalse(holderOfSmallSegment.getPeon().getSegmentsToLoad().contains(smallSegment));
}
+ /**
+ * Servers:
+ * name | segments
+ * -+--
+ * general | large segment
+ * maintenance1 | small segment
+ * maintenance2 | large segment
+ *
+ * After running the rule for the small segment:
+ * general | large & small segments
+ * maintenance1 |
+ * maintenance2 | large segment
+ */
+ @Test
+ public void testBroadcastWithMaintenance()
+ {
+final ForeverBroadcastDistributionRule rule = new
ForeverBroadcastDistributionRule(ImmutableList.of("large_source"));
+
+CoordinatorStats stats = rule.run(
+null,
+DruidCoordinatorRuntimeParams.newBuilder()
+ .withDruidCluster(secondCluster)
+
.withSegmentReplicantLookup(SegmentReplicantLookup.make(secondCluster))
+
.withBalancerReferenceTimestamp(DateTimes.of("2013-01-01"))
+ .withAvailableSegments(Lists.newArrayList(
Review comment:
:white_check_mark:
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] QiuMM commented on issue #6302: Add SQL id, request logs, and metrics
QiuMM commented on issue #6302: Add SQL id, request logs, and metrics URL: https://github.com/apache/incubator-druid/pull/6302#issuecomment-425421548 @gaodayue maybe you can merge the master branch, it works for me. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] egor-ryashin commented on a change in pull request #6349: maintenance mode for Historical
egor-ryashin commented on a change in pull request #6349: maintenance mode for
Historical
URL: https://github.com/apache/incubator-druid/pull/6349#discussion_r221274236
##
File path:
server/src/test/java/org/apache/druid/server/coordinator/rules/LoadRuleTest.java
##
@@ -685,6 +685,210 @@ public void testMaxLoadingQueueSize()
EasyMock.verify(throttler, mockBalancerStrategy);
}
+ /**
+ * 2 servers in different tiers, the first is in maitenance mode.
+ * Should not load a segment to the server in maintenance mode.
+ */
+ @Test
+ public void testLoadDuringMaitenance()
+ {
+final LoadQueuePeon mockPeon1 = createEmptyPeon();
+final LoadQueuePeon mockPeon2 = createOneCallPeonMock();
+
+LoadRule rule = createLoadRule(ImmutableMap.of(
+"tier1", 1,
+"tier2", 1
+));
+
+final DataSegment segment = createDataSegment("foo");
+
+
EasyMock.expect(mockBalancerStrategy.findNewSegmentHomeReplicator(EasyMock.anyObject(),
EasyMock.anyObject()))
+.andDelegateTo(balancerStrategy)
+.times(1);
+
+EasyMock.replay(mockPeon1, mockPeon2, mockBalancerStrategy);
+
+
+DruidCluster druidCluster = new DruidCluster(
+null,
+ImmutableMap.of(
+"tier1",
+Stream.of(
Review comment:
hmm, there is a complication, one test uses its own comparator
org/apache/druid/server/coordinator/DruidCoordinatorBalancerTest.java:520
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] egor-ryashin commented on a change in pull request #6349: maintenance mode for Historical
egor-ryashin commented on a change in pull request #6349: maintenance mode for
Historical
URL: https://github.com/apache/incubator-druid/pull/6349#discussion_r221274488
##
File path:
server/src/test/java/org/apache/druid/server/coordinator/rules/LoadRuleTest.java
##
@@ -685,6 +685,210 @@ public void testMaxLoadingQueueSize()
EasyMock.verify(throttler, mockBalancerStrategy);
}
+ /**
+ * 2 servers in different tiers, the first is in maitenance mode.
+ * Should not load a segment to the server in maintenance mode.
+ */
+ @Test
+ public void testLoadDuringMaitenance()
+ {
+final LoadQueuePeon mockPeon1 = createEmptyPeon();
+final LoadQueuePeon mockPeon2 = createOneCallPeonMock();
+
+LoadRule rule = createLoadRule(ImmutableMap.of(
+"tier1", 1,
+"tier2", 1
+));
+
+final DataSegment segment = createDataSegment("foo");
+
+
EasyMock.expect(mockBalancerStrategy.findNewSegmentHomeReplicator(EasyMock.anyObject(),
EasyMock.anyObject()))
+.andDelegateTo(balancerStrategy)
+.times(1);
+
+EasyMock.replay(mockPeon1, mockPeon2, mockBalancerStrategy);
+
+
+DruidCluster druidCluster = new DruidCluster(
+null,
+ImmutableMap.of(
+"tier1",
+Stream.of(
+createServerHolder("tier1", mockPeon1, true)
+).collect(Collectors.toCollection(() -> new TreeSet<>())),
+"tier2",
+Stream.of(
+createServerHolder("tier2", mockPeon2, false)
+).collect(Collectors.toCollection(() -> new TreeSet<>()))
+)
+);
+
+CoordinatorStats stats = rule.run(
+null,
+DruidCoordinatorRuntimeParams.newBuilder()
+ .withDruidCluster(druidCluster)
+
.withSegmentReplicantLookup(SegmentReplicantLookup.make(druidCluster))
+ .withReplicationManager(throttler)
+
.withBalancerStrategy(mockBalancerStrategy)
+
.withBalancerReferenceTimestamp(DateTimes.of("2013-01-01"))
+
.withAvailableSegments(Collections.singletonList(segment)).build(),
+segment
+);
+
+Assert.assertEquals(1L, stats.getTieredStat(LoadRule.ASSIGNED_COUNT,
"tier2"));
+EasyMock.verify(mockPeon1, mockPeon2, mockBalancerStrategy);
+ }
+
+
+ /**
+ * 2 tiers, 2 servers each, 1 server of the second tier is in maintenance.
+ * Should not load a segment to the server in maintenance mode.
+ */
+ @Test
+ public void testLoadReplicaDuringMaitenance()
+ {
+
EasyMock.expect(throttler.canCreateReplicant(EasyMock.anyString())).andReturn(true).anyTimes();
+
+final LoadQueuePeon mockPeon1 = createEmptyPeon();
+final LoadQueuePeon mockPeon2 = createOneCallPeonMock();
+final LoadQueuePeon mockPeon3 = createOneCallPeonMock();
+final LoadQueuePeon mockPeon4 = createOneCallPeonMock();
+
+LoadRule rule = createLoadRule(ImmutableMap.of(
+"tier1", 2,
+"tier2", 2
+));
+
+final DataSegment segment = createDataSegment("foo");
+
+throttler.registerReplicantCreation(EasyMock.eq("tier2"),
EasyMock.anyString(), EasyMock.anyObject());
+EasyMock.expectLastCall().times(2);
+
+ServerHolder holder1 = createServerHolder("tier1", mockPeon1, true);
+ServerHolder holder2 = createServerHolder("tier1", mockPeon2, false);
+ServerHolder holder3 = createServerHolder("tier2", mockPeon3, false);
+ServerHolder holder4 = createServerHolder("tier2", mockPeon4, false);
+
+EasyMock.expect(mockBalancerStrategy.findNewSegmentHomeReplicator(segment,
ImmutableList.of(holder2)))
+.andReturn(holder2);
+EasyMock.expect(mockBalancerStrategy.findNewSegmentHomeReplicator(segment,
ImmutableList.of(holder3, holder4)))
+.andReturn(holder3);
+EasyMock.expect(mockBalancerStrategy.findNewSegmentHomeReplicator(segment,
ImmutableList.of(holder4)))
+.andReturn(holder4);
+
+EasyMock.replay(throttler, mockPeon1, mockPeon2, mockPeon3, mockPeon4,
mockBalancerStrategy);
+
+
+DruidCluster druidCluster = new DruidCluster(
+null,
+ImmutableMap.of(
+"tier1",
+Stream.of(
+holder1,
+holder2
+).collect(Collectors.toCollection(() -> new TreeSet<>())),
+"tier2",
+Stream.of(
+holder3,
+holder4
+).collect(Collectors.toCollection(() -> new TreeSet<>()))
+)
+);
+
+CoordinatorStats stats = rule.run(
+null,
+DruidCoordinatorRuntimeParams.newBuilder()
+ .withDruidCluster(druidCluster)
+
.withSegmentReplicantLookup(SegmentReplicantLookup.make(druidCluster))
+ .withReplicat
[GitHub] egor-ryashin commented on a change in pull request #6349: maintenance mode for Historical
egor-ryashin commented on a change in pull request #6349: maintenance mode for
Historical
URL: https://github.com/apache/incubator-druid/pull/6349#discussion_r221274524
##
File path:
server/src/test/java/org/apache/druid/server/coordinator/rules/LoadRuleTest.java
##
@@ -685,6 +685,210 @@ public void testMaxLoadingQueueSize()
EasyMock.verify(throttler, mockBalancerStrategy);
}
+ /**
+ * 2 servers in different tiers, the first is in maitenance mode.
+ * Should not load a segment to the server in maintenance mode.
+ */
+ @Test
+ public void testLoadDuringMaitenance()
+ {
+final LoadQueuePeon mockPeon1 = createEmptyPeon();
+final LoadQueuePeon mockPeon2 = createOneCallPeonMock();
+
+LoadRule rule = createLoadRule(ImmutableMap.of(
+"tier1", 1,
+"tier2", 1
+));
+
+final DataSegment segment = createDataSegment("foo");
+
+
EasyMock.expect(mockBalancerStrategy.findNewSegmentHomeReplicator(EasyMock.anyObject(),
EasyMock.anyObject()))
+.andDelegateTo(balancerStrategy)
+.times(1);
+
+EasyMock.replay(mockPeon1, mockPeon2, mockBalancerStrategy);
+
+
+DruidCluster druidCluster = new DruidCluster(
+null,
+ImmutableMap.of(
+"tier1",
+Stream.of(
+createServerHolder("tier1", mockPeon1, true)
+).collect(Collectors.toCollection(() -> new TreeSet<>())),
+"tier2",
+Stream.of(
+createServerHolder("tier2", mockPeon2, false)
+).collect(Collectors.toCollection(() -> new TreeSet<>()))
+)
+);
+
+CoordinatorStats stats = rule.run(
+null,
+DruidCoordinatorRuntimeParams.newBuilder()
+ .withDruidCluster(druidCluster)
+
.withSegmentReplicantLookup(SegmentReplicantLookup.make(druidCluster))
+ .withReplicationManager(throttler)
+
.withBalancerStrategy(mockBalancerStrategy)
+
.withBalancerReferenceTimestamp(DateTimes.of("2013-01-01"))
+
.withAvailableSegments(Collections.singletonList(segment)).build(),
+segment
+);
+
+Assert.assertEquals(1L, stats.getTieredStat(LoadRule.ASSIGNED_COUNT,
"tier2"));
+EasyMock.verify(mockPeon1, mockPeon2, mockBalancerStrategy);
+ }
+
+
+ /**
+ * 2 tiers, 2 servers each, 1 server of the second tier is in maintenance.
+ * Should not load a segment to the server in maintenance mode.
+ */
+ @Test
+ public void testLoadReplicaDuringMaitenance()
+ {
+
EasyMock.expect(throttler.canCreateReplicant(EasyMock.anyString())).andReturn(true).anyTimes();
+
+final LoadQueuePeon mockPeon1 = createEmptyPeon();
+final LoadQueuePeon mockPeon2 = createOneCallPeonMock();
+final LoadQueuePeon mockPeon3 = createOneCallPeonMock();
+final LoadQueuePeon mockPeon4 = createOneCallPeonMock();
+
+LoadRule rule = createLoadRule(ImmutableMap.of(
+"tier1", 2,
+"tier2", 2
+));
+
+final DataSegment segment = createDataSegment("foo");
+
+throttler.registerReplicantCreation(EasyMock.eq("tier2"),
EasyMock.anyString(), EasyMock.anyObject());
+EasyMock.expectLastCall().times(2);
+
+ServerHolder holder1 = createServerHolder("tier1", mockPeon1, true);
+ServerHolder holder2 = createServerHolder("tier1", mockPeon2, false);
+ServerHolder holder3 = createServerHolder("tier2", mockPeon3, false);
+ServerHolder holder4 = createServerHolder("tier2", mockPeon4, false);
+
+EasyMock.expect(mockBalancerStrategy.findNewSegmentHomeReplicator(segment,
ImmutableList.of(holder2)))
+.andReturn(holder2);
+EasyMock.expect(mockBalancerStrategy.findNewSegmentHomeReplicator(segment,
ImmutableList.of(holder3, holder4)))
+.andReturn(holder3);
+EasyMock.expect(mockBalancerStrategy.findNewSegmentHomeReplicator(segment,
ImmutableList.of(holder4)))
+.andReturn(holder4);
+
+EasyMock.replay(throttler, mockPeon1, mockPeon2, mockPeon3, mockPeon4,
mockBalancerStrategy);
+
+
+DruidCluster druidCluster = new DruidCluster(
+null,
+ImmutableMap.of(
+"tier1",
+Stream.of(
+holder1,
+holder2
+).collect(Collectors.toCollection(() -> new TreeSet<>())),
+"tier2",
+Stream.of(
+holder3,
+holder4
+).collect(Collectors.toCollection(() -> new TreeSet<>()))
+)
+);
+
+CoordinatorStats stats = rule.run(
+null,
+DruidCoordinatorRuntimeParams.newBuilder()
+ .withDruidCluster(druidCluster)
+
.withSegmentReplicantLookup(SegmentReplicantLookup.make(druidCluster))
+ .withReplicat
[GitHub] egor-ryashin commented on a change in pull request #6349: maintenance mode for Historical
egor-ryashin commented on a change in pull request #6349: maintenance mode for
Historical
URL: https://github.com/apache/incubator-druid/pull/6349#discussion_r221274585
##
File path:
server/src/test/java/org/apache/druid/server/coordinator/rules/LoadRuleTest.java
##
@@ -685,6 +685,210 @@ public void testMaxLoadingQueueSize()
EasyMock.verify(throttler, mockBalancerStrategy);
}
+ /**
+ * 2 servers in different tiers, the first is in maitenance mode.
+ * Should not load a segment to the server in maintenance mode.
+ */
+ @Test
+ public void testLoadDuringMaitenance()
+ {
+final LoadQueuePeon mockPeon1 = createEmptyPeon();
+final LoadQueuePeon mockPeon2 = createOneCallPeonMock();
+
+LoadRule rule = createLoadRule(ImmutableMap.of(
+"tier1", 1,
+"tier2", 1
+));
+
+final DataSegment segment = createDataSegment("foo");
+
+
EasyMock.expect(mockBalancerStrategy.findNewSegmentHomeReplicator(EasyMock.anyObject(),
EasyMock.anyObject()))
+.andDelegateTo(balancerStrategy)
+.times(1);
+
+EasyMock.replay(mockPeon1, mockPeon2, mockBalancerStrategy);
+
+
+DruidCluster druidCluster = new DruidCluster(
+null,
+ImmutableMap.of(
+"tier1",
+Stream.of(
+createServerHolder("tier1", mockPeon1, true)
+).collect(Collectors.toCollection(() -> new TreeSet<>())),
+"tier2",
+Stream.of(
+createServerHolder("tier2", mockPeon2, false)
+).collect(Collectors.toCollection(() -> new TreeSet<>()))
+)
+);
+
+CoordinatorStats stats = rule.run(
+null,
+DruidCoordinatorRuntimeParams.newBuilder()
+ .withDruidCluster(druidCluster)
+
.withSegmentReplicantLookup(SegmentReplicantLookup.make(druidCluster))
+ .withReplicationManager(throttler)
+
.withBalancerStrategy(mockBalancerStrategy)
+
.withBalancerReferenceTimestamp(DateTimes.of("2013-01-01"))
+
.withAvailableSegments(Collections.singletonList(segment)).build(),
+segment
+);
+
+Assert.assertEquals(1L, stats.getTieredStat(LoadRule.ASSIGNED_COUNT,
"tier2"));
+EasyMock.verify(mockPeon1, mockPeon2, mockBalancerStrategy);
+ }
+
+
+ /**
+ * 2 tiers, 2 servers each, 1 server of the second tier is in maintenance.
+ * Should not load a segment to the server in maintenance mode.
+ */
+ @Test
+ public void testLoadReplicaDuringMaitenance()
+ {
+
EasyMock.expect(throttler.canCreateReplicant(EasyMock.anyString())).andReturn(true).anyTimes();
+
+final LoadQueuePeon mockPeon1 = createEmptyPeon();
+final LoadQueuePeon mockPeon2 = createOneCallPeonMock();
+final LoadQueuePeon mockPeon3 = createOneCallPeonMock();
+final LoadQueuePeon mockPeon4 = createOneCallPeonMock();
+
+LoadRule rule = createLoadRule(ImmutableMap.of(
+"tier1", 2,
+"tier2", 2
+));
+
+final DataSegment segment = createDataSegment("foo");
+
+throttler.registerReplicantCreation(EasyMock.eq("tier2"),
EasyMock.anyString(), EasyMock.anyObject());
+EasyMock.expectLastCall().times(2);
+
+ServerHolder holder1 = createServerHolder("tier1", mockPeon1, true);
+ServerHolder holder2 = createServerHolder("tier1", mockPeon2, false);
+ServerHolder holder3 = createServerHolder("tier2", mockPeon3, false);
+ServerHolder holder4 = createServerHolder("tier2", mockPeon4, false);
+
+EasyMock.expect(mockBalancerStrategy.findNewSegmentHomeReplicator(segment,
ImmutableList.of(holder2)))
+.andReturn(holder2);
+EasyMock.expect(mockBalancerStrategy.findNewSegmentHomeReplicator(segment,
ImmutableList.of(holder3, holder4)))
+.andReturn(holder3);
+EasyMock.expect(mockBalancerStrategy.findNewSegmentHomeReplicator(segment,
ImmutableList.of(holder4)))
+.andReturn(holder4);
+
+EasyMock.replay(throttler, mockPeon1, mockPeon2, mockPeon3, mockPeon4,
mockBalancerStrategy);
+
+
+DruidCluster druidCluster = new DruidCluster(
+null,
+ImmutableMap.of(
+"tier1",
+Stream.of(
+holder1,
+holder2
+).collect(Collectors.toCollection(() -> new TreeSet<>())),
+"tier2",
+Stream.of(
+holder3,
+holder4
+).collect(Collectors.toCollection(() -> new TreeSet<>()))
+)
+);
+
+CoordinatorStats stats = rule.run(
+null,
+DruidCoordinatorRuntimeParams.newBuilder()
+ .withDruidCluster(druidCluster)
+
.withSegmentReplicantLookup(SegmentReplicantLookup.make(druidCluster))
+ .withReplicat
[GitHub] egor-ryashin commented on a change in pull request #6349: maintenance mode for Historical
egor-ryashin commented on a change in pull request #6349: maintenance mode for
Historical
URL: https://github.com/apache/incubator-druid/pull/6349#discussion_r221274451
##
File path:
server/src/test/java/org/apache/druid/server/coordinator/rules/LoadRuleTest.java
##
@@ -685,6 +685,210 @@ public void testMaxLoadingQueueSize()
EasyMock.verify(throttler, mockBalancerStrategy);
}
+ /**
+ * 2 servers in different tiers, the first is in maitenance mode.
+ * Should not load a segment to the server in maintenance mode.
+ */
+ @Test
+ public void testLoadDuringMaitenance()
+ {
+final LoadQueuePeon mockPeon1 = createEmptyPeon();
+final LoadQueuePeon mockPeon2 = createOneCallPeonMock();
+
+LoadRule rule = createLoadRule(ImmutableMap.of(
+"tier1", 1,
+"tier2", 1
+));
+
+final DataSegment segment = createDataSegment("foo");
+
+
EasyMock.expect(mockBalancerStrategy.findNewSegmentHomeReplicator(EasyMock.anyObject(),
EasyMock.anyObject()))
+.andDelegateTo(balancerStrategy)
+.times(1);
+
+EasyMock.replay(mockPeon1, mockPeon2, mockBalancerStrategy);
+
+
+DruidCluster druidCluster = new DruidCluster(
+null,
+ImmutableMap.of(
+"tier1",
+Stream.of(
+createServerHolder("tier1", mockPeon1, true)
+).collect(Collectors.toCollection(() -> new TreeSet<>())),
+"tier2",
+Stream.of(
+createServerHolder("tier2", mockPeon2, false)
+).collect(Collectors.toCollection(() -> new TreeSet<>()))
+)
+);
+
+CoordinatorStats stats = rule.run(
+null,
+DruidCoordinatorRuntimeParams.newBuilder()
+ .withDruidCluster(druidCluster)
+
.withSegmentReplicantLookup(SegmentReplicantLookup.make(druidCluster))
+ .withReplicationManager(throttler)
+
.withBalancerStrategy(mockBalancerStrategy)
+
.withBalancerReferenceTimestamp(DateTimes.of("2013-01-01"))
+
.withAvailableSegments(Collections.singletonList(segment)).build(),
+segment
+);
+
+Assert.assertEquals(1L, stats.getTieredStat(LoadRule.ASSIGNED_COUNT,
"tier2"));
+EasyMock.verify(mockPeon1, mockPeon2, mockBalancerStrategy);
+ }
+
+
+ /**
+ * 2 tiers, 2 servers each, 1 server of the second tier is in maintenance.
+ * Should not load a segment to the server in maintenance mode.
+ */
+ @Test
+ public void testLoadReplicaDuringMaitenance()
+ {
+
EasyMock.expect(throttler.canCreateReplicant(EasyMock.anyString())).andReturn(true).anyTimes();
+
+final LoadQueuePeon mockPeon1 = createEmptyPeon();
+final LoadQueuePeon mockPeon2 = createOneCallPeonMock();
+final LoadQueuePeon mockPeon3 = createOneCallPeonMock();
+final LoadQueuePeon mockPeon4 = createOneCallPeonMock();
+
+LoadRule rule = createLoadRule(ImmutableMap.of(
+"tier1", 2,
+"tier2", 2
+));
+
+final DataSegment segment = createDataSegment("foo");
+
+throttler.registerReplicantCreation(EasyMock.eq("tier2"),
EasyMock.anyString(), EasyMock.anyObject());
+EasyMock.expectLastCall().times(2);
+
+ServerHolder holder1 = createServerHolder("tier1", mockPeon1, true);
+ServerHolder holder2 = createServerHolder("tier1", mockPeon2, false);
+ServerHolder holder3 = createServerHolder("tier2", mockPeon3, false);
+ServerHolder holder4 = createServerHolder("tier2", mockPeon4, false);
+
+EasyMock.expect(mockBalancerStrategy.findNewSegmentHomeReplicator(segment,
ImmutableList.of(holder2)))
+.andReturn(holder2);
+EasyMock.expect(mockBalancerStrategy.findNewSegmentHomeReplicator(segment,
ImmutableList.of(holder3, holder4)))
+.andReturn(holder3);
+EasyMock.expect(mockBalancerStrategy.findNewSegmentHomeReplicator(segment,
ImmutableList.of(holder4)))
+.andReturn(holder4);
+
+EasyMock.replay(throttler, mockPeon1, mockPeon2, mockPeon3, mockPeon4,
mockBalancerStrategy);
+
+
+DruidCluster druidCluster = new DruidCluster(
+null,
+ImmutableMap.of(
+"tier1",
+Stream.of(
+holder1,
+holder2
+).collect(Collectors.toCollection(() -> new TreeSet<>())),
+"tier2",
+Stream.of(
+holder3,
+holder4
+).collect(Collectors.toCollection(() -> new TreeSet<>()))
+)
+);
+
+CoordinatorStats stats = rule.run(
+null,
+DruidCoordinatorRuntimeParams.newBuilder()
+ .withDruidCluster(druidCluster)
+
.withSegmentReplicantLookup(SegmentReplicantLookup.make(druidCluster))
+ .withReplicat
[GitHub] egor-ryashin commented on a change in pull request #6349: maintenance mode for Historical
egor-ryashin commented on a change in pull request #6349: maintenance mode for Historical URL: https://github.com/apache/incubator-druid/pull/6349#discussion_r221274626 ## File path: server/src/test/java/org/apache/druid/server/coordinator/rules/LoadRuleTest.java ## @@ -760,4 +964,36 @@ private static LoadQueuePeon createLoadingPeon(List segments) return mockPeon; } + + private static int serverId = 0; Review comment: :white_check_mark: This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] egor-ryashin commented on a change in pull request #6349: maintenance mode for Historical
egor-ryashin commented on a change in pull request #6349: maintenance mode for
Historical
URL: https://github.com/apache/incubator-druid/pull/6349#discussion_r221274743
##
File path:
server/src/test/java/org/apache/druid/server/http/CoordinatorDynamicConfigTest.java
##
@@ -163,15 +173,16 @@ public void
testDeserializeWithoutMaxSegmentsInNodeLoadingQueue() throws Excepti
CoordinatorDynamicConfig.class
);
-assertConfig(actual, 1, 1, 1, 1, 1, 1, 2, true, ImmutableSet.of(), true,
0);
+assertConfig(actual, 1, 1, 1, 1, 1, 1, 2, true, ImmutableSet.of(), true,
0, ImmutableSet.of(), 0);
}
@Test
public void testBuilderDefaults()
{
CoordinatorDynamicConfig defaultConfig =
CoordinatorDynamicConfig.builder().build();
-assertConfig(defaultConfig, 90, 524288000, 100, 5, 15, 10, 1, false,
ImmutableSet.of(), false, 0);
+assertConfig(defaultConfig, 90, 524288000, 100, 5, 15, 10, 1, false,
ImmutableSet.of(), false, 0, ImmutableSet.of(),
+ 7);
Review comment:
:white_check_mark:
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] egor-ryashin commented on a change in pull request #6349: maintenance mode for Historical
egor-ryashin commented on a change in pull request #6349: maintenance mode for
Historical
URL: https://github.com/apache/incubator-druid/pull/6349#discussion_r221276758
##
File path:
server/src/test/java/org/apache/druid/server/coordinator/rules/LoadRuleTest.java
##
@@ -685,6 +685,210 @@ public void testMaxLoadingQueueSize()
EasyMock.verify(throttler, mockBalancerStrategy);
}
+ /**
+ * 2 servers in different tiers, the first is in maitenance mode.
+ * Should not load a segment to the server in maintenance mode.
+ */
+ @Test
+ public void testLoadDuringMaitenance()
+ {
+final LoadQueuePeon mockPeon1 = createEmptyPeon();
+final LoadQueuePeon mockPeon2 = createOneCallPeonMock();
+
+LoadRule rule = createLoadRule(ImmutableMap.of(
+"tier1", 1,
+"tier2", 1
+));
+
+final DataSegment segment = createDataSegment("foo");
+
+
EasyMock.expect(mockBalancerStrategy.findNewSegmentHomeReplicator(EasyMock.anyObject(),
EasyMock.anyObject()))
+.andDelegateTo(balancerStrategy)
+.times(1);
+
+EasyMock.replay(mockPeon1, mockPeon2, mockBalancerStrategy);
+
+
+DruidCluster druidCluster = new DruidCluster(
+null,
+ImmutableMap.of(
+"tier1",
+Stream.of(
Review comment:
btw, those (not mine) tests run successfully no matter which comparator is
used
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] egor-ryashin commented on a change in pull request #6349: maintenance mode for Historical
egor-ryashin commented on a change in pull request #6349: maintenance mode for
Historical
URL: https://github.com/apache/incubator-druid/pull/6349#discussion_r221276758
##
File path:
server/src/test/java/org/apache/druid/server/coordinator/rules/LoadRuleTest.java
##
@@ -685,6 +685,210 @@ public void testMaxLoadingQueueSize()
EasyMock.verify(throttler, mockBalancerStrategy);
}
+ /**
+ * 2 servers in different tiers, the first is in maitenance mode.
+ * Should not load a segment to the server in maintenance mode.
+ */
+ @Test
+ public void testLoadDuringMaitenance()
+ {
+final LoadQueuePeon mockPeon1 = createEmptyPeon();
+final LoadQueuePeon mockPeon2 = createOneCallPeonMock();
+
+LoadRule rule = createLoadRule(ImmutableMap.of(
+"tier1", 1,
+"tier2", 1
+));
+
+final DataSegment segment = createDataSegment("foo");
+
+
EasyMock.expect(mockBalancerStrategy.findNewSegmentHomeReplicator(EasyMock.anyObject(),
EasyMock.anyObject()))
+.andDelegateTo(balancerStrategy)
+.times(1);
+
+EasyMock.replay(mockPeon1, mockPeon2, mockBalancerStrategy);
+
+
+DruidCluster druidCluster = new DruidCluster(
+null,
+ImmutableMap.of(
+"tier1",
+Stream.of(
Review comment:
btw, those tests run successfully no matter which comparator is used
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] egor-ryashin commented on a change in pull request #6349: maintenance mode for Historical
egor-ryashin commented on a change in pull request #6349: maintenance mode for
Historical
URL: https://github.com/apache/incubator-druid/pull/6349#discussion_r221277398
##
File path:
server/src/test/java/org/apache/druid/server/coordinator/rules/LoadRuleTest.java
##
@@ -685,6 +685,210 @@ public void testMaxLoadingQueueSize()
EasyMock.verify(throttler, mockBalancerStrategy);
}
+ /**
+ * 2 servers in different tiers, the first is in maitenance mode.
+ * Should not load a segment to the server in maintenance mode.
+ */
+ @Test
+ public void testLoadDuringMaitenance()
+ {
+final LoadQueuePeon mockPeon1 = createEmptyPeon();
+final LoadQueuePeon mockPeon2 = createOneCallPeonMock();
+
+LoadRule rule = createLoadRule(ImmutableMap.of(
+"tier1", 1,
+"tier2", 1
+));
+
+final DataSegment segment = createDataSegment("foo");
+
+
EasyMock.expect(mockBalancerStrategy.findNewSegmentHomeReplicator(EasyMock.anyObject(),
EasyMock.anyObject()))
+.andDelegateTo(balancerStrategy)
+.times(1);
+
+EasyMock.replay(mockPeon1, mockPeon2, mockBalancerStrategy);
+
+
+DruidCluster druidCluster = new DruidCluster(
+null,
+ImmutableMap.of(
+"tier1",
+Stream.of(
Review comment:
I need to review those tests before intruducing the new constructor.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] egor-ryashin commented on a change in pull request #6349: maintenance mode for Historical
egor-ryashin commented on a change in pull request #6349: maintenance mode for
Historical
URL: https://github.com/apache/incubator-druid/pull/6349#discussion_r221291652
##
File path:
server/src/test/java/org/apache/druid/server/coordinator/rules/LoadRuleTest.java
##
@@ -685,6 +685,210 @@ public void testMaxLoadingQueueSize()
EasyMock.verify(throttler, mockBalancerStrategy);
}
+ /**
+ * 2 servers in different tiers, the first is in maitenance mode.
+ * Should not load a segment to the server in maintenance mode.
+ */
+ @Test
+ public void testLoadDuringMaitenance()
+ {
+final LoadQueuePeon mockPeon1 = createEmptyPeon();
+final LoadQueuePeon mockPeon2 = createOneCallPeonMock();
+
+LoadRule rule = createLoadRule(ImmutableMap.of(
+"tier1", 1,
+"tier2", 1
+));
+
+final DataSegment segment = createDataSegment("foo");
+
+
EasyMock.expect(mockBalancerStrategy.findNewSegmentHomeReplicator(EasyMock.anyObject(),
EasyMock.anyObject()))
+.andDelegateTo(balancerStrategy)
+.times(1);
+
+EasyMock.replay(mockPeon1, mockPeon2, mockBalancerStrategy);
+
+
+DruidCluster druidCluster = new DruidCluster(
+null,
+ImmutableMap.of(
+"tier1",
+Stream.of(
Review comment:
:white_check_mark:
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] fjy closed pull request #6399: Add missing comma
fjy closed pull request #6399: Add missing comma
URL: https://github.com/apache/incubator-druid/pull/6399
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/docs/content/tutorials/index.md b/docs/content/tutorials/index.md
index bd16066c2fe..66e3a4f642e 100644
--- a/docs/content/tutorials/index.md
+++ b/docs/content/tutorials/index.md
@@ -137,7 +137,7 @@ The sample data has the following columns, and an example
event is shown below:
{
"timestamp":"2015-09-12T20:03:45.018Z",
"channel":"#en.wikipedia",
- "namespace":"Main"
+ "namespace":"Main",
"page":"Spider-Man's powers and equipment",
"user":"foobar",
"comment":"/* Artificial web-shooters */",
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] fjy commented on issue #6399: Add missing comma
fjy commented on issue #6399: Add missing comma URL: https://github.com/apache/incubator-druid/pull/6399#issuecomment-425483925 @adursun thanks for the fix This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid] branch master updated: Add missing comma (#6399)
This is an automated email from the ASF dual-hosted git repository.
fjy pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git
The following commit(s) were added to refs/heads/master by this push:
new 6f44e56 Add missing comma (#6399)
6f44e56 is described below
commit 6f44e568db6cb6626c726c69459e55cc2315c237
Author: adursun
AuthorDate: Fri Sep 28 19:02:36 2018 +0300
Add missing comma (#6399)
---
docs/content/tutorials/index.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/docs/content/tutorials/index.md b/docs/content/tutorials/index.md
index bd16066..66e3a4f 100644
--- a/docs/content/tutorials/index.md
+++ b/docs/content/tutorials/index.md
@@ -137,7 +137,7 @@ The sample data has the following columns, and an example
event is shown below:
{
"timestamp":"2015-09-12T20:03:45.018Z",
"channel":"#en.wikipedia",
- "namespace":"Main"
+ "namespace":"Main",
"page":"Spider-Man's powers and equipment",
"user":"foobar",
"comment":"/* Artificial web-shooters */",
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] jihoonson commented on issue #6350: Remove ConvertSegmentTask, HadoopConverterTask, and ConvertSegmentBackwardsCompatibleTask
jihoonson commented on issue #6350: Remove ConvertSegmentTask, HadoopConverterTask, and ConvertSegmentBackwardsCompatibleTask URL: https://github.com/apache/incubator-druid/issues/6350#issuecomment-425512935 I want to add this to 0.13. if possible because this is an incompatible change which is allowed only in major releases. But, I think it's not necessarily a blocker. We can move this to another major release if it takes too long time to fix. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] jihoonson commented on a change in pull request #6393: Remove ConvertSegmentTask, HadoopConverterTask, and ConvertSegmentBackwardsCompatibleTask
jihoonson commented on a change in pull request #6393: Remove ConvertSegmentTask, HadoopConverterTask, and ConvertSegmentBackwardsCompatibleTask URL: https://github.com/apache/incubator-druid/pull/6393#discussion_r221330881 ## File path: docs/content/ingestion/misc-tasks.md ## @@ -4,68 +4,6 @@ layout: doc_page # Miscellaneous Tasks -## Version Converter Task Review comment: Thanks. Removed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] jihoonson commented on issue #6393: Remove ConvertSegmentTask, HadoopConverterTask, and ConvertSegmentBackwardsCompatibleTask
jihoonson commented on issue #6393: Remove ConvertSegmentTask, HadoopConverterTask, and ConvertSegmentBackwardsCompatibleTask URL: https://github.com/apache/incubator-druid/pull/6393#issuecomment-425514797 @nishantmonu51 can we use reindexing (`IndexTask` + `IngestSegmentFirehose` or `CompactionTask`) for those use cases? I feel `ConvertSegmentTask` is for too specific use cases which are not very common. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] jihoonson commented on issue #6006: Kafka ingestion tasks are not killed and keep spawning after stopping Supervisor
jihoonson commented on issue #6006: Kafka ingestion tasks are not killed and keep spawning after stopping Supervisor URL: https://github.com/apache/incubator-druid/issues/6006#issuecomment-425516823 Hi @frcake, what issue are you seeing exactly? If you mean it takes some time for a dataSource to be appeared in Coordinator console, that is an expected behavior. Coordinator periodically refreshes its metadata and so new dataSources might not appear until it refreshes. Please check http://druid.io/docs/latest/design/coordinator.html and http://druid.io/docs/latest/configuration/index.html#coordinator-operation. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] jihoonson commented on a change in pull request #6396: Implement MapVirtualColumn.makeDimensionSelector properly
jihoonson commented on a change in pull request #6396: Implement
MapVirtualColumn.makeDimensionSelector properly
URL: https://github.com/apache/incubator-druid/pull/6396#discussion_r221338751
##
File path:
extensions-contrib/virtual-columns/src/main/java/org/apache/druid/segment/MapTypeMapVirtualColumnDimensionSelector.java
##
@@ -0,0 +1,142 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.druid.segment;
+
+import com.google.common.base.Predicate;
+import org.apache.druid.query.filter.ValueMatcher;
+import org.apache.druid.query.monomorphicprocessing.RuntimeShapeInspector;
+import org.apache.druid.segment.data.IndexedInts;
+
+import javax.annotation.Nullable;
+import java.util.Map;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+/**
+ * {@link DimensionSelector} for {@link Map} type {@link MapVirtualColumn}.
This dimensionSelector only supports
+ * {@link #getObject()} currently.
+ */
+final class MapTypeMapVirtualColumnDimensionSelector extends
MapVirtualColumnDimensionSelector
+{
+ MapTypeMapVirtualColumnDimensionSelector(
+ DimensionSelector keySelector,
+ DimensionSelector valueSelector
+ )
+ {
+super(keySelector, valueSelector);
+ }
+
+ @Override
+ public IndexedInts getRow()
+ {
+throw new UnsupportedOperationException("Map column doesn't support
aggregation yet");
+ }
+
+ @Override
+ public ValueMatcher makeValueMatcher(@Nullable String value)
+ {
+return new ValueMatcher()
+{
+ @Override
+ public boolean matches()
+ {
+// Map column doesn't match with any string
+return false;
+ }
+
+ @Override
+ public void inspectRuntimeShape(RuntimeShapeInspector inspector)
+ {
+
+ }
+};
+ }
+
+ @Override
+ public ValueMatcher makeValueMatcher(Predicate predicate)
+ {
+return new ValueMatcher()
+{
+ @Override
+ public boolean matches()
+ {
+return false;
+ }
+
+ @Override
+ public void inspectRuntimeShape(RuntimeShapeInspector inspector)
+ {
+
+ }
+};
+ }
+
+ @Override
+ public int getValueCardinality()
+ {
+return CARDINALITY_UNKNOWN;
+ }
+
+ @Nullable
+ @Override
+ public String lookupName(int id)
+ {
+return null;
Review comment:
Fixed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] jihoonson commented on a change in pull request #6396: Implement MapVirtualColumn.makeDimensionSelector properly
jihoonson commented on a change in pull request #6396: Implement
MapVirtualColumn.makeDimensionSelector properly
URL: https://github.com/apache/incubator-druid/pull/6396#discussion_r221338768
##
File path:
extensions-contrib/virtual-columns/src/main/java/org/apache/druid/segment/StringTypeMapVirtualColumnDimensionSelector.java
##
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.druid.segment;
+
+import com.google.common.base.Preconditions;
+import com.google.common.base.Predicate;
+import org.apache.druid.query.filter.ValueMatcher;
+import org.apache.druid.query.monomorphicprocessing.RuntimeShapeInspector;
+import org.apache.druid.segment.data.IndexedInts;
+import org.apache.druid.segment.data.SingleIndexedInt;
+import org.apache.druid.segment.data.ZeroIndexedInts;
+
+import javax.annotation.Nullable;
+import java.util.Objects;
+import java.util.stream.IntStream;
+
+/**
+ * {@link DimensionSelector} for String type {@link MapVirtualColumn}. The
performance has not considered yet and so
+ * it may need to be improved later.
+ */
+final class StringTypeMapVirtualColumnDimensionSelector extends
MapVirtualColumnDimensionSelector
+{
+ private final String subColumnName;
+
+ StringTypeMapVirtualColumnDimensionSelector(
+ DimensionSelector keySelector,
+ DimensionSelector valueSelector,
+ String subColumnName
+ )
+ {
+super(keySelector, valueSelector);
+this.subColumnName = Preconditions.checkNotNull(subColumnName,
"subColumnName");
+ }
+
+ @Override
+ public IndexedInts getRow()
+ {
+final DimensionSelector keySelector = getKeySelector();
+final DimensionSelector valueSelector = getValueSelector();
+
+final IndexedInts keyIndices = keySelector.getRow();
+final IndexedInts valueIndices = valueSelector.getRow();
+
+final int limit = Math.min(keyIndices.size(), valueIndices.size());
+final int valueIndex = IntStream
+.range(0, limit)
+.filter(i ->
subColumnName.equals(keySelector.lookupName(keyIndices.get(i //
subColumnName is never null
+.findAny()
+.orElse(-1);
+if (valueIndex < 0) {
+ return ZeroIndexedInts.instance();
+} else {
+ return new SingleIndexedInt(valueIndex);
+}
+ }
+
+ @Override
+ public ValueMatcher makeValueMatcher(@Nullable String value)
+ {
+return new ValueMatcher()
+{
+ @Override
+ public boolean matches()
+ {
+return Objects.equals(value, getObject());
+ }
+
+ @Override
+ public void inspectRuntimeShape(RuntimeShapeInspector inspector)
+ {
+inspector.visit("keySelector", getKeySelector());
+inspector.visit("valueSelector", getValueSelector());
+inspector.visit("subColumnName", subColumnName);
+ }
+};
+ }
+
+ @Override
+ public ValueMatcher makeValueMatcher(Predicate predicate)
+ {
+return new ValueMatcher()
+{
+ @Override
+ public boolean matches()
+ {
+return predicate.apply((String) getObject());
+ }
+
+ @Override
+ public void inspectRuntimeShape(RuntimeShapeInspector inspector)
+ {
+inspector.visit("keySelector", getKeySelector());
+inspector.visit("valueSelector", getValueSelector());
+inspector.visit("subColumnName", subColumnName);
+ }
+};
+ }
+
+ @Override
+ public int getValueCardinality()
+ {
+// To get the value cardinarlity, we need to first check all keys and
values to find valid pairs, and then find the
+// number of distinct values among them.
+return CARDINALITY_UNKNOWN;
+ }
+
+ @Nullable
+ @Override
+ public String lookupName(int id)
+ {
+final DimensionSelector keySelector = getKeySelector();
+final DimensionSelector valueSelector = getValueSelector();
+
+final IndexedInts keyIndices = keySelector.getRow();
+final IndexedInts valueIndices = valueSelector.getRow();
+
+final int limit = Math.min(keyIndices.size(), valueIndices.size());
+final int valueIndex = IntStream
+.range(0, limit)
+.filter(i ->
subColumnName.equals(keySelector.lookupName(keyIndices.get(i //
subColumnName is never null
+
[GitHub] jihoonson commented on a change in pull request #6396: Implement MapVirtualColumn.makeDimensionSelector properly
jihoonson commented on a change in pull request #6396: Implement
MapVirtualColumn.makeDimensionSelector properly
URL: https://github.com/apache/incubator-druid/pull/6396#discussion_r221338797
##
File path:
processing/src/main/java/org/apache/druid/segment/data/SingleIndexedInt.java
##
@@ -33,6 +33,11 @@ public SingleIndexedInt()
{
}
+ public SingleIndexedInt(int value)
Review comment:
Thanks. Fixed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] jihoonson commented on issue #6396: Implement MapVirtualColumn.makeDimensionSelector properly
jihoonson commented on issue #6396: Implement MapVirtualColumn.makeDimensionSelector properly URL: https://github.com/apache/incubator-druid/pull/6396#issuecomment-425522713 @nishantmonu51 thanks. Addressed your comments. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] jihoonson commented on a change in pull request #6396: Implement MapVirtualColumn.makeDimensionSelector properly
jihoonson commented on a change in pull request #6396: Implement
MapVirtualColumn.makeDimensionSelector properly
URL: https://github.com/apache/incubator-druid/pull/6396#discussion_r221339164
##
File path:
extensions-contrib/virtual-columns/src/main/java/org/apache/druid/segment/MapVirtualColumnDimensionSelector.java
##
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.druid.segment;
+
+import org.apache.druid.common.config.NullHandling;
+import org.apache.druid.query.monomorphicprocessing.RuntimeShapeInspector;
+
+abstract class MapVirtualColumnDimensionSelector implements DimensionSelector
+{
+ private final DimensionSelector keySelector;
+ private final DimensionSelector valueSelector;
+
+ MapVirtualColumnDimensionSelector(
+ DimensionSelector keySelector,
+ DimensionSelector valueSelector
+ )
+ {
+this.keySelector = keySelector;
+this.valueSelector = valueSelector;
+ }
+
+ protected DimensionSelector getKeySelector()
+ {
+return keySelector;
+ }
+
+ protected DimensionSelector getValueSelector()
+ {
+return valueSelector;
+ }
+
+ @Override
+ public double getDouble()
+ {
+assert NullHandling.replaceWithDefault();
+return 0.0;
Review comment:
Sounds good. I think it would be a more general discussion about how to
support `Map` type in Druid.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] jihoonson closed pull request #6203: Add support targetCompactionSizeBytes for compactionTask
jihoonson closed pull request #6203: Add support targetCompactionSizeBytes for
compactionTask
URL: https://github.com/apache/incubator-druid/pull/6203
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/common/src/main/java/org/apache/druid/timeline/partition/PartitionHolder.java
b/common/src/main/java/org/apache/druid/timeline/partition/PartitionHolder.java
index 1ed0ae64bca..5e5f676e42f 100644
---
a/common/src/main/java/org/apache/druid/timeline/partition/PartitionHolder.java
+++
b/common/src/main/java/org/apache/druid/timeline/partition/PartitionHolder.java
@@ -26,6 +26,7 @@
import java.util.Iterator;
import java.util.List;
import java.util.SortedSet;
+import java.util.Spliterator;
import java.util.TreeSet;
/**
@@ -130,6 +131,12 @@ public boolean isComplete()
return holderSet.iterator();
}
+ @Override
+ public Spliterator> spliterator()
+ {
+return holderSet.spliterator();
+ }
+
public Iterable payloads()
{
return Iterables.transform(this, PartitionChunk::getObject);
diff --git
a/indexing-service/src/main/java/org/apache/druid/indexing/common/task/CompactionTask.java
b/indexing-service/src/main/java/org/apache/druid/indexing/common/task/CompactionTask.java
index ea1186aa3a7..cce1a743757 100644
---
a/indexing-service/src/main/java/org/apache/druid/indexing/common/task/CompactionTask.java
+++
b/indexing-service/src/main/java/org/apache/druid/indexing/common/task/CompactionTask.java
@@ -68,6 +68,7 @@
import org.apache.druid.segment.indexing.granularity.GranularitySpec;
import org.apache.druid.segment.loading.SegmentLoadingException;
import org.apache.druid.segment.realtime.firehose.ChatHandlerProvider;
+import org.apache.druid.server.coordinator.DataSourceCompactionConfig;
import org.apache.druid.server.security.AuthorizerMapper;
import org.apache.druid.timeline.DataSegment;
import org.apache.druid.timeline.TimelineObjectHolder;
@@ -86,6 +87,7 @@
import java.util.HashMap;
import java.util.List;
import java.util.Map;
+import java.util.Map.Entry;
import java.util.SortedSet;
import java.util.TreeSet;
import java.util.stream.Collectors;
@@ -101,13 +103,15 @@
private final List segments;
private final DimensionsSpec dimensionsSpec;
private final boolean keepSegmentGranularity;
+ @Nullable
+ private final Long targetCompactionSizeBytes;
+ @Nullable
private final IndexTuningConfig tuningConfig;
private final ObjectMapper jsonMapper;
@JsonIgnore
private final SegmentProvider segmentProvider;
-
@JsonIgnore
- private List indexTaskSpecs;
+ private final PartitionConfigurationManager partitionConfigurationManager;
@JsonIgnore
private final AuthorizerMapper authorizerMapper;
@@ -118,6 +122,9 @@
@JsonIgnore
private final RowIngestionMetersFactory rowIngestionMetersFactory;
+ @JsonIgnore
+ private List indexTaskSpecs;
+
@JsonCreator
public CompactionTask(
@JsonProperty("id") final String id,
@@ -127,6 +134,7 @@ public CompactionTask(
@Nullable @JsonProperty("segments") final List segments,
@Nullable @JsonProperty("dimensions") final DimensionsSpec
dimensionsSpec,
@Nullable @JsonProperty("keepSegmentGranularity") final Boolean
keepSegmentGranularity,
+ @Nullable @JsonProperty("targetCompactionSizeBytes") final Long
targetCompactionSizeBytes,
@Nullable @JsonProperty("tuningConfig") final IndexTuningConfig
tuningConfig,
@Nullable @JsonProperty("context") final Map context,
@JacksonInject ObjectMapper jsonMapper,
@@ -149,9 +157,11 @@ public CompactionTask(
this.keepSegmentGranularity = keepSegmentGranularity == null
? DEFAULT_KEEP_SEGMENT_GRANULARITY
: keepSegmentGranularity;
+this.targetCompactionSizeBytes = targetCompactionSizeBytes;
this.tuningConfig = tuningConfig;
this.jsonMapper = jsonMapper;
this.segmentProvider = segments == null ? new SegmentProvider(dataSource,
interval) : new SegmentProvider(segments);
+this.partitionConfigurationManager = new
PartitionConfigurationManager(targetCompactionSizeBytes, tuningConfig);
this.authorizerMapper = authorizerMapper;
this.chatHandlerProvider = chatHandlerProvider;
this.rowIngestionMetersFactory = rowIngestionMetersFactory;
@@ -181,6 +191,14 @@ public boolean isKeepSegmentGranularity()
return keepSegmentGranularity;
}
+ @Nullable
+ @JsonProperty
+ public Long getTargetCompactionSizeBytes()
+ {
+return targetCompactionSizeBytes;
+ }
+
+ @Nullable
@JsonProperty
public IndexTuningConfig getTuningConfig()
{
@@ -220,9 +238,9 @@ public TaskStatus run(final TaskToolbox toolbox) throws
Exception
indexTaskSpecs = createIngestionSchema(
[incubator-druid] branch master updated: Add support targetCompactionSizeBytes for compactionTask (#6203)
This is an automated email from the ASF dual-hosted git repository.
jihoonson pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git
The following commit(s) were added to refs/heads/master by this push:
new 122caec Add support targetCompactionSizeBytes for compactionTask
(#6203)
122caec is described below
commit 122caec7b187820a0a7fc89e85eb8216dd57df21
Author: Jihoon Son
AuthorDate: Fri Sep 28 11:16:35 2018 -0700
Add support targetCompactionSizeBytes for compactionTask (#6203)
* Add support targetCompactionSizeBytes for compactionTask
* fix test
* fix a bug in keepSegmentGranularity
* fix wrong noinspection comment
* address comments
---
.../druid/timeline/partition/PartitionHolder.java | 7 +
.../druid/indexing/common/task/CompactionTask.java | 205 +--
.../druid/indexing/common/task/IndexTask.java | 182 +++---
.../task/batch/parallel/ParallelIndexSubTask.java | 20 +-
.../indexing/common/task/CompactionTaskTest.java | 387 +++--
.../druid/indexing/common/task/TaskSerdeTest.java | 23 +-
.../coordinator/DataSourceCompactionConfig.java| 3 +-
7 files changed, 693 insertions(+), 134 deletions(-)
diff --git
a/common/src/main/java/org/apache/druid/timeline/partition/PartitionHolder.java
b/common/src/main/java/org/apache/druid/timeline/partition/PartitionHolder.java
index 1ed0ae6..5e5f676 100644
---
a/common/src/main/java/org/apache/druid/timeline/partition/PartitionHolder.java
+++
b/common/src/main/java/org/apache/druid/timeline/partition/PartitionHolder.java
@@ -26,6 +26,7 @@ import com.google.common.collect.Sets;
import java.util.Iterator;
import java.util.List;
import java.util.SortedSet;
+import java.util.Spliterator;
import java.util.TreeSet;
/**
@@ -130,6 +131,12 @@ public class PartitionHolder implements
Iterable>
return holderSet.iterator();
}
+ @Override
+ public Spliterator> spliterator()
+ {
+return holderSet.spliterator();
+ }
+
public Iterable payloads()
{
return Iterables.transform(this, PartitionChunk::getObject);
diff --git
a/indexing-service/src/main/java/org/apache/druid/indexing/common/task/CompactionTask.java
b/indexing-service/src/main/java/org/apache/druid/indexing/common/task/CompactionTask.java
index ea1186a..cce1a74 100644
---
a/indexing-service/src/main/java/org/apache/druid/indexing/common/task/CompactionTask.java
+++
b/indexing-service/src/main/java/org/apache/druid/indexing/common/task/CompactionTask.java
@@ -68,6 +68,7 @@ import
org.apache.druid.segment.indexing.granularity.ArbitraryGranularitySpec;
import org.apache.druid.segment.indexing.granularity.GranularitySpec;
import org.apache.druid.segment.loading.SegmentLoadingException;
import org.apache.druid.segment.realtime.firehose.ChatHandlerProvider;
+import org.apache.druid.server.coordinator.DataSourceCompactionConfig;
import org.apache.druid.server.security.AuthorizerMapper;
import org.apache.druid.timeline.DataSegment;
import org.apache.druid.timeline.TimelineObjectHolder;
@@ -86,6 +87,7 @@ import java.util.Comparator;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
+import java.util.Map.Entry;
import java.util.SortedSet;
import java.util.TreeSet;
import java.util.stream.Collectors;
@@ -101,13 +103,15 @@ public class CompactionTask extends AbstractTask
private final List segments;
private final DimensionsSpec dimensionsSpec;
private final boolean keepSegmentGranularity;
+ @Nullable
+ private final Long targetCompactionSizeBytes;
+ @Nullable
private final IndexTuningConfig tuningConfig;
private final ObjectMapper jsonMapper;
@JsonIgnore
private final SegmentProvider segmentProvider;
-
@JsonIgnore
- private List indexTaskSpecs;
+ private final PartitionConfigurationManager partitionConfigurationManager;
@JsonIgnore
private final AuthorizerMapper authorizerMapper;
@@ -118,6 +122,9 @@ public class CompactionTask extends AbstractTask
@JsonIgnore
private final RowIngestionMetersFactory rowIngestionMetersFactory;
+ @JsonIgnore
+ private List indexTaskSpecs;
+
@JsonCreator
public CompactionTask(
@JsonProperty("id") final String id,
@@ -127,6 +134,7 @@ public class CompactionTask extends AbstractTask
@Nullable @JsonProperty("segments") final List segments,
@Nullable @JsonProperty("dimensions") final DimensionsSpec
dimensionsSpec,
@Nullable @JsonProperty("keepSegmentGranularity") final Boolean
keepSegmentGranularity,
+ @Nullable @JsonProperty("targetCompactionSizeBytes") final Long
targetCompactionSizeBytes,
@Nullable @JsonProperty("tuningConfig") final IndexTuningConfig
tuningConfig,
@Nullable @JsonProperty("context") final Map context,
@JacksonInject ObjectMapper jsonMapper,
@@ -149,9 +157,11 @@ public class CompactionTask extends AbstractTask
this.keepSegmentGranularity = keepSeg
[GitHub] gianm commented on a change in pull request #6302: Add SQL id, request logs, and metrics
gianm commented on a change in pull request #6302: Add SQL id, request logs, and metrics URL: https://github.com/apache/incubator-druid/pull/6302#discussion_r221352359 ## File path: server/src/main/java/org/apache/druid/server/log/AbstractFileRequestLogger.java ## @@ -0,0 +1,116 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.druid.server.log; + +import com.google.common.base.Throwables; +import org.apache.druid.java.util.common.DateTimes; +import org.apache.druid.java.util.common.concurrent.ScheduledExecutors; +import org.apache.druid.java.util.common.guava.CloseQuietly; +import org.joda.time.DateTime; +import org.joda.time.Duration; +import org.joda.time.MutableDateTime; +import org.joda.time.chrono.ISOChronology; + +import java.io.File; +import java.io.FileNotFoundException; +import java.io.IOException; +import java.io.OutputStreamWriter; +import java.util.concurrent.Callable; +import java.util.concurrent.ScheduledExecutorService; + +public abstract class AbstractFileRequestLogger Review comment: Hmm, after thinking about it more, I think it's better to have everything go through one RequestLogger interface. Will write up the reasons why in a separate comment. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] gianm commented on issue #6302: Add SQL id, request logs, and metrics
gianm commented on issue #6302: Add SQL id, request logs, and metrics URL: https://github.com/apache/incubator-druid/pull/6302#issuecomment-425539467 @gaodayue @jon-wei re: https://github.com/apache/incubator-druid/pull/6302#discussion_r221352359, I am thinking it will be best to have a single RequestLogger interface with methods like `logNativeQuery` and `logSqlQuery`. There are three main reasons: 1. It is just as flexible as having multiple interfaces. Imagine a "MultiRequestLogger" that can route different types of logs to different underlying loggers (SQL to kafka, native to file; or different files, etc). Sort of like the "composing" emitter we have. 2. It is more powerful, in that it can do one thing that the multiple interfaces cannot: it can write different types of logs to the same file. 3. Last but not least: it is simpler for users that are just getting started. They set up request logging one time, and all request logs go to the same place (of course, there should be a `logType` field that allows users to understand what is being logged). We could even extend this in the future to add exception logging and other sorts of structured logging (as opposed to log4j, which we use for _unstructured_ logging). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] gianm commented on issue #6302: Add SQL id, request logs, and metrics
gianm commented on issue #6302: Add SQL id, request logs, and metrics URL: https://github.com/apache/incubator-druid/pull/6302#issuecomment-425539501 What do you think? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] leventov opened a new pull request #6400: Optimize data structures in SelectResultValueBuilder
leventov opened a new pull request #6400: Optimize data structures in SelectResultValueBuilder URL: https://github.com/apache/incubator-druid/pull/6400 `MinMaxPriorityQueue` is never needed. Also, when there is no threshold of the number of results, it's practically faster better to aggregate results in an `ArrayList` from the beginning and then sort in the end, than aggregate in a priority queue (because quick sort is considered faster than heap sort). Also, I don't understand the meaning of `pagingIdentifiers` - it's not explained anywhere. Is it important for this Map to put entries in order? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] gianm commented on issue #6400: Optimize data structures in SelectResultValueBuilder
gianm commented on issue #6400: Optimize data structures in SelectResultValueBuilder URL: https://github.com/apache/incubator-druid/pull/6400#issuecomment-425548573 @leventov are you using the Select query today? In general it is worse than the Scan query, except in one way - it supports time ordering. But we can add that to Scan as well. See #5006, #6088. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] leventov commented on issue #6400: Optimize data structures in SelectResultValueBuilder
leventov commented on issue #6400: Optimize data structures in SelectResultValueBuilder URL: https://github.com/apache/incubator-druid/pull/6400#issuecomment-425550207 @gianm no, we don't - I was just passing by this code. However still curious about `pagingIdentifiers` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] leventov commented on a change in pull request #6349: maintenance mode for Historical
leventov commented on a change in pull request #6349: maintenance mode for Historical URL: https://github.com/apache/incubator-druid/pull/6349#discussion_r220495765 ## File path: server/src/main/java/org/apache/druid/server/coordinator/CoordinatorDynamicConfig.java ## @@ -95,6 +100,8 @@ public CoordinatorDynamicConfig( this.killDataSourceWhitelist = parseJsonStringOrArray(killDataSourceWhitelist); this.killPendingSegmentsSkipList = parseJsonStringOrArray(killPendingSegmentsSkipList); this.maxSegmentsInNodeLoadingQueue = maxSegmentsInNodeLoadingQueue; +this.maintenanceList = parseJsonStringOrArray(maintenanceList); +this.maintenanceModeSegmentsPriority = Math.min(10, Math.max(0, maintenanceModeSegmentsPriority)); Review comment: I think it should rather check bounds than saturating, to avoid silent consumption of configuration errors This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] gianm commented on issue #6400: Optimize data structures in SelectResultValueBuilder
gianm commented on issue #6400: Optimize data structures in SelectResultValueBuilder URL: https://github.com/apache/incubator-druid/pull/6400#issuecomment-425550755 I'm not super familiar with how the pagingIdentifiers work exactly, but I think the idea is they represent cursors into specific segments, and are used when you have a paginated select query. I am not sure if the order matters, but I would guess no. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] leventov commented on a change in pull request #6349: maintenance mode for Historical
leventov commented on a change in pull request #6349: maintenance mode for
Historical
URL: https://github.com/apache/incubator-druid/pull/6349#discussion_r221367120
##
File path:
server/src/main/java/org/apache/druid/server/coordinator/helper/DruidCoordinatorBalancer.java
##
@@ -106,35 +104,76 @@ private void balanceTier(
return;
}
-final List toMoveFrom = Lists.newArrayList(servers);
-final List toMoveTo = Lists.newArrayList(servers);
+Map> partitions = servers.stream()
+
.collect(Collectors.partitioningBy(ServerHolder::isMaintenance));
+final List maintenance = partitions.get(true);
+final List general = partitions.get(false);
+log.info("Found %d servers in maintenance, %d general servers",
maintenance.size(), general.size());
-if (toMoveTo.size() <= 1) {
- log.info("[%s]: One or fewer servers found. Cannot balance.", tier);
+if (general.size() == 0) {
+ log.info("[%s]: 0 general servers found. Cannot balance.", tier);
return;
}
int numSegments = 0;
-for (ServerHolder sourceHolder : toMoveFrom) {
+for (ServerHolder sourceHolder : servers) {
numSegments += sourceHolder.getServer().getSegments().size();
}
-
if (numSegments == 0) {
log.info("No segments found. Cannot balance.");
return;
}
-final BalancerStrategy strategy = params.getBalancerStrategy();
final int maxSegmentsToMove =
Math.min(params.getCoordinatorDynamicConfig().getMaxSegmentsToMove(),
numSegments);
+int priority =
params.getCoordinatorDynamicConfig().getMaintenanceModeSegmentsPriority();
+int maintenanceSegmentsToMove = (int) Math.ceil(maxSegmentsToMove *
priority / 10.0);
+log.info("Balancing %d segments for servers in maintenance mode",
maintenanceSegmentsToMove);
+Pair maintenanceResult = balanceServers(params,
maintenance, general, maintenanceSegmentsToMove);
+int generalSegmentsToMove = maxSegmentsToMove - maintenanceResult.lhs;
+log.info("Balancing %d segments for servers in general mode",
generalSegmentsToMove);
+Pair generalResult = balanceServers(params, general,
general, generalSegmentsToMove);
+Pair result = new Pair(
+generalResult.lhs + maintenanceResult.lhs,
+generalResult.rhs + maintenanceResult.rhs
+);
+
+int moved = result.lhs;
+int unmoved = result.rhs;
+if (unmoved == maxSegmentsToMove) {
+ // Cluster should be alive and constantly adjusting
+ log.info("No good moves found in tier [%s]", tier);
+}
+stats.addToTieredStat("unmovedCount", tier, unmoved);
+stats.addToTieredStat("movedCount", tier, moved);
+
+if (params.getCoordinatorDynamicConfig().emitBalancingStats()) {
+ final BalancerStrategy strategy = params.getBalancerStrategy();
+ strategy.emitStats(tier, stats, Lists.newArrayList(servers));
Review comment:
The question is rather, should it still be based on `servers`, as well as
prior to your PR? Not on `general`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] gianm commented on a change in pull request #6094: Introduce SystemSchema tables (#5989)
gianm commented on a change in pull request #6094: Introduce SystemSchema tables (#5989) URL: https://github.com/apache/incubator-druid/pull/6094#discussion_r221369004 ## File path: docs/content/querying/sql.md ## @@ -519,6 +524,101 @@ SELECT * FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_SCHEMA = 'druid' AND TABLE_ |COLLATION_NAME|| |JDBC_TYPE|Type code from java.sql.Types (Druid extension)| +## SYSTEM SCHEMA + +The sys schema provides visibility into Druid segments, servers and tasks. Review comment: I think it's good to put it in quotes, so like: The "sys" schema. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] gianm commented on a change in pull request #6094: Introduce SystemSchema tables (#5989)
gianm commented on a change in pull request #6094: Introduce SystemSchema tables (#5989) URL: https://github.com/apache/incubator-druid/pull/6094#discussion_r221369584 ## File path: docs/content/querying/sql.md ## @@ -481,6 +486,88 @@ SELECT * FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_SCHEMA = 'druid' AND TABLE_ |COLLATION_NAME|| |JDBC_TYPE|Type code from java.sql.Types (Druid extension)| +## SYSTEM SCHEMA + +The SYS schema provides visibility into Druid segments, servers and tasks. +For example to retrieve all segments for datasource "wikipedia", use the query: +```sql +SELECT * FROM sys.segments WHERE datasource = 'wikipedia' +``` + +### SEGMENTS table +Segments table provides details on all Druid segments, whether they are published yet or not. + + +|Column|Notes| +|--|-| +|segment_id|Unique segment identifier| +|datasource|Name of datasource| +|start|Interval start time (in ISO 8601 format)| +|end|Interval end time (in ISO 8601 format)| +|size|Size of segment in bytes| +|version|Version number (generally an ISO8601 timestamp corresponding to when the segment set was first started)| +|partition_num|Partition number (an integer, unique within a datasource+interval+version; may not necessarily be contiguous)| +|num_replicas|Number replicas of this segment currently being served| +|is_published|True if this segment has been published to the metadata store| +|is_available|True if this segment is currently being served by any server| +|is_realtime|True if this segment is being served on a realtime server| +|payload|Jsonified datasegment payload| + +### SERVERS table +Servers table lists all data servers(any server that hosts a segment). It includes both historicals and peons. + +|Column|Notes| +|--|-| +|server|Server name in the form host:port| Review comment: It does support concatenating, but I think we should keep this here. It's basically the primary key of the server table, and is used for joins with the segment_servers table. The other fields (host, plaintext_port, etc) are provided too as conveniences. With system tables, since they're all generated dynamically anyway, it's ok to have some redundancy when it makes the user experience more convenient. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] surekhasaharan commented on a change in pull request #6094: Introduce SystemSchema tables (#5989)
surekhasaharan commented on a change in pull request #6094: Introduce SystemSchema tables (#5989) URL: https://github.com/apache/incubator-druid/pull/6094#discussion_r221369753 ## File path: docs/content/querying/sql.md ## @@ -481,6 +486,88 @@ SELECT * FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_SCHEMA = 'druid' AND TABLE_ |COLLATION_NAME|| |JDBC_TYPE|Type code from java.sql.Types (Druid extension)| +## SYSTEM SCHEMA + +The SYS schema provides visibility into Druid segments, servers and tasks. +For example to retrieve all segments for datasource "wikipedia", use the query: +```sql +SELECT * FROM sys.segments WHERE datasource = 'wikipedia' +``` + +### SEGMENTS table +Segments table provides details on all Druid segments, whether they are published yet or not. + + +|Column|Notes| +|--|-| +|segment_id|Unique segment identifier| +|datasource|Name of datasource| +|start|Interval start time (in ISO 8601 format)| +|end|Interval end time (in ISO 8601 format)| +|size|Size of segment in bytes| +|version|Version number (generally an ISO8601 timestamp corresponding to when the segment set was first started)| +|partition_num|Partition number (an integer, unique within a datasource+interval+version; may not necessarily be contiguous)| +|num_replicas|Number replicas of this segment currently being served| +|is_published|True if this segment has been published to the metadata store| +|is_available|True if this segment is currently being served by any server| +|is_realtime|True if this segment is being served on a realtime server| +|payload|Jsonified datasegment payload| + +### SERVERS table +Servers table lists all data servers(any server that hosts a segment). It includes both historicals and peons. + +|Column|Notes| +|--|-| +|server|Server name in the form host:port| Review comment: Just tried concat, it does not support that call `cannot translate call CONCAT($t1, $t2)`. May be there is a way to support concat in future. But for now users can just do `select server from sys.servers;` :). Also `server` is the primary key of this table and is used in the join table `segment_servers`. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] gianm commented on a change in pull request #6094: Introduce SystemSchema tables (#5989)
gianm commented on a change in pull request #6094: Introduce SystemSchema tables (#5989) URL: https://github.com/apache/incubator-druid/pull/6094#discussion_r221370122 ## File path: docs/content/querying/sql.md ## @@ -519,6 +524,101 @@ SELECT * FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_SCHEMA = 'druid' AND TABLE_ |COLLATION_NAME|| |JDBC_TYPE|Type code from java.sql.Types (Druid extension)| +## SYSTEM SCHEMA + +The sys schema provides visibility into Druid segments, servers and tasks. +For example to retrieve all segments for datasource "wikipedia", use the query: +```sql +SELECT * FROM sys.segments WHERE datasource = 'wikipedia' +``` + +### SEGMENTS table +Segments table provides details on all Druid segments, whether they are published yet or not. + + +|Column|Notes| +|--|-| +|segment_id|Unique segment identifier| +|datasource|Name of datasource| +|start|Interval start time (in ISO 8601 format)| +|end|Interval end time (in ISO 8601 format)| +|size|Size of segment in bytes| +|version|Version number (generally an ISO8601 timestamp corresponding to when the segment set was first started)| +|partition_num|Partition number (an integer, unique within a datasource+interval+version; may not necessarily be contiguous)| +|num_replicas|Number of replicas of this segment currently being served| +|num_rows|Number of rows in current segment, this value could be null if unkown to broker at query time| +|is_published|True if this segment has been published to the metadata store| +|is_available|True if this segment is currently being served by any server| +|is_realtime|True if this segment is being served on a realtime server| +|payload|JSON-serialized datasegment payload| + +### SERVERS table +Servers table lists all data servers(any server that hosts a segment). It includes both historicals and peons. + +|Column|Notes| +|--|-| +|server|Server name in the form host:port| +|host|Hostname of the server| +|plaintext_port|Unsecured port of the server, or -1 if plaintext traffic is disabled| +|tls_port|TLS port of the server, or -1 if TLS is disabled| +|server_type|Type of Druid service. Possible values include: historical, realtime and indexer_executor.| Review comment: It's not the greatest name, but, it's what the servers announce themselves as (legacy reasons). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] gianm commented on a change in pull request #6094: Introduce SystemSchema tables (#5989)
gianm commented on a change in pull request #6094: Introduce SystemSchema
tables (#5989)
URL: https://github.com/apache/incubator-druid/pull/6094#discussion_r221371033
##
File path:
sql/src/main/java/org/apache/druid/sql/calcite/schema/DruidSchema.java
##
@@ -320,25 +322,32 @@ public void awaitInitialization() throws
InterruptedException
private void addSegment(final DruidServerMetadata server, final DataSegment
segment)
{
synchronized (lock) {
- final Map knownSegments =
segmentSignatures.get(segment.getDataSource());
+ final Map knownSegments =
segmentMetadataInfo.get(segment.getDataSource());
if (knownSegments == null || !knownSegments.containsKey(segment)) {
+final long isRealtime = server.segmentReplicatable() ? 0 : 1;
Review comment:
It's a pretty standard way in Druid of differentiating realtime and
non-realtime servers. See CoordinatorBasedSegmentHandoffNotifier, DruidSchema,
and CachingClusteredClient, all of which use this method to determine if
segments are served by realtime servers or not. Maybe we could make this
clearer by adding a new "isRealtimeServer" method.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] surekhasaharan commented on a change in pull request #6094: Introduce SystemSchema tables (#5989)
surekhasaharan commented on a change in pull request #6094: Introduce
SystemSchema tables (#5989)
URL: https://github.com/apache/incubator-druid/pull/6094#discussion_r221371456
##
File path: sql/src/main/java/io/druid/sql/calcite/schema/SystemSchema.java
##
@@ -0,0 +1,537 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package io.druid.sql.calcite.schema;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.FluentIterable;
+import com.google.common.collect.ImmutableMap;
+import com.google.inject.Inject;
+import io.druid.client.DruidServer;
+import io.druid.client.ImmutableDruidDataSource;
+import io.druid.client.TimelineServerView;
+import io.druid.client.coordinator.Coordinator;
+import io.druid.client.indexing.IndexingService;
+import io.druid.client.selector.QueryableDruidServer;
+import io.druid.discovery.DruidLeaderClient;
+import io.druid.indexer.TaskStatusPlus;
+import io.druid.java.util.common.ISE;
+import io.druid.java.util.common.StringUtils;
+import io.druid.java.util.common.logger.Logger;
+import io.druid.java.util.http.client.response.FullResponseHolder;
+import io.druid.segment.column.ValueType;
+import io.druid.server.coordination.ServerType;
+import io.druid.server.security.AuthorizerMapper;
+import io.druid.sql.calcite.table.RowSignature;
+import io.druid.timeline.DataSegment;
+import org.apache.calcite.DataContext;
+import org.apache.calcite.linq4j.Enumerable;
+import org.apache.calcite.linq4j.Linq4j;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.schema.ScannableTable;
+import org.apache.calcite.schema.Table;
+import org.apache.calcite.schema.impl.AbstractSchema;
+import org.apache.calcite.schema.impl.AbstractTable;
+import org.jboss.netty.handler.codec.http.HttpMethod;
+import org.jboss.netty.handler.codec.http.HttpResponseStatus;
+import org.joda.time.DateTime;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+public class SystemSchema extends AbstractSchema
+{
+ private static final Logger log = new Logger(SystemSchema.class);
+
+ public static final String NAME = "sys";
+ private static final String SEGMENTS_TABLE = "segments";
+ private static final String SERVERS_TABLE = "servers";
+ private static final String SEGMENT_SERVERS_TABLE = "segment_servers";
+ private static final String TASKS_TABLE = "tasks";
+ private static final int SEGMENTS_TABLE_SIZE;
+ private static final int SEGMENT_SERVERS_TABLE_SIZE;
+
+ private static final RowSignature SEGMENTS_SIGNATURE = RowSignature
+ .builder()
+ .add("segment_id", ValueType.STRING)
+ .add("datasource", ValueType.STRING)
+ .add("start", ValueType.STRING)
+ .add("end", ValueType.STRING)
+ .add("size", ValueType.LONG)
+ .add("version", ValueType.STRING)
+ .add("partition_num", ValueType.STRING)
+ .add("num_replicas", ValueType.LONG)
+ .add("is_published", ValueType.LONG)
+ .add("is_available", ValueType.LONG)
+ .add("is_realtime", ValueType.LONG)
+ .add("payload", ValueType.STRING)
+ .build();
+
+ private static final RowSignature SERVERS_SIGNATURE = RowSignature
+ .builder()
+ .add("server", ValueType.STRING)
+ .add("scheme", ValueType.STRING)
+ .add("server_type", ValueType.STRING)
+ .add("tier", ValueType.STRING)
+ .add("curr_size", ValueType.LONG)
+ .add("max_size", ValueType.LONG)
+ .build();
+
+ private static final RowSignature SERVERSEGMENTS_SIGNATURE = RowSignature
+ .builder()
+ .add("server", ValueType.STRING)
+ .add("segment_id", ValueType.STRING)
+ .build();
+
+ private static final RowSignature TASKS_SIGNATURE = RowSignature
+ .builder()
+ .add("task_id", ValueType.STRING)
+ .add("type", ValueType.STRING)
+ .add("datasource", ValueType.STR
[GitHub] surekhasaharan commented on a change in pull request #6094: Introduce SystemSchema tables (#5989)
surekhasaharan commented on a change in pull request #6094: Introduce SystemSchema tables (#5989) URL: https://github.com/apache/incubator-druid/pull/6094#discussion_r221371837 ## File path: server/src/main/java/org/apache/druid/client/TimelineServerView.java ## @@ -36,6 +38,8 @@ @Nullable TimelineLookup getTimeline(DataSource dataSource); + Map getClients(); Review comment: renamed the method. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] surekhasaharan commented on a change in pull request #6094: Introduce SystemSchema tables (#5989)
surekhasaharan commented on a change in pull request #6094: Introduce
SystemSchema tables (#5989)
URL: https://github.com/apache/incubator-druid/pull/6094#discussion_r221372115
##
File path:
server/src/main/java/org/apache/druid/server/http/MetadataResource.java
##
@@ -136,6 +137,23 @@ public Response getDatabaseSegmentDataSource(
return Response.status(Response.Status.OK).entity(dataSource).build();
}
+ @GET
+ @Path("/segments")
+ @Produces(MediaType.APPLICATION_JSON)
+ @ResourceFilters(DatasourceResourceFilter.class)
+ public Response getDatabaseSegments()
+ {
+final Collection druidDataSources =
metadataSegmentManager.getInventory();
+final Set metadataSegments = druidDataSources
+.stream()
+.flatMap(t -> t.getSegments().stream())
+.collect(Collectors.toSet());
+
+Response.ResponseBuilder builder = Response.status(Response.Status.OK);
+return builder.entity(metadataSegments).build();
Review comment:
cool, didn't know about `StreamingOutput`, using it now.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] jihoonson commented on a change in pull request #6094: Introduce SystemSchema tables (#5989)
jihoonson commented on a change in pull request #6094: Introduce SystemSchema
tables (#5989)
URL: https://github.com/apache/incubator-druid/pull/6094#discussion_r221366851
##
File path:
sql/src/main/java/org/apache/druid/sql/calcite/schema/SystemSchema.java
##
@@ -0,0 +1,595 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.druid.sql.calcite.schema;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.JavaType;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.FluentIterable;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Iterables;
+import com.google.common.util.concurrent.ListenableFuture;
+import com.google.inject.Inject;
+import org.apache.calcite.DataContext;
+import org.apache.calcite.linq4j.DefaultEnumerable;
+import org.apache.calcite.linq4j.Enumerable;
+import org.apache.calcite.linq4j.Enumerator;
+import org.apache.calcite.linq4j.Linq4j;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.schema.ScannableTable;
+import org.apache.calcite.schema.Table;
+import org.apache.calcite.schema.impl.AbstractSchema;
+import org.apache.calcite.schema.impl.AbstractTable;
+import org.apache.druid.client.DruidServer;
+import org.apache.druid.client.JsonParserIterator;
+import org.apache.druid.client.TimelineServerView;
+import org.apache.druid.client.coordinator.Coordinator;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.client.selector.QueryableDruidServer;
+import org.apache.druid.discovery.DruidLeaderClient;
+import org.apache.druid.indexer.TaskStatusPlus;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.http.client.Request;
+import
org.apache.druid.java.util.http.client.io.AppendableByteArrayInputStream;
+import org.apache.druid.java.util.http.client.response.ClientResponse;
+import
org.apache.druid.java.util.http.client.response.InputStreamResponseHandler;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.server.security.AuthorizerMapper;
+import org.apache.druid.sql.calcite.table.RowSignature;
+import org.apache.druid.timeline.DataSegment;
+import org.jboss.netty.handler.codec.http.HttpMethod;
+import org.jboss.netty.handler.codec.http.HttpResponse;
+
+import javax.servlet.http.HttpServletResponse;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.ExecutionException;
+
+public class SystemSchema extends AbstractSchema
+{
+ private static final Logger log = new Logger(SystemSchema.class);
+
+ public static final String NAME = "sys";
+ private static final String SEGMENTS_TABLE = "segments";
+ private static final String SERVERS_TABLE = "servers";
+ private static final String SEGMENT_SERVERS_TABLE = "segment_servers";
+ private static final String TASKS_TABLE = "tasks";
+ private static final int SEGMENT_SERVERS_TABLE_SIZE;
+
+ private static final RowSignature SEGMENTS_SIGNATURE = RowSignature
+ .builder()
+ .add("segment_id", ValueType.STRING)
+ .add("datasource", ValueType.STRING)
+ .add("start", ValueType.STRING)
+ .add("end", ValueType.STRING)
+ .add("size", ValueType.LONG)
+ .add("version", ValueType.STRING)
+ .add("partition_num", ValueType.STRING)
+ .add("num_replicas", ValueType.LONG)
+ .add("num_rows", ValueType.LONG)
+ .add("is_published", ValueType.LONG)
+ .add("is_available", ValueType.LONG)
+ .add("is_realtime", ValueType.LONG)
+ .add("payload", ValueType.STRING)
+ .build();
+
+ private static final RowSign
[GitHub] jihoonson commented on a change in pull request #6094: Introduce SystemSchema tables (#5989)
jihoonson commented on a change in pull request #6094: Introduce SystemSchema
tables (#5989)
URL: https://github.com/apache/incubator-druid/pull/6094#discussion_r221356186
##
File path:
sql/src/main/java/org/apache/druid/sql/calcite/schema/SystemSchema.java
##
@@ -0,0 +1,595 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.druid.sql.calcite.schema;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.JavaType;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.FluentIterable;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Iterables;
+import com.google.common.util.concurrent.ListenableFuture;
+import com.google.inject.Inject;
+import org.apache.calcite.DataContext;
+import org.apache.calcite.linq4j.DefaultEnumerable;
+import org.apache.calcite.linq4j.Enumerable;
+import org.apache.calcite.linq4j.Enumerator;
+import org.apache.calcite.linq4j.Linq4j;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.schema.ScannableTable;
+import org.apache.calcite.schema.Table;
+import org.apache.calcite.schema.impl.AbstractSchema;
+import org.apache.calcite.schema.impl.AbstractTable;
+import org.apache.druid.client.DruidServer;
+import org.apache.druid.client.JsonParserIterator;
+import org.apache.druid.client.TimelineServerView;
+import org.apache.druid.client.coordinator.Coordinator;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.client.selector.QueryableDruidServer;
+import org.apache.druid.discovery.DruidLeaderClient;
+import org.apache.druid.indexer.TaskStatusPlus;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.http.client.Request;
+import
org.apache.druid.java.util.http.client.io.AppendableByteArrayInputStream;
+import org.apache.druid.java.util.http.client.response.ClientResponse;
+import
org.apache.druid.java.util.http.client.response.InputStreamResponseHandler;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.server.security.AuthorizerMapper;
+import org.apache.druid.sql.calcite.table.RowSignature;
+import org.apache.druid.timeline.DataSegment;
+import org.jboss.netty.handler.codec.http.HttpMethod;
+import org.jboss.netty.handler.codec.http.HttpResponse;
+
+import javax.servlet.http.HttpServletResponse;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.ExecutionException;
+
+public class SystemSchema extends AbstractSchema
+{
+ private static final Logger log = new Logger(SystemSchema.class);
+
+ public static final String NAME = "sys";
+ private static final String SEGMENTS_TABLE = "segments";
+ private static final String SERVERS_TABLE = "servers";
+ private static final String SEGMENT_SERVERS_TABLE = "segment_servers";
Review comment:
`SERVER_SEGMENTS_TABLE` and `server_segments`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] jihoonson commented on a change in pull request #6094: Introduce SystemSchema tables (#5989)
jihoonson commented on a change in pull request #6094: Introduce SystemSchema tables (#5989) URL: https://github.com/apache/incubator-druid/pull/6094#discussion_r221367718 ## File path: server/src/main/java/org/apache/druid/discovery/DruidLeaderClient.java ## @@ -133,6 +134,14 @@ public FullResponseHolder go(Request request) throws IOException, InterruptedExc return go(request, new FullResponseHandler(StandardCharsets.UTF_8)); } + public ListenableFuture goStream( Review comment: Why is this method needed? All callers immediately call `future.get()`. `go()` method is much better because it contains the retry mechanism considering changed leaders. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] jihoonson commented on a change in pull request #6094: Introduce SystemSchema tables (#5989)
jihoonson commented on a change in pull request #6094: Introduce SystemSchema
tables (#5989)
URL: https://github.com/apache/incubator-druid/pull/6094#discussion_r221353249
##
File path:
sql/src/main/java/org/apache/druid/sql/calcite/schema/SystemSchema.java
##
@@ -0,0 +1,595 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.druid.sql.calcite.schema;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.JavaType;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.FluentIterable;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Iterables;
+import com.google.common.util.concurrent.ListenableFuture;
+import com.google.inject.Inject;
+import org.apache.calcite.DataContext;
+import org.apache.calcite.linq4j.DefaultEnumerable;
+import org.apache.calcite.linq4j.Enumerable;
+import org.apache.calcite.linq4j.Enumerator;
+import org.apache.calcite.linq4j.Linq4j;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.schema.ScannableTable;
+import org.apache.calcite.schema.Table;
+import org.apache.calcite.schema.impl.AbstractSchema;
+import org.apache.calcite.schema.impl.AbstractTable;
+import org.apache.druid.client.DruidServer;
+import org.apache.druid.client.JsonParserIterator;
+import org.apache.druid.client.TimelineServerView;
+import org.apache.druid.client.coordinator.Coordinator;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.client.selector.QueryableDruidServer;
+import org.apache.druid.discovery.DruidLeaderClient;
+import org.apache.druid.indexer.TaskStatusPlus;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.http.client.Request;
+import
org.apache.druid.java.util.http.client.io.AppendableByteArrayInputStream;
+import org.apache.druid.java.util.http.client.response.ClientResponse;
+import
org.apache.druid.java.util.http.client.response.InputStreamResponseHandler;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.server.security.AuthorizerMapper;
+import org.apache.druid.sql.calcite.table.RowSignature;
+import org.apache.druid.timeline.DataSegment;
+import org.jboss.netty.handler.codec.http.HttpMethod;
+import org.jboss.netty.handler.codec.http.HttpResponse;
+
+import javax.servlet.http.HttpServletResponse;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.ExecutionException;
+
+public class SystemSchema extends AbstractSchema
+{
+ private static final Logger log = new Logger(SystemSchema.class);
+
+ public static final String NAME = "sys";
+ private static final String SEGMENTS_TABLE = "segments";
+ private static final String SERVERS_TABLE = "servers";
+ private static final String SEGMENT_SERVERS_TABLE = "segment_servers";
+ private static final String TASKS_TABLE = "tasks";
+ private static final int SEGMENT_SERVERS_TABLE_SIZE;
+
+ private static final RowSignature SEGMENTS_SIGNATURE = RowSignature
+ .builder()
+ .add("segment_id", ValueType.STRING)
+ .add("datasource", ValueType.STRING)
+ .add("start", ValueType.STRING)
+ .add("end", ValueType.STRING)
+ .add("size", ValueType.LONG)
+ .add("version", ValueType.STRING)
+ .add("partition_num", ValueType.STRING)
+ .add("num_replicas", ValueType.LONG)
+ .add("num_rows", ValueType.LONG)
+ .add("is_published", ValueType.LONG)
+ .add("is_available", ValueType.LONG)
+ .add("is_realtime", ValueType.LONG)
+ .add("payload", ValueType.STRING)
+ .build();
+
+ private static final RowSign
[GitHub] jihoonson commented on a change in pull request #6094: Introduce SystemSchema tables (#5989)
jihoonson commented on a change in pull request #6094: Introduce SystemSchema
tables (#5989)
URL: https://github.com/apache/incubator-druid/pull/6094#discussion_r221366807
##
File path:
sql/src/main/java/org/apache/druid/sql/calcite/schema/SystemSchema.java
##
@@ -0,0 +1,595 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.druid.sql.calcite.schema;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.JavaType;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.FluentIterable;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Iterables;
+import com.google.common.util.concurrent.ListenableFuture;
+import com.google.inject.Inject;
+import org.apache.calcite.DataContext;
+import org.apache.calcite.linq4j.DefaultEnumerable;
+import org.apache.calcite.linq4j.Enumerable;
+import org.apache.calcite.linq4j.Enumerator;
+import org.apache.calcite.linq4j.Linq4j;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.schema.ScannableTable;
+import org.apache.calcite.schema.Table;
+import org.apache.calcite.schema.impl.AbstractSchema;
+import org.apache.calcite.schema.impl.AbstractTable;
+import org.apache.druid.client.DruidServer;
+import org.apache.druid.client.JsonParserIterator;
+import org.apache.druid.client.TimelineServerView;
+import org.apache.druid.client.coordinator.Coordinator;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.client.selector.QueryableDruidServer;
+import org.apache.druid.discovery.DruidLeaderClient;
+import org.apache.druid.indexer.TaskStatusPlus;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.http.client.Request;
+import
org.apache.druid.java.util.http.client.io.AppendableByteArrayInputStream;
+import org.apache.druid.java.util.http.client.response.ClientResponse;
+import
org.apache.druid.java.util.http.client.response.InputStreamResponseHandler;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.server.security.AuthorizerMapper;
+import org.apache.druid.sql.calcite.table.RowSignature;
+import org.apache.druid.timeline.DataSegment;
+import org.jboss.netty.handler.codec.http.HttpMethod;
+import org.jboss.netty.handler.codec.http.HttpResponse;
+
+import javax.servlet.http.HttpServletResponse;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.ExecutionException;
+
+public class SystemSchema extends AbstractSchema
+{
+ private static final Logger log = new Logger(SystemSchema.class);
+
+ public static final String NAME = "sys";
+ private static final String SEGMENTS_TABLE = "segments";
+ private static final String SERVERS_TABLE = "servers";
+ private static final String SEGMENT_SERVERS_TABLE = "segment_servers";
+ private static final String TASKS_TABLE = "tasks";
+ private static final int SEGMENT_SERVERS_TABLE_SIZE;
+
+ private static final RowSignature SEGMENTS_SIGNATURE = RowSignature
+ .builder()
+ .add("segment_id", ValueType.STRING)
+ .add("datasource", ValueType.STRING)
+ .add("start", ValueType.STRING)
+ .add("end", ValueType.STRING)
+ .add("size", ValueType.LONG)
+ .add("version", ValueType.STRING)
+ .add("partition_num", ValueType.STRING)
+ .add("num_replicas", ValueType.LONG)
+ .add("num_rows", ValueType.LONG)
+ .add("is_published", ValueType.LONG)
+ .add("is_available", ValueType.LONG)
+ .add("is_realtime", ValueType.LONG)
+ .add("payload", ValueType.STRING)
+ .build();
+
+ private static final RowSign
[GitHub] jihoonson commented on a change in pull request #6094: Introduce SystemSchema tables (#5989)
jihoonson commented on a change in pull request #6094: Introduce SystemSchema
tables (#5989)
URL: https://github.com/apache/incubator-druid/pull/6094#discussion_r221368622
##
File path:
sql/src/main/java/org/apache/druid/sql/calcite/schema/SystemSchema.java
##
@@ -0,0 +1,595 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.druid.sql.calcite.schema;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.JavaType;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.FluentIterable;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Iterables;
+import com.google.common.util.concurrent.ListenableFuture;
+import com.google.inject.Inject;
+import org.apache.calcite.DataContext;
+import org.apache.calcite.linq4j.DefaultEnumerable;
+import org.apache.calcite.linq4j.Enumerable;
+import org.apache.calcite.linq4j.Enumerator;
+import org.apache.calcite.linq4j.Linq4j;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.schema.ScannableTable;
+import org.apache.calcite.schema.Table;
+import org.apache.calcite.schema.impl.AbstractSchema;
+import org.apache.calcite.schema.impl.AbstractTable;
+import org.apache.druid.client.DruidServer;
+import org.apache.druid.client.JsonParserIterator;
+import org.apache.druid.client.TimelineServerView;
+import org.apache.druid.client.coordinator.Coordinator;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.client.selector.QueryableDruidServer;
+import org.apache.druid.discovery.DruidLeaderClient;
+import org.apache.druid.indexer.TaskStatusPlus;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.http.client.Request;
+import
org.apache.druid.java.util.http.client.io.AppendableByteArrayInputStream;
+import org.apache.druid.java.util.http.client.response.ClientResponse;
+import
org.apache.druid.java.util.http.client.response.InputStreamResponseHandler;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.server.security.AuthorizerMapper;
+import org.apache.druid.sql.calcite.table.RowSignature;
+import org.apache.druid.timeline.DataSegment;
+import org.jboss.netty.handler.codec.http.HttpMethod;
+import org.jboss.netty.handler.codec.http.HttpResponse;
+
+import javax.servlet.http.HttpServletResponse;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.ExecutionException;
+
+public class SystemSchema extends AbstractSchema
+{
+ private static final Logger log = new Logger(SystemSchema.class);
+
+ public static final String NAME = "sys";
+ private static final String SEGMENTS_TABLE = "segments";
+ private static final String SERVERS_TABLE = "servers";
+ private static final String SEGMENT_SERVERS_TABLE = "segment_servers";
+ private static final String TASKS_TABLE = "tasks";
+ private static final int SEGMENT_SERVERS_TABLE_SIZE;
+
+ private static final RowSignature SEGMENTS_SIGNATURE = RowSignature
+ .builder()
+ .add("segment_id", ValueType.STRING)
+ .add("datasource", ValueType.STRING)
+ .add("start", ValueType.STRING)
+ .add("end", ValueType.STRING)
+ .add("size", ValueType.LONG)
+ .add("version", ValueType.STRING)
+ .add("partition_num", ValueType.STRING)
+ .add("num_replicas", ValueType.LONG)
+ .add("num_rows", ValueType.LONG)
+ .add("is_published", ValueType.LONG)
+ .add("is_available", ValueType.LONG)
+ .add("is_realtime", ValueType.LONG)
+ .add("payload", ValueType.STRING)
+ .build();
+
+ private static final RowSign
[GitHub] jihoonson commented on a change in pull request #6094: Introduce SystemSchema tables (#5989)
jihoonson commented on a change in pull request #6094: Introduce SystemSchema
tables (#5989)
URL: https://github.com/apache/incubator-druid/pull/6094#discussion_r221365114
##
File path:
sql/src/main/java/org/apache/druid/sql/calcite/schema/SegmentMetadataHolder.java
##
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.druid.sql.calcite.schema;
+
+import org.apache.druid.sql.calcite.table.RowSignature;
+
+import javax.annotation.Nullable;
+
+public class SegmentMetadataHolder
+{
+ private final Object lock = new Object();
+ private RowSignature rowSignature;
+ private final long isPublished;
+ private final long isAvailable;
+ private final long isRealtime;
+ private long numReplicas;
+ @Nullable
+ private Long numRows;
+
+
+ public SegmentMetadataHolder(
+ @Nullable RowSignature rowSignature,
+ long isPublished,
+ long isAvailable,
+ long isRealtime,
+ long numReplicas,
+ @Nullable Long numRows
+ )
+ {
+this.rowSignature = rowSignature;
+this.isPublished = isPublished;
+this.isAvailable = isAvailable;
+this.isRealtime = isRealtime;
+this.numReplicas = numReplicas;
+this.numRows = numRows;
+ }
+
+
+ public long isPublished()
+ {
+synchronized (lock) {
+ return isPublished;
+}
+ }
+
+ public long isAvailable()
+ {
+synchronized (lock) {
+ return isAvailable;
+}
+ }
+
+ public long isRealtime()
+ {
+synchronized (lock) {
+ return isRealtime;
+}
+ }
+
+ public long getNumReplicas()
+ {
+synchronized (lock) {
+ return numReplicas;
+}
+ }
+
+ @Nullable
+ public Long getNumRows()
+ {
+synchronized (lock) {
+ return numRows;
+}
+ }
+
+ @Nullable
+ public RowSignature getRowSignature()
+ {
+synchronized (lock) {
+ return rowSignature;
+}
+ }
+
+ public void setRowSignature(RowSignature rowSignature)
Review comment:
I think it's better to create a new instance whenever rowSignature, numRows,
numReplicas need to be updated because we can remove `lock` from this class and
they won't be updated too frequently.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] jihoonson commented on a change in pull request #6094: Introduce SystemSchema tables (#5989)
jihoonson commented on a change in pull request #6094: Introduce SystemSchema
tables (#5989)
URL: https://github.com/apache/incubator-druid/pull/6094#discussion_r221372496
##
File path:
sql/src/main/java/org/apache/druid/sql/calcite/schema/SystemSchema.java
##
@@ -0,0 +1,595 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.druid.sql.calcite.schema;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.JavaType;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.FluentIterable;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Iterables;
+import com.google.common.util.concurrent.ListenableFuture;
+import com.google.inject.Inject;
+import org.apache.calcite.DataContext;
+import org.apache.calcite.linq4j.DefaultEnumerable;
+import org.apache.calcite.linq4j.Enumerable;
+import org.apache.calcite.linq4j.Enumerator;
+import org.apache.calcite.linq4j.Linq4j;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.schema.ScannableTable;
+import org.apache.calcite.schema.Table;
+import org.apache.calcite.schema.impl.AbstractSchema;
+import org.apache.calcite.schema.impl.AbstractTable;
+import org.apache.druid.client.DruidServer;
+import org.apache.druid.client.JsonParserIterator;
+import org.apache.druid.client.TimelineServerView;
+import org.apache.druid.client.coordinator.Coordinator;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.client.selector.QueryableDruidServer;
+import org.apache.druid.discovery.DruidLeaderClient;
+import org.apache.druid.indexer.TaskStatusPlus;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.http.client.Request;
+import
org.apache.druid.java.util.http.client.io.AppendableByteArrayInputStream;
+import org.apache.druid.java.util.http.client.response.ClientResponse;
+import
org.apache.druid.java.util.http.client.response.InputStreamResponseHandler;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.server.security.AuthorizerMapper;
+import org.apache.druid.sql.calcite.table.RowSignature;
+import org.apache.druid.timeline.DataSegment;
+import org.jboss.netty.handler.codec.http.HttpMethod;
+import org.jboss.netty.handler.codec.http.HttpResponse;
+
+import javax.servlet.http.HttpServletResponse;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.ExecutionException;
+
+public class SystemSchema extends AbstractSchema
+{
+ private static final Logger log = new Logger(SystemSchema.class);
+
+ public static final String NAME = "sys";
+ private static final String SEGMENTS_TABLE = "segments";
+ private static final String SERVERS_TABLE = "servers";
+ private static final String SEGMENT_SERVERS_TABLE = "segment_servers";
+ private static final String TASKS_TABLE = "tasks";
+ private static final int SEGMENT_SERVERS_TABLE_SIZE;
+
+ private static final RowSignature SEGMENTS_SIGNATURE = RowSignature
+ .builder()
+ .add("segment_id", ValueType.STRING)
+ .add("datasource", ValueType.STRING)
+ .add("start", ValueType.STRING)
+ .add("end", ValueType.STRING)
+ .add("size", ValueType.LONG)
+ .add("version", ValueType.STRING)
+ .add("partition_num", ValueType.STRING)
+ .add("num_replicas", ValueType.LONG)
+ .add("num_rows", ValueType.LONG)
+ .add("is_published", ValueType.LONG)
+ .add("is_available", ValueType.LONG)
+ .add("is_realtime", ValueType.LONG)
+ .add("payload", ValueType.STRING)
+ .build();
+
+ private static final RowSign
[GitHub] jihoonson commented on a change in pull request #6094: Introduce SystemSchema tables (#5989)
jihoonson commented on a change in pull request #6094: Introduce SystemSchema
tables (#5989)
URL: https://github.com/apache/incubator-druid/pull/6094#discussion_r221372783
##
File path:
sql/src/main/java/org/apache/druid/sql/calcite/schema/SystemSchema.java
##
@@ -0,0 +1,595 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.druid.sql.calcite.schema;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.JavaType;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.FluentIterable;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Iterables;
+import com.google.common.util.concurrent.ListenableFuture;
+import com.google.inject.Inject;
+import org.apache.calcite.DataContext;
+import org.apache.calcite.linq4j.DefaultEnumerable;
+import org.apache.calcite.linq4j.Enumerable;
+import org.apache.calcite.linq4j.Enumerator;
+import org.apache.calcite.linq4j.Linq4j;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.schema.ScannableTable;
+import org.apache.calcite.schema.Table;
+import org.apache.calcite.schema.impl.AbstractSchema;
+import org.apache.calcite.schema.impl.AbstractTable;
+import org.apache.druid.client.DruidServer;
+import org.apache.druid.client.JsonParserIterator;
+import org.apache.druid.client.TimelineServerView;
+import org.apache.druid.client.coordinator.Coordinator;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.client.selector.QueryableDruidServer;
+import org.apache.druid.discovery.DruidLeaderClient;
+import org.apache.druid.indexer.TaskStatusPlus;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.http.client.Request;
+import
org.apache.druid.java.util.http.client.io.AppendableByteArrayInputStream;
+import org.apache.druid.java.util.http.client.response.ClientResponse;
+import
org.apache.druid.java.util.http.client.response.InputStreamResponseHandler;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.server.security.AuthorizerMapper;
+import org.apache.druid.sql.calcite.table.RowSignature;
+import org.apache.druid.timeline.DataSegment;
+import org.jboss.netty.handler.codec.http.HttpMethod;
+import org.jboss.netty.handler.codec.http.HttpResponse;
+
+import javax.servlet.http.HttpServletResponse;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.ExecutionException;
+
+public class SystemSchema extends AbstractSchema
+{
+ private static final Logger log = new Logger(SystemSchema.class);
+
+ public static final String NAME = "sys";
+ private static final String SEGMENTS_TABLE = "segments";
+ private static final String SERVERS_TABLE = "servers";
+ private static final String SEGMENT_SERVERS_TABLE = "segment_servers";
+ private static final String TASKS_TABLE = "tasks";
+ private static final int SEGMENT_SERVERS_TABLE_SIZE;
+
+ private static final RowSignature SEGMENTS_SIGNATURE = RowSignature
+ .builder()
+ .add("segment_id", ValueType.STRING)
+ .add("datasource", ValueType.STRING)
+ .add("start", ValueType.STRING)
+ .add("end", ValueType.STRING)
+ .add("size", ValueType.LONG)
+ .add("version", ValueType.STRING)
+ .add("partition_num", ValueType.STRING)
+ .add("num_replicas", ValueType.LONG)
+ .add("num_rows", ValueType.LONG)
+ .add("is_published", ValueType.LONG)
+ .add("is_available", ValueType.LONG)
+ .add("is_realtime", ValueType.LONG)
+ .add("payload", ValueType.STRING)
+ .build();
+
+ private static final RowSign
[GitHub] jihoonson commented on a change in pull request #6094: Introduce SystemSchema tables (#5989)
jihoonson commented on a change in pull request #6094: Introduce SystemSchema tables (#5989) URL: https://github.com/apache/incubator-druid/pull/6094#discussion_r221365849 ## File path: sql/src/main/java/org/apache/druid/sql/calcite/schema/DruidSchema.java ## @@ -103,9 +104,10 @@ // Protects access to segmentSignatures, mutableSegments, segmentsNeedingRefresh, lastRefresh, isServerViewInitialized private final Object lock = new Object(); - // DataSource -> Segment -> RowSignature for that segment. - // Use TreeMap for segments so they are merged in deterministic order, from older to newer. - private final Map> segmentSignatures = new HashMap<>(); + // DataSource -> Segment -> SegmentMetadataHolder(contains RowSignature) for that segment. Review comment: Looks that this is still needed, but I think the current concurrency control mechanism is too complicated. It's spanned over several classes including `DruidSchema`, `SystemSchema`, and `SegmentMetadataHolder`. This can introduce potential bugs and make maintenance difficult. Can we simplify this? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] jihoonson commented on a change in pull request #6094: Introduce SystemSchema tables (#5989)
jihoonson commented on a change in pull request #6094: Introduce SystemSchema
tables (#5989)
URL: https://github.com/apache/incubator-druid/pull/6094#discussion_r221371431
##
File path: server/src/main/java/org/apache/druid/client/BrokerServerView.java
##
@@ -322,4 +322,10 @@ private void runTimelineCallbacks(final
Function getClients()
+ {
+return clients;
Review comment:
The caller needs only `DruidServer`, not entire map or
`QueryableDruidServer`. Also, it's much better to not return mutable objects.
Please change this to like below:
```java
return clients.values().stream()
.map(queryableDruidServer ->
queryableDruidServer.getServer().toImmutableDruidServer())
.collect(Collectors.toList());
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] jihoonson commented on a change in pull request #6094: Introduce SystemSchema tables (#5989)
jihoonson commented on a change in pull request #6094: Introduce SystemSchema
tables (#5989)
URL: https://github.com/apache/incubator-druid/pull/6094#discussion_r221373401
##
File path:
sql/src/main/java/org/apache/druid/sql/calcite/schema/SystemSchema.java
##
@@ -0,0 +1,595 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.druid.sql.calcite.schema;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.JavaType;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.FluentIterable;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Iterables;
+import com.google.common.util.concurrent.ListenableFuture;
+import com.google.inject.Inject;
+import org.apache.calcite.DataContext;
+import org.apache.calcite.linq4j.DefaultEnumerable;
+import org.apache.calcite.linq4j.Enumerable;
+import org.apache.calcite.linq4j.Enumerator;
+import org.apache.calcite.linq4j.Linq4j;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.schema.ScannableTable;
+import org.apache.calcite.schema.Table;
+import org.apache.calcite.schema.impl.AbstractSchema;
+import org.apache.calcite.schema.impl.AbstractTable;
+import org.apache.druid.client.DruidServer;
+import org.apache.druid.client.JsonParserIterator;
+import org.apache.druid.client.TimelineServerView;
+import org.apache.druid.client.coordinator.Coordinator;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.client.selector.QueryableDruidServer;
+import org.apache.druid.discovery.DruidLeaderClient;
+import org.apache.druid.indexer.TaskStatusPlus;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.http.client.Request;
+import
org.apache.druid.java.util.http.client.io.AppendableByteArrayInputStream;
+import org.apache.druid.java.util.http.client.response.ClientResponse;
+import
org.apache.druid.java.util.http.client.response.InputStreamResponseHandler;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.server.security.AuthorizerMapper;
+import org.apache.druid.sql.calcite.table.RowSignature;
+import org.apache.druid.timeline.DataSegment;
+import org.jboss.netty.handler.codec.http.HttpMethod;
+import org.jboss.netty.handler.codec.http.HttpResponse;
+
+import javax.servlet.http.HttpServletResponse;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.ExecutionException;
+
+public class SystemSchema extends AbstractSchema
+{
+ private static final Logger log = new Logger(SystemSchema.class);
+
+ public static final String NAME = "sys";
+ private static final String SEGMENTS_TABLE = "segments";
+ private static final String SERVERS_TABLE = "servers";
+ private static final String SEGMENT_SERVERS_TABLE = "segment_servers";
+ private static final String TASKS_TABLE = "tasks";
+ private static final int SEGMENT_SERVERS_TABLE_SIZE;
+
+ private static final RowSignature SEGMENTS_SIGNATURE = RowSignature
+ .builder()
+ .add("segment_id", ValueType.STRING)
+ .add("datasource", ValueType.STRING)
+ .add("start", ValueType.STRING)
+ .add("end", ValueType.STRING)
+ .add("size", ValueType.LONG)
+ .add("version", ValueType.STRING)
+ .add("partition_num", ValueType.STRING)
+ .add("num_replicas", ValueType.LONG)
+ .add("num_rows", ValueType.LONG)
+ .add("is_published", ValueType.LONG)
+ .add("is_available", ValueType.LONG)
+ .add("is_realtime", ValueType.LONG)
+ .add("payload", ValueType.STRING)
+ .build();
+
+ private static final RowSign
[GitHub] jihoonson commented on a change in pull request #6094: Introduce SystemSchema tables (#5989)
jihoonson commented on a change in pull request #6094: Introduce SystemSchema
tables (#5989)
URL: https://github.com/apache/incubator-druid/pull/6094#discussion_r221350548
##
File path:
sql/src/main/java/org/apache/druid/sql/calcite/schema/SystemSchema.java
##
@@ -0,0 +1,595 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.druid.sql.calcite.schema;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.JavaType;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.FluentIterable;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Iterables;
+import com.google.common.util.concurrent.ListenableFuture;
+import com.google.inject.Inject;
+import org.apache.calcite.DataContext;
+import org.apache.calcite.linq4j.DefaultEnumerable;
+import org.apache.calcite.linq4j.Enumerable;
+import org.apache.calcite.linq4j.Enumerator;
+import org.apache.calcite.linq4j.Linq4j;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.schema.ScannableTable;
+import org.apache.calcite.schema.Table;
+import org.apache.calcite.schema.impl.AbstractSchema;
+import org.apache.calcite.schema.impl.AbstractTable;
+import org.apache.druid.client.DruidServer;
+import org.apache.druid.client.JsonParserIterator;
+import org.apache.druid.client.TimelineServerView;
+import org.apache.druid.client.coordinator.Coordinator;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.client.selector.QueryableDruidServer;
+import org.apache.druid.discovery.DruidLeaderClient;
+import org.apache.druid.indexer.TaskStatusPlus;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.http.client.Request;
+import
org.apache.druid.java.util.http.client.io.AppendableByteArrayInputStream;
+import org.apache.druid.java.util.http.client.response.ClientResponse;
+import
org.apache.druid.java.util.http.client.response.InputStreamResponseHandler;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.server.security.AuthorizerMapper;
+import org.apache.druid.sql.calcite.table.RowSignature;
+import org.apache.druid.timeline.DataSegment;
+import org.jboss.netty.handler.codec.http.HttpMethod;
+import org.jboss.netty.handler.codec.http.HttpResponse;
+
+import javax.servlet.http.HttpServletResponse;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.ExecutionException;
+
+public class SystemSchema extends AbstractSchema
+{
+ private static final Logger log = new Logger(SystemSchema.class);
+
+ public static final String NAME = "sys";
+ private static final String SEGMENTS_TABLE = "segments";
+ private static final String SERVERS_TABLE = "servers";
+ private static final String SEGMENT_SERVERS_TABLE = "segment_servers";
+ private static final String TASKS_TABLE = "tasks";
+ private static final int SEGMENT_SERVERS_TABLE_SIZE;
+
+ private static final RowSignature SEGMENTS_SIGNATURE = RowSignature
+ .builder()
+ .add("segment_id", ValueType.STRING)
+ .add("datasource", ValueType.STRING)
+ .add("start", ValueType.STRING)
+ .add("end", ValueType.STRING)
+ .add("size", ValueType.LONG)
+ .add("version", ValueType.STRING)
+ .add("partition_num", ValueType.STRING)
+ .add("num_replicas", ValueType.LONG)
+ .add("num_rows", ValueType.LONG)
+ .add("is_published", ValueType.LONG)
+ .add("is_available", ValueType.LONG)
+ .add("is_realtime", ValueType.LONG)
+ .add("payload", ValueType.STRING)
+ .build();
+
+ private static final RowSign
[GitHub] jihoonson commented on a change in pull request #6094: Introduce SystemSchema tables (#5989)
jihoonson commented on a change in pull request #6094: Introduce SystemSchema
tables (#5989)
URL: https://github.com/apache/incubator-druid/pull/6094#discussion_r221353193
##
File path:
sql/src/main/java/org/apache/druid/sql/calcite/schema/SystemSchema.java
##
@@ -0,0 +1,595 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.druid.sql.calcite.schema;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.JavaType;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.FluentIterable;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Iterables;
+import com.google.common.util.concurrent.ListenableFuture;
+import com.google.inject.Inject;
+import org.apache.calcite.DataContext;
+import org.apache.calcite.linq4j.DefaultEnumerable;
+import org.apache.calcite.linq4j.Enumerable;
+import org.apache.calcite.linq4j.Enumerator;
+import org.apache.calcite.linq4j.Linq4j;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.schema.ScannableTable;
+import org.apache.calcite.schema.Table;
+import org.apache.calcite.schema.impl.AbstractSchema;
+import org.apache.calcite.schema.impl.AbstractTable;
+import org.apache.druid.client.DruidServer;
+import org.apache.druid.client.JsonParserIterator;
+import org.apache.druid.client.TimelineServerView;
+import org.apache.druid.client.coordinator.Coordinator;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.client.selector.QueryableDruidServer;
+import org.apache.druid.discovery.DruidLeaderClient;
+import org.apache.druid.indexer.TaskStatusPlus;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.http.client.Request;
+import
org.apache.druid.java.util.http.client.io.AppendableByteArrayInputStream;
+import org.apache.druid.java.util.http.client.response.ClientResponse;
+import
org.apache.druid.java.util.http.client.response.InputStreamResponseHandler;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.server.security.AuthorizerMapper;
+import org.apache.druid.sql.calcite.table.RowSignature;
+import org.apache.druid.timeline.DataSegment;
+import org.jboss.netty.handler.codec.http.HttpMethod;
+import org.jboss.netty.handler.codec.http.HttpResponse;
+
+import javax.servlet.http.HttpServletResponse;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.ExecutionException;
+
+public class SystemSchema extends AbstractSchema
+{
+ private static final Logger log = new Logger(SystemSchema.class);
+
+ public static final String NAME = "sys";
+ private static final String SEGMENTS_TABLE = "segments";
+ private static final String SERVERS_TABLE = "servers";
+ private static final String SEGMENT_SERVERS_TABLE = "segment_servers";
+ private static final String TASKS_TABLE = "tasks";
+ private static final int SEGMENT_SERVERS_TABLE_SIZE;
+
+ private static final RowSignature SEGMENTS_SIGNATURE = RowSignature
+ .builder()
+ .add("segment_id", ValueType.STRING)
+ .add("datasource", ValueType.STRING)
+ .add("start", ValueType.STRING)
+ .add("end", ValueType.STRING)
+ .add("size", ValueType.LONG)
+ .add("version", ValueType.STRING)
+ .add("partition_num", ValueType.STRING)
+ .add("num_replicas", ValueType.LONG)
+ .add("num_rows", ValueType.LONG)
+ .add("is_published", ValueType.LONG)
+ .add("is_available", ValueType.LONG)
+ .add("is_realtime", ValueType.LONG)
+ .add("payload", ValueType.STRING)
+ .build();
+
+ private static final RowSign
[GitHub] jihoonson commented on a change in pull request #6094: Introduce SystemSchema tables (#5989)
jihoonson commented on a change in pull request #6094: Introduce SystemSchema
tables (#5989)
URL: https://github.com/apache/incubator-druid/pull/6094#discussion_r221368670
##
File path:
sql/src/main/java/org/apache/druid/sql/calcite/schema/SystemSchema.java
##
@@ -0,0 +1,595 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.druid.sql.calcite.schema;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.JavaType;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.FluentIterable;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Iterables;
+import com.google.common.util.concurrent.ListenableFuture;
+import com.google.inject.Inject;
+import org.apache.calcite.DataContext;
+import org.apache.calcite.linq4j.DefaultEnumerable;
+import org.apache.calcite.linq4j.Enumerable;
+import org.apache.calcite.linq4j.Enumerator;
+import org.apache.calcite.linq4j.Linq4j;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.schema.ScannableTable;
+import org.apache.calcite.schema.Table;
+import org.apache.calcite.schema.impl.AbstractSchema;
+import org.apache.calcite.schema.impl.AbstractTable;
+import org.apache.druid.client.DruidServer;
+import org.apache.druid.client.JsonParserIterator;
+import org.apache.druid.client.TimelineServerView;
+import org.apache.druid.client.coordinator.Coordinator;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.client.selector.QueryableDruidServer;
+import org.apache.druid.discovery.DruidLeaderClient;
+import org.apache.druid.indexer.TaskStatusPlus;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.http.client.Request;
+import
org.apache.druid.java.util.http.client.io.AppendableByteArrayInputStream;
+import org.apache.druid.java.util.http.client.response.ClientResponse;
+import
org.apache.druid.java.util.http.client.response.InputStreamResponseHandler;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.server.security.AuthorizerMapper;
+import org.apache.druid.sql.calcite.table.RowSignature;
+import org.apache.druid.timeline.DataSegment;
+import org.jboss.netty.handler.codec.http.HttpMethod;
+import org.jboss.netty.handler.codec.http.HttpResponse;
+
+import javax.servlet.http.HttpServletResponse;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.ExecutionException;
+
+public class SystemSchema extends AbstractSchema
+{
+ private static final Logger log = new Logger(SystemSchema.class);
+
+ public static final String NAME = "sys";
+ private static final String SEGMENTS_TABLE = "segments";
+ private static final String SERVERS_TABLE = "servers";
+ private static final String SEGMENT_SERVERS_TABLE = "segment_servers";
+ private static final String TASKS_TABLE = "tasks";
+ private static final int SEGMENT_SERVERS_TABLE_SIZE;
+
+ private static final RowSignature SEGMENTS_SIGNATURE = RowSignature
+ .builder()
+ .add("segment_id", ValueType.STRING)
+ .add("datasource", ValueType.STRING)
+ .add("start", ValueType.STRING)
+ .add("end", ValueType.STRING)
+ .add("size", ValueType.LONG)
+ .add("version", ValueType.STRING)
+ .add("partition_num", ValueType.STRING)
+ .add("num_replicas", ValueType.LONG)
+ .add("num_rows", ValueType.LONG)
+ .add("is_published", ValueType.LONG)
+ .add("is_available", ValueType.LONG)
+ .add("is_realtime", ValueType.LONG)
+ .add("payload", ValueType.STRING)
+ .build();
+
+ private static final RowSign
[GitHub] niketh commented on issue #4033: Query Response format to be based on http 'accept' header & Query Payload content type to be based on 'content-type' header
niketh commented on issue #4033: Query Response format to be based on http 'accept' header & Query Payload content type to be based on 'content-type' header URL: https://github.com/apache/incubator-druid/pull/4033#issuecomment-425559252 @jaihind213 Are you planning to finish this up? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] churrodog commented on issue #4033: Query Response format to be based on http 'accept' header & Query Payload content type to be based on 'content-type' header
churrodog commented on issue #4033: Query Response format to be based on http 'accept' header & Query Payload content type to be based on 'content-type' header URL: https://github.com/apache/incubator-druid/pull/4033#issuecomment-425559751 any plan on getting this in? This would be very useful. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] jihoonson commented on issue #4033: Query Response format to be based on http 'accept' header & Query Payload content type to be based on 'content-type' header
jihoonson commented on issue #4033: Query Response format to be based on http 'accept' header & Query Payload content type to be based on 'content-type' header URL: https://github.com/apache/incubator-druid/pull/4033#issuecomment-425561151 @jaihind213 sorry for the delayed review. The PR looks good to me. Would you please fix the conflicts? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] egor-ryashin commented on a change in pull request #6349: maintenance mode for Historical
egor-ryashin commented on a change in pull request #6349: maintenance mode for
Historical
URL: https://github.com/apache/incubator-druid/pull/6349#discussion_r221383921
##
File path:
server/src/main/java/org/apache/druid/server/coordinator/helper/DruidCoordinatorBalancer.java
##
@@ -106,35 +104,76 @@ private void balanceTier(
return;
}
-final List toMoveFrom = Lists.newArrayList(servers);
-final List toMoveTo = Lists.newArrayList(servers);
+Map> partitions = servers.stream()
+
.collect(Collectors.partitioningBy(ServerHolder::isMaintenance));
+final List maintenance = partitions.get(true);
+final List general = partitions.get(false);
+log.info("Found %d servers in maintenance, %d general servers",
maintenance.size(), general.size());
-if (toMoveTo.size() <= 1) {
- log.info("[%s]: One or fewer servers found. Cannot balance.", tier);
+if (general.size() == 0) {
+ log.info("[%s]: 0 general servers found. Cannot balance.", tier);
return;
}
int numSegments = 0;
-for (ServerHolder sourceHolder : toMoveFrom) {
+for (ServerHolder sourceHolder : servers) {
numSegments += sourceHolder.getServer().getSegments().size();
}
-
if (numSegments == 0) {
log.info("No segments found. Cannot balance.");
return;
}
-final BalancerStrategy strategy = params.getBalancerStrategy();
final int maxSegmentsToMove =
Math.min(params.getCoordinatorDynamicConfig().getMaxSegmentsToMove(),
numSegments);
+int priority =
params.getCoordinatorDynamicConfig().getMaintenanceModeSegmentsPriority();
+int maintenanceSegmentsToMove = (int) Math.ceil(maxSegmentsToMove *
priority / 10.0);
+log.info("Balancing %d segments for servers in maintenance mode",
maintenanceSegmentsToMove);
+Pair maintenanceResult = balanceServers(params,
maintenance, general, maintenanceSegmentsToMove);
+int generalSegmentsToMove = maxSegmentsToMove - maintenanceResult.lhs;
+log.info("Balancing %d segments for servers in general mode",
generalSegmentsToMove);
+Pair generalResult = balanceServers(params, general,
general, generalSegmentsToMove);
+Pair result = new Pair(
+generalResult.lhs + maintenanceResult.lhs,
+generalResult.rhs + maintenanceResult.rhs
+);
+
+int moved = result.lhs;
+int unmoved = result.rhs;
+if (unmoved == maxSegmentsToMove) {
+ // Cluster should be alive and constantly adjusting
+ log.info("No good moves found in tier [%s]", tier);
+}
+stats.addToTieredStat("unmovedCount", tier, unmoved);
+stats.addToTieredStat("movedCount", tier, moved);
+
+if (params.getCoordinatorDynamicConfig().emitBalancingStats()) {
+ final BalancerStrategy strategy = params.getBalancerStrategy();
+ strategy.emitStats(tier, stats, Lists.newArrayList(servers));
Review comment:
Ah, I got it, yes, every segment contributes the cost of its current
location to the Total Cost, if we remove servers in maintenance then it removes
those segments from the calculation too, but those servers are still
operational (moreover they can be removed from the maintenance list at any
time).
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] egor-ryashin commented on a change in pull request #6349: maintenance mode for Historical
egor-ryashin commented on a change in pull request #6349: maintenance mode for
Historical
URL: https://github.com/apache/incubator-druid/pull/6349#discussion_r221383921
##
File path:
server/src/main/java/org/apache/druid/server/coordinator/helper/DruidCoordinatorBalancer.java
##
@@ -106,35 +104,76 @@ private void balanceTier(
return;
}
-final List toMoveFrom = Lists.newArrayList(servers);
-final List toMoveTo = Lists.newArrayList(servers);
+Map> partitions = servers.stream()
+
.collect(Collectors.partitioningBy(ServerHolder::isMaintenance));
+final List maintenance = partitions.get(true);
+final List general = partitions.get(false);
+log.info("Found %d servers in maintenance, %d general servers",
maintenance.size(), general.size());
-if (toMoveTo.size() <= 1) {
- log.info("[%s]: One or fewer servers found. Cannot balance.", tier);
+if (general.size() == 0) {
+ log.info("[%s]: 0 general servers found. Cannot balance.", tier);
return;
}
int numSegments = 0;
-for (ServerHolder sourceHolder : toMoveFrom) {
+for (ServerHolder sourceHolder : servers) {
numSegments += sourceHolder.getServer().getSegments().size();
}
-
if (numSegments == 0) {
log.info("No segments found. Cannot balance.");
return;
}
-final BalancerStrategy strategy = params.getBalancerStrategy();
final int maxSegmentsToMove =
Math.min(params.getCoordinatorDynamicConfig().getMaxSegmentsToMove(),
numSegments);
+int priority =
params.getCoordinatorDynamicConfig().getMaintenanceModeSegmentsPriority();
+int maintenanceSegmentsToMove = (int) Math.ceil(maxSegmentsToMove *
priority / 10.0);
+log.info("Balancing %d segments for servers in maintenance mode",
maintenanceSegmentsToMove);
+Pair maintenanceResult = balanceServers(params,
maintenance, general, maintenanceSegmentsToMove);
+int generalSegmentsToMove = maxSegmentsToMove - maintenanceResult.lhs;
+log.info("Balancing %d segments for servers in general mode",
generalSegmentsToMove);
+Pair generalResult = balanceServers(params, general,
general, generalSegmentsToMove);
+Pair result = new Pair(
+generalResult.lhs + maintenanceResult.lhs,
+generalResult.rhs + maintenanceResult.rhs
+);
+
+int moved = result.lhs;
+int unmoved = result.rhs;
+if (unmoved == maxSegmentsToMove) {
+ // Cluster should be alive and constantly adjusting
+ log.info("No good moves found in tier [%s]", tier);
+}
+stats.addToTieredStat("unmovedCount", tier, unmoved);
+stats.addToTieredStat("movedCount", tier, moved);
+
+if (params.getCoordinatorDynamicConfig().emitBalancingStats()) {
+ final BalancerStrategy strategy = params.getBalancerStrategy();
+ strategy.emitStats(tier, stats, Lists.newArrayList(servers));
Review comment:
Ah, I got it, yes (`servers`), every segment contributes the cost of its
current location to the Total Cost, if we remove servers in maintenance then it
removes those segments from the calculation too, but those servers are still
operational (moreover they can be removed from the maintenance list at any
time).
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] jihoonson commented on issue #5869: Period load rules should include the future by default
jihoonson commented on issue #5869: Period load rules should include the future by default URL: https://github.com/apache/incubator-druid/issues/5869#issuecomment-425578518 @QiuMM thank you for the explanation. Now I got it. But, I'm afraid that it can introduce another issue. The purpose of load/drop rule is to control data retention which means how many data historicals should store locally. And as @gianm said, this problem mostly happens in stream ingestion which data size grows as time goes by. If dropRule works based on a fixed date, people might update the date for the dropRule periodically. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] jihoonson edited a comment on issue #5869: Period load rules should include the future by default
jihoonson edited a comment on issue #5869: Period load rules should include the future by default URL: https://github.com/apache/incubator-druid/issues/5869#issuecomment-425578518 @QiuMM thank you for the explanation. Now I get it. But, I'm afraid that it can introduce another issue. The purpose of load/drop rule is to control data retention which means how many data historicals should store locally. And as @gianm said, this problem mostly happens in stream ingestion which data size grows as time goes by. If dropRule works based on a fixed date, people might update the date for the dropRule periodically. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] surekhasaharan commented on a change in pull request #6094: Introduce SystemSchema tables (#5989)
surekhasaharan commented on a change in pull request #6094: Introduce SystemSchema tables (#5989) URL: https://github.com/apache/incubator-druid/pull/6094#discussion_r221391895 ## File path: benchmarks/src/main/java/org/apache/druid/benchmark/query/SqlBenchmark.java ## @@ -111,16 +114,20 @@ public void setup() .createQueryRunnerFactoryConglomerate(); final QueryRunnerFactoryConglomerate conglomerate = conglomerateCloserPair.lhs; final PlannerConfig plannerConfig = new PlannerConfig(); +final DruidLeaderClient druidLeaderClient = EasyMock.createMock(DruidLeaderClient.class); this.walker = new SpecificSegmentsQuerySegmentWalker(conglomerate).add(dataSegment, index); plannerFactory = new PlannerFactory( CalciteTests.createMockSchema(conglomerate, walker, plannerConfig), +new TestServerInventoryView(walker.getSegments()), Review comment: hm `TestServerInventoryView` is already in `/druid/sql/src/main/java/org/apache/druid/sql/calcite/util`. And not sure if I should rename it in this PR, as it was already existing. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] jihoonson edited a comment on issue #6393: Remove ConvertSegmentTask, HadoopConverterTask, and ConvertSegmentBackwardsCompatibleTask
jihoonson edited a comment on issue #6393: Remove ConvertSegmentTask, HadoopConverterTask, and ConvertSegmentBackwardsCompatibleTask URL: https://github.com/apache/incubator-druid/pull/6393#issuecomment-425578777 I realized the teamcity failure is legit. Fixed it. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] jihoonson commented on issue #6393: Remove ConvertSegmentTask, HadoopConverterTask, and ConvertSegmentBackwardsCompatibleTask
jihoonson commented on issue #6393: Remove ConvertSegmentTask, HadoopConverterTask, and ConvertSegmentBackwardsCompatibleTask URL: https://github.com/apache/incubator-druid/pull/6393#issuecomment-425578777 I realized the teamcity failure legit. Fixed it. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] surekhasaharan commented on a change in pull request #6094: Introduce SystemSchema tables (#5989)
surekhasaharan commented on a change in pull request #6094: Introduce SystemSchema tables (#5989) URL: https://github.com/apache/incubator-druid/pull/6094#discussion_r221392310 ## File path: docs/content/querying/sql.md ## @@ -519,6 +524,101 @@ SELECT * FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_SCHEMA = 'druid' AND TABLE_ |COLLATION_NAME|| |JDBC_TYPE|Type code from java.sql.Types (Druid extension)| +## SYSTEM SCHEMA + +The sys schema provides visibility into Druid segments, servers and tasks. Review comment: Actually, I am a bit confused, and not sure if `sys` would sound better at all places. @gianm do you have an opinion? Should we make is consistent and call it `sys` instead of `system` at all places. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] gianm commented on issue #6327: Druid based on OakIncrementalIndex
gianm commented on issue #6327: Druid based on OakIncrementalIndex URL: https://github.com/apache/incubator-druid/pull/6327#issuecomment-425586851 Hi @sanastas, Does this code run -- and if so how can it be activated? I am thinking we can start helping to review / test by deploying this patch to our test cluster. It will at least provide some more info on stability, perf, etc in our environment. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
