kfaraz commented on code in PR #16889:
URL: https://github.com/apache/druid/pull/16889#discussion_r1983017650
##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/ImmutableWorkerInfo.java:
##########
@@ -225,6 +244,13 @@ private int getWorkerParallelIndexCapacity(double
parallelIndexTaskSlotRatio)
return workerParallelIndexCapacity;
}
+ public Map<String, Integer> incrementTypeSpecificCapacity(String type, int
capacityToAdd)
Review Comment:
This method is a little misleading given that this is an immutable class.
I would advise simply inlining this change at the caller.
```
updatedMap = new
HashMap(immutableWorkerInfo.getCurrCapacityUsedByTaskType()).merge(type,
capacityToAdd, Integer::sum);
```
##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/setup/EqualDistributionWithCategorySpecWorkerSelectStrategy.java:
##########
@@ -74,7 +85,10 @@ public boolean equals(final Object o)
return false;
}
final EqualDistributionWithCategorySpecWorkerSelectStrategy that =
(EqualDistributionWithCategorySpecWorkerSelectStrategy) o;
- return Objects.equals(workerCategorySpec, that.workerCategorySpec);
+ if (!Objects.equals(workerCategorySpec, that.workerCategorySpec)) {
+ return false;
+ }
+ return Objects.equals(taskLimits, that.taskLimits);
Review Comment:
```suggestion
return Objects.equals(workerCategorySpec, that.workerCategorySpec)
&& Objects.equals(taskLimits, that.taskLimits);
```
##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/setup/FillCapacityWithCategorySpecWorkerSelectStrategy.java:
##########
@@ -74,7 +86,10 @@ public boolean equals(final Object o)
return false;
}
final FillCapacityWithCategorySpecWorkerSelectStrategy that =
(FillCapacityWithCategorySpecWorkerSelectStrategy) o;
- return Objects.equals(workerCategorySpec, that.workerCategorySpec);
+ if (!Objects.equals(workerCategorySpec, that.workerCategorySpec)) {
Review Comment:
fix this and hashCode as suggested for other classes.
##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/setup/TaskLimiterUtils.java:
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.overlord.setup;
+
+import org.apache.druid.indexing.common.task.Task;
+
+import java.util.Map;
+
+public class TaskLimiterUtils
Review Comment:
This class won't be needed if the utility method is moved to `TaskLimits`,
where it would be a more natural fit.
##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/setup/TaskLimits.java:
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.overlord.setup;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import org.apache.druid.common.config.Configs;
+
+import javax.annotation.Nullable;
+import java.util.Collections;
+import java.util.Map;
+import java.util.Objects;
+
+public class TaskLimits
+{
+ private final Map<String, Integer> taskLimits;
+ private final Map<String, Double> taskRatios;
+
+ public TaskLimits()
+ {
+ this(Collections.emptyMap(), Collections.emptyMap());
+ }
+
+ @JsonCreator
+ public TaskLimits(
+ @JsonProperty("taskLimits") @Nullable Map<String, Integer> taskLimits,
+ @JsonProperty("taskRatios") @Nullable Map<String, Double> taskRatios
+ )
+ {
+ validateLimits(taskLimits, taskRatios);
+ this.taskLimits = Configs.valueOrDefault(taskLimits,
Collections.emptyMap());
+ this.taskRatios = Configs.valueOrDefault(taskRatios,
Collections.emptyMap());
+ }
+
+ private void validateLimits(Map<String, Integer> taskLimits, Map<String,
Double> taskRatios)
+ {
+ if (taskLimits != null && taskLimits.values().stream().anyMatch(val -> val
< 0)) {
+ throw new IllegalArgumentException("Task limits should be bigger than
0");
+ } else if (taskRatios != null && taskRatios.values().stream().anyMatch(val
-> val < 0 || val > 1)) {
+ throw new IllegalArgumentException("Task ratios should be in the
interval of [0, 1]");
+ }
+ }
+
+ @JsonProperty
+ public Map<String, Integer> getTaskLimits()
+ {
+ return taskLimits;
+ }
+
+ @JsonProperty
+ public Map<String, Double> getTaskRatios()
+ {
+ return taskRatios;
+ }
+
+ @Override
+ public boolean equals(Object o)
+ {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ TaskLimits that = (TaskLimits) o;
+ return Objects.equals(taskLimits, that.taskLimits) &&
Objects.equals(taskRatios, that.taskRatios);
+ }
+
+ @Override
+ public int hashCode()
+ {
+ int result = taskLimits != null ? taskLimits.hashCode() : 0;
+ result = 31 * result + (taskRatios != null ? taskRatios.hashCode() : 0);
+ return result;
Review Comment:
Use `Objects.hashCode` for symmetry with `Objects.equals`.
##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/setup/TaskLimits.java:
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.overlord.setup;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import org.apache.druid.common.config.Configs;
+
+import javax.annotation.Nullable;
+import java.util.Collections;
+import java.util.Map;
+import java.util.Objects;
+
+public class TaskLimits
Review Comment:
Please add a javadoc and keep an `EMPTY` constant instance.
##########
website/.spelling:
##########
@@ -2132,6 +2132,7 @@ sqlQuery
successfulSending
[S]igar
taskBlackListCleanupPeriod
+taskLimits
Review Comment:
You wouldn't need to add it here if you use `taskLimits` in the heading too.
##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/setup/TaskLimits.java:
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.overlord.setup;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import org.apache.druid.common.config.Configs;
+
+import javax.annotation.Nullable;
+import java.util.Collections;
+import java.util.Map;
+import java.util.Objects;
+
+public class TaskLimits
+{
+ private final Map<String, Integer> taskLimits;
+ private final Map<String, Double> taskRatios;
+
+ public TaskLimits()
+ {
+ this(Collections.emptyMap(), Collections.emptyMap());
+ }
+
+ @JsonCreator
+ public TaskLimits(
+ @JsonProperty("taskLimits") @Nullable Map<String, Integer> taskLimits,
+ @JsonProperty("taskRatios") @Nullable Map<String, Double> taskRatios
+ )
+ {
+ validateLimits(taskLimits, taskRatios);
+ this.taskLimits = Configs.valueOrDefault(taskLimits,
Collections.emptyMap());
+ this.taskRatios = Configs.valueOrDefault(taskRatios,
Collections.emptyMap());
+ }
+
+ private void validateLimits(Map<String, Integer> taskLimits, Map<String,
Double> taskRatios)
+ {
+ if (taskLimits != null && taskLimits.values().stream().anyMatch(val -> val
< 0)) {
+ throw new IllegalArgumentException("Task limits should be bigger than
0");
Review Comment:
```suggestion
throw InvalidInput.exception("Task limit for any type must be greater
than zero. Found[%s].", taskLimits);
```
same suggestion for the other exception.
##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/setup/TaskLimiterUtils.java:
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.overlord.setup;
+
+import org.apache.druid.indexing.common.task.Task;
+
+import java.util.Map;
+
+public class TaskLimiterUtils
+{
+
+ /**
+ * Determines whether the given task can be executed based on task limits
and available capacity.
+ *
+ * @param task The task to evaluate.
+ * @param taskLimits The limits and ratios defining task execution
constraints.
+ * @param currentSlotsUsed The current capacity used by tasks of the same
type.
+ * @param totalCapacity The total available capacity across all workers.
+ * @return {@code true} if the task meets the defined limits and capacity
constraints; {@code false} otherwise.
+ */
+ public static boolean canRunTask(Task task, TaskLimits taskLimits, Integer
currentSlotsUsed, Integer totalCapacity)
Review Comment:
I think this need not be a static utility method.
This could very well live as a non-static method in `TaskLimits` itself.
```
public boolean canRunTask(Task task, Integer currentSlotsUsed, Integer
totalCapacity) {
}
```
##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/setup/EqualDistributionWorkerSelectStrategy.java:
##########
@@ -83,7 +95,10 @@ public boolean equals(final Object o)
return false;
}
final EqualDistributionWorkerSelectStrategy that =
(EqualDistributionWorkerSelectStrategy) o;
- return Objects.equals(affinityConfig, that.affinityConfig);
+ if (!Objects.equals(affinityConfig, that.affinityConfig)) {
+ return false;
+ }
+ return Objects.equals(taskLimits, that.taskLimits);
Review Comment:
```suggestion
return Objects.equals(affinityConfig, that.affinityConfig)
&& Objects.equals(taskLimits, that.taskLimits);
```
##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/setup/TaskLimits.java:
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.overlord.setup;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import org.apache.druid.common.config.Configs;
+
+import javax.annotation.Nullable;
+import java.util.Collections;
+import java.util.Map;
+import java.util.Objects;
+
+public class TaskLimits
+{
+ private final Map<String, Integer> taskLimits;
+ private final Map<String, Double> taskRatios;
+
+ public TaskLimits()
+ {
+ this(Collections.emptyMap(), Collections.emptyMap());
Review Comment:
Since we have now moved to Java 11, you may use this instead for brevity.
```suggestion
this(Map.of(), Map.of());
```
##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/setup/FillCapacityWithCategorySpecWorkerSelectStrategy.java:
##########
@@ -32,13 +33,17 @@
public class FillCapacityWithCategorySpecWorkerSelectStrategy implements
WorkerSelectStrategy
{
private final WorkerCategorySpec workerCategorySpec;
+ private final TaskLimits taskLimits;
+
@JsonCreator
public FillCapacityWithCategorySpecWorkerSelectStrategy(
- @JsonProperty("workerCategorySpec") WorkerCategorySpec workerCategorySpec
+ @JsonProperty("workerCategorySpec") WorkerCategorySpec
workerCategorySpec,
+ @JsonProperty("taskLimits") @Nullable TaskLimits taskLimits
)
{
this.workerCategorySpec = workerCategorySpec;
+ this.taskLimits = Configs.valueOrDefault(taskLimits, new TaskLimits());
Review Comment:
Rather than using `new TaskLimits()`, I would advise keeping a
`TaskLimits.EMPTY` instance that can be used everywhere.
##########
indexing-service/src/test/java/org/apache/druid/indexing/overlord/setup/EqualDistributionWorkerSelectStrategyTest.java:
##########
@@ -226,7 +226,7 @@ public void testStrongAffinity()
"bar", ImmutableSet.of("nonexistent-worker")
),
true
- )
+ ), null
Review Comment:
Please put this in the next line. Same comment above.
##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/setup/FillCapacityWorkerSelectStrategy.java:
##########
@@ -81,20 +93,26 @@ public boolean equals(final Object o)
return false;
}
final FillCapacityWorkerSelectStrategy that =
(FillCapacityWorkerSelectStrategy) o;
- return Objects.equals(affinityConfig, that.affinityConfig);
+ if (!Objects.equals(affinityConfig, that.affinityConfig)) {
+ return false;
+ }
+ return Objects.equals(taskLimits, that.taskLimits);
}
@Override
public int hashCode()
{
- return Objects.hash(affinityConfig);
+ int result = affinityConfig != null ? affinityConfig.hashCode() : 0;
+ result = 31 * result + (taskLimits != null ? taskLimits.hashCode() : 0);
+ return result;
Review Comment:
```suggestion
return Objects.hash(affiinityConfig, taskLimits);
```
##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/setup/EqualDistributionWithCategorySpecWorkerSelectStrategy.java:
##########
@@ -74,7 +85,10 @@ public boolean equals(final Object o)
return false;
}
final EqualDistributionWithCategorySpecWorkerSelectStrategy that =
(EqualDistributionWithCategorySpecWorkerSelectStrategy) o;
- return Objects.equals(workerCategorySpec, that.workerCategorySpec);
+ if (!Objects.equals(workerCategorySpec, that.workerCategorySpec)) {
+ return false;
+ }
+ return Objects.equals(taskLimits, that.taskLimits);
Review Comment:
Please update the hashCode method as well.
##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/setup/TaskLimits.java:
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.overlord.setup;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import org.apache.druid.common.config.Configs;
+
+import javax.annotation.Nullable;
+import java.util.Collections;
+import java.util.Map;
+import java.util.Objects;
+
+public class TaskLimits
+{
+ private final Map<String, Integer> taskLimits;
+ private final Map<String, Double> taskRatios;
+
+ public TaskLimits()
+ {
+ this(Collections.emptyMap(), Collections.emptyMap());
+ }
+
+ @JsonCreator
+ public TaskLimits(
+ @JsonProperty("taskLimits") @Nullable Map<String, Integer> taskLimits,
+ @JsonProperty("taskRatios") @Nullable Map<String, Double> taskRatios
Review Comment:
This is a little confusing, since these two fields are already inside a
field named `taskLimits`, right?
I think it would be cleaner if we called these fields something like
`maxCountByType`, `maxRatioByType`.
##########
indexing-service/src/test/java/org/apache/druid/indexing/overlord/setup/EqualDistributionWithAffinityWorkerSelectStrategyTest.java:
##########
@@ -139,12 +144,147 @@ public void testSerde() throws Exception
{
final ObjectMapper objectMapper = TestHelper.makeJsonMapper();
final EqualDistributionWorkerSelectStrategy strategy = new
EqualDistributionWithAffinityWorkerSelectStrategy(
- new AffinityConfig(ImmutableMap.of("foo",
ImmutableSet.of("localhost")), false)
+ new AffinityConfig(ImmutableMap.of("foo",
ImmutableSet.of("localhost")), false),
+ null
);
final WorkerSelectStrategy strategy2 = objectMapper.readValue(
objectMapper.writeValueAsBytes(strategy),
WorkerSelectStrategy.class
);
Assert.assertEquals(strategy, strategy2);
}
+
+ @Test
+ public void testFindWorkerForTaskWithGlobalLimits()
+ {
+ Map<String, Integer> taskLimits = new HashMap<>();
+ taskLimits.put("noop", 2);
+
+ Map<String, Integer> capacityUsed = new HashMap<>();
+ capacityUsed.put("noop", 1);
+ EqualDistributionWorkerSelectStrategy strategy = new
EqualDistributionWithAffinityWorkerSelectStrategy(
+ null,
+ new TaskLimits(taskLimits, null)
+ );
+
+ NoopTask noopTask = NoopTask.forDatasource("foo");
+ ImmutableWorkerInfo worker = strategy.findWorkerForTask(
+ new RemoteTaskRunnerConfig(),
+ ImmutableMap.of(
+ "localhost0",
+ new ImmutableWorkerInfo(
+ new Worker("http", "localhost0", "localhost0", 2, "v1",
WorkerConfig.DEFAULT_CATEGORY), 0,
+ new HashSet<>(),
+ new HashSet<>(),
+ DateTimes.nowUtc()
+ ),
+ "localhost1",
+ new ImmutableWorkerInfo(
+ new Worker("http", "localhost1", "localhost1", 2, "v1",
WorkerConfig.DEFAULT_CATEGORY), 0,
+ 0,
+ capacityUsed,
+ new HashSet<>(),
+ new HashSet<>(),
+ DateTimes.nowUtc()
+ )
+ ),
+ noopTask
+ );
+ Assert.assertNotNull(worker);
+
+ ImmutableWorkerInfo worker1 = strategy.findWorkerForTask(
+ new RemoteTaskRunnerConfig(),
+ ImmutableMap.of(
+ "localhost0",
+ new ImmutableWorkerInfo(
+ new Worker("http", "localhost0", "localhost0", 2, "v1",
WorkerConfig.DEFAULT_CATEGORY), 0,
+ 0,
+ capacityUsed,
+ new HashSet<>(),
+ new HashSet<>(),
Review Comment:
Suggestion:
You may use `Set.of()` instead. Similar, `List.of()` and `Map.of()` are
pretty handy too for creating immutable objects.
##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/setup/WorkerSelectUtils.java:
##########
@@ -145,9 +148,16 @@ public static ImmutableWorkerInfo selectWorker(
private static Map<String, ImmutableWorkerInfo> getRunnableWorkers(
final Task task,
final Map<String, ImmutableWorkerInfo> allWorkers,
- final WorkerTaskRunnerConfig workerTaskRunnerConfig
+ final WorkerTaskRunnerConfig workerTaskRunnerConfig,
+ final TaskLimits taskLimits
)
{
+ if (!TaskLimiterUtils.canRunTask(task,
+ taskLimits,
+ getTotalCapacityUsedByType(allWorkers,
task.getType()), getTotalCapacity(allWorkers)
+ )) {
Review Comment:
style:
```suggestion
if (
!TaskLimiterUtils.canRunTask(
task,
taskLimits,
getTotalCapacityUsedByType(allWorkers, task.getType()),
getTotalCapacity(allWorkers)
)
) {
```
##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/setup/EqualDistributionWorkerSelectStrategy.java:
##########
@@ -83,7 +95,10 @@ public boolean equals(final Object o)
return false;
}
final EqualDistributionWorkerSelectStrategy that =
(EqualDistributionWorkerSelectStrategy) o;
- return Objects.equals(affinityConfig, that.affinityConfig);
+ if (!Objects.equals(affinityConfig, that.affinityConfig)) {
+ return false;
+ }
+ return Objects.equals(taskLimits, that.taskLimits);
Review Comment:
Please update the `hashCode` as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
