[
https://issues.apache.org/jira/browse/STORM-1336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15125622#comment-15125622
]
ASF GitHub Bot commented on STORM-1336:
---------------------------------------
Github user bastiliu commented on a diff in the pull request:
https://github.com/apache/storm/pull/1053#discussion_r51373442
--- Diff:
storm-core/src/jvm/org/apache/storm/container/cgroup/CgroupManager.java ---
@@ -0,0 +1,176 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.container.cgroup;
+
+import org.apache.storm.Config;
+import org.apache.storm.container.ResourceIsolationInterface;
+import org.apache.storm.container.cgroup.core.CpuCore;
+import org.apache.storm.container.cgroup.core.MemoryCore;
+import org.apache.storm.utils.Utils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+public class CgroupManager implements ResourceIsolationInterface {
+
+ public static final Logger LOG =
LoggerFactory.getLogger(CgroupManager.class);
+
+ private CgroupCenter center;
+
+ private Hierarchy h;
+
+ private CgroupCommon rootCgroup;
+
+ private static String rootDir;
+
+ private Map conf;
+
+ public void prepare(Map conf) throws IOException {
+ this.conf = conf;
+ this.rootDir = Config.getCgroupRootDir(this.conf);
+ if (this.rootDir == null) {
+ throw new RuntimeException("Check configuration file. The
supervisor.cgroup.rootdir is missing.");
+ }
+
+ File file = new File(Config.getCgroupStormHierarchyDir(conf) + "/"
+ this.rootDir);
+ if (!file.exists()) {
+ LOG.error("{}/{} is not existing.",
Config.getCgroupStormHierarchyDir(conf), this.rootDir);
+ throw new RuntimeException("Check if cgconfig service starts
or /etc/cgconfig.conf is consistent with configuration file.");
+ }
+ this.center = CgroupCenter.getInstance();
+ if (this.center == null) {
+ throw new RuntimeException("Cgroup error, please check
/proc/cgroups");
+ }
+ this.prepareSubSystem(this.conf);
+ }
+
+ /**
+ * User cfs_period & cfs_quota to control the upper limit use of cpu
core e.g. If making a process to fully use two cpu cores, set cfs_period_us to
+ * 100000 and set cfs_quota_us to 200000
+ */
+ private void setCpuUsageUpperLimit(CpuCore cpuCore, int
cpuCoreUpperLimit) throws IOException {
+
+ if (cpuCoreUpperLimit == -1) {
+ // No control of cpu usage
+ cpuCore.setCpuCfsQuotaUs(cpuCoreUpperLimit);
+ } else {
+ cpuCore.setCpuCfsPeriodUs(100000);
+ cpuCore.setCpuCfsQuotaUs(cpuCoreUpperLimit * 1000);
+ }
+ }
+
+ public String startNewWorker(String workerId, Map resourcesMap) throws
SecurityException {
+ Number cpuNum = (Number) resourcesMap.get("cpu");
+ Number totalMem = null;
+ if (resourcesMap.get("memory") != null) {
+ totalMem = (Number) resourcesMap.get("memory");
+ }
+
+ CgroupCommon workerGroup = new CgroupCommon(workerId, h,
this.rootCgroup);
+ this.center.create(workerGroup);
+
+ if (cpuNum != null) {
+ CpuCore cpuCore = (CpuCore)
workerGroup.getCores().get(SubSystemType.cpu);
+ try {
+ cpuCore.setCpuShares(cpuNum.intValue());
--- End diff --
I am OK for filing follow-up JIRA of hard CPU limit or CPU guarantee. But
we still need a full testing on cpu.share, whether the worker which come back
from idle to busy can retrieve enough cpu resource again instead of starving,
when the other worker has occupied all cpu resources. Actually, we has met this
kind problem on online jobs. That is reason why we choose a hard limit finally.
> Evalute/Port JStorm cgroup support
> ----------------------------------
>
> Key: STORM-1336
> URL: https://issues.apache.org/jira/browse/STORM-1336
> Project: Apache Storm
> Issue Type: New Feature
> Components: storm-core
> Reporter: Robert Joseph Evans
> Assignee: Boyang Jerry Peng
> Labels: jstorm-merger
>
> Supports controlling the upper limit of CPU core usage for a worker using
> cgroups
> Sounds like a good start, will be nice to integrate it with RAS requests too.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
