[dpdk-dev] [RFC] eal: add cgroup-aware resource self discovery
Hi Neil, On 1/26/2016 10:19 PM, Neil Horman wrote: > On Tue, Jan 26, 2016 at 10:22:18AM +0800, Tan, Jianfeng wrote: >> Hi Neil, >> >> On 1/25/2016 9:46 PM, Neil Horman wrote: >>> On Mon, Jan 25, 2016 at 02:49:53AM +0800, Jianfeng Tan wrote: >> ... -- 2.1.4 >>> This doesn't make a whole lot of sense, for several reasons: >>> >>> 1) Applications, as a general rule shouldn't be interrogating the cgroups >>> interface at all. >> The main reason to do this in DPDK is that DPDK obtains resource information >> from sysfs and proc, which are not well containerized so far. And DPDK >> pre-allocates resource instead of on-demand gradual allocating. >> > Not disagreeing with this, just suggesting that: > > 1) Interrogating cgroups really isn't the best way to collect that information > 2) Pre-allocating those resources isn't particularly wise without some > mechanism > to reallocate it, as resource constraints can change (consider your cpuset > getting rewritten) In the case of reallocate, For cpuset, DPDK panics in the initialization if set_affinity fails, but after that, cpuset rewritten will not bring any problem I believe. For memory, a running application uses 2G hugepages, then admin decreases hugetlb cgroup into 1G, the application will not get killed, unless it tries to access more hugepages (I'll double check this). So another way to address this problem is to add an option that DPDK tries best to allocate those resources, and if fails, it just posts a warning and uses those allocated resources, instead of panic. What do you think? > >>> 2) Cgroups aren't the only way in which a cpuset or memoryset can be >>> restricted >>> (the isolcpus command line argument, or a taskset on a parent process for >>> instance, but there are several others). >> Yes, I agree. To enable that, I'd like design the new API for resource self >> discovery in a flexible way. A parameter "type" is used to specify the >> solution to discovery way. In addition, I'm considering to add a callback >> function pointer so that users can write their own resource discovery >> functions. >> > Why? You don't need an API for this, or if you really want one, it can be > very > generic if you use POSIX apis to gather the information. What you have here > is > going to be very linux specific, and will need reimplementing for BSD or other > operating systems. To use the cpuset example, instead of reading and parsing > the mask files in the cgroup filesystem module to find your task and > corresponding mask, just call sched_setaffinity with an all f's mask, then > call > sched_getaffinity. The returned mask will be all the cpus your process is > allowed to execute on, taking into account every limiting filter the system > you > are running on offers. Yes, it makes sense on cpu's side. > > There are simmilar OS level POSIX apis for most resources out there. You > really > don't need to dig through cgroups just to learn what some of those reources > are. > >>> Instead of trying to figure out what cpuset is valid for your process by >>> interrogating the cgroups heirarchy, instead you should follow the >>> proscribed >>> method of calling sched_getaffinity after calling sched_setaffinity. That >>> will >>> give you the canonical cpuset that you are executing on, taking all cpuset >>> filters into account (including cgroups and any other restrictions). Its >>> far >>> simpler as well, as it doesn't require a ton of file/string processing. >> Yes, this way is much better for cpuset discovery. But is there such a >> syscall for hugepages? >> > In what capacity? Interrogating how many hugepages you have, or to what node > they are affined to? Capacity would require reading the requisite proc file, > as > theres no posix api for this resource. Node affinity can be implied by > setting > the numa policy of the dpdk and then writing to /proc/nr_hugepages, as the > kernel will attempt to distribute hugepages evenly among the tasks' numa > policy > configuration. For memory affinity, I believe the existing way of reading /proc/self/pagemap already handle the problem. What I was asking is how much memory (or hugepages in Linux's case) can be used. By the way, what is /proc/nr_hugepages? > > That said, I would advise that you strongly consider not exporting hugepages > as > a resource, as: > > a) Applications generally don't need to know that they are using hugepages, > and > so they dont need to know where said hugepages live, they just allocate memory > via your allocation api and you give them something appropriate But the allocation api provider, DPDK library, needs to know if it's using hugepages or not. > b) Hugepages are a resource that are very specific to Linux, and to X86 Linux > at > that. Some OS implement simmilar resources, but they may have very different > semantics. And other Arches may or may not implement various forms of > compound > paging at all. As the DPDK expands to
[dpdk-dev] [RFC] eal: add cgroup-aware resource self discovery
On Wed, Jan 27, 2016 at 08:02:27PM +0800, Tan, Jianfeng wrote: > Hi Neil, > > On 1/26/2016 10:19 PM, Neil Horman wrote: > >On Tue, Jan 26, 2016 at 10:22:18AM +0800, Tan, Jianfeng wrote: > >>Hi Neil, > >> > >>On 1/25/2016 9:46 PM, Neil Horman wrote: > >>>On Mon, Jan 25, 2016 at 02:49:53AM +0800, Jianfeng Tan wrote: > >>... > -- > 2.1.4 > > > >>>This doesn't make a whole lot of sense, for several reasons: > >>> > >>>1) Applications, as a general rule shouldn't be interrogating the cgroups > >>>interface at all. > >>The main reason to do this in DPDK is that DPDK obtains resource information > >>from sysfs and proc, which are not well containerized so far. And DPDK > >>pre-allocates resource instead of on-demand gradual allocating. > >> > >Not disagreeing with this, just suggesting that: > > > >1) Interrogating cgroups really isn't the best way to collect that > >information > >2) Pre-allocating those resources isn't particularly wise without some > >mechanism > >to reallocate it, as resource constraints can change (consider your cpuset > >getting rewritten) > > In the case of reallocate, > For cpuset, DPDK panics in the initialization if set_affinity fails, but > after that, cpuset rewritten will not bring any problem I believe. Yes, that seems reasonable, but I think you need to update rte_thread_set_affinity to not assume that success in pthread_setaffinity_np means that all cpus in the provided mask are available. That is to say, cpusetp is subsequently stored in lore information after the set, but may not reflect the actual working set of processors, you should follow a successful set with a call to pthread_getaffinity_np to retrieve the actual working cpuset As for subsequent changes to the cpuset, I'm not sure how you want to handle that. I would think that you might want to run a check periodically or alow for a SIGHUP or some other signal to trigger a rescan of your working cpuset so as to keep the application in sync with the system. > For memory, a running application uses 2G hugepages, then admin decreases > hugetlb cgroup into 1G, the application will not get killed, unless it tries > to access more hugepages (I'll double check this). > No, the semantics should be identical to malloc/mmap (if you use the alloc_hugepages api or the mmap api). You should get a NULL return or other no fatal indicator if you allocate more than is available. > So another way to address this problem is to add an option that DPDK tries > best to allocate those resources, and if fails, it just posts a warning and > uses those allocated resources, instead of panic. What do you think? > Yes, that makes sense > > > >>>2) Cgroups aren't the only way in which a cpuset or memoryset can be > >>>restricted > >>>(the isolcpus command line argument, or a taskset on a parent process for > >>>instance, but there are several others). > >>Yes, I agree. To enable that, I'd like design the new API for resource self > >>discovery in a flexible way. A parameter "type" is used to specify the > >>solution to discovery way. In addition, I'm considering to add a callback > >>function pointer so that users can write their own resource discovery > >>functions. > >> > >Why? You don't need an API for this, or if you really want one, it can be > >very > >generic if you use POSIX apis to gather the information. What you have here > >is > >going to be very linux specific, and will need reimplementing for BSD or > >other > >operating systems. To use the cpuset example, instead of reading and parsing > >the mask files in the cgroup filesystem module to find your task and > >corresponding mask, just call sched_setaffinity with an all f's mask, then > >call > >sched_getaffinity. The returned mask will be all the cpus your process is > >allowed to execute on, taking into account every limiting filter the system > >you > >are running on offers. > > Yes, it makes sense on cpu's side. > > > > >There are simmilar OS level POSIX apis for most resources out there. You > >really > >don't need to dig through cgroups just to learn what some of those reources > >are. > > > >>>Instead of trying to figure out what cpuset is valid for your process by > >>>interrogating the cgroups heirarchy, instead you should follow the > >>>proscribed > >>>method of calling sched_getaffinity after calling sched_setaffinity. That > >>>will > >>>give you the canonical cpuset that you are executing on, taking all cpuset > >>>filters into account (including cgroups and any other restrictions). Its > >>>far > >>>simpler as well, as it doesn't require a ton of file/string processing. > >>Yes, this way is much better for cpuset discovery. But is there such a > >>syscall for hugepages? > >> > >In what capacity? Interrogating how many hugepages you have, or to what node > >they are affined to? Capacity would require reading the requisite proc > >file, as > >theres no posix api for this resource. Node affinity can be implied by >
[dpdk-dev] [RFC] eal: add cgroup-aware resource self discovery
Hi Neil, On 1/25/2016 9:46 PM, Neil Horman wrote: > On Mon, Jan 25, 2016 at 02:49:53AM +0800, Jianfeng Tan wrote: ... >> -- >> 2.1.4 >> >> > > This doesn't make a whole lot of sense, for several reasons: > > 1) Applications, as a general rule shouldn't be interrogating the cgroups > interface at all. The main reason to do this in DPDK is that DPDK obtains resource information from sysfs and proc, which are not well containerized so far. And DPDK pre-allocates resource instead of on-demand gradual allocating. > > 2) Cgroups aren't the only way in which a cpuset or memoryset can be > restricted > (the isolcpus command line argument, or a taskset on a parent process for > instance, but there are several others). Yes, I agree. To enable that, I'd like design the new API for resource self discovery in a flexible way. A parameter "type" is used to specify the solution to discovery way. In addition, I'm considering to add a callback function pointer so that users can write their own resource discovery functions. > > Instead of trying to figure out what cpuset is valid for your process by > interrogating the cgroups heirarchy, instead you should follow the proscribed > method of calling sched_getaffinity after calling sched_setaffinity. That > will > give you the canonical cpuset that you are executing on, taking all cpuset > filters into account (including cgroups and any other restrictions). Its far > simpler as well, as it doesn't require a ton of file/string processing. Yes, this way is much better for cpuset discovery. But is there such a syscall for hugepages? Thanks, Jianfeng > > Neil >
[dpdk-dev] [RFC] eal: add cgroup-aware resource self discovery
On Tue, Jan 26, 2016 at 10:22:18AM +0800, Tan, Jianfeng wrote: > > Hi Neil, > > On 1/25/2016 9:46 PM, Neil Horman wrote: > >On Mon, Jan 25, 2016 at 02:49:53AM +0800, Jianfeng Tan wrote: > ... > >>-- > >>2.1.4 > >> > >> > > > >This doesn't make a whole lot of sense, for several reasons: > > > >1) Applications, as a general rule shouldn't be interrogating the cgroups > >interface at all. > > The main reason to do this in DPDK is that DPDK obtains resource information > from sysfs and proc, which are not well containerized so far. And DPDK > pre-allocates resource instead of on-demand gradual allocating. > Not disagreeing with this, just suggesting that: 1) Interrogating cgroups really isn't the best way to collect that information 2) Pre-allocating those resources isn't particularly wise without some mechanism to reallocate it, as resource constraints can change (consider your cpuset getting rewritten) > > > >2) Cgroups aren't the only way in which a cpuset or memoryset can be > >restricted > >(the isolcpus command line argument, or a taskset on a parent process for > >instance, but there are several others). > > Yes, I agree. To enable that, I'd like design the new API for resource self > discovery in a flexible way. A parameter "type" is used to specify the > solution to discovery way. In addition, I'm considering to add a callback > function pointer so that users can write their own resource discovery > functions. > Why? You don't need an API for this, or if you really want one, it can be very generic if you use POSIX apis to gather the information. What you have here is going to be very linux specific, and will need reimplementing for BSD or other operating systems. To use the cpuset example, instead of reading and parsing the mask files in the cgroup filesystem module to find your task and corresponding mask, just call sched_setaffinity with an all f's mask, then call sched_getaffinity. The returned mask will be all the cpus your process is allowed to execute on, taking into account every limiting filter the system you are running on offers. There are simmilar OS level POSIX apis for most resources out there. You really don't need to dig through cgroups just to learn what some of those reources are. > > > >Instead of trying to figure out what cpuset is valid for your process by > >interrogating the cgroups heirarchy, instead you should follow the proscribed > >method of calling sched_getaffinity after calling sched_setaffinity. That > >will > >give you the canonical cpuset that you are executing on, taking all cpuset > >filters into account (including cgroups and any other restrictions). Its far > >simpler as well, as it doesn't require a ton of file/string processing. > > Yes, this way is much better for cpuset discovery. But is there such a > syscall for hugepages? > In what capacity? Interrogating how many hugepages you have, or to what node they are affined to? Capacity would require reading the requisite proc file, as theres no posix api for this resource. Node affinity can be implied by setting the numa policy of the dpdk and then writing to /proc/nr_hugepages, as the kernel will attempt to distribute hugepages evenly among the tasks' numa policy configuration. That said, I would advise that you strongly consider not exporting hugepages as a resource, as: a) Applications generally don't need to know that they are using hugepages, and so they dont need to know where said hugepages live, they just allocate memory via your allocation api and you give them something appropriate b) Hugepages are a resource that are very specific to Linux, and to X86 Linux at that. Some OS implement simmilar resources, but they may have very different semantics. And other Arches may or may not implement various forms of compound paging at all. As the DPDK expands to support more OS'es and arches, it would be nice to ensure that the programming surfaces that you expose have a more broad level of support. Neil > Thanks, > Jianfeng > > > > >Neil > > > >
[dpdk-dev] [RFC] eal: add cgroup-aware resource self discovery
On Mon, Jan 25, 2016 at 02:49:53AM +0800, Jianfeng Tan wrote:
> Current issue: DPDK is not that friendly to container environment usage.
> It's because that it pre-alloc resource like cores and hugepages from cmd
> line options. So for a DPDK application, it's necessary to check how much
> resource is allocated to a container and then use that as an reference.
>
> To address that, this patch introduces two APIs:
>a. rte_eal_res_self_discovery, to query how much resource can be used.
>b. rte_eal_res_self_discovery_apply, to apply self-discovered resource
> into DPDK.
>
> Currently only Linux CGroup is added, similarly, we can add BSD jail as
> well in the future. And even in Linux, there could be other way to query
> and apply resources, like through a centralized daemon.
>
> Known issue: current way to read individual attributes of cgroups directly
> instead of via systemd's API is not a long-term solution. Please refer to
> http://www.freedesktop.org/wiki/Software/systemd/ControlGroupInterface/
> for more information.
>
> Test example:
> a. cgcreate -g cpuset,hugetlb:/test-subgroup
> b. cgset -r cpuset.cpus=2-3 test-subgroup
> c. cgset -r hugetlb.1GB.limit_in_bytes=2147483648 test-subgroup
> d. cgexec -g cpuset,hugetlb:test-subgroup \
> ./examples/l2fwd/build/l2fwd --self-discovery=cgroup -n 4 -- -p 3
>
> Signed-off-by: Jianfeng Tan
> ---
> lib/librte_eal/common/eal_common_options.c | 39
> lib/librte_eal/common/eal_internal_cfg.h| 1 +
> lib/librte_eal/common/eal_options.h | 2 +
> lib/librte_eal/common/include/rte_eal.h | 34 +++
> lib/librte_eal/linuxapp/eal/Makefile| 1 +
> lib/librte_eal/linuxapp/eal/eal_cgroup.c| 294
>
> lib/librte_eal/linuxapp/eal/eal_hugepage_info.c | 5 +
> 7 files changed, 376 insertions(+)
> create mode 100644 lib/librte_eal/linuxapp/eal/eal_cgroup.c
>
> diff --git a/lib/librte_eal/common/eal_common_options.c
> b/lib/librte_eal/common/eal_common_options.c
> index 29942ea..7235473 100644
> --- a/lib/librte_eal/common/eal_common_options.c
> +++ b/lib/librte_eal/common/eal_common_options.c
> @@ -95,6 +95,7 @@ eal_long_options[] = {
> {OPT_VFIO_INTR, 1, NULL, OPT_VFIO_INTR_NUM},
> {OPT_VMWARE_TSC_MAP,0, NULL, OPT_VMWARE_TSC_MAP_NUM },
> {OPT_XEN_DOM0, 0, NULL, OPT_XEN_DOM0_NUM },
> + {OPT_SELF_DISCOVERY,1, NULL, OPT_SELF_DISCOVERY_NUM },
> {0, 0, NULL, 0}
> };
>
> @@ -128,6 +129,7 @@ eal_reset_internal_config(struct internal_config
> *internal_cfg)
> internal_cfg->force_nchannel = 0;
> internal_cfg->hugefile_prefix = HUGEFILE_PREFIX_DEFAULT;
> internal_cfg->hugepage_dir = NULL;
> + internal_cfg->self_discovery = NULL;
> internal_cfg->force_sockets = 0;
> /* zero out the NUMA config */
> for (i = 0; i < RTE_MAX_NUMA_NODES; i++)
> @@ -755,6 +757,24 @@ eal_parse_proc_type(const char *arg)
> }
>
> int
> +__attribute__((weak))
> +rte_eal_res_self_discovery(const char *type __rte_unused,
> +char **p_corelist __rte_unused,
> +uint64_t *p_memory __rte_unused)
> +{
> + return -1;
> +}
> +
> +int
> +__attribute__((weak))
> +rte_eal_res_self_discovery_apply(const char *type __rte_unused,
> + int enable_core __rte_unused,
> + int enable_mem __rte_unused)
> +{
> + return -1;
> +}
> +
> +int
> eal_parse_common_option(int opt, const char *optarg,
> struct internal_config *conf)
> {
> @@ -897,6 +917,25 @@ eal_parse_common_option(int opt, const char *optarg,
> }
> break;
>
> + case OPT_SELF_DISCOVERY_NUM: {
> + char *corelist;
> +
> + if (rte_eal_res_self_discovery(optarg, , NULL) < 0) {
> + RTE_LOG(ERR, EAL, "invalid parameter for --"
> + OPT_SELF_DISCOVERY "\n");
> + return -1;
> + }
> +
> + if (eal_parse_corelist(corelist) < 0) {
> + RTE_LOG(ERR, EAL, "invalid core list\n");
> + return -1;
> + }
> + /* Save it here for memory limit */
> + internal_config.self_discovery = strdup(optarg);
> +
> + break;
> + }
> +
> /* don't know what to do, leave this to caller */
> default:
> return 1;
> diff --git a/lib/librte_eal/common/eal_internal_cfg.h
> b/lib/librte_eal/common/eal_internal_cfg.h
> index 5f1367e..f3c8e31 100644
> --- a/lib/librte_eal/common/eal_internal_cfg.h
> +++ b/lib/librte_eal/common/eal_internal_cfg.h
> @@ -83,6 +83,7 @@ struct internal_config {
> volatile enum rte_intr_mode vfio_intr_mode;
> const char *hugefile_prefix; /**< the base filename of hugetlbfs
> files
[dpdk-dev] [RFC] eal: add cgroup-aware resource self discovery
Current issue: DPDK is not that friendly to container environment usage.
It's because that it pre-alloc resource like cores and hugepages from cmd
line options. So for a DPDK application, it's necessary to check how much
resource is allocated to a container and then use that as an reference.
To address that, this patch introduces two APIs:
a. rte_eal_res_self_discovery, to query how much resource can be used.
b. rte_eal_res_self_discovery_apply, to apply self-discovered resource
into DPDK.
Currently only Linux CGroup is added, similarly, we can add BSD jail as
well in the future. And even in Linux, there could be other way to query
and apply resources, like through a centralized daemon.
Known issue: current way to read individual attributes of cgroups directly
instead of via systemd's API is not a long-term solution. Please refer to
http://www.freedesktop.org/wiki/Software/systemd/ControlGroupInterface/
for more information.
Test example:
a. cgcreate -g cpuset,hugetlb:/test-subgroup
b. cgset -r cpuset.cpus=2-3 test-subgroup
c. cgset -r hugetlb.1GB.limit_in_bytes=2147483648 test-subgroup
d. cgexec -g cpuset,hugetlb:test-subgroup \
./examples/l2fwd/build/l2fwd --self-discovery=cgroup -n 4 -- -p 3
Signed-off-by: Jianfeng Tan
---
lib/librte_eal/common/eal_common_options.c | 39
lib/librte_eal/common/eal_internal_cfg.h| 1 +
lib/librte_eal/common/eal_options.h | 2 +
lib/librte_eal/common/include/rte_eal.h | 34 +++
lib/librte_eal/linuxapp/eal/Makefile| 1 +
lib/librte_eal/linuxapp/eal/eal_cgroup.c| 294
lib/librte_eal/linuxapp/eal/eal_hugepage_info.c | 5 +
7 files changed, 376 insertions(+)
create mode 100644 lib/librte_eal/linuxapp/eal/eal_cgroup.c
diff --git a/lib/librte_eal/common/eal_common_options.c
b/lib/librte_eal/common/eal_common_options.c
index 29942ea..7235473 100644
--- a/lib/librte_eal/common/eal_common_options.c
+++ b/lib/librte_eal/common/eal_common_options.c
@@ -95,6 +95,7 @@ eal_long_options[] = {
{OPT_VFIO_INTR, 1, NULL, OPT_VFIO_INTR_NUM},
{OPT_VMWARE_TSC_MAP,0, NULL, OPT_VMWARE_TSC_MAP_NUM },
{OPT_XEN_DOM0, 0, NULL, OPT_XEN_DOM0_NUM },
+ {OPT_SELF_DISCOVERY,1, NULL, OPT_SELF_DISCOVERY_NUM },
{0, 0, NULL, 0}
};
@@ -128,6 +129,7 @@ eal_reset_internal_config(struct internal_config
*internal_cfg)
internal_cfg->force_nchannel = 0;
internal_cfg->hugefile_prefix = HUGEFILE_PREFIX_DEFAULT;
internal_cfg->hugepage_dir = NULL;
+ internal_cfg->self_discovery = NULL;
internal_cfg->force_sockets = 0;
/* zero out the NUMA config */
for (i = 0; i < RTE_MAX_NUMA_NODES; i++)
@@ -755,6 +757,24 @@ eal_parse_proc_type(const char *arg)
}
int
+__attribute__((weak))
+rte_eal_res_self_discovery(const char *type __rte_unused,
+ char **p_corelist __rte_unused,
+ uint64_t *p_memory __rte_unused)
+{
+ return -1;
+}
+
+int
+__attribute__((weak))
+rte_eal_res_self_discovery_apply(const char *type __rte_unused,
+int enable_core __rte_unused,
+int enable_mem __rte_unused)
+{
+ return -1;
+}
+
+int
eal_parse_common_option(int opt, const char *optarg,
struct internal_config *conf)
{
@@ -897,6 +917,25 @@ eal_parse_common_option(int opt, const char *optarg,
}
break;
+ case OPT_SELF_DISCOVERY_NUM: {
+ char *corelist;
+
+ if (rte_eal_res_self_discovery(optarg, , NULL) < 0) {
+ RTE_LOG(ERR, EAL, "invalid parameter for --"
+ OPT_SELF_DISCOVERY "\n");
+ return -1;
+ }
+
+ if (eal_parse_corelist(corelist) < 0) {
+ RTE_LOG(ERR, EAL, "invalid core list\n");
+ return -1;
+ }
+ /* Save it here for memory limit */
+ internal_config.self_discovery = strdup(optarg);
+
+ break;
+ }
+
/* don't know what to do, leave this to caller */
default:
return 1;
diff --git a/lib/librte_eal/common/eal_internal_cfg.h
b/lib/librte_eal/common/eal_internal_cfg.h
index 5f1367e..f3c8e31 100644
--- a/lib/librte_eal/common/eal_internal_cfg.h
+++ b/lib/librte_eal/common/eal_internal_cfg.h
@@ -83,6 +83,7 @@ struct internal_config {
volatile enum rte_intr_mode vfio_intr_mode;
const char *hugefile_prefix; /**< the base filename of hugetlbfs
files */
const char *hugepage_dir; /**< specific hugetlbfs directory to
use */
+ const char *self_discovery; /**< specific type of self_discovery
*/
unsigned num_hugepage_sizes; /**<
