What i can see in http output is mem_rss_bytes is not coming on rhel7.
Here is the http output :
Output for Agent running on rhel7
[{"container\_id":"8062e683\-204c\-40c2\-87ae\-fcc2c3f71b85","executor\_id":"\*\*\*\*\*","executor\_name":"Command
Executor (Task: \*\*\*\*\*) (Command: sh \-c
'\\*\*\*\*\*\*...')","framework\_id":"edbffd6d\-b274\-4cb1\-b386\-2362ed2af517\-0000","source":"\*\*\*\*\*","statistics":{"cpus\_limit":1.1,"cpus\_system\_time\_secs":0.01,"cpus\_user\_time\_secs":0.03,"mem\_anon\_bytes":0,"mem\_cache\_bytes":0,"mem\_critical\_pressure\_counter":0,"mem\_file\_bytes":0,"mem\_limit\_bytes":1107296256,"mem\_low\_pressure\_counter":0,"mem\_mapped\_file\_bytes":0,"mem\_medium\_pressure\_counter":0,"mem\_rss\_bytes":0,"mem\_swap\_bytes":0,"mem\_total\_bytes":0,"mem\_unevictable\_bytes":0,"timestamp":1475668277.62915},"status":{"executor\_pid":14454}}]
Output for Agent running on Rhel 6
[{"container\_id":"359c0944\-c089\-4d43\-983e\-1f97134fe799","executor\_id":"\*\*\*\*\*","executor\_name":"Command
Executor (Task: \*\*\*\*\*) (Command: sh \-c
'\*\*\*\*\*\*...')","framework\_id":"edbffd6d\-b274\-4cb1\-b386\-2362ed2af517\-0001","source":"\*\*\*\*\*","statistics":{"cpus\_limit":8.1,"cpus\_system\_time\_secs":1.92,"cpus\_user\_time\_secs":6.93,"mem\_limit\_bytes":1107296256,"mem\_rss\_bytes":2329763840,"timestamp":1475670762.73402},"status":{"executor\_pid":31577}}]
Attach are UI screenshot :
Wa002.jpg is for rhel7 and other one is rhel6.
On 5 Oct 2016 4:55 p.m., "haosdent" <haosd...@gmail.com> wrote:
> Hi, @Srikant How about the result of http://${YOUR_AGENT_IP}:5051/containers?
> It is wired that you could saw
>
> ```
> Updated 'memory.limit_in_bytes' to xxx
> ```
>
> in log as you mentioned, but `limit_in_bytes` is still the initialize
> value as you show above.
>
> On Wed, Oct 5, 2016 at 2:04 PM, Srikant Kalani <
> srikant.blackr...@gmail.com> wrote:
>
>> Here are the values -
>> Memory.limit_in_bytes = 1107296256
>> Memory.soft_limit_in_bytes=1107296256
>> Memory.memsw.limit_in_bytes=9223372036854775807
>>
>> I have run the same task on mesos 1.0.1 running on rhel6 and UI then
>> shows task memory usage as 2.2G/1.0G where 2.2 is used and 1.0G is
>> allocated but since we don't have cgroups their so task are not getting
>> killed.
>>
>> On rhel7 UI is showing 0B/1.0G for task memory details.
>>
>> Any idea is this rhel7 fault or do I need to adjust some configurations ?
>> On 4 Oct 2016 21:33, "haosdent" <haosd...@gmail.com> wrote:
>>
>>> Hi, @Srikant
>>>
>>> Hi, @Srikant
>>>
>>> Usually, your task should be killed when over cgroup limit. Would you
>>> enter the `/sys/fs/cgroup/memory/mesos` folder in the agent?
>>> Then check the values in `${YOUR_CONTAINER_ID}/memory.limit_in_bytes`,
>>> `${YOUR_CONTAINER_ID}/memory.soft_limit_in_bytes` and
>>> `${YOUR_CONTAINER_ID}/memory.memsw.limit_in_bytes` and reply in this
>>> email.
>>>
>>> ${YOUR_CONTAINER_ID} is the container id of your task here, you could
>>> find it from the agent log. Or as you said, you only have this one task, so
>>> it should only have one directory under `/sys/fs/cgroup/memory/mesos`.
>>>
>>> Furthermore, would you show the result of
>>> http://${YOUR_AGENT_IP}:5051/containers?
>>> It contains some tasks statistics information as well.
>>>
>>> On Tue, Oct 4, 2016 at 9:00 PM, Srikant Kalani <
>>> srikant.blackr...@gmail.com> wrote:
>>>
>>>> We have upgraded linux from rhel6 to rhel7 and mesos from 0.27 to 1.0.1.
>>>> After upgrade we are not able to see memory used by task which was fine
>>>> in previous version. Due to this cgroups are not effective.
>>>>
>>>> Answers to your questions below :
>>>>
>>>> There is only 1 task running as a appserver which is consuming approx
>>>> 20G mem but this info is not coming in Mesos UI.
>>>> Swaps are enabled in agent start command.
>>>> These flags are used in agent - cgroups_limits_swap=true
>>>> --isolation=cgroups/cpu,cgroups/mem --cgroups_hierachy=/sys/fs/c group
>>>> In agent logs I can see updated memory limit to 33MB for container.
>>>>
>>>> Web UI shows the total memory allocated to framework but it is not
>>>> showing memory used by task.It always shows 0B/33MB.
>>>>
>>>> Not sure if this is rhel7 issue or mesos 1.0.1.
>>>>
>>>> Any suggestions ?
>>>> On 26 Sep 2016 21:55, "haosdent" <haosd...@gmail.com> wrote:
>>>>
>>>>> Hi, @Srikant May you elaborate
>>>>>
>>>>> >We have verified using top command that framework was using 2gB
>>>>> memory while allocated was just 50 mb.
>>>>>
>>>>> * How many running tasks in your framework?
>>>>> * Do you enable or disable swap in the agents?
>>>>> * What's the flags that you launch agents?
>>>>> * Have you saw some thing like `Updated 'memory.limit_in_bytes' to `
>>>>> in the log of agent?
>>>>>
>>>>> On Tue, Sep 27, 2016 at 12:14 AM, Srikant Kalani <
>>>>> srikant.blackr...@gmail.com> wrote:
>>>>>
>>>>>> Hi Greg ,
>>>>>>
>>>>>> Previously we were running Mesos 0.27 on Rhel6 and since we already
>>>>>> have one c group hierarchy for cpu and memory for our production
>>>>>> processes
>>>>>> I'd we were not able to merge two c groups hierarchy on rhel6. Slave
>>>>>> process was not coming up.
>>>>>> Now we have moved to Rhel7 and both mesos master and slave are
>>>>>> running on rhel7 with c group implemented.But we are seeing that mesos UI
>>>>>> not showing the actual memory used by framework.
>>>>>>
>>>>>> Any idea why framework usage of cpu and memory is not coming in UI.
>>>>>> Due to this OS is still not killing the task which are consuming more
>>>>>> memory than the allocated one.
>>>>>> We have verified using top command that framework was using 2gB
>>>>>> memory while allocated was just 50 mb.
>>>>>>
>>>>>> Please suggest.
>>>>>> On 8 Sep 2016 01:53, "Greg Mann" <g...@mesosphere.io> wrote:
>>>>>>
>>>>>>> Hi Srikant,
>>>>>>> Without using cgroups, it won't be possible to enforce isolation of
>>>>>>> cpu/memory on a Linux agent. Could you elaborate a bit on why you aren't
>>>>>>> able to use cgroups currently? Have you tested the existing Mesos cgroup
>>>>>>> isolators in your system?
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Greg
>>>>>>>
>>>>>>> On Tue, Sep 6, 2016 at 9:24 PM, Srikant Kalani <
>>>>>>> srikant.blackr...@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi Guys,
>>>>>>>>
>>>>>>>> We are running Mesos cluster in our development environment. We are
>>>>>>>> seeing the cases where framework uses more amount of resources like
>>>>>>>> cpu and
>>>>>>>> memory then the initial requested resources. When any new framework is
>>>>>>>> registered Mesos calculates the resources on the basis of already
>>>>>>>> offered
>>>>>>>> resources to first framework and it doesn't consider actual resources
>>>>>>>> utilised by previous framework.
>>>>>>>> This is resulting in incorrect calculation of resources.
>>>>>>>> Mesos website says that we should Implement c groups but it is not
>>>>>>>> possible in our case as we have already implemented c groups in other
>>>>>>>> projects and due to Linux restrictions we can't merge two c groups
>>>>>>>> hierarchy.
>>>>>>>>
>>>>>>>> Any idea how we can implement resource Isolation in Mesos ?
>>>>>>>>
>>>>>>>> We are using Mesos 0.27.1
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>> Srikant Kalani
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Best Regards,
>>>>> Haosdent Huang
>>>>>
>>>>
>>>
>>>
>>> --
>>> Best Regards,
>>> Haosdent Huang
>>>
>>
>
>
> --
> Best Regards,
> Haosdent Huang
>
