Lost memory

[Andrea Tomassetti]

Hi everyone,
I've been facing this problem for the past few weeks and I'm out of
ideas of how to analyze it, maybe someone here has some thoughts on
how to address it.
Basically, if I compare the output of `free` and the total RSS
reported by `ps` there are 200 GiB of difference:

$ free
  totalusedfree  shared  buff/cache   available
Mem:  521933440   371541084192460284100   131146328   117933472
Swap:  2097151615208556 5762960

$ ps -eo rss | awk '{ rss += $0 } END { print rss}'
176765072

total used: 386749640
total RSS:  176765072
difference: 209984568 (~200.3 GiB)

I have some programs running that make intense use of networking with
thousands of sockets opened, so I checked sockstat but with no luck:

$ cat /proc/net/sockstat
sockets: used 90319
TCP: inuse 40798 orphan 0 tw 56 alloc 89689 mem 558949 # *4 = 2235796
KiB = ~2.13 GiB
UDP: inuse 31 mem 2
UDPLITE: inuse 0
RAW: inuse 0
FRAG: inuse 0 memory 0

Even /proc/meminfo is quite useless [1].

I started to kill one process at a time till I found the culprit: a
Java process with high network usage. But inspecting this process'
memory consumption didn't give me any conclusion because its memory
consumption appears to be low:

$ sudo pmap -x 268169 | head -n2 | tail -n1; sudo pmap -x 268169 | tail -n 1
Address   Kbytes RSS   Dirty Mode  Mapping
total kB 171403084 23775460 23697900
~ 22.7 GiB

But this is the process that is helding the resources because, once
killed, the memory difference/leak disappears.

How is it possible that no utility or kernel information is capable of
outlining this memory leak? What can I possibly do to investigate it
further? For sure there's a problem releasing resources in the Java
code, but I would like to see it at a kernel level.

Any help is very appreciated.

Thank you in advance,
Andrea


[1] $ cat /proc/meminfo
MemTotal:   521933440 kB
MemFree:20600964 kB
MemAvailable:   118570008 kB
Buffers:   18092 kB
Cached: 125086816 kB
SwapCached:  1792868 kB
Active: 115786784 kB
Inactive:   187218964 kB
Active(anon):   7910 kB
Inactive(anon): 98768932 kB
Active(file):   36642344 kB
Inactive(file): 88450032 kB
Unevictable:   20556 kB
Mlocked:   20556 kB
SwapTotal:  20971516 kB
SwapFree:5762448 kB
Dirty:114392 kB
Writeback: 0 kB
AnonPages:  176129992 kB
Mapped:   315908 kB
Shmem:  4100 kB
KReclaimable:5322088 kB
Slab:   13095480 kB
SReclaimable:5322088 kB
SUnreclaim:  7773392 kB
KernelStack:  135600 kB
PageTables:   496504 kB
NFS_Unstable:  0 kB
Bounce:0 kB
WritebackTmp:  0 kB
CommitLimit:542904956 kB
Committed_AS:   210582988 kB
VmallocTotal:   133143592960 kB
VmallocUsed:  256200 kB
VmallocChunk:  0 kB
Percpu:62976 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
ShmemHugePages:0 kB
ShmemPmdMapped:0 kB
FileHugePages: 0 kB
FilePmdMapped: 0 kB
CmaTotal:  32768 kB
CmaFree:   0 kB
HugePages_Total:   0
HugePages_Free:0
HugePages_Rsvd:0
HugePages_Surp:0
Hugepagesize:   2048 kB
Hugetlb:   0 kB

___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: eBPF Verifier's Design Principles

[Andrea Tomassetti]

Hi Patrick,
there's a lot of work related to security and exploiting the eBPF
verifier out there.

I'm not an expert myself, but the principles you exposed seem right.

Here there's a nice and recent article about eBPF fuzzing:
https://security.googleblog.com/2023/05/introducing-new-way-to-buzz-for-ebpf.html

Best,
Andrea

On Thu, Jun 1, 2023 at 9:48 AM Patrick ZHANG  wrote:
>
> Hi there,
> I am not sure I am doing this in the right way.
> I writing to seek your guidance and expertise regarding on kernel security.
> Specifically, my focus has been on the eBPF environment and its verifier,
> which plays a crucial role in ensuring kernel safety.
>
> While conducting my research, I discovered that there are no official
> documents available that outline the principles of the verifier.
> Consequently, I have endeavored to deduce the kernel safety principles
> ensured by the verifier by studying its code. Based on my analysis, I
> have identified the following principles:
> 1. Control Flow Integrity: It came to my attention that the verifier
> rejects BPF programs containing indirect call instructions (callx). By
> disallowing indirect control flow, the verifier ensures the identification
> of all branch targets, thereby upholding control flow integrity (CFI).
>
> 2. Memory Safety: BPF programs are restricted to accessing predefined
> data, including the stack, maps, and the context. The verifier effectively
> prevents out-of-bounds access and modifies memory access to thwart
> Spectre attacks, thus promoting memory safety.
>
> 3. Prevention of Information Leak: Through a comprehensive analysis of
> all register types, the verifier prohibits the writing of pointer-type
> registers
> into maps. This preventive measure restricts user processes from reading
> kernel pointers, thereby mitigating the risk of information leaks.
>
> 4. Prevention of Denial-of-Service (DoS): The verifier guarantees the
> absence of deadlocks and crashes (e.g., division by zero), while also
> imposing a limit on the execution time of BPF programs (up to 1M
> instructions). These measures effectively prevent DoS attacks.
>
> I would greatly appreciate it if someone could review the aforementioned
> principles to ensure their accuracy and comprehensiveness. If there are
> any additional principles that I may have overlooked, I would be grateful
> for your insights on this matter.
>
> Furthermore, I would like to explore why the static verifier was chosen as
> the means to guarantee kernel security when there are other sandboxing
> techniques that can achieve kernel safety by careful design.
>
> The possible reasons I can think of are that verified (and jitted) BPF
> programs can run at nearly native speed, and that eBPF inherits the verifier
> from cBPF for compatibility reasons.
>
> Thank you very much for your time and attention. I appreciate the  feedback
> and insights.
>
> Best,
> Patrick
>
> ___
> Kernelnewbies mailing list
> Kernelnewbies@kernelnewbies.org
> https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: What to do when your patch gets ignored

[Andrea Tomassetti]

From: Andrea Tomassetti 

Hi all,
thank you very much to have pointed out the footer problem.

It took sometime to my IT dpt to figure out a solution, but
now we have it.

I will send the patch again, as a brand new one. I hope to
receive some replies.

Kind regards,
Andrea

___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: What to do when your patch gets ignored

[Andrea Tomassetti]

Hi Richard,
Thank you very much for your reply, I really appreciate it.

On Fri, Jun 10, 2022 at 2:18 PM Richard  wrote:
>
> Hi Andrea,
>
> Could you include a link to some mailing list archive so people her can
> look at the patch/code without having to google the right version? That
> makes answering a bit easier and people are lazy :)
>
Sure thing! I just didn't want my first message to be marked as spam
because of links. I will add a link to each of the mail in the list I
provided in my previous email.

> For now I can only speculate and from reading your mail two possible
> reasons come to my mind.
>
> 1. Your patch seems pretty big (in its effect/implications) and kernel
> maintainers are usually conservative, caring a lot more for stability
> and reliability than the typical github project. So changing something
> big as your first contribution when you have no reputation makes it more
> difficult and less likely to get applied/merged. Maybe chose something
> smaller, my first commits were understand parts of the tcp code in the
> kernel and writing doc for them
>
> 2. As you said the bcache mailing list is pretty inactive. Maybe the
> project is (semi-) dead? which might mean the maintainer(s) might have
> very little time/motivation to continue it, which would include
> reviewing and working with patches. I might be wrong here I don't know
> the current status of bcachefs but maybe your "error" here was chosing
> bchachefs to contribute and not something in the mainline kernel.
>
Wait, I think there's a small misunderstanding here: I contributed to
the bcache module that it's in the mainline kernel, not to the
bcachefs project. But, apart from this, yes this is something that
crossed my mind: maybe the maintainer is just accepting small bug
fixes (and he's actually reviewing and merging them when they arrive,
he's just ignoring my emails).

At this point, I really hope to have made some _huge_ mistake and that
someone can help me point it out and fix it, so I can maybe stop
speculating on why I'm not getting any answer.

Thank you,
Andrea

> Hope this helps. I might or might not write something more concrete on
> the code if you include a link in your answer
>
> -- Richard
>
>
> On 09/06/2022 15:39, Andrea Tomassetti wrote:
> > I'm writing here as a last resort in the hope that someone can,
> > kindly, help me understand what I'm doing wrong and why I'm being
> > ignored. Let's start from the beginning:
> >
> > On March 8th, I sent my very first patch "[PATCH] bcache: Use bcache
> > without formatting existing device" to the linux-bcache mailing list.
https://marc.info/?l=linux-bcache&m=164675141727884&w=3

> > I was very excited to finally contribute to the Linux kernel. After
> > just one day I received very positive feedback (unfortunately not from
> > the maintainer) and I followed up the same day; fulfilling the
> > requests.
https://marc.info/?l=linux-bcache&m=164682477802826&w=3

> >
> > On March 10th, I submitted the third version of my patch with fixes of
> > some warnings reported by the "kernel test robot". No replies from any
> > of the maintainers.
https://marc.info/?l=linux-bcache&m=164691268020582&w=3

> >
> > On March 22nd, I sent a kind ping: I got no replies from any of the 
> > maintainers.
https://marc.info/?l=linux-bcache&m=164795217223238&w=3

> >
> > On March 28th, I sent the 4th version of the patch.
https://marc.info/?l=linux-bcache&m=164846757707748&w=3

> >
> > On April 21st, I sent a kind ping replying to my last patch message,
> > asking for *any* feedback: I still haven't received any reply.
https://marc.info/?l=linux-bcache&m=165054833923209&w=3

> >
> > I fully understand that it's almost certainly my fault. Should I have
> > sent a RFC instead of sending a PATCH? I really don't know and the
> > worst part is that I will never know unless someone responds to me.
> > I'm willing to learn and ready to take accountability for my mistakes
> > but being ignored prevents me from doing so.
> >
> > The linux-bcache mailing list has zero-to-little activity, so I don't
> > think that my multiple emails got lost and on the other hand it's very
> > difficult to help the maintainer with other patch requests, because
> > there are so few of them (I read this could be a way to encourage the
> > maintainer to respond to your other requests).
> >
> > Should I just give up?
> > Should I resend my PATCH as RFC and hope for the best?
> >
> > I'm open to suggestions.
> >
> > Than

What to do when your patch gets ignored

[Andrea Tomassetti]

I'm writing here as a last resort in the hope that someone can,
kindly, help me understand what I'm doing wrong and why I'm being
ignored. Let's start from the beginning:

On March 8th, I sent my very first patch "[PATCH] bcache: Use bcache
without formatting existing device" to the linux-bcache mailing list.
I was very excited to finally contribute to the Linux kernel. After
just one day I received very positive feedback (unfortunately not from
the maintainer) and I followed up the same day; fulfilling the
requests.

On March 10th, I submitted the third version of my patch with fixes of
some warnings reported by the "kernel test robot". No replies from any
of the maintainers.

On March 22nd, I sent a kind ping: I got no replies from any of the maintainers.

On March 28th, I sent the 4th version of the patch.

On April 21st, I sent a kind ping replying to my last patch message,
asking for *any* feedback: I still haven't received any reply.

I fully understand that it's almost certainly my fault. Should I have
sent a RFC instead of sending a PATCH? I really don't know and the
worst part is that I will never know unless someone responds to me.
I'm willing to learn and ready to take accountability for my mistakes
but being ignored prevents me from doing so.

The linux-bcache mailing list has zero-to-little activity, so I don't
think that my multiple emails got lost and on the other hand it's very
difficult to help the maintainer with other patch requests, because
there are so few of them (I read this could be a way to encourage the
maintainer to respond to your other requests).

Should I just give up?
Should I resend my PATCH as RFC and hope for the best?

I'm open to suggestions.

Thank you very much in advance,
Andrea

-- 







The contents of this email are confidential. If the reader of this 
message is not the intended recipient, you are hereby notified that any 
dissemination, distribution or copying of this communication is strictly 
prohibited. If you have received this communication in error, please notify 
us immediately by replying to this message and deleting it from your 
computer. Thank you. Devo, Inc; a...@devo.com ; 
255 
Main St Suite 702, Cambridge MA USA 02142 
Calle Estébanez Calderón 3-5, 
5th Floor Madrid, Spain 28020


___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies