Re: Kernel memory

[Valdis . Kletnieks]

On Tue, 17 May 2016 12:41:44 +0530, Ronit Halder said:
> Hi,
>
> Where in the memory kernel is located in the boot time?

During which exact phase of the boot, and does it actually matter?  And
physical or virtual address?

(Hint:  If you're not the bootstrap that unpacks the compressed kernel
into memory, or the code that sets up the ASLR code, or are trying to work
around an issue with a hardware device that insists on taking certain
physical addresses for itself, you probably don't really care where the
kernel is)


pgpA8PmnyxbRA.pgp
Description: PGP signature
___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Kernel memory leak

[Anil Shashikumar Belur]

On Saturday 28 June 2014 12:29 PM, Santhosh Kumar wrote:
>
> Is there a way to trace the allocations of memory from different buckets of
> kmalloc ?
You could try out kmemleak -
https://www.kernel.org/doc/Documentation/kmemleak.txt

___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Kernel Memory management

[buyitian]

The answer from manty is totally wrong. 

I suggest that you put real question here.


在 2013年11月18日，15:17，"manty kuma"  写道：

Here is an interesting question(not mine) in SO related to Kernel memory 
management. Most of the points are my questions aswell. It needs ex[ert 
comments. could we try to answer questions posted there.

http://stackoverflow.com/questions/20041212/does-virtual-memory-area-struct-only-comes-into-picture-when-there-is-a-page-fau
___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Kernel Memory

[Vijay Chauhan]

Thanks everyone. I think i got enough information for further study.

Thanks,
Vijay

On Thu, Jun 21, 2012 at 7:10 PM, kishore sheik ahamed
 wrote:
> Hey Vijay
>
> I am a newbie too. Just sharing what I could go through.
>
> It is said that Kernel or atleast a part of kernel needs to be non paged for
> fast interrupt access etc as pinned memory
> Wiki says
>
> Pinned/Locked/Fixed pages
>
> Operating systems have memory areas that are pinned (never swapped to
> secondary storage). For example, interrupt mechanisms rely on an array of
> pointers to their handlers, such as I/O completion and page fault. If the
> pages containing these pointers or the code that they invoke were pageable,
> interrupt-handling would become far more complex and time-consuming,
> particularly in the case of page fault interrupts. Hence, some part of the
> page table structures is not pageable.
>
> Some pages may be pinned for short periods of time, others may be pinned for
> long periods of time, and still others may need to be permanently pinned.
> For example:
>
> The paging supervisor code and drivers for secondary storage devices on
> which pages reside must be permanently pinned, as otherwise paging wouldn't
> even work because the necessary code wouldn't be available.
> Timing-dependent components may be pinned to avoid variable paging delays.
> Data buffers that are accessed directly by peripheral devices that use
> direct memory access or I/O channels must reside in pinned pages while the
> I/O operation is in progress because such devices and the buses to which
> they are attached expect to find data buffers located at physical memory
> addresses; regardless of whether the bus has a memory management unit for
> I/O, transfers cannot be stopped if a page fault occurs and then restarted
> when the page fault has been processed.
>
> There are other two discussion thread which say kernel is non-pageable and
> now due to growing kernel Data structures it is allowed
>
> http://kerneltrap.org/node/6404
>
> http://kerneltrap.org/node/8206
>
>
> Regards
>
> Kishore
>
>
>
> On Thu, Jun 21, 2012 at 5:57 PM, Vijay Chauhan 
> wrote:
>>
>> Hello,
>>
>> I am newbie.
>> It has been said "kernel memory is not pageable"
>> What does it mean? There is no concept of kernel virtual address?
>>
>> Any simple explanation will help me to udnerstand.
>>
>> Thanks,
>> Vijay
>>
>> ___
>> Kernelnewbies mailing list
>> Kernelnewbies@kernelnewbies.org
>> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>
>

___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Kernel Memory

[michi1]

Hi!

On 17:57 Thu 21 Jun , Vijay Chauhan wrote:
> Hello,
> 
> I am newbie.
> It has been said "kernel memory is not pageable"
> What does it mean? There is no concept of kernel virtual address?
> 
> Any simple explanation will help me to udnerstand.

The right term is actually "kernel memory is not swappable". Swapping means
writing inactive memory to disk and then using it for something else. Kernel
memory not being swappable is a design decicion made in the early linux days.
Operating systems which swap kernel memory need to isolate everything which
should not be swappd out (e.g. things needed for swap-in, realtime stuff,
security sensitive data, ...). This is quite a bit of work. I also guess it is
pretty pointless nowadays. Installed memory and is getting so huge that virtual
memory developers have a hard time trying to keep cpu-usage overhead for
swapping user space memory low.

> There is no concept of kernel virtual address?

Kernel memory uses virtual addresses as well. However, these the entire system
memory is continuously mapped somewhere in the virtual address space. The
drawback is that fragmentation turns allocation of large continuous memory
regions into a game of luck.

There is also an virtual address area (vmalloc) which is used to dynamically
map multiple scattered pages to a continuous region. But this is rather slow
and rarely used.

You might want to take a look at: http://lwn.net/Kernel/LDD3/

-Michi
-- 
programing a layer 3+4 network protocol for mesh networks
see http://michaelblizek.twilightparadox.com

___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Kernel Memory

[AFT]

Vijay Chauhan  writes:

> Hello,
>
> I am newbie.
> It has been said "kernel memory is not pageable"
> What does it mean? There is no concept of kernel virtual address?
>

Yes. Kernel works on static adress space.

> Any simple explanation will help me to udnerstand.
>

I'm not sure if you want to understand "how kernel manages memory for
its internal DS". If its the case you should read the following
documents.

1) Read the chapter 8 of Linux Device driver 3rd edition.

http://lwn.net/Kernel/LDD3/

2) To understand slab allocator read the following papers by bonwick

   a) 94 paper describing slab allocator:
   
   
http://static.usenix.org/publications/library/proceedings/bos94/full_papers/bonwick.a

   b) Its followup in 2001
   
   
http://static.usenix.org/event/usenix01/full_papers/bonwick/bonwick_html/index.html

3) These should be enough. But if you want to know detailed architecture
of how virtual memory manager work you should read Gorman's book on
Linux virtual memory manager. Its a free pdf. can be found here:

http://ptgmedia.pearsoncmg.com/images/0131453483/downloads/gorman_book.pdf

Happy hacking.

Cheers
aft


> Thanks,
> Vijay
>
> ___
> Kernelnewbies mailing list
> Kernelnewbies@kernelnewbies.org
> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Kernel Memory

[Arun KS]

Hello Vijay,

On Thu, Jun 21, 2012 at 5:57 PM, Vijay Chauhan  wrote:
> Hello,
>
> I am newbie.
> It has been said "kernel memory is not pageable"
> What does it mean? There is no concept of kernel virtual address?
You might have heard about 3G/1G split. This 1GB is the virtual
address of the kernel. And whenever kernel try to access any address
in this range(for ARM architecture it is  0xC000 - 0x),
there should not be any page fault. That means MMU should be able to
convert your virtual address to physical. This 1GB contains your IO
address, RAM address.

Paging is a mechanism OS uses to pull the data(in pagesizes) to and
fro between system RAM and secondary memory.
Kernel memory is not pageable. This means memory allocated for the
kernel will not be pagged out. If you try to access any memory in
kernel with out creating page tables(this can be done by ioremap) you
will end up in OOPS. The main reason of kernel not being swapable or
pageable is as follows.
Think this way. What will happen if we have paged out that portion of
the logic which decides what to do when a page fault occurs? Who will
take care of the page fault then?

But if a user program hit a page fault(ie accessed address is not in
main memory), kernel will load the page from secondary memory if it is
a valid address. And if the address accesses is illegal, kernel kill
the user application(Segmentation fault).

Thanks,
Arun

>
> Any simple explanation will help me to udnerstand.
>
> Thanks,
> Vijay
>
> ___
> Kernelnewbies mailing list
> Kernelnewbies@kernelnewbies.org
> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Kernel Memory

[kishore sheik ahamed]

Hey Vijay

I am a newbie too. Just sharing what I could go through.

It is said that Kernel or atleast a part of kernel needs to be non paged
for fast interrupt access etc as pinned memory
Wiki says
Pinned/Locked/Fixed pages

Operating systems have memory areas that are pinned (never swapped to
secondary storage). For example,
interruptmechanisms rely on an
array of pointers to their handlers, such as
I/O  completion and page
fault.
If the pages containing these pointers or the code that they invoke were
pageable, interrupt-handling would become far more complex and
time-consuming, particularly in the case of page fault interrupts. Hence,
some part of the page table structures is not pageable.

Some pages may be pinned for short periods of time, others may be pinned
for long periods of time, and still others may need to be permanently
pinned. For example:

   - The paging supervisor code and drivers for secondary storage devices
   on which pages reside must be permanently pinned, as otherwise paging
   wouldn't even work because the necessary code wouldn't be available.
   - Timing-dependent components may be pinned to avoid variable paging
   delays.
   - Data buffers  that are
   accessed directly by peripheral devices that use direct memory
accessor I/O
   channels  must reside in
   pinned pages while the I/O operation is in progress because such devices
   and the buses  to
   which they are attached expect to find data buffers located at physical
   memory addresses; regardless of whether the bus has a memory management
   unit for I/O , transfers cannot be
   stopped if a page fault occurs and then restarted when the page fault has
   been processed.

There are other two discussion thread which say kernel is non-pageable and
now due to growing kernel Data structures it is allowed

http://kerneltrap.org/node/6404

http://kerneltrap.org/node/8206


Regards

Kishore


On Thu, Jun 21, 2012 at 5:57 PM, Vijay Chauhan wrote:

> Hello,
>
> I am newbie.
> It has been said "kernel memory is not pageable"
> What does it mean? There is no concept of kernel virtual address?
>
> Any simple explanation will help me to udnerstand.
>
> Thanks,
> Vijay
>
> ___
> Kernelnewbies mailing list
> Kernelnewbies@kernelnewbies.org
> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>
___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Kernel Memory

[HowKernel StuffWorks]

Hi,
  I am a kernel newbie too, Kernel memory is not pagable, because kernel
itself is responsible for paging. See this discussion,
http://kerneltrap.org/node/6404

  Paging happens for regular processes, i.e each process memory is divided
into a page of certain size(4kb in Linux), so it can swap for another page
that might be needed for that particular moment, it usually replaces an
page that have not been used for long time(see LRU,internal fragmentation).
Kernel is a process too, just like anyother process, but it differs from
others because it directly talks to the hardware and also it is the one
which takes care of paging and LRU algo's. So it does not make much sense,
for a kernel memory to be pageable, because, if it removes the page that
contains that has the paging algorithm, then there is  no way to come back,
you want to retrieve a page from disk_swap_area, but you cannot, because
the page that contains code to retrieve the instructions are paged :)...
This is not just for paging, since kernel controls everything, it is not
advisable to put its own code in swap area(see, virtual memory, virtual
memory = physical memory + swap space).

BTW, i am a kernel newbie too, this is my basic understanding, please feel
free to correct it, if I am wrong

On Thu, Jun 21, 2012 at 8:27 AM, Vijay Chauhan wrote:

> Hello,
>
> I am newbie.
> It has been said "kernel memory is not pageable"
> What does it mean? There is no concept of kernel virtual address?
>
> Any simple explanation will help me to udnerstand.
>
> Thanks,
> Vijay
>
> ___
> Kernelnewbies mailing list
> Kernelnewbies@kernelnewbies.org
> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>
___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: kernel memory allocation

[Dave Hylands]

Hi,

On Thu, Dec 22, 2011 at 6:33 PM, J.Hwan Kim  wrote:
> Hi, everyone
>
> How can I allocated contiguous kernel memory over 128MB ?
> When I use _get_free_pages() function, it returns error.
> I guess the memory size is greater than the amount which the function
> can allocate.

You can use bootmem to reserve the memory at boot time, and then use
the bootmem allocator to alloocate from that reserved memory/

There is also some new code called CMA (Contiguous Memory Allocator)
which hasn't hit the mainline yet, but it could also be used.

See: http://lwn.net/Articles/468044/ for further details.

-- 
Dave Hylands
Shuswap, BC, Canada
http://www.davehylands.com

___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

RE: kernel memory allocation

[Jeff Haran]

> -Original Message-
> From: kernelnewbies-boun...@kernelnewbies.org [mailto:kernelnewbies-
> boun...@kernelnewbies.org] On Behalf Of João Eduardo Luís
> Sent: Friday, June 03, 2011 12:40 PM
> To: Amirali Shambayati
> Cc: kernelnewbies
> Subject: Re: kernel memory allocation
> 
> Hi.
> 
> In future replies, please CC the list.
> 
> Does it panic in during the kmalloc, or afterwards? Are you checking if
> 'newBun' is NULL?
> 
> I may be missing something obvious in that code, but unless you are out of
> memory or with some past corruption, I don't think that should panic the
> kernel.
> 
> In any case, the panic trace should help.
> 
> 
> Cheers.
> 
> ---
> João Eduardo Luís
> gpg key: 477C26E5 from pool.keyserver.eu
> 
> 
> 
> 
> 
> On Jun 3, 2011, at 7:50 PM, Amirali Shambayati wrote:
> 
> > thanks for your guidance. I just want to allocate memory for a struct. I use
> this:
> >
> > struct bundle* newBun;
> > newBun = kmalloc(sizeof(*newBun), GFP_KERNEL);
> >
> > but it goes to panic state.
> >

Just guessing, but the most likely reason for the above call to kmalloc() to 
cause a panic is it's being called in atomic context.

Are you seeing anything like "scheduling while atomic" in the panic back trace?

If so, that's your problem and the solution is most likely to replace the above 
GFP_KERNEL with GFP_ATOMIC. The alternative would be to restructure the code so 
that the allocation doesn't happen in atomic context.

Jeff Haran
Bytemobile




___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: kernel memory allocation

[João Eduardo Luís]

Hello,

Once again, I would pretty much enjoy if you CC'ed the list on replies.


On Jun 3, 2011, at 8:49 PM, Amirali Shambayati wrote:

> I set breakpoint before kmalloc. panic happens after kmalloc.
> 
>> 2011/6/4 Amirali Shambayati 
>> ofcourse it's null before allocate it. I make an instantiation of it. then  
>> I want to allocate memory for it.What else I should do?

I meant, are you checking if 'newBun' is NULL _AFTER_ the kmalloc?

The kmalloc call will return NULL instead of a pointer to newly allocated 
memory whenever an error occurs. If by any chance you are not checking if 
'newBun' is NULL after the kmalloc, then you should do it.

Also, if the panic happens _after_ the kmalloc, and not _during_ the kmalloc, I 
would say there's a slim chance of kmalloc being the cause of the panic.


---
João Eduardo Luís
gpg key: 477C26E5 from pool.keyserver.eu 

PGP.sig
Description: This is a digitally signed message part
___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: kernel memory allocation

[João Eduardo Luís]

Hi.

In future replies, please CC the list.

Does it panic in during the kmalloc, or afterwards? Are you checking if 
'newBun' is NULL?

I may be missing something obvious in that code, but unless you are out of 
memory or with some past corruption, I don't think that should panic the kernel.

In any case, the panic trace should help.


Cheers.

---
João Eduardo Luís
gpg key: 477C26E5 from pool.keyserver.eu 





On Jun 3, 2011, at 7:50 PM, Amirali Shambayati wrote:

> thanks for your guidance. I just want to allocate memory for a struct. I use 
> this: 
> 
> struct bundle* newBun;
> newBun = kmalloc(sizeof(*newBun), GFP_KERNEL);
> 
> but it goes to panic state.
> 
> 2011/6/3 João Eduardo Luís 
> Hi.
> 
> From [1] I'm lead to believe the only difference between the regular 
> kmalloc() arguments and those of kmalloc_node() is the one specifying which 
> node you want to allocate the memory on.
> 
> Aside from the third argument, which seems to be related with NUMA (with 
> which I never worked on kernel-context), I would suggest you to read [2].
> 
> If on the other hand you only want to allocate memory locally, maybe you 
> should use kmalloc() or one of its variants [3].
> 
> 
> Cheers.
> 
> 
> [1] - http://www.kernel.org/doc/htmldocs/kernel-api/API-kmalloc-node.html
> [2] - http://www.kernel.org/doc/htmldocs/kernel-api/API-kcalloc.html
> [3] - http://www.kernel.org/doc/htmldocs/kernel-api/mm.html#id408507
> ---
> João Eduardo Luís
> gpg key: 477C26E5 from pool.keyserver.eu
> 
> 
> 
> 
> 
> On Jun 3, 2011, at 7:07 PM, Amirali Shambayati wrote:
> 
> >
> > Hello all,
> >
> > I just want to allocate memory for a struct instantiation. Would anyone 
> > guide me what arguments I should pass to "kmalloc_node"?
> > Regards,
> > --
> > Amirali Shambayati
> > Bachelor Student
> > Computer Engineering Department
> > Sharif University of Technology
> > Tehran, Iran
> >
> > ___
> > Kernelnewbies mailing list
> > Kernelnewbies@kernelnewbies.org
> > http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
> 
> 
> 
> 
> -- 
> Amirali Shambayati
> Bachelor Student
> Computer Engineering Department
> Sharif University of Technology
> Tehran, Iran
> 



PGP.sig
Description: This is a digitally signed message part
___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: kernel memory allocation

[João Eduardo Luís]

Hi.

From [1] I'm lead to believe the only difference between the regular kmalloc() 
arguments and those of kmalloc_node() is the one specifying which node you want 
to allocate the memory on.

Aside from the third argument, which seems to be related with NUMA (with which 
I never worked on kernel-context), I would suggest you to read [2].

If on the other hand you only want to allocate memory locally, maybe you should 
use kmalloc() or one of its variants [3]. 


Cheers.


[1] - http://www.kernel.org/doc/htmldocs/kernel-api/API-kmalloc-node.html
[2] - http://www.kernel.org/doc/htmldocs/kernel-api/API-kcalloc.html
[3] - http://www.kernel.org/doc/htmldocs/kernel-api/mm.html#id408507
---
João Eduardo Luís
gpg key: 477C26E5 from pool.keyserver.eu 





On Jun 3, 2011, at 7:07 PM, Amirali Shambayati wrote:

> 
> Hello all,
> 
> I just want to allocate memory for a struct instantiation. Would anyone guide 
> me what arguments I should pass to "kmalloc_node"?
> Regards,
> -- 
> Amirali Shambayati
> Bachelor Student
> Computer Engineering Department
> Sharif University of Technology
> Tehran, Iran
> 
> ___
> Kernelnewbies mailing list
> Kernelnewbies@kernelnewbies.org
> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies



PGP.sig
Description: This is a digitally signed message part
___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: kernel memory usage

[naveen yadav]

Yes Bharath, you are right.

On Thu, Dec 30, 2010 at 9:07 AM, Bharath H S  wrote:
> I think Naveen means, all the memory does not appear in the output of
> /proc/meminfo, the reason is kernel memory is discounted and not
> accounted for in the output. May be his goal is to calculate the total
> memory in the system and that output of /proc/meminfo does not give
> that.
>
>
>
> On Wed, Dec 29, 2010 at 9:39 PM, Mulyadi Santosa
>  wrote:
>> On Wed, Dec 29, 2010 at 20:30, naveen yadav  wrote:
>>> Hi all,
>>>
>>> I want to calculate the total kernel memory usage.
>>
>> First, what's your definition of "kernel memory" ? Everything that is
>> mapped inside kernel address space only?
>> --
>> regards,
>>
>> Mulyadi Santosa
>> Freelance Linux trainer and consultant
>>
>> blog: the-hydra.blogspot.com
>> training: mulyaditraining.blogspot.com
>>
>> ___
>> Kernelnewbies mailing list
>> Kernelnewbies@kernelnewbies.org
>> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>>
>

___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: kernel memory usage

[Bharath H S]

I think Naveen means, all the memory does not appear in the output of
/proc/meminfo, the reason is kernel memory is discounted and not
accounted for in the output. May be his goal is to calculate the total
memory in the system and that output of /proc/meminfo does not give
that.



On Wed, Dec 29, 2010 at 9:39 PM, Mulyadi Santosa
 wrote:
> On Wed, Dec 29, 2010 at 20:30, naveen yadav  wrote:
>> Hi all,
>>
>> I want to calculate the total kernel memory usage.
>
> First, what's your definition of "kernel memory" ? Everything that is
> mapped inside kernel address space only?
> --
> regards,
>
> Mulyadi Santosa
> Freelance Linux trainer and consultant
>
> blog: the-hydra.blogspot.com
> training: mulyaditraining.blogspot.com
>
> ___
> Kernelnewbies mailing list
> Kernelnewbies@kernelnewbies.org
> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>

___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: kernel memory usage

[Mulyadi Santosa]

On Wed, Dec 29, 2010 at 20:30, naveen yadav  wrote:
> Hi all,
>
> I want to calculate the total kernel memory usage.

First, what's your definition of "kernel memory" ? Everything that is
mapped inside kernel address space only?
-- 
regards,

Mulyadi Santosa
Freelance Linux trainer and consultant

blog: the-hydra.blogspot.com
training: mulyaditraining.blogspot.com

___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies