Kexec rash kernel memory reservation

[Ronit Halder]

Hi,
kexec reserves memory for loading crash kernel in the boot time for
risk of DMA. I want to know,
How kexec prevents allocation of memory which is adjacent to DMA memory?

regards,
Ronit

___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Kernel memory

[Valdis . Kletnieks]

On Tue, 17 May 2016 12:41:44 +0530, Ronit Halder said:
> Hi,
>
> Where in the memory kernel is located in the boot time?

During which exact phase of the boot, and does it actually matter?  And
physical or virtual address?

(Hint:  If you're not the bootstrap that unpacks the compressed kernel
into memory, or the code that sets up the ASLR code, or are trying to work
around an issue with a hardware device that insists on taking certain
physical addresses for itself, you probably don't really care where the
kernel is)


pgpA8PmnyxbRA.pgp
Description: PGP signature
___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Kernel memory

[Ronit Halder]

Hi,

Where in the memory kernel is located in the boot time?

regards,
Ronit

___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: The limit of kernel memory in 64bit system

[Christoph Lameter]

On Sat, 29 Nov 2014, nick wrote:

> Sorry Yiqun,
> I was thinking of virtual memory so you are correct. I also haven't looked 
> into
> the kernel memory subsystem(s) in a while so I may be a little behind in my 
> knownledge
> of them.
2^1
The amount of memory that can be handled by a processor is available via
/proc/cpuinfo:

processor   : 0
vendor_id   : GenuineIntel
cpu family  : 6
model   : 60
model name  : Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz
stepping: 3
microcode   : 0x16
cpu MHz : 3400.132
cache size  : 8192 KB
clflush size: 64
cache_alignment : 64
address sizes   : 39 bits physical, 48 bits virtual

so this processor support a physical memory space of 2^39 = 512GB and a
virtual address space of 256 TB.

A 64 bit kernel can support what the processor supports.


___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: The limit of kernel memory in 64bit system

[nick]

Sorry Yiqun,
I was thinking of virtual memory so you are correct. I also haven't looked into
the kernel memory subsystem(s) in a while so I may be a little behind in my 
knownledge
of them.
Regards Nick 

On 2014-11-29 03:54 PM, Yiqun Chen wrote:
> Incorrect. The 64 bit machine theoretically supports 2 ^ 64 bytes virtual
> memory space but on x86 only 48 out of 64 address buses are used. So we
> cannot assume the upper limit of physical space.on different platforms.
> And Nick you're confusing with the physical and virtual memory. The usage
> of memory for kernel space is not determined by the ratio. It's defined by
> mapping strategy. You first need to know why this 3 to 1 was used. It's
> because we cannot map all the physical pages onto the corresponding virtual
> address since we need to spare some linear space for user program. But on
> 64 bits platform we have plenty virtual address for the programmer as the
> physical memory is much much smaller than it. On x86_64 we divide
> kernel/user space by setting all the first 16 bits with 0 or 1 as the user
> or kernel space respectfully. So how large would the kernel space be if we
> could use 48-bit linear space?
> 
> On Sat, Nov 29, 2014 at 8:16 PM, nick  wrote:
> 
>> J.Hwan,
>> There is a limit as 64 bit can only support 16 extabytes. Greg is right in
>> that there is no real limit
>> on modern machines because of this. However from my knowledge, system ram
>> is split into a 3 to 1 ratio,
>> where 75 percent is for user space and the only quarter is for the
>> kernel's memory. So on my system with
>> 8 GB of ram, the kernel can use up to 25 percent or 2 GB of ram.
>> Hope this answers your question,
>> Nick
>> On 2014-11-29 12:45 PM, Greg KH wrote:
>>> On Sat, Nov 29, 2014 at 12:25:43AM +0900, J.Hwan Kim wrote:
>>>> Hi, everyone
>>>>
>>>> As far as I know, the kernel memory limit is 1GB in 32bit system.
>>>> Is it also applicable to 64bit system?
>>>> What's the limit of kernel memory in 64bit system?
>>>
>>> There is no limit :)
>>>
>>> What do you really want to know this for?
>>>
>>> greg k-h
>>>
>>> ___
>>> Kernelnewbies mailing list
>>> Kernelnewbies@kernelnewbies.org
>>> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>>>
>>
>> ___
>> Kernelnewbies mailing list
>> Kernelnewbies@kernelnewbies.org
>> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>>
> 
> 
> 

___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: The limit of kernel memory in 64bit system

[Yiqun Chen]

Incorrect. The 64 bit machine theoretically supports 2 ^ 64 bytes virtual
memory space but on x86 only 48 out of 64 address buses are used. So we
cannot assume the upper limit of physical space.on different platforms.
And Nick you're confusing with the physical and virtual memory. The usage
of memory for kernel space is not determined by the ratio. It's defined by
mapping strategy. You first need to know why this 3 to 1 was used. It's
because we cannot map all the physical pages onto the corresponding virtual
address since we need to spare some linear space for user program. But on
64 bits platform we have plenty virtual address for the programmer as the
physical memory is much much smaller than it. On x86_64 we divide
kernel/user space by setting all the first 16 bits with 0 or 1 as the user
or kernel space respectfully. So how large would the kernel space be if we
could use 48-bit linear space?

On Sat, Nov 29, 2014 at 8:16 PM, nick  wrote:

> J.Hwan,
> There is a limit as 64 bit can only support 16 extabytes. Greg is right in
> that there is no real limit
> on modern machines because of this. However from my knowledge, system ram
> is split into a 3 to 1 ratio,
> where 75 percent is for user space and the only quarter is for the
> kernel's memory. So on my system with
> 8 GB of ram, the kernel can use up to 25 percent or 2 GB of ram.
> Hope this answers your question,
> Nick
> On 2014-11-29 12:45 PM, Greg KH wrote:
> > On Sat, Nov 29, 2014 at 12:25:43AM +0900, J.Hwan Kim wrote:
> >> Hi, everyone
> >>
> >> As far as I know, the kernel memory limit is 1GB in 32bit system.
> >> Is it also applicable to 64bit system?
> >> What's the limit of kernel memory in 64bit system?
> >
> > There is no limit :)
> >
> > What do you really want to know this for?
> >
> > greg k-h
> >
> > ___
> > Kernelnewbies mailing list
> > Kernelnewbies@kernelnewbies.org
> > http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
> >
>
> ___
> Kernelnewbies mailing list
> Kernelnewbies@kernelnewbies.org
> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>



-- 
Best regards
Chen, Yiqun
___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: The limit of kernel memory in 64bit system

[nick]

J.Hwan,
There is a limit as 64 bit can only support 16 extabytes. Greg is right in that 
there is no real limit 
on modern machines because of this. However from my knowledge, system ram is 
split into a 3 to 1 ratio,
where 75 percent is for user space and the only quarter is for the kernel's 
memory. So on my system with
8 GB of ram, the kernel can use up to 25 percent or 2 GB of ram.
Hope this answers your question,
Nick
On 2014-11-29 12:45 PM, Greg KH wrote:
> On Sat, Nov 29, 2014 at 12:25:43AM +0900, J.Hwan Kim wrote:
>> Hi, everyone
>>
>> As far as I know, the kernel memory limit is 1GB in 32bit system.
>> Is it also applicable to 64bit system?
>> What's the limit of kernel memory in 64bit system?
> 
> There is no limit :)
> 
> What do you really want to know this for?
> 
> greg k-h
> 
> ___
> Kernelnewbies mailing list
> Kernelnewbies@kernelnewbies.org
> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
> 

___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: The limit of kernel memory in 64bit system

[Greg KH]

On Sat, Nov 29, 2014 at 12:25:43AM +0900, J.Hwan Kim wrote:
> Hi, everyone
> 
> As far as I know, the kernel memory limit is 1GB in 32bit system.
> Is it also applicable to 64bit system?
> What's the limit of kernel memory in 64bit system?

There is no limit :)

What do you really want to know this for?

greg k-h

___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

The limit of kernel memory in 64bit system

[J.Hwan Kim]

Hi, everyone

As far as I know, the kernel memory limit is 1GB in 32bit system.
Is it also applicable to 64bit system?
What's the limit of kernel memory in 64bit system?

Thanks in advance

Regards,

J.Hwan Kim

___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

The kernel memory limit of 64bit system

[J.Hwan Kim]

Hi,  everyone

How much is the kernel memory limit of 64 bit system?
As far as I know, int 32 bit system the kernel memory limit is 1GB.
Is it also applicable to 64bit system?

Thanks in advance

Regards,
J.Hwan Kim

___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: How to mmap kernel memory area to userspace ?

[Valdis . Kletnieks]

On Tue, 19 Aug 2014 17:54:14 +0800, lxgeek said:
>I want to mmap a kernel memory area which sk_buf->data pointer
> into userspace. I want to do this , because this way can reduce a copy
> from kernel to userspace.
>How to fix it? Or, which book or project can help me ?

Look at how other drivers do zero-copy.  Hint: mmap is probably not
the best way to approach it, because that would require userspace to
do something different to use it.


pgpVYdeBOFUh4.pgp
Description: PGP signature
___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

RE: How to mmap kernel memory area to userspace ?

[Jeff Haran]

> -Original Message-
> From: kernelnewbies-boun...@kernelnewbies.org 
> [mailto:kernelnewbies-boun...@kernelnewbies.org] On Behalf Of lxgeek
> Sent: Tuesday, August 19, 2014 2:54 AM
> To: kernelnewbies
> Subject: How to mmap kernel memory area to userspace ?
> 
> hi all:
>    I want to mmap a kernel memory area which sk_buf->data pointer
> into userspace. I want to do this , because this way can reduce a copy
> from kernel to userspace.
>How to fix it? Or, which book or project can help me ?
> 
> Thank you.

One approach that I had some limited success with recently as part of an 
investigative exercise was to mmap /dev/kmem in the context of the process you 
want to access the skb data in. You'll need to figure out the block of kernel 
virtual addresses to map based on your target architecture, but basically it 
needs to cover the kmem cache that the TCP/IP stack allocates skbs from. Then 
you need to get the virtual address of the skbs' data up to user space, somehow 
(for instance create a device that registers hooks with netfilter, steals and 
queues the skbs that get passed to it and then make the virtual addresses in 
the queued skbs available to be read from the device). Then it's just a matter 
of some pointer arithmetic to determine the corresponding address of the skb 
payload in the pointer you get back from the call to mmap().

I am aware of a couple of potential obstacles to contend with here depending on 
your kernel and architecture:

CONFIG_STRICT_DEVMEM - If this kernel config is enabled in your kernel, you 
won't be able to do this. It's a security thing. You'd need to turn this off to 
proceed down this path.
CONFIG_X86_PAT - If your target is X86_64, then this kernel config will prevent 
you from mmap()'ing /dev/kmem, too. It's not clear to me what the effect of 
turning this config off is, I'm no expert in X86_64 memory management. When I 
tried it, the resultant kernel booted ok and seemed to function on our X86_64 
target systems, but just doing read access of the data from the mmap()'ed 
region did occasionally generate error messages from the kernel and eventual 
soft lockups.

For me, it worked well enough to complete the investigation but we didn't try 
to productize this.

Good luck, what you are trying to do is not easy to get right in my experience,

Jeff Haran


___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

How to mmap kernel memory area to userspace ?

[lxgeek]

hi all:
   I want to mmap a kernel memory area which sk_buf->data pointer 
into userspace. I want to do this , because this way can reduce a copy
from kernel to userspace.
   How to fix it? Or, which book or project can help me ?

Thank you.

___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Questions about Kernel Memory that I didn't find answers in Google - Please Help

[Lucas Tanure]

e sections and the address where the
>>> > different sections are supposed to be loaded in memory.   If u replace
>>> the
>>> > vmlinux with the kernel module, eg: ip_tables.ko, then it says:
>>> >
>>> > starting at offset 0x328c blah blah
>>> >
>>> > so the loaded address is with respect to ZERO, but then the actual
>>> module
>>> > address is:
>>> >
>>> > sudo cat /proc/modules |grep ip_table
>>> >
>>> > ip_tables 18106 1 iptable_filter, Live 0xf8bf5000
>>> >
>>> > So all the output from your readelf, just add 0xf8bf5000 to it and you
>>> will
>>> > get the actual virtual address of that section IN MEMORY.
>>> >
>>> > Just only in memory.   In file, the file offset of the section is
>>> different.
>>> > And many parts inside the ELF is also different from memory too:   you
>>> will
>>> > need to add the virtual load address (above) to the offset as specified
>>> > inside the relocation tables (objdump -r), and for each section there
>>> is a
>>> > separate relocation table (all independent from another, meaning that
>>> the
>>> > different section CAN BE loaded to different parts in memory).
>>> >
>>> > Thanks.
>>> >
>>> >
>>> > On Sun, Aug 3, 2014 at 11:59 PM, Lucas Tanure 
>>> wrote:
>>> >>
>>> >> Hi,
>>> >>
>>> >> I'm looking for some site, pdf, book etc, that can answer this
>>> questions.
>>> >> For now I have :
>>> >>
>>> >>
>>> http://unix.stackexchange.com/questions/5124/what-does-the-virtual-kernel-memory-layout-in-dmesg-imply
>>> >>
>>> >>
>>> >> I want to understand a few things about the memory and the execution
>>> >> of Linux kernel.
>>> >> Taking from a X86 and grub I have:
>>> >>
>>> >> 1) Grub loads kernel and root file system in memory, and the vmlinux
>>> >> has the code to decompress it self, right ? linux
>>> >>
>>> >> 2) The address of load kernel is always the same ? And It's at
>>> >> compilation time that is chosen ?
>>> >>
>>> >> 2a) The kernel takes places in 3g-4g memory place, and user space
>>> from 0
>>> >> to 3gb.
>>> >> But if the pc has only 256mb of memory ?
>>> >> And when pc has 16gb of memory, the user space will be split in two ?
>>> >>
>>> >> 2b) And if kernel has soo many modules that needs more than 1gb to
>>> run ?
>>> >>
>>> >> 2c) How we configure all of that memory configs ? make menuconfig and
>>> >> friends ?
>>> >>
>>> >> 3) The function A will call functon B. B is at 0xGG in .text
>>> >> section, but kernel was loaded in address 0xJJ, how A will
>>> >> find B ?
>>> >>
>>> >> 4) Please consider this:
>>> >> $ readelf -S -W vmlinux
>>> >> There are 37 section headers, starting at offset 0xe05718:
>>> >>
>>> >> Section Headers:
>>> >>   [Nr] Name   Type  Address
>>> >> Off Size  ES Flg Lk Inf Al
>>> >>   [ 0]  NULL
>>> >> 00  00 00  0   0  0
>>> >>   [ 1] .text PROGBITS
>>> >> 8100  20 53129a  00  AX  0   0 4096
>>> >>   [ 2] .notes  NOTE
>>> >> 8153129c  73129c 0001d8  00  AX  0   0  4
>>> >>   [ 3] __ex_table   PROGBITS81531480
>>> >>731480 002018  00   A  0   0  8
>>> >>   [ 4] .rodata PROGBITS
>>> >> 8160  80 1655ee 00   A  0   0 64
>>> >>   [ 5] __bug_table PROGBITS817655f0
>>> >>9655f0  005424 00   A  0   0  1
>>> >>   [ 6] .pci_fixup PROGBITS8176aa18
>>> >>  96aa18 002f88  00   A  0   0  8
>>> >>   [ 7] .tracedataPROGBITS

Re: Questions about Kernel Memory that I didn't find answers in Google - Please Help

[Lucas Tanure]

I`m still reading all the resources, but I found a few more.

http://www.ualberta.ca/CNS/RESEARCH/LinuxClusters/mem.html


I didn`t find a few answers yet, but how a module is loaded ?
When you do a insmod the kernel take some pages and load all the module's
sections to these pages ?
How this work  ? Where I can find more information about memory allocation
for a new module.
If I don`t find, I will read the source code for that part, but I try to
find a better way to learn. I know that source code is the best and
updated, but takes more time too.


And when I do a int inside my module I suppose that my int will be inside
that pages that kernel allocated for my new module. Right ??
But if I do this inside kernel code ? It's compiler task do allocate this
memory in sections ? How the kernel runs ? Like my module is driven by
kernel, but how kernel driven it's self ???


Thanks




--
Lucas Tanure
+55 (19) 988176559


On Tue, Aug 5, 2014 at 12:31 AM, Peter Teoh  wrote:

> And Q2:
>
> Just want to comment that the load address has to be fixed initially,
> because unlike normal ELF, after loading ELF, there is a relocation tasks
> done by the linker.   In vmlinuz we cannot have relocation, before
> executing the kernel is the BIOS / uboot / bootloader etc.   One possible
> answer.   Others:
>
> https://groups.google.com/forum/#!topic/comp.os.linux.embedded/0-SAzCqQKFM
>
> And perhaps some of the links below may help you:
>
> http://jianggmulab.blogspot.sg/2010_01_01_archive.html
>
>
> http://stackoverflow.com/questions/5647279/why-does-the-module-start-from-address-0xbf00
>
> http://www.arm.linux.org.uk/developer/memory.txt
>
> http://en.wikipedia.org/wiki/High_memory
>
> bottomline: keep googling.
>
> Q6 and 7 makes no sense to mesorry.
>
>
>
> On Mon, Aug 4, 2014 at 11:22 PM, Lucas Tanure  wrote:
>
>> Thanks!
>>
>> A quick look in all of that show me that there a lot of information
>> about how kernel manage memory.
>> But, I will find the answer for question 2, 6 and 7 in it ?
>>
>> Thanks!
>> --
>> Lucas Tanure
>> +55 (19) 988176559
>>
>>
>> On Sun, Aug 3, 2014 at 8:58 PM, Peter Teoh 
>> wrote:
>> > I like your curiosities and interests in Linux
>> > kernel.
>> http://virtuallyhyper.com/2013/07/rhcsa-and-rhce-chapter-10-the-kernel/
>> >
>> > Instead of answering one by one, I think I will just identify the
>> knowledge
>> > you are lacking:
>> >
>> > Memory management (from both x86/intel and linux kernel perspective).
>> >
>> > There are many many resources out there for you in these area, eg:
>> >
>> > http://en.wikipedia.org/wiki/Page_table
>> > http://en.wikipedia.org/wiki/X86-64
>> >
>> > (both boring, but just understand it well enough)
>> >
>> > http://wiki.osdev.org/Paging   (good explanationunderstand it very
>> very
>> > well).
>> >
>> > The ultimate classic ebook:
>> >
>> > https://www.kernel.org/doc/gorman/pdf/understand.pdf
>> >
>> > And this blog site has tons of good info on intel/memory etc:
>> >
>> >
>> http://duartes.org/gustavo/blog/post/cpu-rings-privilege-and-protection/
>> > http://duartes.org/gustavo/blog/post/anatomy-of-a-program-in-memory/
>> >
>> > http://virtuallyhyper.com/2013/07/rhcsa-and-rhce-chapter-10-the-kernel/
>> >
>> > http://www.cse.psu.edu/~anand/spring01/linux/memory.ppt
>> >
>> > One more thing:
>> >
>> > "readelf -S -W vmlinux" shows u the sections and the address where the
>> > different sections are supposed to be loaded in memory.   If u replace
>> the
>> > vmlinux with the kernel module, eg: ip_tables.ko, then it says:
>> >
>> > starting at offset 0x328c blah blah
>> >
>> > so the loaded address is with respect to ZERO, but then the actual
>> module
>> > address is:
>> >
>> > sudo cat /proc/modules |grep ip_table
>> >
>> > ip_tables 18106 1 iptable_filter, Live 0xf8bf5000
>> >
>> > So all the output from your readelf, just add 0xf8bf5000 to it and you
>> will
>> > get the actual virtual address of that section IN MEMORY.
>> >
>> > Just only in memory.   In file, the file offset of the section is
>> different.
>> > And many parts inside the ELF is also different from memory too:   you
>> will
>> > need to add the virtual load address (above) to the offset as specified
>> > inside the relocation tables (objdump -r), and for each s

Re: Questions about Kernel Memory that I didn't find answers in Google - Please Help

[Peter Teoh]

And Q2:

Just want to comment that the load address has to be fixed initially,
because unlike normal ELF, after loading ELF, there is a relocation tasks
done by the linker.   In vmlinuz we cannot have relocation, before
executing the kernel is the BIOS / uboot / bootloader etc.   One possible
answer.   Others:

https://groups.google.com/forum/#!topic/comp.os.linux.embedded/0-SAzCqQKFM

And perhaps some of the links below may help you:

http://jianggmulab.blogspot.sg/2010_01_01_archive.html

http://stackoverflow.com/questions/5647279/why-does-the-module-start-from-address-0xbf00

http://www.arm.linux.org.uk/developer/memory.txt

http://en.wikipedia.org/wiki/High_memory

bottomline: keep googling.

Q6 and 7 makes no sense to mesorry.



On Mon, Aug 4, 2014 at 11:22 PM, Lucas Tanure  wrote:

> Thanks!
>
> A quick look in all of that show me that there a lot of information
> about how kernel manage memory.
> But, I will find the answer for question 2, 6 and 7 in it ?
>
> Thanks!
> --
> Lucas Tanure
> +55 (19) 988176559
>
>
> On Sun, Aug 3, 2014 at 8:58 PM, Peter Teoh 
> wrote:
> > I like your curiosities and interests in Linux
> > kernel.
> http://virtuallyhyper.com/2013/07/rhcsa-and-rhce-chapter-10-the-kernel/
> >
> > Instead of answering one by one, I think I will just identify the
> knowledge
> > you are lacking:
> >
> > Memory management (from both x86/intel and linux kernel perspective).
> >
> > There are many many resources out there for you in these area, eg:
> >
> > http://en.wikipedia.org/wiki/Page_table
> > http://en.wikipedia.org/wiki/X86-64
> >
> > (both boring, but just understand it well enough)
> >
> > http://wiki.osdev.org/Paging   (good explanationunderstand it very
> very
> > well).
> >
> > The ultimate classic ebook:
> >
> > https://www.kernel.org/doc/gorman/pdf/understand.pdf
> >
> > And this blog site has tons of good info on intel/memory etc:
> >
> > http://duartes.org/gustavo/blog/post/cpu-rings-privilege-and-protection/
> > http://duartes.org/gustavo/blog/post/anatomy-of-a-program-in-memory/
> >
> > http://virtuallyhyper.com/2013/07/rhcsa-and-rhce-chapter-10-the-kernel/
> >
> > http://www.cse.psu.edu/~anand/spring01/linux/memory.ppt
> >
> > One more thing:
> >
> > "readelf -S -W vmlinux" shows u the sections and the address where the
> > different sections are supposed to be loaded in memory.   If u replace
> the
> > vmlinux with the kernel module, eg: ip_tables.ko, then it says:
> >
> > starting at offset 0x328c blah blah
> >
> > so the loaded address is with respect to ZERO, but then the actual module
> > address is:
> >
> > sudo cat /proc/modules |grep ip_table
> >
> > ip_tables 18106 1 iptable_filter, Live 0xf8bf5000
> >
> > So all the output from your readelf, just add 0xf8bf5000 to it and you
> will
> > get the actual virtual address of that section IN MEMORY.
> >
> > Just only in memory.   In file, the file offset of the section is
> different.
> > And many parts inside the ELF is also different from memory too:   you
> will
> > need to add the virtual load address (above) to the offset as specified
> > inside the relocation tables (objdump -r), and for each section there is
> a
> > separate relocation table (all independent from another, meaning that the
> > different section CAN BE loaded to different parts in memory).
> >
> > Thanks.
> >
> >
> > On Sun, Aug 3, 2014 at 11:59 PM, Lucas Tanure  wrote:
> >>
> >> Hi,
> >>
> >> I'm looking for some site, pdf, book etc, that can answer this
> questions.
> >> For now I have :
> >>
> >>
> http://unix.stackexchange.com/questions/5124/what-does-the-virtual-kernel-memory-layout-in-dmesg-imply
> >>
> >>
> >> I want to understand a few things about the memory and the execution
> >> of Linux kernel.
> >> Taking from a X86 and grub I have:
> >>
> >> 1) Grub loads kernel and root file system in memory, and the vmlinux
> >> has the code to decompress it self, right ? linux
> >>
> >> 2) The address of load kernel is always the same ? And It's at
> >> compilation time that is chosen ?
> >>
> >> 2a) The kernel takes places in 3g-4g memory place, and user space from 0
> >> to 3gb.
> >> But if the pc has only 256mb of memory ?
> >> And when pc has 16gb of memory, the user space will be split in two ?
> >>
> >> 2b) And if kernel has soo many modules that needs more than

Re: Questions about Kernel Memory that I didn't find answers in Google - Please Help

[Lucas Tanure]

Thanks!

A quick look in all of that show me that there a lot of information
about how kernel manage memory.
But, I will find the answer for question 2, 6 and 7 in it ?

Thanks!
--
Lucas Tanure
+55 (19) 988176559


On Sun, Aug 3, 2014 at 8:58 PM, Peter Teoh  wrote:
> I like your curiosities and interests in Linux
> kernel.http://virtuallyhyper.com/2013/07/rhcsa-and-rhce-chapter-10-the-kernel/
>
> Instead of answering one by one, I think I will just identify the knowledge
> you are lacking:
>
> Memory management (from both x86/intel and linux kernel perspective).
>
> There are many many resources out there for you in these area, eg:
>
> http://en.wikipedia.org/wiki/Page_table
> http://en.wikipedia.org/wiki/X86-64
>
> (both boring, but just understand it well enough)
>
> http://wiki.osdev.org/Paging   (good explanationunderstand it very very
> well).
>
> The ultimate classic ebook:
>
> https://www.kernel.org/doc/gorman/pdf/understand.pdf
>
> And this blog site has tons of good info on intel/memory etc:
>
> http://duartes.org/gustavo/blog/post/cpu-rings-privilege-and-protection/
> http://duartes.org/gustavo/blog/post/anatomy-of-a-program-in-memory/
>
> http://virtuallyhyper.com/2013/07/rhcsa-and-rhce-chapter-10-the-kernel/
>
> http://www.cse.psu.edu/~anand/spring01/linux/memory.ppt
>
> One more thing:
>
> "readelf -S -W vmlinux" shows u the sections and the address where the
> different sections are supposed to be loaded in memory.   If u replace the
> vmlinux with the kernel module, eg: ip_tables.ko, then it says:
>
> starting at offset 0x328c blah blah
>
> so the loaded address is with respect to ZERO, but then the actual module
> address is:
>
> sudo cat /proc/modules |grep ip_table
>
> ip_tables 18106 1 iptable_filter, Live 0xf8bf5000
>
> So all the output from your readelf, just add 0xf8bf5000 to it and you will
> get the actual virtual address of that section IN MEMORY.
>
> Just only in memory.   In file, the file offset of the section is different.
> And many parts inside the ELF is also different from memory too:   you will
> need to add the virtual load address (above) to the offset as specified
> inside the relocation tables (objdump -r), and for each section there is a
> separate relocation table (all independent from another, meaning that the
> different section CAN BE loaded to different parts in memory).
>
> Thanks.
>
>
> On Sun, Aug 3, 2014 at 11:59 PM, Lucas Tanure  wrote:
>>
>> Hi,
>>
>> I'm looking for some site, pdf, book etc, that can answer this questions.
>> For now I have :
>>
>> http://unix.stackexchange.com/questions/5124/what-does-the-virtual-kernel-memory-layout-in-dmesg-imply
>>
>>
>> I want to understand a few things about the memory and the execution
>> of Linux kernel.
>> Taking from a X86 and grub I have:
>>
>> 1) Grub loads kernel and root file system in memory, and the vmlinux
>> has the code to decompress it self, right ? linux
>>
>> 2) The address of load kernel is always the same ? And It's at
>> compilation time that is chosen ?
>>
>> 2a) The kernel takes places in 3g-4g memory place, and user space from 0
>> to 3gb.
>> But if the pc has only 256mb of memory ?
>> And when pc has 16gb of memory, the user space will be split in two ?
>>
>> 2b) And if kernel has soo many modules that needs more than 1gb to run ?
>>
>> 2c) How we configure all of that memory configs ? make menuconfig and
>> friends ?
>>
>> 3) The function A will call functon B. B is at 0xGG in .text
>> section, but kernel was loaded in address 0xJJ, how A will
>> find B ?
>>
>> 4) Please consider this:
>> $ readelf -S -W vmlinux
>> There are 37 section headers, starting at offset 0xe05718:
>>
>> Section Headers:
>>   [Nr] Name   Type  Address
>> Off Size  ES Flg Lk Inf Al
>>   [ 0]  NULL
>> 00  00 00  0   0  0
>>   [ 1] .text PROGBITS
>> 8100  20 53129a  00  AX  0   0 4096
>>   [ 2] .notes  NOTE
>> 8153129c  73129c 0001d8  00  AX  0   0  4
>>   [ 3] __ex_table   PROGBITS81531480
>>731480 002018  00   A  0   0  8
>>   [ 4] .rodata PROGBITS
>> 8160  80 1655ee 00   A  0   0 64
>>   [ 5] __bug_table PROGBITS817

Re: Questions about Kernel Memory that I didn't find answers in Google - Please Help

[Peter Teoh]

I like your curiosities and interests in Linux kernel.
http://virtuallyhyper.com/2013/07/rhcsa-and-rhce-chapter-10-the-kernel/

Instead of answering one by one, I think I will just identify the knowledge
you are lacking:

Memory management (from both x86/intel and linux kernel perspective).

There are many many resources out there for you in these area, eg:

http://en.wikipedia.org/wiki/Page_table
http://en.wikipedia.org/wiki/X86-64

(both boring, but just understand it well enough)

http://wiki.osdev.org/Paging   (good explanationunderstand it very very
well).

The ultimate classic ebook:

https://www.kernel.org/doc/gorman/pdf/understand.pdf

And this blog site has tons of good info on intel/memory etc:

http://duartes.org/gustavo/blog/post/cpu-rings-privilege-and-protection/
http://duartes.org/gustavo/blog/post/anatomy-of-a-program-in-memory/

http://virtuallyhyper.com/2013/07/rhcsa-and-rhce-chapter-10-the-kernel/

http://www.cse.psu.edu/~anand/spring01/linux/memory.ppt

One more thing:

"readelf -S -W vmlinux" shows u the sections and the address where the
different sections are supposed to be loaded in memory.   If u replace the
vmlinux with the kernel module, eg: ip_tables.ko, then it says:

starting at offset 0x328c blah blah

so the loaded address is with respect to ZERO, but then the actual module
address is:

sudo cat /proc/modules |grep ip_table

ip_tables 18106 1 iptable_filter, Live 0xf8bf5000

So all the output from your readelf, just add 0xf8bf5000 to it and you will
get the actual virtual address of that section IN MEMORY.

Just only in memory.   In file, the file offset of the section is
different.   And many parts inside the ELF is also different from memory
too:   you will need to add the virtual load address (above) to the offset
as specified inside the relocation tables (objdump -r), and for each
section there is a separate relocation table (all independent from another,
meaning that the different section CAN BE loaded to different parts in
memory).

Thanks.


On Sun, Aug 3, 2014 at 11:59 PM, Lucas Tanure  wrote:

> Hi,
>
> I'm looking for some site, pdf, book etc, that can answer this questions.
> For now I have :
>
> http://unix.stackexchange.com/questions/5124/what-does-the-virtual-kernel-memory-layout-in-dmesg-imply
>
>
> I want to understand a few things about the memory and the execution
> of Linux kernel.
> Taking from a X86 and grub I have:
>
> 1) Grub loads kernel and root file system in memory, and the vmlinux
> has the code to decompress it self, right ? linux
>
> 2) The address of load kernel is always the same ? And It's at
> compilation time that is chosen ?
>
> 2a) The kernel takes places in 3g-4g memory place, and user space from 0
> to 3gb.
> But if the pc has only 256mb of memory ?
> And when pc has 16gb of memory, the user space will be split in two ?
>
> 2b) And if kernel has soo many modules that needs more than 1gb to run ?
>
> 2c) How we configure all of that memory configs ? make menuconfig and
> friends ?
>
> 3) The function A will call functon B. B is at 0xGG in .text
> section, but kernel was loaded in address 0xJJ, how A will
> find B ?
>
> 4) Please consider this:
> $ readelf -S -W vmlinux
> There are 37 section headers, starting at offset 0xe05718:
>
> Section Headers:
>   [Nr] Name   Type  Address
> Off Size  ES Flg Lk Inf Al
>   [ 0]  NULL
> 00  00 00  0   0  0
>   [ 1] .text PROGBITS
> 8100  20 53129a  00  AX  0   0 4096
>   [ 2] .notes  NOTE
> 8153129c  73129c 0001d8  00  AX  0   0  4
>   [ 3] __ex_table   PROGBITS81531480
>731480 002018  00   A  0   0  8
>   [ 4] .rodata PROGBITS
> 8160  80 1655ee 00   A  0   0 64
>   [ 5] __bug_table PROGBITS817655f0
>9655f0  005424 00   A  0   0  1
>   [ 6] .pci_fixup PROGBITS8176aa18
>  96aa18 002f88  00   A  0   0  8
>   [ 7] .tracedataPROGBITS8176d9a0
> 96d9a0 3c 00   A  0   0  1
>   [ 8] __ksymtab   PROGBITS8176d9e0
>   96d9e0 00e710 00   A  0   0 16
>   [ 9] __ksymtab_gpl PROGBITS8177c0f0
> 97c0f0  00a150  00   A  0   0 16
>   [10] __kcrctab PROGBITS81786240
>986240 007388 00   A  0   0  8
>   [11] __kcrctab_gpl  PROGBITSfff

Questions about Kernel Memory that I didn't find answers in Google - Please Help

[Lucas Tanure]

Hi,

I'm looking for some site, pdf, book etc, that can answer this questions.
For now I have :
http://unix.stackexchange.com/questions/5124/what-does-the-virtual-kernel-memory-layout-in-dmesg-imply


I want to understand a few things about the memory and the execution
of Linux kernel.
Taking from a X86 and grub I have:

1) Grub loads kernel and root file system in memory, and the vmlinux
has the code to decompress it self, right ? linux

2) The address of load kernel is always the same ? And It's at
compilation time that is chosen ?

2a) The kernel takes places in 3g-4g memory place, and user space from 0 to 3gb.
But if the pc has only 256mb of memory ?
And when pc has 16gb of memory, the user space will be split in two ?

2b) And if kernel has soo many modules that needs more than 1gb to run ?

2c) How we configure all of that memory configs ? make menuconfig and friends ?

3) The function A will call functon B. B is at 0xGG in .text
section, but kernel was loaded in address 0xJJ, how A will
find B ?

4) Please consider this:
$ readelf -S -W vmlinux
There are 37 section headers, starting at offset 0xe05718:

Section Headers:
  [Nr] Name   Type  Address
Off Size  ES Flg Lk Inf Al
  [ 0]  NULL
00  00 00  0   0  0
  [ 1] .text PROGBITS
8100  20 53129a  00  AX  0   0 4096
  [ 2] .notes  NOTE
8153129c  73129c 0001d8  00  AX  0   0  4
  [ 3] __ex_table   PROGBITS81531480
   731480 002018  00   A  0   0  8
  [ 4] .rodata PROGBITS
8160  80 1655ee 00   A  0   0 64
  [ 5] __bug_table PROGBITS817655f0
   9655f0  005424 00   A  0   0  1
  [ 6] .pci_fixup PROGBITS8176aa18
 96aa18 002f88  00   A  0   0  8
  [ 7] .tracedataPROGBITS8176d9a0
96d9a0 3c 00   A  0   0  1
  [ 8] __ksymtab   PROGBITS8176d9e0
  96d9e0 00e710 00   A  0   0 16
  [ 9] __ksymtab_gpl PROGBITS8177c0f0
97c0f0  00a150  00   A  0   0 16
  [10] __kcrctab PROGBITS81786240
   986240 007388 00   A  0   0  8
  [11] __kcrctab_gpl  PROGBITS8178d5c8
 98d5c8 0050a8 00   A  0   0  8
  [12] __ksymtab_strings  PROGBITS81792670
 992670 01cb42   00   A  0   0  1
  [13] __init_rodata   PROGBITS817af1c0
   9af1c0  e8   00   A  0   0 32
  [14] __param  PROGBITS817af2a8
9af2a8 000b00   00   A  0   0  8
  [15] __modverPROGBITS817afda8
   9afda8 000258   00   A  0   0  8
  [16] .dataPROGBITS
8180  a0 0e1180   00  WA  0   0 4096
  [17] .vvarPROGBITS
818e2000  ae2000 001000   00  WA  0   0 16
  [18] .data..percpu   PROGBITS
c0 015300   00  WA  0   0 4096
  [19] .init.text   PROGBITS
818f9000   cf9000  0503ea   00  AX  0   0 16
  [20] .init.data  PROGBITS
8194a000   d4a00009e4c8   00  WA  0   0 4096
  [21] .x86_cpu_dev.initPROGBITS819e84c8
de84c818   00   A  0   0  8
  [22] .parainstructions PROGBITS819e84e0
 de84e000bd3c   00   A  0   0  8
  [23] .altinstructionsPROGBITS819f4220
df4220 005f40   00   A  0   0  1
  [24] .altinstr_replacement  PROGBITS819fa160
  dfa160 001a69   00  AX  0   0  1
  [25] .iommu_table  PROGBITS819fbbd0
 dfbbd0 f0   00   A  0   0  8
  [26] .apicdrivers PROGBITS819fbcc0
 dfbcc0 20   00  WA  0   0  8
  [27] .exit.text PROGBITS819fbce0
   dfbce0 0009bc   00  AX  0   0  1
  [28] .smp_locks  PROGBITS819fd000
dfd000005000   00   A  0   0  4
  [29] .data_nosave  PROGBITS81a02000
 e02000001000   00  WA  0   0  4
  [30] .bss NOBITS
81a03000e03000122000   00  WA  0   0 4096
  [31] .brk  NOBITS
81b25000   e03000425000   00  WA  0   0  1
  [32] .comment   PROGBITS
e0300027   01  MS  0   0  1
  [33] .d

Re: Kernel memory leak

[Anil Shashikumar Belur]

On Saturday 28 June 2014 12:29 PM, Santhosh Kumar wrote:
>
> Is there a way to trace the allocations of memory from different buckets of
> kmalloc ?
You could try out kmemleak -
https://www.kernel.org/doc/Documentation/kmemleak.txt

___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Kernel memory leak

[Santhosh Kumar]

I am suspecting a memory leak in a kernel module that does layer 2
switching of data packets.

In the vmstat -m output the Num and Total keeps going up for the
kmalloc-512..

kmalloc-512   15232  15264512 16

Is this a clear indication of leak in the kernel or can there be false
positives ?

Is there a way to trace the allocations of memory from different buckets of
kmalloc ?

-Santhosh.
___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re:Re: how to detect kernel memory overflow ?

[RS]

Hello


em, this can protect the stack, so what about the memory buffer allocated 
through the kmalloc or vmalloc ?


Thanks,
HeChuan

At 2014-05-29 12:01:10,valdis.kletni...@vt.edu wrote:
>On Thu, 29 May 2014 11:13:09 +0800, RS said:
>> How to detect the kernel memory overflow errors?
>
>With a sufficiently recent gcc, you can build the kernel
>with CONFIG_CC_STACKPROTECTOR=y which will put a canary
>value on the stack and check it for corruption.
___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: how to detect kernel memory overflow ?

[Valdis . Kletnieks]

On Thu, 29 May 2014 11:13:09 +0800, RS said:
> How to detect the kernel memory overflow errors?

With a sufficiently recent gcc, you can build the kernel
with CONFIG_CC_STACKPROTECTOR=y which will put a canary
value on the stack and check it for corruption.


pgpsJAUnjB0ke.pgp
Description: PGP signature
___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

how to detect kernel memory overflow ?

[RS]

Hello


How to detect the kernel memory overflow errors? 
There are many tools to detect the user mode program memory problems, like the 
memcheck(Valgrind),  is there any tools for the kernel ?
The Kmemcheck detects some uses of uninitialized memory, can not detect the 
overflow errors.


Thanks,
HeChuan___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

An idea to capture kernel memory access

[RS]

Hello
I have an idea, to add some changes to the kernel, like the kmemcheck, to help 
to check the kernel memory.


 I call it kernel_module_check_framework, it can check the memory buffer 
overflow errors and others.
The memory buffer is what the user want to monitor, not the whole system's 
memory. User can add/delete the memory buffers to the framework.
The framework provide four interface, register/unregister functions, add/delete 
functions. User can utilize the interface to do some works.


When user add/delete a memory buffer, the framework will store the memory 
information and set the all the pages which contains the memory buffer 
none-present .
Then, when an access to this page, the framework will check whether the access 
is in the monitored buffers. If it is hit, the framework will set the page 
present and execute the interface function(or the hook function), at last 
change the regs->flags to make the CPU to be the single step debugging mode. If 
not, let the kernel to handle it.
Cause the single step debugging  mode, kernel will step into the do_debug 
functions in the traps.c file, and make the page none-present again, at last 
restore the regs->flags.
There, the framework can catch the another access in the same page.
When unregister, the framework will recovery all the pages, and show something.


As mentioned above, the interface function(or the hook function), it is a 
function pointer, at first is NULL, when the user register to the framework, 
the pointer will change to the user's handler. So, the framework's main handler 
is implemented by users.


To implement the framework, I will change the fault.c, traps.c and will add new 
files. It sounds like the kmemcheck, but not the same, my framework intents to 
capture each access in the memory buffers that are dynamically added or deleted 
by users, and let user to handler it. For example, the user can write a module 
to monitor a process's specified memory buffers with the framework, can 
statistic the buffer write,read times and somethings. Or user can develop a 
module to check memory access overflow errors with it.


I don't know whether the design is feasible?  Any one can give some advises?


Thanks, 
HeChuan___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Kernel Memory Growth

[Pietro Paolini]

Hello everyone,

I am experiencing a problem using my Linux 2.6.33 and doing this networking 
test using ab, a tool from Apache which 
helps me to benchmark the performances of a website, actually I am using that 
as a tool for generate a lot of traffic and 
TCP connections.

My topology is :

[AB PC] <==> [LINUX] <==> [WEBSITE]

I run ab like that 

ab -c 300 -n 2 http://10.0.0.103/index.html

and my problem is that after some time my system will reboot to a low memory 
condition, this is caused by the watchdog  I have installed in my system, I 
tried to monitor my applications but I did not see any growth in the memory
usage then I started to investigate if something is happening in the Kernel, 
using slab top I have got:

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME   
 33392  33392 100%1.00K   83484 33392K size-1024
  3716   3716 100%4.00K   37161 14864K size-4096
 34238  33899  99%0.20K   1802   19  7208K skbuff_head_cache
   547538  98%8.00K5471  4376K size-8192
 12488  12488 100%0.27K892   14  3568K nf_conntrack_c0ca4b0c
 12990  12990 100%0.12K433   30  1732K size-128   
  2211   2211 100%0.34K201   11   804K inode_cache
 11387  11368  99%0.06K193   59   772K size-64   
41 41 100%   16.00K 411   656K size-16384 
 4  4 100%  128.00K  41   512K size-131072
  2782   2702  97%0.15K107   26   428K dentry 
  4982   4957  99%0.07K 94   53   376K sysfs_dir_cache
  8814   8558  97%0.03K 78  113   312K size-32
   400400 100%0.50K 508   200K size-512
  1540   1255  81%0.11K 44   35   176K vm_area_struct
   432423  97%0.30K 36   12   144K radix_tree_node

The first line 
 33392  33392 100%1.00K   83484 33392K size-1024

tell me that size-1024 is using quite a lot of memory, how can I understand who 
is using that memory ?

Thanks in advance,
Pietro

___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Pass through kernel memory manager

[Peter Teoh]

the parameter you passed in section start looks weird, given that your
physical memory so limited.   (8K and 128K, 2 different bank? if so then
only one is available at any one time?),

Perhaps some knowledge about linker-script should help:

http://blogs.bu.edu/md/2011/11/15/the-dark-art-of-linker-scripts/

the "1:1" mapping is called identity mapping, and linker script provide a
way for you to load the binary into specific part of the physical memory,



On Sat, Feb 8, 2014 at 4:29 AM, Paul Chavent  wrote:

> Hi
>
> I'm working on an ARM926EJS based SOM (OMAPL138). The ARM has internal
> memory spaces (8k one and 128k one) where i would like to put some code.
>
> I thought to use something like :
>
> void foobar (void) __attribute__ ((section ("bar")));
>
> Then link with
>
> -Wl,--section-start,bar=1000
>
>
> But the Linux loader fails to load this segment.
>
> So, is it worth to try to achieve to run code at desired position ?
>
> Is there any way to tell Linux to 1:1 map some physical regions to
> processes address space ? Perhaps the memmap= kernel parameter ?
>
> Thanks for your help.
>
> Paul.
>
>
> ___
> Kernelnewbies mailing list
> Kernelnewbies@kernelnewbies.org
> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>



-- 
Regards,
Peter Teoh
___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Pass through kernel memory manager

[Paul Chavent]

Hi

I'm working on an ARM926EJS based SOM (OMAPL138). The ARM has internal memory 
spaces (8k one and 128k one) where i would like to put some code.

I thought to use something like :

void foobar (void) __attribute__ ((section ("bar")));

Then link with

-Wl,--section-start,bar=1000


But the Linux loader fails to load this segment.

So, is it worth to try to achieve to run code at desired position ?

Is there any way to tell Linux to 1:1 map some physical regions to processes 
address space ? Perhaps the memmap= kernel parameter ?

Thanks for your help.

Paul.


___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Kernel Memory management

[buyitian]

The answer from manty is totally wrong. 

I suggest that you put real question here.


在 2013年11月18日，15:17，"manty kuma"  写道：

Here is an interesting question(not mine) in SO related to Kernel memory 
management. Most of the points are my questions aswell. It needs ex[ert 
comments. could we try to answer questions posted there.

http://stackoverflow.com/questions/20041212/does-virtual-memory-area-struct-only-comes-into-picture-when-there-is-a-page-fau
___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Kernel Memory management

[manty kuma]

Here is an interesting question(not mine) in SO related to Kernel memory
management. Most of the points are my questions aswell. It needs ex[ert
comments. could we try to answer questions posted there.

http://stackoverflow.com/questions/20041212/does-virtual-memory-area-struct-only-comes-into-picture-when-there-is-a-page-fau
___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Why is kernel memory not pagable ?

[Anatol Pomozov]

Hi

On Mon, Oct 21, 2013 at 8:53 AM, Nav Kamal  wrote:
> Am very new to kernel programming and one of the concept says that kernel
> memory isn't pagable ? Can please someone explain its reason ?

If we try to page off kernel structures we might have a deadlock
trying to bring it back. Imagine you removed from memory for disk
drivers code text, or swapped data needed for paging algorithm. Now to
bring any data back you need the code that does not present in memory
and should be paged in -> deadlock.

___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Why is kernel memory not pagable ?

[Nav Kamal]

Am very new to kernel programming and one of the concept says that kernel
memory isn't pagable ? Can please someone explain its reason ?
___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: What rationale behind physical and virtual kernel memory layout ?

[Arun KS]

Hi Telenn,

On Sat, Oct 20, 2012 at 12:48 AM, telenn barz  wrote:

> Hello Arun,
>
> Thanks for answering.
>
> On Fri, Oct 19, 2012 at 3:33 PM, Arun KS  wrote:
>
>>
>> On Fri, Oct 19, 2012 at 3:03 PM, telenn barz wrote:
>>
>>>
>>> Q: In the 4G/4G split case, I don't see why we have to necessarily flush
>>> the TLB when switching from user-space to kernel-space ? Why the TLB
>>> couldn't be shared across one user-space to kernel-space switch, and be
>>> flushed only every two switches ?
>>>
>>
>> In 4G/4G split, kernel space(KS) and user space(US) both have 4GB address
>> space. So there can be a same virtual address present in KS & US which
>> corresponds to two different physical addressess.
>> When switching from US to KS or vice versa, the pointer to page table has
>> to be changed in the MMU. So TLB which is just a cache for the page tables
>> has to be flushed aswell.
>>
>> Yes you're right.
>
> So considering the traditional 3G/1G split, another question is why the
> kernel restricts itself to a "window" of 128MB on high memory. Can we
> imagine rather that the kernel would simply allow itself to dynamically use
> some virtual pages of the 3G user-space part (while changing their access
> rights of course) ?
>
128MB  of kernel linear address space is for mapping high mem pages. This
is a temporary mapping, you have to unmap this 128MB to map another 128MB.
AFAIK you can change this configuration by changing your lowmem area,
vmalloc area and device io memory area which are mapped to the kernel
virtual space.


Thanks,
Arun

>
> Thanks,
> Telenn
>
___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: What rationale behind physical and virtual kernel memory layout ?

[telenn barz]

Hello Arun,

Thanks for answering.

On Fri, Oct 19, 2012 at 3:33 PM, Arun KS  wrote:

>
> On Fri, Oct 19, 2012 at 3:03 PM, telenn barz wrote:
>
>>
>> Q: In the 4G/4G split case, I don't see why we have to necessarily flush
>> the TLB when switching from user-space to kernel-space ? Why the TLB
>> couldn't be shared across one user-space to kernel-space switch, and be
>> flushed only every two switches ?
>>
>
> In 4G/4G split, kernel space(KS) and user space(US) both have 4GB address
> space. So there can be a same virtual address present in KS & US which
> corresponds to two different physical addressess.
> When switching from US to KS or vice versa, the pointer to page table has
> to be changed in the MMU. So TLB which is just a cache for the page tables
> has to be flushed aswell.
>
> Yes you're right.

So considering the traditional 3G/1G split, another question is why the
kernel restricts itself to a "window" of 128MB on high memory. Can we
imagine rather that the kernel would simply allow itself to dynamically use
some virtual pages of the 3G user-space part (while changing their access
rights of course) ?

Thanks,
Telenn
___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: What rationale behind physical and virtual kernel memory layout ?

[Arun KS]

Hi Telenn,

On Fri, Oct 19, 2012 at 3:03 PM, telenn barz  wrote:

> Hi all,
>
> This a vast topic. But I believe it's worthwhile exposing in greater
> detail the "Why this design has been chosen ?", before the "How this design
> has been implemented ?". And I think this is a common lack (or at least not
> enough developed) in documentations, even in the more outstanding of them.
>
> Let's try to be more precise :
>
> A] About the direct mapping of the first 896MB of the virtual kernel space
> to the first 896MB of the physical memory
>
> Thus the kernel is able to handle a big virtual memory area mapped to a
> zone of contiguous physical addresses. This is a need for some peripheral
> devices which are unable to deal with paged memory. So all right, I can
> understand that (ref: "Avoiding - and fixing - memory 
> fragmentation"
> at lwn.net). But this article also says this is required for large kernel
> data structures...
>
> Q: ...Here I don't get the point : why wouldn't it be possible for the
> kernel to handle its structures through paged memory not necessarily
> physically contiguous ?
>
> In the bible "Understanding the Linux Kernel" third edition, at section
> "8.3 Non-contiguous Memory Area Management" it is written : "[...] it is
> preferable to map areas into sets of contiguous page frames, thus making
> better use of the cache and achieving lower average memory access times".
>
> Q: Is this assertion still valid on modern architecture ? Can please
> someone explain in further detail the theory behind it, or point me to
> relevant documentations ?
>
> Q: Do you see any other reason for this physically-contiguous memory
> requirement ?
>
> B] About the 3G/1G split of process' virtual address space on 32-bit
> architecture
>
> The article "Virtual Memory I: the problem"
> at lwn.net says it has to see with the TLB : by sharing the process'
> virtual space, we also share the TLB and thus we avoid a costly TLB flush
> at each user-space to kernel-space switch. The article invokes this reason
> of degradation performance when talking about the 4G/4G split patch (from
> Ingo Molnar).
>
> Q: In the 4G/4G split case, I don't see why we have to necessarily flush
> the TLB when switching from user-space to kernel-space ? Why the TLB
> couldn't be shared across one user-space to kernel-space switch, and be
> flushed only every two switches ?
>

In 4G/4G split, kernel space(KS) and user space(US) both have 4GB address
space. So there can be a same virtual address present in KS & US which
corresponds to two different physical addressess.
When switching from US to KS or vice versa, the pointer to page table has
to be changed in the MMU. So TLB which is just a cache for the page tables
has to be flushed aswell.

thanks,
Arun


>
> The TLB is small, Ok... but is it really a matter of TLB size ? I mean the
> kernel can rely on big pages for backing its virtual space : this is
> already the case for Intel Pentium (ref: "Understanding the Linux Kernel"
> third edition) which uses 4MB pages, or for Freescale e500v2/e500mc family
> which uses 256MB pages. On this architecture, if it had to map 4GB of
> physical memory, it could even use a single huge page, thus occupying a
> single page table entry, and so a single TLB entry...
>
> Thanks to anyone for bringing a gentle breeze of cleverness on my cloudy
> brain,
> Telenn
>
> ___
> Kernelnewbies mailing list
> Kernelnewbies@kernelnewbies.org
> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>
>
___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

What rationale behind physical and virtual kernel memory layout ?

[telenn barz]

Hi all,

This a vast topic. But I believe it's worthwhile exposing in greater detail
the "Why this design has been chosen ?", before the "How this design has
been implemented ?". And I think this is a common lack (or at least not
enough developed) in documentations, even in the more outstanding of them.

Let's try to be more precise :

A] About the direct mapping of the first 896MB of the virtual kernel space
to the first 896MB of the physical memory

Thus the kernel is able to handle a big virtual memory area mapped to a
zone of contiguous physical addresses. This is a need for some peripheral
devices which are unable to deal with paged memory. So all right, I can
understand that (ref: "Avoiding - and fixing - memory
fragmentation"
at lwn.net). But this article also says this is required for large kernel
data structures...

Q: ...Here I don't get the point : why wouldn't it be possible for the
kernel to handle its structures through paged memory not necessarily
physically contiguous ?

In the bible "Understanding the Linux Kernel" third edition, at section
"8.3 Non-contiguous Memory Area Management" it is written : "[...] it is
preferable to map areas into sets of contiguous page frames, thus making
better use of the cache and achieving lower average memory access times".

Q: Is this assertion still valid on modern architecture ? Can please
someone explain in further detail the theory behind it, or point me to
relevant documentations ?

Q: Do you see any other reason for this physically-contiguous memory
requirement ?

B] About the 3G/1G split of process' virtual address space on 32-bit
architecture

The article "Virtual Memory I: the problem "
at lwn.net says it has to see with the TLB : by sharing the process'
virtual space, we also share the TLB and thus we avoid a costly TLB flush
at each user-space to kernel-space switch. The article invokes this reason
of degradation performance when talking about the 4G/4G split patch (from
Ingo Molnar).

Q: In the 4G/4G split case, I don't see why we have to necessarily flush
the TLB when switching from user-space to kernel-space ? Why the TLB
couldn't be shared across one user-space to kernel-space switch, and be
flushed only every two switches ?

The TLB is small, Ok... but is it really a matter of TLB size ? I mean the
kernel can rely on big pages for backing its virtual space : this is
already the case for Intel Pentium (ref: "Understanding the Linux Kernel"
third edition) which uses 4MB pages, or for Freescale e500v2/e500mc family
which uses 256MB pages. On this architecture, if it had to map 4GB of
physical memory, it could even use a single huge page, thus occupying a
single page table entry, and so a single TLB entry...

Thanks to anyone for bringing a gentle breeze of cleverness on my cloudy
brain,
Telenn
___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Kernel Memory

[Vijay Chauhan]

Thanks everyone. I think i got enough information for further study.

Thanks,
Vijay

On Thu, Jun 21, 2012 at 7:10 PM, kishore sheik ahamed
 wrote:
> Hey Vijay
>
> I am a newbie too. Just sharing what I could go through.
>
> It is said that Kernel or atleast a part of kernel needs to be non paged for
> fast interrupt access etc as pinned memory
> Wiki says
>
> Pinned/Locked/Fixed pages
>
> Operating systems have memory areas that are pinned (never swapped to
> secondary storage). For example, interrupt mechanisms rely on an array of
> pointers to their handlers, such as I/O completion and page fault. If the
> pages containing these pointers or the code that they invoke were pageable,
> interrupt-handling would become far more complex and time-consuming,
> particularly in the case of page fault interrupts. Hence, some part of the
> page table structures is not pageable.
>
> Some pages may be pinned for short periods of time, others may be pinned for
> long periods of time, and still others may need to be permanently pinned.
> For example:
>
> The paging supervisor code and drivers for secondary storage devices on
> which pages reside must be permanently pinned, as otherwise paging wouldn't
> even work because the necessary code wouldn't be available.
> Timing-dependent components may be pinned to avoid variable paging delays.
> Data buffers that are accessed directly by peripheral devices that use
> direct memory access or I/O channels must reside in pinned pages while the
> I/O operation is in progress because such devices and the buses to which
> they are attached expect to find data buffers located at physical memory
> addresses; regardless of whether the bus has a memory management unit for
> I/O, transfers cannot be stopped if a page fault occurs and then restarted
> when the page fault has been processed.
>
> There are other two discussion thread which say kernel is non-pageable and
> now due to growing kernel Data structures it is allowed
>
> http://kerneltrap.org/node/6404
>
> http://kerneltrap.org/node/8206
>
>
> Regards
>
> Kishore
>
>
>
> On Thu, Jun 21, 2012 at 5:57 PM, Vijay Chauhan 
> wrote:
>>
>> Hello,
>>
>> I am newbie.
>> It has been said "kernel memory is not pageable"
>> What does it mean? There is no concept of kernel virtual address?
>>
>> Any simple explanation will help me to udnerstand.
>>
>> Thanks,
>> Vijay
>>
>> ___
>> Kernelnewbies mailing list
>> Kernelnewbies@kernelnewbies.org
>> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>
>

___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Kernel Memory

[michi1]

Hi!

On 17:57 Thu 21 Jun , Vijay Chauhan wrote:
> Hello,
> 
> I am newbie.
> It has been said "kernel memory is not pageable"
> What does it mean? There is no concept of kernel virtual address?
> 
> Any simple explanation will help me to udnerstand.

The right term is actually "kernel memory is not swappable". Swapping means
writing inactive memory to disk and then using it for something else. Kernel
memory not being swappable is a design decicion made in the early linux days.
Operating systems which swap kernel memory need to isolate everything which
should not be swappd out (e.g. things needed for swap-in, realtime stuff,
security sensitive data, ...). This is quite a bit of work. I also guess it is
pretty pointless nowadays. Installed memory and is getting so huge that virtual
memory developers have a hard time trying to keep cpu-usage overhead for
swapping user space memory low.

> There is no concept of kernel virtual address?

Kernel memory uses virtual addresses as well. However, these the entire system
memory is continuously mapped somewhere in the virtual address space. The
drawback is that fragmentation turns allocation of large continuous memory
regions into a game of luck.

There is also an virtual address area (vmalloc) which is used to dynamically
map multiple scattered pages to a continuous region. But this is rather slow
and rarely used.

You might want to take a look at: http://lwn.net/Kernel/LDD3/

-Michi
-- 
programing a layer 3+4 network protocol for mesh networks
see http://michaelblizek.twilightparadox.com

___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Kernel Memory

[AFT]

Vijay Chauhan  writes:

> Hello,
>
> I am newbie.
> It has been said "kernel memory is not pageable"
> What does it mean? There is no concept of kernel virtual address?
>

Yes. Kernel works on static adress space.

> Any simple explanation will help me to udnerstand.
>

I'm not sure if you want to understand "how kernel manages memory for
its internal DS". If its the case you should read the following
documents.

1) Read the chapter 8 of Linux Device driver 3rd edition.

http://lwn.net/Kernel/LDD3/

2) To understand slab allocator read the following papers by bonwick

   a) 94 paper describing slab allocator:
   
   
http://static.usenix.org/publications/library/proceedings/bos94/full_papers/bonwick.a

   b) Its followup in 2001
   
   
http://static.usenix.org/event/usenix01/full_papers/bonwick/bonwick_html/index.html

3) These should be enough. But if you want to know detailed architecture
of how virtual memory manager work you should read Gorman's book on
Linux virtual memory manager. Its a free pdf. can be found here:

http://ptgmedia.pearsoncmg.com/images/0131453483/downloads/gorman_book.pdf

Happy hacking.

Cheers
aft


> Thanks,
> Vijay
>
> ___
> Kernelnewbies mailing list
> Kernelnewbies@kernelnewbies.org
> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Kernel Memory

[Arun KS]

Hello Vijay,

On Thu, Jun 21, 2012 at 5:57 PM, Vijay Chauhan  wrote:
> Hello,
>
> I am newbie.
> It has been said "kernel memory is not pageable"
> What does it mean? There is no concept of kernel virtual address?
You might have heard about 3G/1G split. This 1GB is the virtual
address of the kernel. And whenever kernel try to access any address
in this range(for ARM architecture it is  0xC000 - 0x),
there should not be any page fault. That means MMU should be able to
convert your virtual address to physical. This 1GB contains your IO
address, RAM address.

Paging is a mechanism OS uses to pull the data(in pagesizes) to and
fro between system RAM and secondary memory.
Kernel memory is not pageable. This means memory allocated for the
kernel will not be pagged out. If you try to access any memory in
kernel with out creating page tables(this can be done by ioremap) you
will end up in OOPS. The main reason of kernel not being swapable or
pageable is as follows.
Think this way. What will happen if we have paged out that portion of
the logic which decides what to do when a page fault occurs? Who will
take care of the page fault then?

But if a user program hit a page fault(ie accessed address is not in
main memory), kernel will load the page from secondary memory if it is
a valid address. And if the address accesses is illegal, kernel kill
the user application(Segmentation fault).

Thanks,
Arun

>
> Any simple explanation will help me to udnerstand.
>
> Thanks,
> Vijay
>
> ___
> Kernelnewbies mailing list
> Kernelnewbies@kernelnewbies.org
> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Kernel Memory

[kishore sheik ahamed]

Hey Vijay

I am a newbie too. Just sharing what I could go through.

It is said that Kernel or atleast a part of kernel needs to be non paged
for fast interrupt access etc as pinned memory
Wiki says
Pinned/Locked/Fixed pages

Operating systems have memory areas that are pinned (never swapped to
secondary storage). For example,
interrupt<http://en.wikipedia.org/wiki/Interrupt>mechanisms rely on an
array of pointers to their handlers, such as
I/O <http://en.wikipedia.org/wiki/I/O> completion and page
fault<http://en.wikipedia.org/wiki/Page_fault>.
If the pages containing these pointers or the code that they invoke were
pageable, interrupt-handling would become far more complex and
time-consuming, particularly in the case of page fault interrupts. Hence,
some part of the page table structures is not pageable.

Some pages may be pinned for short periods of time, others may be pinned
for long periods of time, and still others may need to be permanently
pinned. For example:

   - The paging supervisor code and drivers for secondary storage devices
   on which pages reside must be permanently pinned, as otherwise paging
   wouldn't even work because the necessary code wouldn't be available.
   - Timing-dependent components may be pinned to avoid variable paging
   delays.
   - Data buffers <http://en.wikipedia.org/wiki/Data_buffer> that are
   accessed directly by peripheral devices that use direct memory
access<http://en.wikipedia.org/wiki/Direct_memory_access>or I/O
   channels <http://en.wikipedia.org/wiki/I/O_channel> must reside in
   pinned pages while the I/O operation is in progress because such devices
   and the buses <http://en.wikipedia.org/wiki/Bus_%28computing%29> to
   which they are attached expect to find data buffers located at physical
   memory addresses; regardless of whether the bus has a memory management
   unit for I/O <http://en.wikipedia.org/wiki/IOMMU>, transfers cannot be
   stopped if a page fault occurs and then restarted when the page fault has
   been processed.

There are other two discussion thread which say kernel is non-pageable and
now due to growing kernel Data structures it is allowed

http://kerneltrap.org/node/6404

http://kerneltrap.org/node/8206


Regards

Kishore


On Thu, Jun 21, 2012 at 5:57 PM, Vijay Chauhan wrote:

> Hello,
>
> I am newbie.
> It has been said "kernel memory is not pageable"
> What does it mean? There is no concept of kernel virtual address?
>
> Any simple explanation will help me to udnerstand.
>
> Thanks,
> Vijay
>
> ___
> Kernelnewbies mailing list
> Kernelnewbies@kernelnewbies.org
> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>
___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Kernel Memory

[HowKernel StuffWorks]

Hi,
  I am a kernel newbie too, Kernel memory is not pagable, because kernel
itself is responsible for paging. See this discussion,
http://kerneltrap.org/node/6404

  Paging happens for regular processes, i.e each process memory is divided
into a page of certain size(4kb in Linux), so it can swap for another page
that might be needed for that particular moment, it usually replaces an
page that have not been used for long time(see LRU,internal fragmentation).
Kernel is a process too, just like anyother process, but it differs from
others because it directly talks to the hardware and also it is the one
which takes care of paging and LRU algo's. So it does not make much sense,
for a kernel memory to be pageable, because, if it removes the page that
contains that has the paging algorithm, then there is  no way to come back,
you want to retrieve a page from disk_swap_area, but you cannot, because
the page that contains code to retrieve the instructions are paged :)...
This is not just for paging, since kernel controls everything, it is not
advisable to put its own code in swap area(see, virtual memory, virtual
memory = physical memory + swap space).

BTW, i am a kernel newbie too, this is my basic understanding, please feel
free to correct it, if I am wrong

On Thu, Jun 21, 2012 at 8:27 AM, Vijay Chauhan wrote:

> Hello,
>
> I am newbie.
> It has been said "kernel memory is not pageable"
> What does it mean? There is no concept of kernel virtual address?
>
> Any simple explanation will help me to udnerstand.
>
> Thanks,
> Vijay
>
> ___
> Kernelnewbies mailing list
> Kernelnewbies@kernelnewbies.org
> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>
___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Kernel Memory

[Vijay Chauhan]

Hello,

I am newbie.
It has been said "kernel memory is not pageable"
What does it mean? There is no concept of kernel virtual address?

Any simple explanation will help me to udnerstand.

Thanks,
Vijay

___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: kernel memory allocation

[Dave Hylands]

Hi,

On Thu, Dec 22, 2011 at 6:33 PM, J.Hwan Kim  wrote:
> Hi, everyone
>
> How can I allocated contiguous kernel memory over 128MB ?
> When I use _get_free_pages() function, it returns error.
> I guess the memory size is greater than the amount which the function
> can allocate.

You can use bootmem to reserve the memory at boot time, and then use
the bootmem allocator to alloocate from that reserved memory/

There is also some new code called CMA (Contiguous Memory Allocator)
which hasn't hit the mainline yet, but it could also be used.

See: http://lwn.net/Articles/468044/ for further details.

-- 
Dave Hylands
Shuswap, BC, Canada
http://www.davehylands.com

___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

kernel memory allocation

[J.Hwan Kim]

Hi, everyone

How can I allocated contiguous kernel memory over 128MB ?
When I use _get_free_pages() function, it returns error.
I guess the memory size is greater than the amount which the function
can allocate.

Thanks in advance.

Best Regards,
J.Hwan Kim


___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: About kernel memory limit

[Geraint Yang]

Thank you !
I will try kmap/kunmap later.


On Fri, Nov 11, 2011 at 4:20 PM, Dave Hylands  wrote:

> Hi Geraint,
>
> On Thu, Nov 10, 2011 at 8:44 PM, Geraint Yang 
> wrote:
> > Hi Dave,
> > Thank you for your help !
> > Does it mean that I could use all of the memory my computer has? But one
> of
> > my classmates told me that kernel could only use 1G from a 4G
> > memory.computer...Is there anything I have misunderstood ?
>
> There is a certain amount (exact amount depends on the architecture
> and configuration) that can be accessed directly all of the time.
>
> Then there is high memory, which can be accessed using kmap/kunmap.
> See http://linux-mm.org/HighMemory
>
> If you have a 64-bit system, then you can access all of the memory
> directly.
>
> --
> Dave Hylands
> Shuswap, BC, Canada
> http://www.davehylands.com
>



-- 
Geraint Yang
Tsinghua University Department of Computer Science and Technology
___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: About kernel memory limit

[Dave Hylands]

Hi Geraint,

On Thu, Nov 10, 2011 at 8:44 PM, Geraint Yang  wrote:
> Hi Dave,
> Thank you for your help !
> Does it mean that I could use all of the memory my computer has? But one of
> my classmates told me that kernel could only use 1G from a 4G
> memory.computer...Is there anything I have misunderstood ?

There is a certain amount (exact amount depends on the architecture
and configuration) that can be accessed directly all of the time.

Then there is high memory, which can be accessed using kmap/kunmap.
See http://linux-mm.org/HighMemory

If you have a 64-bit system, then you can access all of the memory directly.

-- 
Dave Hylands
Shuswap, BC, Canada
http://www.davehylands.com

___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

RE: About kernel memory limit

[adheer chandravanshi]

Try this command:

dmesg | grep Memory

For  me the output is:
[0.00] Memory: 2007408k/2051636k available (4678k kernel code, 42932k 
reserved, 2124k data, 668k init, 1142332k highmem)

Look here for more details:
https://bbs.archlinux.org/viewtopic.php?id=14412

~ Adheer

From: geraint0...@gmail.com
Date: Fri, 11 Nov 2011 13:28:53 +0800
Subject: Re: About kernel memory limit
To: g...@uniserve.com
CC: dhyla...@gmail.com; kernelnewbies@kernelnewbies.org

Thanks. 
But I think that command 'free' just tell the memory used in kernel space and 
user space.It is still unknown to us that how much memory is used by kernel.



On Fri, Nov 11, 2011 at 1:15 PM, Dave Stevens  wrote:


Quoting Geraint Yang :






Hi Dave,

Thank you for your help !

Does it mean that I could use all of the memory my computer has? But one of

my classmates told me that kernel could only use 1G from a 4G

memory.computer...Is there anything I have misunderstood ?




I'm sitting in front of a Ubuntu box with 8G installed, uname -a shows:



Linux roger-System-Product-Name 3.0.0-12-generic #20-Ubuntu SMP Fri Oct 7 
14:56:25 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux



so kernel 3 and free: shows

 total   used   free sharedbuffers cached

Mem:   819250029076565284844  0 1620601915540

-/+ buffers/cache: 8300567362444

Swap:  7812092  07812092



so 2.9G of 8 in use



Dave












On Fri, Nov 11, 2011 at 4:58 AM, Dave Hylands  wrote:




Hi,



On Thu, Nov 10, 2011 at 6:23 AM, Geraint Yang 

wrote:

> Hi there,

> I am a newbie to Linux kernel programming. I am going to make a module

> which will cost much memory in kernel, I just want to know how much

> memory I can get by calling memory allocate API in kernel.



All of it.



>From kernel space, you can completely exhaust memory to the point of

making your system unusable.



>From kernel space you have vmalloc memory and kmalloc memory (plus a

couple other memory spaces). Depending on how things are configured,

it's possible to exhaust vmalloc memory even though there is memory

available to be kmalloc'd.



--

Dave Hylands

Shuswap, BC, Canada

http://www.davehylands.com










--

Geraint Yang

Tsinghua University Department of Computer Science and Technology










-- 

It is told that such are the aerodynamics and wing loading of the bumblebee 
that, in principle, it cannot fly...if all this be true...life among bumblebees 
must bear a remarkable resemblance to life in the United States.





-- John Kenneth Galbraith, in American Capitalism: The Concept of 
Countervailing Power







-- 
Geraint Yang 
Tsinghua University Department of Computer Science and Technology




___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies   
  ___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: About kernel memory limit

[Geraint Yang]

Thanks.
But I think that command 'free' just tell the memory used in kernel space
and user space.It is still unknown to us that how much memory is used by
kernel.



On Fri, Nov 11, 2011 at 1:15 PM, Dave Stevens  wrote:

> Quoting Geraint Yang :
>
>  Hi Dave,
>> Thank you for your help !
>> Does it mean that I could use all of the memory my computer has? But one
>> of
>> my classmates told me that kernel could only use 1G from a 4G
>> memory.computer...Is there anything I have misunderstood ?
>>
>
> I'm sitting in front of a Ubuntu box with 8G installed, uname -a shows:
>
> Linux roger-System-Product-Name 3.0.0-12-generic #20-Ubuntu SMP Fri Oct 7
> 14:56:25 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux
>
> so kernel 3 and free: shows
> total   used   free sharedbuffers cached
> Mem:   819250029076565284844  0 1620601915540
> -/+ buffers/cache: 8300567362444
> Swap:  7812092  07812092
>
> so 2.9G of 8 in use
>
> Dave
>
>
>
>
>
>>
>> On Fri, Nov 11, 2011 at 4:58 AM, Dave Hylands  wrote:
>>
>>  Hi,
>>>
>>> On Thu, Nov 10, 2011 at 6:23 AM, Geraint Yang 
>>> wrote:
>>> > Hi there,
>>> > I am a newbie to Linux kernel programming. I am going to make a module
>>> > which will cost much memory in kernel, I just want to know how much
>>> > memory I can get by calling memory allocate API in kernel.
>>>
>>> All of it.
>>>
>>> From kernel space, you can completely exhaust memory to the point of
>>> making your system unusable.
>>>
>>> From kernel space you have vmalloc memory and kmalloc memory (plus a
>>> couple other memory spaces). Depending on how things are configured,
>>> it's possible to exhaust vmalloc memory even though there is memory
>>> available to be kmalloc'd.
>>>
>>> --
>>> Dave Hylands
>>> Shuswap, BC, Canada
>>> http://www.davehylands.com
>>>
>>>
>>
>>
>> --
>> Geraint Yang
>> Tsinghua University Department of Computer Science and Technology
>>
>>
>
>
> --
> It is told that such are the aerodynamics and wing loading of the
> bumblebee that, in principle, it cannot fly...if all this be true...life
> among bumblebees must bear a remarkable resemblance to life in the United
> States.
>
> -- John Kenneth Galbraith, in American Capitalism: The Concept of
> Countervailing Power
>
>
>


-- 
Geraint Yang
Tsinghua University Department of Computer Science and Technology
___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: About kernel memory limit

[Dave Stevens]

Quoting Geraint Yang :

> Hi Dave,
> Thank you for your help !
> Does it mean that I could use all of the memory my computer has? But one of
> my classmates told me that kernel could only use 1G from a 4G
> memory.computer...Is there anything I have misunderstood ?

I'm sitting in front of a Ubuntu box with 8G installed, uname -a shows:

Linux roger-System-Product-Name 3.0.0-12-generic #20-Ubuntu SMP Fri  
Oct 7 14:56:25 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux

so kernel 3 and free: shows
  total   used   free sharedbuffers cached
Mem:   819250029076565284844  0 1620601915540
-/+ buffers/cache: 8300567362444
Swap:  7812092  07812092

so 2.9G of 8 in use

Dave



>
>
> On Fri, Nov 11, 2011 at 4:58 AM, Dave Hylands  wrote:
>
>> Hi,
>>
>> On Thu, Nov 10, 2011 at 6:23 AM, Geraint Yang 
>> wrote:
>> > Hi there,
>> > I am a newbie to Linux kernel programming. I am going to make a module
>> > which will cost much memory in kernel, I just want to know how much
>> > memory I can get by calling memory allocate API in kernel.
>>
>> All of it.
>>
>> From kernel space, you can completely exhaust memory to the point of
>> making your system unusable.
>>
>> From kernel space you have vmalloc memory and kmalloc memory (plus a
>> couple other memory spaces). Depending on how things are configured,
>> it's possible to exhaust vmalloc memory even though there is memory
>> available to be kmalloc'd.
>>
>> --
>> Dave Hylands
>> Shuswap, BC, Canada
>> http://www.davehylands.com
>>
>
>
>
> --
> Geraint Yang
> Tsinghua University Department of Computer Science and Technology
>



-- 
It is told that such are the aerodynamics and wing loading of the  
bumblebee that, in principle, it cannot fly...if all this be  
true...life among bumblebees must bear a remarkable resemblance to  
life in the United States.

-- John Kenneth Galbraith, in American Capitalism: The Concept of  
Countervailing Power



___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: About kernel memory limit

[Haojian Zhuang]

On Fri, Nov 11, 2011 at 12:44 PM, Geraint Yang  wrote:
> Hi Dave,
> Thank you for your help !
> Does it mean that I could use all of the memory my computer has? But one of
> my classmates told me that kernel could only use 1G from a 4G
> memory.computer...Is there anything I have misunderstood ?
>

Kernel can operate 896MB directly. If your physical memory is larger
than this memory size, you can use high memory. So kernel needs to map
them first, then use it. The mapped size is limited in 128MB at the
same time.

Thanks
Haojian

___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: About kernel memory limit

[Geraint Yang]

Hi Dave,
Thank you for your help !
Does it mean that I could use all of the memory my computer has? But one of
my classmates told me that kernel could only use 1G from a 4G
memory.computer...Is there anything I have misunderstood ?


On Fri, Nov 11, 2011 at 4:58 AM, Dave Hylands  wrote:

> Hi,
>
> On Thu, Nov 10, 2011 at 6:23 AM, Geraint Yang 
> wrote:
> > Hi there,
> > I am a newbie to Linux kernel programming. I am going to make a module
> > which will cost much memory in kernel, I just want to know how much
> > memory I can get by calling memory allocate API in kernel.
>
> All of it.
>
> From kernel space, you can completely exhaust memory to the point of
> making your system unusable.
>
> From kernel space you have vmalloc memory and kmalloc memory (plus a
> couple other memory spaces). Depending on how things are configured,
> it's possible to exhaust vmalloc memory even though there is memory
> available to be kmalloc'd.
>
> --
> Dave Hylands
> Shuswap, BC, Canada
> http://www.davehylands.com
>



-- 
Geraint Yang
Tsinghua University Department of Computer Science and Technology
___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: About kernel memory limit

[Dave Hylands]

Hi,

On Thu, Nov 10, 2011 at 6:23 AM, Geraint Yang  wrote:
> Hi there,
> I am a newbie to Linux kernel programming. I am going to make a module
> which will cost much memory in kernel, I just want to know how much
> memory I can get by calling memory allocate API in kernel.

All of it.

>From kernel space, you can completely exhaust memory to the point of
making your system unusable.

>From kernel space you have vmalloc memory and kmalloc memory (plus a
couple other memory spaces). Depending on how things are configured,
it's possible to exhaust vmalloc memory even though there is memory
available to be kmalloc'd.

-- 
Dave Hylands
Shuswap, BC, Canada
http://www.davehylands.com

___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

About kernel memory limit

[Geraint Yang]

Hi there,
I am a newbie to Linux kernel programming. I am going to make a module
which will cost much memory in kernel, I just want to know how much
memory I can get by calling memory allocate API in kernel.
Any help will be appreaciated.
___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Paging of Kernel Memory

[Rik van Riel]

On 09/07/2011 12:41 AM, Mulyadi Santosa wrote:
> Hi :)
>
> On Wed, Sep 7, 2011 at 09:44, ashish anand  wrote:
>> Hi
>> on wed 7th sep  Christopher Harvey wrote
 It means that it can't be swapped to your swap partition, even if
 you're not using it.
>>
>>   this thing I understood it pretty well but what about the line
>> "Therefore, every byte of
>>   memory you consume is one less byte of available physical memory".What is
>> the meaning of this line and why it is so.
>
> could you please next time cut out the unrelated message? :)
>
> Anyway, that passage means that the bigger your kernel image is, the
> lesser your free memory are. This is simply because your kernel image
> is entirely loaded and locked in RAM.

This was a concern back in the day where systems had 4MB of
RAM and the kernel took up 1MB.

However, nowadays the kernel takes up a few MB in a system
with several GB of memory, so it really no longer matters...

-- 
All rights reversed.

___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Paging of Kernel Memory

[Asutosh Das]

Hi
I think it also can mean that there is a 1-1 mapping in the kernel(except
for high mem) as there cannot be any page-faults in the kernel. So for
memory allocated in kernel, a physical memory mapping exists, thus lessening
the available physical memory.


On 6 September 2011 17:48, Vijay Chauhan  wrote:

> Hi,
>
> i was going through the Linux Kernel Development book. It mention the
> following:
>
> "Additionally, kernel memory is not pageable.Therefore, every byte of
> memory you consume is one less byte of available physical memory."
>
> What is the meaning of 'Kernel memory is not pageable'? Anyone can
> give details would help me in understanding.
>
> Thanks,
> Vijay
>
> ___
> Kernelnewbies mailing list
> Kernelnewbies@kernelnewbies.org
> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>



-- 
~/asd
___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Paging of Kernel Memory

[Mulyadi Santosa]

Hi :)

On Wed, Sep 7, 2011 at 09:44, ashish anand  wrote:
> Hi
> on wed 7th sep  Christopher Harvey wrote
>>>It means that it can't be swapped to your swap partition, even if
>>>you're not using it.
>
>  this thing I understood it pretty well but what about the line
> "Therefore, every byte of
>  memory you consume is one less byte of available physical memory".What is
> the meaning of this line and why it is so.

could you please next time cut out the unrelated message? :)

Anyway, that passage means that the bigger your kernel image is, the
lesser your free memory are. This is simply because your kernel image
is entirely loaded and locked in RAM.

-- 
regards,

Mulyadi Santosa
Freelance Linux trainer and consultant

blog: the-hydra.blogspot.com
training: mulyaditraining.blogspot.com

___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Paging of Kernel Memory

[ashish anand]

Hi
on wed 7th sep  Christopher Harvey wrote

>>It means that it can't be swapped to your swap partition, even if
>>you're not using it.

 this thing I understood it pretty well but what about the line
"Therefore, every byte of
 memory you consume is one less byte of available physical memory".What is
the meaning of this line and why it is so.


On Tue, Sep 6, 2011 at 9:30 PM, wrote:

> Send Kernelnewbies mailing list submissions to
>kernelnewbies@kernelnewbies.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
>http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
> or, via email, send a message with subject or body 'help' to
>kernelnewbies-requ...@kernelnewbies.org
>
> You can reach the person managing the list at
>kernelnewbies-ow...@kernelnewbies.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Kernelnewbies digest..."
>
>
> Today's Topics:
>
>   1. Re: Re: Regarding threaded irq and normal irq (anish singh)
>   2. CRC calculation in Module.symvers (Shakthi Kannan)
>   3. Re: Finding physical and logical core number in a kernel
>  module ? (Alexandre Courbot)
>   4. How to identify Connected USB Storage device (Harddisk, flash
>  Memoryor SSD) (mani)
>   5. Paging of Kernel Memory (Vijay Chauhan)
>   6. Re: Paging of Kernel Memory (Christopher Harvey)
>
>
> --
>
> Message: 1
> Date: Tue, 6 Sep 2011 13:57:22 +0530
> From: anish singh 
> Subject: Re: Re: Regarding threaded irq and normal irq
> To: Mulyadi Santosa 
> Cc: Greg KH , t...@linutronix.de,   sandeep kumar
>, kernelnewbies@kernelnewbies.org
> Message-ID:
> >
> Content-Type: text/plain; charset=UTF-8
>
> On Tue, Sep 6, 2011 at 10:59 AM, anish singh
>  wrote:
> > On Tue, Sep 6, 2011 at 10:51 AM, Mulyadi Santosa
> >  wrote:
> >> Hi :)
> >>
> >> On 06/09/2011, sandeep kumar  wrote:
> >>> Hmm..But when i tried to take a mutex lock in threaded_irq, it is
> throwing a
> >>> warning message
> >>> "BUG: sleeping function called from invalid context"... So i was
> wondering
> >>> which way it is..
> >>
> >> Please don't top post :)
> >>
> >> Anyway, regarding that "BUG:" message, could it be that mutex function
> >> mistakenly assume it's called inside top half, thus bails out since
> > How can mutex assume? IMHO the message would have printed
> > because sleeping would be forbidden there but again AFAIK in the threaded
> > handler we can do I2C transactions which can sleep.So kind of
> > paradox there.
>
> >>something like using in_irq() inside the threaded irq handler?
> I think after going thorough the code of mutex_lock we can find out the
> reason why it is printing the warning eventhough we are calling it in a
> threaded handler i.e. kernel thread.
>
> >From the code:
> 126 /**
> 127  * might_sleep - annotation for functions that can sleep
> 128  *
> 129  * this macro will print a stack trace if it is executed in an atomic
> 130  * context (spinlock, irq-handler, ...).
>
> 129 line no is important.
> This might_sleep is called by mutex_lock function and in the defintion
> of __might_sleep  i found below code.I think this below if condition is
> false because irqs_disabled is true.This causes the warning.
>
> if ((preempt_count_equals(preempt_offset) && !irqs_disabled()) ||
> system_state != SYSTEM_RUNNING || oops_in_progress)
>   return;
>
> Hope i am right?
>
> > ps:I have seen the code where sandeep is doing the taking the mutex lock.
> > He is taking mutex lock at the beginging of the handler and releasing
> > at the end.
> >> sleeping is forbidden there? which mutex function do you use? (sorry I
> >> don't follow every locking functions exist now)
> >>
> >> --
> >> regards,
> >>
> >> Mulyadi Santosa
> >> Freelance Linux trainer and consultant
> >>
> >> blog: the-hydra.blogspot.com
> >> training: mulyaditraining.blogspot.com
> >>
> >> ___
> >> Kernelnewbies mailing list
> >> Kernelnewbies@kernelnewbies.org
> >> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
> >>
> >
>
>
>
> --
>
> Message: 2
> Date: Tue, 6 Sep 2011 12:41:24 +0530
> From: Shakthi Kannan 
> Sub

Re: Paging of Kernel Memory

[Christopher Harvey]

On Tue, 6 Sep 2011 17:48:12 +0530, Vijay Chauhan wrote:
> Hi,
>
> i was going through the Linux Kernel Development book. It mention the
> following:
>
> "Additionally, kernel memory is not pageable.Therefore, every byte of
> memory you consume is one less byte of available physical memory."
>
> What is the meaning of 'Kernel memory is not pageable'? Anyone can
> give details would help me in understanding.
>
> Thanks,
> Vijay
>
It means that it can't be swapped to your swap partition, even if 
you're not using it.

___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Paging of Kernel Memory

[Vijay Chauhan]

Hi,

i was going through the Linux Kernel Development book. It mention the following:

"Additionally, kernel memory is not pageable.Therefore, every byte of
memory you consume is one less byte of available physical memory."

What is the meaning of 'Kernel memory is not pageable'? Anyone can
give details would help me in understanding.

Thanks,
Vijay

___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Query regarding kernel memory

[Dave Hylands]

Hi Vaibhav,

On Mon, Jul 25, 2011 at 9:40 PM, Vaibhav Jain  wrote:
> Hi Dave, Santhosh,
>
> Thanks for the reply! I was talking about the following paragraph from the
> reference you provided :
>
> The kernel (on the x86 architecture, in the default configuration) splits
> the 4-GB virtual address space between user-space and the kernel; the same
> set of mappings is used in both contexts. A typical split dedicates 3 GB to
> user space, and 1 GB for kernel space.* The kernel’s code and data
> structures must fit into that space, but the biggest consumer of kernel
> address space is virtual mappings for physical memory. The
> kernel cannot directly manipulate memory that is not mapped into the
> kernel’s address space. The kernel, in other words, needs its own virtual
> address for any memory it must touch directly. Thus, for many years, the
> maximum amount of physical memory that could be handled by the kernel was
> the amount that could be mapped into the kernel’s portion of the virtual
> address space, minus the space needed for the kernel code itself. As a
> result, x86-based Linux systems could work with a maximum of a little under
> 1 GB of physical memory.
>
> I am still not clear about the sentences in bold. Why is  the space needed
> for kernel code subtracted from the amount that could be mapped
> into kernel's portion of virtual address space ?

Well the kernel code needs to run from the kernel virtual space.

> Also , what difference does
> the fact - "biggest consumer of kernel address space is virtual mappings for
> physical memory" make in the amount of memory that can be handled by the
> kernel.

In order to access memory you need to have MMU tables. These MMU
tables are what maps the virrtual to physical memory.

MMU entries are needed for every single page. I don't quite think that
they take up a huge amount of memory. There is also a page structure
for every page of physical memory.

-- 
Dave Hylands
Shuswap, BC, Canada
http://www.davehylands.com

___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Query regarding kernel memory

[Vaibhav Jain]

Hi Dave, Santhosh,

Thanks for the reply! I was talking about the following paragraph from the
reference you provided :

The kernel (on the x86 architecture, in the default configuration) splits
the 4-GB virtual address space between user-space and the kernel; the same
set of mappings is used in both contexts. A typical split dedicates 3 GB to
user space, and 1 GB for kernel space.* The kernel’s code and data
structures must fit into that space, but the biggest consumer of kernel
address space is virtual mappings for physical memory. The
kernel cannot directly manipulate memory that is not mapped into the
kernel’s address space. The kernel, in other words, needs its own virtual
address for any memory it must touch directly. *Thus, for many years, the
maximum amount of physical memory that could be handled by the kernel was
the amount that could be mapped into the kernel’s portion of the virtual
address space, minus the space needed for the kernel code itself. As a
result, x86-based Linux systems could work with a maximum of a little under
1 GB of physical memory.*

I am still not clear about the sentences in bold. Why is  the space needed
for kernel code subtracted from the amount that could be mapped
into kernel's portion of virtual address space ? Also , what difference does
the fact - "biggest consumer of kernel address space is virtual mappings for
physical memory" make in the amount of memory that can be handled by the
kernel.
I am little confused.

Thanks
Vaibhav Jain

On Mon, Jul 25, 2011 at 5:51 PM, Dave Hylands  wrote:

> Hi Vaibhav,
>
> On Mon, Jul 25, 2011 at 3:17 PM, Vaibhav Jain  wrote:
> > Hi,
> >
> > I read a few articles on linux virtual memory management such as this one
> :
> > http://lwn.net/Articles/75174/
> >
> > which say that earlier linux kernel could only use memory slightly below
> 1
> > GB. They have
> > given the reason for it but I am unable to understand.They further
> describe
> > the use of High memory and low memory.
> > Could anybody please explain the reason for kernel not being able to use
> the
> > 1 GB completely?
> > Also please provide references for high memory and low memory.
>
> My numbers/comments are for the ARM processor, the x86 may be slightly
> different.
>
> The typical configuration for the kernel has addresses from 0x
> through 0xC000 given to user space (it's actually a small amount
> less than 3 GB since modules are loaded in the space just before
> 0xC000).
>
> That leaves 0xC000 to 0x for kernel virtual memory (or 1
> Gb). Now devices need some I/O space, which takes away from the 1Gb.
>
> I think what you're calling low memory is kernel logical memory. See
> http://lwn.net/images/pdf/LDD3/ch15.pdf on page 414 (not the 414th
> page of the PDF, but the page with 414 printed on the bottom).
>
> High memory is memory which is not directly accessible by the kernel.
> You need to use kmap/kunmap to map the memory into the kernel virtual
> memory space. Low memory is always accessible by the kernel.
>
> So user-mode programs get allocated from high-memory (if high memory
> exists) since the kernel doesn't typically need to access the
> user-space memory.
>
> It is possible to set some CONFIG options and have the 3Gb/1Gb split
> changed to 2Gb/2Gb or 1Gb/3Gb, but 3Gb/1Gb is the normal default.
>
> --
> Dave Hylands
> Shuswap, BC, Canada
> http://www.davehylands.com
>
___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Query regarding kernel memory

[santhosh kumars]

hi,

On Tue, Jul 26, 2011 at 3:47 AM, Vaibhav Jain  wrote:
> Hi,
>
> I read a few articles on linux virtual memory management such as this one :
> http://lwn.net/Articles/75174/
>
> which say that earlier linux kernel could only use memory slightly below 1
> GB. They have
In Linux to separate the user space and kernel space,total linear
address range is divided into  two parts
1GB for kernel space and 3GB for user space.This not fixed you can
configure it to 2GB and 2GB also.
> given the reason for it but I am unable to understand.They further describe
> the use of High memory and low memory.
> Could anybody please explain the reason for kernel not being able to use the
> 1 GB completely?
The highest 128 MB of linear addresses are left available for several
kinds of mappings.So the kernel address space left
for mapping the RAM 1GB-128MB=896MB. During initialization phase
liinux maps RAM window of 896MB into the kernel
linear address space of 896 MB(its kind of one to one mapping for
which page table entries are fixed) .The other 128 MB of linear
addresses always left available because the kernel uses them to
implement non contiguous memory allocation and fix-mapped linear
address(for this range page table changes accordingly map and unmap ).
> Also please provide references for high memory and low memory.
please refer to the memory management chapter of understanding linux kernel.
>
>
> Thanks
> Vaibhav Jain
>
> ___
> Kernelnewbies mailing list
> Kernelnewbies@kernelnewbies.org
> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>
>

___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Query regarding kernel memory

[Dave Hylands]

Hi Vaibhav,

On Mon, Jul 25, 2011 at 3:17 PM, Vaibhav Jain  wrote:
> Hi,
>
> I read a few articles on linux virtual memory management such as this one :
> http://lwn.net/Articles/75174/
>
> which say that earlier linux kernel could only use memory slightly below 1
> GB. They have
> given the reason for it but I am unable to understand.They further describe
> the use of High memory and low memory.
> Could anybody please explain the reason for kernel not being able to use the
> 1 GB completely?
> Also please provide references for high memory and low memory.

My numbers/comments are for the ARM processor, the x86 may be slightly
different.

The typical configuration for the kernel has addresses from 0x
through 0xC000 given to user space (it's actually a small amount
less than 3 GB since modules are loaded in the space just before
0xC000).

That leaves 0xC000 to 0x for kernel virtual memory (or 1
Gb). Now devices need some I/O space, which takes away from the 1Gb.

I think what you're calling low memory is kernel logical memory. See
http://lwn.net/images/pdf/LDD3/ch15.pdf on page 414 (not the 414th
page of the PDF, but the page with 414 printed on the bottom).

High memory is memory which is not directly accessible by the kernel.
You need to use kmap/kunmap to map the memory into the kernel virtual
memory space. Low memory is always accessible by the kernel.

So user-mode programs get allocated from high-memory (if high memory
exists) since the kernel doesn't typically need to access the
user-space memory.

It is possible to set some CONFIG options and have the 3Gb/1Gb split
changed to 2Gb/2Gb or 1Gb/3Gb, but 3Gb/1Gb is the normal default.

-- 
Dave Hylands
Shuswap, BC, Canada
http://www.davehylands.com

___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Query regarding kernel memory

[Vaibhav Jain]

Hi,

I read a few articles on linux virtual memory management such as this one :
http://lwn.net/Articles/75174/

which say that earlier linux kernel could only use memory slightly below 1
GB. They have
given the reason for it but I am unable to understand.They further describe
the use of High memory and low memory.
Could anybody please explain the reason for kernel not being able to use the
1 GB completely?
Also please provide references for high memory and low memory.


Thanks
Vaibhav Jain
___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

RE: kernel memory allocation

[Jeff Haran]

> -Original Message-
> From: kernelnewbies-boun...@kernelnewbies.org [mailto:kernelnewbies-
> boun...@kernelnewbies.org] On Behalf Of João Eduardo Luís
> Sent: Friday, June 03, 2011 12:40 PM
> To: Amirali Shambayati
> Cc: kernelnewbies
> Subject: Re: kernel memory allocation
> 
> Hi.
> 
> In future replies, please CC the list.
> 
> Does it panic in during the kmalloc, or afterwards? Are you checking if
> 'newBun' is NULL?
> 
> I may be missing something obvious in that code, but unless you are out of
> memory or with some past corruption, I don't think that should panic the
> kernel.
> 
> In any case, the panic trace should help.
> 
> 
> Cheers.
> 
> ---
> João Eduardo Luís
> gpg key: 477C26E5 from pool.keyserver.eu
> 
> 
> 
> 
> 
> On Jun 3, 2011, at 7:50 PM, Amirali Shambayati wrote:
> 
> > thanks for your guidance. I just want to allocate memory for a struct. I use
> this:
> >
> > struct bundle* newBun;
> > newBun = kmalloc(sizeof(*newBun), GFP_KERNEL);
> >
> > but it goes to panic state.
> >

Just guessing, but the most likely reason for the above call to kmalloc() to 
cause a panic is it's being called in atomic context.

Are you seeing anything like "scheduling while atomic" in the panic back trace?

If so, that's your problem and the solution is most likely to replace the above 
GFP_KERNEL with GFP_ATOMIC. The alternative would be to restructure the code so 
that the allocation doesn't happen in atomic context.

Jeff Haran
Bytemobile




___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: kernel memory allocation

[João Eduardo Luís]

Hello,

Once again, I would pretty much enjoy if you CC'ed the list on replies.


On Jun 3, 2011, at 8:49 PM, Amirali Shambayati wrote:

> I set breakpoint before kmalloc. panic happens after kmalloc.
> 
>> 2011/6/4 Amirali Shambayati 
>> ofcourse it's null before allocate it. I make an instantiation of it. then  
>> I want to allocate memory for it.What else I should do?

I meant, are you checking if 'newBun' is NULL _AFTER_ the kmalloc?

The kmalloc call will return NULL instead of a pointer to newly allocated 
memory whenever an error occurs. If by any chance you are not checking if 
'newBun' is NULL after the kmalloc, then you should do it.

Also, if the panic happens _after_ the kmalloc, and not _during_ the kmalloc, I 
would say there's a slim chance of kmalloc being the cause of the panic.


---
João Eduardo Luís
gpg key: 477C26E5 from pool.keyserver.eu 

PGP.sig
Description: This is a digitally signed message part
___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: kernel memory allocation

[João Eduardo Luís]

Hi.

In future replies, please CC the list.

Does it panic in during the kmalloc, or afterwards? Are you checking if 
'newBun' is NULL?

I may be missing something obvious in that code, but unless you are out of 
memory or with some past corruption, I don't think that should panic the kernel.

In any case, the panic trace should help.


Cheers.

---
João Eduardo Luís
gpg key: 477C26E5 from pool.keyserver.eu 





On Jun 3, 2011, at 7:50 PM, Amirali Shambayati wrote:

> thanks for your guidance. I just want to allocate memory for a struct. I use 
> this: 
> 
> struct bundle* newBun;
> newBun = kmalloc(sizeof(*newBun), GFP_KERNEL);
> 
> but it goes to panic state.
> 
> 2011/6/3 João Eduardo Luís 
> Hi.
> 
> From [1] I'm lead to believe the only difference between the regular 
> kmalloc() arguments and those of kmalloc_node() is the one specifying which 
> node you want to allocate the memory on.
> 
> Aside from the third argument, which seems to be related with NUMA (with 
> which I never worked on kernel-context), I would suggest you to read [2].
> 
> If on the other hand you only want to allocate memory locally, maybe you 
> should use kmalloc() or one of its variants [3].
> 
> 
> Cheers.
> 
> 
> [1] - http://www.kernel.org/doc/htmldocs/kernel-api/API-kmalloc-node.html
> [2] - http://www.kernel.org/doc/htmldocs/kernel-api/API-kcalloc.html
> [3] - http://www.kernel.org/doc/htmldocs/kernel-api/mm.html#id408507
> ---
> João Eduardo Luís
> gpg key: 477C26E5 from pool.keyserver.eu
> 
> 
> 
> 
> 
> On Jun 3, 2011, at 7:07 PM, Amirali Shambayati wrote:
> 
> >
> > Hello all,
> >
> > I just want to allocate memory for a struct instantiation. Would anyone 
> > guide me what arguments I should pass to "kmalloc_node"?
> > Regards,
> > --
> > Amirali Shambayati
> > Bachelor Student
> > Computer Engineering Department
> > Sharif University of Technology
> > Tehran, Iran
> >
> > ___
> > Kernelnewbies mailing list
> > Kernelnewbies@kernelnewbies.org
> > http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
> 
> 
> 
> 
> -- 
> Amirali Shambayati
> Bachelor Student
> Computer Engineering Department
> Sharif University of Technology
> Tehran, Iran
> 



PGP.sig
Description: This is a digitally signed message part
___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: kernel memory allocation

[João Eduardo Luís]

Hi.

From [1] I'm lead to believe the only difference between the regular kmalloc() 
arguments and those of kmalloc_node() is the one specifying which node you want 
to allocate the memory on.

Aside from the third argument, which seems to be related with NUMA (with which 
I never worked on kernel-context), I would suggest you to read [2].

If on the other hand you only want to allocate memory locally, maybe you should 
use kmalloc() or one of its variants [3]. 


Cheers.


[1] - http://www.kernel.org/doc/htmldocs/kernel-api/API-kmalloc-node.html
[2] - http://www.kernel.org/doc/htmldocs/kernel-api/API-kcalloc.html
[3] - http://www.kernel.org/doc/htmldocs/kernel-api/mm.html#id408507
---
João Eduardo Luís
gpg key: 477C26E5 from pool.keyserver.eu 





On Jun 3, 2011, at 7:07 PM, Amirali Shambayati wrote:

> 
> Hello all,
> 
> I just want to allocate memory for a struct instantiation. Would anyone guide 
> me what arguments I should pass to "kmalloc_node"?
> Regards,
> -- 
> Amirali Shambayati
> Bachelor Student
> Computer Engineering Department
> Sharif University of Technology
> Tehran, Iran
> 
> ___
> Kernelnewbies mailing list
> Kernelnewbies@kernelnewbies.org
> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies



PGP.sig
Description: This is a digitally signed message part
___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

kernel memory allocation

[Amirali Shambayati]

Hello all,

I just want to allocate memory for a struct instantiation. Would anyone
guide me what arguments I should pass to "kmalloc_node"?
Regards,
-- 
Amirali Shambayati
Bachelor Student
Computer Engineering Department
Sharif University of Technology
Tehran, Iran
___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Check Kernel Memory Allocation

[Dev Null]

Hello,

I wanted to know how much kernel and userspace memory is available on the
Linux system during run-time either by some C code or some procfile.

Do you know how to do that?

Thanks,
Dev
___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Check Kernel Memory Allocation

[Dave Hylands]

Hi Dev,

> I wanted to know how much kernel and userspace memory is available on the
> Linux system during run-time either by some C code or some procfile.
>
> Do you know how to do that?

You can look in /proc/meminfo
http://unixfoo.blogspot.com/2008/02/know-about-procmeminfo.html

The memory is shared between userspace and kernel space and the amount
of memory available changes dynamically as the system is running.

Be aware that Cached memory is quite dynamic, and the system will
throw away cache pages to make space if it needs to.

-- 
Dave Hylands
Shuswap, BC, Canada
http://www.davehylands.com

___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Check Kernel Memory Allocation

[Mulyadi Santosa]

On Fri, Jun 3, 2011 at 01:05, Dev Null  wrote:
> I wanted to know how much kernel and userspace memory is available on the
> Linux system during run-time either by some C code or some procfile.

have you checked /proc/meminfo and its handler?

-- 
regards,

Mulyadi Santosa
Freelance Linux trainer and consultant

blog: the-hydra.blogspot.com
training: mulyaditraining.blogspot.com

___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Fwd: Check Kernel Memory Allocation

[Dev Null]

-- Forwarded message --
From: Dev Null 
Date: Tue, May 31, 2011 at 4:08 PM
Subject: Check Kernel Memory Allocation
To: kernelnewbies Newbies 


Hello,

I wanted to know how much kernel and userspace memory is available on the
Linux system during run-time either by some C code or some procfile.

Do you know how to do that?

Thanks,
Dev
___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: kernel memory usage

[naveen yadav]

Yes Bharath, you are right.

On Thu, Dec 30, 2010 at 9:07 AM, Bharath H S  wrote:
> I think Naveen means, all the memory does not appear in the output of
> /proc/meminfo, the reason is kernel memory is discounted and not
> accounted for in the output. May be his goal is to calculate the total
> memory in the system and that output of /proc/meminfo does not give
> that.
>
>
>
> On Wed, Dec 29, 2010 at 9:39 PM, Mulyadi Santosa
>  wrote:
>> On Wed, Dec 29, 2010 at 20:30, naveen yadav  wrote:
>>> Hi all,
>>>
>>> I want to calculate the total kernel memory usage.
>>
>> First, what's your definition of "kernel memory" ? Everything that is
>> mapped inside kernel address space only?
>> --
>> regards,
>>
>> Mulyadi Santosa
>> Freelance Linux trainer and consultant
>>
>> blog: the-hydra.blogspot.com
>> training: mulyaditraining.blogspot.com
>>
>> ___
>> Kernelnewbies mailing list
>> Kernelnewbies@kernelnewbies.org
>> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>>
>

___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: kernel memory usage

[Bharath H S]

I think Naveen means, all the memory does not appear in the output of
/proc/meminfo, the reason is kernel memory is discounted and not
accounted for in the output. May be his goal is to calculate the total
memory in the system and that output of /proc/meminfo does not give
that.



On Wed, Dec 29, 2010 at 9:39 PM, Mulyadi Santosa
 wrote:
> On Wed, Dec 29, 2010 at 20:30, naveen yadav  wrote:
>> Hi all,
>>
>> I want to calculate the total kernel memory usage.
>
> First, what's your definition of "kernel memory" ? Everything that is
> mapped inside kernel address space only?
> --
> regards,
>
> Mulyadi Santosa
> Freelance Linux trainer and consultant
>
> blog: the-hydra.blogspot.com
> training: mulyaditraining.blogspot.com
>
> ___
> Kernelnewbies mailing list
> Kernelnewbies@kernelnewbies.org
> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>

___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: kernel memory usage

[Mulyadi Santosa]

On Wed, Dec 29, 2010 at 20:30, naveen yadav  wrote:
> Hi all,
>
> I want to calculate the total kernel memory usage.

First, what's your definition of "kernel memory" ? Everything that is
mapped inside kernel address space only?
-- 
regards,

Mulyadi Santosa
Freelance Linux trainer and consultant

blog: the-hydra.blogspot.com
training: mulyaditraining.blogspot.com

___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

kernel memory usage

[naveen yadav]

Hi all,

I want to calculate the total kernel memory usage.

cat /proc/meminfo
MemTotal:  4131168 kB
MemFree:336848 kB
Buffers:261776 kB
Cached:3115752 kB
SwapCached:264 kB
Active:1536344 kB
Inactive:  1922636 kB
HighTotal: 3260052 kB
HighFree:   149984 kB
LowTotal:   871116 kB
LowFree:186864 kB
SwapTotal: 2072376 kB
SwapFree:  2070544 kB
Dirty:   20956 kB
Writeback:   0 kB
AnonPages:   81080 kB
Mapped:  21216 kB
Slab:   205672 kB
SReclaimable:   184296 kB
SUnreclaim:  21376 kB
PageTables:   5848 kB
NFS_Unstable:0 kB
Bounce:  0 kB
WritebackTmp:0 kB
CommitLimit:   4137960 kB
Committed_AS:   245100 kB
VmallocTotal:   114680 kB
VmallocUsed:  4724 kB
VmallocChunk:   109804 kB
HugePages_Total: 0
HugePages_Free:  0
HugePages_Rsvd:  0
HugePages_Surp:  0
Hugepagesize: 2048 kB
DirectMap4k:  4096 kB
DirectMap2M:913408 kB

so I want to know the method by which I can know the amout of memory
kernel is using at any instance of time. How can i find this.
since as far as i know vmallocused =vmalloc allocation+ ioremap. also
ioremap can be allocated multiple times. So is there any method to do
exact calculation.

Thanks

___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies