Re: Need help understanding memory models, cpu modes and address translation
On Sat, Jul 16, 2011 at 12:06 AM, Joel A Fernandes wrote: > On Fri, Jul 15, 2011 at 10:04 PM, Vaibhav Jain wrote: >> Hi, >> >> Actually I have read that book. But when i started reading other books such >> as those on assembly >> they had these concpets of Real Mode , Protected Mode , Flat Memory model, >> Segmented Memory model which are specific to intel 32-bit architecture >> and which has got me highly confused. So I am looking for references that >> explain these concepts in depth. >> > > Then you should read the book "Understanding The Linux Kernel", and > Intel's reference manuals which document these features. > > Also this is the book we read in Undergrad for x86 internals :) It was > considered to be the bible for the topic (atleast at the time) > http://www.amazon.com/Intel-Microprocessors-80186-80286-80386/dp/0132606704/ref=sr_1_1?s=books&ie=UTF8&qid=1310792692&sr=1-1 > I'd like to also add that the above mentioned book covers a lot of fundamentals and is a good starting point. Thanks Joel ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: Need help understanding memory models, cpu modes and address translation
On Sat, Jul 16, 2011 at 8:34 AM, Vaibhav Jain wrote: > > Hi, > > Actually I have read that book. But when i started reading other books such > as those on assembly > they had these concpets of Real Mode , Protected Mode , Flat Memory model, > Segmented Memory model which are specific to intel 32-bit architecture > and which has got me highly confused. So I am looking for references that > explain these concepts in depth. What I suggest is. You should download a datasheet for Intel 8086, designed by Intel in late 1970's. Read espacially the memory organization part. This will give you some basic understanding of read mode and segmentation stuff and how it all started. The current processor manuals are too complex to start with. Link: http://datasheets.chipdb.org/Intel/x86/808x/datashts/8086/ > > Thanks > Vaibhav Jain > > > On Thu, Jul 14, 2011 at 9:26 PM, Kaustubh Ashtekar > wrote: >> >> I think you should first read the book >> Operating System Concepts by Silberschatz Galvin. >> It is a general book on OS concepts and will help you gain understanding of >> these concepts. >> -Kaustubh >> On Thu, Jul 14, 2011 at 1:17 AM, Vaibhav Jain wrote: >>> >>> Hi, >>> >>> Thanks a lot for the resources! I just had a cursory look at them and could >>> see that they explain virtual memory and >>> address translation in depth. Just wanted to ask if they explain the >>> concepts Memory models (flat, segmented ) and cpu modes >>> (real,protected) equally well. I am more confused about >>> these concepts. >>> >>> Thanks >>> Vaibhav Jain >>> >>> On Wed, Jul 13, 2011 at 10:40 AM, Anuz Pratap Singh Tomar >>> wrote: On Wed, Jul 13, 2011 at 6:29 PM, Jeff Haran wrote: > > > > > > From: kernelnewbies-boun...@kernelnewbies.org > [mailto:kernelnewbies-boun...@kernelnewbies.org] On Behalf Of Vaibhav Jain > Sent: Wednesday, July 13, 2011 1:38 AM > To: Daniel Baluta > Cc: kernelnewbies@kernelnewbies.org > Subject: Re: Need help understanding memory models,cpu modes and address > translation > > > > Hi, > > Thanks for the link.I really appreciate but I need something more basic > and something that > explains these concepts from a broader perspective and not in the context > of a > particular cpu architecture.Please send me more such links if you come > across any. > > Thanks > Vaibhav Jain > > On Wed, Jul 13, 2011 at 1:14 AM, Daniel Baluta > wrote: > > Hi, > > > I am eager to understand the basics of Memory models (flat, segmented > > etc) > > , CPU modes (real,protected) > > and address translation (physical to logical etc.) and how all of them > > work > > together. I am very confused > > about this and would really appreciate if someone could provide good > > references to these topics. > > You may find useful information inside i366 Programmers Manual. > Anyhow, reading materials is the first step in understanding these > concepts. You will have to actually read/write/debug pieces of code > related to them. > > thanks, > Daniel. > > [1] pdos.csail.mit.edu/6.828/2010/readings/i386.pdf > > > > You might want to try “Understanding the Linux Virtual Memory Manager” by > Mel Gorman. It’s freely available on the web. > > > > It’s Linux specific, but since Linux needs to run on a multitude of > different processors with different memory management hardware, the > approach it takes to doing so in inherently broad. > > > > A lot of basic concepts can be learnt from JIm Turley's book on 8086 Other book is modern operating systems by Tanenbaum which explains everything in much detail with code snippets. >>> >>> >>> ___ >>> Kernelnewbies mailing list >>> Kernelnewbies@kernelnewbies.org >>> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies >>> >> > ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: Need help understanding memory models, cpu modes and address translation
Hi Mulyadi, Good to read your posts. It has certainly been a long time and it feels good to be back on this list! > On Sat, Jul 16, 2011 at 10:14, Vaibhav Jain wrote: > >> Could somebody please state the difference clearly for me and explain how >> these two work together. I would >> really appreciate if someone could explain the whole chain from generation >> of addresses by compiler and then translation of >> those addresses in case of Segmentation working along with Paging. > > when you generate object code from your source code (let's say in C) > using gcc, first your code and variables (data) are turn into Position > Independent Code. It means, it is just an offset. If there is an > offset, surely we need base address, right? But not at this object > (resulting in .o) stage. > > Then in reach producing final ELF binary (executable). Using known > standart ELF rule, those offset are turn into final final address. So > let's, code are placed starting at 0x080499f0 and so on. Very nice explanation! > > When that binary is loaded into memory, loader (ld.so) take that > information and use it as a clue on where to put the code and data. > Using standart mmap() syscall, memory area is reserved and data/code > is loaded there. The exception is stack, where it is allocated > dynamically (and grows down, for Intel arch) starting at the upper > limit of user space (near 3 GiB). > I'm just a little troubled by this bit. AFAIK, the kernel takes ELF executables and loads them into the appropriate sections after parsing the ELF tables and headers. Correct me if I'm wrong? Ofcourse, ld.so takes care of dynamic linking/loading shared libraries into the address space. Thanks, Joel ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: Need help understanding memory models, cpu modes and address translation
On Fri, Jul 15, 2011 at 10:04 PM, Vaibhav Jain wrote: > Hi, > > Actually I have read that book. But when i started reading other books such > as those on assembly > they had these concpets of Real Mode , Protected Mode , Flat Memory model, > Segmented Memory model which are specific to intel 32-bit architecture > and which has got me highly confused. So I am looking for references that > explain these concepts in depth. > Then you should read the book "Understanding The Linux Kernel", and Intel's reference manuals which document these features. Also this is the book we read in Undergrad for x86 internals :) It was considered to be the bible for the topic (atleast at the time) http://www.amazon.com/Intel-Microprocessors-80186-80286-80386/dp/0132606704/ref=sr_1_1?s=books&ie=UTF8&qid=1310792692&sr=1-1 Regards, Joel ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: Need help understanding memory models, cpu modes and address translation
Hello Vaibhav... What you asked, is something really difficult to summarize in short...but let me try... On Sat, Jul 16, 2011 at 10:14, Vaibhav Jain wrote: > Hi, > Thanks a lot for all the links! > I read the two articles below by Gustavo but I am still very confused about > segmentation. One of the articles below says that > the segments in virtual address space are different from 'intel-style > segments' whereas I used to think of them as being the same. I am not really sure what segments in virtual address space really means. What we have in virtual address space is AFAIK kernel space and user space. User space, is a space where OS put user and data from your casual programs (e.g your games, your word processors etc). Assuming it's in x86 32 bit, the range in virtual address is between address 0 and slightly under 3 GiB. Whereas kernel space, is a space where OS put its own data and code (i.e interrupt handler, system call handler, page tables). Again in x86 32 bit, it lies between 3 GiB up to 4 GiB. > Could somebody please state the difference clearly for me and explain how > these two work together. I would > really appreciate if someone could explain the whole chain from generation > of addresses by compiler and then translation of > those addresses in case of Segmentation working along with Paging. when you generate object code from your source code (let's say in C) using gcc, first your code and variables (data) are turn into Position Independent Code. It means, it is just an offset. If there is an offset, surely we need base address, right? But not at this object (resulting in .o) stage. Then in reach producing final ELF binary (executable). Using known standart ELF rule, those offset are turn into final final address. So let's, code are placed starting at 0x080499f0 and so on. When that binary is loaded into memory, loader (ld.so) take that information and use it as a clue on where to put the code and data. Using standart mmap() syscall, memory area is reserved and data/code is loaded there. The exception is stack, where it is allocated dynamically (and grows down, for Intel arch) starting at the upper limit of user space (near 3 GiB). The new feature, called PIE (position independent code) allow loader to do further address reorganization. So instead of let's say taking 0x080499f0 for start of code segment for granted, it can relocate it to another address (but still in user space). However, in order to do that, every addresing done by the code must be compiled differently, so instead using constant address, now it uses combination of base register and offset AFAIK. Code runs, data addressed, they are all under virtual address. But memory is pointed using real address. Here comes the work of MMU (Memory Management Unit). With the help of tables of address translation called page tables, virtual address is translated into real address. The detail is left for you for further research. But enough to say it works like lookup table, e.g "12" in virtual address could means real address 10012 and so on. Please notice that continous address in virtual address doesn't neccessarily means continous address in real address e.g 12-20 in virtual address could means 10012 up to 10016 then 40017 up to 40020. Getting clearer now? :D -- regards, Mulyadi Santosa Freelance Linux trainer and consultant blog: the-hydra.blogspot.com training: mulyaditraining.blogspot.com ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: Need help understanding memory models, cpu modes and address translation
Hi, Thanks a lot for all the links! I read the two articles below by Gustavo but I am still very confused about segmentation. One of the articles below says that the segments in virtual address space are different from 'intel-style segments' whereas I used to think of them as being the same. Could somebody please state the difference clearly for me and explain how these two work together. I would really appreciate if someone could explain the whole chain from generation of addresses by compiler and then translation of those addresses in case of Segmentation working along with Paging. I am unable to comprehend how a compiler generates address for a segmented memory model. Explaining the whole chain, I think , can serve the purpose. Thanks! Vaibhav Jain On Wed, Jul 13, 2011 at 7:46 PM, Joel A Fernandes wrote: > On Wed, Jul 13, 2011 at 9:41 PM, Mulyadi Santosa > wrote: > > On Wed, Jul 13, 2011 at 14:33, Vaibhav Jain wrote: > >> Hi, > >> > >> I am eager to understand the basics of Memory models (flat, segmented > etc) > >> , CPU modes (real,protected) > >> and address translation (physical to logical etc.) and how all of them > work > >> together. > > > > maybe something like this? > > > > http://tldp.org/HOWTO/KernelAnalysis-HOWTO.html > > This should nail it :) > > http://duartes.org/gustavo/blog/post/how-the-kernel-manages-your-memory > http://duartes.org/gustavo/blog/post/anatomy-of-a-program-in-memory > > Thanks, > Joel > ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: Need help understanding memory models, cpu modes and address translation
Hi, Actually I have read that book. But when i started reading other books such as those on assembly they had these concpets of Real Mode , Protected Mode , Flat Memory model, Segmented Memory model which are specific to intel 32-bit architecture and which has got me highly confused. So I am looking for references that explain these concepts in depth. Thanks Vaibhav Jain On Thu, Jul 14, 2011 at 9:26 PM, Kaustubh Ashtekar wrote: > I think you should first read the book > Operating System Concepts by Silberschatz Galvin. > > It is a general book on OS concepts and will help you gain understanding of > these concepts. > > -Kaustubh > > On Thu, Jul 14, 2011 at 1:17 AM, Vaibhav Jain wrote: > >> Hi, >> >> Thanks a lot for the resources! I just had a cursory look at them and >> could see that they explain virtual memory and >> address translation in depth. Just wanted to ask if they explain the >> concepts Memory models (flat, segmented ) and cpu modes >> (real,protected) equally well. I am more confused about >> these concepts. >> >> Thanks >> Vaibhav Jain >> >> On Wed, Jul 13, 2011 at 10:40 AM, Anuz Pratap Singh Tomar < >> chambilketha...@gmail.com> wrote: >> >>> >>> >>> On Wed, Jul 13, 2011 at 6:29 PM, Jeff Haran wrote: >>> ** ** ** ** *From:* kernelnewbies-boun...@kernelnewbies.org [mailto: kernelnewbies-boun...@kernelnewbies.org] *On Behalf Of *Vaibhav Jain *Sent:* Wednesday, July 13, 2011 1:38 AM *To:* Daniel Baluta *Cc:* kernelnewbies@kernelnewbies.org *Subject:* Re: Need help understanding memory models,cpu modes and address translation ** ** Hi, Thanks for the link.I really appreciate but I need something more basic and something that explains these concepts from a broader perspective and not in the context of a particular cpu architecture.Please send me more such links if you come across any. Thanks Vaibhav Jain On Wed, Jul 13, 2011 at 1:14 AM, Daniel Baluta wrote: Hi, > I am eager to understand the basics of Memory models (flat, segmented etc) > , CPU modes (real,protected) > and address translation (physical to logical etc.) and how all of them work > together. I am very confused > about this and would really appreciate if someone could provide good > references to these topics. You may find useful information inside i366 Programmers Manual. Anyhow, reading materials is the first step in understanding these concepts. You will have to actually read/write/debug pieces of code related to them. thanks, Daniel. [1] pdos.csail.mit.edu/6.828/2010/readings/i386.pdf ** ** You might want to try “Understanding the Linux Virtual Memory Manager” by Mel Gorman. It’s freely available on the web. ** ** It’s Linux specific, but since Linux needs to run on a multitude of different processors with different memory management hardware, the approach it takes to doing so in inherently broad. ** ** A lot of basic concepts can be learnt from JIm Turley's book on 8086 >>> Other book is modern operating systems by Tanenbaum which explains >>> everything in much detail with code snippets. >>> >>> >>> >> >> ___ >> Kernelnewbies mailing list >> Kernelnewbies@kernelnewbies.org >> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies >> >> > ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
LinuxCon Japan Videos
Can anybody point me at this year's LinuxCon Japan video recordings? Are they available somewhere? I'm aware of the Linus interview, I am intrested in talks and panels. --paul ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
How to recognize synchronous write requests
Hi all, As much as I know, type of an IO request can be realized by "DATA_DIR" macro,but I wanted to know if it is possible to distinguish a read or write is synchronous or asychronous. I'm also using "blktrace" to extract requests dispatching to disk. blktrace, determines synchrnous writes as "ws" and asynchronous writes as "w". Regards, -- Amirali Shambayati Bachelor Student Computer Engineering Department Sharif University of Technology Tehran, Iran ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: change kthread priority
On Mon, Jul 11, 2011 at 12:45 PM, loody wrote:
> Hi:
>
> 2011/7/11 loody :
>> hi:
>>
>>
>> 2011/7/11 Dave Hylands :
>>> Hi Milody,
>>>
>>> On Sun, Jul 10, 2011 at 8:55 PM, loody wrote:
hi:
2011/7/11 Mulyadi Santosa :
> Hi...
>
>>>
>>> ...snip...
>>>
> AFAIK they are user mode libc functions...
I found there are the same export_symbols in kernel.
I use them to dynamically change the priority.
It works but I found something interesting:
1. if thread use schedule algorithm of SCHED_NORMAL, it cannot change
the priority.
2. can we set the priority to any number less than
(MAX_USER_RT_PRIO-1)?
( I tried to set the priority as 99, but it fail)
>>>
>>> You need to change the scheduling algorithim to be SCHED_FIFO or
>>> SCHED_RR in order to use the real-time priorities.
>> I saw kernel restrict the algorithm as SCHED_FIFO and SCHED_RR to
>> change the priority.
>> What are the differences between rt_priority, prio, static_prio and
>> normal_prio?
>> sched_setscheduler seems only change rt_priority.
>>
>> normal_prio is MAX_RT_PRIO-1 - t_priority
>> prio will be min(normal_prio, task_top_pi_waiter)
>>
>> what is the real priority that kthread runs after calling
>> sched_setscheduler ?
>>
>>
>>>
3. can we get the name of thread from task_struct or pid?
(I use "cat /proc/338/status", I can see the name of thread. But I
look around the task_struct, I cannot find any name information in
it.)
>>>
>>> You can use the comm field from the task struct. This can be set using
>>> the prctl API and passing in PR_SET_NAME. You can see the task name by
>>> doing something like
>>>
>>> ps -T -o pid,ppid,user,time,comm
>>>
>>> Normally, ps shows the args column.
>>>
>>> --
> Since there are below schedule algorithms in kernel,
> SCHED_NORMAL
> SCHED_FIFO
> SCHED_RR
> SCHED_BATCH
> when and how they circular around?
> or each of them has a fix cpu time slice to go through?
The base scheduler code(kernel/sched.c) iterates over each scheduler
class(SCHED_NORMAL,SCHED_FIFO,SCHED_RR) in order of priority.The
highest priority scheduler class that has a runnable process wins.
below is the code.
/*
* Pick up the highest-prio task:
*/
static inline struct task_struct *
pick_next_task(struct rq *rq)
{
const struct sched_class *class;
struct task_struct *p;
/*
* Optimization: we know that if all tasks are in
* the fair class we can call that function directly:
*/
if (likely(rq->nr_running == rq->cfs.nr_running)) {
p = fair_sched_class.pick_next_task(rq);
if (likely(p))
return p;
}
class = sched_class_highest;
for ( ; ; ) {
p = class->pick_next_task(rq);
if (p)
return p;
/*
* Will never be NULL as the idle class always
* returns a non-NULL p:
*/
class = class->next;
}
}
> BR,
> cckuo
>
> ___
> Kernelnewbies mailing list
> Kernelnewbies@kernelnewbies.org
> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>
>
___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: Questions regarding console drivers..
Hi sandeep, On Wed, Jul 13, 2011 at 4:47 PM, sandeep kumar wrote: > Hi all, > 4) How to know how many number of consoles are running at a time? /sys/class/vtconsole/ http://www.linux.it/~rubini/docs/sercons/sercons.html ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
