Re: Where did vm_operations_struct->unmap in 2.4.0 go?

[David Woodhouse]

[EMAIL PROTECTED] said:
> That is a lot of work for a very few special cases.  OTOH, you could
> just add a few lines of __initcall code in two source files (which I
> did when I wrote inter_module_xxx) and swap the order of 3 lines in
> drivers/mtd/Makefile.  Guess which alternative I am going for? 

I've already worked round it for the 2.[024] case by reintroducing the
ifdefs. I assume here that we're planning for 2.5. As part of the many
changes that are going to be introduced during 2.5, this shouldn't be too
intrusive.

Once it's actually usable for the common case, it won't just be 'a very few 
special cases' any more. But that's all 2.5 material.

--
dwmw2


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct-unmap in 2.4.0 go?

[David Woodhouse]

[EMAIL PROTECTED] said:
 That is a lot of work for a very few special cases.  OTOH, you could
 just add a few lines of __initcall code in two source files (which I
 did when I wrote inter_module_xxx) and swap the order of 3 lines in
 drivers/mtd/Makefile.  Guess which alternative I am going for? 

I've already worked round it for the 2.[024] case by reintroducing the
ifdefs. I assume here that we're planning for 2.5. As part of the many
changes that are going to be introduced during 2.5, this shouldn't be too
intrusive.

Once it's actually usable for the common case, it won't just be 'a very few 
special cases' any more. But that's all 2.5 material.

--
dwmw2


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct->unmap in 2.4.0 go?

[Keith Owens]

On Sun, 14 Jan 2001 13:47:29 -0800 (PST), 
Linus Torvalds <[EMAIL PROTECTED]> wrote:
>On Sun, 14 Jan 2001, David Woodhouse wrote:
>> That's the one flaw in the inter_module_get() stuff - we could do with a
>> way to put entries in the table at _compile_ time, rather than _only_ at
>> run time. 
>
>Ok, I can buy that. Not having to initialize explicitly would be nice, but
>if so we should make module loading do it automatically too ...

It might be nice but it is also expensive.  Adding static
inter_module_xxx tables requires

* changes to linux/modules.h to define the new table format and
* changes to vmlinux.lds for _every_ architecture to bring all the
  static tables together in vmlinux and
* new initialisation code in module.c to read and load all the static
  tables at boot time and
* extra code in modutils to find any static tables in modules and
* an extension to struct modules to let modutils pass information about
  the static tables to the kernel and
* the kernel code will only work with an upgraded modutils.

That is a lot of work for a very few special cases.  OTOH, you could
just add a few lines of __initcall code in two source files (which I
did when I wrote inter_module_xxx) and swap the order of 3 lines in
drivers/mtd/Makefile.  Guess which alternative I am going for?

IMHO any automatic method that relies on ELF sections and/or modutils
support is the wrong approach, it is a complex solution with external
dependencies when we already have a simple solution with no external
dependencies.  What next, static tables for file system registration,
for device registration?

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct->unmap in 2.4.0 go?

[David Woodhouse]

On Sun, 14 Jan 2001, Linus Torvalds wrote:
> On Sun, 14 Jan 2001, David Woodhouse wrote:
> > That's the one flaw in the inter_module_get() stuff - we could do with a
> > way to put entries in the table at _compile_ time, rather than _only_ 
> > at run time. 

> Ok, I can buy that. Not having to initialize explicitly would be nice, but
> if so we should make module loading do it automatically too, so that we
> don't generate unnecessary differences between module and compiled in (ie
> I'd rather avoid the situation that "if you're a module, you need to
> explicitly export your inter_module_stuff(), while if you're compiled-in
> it will be exported automatically").

Yep. Modutils can probably handle that case without too much difficulty, 
if we go with sticking the static inter_module_entries in a special ELF 
section as I originally suggested. Keith?

-- 
dwmw2




-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct->unmap in 2.4.0 go?

[Linus Torvalds]

On Sun, 14 Jan 2001, David Woodhouse wrote:
> 
> That's the one flaw in the inter_module_get() stuff - we could do with a
> way to put entries in the table at _compile_ time, rather than _only_ at
> run time. 

Ok, I can buy that. Not having to initialize explicitly would be nice, but
if so we should make module loading do it automatically too, so that we
don't generate unnecessary differences between module and compiled in (ie
I'd rather avoid the situation that "if you're a module, you need to
explicitly export your inter_module_stuff(), while if you're compiled-in
it will be exported automatically").

Linus

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct->unmap in 2.4.0 go?

[David Woodhouse]

On Sun, 14 Jan 2001, Linus Torvalds wrote:

> This is what "request_module()" and "kmod" is all about. Once we probe the
> hardware, the drievr itself can ask for more drivers.
> 
> I completely fail to see the arguments that have been brought up for drm
> doing ugly things. The code should simply do
> 
>   drm_agp_head_t * head = inter_module_get("drm_agp");
> 
>   if (!head) {
>   request_module("drm-agp");
>   head = inter_module_get("drm_agp");
>   if (!head)
>   return -ENOAGP;
>   }
> 
> and be done with it. THE ABOVE MAKES SENSE. The code says _exactly_ what
> the module wants to do: it wants to find the AGP support, and if it cannot
> find the AGP support it wants to load them.

It's the same with CFI command-set-specific code. Except that the 
command-set specific code didn't previously have to be initialised at all, 
and now we've got to initialise it (and have it call 
inter_module_register) before it's required by the cfi_probe code.

The difference here is that while drm_agp actually had to do some hardware
initialisation, the CFI command set handlers didn't - the only thing their
module_init routine does is call inter_module_register(). So we've
introduced the init order dependencies where previously they weren't
necessary.

That's the one flaw in the inter_module_get() stuff - we could do with a
way to put entries in the table at _compile_ time, rather than _only_ at
run time. 

While I accept that we can't eliminate init order dependencies completely,
I still think we should avoid them where it's possible. Which it would be
in this case, without much difficulty at all.

-- 
dwmw2


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct->unmap in 2.4.0 go?

[Linus Torvalds]

On Sun, 14 Jan 2001, David Woodhouse wrote:
> 
> But in the case of the CFI probe code and also I believe DRM, we don't
> actually know precisely which feature we're going to require until we've
> done the hardware probe at runtime.

That's ok.

This is what "request_module()" and "kmod" is all about. Once we probe the
hardware, the drievr itself can ask for more drivers.

I completely fail to see the arguments that have been brought up for drm
doing ugly things. The code should simply do

drm_agp_head_t * head = inter_module_get("drm_agp");

if (!head) {
request_module("drm-agp");
head = inter_module_get("drm_agp");
if (!head)
return -ENOAGP;
}

and be done with it. THE ABOVE MAKES SENSE. The code says _exactly_ what
the module wants to do: it wants to find the AGP support, and if it cannot
find the AGP support it wants to load them.

The arguments about how the user should load things in some specific order
or whatever are complete crap. All the support is there, and whining about
it is not going to change my opinion in the least.

Linus


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct->unmap in 2.4.0 go?

[David Woodhouse]

On Sun, 14 Jan 2001, Linus Torvalds wrote:

> Note that previously there _were_ order dependencies. In fact, I consider
> it very tasteless to have modules that act differently on whether another
> module is loaded. I saw some arguments saying that this is th "right
> thing", and I disagree completely.

The only valid behaviour I can think of is...

 if (!feature_present)
try_to_load_it();

 if (feature_present)
use_it();
 else
if (we_can_live_without())
deal_with_it();
else
whinge_and_die();

In this case, it's not really depending on whether the desired feature has
been loaded yet or not. It's depending on whether the desired feature is
available or not. In this sense, inter_module_get_request() is an
improvement, because it makes that explicit.

Obviously, in the case where we'd eventually just whinge_and_die(),
usually it's best to just make this code depend on whatever feature it is
that's being used, by referencing it directly.

But in the case of the CFI probe code and also I believe DRM, we don't
actually know precisely which feature we're going to require until we've
done the hardware probe at runtime. We don't want the generic code having
a hard dependency on each possible submodule, and doing it with ifdefs
according to what happens to be compiled in is just ugly. So the above
logic was useful, and get_module_symbol(), even though it wasn't
wonderful, provided it.

The reason you didn't get the current CFI code with the rest of the update 
I gave you for 2.4.0-test12 is because I'm intending to rewrite it first, 
and hopefully deal with this issue in a better way while I'm at it.

But as it stands, the best option I can see is to have the generic probe
code have ifdefs on the chipset-specific options, and reference only the
ones which are present. It's not the end of the world, and as rmk 
suggests, many embedded systems are run without module support in 
production anyway. 

--
dwmw2



-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct->unmap in 2.4.0 go?

[Linus Torvalds]

On Sun, 14 Jan 2001, David Woodhouse wrote:
> 
> But I have no particular attachment to it. All I'm asking for is a way to
> avoid having init order dependencies where previously there was no need
> for them, by having a way to put entries in the inter_module_get() table
> at compile time.

Note that previously there _were_ order dependencies. In fact, I consider
it very tasteless to have modules that act differently on whether another
module is loaded. I saw some arguments saying that this is th "right
thing", and I disagree completely.

Linus

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct->unmap in 2.4.0 go?

[David Woodhouse]

On 13 Jan 2001, Linus Torvalds wrote:

> You miss _entirely_ the reason why "get_module_symbol()" was removed,
> and why I will not _ever_ accept it coming back.
> 
> Hint #1: get_MODULE_symbol().
> Hint #2: compiled in functionality.

Er,... forgive me if I'm being overly dense here, but I can't see anything 
fundamentally wrong in the above that wouldn't be fixed with a 
judicious application of s/module_//

But I have no particular attachment to it. All I'm asking for is a way to
avoid having init order dependencies where previously there was no need
for them, by having a way to put entries in the inter_module_get() table
at compile time.

-- 
dwmw2


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct->unmap in 2.4.0 go?

[Kai Henningsen]

[EMAIL PROTECTED] (Keith Owens)  wrote on 12.01.01 in <[EMAIL PROTECTED]>:

> On Fri, 12 Jan 2001 13:01:12 +0100,
> Daniel Phillips <[EMAIL PROTECTED]> wrote:
> >Keith Owens wrote:
> >> I want to completely remove this multi layered method for setting
> >> initialisation order and go back to basics.  I want the programmer to
> >> say "initialise E and F after G, H and I".  The kernel build system
> >> works out the directed graph of initialisation order then controls the
> >> execution of startup code to satisfy this graph.
> >
> >I don't doubt you will come up with a workable solution at build time.
> >However, working out a valid graph at execution time is trivial and
> >efficient, given a list of precedence relations of the kind you're
> >suggesting.  In fact you don't even have to work out the graph before
> >starting the initialization, it's also trivial to keep a count of
> >unsatisfied initialization conditions at the beginning of each
> >initialization sequence and block until the count goes to zero.  (In
> >essence, evaluate a priority sort on the fly.)
>
> It is safer to evaluate the graph at link time in case somebody
> mistakenly codes a dependency loop, that is an abort case.  Finding
> that you have a loop at load time and aborting the kernel is a little
> too drastic for my tastes.
>
> In any case it is academic.  Linus insists on link order being
> explicitly and completely specified by the programmer and he likes the
> link order being implied by Makefile order.  So there is no point in
> working on a better system.

I'm not so sure about that. I think it _should_ be possible to do both,  
and get better documentation at the same time.

How about this:

* Invent some method for modules to declare these dependencies. Maybe even
  in /** */ type comments, so it goes right into the documentation.

* Write a program to collect all dependencies, do a tsort on them
  (alerting the developer if a loop is found), determining a reasonable
  initialization order, and spitting out the relevant Makefiles or
  Makefile fragments to *create* that initialization order the Linus way.

Or in other words, don't do it at runtime, don't do it at compile time, do  
it at develop time. (Or patch time, perhaps.)

MfG Kai
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct->unmap in 2.4.0 go?

[David Woodhouse]

On Sun, 14 Jan 2001, Keith Owens wrote:

> Note I said allowed, not supported.  I refuse to support any binary
> only modules, my standard response to problems logged against binary
> modules is "remove them and reproduce the problem".  Checking for ABI
> violations is not supporting binary modules, it is detecting that they
> are stuffed and telling the user to go pester their supplier instead of
> polluting l-k with questions that will be ignored.

Sensible. As long as it doesn't give rise to reports of the type "modutils
didn't whinge so it's not the binary-only module's fault."

-- 
dwmw2


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct->unmap in 2.4.0 go?

[Keith Owens]

On Sun, 14 Jan 2001 09:43:21 + (GMT), 
David Woodhouse <[EMAIL PROTECTED]> wrote:
>On Sun, 14 Jan 2001, Keith Owens wrote:
>> That forces the maintenance load back onto the binary supplier and
>> removes the questions from l-k, including many of the oops reports with
>> binary only drivers in the module list.
>
>No. The correct response to that is _already_ "You have a binary-only
>module. Even in the kernel it was compiled against, you are not supported.
>Goodbye".

I wish that Linus had never agreed to binary only modules.  But as long
as they are allowed, I want to detect problems with binary only modules
before they hit the rest of the kernel and end up as questions on l-k.

Note I said allowed, not supported.  I refuse to support any binary
only modules, my standard response to problems logged against binary
modules is "remove them and reproduce the problem".  Checking for ABI
violations is not supporting binary modules, it is detecting that they
are stuffed and telling the user to go pester their supplier instead of
polluting l-k with questions that will be ignored.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct->unmap in 2.4.0 go?

[David Woodhouse]

On Sun, 14 Jan 2001, Keith Owens wrote:

> This is becoming more important as the kernel moves towards hot
> plugging devices, especially for binary only drivers.  It is far better
> for the kernel community if modutils can say "cannot load module foo
> because its interfaces do not match the kernel, upgrade module foo".
> That forces the maintenance load back onto the binary supplier and
> removes the questions from l-k, including many of the oops reports with
> binary only drivers in the module list.

No. The correct response to that is _already_ "You have a binary-only
module. Even in the kernel it was compiled against, you are not supported.
Goodbye".

To quote our Lord and Master:

(http://lwn.net/1999/0211/a/lt-afs.html)
>> I will strive for binary compatibility for modules, but I _expect_
>> that it will be broken.  It's just too easy to have to make changes
>> that break binary-only modules, and I have too little incentive to try
>> to avoid it.
>>
>> If people feel this is a problem, I see a few alternatives:
>>  - don't use stuff with binary-only modules. Just say no.
>>  - work hard at making a source-version of the thing available (it
>>doesn't have to be under the GPL if it's a module, but it has to be
>>available as source so that it can be recompiled).
>>  - don't upgrade
>>  - drop Linux

(http://lwn.net/1999/0211/a/lt-binary.html)
>> Basically, I want people to know that when they use binary-only
>> modules, it's THEIR problem.  I want people to know that in their
>> bones, and I want it shouted out from the rooftops.  I want people to
>> wake up in a cold sweat every once in a while if they use binary-only
>> modules.


[EMAIL PROTECTED] wrote:
> Ignore the fact that the existing module symbol version implementation
> is broken as designed.  http://gear.torque.net/kbuild/archive/1280.html
> lists the major problems with make dep, genksyms has all those problems
> plus several of its own.  As part of the Makefile rewrite for 2.5, I am
> redesigning module symbol versions from scratch.
>
> I agree that inter_module_xxx does not check ABI.  That was not for
> lack of trying, but it cannot be done in 2.4, it needs a major redesign
> of module symbols and the makefiles.  It will be possible in 2.5.

This is a good thing, as long as it doesn't get in the way of real
functionality. We don't _need_ to make life easier for people running
binary-only modules. But if we can do it without making life harder for
real people, then that's nice.

-- 
dwmw2


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct-unmap in 2.4.0 go?

[David Woodhouse]

On Sun, 14 Jan 2001, Keith Owens wrote:

 This is becoming more important as the kernel moves towards hot
 plugging devices, especially for binary only drivers.  It is far better
 for the kernel community if modutils can say "cannot load module foo
 because its interfaces do not match the kernel, upgrade module foo".
 That forces the maintenance load back onto the binary supplier and
 removes the questions from l-k, including many of the oops reports with
 binary only drivers in the module list.

No. The correct response to that is _already_ "You have a binary-only
module. Even in the kernel it was compiled against, you are not supported.
Goodbye".

To quote our Lord and Master:

(http://lwn.net/1999/0211/a/lt-afs.html)
 I will strive for binary compatibility for modules, but I _expect_
 that it will be broken.  It's just too easy to have to make changes
 that break binary-only modules, and I have too little incentive to try
 to avoid it.

 If people feel this is a problem, I see a few alternatives:
  - don't use stuff with binary-only modules. Just say no.
  - work hard at making a source-version of the thing available (it
doesn't have to be under the GPL if it's a module, but it has to be
available as source so that it can be recompiled).
  - don't upgrade
  - drop Linux

(http://lwn.net/1999/0211/a/lt-binary.html)
 Basically, I want people to know that when they use binary-only
 modules, it's THEIR problem.  I want people to know that in their
 bones, and I want it shouted out from the rooftops.  I want people to
 wake up in a cold sweat every once in a while if they use binary-only
 modules.


[EMAIL PROTECTED] wrote:
 Ignore the fact that the existing module symbol version implementation
 is broken as designed.  http://gear.torque.net/kbuild/archive/1280.html
 lists the major problems with make dep, genksyms has all those problems
 plus several of its own.  As part of the Makefile rewrite for 2.5, I am
 redesigning module symbol versions from scratch.

 I agree that inter_module_xxx does not check ABI.  That was not for
 lack of trying, but it cannot be done in 2.4, it needs a major redesign
 of module symbols and the makefiles.  It will be possible in 2.5.

This is a good thing, as long as it doesn't get in the way of real
functionality. We don't _need_ to make life easier for people running
binary-only modules. But if we can do it without making life harder for
real people, then that's nice.

-- 
dwmw2


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct-unmap in 2.4.0 go?

[Keith Owens]

On Sun, 14 Jan 2001 09:43:21 + (GMT), 
David Woodhouse [EMAIL PROTECTED] wrote:
On Sun, 14 Jan 2001, Keith Owens wrote:
 That forces the maintenance load back onto the binary supplier and
 removes the questions from l-k, including many of the oops reports with
 binary only drivers in the module list.

No. The correct response to that is _already_ "You have a binary-only
module. Even in the kernel it was compiled against, you are not supported.
Goodbye".

I wish that Linus had never agreed to binary only modules.  But as long
as they are allowed, I want to detect problems with binary only modules
before they hit the rest of the kernel and end up as questions on l-k.

Note I said allowed, not supported.  I refuse to support any binary
only modules, my standard response to problems logged against binary
modules is "remove them and reproduce the problem".  Checking for ABI
violations is not supporting binary modules, it is detecting that they
are stuffed and telling the user to go pester their supplier instead of
polluting l-k with questions that will be ignored.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct-unmap in 2.4.0 go?

[Kai Henningsen]

[EMAIL PROTECTED] (Keith Owens)  wrote on 12.01.01 in [EMAIL PROTECTED]:

 On Fri, 12 Jan 2001 13:01:12 +0100,
 Daniel Phillips [EMAIL PROTECTED] wrote:
 Keith Owens wrote:
  I want to completely remove this multi layered method for setting
  initialisation order and go back to basics.  I want the programmer to
  say "initialise E and F after G, H and I".  The kernel build system
  works out the directed graph of initialisation order then controls the
  execution of startup code to satisfy this graph.
 
 I don't doubt you will come up with a workable solution at build time.
 However, working out a valid graph at execution time is trivial and
 efficient, given a list of precedence relations of the kind you're
 suggesting.  In fact you don't even have to work out the graph before
 starting the initialization, it's also trivial to keep a count of
 unsatisfied initialization conditions at the beginning of each
 initialization sequence and block until the count goes to zero.  (In
 essence, evaluate a priority sort on the fly.)

 It is safer to evaluate the graph at link time in case somebody
 mistakenly codes a dependency loop, that is an abort case.  Finding
 that you have a loop at load time and aborting the kernel is a little
 too drastic for my tastes.

 In any case it is academic.  Linus insists on link order being
 explicitly and completely specified by the programmer and he likes the
 link order being implied by Makefile order.  So there is no point in
 working on a better system.

I'm not so sure about that. I think it _should_ be possible to do both,  
and get better documentation at the same time.

How about this:

* Invent some method for modules to declare these dependencies. Maybe even
  in /** */ type comments, so it goes right into the documentation.

* Write a program to collect all dependencies, do a tsort on them
  (alerting the developer if a loop is found), determining a reasonable
  initialization order, and spitting out the relevant Makefiles or
  Makefile fragments to *create* that initialization order the Linus way.

Or in other words, don't do it at runtime, don't do it at compile time, do  
it at develop time. (Or patch time, perhaps.)

MfG Kai
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct-unmap in 2.4.0 go?

[David Woodhouse]

On 13 Jan 2001, Linus Torvalds wrote:

 You miss _entirely_ the reason why "get_module_symbol()" was removed,
 and why I will not _ever_ accept it coming back.
 
 Hint #1: get_MODULE_symbol().
 Hint #2: compiled in functionality.

Er,... forgive me if I'm being overly dense here, but I can't see anything 
fundamentally wrong in the above that wouldn't be fixed with a 
judicious application of s/module_//

But I have no particular attachment to it. All I'm asking for is a way to
avoid having init order dependencies where previously there was no need
for them, by having a way to put entries in the inter_module_get() table
at compile time.

-- 
dwmw2


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct-unmap in 2.4.0 go?

[Linus Torvalds]

On Sun, 14 Jan 2001, David Woodhouse wrote:
 
 But I have no particular attachment to it. All I'm asking for is a way to
 avoid having init order dependencies where previously there was no need
 for them, by having a way to put entries in the inter_module_get() table
 at compile time.

Note that previously there _were_ order dependencies. In fact, I consider
it very tasteless to have modules that act differently on whether another
module is loaded. I saw some arguments saying that this is th "right
thing", and I disagree completely.

Linus

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct-unmap in 2.4.0 go?

[David Woodhouse]

On Sun, 14 Jan 2001, Linus Torvalds wrote:

 Note that previously there _were_ order dependencies. In fact, I consider
 it very tasteless to have modules that act differently on whether another
 module is loaded. I saw some arguments saying that this is th "right
 thing", and I disagree completely.

The only valid behaviour I can think of is...

 if (!feature_present)
try_to_load_it();

 if (feature_present)
use_it();
 else
if (we_can_live_without())
deal_with_it();
else
whinge_and_die();

In this case, it's not really depending on whether the desired feature has
been loaded yet or not. It's depending on whether the desired feature is
available or not. In this sense, inter_module_get_request() is an
improvement, because it makes that explicit.

Obviously, in the case where we'd eventually just whinge_and_die(),
usually it's best to just make this code depend on whatever feature it is
that's being used, by referencing it directly.

But in the case of the CFI probe code and also I believe DRM, we don't
actually know precisely which feature we're going to require until we've
done the hardware probe at runtime. We don't want the generic code having
a hard dependency on each possible submodule, and doing it with ifdefs
according to what happens to be compiled in is just ugly. So the above
logic was useful, and get_module_symbol(), even though it wasn't
wonderful, provided it.

The reason you didn't get the current CFI code with the rest of the update 
I gave you for 2.4.0-test12 is because I'm intending to rewrite it first, 
and hopefully deal with this issue in a better way while I'm at it.

But as it stands, the best option I can see is to have the generic probe
code have ifdefs on the chipset-specific options, and reference only the
ones which are present. It's not the end of the world, and as rmk 
suggests, many embedded systems are run without module support in 
production anyway. 

--
dwmw2



-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct-unmap in 2.4.0 go?

[Linus Torvalds]

On Sun, 14 Jan 2001, David Woodhouse wrote:
 
 But in the case of the CFI probe code and also I believe DRM, we don't
 actually know precisely which feature we're going to require until we've
 done the hardware probe at runtime.

That's ok.

This is what "request_module()" and "kmod" is all about. Once we probe the
hardware, the drievr itself can ask for more drivers.

I completely fail to see the arguments that have been brought up for drm
doing ugly things. The code should simply do

drm_agp_head_t * head = inter_module_get("drm_agp");

if (!head) {
request_module("drm-agp");
head = inter_module_get("drm_agp");
if (!head)
return -ENOAGP;
}

and be done with it. THE ABOVE MAKES SENSE. The code says _exactly_ what
the module wants to do: it wants to find the AGP support, and if it cannot
find the AGP support it wants to load them.

The arguments about how the user should load things in some specific order
or whatever are complete crap. All the support is there, and whining about
it is not going to change my opinion in the least.

Linus


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct-unmap in 2.4.0 go?

[David Woodhouse]

On Sun, 14 Jan 2001, Linus Torvalds wrote:

 This is what "request_module()" and "kmod" is all about. Once we probe the
 hardware, the drievr itself can ask for more drivers.
 
 I completely fail to see the arguments that have been brought up for drm
 doing ugly things. The code should simply do
 
   drm_agp_head_t * head = inter_module_get("drm_agp");
 
   if (!head) {
   request_module("drm-agp");
   head = inter_module_get("drm_agp");
   if (!head)
   return -ENOAGP;
   }
 
 and be done with it. THE ABOVE MAKES SENSE. The code says _exactly_ what
 the module wants to do: it wants to find the AGP support, and if it cannot
 find the AGP support it wants to load them.

It's the same with CFI command-set-specific code. Except that the 
command-set specific code didn't previously have to be initialised at all, 
and now we've got to initialise it (and have it call 
inter_module_register) before it's required by the cfi_probe code.

The difference here is that while drm_agp actually had to do some hardware
initialisation, the CFI command set handlers didn't - the only thing their
module_init routine does is call inter_module_register(). So we've
introduced the init order dependencies where previously they weren't
necessary.

That's the one flaw in the inter_module_get() stuff - we could do with a
way to put entries in the table at _compile_ time, rather than _only_ at
run time. 

While I accept that we can't eliminate init order dependencies completely,
I still think we should avoid them where it's possible. Which it would be
in this case, without much difficulty at all.

-- 
dwmw2


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct-unmap in 2.4.0 go?

[Linus Torvalds]

On Sun, 14 Jan 2001, David Woodhouse wrote:
 
 That's the one flaw in the inter_module_get() stuff - we could do with a
 way to put entries in the table at _compile_ time, rather than _only_ at
 run time. 

Ok, I can buy that. Not having to initialize explicitly would be nice, but
if so we should make module loading do it automatically too, so that we
don't generate unnecessary differences between module and compiled in (ie
I'd rather avoid the situation that "if you're a module, you need to
explicitly export your inter_module_stuff(), while if you're compiled-in
it will be exported automatically").

Linus

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct-unmap in 2.4.0 go?

[David Woodhouse]

On Sun, 14 Jan 2001, Linus Torvalds wrote:
 On Sun, 14 Jan 2001, David Woodhouse wrote:
  That's the one flaw in the inter_module_get() stuff - we could do with a
  way to put entries in the table at _compile_ time, rather than _only_ 
  at run time. 

 Ok, I can buy that. Not having to initialize explicitly would be nice, but
 if so we should make module loading do it automatically too, so that we
 don't generate unnecessary differences between module and compiled in (ie
 I'd rather avoid the situation that "if you're a module, you need to
 explicitly export your inter_module_stuff(), while if you're compiled-in
 it will be exported automatically").

Yep. Modutils can probably handle that case without too much difficulty, 
if we go with sticking the static inter_module_entries in a special ELF 
section as I originally suggested. Keith?

-- 
dwmw2




-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct-unmap in 2.4.0 go?

[Keith Owens]

On Sun, 14 Jan 2001 13:47:29 -0800 (PST), 
Linus Torvalds [EMAIL PROTECTED] wrote:
On Sun, 14 Jan 2001, David Woodhouse wrote:
 That's the one flaw in the inter_module_get() stuff - we could do with a
 way to put entries in the table at _compile_ time, rather than _only_ at
 run time. 

Ok, I can buy that. Not having to initialize explicitly would be nice, but
if so we should make module loading do it automatically too ...

It might be nice but it is also expensive.  Adding static
inter_module_xxx tables requires

* changes to linux/modules.h to define the new table format and
* changes to vmlinux.lds for _every_ architecture to bring all the
  static tables together in vmlinux and
* new initialisation code in module.c to read and load all the static
  tables at boot time and
* extra code in modutils to find any static tables in modules and
* an extension to struct modules to let modutils pass information about
  the static tables to the kernel and
* the kernel code will only work with an upgraded modutils.

That is a lot of work for a very few special cases.  OTOH, you could
just add a few lines of __initcall code in two source files (which I
did when I wrote inter_module_xxx) and swap the order of 3 lines in
drivers/mtd/Makefile.  Guess which alternative I am going for?

IMHO any automatic method that relies on ELF sections and/or modutils
support is the wrong approach, it is a complex solution with external
dependencies when we already have a simple solution with no external
dependencies.  What next, static tables for file system registration,
for device registration?

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct->unmap in 2.4.0 go?

[Linus Torvalds]

In article <[EMAIL PROTECTED]>,
David Woodhouse  <[EMAIL PROTECTED]> wrote:
>
>We don't need to overdesign it. get_module_symbol() basically provided
>this for us. The only thing really wrong with it was the lack of use
>count handling, which I fixed a while ago.

NO NO NO!

You miss _entirely_ the reason why "get_module_symbol()" was removed,
and why I will not _ever_ accept it coming back.

Hint #1: get_MODULE_symbol().

Hint #2: compiled in functionality.

The fact is, that get_module_symbol() was seriously and totally
mis-designed from the very beginning, and it was removed for THAT
reason, and it had nothing at all to do with the count handling.

So stop this discussion. It's not coming back. Live with the current
interfaces.

Linus

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct->unmap in 2.4.0 go?

[Keith Owens]

On Sat, 13 Jan 2001 15:09:40 + (GMT), 
David Woodhouse <[EMAIL PROTECTED]> wrote:
>Lack of [module symbol] runtime typechecking isn't a showstopper.

It is when users try to insert modules from kernel A into kernel B when
the ABI changed between A and B.  This is not type checking to catch
kernel programmers, it is ABI checking to catch user errors.

This is becoming more important as the kernel moves towards hot
plugging devices, especially for binary only drivers.  It is far better
for the kernel community if modutils can say "cannot load module foo
because its interfaces do not match the kernel, upgrade module foo".
That forces the maintenance load back onto the binary supplier and
removes the questions from l-k, including many of the oops reports with
binary only drivers in the module list.

Module symbol versions are the only way to catch ABI changes.  I do not
want to add a mechanism for accessing symbols dynamically if it cannot
detect ABI changes, it leaves the kernel open to difficult to diagnose
user errors.  I'm doing the hard work now to save everybody time later.

Ignore the fact that the existing module symbol version implementation
is broken as designed.  http://gear.torque.net/kbuild/archive/1280.html
lists the major problems with make dep, genksyms has all those problems
plus several of its own.  As part of the Makefile rewrite for 2.5, I am
redesigning module symbol versions from scratch.

I agree that inter_module_xxx does not check ABI.  That was not for
lack of trying, but it cannot be done in 2.4, it needs a major redesign
of module symbols and the makefiles.  It will be possible in 2.5.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct->unmap in 2.4.0 go?

[Russell King]

David Woodhouse writes:
> We don't need to overdesign it. get_module_symbol() basically provided
> this for us. The only thing really wrong with it was the lack of use
> count handling, which I fixed a while ago.

And the fact that it doesn't work if you turn module support off, which
you'd want to do on an embedded kernel.  Unfortunately, this is one of
the times when you do want the MTD stuff.  You either have to put up
with no MTD support, write your own, or put up with the extra space of
module symbols.

Therefore, get_module_symbol() as it stood was the wrong interface to
use, and I completely agree with Keiths decision to remove it.  However,
I'm not sure that the inter_* stuff that replaced it is much better for
the reasons David has highlighted previously wrt link ordering.
   _
  |_| - ---+---+-
  |   | Russell King[EMAIL PROTECTED]  --- ---
  | | | | http://www.arm.linux.org.uk/personal/aboutme.html   /  /  |
  | +-+-+ --- -+-
  /   |   THE developer of ARM Linux  |+| /|\
 /  | | | ---  |
+-+-+ -  /\\\  |
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct->unmap in 2.4.0 go?

[David Woodhouse]

On Sat, 13 Jan 2001, Keith Owens wrote:

> BTW, modutils cannot automatically fill in upward references when a
> module is loaded.  A reference is a use count, an automatic reference
> would be an automatic use count with no way of removing it.  Code that
> calls upwards to a symbol must perform an overt action to get the
> reference and cope with the case when the symbol is not there.  Think
> of it as a probe, "do I have facility XXX yet?".  It is up to the
> caller to probe as often as required.  Hot plugging for symbols, wheee!

We don't need to overdesign it. get_module_symbol() basically provided
this for us. The only thing really wrong with it was the lack of use
count handling, which I fixed a while ago.

Lack of runtime typechecking isn't a showstopper. Otherwise we'd have
thrown out GCC by now and rewritten the kernel in Modula-3.

That leaves the IA64 problem.

> Fixing this would mean tweaking get_module_symbol() on IA64 to
> recognise that the symbol is a function, build a function descriptor on
> the fly and return the address of the descriptor.  But the information
> about the types of symbols is not available in the kernel nor in
> modules after they are loaded, get_module_symbol() cannot differentiate
> between functions and anything else.  There is also the small matter of
> grubbing around in the arch dependent bit of struct modules to find the
> global pointer for the target function, more complexity.

This is already handled by modutils, presumably. How? By 'grubbing arouund
in the arch dependent bit of struct modules'? There's already a handful of
gp handling surrounded by #ifdef __alpha__ in module.c which doesn't seem
too unreasonable.

The caller knows what it's expecting to find. How about separate
get_module_function() and get_module_data() routines? Which are identical
on most architectures, but on (Alpha and?) IA64 could be defined to return
an appropriate structure.

> get_module_symbol() was usually used with a cast from unsigned long to
> some type that was defined in the calling .c file.  That made the
> caller code responsible for using the correct declaration.  It is
> better to define interfaces as shared data in a shared header.  Not
> perfect, but better.

We could quite happily define the function type in a shared header file,
and coding the original function and the subsequent cast from
get_module_symbol() using that definition.

Conversely, nothing stops you from using only local definitions for
the inter_module_xxx objects, rather than a single shared definition.

Nothing's changed. You just changed the users to use shared definitions
while you converted them to inter_module_xxx. But there's no fundamental
difference in the interface used, in this respect.

> With inter_module_xxx you have no type checking on the registered data.
> But you do not have to use EXPORT_SYMBOL_NOVERS on the shared symbols
> which means that any other users of the symbols will still get type
> checking.  Again, not perfect, but better.

Slightly. But for the cases where inter_module_xxx or get_module_symbol
are used,

A. AFAIK there are no such 'direct' users who get the
benefit of the runtime typechecking.

B. The authors are already having to pay attention to any
changes in the type of the exported data, rather than
just pretending that they're writing Java code and
expecting the runtime system to wipe the dribble
from their chins.

-- 
dwmw2


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct->unmap in 2.4.0 go?

[Keith Owens]

On Sat, 13 Jan 2001 12:46:00 +0100, 
Christian Zander <[EMAIL PROTECTED]> wrote:
>I see what you mean. What do you suggest should be done in the context of
>the driver? As you can easily tell, I'm not overly familiar with the
>internal workings of the kernel. That and the mere impossibility to get
>any kind of help at the mere mention of the Nvidia driver module ("go bitch
>at nvidia", "who cares", ...) do not exactly make it easier to fix problems
>that arise from changes to the kernel.

Hmm, can I charge Nvidia for this fix?

The only reason you are looking at symbols is to map Nvidia registry
names to module symbols.  There are 9 registry names, 4 of which are
#ifdeffed out.

MODULE_PARM(NVreg_resman_debuglevel, "i");
MODULE_PARM(NVreg_VideoMemoryTypeOverride, "i");#ifdeffed out
MODULE_PARM(NVreg_EnableVia4x, "i");
MODULE_PARM(NVreg_ReqAGPRate, "i"); #ifdeffed out
MODULE_PARM(NVreg_SkipBiosPost, "i");   #ifdeffed out
MODULE_PARM(NVreg_UseKernelAGP, "i");
MODULE_PARM(NVreg_UpdateKernelAGP, "i");#ifdeffed out
MODULE_PARM(NVreg_ReqAGPSBA, "i");
MODULE_PARM(NVreg_ReqAGPFW, "i");

Simple fix is an array to map names to addresses.

struct {
const char *name;
int *value;
} linux_registry[] = {
{ "resman_debuglevel",  _resman_debuglevel },
{ "EnableVia4x",_EnableVia4x },
{ "UseKernelAGP",   _UseKernelAGP },
{ "ReqAGPSBA",  _ReqAGPSBA },
{ "ReqAGPFW",   _ReqAGPFW },
};

Changing osRegistryLookup to scan that array for the registry name and
return the address of the corresponding variable is left as an exercise
for the reader.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct->unmap in 2.4.0 go?

[Keith Owens]

On Sat, 13 Jan 2001 10:46:44 + (GMT), 
David Woodhouse <[EMAIL PROTECTED]> wrote:
>On Sat, 13 Jan 2001, Keith Owens wrote:
>Actually, my testing showed that modutils was quite OK with symbols which
>may or may not be present. But compiling such code into the kernel, at
>least on MIPS and m68k, didn't work.

Weak symbols were added to gcc somewhere between 2.7.2.3 and 2.91.66.
At least two architectures are using versions of gcc that predate (by a
few days) the addition of weak symbols.  Davem hit this problem on
sparc with the weak references to kallsyms, he had to define the
symbols instead of letting them resolve to zero, gcc on sparc silently
ignored weak.

>I doubt this would have implemented weak symbols completely, though.
>Fixing up the reference in weaktest.o if myfun.o was loaded later, etc.
>And we don't really want to 'fix' that either.

Weak is not enough.  We need dynamic symbol binding if we plan to
support a cooperative model for objects instead of a strict hierarchic
model.

BTW, modutils cannot automatically fill in upward references when a
module is loaded.  A reference is a use count, an automatic reference
would be an automatic use count with no way of removing it.  Code that
calls upwards to a symbol must perform an overt action to get the
reference and cope with the case when the symbol is not there.  Think
of it as a probe, "do I have facility XXX yet?".  It is up to the
caller to probe as often as required.  Hot plugging for symbols, wheee!

>> That last point is especially important for IA64 where function
>> pointers do not reference the function directly, instead they point to
>> a function descriptor with two fields, one of which is the function
>> address.  Casting the unsigned long address of a function into a
>> function pointer fails miserably on IA64, and gcc does not even give
>> any warnings.  foo = (int (*)(int))get_module_symbol(NULL, "funcname")
>> is architecture dependent.
>
>But fixable.

Probably not.  The generated IA64 object code for this case is
completely wrong, not surprising since we are lying to gcc.

x.o: file format elf64-ia64-little

Disassembly of section .text:

 :
unsigned long value = 0xc0002000;   // example, not a real ia64 address

void test1(void)
{
   0:  [MII]   alloc r34=ar.pfs,4,4,0
void (*test2)(void);
   6:  mov r35=r12  // 
   c:  adds r12=-16,r12 // adjust stack pointer
  10:  [MII]   nop.m 0x0
test2 = (void (*)(void))value;
  16:  mov r33=b0   // save return address
12: GPREL22 value
  1c:  addl r14=0,r1;;  // 
  20:  [MMI]   ld8 r14=[r14];;  // value
  26:  st8 [r35]=r14// test2 = value 0xc0002000
  2c:  nop.i 0x0
test2();
  30:  [MII]   ld8 r14=[r35]// read test2, 0xc0002000
  36:  mov r32=r1;; // save current global pointer
  3c:  nop.i 0x0
  40:  [MII]   ld8 r15=[r14]// dereference test2, wrong
  46:  adds r14=8,r14;; // test2 + 8, 0xc0002008
  4c:  nop.i 0x0
  50:  [MII]   ld8 r1=[r14] // new gp from 0xc0002008, wrong
  56:  mov b6=r15;; // & target function, wrong
  5c:  nop.i 0x0
  60:  [MIB]   nop.m 0x0
  66:  nop.i 0x0
  6c:  br.call.sptk.many b0=b6;; // call indirect function, oops
  70:  [MII]   mov r1=r32   // restore gp

By casting 'test2 = (void (*)(void))value' we claim that value is the
address of the the function descriptor which must contain

  { actual address of function, global data pointer for function }

gcc trusts us and tries to use the data at location 0xc0002000 as a
function descriptor.  Because get_module_symbol() returns the address
of the first instruction in a function, that code would load the actual
address and global pointer from the first 16 bytes of the function's
code area.  Needless to say, it does not work.

Fixing this would mean tweaking get_module_symbol() on IA64 to
recognise that the symbol is a function, build a function descriptor on
the fly and return the address of the descriptor.  But the information
about the types of symbols is not available in the kernel nor in
modules after they are loaded, get_module_symbol() cannot differentiate
between functions and anything else.  There is also the small matter of
grubbing around in the arch dependent bit of struct modules to find the
global pointer for the target function, more complexity.

>But what about keeping a separate table, with PUBLISH_SYMBOL() or
>something slightly more sensibly named? That way, get_published_symbol()
>can only get at those symbols which were supposed to be listed, and if
>someone 

Re: Where did vm_operations_struct->unmap in 2.4.0 go?

[Christian Zander]

On Sat, Jan 13, 2001 at 12:11:31PM +1100, [EMAIL PROTECTED] wrote:
> My apologies.  I read the patch, not the full source code and the patch
> does not have enough programming context to show that the driver is
> only searching its own symbol space.  In my own defense, the references
> to spinlock_t unload_lock and MOD_CAN_QUERY(mp) in the patch are highly
> misleading, those statements only make sense when you are looking at a
> symbol table for another module.  When searching your own symbol table
> the current module must be live with a non-zero use count, not being
> unloaded and it can always be queried.
> 
> >Contrary to what you're saying, the patch does not just inline the old
> >get_module_symbol algorithm nor does it access any of module.c's internal
> >data.
> 
> unload_lock and MOD_CAN_QUERY were copied verbatim from the old
> get_module_symbol, even though they are completely unnecessary.  That
> looks like inlining the old algorithm to me.
> 
> struct module_symbol, mp->nsyms and mp->syms are module.c internal
> data.  If it is ever necessary to change those structures, nothing
> outside module.c, the 32/64 handlers for module system calls and
> modutils should be affected.  Now if I change module_symbol, other bits
> of the kernel will unexpectedly break, this is not good.

I see what you mean. What do you suggest should be done in the context of
the driver? As you can easily tell, I'm not overly familiar with the 
internal workings of the kernel. That and the mere impossibility to get 
any kind of help at the mere mention of the Nvidia driver module ("go bitch
at nvidia", "who cares", ...) do not exactly make it easier to fix problems 
that arise from changes to the kernel. 

--
--
 christian zander  we come to bury dos, not to praise it.
 [EMAIL PROTECTED]-- paul vojta
--

 PGP signature

Re: Where did vm_operations_struct->unmap in 2.4.0 go?

[David Woodhouse]

On Sat, 13 Jan 2001, Keith Owens wrote:

> Over emphasis for humorous effect.  Must remember to add smiley.

Heh. But it does deserve to get into the fortune file.

> What this patch and David Woodhouse's comments show is that I need to
> look at a generic and safe mechanism for kernel/module symbol lookup.
> The existing static mechanism works for fixed symbol names but does not
> work for symbol names that are generated at run time nor for symbols
> that may or may not be present.

Actually, my testing showed that modutils was quite OK with symbols which
may or may not be present. But compiling such code into the kernel, at
least on MIPS and m68k, didn't work.

cat >weaktest.c <

extern char *myfun(void) __attribute__((weak));

int init_module(void)
{
char *txt= "myfun() not present\n";

if (myfun)
txt = myfun();

printk(txt);
return 0;
}
EOF
cat > myfun.c <

char *myfun(void)
{
return "Hello World\n";
}
EOF

I doubt this would have implemented weak symbols completely, though.
Fixing up the reference in weaktest.o if myfun.o was loaded later, etc.
And we don't really want to 'fix' that either.

So it'd still have needed request_module(); get_module_symbol() if it
found that myfun wasn't present and it needed it. So it might as well have
used get_module_symbol() from the start instead of the weak declaration.

> get_module_symbol() "worked" but was horribly unsafe.  It broke with
> module versions, it did zero type checking which left the code open to
> version skew and it assumed that all addresses are equivalent to an
> unsigned long.
>
> That last point is especially important for IA64 where function
> pointers do not reference the function directly, instead they point to
> a function descriptor with two fields, one of which is the function
> address.  Casting the unsigned long address of a function into a
> function pointer fails miserably on IA64, and gcc does not even give
> any warnings.  foo = (int (*)(int))get_module_symbol(NULL, "funcname")
> is architecture dependent.

But fixable.

> Using EXPORT_SYMBOL_NOVERS() to "fix" the modversions problem for
> get_module_symbol() removes all inter module checks on the relevant
> symbols.  Not just for the caller of get_module_symbol for all modules
> that access those symbols.  This leaves too much code open to version
> skew and is not acceptable.

I'm not sure there's anything which was expected to be obtained by
get_module_symbol() which was also obtained by normal linking. The nature
of these routines is that they're optional. Usually, the routine would be
optional for all callers or it'd be mandatory for all callers. Rarely a
mixture.

But what about keeping a separate table, with PUBLISH_SYMBOL() or
something slightly more sensibly named? That way, get_published_symbol()
can only get at those symbols which were supposed to be listed, and if
someone really wants EXPORT_SYMBOL() and PUBLISH_SYMBOL() they can do
that.

> > inter_module_xxx is modversions safe.  It still does no type checking
> because it uses void * for the data structure, but the exporter and
> user have to declare their common data area which reduces the chance of
> version skew.

I'm not sure I follow. The exporter and the user have to each declare
their common data area, which means they don't have to declare it
identically, and there's just as much chance of version skew as before,
surely? With get_module_symbol both had to declare it, too.

And 'modversions safe' just means that there's no attempt to mangle the
names so it's identical to the EXPORT_SYMBOL_NOVERS case above?

> I am still not happy about this possibility of skew but
> anything is better than no checks at all.  Passing a data structure
> which contains real declarations for function pointers instead of
> assuming you can cast a number to a function pointer makes
> inter_module_xxx architecture independent.

On the other hand, it's also simple enough to define a macro which does
the arch-dependent equivalent of the 'foo=(fnptr_t)get_module_symbol()'
above, isn't it? But I'm not particularly attached to the method. The
static initialisation is what I miss.

> I will look at a general kernel and module symbol lookup routine that
> does the job properly.

Thanks.

> The hard part is making sure that the provider and consumer have
> exactly the same types for a symbol.

That would be useful, but it's not 100% imperative. Runtime type-checking
is a nice sanity check where it's almost free, as it was in the
modversions case, but I don't want to program in Java.

-- 
dwmw2

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct-unmap in 2.4.0 go?

[David Woodhouse]

On Sat, 13 Jan 2001, Keith Owens wrote:

 Over emphasis for humorous effect.  Must remember to add smiley.

Heh. But it does deserve to get into the fortune file.

 What this patch and David Woodhouse's comments show is that I need to
 look at a generic and safe mechanism for kernel/module symbol lookup.
 The existing static mechanism works for fixed symbol names but does not
 work for symbol names that are generated at run time nor for symbols
 that may or may not be present.

Actually, my testing showed that modutils was quite OK with symbols which
may or may not be present. But compiling such code into the kernel, at
least on MIPS and m68k, didn't work.

cat weaktest.c EOF
#include linux/module.h

extern char *myfun(void) __attribute__((weak));

int init_module(void)
{
char *txt= "myfun() not present\n";

if (myfun)
txt = myfun();

printk(txt);
return 0;
}
EOF
cat  myfun.c EOF
#include linux/module.h

char *myfun(void)
{
return "Hello World\n";
}
EOF

I doubt this would have implemented weak symbols completely, though.
Fixing up the reference in weaktest.o if myfun.o was loaded later, etc.
And we don't really want to 'fix' that either.

So it'd still have needed request_module(); get_module_symbol() if it
found that myfun wasn't present and it needed it. So it might as well have
used get_module_symbol() from the start instead of the weak declaration.

 get_module_symbol() "worked" but was horribly unsafe.  It broke with
 module versions, it did zero type checking which left the code open to
 version skew and it assumed that all addresses are equivalent to an
 unsigned long.

 That last point is especially important for IA64 where function
 pointers do not reference the function directly, instead they point to
 a function descriptor with two fields, one of which is the function
 address.  Casting the unsigned long address of a function into a
 function pointer fails miserably on IA64, and gcc does not even give
 any warnings.  foo = (int (*)(int))get_module_symbol(NULL, "funcname")
 is architecture dependent.

But fixable.

 Using EXPORT_SYMBOL_NOVERS() to "fix" the modversions problem for
 get_module_symbol() removes all inter module checks on the relevant
 symbols.  Not just for the caller of get_module_symbol for all modules
 that access those symbols.  This leaves too much code open to version
 skew and is not acceptable.

I'm not sure there's anything which was expected to be obtained by
get_module_symbol() which was also obtained by normal linking. The nature
of these routines is that they're optional. Usually, the routine would be
optional for all callers or it'd be mandatory for all callers. Rarely a
mixture.

But what about keeping a separate table, with PUBLISH_SYMBOL() or
something slightly more sensibly named? That way, get_published_symbol()
can only get at those symbols which were supposed to be listed, and if
someone really wants EXPORT_SYMBOL() and PUBLISH_SYMBOL() they can do
that.

  inter_module_xxx is modversions safe.  It still does no type checking
 because it uses void * for the data structure, but the exporter and
 user have to declare their common data area which reduces the chance of
 version skew.

I'm not sure I follow. The exporter and the user have to each declare
their common data area, which means they don't have to declare it
identically, and there's just as much chance of version skew as before,
surely? With get_module_symbol both had to declare it, too.

And 'modversions safe' just means that there's no attempt to mangle the
names so it's identical to the EXPORT_SYMBOL_NOVERS case above?

 I am still not happy about this possibility of skew but
 anything is better than no checks at all.  Passing a data structure
 which contains real declarations for function pointers instead of
 assuming you can cast a number to a function pointer makes
 inter_module_xxx architecture independent.

On the other hand, it's also simple enough to define a macro which does
the arch-dependent equivalent of the 'foo=(fnptr_t)get_module_symbol()'
above, isn't it? But I'm not particularly attached to the method. The
static initialisation is what I miss.

 I will look at a general kernel and module symbol lookup routine that
 does the job properly.

Thanks.

 The hard part is making sure that the provider and consumer have
 exactly the same types for a symbol.

That would be useful, but it's not 100% imperative. Runtime type-checking
is a nice sanity check where it's almost free, as it was in the
modversions case, but I don't want to program in Java.

-- 
dwmw2

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct-unmap in 2.4.0 go?

[Christian Zander]

On Sat, Jan 13, 2001 at 12:11:31PM +1100, [EMAIL PROTECTED] wrote:
 My apologies.  I read the patch, not the full source code and the patch
 does not have enough programming context to show that the driver is
 only searching its own symbol space.  In my own defense, the references
 to spinlock_t unload_lock and MOD_CAN_QUERY(mp) in the patch are highly
 misleading, those statements only make sense when you are looking at a
 symbol table for another module.  When searching your own symbol table
 the current module must be live with a non-zero use count, not being
 unloaded and it can always be queried.
 
 Contrary to what you're saying, the patch does not just inline the old
 get_module_symbol algorithm nor does it access any of module.c's internal
 data.
 
 unload_lock and MOD_CAN_QUERY were copied verbatim from the old
 get_module_symbol, even though they are completely unnecessary.  That
 looks like inlining the old algorithm to me.
 
 struct module_symbol, mp-nsyms and mp-syms are module.c internal
 data.  If it is ever necessary to change those structures, nothing
 outside module.c, the 32/64 handlers for module system calls and
 modutils should be affected.  Now if I change module_symbol, other bits
 of the kernel will unexpectedly break, this is not good.

I see what you mean. What do you suggest should be done in the context of
the driver? As you can easily tell, I'm not overly familiar with the 
internal workings of the kernel. That and the mere impossibility to get 
any kind of help at the mere mention of the Nvidia driver module ("go bitch
at nvidia", "who cares", ...) do not exactly make it easier to fix problems 
that arise from changes to the kernel. 

--
--
 christian zander  we come to bury dos, not to praise it.
 [EMAIL PROTECTED]-- paul vojta
--

 PGP signature

Re: Where did vm_operations_struct-unmap in 2.4.0 go?

[Keith Owens]

On Sat, 13 Jan 2001 10:46:44 + (GMT), 
David Woodhouse [EMAIL PROTECTED] wrote:
On Sat, 13 Jan 2001, Keith Owens wrote:
Actually, my testing showed that modutils was quite OK with symbols which
may or may not be present. But compiling such code into the kernel, at
least on MIPS and m68k, didn't work.

Weak symbols were added to gcc somewhere between 2.7.2.3 and 2.91.66.
At least two architectures are using versions of gcc that predate (by a
few days) the addition of weak symbols.  Davem hit this problem on
sparc with the weak references to kallsyms, he had to define the
symbols instead of letting them resolve to zero, gcc on sparc silently
ignored weak.

I doubt this would have implemented weak symbols completely, though.
Fixing up the reference in weaktest.o if myfun.o was loaded later, etc.
And we don't really want to 'fix' that either.

Weak is not enough.  We need dynamic symbol binding if we plan to
support a cooperative model for objects instead of a strict hierarchic
model.

BTW, modutils cannot automatically fill in upward references when a
module is loaded.  A reference is a use count, an automatic reference
would be an automatic use count with no way of removing it.  Code that
calls upwards to a symbol must perform an overt action to get the
reference and cope with the case when the symbol is not there.  Think
of it as a probe, "do I have facility XXX yet?".  It is up to the
caller to probe as often as required.  Hot plugging for symbols, wheee!

 That last point is especially important for IA64 where function
 pointers do not reference the function directly, instead they point to
 a function descriptor with two fields, one of which is the function
 address.  Casting the unsigned long address of a function into a
 function pointer fails miserably on IA64, and gcc does not even give
 any warnings.  foo = (int (*)(int))get_module_symbol(NULL, "funcname")
 is architecture dependent.

But fixable.

Probably not.  The generated IA64 object code for this case is
completely wrong, not surprising since we are lying to gcc.

x.o: file format elf64-ia64-little

Disassembly of section .text:

 test1:
unsigned long value = 0xc0002000;   // example, not a real ia64 address

void test1(void)
{
   0:  [MII]   alloc r34=ar.pfs,4,4,0
void (*test2)(void);
   6:  mov r35=r12  // test2
   c:  adds r12=-16,r12 // adjust stack pointer
  10:  [MII]   nop.m 0x0
test2 = (void (*)(void))value;
  16:  mov r33=b0   // save return address
12: GPREL22 value
  1c:  addl r14=0,r1;;  // value
  20:  [MMI]   ld8 r14=[r14];;  // value
  26:  st8 [r35]=r14// test2 = value 0xc0002000
  2c:  nop.i 0x0
test2();
  30:  [MII]   ld8 r14=[r35]// read test2, 0xc0002000
  36:  mov r32=r1;; // save current global pointer
  3c:  nop.i 0x0
  40:  [MII]   ld8 r15=[r14]// dereference test2, wrong
  46:  adds r14=8,r14;; // test2 + 8, 0xc0002008
  4c:  nop.i 0x0
  50:  [MII]   ld8 r1=[r14] // new gp from 0xc0002008, wrong
  56:  mov b6=r15;; //  target function, wrong
  5c:  nop.i 0x0
  60:  [MIB]   nop.m 0x0
  66:  nop.i 0x0
  6c:  br.call.sptk.many b0=b6;; // call indirect function, oops
  70:  [MII]   mov r1=r32   // restore gp

By casting 'test2 = (void (*)(void))value' we claim that value is the
address of the the function descriptor which must contain

  { actual address of function, global data pointer for function }

gcc trusts us and tries to use the data at location 0xc0002000 as a
function descriptor.  Because get_module_symbol() returns the address
of the first instruction in a function, that code would load the actual
address and global pointer from the first 16 bytes of the function's
code area.  Needless to say, it does not work.

Fixing this would mean tweaking get_module_symbol() on IA64 to
recognise that the symbol is a function, build a function descriptor on
the fly and return the address of the descriptor.  But the information
about the types of symbols is not available in the kernel nor in
modules after they are loaded, get_module_symbol() cannot differentiate
between functions and anything else.  There is also the small matter of
grubbing around in the arch dependent bit of struct modules to find the
global pointer for the target function, more complexity.

But what about keeping a separate table, with PUBLISH_SYMBOL() or
something slightly more sensibly named? That way, get_published_symbol()
can only get at those symbols which were supposed to be listed, and if
someone really wants 

Re: Where did vm_operations_struct-unmap in 2.4.0 go?

[Keith Owens]

On Sat, 13 Jan 2001 12:46:00 +0100, 
Christian Zander [EMAIL PROTECTED] wrote:
I see what you mean. What do you suggest should be done in the context of
the driver? As you can easily tell, I'm not overly familiar with the
internal workings of the kernel. That and the mere impossibility to get
any kind of help at the mere mention of the Nvidia driver module ("go bitch
at nvidia", "who cares", ...) do not exactly make it easier to fix problems
that arise from changes to the kernel.

Hmm, can I charge Nvidia for this fix?

The only reason you are looking at symbols is to map Nvidia registry
names to module symbols.  There are 9 registry names, 4 of which are
#ifdeffed out.

MODULE_PARM(NVreg_resman_debuglevel, "i");
MODULE_PARM(NVreg_VideoMemoryTypeOverride, "i");#ifdeffed out
MODULE_PARM(NVreg_EnableVia4x, "i");
MODULE_PARM(NVreg_ReqAGPRate, "i"); #ifdeffed out
MODULE_PARM(NVreg_SkipBiosPost, "i");   #ifdeffed out
MODULE_PARM(NVreg_UseKernelAGP, "i");
MODULE_PARM(NVreg_UpdateKernelAGP, "i");#ifdeffed out
MODULE_PARM(NVreg_ReqAGPSBA, "i");
MODULE_PARM(NVreg_ReqAGPFW, "i");

Simple fix is an array to map names to addresses.

struct {
const char *name;
int *value;
} linux_registry[] = {
{ "resman_debuglevel",  NVreg_resman_debuglevel },
{ "EnableVia4x",NVreg_EnableVia4x },
{ "UseKernelAGP",   NVreg_UseKernelAGP },
{ "ReqAGPSBA",  NVreg_ReqAGPSBA },
{ "ReqAGPFW",   NVreg_ReqAGPFW },
};

Changing osRegistryLookup to scan that array for the registry name and
return the address of the corresponding variable is left as an exercise
for the reader.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct-unmap in 2.4.0 go?

[David Woodhouse]

On Sat, 13 Jan 2001, Keith Owens wrote:

 BTW, modutils cannot automatically fill in upward references when a
 module is loaded.  A reference is a use count, an automatic reference
 would be an automatic use count with no way of removing it.  Code that
 calls upwards to a symbol must perform an overt action to get the
 reference and cope with the case when the symbol is not there.  Think
 of it as a probe, "do I have facility XXX yet?".  It is up to the
 caller to probe as often as required.  Hot plugging for symbols, wheee!

We don't need to overdesign it. get_module_symbol() basically provided
this for us. The only thing really wrong with it was the lack of use
count handling, which I fixed a while ago.

Lack of runtime typechecking isn't a showstopper. Otherwise we'd have
thrown out GCC by now and rewritten the kernel in Modula-3.

That leaves the IA64 problem.

 Fixing this would mean tweaking get_module_symbol() on IA64 to
 recognise that the symbol is a function, build a function descriptor on
 the fly and return the address of the descriptor.  But the information
 about the types of symbols is not available in the kernel nor in
 modules after they are loaded, get_module_symbol() cannot differentiate
 between functions and anything else.  There is also the small matter of
 grubbing around in the arch dependent bit of struct modules to find the
 global pointer for the target function, more complexity.

This is already handled by modutils, presumably. How? By 'grubbing arouund
in the arch dependent bit of struct modules'? There's already a handful of
gp handling surrounded by #ifdef __alpha__ in module.c which doesn't seem
too unreasonable.

The caller knows what it's expecting to find. How about separate
get_module_function() and get_module_data() routines? Which are identical
on most architectures, but on (Alpha and?) IA64 could be defined to return
an appropriate structure.

 get_module_symbol() was usually used with a cast from unsigned long to
 some type that was defined in the calling .c file.  That made the
 caller code responsible for using the correct declaration.  It is
 better to define interfaces as shared data in a shared header.  Not
 perfect, but better.

We could quite happily define the function type in a shared header file,
and coding the original function and the subsequent cast from
get_module_symbol() using that definition.

Conversely, nothing stops you from using only local definitions for
the inter_module_xxx objects, rather than a single shared definition.

Nothing's changed. You just changed the users to use shared definitions
while you converted them to inter_module_xxx. But there's no fundamental
difference in the interface used, in this respect.

 With inter_module_xxx you have no type checking on the registered data.
 But you do not have to use EXPORT_SYMBOL_NOVERS on the shared symbols
 which means that any other users of the symbols will still get type
 checking.  Again, not perfect, but better.

Slightly. But for the cases where inter_module_xxx or get_module_symbol
are used,

A. AFAIK there are no such 'direct' users who get the
benefit of the runtime typechecking.

B. The authors are already having to pay attention to any
changes in the type of the exported data, rather than
just pretending that they're writing Java code and
expecting the runtime system to wipe the dribble
from their chins.

-- 
dwmw2


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct-unmap in 2.4.0 go?

[Russell King]

David Woodhouse writes:
 We don't need to overdesign it. get_module_symbol() basically provided
 this for us. The only thing really wrong with it was the lack of use
 count handling, which I fixed a while ago.

And the fact that it doesn't work if you turn module support off, which
you'd want to do on an embedded kernel.  Unfortunately, this is one of
the times when you do want the MTD stuff.  You either have to put up
with no MTD support, write your own, or put up with the extra space of
module symbols.

Therefore, get_module_symbol() as it stood was the wrong interface to
use, and I completely agree with Keiths decision to remove it.  However,
I'm not sure that the inter_* stuff that replaced it is much better for
the reasons David has highlighted previously wrt link ordering.
   _
  |_| - ---+---+-
  |   | Russell King[EMAIL PROTECTED]  --- ---
  | | | | http://www.arm.linux.org.uk/personal/aboutme.html   /  /  |
  | +-+-+ --- -+-
  /   |   THE developer of ARM Linux  |+| /|\
 /  | | | ---  |
+-+-+ -  /\\\  |
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct-unmap in 2.4.0 go?

[Keith Owens]

On Sat, 13 Jan 2001 15:09:40 + (GMT), 
David Woodhouse [EMAIL PROTECTED] wrote:
Lack of [module symbol] runtime typechecking isn't a showstopper.

It is when users try to insert modules from kernel A into kernel B when
the ABI changed between A and B.  This is not type checking to catch
kernel programmers, it is ABI checking to catch user errors.

This is becoming more important as the kernel moves towards hot
plugging devices, especially for binary only drivers.  It is far better
for the kernel community if modutils can say "cannot load module foo
because its interfaces do not match the kernel, upgrade module foo".
That forces the maintenance load back onto the binary supplier and
removes the questions from l-k, including many of the oops reports with
binary only drivers in the module list.

Module symbol versions are the only way to catch ABI changes.  I do not
want to add a mechanism for accessing symbols dynamically if it cannot
detect ABI changes, it leaves the kernel open to difficult to diagnose
user errors.  I'm doing the hard work now to save everybody time later.

Ignore the fact that the existing module symbol version implementation
is broken as designed.  http://gear.torque.net/kbuild/archive/1280.html
lists the major problems with make dep, genksyms has all those problems
plus several of its own.  As part of the Makefile rewrite for 2.5, I am
redesigning module symbol versions from scratch.

I agree that inter_module_xxx does not check ABI.  That was not for
lack of trying, but it cannot be done in 2.4, it needs a major redesign
of module symbols and the makefiles.  It will be possible in 2.5.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct-unmap in 2.4.0 go?

[Linus Torvalds]

In article [EMAIL PROTECTED],
David Woodhouse  [EMAIL PROTECTED] wrote:

We don't need to overdesign it. get_module_symbol() basically provided
this for us. The only thing really wrong with it was the lack of use
count handling, which I fixed a while ago.

NO NO NO!

You miss _entirely_ the reason why "get_module_symbol()" was removed,
and why I will not _ever_ accept it coming back.

Hint #1: get_MODULE_symbol().

Hint #2: compiled in functionality.

The fact is, that get_module_symbol() was seriously and totally
mis-designed from the very beginning, and it was removed for THAT
reason, and it had nothing at all to do with the count handling.

So stop this discussion. It's not coming back. Live with the current
interfaces.

Linus

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct->unmap in 2.4.0 go?

[Keith Owens]

On Fri, 12 Jan 2001 20:11:30 +0100, 
Christian Zander <[EMAIL PROTECTED]> wrote:
>Saying that I should have made use of this mechanism for the specific
>code in the Nvidia driver that we are talking about clearly shows that
>you didn't look at it. The module used get_module_symbol to search its
>own symbol table for parameters that may have been passed to it at load
>time.

My apologies.  I read the patch, not the full source code and the patch
does not have enough programming context to show that the driver is
only searching its own symbol space.  In my own defense, the references
to spinlock_t unload_lock and MOD_CAN_QUERY(mp) in the patch are highly
misleading, those statements only make sense when you are looking at a
symbol table for another module.  When searching your own symbol table
the current module must be live with a non-zero use count, not being
unloaded and it can always be queried.

>Contrary to what you're saying, the patch does not just inline the old
>get_module_symbol algorithm nor does it access any of module.c's internal
>data.

unload_lock and MOD_CAN_QUERY were copied verbatim from the old
get_module_symbol, even though they are completely unnecessary.  That
looks like inlining the old algorithm to me.

struct module_symbol, mp->nsyms and mp->syms are module.c internal
data.  If it is ever necessary to change those structures, nothing
outside module.c, the 32/64 handlers for module system calls and
modutils should be affected.  Now if I change module_symbol, other bits
of the kernel will unexpectedly break, this is not good.

>> Whoever coded that patch should be taken out and shot, hung, drawn and
>> quartered then forced to write COBOL for the rest of their natural
>> life.
>
>Excellent comment - it is just as appropriate as it is helpful.

Over emphasis for humorous effect.  Must remember to add smiley.


What this patch and David Woodhouse's comments show is that I need to
look at a generic and safe mechanism for kernel/module symbol lookup.
The existing static mechanism works for fixed symbol names but does not
work for symbol names that are generated at run time nor for symbols
that may or may not be present.

get_module_symbol() "worked" but was horribly unsafe.  It broke with
module versions, it did zero type checking which left the code open to
version skew and it assumed that all addresses are equivalent to an
unsigned long.

That last point is especially important for IA64 where function
pointers do not reference the function directly, instead they point to
a function descriptor with two fields, one of which is the function
address.  Casting the unsigned long address of a function into a
function pointer fails miserably on IA64, and gcc does not even give
any warnings.  foo = (int (*)(int))get_module_symbol(NULL, "funcname")
is architecture dependent.

Using EXPORT_SYMBOL_NOVERS() to "fix" the modversions problem for
get_module_symbol() removes all inter module checks on the relevant
symbols.  Not just for the caller of get_module_symbol for all modules
that access those symbols.  This leaves too much code open to version
skew and is not acceptable.

inter_module_xxx is modversions safe.  It still does no type checking
because it uses void * for the data structure, but the exporter and
user have to declare their common data area which reduces the chance of
version skew.  I am still not happy about this possibility of skew but
anything is better than no checks at all.  Passing a data structure
which contains real declarations for function pointers instead of
assuming you can cast a number to a function pointer makes
inter_module_xxx architecture independent.


I will look at a general kernel and module symbol lookup routine that
does the job properly.  The hard part is making sure that the provider
and consumer have exactly the same types for a symbol.  Both
get_module_symbol and inter_module_xxx completely bypass the
modversions checks and are wide open to undetectable version skew,
although inter_module_xxx is a little bit safer.  Any replacement for
these functions must be able to do type checking at run time, which
probably means it is 2.5 code.  And yes, David, it should be able to
handle static data.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct->unmap in 2.4.0 go?

[Christian Zander]

> >> I ran into this while hacking the Nvidia kernel driver to work with
> >> 2.4.0.  I got the driver working but it's not 100%
> >> 
> >> Also where did get_module_symbol() and put_module_symbol() go?
> >
> >Patches for the NVIDIA binary X drivers following all these kernel
> >changes can be gotten from IRC server irc.openprojects.net, channel
> >#nvidia. Or from http://ex.shafted.com.au/nvidia/
> 
> And what a pile of crud those patches are!!  Instead of using the clean
> replacement interface for get_module_symbol, nvidia/patch-2.4.0-PR hard
> codes the old get_module_symbol algorithm as inline code.
> 
> This patch violates the modules interface by accessing modules.c
> internal data.  It still suffers from all the problems that
> get_module_symbol had.  Because it is hard coded as inline code instead
> of a common function, will be much harder to fix when it breaks.
> 

The way I understand the inter_module mechanism, module A registers one
or several of its symbols using inter_module_register to make it or them 
available to other modules. Module B can then request any of the symbols
with inter_module_request and get a pointer. The inter_module mechanism 
guarantees that the symbol will be available until module B decides that 
the symbol is no longer needed and releases it by calling inter_module_put.

Saying that I should have made use of this mechanism for the specific
code in the Nvidia driver that we are talking about clearly shows that
you didn't look at it. The module used get_module_symbol to search its
own symbol table for parameters that may have been passed to it at load
time. Arguably, this is bad practise, but it is also the reason why using
your mechanism doesn't make any sense. Obviously, the module wouldn't
want to register private data to request it later on; the information
that would have to be passed to inter_module_register is the same that
the code in question intends to retrieve in the first place.

Contrary to what you're saying, the patch does not just inline the old
get_module_symbol algorithm nor does it access any of module.c's internal
data. What is does is to browse the list of the modules's _own_ symbols 
looking for a match. If it finds one, it returns the desired data.

> Whoever coded that patch should be taken out and shot, hung, drawn and
> quartered then forced to write COBOL for the rest of their natural
> life.

Excellent comment - it is just as appropriate as it is helpful.

--
--
 christian zander  we come to bury dos, not to praise it.
 [EMAIL PROTECTED]-- paul vojta
--

 PGP signature

Re: Where did vm_operations_struct->unmap in 2.4.0 go?

[David Woodhouse]

[EMAIL PROTECTED] said:
>  You just proved my point.  It is extremely difficult to deduce the
> required initialisation order by reading an undocumented Makefile
> where the init order is implemented as a side effect of selection
> order.  The existing method implies link order when none is required. 

I agree entirely. But you're confusing the debate on how to satisfy init 
order dependencies with my desire to avoid them altogether (for certain 
situations).



[EMAIL PROTECTED] said:
> I'm no filesystem expert but it appears that some of those core
> initialisation routines must be executed before most filesystems can
> start.  In particular, filelock_init() must be executed before any
> filesystem that supports locks is initialised.

No, before any filesystem that supports locks is _mounted_. Big difference. 
But I'm picking holes in your examples again. I accept that in some cases 
it is necessary, but I still think it's best to avoid it where possible.


[EMAIL PROTECTED] said:
> Firstly inter_module_xxx is only used by that very small subset of
> code where there are no constraints on whether the caller and callee
> can be in kernel, in modules or a mixture _and_ some of the objects
> are optional.  It is a special case because most code handles this
> problem through CONFIG options. If X needs (Y, Z) and X is built into the
> kernel then (Y, Z) must also be built into the kernel.  If Y or Z are
> optional then control the calls to those functions with CONFIG options.
> Almost all of the kernel handles it properly though CONFIG, so
> inter_module_xxx is already a process to handle a few special cases.

'properly through CONFIG'? I thought you agreed that doing it through
preprocessor options was ugly, and that it was preferable to get rid of such
things and deal with the presence or absence of such code cleanly through
some mechanism with similar semantics to inter_module_get(). The 'special 
case' code to which you refer is in fact the first set of such code to 
attempt to deal with this properly rather than just giving up and hacking 
it with preprocessor options.

Stepping back a moment and considering it, what we actually appear to be 
doing is trying to reinvent weak symbols, as far as I can tell. 


[EMAIL PROTECTED] said:
>  Secondly you want static inter_module_xxx tables for a small subset
> of these special cases, where the source has no other initialisation
> code. This is piling special cases on special cases.  Adding static
> inter_module_xxx tables requires 

I want the registration method not to impose init order restrictions where 
previously there were none. Where the previous code already had init order 
ugliness, it's not so much of a problem. But replacing the safe 
get_module_symbol() with the unsafe inter_module_get() just because the AGP 
author forgot to use EXPORT_SYMBOL_NOVERS to export the symbols doesn't 
really strike me as being useful. 

The only real difference between the two, other than the module symbol
mangling, appears to be that inter_module_get() looks stuff up in a dynamic
table which is made at runtime, where get_module_symbol() looks it up in a
static table which is made at compile (or module load) time. That was a
backward step, and wasn't necessary. 

If you object to having both the runtime inter_module_register() and the 
compile-time one, then ditch the dynamic one. Both current users of 
inter_module_xxx would be able to use the static version. 

But to be honest, I'd settle for ditching the whole blinkin' lot and 
getting weak symbols working right, if I could.

--
dwmw2


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct->unmap in 2.4.0 go?

[Keith Owens]

On Fri, 12 Jan 2001 13:01:12 +0100, 
Daniel Phillips <[EMAIL PROTECTED]> wrote:
>Keith Owens wrote:
>> I want to completely remove this multi layered method for setting
>> initialisation order and go back to basics.  I want the programmer to
>> say "initialise E and F after G, H and I".  The kernel build system
>> works out the directed graph of initialisation order then controls the
>> execution of startup code to satisfy this graph.
>
>I don't doubt you will come up with a workable solution at build time. 
>However, working out a valid graph at execution time is trivial and
>efficient, given a list of precedence relations of the kind you're
>suggesting.  In fact you don't even have to work out the graph before
>starting the initialization, it's also trivial to keep a count of
>unsatisfied initialization conditions at the beginning of each
>initialization sequence and block until the count goes to zero.  (In
>essence, evaluate a priority sort on the fly.)

It is safer to evaluate the graph at link time in case somebody
mistakenly codes a dependency loop, that is an abort case.  Finding
that you have a loop at load time and aborting the kernel is a little
too drastic for my tastes.

In any case it is academic.  Linus insists on link order being
explicitly and completely specified by the programmer and he likes the
link order being implied by Makefile order.  So there is no point in
working on a better system.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct->unmap in 2.4.0 go?

[Daniel Phillips]

Keith Owens wrote:
> I want to completely remove this multi layered method for setting
> initialisation order and go back to basics.  I want the programmer to
> say "initialise E and F after G, H and I".  The kernel build system
> works out the directed graph of initialisation order then controls the
> execution of startup code to satisfy this graph.

I don't doubt you will come up with a workable solution at build time. 
However, working out a valid graph at execution time is trivial and
efficient, given a list of precedence relations of the kind you're
suggesting.  In fact you don't even have to work out the graph before
starting the initialization, it's also trivial to keep a count of
unsatisfied initialization conditions at the beginning of each
initialization sequence and block until the count goes to zero.  (In
essence, evaluate a priority sort on the fly.)

--
Daniel
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct->unmap in 2.4.0 go?

[Keith Owens]

On Fri, 12 Jan 2001 10:27:34 + (GMT), 
David Woodhouse <[EMAIL PROTECTED]> wrote:
>On Fri, 12 Jan 2001, Keith Owens wrote:
>> A typical graph would have scsi disk depends on scsi host adaptor group
>> which depends on pci.
>
>No. sd will happily take over any existing devices when as and when they
>arrive. It doesn't have to be loaded last. Likewise, in theory at least,
>host adaptor drivers using the new PCI driver code would respond correctly
>to the PCI code being initialised (and calling their ->probe routine)
>later, although that doesn't happen now.

You just proved my point.  It is extremely difficult to deduce the
required initialisation order by reading an undocumented Makefile where
the init order is implemented as a side effect of selection order.  The
existing method implies link order when none is required.

>> Most of the objects have fairly simple execution dependencies, e.g.
>> all file systems depend on core fs code having already executed.
>
>Er... Why? They call register_filesystem() which uses a lock which is
>staticly initialised. Don't order stuff for the sake of it. If there are
>filesystems which _do_ require the VFS to be initialised first, those
>filesystems can be marked as such. I'm not aware of any.

I was using scsi and fs as examples, no need to pick holes in the
examples.  But ...

  fs/buffer.c:module_init(bdflush_init)
  fs/pipe.c:module_init(init_pipe_fs)
  fs/fcntl.c:module_init(fasync_init)
  fs/locks.c:module_init(filelock_init)
  fs/dnotify.c:module_init(dnotify_init)

I'm no filesystem expert but it appears that some of those core
initialisation routines must be executed before most filesystems can
start.  In particular, filelock_init() must be executed before any
filesystem that supports locks is initialised.

>All I want is a way to staticly add entries to the
>inter_module_xxx tables at compile time, because I _have_ done the
>analysis, and I'm saying that's when they should be made available.

Firstly inter_module_xxx is only used by that very small subset of code
where there are no constraints on whether the caller and callee can be
in kernel, in modules or a mixture _and_ some of the objects are
optional.  It is a special case because most code handles this problem
through CONFIG options.  If X needs (Y, Z) and X is built into the
kernel then (Y, Z) must also be built into the kernel.  If Y or Z are
optional then control the calls to those functions with CONFIG options.
Almost all of the kernel handles it properly though CONFIG, so
inter_module_xxx is already a process to handle a few special cases.

Secondly you want static inter_module_xxx tables for a small subset of
these special cases, where the source has no other initialisation code.
This is piling special cases on special cases.  Adding static
inter_module_xxx tables requires

* changes to linux/modules.h to define the new table format and
* changes to vmlinux.lds for _every_ architecture to bring all the
  static tables together in vmlinux and
* new initialisation code in module.c to read and load all the static
  tables at boot time and
* extra code in modutils to find any static tables in modules and
* an extension to struct modules to let modutils pass information about
  the static tables to the kernel and
* the kernel code will only work with an upgraded modutils.

That is a lot of work for a very few special cases.  OTOH, you could
just swap the order of 3 lines in drivers/mtd/Makefile.  Guess which
alternative I am going for?

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct->unmap in 2.4.0 go?

[David Woodhouse]

On Fri, 12 Jan 2001, Keith Owens wrote:

> People need to realise that the problem is initialisation order,
> nothing more, nothing less.  You have to determine and document the
> startup requirements for your code.

This is true. But I'd also agree with the implication which you probably
didn't mean to make there - that initialisation order is a problem. :)

Where an init ordering is required, it must be documented and the
Makefiles set up accordingly. But I believe that we should also try to
avoid requiring such ordering where possible, too.


> It is no wonder that people have problems getting the initialisation
> order correct.

True. And while that situation continues, I desire to avoid the issue
completely by not having any dependencies on init order.

> I want to completely remove this multi layered method for setting
> initialisation order and go back to basics.  I want the programmer to
> say "initialise E and F after G, H and I".  The kernel build system
> works out the directed graph of initialisation order then controls the
> execution of startup code to satisfy this graph.

But the fewer such constraints there are, the better. We don't want
everyone starting to impose unnecessary link order restrictions instead of
thinking about the code a little more and just eliminating them
completely.

> A typical graph would have scsi disk depends on scsi host adaptor group
> which depends on pci.

No. sd will happily take over any existing devices when as and when they
arrive. It doesn't have to be loaded last. Likewise, in theory at least,
host adaptor drivers using the new PCI driver code would respond correctly
to the PCI code being initialised (and calling their ->probe routine)
later, although that doesn't happen now.

Why have these dependencies where they're not necessary?

> Within the scsi host adaptor group you might need to initialise one
> driver before another, so just declare the few inter-driver
> dependencies.

And there are few. And there should _remain_ few. We shouldn't start
imposing link order restrictions on other code which doesn't really need
it.

> Most of the objects have fairly simple execution dependencies, e.g.
> all file systems depend on core fs code having already executed.

Er... Why? They call register_filesystem() which uses a lock which is
staticly initialised. Don't order stuff for the sake of it. If there are
filesystems which _do_ require the VFS to be initialised first, those
filesystems can be marked as such. I'm not aware of any.

> [1] vfat is one obvious exception, it needs dos first.  Also the first
> few built in file systems must execute in a defined order because
> that in turn controls the probe order for mount.  But this order
> should be explicitly declared, not as a side effect of the line
> order in fs/Makefile.

If the fat_inode_hashtable were staticly initialised, that one wouldn't be
necessary either. But unfortunately that would be quite ugly.

I don't want to get involved in the link order stuff. I would very much
like to avoid having such dependencies. All I want is weak symbols. I
want to call a function directly if it's present, and if not I want to
attempt to load its module. Without having my probe module have a hard
dependency on _all_ the submodules it may decide to ask for.

get_module_symbol() did that for me, and was perfectly acceptable. I
needed to use EXPORT_SYMBOL_NOVERS to export the functions for the
individual command sets. Big deal.

inter_module_get() almost does that for me, but it imposes link order
dependencies, which I want to avoid because I think in this case they're
not necessary. All I want is a way to staticly add entries to the
inter_module_xxx tables at compile time, because I _have_ done the
analysis, and I'm saying that's when they should be made available.

Alternatively, show me how to get weak symbols actually working on all
architectures, and I'll forget the whole thing and be happy.

-- 
dwmw2


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct-unmap in 2.4.0 go?

[David Woodhouse]

On Fri, 12 Jan 2001, Keith Owens wrote:

 People need to realise that the problem is initialisation order,
 nothing more, nothing less.  You have to determine and document the
 startup requirements for your code.

This is true. But I'd also agree with the implication which you probably
didn't mean to make there - that initialisation order is a problem. :)

Where an init ordering is required, it must be documented and the
Makefiles set up accordingly. But I believe that we should also try to
avoid requiring such ordering where possible, too.


 It is no wonder that people have problems getting the initialisation
 order correct.

True. And while that situation continues, I desire to avoid the issue
completely by not having any dependencies on init order.

 I want to completely remove this multi layered method for setting
 initialisation order and go back to basics.  I want the programmer to
 say "initialise E and F after G, H and I".  The kernel build system
 works out the directed graph of initialisation order then controls the
 execution of startup code to satisfy this graph.

But the fewer such constraints there are, the better. We don't want
everyone starting to impose unnecessary link order restrictions instead of
thinking about the code a little more and just eliminating them
completely.

 A typical graph would have scsi disk depends on scsi host adaptor group
 which depends on pci.

No. sd will happily take over any existing devices when as and when they
arrive. It doesn't have to be loaded last. Likewise, in theory at least,
host adaptor drivers using the new PCI driver code would respond correctly
to the PCI code being initialised (and calling their -probe routine)
later, although that doesn't happen now.

Why have these dependencies where they're not necessary?

 Within the scsi host adaptor group you might need to initialise one
 driver before another, so just declare the few inter-driver
 dependencies.

And there are few. And there should _remain_ few. We shouldn't start
imposing link order restrictions on other code which doesn't really need
it.

 Most of the objects have fairly simple execution dependencies, e.g.
 all file systems depend on core fs code having already executed.

Er... Why? They call register_filesystem() which uses a lock which is
staticly initialised. Don't order stuff for the sake of it. If there are
filesystems which _do_ require the VFS to be initialised first, those
filesystems can be marked as such. I'm not aware of any.

 [1] vfat is one obvious exception, it needs dos first.  Also the first
 few built in file systems must execute in a defined order because
 that in turn controls the probe order for mount.  But this order
 should be explicitly declared, not as a side effect of the line
 order in fs/Makefile.

If the fat_inode_hashtable were staticly initialised, that one wouldn't be
necessary either. But unfortunately that would be quite ugly.

I don't want to get involved in the link order stuff. I would very much
like to avoid having such dependencies. All I want is weak symbols. I
want to call a function directly if it's present, and if not I want to
attempt to load its module. Without having my probe module have a hard
dependency on _all_ the submodules it may decide to ask for.

get_module_symbol() did that for me, and was perfectly acceptable. I
needed to use EXPORT_SYMBOL_NOVERS to export the functions for the
individual command sets. Big deal.

inter_module_get() almost does that for me, but it imposes link order
dependencies, which I want to avoid because I think in this case they're
not necessary. All I want is a way to staticly add entries to the
inter_module_xxx tables at compile time, because I _have_ done the
analysis, and I'm saying that's when they should be made available.

Alternatively, show me how to get weak symbols actually working on all
architectures, and I'll forget the whole thing and be happy.

-- 
dwmw2


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct-unmap in 2.4.0 go?

[Keith Owens]

On Fri, 12 Jan 2001 10:27:34 + (GMT), 
David Woodhouse [EMAIL PROTECTED] wrote:
On Fri, 12 Jan 2001, Keith Owens wrote:
 A typical graph would have scsi disk depends on scsi host adaptor group
 which depends on pci.

No. sd will happily take over any existing devices when as and when they
arrive. It doesn't have to be loaded last. Likewise, in theory at least,
host adaptor drivers using the new PCI driver code would respond correctly
to the PCI code being initialised (and calling their -probe routine)
later, although that doesn't happen now.

You just proved my point.  It is extremely difficult to deduce the
required initialisation order by reading an undocumented Makefile where
the init order is implemented as a side effect of selection order.  The
existing method implies link order when none is required.

 Most of the objects have fairly simple execution dependencies, e.g.
 all file systems depend on core fs code having already executed.

Er... Why? They call register_filesystem() which uses a lock which is
staticly initialised. Don't order stuff for the sake of it. If there are
filesystems which _do_ require the VFS to be initialised first, those
filesystems can be marked as such. I'm not aware of any.

I was using scsi and fs as examples, no need to pick holes in the
examples.  But ...

  fs/buffer.c:module_init(bdflush_init)
  fs/pipe.c:module_init(init_pipe_fs)
  fs/fcntl.c:module_init(fasync_init)
  fs/locks.c:module_init(filelock_init)
  fs/dnotify.c:module_init(dnotify_init)

I'm no filesystem expert but it appears that some of those core
initialisation routines must be executed before most filesystems can
start.  In particular, filelock_init() must be executed before any
filesystem that supports locks is initialised.

All I want is a way to staticly add entries to the
inter_module_xxx tables at compile time, because I _have_ done the
analysis, and I'm saying that's when they should be made available.

Firstly inter_module_xxx is only used by that very small subset of code
where there are no constraints on whether the caller and callee can be
in kernel, in modules or a mixture _and_ some of the objects are
optional.  It is a special case because most code handles this problem
through CONFIG options.  If X needs (Y, Z) and X is built into the
kernel then (Y, Z) must also be built into the kernel.  If Y or Z are
optional then control the calls to those functions with CONFIG options.
Almost all of the kernel handles it properly though CONFIG, so
inter_module_xxx is already a process to handle a few special cases.

Secondly you want static inter_module_xxx tables for a small subset of
these special cases, where the source has no other initialisation code.
This is piling special cases on special cases.  Adding static
inter_module_xxx tables requires

* changes to linux/modules.h to define the new table format and
* changes to vmlinux.lds for _every_ architecture to bring all the
  static tables together in vmlinux and
* new initialisation code in module.c to read and load all the static
  tables at boot time and
* extra code in modutils to find any static tables in modules and
* an extension to struct modules to let modutils pass information about
  the static tables to the kernel and
* the kernel code will only work with an upgraded modutils.

That is a lot of work for a very few special cases.  OTOH, you could
just swap the order of 3 lines in drivers/mtd/Makefile.  Guess which
alternative I am going for?

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct-unmap in 2.4.0 go?

[Daniel Phillips]

Keith Owens wrote:
 I want to completely remove this multi layered method for setting
 initialisation order and go back to basics.  I want the programmer to
 say "initialise E and F after G, H and I".  The kernel build system
 works out the directed graph of initialisation order then controls the
 execution of startup code to satisfy this graph.

I don't doubt you will come up with a workable solution at build time. 
However, working out a valid graph at execution time is trivial and
efficient, given a list of precedence relations of the kind you're
suggesting.  In fact you don't even have to work out the graph before
starting the initialization, it's also trivial to keep a count of
unsatisfied initialization conditions at the beginning of each
initialization sequence and block until the count goes to zero.  (In
essence, evaluate a priority sort on the fly.)

--
Daniel
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct-unmap in 2.4.0 go?

[Keith Owens]

On Fri, 12 Jan 2001 13:01:12 +0100, 
Daniel Phillips [EMAIL PROTECTED] wrote:
Keith Owens wrote:
 I want to completely remove this multi layered method for setting
 initialisation order and go back to basics.  I want the programmer to
 say "initialise E and F after G, H and I".  The kernel build system
 works out the directed graph of initialisation order then controls the
 execution of startup code to satisfy this graph.

I don't doubt you will come up with a workable solution at build time. 
However, working out a valid graph at execution time is trivial and
efficient, given a list of precedence relations of the kind you're
suggesting.  In fact you don't even have to work out the graph before
starting the initialization, it's also trivial to keep a count of
unsatisfied initialization conditions at the beginning of each
initialization sequence and block until the count goes to zero.  (In
essence, evaluate a priority sort on the fly.)

It is safer to evaluate the graph at link time in case somebody
mistakenly codes a dependency loop, that is an abort case.  Finding
that you have a loop at load time and aborting the kernel is a little
too drastic for my tastes.

In any case it is academic.  Linus insists on link order being
explicitly and completely specified by the programmer and he likes the
link order being implied by Makefile order.  So there is no point in
working on a better system.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct-unmap in 2.4.0 go?

[David Woodhouse]

[EMAIL PROTECTED] said:
  You just proved my point.  It is extremely difficult to deduce the
 required initialisation order by reading an undocumented Makefile
 where the init order is implemented as a side effect of selection
 order.  The existing method implies link order when none is required. 

I agree entirely. But you're confusing the debate on how to satisfy init 
order dependencies with my desire to avoid them altogether (for certain 
situations).



[EMAIL PROTECTED] said:
 I'm no filesystem expert but it appears that some of those core
 initialisation routines must be executed before most filesystems can
 start.  In particular, filelock_init() must be executed before any
 filesystem that supports locks is initialised.

No, before any filesystem that supports locks is _mounted_. Big difference. 
But I'm picking holes in your examples again. I accept that in some cases 
it is necessary, but I still think it's best to avoid it where possible.


[EMAIL PROTECTED] said:
 Firstly inter_module_xxx is only used by that very small subset of
 code where there are no constraints on whether the caller and callee
 can be in kernel, in modules or a mixture _and_ some of the objects
 are optional.  It is a special case because most code handles this
 problem through CONFIG options. If X needs (Y, Z) and X is built into the
 kernel then (Y, Z) must also be built into the kernel.  If Y or Z are
 optional then control the calls to those functions with CONFIG options.
 Almost all of the kernel handles it properly though CONFIG, so
 inter_module_xxx is already a process to handle a few special cases.

'properly through CONFIG'? I thought you agreed that doing it through
preprocessor options was ugly, and that it was preferable to get rid of such
things and deal with the presence or absence of such code cleanly through
some mechanism with similar semantics to inter_module_get(). The 'special 
case' code to which you refer is in fact the first set of such code to 
attempt to deal with this properly rather than just giving up and hacking 
it with preprocessor options.

Stepping back a moment and considering it, what we actually appear to be 
doing is trying to reinvent weak symbols, as far as I can tell. 


[EMAIL PROTECTED] said:
  Secondly you want static inter_module_xxx tables for a small subset
 of these special cases, where the source has no other initialisation
 code. This is piling special cases on special cases.  Adding static
 inter_module_xxx tables requires 

I want the registration method not to impose init order restrictions where 
previously there were none. Where the previous code already had init order 
ugliness, it's not so much of a problem. But replacing the safe 
get_module_symbol() with the unsafe inter_module_get() just because the AGP 
author forgot to use EXPORT_SYMBOL_NOVERS to export the symbols doesn't 
really strike me as being useful. 

The only real difference between the two, other than the module symbol
mangling, appears to be that inter_module_get() looks stuff up in a dynamic
table which is made at runtime, where get_module_symbol() looks it up in a
static table which is made at compile (or module load) time. That was a
backward step, and wasn't necessary. 

If you object to having both the runtime inter_module_register() and the 
compile-time one, then ditch the dynamic one. Both current users of 
inter_module_xxx would be able to use the static version. 

But to be honest, I'd settle for ditching the whole blinkin' lot and 
getting weak symbols working right, if I could.

--
dwmw2


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct-unmap in 2.4.0 go?

[Christian Zander]

  I ran into this while hacking the Nvidia kernel driver to work with
  2.4.0.  I got the driver working but it's not 100%
  
  Also where did get_module_symbol() and put_module_symbol() go?
 
 Patches for the NVIDIA binary X drivers following all these kernel
 changes can be gotten from IRC server irc.openprojects.net, channel
 #nvidia. Or from http://ex.shafted.com.au/nvidia/
 
 And what a pile of crud those patches are!!  Instead of using the clean
 replacement interface for get_module_symbol, nvidia/patch-2.4.0-PR hard
 codes the old get_module_symbol algorithm as inline code.
 
 This patch violates the modules interface by accessing modules.c
 internal data.  It still suffers from all the problems that
 get_module_symbol had.  Because it is hard coded as inline code instead
 of a common function, will be much harder to fix when it breaks.
 

The way I understand the inter_module mechanism, module A registers one
or several of its symbols using inter_module_register to make it or them 
available to other modules. Module B can then request any of the symbols
with inter_module_request and get a pointer. The inter_module mechanism 
guarantees that the symbol will be available until module B decides that 
the symbol is no longer needed and releases it by calling inter_module_put.

Saying that I should have made use of this mechanism for the specific
code in the Nvidia driver that we are talking about clearly shows that
you didn't look at it. The module used get_module_symbol to search its
own symbol table for parameters that may have been passed to it at load
time. Arguably, this is bad practise, but it is also the reason why using
your mechanism doesn't make any sense. Obviously, the module wouldn't
want to register private data to request it later on; the information
that would have to be passed to inter_module_register is the same that
the code in question intends to retrieve in the first place.

Contrary to what you're saying, the patch does not just inline the old
get_module_symbol algorithm nor does it access any of module.c's internal
data. What is does is to browse the list of the modules's _own_ symbols 
looking for a match. If it finds one, it returns the desired data.

 Whoever coded that patch should be taken out and shot, hung, drawn and
 quartered then forced to write COBOL for the rest of their natural
 life.

Excellent comment - it is just as appropriate as it is helpful.

--
--
 christian zander  we come to bury dos, not to praise it.
 [EMAIL PROTECTED]-- paul vojta
--

 PGP signature

Re: Where did vm_operations_struct-unmap in 2.4.0 go?

[Keith Owens]

On Fri, 12 Jan 2001 20:11:30 +0100, 
Christian Zander [EMAIL PROTECTED] wrote:
Saying that I should have made use of this mechanism for the specific
code in the Nvidia driver that we are talking about clearly shows that
you didn't look at it. The module used get_module_symbol to search its
own symbol table for parameters that may have been passed to it at load
time.

My apologies.  I read the patch, not the full source code and the patch
does not have enough programming context to show that the driver is
only searching its own symbol space.  In my own defense, the references
to spinlock_t unload_lock and MOD_CAN_QUERY(mp) in the patch are highly
misleading, those statements only make sense when you are looking at a
symbol table for another module.  When searching your own symbol table
the current module must be live with a non-zero use count, not being
unloaded and it can always be queried.

Contrary to what you're saying, the patch does not just inline the old
get_module_symbol algorithm nor does it access any of module.c's internal
data.

unload_lock and MOD_CAN_QUERY were copied verbatim from the old
get_module_symbol, even though they are completely unnecessary.  That
looks like inlining the old algorithm to me.

struct module_symbol, mp-nsyms and mp-syms are module.c internal
data.  If it is ever necessary to change those structures, nothing
outside module.c, the 32/64 handlers for module system calls and
modutils should be affected.  Now if I change module_symbol, other bits
of the kernel will unexpectedly break, this is not good.

 Whoever coded that patch should be taken out and shot, hung, drawn and
 quartered then forced to write COBOL for the rest of their natural
 life.

Excellent comment - it is just as appropriate as it is helpful.

Over emphasis for humorous effect.  Must remember to add smiley.


What this patch and David Woodhouse's comments show is that I need to
look at a generic and safe mechanism for kernel/module symbol lookup.
The existing static mechanism works for fixed symbol names but does not
work for symbol names that are generated at run time nor for symbols
that may or may not be present.

get_module_symbol() "worked" but was horribly unsafe.  It broke with
module versions, it did zero type checking which left the code open to
version skew and it assumed that all addresses are equivalent to an
unsigned long.

That last point is especially important for IA64 where function
pointers do not reference the function directly, instead they point to
a function descriptor with two fields, one of which is the function
address.  Casting the unsigned long address of a function into a
function pointer fails miserably on IA64, and gcc does not even give
any warnings.  foo = (int (*)(int))get_module_symbol(NULL, "funcname")
is architecture dependent.

Using EXPORT_SYMBOL_NOVERS() to "fix" the modversions problem for
get_module_symbol() removes all inter module checks on the relevant
symbols.  Not just for the caller of get_module_symbol for all modules
that access those symbols.  This leaves too much code open to version
skew and is not acceptable.

inter_module_xxx is modversions safe.  It still does no type checking
because it uses void * for the data structure, but the exporter and
user have to declare their common data area which reduces the chance of
version skew.  I am still not happy about this possibility of skew but
anything is better than no checks at all.  Passing a data structure
which contains real declarations for function pointers instead of
assuming you can cast a number to a function pointer makes
inter_module_xxx architecture independent.


I will look at a general kernel and module symbol lookup routine that
does the job properly.  The hard part is making sure that the provider
and consumer have exactly the same types for a symbol.  Both
get_module_symbol and inter_module_xxx completely bypass the
modversions checks and are wide open to undetectable version skew,
although inter_module_xxx is a little bit safer.  Any replacement for
these functions must be able to do type checking at run time, which
probably means it is 2.5 code.  And yes, David, it should be able to
handle static data.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct->unmap in 2.4.0 go?

[Keith Owens]

On Fri, 12 Jan 2001 03:12:47 +0100, 
Ingo Oeser <[EMAIL PROTECTED]> wrote:
>So why don't we use sth. like depmod for these issues and get the
>link order automagically (like we get module load order)?

depmod handles dependencies on symbols.  Module Y needs a symbol from
module X so modprobe must load X before Y.  This is a link/load problem
which depmod handles fairly well.

The initialisation order is a dependency on actions, not on symbols.
Code F cannot start until code E has initialised so execute E before F.
This should have *NOTHING* to do with link order, but it is implemented
as a side effect of link ordering which confuses people.

People need to realise that the problem is initialisation order,
nothing more, nothing less.  You have to determine and document the
startup requirements for your code.  Only you know what the startup
pre-requisites for your code are, there is no way for another program
to determine this from the source.  Document your startup requirements,
implement according to the documentation and your problems go away.

Initialisation order is not the problem, the implementation is the
problem.  The method currently used to control initialisation order
sucks.  It is better than the original method (lots of conditional
calls in main.c) but it still sucks.

* Initialisation order is set by the order of structures in section
  .initcall.init.
* The order of the structures in .initcall.init is set by the order
  that objects are linked into vmlinux.
* The link order for vmlinux is derived from a combination of line
  order within a Makefile plus an overriding directory link order from
  the top level Makefile and parent Makefiles.
* Because order is a side effect of adding a line to a Makefile, the
  order requirements are rarely documented.

It is no wonder that people have problems getting the initialisation
order correct.

I want to completely remove this multi layered method for setting
initialisation order and go back to basics.  I want the programmer to
say "initialise E and F after G, H and I".  The kernel build system
works out the directed graph of initialisation order then controls the
execution of startup code to satisfy this graph.

That still means controlling link order to achieve the required result.
But with my approach the complexity would be handled by kbuild based on
explicit rules which are documented in the local Makefile, instead of
the complexity being handled by programmer via implicit rules scattered
over several layers of Makefiles.

A typical graph would have scsi disk depends on scsi host adaptor group
which depends on pci.  Within the scsi host adaptor group you might
need to initialise one driver before another, so just declare the few
inter-driver dependencies.  kbuild would automatically initialise pci
then the scsi host adaptors (in the correct order) then scsi disk.

Most of the objects have fairly simple execution dependencies, e.g.
all file systems depend on core fs code having already executed.  There
are no dependencies between most file systems so most file systems
could initialise in any order[1] which means they could be linked in
any order within the file system group.

I am ready and willing to code this method, it would make kbuild a lot
easier to code, as well as making future maintainence a lot easier.
Linus refuses to accept this approach.  He insists that kernel coders
explicitly specify the link order for everything, via Makefile
order[2].  As long as Linus insists on kernel coders explicitly
controlling the entire link order, we are stuck with the current
method.  I have tried to change his mind without success.

[1] vfat is one obvious exception, it needs dos first.  Also the first
few built in file systems must execute in a defined order because
that in turn controls the probe order for mount.  But this order
should be explicitly declared, not as a side effect of the line
order in fs/Makefile.

[2] http://www.mail-archive.com/linux-kernel@vger.kernel.org/msg10520.html

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct->unmap in 2.4.0 go?

[Ingo Oeser]

On Fri, Jan 12, 2001 at 12:14:44AM +1100, Keith Owens wrote:
> >What happens when we get a loop in init order because of binding and other init
> >order conflicts?
> 
> The kernel does not support circular dependencies between providers and
> consumers.  It does not matter whether they are built into vmlinux or
> loaded as modules, there can be no loops in the directed graph of
> dependencies.  It just does not make sense.

So why don't we use sth. like depmod for these issues and get the
link order automagically (like we get module load order)?

Keith: Perhaps you could explain, why this is impossible.

Regards

Ingo Oeser
-- 
10.+11.03.2001 - 3. Chemnitzer LinuxTag 
    come and join the fun   
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct->unmap in 2.4.0 go?

[David Woodhouse]

[EMAIL PROTECTED] said:
> So you want two services, one static for code that does not do any
> initialisation and one dynamic for code that does do initialisation.
> Can you imagine the fun when somebody adds startup code to a routine
> that was using static registration? 

Oh come on. If you change a module from being 'self-contained' and 
registered at compile time to requiring initialisation it's hardly 
unreasonable to expect you switch the registration too.

Besides, I'm not going to allow any link order dependencies into code I
maintain without them being impossible to avoid - and if anyone's thought
about the problem hard enough to convince me to accept such a change,
they'll have noticed the need to change the registration.

> Oh dear, I added init code so I have to remember to change from static 
> to dynamic registration, and that affects the link order so now I have 
> to tweak the Makefile.

cf. "Oh dear, I added init code but put it _after_ the registration instead
of before, so stuff blows up."

Neither of these two programmers will get their code into anything I 
maintain.

cf. "Oh dear, I need registration, but I have to remember that
inter_module_xxx can't do static registration so now I have to tweak the
Makefile."

[EMAIL PROTECTED] said:
> Stick to one method that works for all routines, dynamic registration.

It doesn't work for all routines. It introduces unnecessary brokenness - 
link order dependencies - where previously there were none.

> If that imposes the occasional need for a couple of extra calls in
> some routines and for people to think about initialisation order right
> from the start then so be it, it is a small price to pay for long term
> stability and ease of maintenance.

I'm thinking about link order. If I _wasn't_ thinking about link order, 
then I'd just put the lines in the 'right' order in the Makefile and put up 
with it. But I'm thinking about it, and I object to it. It is absolutely 
unnecessary in this case.

As far as I'm concerned, fixing the registration problems introduced by the
dynamic inter_module_register is a small price to pay for long term
stability and ease of maintenance :)

--
dwmw2


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct->unmap in 2.4.0 go?

[Keith Owens]

On Thu, 11 Jan 2001 13:09:13 + (GMT), 
Alan Cox <[EMAIL PROTECTED]> wrote:
>> Stick to one method that works for all routines, dynamic registration.
>> If that imposes the occasional need for a couple of extra calls in some
>> routines and for people to think about initialisation order right from
>> the start then so be it, it is a small price to pay for long term
>> stability and ease of maintenance.
>
>What happens when we get a loop in init order because of binding and other init
>order conflicts?

The kernel does not support circular dependencies between providers and
consumers.  It does not matter whether they are built into vmlinux or
loaded as modules, there can be no loops in the directed graph of
dependencies.  It just does not make sense.

A while ago there was accidentally a loop between two ppp related
modules, each needed a routine in the other module.  modprobe would not
load them.  Even if it could have loaded them, it would have been
impossible to unload, both modules would have had a use count on the
other.  The fix was to remove the incorrect loop, it was a programming
error.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct->unmap in 2.4.0 go?

[Alan Cox]

> Stick to one method that works for all routines, dynamic registration.
> If that imposes the occasional need for a couple of extra calls in some
> routines and for people to think about initialisation order right from
> the start then so be it, it is a small price to pay for long term
> stability and ease of maintenance.

What happens when we get a loop in init order because of binding and other init
order conflicts?
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct->unmap in 2.4.0 go?

[Keith Owens]

On Thu, 11 Jan 2001 12:32:10 +, 
David Woodhouse <[EMAIL PROTECTED]> wrote:
>I'm not suggesting that we change it drastically, only that we add 
>the option of static (compile-time) registration for those entries which 
>require it. 

So you want two services, one static for code that does not do any
initialisation and one dynamic for code that does do initialisation.
Can you imagine the fun when somebody adds startup code to a routine
that was using static registration?  Oh dear, I added init code so I
have to remember to change from static to dynamic registration, and
that affects the link order so now I have to tweak the Makefile.
Thanks, but no thanks!

Stick to one method that works for all routines, dynamic registration.
If that imposes the occasional need for a couple of extra calls in some
routines and for people to think about initialisation order right from
the start then so be it, it is a small price to pay for long term
stability and ease of maintenance.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct->unmap in 2.4.0 go?

[David Woodhouse]

[EMAIL PROTECTED] said:
>  Q. With your suggested static method, what happens when Y initialises
>before X, calls inter_module_get, retrieves X's static data and
>starts to use it before X has initialised? 
> A. Oops!

No. You'd explicitly only use the static registration when object X doesn't 
_need_ initialisation, which is the case for my code.

As it is, I've had to add completely now init routines for 
modules which didn't have them before, and all those init routines do is 
inter_module_register() and pray that they're called in time.


[EMAIL PROTECTED] said:
>  The whole point of registration methods is that the owner of the data
> decides when they are ready to provide the service.  Ensuring that
> code is initialised in the correct order, with providers starting
> before consumers, is a fact of life.

I have decided when I'm ready to provide the service. At compile time. 
Ensuring that the code is initialised in the correct order is not a fact of 
life. It's an artifact of the limited functionality of the new setup.


[EMAIL PROTECTED] said:
> It would be much nicer to define ordering sets.  Code in driver foo
> needs the code in driver bar to initialise first.  cfi_probe cannot
> initialise until cfi_cmdset_0001 and cfi_cmdset_0002 have initialised.
> Declare it that way so it is clear what is going on and why, instead
> of being implied by the Makefile order via three layers of
> indirection.

For cases where the code really does need initialisation, that is true. It 
should be done properly rather than just implicitly ordered by the 
Makefile. But in this case, the command set drivers don't need 
initialisation. They just provide a function for use by the generic CFI 
init code. They don't _need_ to initialise themselves beforehand.

The dynamic registration has introduced this ordering dependency when 
previously there was none.


>  The problem is, Linus likes the current method. 

Then he either hasn't considered this particular case, or he's wrong. It 
happens. I'm not suggesting that we change it drastically, only that we add 
the option of static (compile-time) registration for those entries which 
require it. 

Actually, I'd rather do this with weak symbols. Something along the lines 
of...

 extern cfi_cmdset_fn_t cfi_cmdset_0001 __attribute__((weak));
 extern cfi_cmdset_fn_t cfi_cmdset_0002 __attribute__((weak));


 ...

 if (cfi_cmdset_0001 && chip_is_type_1())
return cfi_cmdset_0001(args...);
 if (cfi_cmdset_0002 && chip_is_type_2())
return cfi_cmdset_0002(args...);

 return cfi_cmdset_load_module(args...)

Unfortunately, I couldn't get it to work reliably on anything but x86. 




--
dwmw2


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct->unmap in 2.4.0 go?

[Keith Owens]

On Thu, 11 Jan 2001 11:42:24 +, 
David Woodhouse <[EMAIL PROTECTED]> wrote:
>Taking away get_module_symbol() and providing a replacement which has link 
>order problems wasn't really very sensible.
>
>It's too late to do the sensible thing and deprecate the old version rather 
>than having a 'flag day'. But can we at least fix the link order crap?
>
>struct static_inter_module_entry {
>   const char *im_name;
>   const void *userdata;
>};
>
>#define inter_module_register_static(x,y) \
> static struct static_inter_module_entry __ime_##x \
>   __attribute__((unused,__section__(".intermodule")) \
>   = { #x, y };
>
>.. and the obvious for looking in that table in inter_module_get().

If object X registers data for object Y to use then X _must_ initialise
before Y.  It does not matter whether the registration method is static
or dynamic, the initialisation order must be observed.

Q. With your suggested static method, what happens when Y initialises
   before X, calls inter_module_get, retrieves X's static data and
   starts to use it before X has initialised?
A. Oops!

The whole point of registration methods is that the owner of the data
decides when they are ready to provide the service.  Ensuring that code
is initialised in the correct order, with providers starting before
consumers, is a fact of life.

I dislike the method that the kernel uses to control initialisation
order, but that is an entirely separate problem from inter_module_xxx.
What we really want at startup is a correct initialisation order.  What
we have is the order that objects are selected in a Makefile which maps
to the link order of objects in vmlinux which maps to the listed order
of init routines in section .init.text which maps to initialisation
order.  The mechanism is three layers away from the problem and it is
difficult to understand for many people.

It would be much nicer to define ordering sets.  Code in driver foo
needs the code in driver bar to initialise first.  cfi_probe cannot
initialise until cfi_cmdset_0001 and cfi_cmdset_0002 have initialised.
Declare it that way so it is clear what is going on and why, instead of
being implied by the Makefile order via three layers of indirection.
Then let the kernel build system do whatever it takes to honour the
documented initialisation order.

The problem is, Linus likes the current method.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct->unmap in 2.4.0 go?

[David Woodhouse]

[EMAIL PROTECTED] said:
>  And what a pile of crud those patches are!!  Instead of using the
> clean replacement interface for get_module_symbol, nvidia/
> patch-2.4.0-PR hard codes the old get_module_symbol algorithm as
> inline code.

Taking away get_module_symbol() and providing a replacement which has link 
order problems wasn't really very sensible.

You've changed a lookup in a static table built at compile time to a lookup 
in a dynamic table which has to be built in the right order at runtime.

It's too late to do the sensible thing and deprecate the old version rather 
than having a 'flag day'. But can we at least fix the link order crap?

struct static_inter_module_entry {
const char *im_name;
const void *userdata;
};

#define inter_module_register_static(x,y) \
 static struct static_inter_module_entry __ime_##x \
__attribute__((unused,__section__(".intermodule")) \
= { #x, y };

.. and the obvious for looking in that table in inter_module_get().


--
dwmw2


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct-unmap in 2.4.0 go?

[David Woodhouse]

[EMAIL PROTECTED] said:
  And what a pile of crud those patches are!!  Instead of using the
 clean replacement interface for get_module_symbol, nvidia/
 patch-2.4.0-PR hard codes the old get_module_symbol algorithm as
 inline code.

Taking away get_module_symbol() and providing a replacement which has link 
order problems wasn't really very sensible.

You've changed a lookup in a static table built at compile time to a lookup 
in a dynamic table which has to be built in the right order at runtime.

It's too late to do the sensible thing and deprecate the old version rather 
than having a 'flag day'. But can we at least fix the link order crap?

struct static_inter_module_entry {
const char *im_name;
const void *userdata;
};

#define inter_module_register_static(x,y) \
 static struct static_inter_module_entry __ime_##x \
__attribute__((unused,__section__(".intermodule")) \
= { #x, y };

.. and the obvious for looking in that table in inter_module_get().


--
dwmw2


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct-unmap in 2.4.0 go?

[Keith Owens]

On Thu, 11 Jan 2001 11:42:24 +, 
David Woodhouse [EMAIL PROTECTED] wrote:
Taking away get_module_symbol() and providing a replacement which has link 
order problems wasn't really very sensible.

It's too late to do the sensible thing and deprecate the old version rather 
than having a 'flag day'. But can we at least fix the link order crap?

struct static_inter_module_entry {
   const char *im_name;
   const void *userdata;
};

#define inter_module_register_static(x,y) \
 static struct static_inter_module_entry __ime_##x \
   __attribute__((unused,__section__(".intermodule")) \
   = { #x, y };

.. and the obvious for looking in that table in inter_module_get().

If object X registers data for object Y to use then X _must_ initialise
before Y.  It does not matter whether the registration method is static
or dynamic, the initialisation order must be observed.

Q. With your suggested static method, what happens when Y initialises
   before X, calls inter_module_get, retrieves X's static data and
   starts to use it before X has initialised?
A. Oops!

The whole point of registration methods is that the owner of the data
decides when they are ready to provide the service.  Ensuring that code
is initialised in the correct order, with providers starting before
consumers, is a fact of life.

I dislike the method that the kernel uses to control initialisation
order, but that is an entirely separate problem from inter_module_xxx.
What we really want at startup is a correct initialisation order.  What
we have is the order that objects are selected in a Makefile which maps
to the link order of objects in vmlinux which maps to the listed order
of init routines in section .init.text which maps to initialisation
order.  The mechanism is three layers away from the problem and it is
difficult to understand for many people.

It would be much nicer to define ordering sets.  Code in driver foo
needs the code in driver bar to initialise first.  cfi_probe cannot
initialise until cfi_cmdset_0001 and cfi_cmdset_0002 have initialised.
Declare it that way so it is clear what is going on and why, instead of
being implied by the Makefile order via three layers of indirection.
Then let the kernel build system do whatever it takes to honour the
documented initialisation order.

The problem is, Linus likes the current method.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct-unmap in 2.4.0 go?

[David Woodhouse]

[EMAIL PROTECTED] said:
  Q. With your suggested static method, what happens when Y initialises
before X, calls inter_module_get, retrieves X's static data and
starts to use it before X has initialised? 
 A. Oops!

No. You'd explicitly only use the static registration when object X doesn't 
_need_ initialisation, which is the case for my code.

As it is, I've had to add completely now init routines for 
modules which didn't have them before, and all those init routines do is 
inter_module_register() and pray that they're called in time.


[EMAIL PROTECTED] said:
  The whole point of registration methods is that the owner of the data
 decides when they are ready to provide the service.  Ensuring that
 code is initialised in the correct order, with providers starting
 before consumers, is a fact of life.

I have decided when I'm ready to provide the service. At compile time. 
Ensuring that the code is initialised in the correct order is not a fact of 
life. It's an artifact of the limited functionality of the new setup.


[EMAIL PROTECTED] said:
 It would be much nicer to define ordering sets.  Code in driver foo
 needs the code in driver bar to initialise first.  cfi_probe cannot
 initialise until cfi_cmdset_0001 and cfi_cmdset_0002 have initialised.
 Declare it that way so it is clear what is going on and why, instead
 of being implied by the Makefile order via three layers of
 indirection.

For cases where the code really does need initialisation, that is true. It 
should be done properly rather than just implicitly ordered by the 
Makefile. But in this case, the command set drivers don't need 
initialisation. They just provide a function for use by the generic CFI 
init code. They don't _need_ to initialise themselves beforehand.

The dynamic registration has introduced this ordering dependency when 
previously there was none.


  The problem is, Linus likes the current method. 

Then he either hasn't considered this particular case, or he's wrong. It 
happens. I'm not suggesting that we change it drastically, only that we add 
the option of static (compile-time) registration for those entries which 
require it. 

Actually, I'd rather do this with weak symbols. Something along the lines 
of...

 extern cfi_cmdset_fn_t cfi_cmdset_0001 __attribute__((weak));
 extern cfi_cmdset_fn_t cfi_cmdset_0002 __attribute__((weak));


 ...

 if (cfi_cmdset_0001  chip_is_type_1())
return cfi_cmdset_0001(args...);
 if (cfi_cmdset_0002  chip_is_type_2())
return cfi_cmdset_0002(args...);

 return cfi_cmdset_load_module(args...)

Unfortunately, I couldn't get it to work reliably on anything but x86. 




--
dwmw2


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct-unmap in 2.4.0 go?

[Keith Owens]

On Thu, 11 Jan 2001 12:32:10 +, 
David Woodhouse [EMAIL PROTECTED] wrote:
I'm not suggesting that we change it drastically, only that we add 
the option of static (compile-time) registration for those entries which 
require it. 

So you want two services, one static for code that does not do any
initialisation and one dynamic for code that does do initialisation.
Can you imagine the fun when somebody adds startup code to a routine
that was using static registration?  Oh dear, I added init code so I
have to remember to change from static to dynamic registration, and
that affects the link order so now I have to tweak the Makefile.
Thanks, but no thanks!

Stick to one method that works for all routines, dynamic registration.
If that imposes the occasional need for a couple of extra calls in some
routines and for people to think about initialisation order right from
the start then so be it, it is a small price to pay for long term
stability and ease of maintenance.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct-unmap in 2.4.0 go?

[Alan Cox]

 Stick to one method that works for all routines, dynamic registration.
 If that imposes the occasional need for a couple of extra calls in some
 routines and for people to think about initialisation order right from
 the start then so be it, it is a small price to pay for long term
 stability and ease of maintenance.

What happens when we get a loop in init order because of binding and other init
order conflicts?
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct-unmap in 2.4.0 go?

[Keith Owens]

On Thu, 11 Jan 2001 13:09:13 + (GMT), 
Alan Cox [EMAIL PROTECTED] wrote:
 Stick to one method that works for all routines, dynamic registration.
 If that imposes the occasional need for a couple of extra calls in some
 routines and for people to think about initialisation order right from
 the start then so be it, it is a small price to pay for long term
 stability and ease of maintenance.

What happens when we get a loop in init order because of binding and other init
order conflicts?

The kernel does not support circular dependencies between providers and
consumers.  It does not matter whether they are built into vmlinux or
loaded as modules, there can be no loops in the directed graph of
dependencies.  It just does not make sense.

A while ago there was accidentally a loop between two ppp related
modules, each needed a routine in the other module.  modprobe would not
load them.  Even if it could have loaded them, it would have been
impossible to unload, both modules would have had a use count on the
other.  The fix was to remove the incorrect loop, it was a programming
error.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct-unmap in 2.4.0 go?

[David Woodhouse]

[EMAIL PROTECTED] said:
 So you want two services, one static for code that does not do any
 initialisation and one dynamic for code that does do initialisation.
 Can you imagine the fun when somebody adds startup code to a routine
 that was using static registration? 

Oh come on. If you change a module from being 'self-contained' and 
registered at compile time to requiring initialisation it's hardly 
unreasonable to expect you switch the registration too.

Besides, I'm not going to allow any link order dependencies into code I
maintain without them being impossible to avoid - and if anyone's thought
about the problem hard enough to convince me to accept such a change,
they'll have noticed the need to change the registration.

 Oh dear, I added init code so I have to remember to change from static 
 to dynamic registration, and that affects the link order so now I have 
 to tweak the Makefile.

cf. "Oh dear, I added init code but put it _after_ the registration instead
of before, so stuff blows up."

Neither of these two programmers will get their code into anything I 
maintain.

cf. "Oh dear, I need registration, but I have to remember that
inter_module_xxx can't do static registration so now I have to tweak the
Makefile."

[EMAIL PROTECTED] said:
 Stick to one method that works for all routines, dynamic registration.

It doesn't work for all routines. It introduces unnecessary brokenness - 
link order dependencies - where previously there were none.

 If that imposes the occasional need for a couple of extra calls in
 some routines and for people to think about initialisation order right
 from the start then so be it, it is a small price to pay for long term
 stability and ease of maintenance.

I'm thinking about link order. If I _wasn't_ thinking about link order, 
then I'd just put the lines in the 'right' order in the Makefile and put up 
with it. But I'm thinking about it, and I object to it. It is absolutely 
unnecessary in this case.

As far as I'm concerned, fixing the registration problems introduced by the
dynamic inter_module_register is a small price to pay for long term
stability and ease of maintenance :)

--
dwmw2


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct-unmap in 2.4.0 go?

[Ingo Oeser]

On Fri, Jan 12, 2001 at 12:14:44AM +1100, Keith Owens wrote:
 What happens when we get a loop in init order because of binding and other init
 order conflicts?
 
 The kernel does not support circular dependencies between providers and
 consumers.  It does not matter whether they are built into vmlinux or
 loaded as modules, there can be no loops in the directed graph of
 dependencies.  It just does not make sense.

So why don't we use sth. like depmod for these issues and get the
link order automagically (like we get module load order)?

Keith: Perhaps you could explain, why this is impossible.

Regards

Ingo Oeser
-- 
10.+11.03.2001 - 3. Chemnitzer LinuxTag http://www.tu-chemnitz.de/linux/tag
come and join the fun   
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct-unmap in 2.4.0 go?

[Keith Owens]

On Fri, 12 Jan 2001 03:12:47 +0100, 
Ingo Oeser [EMAIL PROTECTED] wrote:
So why don't we use sth. like depmod for these issues and get the
link order automagically (like we get module load order)?

depmod handles dependencies on symbols.  Module Y needs a symbol from
module X so modprobe must load X before Y.  This is a link/load problem
which depmod handles fairly well.

The initialisation order is a dependency on actions, not on symbols.
Code F cannot start until code E has initialised so execute E before F.
This should have *NOTHING* to do with link order, but it is implemented
as a side effect of link ordering which confuses people.

People need to realise that the problem is initialisation order,
nothing more, nothing less.  You have to determine and document the
startup requirements for your code.  Only you know what the startup
pre-requisites for your code are, there is no way for another program
to determine this from the source.  Document your startup requirements,
implement according to the documentation and your problems go away.

Initialisation order is not the problem, the implementation is the
problem.  The method currently used to control initialisation order
sucks.  It is better than the original method (lots of conditional
calls in main.c) but it still sucks.

* Initialisation order is set by the order of structures in section
  .initcall.init.
* The order of the structures in .initcall.init is set by the order
  that objects are linked into vmlinux.
* The link order for vmlinux is derived from a combination of line
  order within a Makefile plus an overriding directory link order from
  the top level Makefile and parent Makefiles.
* Because order is a side effect of adding a line to a Makefile, the
  order requirements are rarely documented.

It is no wonder that people have problems getting the initialisation
order correct.

I want to completely remove this multi layered method for setting
initialisation order and go back to basics.  I want the programmer to
say "initialise E and F after G, H and I".  The kernel build system
works out the directed graph of initialisation order then controls the
execution of startup code to satisfy this graph.

That still means controlling link order to achieve the required result.
But with my approach the complexity would be handled by kbuild based on
explicit rules which are documented in the local Makefile, instead of
the complexity being handled by programmer via implicit rules scattered
over several layers of Makefiles.

A typical graph would have scsi disk depends on scsi host adaptor group
which depends on pci.  Within the scsi host adaptor group you might
need to initialise one driver before another, so just declare the few
inter-driver dependencies.  kbuild would automatically initialise pci
then the scsi host adaptors (in the correct order) then scsi disk.

Most of the objects have fairly simple execution dependencies, e.g.
all file systems depend on core fs code having already executed.  There
are no dependencies between most file systems so most file systems
could initialise in any order[1] which means they could be linked in
any order within the file system group.

I am ready and willing to code this method, it would make kbuild a lot
easier to code, as well as making future maintainence a lot easier.
Linus refuses to accept this approach.  He insists that kernel coders
explicitly specify the link order for everything, via Makefile
order[2].  As long as Linus insists on kernel coders explicitly
controlling the entire link order, we are stuck with the current
method.  I have tried to change his mind without success.

[1] vfat is one obvious exception, it needs dos first.  Also the first
few built in file systems must execute in a defined order because
that in turn controls the probe order for mount.  But this order
should be explicitly declared, not as a side effect of the line
order in fs/Makefile.

[2] http://www.mail-archive.com/linux-kernel@vger.kernel.org/msg10520.html

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct->unmap in 2.4.0 go?

[Keith Owens]

On Thu, 11 Jan 2001 16:38:50 +1100, 
Antony Suter <[EMAIL PROTECTED]> wrote:
>Allen Unueco wrote:
>> I ran into this while hacking the Nvidia kernel driver to work with
>> 2.4.0.  I got the driver working but it's not 100%
>> 
>> Also where did get_module_symbol() and put_module_symbol() go?
>
>Patches for the NVIDIA binary X drivers following all these kernel
>changes can be gotten from IRC server irc.openprojects.net, channel
>#nvidia. Or from http://ex.shafted.com.au/nvidia/

And what a pile of crud those patches are!!  Instead of using the clean
replacement interface for get_module_symbol, nvidia/patch-2.4.0-PR hard
codes the old get_module_symbol algorithm as inline code.

This patch violates the modules interface by accessing modules.c
internal data.  It still suffers from all the problems that
get_module_symbol had.  Because it is hard coded as inline code instead
of a common function, will be much harder to fix when it breaks.

Whoever coded that patch should be taken out and shot, hung, drawn and
quartered then forced to write COBOL for the rest of their natural
life.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct->unmap in 2.4.0 go?

[Antony Suter]

Allen Unueco wrote:
> 
> Sometime around test10 or test11 unmap left vm_operations_struct.  The
> comment implies its there but it's gone.  Where did it go?
> 
> How do I get a call back for a page unmap?
> 
> I ran into this while hacking the Nvidia kernel driver to work with
> 2.4.0.  I got the driver working but it's not 100%
> 
> Also where did get_module_symbol() and put_module_symbol() go?
> 
> -Allen

Patches for the NVIDIA binary X drivers following all these kernel
changes can be gotten from IRC server irc.openprojects.net, channel
#nvidia. Or from http://ex.shafted.com.au/nvidia/

--
- Antony Suter  ([EMAIL PROTECTED])  "ExWired"  openpgp:71ADFC87
- "...to condense fact from the vapor of nuance."
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct-unmap in 2.4.0 go?

[Antony Suter]

Allen Unueco wrote:
 
 Sometime around test10 or test11 unmap left vm_operations_struct.  The
 comment implies its there but it's gone.  Where did it go?
 
 How do I get a call back for a page unmap?
 
 I ran into this while hacking the Nvidia kernel driver to work with
 2.4.0.  I got the driver working but it's not 100%
 
 Also where did get_module_symbol() and put_module_symbol() go?
 
 -Allen

Patches for the NVIDIA binary X drivers following all these kernel
changes can be gotten from IRC server irc.openprojects.net, channel
#nvidia. Or from http://ex.shafted.com.au/nvidia/

--
- Antony Suter  ([EMAIL PROTECTED])  "ExWired"  openpgp:71ADFC87
- "...to condense fact from the vapor of nuance."
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct-unmap in 2.4.0 go?

[Keith Owens]

On Thu, 11 Jan 2001 16:38:50 +1100, 
Antony Suter [EMAIL PROTECTED] wrote:
Allen Unueco wrote:
 I ran into this while hacking the Nvidia kernel driver to work with
 2.4.0.  I got the driver working but it's not 100%
 
 Also where did get_module_symbol() and put_module_symbol() go?

Patches for the NVIDIA binary X drivers following all these kernel
changes can be gotten from IRC server irc.openprojects.net, channel
#nvidia. Or from http://ex.shafted.com.au/nvidia/

And what a pile of crud those patches are!!  Instead of using the clean
replacement interface for get_module_symbol, nvidia/patch-2.4.0-PR hard
codes the old get_module_symbol algorithm as inline code.

This patch violates the modules interface by accessing modules.c
internal data.  It still suffers from all the problems that
get_module_symbol had.  Because it is hard coded as inline code instead
of a common function, will be much harder to fix when it breaks.

Whoever coded that patch should be taken out and shot, hung, drawn and
quartered then forced to write COBOL for the rest of their natural
life.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct->unmap in 2.4.0 go?

[Keith Owens]

On Tue, 09 Jan 2001 19:27:24 -0800, 
Allen Unueco <[EMAIL PROTECTED]> wrote:
>Also where did get_module_symbol() and put_module_symbol() go?

http://www.mail-archive.com/linux-kernel@vger.kernel.org/msg08791.html
http://www.mail-archive.com/linux-kernel@vger.kernel.org/msg11497.html

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Where did vm_operations_struct->unmap in 2.4.0 go?

[Allen Unueco]

Sometime around test10 or test11 unmap left vm_operations_struct.  The
comment implies its there but it's gone.  Where did it go? 

How do I get a call back for a page unmap?

I ran into this while hacking the Nvidia kernel driver to work with
2.4.0.  I got the driver working but it's not 100%

Also where did get_module_symbol() and put_module_symbol() go?

-Allen
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Where did vm_operations_struct-unmap in 2.4.0 go?

[Allen Unueco]

Sometime around test10 or test11 unmap left vm_operations_struct.  The
comment implies its there but it's gone.  Where did it go? 

How do I get a call back for a page unmap?

I ran into this while hacking the Nvidia kernel driver to work with
2.4.0.  I got the driver working but it's not 100%

Also where did get_module_symbol() and put_module_symbol() go?

-Allen
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: Where did vm_operations_struct-unmap in 2.4.0 go?

[Keith Owens]

On Tue, 09 Jan 2001 19:27:24 -0800, 
Allen Unueco [EMAIL PROTECTED] wrote:
Also where did get_module_symbol() and put_module_symbol() go?

http://www.mail-archive.com/linux-kernel@vger.kernel.org/msg08791.html
http://www.mail-archive.com/linux-kernel@vger.kernel.org/msg11497.html

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/