Re: [Proposal for trunk] Extending RewriteMap, Reusing ip network matches
On 18.12.2009 20:24, William A. Rowe Jr. wrote:
Rainer Jung wrote:
On 17.12.2009 23:14, William A. Rowe Jr. wrote:
Rainer Jung wrote:
1) Extending RewriteMap
===
I plan to extend key file handling in text file RewriteMap. At the
moment keys are always matched as exact strings against the map. I want
to add the ability to alternatively
a) match via regexp (and replace backreferences in the found values)
b) match via ip network notation like e.g. used in Require ip ...
This could be expressed in the configuration by adding another token
after type:source, e.g. exact, regexp, ip, where exact is the
default.
-1; you are going to completely hide the performance penalty from the
user's attention. In order to identify this properly, please extend with
an alternate directive to activate this (slower) mapping, e.g.
RewriteList
or some such.
Really? The feature is off by default, users need to activate by adding
one of the new match types. Old configs will not see a performance penalty.
The penalty when using the new match type can easily be documented the
same way I would need to documnt RewriteList. I'm not sure whether
adding a new directive is the way to go, because the purpose of the list
is the same as of the map. Only the type of key matching changes. I'm
afraid it will lead to some confusion.
Reviewers are more likely to catch a misconfig if you use RewriteList rather
than some extra args. This is why I deprecatedContainer ~ foo so long
ago, and replaced it withContainerMatch foo.
Do you have an comment on the other part of the proposal, namely adding
ip subnet match to RewriteCond and SetEnvIf by using a magic prefix
character, like e.g.
RewriteCond %{REMOTE_ADDR} @195.227.30.128/25
SetEnvIf Remote_Addr @195.227.30.128/25 isTrusted
It seems you wouldn't like that as well in analogy to the ~ operator?
Any other syntax idea how to express an ip subnet match instead of a regexp?
RewriteList's will take only a subset of the storage types, right?
Yes, I was only planning to add it to txt (and rnd) types. int and
prg don't make sense for this. Concerning the dbm storage type ...
You aren't planning on using/reading large, nearly empty hash sets, are you?
... we would need to iterate over the entries during initial load to
precompile the regexps and replace the dbm storage with an in-memory
regexp array. Even if the dbm were not to large, that would pose the
problems that the order of the dbm entries retrieved during iteration is
not well-defined. So I'd say no, there's no good way of combining this
with dbm.
Similarly for dbd we would need an additional ordered retrieval sql
statement as part of the config and we would loose the ability to use
data updated in the database without restart, because there are no file
timestamps we can check to detect such data updates. So again I'd say
there's not good way of implementing this for a dbd based RewriteMap.
Regards,
Rainer
Re: [Proposal for trunk] Extending RewriteMap, Reusing ip network matches
On 17.12.2009 23:14, William A. Rowe Jr. wrote: Rainer Jung wrote: 1) Extending RewriteMap === I plan to extend key file handling in text file RewriteMap. At the moment keys are always matched as exact strings against the map. I want to add the ability to alternatively a) match via regexp (and replace backreferences in the found values) b) match via ip network notation like e.g. used in Require ip ... This could be expressed in the configuration by adding another token after type:source, e.g. exact, regexp, ip, where exact is the default. -1; you are going to completely hide the performance penalty from the user's attention. In order to identify this properly, please extend with an alternate directive to activate this (slower) mapping, e.g. RewriteList or some such. Really? The feature is off by default, users need to activate by adding one of the new match types. Old configs will not see a performance penalty. The penalty when using the new match type can easily be documented the same way I would need to documnt RewriteList. I'm not sure whether adding a new directive is the way to go, because the purpose of the list is the same as of the map. Only the type of key matching changes. I'm afraid it will lead to some confusion. Regards, Rainer
Re: [Proposal for trunk] Extending RewriteMap, Reusing ip network matches
Rainer Jung wrote: On 17.12.2009 23:14, William A. Rowe Jr. wrote: Rainer Jung wrote: 1) Extending RewriteMap === I plan to extend key file handling in text file RewriteMap. At the moment keys are always matched as exact strings against the map. I want to add the ability to alternatively a) match via regexp (and replace backreferences in the found values) b) match via ip network notation like e.g. used in Require ip ... This could be expressed in the configuration by adding another token after type:source, e.g. exact, regexp, ip, where exact is the default. -1; you are going to completely hide the performance penalty from the user's attention. In order to identify this properly, please extend with an alternate directive to activate this (slower) mapping, e.g. RewriteList or some such. Really? The feature is off by default, users need to activate by adding one of the new match types. Old configs will not see a performance penalty. The penalty when using the new match type can easily be documented the same way I would need to documnt RewriteList. I'm not sure whether adding a new directive is the way to go, because the purpose of the list is the same as of the map. Only the type of key matching changes. I'm afraid it will lead to some confusion. Reviewers are more likely to catch a misconfig if you use RewriteList rather than some extra args. This is why I deprecated Container ~ foo so long ago, and replaced it with ContainerMatch foo. RewriteList's will take only a subset of the storage types, right? You aren't planning on using/reading large, nearly empty hash sets, are you?
[Proposal for trunk] Extending RewriteMap, Reusing ip network matches
1) Extending RewriteMap
===
I plan to extend key file handling in text file RewriteMap. At the
moment keys are always matched as exact strings against the map. I want
to add the ability to alternatively
a) match via regexp (and replace backreferences in the found values)
b) match via ip network notation like e.g. used in Require ip ...
This could be expressed in the configuration by adding another token
after type:source, e.g. exact, regexp, ip, where exact is the
default.
Of course a) can be achieved without RewriteMap with a list of
RewriteRules, but if you need a long list, the map is easier to maintain
and understand, and it's also reloaded atomatically.
For b) there is no other solution at the moment.
2) Further reuse of ip network matches
==
a) RewriteCond
Furthermore I would like to extend RewriteCond with another CondPattern
that matches ip addresses in the way noted under b) above, i.e. using ip
address and network notation. It is very clumsy and not robust to try to
do that via regexps. For that we would need to decide on another prefix
character to use in the CondPattern, like e.g.
RewriteCond %{REMOTE_ADDR} @195.227.30.128/25
The preifx character @ needs to be chosen carefully to prevent most
existing RewriteCond lines from being broken (in case they already start
with this prefix). Alternatively it could be done using a new flag.
b) SetEnvIf
The same extension could be added to SetEnvIf:
SetEnvIf Remote_Addr @195.227.30.128/25 isTrusted
Any comments? I'm willing to work on the implementation, but I'd like to
hear, whether there are arguments against it.
Thanks!
Rainer
RE: [Proposal for trunk] Extending RewriteMap, Reusing ip network matches
-Original Message- From: Rainer Jung Sent: Donnerstag, 17. Dezember 2009 13:38 To: dev@httpd.apache.org Subject: [Proposal for trunk] Extending RewriteMap, Reusing ip network matches 1) Extending RewriteMap === I plan to extend key file handling in text file RewriteMap. At the moment keys are always matched as exact strings against the map. I want to add the ability to alternatively a) match via regexp (and replace backreferences in the found values) Can you give an example for this? Where are these regexps? In the map file or in the expression of the Map in the RewriteRule? Regards Rüdiger
Re: [Proposal for trunk] Extending RewriteMap, Reusing ip network matches
On 17.12.2009 14:32, Plüm, Rüdiger, VF-Group wrote: -Original Message- From: Rainer Jung Sent: Donnerstag, 17. Dezember 2009 13:38 To: dev@httpd.apache.org Subject: [Proposal for trunk] Extending RewriteMap, Reusing ip network matches 1) Extending RewriteMap === I plan to extend key file handling in text file RewriteMap. At the moment keys are always matched as exact strings against the map. I want to add the ability to alternatively a) match via regexp (and replace backreferences in the found values) Can you give an example for this? Where are these regexps? In the map file or in the expression of the Map in the RewriteRule? The regular expressions are the first columns in the map file. The actual keys searched for will be normal strings matched against those regexps. The result returned is the second column with backreferences resolved against the match from the first column and the key. Regards, Rainer
RE: [Proposal for trunk] Extending RewriteMap, Reusing ip network matches
-Original Message- From: Rainer Jung Sent: Donnerstag, 17. Dezember 2009 15:56 To: dev@httpd.apache.org Subject: Re: [Proposal for trunk] Extending RewriteMap, Reusing ip network matches On 17.12.2009 14:32, Plüm, Rüdiger, VF-Group wrote: -Original Message- From: Rainer Jung Sent: Donnerstag, 17. Dezember 2009 13:38 To: dev@httpd.apache.org Subject: [Proposal for trunk] Extending RewriteMap, Reusing ip network matches 1) Extending RewriteMap === I plan to extend key file handling in text file RewriteMap. At the moment keys are always matched as exact strings against the map. I want to add the ability to alternatively a) match via regexp (and replace backreferences in the found values) Can you give an example for this? Where are these regexps? In the map file or in the expression of the Map in the RewriteRule? The regular expressions are the first columns in the map file. The actual keys searched for will be normal strings matched against those regexps. The result returned is the second column with backreferences resolved against the match from the first column and the key. I see some performance issue with this approach since each time one thing in the file changes *all* regexps need to be recompiled. If it is a large file that changes frequently this can eat up a considerable amount of cycles. Regards Rüdiger
Re: [Proposal for trunk] Extending RewriteMap, Reusing ip network matches
On 17.12.2009 16:02, Plüm, Rüdiger, VF-Group wrote: -Original Message- From: Rainer Jung Sent: Donnerstag, 17. Dezember 2009 15:56 To: dev@httpd.apache.org Subject: Re: [Proposal for trunk] Extending RewriteMap, Reusing ip network matches On 17.12.2009 14:32, Plüm, Rüdiger, VF-Group wrote: -Original Message- From: Rainer Jung Sent: Donnerstag, 17. Dezember 2009 13:38 To: dev@httpd.apache.org Subject: [Proposal for trunk] Extending RewriteMap, Reusing ip network matches 1) Extending RewriteMap === I plan to extend key file handling in text file RewriteMap. At the moment keys are always matched as exact strings against the map. I want to add the ability to alternatively a) match via regexp (and replace backreferences in the found values) Can you give an example for this? Where are these regexps? In the map file or in the expression of the Map in the RewriteRule? The regular expressions are the first columns in the map file. The actual keys searched for will be normal strings matched against those regexps. The result returned is the second column with backreferences resolved against the match from the first column and the key. I see some performance issue with this approach since each time one thing in the file changes *all* regexps need to be recompiled. If it is a large file that changes frequently this can eat up a considerable amount of cycles. Right, I noticed that to, when I just now started to implement to learn more about possible problems. OTOH having the rules in one externally managed file can still be nice even without using the reloading ability. And if you do reload it, recompiling the regexps will likely be cheaper than a full server restart because of changed RewriteRules. At the moment as long as a key hasn't been looked up before (i.e. is not cached), the file is opened for each lookup and read line by line. When using regexps it would be necessary to keep the compiled regexps in memory after the first file read (to avoid continuous recompilation). So the first access to the file is expensive and we would also need some more memory. The impl wouldn't change the behaviour of the existing exact match case though. Regards, Rainer
Re: [Proposal for trunk] Extending RewriteMap, Reusing ip network matches
Rainer Jung wrote: 1) Extending RewriteMap === I plan to extend key file handling in text file RewriteMap. At the moment keys are always matched as exact strings against the map. I want to add the ability to alternatively a) match via regexp (and replace backreferences in the found values) b) match via ip network notation like e.g. used in Require ip ... This could be expressed in the configuration by adding another token after type:source, e.g. exact, regexp, ip, where exact is the default. -1; you are going to completely hide the performance penalty from the user's attention. In order to identify this properly, please extend with an alternate directive to activate this (slower) mapping, e.g. RewriteList or some such.
