Re: [Tutor] I Give Up. - Follow up post
> Comes from wanting to minimize how much stuff I have loaded into > memory, and my impression is, a dictionary of 2 elements (name:value > pairs) > takes up more space in memory than two single variables (name points > to value in memory), Dangerous assumption since Python ultimately uses dictionaries for *all* its variables, its just that some of them are somewhat hidden from view... So evern if you declare foo = 42 it gets turned into local_vars[foo'] = 42 or something similar So by declaring your own dictionary the only overhead is the extra dictionary entry to point at your explicit doictionary. Dictionaries in Python are pretty efficient since they are the bedrock of the interpreter. HTH, Alan G. ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] I Give Up. - Follow up post
> > The other alternative is to provide a tool for editing the file in which > case the format is less important sionce hiumans "never" need to > touch it. > Heh. good analysis.. now I feel a little dumb- I should have thought of this *LONG* ago. yeah I knew about the xml parser , and how to do xml markup (it's one of the methods we use on intranet communications channels for certain scriptable tasks) ... My sticking point was, I wanted it to be super easy for a newbie or beginner or "sysadmin trainee" to install and set up the monitoring without needing to know much more than maybe basic command line.. That's when it finally hit me (thanks to your commentary below) , why use a config file at all? all I have to do is create a simple "setup" utility that will write the "configuration" data I need in *correct and perfect* python syntax, which I can simply reload, or re-import into the daemon ... no need to even have a text editor, all I need is a little shell script or python utility to ask for and wait for the user's input at each prompt, and check data at each step to get it right.. Yep, I think I was *absolutely* over-thinking the issue here. :) > How about command line tools? > Somewhat like the commonly found adduser command in Unix? > Yeah, basically you carry values in a dictionary named by keyname , but.. there have been situations where I need the key name as the variable name , I.E. config_options[name] = value could become name = value as if it was explicitly defined that way > > > While I understand the usefulness of this in an interactive environment > I'm puzzled about how this would work in a pre-written script. If you are > creating variables from a config file how do you know what those variables > are called? If you don;lt know how can you reference them later in the > code? And if you don;t reference them of what value is the variable > name? Therefore the logical conclusion(to me!) is that you must > know what names you expect to read and therefore can use a dictionary? > > I'm not sure I understand where exactly you are having the problems > (other than the admnittedly poor configparser documentation! - the > quality of python docs tends to be directly proportional to its frequency > of use and inversely proportional to age - more recent modules tend to > be better documented than ancient ones. Unfortunately for you config > parser doesn't seem to be used that much and has been there for ever!) Yeah this kind of confused the whole issue- and snowballed a little bit.. got off track .. Point originally was, I wanted to be able to read in a config file as a name => value pair , and then actually have a variable named by name in the script, which you cannot do using a dictionary, you could use the key, yes, but at the time it didnt make sense to me to have 4 or 5 dictionaries laying around in memory when they only have one or two keys , and I wanted to use single variables.. Comes from wanting to minimize how much stuff I have loaded into memory, and my impression is, a dictionary of 2 elements (name:value pairs) takes up more space in memory than two single variables (name points to value in memory), and the conversation just ultimately got mixed up, and I couldnt even remember, in the end why I needed the variable names Vs. a dict, until I went back to SubVersion and dug out the first 2 versions of my code :) (Got to love Revision control systems! ) Anyhow, that's now a moot point. :) I have no valid reason at this point to insist on knowing variable names. However to make a point, my original thinking was along the lines of a way to do variable variables, (there are certain instances where they can be handy) , however since python has no $ to start a variable, I saw no way it was possible... Oh well I have yet to get back in this project, it's working as needed in Perl (with a somewhat kludgy configuration, and really, IMHO, more code than is really necesary for something so simple as this) but it does the job, and I have other things I must do :) > > HTH, > > Alan Gauld > Author of the Learn to Program web site > http://www.freenetpages.co.uk/hp/alan.gauld > > > !DSPAM:44ab7d4d293088057919449! > > ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
[Tutor] I Give Up. - Follow up post
Dear Brian, The best parser is python itself :) let's make ports.py with your original content: http = 80 https = 443 http1 = 81 smtp = 25 smtp2 = 587 In this case, you can import ports.py with simple >>> import ports >>> ports.http 80 >>> You don't need to define a new file format, just use the python syntax and it will work. The only problem is that the format has to follow the python systax. So the next wont work ## Ports.py http = 80 https = 443 http1 = 81 smtp = 25 smtp2 = 587 >>> import ports Traceback (most recent call last): File "", line 1, in ? File "ports.py", line 5 smtp = 25 ^ SyntaxError: invalid syntax >>> The same method can be used for making more complex ini files. I like it very much. It is because Python is a dynamic language. Yours sincerely, __ Janos Juhasz ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] I Give Up. - Follow up post
>>
>> ### machine_config.py
>> machines = { 'apache': ('http', 80),
>> ## fill me in with the config of other machines
>> }
>>
>>
>> could be used as your configuration file format. The value portion
>> of a
> Umm well it would work nicely . *if* the configuration never
> changed- on the other hand, if it is to do as intended, the user of
> the application needs to be able to add and edit the configuration.
Ok, would an XML format work better? There are several good XML
parsers available.
etc...
or something similar?
> it is easier for a non programmer to edit a configuration that makes
> some sort of logical sense and is easy to edit, without screwing up
> (need to make it more "forgiving" of things like line endings,
> spacing between name , = and value elements, etc)
The other alternative is to provide a tool for editing the file in
which
case the format is less important sionce hiumans "never" need to
touch it.
>>> believe me I have tried dictionaries, Ive tried parsing the file
>>> by other means, but the only way I could get results I needed was
>>> through configparser
If you are using ini file format config parser is the right choice.
But there are other file formats and making the data match the
problem is usually the biggest breakthrough in creating clean code.
So although ini format might be simplest to maintain by non technical
users if it forces you to ju7mp through hoops mayber a slihghtly more
complex format would be better.
(An example is the XF86 setup file used in Linux which uses a
syntax somewhat like a Python dictionary mixed with an ini file)
> Another option I had was to do a web based interface for setting up
> additional machines, etc, but web based is non-ideal for this
> application.
How about command line tools?
Somewhat like the commonly found adduser command in Unix?
>>> Yeah, basically you carry values in a dictionary named by keyname
>>> , but.. there have been situations where I need the key name as
>>> the variable name , I.E. config_options[name] = value could become
>>>
>>> name = value as if it was explicitly defined that way
While I understand the usefulness of this in an interactive
environment
I'm puzzled about how this would work in a pre-written script. If you
are
creating variables from a config file how do you know what those
variables
are called? If you don;lt know how can you reference them later in the
code? And if you don;t reference them of what value is the variable
name? Therefore the logical conclusion(to me!) is that you must
know what names you expect to read and therefore can use a dictionary?
Typically the examples I've seen are where people use a known
name as the root and append a sequence number:
foo1 = 43
foo2 = 45
etc
then they use a loop later in the code to reconstitute the variable
names.
But if a list is used inside a dictionary the same effect is achieved
with
about the same or less work:
vars['port'].append(42)
vars['port'].append(45)
for port in vars['port']:
use port here.
The other scenario that often arises is that you know the variable
names
you expect to find and only use those names, thus if a user causes
an unexpected name you just ignore it.
In this case you can create a dictionary of the known variable names
for validation purposes and link them to a set of helper functions
(one
per variable) for storing the values in your variables:
Something like this:
machine = [] # list variable
port = None # scalar value
def addMachine(value): machine.append(value)
def addPort(value):
global port
port = int(value)
vars = {'machine':addMachine,'port':addPort}
name = parseConfig()
if name in vars.keys():
vars[name](parseConfig())
And if you don't like the helper functions you could use lambdas:
vars = {'machine': lambda val: machine.append(val),
'port':lambda val: port = int(val)
}
One you finish parsing the data you can now use machine and
port as normal variables, since that's what they are.
(Note: by converting the string value to an int I remove the issue
of extraneous spaces. Not a universal cure but a step forward)
>> Can you show us the situation you're talking about that requires
>> this?
I'm not sure the stuff above actually addresses what you need,
but it might give some ideas? Like Danny I'd like to see an example
where the dictionary approach wouldn't work.
> Sure, I could also do this by reading a config file and parsing it ,
> but ..
>
> Say you split at the = , and the config file is like
>
> http = 80 #standard http port
> https=443#https port
>
> Are you going to be 100% certain that when you call
> watch_hosts[http], that you are gonna GET string "80"?
> what about the space between http and = ?
Yes, user errors like added spaces, mixed case, dropped or wrong
syntax(semi-colons/colons etc) are perenni
Re: [Tutor] I Give Up. - Follow up post
OK .. so far so good.. :)
> ultimately want is not meshing well together.
>
> Let's clarify the requirement: you want to have a mapping from services
> to their configurations. Ignoring the format of the configuration file
> for the moment, it sounds like you ultimately want to parse the
> configruation and get:
>
> ### machine_config.py
> machines = { 'apache': ('http', 80),
> ## fill me in with the config of other machines
> }
>
>
> could be used as your configuration file format. The value portion of a
> dictionary can be an arbitrary value. Here, we map a machine's name to
> a tuple containing the protocol and port. No string hackery is involved
> here.
Umm well it would work nicely . *if* the configuration never changed- on
the other hand, if it is to do as intended, the user of the application
needs to be able to add and edit the configuration.
Unless they are well versed with python and/or programming and syntax,
(in which case they probably arent going to use this script anyway)
it is easier for a non programmer to edit a configuration that makes
some sort of logical sense and is easy to edit, without screwing up
(need to make it more "forgiving" of things like line endings, spacing
between name , = and value elements, etc)
So the ideal format is a configuration file where they can set up
(within documented settings parameters) additional machines or protocols
or ports to suit their own network or layout..
Easiest = a simple file where I set name = value and import it.
HOWEVER, if it is to be imported, it must follow proper python syntax,
correct? so if it doesnt, they break it
an INI file on the other hand is a bit more forgiving, thus the choice
to use it.
>
>
>
>> believe me I have tried dictionaries, Ive tried parsing the file by
>> other means, but the only way I could get results I needed was through
>> configparser
>
>
> I think you're making this problem too hard for yourself.
>
perhaps, when I get back to the project I plan to do a code review of
code to date..
I have no problem with refactoring if I can find a better, momre
effective way, but it *must* also be user friendly (ease of use and
configuration for a non-programmer)
Another option I had was to do a web based interface for setting up
additional machines, etc, but web based is non-ideal for this application.
>
>
>> Yeah, basically you carry values in a dictionary named by keyname ,
>> but.. there have been situations where I need the key name as the
>> variable name , I.E. config_options[name] = value could become
>>
>> name = value as if it was explicitly defined that way
>
>
>
> Can you show us the situation you're talking about that requires this?
if cfg.has_section("Hosts"):
hostnames = cfg.items("Hosts")
for hostname in hostnames:
watch_hosts[hostname[0]] = hostname[1]
would result in like
watch_hosts = {"http" : "80","https" : "443"}
Sure, I could also do this by reading a config file and parsing it , but ..
Say you split at the = , and the config file is like
http = 80 #standard http port
https=443#https port
Are you going to be 100% certain that when you call watch_hosts[http],
that you are gonna GET string "80"?
what about the space between http and = ?
if you wanted to check if watch_hosts[keyname] == "http" , it would
never work, right?(because if you split at =, you have string "http " <-
see the space char?)
if I, however use configparser, I will always get "http" and "80" where
I expect them to be , I dont have to worry (or send values through a
cleanup or trim() function) if it has a comment for the line or not, I
dont need to worry if there is whitespace. right?
This is intended for an end user to be able to edit a config file with a
minimum of trouble, and be able to leave comments so they remember what
each is for.. :)
(and if they do mess up somehow, I can catch that kind of thing in the
code)
I just didnt see any more effective way to get this than to use
ConfigParser as opposed to attempting to parse a file using regular
expressions , trim, split, etc
>
>
>
>> It's difficult to grasp or even effectively explain the concept or idea,
>
>
> Try to do so. I think this is a real point that needs to be cleared up.
I understand the concept/idea behind it, I think above is about as good
an explanation as it gets - I tried to express how Im
thinking/approaching this issue. Keeping in mind this is going to be
edited by someone who may or may not know how to write a single line of
program code.. :)
>
> Take a look at:
>
> http://www.python.org/doc/faq/general.html#why-isn-t-there-a-switch-or-case-statement-in-python
>
>
Thanks. reading up on it now.
>
> If you need more examples, ask, and someone here on the list will be
> happy to help.
>
> !DSPAM:44aae7ac185203757830825!
>
>
_
Re: [Tutor] I Give Up. - Follow up post
> Now, if I want to iterate over a list of machines , and check each
> machine for whatever it was set for (in the config file)
> So the config file requires http to be the first part of the string ,
> followed by something to make it unique (I.E. http1, http2, http3)
Hi Brian,
Have you considered using something other than the INI-style file format?
ConfigParser works best when there is a fixed, known set of keys: it
doesn't work so well when you want to use an arbitrary collection where
you do not know the keys in advance.
Bluntly put: it does sound like the INI file format and what you
ultimately want is not meshing well together.
Let's clarify the requirement: you want to have a mapping from services to
their configurations. Ignoring the format of the configuration file for
the moment, it sounds like you ultimately want to parse the configruation
and get:
{ 'apache' : ('http', 80),
... }
where a 'machine' is the key into the corresponding (protocol, port)
value.
If you are just prototyping this, a 'module' file like:
### machine_config.py
machines = { 'apache': ('http', 80),
## fill me in with the config of other machines
}
could be used as your configuration file format. The value portion of a
dictionary can be an arbitrary value. Here, we map a machine's name to a
tuple containing the protocol and port. No string hackery is involved
here.
> believe me I have tried dictionaries, Ive tried parsing the file by
> other means, but the only way I could get results I needed was through
> configparser
I think you're making this problem too hard for yourself.
> Yeah, basically you carry values in a dictionary named by keyname ,
> but.. there have been situations where I need the key name as the
> variable name , I.E. config_options[name] = value could become
>
> name = value as if it was explicitly defined that way
Can you show us the situation you're talking about that requires this?
> It's difficult to grasp or even effectively explain the concept or idea,
Try to do so. I think this is a real point that needs to be cleared up.
> Agreed. however I just set this as a preliminary - in no way is this
> code finished. :) I actually intended to build a function to do this.
> (In perl, I use Switch/Case statements, does Python have anything
> similar? - with switch/case, you can set a final default "catch-all" at
> which point I log a "unknown event" message )
Take a look at:
http://www.python.org/doc/faq/general.html#why-isn-t-there-a-switch-or-case-statement-in-python
If you need more examples, ask, and someone here on the list will be happy
to help.
___
Tutor maillist - Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] I Give Up. - Follow up post
Danny Yoo wrote:
>> I tried it by opening a file , but could find no way to do variable
>> variables
>
>
>
> Hi Brian,
>
> Look for the concept of dictionaries in Python. "Variable variables" in
> languages like PHP and Perl are doable in Python, but a dictionary
> usually handles such situations in a safer way.
Yeah I am familiar with the concept , and it would work nicely for some
applications, but I couldnt make it work the way I need it to for this
application :)
Reason: It's more to do with the further functions in the application
that I did not paste in the follow up :)
The issue, again is that I want to call a variable by *name* , but I do
not *KNOW* what that variable's name is, exactly
For example, say I have apache running on port 80, apache-ssl on 443,
apache2 on port 81, and Caudium on port 8080 (yes I do have a machine
running not only those servers, but AOLServer, simultaneously)
If I want to configure a protocol to check, *for multiple machines* I
might do
(pseudo function)
Check_Host(machine,protocol,port,last_results)
Now, if I want to iterate over a list of machines , and check each
machine for whatever it was set for (in the config file)
I cannot obviously have a config that says
(Protocol = Port)
http = 80
http = 81
http = 8080
smtp = 25
So the config file requires http to be the first part of the string ,
followed by something to make it unique (I.E. http1, http2, http3)
Now, if I pass each value in turn to check_host, how do I tell what type
of check to make ? (Obviously if I want to get a string from a web page,
I need to SEND a GET request, but if I am checking smtp, I cannot use
that same function, SO )
I check for the base of the configuration (if http in protocol: then run
http request, else something else)
so that I can run the correct request type on the correct port , and
maintain a log, but I cannot do that as effectively if it is a
dictionary or list or even a foreach ..
As I said, I have looked at this from every angle I could see, but using
dictionary (which was my *first* choice) did not work the way I needed
it to, nor did simply reading a file to get name value pairs I needed
the name to be an actual variable name..
believe me I have tried dictionaries, Ive tried parsing the file by
other means, but the only way I could get results I needed was through
configparser
>
>> and then read it in as a file , which builds an array (AKA dictionary)
>> and then just assign thus:
>>
>> foreach ($filearray as $key => $value)
>> {
>> $$key = $value
>> }
>
> This is not safe in Python, because the $key given could easily be the
> name of something that shouldn't be treated as configuration, such as
> the built-in functions. It also makes debugging a bit harder if we have
> variables in a program that aren't explicitely named.
Yes, I am well aware of the dangers of taking values unchecked into an
array such as this method shows (simplified).. I rarely ever use it, or
if I do, it is on *known* clean data (example, pulling config data out
of a database , reading a .ini file, etc.. not untrusted user input)
I have seen people use this method to extract POST and GET data for
example (untrusted user input) , which I find horrifying :)
>
> The safe way to do this is to sandbox these variables in a dictionary.
> Conceptually, the pseudocode looks like:
>
>
> config_options = {}
> for (name, value) in the file:
> config_options[name] = value
>
>
> and the real code to do this doesn't look too much different.
>
Yeah, basically you carry values in a dictionary named by keyname ,
but.. there have been situations where I need the key name as the
variable name , I.E. config_options[name] = value could become
name = value as if it was explicitly defined that way
It's difficult to grasp or even effectively explain the concept or idea,
but sometimes you need to KNOW the key name, *without* knowing the key
name :) (confused yet? hahaha)
>
>
> Let's look at some of the config-reading code:
>
>> if cfg.has_section("Parameters"):
>> myparams = cfg.items("Parameters")
>> for item in myparams:
>> parameter[item[0]] = item[1]
>> else:
>> log_error("Parameters","not found")
>> if cfg.has_section("Ports"):
>> ports = cfg.items("Ports")
>> for port in ports:
>> watch_port[port[0]] = port[1]
>> else:
>> log_error("Ports","Not Found")
>> if cfg.has_section("Hosts"):
>> hostnames = cfg.items("Hosts")
>> for hostname in hostnames:
>> watch_hosts[hostname[0]] = hostname[1]
>> else:
>> log_error("Hosts","Not Found")
>> if cfg.has_section("IP Addresses"):
>> ips = cfg.items("IP Addresses")
>> for ip in ips:
>> watch_ips[ip[0]] = ip[1]
>> else:
>> log_error("IP Addresses","Not Found")
>>
Re: [Tutor] I Give Up. - Follow up post
> I tried it by opening a file , but could find no way to do variable
> variables
Hi Brian,
Look for the concept of dictionaries in Python. "Variable variables" in
languages like PHP and Perl are doable in Python, but a dictionary usually
handles such situations in a safer way.
> and then read it in as a file , which builds an array (AKA dictionary)
> and then just assign thus:
>
> foreach ($filearray as $key => $value)
> {
> $$key = $value
> }
This is not safe in Python, because the $key given could easily be the
name of something that shouldn't be treated as configuration, such as the
built-in functions. It also makes debugging a bit harder if we have
variables in a program that aren't explicitely named.
The safe way to do this is to sandbox these variables in a dictionary.
Conceptually, the pseudocode looks like:
config_options = {}
for (name, value) in the file:
config_options[name] = value
and the real code to do this doesn't look too much different.
Let's look at some of the config-reading code:
> if cfg.has_section("Parameters"):
> myparams = cfg.items("Parameters")
> for item in myparams:
> parameter[item[0]] = item[1]
> else:
> log_error("Parameters","not found")
> if cfg.has_section("Ports"):
> ports = cfg.items("Ports")
> for port in ports:
> watch_port[port[0]] = port[1]
> else:
> log_error("Ports","Not Found")
> if cfg.has_section("Hosts"):
> hostnames = cfg.items("Hosts")
> for hostname in hostnames:
> watch_hosts[hostname[0]] = hostname[1]
> else:
> log_error("Hosts","Not Found")
> if cfg.has_section("IP Addresses"):
> ips = cfg.items("IP Addresses")
> for ip in ips:
> watch_ips[ip[0]] = ip[1]
> else:
> log_error("IP Addresses","Not Found")
> if cfg.has_section("Alerts"):
> alerts_to = cfg.items("Alerts")
> else:
> log_error("Hosts","Not Found")
There's a lot of repetition here. When we have the temptation to copy and
paste, try to see if a helper function can do some lifting.
In fact, there's a bug there because of the copy-and-paste. If there are
no Alerts, the system will spuriously blame Hosts. I resent being blamed
for things I don't do, and I'm sure my programs feel the same. *grin*
Here's a possible refactoring:
#
def read_config_section(section_name, output_dictionary):
if cfg.has_section(section_name)
section = cfg.items(section_name)
else:
log_error(section_name, "Not Found")
for key_value in section:
output_dictionary[key_value[0]] = key_value[1]
#
Once we have this, we can then do:
read_config_section("Parameters", parameter)
read_config_section("Ports", watch_port)
...
and eliminate a lot of that repetitive code.
Does this make sense? Feel free to ask more questions.
___
Tutor maillist - Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] I Give Up. - Follow up post
OK see, what I was doing originally (and I found time to finally get it
partially working)
I have a configuration file that may be edited by webmaster thus
[Ports]
http = 80
https = 443
http1 = 81
smtp = 25
smtp2 = 587
(the above is a small example, it could be *anything* )
Now I have a function that takes machine, protocol, and a dictionary of
last alerts (would be empty if none)
The whole is a small multiple machine monitoring script, By the way...
The configuration file needed to be read by config parser and assign the
config name as if it were a variable, and the value along with it
I tried it by opening a file , but could find no way to do variable
variables
(In PHP, you can do something to the effect of:
name = value
name1 = value1
and then read it in as a file , which builds an array (AKA dictionary)
and then just assign thus:
foreach ($filearray as $key => $value)
{
$$key = $value
}
in python I could see no way to do so, so I ended up with configparser,
which I *finally* got working - working code below
I am still working on this , as I wanna do it in python (currently
operational in Perl , and running as a daemon, which I want this code to
do eventually)
So, the below is a working example of reading a config file in python
using configparser
Also note: I found that ConfigParser != configparser (it is case sensitive)
Just wanted to post this back in case anyone else needs to figure out
how to use configparser in python.
import ConfigParser
cfg = ConfigParser.SafeConfigParser()
cfg.read("/home/brian/pymon.cfg")
if cfg.has_section("Parameters"):
myparams = cfg.items("Parameters")
for item in myparams:
parameter[item[0]] = item[1]
else:
log_error("Parameters","not found")
if cfg.has_section("Ports"):
ports = cfg.items("Ports")
for port in ports:
watch_port[port[0]] = port[1]
else:
log_error("Ports","Not Found")
if cfg.has_section("Hosts"):
hostnames = cfg.items("Hosts")
for hostname in hostnames:
watch_hosts[hostname[0]] = hostname[1]
else:
log_error("Hosts","Not Found")
if cfg.has_section("IP Addresses"):
ips = cfg.items("IP Addresses")
for ip in ips:
watch_ips[ip[0]] = ip[1]
else:
log_error("IP Addresses","Not Found")
if cfg.has_section("Alerts"):
alerts_to = cfg.items("Alerts")
else:
log_error("Hosts","Not Found")
print parameter
print watch_port
print watch_hosts
print watch_ips
print alerts_to
___
Tutor maillist - Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor
