Re: Keepin constants, configuration values, etc. in Python - dedicated module or what?
On Tuesday, September 30, 2014 10:09:52 PM UTC+5:30, Rustom Mody wrote: On Tuesday, September 30, 2014 8:48:15 PM UTC+5:30, c...@isbd.net wrote: Rustom Mody wrote: On Tuesday, September 30, 2014 5:18:31 PM UTC+5:30, Chris wrote: I would actually quite like to keep the configuration data separate from the code as it would simplify using the data at the 'home' end of things as I'd just need to copy the configuration file across. This was why the database approach appealed at first as all I need to do is copy the database and everything is in there. Of course Are there any better ways of doing this? E.g. some sort of standard configuration file format that Python knows about? Umm this is getting to be a FAQ... Maybe it should go up somewhere? Yes there are dozens: - ini - csv - json - yml - xml - pickle - And any DBMS of your choice I guess Ive forgotten as many as Ive listed!! Yes, I know, I've found most of those. I'm really asking for help in choosing which to use. I think I can reject some quite quickly:- xml - horrible, nasty to edit, etc. I don't like XML! :-) Heh! Youve proved yourself a pythonista! ini - doesn't work so well with lists/dictionaries (though possible) csv - rather difficult to edit Have you tried with comma=tab? yml - front runner if I go for configuration files Yeah my favorite as well json - one of the most likely possibilities, but prefer yml Seems to be most popular nowadays -- maybe related to being almost yaml and in the standard lib pickle - not user editable as I understand it Well not in any reasonably pleasant way! What I'm really asking for is how to choose between:- python - just keep config in the modules/classes, not easy to use at 'both ends' (home and remote), otherwise quite simple Can work at a trivial level. As soon as things get a bit larger data and code mixed up is a recipe for mess up. Just came across this https://github.com/henriquebastos/python-decouple/ -- https://mail.python.org/mailman/listinfo/python-list
Re: Keepin constants, configuration values, etc. in Python - dedicated module or what?
On 09/30/2014 10:55 AM, c...@isbd.net wrote: Dave Angel da...@davea.name wrote: name. And presumably you never remove an old name from the config. The only things really likely to change (and may change regularly) are the conversion factors, drifting voltage references etc. will inevitably require these to be recalibrated every so often. Other than that it's just typos or me deciding to change the name of something. Right there you have an important design decision. You then presumably have to have a way that each particular data point can record what configuration it's associated with. If you just change the calibration data externally, then all the old records are invalidated. More troublesome is adding a new kind of data. You have to decide whether it's worth generalizing the code to anticipate the change, or just admit that the code will grow at that point and that old code won't deal with the new data. There's a separate module for each input type (just ADC and 1-wire at the moment), I don't anticipate any more though I suppose there might be digital inputs one day. So a configuration [file] for each type seems to make sense, generalising it would be more trouble than it's worth I think. -- DaveA -- https://mail.python.org/mailman/listinfo/python-list
Re: Keepin constants, configuration values, etc. in Python - dedicated module or what?
Chris Angelico wrote: I'd agree, where trivial limits is defined by each individual item. Going with straight Python code is fine for huge projects with long config files, as long as each config entry is itself simple. You even get a form of #include: from otherfile import *. I would argue the opposite. If I have a large Python project, with big config files, then the added complexity of moving the config data into separate files using a non-executable format (say, JSON or INI) is minimal, and keeping the data and code separate will pay off. But for small Python projects, with only a few data values, keeping them separate is overkill. So, to the Original Poster, I think that depending on the size of your program and the amount of config data you have to deal with, there's nothing wrong with including it directly in the module. -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: Keepin constants, configuration values, etc. in Python - dedicated module or what?
On Thu, Oct 2, 2014 at 12:49 AM, Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: Chris Angelico wrote: I'd agree, where trivial limits is defined by each individual item. Going with straight Python code is fine for huge projects with long config files, as long as each config entry is itself simple. You even get a form of #include: from otherfile import *. I would argue the opposite. Welcome to design debates, where there are as many valid and justifiable views as there are participants :) Though I wasn't precluding small config files from being code - just saying that I don't think size of project is a factor at all. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Keepin constants, configuration values, etc. in Python - dedicated module or what?
I am developing some code which runs on a (remote from me most of the
time) Beaglebone Black single board computer. It reads various items
of data (voltages, currents, temperatures, etc.) using both a 1-wire
bus system and the Beaglebone's ADC inputs. The values are stored
at hourly intervals into a database which is regularly rsync'ed
across to my home system where I use the values for monitoring etc.
Associated with each value are various parameters, i.e. for each ADC
input of the Beaglebone there is the ADC input number, a short name
for the value, a conversion factor and a description. Similarly for
the 1-wire inputs there is the 1-wire filesystem filename, a short
name for the data and a longer description.
I am puzzling where and how to keep these configuration values. My
current design has them in dedicated tables in the database but this
is rather clumsy in many ways as there's an overhead reading them
every time the program needs them and changing them isn't particularly
convenient at the Beaglebone doesn't have a GUI so it has to be done
using SQL from the command line.
Does it make sense to add them to the modules which handle the reading
of the inputs? I already have modules defining classes called Adc and
OneWire, is it a reasonable approach to add the configuration to these
as, probably, dictionaries? Modifying it would be pretty simple -
just edit the python source (one of the easiest things for me to do on
the Beaglebone as my sole access is via ssh/command line).
Thus I'd have something like (apologies for any syntax errors):-
cfg = { LeisureVolts: [AIN0, 0.061256, Leisure Battery Voltage],
StarterVolts: [AIN1, 0.060943, Starter Battery Voltage],
LeisureAmps1: [AIN2, 0.423122, Leisure Battery Current}
(It might be better to makes those lists dictionaries, but it shows
the idea)
Are there any better ways of doing this? E.g. some sort of standard
configuration file format that Python knows about? I would actually
quite like to keep the configuration data separate from the code as it
would simplify using the data at the 'home' end of things as I'd just
need to copy the configuration file across. This was why the database
approach appealed at first as all I need to do is copy the database
and everything is in there.
I'm not really expecting quick/glib answers, just some ideas and
comments on the various ways of doing this and their advantages and
disadvantages.
--
Chris Green
·
--
https://mail.python.org/mailman/listinfo/python-list
Re: Keepin constants, configuration values, etc. in Python - dedicated module or what?
On Tuesday, September 30, 2014 5:18:31 PM UTC+5:30, Chris wrote: I would actually quite like to keep the configuration data separate from the code as it would simplify using the data at the 'home' end of things as I'd just need to copy the configuration file across. This was why the database approach appealed at first as all I need to do is copy the database and everything is in there. Of course Are there any better ways of doing this? E.g. some sort of standard configuration file format that Python knows about? Umm this is getting to be a FAQ... Maybe it should go up somewhere? Yes there are dozens: - ini - csv - json - yml - xml - pickle - And any DBMS of your choice I guess Ive forgotten as many as Ive listed!! -- https://mail.python.org/mailman/listinfo/python-list
Re: Keepin constants, configuration values, etc. in Python - dedicated module or what?
Rustom Mody rustompm...@gmail.com wrote: On Tuesday, September 30, 2014 5:18:31 PM UTC+5:30, Chris wrote: I would actually quite like to keep the configuration data separate from the code as it would simplify using the data at the 'home' end of things as I'd just need to copy the configuration file across. This was why the database approach appealed at first as all I need to do is copy the database and everything is in there. Of course Are there any better ways of doing this? E.g. some sort of standard configuration file format that Python knows about? Umm this is getting to be a FAQ... Maybe it should go up somewhere? Yes there are dozens: - ini - csv - json - yml - xml - pickle - And any DBMS of your choice I guess Ive forgotten as many as Ive listed!! Yes, I know, I've found most of those. I'm really asking for help in choosing which to use. I think I can reject some quite quickly:- ini - doesn't work so well with lists/dictionaries (though possible) csv - rather difficult to edit json - one of the most likely possibilities, but prefer yml yml - front runner if I go for configuration files xml - horrible, nasty to edit, etc. I don't like XML! :-) pickle - not user editable as I understand it dbms - current solution in sqlite3 as described What I'm really asking for is how to choose between:- DBMS - present solution, keeps config with data easily but more code and less easy to change yml - easy to edit config, keeps data separate from code but needs YAML installed and separate files to manage python - just keep config in the modules/classes, not easy to use at 'both ends' (home and remote), otherwise quite simple My requirements are:- Easy to change, i.e.human readable format which can be edited simply as I have to do this via a terminal/ssh. Easy to use at local end as well as remote end, the remote end is where it lives and is the 'driver' as it were but I need to know the configuration at the local end as well. -- Chris Green · -- https://mail.python.org/mailman/listinfo/python-list
Re: Keepin constants, configuration values, etc. in Python - dedicated module or what?
might this be of interest (though old)? https://wiki.python.org/moin/ConfigParserShootout Cheers Jon N -- https://mail.python.org/mailman/listinfo/python-list
Re: Keepin constants, configuration values, etc. in Python - dedicated module or what?
On 9/30/2014 7:35 AM, c...@isbd.net wrote:
Thus I'd have something like (apologies for any syntax errors):-
cfg = { LeisureVolts: [AIN0, 0.061256, Leisure Battery Voltage],
StarterVolts: [AIN1, 0.060943, Starter Battery Voltage],
LeisureAmps1: [AIN2, 0.423122, Leisure Battery Current}
(It might be better to makes those lists dictionaries, but it shows
the idea)
Using configparser.ConfigParser to read an ini-format seems like a good
idea. I use it for my own numerous fixed format data file definitions,
and it's been convenient and even extensible.
[LeisureVolts]
Description=Leisure Battery Voltage
Code=AIN0
Value=0.061256
[StarterVolts]
Description=Starter Battery Voltage
Code=AIN1
Value=0.060943
[LeisureAmps1]
Description=Leisure Battery Current
Code=AIN2
Value=0.423122
I've set it up so I can understand abbreviations for the field names,
for when I want to be lazy.
Some of my values are dictionaries, which looks like this in my files
(an alternate spelling of one of the above entries):
LeisureVolts=desc:Leisure Battery Voltage
code:AIN2
value:0.061256
It's simple to hook into ConfigParser to get whatever meaning you'd
like, and whatever verification you'd find necessary.
--
Neil Cerutti
--
https://mail.python.org/mailman/listinfo/python-list
Re: Keepin constants, configuration values, etc. in Python - dedicated module or what?
On Tuesday, September 30, 2014 8:48:15 PM UTC+5:30, c...@isbd.net wrote: Rustom Mody wrote: On Tuesday, September 30, 2014 5:18:31 PM UTC+5:30, Chris wrote: I would actually quite like to keep the configuration data separate from the code as it would simplify using the data at the 'home' end of things as I'd just need to copy the configuration file across. This was why the database approach appealed at first as all I need to do is copy the database and everything is in there. Of course Are there any better ways of doing this? E.g. some sort of standard configuration file format that Python knows about? Umm this is getting to be a FAQ... Maybe it should go up somewhere? Yes there are dozens: - ini - csv - json - yml - xml - pickle - And any DBMS of your choice I guess Ive forgotten as many as Ive listed!! Yes, I know, I've found most of those. I'm really asking for help in choosing which to use. I think I can reject some quite quickly:- xml - horrible, nasty to edit, etc. I don't like XML! :-) Heh! Youve proved yourself a pythonista! ini - doesn't work so well with lists/dictionaries (though possible) csv - rather difficult to edit Have you tried with comma=tab? yml - front runner if I go for configuration files Yeah my favorite as well json - one of the most likely possibilities, but prefer yml Seems to be most popular nowadays -- maybe related to being almost yaml and in the standard lib pickle - not user editable as I understand it Well not in any reasonably pleasant way! What I'm really asking for is how to choose between:- snipped python - just keep config in the modules/classes, not easy to use at 'both ends' (home and remote), otherwise quite simple Can work at a trivial level. As soon as things get a bit larger data and code mixed up is a recipe for mess up. -- https://mail.python.org/mailman/listinfo/python-list
Re: Keepin constants, configuration values, etc. in Python - dedicated module or what?
On Wed, Oct 1, 2014 at 2:39 AM, Rustom Mody rustompm...@gmail.com wrote: python - just keep config in the modules/classes, not easy to use at 'both ends' (home and remote), otherwise quite simple Can work at a trivial level. As soon as things get a bit larger data and code mixed up is a recipe for mess up. True, but it's certainly possible to break out the config data into an importable module that conceptually just provides constants. Technically it's code, yes, but it'll normally be code that looks like your standard name = value config file: # docs for first option # more docs # examples # etcetera first_option =123 # docs for second option second_option = 234 Is that Python code, or is it a sectionless INI file, or what? There's no difference. And you get expressions for free - simple stuff like 7*24*60*60 to represent the number of seconds in a week (for people who aren't intimately familiar with 604800), or calculations relative to previous data, or whatever. Sometimes it's helpful to have just a little code in your data. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Keepin constants, configuration values, etc. in Python - dedicated module or what?
On Tuesday, September 30, 2014 10:22:12 PM UTC+5:30, Chris Angelico wrote: On Wed, Oct 1, 2014 at 2:39 AM, Rustom Mody wrote: python - just keep config in the modules/classes, not easy to use at 'both ends' (home and remote), otherwise quite simple Can work at a trivial level. As soon as things get a bit larger data and code mixed up is a recipe for mess up. True, but it's certainly possible to break out the config data into an importable module that conceptually just provides constants. Technically it's code, yes, but it'll normally be code that looks like your standard name = value config file: # docs for first option # more docs # examples # etcetera first_option =123 # docs for second option second_option = 234 Is that Python code, or is it a sectionless INI file, or what? Yeah I was going to say that this is possible There's no difference. But there is! Its code that looks like data. And you get expressions for free - simple stuff like 7*24*60*60 to represent the number of seconds in a week (for people who aren't intimately familiar with 604800), or calculations relative to previous data, or whatever. Sometimes it's helpful to have just a little code in your data. Not free at all. Power of code means cost of code See http://www.w3.org/2001/tag/doc/leastPower.html I'd reiterate though what I first said: In this case its probably ok if the code (=data) does not cross trivial limits -- https://mail.python.org/mailman/listinfo/python-list
Re: Keepin constants, configuration values, etc. in Python - dedicated module or what?
On Wed, Oct 1, 2014 at 3:01 AM, Rustom Mody rustompm...@gmail.com wrote: And you get expressions for free - simple stuff like 7*24*60*60 to represent the number of seconds in a week (for people who aren't intimately familiar with 604800), or calculations relative to previous data, or whatever. Sometimes it's helpful to have just a little code in your data. Not free at all. Power of code means cost of code See http://www.w3.org/2001/tag/doc/leastPower.html It's free once you've already decided (for other reasons) to make your config file use Python syntax, at which point you've already paid the cost of code. But you're quite right, allowing code does have costs. Most notably, you have to quote your strings; although the direct benefits you get (line continuation, unambiguous handling of leading/trailing spaces, etc) may outweigh that on their own. I'd reiterate though what I first said: In this case its probably ok if the code (=data) does not cross trivial limits I'd agree, where trivial limits is defined by each individual item. Going with straight Python code is fine for huge projects with long config files, as long as each config entry is itself simple. You even get a form of #include: from otherfile import *. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Keepin constants, configuration values, etc. in Python - dedicated module or what?
On Tuesday, September 30, 2014 10:46:21 PM UTC+5:30, Chris Angelico wrote: On Wed, Oct 1, 2014 at 3:01 AM, Rustom Mody wrote: And you get expressions for free - simple stuff like 7*24*60*60 to represent the number of seconds in a week (for people who aren't intimately familiar with 604800), or calculations relative to previous data, or whatever. Sometimes it's helpful to have just a little code in your data. Not free at all. Power of code means cost of code See http://www.w3.org/2001/tag/doc/leastPower.html It's free once you've already decided (for other reasons) to make your config file use Python syntax, at which point you've already paid the cost of code. But you're quite right, allowing code does have costs. Most notably, you have to quote your strings; although the direct benefits you get (line continuation, unambiguous handling of leading/trailing spaces, etc) may outweigh that on their own. I'd reiterate though what I first said: In this case its probably ok if the code (=data) does not cross trivial limits I'd agree, where trivial limits is defined by each individual item. Going with straight Python code is fine for huge projects with long config files, as long as each config entry is itself simple. You even get a form of #include: from otherfile import *. Well in programming we sometimes use strict/formal methods and sometimes more informal. In this case using configparser or pyyaml or whatever means that the config file's syntax is rigorously fixed by that library On the other hand if you impose a convention: Constants file-module has NOTHING but constants, thats a non-formal convention and thats ok. If however you mix it up with other (real) code, you'll get a bloody mess. This kind of stuff 7*24*60*60 is borderline and in my experience its a slippery slope that ends up being more trouble than its worth. Experience being emacs where because in lisp code and data are the same, all kinds of design messes are perpetrated -- https://mail.python.org/mailman/listinfo/python-list
Re: Keepin constants, configuration values, etc. in Python - dedicated module or what?
Rustom Mody rustompm...@gmail.com wrote: # docs for first option # more docs # examples # etcetera first_option =123 # docs for second option second_option = 234 Is that Python code, or is it a sectionless INI file, or what? Yeah I was going to say that this is possible There's no difference. But there is! Its code that looks like data. The main trouble with this approach is that I need some way to have the python/config file available at the 'home' end of this as well as at the 'remote' end. I guess I could write a copy of the file into the database but then I have the editing issue again, changing it becomes messy. If it's not in the database then how do I 'tie' it to the data? -- Chris Green · -- https://mail.python.org/mailman/listinfo/python-list
Re: Keepin constants, configuration values, etc. in Python - dedicated module or what?
On Wed, Oct 1, 2014 at 3:31 AM, Rustom Mody rustompm...@gmail.com wrote: On the other hand if you impose a convention: Constants file-module has NOTHING but constants, thats a non-formal convention and thats ok. If however you mix it up with other (real) code, you'll get a bloody mess. This kind of stuff 7*24*60*60 is borderline and in my experience its a slippery slope that ends up being more trouble than its worth. Agreed, and which side of the border it's on is a matter of opinion. Suppose you were writing an INI-style parser - would you include this feature? I probably wouldn't include arithmetic, as it's usually more effort than it's worth, but it isn't a bad feature inherently. I'd definitely not include backreferencing (FOO = bar, followed by FOO_LEN = len(FOO) or something), or the ability to chain assignments (FOO = BAR = 0), so those would be advised against in the convention docs. On the other hand, a quoted string requirement (foo\nbar) makes a huge amount of sense in certain contexts, and I can imagine implementing quite a lot of a programming language's string literal syntax. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Keepin constants, configuration values, etc. in Python - dedicated module or what?
On Wed, Oct 1, 2014 at 3:30 AM, c...@isbd.net wrote: The main trouble with this approach is that I need some way to have the python/config file available at the 'home' end of this as well as at the 'remote' end. I guess I could write a copy of the file into the database but then I have the editing issue again, changing it becomes messy. If it's not in the database then how do I 'tie' it to the data? That's a design question. Maybe it's better for you to do your config in the database. I usually find that these config files include database credentials (server, port, user name, password), so they have to be (a) outside the database, (b) outside source control, and (c) separately configurable for test and production systems, even if they run on the exact same hardware. So for those situations, it makes more sense to have them in a script or INI file, rather than the database. Your situation may well be completely different. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Keepin constants, configuration values, etc. in Python - dedicated module or what?
Neil D. Cerutti ne...@norwich.edu wrote:
On 9/30/2014 7:35 AM, c...@isbd.net wrote:
Thus I'd have something like (apologies for any syntax errors):-
cfg = { LeisureVolts: [AIN0, 0.061256, Leisure Battery Voltage],
StarterVolts: [AIN1, 0.060943, Starter Battery Voltage],
LeisureAmps1: [AIN2, 0.423122, Leisure Battery Current}
(It might be better to makes those lists dictionaries, but it shows
the idea)
Using configparser.ConfigParser to read an ini-format seems like a good
idea. I use it for my own numerous fixed format data file definitions,
and it's been convenient and even extensible.
[LeisureVolts]
Description=Leisure Battery Voltage
Code=AIN0
Value=0.061256
[StarterVolts]
Description=Starter Battery Voltage
Code=AIN1
Value=0.060943
[LeisureAmps1]
Description=Leisure Battery Current
Code=AIN2
Value=0.423122
That's OK except that it doesn't associate the different sections in
any way, I need another level which includes these to indicate that
these are values read from the ADC. There's another collection of
config data for values read from the 1-wire bus.
I guess I could have two config files, one for ADC and one for 1-wire.
In fact I think that's the best approach so far. Ini format files are
human readable (and more to the point editable), I can store them in
the same directory as the database so they will get carried around
with the data and the configparser module is built-in.
--
Chris Green
·
--
https://mail.python.org/mailman/listinfo/python-list
