Re: Line continuation and comments
Robert Latest wrote: > Paul Bryan wrote: >> Adding to this, there should be no reason now in recent versions of >> Python to ever use line continuation. Black goes so far as to state >> "backslashes are bad and should never be used": >> >> https://black.readthedocs.io/en/stable/the_black_code_style/ > future_style.html#using-backslashes-for-with-statements > > Then I wonder how Mr. Black would go about these long "dot chaining" > expressions that packages like pandas and sqlalchemy require. Just found out that parentheses work there, too. -- https://mail.python.org/mailman/listinfo/python-list
Re: Line continuation and comments
Paul Bryan wrote: > Adding to this, there should be no reason now in recent versions of > Python to ever use line continuation. Black goes so far as to state > "backslashes are bad and should never be used": > > https://black.readthedocs.io/en/stable/the_black_code_style/ future_style.html#using-backslashes-for-with-statements Then I wonder how Mr. Black would go about these long "dot chaining" expressions that packages like pandas and sqlalchemy require. -- https://mail.python.org/mailman/listinfo/python-list
Re: Line continuation and comments
Edmondo Giovannozzi wrote: > Il giorno mercoledì 22 febbraio 2023 alle 09:50:14 UTC+1 Robert Latest ha > scritto: >> I found myself building a complicated logical condition with many ands and >> ors which I made more manageable by putting the various terms on individual >> lines and breaking them with the "\" line continuation character. In this >> context it would have been nice to be able to add comments to lines terms >> which of course isn't possible because the backslash must be the last >> character on the line. >> >> Question: If the Python syntax were changed to allow comments after >> line-ending >> backslashes, would it break any existing code? I can't think of an example. > > Well you can if you use parenthesis like in: > x = 5 > a = (x > 3 and > # x < 21 or > x > 100 > ) > You don't need the "\" to continue a line in this case I like that. Never thought of it. -- https://mail.python.org/mailman/listinfo/python-list
Line continuation and comments
I found myself building a complicated logical condition with many ands and ors which I made more manageable by putting the various terms on individual lines and breaking them with the "\" line continuation character. In this context it would have been nice to be able to add comments to lines terms which of course isn't possible because the backslash must be the last character on the line. Question: If the Python syntax were changed to allow comments after line-ending backslashes, would it break any existing code? I can't think of an example. -- https://mail.python.org/mailman/listinfo/python-list
Why can't the pointer in a PyCapsule be NULL?
Hi all, the question is in the subject. I'd like the pointer to be able to be NULL because that would make my code slightly cleaner. No big deal though. -- https://mail.python.org/mailman/listinfo/python-list
Re: Why can't the pointer in a PyCapsule be NULL?
Stefan Ram wrote: > Robert Latest writes: >>the question is in the subject. I'd like the pointer to be able to be NULL >>because that would make my code slightly cleaner. No big deal though. > > In Usenet, it is considered good style to have all relevant > content in the body. Makes sense. > . On a superficial level, the answer is: "Because > PyCapsule_GetPointer uses NULL to indicate failure." Makes sense, too. Thanks. -- https://mail.python.org/mailman/listinfo/python-list
Re: Nonuniform PRNG?
One thing you could do is to apply von Neumann de-biasing to convert a string of output bits from your biased PRNG to an unbiased string, and test the de-biased output. If such tests pass I don't know that you can be satisfied thaty your biased PRNG is close to a theorieical biased random bit stream, but if they fail that should indicate a problem. Robert E. Beaudoin On Wed, 7 Dec 2022 11:05:53 -0500 David Lowry-Duda wrote: > Inspired by the recent thread about pseudorandom number generators on > python-ideas (where I also mistakenly first wrote this message), I > began to wonder: suppose that I had a pseudorandom number generator > that attempted to generate a nonuniform distribution. Suppose for > instance that it was to generate a 0 bit 2/3 of the time, and a 1 bit > 1/3 of the time. > > How would one go about testing this PRNG against an idealized > (similarly biased) PRNG? > > - DLD -- https://mail.python.org/mailman/listinfo/python-list
Re: Need help with custom string formatter
Cameron Simpson wrote: > Stefan's code implements it's own format_field and falls back to the > original format_field(). That's standard subclassing practice, and worth > doing reflexively more of the time - it avoids _knowing_ that > format_field() just calls format(). > > So I'd take Stefan's statement above to imply that calling format() > directly should work. Yup, makes sense. -- https://mail.python.org/mailman/listinfo/python-list
Re: Need help with custom string formatter
Stefan Ram wrote: [the solution] thanks, right on the spot. I had already figured out that format_field() is the one method I need, and thanks for the str.translate method. I knew that raking seven RE's across the same string HAD to be stupid. Have a nice weekend! -- https://mail.python.org/mailman/listinfo/python-list
Need help with custom string formatter
Hi all,
I would like to modify the standard str.format() in a way that when the
input field is of type str, there is some character replacement, and the
string gets padded or truncated to the given field width. Basically like
this:
fmt = MagicString('<{s:6}>')
print(fmt.format(s='Äußerst'))
Output:
I've written a function fix_format() which, given a string and a field width,
does just that. However, I find myself unable to implement a Formatter that
uses this function in the intened way. See the example below, I hope I
sprinkled it with enough comments to make my intents clear. Thanks for any
enlightenment. The interesting part starts somewhere in the middle.
### Self contained example
import re
from string import Formatter
_replacements = [(re.compile(rx), repl) for rx, repl in (\
('Ä', 'Ae'),
('ä', 'ae'),
('Ö', 'Oe'),
('ö', 'oe'),
('Ü', 'Ue'),
('ü', 'ue'),
('ß', 'ss'))]
def fix_format(text, width):
# Seven regex passes seems awfully inefficient. I can't think of a
# better way. Besides the point though.
for rx, repl in _replacements:
text = re.sub(rx, repl, text)
# return truncated / padded version of string
return text[:width] + ' ' * max(0, width - len(text))
class Obj():
"""I'm just an object with some attributes"""
def __init__(self, **kw):
self.__dict__.update(kw)
o = Obj(x="I am X, and I'm too long",
y="ÄÖÜ Ich bin auch zu lang")
z = 'Pad me!'
format_spec = '<{o.x:6}>\n<{o.y:6}>\n<{z:10}>'
# Standard string formatting
print('Standard string formatting:')
print(format_spec.format(o=o, z=z))
# Demonstrate fix_format()
print('\nWanted output:')
print('<' + fix_format(o.x, 6) + '>')
print('<' + fix_format(o.y, 6) + '>')
print('<' + fix_format(z, 10) + '>')
# This is where my struggle begins. #
class MagicString(Formatter):
def __init__(self, format_spec):
self.spec = format_spec
super().__init__()
def format(self, **kw):
return(self.vformat(self.spec, [], kw))
def get_field(self, name, a, kw):
# Compound fields have a dot:
obj_name, _, key = name.partition('.')
obj = getattr(kw[obj_name], key) if key else kw[obj_name]
if isinstance(obj, str):
# Here I would like to call fix_format(), but I don't know where
# to get the field width.
print('get_field(): <' + obj + '>')
else:
# Here I'd like to use the "native" formatter of whatever type
# the field is.
pass
return obj, key
def get_value(self, key, a, kw):
'''I don't understand what this method is for, it never gets called'''
raise NotImplementedError
fmt = MagicString(format_spec)
print('\nReal output:')
print(fmt.format(o=o, z=z))
# Weirdly, somewhere on the way the standard formatting kicks in, too, as
# the 'Pad me!' string does get padded (which must be some postprocessing,
# as the string is still unpadded when passed into get_field())
--
https://mail.python.org/mailman/listinfo/python-list
Re: Need help with custom string formatter
Hi Stefan, I have now implemented a version of this, works nicely. I have a few minor questions / remarks: > result += ' ' *( length - len( result )) Nice, I didn't know that one could multiply strings by negative numbers without error. > def __init__( self ): > super().__init__() Isn't this a no-op? Probably a leftover from my stuff. > def format_field( self, value, format_string ): > if re.match( r'\d+', format_string )and type( value )== str: Why do you prefer re.match(r'\d+', x) over x.isdigit()? > return super().format_field( value, format_string ) Why do you prefer super().format_field() over plain format()? The doc says: "format_field() simply calls format()." So I figured I might do the same. Thanks! -- https://mail.python.org/mailman/listinfo/python-list
Re: xml.etree and namespaces -- why?
Jon Ribbens wrote:
> That's because you *always* need to know the URI of the namespace,
> because that's its only meaningful identifier. If you assume that a
> particular namespace always uses the same prefix then your code will be
> completely broken. The following two pieces of XML should be understood
> identically:
>
> http://www.inkscape.org/namespaces/inkscape";>
>
>
> and:
>
> http://www.inkscape.org/namespaces/inkscape";>
>
>
> So you can see why e.get('inkscape:label') cannot possibly work, and why
> e.get('{http://www.inkscape.org/namespaces/inkscape}label') makes sense.
I get it. It does.
> The xml.etree author obviously knew that this was cumbersome, and
> hence you can do something like:
>
> namespaces = {'inkspace': 'http://www.inkscape.org/namespaces/inkscape'}
> element = root.find('inkspace:foo', namespaces)
>
> which will work for both of the above pieces of XML.
Makes sense. It forces me to make up my own prefixes which I can then safely
use in my code rather than relying on the xml's generator to not change their
prefixes.
BTW, I only now thought to look at what actually is at Inkscape's namespace
URI, and it turns out to be quite a nice explanation of what a namespace is and
why it looks like a URL.
--
https://mail.python.org/mailman/listinfo/python-list
xml.etree and namespaces -- why?
Hi all,
For the impatient: Below the longish text is a fully self-contained Python
example that illustrates my problem.
I'm struggling to understand xml.etree's handling of namespaces. I'm trying to
parse an Inkscape document which uses several namespaces. From etree's
documentation:
If the XML input has namespaces, tags and attributes with prefixes in the
form prefix:sometag get expanded to {uri}sometag where the prefix is
replaced by the full URI.
Which means that given an Element e, I cannot directly access its attributes
using e.get() because in order to do that I need to know the URI of the
namespace. So rather than doing this (see example below):
label = e.get('inkscape:label')
I need to do this:
label = e.get('{' + uri_inkscape_namespace + '}label')
...which is the method mentioned in etree's docs:
One way to search and explore this XML example is to manually add the URI
to every tag or attribute in the xpath of a find() or findall().
[...]
A better way to search the namespaced XML example is to create a
dictionary with your own prefixes and use those in the search functions.
Good idea! Better yet, that dictionary or rather, its reverse, already exists,
because etree has used it to unnecessarily mangle the namespaces in the first
place. The documentation doesn't mention where it can be found, but we can
just use the 'xmlns:' attributes of the root element to rebuild it. Or
so I thought, until I found out that etree deletes exactly these attributes
before handing the element to the user.
I'm really stumped here. Apart from the fact that I think XML is bloated shit
anyway and has no place outside HTML, I just don't get the purpose of etree's
way of working:
1) Evaluate 'xmlns:' attributes of the element
2) Use that info to replace the existing prefixes by {uri}
3) Realizing that using {uri} prefixes is cumbersome, suggest to
the user to build their own prefix -> uri dictionary
to undo the effort of doing 1) and 2)
4) ...but witholding exactly the information that existed in the original
document by deleting the 'xmlns:' attributes from the tag
Why didn't they leave the whole damn thing alone? Keep intact and keep
the attribute 'prefix:key' literally as they are. For anyone wanting to use
the {uri} prefixes (why would they) they could have thrown in a helper
function for the prefix->URI translation.
I'm assuming that etree's designers knew what they were doing in order to make
my life easier when dealing with XML. Maybe I'm missing the forest for the
trees. Can anybody enlighten me? Thanks!
self-contained example
import xml.etree.ElementTree as ET
def test_svg(xml):
root = ET.fromstring(xml)
for e in root.iter():
print(e.tag) # tags are shown prefixed with {URI}
if e.tag.endswith('svg'):
# Since namespaces are defined inside the tag, let's use the info
# from the 'xmlns:' attributes to undo etree's URI prefixing
print('Element :')
for k, v in e.items():
print(' %s: %s' % (k, v))
# ...but alas: the 'xmlns:' attributes have been deleted by the parser
xml = '''
http://www.inkscape.org/namespaces/inkscape";
xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd";
xmlns="http://www.w3.org/2000/svg";
xmlns:svg="http://www.w3.org/2000/svg";>
'''
if __name__ == '__main__':
test_svg(xml)
--
https://mail.python.org/mailman/listinfo/python-list
Re: for -- else: what was the motivation?
wrote:
> I had another crazy thought that I AM NOT ASKING anyone to do. OK?
>
> I was wondering about a sort of catch method you could use that generates a
> pseudo-signal only when the enclosed preceding loop exits normally as a
> sort of way to handle the ELSE need without the use of a keyword known by
> the language. All you would need is an object of the right kind that is
> thrown and optionally caught.
(untested)
try:
while condition:
if not do_something():
raise RuntimeError
except RuntimeError:
pass
else:
print('Loop exited normally')
Ironically, this again relies on the much-used "else" and adds the overhead of
exception handling. Also from a natural language perspective I find the "try
... except ... else" clause just as questionable as "while ... else." Since
we're discussing weird keywords: Maybe we can find another use for "finally."
In fact, one could argue that "while ... finally" could make just as much sense
as "while ... else" (but I won't).
> Of course, even if I fleshed this out and even if anyone thought it made
> sense, there is no such need now as Python has made a choice that meets the
> need even if few may dare use it or even know about it! LOL!
--
https://mail.python.org/mailman/listinfo/python-list
Re: What to use for finding as many syntax errors as possible.
Antoon Pardon wrote: > I would like a tool that tries to find as many syntax errors as possible > in a python file. I'm puzzled as to when such a tool would be needed. How many syntax errors can you realistically put into a single Python file before compiling it for the first time? -- https://mail.python.org/mailman/listinfo/python-list
Re: What to use for finding as many syntax errors as possible.
Michael F. Stemper wrote: > How does one declare a variable in python? Sometimes it'd be nice to > be able to have declarations and any undeclared variable be flagged. To my knowledge, the closest to that is using __slots__ in class definitions. Many a time have I assigned to misspelled class members until I discovered __slots__. -- https://mail.python.org/mailman/listinfo/python-list
Re: for -- else: what was the motivation?
Axy wrote: >> Also not really a justification for "shortest block first". Wanting >> some elaboration on that. What's the value in it? > > Well, the value is productivity. No need to save puzzles "what this > hanging else belongs to?" If you find yourself asking that question, the if-block is probably too long to begin with. > Code small things first and return early, same > as taking a test: do easy and quick things first and boring and > difficult ones later. Yes, but in that case you have a very long indented "else" block, and at the point where the unindent happens you are scratching your head again like before. Better to immediately return or break and not to use any "else" block at all. -- https://mail.python.org/mailman/listinfo/python-list
Re: What to use for finding as many syntax errors as possible.
wrote: > Cameron, > > Your suggestion makes me shudder! Me, too > Removing all earlier lines of code is often guaranteed to generate errors as > variables you are using are not declared or initiated, modules are not > imported and so on. all of which aren't syntax errors, so the method should still work. Ugly as hell though. I can't think of a reason to want to find multiple syntax errors in a file. -- https://mail.python.org/mailman/listinfo/python-list
Re: for -- else: what was the motivation?
Grant Edwards wrote: > I've followed that advice for several decades. I find it much easier > to read code that's organized that way -- particularly when the > difference in block sizes is large (e.g. the first block is one line, > and the second is a a hundred). If any conditionally executed blocks is a hundred lines, I believe your code needs refactoring. I know mine does. Either the long block should go into an extra function, or you do a "fail and bail" (just learned that phrase). -- https://mail.python.org/mailman/listinfo/python-list
Re: for -- else: what was the motivation?
Chris Angelico wrote: > Yes, I'm aware that code readability becomes irrelevant for > short-duration projects. Beside the point. I'm wondering how important > it really is to have the shortest block first. I usually put the most expected / frequent / not negated block first if the whole if/else statement is not "too long". Sometimes whatever you want to do becomes pointless if a certain conditions is not met, in which case I do an early break or return and have no else block at all. > Given that for-else is an excellent, if rarely-used, construct I knew it existed but coming from C I never thought to exploit it. I know I wrote loops like this: found = None while not found: found = search(something) if found: break if not found: complain() Need to look into using "else" in these cases. -- https://mail.python.org/mailman/listinfo/python-list
Re: Implementation of an lru_cache() decorator that ignores the first argument
Hi Chris and dh, thanks for your --as usually-- thoughtful and interesting answers. Indeed, when doing these web applications I find that there are several layers of useful, maybe less useful, and unknown caching. Many of my requests rely on a notoriously unreliable read-only database outside of my control, so I cache the required data into a local DB on my server, then I do some in-memory caching of expensive data plots because I haven't figured out how to reliably exploit the client-side caching ... then every middleware on that path may or may not implement its own version of clever or not-so-clever caching. Probably not a good idea to try and outsmart that by adding yet another thing that may break or not be up-to-date at the wrong moment. That said, the only caching that SQLAlchemy does (to my knowledge) is that it stores retrieved DB items by their primary keys in the session. Not worth much since the session gets created and dumped on each request by SQA's unit of work paradigm. But the DB backend itself may be caching repeated queries. Back to Python-theory: The "Cloak" object is the only way I could think of to sneak changing data past lru_cache's key lookup mechanism. Is there some other method? Just curious. -- https://mail.python.org/mailman/listinfo/python-list
Implementation of an lru_cache() decorator that ignores the first argument
Hi all,
in a (Flask) web application I often find that many equal (SQLAlchemy) queries
are executed across subsequent requests. So I tried to cache the results of
those queries on the module level like this:
@lru_cache()
def query_db(db, args):
# do the "expensive" query
return result
This obviously doesn't work because each request uses a new database session,
so the db argument always changes from one call to the next, triggering a new
query against the database. But even if that weren't so, the function would
keep returning the same value forever (unless it's kicked out of the cache) and
not reflect the (infrequent) changes on the database. So what I need is some
decorator that can be used like this:
@lru_ignore_first(timeout=10)
def query_db(db, args):
# do the "expensive" query
return result
This is what I came up with. I'm quite happy with it so far. Question: Am I
being too clever? is it too complicated? Am I overlooking something that will
come back and bite me later? Thanks for any comments!
from functools import wraps, lru_cache
from time import time, sleep
def lru_ignore_first(timeout=0, **lru_args):
class TimeCloak():
'''All instances compare equal until timeout expires'''
__slots__ = ('x', 't', 'timeout')
def __init__(self, timeout):
self.timeout = timeout
self.t = 0
self.x = None
def __hash__(self):
return self.t
def __eq__(self, other):
return self.t == other.t
def update(self, x):
self.x = x
if self.timeout:
t = int(time())
if t >= self.t + self.timeout:
self.t = t
cloak = TimeCloak(timeout)
def decorator(func):
@lru_cache(**lru_args)
def worker(cloak, *a, **b):
return func(cloak.x, *a, **b)
@wraps(func)
def wrapped(first, *a, **kw):
cloak.update(first)
return worker(cloak, *a, **kw)
return wrapped
return decorator
@lru_ignore_first(3)
def expensive(first, par):
'''This takes a long time'''
print('Expensive:', first, par)
return par * 2
for i in range(10):
r = expensive(i, 100)
sleep(1)
print(r)
--
https://mail.python.org/mailman/listinfo/python-list
Fwd: Could not load correctly
Forwarded Message Subject:Could not load correctly Date: Sat, 21 May 2022 10:58:39 -0400 From: Robert Loomis Reply-To: b...@loomisengineering.com To: python-list@python.org I am new to python.I tried to download it to a virtual environment since I have been learning on another version. I downloaded version 3.10.4 into my windows10 operating system into directory c:\Users\Bob\PyVer\Py3913 and it said it was successful.I went to C:\Users\Bob\PyProj and made my environment by c:\Users\Bob\PyVer\Py3913\python -m venv my_env.I then activated it by my_env\Scripts\activate and it came back with a prompt.Then I tried to test it by typing python and I got what is below. What did I do wrong? Thank you, Bob Loomis -- This email has been checked for viruses by AVG. https://www.avg.com -- https://mail.python.org/mailman/listinfo/python-list
Re: SQLAlchemy: JSON vs. PickleType vs. raw string for serialised data
Loris Bennett wrote: > Thanks for the various suggestions. The data I need to store is just a > dict with maybe 3 or 4 keys and short string values probably of less > than 32 characters each per event. The traffic on the DB is going to be > very low, creating maybe a dozen events a day, mainly triggered via a > command-line interface, although I will probably set up one or two cron > jobs, each of which might generate another 0 to maybe 5 records a day. > > I could go for JSON (or rather LONGSTRING, as JSON is just an alias for > LONGSTRING, but JSON is not available on the version of MariaDB I am > using). However, that seems like overkill, since I am never going to > have to store anything near 4 GB in the field. So I should probably in > fact just use say VARCHAR(255). > > WDYT? Using TypeDecorator to transparently convert between a dict and its JSON string representation and MutableDict to track changes, you will get a completely transparent attribute that works just like a dict. Make sure to check that the generated JSON fits into your column width. I once got bitten by the fact that VARCHAR(x) can hold only x/4 characters in utf8mb4 character set. -- https://mail.python.org/mailman/listinfo/python-list
Re: SQLAlchemy: JSON vs. PickleType vs. raw string for serialised data
Albert-Jan Roskam wrote: > The event may have arbitrary, but dict-like data associated with it, > which I want to add in the field 'info'. This data never needs to be > modified, once the event has been inserted into the DB. > > What type should the info field have? JSON, PickleType, String, or > something else? > > I couldn't find any really reliable sounding information about the > relative > pros and cons, apart from a Reddit thread claiming that pickled dicts > are larger than dicts converted to JSON or String. I've done exactly this. Since my data was strictly ASCII I decided to go for JSON. But in the end you're the only one who can decide this because only you know the data. That's why you won't find any hard and fast rule for this. -- https://mail.python.org/mailman/listinfo/python-list
Re: Threading question .. am I doing this right?
Chris Angelico wrote: > I'm still curious as to the workload (requests per second), as it might still > be worth going for the feeder model. But if your current system works, then > it may be simplest to debug that rather than change. It is by all accounts a low-traffic situation, maybe one request/second. But the view in question opens four plots on one page, generating four separate requests. So with only two clients and a blocking DB connection, the whole application with eight uwsgi worker threads comes down. Now with the "extra load thread" modification, the app worked fine for several days with only two threads. Out of curiosity I tried the "feeder thread" approach with a dummy thread that just sleeps and logs something every few seconds, ten times total. For some reason it sometimes hangs after eight or nine loops, and then uwsgi cannot restart gracefully probably because it is still waiting for that thread to finish. Also my web app is built around setting up the DB connections in the request context, so using an extra thread outside that context would require doubling some DB infrastructure. Probably not worth it at this point. -- https://mail.python.org/mailman/listinfo/python-list
Re: Best way to check if there is internet?
Chris Angelico wrote: > Every language learns from every other. Except Visual Basic, which didn't learn anything from anywhere, and all that can be learned from it is how not to do it. Ugh. -- https://mail.python.org/mailman/listinfo/python-list
Re: Threading question .. am I doing this right?
Greg Ewing wrote: > * If more than one thread calls get_data() during the initial > cache filling, it looks like only one of them will wait for > the thread -- the others will skip waiting altogether and > immediately return None. Right. But that needs to be dealt with somehow. No data is no data. > * Also if the first call to get_data() times out it will > return None (although maybe that's acceptable if the caller > is expecting it). Right. Needs to be dealt with. > * The caller of get_data() is getting an object that could > be changed under it by a future update. I don't think that's a problem. If it turns out to be one I'll create a copy of the data while I hold the lock and pass that back. > -- https://mail.python.org/mailman/listinfo/python-list
Re: Threading question .. am I doing this right?
Chris Angelico wrote: > Depending on your database, this might be counter-productive. A > PostgreSQL database running on localhost, for instance, has its own > caching, and data transfers between two apps running on the same > computer can be pretty fast. The complexity you add in order to do > your own caching might be giving you negligible benefit, or even a > penalty. I would strongly recommend benchmarking the naive "keep going > back to the database" approach first, as a baseline, and only testing > these alternatives when you've confirmed that the database really is a > bottleneck. "Depending on your database" is the key phrase. This is not "my" database that is running on localhost. It is an external MSSQL server that I have no control over and whose requests frequently time out. > Hmm, it's complicated. There is another approach, and that's to > completely invert your thinking: instead of "request wants data, so > let's get data", have a thread that periodically updates your cache > from the database, and then all requests return from the cache, > without pinging the requester. Downside: It'll be requesting fairly > frequently. Upside: Very simple, very easy, no difficulties debugging. I'm using a similar approach in other places, but there I actually have a separate process that feeds my local, fast DB with unwieldy data. But that is not merely replicating, it actually preprocesses and "adds value" to the data, and the data is worth retaining on my server. I didn't want to take that approach in this instance because it is a bit too much overhead for essentially "throwaway" stuff. I like the idea of starting a separated "timed" thread in the same application. Need to think about that. Background: The clients are SBCs that display data on screens distributed throughout a manufacturing facility. They periodically refresh every few minutes. Occasionally the requests would pile up waiting for the databsase, so that some screens displayed error messages for a minute or two. Nobody cares but my pride was piqued and the error logs filled up. I've had my proposed solution running for a few days now without errors. For me that's enough but I wanted to ask you guys if I made some logical mistakes. -- https://mail.python.org/mailman/listinfo/python-list
Threading question .. am I doing this right?
I have a multi-threaded application (a web service) where several threads need data from an external database. That data is quite a lot, but it is almost always the same. Between incoming requests, timestamped records get added to the DB. So I decided to keep an in-memory cache of the DB records that gets only "topped up" with the most recent records on each request: from threading import Lock, Thread class MyCache(): def __init__(self): self.cache = None self.cache_lock = Lock() def _update(self): new_records = query_external_database() if self.cache is None: self.cache = new_records else: self.cache.extend(new_records) def get_data(self): with self.cache_lock: self._update() return self.cache my_cache = MyCache() # module level This works, but even those "small" queries can sometimes hang for a long time, causing incoming requests to pile up at the "with self.cache_lock" block. Since it is better to quickly serve the client with slightly outdated data than not at all, I came up with the "impatient" solution below. The idea is that an incoming request triggers an update query in another thread, waits for a short timeout for that thread to finish and then returns either updated or old data. class MyCache(): def __init__(self): self.cache = None self.thread_lock = Lock() self.update_thread = None def _update(self): new_records = query_external_database() if self.cache is None: self.cache = new_records else: self.cache.extend(new_records) def get_data(self): if self.cache is None: timeout = 10 # allow more time to get initial batch of data else: timeout = 0.5 with self.thread_lock: if self.update_thread is None or not self.update_thread.is_alive(): self.update_thread = Thread(target=self._update) self.update_thread.start() self.update_thread.join(timeout) return self.cache my_cache = MyCache() My question is: Is this a solid approach? Am I forgetting something? For instance, I believe that I don't need another lock to guard self.cache.append() because _update() can ever only run in one thread at a time. But maybe I'm overlooking something. -- https://mail.python.org/mailman/listinfo/python-list
Re: Best way to check if there is internet?
Abdur-Rahmaan Janhangeer wrote: > I've got my answers but the 'wandering' a bit > on this thread is at least connected to the topic ^^. > > Last question: If we check for wifi or ethernet cable connection? > > Wifi sometimes tell of connected but no internet connection. > So it detected that there is or there is no internet ... Your application obviously needs to make a certain network connection. Why don't you just check if the connection can be made? Why would you care if it's via cable or Wifi? -- https://mail.python.org/mailman/listinfo/python-list
Re: Best way to check if there is internet?
Abdur-Rahmaan Janhangeer wrote: > Well, nice perspective. > > It's a valid consideration, sound theory > but poor practicality according to me. On the contrary: It is absolutely the right and only way to do it. > It you view it like this then between the moment > we press run or enter key on the terminal, > our Python interpreter might get deleted. Yeah, but by your logic you'd have to check the existence of /usr/bin/python each time before you run it. > Though it's nice to go in and catch exceptions, > if you have a long operation upcoming it might be nice > to see if your key element is present. That is true, but you still must make sure that your long operation fails in a controlled manner if the necessary ressource becomes unavailable in the meantime. > Checking by exceptions is what the snippet > i shared does. But we check for the ability to do it > before we start. Nothing wrong with checking before. > Of course, it can change in the seconds that follow. But it's too much pure > logic at work. No. It's the correct way of doing it. -- https://mail.python.org/mailman/listinfo/python-list
Re: Long running process - how to speed up?
Shaozhong SHI wrote: > Can it be divided into several processes? I'd do it like this: from time import sleep from threading import Thread t = Thread(target=lambda: sleep(1)) t.run() # do your work here t.wait() -- https://mail.python.org/mailman/listinfo/python-list
Re: "undefined symbol" in C extension module
Dan Stromberg wrote: > Perhaps try: > https://stromberg.dnsalias.org/svn/find-sym/trunk > > It tries to find symbols in C libraries. > > In this case, I believe you'll find it in -lpythonx.ym Thanks! Found out that ldd produces many errors also with working python libraries. Turns out I tried to revive a package from v2.7. Need to adapt it to 3.x -- https://mail.python.org/mailman/listinfo/python-list
"undefined symbol" in C extension module
Hi guys, I've written some CPython extension modules in the past without problems. Now after moving to a new Archlinux box with Python3.10 installed, I can't build them any more. Or rather, I can build them but not use them due to "undefined symbols" during linking. Here's ldd's output when used on the "spam" example library from the docs: linux-vdso.so.1 (0x7ffe2454a000) libc.so.6 => /usr/lib/libc.so.6 (0x7fb6b6eb9000) /usr/lib64/ld-linux-x86-64.so.2 (0x7fb6b70a4000) undefined symbol: PyObject_Init (./build/lib.linux-x86_64-3.10/ spam.cpython-310-x86_64-linux-gnu.so) [and more of the same] It seems that I need to install some library to make this work, but which one? -- https://mail.python.org/mailman/listinfo/python-list
Re: Advanced ways to get object information from within python
Julius Hamilton wrote: > dir(scrapy) shows this: > > ['Field', 'FormRequest', 'Item', 'Request', 'Selector', 'Spider', > '__all__', '__builtins__', '__cached__', '__doc__', '__file__', > '__loader__', '__name__', '__package__', '__path__', '__spec__', > '__version__', '_txv', 'exceptions', 'http', 'item', 'link', > 'linkextractors', 'selector', 'signals', 'spiders', 'twisted_version', > 'utils', 'version_info'] > > I wish there was a convenient way for me to know what all of these are. ['%s: %s' % (x, type(getattr(scrapy, x))) for x in dir(scrapy)] -- https://mail.python.org/mailman/listinfo/python-list
Type annotation pitfall
Hi all,
this just caused me several hours of my life until I could whittle it down to
this minimal example. Simple question: Why is the x member of object "foo"
modified by initializing "bar"?
Obviously, initializing foo with None doesn't set foo.x at all. So I guess x
stays a class property, not an instance property. And instantiating bar just
added an item to the class property but didn't instantiate a new set. So
basically, neither foo nor bar ever had their "own" x, right?
Oooohh, dangerous! Never use mutable types in type hint, unless it's in
explicit dataclasses (which automatically add an appropriate __init__()?)
Now I must fine-comb all my code for more of these.
class Foo():
x : set = set()
def __init__(self, s):
if s:
self.x.add(s)
foo = Foo(None)
print(foo.x) # prints 'set()'
bar = Foo('abc')
print(foo.x) # prints '{'abc'}
--
https://mail.python.org/mailman/listinfo/python-list
Re: How to "cast" an object to a derived class?
Stefan Ram wrote: > Robert Latest writes: But how can I "promote" a >>given Opaque instance to the derived class? > > Sometimes, one can use containment instead of inheritance. Nah, doesn't work in my case. I'm trying to write a wrapper around xml.etree.ElemenTree and .Element to circumvent its idiotic namespace handling. For that I need inheritance since I want to override the find..() functions to return my derived MyElement classes. I think it could work but I somehow need to convert the root Element to a MyElement. -- https://mail.python.org/mailman/listinfo/python-list
How to "cast" an object to a derived class?
Hi all, let's assume I'm using a module that defines some class "Opaque" and also a function that creates objects of that type. In my program I subclass that type because I want some extra functionality. But how can I "promote" a given Opaque instance to the derived class? Of course I could just create an instance of the derived type and copy all attributes from the original object to the new one, but that either breaks when the opaque.py module changes, or it requires introspection. It's easily done of course but seems overly clumsy. Is there a more Pythonic way? # this is the module opaque.py class Opaque(): def __init__(self, x): assert isinstance(x, int) self.n = x def make_opaque(): return Opaque(0) # this is my program import opaque class MyClass(opaque.Opaque): # generic __init__ b/c I don't want to have to know anything about that # class def __init__(self, *a, **k): super().__init__(*a, *k) self.__something = 0 def get_something(self): return self.something op = opaque.make_opaque() # But I want to have this as type MyClass. This obviously doesn't work: my_object = MyClass(op) # But what does? -- https://mail.python.org/mailman/listinfo/python-list
Re: Ad-hoc SQL query builder for Python3?
Rich Shepard wrote: > For those interested I've found a couple of possibilities: PyPika and > SQLbuilder. I'll be looking deeply into them to learn their capabilities. In case nobody mentioned it before, don't forget to take a look at SQLAlchemy. The object-relational-mapper (ORM) creates a 1:1 mapping of Python objects to SQL table rows. -- robert -- https://mail.python.org/mailman/listinfo/python-list
Re: .title() - annoying mistake
Benjamin Schollnick wrote: > I’m sorry, but it’s as if he’s arguing for the sake of arguing. It’s > starting to feel very unproductive, and unnecessary. That was never five minutes just now! robert -- https://mail.python.org/mailman/listinfo/python-list
Re: .title() - annoying mistake
Chris Angelico wrote: > Cool thing is, nobody in Python needs to maintain anything here. That's great because I'm actually having trouble with sending log messages over the socket conection you helped me with, would you mind having a look? Regards, robert -- https://mail.python.org/mailman/listinfo/python-list
Re: How to set up a 'listening' Unix domain socket
> Chris Angelico wrote:
[Helpful stuff]
I'm actually trying to implement a SocketHandler for a Python logger. However,
I can't get my handler on the client side to send anything. Do I need to
subclass logging.SocketHandler and fill the various methods with meaning? The
documentation doesn't say so.
I've noticed that when the server is not running, logging.SocketHandler creates
the socket. I don't understand why it would; isn't it the server's job to
create the socket?
## CLIENT
import socket
import pickle
from logging import getLogger
from logging.handlers import SocketHandler
SOCKET = '/tmp/test.socket'
# This doesn't send anything to the socket
_log = getLogger(__name__)
_log.addHandler(SocketHandler(None, SOCKET))
_log.error('Logging something')
# String gets sent to socket
sock = socket.socket(socket.AF_UNIX)
sock.connect(SOCKET)
sock.send(b"Hello, world")
sock.close()
# Pickled object works fine
sock = socket.socket(socket.AF_UNIX)
sock.connect(SOCKET)
sock.send(pickle.dumps(dict(a=1)))
sock.close()
## SERVER
import os
import pickle
from socketserver import UnixStreamServer, StreamRequestHandler
SOCKET = '/tmp/test.socket'
class Handler(StreamRequestHandler):
def handle(self):
data = self.rfile.read()
try:
obj = pickle.loads(data)
except:
print('Unpickled: ', data)
else:
print('Pickled: ', type(obj))
if os.path.exists(SOCKET):
os.unlink(SOCKET)
with UnixStreamServer(SOCKET, Handler) as server:
server.serve_forever()
--
https://mail.python.org/mailman/listinfo/python-list
Re: .title() - annoying mistake
Karsten Hilbert wrote: > and life with that wart. Perfectly willing to as long as everybody agrees it's a wart ;-) robert -- https://mail.python.org/mailman/listinfo/python-list
Re: .title() - annoying mistake
Chris Angelico wrote:
> There are a small number of characters which, when case folded, become
> more than one character. The sharp S from German behaves thusly:
>
>>>> "ß".upper(), "ß".lower(), "ß".casefold(), "ß".title()
> ('SS', 'ß', 'ss', 'Ss')
Now we're getting somewhere. I'm a native German speaker and I can tell you
that this doesn't happen in the real world, simply because 'ß' never appears at
the beginning of a word and thus is never "title cased". The only occurence of
uppercase 'ß' is in all-caps text, which Python handles properly:
'bißchen'.upper()
'BISSCHEN'
...that is, properly until 2008, when the capital 'ß' was officially introduced
into German ortography:
https://en.wikipedia.org/wiki/%C3%9F#Capital_form
"Traditionally, ß did not have a capital form, although some type designers
introduced de facto capitalized variants of ß. In 2017, the Council for German
Orthography ultimately adopted capital ß, ẞ, into German orthography, ending a
long orthographic debate.[3] [...] The capital variant (U+1E9E ẞ LATIN CAPITAL
LETTER SHARP S) was encoded by ISO 10646 in 2008." So Python 3.6.8 is about 12
years behind.
As a German I also appreciate the reduced occurence of the letter combination
'SS'.
That said, the concept of "title casing" doesn't even exist in German. Titles
are spelt just like any regular sentence. I know only two definitions of the
concept "title case":
1) From Wikipedia
"Title case or headline case is a style of capitalization used for rendering
the titles of published works or works of art in English. [...]"
2) From Python (paraphrased):
"Perform an arbitrary (but defined) operation on the characters of a string."
I don't mind .title() being in Python. I would very much mind to be the person
in charge of maintaining it and having to port it into new versions of Python,
always keeping an eye on the evolution of Unicode or other standards (see
above).
It probably just comes down to me not being able to conjure up a single
sensible use case for .title() as well as the whole concept of "title casing"
in the context of a programming language.
> The neat thing about Unicode is
[many things]
> The documentation sometimes shorthands things with terms like "upper
> case" and "lower case", but that's partly because being pedantically
> correct in a docstring doesn't actually help anything, and the code
> itself IS correct.
...but hard to maintain and useless. I just love to hate .title() ;-)
robert
--
https://mail.python.org/mailman/listinfo/python-list
Re: How to set up a 'listening' Unix domain socket
Chris Angelico wrote:
>
> Hmm, your formatting's messed up, but the code looks fine to me. (Be aware
> that you seem to have a "selr" where it should be "self".)
Didn't catch that because my program didn't even get to that point ;-)
>
>> However, when I try to send somthing to that socket, I get this error
>> message:
>>
>> $ echo "Hello" | socat - UNIX-SENDTO:/tmp/test.socket 2021/03/22 11:03:22
>> socat[2188] E sendto(5, 0x55a22f414990, 6, 0, AF=1 "/tmp/test.socket", 18):
>> Protocol wrong type for socket
>>
>
> Not familiar with socat, but here's some simple Python code to trigger your
> server:
>
>>>> import socket sock = socket.socket(socket.AF_UNIX)
>>>> sock.connect("/tmp/test.socket") sock.send(b"Hello, world")
> 12
>>>> sock.close()
>>>>
Works perfectly, thanks! I'm probably not using socat right.
robert
--
https://mail.python.org/mailman/listinfo/python-list
Re: .title() - annoying mistake
Chris Angelico wrote: > If you still, after all these posts, have not yet understood that > title-casing *a single character* is a significant thing, I must admit I have no idea what title-casing even is, but I'm eager to learn. The documentation only talks about "words" and "first characters" and "remaining characters." So a single character gets converted to uppercase, whatever that may mean in the context of .title(). The .upper() method is different in that it only applies to "cased" characters, so .title() may or may not work differently on single characters. robert -- https://mail.python.org/mailman/listinfo/python-list
How to set up a 'listening' Unix domain socket
Hello, I'm trying to set up a server that receives data on a Unix domain socket using the code below. import os from socketserver import UnixStreamServer, StreamRequestHandler SOCKET = '/tmp/test.socket' class Handler(StreamRequestHandler): def handle(self): data = selr.rfile.read() print(data) if os.path.exists(SOCKET): os.unlink(SOCKET) with UnixStreamServer(SOCKET, Handler) as server: server.serve_forever() However, when I try to send somthing to that socket, I get this error message: $ echo "Hello" | socat - UNIX-SENDTO:/tmp/test.socket 2021/03/22 11:03:22 socat[2188] E sendto(5, 0x55a22f414990, 6, 0, AF=1 "/tmp/test.socket", 18): Protocol wrong type for socket I don't understand that error. Here's /tmp/test.socket: $ stat /tmp/test.socket File: ‘/tmp/test.socket’ Size: 0 Blocks: 0 IO Block: 4096 socket Device: fd00h/64768dInode: 201443577 Links: 1 Access: (0775/srwxrwxr-x) Uid: ( 1001/ dh) Gid: ( 1001/ dh) Context: unconfined_u:object_r:user_tmp_t:s0 Access: 2021-03-22 09:47:49.516298102 +0100 Modify: 2021-03-22 09:47:49.516298102 +0100 Change: 2021-03-22 09:47:49.516298102 +0100 Birth: - Sadly all examples I can find on the web are for TCP sockets, not Unix domain. Any tips? robert -- https://mail.python.org/mailman/listinfo/python-list
Re: .title() - annoying mistake
Grant Edwards wrote: > On 2021-03-20, Robert Latest via Python-list wrote: >> Mats Wichmann wrote: >>> The problem is that there isn't a standard for title case, >> >> The problem is that we owe the very existence of the .title() method to too >> much weed being smoked during Python development. It makes specific >> assumptions about a specific use case of one specific language. It doesn't >> get more idiotic, frankly. > > Ah, you've never used PHP then? > > I haven't checked but it's a fair bit that PHP has 3-4 different built-in > ways to do it, and they're all broken in interestingly unpredictable ways. I believe that 100%. PHP is the reason I switched to Python/WSGI, and I'm loving it. robert -- https://mail.python.org/mailman/listinfo/python-list
Re: .title() - annoying mistake
Benjamin Schollnick wrote: > >> I agree with everything you say. Especially the open source part. But >> wouldn't you agree that .title() with all its arbitrary specificity to >> appear in the very core of a general purpose language is quite an oddity? > > No, because it book ends the issue. > > Upper - Converts everything to uppercase Lower - Converts everything to > lowercase > > Title - Covers the cases in-between upper/lower. My only issue is that I completely fail to see how this function would be useful enough to warrant the inclusion into the *core* of a general-purpose language, including its misleading name. The fact that the function's behavior is correctly documented doesn't make its very existence less bewildering to me. Consider this function: def add_seventeen(n): '''Return n with 16.8 added''' return n + 16.8 It's like .title(): It does almost the thing its name suggests, it is correctly documented, it is useful to anybody who happens to want 16.8 added to numbers, and it might erroneously be used by someone who wants exactly 17 added and didn't bother to read the docs. > And as I mentioned the sheer amount of work that would be needed would > probably cover a phd dissertation or more… It’s a huge problem set to > respect one language, let alone all languages. And that's why I believe that such a function should be delegated to a natural language-processing library and not the core of Python. > So the only answer is to not respect the languages, and leave that up to a > later implementation or for someone else to assist in adding in support. And that too. > Heck, how do we prevent it from titlecasing abbreviations? (This is plain > text not XML…. If it was XML it would be easy!) And that too. BTW I have no beef with the person who invented .title() nor with anybody who uses it. I know that everybody can join the Python development community and propose the removal of .title() and the inclusion of add_seventeen(). That said, I doubt that .title() would make it into Python today if it weren't there already. I'm having fun with this. robert -- https://mail.python.org/mailman/listinfo/python-list
Re: .title() - annoying mistake
Chris Angelico wrote: > On Sun, Mar 21, 2021 at 10:31 PM Robert Latest via Python-list > wrote: >> Yes, I get that. But the purpose it (improperly) serves only makes sense in >> the English language. > > Why? Do titles not exist in other languages? Does no other language > capitalize words in book or other titles? I wonder if .title() properly capitalizes titles in any language. It doesn't in English (nor does it purport to), so it begs the question why it is there in the first place. German and Spanish don't have any special capitalization rules for titles; I don't know about any other languages. robert -- https://mail.python.org/mailman/listinfo/python-list
Re: .title() - annoying mistake
Benjamin Schollnick wrote: > > I’m sorry Robert, but just because it doesn’t meet your requirements, doesn’t > mean it’s useless. > > I use .title to normalize strings for data comparison, all the time. It’s a > perfect alternative to using .UPPER or .lower. > > Right in the documentation, it specifically acknowledges .title working in > foreign character sets. > > So while I feel for the fact that it doesn’t met your requirements, please > keep in mind, it does meet other peoples requirements. > > As with everything here, it’s open source. If you feel that there should be > a revised version that does met your requirements create it, or gather a > bunch of people and go the route of SCANDIR and open-source it, and petition > that it be moved into the standard library. > > Since this seems to be bugging you this much, come up with a solution. I agree with everything you say. Especially the open source part. But wouldn't you agree that .title() with all its arbitrary specificity to appear in the very core of a general purpose language is quite an oddity? robert -- https://mail.python.org/mailman/listinfo/python-list
Re: .title() - annoying mistake
Chris Angelico wrote: > On Sun, Mar 21, 2021 at 4:31 AM Robert Latest via Python-list > wrote: >> >> Mats Wichmann wrote: >> > The problem is that there isn't a standard for title case, >> >> The problem is that we owe the very existence of the .title() method to too >> much weed being smoked during Python development. It makes specific >> assumptions about a specific use case of one specific language. It doesn't >> get more idiotic, frankly. >> > > The problem is that you haven't read the documentation :) It very carefully > does NOT define itself by language, and its behaviour is identical regardless > of the language used. The documentation says: "The algorithm uses a simple language-independent definition of a word as groups of consecutive letters..." Yes, I get that. But the purpose it (improperly) serves only makes sense in the English language. Which is also the reason they called it title() and not capitalize_words(). Frankly, I can't think of any situation where this method would have any use -- in any language, including English. It is just a completely arbitrary feature, as would be a function that capitalizes only the last letter of each word. robert -- https://mail.python.org/mailman/listinfo/python-list
Re: .title() - annoying mistake
Mats Wichmann wrote: > The problem is that there isn't a standard for title case, The problem is that we owe the very existence of the .title() method to too much weed being smoked during Python development. It makes specific assumptions about a specific use case of one specific language. It doesn't get more idiotic, frankly. robert -- https://mail.python.org/mailman/listinfo/python-list
[SQLAlchemy] Struggling with association_proxy
I'm trying to implement a many-to-many relationship that associates Baskets
with Items via an association object called Link which holds the quantity of
each item. I've done that in SQLAlchemy in a very pedestrian way, such as when
I want to have six eggs in a basket:
1. Find ID of Item with name 'egg'
2. See if there is an association object with the egg ID and the basket ID
3a. if yes, set its quantity to 6
3b if no, create it with quantity 6 and add it to the items colletion in basket
The association_proxy documentation suggests that this could be done elegantly
in such a way that I could simply write
basket.contents['egg'] = 6
and be done with it. I've tried to follow the documentation at
https://docs.sqlalchemy.org/en/14/orm/extensions/associationproxy.html
but I don't understand it: It keeps creating new keyword instances rather
re-using existing ones, thus defeating the many-to-many idea. Here's what I've
come up so far, but it predictably fails because I don't want it to create new
Items on its own:
from sqlalchemy import create_engine, Column, Integer,\
String, ForeignKey
from sqlalchemy.ext.associationproxy import association_proxy
from sqlalchemy.orm import relationship, sessionmaker, backref
from sqlalchemy.orm.collections import attribute_mapped_collection
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
class Basket(Base):
__tablename__ = 'basket'
id= Column(Integer, primary_key=True)
contents = association_proxy('basket_contents', 'id')
class Link(Base):
__tablename__ = 'link'
item_id = Column(ForeignKey('item.id'), primary_key=True)
basket_id = Column(ForeignKey('basket.id'), primary_key=True)
quantity = Column(Integer)
basket = relationship(Basket, backref=backref('basket_contents',
collection_class=attribute_mapped_collection('quantity')))
item = relationship('Item')
name = association_proxy('item', 'name')
def __init__(self, name, quantity):
# this doesn't work b/c it calls Item.__init__() rather than
# looking for an existing Item
self.name = name
self.quantity = quantity
class Item(Base):
__tablename__ = 'item'
id= Column(Integer, primary_key=True)
name = Column(String(10), unique=True)
weight= Column(Integer)
color = String(10)
engine = create_engine('sqlite://')
Base.metadata.create_all(engine)
Session = sessionmaker(engine)
db = Session()
egg = Item(name='egg', weight=50, color='white')
b = Basket()
# fails because in Link.__init__(), SQLAlchemy wants to create a new Item
# rather than using the existing one.
b.contents['egg'] = 6
db.add(b)
db.commit()
--
https://mail.python.org/mailman/listinfo/python-list
Re: A 35mm film camera represented in Python object
D.M. Procida wrote: > Hi everyone, I've created <https://github.com/evildmp/C-is-for-Camera> - > a representation of a Canonet G-III QL17 in Python. [...] > The Canonet G-III QL17 is one of my favourites. One of my reasons for > writing this code is to appreciate the intricate mechanical logic > embodied in the machine. I love both photography with mechanical camears (The Nikon FE2 being my favorite) and programming. So I absolutely love your project. Also I think its totally nuts. I won't spend a second looking at your code or get otherwise involved, but I wish you best of luck! robert -- https://mail.python.org/mailman/listinfo/python-list
Re: How to implement logging for an imported module?
Richard Damon wrote:
> On 3/8/21 4:16 AM, Robert Latest via Python-list wrote:
>> Joseph L. Casale wrote:
>>>> I couldn't find any information on how to implement logging in a library
>>>> that doesn't know the name of the application that uses it. How is that
>>>> done?
>>> That's not how it works, it is the opposite. You need to know the name of
>>> its logger, and since you imported it, you do.
>> [much snipping]
>>
>>> Last word of advice, don't fight it by hacking up or patching (somehow?),
>>> it will simply not work right for any other case even slightly different
>>> than the one you somehow beat into submission.
>> I didn't waht to hack the logging system, it's just that I wasn't sure of
>> its design principles. I had hoped that if I set up a logger (including
>> levels and formatter) in my main app, the loggers in the imported modules
>> would somwhow automagically follow suit. Now I understand that for each
>> imported module I must import its logger, too, and decide how to deal with
>> its messages.
>>
>>
> Each instance of the logger inherents from a 'parent' logger, except for the
> top level logger which has the name None (as in the singleton of NoneType),
> and unless told otherwise will inherit it properties from its parent.
>
> Thus, if you get the root logger with logging.getLogger() you can set
> properties there, and unless a child logger has specifical been told not to
> inherit or has been specifically given a different value.
>
> General convention is that modules will use their name as the name of their
> logger, as that is generally unique.
>
I must admit I'm still struggling with the very basics of logging. I don't
understand the behavior of the code samples below at all, see comments.
It seems that logging.debug() et al have some side effects that are required
to get a logger to notice its level in the first place.
# Example 1:
# Why does the logger "mylog" require
# a call to the root logger in order to work?
import logging
mylog = logging.getLogger('foo')
mylog.setLevel(logging.DEBUG)
mylog.debug('1 mylog.debug()') # prints nothing
logging.debug('2 logging.debug()') # prints nothing
mylog.debug('3 mylog.debug()') # works
# Example 2:
# Why do I have to call 'logging.debug' before the root
# logger works?
import logging
mylog = logging.getLogger() # now mylog is the root logger
mylog.setLevel(logging.DEBUG) # setting level of root logger
mylog.debug('1 mylog.debug()') # prints nothing, why?
logging.debug('2 logging.debug()') # works
mylog.debug('3 mylog.debug()') # works
--
https://mail.python.org/mailman/listinfo/python-list
Re: Why assert is not a function?
Chris Angelico (and oters) wrote: [interesting stuff] I'm a late contributor here, but I'd just say: If you'd prefer a function for assert, just define a function that does what you want and be done with it. Much harder to do if assert() were a built-in function but you'd rather have a keyword ;-) robert -- https://mail.python.org/mailman/listinfo/python-list
Re: How to implement logging for an imported module?
Joseph L. Casale wrote: >> I couldn't find any information on how to implement logging in a library that >> doesn't know the name of the application that uses it. How is that done? > > That's not how it works, it is the opposite. You need to know the name of its > logger, and since you imported it, you do. [much snipping] > Last word of advice, don't fight it by hacking up or patching (somehow?), it > will simply not work right for any other case even slightly different than > the one you somehow beat into submission. I didn't waht to hack the logging system, it's just that I wasn't sure of its design principles. I had hoped that if I set up a logger (including levels and formatter) in my main app, the loggers in the imported modules would somwhow automagically follow suit. Now I understand that for each imported module I must import its logger, too, and decide how to deal with its messages. > I hope that helps, Much appreciated, robert -- https://mail.python.org/mailman/listinfo/python-list
How to implement logging for an imported module?
Hello,
I'm trying to add logging to a module that gets imported by another module. But
I only get it to work right if the imported module knows the name of the
importing module. The example given in the "Logging Cookbook" also rely on this
fact.
https://docs.python.org/3/howto/logging-cookbook.html#logging-cookbook
I couldn't find any information on how to implement logging in a library that
doesn't know the name of the application that uses it. How is that done?
Here's some example code consisting of the main module foo.py and imported
modules bar.py and baz.py
### foo.py
import logging
import bar, baz
formatter = logging.Formatter(
'%(asctime)s - %(name)s - %(levelname)s - %(message)s')
ch = logging.StreamHandler()
ch.setFormatter(formatter)
logger = logging.getLogger('foo')
logger.addHandler(ch)
logger.setLevel(logging.DEBUG)
logger.debug('debug from <%s>', __name__)
logger.info('info from <%s>', __name__)
logger.warning('warning from <%s>', __name__)
logger.error('error from <%s>', __name__)
bar.func()
baz.func()
### bar.py
'''This "generic" approach doesn't honor loglevel or formats
when imported by another module'''
import logging
l = logging.getLogger(__name__)
def func():
l.debug('debug from <%s>', __name__)
l.info('info from <%s>', __name__)
l.warning('warning from <%s>', __name__)
l.error('error from <%s>', __name__)
### baz.py
'''This only works if the importing module is named 'foo', which
precludes its use as a library module'''
import logging
l = logging.getLogger('foo.baz')
def func():
l.debug('debug from <%s>', __name__)
l.info('info from <%s>', __name__)
l.warning('warning from <%s>', __name__)
l.error('error from <%s>', __name__)
--
https://mail.python.org/mailman/listinfo/python-list
Re: Why am I unable to using rsync to install modules and appear to have to use pip install instead?
Ok this was due to an install of miniconda then choosing to unconditionally install an older version of cryptography. > On Feb 10, 2021, at 3:40 PM, Robert Nicholson > wrote: > > Just reinstalling cryptography with pip install seems to have fixed my issue. > > Any pointers on why? > >> On Feb 10, 2021, at 3:21 PM, Robert Nicholson >> wrote: >> >> I’m using Python 3.7.0 >> >> so I have multiple environments all on the same architecture with the same >> python release using anonconda. >> >> What I discovered was that if I install the cryptography module in my dev / >> uat and then tried to synchronize to production server using rsync I ended >> up with errors relating to dependencies of the cryptography module. >> >> Specifically >> >> Cannot import name ‘_get_backend’ >> >> these errors appear to have gone away if I actually use pip install >> cryptography==3.2.1 instead of using rsync from an existing install machine. >> >> why is this the case? >> >> >> > -- https://mail.python.org/mailman/listinfo/python-list
Re: Why am I unable to using rsync to install modules and appear to have to use pip install instead?
Just reinstalling cryptography with pip install seems to have fixed my issue. Any pointers on why? > On Feb 10, 2021, at 3:21 PM, Robert Nicholson > wrote: > > I’m using Python 3.7.0 > > so I have multiple environments all on the same architecture with the same > python release using anonconda. > > What I discovered was that if I install the cryptography module in my dev / > uat and then tried to synchronize to production server using rsync I ended up > with errors relating to dependencies of the cryptography module. > > Specifically > > Cannot import name ‘_get_backend’ > > these errors appear to have gone away if I actually use pip install > cryptography==3.2.1 instead of using rsync from an existing install machine. > > why is this the case? > > > -- https://mail.python.org/mailman/listinfo/python-list
Why am I unable to using rsync to install modules and appear to have to use pip install instead?
I’m using Python 3.7.0 so I have multiple environments all on the same architecture with the same python release using anonconda. What I discovered was that if I install the cryptography module in my dev / uat and then tried to synchronize to production server using rsync I ended up with errors relating to dependencies of the cryptography module. Specifically Cannot import name ‘_get_backend’ these errors appear to have gone away if I actually use pip install cryptography==3.2.1 instead of using rsync from an existing install machine. why is this the case? -- https://mail.python.org/mailman/listinfo/python-list
Tkinter performance issues between Windows and Linux
Hi community,
I have developed a tkinter GUI component, Python v3.7. It runs very well in
Linux but seeing a huge performance impact in Windows 10. While in Linux an
almost real-time performance is achieved, in Windows it is slow to an
unusable level.
The code is somewhat stripped down from the original, but the performance
difference is the same anyway. The columns can be resized by clicking on
the column border and dragging it. Resizing works only for the top row (but
it resizes the entire column).
In this demo, all bindings are avoided to exclude influence on the
component performance and thus not included. If you resize the window
(i.e., if you maximize it), you must call the function table.fit() from
IDLE shell.
Does anyone know where is this huge difference in performance coming from?
Can anything be done about it?
Thank you,
--
Robert Okadar
IT Consultant
Schedule an *online meeting <https://calendly.com/aranea-network/60min>* with
me!
Visit *aranea-mreze.hr* <http://aranea-mreze.hr> or call
* +385 91 300 8887*
import tkinter
class Resizer(tkinter.Frame):
def __init__(self, info_grid, master, **cnf):
self.table_grid = info_grid
tkinter.Frame.__init__(self, master, **cnf)
self.bind('', self.resize_column)
self.bind('', self.resize_start)
self.bind('', self.resize_end)
self._resizing = False
self.bind('', self.onDestroyEvent)
def onDestroyEvent(self, event):
self.table_grid = []
def resize_column(self, event, width = None):
#if self._resizing:
top = self.table_grid.Top
grid = self.table_grid._grid
col = self.master.grid_info()["column"]
if not width:
width = self._width + event.x_root - self._x_root
top.columnconfigure(col, minsize = width)
grid.columnconfigure(col, minsize = width)
def resize_start(self, event):
top = self.table_grid.Top
self._resizing = True
self._x_root = event.x_root
col = self.master.grid_info()["column"]
self._width = top.grid_bbox(row = 0, column = col)[2]
#print event.__dict__
col = self.master.grid_info()["column"]
#print top.grid_bbox(row = 0, column = col)
def resize_end(self, event):
pass
#self.table_grid.xscrollcommand()
#self.table_grid.column_resize_callback(col, self)
class Table(tkinter.Frame):
def __init__(self, master, columns = 10, rows = 20, width = 100,**kw):
tkinter.Frame.__init__(self, master, **kw)
self.columns = []
self._width = width
self._grid = grid = tkinter.Frame(self, bg = "#CC")
self.Top = top = tkinter.Frame(self, bg = "#DD")
self.create_top(columns)
self.create_grid(rows)
#self.bind('', self.on_table_configure)
#self.bind('', self.on_table_map)
top.pack(anchor = 'nw')#, expand = 1, fill = "both")
grid.pack(anchor = 'nw')#fill = "both",expand = 1
def on_table_map(self, event):
theight = self.winfo_height()
def fit(self):#on_table_configure(self, event):
i = 0
for frame in self.Top.grid_slaves(row = 0):
frame.resizer.resize_column(None, width = frame.winfo_width())
i += 1
theight = self.winfo_height()
fheight = self._grid.winfo_height() + self.Top.winfo_height()
#print('', theight, fheight)
if theight > fheight:
rheight = self.grid_array[0][0].winfo_height()
ammount = int((-fheight + theight) / rheight)
#print(rheight, ammount)
for i in range(ammount):
self.add_row()
self.update()
def add_row(self, ammount = 1):
columnsw = self.columns
row = []
i = len(self.grid_array)
for j in range(len(columnsw)):
bg = self.bgcolor0
if i % 2 == 1:
bg = self.bgcolor1
entry = tkinter.Label(self._grid, bg = bg, text = '%i %i' % (i, j))
entry.grid(row = i, column = j, sticky = "we", padx = 2)
row.append(entry)
self.grid_array.append(row)
bgcolor0 = "#FF"
bgcolor1 = "#EE"
def create_grid(self, height):
#grid.grid(row = 0, column = 0, sticky = "nsew")
columnsw = self.columns# = self.Top.grid_slaves(row = 1)
self.grid_array = []
for i in range(height):
row = []
for j in range(len(columnsw)):
bg = self.bgcolor0
if i % 2 == 1:
bg = self.bgcolor1
#entry = self.EntryClass(False, self,
Re: Python2.7 unicode conundrum
Richard Damon wrote: > Why do you say it has been convert to 'Latin'. The string prints as > being Unicode. Internally Python doesn't store strings as UTF-8, but as > plain Unicode (UCS-2 or UCS-4 as needed), and code-point E4 is the > character you want. You're right, this wasn't the minimal example for my problem after all. Turns out that the actual issue is somewhere between SQLAlchemy and MySQL. I took a more specific questioon overt to stackoverflow.com Thanks robert -- https://mail.python.org/mailman/listinfo/python-list
Python2.7 unicode conundrum
Hi folks, what semmingly started out as a weird database character encoding mix-up could be boiled down to a few lines of pure Python. The source-code below is real utf8 (as evidenced by the UTF code point 'c3 a4' in the third line of the hexdump). When just printed, the string "s" is displayed correctly as 'ä' (a umlaut), but the string representation shows that it seems to have been converted to latin-1 'e4' somewhere on the way. How can this be avoided? dh@jenna:~/python$ cat unicode.py # -*- encoding: utf8 -*- s = u'ä' print(s) print((s, )) dh@jenna:~/python$ hd unicode.py 23 20 2d 2a 2d 20 65 6e 63 6f 64 69 6e 67 3a 20 |# -*- encoding: | 0010 75 74 66 38 20 2d 2a 2d 0a 0a 73 20 3d 20 75 27 |utf8 -*-..s = u'| 0020 c3 a4 27 0a 0a 70 72 69 6e 74 28 73 29 0a 70 72 |..'..print(s).pr| 0030 69 6e 74 28 28 73 2c 20 29 29 0a 0a |int((s,))..| 003c dh@jenna:~/python$ python unicode.py ä (u'\xe4',) dh@jenna:~/python$ -- https://mail.python.org/mailman/listinfo/python-list
Re: on the prng behind random.random()
Dennis Lee Bieber writes: > On Mon, 19 Nov 2018 19:05:44 -0200, Robert Girault declaimed > the following: > >>I mean the fact that with 624 samples from the generator, you can >>determine the rest of the sequence completely. > > Being able to predict the sequence after a large sampling does not mean > that the /distribution of values/ is not (pseudo-) random. The problem with determining its sequence is that it might defeat its purpose. If you use mt19937 to select a pivot in random Quicksort for example (where you plan to spend n lg n time in sorting), we can frustrate your plans and force it into n^2 every time, an effective DoS attack on your software. > After all, pretty much all random number generators will produce the > same sequence if given the same starting seed... You are, in effect, > treating your 624 samples as a very large seed... I think I disagree with your take here. With mt19937, given ANY seed, I can eventually predict all the sequence without having to query the oracle any further. If you're just writing a toy software, even K&R PRNG works just fine. If you're writing a weather simulation, I suppose you need real random-like properties and still need your generator to be reproducible. If you're using random Quicksort, you do need unpredictability and reproducibility. If you're writing a crypto application, then you need something way stronger. We need all of them. But mt19937 is now useful only in toy software. -- https://mail.python.org/mailman/listinfo/python-list
Re: on the prng behind random.random()
Chris Angelico writes: > On Tue, Nov 20, 2018 at 7:31 AM Robert Girault wrote: >> Nice. So Python's random.random() does indeed use mt19937. Since it's >> been broken for years, why isn't it replaced by something newer like >> ChaCha20? Is it due to backward compatibility? That would make sense. > > What exactly do you mean by "broken"? I mean the fact that with 624 samples from the generator, you can determine the rest of the sequence completely. Sorry about mentioning ChaCha20. That was misleading. I should've said something newer like mrtg32k3a or xorshift*. > If you're generating random numbers for any sort of security purpose, > you probably should look at this: > > https://docs.python.org/3/library/secrets.html > > (New in 3.6, though, hence the "probably". If you need to support 3.5 > or older - including 2.7 - then you can't use that.) Thanks for the reference! I'm not particularly interested in security at the moment, but I would like an expert's confirmation that some of these algorithms arent't replaced due to backward compatibility. We could easily replace them, but I think we shouldn't: some people still depend on these algorithms for their experiment. Are there other reasons? -- https://mail.python.org/mailman/listinfo/python-list
Re: on the prng behind random.random()
Peter Otten <__pete...@web.de> writes: > Robert Girault wrote: > >> Looking at its source code, it seems the PRNG behind random.random() is >> Mersenne Twister, but I'm not sure. It also seems that random.random() >> is using /dev/urandom. Can someone help me to read that source code? >> >> I'm talking about CPython, by the way. I'm reading >> >> https://github.com/python/cpython/blob/master/Lib/random.py >> >> The initial comment clearly says it's Mersenne Twister, but the only >> random() function there seems to call _urandom(), which I suppose is an >> interface to /dev/urandom. >> >> What am I missing here? > > There's a class random.Random which is instantiated at the end of the file, > and random() is bound to the corresponding method: > > _inst = Random() > ... > random = _inst.random > > The Random class inherits from _random.Random [...] Thanks. I missed that. > which is implemented in C and does most of the actual work. If you can > read C: > > https://github.com/python/cpython/blob/master/Modules/_randommodule.c > > The most relevant part seems to be genrand_int32() which is wrapped by > random_random() that actually implenents the _random.Random.random() method. Nice. So Python's random.random() does indeed use mt19937. Since it's been broken for years, why isn't it replaced by something newer like ChaCha20? Is it due to backward compatibility? That would make sense. Do you know who broke mt19937 and when? I'd love to read the reference. Thank you! -- https://mail.python.org/mailman/listinfo/python-list
on the prng behind random.random()
Looking at its source code, it seems the PRNG behind random.random() is Mersenne Twister, but I'm not sure. It also seems that random.random() is using /dev/urandom. Can someone help me to read that source code? I'm talking about CPython, by the way. I'm reading https://github.com/python/cpython/blob/master/Lib/random.py The initial comment clearly says it's Mersenne Twister, but the only random() function there seems to call _urandom(), which I suppose is an interface to /dev/urandom. What am I missing here? -- https://mail.python.org/mailman/listinfo/python-list
Re: Package directory question
Ben Finney wrote:
> Robert Latest via Python-list writes:
>
>> Because the main.py script needs to import the tables.py module from
>> backend, I put this at the top if main.py:
>>
>>sys.path.append('../..')
>>import jobwatch.backend.tables as tables
>>
>> My question is: Is this the way it should be done? It looks fishy. The
>> only alternative I could come up with is to put a symlink to tables.py
>> into the frontend directory, which also seems fishy.
>
> Your fish-sense is working correctly. Both of those are hard-coding the
> path, when the Python import mechanism is designed so you don't do that.
[...]
> * To install for use while also developing, add the ‘--editable’ option.
Ah, that's what I needed. Of course the problem I had was only present
during development. I haven't really looked into pip yet, so far I've
been using only "python setup.py install".
Thanks,
robert
--
https://mail.python.org/mailman/listinfo/python-list
Package directory question
From: Robert Latest
Hello,
I'm building an application which consists of two largely distinct parts, a
frontend and a backend. The directory layout is like this:
|-- jobwatch
| |-- backend
| | |-- backend.py
| | |-- __init__.py
| | `-- tables.py
| |-- frontend
| | |-- __init__.py
| | |-- main.py
| `-- __init__.py
`-- setup.py
Because the main.py script needs to import the tables.py module from backend, I
put this at the top if main.py:
sys.path.append('../..')
import jobwatch.backend.tables as tables
My question is: Is this the way it should be done? It looks fishy. The only
alternative I could come up with is to put a symlink to tables.py into the
frontend directory, which also seems fishy. Eventually I want to package all
this up neatly to be able to use only little wrapper scripts for the backend
(running as a service) and the frontend (a wsgi app).
Any thoughts?
Thanks
robert
--- BBBS/Li6 v4.10 Toy-3
* Origin: Prism bbs (1:261/38)
--
https://mail.python.org/mailman/listinfo/python-list
Package directory question
Hello,
I'm building an application which consists of two largely distinct
parts, a frontend and a backend. The directory layout is like this:
|-- jobwatch
| |-- backend
| | |-- backend.py
| | |-- __init__.py
| | `-- tables.py
| |-- frontend
| | |-- __init__.py
| | |-- main.py
| `-- __init__.py
`-- setup.py
Because the main.py script needs to import the tables.py module from
backend, I put this at the top if main.py:
sys.path.append('../..')
import jobwatch.backend.tables as tables
My question is: Is this the way it should be done? It looks fishy. The
only alternative I could come up with is to put a symlink to tables.py
into the frontend directory, which also seems fishy. Eventually I want
to package all this up neatly to be able to use only little wrapper
scripts for the backend (running as a service) and the frontend (a wsgi
app).
Any thoughts?
Thanks
robert
--
https://mail.python.org/mailman/listinfo/python-list
Re: Weird side effect of default parameter
Steven D'Aprano wrote: > Python function default values use *early binding*: the default parameter > is evaluated, ONCE, when the function is defined, and that value is used > each time it is needed. Thanks, "early binding" was the clue I was missing. robert -- https://mail.python.org/mailman/listinfo/python-list
Weird side effect of default parameter
Hello, I don't understand the behavior of the code below. Why does the dict property "a" of both objects contain the same keys? This is only if "a=dict" is in the initializer. If I put self.a = dict() into the init function, I get two separate dicts class Foo(object): def __init__(self, x, a=dict()): self.x = x self.a = a self.a[x] = x c = Foo(1) d = Foo(2) print(c.__dict__) print(d.__dict__) robert -- https://mail.python.org/mailman/listinfo/python-list
Re: Tips or strategies to understanding how CPython works under the hood
Thanks all for the links and suggestions, they are greatly appreciated. I might be programming for a long time (relative to my age) but I haven't touched much on compilers or interpreters. Inspired a but by Python's interpreter I wrote a little bytecode interpreter in C (maybe should have upgrade to C++ but not yet), I enter custom bytes into a program array, currently can jump, add/subtract, print values on stack and halt the interpreter. I'm happy with what I've done so far. A compiler is out of my reach for the moment so I'm going to research that after learning a good bit of Python's internals. On Wed 10 Jan 2018, 12:13 bartc, wrote: > On 09/01/2018 20:12, Alain Ketterlin wrote: > > ElChino writes: > > > >> Chris Angelico wrote: > >> > >>> CPython is a stack-based interpreter, which means it loads values onto > >>> an (invisible) internal stack, processes values at the top of the > >>> stack, and removes them when it's done. > >> > >> Is this similar to how Lua operates too? > > > > No. Lua uses a register-based (virtual) machine. See > > > > https://www.lua.org/doc/jucs05.pdf > > "Registers are kept in the run-time stack ... an array". > > So it sounds like a stack is still used, but instructions directly > access specific slots on the stack, within a particular register window. > > It means there need be fewer instructions to implement some code, but > each has more operands. > > Also interesting is that the comparison operators include only EQ, LT > and LE. There is no NE, so no issues such as those discussed recently. > > > > > I think Lua was the first language in widespread use to move to a > > register-based machine. > > I believe stack-based byte-code, which is very easy to generate, can be > transformed to some 'register'-based version, if performance is the > motivation. > > (But I'm not convinced that register-based is necessarily faster. Lua is > quite fast, especially LuaJIT, but the language is also smaller and > simpler.) > > -- > bartc > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Tips or strategies to understanding how CPython works under the hood
Hey all, Been subscribed to this thread for a while but haven't contributed much. One of my ultimate goals this year is to get under the hood of CPython and get a decent understanding of mechanics Guido and the rest of you wonderful people have designed and implemented. I've been programming in python for nearly 10 years now and while I can write a mean Python script, I've been becoming more and more interested in low level operations or complex C programs so I thought I could spread my love of both to make a difference for me and others. So besides just grabbing a chunk of CPython source code and digesting it, I was wondering if those of you have read and understood the source code, do you have any tips or good starting points? Robert -- https://mail.python.org/mailman/listinfo/python-list
Re: Is it useful to set a fraction number here to the mask value?
On Thursday, November 30, 2017 at 6:17:05 PM UTC-5, Robert wrote:
> Hi,
>
> I am new to Python. Now I follow a thread on mask array usage on line:
>
>
> https://stackoverflow.com/questions/31563970/fitting-a-binomial-distribution-with-pymc-raises-zeroprobability-error-for-certa
>
>
> I understand the problem, but I don't understand the answer follow the link.
>
> Because the 'mask' array is composed of integer, if it is assigned a fraction
> number as suggested a 1.5, it will be formatted to an integer 1.
>
>
> observed_values = sp.random.binomial(n = 10.0, p = 0.1, size = 100)
> ...
> mask = sp.zeros_like(observed_values)
>
>
> Are you clear the answer's meaning?
>
>
> "You can give it a non-integer value in order to avoid the problem that you
> cite.
> For example, if you fill with, say, 1.5 that should work."
>
>
>
>
> Thanks in advance
Excuse me for the top post. Now I find the more specific question from the below
link:
https://pymc-devs.github.io/pymc/tutorial.html
At almost the bottom part of the web page, when I run the script
"masked_values = masked_array(disasters_array, mask=disasters_array==-999)"
has an error:
... masked_values = masked_array(disasters_array, mask=disasters_array==-999)
Traceback (most recent call last):
File "", line 7, in
NameError: name 'masked_array' is not defined
I use Python 2.7 on Ubuntu 16.04, 64-bit. Is there an error in the pymc
tutorial web page?
Can you tell me what is wrong?
Thanks,
===
# Switchpoint
switch = DiscreteUniform('switch', lower=0, upper=110)
# Early mean
early_mean = Exponential('early_mean', beta=1)
# Late mean
late_mean = Exponential('late_mean', beta=1)
@deterministic(plot=False)
def rate(s=switch, e=early_mean, l=late_mean):
"""Allocate appropriate mean to time series"""
out = np.empty(len(disasters_array))
# Early mean prior to switchpoint
out[:s] = e
# Late mean following switchpoint
out[s:] = l
return out
# The inefficient way, using the Impute function:
# D = Impute('D', Poisson, disasters_array, mu=r)
#
# The efficient way, using masked arrays:
# Generate masked array. Where the mask is true,
# the value is taken as missing.
masked_values = masked_array(disasters_array, mask=disasters_array==-999)
# Pass masked array to data stochastic, and it does the right thing
disasters = Poisson('disasters', mu=rate, value=masked_values, observed=True)
--
https://mail.python.org/mailman/listinfo/python-list
Is it useful to set a fraction number here to the mask value?
Hi, I am new to Python. Now I follow a thread on mask array usage on line: https://stackoverflow.com/questions/31563970/fitting-a-binomial-distribution-with-pymc-raises-zeroprobability-error-for-certa I understand the problem, but I don't understand the answer follow the link. Because the 'mask' array is composed of integer, if it is assigned a fraction number as suggested a 1.5, it will be formatted to an integer 1. observed_values = sp.random.binomial(n = 10.0, p = 0.1, size = 100) ... mask = sp.zeros_like(observed_values) Are you clear the answer's meaning? "You can give it a non-integer value in order to avoid the problem that you cite. For example, if you fill with, say, 1.5 that should work." Thanks in advance -- https://mail.python.org/mailman/listinfo/python-list
Re: Has anyone worked on docker with windows
i was also of the same opinion , but docker is available on windows too https://www.docker.com/docker-windows On Wed, Nov 29, 2017 at 12:22 PM, Percival John Hackworth wrote: > On 28-Nov-2017, Robert Clove wrote > (in article): > > > Hi, > > > > what am i trying to achieve is, container of windows with an application > > like slack on it. > > Does window container has an UI? > > > > Has anyone worked on it, is it feasible? > > AFAIK, docker is linux/CoreOS only. Ask docker.com this question. AFAIK, > the > only way to get docker to run on Windows is to run docker-machine with > virtual box. That's a coreOS VM. > > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Has anyone worked on docker with windows
Hi, what am i trying to achieve is, container of windows with an application like slack on it. Does window container has an UI? Has anyone worked on it, is it feasible? -- https://mail.python.org/mailman/listinfo/python-list
What use is of this 'cast=float ,'?
Hi, I read below code snippet on line. I am interested in the second of the last line. cast=float , I've tried it in Python. Even simply with float it has no error, but what use is it? I do see a space before the comma ','. Is it a typo or not? Thanks, self.freqslider=forms.slider( parent=self.GetWin( ), sizer=freqsizer, value=self.freq, callback= self.setfreq, minimum=−samprate/2, maximum=samprate/2, num_steps=100, style=wx.SL_HORIZONTAL, cast=float , proportion=1, ) -- https://mail.python.org/mailman/listinfo/python-list
.Re: scanf string in python
> > I have a string which is returned by a C extension. > > > > mystring = '(1,2,3)' > > > > HOW can I read the numbers in python ? > > re.findall seems the safest and easiest solution: > > >>> re.findall(r'(\d+)', '(1, 2, 3)') > ['1', '2', '3'] > >>> map(int, re.findall(r'(\d+)', '(1, 2, 3)')) > [1, 2, 3] '(1,2,3)'.scan(/\d+/).map &:to_i ===> [1, 2, 3] -- [T]he attackers tore off the woman's clothes and raped her until five others arrived The new arrivals took turns having sex with her and then sodomized her At gunpoint, the assailants forced the mother and son to have sex. www.sun-sentinel.com/local/palm-beach/sfl-flpdunbar0822nbaug22-story.html -- https://mail.python.org/mailman/listinfo/python-list
Re: Temporary variables in list comprehensions
On 1/8/2017, Steven D'Aprano wrote:
> Suppose you have an expensive calculation that gets used two or
> more times in a loop. The obvious way to avoid calculating it
> twice in an ordinary loop is with a temporary variable:
>
> result = []
> for x in data:
> tmp = expensive_calculation(x)
> result.append((tmp, tmp+1))
>
>
> But what if you are using a list comprehension? Alas, list comps
> don't let you have temporary variables, so you have to write
> this:
>
> [(expensive_calculation(x), expensive_calculation(x) + 1) for x in data]
>
> Or do you? ... no, you don't!
>
> [(tmp, tmp + 1) for x in data for tmp in [expensive_calculation(x)]]
[2,3,5].map{|n| tmp = Math.sqrt n; [tmp, tmp+1]}
===>
[[1.4142135623730951, 2.414213562373095],
[1.7320508075688772, 2.732050807568877],
[2.23606797749979, 3.23606797749979]]
--
I don't believe in western morality, i.e. don't kill civilians or children
The only way to fight a moral war is the Jewish way: Destroy their holy sites.
Kill men, women, and children (and cattle). --- Rabbi Manis Friedman
web.archive.org/web/20090605154706/http://www.momentmag.com/Exclusive/2009/2009-06/200906-Ask_Rabbis.html
archive.org/download/DavidDukeVideo/TheZionistMatrixOfPowerddhd.ogv
--
https://mail.python.org/mailman/listinfo/python-list
Re: sorting list python
On 1/18/2017, Peter Otten wrote:
> with partite.txt looking like this
>
> > 74' Kessie'
> > 90' + 4' D'alessandro
> > 51' Mchedlidze
> > 54' Banega
> > 56' Icardi
> > 65' Icardi
> > 14' Sau
>
>
> Assuming you want to perform a numerical sort on the numbers before the '
> you can just apply sorted
>
> with open(...) as f:
> print("".join(sorted(f))
>
> as long as all number strings have the same length.
>
> If that's not the case python's sorted() has decorate-sort-undecorate
> capabilities built in -- no need to do it manually:
>
> with open("partite.txt") as f:
> by_number = sorted(f, key=lambda line: int(line.partition("'")[0]))
> print("".join(by_number))
puts IO.readlines('foo.txt').sort_by( &:to_i )
14' Sau
51' Mchedlidze
54' Banega
56' Icardi
65' Icardi
74' Kessie'
90' + 4' D'alessandro
==>nil
--
Jews totally run Hollywood But I don't care if Americans think we're
running the news media, Hollywood, Wall Street, or the government. I just care
that we get to keep running them. --- Joel Stein
articles.latimes.com/2008/dec/19/opinion/oe-stein19
archive.org/download/DavidDukeTv/DoJewsControlTheMediaTheLaTimesSaysYes.flv
--
https://mail.python.org/mailman/listinfo/python-list
Re: How to flatten only one sub list of list of lists
On 3/1/2017, Sayth Renshaw wrote: > How can I flatten just a specific sublist of each list in a list of lists? > > So if I had this data > > > [ ['46295', 'Montauk', '3', '60', '85', ['19', '5', '1', '0 $277790.00']], > ['46295', 'Dark Eyes', '5', '59', '83', ['6', '4', '1', '0 $105625.00']], > ['46295', 'Machinegun Jubs', '6', '53', '77', ['6', '2', '1', '1 > $71685.00']], > ['46295', 'Zara Bay', '1', '53', '77', ['12', '2', '3', '3 $112645.00']]] > > > How can I make it be > > > [ ['46295', 'Montauk', '3', '60', '85', '19', '5', '1', '0 $277790.00'], > ['46295', 'Dark Eyes', '5', '59', '83', '6', '4', '1', '0 $105625.00'], > ['46295', 'Machinegun Jubs', '6', '53', '77', '6', '2', '1', '1 > $71685.00'], > ['46295', 'Zara Bay', '1', '53', '77', '12', '2', '3', '3 $112645.00']] > > Been looking around but most solutions just entirely flatten everything. > This was popular on SO but yeah it flattens everything I want to be more > selective > > def flatten(lst): > for elem in lst: > if type(elem) in (tuple, list): > for i in flatten(elem): > yield i > else: > yield elem > [['46295', 'Montauk', '3', '60', '85', ['19', '5', '1', '0 $277790.00']], ['46295', 'Dark Eyes', '5', '59', '83', ['6', '4', '1', '0 $105625.00']], ['46295', 'Machinegun Jubs', '6', '53', '77', ['6', '2', '1', '1 $71685.00']], ['46295', 'Zara Bay', '1', '53', '77', ['12', '2', '3', '3 $112645.00']]]. map( &:flatten ) ===> [["46295", "Montauk", "3", "60", "85", "19", "5", "1", "0 $277790.00"], ["46295", "Dark Eyes", "5", "59", "83", "6", "4", "1", "0 $105625.00"], ["46295", "Machinegun Jubs", "6", "53", "77", "6", "2", "1", "1 $71685.00"], ["46295", "Zara Bay", "1", "53", "77", "12", "2", "3", "3 $112645.00"]] -- [T]he driving force behind mass immigration is the organized Jewish community, which feels its interests are best served by a "diverse" society divided between antagonistic groups that can be easily dominated by their cohesive and privileged tribe. http://www.renseradioarchives.com/dduke/ -- https://mail.python.org/mailman/listinfo/python-list
Re: Better way to do this dict comprehesion
On 3/7/2017, Sayth Renshaw wrote:
> I have got this dictionary comprehension and it
> works but how can I do it better?
>
> from collections import Counter
>
> def find_it(seq):
> counts = dict(Counter(seq))
> a = [(k, v) for k,v in counts.items() if v % 3 == 0]
> return a[0][0]
>
> test_seq = [20,1,-1,2,-2,3,3,5,5,1,2,4,20,4,-1,-2,5]
>
> so this returns 5 which is great and the point
> of the problem I was doing.
>
> Can also do it like this
> def find_it(seq):
> counts = dict(Counter(seq))
> a = [(k) for k,v in counts.items() if v % 3 == 0]
> return a[0]
>
> But the given problem states there will always
> only be one number appearing an odd number of
> times given that is there a neater way to get
> the answer?
[20,1,-1,2,-2,3,3,5,5,1,2,4,20,4,-1,-2,5].group_by{|x| x}.
find{|k,v| v.size.odd?}.first
===>
5
--
Goyim were born only to serve us They will work, they will plow, they will
reap. We will sit like an effendi and eat. --- Rabbi Ovadia Yosef
web.archive.org/web/20101020044210/http://www.jpost.com/JewishWorld/JewishNews/Article.aspx?id=191782
archive.org/download/DavidDuke_videos/TopRabbiExposesJewishRacism-cybsdwjezqi.ogv
--
https://mail.python.org/mailman/listinfo/python-list
Re: PyPy2.7 and PyPy3.5 v5.7 - two in one release
Been meaning to give Pypy a try for a while, tonight may be the time On Tue 21 Mar 2017, 20:32 , wrote: > Hopefully this > https://morepypy.blogspot.co.uk/2017/03/pypy27-and-pypy35-v57-two-in-one-release.html > is rather more interesting for some than blatant trolling about spaces vs > tabs. > > Kindest regards. > > Mark Lawrence. > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Where to start in the field of AI with Python
Hi all, I've been given a good bit of free time at the moment and I really want to get into the field of AI with my time. I just want to get into the basics for the moment, eventually getting into stuff like machine learning and NLP (Natural Language Processing). I was wondering if any of you fine people know some good resources of where to start as I'm finding it hard to procure decent content. If any of you know of any resources in the field please send them my way. Regards, Robert -- https://mail.python.org/mailman/listinfo/python-list
Oracle Database
What is the easiest way to connect to an Oracle Database using python in order to run queries? -- https://mail.python.org/mailman/listinfo/python-list
Re: Using re to perform grep functionality in Python
Thanks, Chris. That was nice and easy and very simple. -- https://mail.python.org/mailman/listinfo/python-list
Using re to perform grep functionality in Python
Hi All,
I'm relatively new to Python, and I am having some trouble with one of my
scripts. Basically, this script connects to a server via ssh, runs Dell's
omreport output, and then externally pipes it to a mail script in cron. The
script uses an external call to grep via subprocess, but I would like to
internalize everything to run in Python. I've read that re can accomplish the
filtering, but I am having trouble getting it to duplicate what I already have.
# USING EXTERNAL GREP #
# display RAID controller info
cmd = ['ssh r...@server.ip -C \"/opt/dell/srvadmin/sbin/omreport storage
vdisk\"']
print '## Array Status ##'
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
if call(["grep", "-i", "ID"], stdin=p.stdout) != 0:
print final
sys.stdout.write(p)
p.wait()
## END ###
This gives me:
### Array Status ##
ID: 0
Layout: RAID-1
Associated Fluid Cache State : Not Applicable
vs the full output of:
## Array Status ##
List of Virtual Disks in the System
Controller PERC H310 Mini (Embedded)
ID: 0
Status: Ok
Name : Main
State : Ready
Hot Spare Policy violated : Not Assigned
Encrypted : Not Applicable
Layout: RAID-1
Size : 2,794.00 GB (334656256 bytes)
T10 Protection Information Status : No
Associated Fluid Cache State : Not Applicable
Device Name : /dev/sda
Bus Protocol : SATA
Media : HDD
Read Policy : No Read Ahead
Write Policy : Write Through
Cache Policy : Not Applicable
Stripe Element Size : 64 KB
Disk Cache Policy : Enabled
# END
I've been tinkering around with re, and the only code that I've come up with
that doesn't give me a type error is this:
cmd = ['ssh r...@server.ip -C \"/opt/dell/srvadmin/sbin/omreport storage
vdisk\"']
print '## Array Status ##'
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
for line in p.stdout:
final = re.findall('ID', line, re.DOTALL)
print final
p.wait()
Which returns this:
## Array Status ##
[]
[]
[]
['ID']
[]
[]
[]
[]
[]
['ID']
[]
[]
[]
[]
[]
[]
[]
[]
[]
[]
[]
Obviously, not what I want. Can anyone feed some input?
--
https://mail.python.org/mailman/listinfo/python-list
problems installing pylab
Dear Python I am trying to install pylab alongside python 3.6. However when I type python -m pip install pylab I get the message No module named site In the documentation [documentation for installing python modules in python 3.6.0 documentation] it says: The above example assumes that the option to adjust the system PATH environment variable was selected when installing python. How do I do this? I am running Windows 10 Looking forward to hearing from you Bob -- https://mail.python.org/mailman/listinfo/python-list
Re: What meaning is "if k in [0, len(n_trials) - 1] else None"?
On Saturday, December 3, 2016 at 6:09:02 PM UTC-5, Robert wrote:
> Hi,
>
> I am trying to understand the meaning of the below code snippet. Though I have
> a Python IDLE at computer, I can't get a way to know below line:
>
> if k in [0, len(n_trials) - 1] else None
>
> I feel it is strange for what returns when the 'if' condition is true?
> The second part 'None' is clear to me though.
>
> Could you explain it to me?
>
>
> thanks,
>
>
>
>
>
>
>
> %matplotlib inline
> from IPython.core.pylabtools import figsize
> import numpy as np
> from matplotlib import pyplot as plt
> figsize(11, 9)
>
> import scipy.stats as stats
>
> dist = stats.beta
> n_trials = [0, 1, 2, 3, 4, 5, 8, 15, 50, 500]
> data = stats.bernoulli.rvs(0.5, size=n_trials[-1])
> x = np.linspace(0, 1, 100)
>
> # For the already prepared, I'm using Binomial's conj. prior.
> for k, N in enumerate(n_trials):
> sx = plt.subplot(len(n_trials) / 2, 2, k + 1)
> plt.xlabel("$p$, probability of heads") \
> if k in [0, len(n_trials) - 1] else None
> plt.setp(sx.get_yticklabels(), visible=False)
> heads = data[:N].sum()
> y = dist.pdf(x, 1 + heads, 1 + N - heads)
> plt.plot(x, y, label="observe %d tosses,\n %d heads" % (N, heads))
> plt.fill_between(x, 0, y, color="#348ABD", alpha=0.4)
> plt.vlines(0.5, 0, 4, color="k", linestyles="--", lw=1)
>
> leg = plt.legend()
> leg.get_frame().set_alpha(0.4)
> plt.autoscale(tight=True
I just notice that there is a slash character (\) before the if line.
What is it for?
I've learn Python for a while, but I don't use it for more than 2 years now.
Thanks.
--
https://mail.python.org/mailman/listinfo/python-list
What meaning is "if k in [0, len(n_trials) - 1] else None"?
Hi,
I am trying to understand the meaning of the below code snippet. Though I have
a Python IDLE at computer, I can't get a way to know below line:
if k in [0, len(n_trials) - 1] else None
I feel it is strange for what returns when the 'if' condition is true?
The second part 'None' is clear to me though.
Could you explain it to me?
thanks,
%matplotlib inline
from IPython.core.pylabtools import figsize
import numpy as np
from matplotlib import pyplot as plt
figsize(11, 9)
import scipy.stats as stats
dist = stats.beta
n_trials = [0, 1, 2, 3, 4, 5, 8, 15, 50, 500]
data = stats.bernoulli.rvs(0.5, size=n_trials[-1])
x = np.linspace(0, 1, 100)
# For the already prepared, I'm using Binomial's conj. prior.
for k, N in enumerate(n_trials):
sx = plt.subplot(len(n_trials) / 2, 2, k + 1)
plt.xlabel("$p$, probability of heads") \
if k in [0, len(n_trials) - 1] else None
plt.setp(sx.get_yticklabels(), visible=False)
heads = data[:N].sum()
y = dist.pdf(x, 1 + heads, 1 + N - heads)
plt.plot(x, y, label="observe %d tosses,\n %d heads" % (N, heads))
plt.fill_between(x, 0, y, color="#348ABD", alpha=0.4)
plt.vlines(0.5, 0, 4, color="k", linestyles="--", lw=1)
leg = plt.legend()
leg.get_frame().set_alpha(0.4)
plt.autoscale(tight=True
--
https://mail.python.org/mailman/listinfo/python-list
Options for stdin and stdout when using pdb debugger
I would like to use pdb in an application where it isn't possible to use sys.stdin for input. I've read in the documentation for pdb.Pdb that a file object can be used instead of sys.stdin. Unfortunately, I'm not clear about my options for the file object. I've looked at rpdb on PyPI, which re-routes stdin and stdout to a socket handler. I can connect to the socket with telnet. This works well. I'm just curious if there are other options? Thank you! -- https://mail.python.org/mailman/listinfo/python-list
Survey About Package Requirements Management
Hey all, To clarify my title quickly, this survey is not about replacing pip, as yarn (https://github.com/yarnpkg/yarn) did recently with npm. This is about helping you throughout the lifespan of projects you maintain keep a handle on the dependencies it has. Where I work, we've already made something internally, and we're trying to figure out how useful this is to other people. So if you could fill out this survey (https://docs.google.com/forms/d/e/1FAIpQLSfuXl-UCIJ4PEyQCAWLBB1zPG3Ay86lOxBblV5YJCiFVldFGg/viewform?c=0&w=1), it would help us gauge interest. Thanks all! -- https://mail.python.org/mailman/listinfo/python-list
Has any one automated the vmware-vra setup using python?
-- https://mail.python.org/mailman/listinfo/python-list
Find Nested group in LDAP by this i mean group in group
Hi, I was looking for search query in LDAP for nested group memebership. It would be great if someone can provide the python code for the same. Regards -- https://mail.python.org/mailman/listinfo/python-list
memberof example using ldap
Hi, I have to find if user is the member of a group, for this i am using the following query (&(objectClass=user)(sAMAccountName=yourUserName) (memberof=CN=YourGroup,OU=Users,DC=YourDomain,DC=com)) (memberof=CN=YourGroup,OU=Users,DC=YourDomain,DC=com)== values from distinguished name of your group The above query doesn't work fine even if group doesn't exist, It always says that user is member of -- https://mail.python.org/mailman/listinfo/python-list
