Re: Line continuation and comments

[Robert Latest via Python-list]

Robert Latest wrote:
> Paul Bryan wrote:
>> Adding to this, there should be no reason now in recent versions of
>> Python to ever use line continuation. Black goes so far as to state
>> "backslashes are bad and should never be used":
>>
>> https://black.readthedocs.io/en/stable/the_black_code_style/
>   future_style.html#using-backslashes-for-with-statements
>
> Then I wonder how Mr. Black would go about these long "dot chaining"
> expressions that packages like pandas and sqlalchemy require.

Just found out that parentheses work there, too.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Line continuation and comments

[Robert Latest via Python-list]

Paul Bryan wrote:
> Adding to this, there should be no reason now in recent versions of
> Python to ever use line continuation. Black goes so far as to state
> "backslashes are bad and should never be used":
>
> https://black.readthedocs.io/en/stable/the_black_code_style/
  future_style.html#using-backslashes-for-with-statements

Then I wonder how Mr. Black would go about these long "dot chaining"
expressions that packages like pandas and sqlalchemy require.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Line continuation and comments

[Robert Latest via Python-list]

Edmondo Giovannozzi wrote:
> Il giorno mercoledì 22 febbraio 2023 alle 09:50:14 UTC+1 Robert Latest ha
> scritto:
>> I found myself building a complicated logical condition with many ands and
>> ors which I made more manageable by putting the various terms on individual
>> lines and breaking them with the "\" line continuation character. In this
>> context it would have been nice to be able to add comments to lines terms
>> which of course isn't possible because the backslash must be the last
>> character on the line. 
>> 
>> Question: If the Python syntax were changed to allow comments after
>> line-ending 
>> backslashes, would it break any existing code? I can't think of an example.
>
> Well you can if you use parenthesis like in:
> x = 5
> a = (x > 3 and
> # x < 21 or
>  x > 100
>  )
> You don't need the "\" to continue a line in this case

I like that. Never thought of it.
-- 
https://mail.python.org/mailman/listinfo/python-list

Line continuation and comments

[Robert Latest via Python-list]

I found myself building a complicated logical condition with many ands and ors
which I made more manageable by putting the various terms on individual lines
and breaking them with the "\" line continuation character. In this context it
would have been nice to be able to add comments to lines terms which of course
isn't possible because the backslash must be the last character on the line.

Question: If the Python syntax were changed to allow comments after line-ending
backslashes, would it break any existing code? I can't think of an example.
-- 
https://mail.python.org/mailman/listinfo/python-list

Why can't the pointer in a PyCapsule be NULL?

[Robert Latest via Python-list]

Hi all,

the question is in the subject. I'd like the pointer to be able to be NULL
because that would make my code slightly cleaner. No big deal though.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Why can't the pointer in a PyCapsule be NULL?

[Robert Latest via Python-list]

Stefan Ram wrote:
> Robert Latest  writes:
>>the question is in the subject. I'd like the pointer to be able to be NULL
>>because that would make my code slightly cleaner. No big deal though.
>
>   In Usenet, it is considered good style to have all relevant
>   content in the body.

Makes sense.

>   . On a superficial level, the answer is: "Because
>   PyCapsule_GetPointer uses NULL to indicate failure."

Makes sense, too. Thanks.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Need help with custom string formatter

[Robert Latest via Python-list]

Cameron Simpson wrote:
> Stefan's code implements it's own format_field and falls back to the 
> original format_field(). That's standard subclassing practice, and worth 
> doing reflexively more of the time - it avoids _knowing_ that 
> format_field() just calls format().
>
> So I'd take Stefan's statement above to imply that calling format() 
> directly should work.

Yup, makes sense.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Need help with custom string formatter

[Robert Latest via Python-list]

Stefan Ram wrote:

[the solution]

thanks, right on the spot. I had already figured out that format_field() is the
one method I need, and thanks for the str.translate method. I knew that raking
seven RE's across the same string HAD to be stupid.

Have a nice weekend!

-- 
https://mail.python.org/mailman/listinfo/python-list

Need help with custom string formatter

[Robert Latest via Python-list]

Hi all,

I would like to modify the standard str.format() in a way that when the
input field is of type str, there is some character replacement, and the
string gets padded or truncated to the given field width. Basically like
this:

fmt = MagicString('<{s:6}>')
print(fmt.format(s='Äußerst'))

Output:


I've written a function fix_format() which, given a string and a field width,
does just that. However, I find myself unable to implement a Formatter that
uses this function in the intened way. See the example below, I hope I
sprinkled it with enough comments to make my intents clear.  Thanks for any
enlightenment. The interesting part starts somewhere in the middle.

### Self contained example
import re
from string import Formatter

_replacements = [(re.compile(rx), repl) for rx, repl in (\
('Ä', 'Ae'),
('ä', 'ae'),
('Ö', 'Oe'),
('ö', 'oe'),
('Ü', 'Ue'),
('ü', 'ue'),
('ß', 'ss'))]

def fix_format(text, width):

# Seven regex passes seems awfully inefficient. I can't think of a
# better way. Besides the point though.
for rx, repl in _replacements:
text = re.sub(rx, repl, text)

# return truncated / padded version of string
return text[:width] + ' ' * max(0, width - len(text))

class Obj():
"""I'm just an object with some attributes"""
def __init__(self, **kw):
self.__dict__.update(kw)

o = Obj(x="I am X, and I'm too long",
y="ÄÖÜ Ich bin auch zu lang")
z = 'Pad me!'

format_spec = '<{o.x:6}>\n<{o.y:6}>\n<{z:10}>'

# Standard string formatting
print('Standard string formatting:')
print(format_spec.format(o=o, z=z))

# Demonstrate fix_format()
print('\nWanted output:')
print('<' + fix_format(o.x, 6) + '>')
print('<' + fix_format(o.y, 6) + '>')
print('<' + fix_format(z, 10) + '>')

# This is where my struggle begins. #

class MagicString(Formatter):
def __init__(self, format_spec):
self.spec = format_spec
super().__init__()

def format(self, **kw):
return(self.vformat(self.spec, [], kw))

def get_field(self, name, a, kw):
# Compound fields have a dot:
obj_name, _, key = name.partition('.')
obj = getattr(kw[obj_name], key) if key else kw[obj_name]
if isinstance(obj, str):
# Here I would like to call fix_format(), but I don't know where
# to get the field width.
print('get_field(): <' + obj + '>')
else:
# Here I'd like to use the "native" formatter of whatever type
# the field is.
pass
return obj, key

def get_value(self, key, a, kw):
'''I don't understand what this method is for, it never gets called'''
raise NotImplementedError

fmt = MagicString(format_spec)
print('\nReal output:')
print(fmt.format(o=o, z=z))

# Weirdly, somewhere on the way the standard formatting kicks in, too, as
# the 'Pad me!' string does get padded (which must be some postprocessing,
# as the string is still unpadded when passed into get_field())

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Need help with custom string formatter

[Robert Latest via Python-list]

Hi Stefan, 

I have now implemented a version of this, works nicely. I have a few minor
questions / remarks:

>   result += ' ' *( length - len( result ))

Nice, I didn't know that one could multiply strings by negative numbers without
error.

> def __init__( self ):
> super().__init__()

Isn't this a no-op? Probably a leftover from my stuff.

> def format_field( self, value, format_string ):
> if re.match( r'\d+', format_string )and type( value )== str:

Why do you prefer re.match(r'\d+', x) over x.isdigit()?

> return super().format_field( value, format_string )

Why do you prefer super().format_field() over plain format()? The doc says:
"format_field() simply calls format()." So I figured I might do the same.

Thanks!
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: xml.etree and namespaces -- why?

[Robert Latest via Python-list]

Jon Ribbens wrote:
> That's because you *always* need to know the URI of the namespace,
> because that's its only meaningful identifier. If you assume that a
> particular namespace always uses the same prefix then your code will be
> completely broken. The following two pieces of XML should be understood
> identically:
>
> http://www.inkscape.org/namespaces/inkscape;>
>   
>
> and:
>
> http://www.inkscape.org/namespaces/inkscape;>
>   
>
> So you can see why e.get('inkscape:label') cannot possibly work, and why
> e.get('{http://www.inkscape.org/namespaces/inkscape}label') makes sense.

I get it. It does.

> The xml.etree author obviously knew that this was cumbersome, and
> hence you can do something like:
>
> namespaces = {'inkspace': 'http://www.inkscape.org/namespaces/inkscape'}
> element = root.find('inkspace:foo', namespaces)
>
> which will work for both of the above pieces of XML.

Makes sense. It forces me to make up my own prefixes which I can then safely
use in my code rather than relying on the xml's generator to not change their
prefixes.

BTW, I only now thought to look at what actually is at Inkscape's namespace
URI, and it turns out to be quite a nice explanation of what a namespace is and
why it looks like a URL.
-- 
https://mail.python.org/mailman/listinfo/python-list

xml.etree and namespaces -- why?

[Robert Latest via Python-list]

Hi all,

For the impatient: Below the longish text is a fully self-contained Python
example that illustrates my problem.

I'm struggling to understand xml.etree's handling of namespaces. I'm trying to
parse an Inkscape document which uses several namespaces. From etree's
documentation:

If the XML input has namespaces, tags and attributes with prefixes in the
form prefix:sometag get expanded to {uri}sometag where the prefix is
replaced by the full URI.

Which means that given an Element e, I cannot directly access its attributes
using e.get() because in order to do that I need to know the URI of the
namespace. So rather than doing this (see example below):

label = e.get('inkscape:label')

I need to do this:

label = e.get('{' + uri_inkscape_namespace + '}label')

...which is the method mentioned in etree's docs:

One way to search and explore this XML example is to manually add the URI
to every tag or attribute in the xpath of a find() or findall().
[...]
A better way to search the namespaced XML example is to create a
dictionary with your own prefixes and use those in the search functions.

Good idea! Better yet, that dictionary or rather, its reverse, already exists,
because etree has used it to unnecessarily mangle the namespaces in the first
place. The documentation doesn't mention where it can be found, but we can
just use the 'xmlns:' attributes of the  root element to rebuild it. Or
so I thought, until I found out that etree deletes exactly these attributes
before handing the  element to the user.

I'm really stumped here. Apart from the fact that I think XML is bloated shit
anyway and has no place outside HTML, I just don't get the purpose of etree's
way of working:

1) Evaluate 'xmlns:' attributes of the  element
2) Use that info to replace the existing prefixes by {uri}
3) Realizing that using {uri} prefixes is cumbersome, suggest to
   the user to build their own prefix -> uri dictionary
   to undo the effort of doing 1) and 2)
4) ...but witholding exactly the information that existed in the original
   document by deleting the 'xmlns:' attributes from the  tag

Why didn't they leave the whole damn thing alone? Keep  intact and keep
the attribute 'prefix:key' literally as they are. For anyone wanting to use
the {uri} prefixes (why would they) they could have thrown in a helper
function for the prefix->URI translation.

I'm assuming that etree's designers knew what they were doing in order to make
my life easier when dealing with XML. Maybe I'm missing the forest for the
trees. Can anybody enlighten me? Thanks!


 self-contained example
import xml.etree.ElementTree as ET

def test_svg(xml):
root = ET.fromstring(xml)
for e in root.iter():
print(e.tag) # tags are shown prefixed with {URI}
if e.tag.endswith('svg'):
# Since namespaces are defined inside the  tag, let's use the info
# from the 'xmlns:' attributes to undo etree's URI prefixing
print('Element :')
for k, v in e.items():
print('  %s: %s' % (k, v))
# ...but alas: the 'xmlns:' attributes have been deleted by the parser

xml = '''


http://www.inkscape.org/namespaces/inkscape;
   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd;
   xmlns="http://www.w3.org/2000/svg;
   xmlns:svg="http://www.w3.org/2000/svg;>
  
  
  

  

'''

if __name__ == '__main__':
test_svg(xml)
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: for -- else: what was the motivation?

[Robert Latest via Python-list]

 wrote:
> I had another crazy thought that I AM NOT ASKING anyone to do. OK?
>
> I was wondering about a sort of catch method you could use that generates a
> pseudo-signal only when the enclosed preceding  loop exits normally as a
> sort of way to handle the ELSE need without the use of a keyword known by
> the language. All you would need is an object of the right kind that is
> thrown and optionally caught.


(untested)

try:
while condition:
if not do_something():
raise RuntimeError
except RuntimeError:
pass
else:
print('Loop exited normally')

Ironically, this again relies on the much-used "else" and adds the overhead of
exception handling. Also from a natural language perspective I find the "try
...  except ... else" clause just as questionable as "while ... else." Since
we're discussing weird keywords: Maybe we can find another use for "finally."
In fact, one could argue that "while ... finally" could make just as much sense
as "while ... else" (but I won't).

> Of course, even if I fleshed this out and even if anyone thought it made
> sense, there is no such need now as Python has made a choice that meets the
> need even if few may dare use it or even know about it! LOL!
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What to use for finding as many syntax errors as possible.

[Robert Latest via Python-list]

Antoon Pardon wrote:
> I would like a tool that tries to find as many syntax errors as possible 
> in a python file.

I'm puzzled as to when such a tool would be needed. How many syntax errors can
you realistically put into a single Python file before compiling it for the
first time?

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What to use for finding as many syntax errors as possible.

[Robert Latest via Python-list]

Michael F. Stemper wrote:
> How does one declare a variable in python? Sometimes it'd be nice to
> be able to have declarations and any undeclared variable be flagged.

To my knowledge, the closest to that is using __slots__ in class definitions.
Many a time have I assigned to misspelled class members until I discovered
__slots__.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: for -- else: what was the motivation?

[Robert Latest via Python-list]

Axy wrote:
>> Also not really a justification for "shortest block first". Wanting
>> some elaboration on that. What's the value in it?
>
> Well, the value is productivity. No need to save puzzles "what this 
> hanging else belongs to?"

If you find yourself asking that question, the if-block is probably too long to
begin with.

> Code small things first and return early, same 
> as taking a test: do easy and quick things first and boring and 
> difficult ones later.

Yes, but in that case you have a very long indented "else" block, and at the
point where the unindent happens you are scratching your head again like
before. Better to immediately return or break and not to use any "else" block
at all.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What to use for finding as many syntax errors as possible.

[Robert Latest via Python-list]

 wrote:
> Cameron,
>
> Your suggestion makes me shudder!

Me, too

> Removing all earlier lines of code is often guaranteed to generate errors as
> variables you are using are not declared or initiated, modules are not
> imported and so on.

all of which aren't syntax errors, so the method should still work. Ugly as
hell though. I can't think of a reason to want to find multiple syntax errors
in a file.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: for -- else: what was the motivation?

[Robert Latest via Python-list]

Grant Edwards wrote:
> I've followed that advice for several decades. I find it much easier
> to read code that's organized that way -- particularly when the
> difference in block sizes is large (e.g. the first block is one line,
> and the second is a a hundred).

If any conditionally executed blocks is a hundred lines, I believe your code
needs refactoring. I know mine does. Either the long block should go into an
extra function, or you do a "fail and bail" (just learned that phrase).
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: for -- else: what was the motivation?

[Robert Latest via Python-list]

Chris Angelico wrote:
> Yes, I'm aware that code readability becomes irrelevant for
> short-duration projects. Beside the point. I'm wondering how important
> it really is to have the shortest block first.

I usually put the most expected / frequent / not negated block first if the
whole if/else statement is not "too long". Sometimes whatever you want to do
becomes pointless if a certain conditions is not met, in which case I do an
early break or return and have no else block at all.

> Given that for-else is an excellent, if rarely-used, construct

I knew it existed but coming from C I never thought to exploit it. I know I
wrote loops like this:

found = None
while not found:
found = search(something)
if found:
break
if not found:
complain()

Need to look into using "else" in these cases.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Implementation of an lru_cache() decorator that ignores the first argument

[Robert Latest via Python-list]

Hi Chris and dh,

thanks for your --as usually-- thoughtful and interesting answers. Indeed, when
doing these web applications I find that there are several layers of useful,
maybe less useful, and unknown caching. Many of my requests rely on a
notoriously unreliable read-only database outside of my control, so I cache the
required data into a local DB on my server, then I do some in-memory caching of
expensive data plots because I haven't figured out how to reliably exploit the
client-side caching ... then every middleware on that path may or may not
implement its own version of clever or not-so-clever caching. Probably not a
good idea to try and outsmart that by adding yet another thing that may break
or not be up-to-date at the wrong moment.

That said, the only caching that SQLAlchemy does (to my knowledge) is that it
stores retrieved DB items by their primary keys in the session. Not worth much
since the session gets created and dumped on each request by SQA's unit of work
paradigm. But the DB backend itself may be caching repeated queries.

Back to Python-theory: The "Cloak" object is the only way I could think of to
sneak changing data past lru_cache's key lookup mechanism. Is there some other
method? Just curious.

-- 
https://mail.python.org/mailman/listinfo/python-list

Implementation of an lru_cache() decorator that ignores the first argument

[Robert Latest via Python-list]

Hi all,

in a (Flask) web application I often find that many equal (SQLAlchemy) queries
are executed across subsequent requests. So I tried to cache the results of
those queries on the module level like this:

@lru_cache()
def query_db(db, args):
# do the "expensive" query
return result

This obviously doesn't work because each request uses a new database session,
so the db argument always changes from one call to the next, triggering a new
query against the database. But even if that weren't so, the function would
keep returning the same value forever (unless it's kicked out of the cache) and
not reflect the (infrequent) changes on the database. So what I need is some
decorator that can be used like this:

@lru_ignore_first(timeout=10)
def query_db(db, args):
# do the "expensive" query
return result

This is what I came up with. I'm quite happy with it so far.  Question: Am I
being too clever? is it too complicated? Am I overlooking something that will
come back and bite me later? Thanks for any comments!

from functools import wraps, lru_cache
from time import time, sleep

def lru_ignore_first(timeout=0, **lru_args):

class TimeCloak():
'''All instances compare equal until timeout expires'''
__slots__ = ('x', 't', 'timeout')

def __init__(self, timeout):
self.timeout = timeout
self.t = 0
self.x = None

def __hash__(self):
return self.t

def __eq__(self, other):
return self.t == other.t

def update(self, x):
self.x = x
if self.timeout:
t = int(time())
if t >= self.t + self.timeout:
self.t = t

cloak = TimeCloak(timeout)

def decorator(func):

@lru_cache(**lru_args)
def worker(cloak, *a, **b):
return func(cloak.x, *a, **b)

@wraps(func)
def wrapped(first, *a, **kw):
cloak.update(first)
return worker(cloak, *a, **kw)

return wrapped

return decorator

@lru_ignore_first(3)
def expensive(first, par):
'''This takes a long time'''
print('Expensive:', first, par)
return par * 2

for i in range(10):
r = expensive(i, 100)
sleep(1)
print(r)
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: SQLAlchemy: JSON vs. PickleType vs. raw string for serialised data

[Robert Latest via Python-list]

Loris Bennett wrote:
> Thanks for the various suggestions.  The data I need to store is just a
> dict with maybe 3 or 4 keys and short string values probably of less
> than 32 characters each per event.  The traffic on the DB is going to be
> very low, creating maybe a dozen events a day, mainly triggered via a
> command-line interface, although I will probably set up one or two cron
> jobs, each of which might generate another 0 to maybe 5 records a day.
>
> I could go for JSON (or rather LONGSTRING, as JSON is just an alias for
> LONGSTRING, but JSON is not available on the version of MariaDB I am
> using).  However, that seems like overkill, since I am never going to
> have to store anything near 4 GB in the field.  So I should probably in
> fact just use say VARCHAR(255).
>
> WDYT?

Using TypeDecorator to transparently convert between a dict and its JSON string
representation and MutableDict to track changes, you will get a completely
transparent attribute that works just like a dict. Make sure to check that the
generated JSON fits into your column width. I once got bitten by the fact that
VARCHAR(x) can hold only x/4 characters in utf8mb4 character set.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: SQLAlchemy: JSON vs. PickleType vs. raw string for serialised data

[Robert Latest via Python-list]

Albert-Jan Roskam wrote:
>  The event may have arbitrary, but dict-like data associated with it,
>  which I want to add in the field 'info'.  This data never needs to be
>  modified, once the event has been inserted into the DB.
>
>  What type should the info field have?  JSON, PickleType, String, or
>  something else?
>
>  I couldn't find any really reliable sounding information about the
>  relative
>  pros and cons, apart from a Reddit thread claiming that pickled dicts
>  are larger than dicts converted to JSON or String.

I've done exactly this. Since my data was strictly ASCII I decided to go for
JSON. But in the end you're the only one who can decide this because only you
know the data. That's why you won't find any hard and fast rule for this.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Threading question .. am I doing this right?

[Robert Latest via Python-list]

Chris Angelico wrote:
> I'm still curious as to the workload (requests per second), as it might still
> be worth going for the feeder model. But if your current system works, then
> it may be simplest to debug that rather than change.

It is by all accounts a low-traffic situation, maybe one request/second. But
the view in question opens four plots on one page, generating four separate
requests. So with only two clients and a blocking DB connection, the whole
application with eight uwsgi worker threads comes down. Now with the "extra
load thread" modification, the app worked fine for several days with only two
threads.

Out of curiosity I tried the "feeder thread" approach with a dummy thread that
just sleeps and logs something every few seconds, ten times total. For some
reason it sometimes hangs after eight or nine loops, and then uwsgi cannot
restart gracefully probably because it is still waiting for that thread to
finish. Also my web app is built around setting up the DB connections in the
request context, so using an extra thread outside that context would require
doubling some DB infrastructure. Probably not worth it at this point.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Best way to check if there is internet?

[Robert Latest via Python-list]

Chris Angelico wrote:
> Every language learns from every other.

Except Visual Basic, which didn't learn anything from anywhere, and all that
can be learned from it is how not to do it. Ugh.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Threading question .. am I doing this right?

[Robert Latest via Python-list]

Greg Ewing wrote:
> * If more than one thread calls get_data() during the initial
> cache filling, it looks like only one of them will wait for
> the thread -- the others will skip waiting altogether and
> immediately return None.

Right. But that needs to be dealt with somehow. No data is no data.

> * Also if the first call to get_data() times out it will
> return None (although maybe that's acceptable if the caller
> is expecting it).

Right. Needs to be dealt with.

> * The caller of get_data() is getting an object that could
> be changed under it by a future update.

I don't think that's a problem. If it turns out to be one I'll create a copy of
the data while I hold the lock and pass that back.
>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Threading question .. am I doing this right?

[Robert Latest via Python-list]

Chris Angelico wrote:
> Depending on your database, this might be counter-productive. A
> PostgreSQL database running on localhost, for instance, has its own
> caching, and data transfers between two apps running on the same
> computer can be pretty fast. The complexity you add in order to do
> your own caching might be giving you negligible benefit, or even a
> penalty. I would strongly recommend benchmarking the naive "keep going
> back to the database" approach first, as a baseline, and only testing
> these alternatives when you've confirmed that the database really is a
> bottleneck.

"Depending on your database" is the key phrase. This is not "my" database that
is running on localhost. It is an external MSSQL server that I have no control
over and whose requests frequently time out.

> Hmm, it's complicated. There is another approach, and that's to
> completely invert your thinking: instead of "request wants data, so
> let's get data", have a thread that periodically updates your cache
> from the database, and then all requests return from the cache,
> without pinging the requester. Downside: It'll be requesting fairly
> frequently. Upside: Very simple, very easy, no difficulties debugging.

I'm using a similar approach in other places, but there I actually have a
separate process that feeds my local, fast DB with unwieldy data. But that is
not merely replicating, it actually preprocesses and "adds value" to the data,
and the data is worth retaining on my server. I didn't want to take that
approach in this instance because it is a bit too much overhead for essentially
"throwaway" stuff. I like the idea of starting a separated "timed" thread in
the same application. Need to think about that.

Background: The clients are SBCs that display data on screens distributed
throughout a manufacturing facility. They periodically refresh every few
minutes. Occasionally the requests would pile up waiting for the databsase, so
that some screens displayed error messages for a minute or two. Nobody cares
but my pride was piqued and the error logs filled up.

I've had my proposed solution running for a few days now without errors. For me
that's enough but I wanted to ask you guys if I made some logical mistakes.
-- 
https://mail.python.org/mailman/listinfo/python-list

Threading question .. am I doing this right?

[Robert Latest via Python-list]

I have a multi-threaded application (a web service) where several threads need
data from an external database. That data is quite a lot, but it is almost
always the same. Between incoming requests, timestamped records get added to
the DB.

So I decided to keep an in-memory cache of the DB records that gets only
"topped up" with the most recent records on each request:


from threading import Lock, Thread


class MyCache():
def __init__(self):
self.cache = None
self.cache_lock = Lock()

def _update(self):
new_records = query_external_database()
if self.cache is None:
self.cache = new_records
else:
self.cache.extend(new_records)

def get_data(self):
with self.cache_lock:
self._update()

return self.cache

my_cache = MyCache() # module level


This works, but even those "small" queries can sometimes hang for a long time,
causing incoming requests to pile up at the "with self.cache_lock" block.

Since it is better to quickly serve the client with slightly outdated data than
not at all, I came up with the "impatient" solution below. The idea is that an
incoming request triggers an update query in another thread, waits for a short
timeout for that thread to finish and then returns either updated or old data.

class MyCache():
def __init__(self):
self.cache = None
self.thread_lock = Lock()
self.update_thread = None

def _update(self):
new_records = query_external_database()
if self.cache is None:
self.cache = new_records
else:
self.cache.extend(new_records)

def get_data(self):
if self.cache is None:
timeout = 10 # allow more time to get initial batch of data
else:
timeout = 0.5
with self.thread_lock:
if self.update_thread is None or not self.update_thread.is_alive():
self.update_thread = Thread(target=self._update)
self.update_thread.start()
self.update_thread.join(timeout)

return self.cache

my_cache = MyCache()

My question is: Is this a solid approach? Am I forgetting something? For
instance, I believe that I don't need another lock to guard self.cache.append()
because _update() can ever only run in one thread at a time. But maybe I'm
overlooking something.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Best way to check if there is internet?

[Robert Latest via Python-list]

Abdur-Rahmaan Janhangeer wrote:
> I've got my answers but the 'wandering' a bit
> on this thread is at least connected to the topic ^^.
>
> Last question: If we check for wifi or ethernet cable connection?
>
> Wifi sometimes tell of connected but no internet connection.
> So it detected that there is or there is no internet ...

Your application obviously needs to make a certain network connection. Why
don't you just check if the connection can be made? Why would you care if it's
via cable or Wifi?
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Best way to check if there is internet?

[Robert Latest via Python-list]

Abdur-Rahmaan Janhangeer wrote:
> Well, nice perspective.
>
> It's a valid consideration, sound theory
> but poor practicality according to me.

On the contrary: It is absolutely the right and only way to do it.

> It you view it like this then between the moment
> we press run or enter key on the terminal,
> our Python interpreter might get deleted.

Yeah, but by your logic you'd have to check the existence of /usr/bin/python
each time before you run it.

> Though it's nice to go in and catch exceptions,
> if you have a long operation upcoming it might be nice
> to see if your key element is present.

That is true, but you still must make sure that your long operation fails in a
controlled manner if the necessary ressource becomes unavailable in the
meantime.

> Checking by exceptions is what the snippet
> i shared does. But we check for the ability to do it
> before we start.

Nothing wrong with checking before.

> Of course, it can change in the seconds that follow. But it's too much pure
> logic at work.

No. It's the correct way of doing it.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Long running process - how to speed up?

[Robert Latest via Python-list]

Shaozhong SHI wrote:
> Can it be divided into several processes?

I'd do it like this:

from time import sleep
from threading import Thread

t = Thread(target=lambda: sleep(1))
t.run()

# do your work here

t.wait()

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: "undefined symbol" in C extension module

[Robert Latest via Python-list]

Dan Stromberg wrote:
> Perhaps try:
> https://stromberg.dnsalias.org/svn/find-sym/trunk
>
> It tries to find symbols in C libraries.
>
> In this case, I believe you'll find it in -lpythonx.ym

Thanks! Found out that ldd produces many errors also with working python
libraries. Turns out I tried to revive a package from v2.7. Need to adapt
it to 3.x
-- 
https://mail.python.org/mailman/listinfo/python-list

"undefined symbol" in C extension module

[Robert Latest via Python-list]

Hi guys,

I've written some CPython extension modules in the past without problems. Now
after moving to a new Archlinux box with Python3.10 installed, I can't build
them any more. Or rather, I can build them but not use them due to "undefined
symbols" during linking. Here's ldd's output when used on the "spam" example
library from the docs:

linux-vdso.so.1 (0x7ffe2454a000)
libc.so.6 => /usr/lib/libc.so.6 (0x7fb6b6eb9000)
/usr/lib64/ld-linux-x86-64.so.2 (0x7fb6b70a4000)
undefined symbol: PyObject_Init (./build/lib.linux-x86_64-3.10/
spam.cpython-310-x86_64-linux-gnu.so)

[and more of the same]

It seems that I need to install some library to make this work, but which one?
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Advanced ways to get object information from within python

[Robert Latest via Python-list]

Julius Hamilton wrote:
> dir(scrapy) shows this:
>
> ['Field', 'FormRequest', 'Item', 'Request', 'Selector', 'Spider',
> '__all__', '__builtins__', '__cached__', '__doc__', '__file__',
> '__loader__', '__name__', '__package__', '__path__', '__spec__',
> '__version__', '_txv', 'exceptions', 'http', 'item', 'link',
> 'linkextractors', 'selector', 'signals', 'spiders', 'twisted_version',
> 'utils', 'version_info']
>
> I wish there was a convenient way for me to know what all of these are.

['%s: %s' % (x, type(getattr(scrapy, x))) for x in dir(scrapy)]

-- 
https://mail.python.org/mailman/listinfo/python-list

Type annotation pitfall

[Robert Latest via Python-list]

Hi all,

this just caused me several hours of my life until I could whittle it down to
this minimal example. Simple question: Why is the x member of object "foo"
modified by initializing "bar"?

Obviously, initializing foo with None doesn't set foo.x at all. So I guess x
stays a class property, not an instance property. And instantiating bar just
added an item to the class property but didn't instantiate a new set. So
basically, neither foo nor bar ever had their "own" x, right?

Oooohh, dangerous! Never use mutable types in type hint, unless it's in
explicit dataclasses (which automatically add an appropriate __init__()?)

Now I must fine-comb all my code for more of these.

class Foo():
x : set = set()

def __init__(self, s):
if s:
self.x.add(s)

foo = Foo(None)
print(foo.x) # prints 'set()'
bar = Foo('abc')
print(foo.x) # prints '{'abc'}

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to "cast" an object to a derived class?

[Robert Latest via Python-list]

Stefan Ram wrote:
> Robert Latest  writes: But how can I "promote" a
>>given Opaque instance to the derived class?
>
>   Sometimes, one can use containment instead of inheritance.

Nah, doesn't work in my case. I'm trying to write a wrapper around
xml.etree.ElemenTree and .Element  to circumvent its idiotic namespace
handling. For that I need inheritance since I want to override the find..()
functions to return my derived MyElement classes. I think it could work but I
somehow need to convert the root Element to a MyElement.

-- 
https://mail.python.org/mailman/listinfo/python-list

How to "cast" an object to a derived class?

[Robert Latest via Python-list]

Hi all, let's assume I'm using a module that defines some class "Opaque" and
also a function that creates objects of that type. In my program I subclass
that type because I want some extra functionality. But how can I "promote" a
given Opaque instance to the derived class? Of course I could just create an
instance of the derived type and copy all attributes from the original object
to the new one, but that either breaks when the opaque.py module changes, or it
requires introspection. It's easily done of course but seems overly clumsy. Is
there a more Pythonic way?

# this is the module opaque.py
class Opaque(): def __init__(self, x): assert isinstance(x, int) self.n = x

def make_opaque(): return Opaque(0)

# this is my program
import opaque class MyClass(opaque.Opaque):
# generic __init__ b/c I don't want to have to know anything about that
# class
def __init__(self, *a, **k): super().__init__(*a, *k) self.__something = 0

def get_something(self): return self.something

op = opaque.make_opaque()

# But I want to have this as type MyClass. This obviously doesn't work:
my_object = MyClass(op)
# But what does?
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Ad-hoc SQL query builder for Python3?

[Robert Latest via Python-list]

Rich Shepard wrote:
> For those interested I've found a couple of possibilities: PyPika and
> SQLbuilder. I'll be looking deeply into them to learn their capabilities.

In case nobody mentioned it before, don't forget to take a look at SQLAlchemy.
The object-relational-mapper (ORM) creates a 1:1 mapping of Python objects to
SQL table rows.

-- 
robert
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: .title() - annoying mistake

[Robert Latest via Python-list]

Benjamin Schollnick wrote:

> I’m sorry, but it’s as if he’s arguing for the sake of arguing.  It’s
> starting to feel very unproductive, and unnecessary.

That was never five minutes just now!

robert
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: .title() - annoying mistake

[Robert Latest via Python-list]

Chris Angelico wrote:
> Cool thing is, nobody in Python needs to maintain anything here.

That's great because I'm actually having trouble with sending log messages over
the socket conection you helped me with, would you mind having a look?

Regards,
robert
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to set up a 'listening' Unix domain socket

[Robert Latest via Python-list]

> Chris Angelico wrote:

[Helpful stuff]

I'm actually trying to implement a SocketHandler for a Python logger. However,
I can't get my handler on the client side to send anything. Do I need to
subclass logging.SocketHandler and fill the various methods with meaning? The
documentation doesn't say so.

I've noticed that when the server is not running, logging.SocketHandler creates
the socket. I don't understand why it would; isn't it the server's job to
create the socket?

## CLIENT 

import socket
import pickle
from logging import getLogger
from logging.handlers import SocketHandler

SOCKET = '/tmp/test.socket'

# This doesn't send anything to the socket
_log = getLogger(__name__)
_log.addHandler(SocketHandler(None, SOCKET))
_log.error('Logging something')

# String gets sent to socket
sock = socket.socket(socket.AF_UNIX)
sock.connect(SOCKET)
sock.send(b"Hello, world")
sock.close()

# Pickled object works fine
sock = socket.socket(socket.AF_UNIX)
sock.connect(SOCKET)
sock.send(pickle.dumps(dict(a=1)))
sock.close()


## SERVER 

import os
import pickle
from socketserver import UnixStreamServer, StreamRequestHandler

SOCKET = '/tmp/test.socket'

class Handler(StreamRequestHandler):

def handle(self):
data = self.rfile.read()
try:
obj = pickle.loads(data)
except:
print('Unpickled: ', data)
else:
print('Pickled: ', type(obj))

if os.path.exists(SOCKET):
os.unlink(SOCKET)
with UnixStreamServer(SOCKET, Handler) as server:
server.serve_forever()

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: .title() - annoying mistake

[Robert Latest via Python-list]

Karsten Hilbert wrote:
> and life with that wart.

Perfectly willing to as long as everybody agrees it's a wart ;-)

robert
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: .title() - annoying mistake

[Robert Latest via Python-list]

Chris Angelico wrote:
> There are a small number of characters which, when case folded, become
> more than one character. The sharp S from German behaves thusly:
>
 "ß".upper(), "ß".lower(), "ß".casefold(), "ß".title()
> ('SS', 'ß', 'ss', 'Ss')

Now we're getting somewhere. I'm a native German speaker and I can tell you
that this doesn't happen in the real world, simply because 'ß' never appears at
the beginning of a word and thus is never "title cased". The only occurence of
uppercase 'ß' is in all-caps text, which Python handles properly:

'bißchen'.upper()
'BISSCHEN'

...that is, properly until 2008, when the capital 'ß' was officially introduced
into German ortography:

https://en.wikipedia.org/wiki/%C3%9F#Capital_form
"Traditionally, ß did not have a capital form, although some type designers
introduced de facto capitalized variants of ß. In 2017, the Council for German
Orthography ultimately adopted capital ß, ẞ, into German orthography, ending a
long orthographic debate.[3] [...] The capital variant (U+1E9E ẞ LATIN CAPITAL
LETTER SHARP S) was encoded by ISO 10646 in 2008." So Python 3.6.8 is about 12
years behind.

As a German I also appreciate the reduced occurence of the letter combination
'SS'.

That said, the concept of "title casing" doesn't even exist in German. Titles
are spelt just like any regular sentence. I know only two definitions of the
concept "title case":

1) From Wikipedia
"Title case or headline case is a style of capitalization used for rendering
the titles of published works or works of art in English. [...]"

2) From Python (paraphrased):
"Perform an arbitrary (but defined) operation on the characters of a string."

I don't mind .title() being in Python. I would very much mind to be the person
in charge of maintaining it and having to port it into new versions of Python,
always keeping an eye on the evolution of Unicode or other standards (see
above).

It probably just comes down to me not being able to conjure up a single
sensible use case for .title() as well as the whole concept of "title casing"
in the context of a programming language.

> The neat thing about Unicode is 

[many things]

> The documentation sometimes shorthands things with terms like "upper
> case" and "lower case", but that's partly because being pedantically
> correct in a docstring doesn't actually help anything, and the code
> itself IS correct.

...but hard to maintain and useless. I just love to hate .title() ;-)

robert
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to set up a 'listening' Unix domain socket

[Robert Latest via Python-list]

Chris Angelico wrote:
>
> Hmm, your formatting's messed up, but the code looks fine to me. (Be aware
> that you seem to have a "selr" where it should be "self".)

Didn't catch that because my program didn't even get to that point ;-)

>
>> However, when I try to send somthing to that socket, I get this error
>> message:
>>
>> $ echo "Hello" | socat - UNIX-SENDTO:/tmp/test.socket 2021/03/22 11:03:22
>> socat[2188] E sendto(5, 0x55a22f414990, 6, 0, AF=1 "/tmp/test.socket", 18):
>> Protocol wrong type for socket
>>
>
> Not familiar with socat, but here's some simple Python code to trigger your
> server:
>
 import socket sock = socket.socket(socket.AF_UNIX)
 sock.connect("/tmp/test.socket") sock.send(b"Hello, world")
> 12
 sock.close()


Works perfectly, thanks! I'm probably not using socat right.

robert
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: .title() - annoying mistake

[Robert Latest via Python-list]

Chris Angelico wrote:
> If you still, after all these posts, have not yet understood that
> title-casing *a single character* is a significant thing,

I must admit I have no idea what title-casing even is, but I'm eager to learn.
The documentation only talks about "words" and "first characters" and
"remaining characters." So a single character gets converted to uppercase,
whatever that may mean in the context of .title(). The .upper() method is
different in that it only applies to "cased" characters, so .title() may or may
not work differently on single characters. 

robert
-- 
https://mail.python.org/mailman/listinfo/python-list

How to set up a 'listening' Unix domain socket

[Robert Latest via Python-list]

Hello,

I'm trying to set up a server that receives data on a Unix domain socket using
the code below.

import os from socketserver import UnixStreamServer, StreamRequestHandler

SOCKET = '/tmp/test.socket'

class Handler(StreamRequestHandler):

def handle(self): data = selr.rfile.read() print(data)

if os.path.exists(SOCKET): os.unlink(SOCKET) with UnixStreamServer(SOCKET,
Handler) as server: server.serve_forever()


However, when I try to send somthing to that socket, I get this error message:

$ echo "Hello" | socat - UNIX-SENDTO:/tmp/test.socket 2021/03/22 11:03:22
socat[2188] E sendto(5, 0x55a22f414990, 6, 0, AF=1 "/tmp/test.socket", 18):
Protocol wrong type for socket

I don't understand that error. Here's /tmp/test.socket:

$ stat /tmp/test.socket
  File: ‘/tmp/test.socket’
  Size: 0   Blocks: 0  IO Block: 4096   socket
Device: fd00h/64768dInode: 201443577   Links: 1
Access: (0775/srwxrwxr-x)  Uid: ( 1001/  dh)   Gid: ( 1001/  dh)
Context: unconfined_u:object_r:user_tmp_t:s0
Access: 2021-03-22 09:47:49.516298102 +0100
Modify: 2021-03-22 09:47:49.516298102 +0100
Change: 2021-03-22 09:47:49.516298102 +0100
 Birth: -

Sadly all examples I can find on the web are for TCP sockets, not Unix domain.

Any tips?
robert
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: .title() - annoying mistake

[Robert Latest via Python-list]

Grant Edwards wrote:
> On 2021-03-20, Robert Latest via Python-list  wrote:
>> Mats Wichmann wrote:
>>> The problem is that there isn't a standard for title case,
>>
>> The problem is that we owe the very existence of the .title() method to too
>> much weed being smoked during Python development. It makes specific
>> assumptions about a specific use case of one specific language. It doesn't
>> get more idiotic, frankly.
>
> Ah, you've never used PHP then?
>
> I haven't checked but it's a fair bit that PHP has 3-4 different built-in
> ways to do it, and they're all broken in interestingly unpredictable ways.

I believe that 100%. PHP is the reason I switched to Python/WSGI, and I'm
loving it. 

robert
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: .title() - annoying mistake

[Robert Latest via Python-list]

Benjamin Schollnick wrote:
>
>> I agree with everything you say. Especially the open source part. But
>> wouldn't you agree that .title() with all its arbitrary specificity to
>> appear in the very core of a general purpose language is quite an oddity?
>
> No, because it book ends the issue.
>
> Upper - Converts everything to uppercase Lower - Converts everything to
> lowercase
>
> Title - Covers the cases in-between upper/lower.  

My only issue is that I completely fail to see how this function would be
useful enough to warrant the inclusion into the *core* of a general-purpose
language, including its misleading name. The fact that the function's behavior
is correctly documented doesn't make its very existence less bewildering to me.
Consider this function:

def add_seventeen(n): '''Return n with 16.8 added''' return n + 16.8

It's like .title(): It does almost the thing its name suggests, it is correctly
documented, it is useful to anybody who happens to want 16.8 added to numbers,
and it might erroneously be used by someone who wants exactly 17 added and
didn't bother to read the docs.

> And as I mentioned the sheer amount of work that would be needed would
> probably cover a phd dissertation or more…  It’s a huge problem set to
> respect one language, let alone all languages.  

And that's why I believe that such a function should be delegated to a natural
language-processing library and not the core of Python.

> So the only answer is to not respect the languages, and leave that up to a
> later implementation or for someone else to assist in adding in support.

And that too.

> Heck, how do we prevent it from titlecasing abbreviations?  (This is plain
> text not XML….  If it was XML it would be easy!)

And that too.

BTW I have no beef with the person who invented .title() nor with anybody who
uses it. I know that everybody can join the Python development community and
propose the removal of .title() and the inclusion of add_seventeen(). That
said, I doubt that .title() would make it into Python today if it weren't there
already. I'm having fun with this.

robert
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: .title() - annoying mistake

[Robert Latest via Python-list]

Chris Angelico wrote:
> On Sun, Mar 21, 2021 at 10:31 PM Robert Latest via Python-list
> wrote:
>> Yes, I get that. But the purpose it (improperly) serves only makes sense in
>> the English language.
>
> Why? Do titles not exist in other languages? Does no other language
> capitalize words in book or other titles?

I wonder if .title() properly capitalizes titles in any language. It doesn't in
English (nor does it purport to), so it begs the question why it is there in
the first place. German and Spanish don't have any special capitalization rules
for titles; I don't know about any other languages. 

robert
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: .title() - annoying mistake

[Robert Latest via Python-list]

Benjamin Schollnick wrote:
>
> I’m sorry Robert, but just because it doesn’t meet your requirements, doesn’t
> mean it’s useless.
>
> I use .title to normalize strings for data comparison, all the time.  It’s a
> perfect alternative to using .UPPER or .lower.  
>
> Right in the documentation, it specifically acknowledges .title working in
> foreign character sets. 
>
> So while I feel for the fact that it doesn’t met your requirements, please
> keep in mind, it does meet other peoples requirements.  
>
> As with everything here, it’s open source.  If you feel that there should be
> a revised version that does met your requirements create it, or gather a
> bunch of people and go the route of SCANDIR and open-source it, and petition
> that it be moved into the standard library.
>
> Since this seems to be bugging you this much, come up with a solution.  

I agree with everything you say. Especially the open source part. But wouldn't
you agree that .title() with all its arbitrary specificity to appear in the
very core of a general purpose language is quite an oddity?

robert
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: .title() - annoying mistake

[Robert Latest via Python-list]

Chris Angelico wrote:
> On Sun, Mar 21, 2021 at 4:31 AM Robert Latest via Python-list
> wrote:
>>
>> Mats Wichmann wrote:
>> > The problem is that there isn't a standard for title case,
>>
>> The problem is that we owe the very existence of the .title() method to too
>> much weed being smoked during Python development. It makes specific
>> assumptions about a specific use case of one specific language. It doesn't
>> get more idiotic, frankly.
>>
>
> The problem is that you haven't read the documentation :) It very carefully
> does NOT define itself by language, and its behaviour is identical regardless
> of the language used.

The documentation says: "The algorithm uses a simple language-independent
definition of a word as groups of consecutive letters..."

Yes, I get that. But the purpose it (improperly) serves only makes sense in the
English language. Which is also the reason they called it title() and not
capitalize_words(). Frankly, I can't think of any situation where this method
would have any use -- in any language, including English. It is just a
completely arbitrary feature, as would be a function that capitalizes only the
last letter of each word.

robert
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: .title() - annoying mistake

[Robert Latest via Python-list]

Mats Wichmann wrote:
> The problem is that there isn't a standard for title case,

The problem is that we owe the very existence of the .title() method to too
much weed being smoked during Python development. It makes specific assumptions
about a specific use case of one specific language. It doesn't get more
idiotic, frankly.

robert
-- 
https://mail.python.org/mailman/listinfo/python-list

[SQLAlchemy] Struggling with association_proxy

[Robert Latest via Python-list]

I'm trying to implement a many-to-many relationship that associates Baskets
with Items via an association object called Link which holds the quantity of
each item. I've done that in SQLAlchemy in a very pedestrian way, such as when
I want to have six eggs in a basket:

1. Find ID of Item with name 'egg'
2. See if there is an association object with the egg ID and the basket ID
3a. if yes, set its quantity to 6
3b if no, create it with quantity 6 and add it to the items colletion in basket

The association_proxy documentation suggests that this could be done elegantly
in such a way that I could simply write

basket.contents['egg'] = 6

and be done with it. I've tried to follow the documentation at 
https://docs.sqlalchemy.org/en/14/orm/extensions/associationproxy.html
but I don't understand it: It keeps creating new keyword instances rather
re-using existing ones, thus defeating the many-to-many idea. Here's what I've
come up so far, but it predictably fails because I don't want it to create new
Items on its own:

from sqlalchemy import create_engine, Column, Integer,\
String, ForeignKey
from sqlalchemy.ext.associationproxy import association_proxy
from sqlalchemy.orm import relationship, sessionmaker, backref
from sqlalchemy.orm.collections import attribute_mapped_collection
from sqlalchemy.ext.declarative import declarative_base

Base = declarative_base()

class Basket(Base):
__tablename__ = 'basket'
id= Column(Integer, primary_key=True)
contents = association_proxy('basket_contents', 'id')

class Link(Base):
__tablename__ = 'link'
item_id   = Column(ForeignKey('item.id'), primary_key=True)
basket_id = Column(ForeignKey('basket.id'), primary_key=True)
quantity  = Column(Integer)
basket = relationship(Basket, backref=backref('basket_contents',
collection_class=attribute_mapped_collection('quantity')))
item = relationship('Item')
name = association_proxy('item', 'name')

def __init__(self, name, quantity):
# this doesn't work b/c it calls Item.__init__() rather than
# looking for an existing Item
self.name = name
self.quantity = quantity

class Item(Base):
__tablename__ = 'item'
id= Column(Integer, primary_key=True)
name  = Column(String(10), unique=True)
weight= Column(Integer)
color = String(10)

engine = create_engine('sqlite://')
Base.metadata.create_all(engine)
Session = sessionmaker(engine)

db = Session()

egg = Item(name='egg', weight=50, color='white')

b = Basket()

# fails because in Link.__init__(), SQLAlchemy wants to create a new Item
# rather than using the existing one.
b.contents['egg'] = 6

db.add(b)

db.commit()

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: A 35mm film camera represented in Python object

[Robert Latest via Python-list]

D.M. Procida wrote:
> Hi everyone, I've created  -
> a representation of a Canonet G-III QL17 in Python.

[...]

> The Canonet G-III QL17 is one of my favourites. One of my reasons for
> writing this code is to appreciate the intricate mechanical logic
> embodied in the machine.

I love both photography with mechanical camears (The Nikon FE2 being my
favorite) and programming. So I absolutely love your project. Also I think its
totally nuts. I won't spend a second looking at your code or get otherwise
involved, but I wish you best of luck!

robert
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to implement logging for an imported module?

[Robert Latest via Python-list]

Richard Damon wrote:
> On 3/8/21 4:16 AM, Robert Latest via Python-list wrote:
>> Joseph L. Casale wrote:
>>>> I couldn't find any information on how to implement logging in a library
>>>> that doesn't know the name of the application that uses it. How is that
>>>> done?
>>> That's not how it works, it is the opposite. You need to know the name of
>>> its logger, and since you imported it, you do.
>> [much snipping]
>>
>>> Last word of advice, don't fight it by hacking up or patching (somehow?),
>>> it will simply not work right for any other case even slightly different
>>> than the one you somehow beat into submission.
>> I didn't waht to hack the logging system, it's just that I wasn't sure of
>> its design principles. I had hoped that if I set up a logger (including
>> levels and formatter) in my main app, the loggers in the imported modules
>> would somwhow automagically follow suit. Now I understand that for each
>> imported module I must import its logger, too, and decide how to deal with
>> its messages.
>>
>>
> Each instance of the logger inherents from a 'parent' logger, except for the
> top level logger which has the name None (as in the singleton of NoneType),
> and unless told otherwise will inherit it properties from its parent.
>
> Thus, if you get the root logger with logging.getLogger() you can set
> properties there, and unless a child logger has specifical been told not to
> inherit or has been specifically given a different value.
>
> General convention is that modules will use their name as the name of their
> logger, as that is generally unique.
>

I must admit I'm still struggling with the very basics of logging. I don't
understand the behavior of the code samples below at all, see comments.
It seems that logging.debug() et al have some side effects that are required
to get a logger to notice its level in the first place.


# Example 1:
# Why does the logger "mylog" require
# a call to the root logger in order to work?

import logging

mylog = logging.getLogger('foo')
mylog.setLevel(logging.DEBUG)

mylog.debug('1 mylog.debug()')   # prints nothing
logging.debug('2 logging.debug()') # prints nothing
mylog.debug('3 mylog.debug()')   # works


# Example 2:
# Why do I have to call 'logging.debug' before the root
# logger works?

import logging

mylog = logging.getLogger() # now mylog is the root logger
mylog.setLevel(logging.DEBUG) # setting level of root logger

mylog.debug('1 mylog.debug()')   # prints nothing, why?
logging.debug('2 logging.debug()') # works
mylog.debug('3 mylog.debug()')   # works

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Why assert is not a function?

[Robert Latest via Python-list]

Chris Angelico (and oters) wrote:

[interesting stuff]

I'm a late contributor here, but I'd just say: If you'd prefer a function for
assert, just define a function that does what you want and be done with it.
Much harder to do if assert() were a built-in function but you'd rather have a
keyword ;-)

robert
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to implement logging for an imported module?

[Robert Latest via Python-list]

Joseph L. Casale wrote:
>> I couldn't find any information on how to implement logging in a library that
>> doesn't know the name of the application that uses it. How is that done?
>
> That's not how it works, it is the opposite. You need to know the name of its
> logger, and since you imported it, you do.

[much snipping]

> Last word of advice, don't fight it by hacking up or patching (somehow?), it
> will simply not work right for any other case even slightly different than
> the one you somehow beat into submission.

I didn't waht to hack the logging system, it's just that I wasn't sure of its
design principles. I had hoped that if I set up a logger (including levels and
formatter) in my main app, the loggers in the imported modules would somwhow
automagically follow suit. Now I understand that for each imported module I
must import its logger, too, and decide how to deal with its messages.

> I hope that helps,

Much appreciated,
robert
-- 
https://mail.python.org/mailman/listinfo/python-list

How to implement logging for an imported module?

[Robert Latest via Python-list]

Hello,

I'm trying to add logging to a module that gets imported by another module. But
I only get it to work right if the imported module knows the name of the
importing module. The example given in the "Logging Cookbook" also rely on this
fact.

https://docs.python.org/3/howto/logging-cookbook.html#logging-cookbook

I couldn't find any information on how to implement logging in a library that
doesn't know the name of the application that uses it. How is that done?

Here's some example code consisting of the main module foo.py and imported
modules bar.py and baz.py


### foo.py
import logging

import bar, baz

formatter = logging.Formatter(
'%(asctime)s - %(name)s - %(levelname)s - %(message)s')

ch = logging.StreamHandler()
ch.setFormatter(formatter)

logger = logging.getLogger('foo')
logger.addHandler(ch)
logger.setLevel(logging.DEBUG)

logger.debug('debug from <%s>', __name__)
logger.info('info from <%s>', __name__)
logger.warning('warning from <%s>', __name__)
logger.error('error from <%s>', __name__)

bar.func()
baz.func()

### bar.py
'''This "generic" approach doesn't honor loglevel or formats
when imported by another module'''
import logging

l = logging.getLogger(__name__)
def func():
l.debug('debug from <%s>', __name__)
l.info('info from <%s>', __name__)
l.warning('warning from <%s>', __name__)
l.error('error from <%s>', __name__)

### baz.py
'''This only works if the importing module is named 'foo', which
precludes its use as a library module'''
import logging

l = logging.getLogger('foo.baz')
def func():
l.debug('debug from <%s>', __name__)
l.info('info from <%s>', __name__)
l.warning('warning from <%s>', __name__)
l.error('error from <%s>', __name__)

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python2.7 unicode conundrum

[Robert Latest via Python-list]

Richard Damon wrote:
> Why do you say it has been convert to 'Latin'. The string prints as
> being Unicode. Internally Python doesn't store strings as UTF-8, but as
> plain Unicode (UCS-2 or UCS-4 as needed), and code-point E4 is the
> character you want.

You're right, this wasn't the minimal example for my problem after all.
Turns out that the actual issue is somewhere between SQLAlchemy and
MySQL. I took a more specific questioon overt to stackoverflow.com

Thanks
robert
-- 
https://mail.python.org/mailman/listinfo/python-list

Python2.7 unicode conundrum

[Robert Latest via Python-list]

Hi folks,
what semmingly started out as a weird database character encoding mix-up
could be boiled down to a few lines of pure Python. The source-code
below is real utf8 (as evidenced by the UTF code point 'c3 a4' in the
third line of the hexdump). When just printed, the string "s" is
displayed correctly as 'ä' (a umlaut), but the string representation
shows that it seems to have been converted to latin-1 'e4' somewhere on
the way.
How can this be avoided?

dh@jenna:~/python$ cat unicode.py
# -*- encoding: utf8 -*-

s = u'ä'

print(s)
print((s, ))

dh@jenna:~/python$ hd unicode.py 
  23 20 2d 2a 2d 20 65 6e  63 6f 64 69 6e 67 3a 20  |# -*- encoding: |
0010  75 74 66 38 20 2d 2a 2d  0a 0a 73 20 3d 20 75 27  |utf8 -*-..s = u'|
0020  c3 a4 27 0a 0a 70 72 69  6e 74 28 73 29 0a 70 72  |..'..print(s).pr|
0030  69 6e 74 28 28 73 2c 20  29 29 0a 0a  |int((s,))..|
003c
dh@jenna:~/python$ python unicode.py
ä
(u'\xe4',)
dh@jenna:~/python$



-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Package directory question

[Robert Latest via Python-list]

Ben Finney wrote:
> Robert Latest via Python-list  writes:
>
>> Because the main.py script needs to import the tables.py module from
>> backend, I put this at the top if main.py:
>>
>>sys.path.append('../..')
>>import jobwatch.backend.tables as tables
>>
>> My question is: Is this the way it should be done? It looks fishy. The
>> only alternative I could come up with is to put a symlink to tables.py
>> into the frontend directory, which also seems fishy.
>
> Your fish-sense is working correctly. Both of those are hard-coding the
> path, when the Python import mechanism is designed so you don't do that.

[...]

> * To install for use while also developing, add the ‘--editable’ option.

Ah, that's what I needed. Of course the problem I had was only present
during development. I haven't really looked into pip yet, so far I've
been using only "python setup.py install".

Thanks,
robert
-- 
https://mail.python.org/mailman/listinfo/python-list

Package directory question

[Robert Latest via Python-list]

Hello,

I'm building an application which consists of two largely distinct
parts, a frontend and a backend. The directory layout is like this:

|-- jobwatch
|   |-- backend
|   |   |-- backend.py
|   |   |-- __init__.py
|   |   `-- tables.py
|   |-- frontend
|   |   |-- __init__.py
|   |   |-- main.py
|   `-- __init__.py
`-- setup.py

Because the main.py script needs to import the tables.py module from
backend, I put this at the top if main.py:

   sys.path.append('../..')
   import jobwatch.backend.tables as tables

My question is: Is this the way it should be done? It looks fishy. The
only alternative I could come up with is to put a symlink to tables.py
into the frontend directory, which also seems fishy. Eventually I want
to package all this up neatly to be able to use only little wrapper
scripts for the backend (running as a service) and the frontend (a wsgi
app).

Any thoughts?

Thanks
robert
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Weird side effect of default parameter

[Robert Latest via Python-list]

Steven D'Aprano wrote:
> Python function default values use *early binding*: the default parameter 
> is evaluated, ONCE, when the function is defined, and that value is used 
> each time it is needed.

Thanks, "early binding" was the clue I was missing.

robert
-- 
https://mail.python.org/mailman/listinfo/python-list

Weird side effect of default parameter

[Robert Latest via Python-list]

Hello,

I don't understand the behavior of the code below. Why does the dict property
"a" of both objects contain the same keys? This is only if "a=dict" is in
the initializer. If I put self.a = dict() into the init function, I get two
separate dicts



class Foo(object):
def __init__(self, x, a=dict()):
self.x = x
self.a = a
self.a[x] = x


c = Foo(1)
d = Foo(2)

print(c.__dict__)
print(d.__dict__)


robert
-- 
https://mail.python.org/mailman/listinfo/python-list

Why doesn't this method have access to its "self" argument?

[Robert Latest via Python-list]

Hi all,

I'm still trying to find a simple way to implement a method (but not the
full class) in C. Suggestions I have recived so far are good, but seem to be
over the top for such a (seemingly) simple problem. I came up with the idea
of implementing the class in Python, and just setting the method after the
actual class definition. See example below. The "py_method" is added to the
class later on, and it seems to become a proper method of the class (and its
instances) just as if it had been defined inside the class itself. Of course
it also has access to its "self" argument.

If I do the same thing with a method implemented in the external module
"cmethod", and I call that method on an instance, it gets passed a NULL
pointer as its "self" argument and so has no access to the instance's "data"
attribute.

I found a workaround using a wrapper method which calls a C function,
passing the instance as a separate argument. It works, and I cannot see any
disadvantage. It's just not as elegant as I'd like it to be, and I don't
understand WHY the C "method" doesn't receive a pointer to the Python
instance. Maybe somebody can clarify.


Here's what happens when I build, install and run the minimal example below:

rl@dc:~/c/cwsf/python_module$ python test.py
py_method(): <__main__.Test instance at 0xb71f5a2c>
c_method(): self at (nil)
c_function(): self at (nil), instance at 0xb71f5a2c
<__main__.Test instance at 0xb71f5a2c>
'Some data'
c_function(): self at (nil), instance at 0xb71f5a2c
<__main__.Test instance at 0xb71f5a2c>
'New Data'
rl@dc:~/c/cwsf/python_module$

Minimal example files:


== test.py ===

import cmethod

class Test():
def __init__(self):
self.data = "Some data"

def wrapper(self):
return cmethod.c_function(self)

def py_method(self):
print "py_method(): %s" % repr(self)

# add methods to the class "after the fact"
Test.py_method = py_method
Test.c_method = cmethod.c_method

foo = Test()
foo.py_method() # works as expected
foo.c_method()  # is passed NULL in its self argument, why?
foo.wrapper()   # works fine, updates "data" attribute
foo.wrapper()   # works fine, sees updated "data" attribute

 cmethod.c ===

#include 
#include 

static PyObject *c_method(PyObject *self, PyObject *args) {
(void) args; /* not used */
fprintf(stderr, "c_method(): self at %p\n",
(void*) self); /* always prints 0x0 */
Py_INCREF(Py_None);
return Py_None;
}

static PyObject *c_function(PyObject *self, PyObject *args) {
PyObject *instance;
PyObject *o;
/* Retrieve instance from wrapper through argument tuple: WORKS */
PyArg_UnpackTuple(args, "c_function", 1, 1, );
fprintf(stderr, "c_function(): self at %p, instance at %p\n",
(void*) self, (void*) instance);
PyObject_Print(instance, stderr, 0); fputc('\n', stderr);
/* Get and set attributes of instance: WORKS */
o = PyObject_GetAttrString(instance, "data");
PyObject_Print(o, stderr, 0); fputc('\n', stderr);
PyObject_SetAttrString(instance, "data",
PyString_FromString("New Data"));
/* Side question: Do I have to DECREF "o"? */
Py_INCREF(Py_None);
return Py_None;
}

static PyMethodDef methods[] =
{
 {"c_method", c_method, METH_VARARGS, "I want to be a class method"},
 {"c_function", c_function, METH_VARARGS, "I am a function"},
 {NULL, NULL, 0, NULL}
};

PyMODINIT_FUNC initcmethod(void)
{
 (void) Py_InitModule("cmethod", methods);
}


 setup_cmethod.py ===

from distutils.core import setup, Extension

module1 = Extension('cmethod', sources = ['cmethod.c'])

setup (name = 'cmethod',
version = '0.1',
description = 'struggling to implement a class method in C',
ext_modules = [module1])


robert
-- 
https://mail.python.org/mailman/listinfo/python-list