Synchronise annotations -> docstring

[Albert-Jan Roskam via Python-list]

   Hi,
   Are there any tools that check whether type annotations and Numpydoc
   strings are consistent?
   I did find this Vim
   plugin: https://lxyuan0420.github.io/posts/til-vim-pydocstring-plugin.
   Looks incredibly useful, but I haven't tried it yet.
   Thanks!
   AJ
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: ListAdmin: Is list/archive working correctly?

[Albert-Jan Roskam via Python-list]

   I also think that list/archive isn't working properly. Very little emails.
   Before, this was quite a busy list.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Error codes

[Albert-Jan Roskam via Python-list]

   On Aug 13, 2024 15:29, Barry Scott via Python-list
wrote:

 > Could not find file 'C:\Users\Charl\OneDrive\Documents\The Sims 4 Mod
 
Constructor\Projects\MetalMummysMods_Ehlers-DanlosMod\Python\__pycache__\MetalMummysMods_Ehlers-DanlosMod.cpython-37.pyc'.
 > Element ID: (No Element)
 > Element Name: (No Element)

   
   Wild guess: do you have sufficicient permissions to write the .pyc files?
   What happens if you run the program with the -B option?
   https://docs.python.org/3/using/cmdline.html#envvar-PYTHONDONTWRITEBYTECODE
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Best use of "open" context manager

[Albert-Jan Roskam via Python-list]

   Or like below, although pylint complains about this: "consider using
   with". Less indentation this way.
   f = None
   try:
   f = open(FILENAME)
   records = f.readlines()
   except Exception:
   sys.exit(1)
   finally:
   if f is not None:
   f.close()
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Suggested python feature: allowing except in context maneger

[Albert-Jan Roskam via Python-list]

 The example exception is not what bothers me. The syntax change is
 nowhere near as useful as `with` and context managers. They provide an
 excellent idiom for resource usage and release.

 Your suggestion complicates the `with` statement and brings only a tiny
 indentation reduction over the `with`-inside-`try` idiom. It brings no
 semantic changes or new features.

   
   I also don't see the added value. If you desperately want to get rid of an
   indentation level, you could use an except
   hook. https://docs.python.org/3/library/sys.html#sys.excepthook
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: pathlib.Path.is_file vs os.path.isfile difference

[Albert-Jan Roskam via Python-list]

   On Mar 10, 2024 12:59, Thomas Passin via Python-list
wrote:

 On 3/10/2024 6:17 AM, Barry wrote:
 >
 >
 >> On 8 Mar 2024, at 23:19, Thomas Passin via Python-list
  wrote:
 >>
 >> We just learned a few posts back that it might be specific to Linux;
 I ran it on Windows.
 >
 > Depending on the exact win32 api used there is a 257 limit on windows.
 > The 257 includes 2 for the device, C:, and 255 for the path part that
 will use 1 for the leading \. Getting an error for a name that is 255 is
 not surprising.
 >
 > Other api allow for 65535 limit, not sure on its additional limits.

 I seem to remember there is a setting to allow longer paths, but I
 forget any details.

   =
   You mean the "\\?\" prefix?
   
https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation?tabs=registry
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: pathlib.Path.is_file vs os.path.isfile difference

[Albert-Jan Roskam via Python-list]

   On Mar 8, 2024 19:35, Thomas Passin via Python-list
wrote:

 On 3/8/2024 1:03 PM, Albert-Jan Roskam via Python-list wrote:
 > Hi,
 > I was replacing some os.path stuff with Pathlib and I discovered
 this:
 > Path(256 * "x").is_file()  # OSError
 > os.path.isfile(256 * "x")  # bool
 > Is this intended? Does pathlib try to resemble os.path as closely
 as
 > possible?

 You must have an very old version of Python.  I'm running 3.12.2 and it
 returns False.  Either that or that path name exists and throws some
 kind of unexpected exception.

   
   Hi, I tested this with Python 3.8. Good to know that this was fixed!
-- 
https://mail.python.org/mailman/listinfo/python-list

pathlib.Path.is_file vs os.path.isfile difference

[Albert-Jan Roskam via Python-list]

   Hi,
   I was replacing some os.path stuff with Pathlib and I discovered this:
   Path(256 * "x").is_file()  # OSError
   os.path.isfile(256 * "x")  # bool
   Is this intended? Does pathlib try to resemble os.path as closely as
   possible?
   Best wishes,
   Albert-Jan
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Postgresql equivalent of Python's timeit?

[Albert-Jan Roskam via Python-list]

   On Sep 15, 2023 19:45, "Peter J. Holzer via Python-list"
wrote:

 On 2023-09-15 17:42:06 +0200, Albert-Jan Roskam via Python-list wrote:
 >    This is more related to Postgresql than to Python, I hope this is
 ok.
 >    I want to measure Postgres queries N times, much like Python timeit
 >    (https://docs.python.org/3/library/timeit.html). I know about
 EXPLAIN
 >    ANALYZE and psql \timing, but there's quite a bit of variation in
 the
 >    times. Is there a timeit-like function in Postgresql?

 Why not simply call it n times from Python?

 (But be aware that calling the same query n times in a row is likely to
 be
 unrealistically fast because most of the data will already be in
 memory.)

   =
   Thanks, I'll give this a shot. Hopefully the caching is not an issue if I
   don't re-use the same database connection.
-- 
https://mail.python.org/mailman/listinfo/python-list

Postgresql equivalent of Python's timeit?

[Albert-Jan Roskam via Python-list]

   Hi,
   This is more related to Postgresql than to Python, I hope this is ok.
   I want to measure Postgres queries N times, much like Python timeit
   (https://docs.python.org/3/library/timeit.html). I know about EXPLAIN
   ANALYZE and psql \timing, but there's quite a bit of variation in the
   times. Is there a timeit-like function in Postgresql?
   Thanks!
   Albert-Jan
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: LRU cache

[Albert-Jan Roskam]

   On Feb 18, 2023 17:28, Rob Cliffe via Python-list 
   wrote:

 On 18/02/2023 15:29, Thomas Passin wrote:
 > On 2/18/2023 5:38 AM, Albert-Jan Roskam wrote:
 >>     I sometimes use this trick, which I learnt from a book by
 Martelli.
 >>     Instead of try/except, membership testing with "in"
 >> (__contains__) might
 >>     be faster. Probably "depends". Matter of measuring.
 >>     def somefunc(arg, _cache={}):
 >>         if len(_cache) > 10 ** 5:
 >>             _cache.pop()
 >>         try:
 >>             return _cache[arg]
 >>         except KeyError:
 >>             result = expensivefunc(arg)
 >>             _cache[arg] = result
 >>             return result
 >>     Albert-Jan
 >
 > _cache.get(arg) should be a little faster and use slightly fewer
 > resources than the try/except.
 >
 Provided that you can provide a default value to get() which will never
 be a genuine "result".

   =
   This might be better than None:
   _cache.get(arg, Ellipsis)
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: LRU cache

[Albert-Jan Roskam]

   I sometimes use this trick, which I learnt from a book by Martelli.
   Instead of try/except, membership testing with "in" (__contains__) might
   be faster. Probably "depends". Matter of measuring.
   def somefunc(arg, _cache={}):
       if len(_cache) > 10 ** 5:
           _cache.pop()
       try:
           return _cache[arg]
       except KeyError:
           result = expensivefunc(arg)
           _cache[arg] = result
           return result
   Albert-Jan
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Fast lookup of bulky "table"

[Albert-Jan Roskam]

   On Jan 15, 2023 05:26, Dino  wrote:

 Hello, I have built a PoC service in Python Flask for my work, and - now
 that the point is made - I need to make it a little more performant (to
 be honest, chances are that someone else will pick up from where I left
 off, and implement the same service from scratch in a different language
 (GoLang? .Net? Java?) but I am digressing).

   ===
   Hi,
   * I'd start by measuring where your program spends its
   time: https://docs.python.org/3/library/profile.html
   * It might be useful try DuckDB instead of
   Sqlite. https://duckdb.org/why_duckdb.html
   Best wishes,
   AJ
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to enter escape character in a positional string argument from the command line?

[Albert-Jan Roskam]

   On Dec 21, 2022 06:01, Chris Angelico  wrote:

 On Wed, 21 Dec 2022 at 15:28, Jach Feng  wrote:
 > That's what I am taking this path under Windows now, the ultimate
 solution before Windows has shell similar to bash:-)

 Technically, Windows DOES have a shell similar to bash. It's called
 bash. :) The trouble is, most people use cmd.exe instead.

   =
   I use Git Bash quite a lot: https://gitforwindows.org/
   Is that the one you're referring to?
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Keeping a list of records with named fields that can be updated

[Albert-Jan Roskam]

   On Dec 15, 2022 10:21, Peter Otten <__pete...@web.de> wrote:

 >>> from collections import namedtuple
 >>> Row = namedtuple("Row", "foo bar baz")
 >>> row = Row(1, 2, 3)
 >>> row._replace(bar=42)
 Row(foo=1, bar=42, baz=3)

   
   Ahh, I always thought these are undocumented methods, but: "In addition to
   the methods inherited from tuples, named tuples support three additional
   methods and two attributes. To prevent conflicts with field names, the
   method and attribute names start with an underscore."
   
https://docs.python.org/3/library/collections.html#collections.somenamedtuple._make
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Yaml.unsafe_load error

[Albert-Jan Roskam]

   On Oct 19, 2022 13:02, Albert-Jan Roskam  wrote:

    Hi,
    I am trying to create a celery.schedules.crontab object from an
 external
    yaml file. I can successfully create an instance from a dummy class
 "Bar",
    but the crontab class seems call __setstate__ prior to __init__. I
 have no
    idea how to solve this. Any ideas? See code below.
    Thanks!
    Albert-Jan

    

     

    # what is the correct way for the next line?

    >>> yaml.unsafe_load('!!python/object:celery.schedules.crontab\n  
 hour:
    3\n   minute: 30')

   

   
   Reading the source a bit more, me thinks it might be:
   yaml.unsafe_load('!!python/object/apply:celery.schedules.crontab\nkwds:\n 
    hour: 3\n   minute: 30')
   I did not yet test this, though.
-- 
https://mail.python.org/mailman/listinfo/python-list

Yaml.unsafe_load error

[Albert-Jan Roskam]

   Hi,
   I am trying to create a celery.schedules.crontab object from an external
   yaml file. I can successfully create an instance from a dummy class "Bar",
   but the crontab class seems call __setstate__ prior to __init__. I have no
   idea how to solve this. Any ideas? See code below.
   Thanks!
   Albert-Jan

   Python 3.6.8 (default, Nov 16 2020, 16:55:22)

   [GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] on linux

   Type "help", "copyright", "credits" or "license" for more information.

   >>> import yaml

   >>> from celery.schedules import crontab

   >>> crontab(hour=3, minute=0)

   

   >>> yaml.unsafe_load('!!python/name:celery.schedules.crontab')

   

   >>> yaml.safe_load('celery.schedules.crontab:\n   hour: 3\n   minute:
   0\n')

   {'celery.schedules.crontab': {'hour': 3, 'minute': 0}}

   >>> class Bar:

   ... def __init__(self, x, y):

   ... pass

   ...

   >>> bar = yaml.unsafe_load('!!python/object:__main__.Bar\n   x: 42\n   y:
   666')

   >>> bar

   <__main__.Bar object at 0x7f43b464bb38>

   >>> bar.x

   42

    

   # what is the correct way for the next line?

   >>> yaml.unsafe_load('!!python/object:celery.schedules.crontab\n   hour:
   3\n   minute: 30')

   Traceback (most recent call last):

     File "", line 1, in 

     File
   
"/home/albertjan@mycompany/envs/myenv/lib64/python3.6/site-packages/yaml/__init__.py",
   line 182, in unsafe_load

       return load(stream, UnsafeLoader)

     File
   
"/home/albertjan@mycompany/envs/myenv/lib64/python3.6/site-packages/yaml/__init__.py",
   line 114, in load

       return loader.get_single_data()

     File
   
"/home/albertjan@mycompany/envs/myenv/lib64/python3.6/site-packages/yaml/constructor.py",
   line 51, in get_single_data

       return self.construct_document(node)

     File
   
"/home/albertjan@mycompany/envs/myenv/lib64/python3.6/site-packages/yaml/constructor.py",
   line 60, in construct_document

       for dummy in generator:

     File
   
"/home/albertjan@mycompany/envs/myenv/lib64/python3.6/site-packages/yaml/constructor.py",
   line 621, in construct_python_object

       self.set_python_instance_state(instance, state)

     File
   
"/home/albertjan@mycompany/envs/myenv/lib64/python3.6/site-packages/yaml/constructor.py",
   line 727, in set_python_instance_state

       instance, state, unsafe=True)

     File
   
"/home/albertjan@mycompany/envs/myenv/lib64/python3.6/site-packages/yaml/constructor.py",
   line 597, in set_python_instance_state

       instance.__setstate__(state)

     File
   
"/home/albertjan@mycompany/envs/myenv/lib/python3.6/site-packages/celery/schedules.py",
   line 541, in __setstate__

       super().__init__(**state)

   TypeError: __init__() got an unexpected keyword argument 'hour'
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Find the path of a shell command

[Albert-Jan Roskam]

   On Oct 14, 2022 18:19, "Peter J. Holzer"  wrote:

 On 2022-10-14 07:40:14 -0700, Dan Stromberg wrote:
 > Alternatively, you can "ps axfwwe" (on Linux) to see environment
 > variables, and check what the environment of cron (or similar) is.  It
 > is this environment (mostly) that cronjobs will inherit.

 The simplest (and IMHO also most reliable) way to find out the
 environment a cronjob has is to write a cronjob which just dumps the
 environment.

   

   =
   Lately I've been using systemd timers instead of cronjobs. They are easier
   to debug (journalctl) but require a bit more work to write. Systemd is
   available on Fedora & friends and Debian based systems, maybe more. It has
   no builtin MAILTO. I use an OnFailure stanza to send a Slack message with
   curl instead.
   https://www.freedesktop.org/software/systemd/man/systemd.timer.html
   https://unix.stackexchange.com/questions/278564/cron-vs-systemd-timers
-- 
https://mail.python.org/mailman/listinfo/python-list

Book/resource recommendation about Celery?

[Albert-Jan Roskam]

   Hi,
   I'm using Flask + Celery + RabbitMQ. Can anyone recommend a good book or
   other resource about Celery? 
   Thanks!
   Albert-Jan
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Register multiple excepthooks?

[Albert-Jan Roskam]

   On Aug 1, 2022 19:34, Dieter Maurer  wrote:

 Albert-Jan Roskam wrote at 2022-7-31 11:39 +0200:
 >   I have a function init_logging.log_uncaught_errors() that I use for
 >   sys.excepthook. Now I also want to call another function
 (ffi.dlclose())
 >   upon abnormal termination. Is it possible to register multiple
 >   excepthooks, like with atexit.register? Or should I rename/redefine
 >   log_uncaught_errors() so it does both things?

 `sys.excepthook` is a single function (not a list of them).
 This means: at any moment a single `excepthook` is effective.

 If you need a modular design, use a dispatcher function
 as your `excepthook` associated with a registry (e.g. a `list`).
 The dispatcher can then call all registered function.

   
   Thank you both. I'll give this a try. I think it would be nice if the
   standard library function atexit.register would be improved, such that the
   registered functions would not only be called upon (a) normal program
   termination, but that one could also register functions that are called
   (b) upon error (c) unconditionally. Much like (a) try - (b) except - (c)
   finally.
-- 
https://mail.python.org/mailman/listinfo/python-list

Register multiple excepthooks?

[Albert-Jan Roskam]

   Hi,
   I have a function init_logging.log_uncaught_errors() that I use for
   sys.excepthook. Now I also want to call another function (ffi.dlclose())
   upon abnormal termination. Is it possible to register multiple
   excepthooks, like with atexit.register? Or should I rename/redefine
   log_uncaught_errors() so it does both things?
   Thanks!
   Albert-Jan
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Tuple unpacking inside lambda expressions

[Albert-Jan Roskam]

   On Apr 20, 2022 13:01, Sam Ezeh  wrote:

 I went back to the code recently and I remembered what the problem was.

 I was using multiprocessing.Pool.pmap which takes a callable (the
 lambda here) so I wasn't able to use comprehensions or starmap

 Is there anything for situations like these?

   =
   Could it simply be:
   multiprocessing.Pool.pmap(lambda job: result.process(*job), jobs)
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: flask app convert sql query to python plotly.

[Albert-Jan Roskam]

   On Apr 2, 2022 20:50, Abdellah ALAOUI ISMAILI 
   wrote:

 i would like to convert in my flask app an SQL query to an plotly pie
 chart using pandas. this is my code :

 def query_tickets_status() :
     query_result = pd.read_sql ("""
     SELECT COUNT(*)count_status, status
     FROM tickets
     GROUP BY status""", con = mydc_db)
     return query_result

 labels_statut = query_tickets_status['status']
 values_statut = query_tickets_status['count_status']

   ==
   I think you mean:
   labels_statut = query_tickets_status()['status']
   values_statut = query_tickets_status()['count_status']
-- 
https://mail.python.org/mailman/listinfo/python-list

Fwd: dict.get_deep()

[Albert-Jan Roskam]

   -- Forwarded message --
   From: Marco Sulla 
   Date: Apr 2, 2022 22:44
   Subject: dict.get_deep()
   To: Python List <>
   Cc:

 A proposal. Very often dict are used as a deeply nested carrier of
 data, usually decoded from JSON. 

 data["users"][0]["address"]["street"]

   I have been using jsonpath expressions for that kind of stuff lately.
   Works really well.
   https://pypi.org/project/jsonpath-ng/
-- 
https://mail.python.org/mailman/listinfo/python-list

Marshmallow: json-to-schema helper?

[Albert-Jan Roskam]

   Hi,
   I'm looking for a convenience function to convert a Marshmallow schema
   into a valid Python class definition. That is, I want to generate python
   code (class MySchema.. etc) that I could write to a .py file. Does this
   exist? I tried the code below, but that is not the intended use of
   marshmallow.Schema.from_dict or inspect.getsource.
   from inspect import getsource
   from marshmallow import Schema
   d = dict(something='potentially', very='complicated')
   schema = Schema.from_dict(d)
   python_class_def_as_str = getsource(schema())  # OSError
   
https://marshmallow.readthedocs.io/en/stable/api_reference.html#marshmallow.Schema.from_dict
   https://docs.python.org/3/library/inspect.html#inspect.getsource
   Thanks!
   Albert-Jan
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: SQLAlchemy: JSON vs. PickleType vs. raw string for serialised data

[Albert-Jan Roskam]

   On Feb 28, 2022 10:11, Loris Bennett  wrote:

 Hi,

 I have an SQLAlchemy class for an event:

   class UserEvent(Base):
   __tablename__ = "user_events"

   id = Column('id', Integer, primary_key=True)
   date = Column('date', Date, nullable=False)
   uid = Column('gid', String(64), ForeignKey('users.uid'),
 nullable=False)
   info = ??

 The event may have arbitrary, but dict-like data associated with it,
 which I want to add in the field 'info'.  This data never needs to be
 modified, once the event has been inserted into the DB.

 What type should the info field have?  JSON, PickleType, String, or
 something else?

 I couldn't find any really reliable sounding information about the
 relative
 pros and cons, apart from a Reddit thread claiming that pickled dicts
 are larger than dicts converted to JSON or String.

 Cheers,

 Loris

   
   I think you need a
   BLOB. 
https://docs.sqlalchemy.org/en/14/core/type_basics.html#sqlalchemy.types.LargeBinary
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: One-liner to merge lists?

[Albert-Jan Roskam]

 If you don't like the idea of 'adding' strings you can 'concat'enate:

 >>> items = [[1,2,3], [4,5], [6]]
 >>> functools.reduce(operator.concat, items)
 [1, 2, 3, 4, 5, 6]
 >>> functools.reduce(operator.iconcat, items, [])
 [1, 2, 3, 4, 5, 6]

 The latter is the functional way to spell your for... extend() loop.
 Don't forget to provide the initial value in that case lest you modify
 the input:

 >> functools.reduce(operator.iconcat, items)  # wrong
 [1, 2, 3, 4, 5, 6]
 >>> items
 [[1, 2, 3, 4, 5, 6], [4, 5], [6]]  # oops

 --
 https://mail.python.org/mailman/listinfo/python-list

   
   Was also thinking about reduce, though this one uses a dunder method:
   from functools import reduce
   d = {1: ['aaa', 'bbb', 'ccc'], 2: ['fff', 'ggg']}
   print(reduce(list.__add__, list(d.values(
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Long running process - how to speed up?

[Albert-Jan Roskam]

   On Feb 19, 2022 12:28, Shaozhong SHI  wrote:

 I have a cvs file of 932956 row and have to have time.sleep in a Python
 script.  It takes a long time to process.

 How can I speed up the processing?  Can I do multi-processing?

   
   Perhaps a dask df: 
   https://docs.dask.org/en/latest/generated/dask.dataframe.read_csv.html
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Error installing requirements

[Albert-Jan Roskam]

   On Feb 18, 2022 08:23, Saruni David  wrote:
   >> Christian Gohlke's site has a Pillow .whl for python
   2.7: https://www.lfd.uci.edu/~gohlke/pythonlibs/#pillow
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Pypy with Cython

[Albert-Jan Roskam]

   On Feb 3, 2022 17:01, Dan Stromberg  wrote:

 > The best answer to "is this slower on

 > Pypy" is probably to measure.
 > Sometimes it makes sense to rewrite C

 > extension modules in pure python for pypy.

   
   Hi Dan, thanks. What profiler do you recommend I normally use cProfile,
   but I was thinking about this
   one: https://pyinstrument.readthedocs.io/en/latest/index.html
-- 
https://mail.python.org/mailman/listinfo/python-list

Pypy with Cython

[Albert-Jan Roskam]

   Hi,
   I inherited a fairly large codebase that I need to port to Python 3. Since
   the program was running quite slow I am also running the unittests against
   pypy3.8. It's a long running program that does lots of pairwise
   comparisons of string values in two files. Some parts of the program (e.g
   a modulo 11 digit check) are implemented in Cython. Should I use pure
   Python instead when using Pypy? I compiled the Cython modules for pypy and
   they work, but I'm afraid they might just slow things down.
   Thanks!
   Albert-Jan
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Waht do you think about my repeated_timer class

[Albert-Jan Roskam]

   On Feb 2, 2022 23:31, Barry  wrote:

 > On 2 Feb 2022, at 21:12, Marco Sulla 
 wrote:
 >
 > You could add a __del__ that calls stop :)

 Didn't python3 make this non deterministic when del is called?

 I thought the recommendation is to not rely on __del__ in python3 code.

   ==>
   Adding __del__ also poses chalenges is you would like to support pypy:
   "There are a few extra implications from the difference in the GC. Most
   notably, if an object has a __del__, the __del__ is never called more than
   once in PyPy; but CPython will call the same __del__ several times if the
   object is resurrected and dies again (at least it is reliably so in older
   CPythons; newer CPythons try to call destructors not more than once, but
   there are counter-examples). The __del__ methods are called in "the right"
   order if they are on objects pointing to each other, as in CPython, but
   unlike CPython, if there is a dead cycle of objects referencing each
   other, their __del__ methods are called anyway; CPython would instead put
   them into the list garbage of the gc module."
   https://doc.pypy.org/en/latest/cpython_differences.html
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Gunicorn - HTTP and HTTPS in the same instance?

[Albert-Jan Roskam]

   I always use NGINX for this. Run Flask/Gunicorn on localhost:5000 and have
   NGINX rewrite https requests to localhost requests. In nginx.conf I
   automatically redirect every http request to https. Static files are
   served by NGINX, not by Gunicorn, which is faster. NGINX also allows you
   to easily set the transfer encoding to gzip
   Albert-Jan
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Call julia from Python: which package?

[Albert-Jan Roskam]

   Hi all,
   Thank you very much for your valuable replies! I will definitely do some
   tracing to see where the bottlenecks really are. It's good to know that
   pypy is still alive and kicking, I thought it was stuck in py2.7. I will
   also write a mini program during the holiday to see how this Julia/Python
   interaction might work. The little bit of experience with Julia more or
   less coincides with what Oscar mentioned: a lot of "warm up" time. This is
   actually a py2.7 project that I inherited. I was asked to convert it to
   py3.8.
   Thanks and merry xmas!
   Albert-Jan
-- 
https://mail.python.org/mailman/listinfo/python-list

Call julia from Python: which package?

[Albert-Jan Roskam]

Hi,

I have a Python program that uses Tkinter for its GUI. It's rather slow so I 
hope to replace many or all of the non-GUI parts by Julia code. Has anybody 
experience with this? Any packages you can recommend? I found three 
alternatives:

* https://pyjulia.readthedocs.io/en/latest/usage.html#
* https://pypi.org/project/juliacall/
* https://github.com/JuliaPy/PyCall.jl

Thanks in advance!

Albert-Jan
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to apply a self defined function in Pandas

[Albert-Jan Roskam]

 > df['URL'] = df.apply(lambda x:  connect(df['URL']), axis=1)

   I think you need axis=0. Or use the Series, df['URL'] =
   df.URL.apply(connect)
-- 
https://mail.python.org/mailman/listinfo/python-list

Ansible, pip and virtualenv

[Albert-Jan Roskam]

   Hi
   I wrote an Ansible .yml to deploy a Flask webapp. I use python 3.6 for the
   ansible-playbook executable. The yml starts with some yum installs,
   amongst which python-pip. That installs an ancient pip version (v9). Then
   I create a virtualenv where I use a requirements.txt for pip install -r.
   Should I precede that step with a separate --upgrade pip step? Or should
   pip just be in my requirements.txt?
   Separate question: what locations do I need to specify in my pip
   --trusted-host list? Could it be that this has recently changed? I
   suddenly got SSL errors.
   Thanks in advance!
   Albert-Jan
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Tracing in a Flask application

[Albert-Jan Roskam]

   Hi,
   logging.basicConfig(level="DEBUG")
   ..in e.g  __init__.py
   AJ
   On 4 Aug 2021 23:26, Javi D R  wrote:

 Hi

 I would like to do some tracing in a flask. I have been able to trace
 request in plain python requests using sys.settrace(), but this doesnt
 work
 with Flask.

 Moreover, what i want to trace is in a flask application, when an
 endpoint
 is called, what was the request, which parts of the code was executed (i
 think i can still do it with settrace) and what is the response sent by
 the
 application

 Can you help me to understand which services i can use to do this?

 Thanks
 --
 https://mail.python.org/mailman/listinfo/python-list
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: argparse support of/by argparse

[Albert-Jan Roskam]

 >>> [1] https://pypi.org/project/clize/

   I use and like docopt (https://github.com/docopt/docopt). Is clize a
   better choice?
-- 
https://mail.python.org/mailman/listinfo/python-list

Async code across Python versions

[Albert-Jan Roskam]

   Hi,
   I just started using async functions and I was wondering if there are any
   best practices to deal with developments in this area for Python 3.6 and
   up. I'm currently using Python 3.6 but I hope I can upgrade to 3.8 soon.
   Do I have to worry that my code might break? If so, is it better to use
   3rd party libraries? It seems that things get a little easier with newer
   Python versions, so it might also a reason to simplify the code.
   Cheers!
   Albert-Jan
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Async requests library with NTLM auth support?

[Albert-Jan Roskam]

   > Asyncio and httpx [1] look promising but don't seem to support ntlm. Any
   tips?
   ==> https://pypi.org/project/httpx-ntlm/
   Not sure how I missed this in the first place. :-)
-- 
https://mail.python.org/mailman/listinfo/python-list

Async requests library with NTLM auth support?

[Albert-Jan Roskam]

   Hi,
   I need to make thousands of requests that require ntlm authentication so I
   was hoping to do them asynchronously. With synchronous requests I use
   requests/requests_ntlm. Asyncio and httpx [1] look promising but don't
   seem to support ntlm. Any tips?
   Cheers!
   Albert-Jan
   [1] https://www.python-httpx.org/
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: .title() - annoying mistake

[Albert-Jan Roskam]

   On 20 Mar 2021 23:47, Cameron Simpson  wrote:

 On 20Mar2021 12:53, Sibylle Koczian  wrote:
 >Am 20.03.2021 um 09:34 schrieb Alan Bawden:
 >>The real reason Python strings support a .title() method is surely
 >>because Unicode supports upper, lower, _and_ title case letters, and
 >>tells you how to map between them. [...]
 >>
 >But that's exactly what he's doing, with a result which is documented,
 >but not really satisfactory.

   
   This would be a good
   start: 
https://apastyle.apa.org/style-grammar-guidelines/capitalization/title-case
   It could be locale-dependent. What I also don't like about .title() is
   that it messes up abbreviations ("Oecd")
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Packaging/MANIFEST.in: Incude All, Exclude .gitignore

[Albert-Jan Roskam]

   you could call a simple bash script in a git hook that syncs your
   MANIFEST.in with your .gitignore. Something like:
   echo -n "exclude " > MANIFEST.in
   cat .gitignore | tr '\n' ' ' >> MANIFEST.in
   echo "graft $(readlink -f ./keep/this)" >> MANIFEST.in
   https://docs.python.org/2/distutils/sourcedist.html#commands
   On 1 Mar 2021 06:05, Abdur-Rahmaan Janhangeer 
   wrote:

 Greetings list,

 SInce i have a .gitignore, how do i exclude
 all files and folders listed by my gitignore?
 How do i include everything by default?

 Kind Regards,

 Abdur-Rahmaan Janhangeer
 about  | blog
 
 github 
 Mauritius
 --
 https://mail.python.org/mailman/listinfo/python-list
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Win32api problems

[Albert-Jan Roskam]

On 22 Oct 2019 11:23, GerritM  wrote:

> ImportError: DLL load failed: The specified > procedure could not be found.

I've had the same error before and I solved it by adding the location where the 
win32 dlls live to PATH. Maybe PATH gets messed up during the installation of 
something.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: python2 vs python3

[Albert-Jan Roskam]

On 18 Oct 2019 20:36, Chris Angelico  wrote:

On Sat, Oct 19, 2019 at 5:29 AM Jagga Soorma  wrote:
>
> Hello,
>
> I am writing my second python script and got it to work using
> python2.x.  However, realized that I should be using python3 and it
> seems to fail with the following message:
>
> --
> Traceback (most recent call last):
>   File "test_script.py", line 29, in 
> test_cmd = ("diskcmd -u " + x + " | grep -v '\*' | awk '{print $1,
> $3, $4, $9, $10}'" )
> TypeError: Can't convert 'bytes' object to str implicitly
> --
>
> I then run this command and save the output like this:
>
> --
> test_info = (subprocess.check_output( test_cmd,
> stderr=subprocess.STDOUT, shell=True )).splitlines()
> --
>
> Looks like the command output is in bytes and I can't simply wrap that
> around str().  Thanks in advance for your help with this.

>That's correct. The output of the command >is, by default, given to you
>in bytes.

Do you happen to know why this is the default?  And is there a reliable way to 
figure out the encoding? On posix, it's probably utf8, but on windows I usually 
use cp437, but knowing windows, it could be any codepage (you can even change 
it with chcp.exe)
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: sqlalchemy & #temp tables

[Albert-Jan Roskam]

On 8 Oct 2019 07:49, Frank Millman  wrote:

On 2019-10-07 5:30 PM, Albert-Jan Roskam wrote:
> Hi,
>
> I am using sqlalchemy (SA) to access a MS SQL Server database (python 3.5, 
> Win 10). I would like to use a temporary table (preferably #local, but 
> ##global would also be an option) to store results of a time-consuming query. 
> In other queries I'd like to access the temporary table again in various 
> places in my Flask app. How do I do that, given that SA closes the connection 
> after each request?
>
> I can do:
> with engine.connect() as con:
>  con.execute('select * into #tmp from tbl')
>  con.execute('select  * from #tmp')
>
> ... but that's limited to the scope of the context manager.
>
> Oh, I don't have rights to create a 'real' table. :-(
>
> Thanks!
>
> Albert-Jan
>


>I do not use SA, but I have written my app to >support Sql Server,
>PostgreSQL and sqlite3 as backend >databases. However, no matter which
>one is in use, I also use sqlite3 as an in->memory database to store
>temporary information.


Hi,

I tried your approach today but I ran into problems due to differences between 
the MS SQL and Sqlite dialect. However, I just came across this page: 
https://docs.sqlalchemy.org/en/13/dialects/sqlite.html#using-temporary-tables-with-sqlite.
 I haven't tried it yey, but using a StaticPool might work.


# maintain the same connection across all threads
from sqlalchemy.pool import StaticPool
engine = create_engine('sqlite:///mydb.db',
poolclass=StaticPool)
-- 
https://mail.python.org/mailman/listinfo/python-list

sqlalchemy & #temp tables

[Albert-Jan Roskam]

Hi,

I am using sqlalchemy (SA) to access a MS SQL Server database (python 3.5, Win 
10). I would like to use a temporary table (preferably #local, but ##global 
would also be an option) to store results of a time-consuming query. In other 
queries I'd like to access the temporary table again in various places in my 
Flask app. How do I do that, given that SA closes the connection after each 
request?

I can do:
with engine.connect() as con:
con.execute('select * into #tmp from tbl')
con.execute('select  * from #tmp')

... but that's limited to the scope of the context manager.

Oh, I don't have rights to create a 'real' table. :-(

Thanks!

Albert-Jan

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Funny code

[Albert-Jan Roskam]

On 26 Sep 2019 10:28, Christian Gollwitzer  wrote:

Am 26.09.19 um 08:34 schrieb ast:
> Hello
>
> A line of code which produce itself when executed
>
>  >>> s='s=%r;print(s%%s)';print(s%s)
> s='s=%r;print(s%%s)';print(s%s)
>
> Thats funny !

==> Also impressive, a 128-language quine:
https://github.com/mame/quine-relay

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: [Tutor] Most efficient way to replace ", " with "." in a array and/or dataframe

[Albert-Jan Roskam]

On 22 Sep 2019 04:27, Cameron Simpson  wrote:

On 21Sep2019 20:42, Markos  wrote:
>I have a table.csv file with the following structure:
>
>, Polyarene conc ,, mg L-1 ,,,
>Spectrum, Py, Ace, Anth,
>1, "0,456", "0,120", "0,168"
>2, "0,456", "0,040", "0,280"
>3, "0,152", "0,200", "0,280"
>
>I open as dataframe with the command:
>data = pd.read_csv ('table.csv', sep = ',', skiprows = 1)
[...]
>And the data_array variable gets the fields in string format:
>[['0,456' '0,120' '0,168']
[...]

>Please see the documentation for the >read_csv function here:

> https://pandas.pydata.org/pandas

>docs/stable/reference/api/pandas.read_cs> 
>v.html?highlight=read_csv#pandas.read_csv

Do you think it's a deliberate design choice that decimal and thousands where 
used here as params, and not a 'locale' param? It seems nice to be able to 
specify e.g. locale='dutch' and then all the right lc_numeric, lc_monetary, 
lc_time where used. Or even locale='nl_NL.1252' and you also wouldn't need 
'encoding' as a separate param. Or might that be bad on windows where there's 
no locale-gen? Just wondering...

Albert-Jan

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Document Entire Apps

[Albert-Jan Roskam]

On 15 Sep 2019 07:00, Sinardy Gmail  wrote:

I understand that we can use pydoc to document procedures how about the 
relationship between packages and dependencies ?



==》 Check out snakefood to generate dependency graphs: 
http://furius.ca/snakefood/. Also, did you discover sphinx already?

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Creating time stamps

[Albert-Jan Roskam]

On 22 Jul 2019 23:12, Skip Montanaro  wrote:

Assuming you're using Python 3, why not use an f-string?

>>> dt = datetime.datetime.now()
>>> dt.strftime("%Y-%m-%d %H:%M")
'2019-07-22 16:10'
>>> f"{dt:%Y-%m-%d %H:%M}"
'2019-07-22 16:10'

===》》 Or if you're running < Python 3.6 (no f strings): format(datetime.now(), 
"%Y-%m-%d %H:%M")
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Most "pythonic" syntax to use for an API client library

[Albert-Jan Roskam]

On 29 Apr 2019 07:18, DL Neil  wrote:

On 29/04/19 4:52 PM, Chris Angelico wrote:
> On Mon, Apr 29, 2019 at 2:43 PM DL Neil  
> wrote:
>>
>> On 29/04/19 3:55 PM, Chris Angelico wrote:
>>> On Mon, Apr 29, 2019 at 1:43 PM DL Neil  
>>> wrote:
 Well, seeing you ask: a more HTTP-ish approach *might* be:

 api.update.customer( 1, name='Bob' )

 ie
 api.verb.subject( adjectives and adverbs )

 Thus:
 api_label/intro/ID.what_we're_going_to_do.who/what_we'll_do_it_to(
 customerID, support_data)

 Yet, it doesn't really *look right* does it?
 (and someone might complain about mixing the 'different' 
 variable-values...)

>>>
>>> The point here is not to make an HTTP-like interface, but a
>>> Python-like interface :)
>>

I recently used Python properties (getter, setter, deleter) for GET, POST/PUT, 
DELETE, respectively. I didn't need any other http methods.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What Python related git pre-commit hooks are you using?

[Albert-Jan Roskam]

On 18 Nov 2018 20:33, Malcolm Greene  wrote:

>Curious to learn what Python related git >pre-commit hooks people are using? 
>What >hooks have you found useful and which >hooks have you tried

I use Python to reject large commits (pre-commit hook): 
http://code.activestate.com/recipes/578883-git-pre-commit-hook-to-reject-large-files-using-py/

I've also used hooks to trigger a Sphinx doc build, and for tox.
-- 
https://mail.python.org/mailman/listinfo/python-list

[OT] master/slave debate in Python

[Albert-Jan Roskam]

*sigh*. I'm with Hettinger on this.

https://www.theregister.co.uk/2018/09/11/python_purges_master_and_slave_in_political_pogrom/
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: python 3.7 - I try to close the thread without closing the GUI is it possible?

[Albert-Jan Roskam]

 > I try to close the thread without closing the GUI is it possible?


Qthread seems to be worth investigating:
https://medium.com/@webmamoffice/getting-started-gui-s-with-python-pyqt-qthread-class-1b796203c18c
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: user defined modules

[Albert-Jan Roskam]

On 5 Jun 2018 09:32, Steven D'Aprano  
wrote:

On Mon, 04 Jun 2018 20:13:32 -0700, Sharan Basappa wrote:

> Is there a specific location where user defined modules need to be kept?
> If not, do we need to specify search location so that Python interpreter
> can find it?

Python modules used as scripts can be run from anywhere, by pointing the
interpreter at the script:

python /path/to/my/script.py


But Python modules uses as libraries, to be imported by other modules,
have to be on the Python search path. You can add extra  paths to the
Python search path from the shell by setting the environment variable
PYTHONPATH to a colon-separated list of paths. On Linux, I do this in
my .bashrc config file:

export PYTHONPATH="paths:to:add"

In the Python interpreter, you can query and modify the search path by
importing sys and looking at sys.path. (But you should not do so unless
you really know what you are doing.)

The default search path is set by the site module:

https://docs.python.org/3/library/site.html

but again, you should not mess with this unless you know what you are
doing.

There are some per-user directories which are automatically added to the
search path. I can't find the existing documentation for them, but a good
place to start is the PEP that introduced the feature:

https://www.python.org/dev/peps/pep-0370/


Apart from setting the PYTHONPATH environment variable, the best way to
add extra paths to is to install a .pth file.

If you run both Python 2 and 3, than .pth might be a better choice. With 
Pythonpath, you run the risk of e.g. importing python3-incompatible code.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Extract data from multiple text files

[Albert-Jan Roskam]

On May 15, 2018 14:12, mahesh d  wrote:

import glob,os

import errno

path = 'C:/Users/A-7993\Desktop/task11/sample emails/'

files = glob.glob(path)

'''for name in files:

print(str(name))

if name.endswith(".txt"):

   print(name)'''

for file in os.listdir(path):

print(file)

if file.endswith(".txt"):

print(os.path.join(path, file))

print(file)

try:

with open(file) as f:

msg = f.read()

print(msg)

except IOError as exc:

if exc.errno != errno.EISDIR:

raise


In the above program . Getting lot of errors . My intention is read the
list of the text files in a folder . Print them


How can resolve those error
--
https://mail.python.org/mailman/listinfo/python-list

Try:
path = 'C:/Users/A-7993/Desktop/task11/sample emails/*.msg'
msg_files = glob.glob(path)
print(msg_files)
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Extract data

[Albert-Jan Roskam]

On May 15, 2018 08:54, Steven D'Aprano  
wrote:

On Tue, 15 May 2018 11:53:47 +0530, mahesh d wrote:

> Hii.
>
>  I have folder.in that folder some files .txt and some files .msg files.
>  .
> My requirement is reading those file contents . Extract data in that
> files .

Reading .msg can be done with win32com or 
https://github.com/mattgwwalker/msg-extractor

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: The basics of the logging module mystify me

[Albert-Jan Roskam]

On Apr 19, 2018 03:03, Skip Montanaro  wrote:
>
>
> I really don't like the logging module, but it looks like I'm stuck
> with it. Why aren't simple/obvious things either simple or obvious?

Agreed. One thing that, in my opinion, ought to be added to the docs is sample 
code to log uncaught exceptions using an excepthook, with correctly formatted 
traceback.

Another thing I always do is to define a custom log level named 'message', 
which is always logged. Basically it's logging.FATAL + 1, but with a friendlier 
label. Would be a nice enhancement of the logging module IMHO.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: RE newbie question

[Albert-Jan Roskam]

On Apr 18, 2018 21:42, TUA  wrote:
>
> import re
>
> compval = 'A123456_8'
> regex = '[a-zA-Z]\w{0,7}'
>
> if re.match(regex, compval):
>print('Yes')
> else:
>print('No')
>
>
> My intention is to implement a max. length of 8 for an input string. The 
> above works well in all other respects, but does allow for strings that are 
> too long.
>
> What is the proper way to fix this?

Use a $ sign at the end of the regex
-- 
https://mail.python.org/mailman/listinfo/python-list

Flask test generator code review?

[Albert-Jan Roskam]

Hi,

I am writing my first unittests for a Flask app. First modest goal is to test 
whether a selected subset of the templates return the expected status 200. 
I am using a nose test generator in a class for this. Is the code below the 
correct way to do this? And is there a way to dynamically set the docstring of 
test_generator? This would make the nosetests output a bit more understandable. 

Thanks!
Albert-Jan

import os
import sys
from os.path import splitext
from http import HTTPStatus as status

import nose

from MyFabulousApp import app

app.testing = True
template_folder = app.config['TEMPLATE_FOLDER']


class Test_MyFabulousApp_HTTP_Status_OK:

def __init__(self):
self.setup()   # with unittest, setUp is called automatically, but not 
with nose

def setup(self):
self.client = app.test_client()
self.client.post('/login', follow_redirects=True)

def teardown(self):
self.client.post('/logout', follow_redirects=True)

def test_generator(self):
"""Does template return HTTP Status 200?"""
def the_test(self, template):
# the following line throws an error: AttributeError: attribute 
'__doc__' of 'method' objects is not writable
#self.test_generator.__doc__ = 'Does template "%s" return HTTP 
Status 200?' % template
respons = self.client.get('/' + template)
actual = respons.status_code
desired = status.OK.value
assert actual == desired, \
   'Template "%s" returns status code %d' % (template, actual)
templates = [splitext(item)[0] for item in os.listdir(template_folder)]
for template in templates:
yield the_test, self, template


if __name__ == '__main__':
nose.run(defaultTest=__name__, argv=[sys.argv[0], '__main__', 
'--verbosity=2'])
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to write partial of a buffer which was returned from a C function to a file?

[Albert-Jan Roskam]

On Apr 12, 2018 09:39, jf...@ms4.hinet.net wrote:
>
> Chris Angelico於 2018年4月12日星期四 UTC+8下午1時31分35秒寫道：
> > On Thu, Apr 12, 2018 at 2:16 PM,   wrote:
> > > This C function returns a buffer which I declared it as a 
> > > ctypes.c_char_p. The buffer has size 0x1 bytes long and the valid 
> > > data may vary from a few bytes to the whole size.
> > >
> > > In every call I know how much the valid data size is, but I suppose I 
> > > can't use slice to get it because there may be zero byte in it. What to 
> > > do?
> > >
> >
> > You suppose? Or have you tested it?
> >
> > ChrisA
>
> Yes, I had test it once before. Now, I re-do it again to make sure. After a 
> call which returns 3 bytes of data, I use len(buf) to check the length and 
> get the number 24. I can see the first 24 bytes of data by using buf[:30] but 
> buf[24] will cause an "index out of range" error. I don't know how to see 
> what the buf[24] exactly is but I suppose it might be a zero byte.

Aren't you looking for the .value or the .raw property?
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Pandas, create new column if previous column(s) are not in [None, '', np.nan]

[Albert-Jan Roskam]

On Apr 11, 2018 20:52, zljubi...@gmail.com wrote:
>
> I have a dataframe:
>
> import pandas as pd
> import numpy as np
>
> df = pd.DataFrame( { 'A' : ['a', 'b', '', None, np.nan],
>  'B'  : [None, np.nan, 'a', 'b', '']})
>
>   A B
> 0 a  None
> 1 b   NaN
> 2   a
> 3  None b
> 4   NaN
>
>
> I would like to create column C in the following way:
> column C = column B if column B is not in [None, '', np.nan]
> else column A
>
> How to do that?
>
> I tried:
>
> df['C'] = df[['A', 'B']].apply(lambda x: x[1] if x[1] in [None, '', np.nan] 
> else x[0])
>
> but I got all np.nan's.

This is another approach:
https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.replace.html
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Numpy and Terabyte data

[Albert-Jan Roskam]

On Jan 2, 2018 18:27, Rustom Mody  wrote:
>
> Someone who works in hadoop asked me:
>
> If our data is in terabytes can we do statistical (ie numpy pandas etc)
> analysis on it?
>
> I said: No (I dont think so at least!) ie I expect numpy (pandas etc)
> to not work if the data does not fit in memory
>
> Well sure *python* can handle (streams of) terabyte data I guess
> *numpy* cannot
>
> Is there a more sophisticated answer?
>
> ["Terabyte" is a just a figure of speech for "too large for main memory"]

Have a look at Pyspark and pyspark.ml. Pyspark has its own kind of DataFrame. 
Very, very cool stuff.

Dask DataFrames have been mentioned already.

numpy has memmapped arrays: 
https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.memmap.html
-- 
https://mail.python.org/mailman/listinfo/python-list

Book recommendation for Spark/Pyspark?

[Albert-Jan Roskam]

Hi,


Can anybody recommend a good, preferably recent, book about Spark and Pyspark? 
I am using Pyspark now, but I am looking for a book that also gives a thorough 
background about Spark itself. I've been looking around on e.g. Amazon but, as 
the saying goes, one can't judge a book by its cover.


Thanks!


Albert-Jan
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Compare files excel

[Albert-Jan Roskam]

(sorry for top posting)

Try:
df1['difference'] = (df1 == df2).all(axis=1)

From: Python-list  on 
behalf of Smith 
Sent: Saturday, July 22, 2017 7:47:59 AM
To: python-list@python.org
Subject: Compare files excel

Hello to all,
I should compare two excel files with pandas.
Who can help me?


Do you have any links?


i tried this, but not working
import pandas as pd
df1 = pd.read_excel('excel1.xlsx')
df2 = pd.read_excel('excel2.xlsx')
difference = df1[df1!=df2]
print (difference)

Thank you
--
https://mail.python.org/mailman/listinfo/python-list
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Test 0 and false since false is 0

[Albert-Jan Roskam]

From: Python-list  on 
behalf of Dan Sommers 
Sent: Friday, July 7, 2017 2:46 AM
To: python-list@python.org
Subject: Re: Test 0 and false since false is 0
    
On Thu, 06 Jul 2017 19:29:00 -0700, Sayth Renshaw wrote:

> I have tried or conditions of v == False etc but then the 0's being
> false also aren't moved. How can you check this at once?

Maybe this will help:

    Python 3.5.3+ (default, Jun  7 2017, 23:23:48) 
    [GCC 6.3.0 20170516] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> False == 0
    True
    >>> False is 0
    False


=> Just wondering: Is this 'is' test depending on an implementation detail 
of cPython (small ints, I forgot how small 0-255 maybe, are  singletons)?
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: memory leak with re.match

[Albert-Jan Roskam]

From: Python-list  on 
behalf of Mayling ge 
Sent: Tuesday, July 4, 2017 9:01 AM
To: python-list
Subject: memory leak with re.match
    
   Hi,

   My function is in the following way to handle file line by line. There are
   multiple error patterns  defined and  need to apply  to each  line. I  use
   multiprocessing.Pool to handle the file in block.

   The memory usage increases to 2G for a 1G file. And stays in 2G even after
   the file processing. File closed in the end.

   If I comment  out the  call to re_pat.match,  memory usage  is normal  and
   keeps under 100Mb.

   am I using re in a wrong way? I cannot figure out a way to fix the  memory
   leak. And I googled .

   def line_match(lines, errors)

  

   lines = list(itertools.islice(fo, line_per_proc))

===> do you really need to listify the iterator?
   if not lines:

   break

   result = p.apply_async(line_match, args=(errors, lines))

===> the signature of line_match is (lines, errors), in args you do (errors, 
lines)

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Combining 2 data series into one

[Albert-Jan Roskam]

Hi,

Does your code run on a sample of the data?
Does your code have categorical data in it? If so: 
https://pandas.pydata.org/pandas-docs/stable/categorical.html. Also, check out 
http://www.pytables.org.

Albert-Jan

From: Python-list  on 
behalf of Bhaskar Dhariyal 
Sent: Thursday, June 29, 2017 4:34:56 AM
To: python-list@python.org
Subject: Re: Combining 2 data series into one

On Wednesday, 28 June 2017 23:43:57 UTC+5:30, Albert-Jan Roskam  wrote:
> (sorry for top posting)
> Yes, I'd try pd.concat([df1, df2]).
> Or this:
> df['both_names'] = df.apply(lambda row: row.name + ' ' + row.surname, axis=1)
> 
> From: Python-list  on 
> behalf of Paul Barry 
> Sent: Wednesday, June 28, 2017 12:30:25 PM
> To: Bhaskar Dhariyal
> Cc: python-list@python.org
> Subject: Re: Combining 2 data series into one
>
> Maybe look at using .concat instead of +
>
> See:
> http://nbviewer.jupyter.org/github/jakevdp/PythonDataScienceHandbook/blob/master/notebooks/03.06-Concat-And-Append.ipynb
>
> On 28 June 2017 at 13:02, Paul Barry  wrote:
>
> >
> > Maybe try your code on a sub-set of your data - perhaps 1000 lines of
> > data? - to see if that works.
> >
> > Anyone else on the list suggest anything to try here?
> >
> > On 28 June 2017 at 12:50, Bhaskar Dhariyal 
> > wrote:
> >
> >> No it didn't work. I am getting memory error. Using 32GB RAM system
> >>
> >> On Wed, Jun 28, 2017 at 5:17 PM, Paul Barry 
> >> wrote:
> >>
> >>> On the line that's failing, your code is this:
> >>>
> >>> combinedX=combinedX+dframe['tf']
> >>>
> >>> which uses combinedX on both sides of the assignment statement - note
> >>> that Python is reporting a 'MemoryError", which may be happening due to
> >>> this "double use" (maybe).  What happens if you create a new dataframe,
> >>> like this:
> >>>
> >>> newX = combinedX + dframe['tf']
> >>>
> >>> Regardless, it looks like you are doing a dataframe merge.  Jake V's
> >>> book has an excellent section on it here: http://nbviewer.jupyter.
> >>> org/github/jakevdp/PythonDataScienceHandbook/blob/master/not
> >>> ebooks/03.07-Merge-and-Join.ipynb - this should take about 20 minutes
> >>> to read, and may be of use to you.
> >>>
> >>> Paul.
> >>>
> >>>
> >>>
> >>> On 28 June 2017 at 12:19, Bhaskar Dhariyal 
> >>> wrote:
> >>>
> >>>> On Wednesday, 28 June 2017 14:43:48 UTC+5:30, Paul Barry  wrote:
> >>>> > This should do it:
> >>>> >
> >>>> > >>> import pandas as pd
> >>>> > >>>
> >>>> > >>> df1 = pd.DataFrame(['bhaskar', 'Rohit'], columns=['first_name'])
> >>>> > >>> df1
> >>>> >   first_name
> >>>> > 0bhaskar
> >>>> > 1  Rohit
> >>>> > >>> df2 = pd.DataFrame(['dhariyal', 'Gavval'], columns=['last_name'])
> >>>> > >>> df2
> >>>> >   last_name
> >>>> > 0  dhariyal
> >>>> > 1Gavval
> >>>> > >>> df = pd.DataFrame()
> >>>> > >>> df['name'] = df1['first_name'] + ' ' + df2['last_name']
> >>>> > >>> df
> >>>> >name
> >>>> > 0  bhaskar dhariyal
> >>>> > 1  Rohit Gavval
> >>>> > >>>
> >>>> >
> >>>> > Again, I draw your attention to Jake VanderPlas's excellent book,
> >>>> which is
> >>>> > available for free on the web.  All of these kind of data
> >>>> manipulations are
> >>>> > covered there:  https://github.com/jakevdp/PythonDataScienceHandbook
> >>>> - the
> >>>> > hard copy is worth owning too (if you plan to do a lot of work using
> >>>> > numpy/pandas).
> >>>> >
> >>>> > I'd also recommend the upcoming 2nd edition of Wes McKinney's "Python
> >>>> for
> >>>> > Data Analysis" book - I've just finished tech reviewing it for
> >>>> O'Reilly,
> >&g

Re: Combining 2 data series into one

[Albert-Jan Roskam]

(sorry for top posting)
Yes, I'd try pd.concat([df1, df2]).
Or this:
df['both_names'] = df.apply(lambda row: row.name + ' ' + row.surname, axis=1)

From: Python-list  on 
behalf of Paul Barry 
Sent: Wednesday, June 28, 2017 12:30:25 PM
To: Bhaskar Dhariyal
Cc: python-list@python.org
Subject: Re: Combining 2 data series into one

Maybe look at using .concat instead of +

See:
http://nbviewer.jupyter.org/github/jakevdp/PythonDataScienceHandbook/blob/master/notebooks/03.06-Concat-And-Append.ipynb

On 28 June 2017 at 13:02, Paul Barry  wrote:

>
> Maybe try your code on a sub-set of your data - perhaps 1000 lines of
> data? - to see if that works.
>
> Anyone else on the list suggest anything to try here?
>
> On 28 June 2017 at 12:50, Bhaskar Dhariyal 
> wrote:
>
>> No it didn't work. I am getting memory error. Using 32GB RAM system
>>
>> On Wed, Jun 28, 2017 at 5:17 PM, Paul Barry 
>> wrote:
>>
>>> On the line that's failing, your code is this:
>>>
>>> combinedX=combinedX+dframe['tf']
>>>
>>> which uses combinedX on both sides of the assignment statement - note
>>> that Python is reporting a 'MemoryError", which may be happening due to
>>> this "double use" (maybe).  What happens if you create a new dataframe,
>>> like this:
>>>
>>> newX = combinedX + dframe['tf']
>>>
>>> Regardless, it looks like you are doing a dataframe merge.  Jake V's
>>> book has an excellent section on it here: http://nbviewer.jupyter.
>>> org/github/jakevdp/PythonDataScienceHandbook/blob/master/not
>>> ebooks/03.07-Merge-and-Join.ipynb - this should take about 20 minutes
>>> to read, and may be of use to you.
>>>
>>> Paul.
>>>
>>>
>>>
>>> On 28 June 2017 at 12:19, Bhaskar Dhariyal 
>>> wrote:
>>>
 On Wednesday, 28 June 2017 14:43:48 UTC+5:30, Paul Barry  wrote:
 > This should do it:
 >
 > >>> import pandas as pd
 > >>>
 > >>> df1 = pd.DataFrame(['bhaskar', 'Rohit'], columns=['first_name'])
 > >>> df1
 >   first_name
 > 0bhaskar
 > 1  Rohit
 > >>> df2 = pd.DataFrame(['dhariyal', 'Gavval'], columns=['last_name'])
 > >>> df2
 >   last_name
 > 0  dhariyal
 > 1Gavval
 > >>> df = pd.DataFrame()
 > >>> df['name'] = df1['first_name'] + ' ' + df2['last_name']
 > >>> df
 >name
 > 0  bhaskar dhariyal
 > 1  Rohit Gavval
 > >>>
 >
 > Again, I draw your attention to Jake VanderPlas's excellent book,
 which is
 > available for free on the web.  All of these kind of data
 manipulations are
 > covered there:  https://github.com/jakevdp/PythonDataScienceHandbook
 - the
 > hard copy is worth owning too (if you plan to do a lot of work using
 > numpy/pandas).
 >
 > I'd also recommend the upcoming 2nd edition of Wes McKinney's "Python
 for
 > Data Analysis" book - I've just finished tech reviewing it for
 O'Reilly,
 > and it is very good, too - highly recommended.
 >
 > Regards.
 >
 > Paul.
 >
 > On 28 June 2017 at 07:11, Bhaskar Dhariyal >>> >
 > wrote:
 >
 > > Hi!
 > >
 > > I have 2 dataframe i.e. df1['first_name'] and df2['last_name']. I
 want to
 > > make it as df['name']. How to do it using pandas dataframe.
 > >
 > > first_name
 > > --
 > > bhaskar
 > > Rohit
 > >
 > >
 > > last_name
 > > ---
 > > dhariyal
 > > Gavval
 > >
 > > should appear as
 > >
 > > name
 > > --
 > > bhaskar dhariyal
 > > Rohit Gavval
 > >
 > >
 > >
 > > Thanks
 > > --
 > > https://mail.python.org/mailman/listinfo/python-list
 > >
 >
 >
 >
 > --
 > Paul Barry, t: @barrypj  - w:
 > http://paulbarry.itcarlow.ie - e: paul.ba...@itcarlow.ie
 > Lecturer, Computer Networking: Institute of Technology, Carlow,
 Ireland.

 https://drive.google.com/open?id=0Bw2Avni0DUa3aFJKdC1Xd2trM2c
 link to code
 --
 https://mail.python.org/mailman/listinfo/python-list

>>>
>>>
>>>
>>> --
>>> Paul Barry, t: @barrypj  - w:
>>> http://paulbarry.itcarlow.ie - e: paul.ba...@itcarlow.ie
>>> Lecturer, Computer Networking: Institute of Technology, Carlow, Ireland.
>>>
>>
>>
>
>
> --
> Paul Barry, t: @barrypj  - w:
> http://paulbarry.itcarlow.ie - e: paul.ba...@itcarlow.ie
> Lecturer, Computer Networking: Institute of Technology, Carlow, Ireland.
>



--
Paul Barry, t: @barrypj  - w:
http://paulbarry.itcarlow.ie - e: paul.ba...@itcarlow.ie
Lecturer, Computer Networking: Institute of Technology, Carlow, Ireland.
--
https://mail.python.org/mailman/listinfo/python-list
-- 
https://mail.python.org/mailman/listinfo/python-list

Fw: Unable to convert pandas object to string

[Albert-Jan Roskam]

From: Albert-Jan Roskam 
Sent: Saturday, June 24, 2017 11:26:26 AM
To: Paul Barry
Subject: Re: Unable to convert pandas object to string

(sorry for top posting)

Try using fillna('') to convert np.nan into empty strings. df['desc'] = 
df.desc.fillna(''). Btw, np.object already is what best approximates str. I 
wish np.object had its own sentinel value for missing data instead of np.nan, 
which is a float.

From: Python-list  on 
behalf of Paul Barry 
Sent: Saturday, June 24, 2017 9:44:54 AM
To: Bhaskar Dhariyal
Cc: python-list@python.org
Subject: Re: Unable to convert pandas object to string

Any chance you could post one line of data so we can see what we have to
work with?

Also - have you taken a look at Jake VanderPlas's notebooks? There's lot of
help with pandas to be found there:
https://github.com/jakevdp/PythonDataScienceHandbook

Paul.

On 24 June 2017 at 10:32, Bhaskar Dhariyal 
wrote:

> 
> Int64Index: 171594 entries, 0 to 63464
> Data columns (total 7 columns):
> project_id  171594 non-null object
> desc171594 non-null object
> goal171594 non-null float64
> keywords171594 non-null object
> diff_creat_laun 171594 non-null int64
> diff_laun_status171594 non-null int64
> diff_status_dead171594 non-null int64
> dtypes: float64(1), int64(3), object(3)
>
> not able to convert desc and keywords to string for preprocessing.
> Tried astype(str). Please help
> --
> https://mail.python.org/mailman/listinfo/python-list
>



--
Paul Barry, t: @barrypj <https://twitter.com/barrypj> - w:
http://paulbarry.itcarlow.ie - e: paul.ba...@itcarlow.ie
Lecturer, Computer Networking: Institute of Technology, Carlow, Ireland.
--
https://mail.python.org/mailman/listinfo/python-list
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to get Buttons to work

[Albert-Jan Roskam]

(sorry for top-posting). UpdateRecords and the other functions need to be 
nested so they fall under your class. Right now they are functions, not methods.

AJ

From: Python-list  on 
behalf of horgan.ant...@gmail.com 
Sent: Saturday, April 22, 2017 12:45:09 PM
To: python-list@python.org
Subject: How to get Buttons to work

I am busy learning Python and I want to make a simple program that connects to 
a database to locate the information of Doctors. Now the as far as I can see 
everything works fine, database connects, info gets displayed, but the buttons 
don't want to work. Please see code and error below.

Any help will be greatly appreciated.

Error:

Traceback (most recent call last):
  File "C:\Users\Bl@h\Desktop\New INF\CallDoctorLocator.py", line 57, in 

myapp = MyForm()
  File "C:\Users\Bl@h\Desktop\New INF\CallDoctorLocator.py", line 29, in 
__init__
QtCore.QObject.connect(self.ui.ButtonUpdate, QtCore.SIGNAL('clicked()' ), 
self.UpdateRecords)
AttributeError: 'MyForm' object has no attribute 'UpdateRecords'
>>>
Code:

#CallDoctorLocator import sys from DoctorLocator import * from PyQt4 import 
QtSql, QtGui

#Create Connection to the Database def createConnection():
db = QtSql.QSqlDatabase.addDatabase('QMYSQL')
db.setHostName('localhost')
db.setDatabaseName('healthcare')
db.setUserName('root')
db.setPassword('3364834')
db.open()
print (db.lastError().text())
return True

class MyForm(QtGui.QDialog):
def __init__(self, parent=None):
QtGui.QWidget.__init__(self, parent)
QtGui.QWidget.__init__(self, parent)
self.ui = Ui_Dialog()
self.ui.setupUi(self)
self.model = QtSql.QSqlTableModel(self)
self.model.setTable("doctors")
self.model.setEditStrategy(QtSql.QSqlTableModel.OnManualSubmit)
self.model.select()
self.ui.tableView.setModel(self.model)
QtCore.QObject.connect(self.ui.ButtonUpdate, QtCore.SIGNAL('clicked()'
), self.UpdateRecords)
QtCore.QObject.connect(self.ui.buttonCancel, QtCore.SIGNAL('clicked()'
), self.CancelChanges)
QtCore.QObject.connect(self.ui.buttonAdd, QtCore.SIGNAL('clicked()' ),
self.AddRecord)
QtCore.QObject.connect(self.ui.buttonDelete, QtCore.SIGNAL('clicked()'
), self.DeleteRecord)
QtCore.QObject.connect(self.ui.buttonSearch, QtCore.SIGNAL('clicked()'
), self.SearchRecords)


def UpdateRecords(self):
self.model.AddRecord()

def CancelChanges(self):
self.model.revertAll()

def UpdateRecords (self):
self.model.insertRow(self.ui.tableView.currentIndex().row())

def DeleteRecords(self):
self.model.removeRow(self.ui.tableView.currentIndex().row())
self.model.AddRecord()

def SearchRecords(self):
self.model.setFilter("Name like '" + self.ui.Name.text()+"%'")

if __name__ == "__main__":
app = QtGui.QApplication(sys.argv)
if not createConnection():
sys.exit(1)
myapp = MyForm()
myapp.show()
sys.exit(app.exec_())
--
https://mail.python.org/mailman/listinfo/python-list
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: OrderedDict with kwds

[Albert-Jan Roskam]

From: eryk sun 
Sent: Saturday, April 22, 2017 7:59 AM
To: Python Main
Cc: Albert-Jan Roskam
Subject: Re: OrderedDict with kwds
    
On Fri, Apr 21, 2017 at 6:08 PM, Albert-Jan Roskam
 wrote:
> Would the insertion order be preserved if the last line were to be
> replaced with:
>
> if kwds:
> for k, v in kwds.items():
> self[k] = v
> if args:
> self.__update(*args)  # no **kwds!

The basic problem is that kwds is a regular, unordered dict:

    def f(**kwds):
    print(type(kwds))

    >>> f()
    


> Hi Eryk,

Yes, I realized this later that evening (probably thanks to can of cold beer 
:-)). But there is hope: https://www.python.org/dev/peps/pep-0468/ . Do you 
know if there is/will be a "from __future__" to backport that behavior? We're 
using Python 3.5 now.

Thank you!

Albert-Jan
-- 
https://mail.python.org/mailman/listinfo/python-list

OrderedDict with kwds

[Albert-Jan Roskam]

For regular dicts I like to use the dict() function because the code is easier 
to write and read. But OrderedDict() is not equivalent to dict():
In the docstring of collections.OrderedDict it says "keyword arguments are not 
recommended because their insertion order is arbitrary" 
(https://github.com/python/cpython/blob/3.6/Lib/collections/__init__.py)

It took me while to realize that. What is the best way to use keywords to 
create an ordered dict, while maintaining insertion order?

Below is OrderedDict.__init__. Would the insertion order be preserved if the 
last line were to be replaced with:

if kwds:
for k, v in kwds.items():
self[k] = v
if args:
self.__update(*args)  # no **kwds!


###
class OrderedDict(dict):
'Dictionary that remembers insertion order'

def __init__(*args, **kwds):
'''Initialize an ordered dictionary.  The signature is the same as
regular dictionaries, but keyword arguments are not recommended because
their insertion order is arbitrary.
'''
if not args:
raise TypeError("descriptor '__init__' of 'OrderedDict' object "
"needs an argument")
self, *args = args
if len(args) > 1:
raise TypeError('expected at most 1 arguments, got %d' % len(args))
try:
self.__root
except AttributeError:
self.__hardroot = _Link()
self.__root = root = _proxy(self.__hardroot)
root.prev = root.next = root
self.__map = {}
self.__update(*args, **kwds)

Thanks!

Albert-Jan
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How coding in Python is bad for you

[Albert-Jan Roskam]

sola dosis facit venenum ~ Paracelsus (1493-1541)

From: Python-list  on 
behalf of alister 
Sent: Monday, January 23, 2017 8:32:49 PM
To: python-list@python.org
Subject: Re: How coding in Python is bad for you

On Tue, 24 Jan 2017 07:19:42 +1100, Chris Angelico wrote:

> On Tue, Jan 24, 2017 at 6:59 AM, Grant Edwards
>  wrote:
>> On 2017-01-23, breamore...@gmail.com  wrote:
>>
>>> The article is here http://lenkaspace.net/index.php/blog/show/111
>>
>> I don't really think any of his points are valid, but one way that
>> programming in Python is bad for you:
>>
>>  * It reduces your tolerance for progamming in PHP zero.  If you end
>>up assigned to a PHP project, you end up miserable and unpleasant to
>>work with or just plain unemployed.
>
> I believe that's "bad for you" in the sense that chocolate is bad for
> you.
>
> It isn't.
>
> ChrisA

chocolate is a poison (lethal dose for a human approx 22lb)



--
Make headway at work.  Continue to let things deteriorate at home.
--
https://mail.python.org/mailman/listinfo/python-list
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Adding colormaps?

[Albert-Jan Roskam]

(sorry for top-posting)
I does not appear to be possible in matplolibrc (1). But you can use 
matplotlib.cm.register_cmap to register new cmaps (2) such as these (3).

(Note: I did not try this)

(1)http://matplotlib.org/1.4.0/users/customizing.html
(2)http://matplotlib.org/api/cm_api.html
(3)https://github.com/BIDS/colormap/blob/master/colormaps.py

From: Python-list  on 
behalf of Martin Schöön 
Sent: Saturday, January 21, 2017 8:42:29 PM
To: python-list@python.org
Subject: Re: Adding colormaps?

Den 2017-01-21 skrev Gilmeh Serda :
> On Wed, 18 Jan 2017 21:41:34 +, Martin Schöön wrote:
>
>> What I would like to do is to add the perceptually uniform sequential
>> colormaps introduced in version 1.5.something. I would like to do this
>> without breaking my Debian system in which Matplotlib version 1.4.2 is
>> the newest version available in the repo.
>
> Haven't checked, but I assume you can get the source. Compile it but
> don't install it and then use the result in virtualenv, maybe?
>
I am hoping for directions to a config file to download and place
somewhere...

/Martin
--
https://mail.python.org/mailman/listinfo/python-list
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Is that forwards first or backwards first? (Re: unintuitive for-loop behavior)

[Albert-Jan Roskam]

(Sorry for top-posting)

Yep, part of the baby's hardware. Also, the interface is not limited to visual 
and auditory information:
http://www.scientificamerican.com/article/pheromones-sex-lives/

From: Python-list  on 
behalf of Steven D'Aprano 
Sent: Tuesday, October 4, 2016 7:00:48 AM
To: python-list@python.org
Subject: Re: Is that forwards first or backwards first? (Re: unintuitive 
for-loop behavior)

On Tuesday 04 October 2016 14:51, Michael Torrie wrote:

> On 10/03/2016 08:21 PM, Steve D'Aprano wrote:
>> On Tue, 4 Oct 2016 05:48 am, Michael Torrie wrote:
>>
>>> There is that old, but false, saying that the only intuitive interface
>>> is the nipple.  Turns out everything, even that, is learned
>>
>> Citation required.
>
> Sure, just ask a nursing woman.
[...]
> Sucking seems to be instinctive but the actual interface is learned
> (albeit very quickly) by experience.

You say tomahto, I say tomarto. It sounds like we're using different language
to describe the same thing.

Babies do have an instinct to suck if you stick a nipple (or a finger) in their
mouth, or even if you merely touch them on the lips or cheek:

http://www.medicinenet.com/script/main/art.asp?articlekey=5392

https://en.wikipedia.org/wiki/Primitive_reflexes#Rooting_reflex

The rooting instinct, together with the sucking instinct, is present in all
healthy babies. I would expect that only the most severe development
abnormalities would prevent instincts as important for survival as these two.

Between the rooting and sucking instincts, I consider "the only intuitive
interface is the nipple" is correct. However, that doesn't necessarily mean
that babies will suckle well: there are all sorts of reasons why babies don't
breast-feed well, e.g.:

http://www.cyh.com/HealthTopics/HealthTopicDetails.aspx?p=114&np=302&id=1960


> Babies don't just see a nipple and know what it's for.

Of course the instinct isn't *visual* -- babies may not open their eyes for
many minutes after birth (one study found that about 3% of babies hadn't opened
their eyes within 20 minutes, the maximum time allotted) which may be long
after their first suckle.

Nevertheless, there are senses other than sight.



--
Steven
git gets easier once you get the basic idea that branches are homeomorphic
endofunctors mapping submanifolds of a Hilbert space.

--
https://mail.python.org/mailman/listinfo/python-list
-- 
https://mail.python.org/mailman/listinfo/python-list

RE: re.search - Pattern matching review ( Apologies re sending)

[Albert-Jan Roskam]

> Date: Sat, 28 May 2016 23:48:16 +0530
> Subject: re.search - Pattern matching review ( Apologies re sending)
> From: ganesh1...@gmail.com
> To: python-list@python.org
> 
> Dear Python friends,
> 
> I am  on Python 2.7 and Linux . I am trying to extract the address
> "1,5,147456:8192" from the below stdout using re.search
> 
> (Pdb) stdout
> 'linux-host-machine-1: Block Address for 1,5,27320320:8192 (block
> 1,5,147456:8192) --\nlinux-host-machine-1: magic
> 0xdeaff2fe mark_cookie 0x300a\n'
> (Pdb) type(stdout)
> 
> 
> Here is the code I have come up with, this looks buggy please review
> the same and suggest any better ways  to code.
> 
> Could we use splitlines() or re.complie() etc , my intention is to
> match 1,5,147456:8192 and return the same.
> 
> 
> #Sample code
> 
> import re
> import subprocess_run
> 
> def get_block():
> try:
> cmd = "get_block_info -l"
> # stdout is the output retrieved by subprocess.Popen()
> stdout, stderr, exitcode = subprocess_run(cmd)
> search_pat = 'Block Address.* \(block (\d+),(\d+),(\d+):(\d+)'
> matched = re.search(search_pat, stdout)
> block = (int(matched.group(1)),
>int(matched.group(2)),
>int(matched.group(3)),
>int(matched.group(4)),
>   )
Perhaps:map(int,  re.search(search_pat, stdout).groups())
Or re.findall
> except IOError, e:
> logging.warning('Error reading lines from "%s" (%s).'
> % (cmd, e))
> 
> if block is None:
>logging.error("block not found")
>return False
> logging.info("block not found")
> return block
> 
> Regards,
> 
> Ganesh
> -- 
> https://mail.python.org/mailman/listinfo/python-list
  
-- 
https://mail.python.org/mailman/listinfo/python-list

RE: Remove directory tree without following symlinks

[Albert-Jan Roskam]

> From: eryk...@gmail.com
> Date: Sat, 23 Apr 2016 15:22:35 -0500
> Subject: Re: Remove directory tree without following symlinks
> To: python-list@python.org
> 
> On Sat, Apr 23, 2016 at 4:34 AM, Albert-Jan Roskam
>  wrote:
>>
>>> From: eryk...@gmail.com
>>> Date: Fri, 22 Apr 2016 13:28:01 -0500
>>> On Fri, Apr 22, 2016 at 12:39 PM, Albert-Jan Roskam
>>>  wrote:
>>>> FYI, Just today I found out that shutil.rmtree raises a WindowsError if
>>>> the dir is read-only (or its contents). Using 'ignore_errors', won't help.
>>>> Sure, no error is raised, but the dir is not deleted either! A 'force' 
>>>> option
>>>> would be a nice improvement.
>>>
>>> Use the onerror handler to call os.chmod(path, stat.S_IWRITE). For
>>> example, see pip's rmtree_errorhandler:
>>>
>>> https://github.com/pypa/pip/blob/8.1.1/pip/utils/__init__.py#L105
>>
>> Thanks, that looks useful indeed. I thought about os.chmod, but with
>> os.walk. That seemed expensive. So I used subprocess.call('rmdir "%s" /s /q'
>> % dirname). That's Windows only, of course, but aside of that, is using
>> subprocess less preferable?
> 
> I assume you used shell=True in the above call, and not an external
> rmdir.exe. There are security concerns with using the shell if you're
> not in complete control of the command line.
> 
> As to performance, cmd's rmdir wins without question, not only because
> it's implemented in C, but also because it uses the stat data from the
> WIN32_FIND_DATA returned by FindFirstFile/FindNextFile to check for
> FILE_ATTRIBUTE_DIRECTORY and FILE_ATTRIBUTE_READONLY.
> 
> On the other hand, Python wins when it comes to working with deeply
> nested directories. Paths in cmd are limited to MAX_PATH characters.
> rmdir uses DOS 8.3 short names (i.e. cAlternateFileName in
> WIN32_FIND_DATA), but that could still exceed MAX_PATH for a deeply
> nested tree, or the volume may not even have 8.3 DOS filenames.
> shutil.rmtree allows you to work around the DOS limit by prefixing the
> path with "\\?\". For example:
> 
>>>> subprocess.call(r'rmdir /q/s Z:\Temp\long', shell=True)
> The path Z:\Temp\long\aa
> 
> 
> 
> a is too long.
> 0
> 
>>>> shutil.rmtree(r'\\?\Z:\Temp\long')
>>>> os.path.exists(r'Z:\Temp\long')
> False
> 
> Using "\\?\" requires a path that's fully qualified, normalized
> (backslash only), and unicode (i.e. decode a Python 2 str).

Aww, I kinda forgot about that already, but I came across this last year [1]. 
Apparently, 
shutil.rmtree(very_long_path) failed under Win 7, even with the "silly prefix". 
I believe very_long_path was a Python2-str.
It seems useful if shutil or os.path would automatically prefix paths with 
"\\?\". It is rarely really needed, though.
(in my case it was needed to copy a bunch of MS Outlook .msg files, which 
automatically get the subject line as the filename, and perhaps
the first sentence of the mail of the mail has no subject).

[1] https://mail.python.org/pipermail/python-list/2015-June/693156.html

  
-- 
https://mail.python.org/mailman/listinfo/python-list

RE: Remove directory tree without following symlinks

[Albert-Jan Roskam]

> From: eryk...@gmail.com
> Date: Fri, 22 Apr 2016 13:28:01 -0500
> Subject: Re: Remove directory tree without following symlinks
> To: python-list@python.org
> 
> On Fri, Apr 22, 2016 at 12:39 PM, Albert-Jan Roskam
>  wrote:
> > FYI, Just today I found out that shutil.rmtree raises a WindowsError if the 
> > dir is read-
> > only (or its contents). Using 'ignore_errors', won't help. Sure, no error 
> > is raised, but the
> > dir is not deleted either! A 'force' option would be a nice improvement.
> 
> Use the onerror handler to call os.chmod(path, stat.S_IWRITE). For
> example, see pip's rmtree_errorhandler:
> 
> https://github.com/pypa/pip/blob/8.1.1/pip/utils/__init__.py#L105

Thanks, that looks useful indeed. I thought about os.chmod, but with os.walk. 
That seemed expensive. So I used subprocess.call('rmdir "%s" /s /q' % dirname). 
That's Windows only, of course, but aside of that, is using subprocess less 
preferable?Fun fact: I used it to remove .svn dirs, just like what is mentioned 
in the pip comments :-)

  
-- 
https://mail.python.org/mailman/listinfo/python-list

RE: Remove directory tree without following symlinks

[Albert-Jan Roskam]

> From: st...@pearwood.info
> Subject: Re: Remove directory tree without following symlinks
> Date: Sat, 23 Apr 2016 03:14:12 +1000
> To: python-list@python.org
> 
> On Sat, 23 Apr 2016 01:09 am, Random832 wrote:
> 
> > On Fri, Apr 22, 2016, at 10:56, Steven D'Aprano wrote:
> >> What should I use for "remove_tree"? Do I have to write my own, or does a
> >> solution already exist?
> > 
> > In the os.walk documentation it provides a simple recipe and also
> > mentions shutil.rmtree
> 
> Thanks for that.

FYI, Just today I found out that shutil.rmtree raises a WindowsError if the dir 
is read-only (or its contents). Using 'ignore_errors', won't help. Sure, no 
error is raised, but the dir is not deleted either! A 'force' option would be a 
nice improvement.



> The os.walk recipe is described as a simple version of shutil.rmtree. The
> documentation for rmtree seems lacking to me, but after testing it, it
> appears to work as I want it: it removes symbolic links, it does not follow
> them.
>
> Is anyone else able to confirm that my understanding is correct? If so, the
> documentation should probably be a bit clearer.
> 
> 
> 
> -- 
> Steven
> 
> -- 
> https://mail.python.org/mailman/listinfo/python-list
  
-- 
https://mail.python.org/mailman/listinfo/python-list

RE: read datas from sensors and plotting

[Albert-Jan Roskam]

> From: ran...@nospam.it
> Subject: read datas from sensors and plotting
> Date: Sun, 17 Apr 2016 18:46:25 +0200
> To: python-list@python.org
> 
> I'm reading in python some values from some sensors and I write them in 
> a csv file.
> My problem now is to use this datas to plot a realtime graph for a 
> example in a web server.
> Is it possible to read in the same time the values, writing in the file 
> and plot them in a webpage with python?


tail -F data.log | python myprogram.py
http://stackoverflow.com/questions/1712276/tail-read-a-growing-dynamic-file-and-extract-two-columns-and-then-print-a-graphhttp://code.activestate.com/recipes/577968-log-watcher-tail-f-log/
  
-- 
https://mail.python.org/mailman/listinfo/python-list

RE: extract rar

[Albert-Jan Roskam]

> Date: Fri, 1 Apr 2016 13:22:12 -0600
> Subject: extract rar
> From: fanjianl...@gmail.com
> To: python-list@python.org
> 
> Hello everyone,
> 
> I am wondering is there any way to extract rar files by python without
> WinRAR software?
> 
> I tried Archive() and patool, but seems they required the WinRAR software.

Perhaps 7-zip in a Python 
subprocess:http://superuser.com/questions/458643/unzip-rar-from-command-line-with-7-zip/464128
  
-- 
https://mail.python.org/mailman/listinfo/python-list

RE: A tool to add diagrams to sphinx docs

[Albert-Jan Roskam]

> Subject: Re: A tool to add diagrams to sphinx docs
> From: irmen.nos...@xs4all.nl
> Date: Fri, 1 Apr 2016 18:26:48 +0200
> To: python-list@python.org
> 
> On 1-4-2016 17:59, George Trojan - NOAA Federal wrote:
> > What graphics editor would you recommend to create diagrams that can be
> > included in sphinx made documentation? In the past I used xfig, but was not
> > happy with font quality. My understanding is the diagrams would be saved in
> > a .png file and I should use an image directive in the relevant .rst file.
> > 
> > George
> > 
> 
> I've used .png successfully for a few images in my sphinx docs.
> However if you're concerned with font quality, a better option is perhaps to 
> use a
> vector format like SVG instead. You can create them using a tool like 
> Inkskape or
> Illustrator. I haven't tried to use .svg in sphinx myself yet but I guess it 
> simply
> embeds/links it into the resulting html so a recent web browser should 
> display them just
> fine.
I used .svg for dependency graphs made with snakefood. Looked good in html, 
though they essily get too detailed. The .rst:

.. figure:: images/my_image.svg:width: 100% 
  
-- 
https://mail.python.org/mailman/listinfo/python-list

RE: Effects of caching frequently used objects, was Re: Explaining names vs variables in Python

[Albert-Jan Roskam]

> To: python-list@python.org
> From: __pete...@web.de
> Subject: Effects of caching frequently used objects, was Re: Explaining  
> names  vs variables  in Python
> Date: Wed, 2 Mar 2016 10:12:48 +0100
> 
> Salvatore DI DIO wrote:
> 
> > Hello,
> > 
> > I know Python does not have variables, but names.
> > Multiple names cant then be bound to the same objects.
> > 
> > So this behavior
> > 
>  b = 234
>  v = 234
>  b is v
> > True
> > 
> > according to the above that is ok
> > 
> > 
> > 
> > But where is the consistency ? if I try :
> > 
>  v = 890
>  w = 890
>  v is w
> > False
> > 
> > It is a little difficult to explain this behavior to a newcommer in Python
> > 
> > Can someone give me the right argument to expose ?
> 
> You should not bother with object identity for objects other than None.


A little late to the party, but: how about Ellipsis? Shouldn't "is" also be 
used for that one? (It's rare, I know :))
Albert-Jan

  
-- 
https://mail.python.org/mailman/listinfo/python-list

RE: looping and searching in numpy array

[Albert-Jan Roskam]

> From: sjeik_ap...@hotmail.com
> To: heml...@gmail.com; python-list@python.org
> Subject: RE: looping and searching in numpy array
> Date: Sun, 13 Mar 2016 13:51:23 +



> 
> Hi, I suppose you have seen this already (in particular the first link): 
> http://numpy-discussion.10968.n7.nabble.com/Implementing-a-quot-find-first-quot-style-function-td33085.htmlI
>  don't thonk it's part of numpy yet.
> Albert-Jan

sorry, the correct url is: 
http://numpy-discussion.10968.n7.nabble.com/Implementing-a-quot-find-first-quot-style-function-td33085.html

-- 
https://mail.python.org/mailman/listinfo/python-list

RE: looping and searching in numpy array

[Albert-Jan Roskam]

> Date: Thu, 10 Mar 2016 08:48:48 -0800
> Subject: Re: looping and searching in numpy array
> From: heml...@gmail.com
> To: python-list@python.org
> 
> On Thursday, March 10, 2016 at 2:02:57 PM UTC+1, Peter Otten wrote:
> > Heli wrote:
> > 
> > > Dear all,
> > > 
> > > I need to loop over a numpy array and then do the following search. The
> > > following is taking almost 60(s) for an array (npArray1 and npArray2 in
> > > the example below) with around 300K values.
> > > 
> > > 
> > > for id in np.nditer(npArray1):
> > >   
> > >newId=(np.where(npArray2==id))[0][0]
> > > 
> > > 
> > > Is there anyway I can make the above faster? I need to run the script
> > > above on much bigger arrays (50M). Please note that my two numpy arrays in
> > > the lines above, npArray1 and npArray2  are not necessarily the same size,
> > > but they are both 1d.
> > 
> > You mean you are looking for the index of the first occurence in npArray2 
> > for every value of npArray1?
> > 
> > I don't know how to do this in numpy (I'm not an expert), but even basic 
> > Python might be acceptable:
> > 
> > lookup = {}
> > for i, v in enumerate(npArray2):
> > if v not in lookup:
> > lookup[v] = i
> > 
> > for v in npArray1:
> > print(lookup.get(v, ""))
> > 
> > That way you iterate once (in Python) instead of 2*len(npArray1) times (in 
> > C) over npArray2.
> 
> Dear Peter, 
> 
> Thanks for your reply. This really helped. It reduces the script time from 
> 61(s) to 2(s). 
> 
> I am still very interested in knowing the correct numpy way to do this, but 
> till then your fix works great. 


Hi, I suppose you have seen this already (in particular the first link): 
http://numpy-discussion.10968.n7.nabble.com/Implementing-a-quot-find-first-quot-style-function-td33085.htmlI
 don't thonk it's part of numpy yet.
Albert-Jan
  
-- 
https://mail.python.org/mailman/listinfo/python-list

RE: reversed(zip(...)) not working as intended

[Albert-Jan Roskam]

(Sorry for top-posting)

 No TypeError here:

Python 2.7.2 (default, Nov  2 2015, 01:07:37) [GCC 4.9 20140827 (prerelease)] 
on linux4
Type "help", "copyright", "credits" or "license" for more information.
>>> ten = range(10)
>>> reversed(zip(ten, ten))

>>> list(reversed(zip(ten, ten)))
[(9, 9), (8, 8), (7, 7), (6, 6), (5, 5), (4, 4), (3, 3), (2, 2), (1, 1), (0, 0)]
>>>

> To: python-list@python.org
> From: srku...@mail.de
> Subject: reversed(zip(...)) not working as intended
> Date: Sun, 6 Mar 2016 19:29:59 +0100
> 
> Hi,
> 
> what's the reason that reversed(zip(...)) raises as a TypeError?
> 
> Would allowing reversed to handle zip and related functions lead to 
> strange errors?
> 
> Best,
> Sven
> -- 
> https://mail.python.org/mailman/listinfo/python-list
  
-- 
https://mail.python.org/mailman/listinfo/python-list

RE: SQLite

[Albert-Jan Roskam]

(Sorry for top posting)

IIRC, you have to do
sudo apt-get install build-essential python-dev
... then re-compile python

> To: python-list@python.org
> From: k.d.jant...@mailbox.org
> Subject: SQLite
> Date: Sun, 21 Feb 2016 18:11:18 +0100
> 
>Hello,
> 
>I have downloaded Python3.5.1 as .targz, compiled it(configure, make,...)
>and it works
>(under Debian Wheezy AMD64) up to the moment I wanted to use SQLite.
> 
>I get the following message:
>===
>jantzen@PC4:~$ python
>Python 3.5.0 (default, Dec  2 2015, 14:16:16)
>[GCC 4.7.2] on linux
>Type "help", "copyright", "credits" or "license" for more information.
>>>> import sqlite3
>Traceback (most recent call last):
>  File "", line 1, in 
>  File "/usr/local/lib/python3.5/sqlite3/__init__.py", line 23, in
>
>from sqlite3.dbapi2 import *
>  File "/usr/local/lib/python3.5/sqlite3/dbapi2.py", line 27, in 
>from _sqlite3 import *
>ImportError: No module named '_sqlite3'
>===
> 
>Obviously something is missing.
>How do I solve the problem? Where do I find this module?
> 
>Thanks for a hint.
>--
> 
>K.D.J.
> -- 
> https://mail.python.org/mailman/listinfo/python-list
  
-- 
https://mail.python.org/mailman/listinfo/python-list

RE: libre office

[Albert-Jan Roskam]

> From: ji...@frontier.com
> To: python-list@python.org
> Subject: libre office
> Date: Tue, 19 Jan 2016 17:01:40 -0600
> 
> How do I get data from libre office using python?

Does this help?http://www.openoffice.org/udk/python/python-bridge.html  
  
-- 
https://mail.python.org/mailman/listinfo/python-list

RE: Screenshots in Sphinx docs

[Albert-Jan Roskam]

> To: python-list@python.org
> From: tjre...@udel.edu
> Subject: Re: Screenshots in Sphinx docs
> Date: Mon, 14 Dec 2015 14:01:03 -0500
> 
> On 12/14/2015 11:31 AM, Albert-Jan Roskam wrote:
> 
> > I'd like to include up-to-date screenshots (of a tkinter app)
>  > into my Sphinx documentation.
> 
> If you manually take screenshots with *any* screen grabber and save in 
> an appropriate format, this is apparently trivial -- use the ..image 
> directive.  From the below, it appears that what you want is to have a 
> literally up-to-date screenshot taken automatically during the doc build.
> 
> This requires that one be able to automate getting the application into 
> the exact display state one wants to capture.  You can probably do that 
> with a tkinter app if you write it with that possibility in mind.  In 
> particular, you must keep a Python reference to every widget you want to 
> manipulate, even if not needed for normal program operation.
> 
> There is also an issue with grabbing the whole screen versus only a 
> program-specific window.

I need only a few screens. I think I will call my tkinter app with 
subprocess.Popen, wait until it's loaded,
grab the image, then kill it. Then I indeed wanted to use the ..image directive.

>  > This looks ok:
> > https://pypi.python.org/pypi/sphinxcontrib-programscreenshot
> 
> This (automatically) takes 'screenshots' on a no-screen (headless) *nix 
> system (during doc build) by redirecting X-windows output to a 
> pseudo-screen program.  Rather clever, and system-specific.
> 
> > BUT I need something that works on Windows (Python 2.7).
>  > Can any recommend an approach? I thought about using PIL:
> 
> Get the pillow fork/upgrade on pypi.

Thanks for the tip! So good ol' PIL is no longer maintained?

 
> > http://www.varesano.net/blog/fabio/capturing%20screen%20image%20python%20and%20pil%20windows
> 
> Or look into Windows screen grabber programs, of which there are many.
> 
> -- 
> Terry Jan Reedy
> 
> -- 
> https://mail.python.org/mailman/listinfo/python-list
  
-- 
https://mail.python.org/mailman/listinfo/python-list

Screenshots in Sphinx docs

[Albert-Jan Roskam]

Hello,

I'd like to include up-to-date screenshots (of a tkinter app) into my Sphinx 
documentation. This looks ok:
https://pypi.python.org/pypi/sphinxcontrib-programscreenshot
BUT I need something that works on Windows (Python 2.7). Can any recommend an 
approach? I thought about using PIL: 
http://www.varesano.net/blog/fabio/capturing%20screen%20image%20python%20and%20pil%20windows

Thanks!

Albert-Jan
  
-- 
https://mail.python.org/mailman/listinfo/python-list

RE: Unicode failure

[Albert-Jan Roskam]

I think you need to use a raw unicode string, ur

>>> unicodedata.name(ur'\u2122')
'TRADE MARK SIGN'

> Date: Fri, 4 Dec 2015 13:07:38 -0500
> From: da...@vybenetworks.com
> To: python-list@python.org
> Subject: Unicode failure
> 
> I thought that going to Python 3.4 would solve my Unicode issues but it
> seems I still don't understand this stuff.  Here is my script.
> 
> #! /usr/bin/python3
> # -*- coding: UTF-8 -*-
> import sys 
> print(sys.getdefaultencoding()) 
> print(u"\N{TRADE MARK SIGN}") 
> 
> And here is my output.
> 
> utf-8
> Traceback (most recent call last):
>   File "./g", line 5, in 
> print(u"\N{TRADE MARK SIGN}")
> UnicodeEncodeError: 'ascii' codec can't encode character '\u2122' in
> position 0: ordinal not in range(128)
> 
> What am I missing?
> 
> TIA.
> 
> -- 
> D'Arcy J.M. Cain
> Vybe Networks Inc.
> http://www.VybeNetworks.com/
> IM:da...@vex.net VoIP: sip:da...@vybenetworks.com
> -- 
> https://mail.python.org/mailman/listinfo/python-list
  
-- 
https://mail.python.org/mailman/listinfo/python-list

RE: counting unique numpy subarrays

[Albert-Jan Roskam]

Hi

(Sorry for topposting)

numpy.ravel is faster than numpy.flatten (no copy)
numpy.empty is faster than numpy.zeros
numpy.fromiter might be useful to avoid the loop (just a hunch)

Albert-Jan

> From: duncan@invalid.invalid
> Subject: counting unique numpy subarrays
> Date: Fri, 4 Dec 2015 19:43:35 +
> To: python-list@python.org
> 
> Hello,
>   I'm trying to find a computationally efficient way of identifying
> unique subarrays, counting them and returning an array containing only
> the unique subarrays and a corresponding 1D array of counts. The
> following code works, but is a bit slow.
> 
> ###
> 
> from collections import Counter
> import numpy
> 
> def bag_data(data):
> # data (a numpy array) is bagged along axis 0
> # returns concatenated array and corresponding array of counts
> vec_shape = data.shape[1:]
> counts = Counter(tuple(arr.flatten()) for arr in data)
> data_out = numpy.zeros((len(counts),) + vec_shape)
> cnts = numpy.zeros((len(counts,)))
> for i, (tup, cnt) in enumerate(counts.iteritems()):
> data_out[i] = numpy.array(tup).reshape(vec_shape)
> cnts[i] =  cnt
> return data_out, cnts
> 
> ###
> 
> I've been looking through the numpy docs, but don't seem to be able to
> come up with a clean solution that avoids Python loops. TIA for any
> useful pointers. Cheers.
> 
> Duncan
> -- 
> https://mail.python.org/mailman/listinfo/python-list
  
-- 
https://mail.python.org/mailman/listinfo/python-list

RE: I'm using Sphinx, but is there a UML auto generator

[Albert-Jan Roskam]

(Sorry for top-posting, mobile hotmail sie sucks). This is cool, although it's 
not a Sphinx directive. You use insert the resulting graph in the .rst of 
course:
http://furius.ca/snakefood/

Date: Fri, 2 Oct 2015 11:14:59 -0300
Subject: I'm using Sphinx, but is there a UML auto generator
From: gilcanmach...@gmail.com
To: python-list@python.org

Hi,

I'm using Sphinx as a doc tool.

Amazing.

But I need a way to describe, for example, that, inside a Person class, there's 
a method called changePassword which depends of the execution of the method 
checkPermissions, and checkPermissions depends on the execution of two other 
methods.

I'm using a graphic tool like OpenDraw to do the job, but I guess there's a 
better way of doing this.

Is there any UML auto generator for python that do that?

If not, any suggestion?

By the way, I don't know UML (only the very basic stuff), I'm only trying to 
find a good way to describe the methods relations inside my class.

[]
Gil



-- 
https://mail.python.org/mailman/listinfo/python-list
  -- 
https://mail.python.org/mailman/listinfo/python-list

RE: Python CI and CD support available on Semaphore (feedback appreciated)

[Albert-Jan Roskam]

> To: python-list@python.org
> From: tjre...@udel.edu
> Subject: Re: Python CI and CD support available on Semaphore (feedback 
> appreciated)
> Date: Tue, 8 Sep 2015 14:46:28 -0400
> 
> On 9/8/2015 6:27 AM, Filip Komnenović wrote:
> 
> > We have recently launched Python support on our continuous integration and 
> > deployment service and are looking for communities feedback. If you're up 
> > for it, please give a test drive to our service with your Python projects 
> > and give us your thoughts on what we could further improve upon.
> 
> Have you considered integrating with bitbucket as well as github?


Bitbucket is supported, not just Github (like most CIs). The information is 
kinda hidden (or at least not as clearly visible as the Github integration):
https://semaphoreci.com/docs/adding-github-bitbucket-project-to-semaphore.html

I see Ubuntu 14.04 LTS 64-bit VMs are used. I would love to be able to run my 
tests on Linux, Mac OS and Windows in one go.

> 
> > Official blog post announcing Python support: http://bit.ly/1KYrVsT
> > Our website: https://semaphoreci.com/
> 
> > BTW we offer free plans for open source projects and have limited free 
> > plans for private projects.
> 
> -- 
> Terry Jan Reedy
> 
> 
> -- 
> https://mail.python.org/mailman/listinfo/python-list
  -- 
https://mail.python.org/mailman/listinfo/python-list

RE: -2146826246 in win32com.client for empty #N/A cell in Excel

[Albert-Jan Roskam]

> Date: Sun, 16 Aug 2015 09:53:32 -0700
> Subject: -2146826246 in win32com.client for empty #N/A cell in Excel
> From: sven.bo...@gmail.com
> To: python-list@python.org
> 
> 
> Anyone know how to handle "#N/A" in Excel from win32com.client.
> 
> I'm extracting data from an Excel file using win32com.client. Everything 
> works fine except for when the value "#N/A" is entered in excel. An empty 
> cell. I assumed I do something as 
> 
> if ws.Cells(r, c).Value is None:
> ...
> 
> But that doesn't seem to work. When I debug the piece of code while handling 
> #N/A in a cell the type of the cell according to win32com.client is int and 
> the value in the cell is -2146826246. Chances are small just this number will 
> appear in Excel, but it looks dirty to depend on that value to decide if a 
> cell is empty. Looked around the net for a solution, but nothing came up so 
> far.
> 
> Anyone knows how to handle a "#N/A" cell in Excel in the proper way?
> 
> Regards,
> Sven
> -- 
> https://mail.python.org/mailman/listinfo/python-list


Hello,
Does that number happen to be -1 * sys.maxint?
Regards,Albert-Jan

  -- 
https://mail.python.org/mailman/listinfo/python-list

Re: windows and file names > 256 bytes

[Albert-Jan Roskam]

 

> import os import shutil import sys
>  
> # create an insanely long directory tree p = os.getenv("TEMP")
> #p = ur"\\server\share\blah\temp"
> tmpdir = p os.chdir(tmpdir)
> for i in xrange(1000):
> tmpdir = os.path.join(tmpdir, "sub") os.mkdir("?\\" + tmpdir)
> #os.mkdir(u"?\\UNC" + tmpdir[1:])
>  
> # write a file to it deep = "?\\" + os.path.join(tmpdir, "deep.txt")
> assert os.path.exists(deep)

sorry, this "assert" should of course follow 'with open(..' 

---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: windows and file names > 256 bytes

[Albert-Jan Roskam]

On Thu, 25 Jun 2015 14:37:55 +0100, Tim Golden wrote:

> On 25/06/2015 14:35, Michael Torrie wrote:
>> On 06/25/2015 06:34 AM, Tim Golden wrote:
>>> On 25/06/2015 13:04, Joonas Liik wrote:
 It sounds to me more like it is possible to use long file names on
 windows but it is a pain and in python, on windows it is basically
 impossible.
>>>
>>> Certainly not impossible: you could write your own wrapper function:
>>>
>>> def extended_path(p):
>>> return r"\\?\%s" % os.path.abspath(p)
>>>
>>> where you knew that there was a possibility of long paths and that an
>>> absolute path would work.
>> 
>> The OP mentions that even when he manually supplies extended paths,
>> os.mkdir, os.getsize, and shutil.rmtree return errors for him in Python
>> 2.7.  So there's more to this problem.
>> 
>> 
> He's probably not passing unicode strings: the extended path only works
> for unicode string. For 3.x that's what you do by default.

Hi all,

Thanks for your replies. I've been messing with this a bit more. I 
created a little test script (see below). However, this only works with 
drive letters, not with UNC paths. I tried using os.chdir, DOS pushd, 
subst, net use but they all don't seem to work with with long paths. 
I finally managed to remove an absurdly long dir with shutil.rmtree, 
after changing sys.setrecursionlimit. But my main goal was to get the 
file size (and, actually, also the file owner) of a long file name on XP.

import os
import shutil
import sys
 
# create an insanely long directory tree
p = os.getenv("TEMP")
#p = ur"\\server\share\blah\temp"
tmpdir = p
os.chdir(tmpdir)
for i in xrange(1000):
tmpdir = os.path.join(tmpdir, "sub")
os.mkdir("?\\" + tmpdir)
#os.mkdir(u"?\\UNC" + tmpdir[1:])
 
# write a file to it
deep = "?\\" + os.path.join(tmpdir, "deep.txt")
assert os.path.exists(deep)
with open(deep, "w") as f:
f.write("Deep!\r\n")
 
# try if the file size can be determined (requires special \\?\ notation)
print " %d bytes" % os.path.getsize(deep)
 
# now delete the whole directory and its contents.
path = "?\\" + os.path.join(p, "sub")
path = path.decode(sys.getfilesystemencoding())
sys.setrecursionlimit(10 ** 7)  # net use, pushd, subst will not work
shutil.rmtree(path)

Any feedback is welcome. I will post the solution somewhere so somebody 
else will be spared this nuisance. :-)

Regards,
Albert-Jan


---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus


-- 
https://mail.python.org/mailman/listinfo/python-list