Re: cannot open file with non-ASCII filename

2015-12-15 Thread Ulli Horlacher
Laura Creighton  wrote:

> PyPy wrote its own pyreadline.
> You can get it here. https://bitbucket.org/pypy/pyrepl

As far as I can see, it has no getkey function.
My users do not hit ENTER after drag&drop or copy&paste files.
I need an input function with a timeout.


-- 
Ullrich Horlacher  Server und Virtualisierung
Rechenzentrum IZUS/TIK E-Mail: horlac...@tik.uni-stuttgart.de
Universitaet Stuttgart Tel:++49-711-68565868
Allmandring 30aFax:++49-711-682357
70550 Stuttgart (Germany)  WWW:http://www.tik.uni-stuttgart.de/
-- 
https://mail.python.org/mailman/listinfo/python-list


[no subject]

2015-12-15 Thread Malik Brahimi
Basically,  I'm trying to make an event based system with pywin32 that
handles spooled files that are to be printed. Users often print under the
impression that their document has yet to emerge from the printer when it
is in fact in queue. I'm creating a script that polls print jobs,  saves
them in a Mongo DB as needed, and prompts the user to reconsider another
print job for a duplicate document.

However,  I'm running into some problems distinguishing documents as unique
because the print job could have the same document name but the files could
be in different directories.  Additionally,  I have no idea how to
temporarily halt spooling until the user has confirmed their decision.

Does anyone have any idea how to distinguish documents as unique and how to
temporarily halt spooling as described?
-- 
https://mail.python.org/mailman/listinfo/python-list


Need help - How to identify the cell display format?

2015-12-15 Thread Ezhilarasan Chandrasekar
Hi folks,

I just want to find the cell display format in Excel. I have a Expected
excel file and Actual Excel file.

I have some knowledge about, how to check the cell value, cell font,
alignment. But I also want to know about what type of cell format is being
used.

For example: If the cell value has "*04:05:00 AM*", but it displays as "
*04:05:00*"(AM notation not needed) which is of format "*hh:mm:ss*"

I want to identify the cell format *(hh:mm:ss)* from Expected Excel file
and compare with the Actual Excel file

Please help me!

--
Warm regards,
Ezhilarasan
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Need help - How to identify the cell display format?

2015-12-15 Thread Gonzalo V
Are you working with time series?

saludos,
desde un móvil.
El dic 15, 2015 9:41 a.m., "Ezhilarasan Chandrasekar" 
escribió:

> Hi folks,
>
> I just want to find the cell display format in Excel. I have a Expected
> excel file and Actual Excel file.
>
> I have some knowledge about, how to check the cell value, cell font,
> alignment. But I also want to know about what type of cell format is being
> used.
>
> For example: If the cell value has "*04:05:00 AM*", but it displays as "
> *04:05:00*"(AM notation not needed) which is of format "*hh:mm:ss*"
>
> I want to identify the cell format *(hh:mm:ss)* from Expected Excel file
> and compare with the Actual Excel file
>
> Please help me!
>
> --
> Warm regards,
> Ezhilarasan
> --
> https://mail.python.org/mailman/listinfo/python-list
>
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Try: rather than if :

2015-12-15 Thread jmp

On 12/14/2015 11:38 PM, Vincent Davis wrote:

In the code below try is used to check if handle has the attribute name. It
seems an if statement could be used. Is there reason one way would be
better than another?

def write_header(self):
 handle = self.handle
 try:
 handle.write("# Report_file: %s\n" % handle.name)
 except AttributeError:
 pass
 handle.write("\n")

The specific use case I noticed this was
https://github.com/biopython/biopython/blob/master/Bio/AlignIO/EmbossIO.py#L38

Vincent Davis



Nothing wrong with the try block. However other forms may be 
shorter/more readable.
Since there is 2 attribute lookups in the try block, (write and name) 
it's not clear which one you want to catch (except for the line 
following the except clause but it could not be the case).


Here are 2 alternative forms

1/ handle.write(handle.name if hasattr(handle, 'name') else '')
2/ handle.write(getattr(handle, 'name', ''))

Here's the best solution imo:
3/ assume handle has always a name attribute and don't write code 
handling attribute existence. Instead handle the attribute values:


class Handle:
  def __init__(self):
self.name= None # and the problem is gone

if handle.name : handle.write(handle.name)


By the way, in the use case you've linked, it is written
'handle = self.handle'
at every beginning of each method. Don't follow that rule.


Jm


--
https://mail.python.org/mailman/listinfo/python-list


wrapper api / proxying third party exception?

2015-12-15 Thread Benjamin Risher
Ok, here's my issue.  I have a plugin framework I wrote that relies on a third 
party library that I can't change.  As a part of that framework, I wrote a 
small stub library to proxy calls to the third party library, so that if their 
api ever changes, plugins written against the stub won't have to change, I'll 
just have to update the back-half that interfaces with the library.  I think 
this is called an adapter design pattern...?  

plugin --> my api --> their api --> their library 

It works, for the most part.  Most of my classes look similar to this

class MYClass:
def __init__(self, *args, **kwargs):
self.__theirclass = TheirClass(*args, **kwargs)

def __getattr__(self, attr):
return getattr(self.__theirclass, attr)

What I'm having issue with, is their exception class.  

1.  A plugin calls a stub in my library
2.  That stub calls the corresponding call in the third party library
3.  The call raises an exception from the library
4.  The plugin can't catch the exception without importing directly from the 
third party library

What is the correct way to allow plugins to import my stub exception, but catch 
the ones raised by the library?  I was looking at monkeying with subclasshook 
or something along those lines, but it doesn't seem like the best approach.  
Hoping for some help from here.  

Please let me know if you need any clarification to help!

Thank you.
-- 
https://mail.python.org/mailman/listinfo/python-list


subprocess.call with non-ASCII arguments?

2015-12-15 Thread Ulli Horlacher
I want to create a zip file within a Python 2.7 program on windows.

My code:

  cmd = ['7za.exe','a','-tzip',archive] + files
  status = subprocess.call(cmd)

leads to:

  File "fexit.py", line 971, in sendfile_retry
status = subprocess.call(cmd)
  File "C:\Python27\lib\subprocess.py", line 522, in call
return Popen(*popenargs, **kwargs).wait()
  File "C:\Python27\lib\subprocess.py", line 710, in __init__
errread, errwrite)
  File "C:\Python27\lib\subprocess.py", line 958, in _execute_child
startupinfo)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xf6' in position 87:
 ordinal not in range(128)


This is because the array "files" contains filenames with non-ASCII
characters.

So, the problem is in subprocess.py, which I cannot modify.

What can I do?


-- 
Ullrich Horlacher  Server und Virtualisierung
Rechenzentrum IZUS/TIK E-Mail: horlac...@tik.uni-stuttgart.de
Universitaet Stuttgart Tel:++49-711-68565868
Allmandring 30aFax:++49-711-682357
70550 Stuttgart (Germany)  WWW:http://www.tik.uni-stuttgart.de/
-- 
https://mail.python.org/mailman/listinfo/python-list


Urgent: Last call for the CfP of PythonFOSDEM 2016

2015-12-15 Thread Stephane Wirtel
Hi all

Because the deadline is imminent and because we have only received some
proposals, we have extended the current deadline. The new submission deadline is
2015-12-20.

Call For Proposals
==

This is the official call for sessions for the Python devroom at FOSDEM 2016.

FOSDEM is the Free and Open source Software Developers' European Meeting, a free
and non-commercial two-day week-end that offers open source contributors a place
to meet, share ideas and collaborate.

It's the biggest event in Europe with +5000 hackers, +400 speakers.

For this edition, Python will be represented by its Community.
If you want to discuss with a lot of Python Users, it's the place to be!

Important dates
===

* Submission deadlines: 2015-12-20
* Acceptance notifications: 2015-12-24

Practical
=

* The duration for talks will be 30 minutes, including presentations and
questions and answers.
* Presentation can be recorded and streamed, sending your proposal implies
giving permission to be recorded.
* A mailing list for the Python devroom is available for discussions about
devroom organisation. You can register at this address:
https://lists.fosdem.org/listinfo/python-devroom

How to submit
=

All submissions are made in the Pentabarf event planning tool at
https://penta.fosdem.org/submission/FOSDEM16

When submitting your talk in Pentabarf, make sure to select the Python devroom
as the Track.

Of course, if you already have a user account, please reuse it.

Questions
=

Any questions, please sned an email to info AT python-fosdem DOT org

Thank you for submitting your sessions and see you soon in Brussels to talk
about Python.

If you want to keep informed for this edition, you can follow our twitter
account @PythonFOSDEM.

* FOSDEM 2016: https://fosdem.org/2016
* Python Devroom: http://python-fosdem.org
* Twitter: https://twitter.com/PythonFOSDEM

Thank you so much,

Stephane

-- 
Stéphane Wirtel - http://wirtel.be - @matrixise
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Need help - How to identify the cell display format?

2015-12-15 Thread Larry Martell
On Tue, Dec 15, 2015 at 2:29 AM, Ezhilarasan Chandrasekar
 wrote:
> Hi folks,
>
> I just want to find the cell display format in Excel. I have a Expected
> excel file and Actual Excel file.
>
> I have some knowledge about, how to check the cell value, cell font,
> alignment. But I also want to know about what type of cell format is being
> used.
>
> For example: If the cell value has "*04:05:00 AM*", but it displays as "
> *04:05:00*"(AM notation not needed) which is of format "*hh:mm:ss*"
>
> I want to identify the cell format *(hh:mm:ss)* from Expected Excel file
> and compare with the Actual Excel file

You can do this with openpyxl, for example:

from openpyxl.reader.excel import load_workbook

book = load_workbook(filename='transactions.xlsx')
sheet = book.get_sheet_by_name('transactions')
print sheet.cell("A12").style.number_format
print sheet.cell("A13").style.number_format
-- 
https://mail.python.org/mailman/listinfo/python-list


Urgent: Last call for the CfP of PythonFOSDEM 2016

2015-12-15 Thread Stephane Wirtel
Hi all

Because the deadline is imminent and because we have only received some
proposals, we have extended the current deadline. The new submission deadline is
2015-12-20.

Call For Proposals
==

This is the official call for sessions for the Python devroom at FOSDEM 2016.

FOSDEM is the Free and Open source Software Developers' European Meeting, a free
and non-commercial two-day week-end that offers open source contributors a place
to meet, share ideas and collaborate.

It's the biggest event in Europe with +5000 hackers, +400 speakers.

For this edition, Python will be represented by its Community.
If you want to discuss with a lot of Python Users, it's the place to be!

Important dates
===

* Submission deadlines: 2015-12-20
* Acceptance notifications: 2015-12-24

Practical
=

* The duration for talks will be 30 minutes, including presentations and
questions and answers.
* Presentation can be recorded and streamed, sending your proposal implies
giving permission to be recorded.
* A mailing list for the Python devroom is available for discussions about
devroom organisation. You can register at this address:
https://lists.fosdem.org/listinfo/python-devroom

How to submit
=

All submissions are made in the Pentabarf event planning tool at
https://penta.fosdem.org/submission/FOSDEM16

When submitting your talk in Pentabarf, make sure to select the Python devroom
as the Track.

Of course, if you already have a user account, please reuse it.

Questions
=

Any questions, please sned an email to info AT python-fosdem DOT org

Thank you for submitting your sessions and see you soon in Brussels to talk
about Python.

If you want to keep informed for this edition, you can follow our twitter
account @PythonFOSDEM.

* FOSDEM 2016: https://fosdem.org/2016
* Python Devroom: http://python-fosdem.org
* Twitter: https://twitter.com/PythonFOSDEM

Thank you so much,

Stephane

-- 
Stéphane Wirtel - http://wirtel.be - @matrixise
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: cannot open file with non-ASCII filename

2015-12-15 Thread Laura Creighton
In a message of Tue, 15 Dec 2015 08:26:37 +, Ulli Horlacher writes:
>Laura Creighton  wrote:
>
>> PyPy wrote its own pyreadline.
>> You can get it here. https://bitbucket.org/pypy/pyrepl
>
>As far as I can see, it has no getkey function.
>My users do not hit ENTER after drag&drop or copy&paste files.
>I need an input function with a timeout.

Right, then this isn't going to work.  Sorry about that.

Laura

-- 
https://mail.python.org/mailman/listinfo/python-list


subprocess.call with non-ASCII arguments?

2015-12-15 Thread Ulli Horlacher
(My first posting seems to got lost)

I want to create a zip file within a Python 2.7 program on windows.

My code:

  cmd = ['7za.exe','a','-tzip',archive] + files
  status = subprocess.call(cmd)

leads to:

  File "fexit.py", line 971, in sendfile_retry
status = subprocess.call(cmd)
  File "C:\Python27\lib\subprocess.py", line 522, in call
return Popen(*popenargs, **kwargs).wait()
  File "C:\Python27\lib\subprocess.py", line 710, in __init__
errread, errwrite)
  File "C:\Python27\lib\subprocess.py", line 958, in _execute_child
startupinfo)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xf6' in position 87:
 ordinal not in range(128)


This is because the array "files" contains filenames with non-ASCII
characters.

So, the problem is in subprocess.py, which I cannot modify.


Instead of calling a 7z subprocess with non-ASCII arguments I tried to
call it with a listfile: it starts with a "@" and contains the names of
the files to be packed into the arcive. It is a special 7z feature.

New code:

  fileslist = archive + '.list'
  flo = open(fileslist,'w')
  for file in files: print(file,file=flo)
  flo.close()
  cmd = ['7za.exe','a','-tzip',archive,'@'+fileslist]
  status = subprocess.call(cmd)


But with that I get a new error:

  File "fexit.py", line 959, in sendfile_retry
for file in files: print(file,file=flo)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xf6' in position 8:
  ordinal not in range(128)


I get the same error message, when i use:
  flo = open(fileslist,'wb')
  

How can I tell open() or print() that I want to write non-ASCII ?



-- 
Ullrich Horlacher  Server und Virtualisierung
Rechenzentrum IZUS/TIK E-Mail: horlac...@tik.uni-stuttgart.de
Universitaet Stuttgart Tel:++49-711-68565868
Allmandring 30aFax:++49-711-682357
70550 Stuttgart (Germany)  WWW:http://www.tik.uni-stuttgart.de/
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: subprocess.call with non-ASCII arguments?

2015-12-15 Thread Ulli Horlacher
Ulli Horlacher  wrote:

> Instead of calling a 7z subprocess with non-ASCII arguments I tried to
> call it with a listfile: it starts with a "@" and contains the names of
> the files to be packed into the arcive. It is a special 7z feature.
> 
> New code:
> 
>   fileslist = archive + '.list'
>   flo = open(fileslist,'w')
>   for file in files: print(file,file=flo)
>   flo.close()
>   cmd = ['7za.exe','a','-tzip',archive,'@'+fileslist]
>   status = subprocess.call(cmd)
> 
> 
> But with that I get a new error:
> 
>   File "fexit.py", line 959, in sendfile_retry
> for file in files: print(file,file=flo)
> UnicodeEncodeError: 'ascii' codec can't encode character u'\xf6' in position 
> 8:
>   ordinal not in range(128)

I found a partial solution:

for file in files: print(file.encode('utf8'),file=flo)

But this works only for files I get from Tk askopenfilename(), not for
files from sys.argv[]
Then I see:

S:\>python fexit.py -a x.zip C:\Users\admin\_XöX.exe .
files selected:

"C:\Users\admin\_X÷X.exe"
2015-12-07 16:17:15
114 kB

Traceback (most recent call last):
  File "fexit.py", line 2166, in 
wexit(main())
  File "fexit.py", line 260, in main
status = sendfile_retry(files,recipient,comment)
  File "fexit.py", line 959, in sendfile_retry
for file in files: print(file.encode('utf8'),file=flo)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xf6 in position 17:
 ordinal not in range(128)


-- 
Ullrich Horlacher  Server und Virtualisierung
Rechenzentrum IZUS/TIK E-Mail: horlac...@tik.uni-stuttgart.de
Universitaet Stuttgart Tel:++49-711-68565868
Allmandring 30aFax:++49-711-682357
70550 Stuttgart (Germany)  WWW:http://www.tik.uni-stuttgart.de/
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: subprocess.call with non-ASCII arguments?

2015-12-15 Thread Laura Creighton
In a message of Tue, 15 Dec 2015 14:25:50 +, Ulli Horlacher writes:
>(My first posting seems to got lost)
>
>I want to create a zip file within a Python 2.7 program on windows.
>
>My code:
>
>  cmd = ['7za.exe','a','-tzip',archive] + files
>  status = subprocess.call(cmd)
>
>leads to:
>
>  File "fexit.py", line 971, in sendfile_retry
>status = subprocess.call(cmd)
>  File "C:\Python27\lib\subprocess.py", line 522, in call
>return Popen(*popenargs, **kwargs).wait()
>  File "C:\Python27\lib\subprocess.py", line 710, in __init__
>errread, errwrite)
>  File "C:\Python27\lib\subprocess.py", line 958, in _execute_child
>startupinfo)
>UnicodeEncodeError: 'ascii' codec can't encode character u'\xf6' in position 
>87:
> ordinal not in range(128)
>
>
>This is because the array "files" contains filenames with non-ASCII
>characters.
>
>So, the problem is in subprocess.py, which I cannot modify.
>
>
>Instead of calling a 7z subprocess with non-ASCII arguments I tried to
>call it with a listfile: it starts with a "@" and contains the names of
>the files to be packed into the arcive. It is a special 7z feature.
>
>New code:
>
>  fileslist = archive + '.list'
>  flo = open(fileslist,'w')
>  for file in files: print(file,file=flo)
>  flo.close()
>  cmd = ['7za.exe','a','-tzip',archive,'@'+fileslist]
>  status = subprocess.call(cmd)
>
>
>But with that I get a new error:
>
>  File "fexit.py", line 959, in sendfile_retry
>for file in files: print(file,file=flo)
>UnicodeEncodeError: 'ascii' codec can't encode character u'\xf6' in position 8:
>  ordinal not in range(128)
>
>
>I get the same error message, when i use:
>  flo = open(fileslist,'wb')
>  
>
>How can I tell open() or print() that I want to write non-ASCII ?

see if setting the environment variable PYTHONIOENCODING
https://docs.python.org/2/using/cmdline.html

works for you.  No promises, not a windows user.

Laura
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Question about figure plot

2015-12-15 Thread Robert Kern

On 2015-12-15 02:43, Robert wrote:

Hi,

When I run the following code, there is no figure shown in the end.


//
import pymc
import numpy as np

n = 5*np.ones(4,dtype=int)
x = np.array([-.86,-.3,-.05,.73])

alpha = pymc.Normal('alpha',mu=0,tau=.01)
beta = pymc.Normal('beta',mu=0,tau=.01)

@pymc.deterministic
def theta(a=alpha, b=beta):
 """theta = logit^{-1}(a+b)"""
 return pymc.invlogit(a+b*x)

d = pymc.Binomial('d', n=n, p=theta, value=np.array([0.,1.,3.,5.]),\
 observed=True)

import pymc
import mymodel

S = pymc.MCMC(mymodel, db='pickle')
S.sample(iter=1, burn=5000, thin=2)
pymc.Matplot.plot(S)



I find that the figures are shown after these two lines by myself:
*
import matplotlib.pyplot as plt
plt.show()

I have searched around and have not found some explanation about it.
The plot function here is different from Matlab's. Is there better ways than
my last two lines? (I am not confident whether my last two lines is the
only choice.


No, that's right. pymc.Matplot.plot() uses matplotlib's pyplot API underneath. 
pyplot can run in two different modes: interactive and non-interactive. When 
used in a standalone script, like I assume here, it defaults to non-interactive. 
That means that it will not raise any plot windows until you call plt.show().


  http://matplotlib.org/faq/usage_faq.html#what-is-interactive-mode

See any of the examples here (note: "pylab" is the essentially the same as 
"pyplot" for these purposes):


  http://matplotlib.org/examples/pylab_examples/index.html

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

--
https://mail.python.org/mailman/listinfo/python-list


Re: cannot open file with non-ASCII filename

2015-12-15 Thread eryk sun
On Tue, Dec 15, 2015 at 2:26 AM, Ulli Horlacher
 wrote:
> Laura Creighton  wrote:
>
>> PyPy wrote its own pyreadline.
>> You can get it here. https://bitbucket.org/pypy/pyrepl
>
> As far as I can see, it has no getkey function.
> My users do not hit ENTER after drag&drop or copy&paste files.
> I need an input function with a timeout.

pyreadline looked promising for its extensive ctypes implementation of
the Windows console API [1], wrapped by high-level methods such as
peek, getchar, and getkeypress. It turns out it ignores the event
sequences you need for alt+numpad input (used when a file is dragged
into the console). You'd have to modify its console and keysyms
modules to make it work. It would be a useful enhancement, so probably
your patches would be accepted upstream.

AFAICT, pyrepl has no Windows support. Check the TODO [2]:

> + port to windows

[1]: 
https://github.com/pyreadline/pyreadline/blob/master/pyreadline/console/console.py
[2]: 
https://bitbucket.org/pypy/pyrepl/src/62f2256014af7b74b97c00827f1a7789e00dd814/TODO?at=v0.8.4
-- 
https://mail.python.org/mailman/listinfo/python-list


RE: Windows 10 and PYODBC

2015-12-15 Thread paul.hermeneutic
On Dec 13, 2015 7:20 AM, "William Abdo"  wrote:
>
> Problem Resolved.
> I have fixed the Oracle connection issue under Windows 10 with cx_Oracle .
> PYODBC was only failing on the Oracle connection and worked fine on MS
SQL under Windows 10.

Please tell us what the fix is. Thanks.
-- 
https://mail.python.org/mailman/listinfo/python-list


subclassing collections.Counter

2015-12-15 Thread Pavlos Parissis
Hi,

I need to store values for metrics and return the average for some
and the sum for the rest. Thus, I thought I could extend
collections.Counter class by returning averages for some keys.

My class modifies the update() to increment a counter and the
__getitem__ to perform the calculation. But, I get RuntimeError: maximum
recursion depth exceeded as I access an attribute inside
__getitem__.

Does anyone has an idea how I can achieve this?


Example::
In [7]: metrics = {'f':6, 'g':1}

In [8]: avg_metrics = ['f']

In [9]: a = CounterExt(avg_metrics=avg_metrics)

In [10]: a.update(metrics)

In [11]: a
Out[11]: CounterExt({'f': 6, 'g': 1})

In [12]: a.update(metrics)

In [13]: a
Out[13]: CounterExt({'f': 12, 'g': 2})

In [14]: a['f']
.
RuntimeError: maximum recursion depth exceeded


Code::
from collections import Counter
from collections.abc import Mapping
class CounterExt(Counter):
 def __init__(self, iterable=None, avg_metrics=None, **kwds):
self._counter = 1  # a zero value will raise ZeroDivisionError
self.avg_metrics = avg_metrics
super().__init__()
self.update(iterable, **kwds)

def _count_elements(self,  iterable):
"""Tally elements from the iterable.

This is copied from collections/__init__.py
original code::
def _count_elements(mapping, iterable):
'Tally elements from the iterable.'
mapping_get = mapping.get
for elem in iterable:
mapping[elem] = mapping_get(elem, 0) + 1
"""
for elem in iterable:
self[elem] = self.get(elem, 0) + 1

def __getitem__(self, key):
if (self.avg_metrics is not None and key in self.avg_metrics):
return self[key] / self._counter
else:
return self[key]

def update(self, iterable=None, **kwds):
if iterable is not None:
if isinstance(iterable, Mapping):
if self:
self_get = self.get
self._counter +=1  # local modification
for elem, count in iterable.items():
self[elem] = count + self_get(elem, 0)
else:
super().update(iterable)
else:
self._count_elements(iterable)
if kwds:
self.update(kwds)





signature.asc
Description: OpenPGP digital signature
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: subclassing collections.Counter

2015-12-15 Thread Peter Otten
Pavlos Parissis wrote:

> I need to store values for metrics and return the average for some
> and the sum for the rest. Thus, I thought I could extend
> collections.Counter class by returning averages for some keys.
> 
> My class modifies the update() to increment a counter and the
> __getitem__ to perform the calculation. But, I get RuntimeError: maximum
> recursion depth exceeded as I access an attribute inside
> __getitem__.
> 
> Does anyone has an idea how I can achieve this?

> class CounterExt(Counter):

> def __getitem__(self, key):
> if (self.avg_metrics is not None and key in self.avg_metrics):
> return self[key] / self._counter
> else:
> return self[key]

self[key] will call the CounterExt.__getitem__() method again. Use 
super().__getitem__(key) instead to invoke Counter.__getitem__().



-- 
https://mail.python.org/mailman/listinfo/python-list


Re: subclassing collections.Counter

2015-12-15 Thread Pavlos Parissis
On 15/12/2015 05:08 μμ, Peter Otten wrote:
> Pavlos Parissis wrote:
> 
>> I need to store values for metrics and return the average for some
>> and the sum for the rest. Thus, I thought I could extend
>> collections.Counter class by returning averages for some keys.
>>
>> My class modifies the update() to increment a counter and the
>> __getitem__ to perform the calculation. But, I get RuntimeError: maximum
>> recursion depth exceeded as I access an attribute inside
>> __getitem__.
>>
>> Does anyone has an idea how I can achieve this?
> 
>> class CounterExt(Counter):
> 
>> def __getitem__(self, key):
>> if (self.avg_metrics is not None and key in self.avg_metrics):
>> return self[key] / self._counter
>> else:
>> return self[key]
> 
> self[key] will call the CounterExt.__getitem__() method again. Use 
> super().__getitem__(key) instead to invoke Counter.__getitem__().
> 
> 
> 

I applied your suggestion and worked!
   def __getitem__(self, key):
if (self.avg_metrics is not None and key in self.avg_metrics):
return super().__getitem__(key) / self.__counter
else:
return super().__getitem__(key)

Thank you very much,
Pavlos



signature.asc
Description: OpenPGP digital signature
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: subclassing collections.Counter

2015-12-15 Thread Ian Kelly
On Tue, Dec 15, 2015 at 8:49 AM, Pavlos Parissis
 wrote:
> Hi,
>
> I need to store values for metrics and return the average for some
> and the sum for the rest. Thus, I thought I could extend
> collections.Counter class by returning averages for some keys.

Leave Counter out of it, as this is not what it's designed for. Write
a custom Metrics class, with each attribute being a pseudo-collection
that maintains a sum or average.
-- 
https://mail.python.org/mailman/listinfo/python-list


Help on code comprehension from an example project of pymc

2015-12-15 Thread Robert
Hi,

I find the useful small code project for me:
#https://users.obs.carnegiescience.edu/cburns/ipynbs/PyMC.html

It runs as expected.

When I review the code, I find 'data' in the original line:

data = pymc.Normal('data', mu=model, tau=tau, value=z_obs, observed=True)

has not been referenced thereafter.
If I comment out the line as:

#data = pymc.Normal('data', mu=model, tau=tau, value=z_obs, observed=True)

the result is ugly different from the original.

If I change it to:

pymc.Normal('data', mu=model, tau=tau, value=z_obs, observed=True)


it still runs as the original. Reading the last half part 
(after data= line), I cannot see anything uses 'data', though I suspect
below line may use it:
sampler = pymc.MCMC([alpha,betax,betay,eps,model,tau,z_obs,x_true,y_true])

This is quite different from my any other language experience.

Could you help me on this?

Thanks,





--
#https://users.obs.carnegiescience.edu/cburns/ipynbs/PyMC.html
from numpy import *
Nobs = 20
x_true = random.uniform(0,10, size=Nobs)
y_true = random.uniform(-1,1, size=Nobs)
alpha_true = 0.5
beta_x_true = 1.0
beta_y_true = 10.0
eps_true = 0.5
z_true = alpha_true + beta_x_true*x_true + beta_y_true*y_true
z_obs = z_true + random.normal(0, eps_true, size=Nobs)


import pymc
# define the parameters with their associated priors

#alpha = pymc.Normal('alpha',mu=0,tau=.01)
#beta = pymc.Normal('beta',mu=0,tau=.01)

alpha = pymc.Uniform('alpha', -100,100, value=median(z_obs))
betax = pymc.Uniform('betax', -100,100, value=std(z_obs)/std(x_true))
betay = pymc.Uniform('betay', -100,100, value=std(z_obs)/std(y_true))
eps   = pymc.Uniform('eps',  0,100, value=0.01)

# Now define the model
@pymc.deterministic
def model(alpha=alpha, betax=betax, betay=betay, x=x_true, y=y_true):
return alpha + betax*x + betay*y

# pymc parametrizes the width of the normal distribution by tau=1/sigma**2
@pymc.deterministic
def tau(eps=eps):
return power(eps, -2)

# Lastly relate the model/parameters to the data
#data = pymc.Normal('data', mu=model, tau=tau, value=z_obs, observed=True)
pymc.Normal('data0', mu=model, tau=tau, value=z_obs, observed=True)

sampler = pymc.MCMC([alpha,betax,betay,eps,model,tau,z_obs,x_true,y_true])
sampler.use_step_method(pymc.AdaptiveMetropolis, [alpha,betax,betay,eps],
scales={alpha:0.1, betax:0.1, betay:1.0, eps:0.1})
sampler.sample(iter=1)

pymc.Matplot.plot(sampler)

sampler.sample(iter=1)

alpha.summary()

m_alpha = median(alpha.trace())
m_betax = median(betax.trace())
m_betay = median(betay.trace())
m_eps = median(eps.trace())


import matplotlib.pyplot as plt
plt.figure(figsize=(12,6))
plt.subplot(1,2,1)
plt.plot(x_true, z_obs-m_alpha-m_betay*y_true, 'o')
plt.xlabel('X')
plt.ylabel('Z - alpha - beta_y y')
# Now plot the model
xx = array([x_true.min(), x_true.max()])
plt.plot(xx, xx*m_betax)
plt.plot(xx, xx*m_betax + m_eps, '--', color='k')
plt.plot(xx, xx*m_betax - m_eps, '--', color='k')
plt.subplot(1,2,2)
plt.plot(y_true, z_obs-m_alpha-m_betax*x_true, 'o')
plt.xlabel('Y')
plt.ylabel('Z - alpha - beta_x x')
yy = array([y_true.min(), y_true.max()])
plt.plot(yy, yy*m_betay)
plt.plot(yy, yy*m_betay + m_eps, '--', color='k')
plt.plot(yy, yy*m_betay - m_eps, '--', color='k')
plt.show()
-- 
https://mail.python.org/mailman/listinfo/python-list


RE: Windows 10 and PYODBC

2015-12-15 Thread paul.hermeneutic
On Dec 15, 2015 9:22 AM, "William Abdo"  wrote:
>
> So I started a search for an Oracle based ODBC client since PYODBC is
still working with the Microsoft DB’s, I tried cx_Oracle and it worked
perfectly after I managed to get the parameters correct on the call,

It sounds like PYODBC cannot connect to Oracle on Windows 10. Is that
correct?
-- 
https://mail.python.org/mailman/listinfo/python-list


RE: Windows 10 and PYODBC

2015-12-15 Thread William Abdo
Hi Paul Hermeneutic,
The issue was it could not establish a connection , it threw  an error 
ORA-01019 (1019) .
So after some research it began to look like it was unable to connect via the 
TNS Names. I was unable to verify this since it was just dying and not logging 
the issue.
So I started a search for an Oracle based ODBC client since PYODBC is still 
working with the Microsoft DB’s, I tried cx_Oracle and it worked perfectly 
after I managed to get the parameters correct on the call,
It utilized all the syntax for the most part of PYODBC calls so only the 
connection information needed to be altered.
Using parts of a snippet found on the internet that helped me.
Since I still needed PYODBC to talk to SQL Server I found that it worked best 
if cx_Oracle’s import statement is Place before other ODBC clients. At least it 
appeared that way to me in my testing.
See Below:
import cx_Oracle #Place before other ODBC Clients
pconnObj  = cx_Oracle.connect('User/Password@TNSname')  #replace with your 
settings
pcur = pconnObj.cursor()
if pcur.execute('select * from dual'):
print "Yeah, it works!!!"
else:
print "Failure"
pcur.close()


I named the cursor the same as I had with PYODBC and no other changes were 
necessary
.fetchall()
.fetchone()
All worked

Returnd values requested also as PYODBC did:
prows = pcur.execute("SELECT name,TableID,LastAlteredDate  FROM myTable where 
tableType = '1' order by name").fetchall()
print "#rows=" + str(len(prows)) + '\n'
tscnt = len(prows)

Good Luck!
If you have more questions just ask.

Respectfully,

William Abdo
Software App Engineer II
NTT America, an NTT Communications Company
Office:   +1 561.912.2434
Email:w.a...@ntta.com
[https://rvip.team-center.net/externals/images/email/ntta.png]
[https://rvip.team-center.net/externals/images/email/ntta-twitter.png]
  [https://rvip.team-center.net/externals/images/email/ntta-linkedin.png] 
   
[https://rvip.team-center.net/externals/images/email/ntta-facebook.png] 
   
[https://rvip.team-center.net/externals/images/email/ntta-youtube.png] 






From: paul.hermeneu...@gmail.com [mailto:paul.hermeneu...@gmail.com]
Sent: Tuesday, December 15, 2015 10:43 AM
To: William Abdo
Cc: python-list@python.org
Subject: RE: Windows 10 and PYODBC


On Dec 13, 2015 7:20 AM, "William Abdo" 
mailto:w.a...@ntta.com>> wrote:
>
> Problem Resolved.
> I have fixed the Oracle connection issue under Windows 10 with cx_Oracle .
> PYODBC was only failing on the Oracle connection and worked fine on MS SQL 
> under Windows 10.

Please tell us what the fix is. Thanks.


This email message is intended for the use of the person to whom it has been 
sent, and may contain information that is confidential or legally protected. If 
you are not the intended recipient or have received this message in error, you 
are not authorized to copy, distribute, or otherwise use this message or its 
attachments. Please notify the sender immediately by return e-mail and 
permanently delete this message and any attachments. NTT America makes no 
warranty that this email is error or virus free. Thank you.


-- 
https://mail.python.org/mailman/listinfo/python-list


Re: subclassing collections.Counter

2015-12-15 Thread Pavlos Parissis
On 15/12/2015 05:11 μμ, Ian Kelly wrote:
> On Tue, Dec 15, 2015 at 8:49 AM, Pavlos Parissis
>  wrote:
>> Hi,
>>
>> I need to store values for metrics and return the average for some
>> and the sum for the rest. Thus, I thought I could extend
>> collections.Counter class by returning averages for some keys.
> 
> Leave Counter out of it, as this is not what it's designed for. Write
> a custom Metrics class, with each attribute being a pseudo-collection
> that maintains a sum or average.
> 

But then I will have to override a lot of magic methods, right?
What is the real problem of extending Counter in the way I did?

Cheers,
Pavlos



signature.asc
Description: OpenPGP digital signature
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: subclassing collections.Counter

2015-12-15 Thread Pavlos Parissis
On 15/12/2015 05:18 μμ, Pavlos Parissis wrote:
> On 15/12/2015 05:08 μμ, Peter Otten wrote:
>> Pavlos Parissis wrote:
>>
>>> I need to store values for metrics and return the average for some
>>> and the sum for the rest. Thus, I thought I could extend
>>> collections.Counter class by returning averages for some keys.
>>>
>>> My class modifies the update() to increment a counter and the
>>> __getitem__ to perform the calculation. But, I get RuntimeError: maximum
>>> recursion depth exceeded as I access an attribute inside
>>> __getitem__.
>>>
>>> Does anyone has an idea how I can achieve this?
>>
>>> class CounterExt(Counter):
>>
>>> def __getitem__(self, key):
>>> if (self.avg_metrics is not None and key in self.avg_metrics):
>>> return self[key] / self._counter
>>> else:
>>> return self[key]
>>
>> self[key] will call the CounterExt.__getitem__() method again. Use 
>> super().__getitem__(key) instead to invoke Counter.__getitem__().
>>
>>
>>
> 
> I applied your suggestion and worked!
>def __getitem__(self, key):
> if (self.avg_metrics is not None and key in self.avg_metrics):
> return super().__getitem__(key) / self.__counter
> else:
> return super().__getitem__(key)
> 
> Thank you very much,
> Pavlos
> 

Calling items() over the object doesn't call __getitem__ and I can't
find in the doc which method I need to change, any ideas?

Cheers,
Pavlos



signature.asc
Description: OpenPGP digital signature
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: cannot open file with non-ASCII filename

2015-12-15 Thread Ulli Horlacher
eryk sun  wrote:

> pyreadline looked promising for its extensive ctypes implementation of
> the Windows console API [1], wrapped by high-level methods such as
> peek, getchar, and getkeypress. It turns out it ignores the event
> sequences you need for alt+numpad input (used when a file is dragged
> into the console). You'd have to modify its console and keysyms
> modules to make it work. It would be a useful enhancement, so probably
> your patches would be accepted upstream.

Ehhh... I started Python programming some weeks ago and I know nearly
nothing about Windows. I am a UNIX and VMS guy :-)

I am far away from delivering patches for Windows system programming.

-- 
Ullrich Horlacher  Server und Virtualisierung
Rechenzentrum IZUS/TIK E-Mail: horlac...@tik.uni-stuttgart.de
Universitaet Stuttgart Tel:++49-711-68565868
Allmandring 30aFax:++49-711-682357
70550 Stuttgart (Germany)  WWW:http://www.tik.uni-stuttgart.de/
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: subclassing collections.Counter

2015-12-15 Thread Ian Kelly
On Tue, Dec 15, 2015 at 9:20 AM, Pavlos Parissis
 wrote:
> On 15/12/2015 05:11 μμ, Ian Kelly wrote:
>> On Tue, Dec 15, 2015 at 8:49 AM, Pavlos Parissis
>>  wrote:
>>> Hi,
>>>
>>> I need to store values for metrics and return the average for some
>>> and the sum for the rest. Thus, I thought I could extend
>>> collections.Counter class by returning averages for some keys.
>>
>> Leave Counter out of it, as this is not what it's designed for. Write
>> a custom Metrics class, with each attribute being a pseudo-collection
>> that maintains a sum or average.
>>
>
> But then I will have to override a lot of magic methods, right?
> What is the real problem of extending Counter in the way I did?

Only the ones that you have use for. So far, you haven't indicated
that you need any. All you said about your use case was "I need to
store values for metrics."

If you want your metrics container to act like a dict, then my
suggestion would be to just use a dict, with pseudo-collections for
the values as above.

> Calling items() over the object doesn't call __getitem__ and I can't
> find in the doc which method I need to change, any ideas?

You can find the source at
https://hg.python.org/cpython/file/3.5/Lib/collections/__init__.py.
Counter subclasses dict but doesn't override items, so if you want to
change the behavior of items then you'll probably have to override
items.
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: subclassing collections.Counter

2015-12-15 Thread Pavlos Parissis
On 15/12/2015 06:22 μμ, Ian Kelly wrote:
> On Tue, Dec 15, 2015 at 9:20 AM, Pavlos Parissis
>  wrote:
>> On 15/12/2015 05:11 μμ, Ian Kelly wrote:
>>> On Tue, Dec 15, 2015 at 8:49 AM, Pavlos Parissis
>>>  wrote:
 Hi,

 I need to store values for metrics and return the average for some
 and the sum for the rest. Thus, I thought I could extend
 collections.Counter class by returning averages for some keys.
>>>
>>> Leave Counter out of it, as this is not what it's designed for. Write
>>> a custom Metrics class, with each attribute being a pseudo-collection
>>> that maintains a sum or average.
>>>
>>
>> But then I will have to override a lot of magic methods, right?
>> What is the real problem of extending Counter in the way I did?
> 
> Only the ones that you have use for. So far, you haven't indicated
> that you need any. All you said about your use case was "I need to
> store values for metrics."
> 

I needed the update() to do the addition rather the override.

> If you want your metrics container to act like a dict, then my
> suggestion would be to just use a dict, with pseudo-collections for
> the values as above.
> 

If I understood you correctly, you are saying store all metrics in a
dict and have a counter key as well to store the times metrics are
pushed in, and then have a function to do the math. Am I right?

>> Calling items() over the object doesn't call __getitem__ and I can't
>> find in the doc which method I need to change, any ideas?
> 
> You can find the source at
> https://hg.python.org/cpython/file/3.5/Lib/collections/__init__.py.
> Counter subclasses dict but doesn't override items, so if you want to
> change the behavior of items then you'll probably have to override
> items.
> 

I thought that items() calls a magic method, but it is actually a method
itself. Thanks for highlight this.

Cheers,
Pavlos



signature.asc
Description: OpenPGP digital signature
-- 
https://mail.python.org/mailman/listinfo/python-list


netcdf 4 time

2015-12-15 Thread jorge . conrado



Hi,


I'm reading ndetcdf 4 data from NCEP Reanalyisis project.


I use to read the data: air_2010_2014_850hPa_d.nc

is four year daily data.

mport matplotlib.pylab as plt
from netCDF4 import Dataset, num2date, date2index, date2num
import pandas as pd
from mpl_toolkits.basemap import Basemap
import numpy as np
from jdcal import gcal2jd, jd2gcal

f = Dataset('air_2010_2014_850hPa_d.nc')

air = f.variables['air'][:]
time = f.variables['time']
lon = f.variables['lon'][:]
lat = f.variables['lat'][:]


lon, lat = np.meshgrid(lon, lat)


dates = num2date(time[:], time.units)


print dates


This is the date values:


[datetime.datetime(2010, 1, 1, 0, 0) datetime.datetime(2010, 1, 2, 0, 0)
 datetime.datetime(2010, 1, 3, 0, 0) ...,
 datetime.datetime(2014, 12, 29, 0, 0)
 datetime.datetime(2014, 12, 30, 0, 0)
 datetime.datetime(2014, 12, 31, 0, 0)]


Please, how can I get an arrya for each year, monht and day from the 
dates values and how can I get an specific day for my dates array.



Please anyone have any example of date2index for the netcdf 4.



This is the time for my netcdf data.


float64 time(time)
long_name: Time
delta_t: -00-01 00:00:00
avg_period: -00-01 00:00:00
standard_name: time
axis: T
units: hours since 1800-01-01 00:00:0.0
actual_range: [ 1840824.  1884624.]
unlimited dimensions: time
current shape = (1826,)
filling on, default _FillValue of 9.96920996839e+36 used




Thanks in advance,


Conrado
--
https://mail.python.org/mailman/listinfo/python-list


Re: subprocess.call with non-ASCII arguments?

2015-12-15 Thread Ulli Horlacher
Dennis Lee Bieber  wrote:

> Python has a zipfile library that is portable between OS. Along with
> libraries for gzip, bzip2, and tarfiles...

Ohh.. this is new to me!

https://docs.python.org/2/library/tarfile.html
https://docs.python.org/2/library/zipfile.html

What is missing in the documentation:
Is the tar/zip file generation done in memory or will it be written
directly to disk?

My program creates zip or tar files up to TB size. This cannot be done in
memory.


-- 
Ullrich Horlacher  Server und Virtualisierung
Rechenzentrum IZUS/TIK E-Mail: horlac...@tik.uni-stuttgart.de
Universitaet Stuttgart Tel:++49-711-68565868
Allmandring 30aFax:++49-711-682357
70550 Stuttgart (Germany)  WWW:http://www.tik.uni-stuttgart.de/
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: subclassing collections.Counter

2015-12-15 Thread Ian Kelly
On Tue, Dec 15, 2015 at 10:43 AM, Pavlos Parissis
 wrote:
>> If you want your metrics container to act like a dict, then my
>> suggestion would be to just use a dict, with pseudo-collections for
>> the values as above.
>>
>
> If I understood you correctly, you are saying store all metrics in a
> dict and have a counter key as well to store the times metrics are
> pushed in, and then have a function to do the math. Am I right?

That would work, although I was actually thinking of something like this:

class SummedMetric:
def __init__(self):
self.total = 0
self.count = 0

@property
def average(self):
return self.total / self.count

def add(self, value):
self.total += value
self.count += 1

metrics = {}
for metric_name in all_metrics:
metrics[metric_name] = SummedMetric()

For averaged metrics, look at metrics['f'].average, otherwise look at
metrics['f'].total.
-- 
https://mail.python.org/mailman/listinfo/python-list


RE: Windows 10 and PYODBC

2015-12-15 Thread paul.hermeneutic
On Dec 15, 2015 10:34 AM, "William Abdo"  wrote:
>
> Yes Paul Hermeneutic , that is correct.
>
> I tried everything I could  however,  I was unable to make PYODBC  talk
to Oracle under Windows 10.

It would be of help to everyone if you would file a bug report on the issue
tracker. http://bugs.python.org/
-- 
https://mail.python.org/mailman/listinfo/python-list


RE: Windows 10 and PYODBC

2015-12-15 Thread paul.hermeneutic
On Dec 15, 2015 12:00 PM, "William Abdo"  wrote:
>
> As you wish,
>
> [issue577] PYODBC will not talk to Oracle under Windows 10.

Where is this issue filed? I do not see it on http://bugs.python.org/
-- 
https://mail.python.org/mailman/listinfo/python-list


RE: Screenshots in Sphinx docs

2015-12-15 Thread Albert-Jan Roskam

> To: python-list@python.org
> From: tjre...@udel.edu
> Subject: Re: Screenshots in Sphinx docs
> Date: Mon, 14 Dec 2015 14:01:03 -0500
> 
> On 12/14/2015 11:31 AM, Albert-Jan Roskam wrote:
> 
> > I'd like to include up-to-date screenshots (of a tkinter app)
>  > into my Sphinx documentation.
> 
> If you manually take screenshots with *any* screen grabber and save in 
> an appropriate format, this is apparently trivial -- use the ..image 
> directive.  From the below, it appears that what you want is to have a 
> literally up-to-date screenshot taken automatically during the doc build.
> 
> This requires that one be able to automate getting the application into 
> the exact display state one wants to capture.  You can probably do that 
> with a tkinter app if you write it with that possibility in mind.  In 
> particular, you must keep a Python reference to every widget you want to 
> manipulate, even if not needed for normal program operation.
> 
> There is also an issue with grabbing the whole screen versus only a 
> program-specific window.

I need only a few screens. I think I will call my tkinter app with 
subprocess.Popen, wait until it's loaded,
grab the image, then kill it. Then I indeed wanted to use the ..image directive.

>  > This looks ok:
> > https://pypi.python.org/pypi/sphinxcontrib-programscreenshot
> 
> This (automatically) takes 'screenshots' on a no-screen (headless) *nix 
> system (during doc build) by redirecting X-windows output to a 
> pseudo-screen program.  Rather clever, and system-specific.
> 
> > BUT I need something that works on Windows (Python 2.7).
>  > Can any recommend an approach? I thought about using PIL:
> 
> Get the pillow fork/upgrade on pypi.

Thanks for the tip! So good ol' PIL is no longer maintained?

 
> > http://www.varesano.net/blog/fabio/capturing%20screen%20image%20python%20and%20pil%20windows
> 
> Or look into Windows screen grabber programs, of which there are many.
> 
> -- 
> Terry Jan Reedy
> 
> -- 
> https://mail.python.org/mailman/listinfo/python-list
  
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: subclassing collections.Counter

2015-12-15 Thread Pavlos Parissis
On 15/12/2015 06:55 μμ, Ian Kelly wrote:
> On Tue, Dec 15, 2015 at 10:43 AM, Pavlos Parissis
>  wrote:
>>> If you want your metrics container to act like a dict, then my
>>> suggestion would be to just use a dict, with pseudo-collections for
>>> the values as above.
>>>
>>
>> If I understood you correctly, you are saying store all metrics in a
>> dict and have a counter key as well to store the times metrics are
>> pushed in, and then have a function to do the math. Am I right?
> 
> That would work, although I was actually thinking of something like this:
> 
> class SummedMetric:
> def __init__(self):
> self.total = 0
> self.count = 0
> 
> @property
> def average(self):
> return self.total / self.count
> 
> def add(self, value):
> self.total += value
> self.count += 1
> 
> metrics = {}
> for metric_name in all_metrics:
> metrics[metric_name] = SummedMetric()
> 
> For averaged metrics, look at metrics['f'].average, otherwise look at
> metrics['f'].total.
> 

With this approach I will have for each metric 1 object, which could
cause performance issues for my case.

Let me bring some context on what I am trying to do here.
I want to provide a fast retrieval and processing of statistics metrics
for HAProxy.

HAProxy exposes stats over a UNIX socket(stats socket).
HAProxy is a multi-process daemon and each process can only be accessed
by a distinct stats socket. There isn't any shared memory for all these
processes. That means that if a frontend or backend is managed by more
than one processes, you have to collect metrics from all processes and
do the sum or average based on type of the metric.

stats are provided in a CSV format:
https://gist.github.com/unixsurfer/ba7e3bb3f3f79dcea686

there is 1 line per frontend and backend. For servers is a bit more
complicated.

When there are 100 lines per process, it is easy to do the work even in
setups with 24 processes(24 *100=2.4K lines). But, there are a lot of
cases where a stats socket will return 10K lines, due to the amount of
backends and servers in backends. This is 240K lines to process and
provide stats per 10secs or 5 secs.

My plan is to split the processing from the collection.
A program will connect to all UNIX sockets asynchronously and dump the
CSV to files, one per socket, and group them by EPOCH time.
It will dump all files under 1 directory which will have as name the
time of the retrieval.

Another program in multi-process mode[1], will pick those files and
parse them in sequentially to perform the aggregation. For this program
I needed the CounterExt.

I will try your approach as well as it is very simple and it does the
work with fewer lines:-) I will compare both in terms of performance and
select the fastest.

Thank you very much for your assistance, very much appreciated.



[1] pseudo-code
from multiprocessing import Process, Queue
import pyinotify

wm = pyinotify.WatchManager()  # Watch Manager
mask = pyinotify.IN_CREATE  # watched events

class EventHandler(pyinotify.ProcessEvent):
def __init__(self,  queue):
self.queue = queue

def process_IN_CREATE(self, event):
self.queue.put(event.pathname)

def work(queue):
while True:
job = queue.get()
if job == 'STOP':
break
print(job)

def main():
pnum = 10
queue = Queue()
plist = []
for i in range(pnum):
p = Process(target=work, args=(queue,))
p.start()
plist.append(p)

handler = EventHandler(queue)
notifier = pyinotify.Notifier(wm, handler)
wdd = wm.add_watch('/tmp/test', mask, rec=True)
notifier.loop()





signature.asc
Description: OpenPGP digital signature
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Help on code comprehension from an example project of pymc

2015-12-15 Thread Chris Angelico
On Wed, Dec 16, 2015 at 3:15 AM, Robert  wrote:
> When I review the code, I find 'data' in the original line:
>
> data = pymc.Normal('data', mu=model, tau=tau, value=z_obs, observed=True)
>
> has not been referenced thereafter.
> If I comment out the line as:
>
> #data = pymc.Normal('data', mu=model, tau=tau, value=z_obs, observed=True)
>
> the result is ugly different from the original.
>
> If I change it to:
>
> pymc.Normal('data', mu=model, tau=tau, value=z_obs, observed=True)
>
> it still runs as the original.
>
> This is quite different from my any other language experience.
>
> Could you help me on this?

What this suggests is that the call you're doing returns something (as
every function call must, unless it raises an exception or doesn't
terminate), but it also has side effects. You're ignoring the return
value (which is why "data = " doesn't affect your code), but you need
the side effects. Unfortunately this is the case with quite a lot of
Python's high end math/stats modules, presumably because their APIs
are lifted directly from other languages; have a look at pyplot, for
instance, where a more Pythonic API would be to instantiate a plot
object every time. (In that particular example, I believe that's an
option; but most examples just use the module-level default.)

It's my guess (based on skimming the docs) that the objects passed in,
including those referenced as function default arguments, are getting
mutated. But I'm no expert on pymc. It's equally likely that
module-level (global) state is being changed.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Help on code comprehension from an example project of pymc

2015-12-15 Thread Terry Reedy

On 12/15/2015 11:15 AM, Robert wrote:

Hi,

I find the useful small code project for me:
#https://users.obs.carnegiescience.edu/cburns/ipynbs/PyMC.html

It runs as expected.

When I review the code, I find 'data' in the original line:

data = pymc.Normal('data', mu=model, tau=tau, value=z_obs, observed=True)

has not been referenced thereafter.


If the function is called strictly for its side-effect, then it would be 
normal to not keep the 'return' value.  Code checkers will catch this 
and warn.  Just because code is make available, does not mean it follows 
the best style.  Perhaps the programmer though 'data' might be needed 
before writing the rest.



If I comment out the line as:

#data = pymc.Normal('data', mu=model, tau=tau, value=z_obs, observed=True)

the result is ugly different from the original.

If I change it to:

pymc.Normal('data', mu=model, tau=tau, value=z_obs, observed=True)

it still runs as the original.




--
Terry Jan Reedy

--
https://mail.python.org/mailman/listinfo/python-list


Re: How can I remove the first line of a multi-line string?

2015-12-15 Thread sgardne
On Monday, September 2, 2013 at 11:53:32 AM UTC-5, MRAB wrote:
> On 02/09/2013 17:12, Chris "Kwpolska" Warrick wrote:
> > On Mon, Sep 2, 2013 at 6:06 PM, Anthony Papillion  > gmail.com> wrote:
> >> Hello Everyone,
> >>
> >> I have a multi-line string and I need to remove the very first line from
> >> it. How can I do that? I looked at StringIO but I can't seem to figure
> >> out how to properly use it to remove the first line. Basically, I want
> >> to toss the first line but keep everything else.  Can anyone put me on
> >> the right path? I know it is probably easy but I'm still learning Python
> >> and don't have all the string functions down yet.
> >>
> >> Thanks,
> >> Anthony
> >> --
> >> http://mail.python.org/mailman/listinfo/python-list
> >
> > Use split() and join() methods of strings, along with slicing.  Like this:
> >
> >  fullstring = """foo
> >  bar
> >  baz"""
> >
> >  sansfirstline = '\n'.join(fullstring.split('\n')[1:])
> >
> > The last line does this:
> > 1. fullstring.split('\n') turns it into a list of ['foo', 'bar', 'baz']
> > 2. the [1:] slice removes the first element, making it ['bar', 'baz']
> > 3. Finally, '\n'.join() turns the list into a string separated by
> > newlines ("""bar
> > baz""")
> >
> Another way is to use .partition:
> 
>  >>> fullstring = """foo\nbar\nbaz"""
>  >>> fullstring.partition("\n")[2]
> 'bar\nbaz'

I realize this is very old, but thanks for posting this reply. I like this 
answer the best.
-- 
https://mail.python.org/mailman/listinfo/python-list


error reading api with urllib

2015-12-15 Thread simian336
Hi,

I am pretty new to python. I have a project reading an api with urllib. The 
problem is I have to sections of code almost exactly the same. The first url 
works great. They second one fails.

If I manually copy and paste the url in the browser ti works great.
The error I get back is...

Bad Request
b''
I have manipulated it a few ways and I sometimes she bad url type b'http

I have attempted to try to pass it as ascii and unicode but may not have been 
correct.

Thanks for any suggestions.

Simian




#Getting Individual Customer services Info
custinfostart='http://192.17.3.17:7780/dataservices/cdk?SP=md_cst.get_customer(%270003%27,%27'
custinforend ='%27,?,?int%20status)'
CustLink=custinfostart+curr_cust+custinforend
#print(CustLink)

req=urllib.request.Request(CustLink)
resp=urllib.request.urlopen(req)
respData=resp.read()
CustServ.write('Customer ID:'+curr_cust+'\n')
CustServ.write(str(respData)+'\n')

#*
#Getting Device Info
custinfostart='http://192.17.3.17:7780/dataservices/cdk?SP=md_dev.get_device(%270003%27,%272%27,%27'
custinforend ='%27,%27n%27,%271%27,%27100%27,?,?,?int status)'

#custinfostart="http://192.17.3.17:7780/dataservices/cdk?SP=md_dev.get_device('0003','2','"
#custinforend ="','n','1','100',?,?,?int status)"

print(custinfostart)
print(curr_cust)
print(custinforend)
CustLink=custinfostart+curr_cust+custinforend
#CustLink=CustLink.encode('ascii', 'ignore')
print('test')
print(CustLink)
 
# Code Fails Here ##

req=urllib.request.Request(CustLink)
#resp=urllib.request.urlopen(req)
try: resp=urllib.request.urlopen(req)
except urllib.error.URLError as e:
print(e.reason)
respData=resp.read()
print(respData)
sys.exit("Device Error")

respData=resp.read()
DeviceInfo.write(str(respData)+'\n')
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: error reading api with urllib

2015-12-15 Thread Simian
I added 

except urllib.error.HTTPError as e:
 print('HTTP Errpr')
 print('Error code: ', e.code)

to my try and I recieve...

400: ('Bad Request',
 'Bad request syntax or unsupported method'),

but processing the string with a browser works fine.

Simi
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: TypeError: 'float' object is not iterable

2015-12-15 Thread scarrera53
Someone stealing my points, I don't know how someone do it, but they had stolen 
some of my points.
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: TypeError: 'float' object is not iterable

2015-12-15 Thread Chris Angelico
On Wed, Dec 16, 2015 at 4:40 PM,   wrote:
> Someone stealing my points, I don't know how someone do it, but they had 
> stolen some of my points.

They were floating. You should fix them down.

Also, you're replying to a years-old thread with no context.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: wrapper api / proxying third party exception?

2015-12-15 Thread dieter
Benjamin Risher  writes:
> Ok, here's my issue.  I have a plugin framework I wrote that relies on a 
> third party library that I can't change.  As a part of that framework, I 
> wrote a small stub library to proxy calls to the third party library, so that 
> if their api ever changes, plugins written against the stub won't have to 
> change, I'll just have to update the back-half that interfaces with the 
> library.  I think this is called an adapter design pattern...?  
>
> plugin --> my api --> their api --> their library 
>
> It works, for the most part.  Most of my classes look similar to this
>
> class MYClass:
> def __init__(self, *args, **kwargs):
> self.__theirclass = TheirClass(*args, **kwargs)
>
> def __getattr__(self, attr):
> return getattr(self.__theirclass, attr)
>
> What I'm having issue with, is their exception class.  
>
> 1.  A plugin calls a stub in my library
> 2.  That stub calls the corresponding call in the third party library
> 3.  The call raises an exception from the library
> 4.  The plugin can't catch the exception without importing directly from the 
> third party library
>
> What is the correct way to allow plugins to import my stub exception, but 
> catch the ones raised by the library?

I would approach this as follows:

 * define an exception hiearchy for your wrapper with a common root class
   and document those hiearchy to be used for the plugins

 * in your wrapper, catch all exceptions

   - if it is one of your execptions, you know what to do

   - if it is a general Python exception, it likely indicates
 a programming error in the plugin or library -- handle
 as appropriately (personally, I would "reraise")

   - if it is something else, it likely indicates that
 the plugin violated its contract (towards your wrapper API) --
 handle as appropriate (again, I would "reraise").

-- 
https://mail.python.org/mailman/listinfo/python-list