Zipping a dictionary whose values are lists

[tkpmep]

I using Python 3.2 and have a dictionary
>>> d = {0:[1,2], 1:[1,2,3], 2:[1,2,3,4]}

whose values are lists I would like to zip into a list of tuples. If I 
explicitly write:
>>> list(zip([1,2], [1,2,3], [1,2,3,4])
[(1, 1, 1), (2, 2, 2)]

I get exactly what I want. On the other hand, I have tried

>>>list(zip(d))
[(0,), (1,), (2,)]

>>> list(zip(d.values()))
[([1, 2],), ([1, 2, 3],), ([1, 2, 3, 4],)]

>>> list(zip(d[i] for i in d))
[([1, 2],), ([1, 2, 3],), ([1, 2, 3, 4],)]

>>> list(zip(*d))
Traceback (most recent call last):
  File "", line 1, in 
list(zip(*d))
TypeError: zip argument #1 must support iteration

and nothing quite works. What am I doing wrong?

Sincerely

Thomas Philips
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Finding subsets for a robust regression

[tkpmep]

Thank you everyone, for your input. The help is much appreciated.

Thomas  Philips

--
http://mail.python.org/mailman/listinfo/python-list

Finding subsets for a robust regression

[tkpmep]

I have coded a robust (Theil-Sen) regression routine which takes as
inputs two lists of numbers, x and y, and returns a robust estimate of
the slope and intercept of the best robust straight line fit.

In a pre-processing phase, I create two new lists, x1 and y1; x1 has
only the unique values in x, and for each unique value in x1, y1 has
the median of all such values in x. My code follows, and it seems a
bit clumsy - is there a cleaner way to do it? By the way, I'd be more
than happy to share the code for the entire algorithm - just let me
know and I will post it here.

Thanks in advance

Thomas Philips

d = {}  #identify unique instances of x and y
for xx,yy in zip(x,y):
if xx in d:
d[xx].append(yy)
else:
d[xx] = [yy]

x1 = [] #unique instances of x and y
y1 = [] #median(y) for each unique value of x
for xx,yy in d.iteritems():
x1.append(xx)
l = len(yy)
if l == 1:
y1.append(yy[0])
else:
yy.sort()
y1.append( (yy[l//2-1] + yy[l//2])/2.0 if l % 2 == 0 else
yy[l//2] )
--
http://mail.python.org/mailman/listinfo/python-list

Detecting the first time I open/append to a file

[tkpmep]

I have a simulation that runs many times with different parameters,
and I want to aggregate the output into a  single file with one rub: I
want a header to be written only the first time. My program looks a
bit like this:

def main():
for param in range(10):
simulate(param)

def simulate(parameter):
'Lots of code followed by:
with open(summaryFn, 'ab') as f:
writer = csv.writer(f)
writer.writerow(header)
writer.writerow(Sigma)


If I can sense that the file is being created in the first iteration,
I can then use an if statement to decide whether or not I need to
write the header. Question: how can I tell if the file is being
created or if this its the first iteration? It's unrealistic to test
the value of the parameter as in the real problem, there are many
nested loops in main, and the bounds on the loop indices may change.

Thanks in advance for your assistance

Thomas  Philips
--
http://mail.python.org/mailman/listinfo/python-list

Re: Identifying the start of good data in a list

[tkpmep]

On Aug 26, 7:23 pm, Emile van Sebille <[EMAIL PROTECTED]> wrote:
> [EMAIL PROTECTED] wrote:
> > I have a list that starts with zeros, has sporadic data, and then has
> > good data. I define the point at  which the data turns good to be the
> > first index with a non-zero entry that is followed by at least 4
> > consecutive non-zero data items (i.e. a week's worth of non-zero
> > data). For example, if my list is [0, 0, 1, 0, 1, 2, 3, 4, 5, 6, 7, 8,
> > 9], I would define the point at which data turns good to be 4 (1
> > followed by 2, 3, 4, 5).
>
> > I have a simple algorithm to identify this changepoint, but it looks
> > crude: is there a cleaner, more elegant way to do this?
>
>  >>> for ii,dummy in enumerate(retHist):
> ...     if 0 not in retHist[ii:ii+5]:
> ...         break
>
>  >>> del retHist[:ii]
>
> Well, to the extent short and sweet is elegant...
>
> Emile

This is just what the doctor ordered. Thank you, everyone, for the
help.

Sincerely

Thomas Philips
--
http://mail.python.org/mailman/listinfo/python-list

Identifying the start of good data in a list

[tkpmep]

I have a list that starts with zeros, has sporadic data, and then has
good data. I define the point at  which the data turns good to be the
first index with a non-zero entry that is followed by at least 4
consecutive non-zero data items (i.e. a week's worth of non-zero
data). For example, if my list is [0, 0, 1, 0, 1, 2, 3, 4, 5, 6, 7, 8,
9], I would define the point at which data turns good to be 4 (1
followed by 2, 3, 4, 5).

I have a simple algorithm to identify this changepoint, but it looks
crude: is there a cleaner, more elegant way to do this?

flag = True
i=-1
j=0
while flag and i < len(retHist)-1:
i += 1
if retHist[i] == 0:
j = 0
else:
j += 1
if j == 5:
flag = False

del retHist[:i-4]

Thanks in advance for your help

Thomas Philips
--
http://mail.python.org/mailman/listinfo/python-list

Getting up and running with Python on a Mac

[tkpmep]

I've just bought an iMac (OS X 10.5.2, will almost immediately jump to
10.5.3), and am looking to install Python on it, and to use it with
XCode, Apple's IDE. Some googling suggests that a number of people
have had trouble getting Python to run satisfactorily on their Macs.
This is my first Mac, and I'd appreciate some guidance on what to do
(and what not to) when installing Python and potential problems to
keep an eye open for. I want to do a fair bit of scientific /
numerical computing, so it would seem that SAGE ot the Enthought
Python distribution would seem to be the most relevant  - I'd
appreciate your guidance on getting Python to run on a Mac with a
particular focus on these two distributions.

Thank you in advance

Thomas Philips
--
http://mail.python.org/mailman/listinfo/python-list

Rpy - partially blank R window

[tkpmep]

I have just installed R and Rpy, and am experiencing an odd problem
when driving R from Python - if I create a plot in R, the portion of
the plot window that lies under the IDLE window in which I type my
Python code remains blank. So if I type

>>> from rpy import *
>>> x = range(10)
>>> y = [i ** 2 for i in x]
>>> r.plot(x,y)


I get a plot of y vs.  x, but only the top right portion of the plot
shows any points, axes etc. The bottom left corner (the portion that
lay under the IDLE window) is blank. Is this a known problem?If so, is
there a fix?

Sincerely

Thomas Philips
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Cannot start RPy - need win32api

[tkpmep]

Thanks a mill - works like a charm!
-- 
http://mail.python.org/mailman/listinfo/python-list

Cannot start RPy - need win32api

[tkpmep]

I'm running Python 2.5.2 on Windows XP and need to interface with R,
so I downloaded the R 2.6.2 statistical package and installed it, and
did the same for RPy 1.02 (i made sure I got the version for Python
2.5 and R 2.62.). When I go to the Python command line and type

>>> from rpy import *
I get the following error message:

Traceback (most recent call last):
  File "", line 1, in 
from rpy import *
  File "C:\Python25\lib\site-packages\rpy.py", line 88, in 
import win32api
ImportError: No module named win32api
>>>

What on earth is win32api, where can I find it, and where ought I to
put it?

Thanks in advance

Thomas Philips
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Dispatching functions from a dictionary

[tkpmep]

Paul, George,

Thanks a mill - the help is greatly appreciated.

Thomas Philips

-- 
http://mail.python.org/mailman/listinfo/python-list

Dispatching functions from a dictionary

[tkpmep]

To keep a simulation tidy, I created a dispatcher that generates
random variables drawn from various distributions as follows:

import random

RVType = 1  #Type of random variable - pulled from RVDict

RVDict= {'1': random.betavariate(1,1), '2': random.expovariate(1),
 '3': random.gammavariate(1,1),  '4': random.gauss(0,1),
 '5': random.lognormvariate(1,1), '6':
random.paretovariate(1),
 '7': random.uniform( -1,1), '8':
random.weibullvariate(1,2) }

x = []
y=[]

rv = RVDict[str(RVType)]
for i in range(N):
x.append(rv)
y.append(rv)


Oddly, x and y get filled with a single value repeated N times. I
expected to get a different random number appear each time I called
rv ,but this does not happen. Instead, my first call to rv generates a
random number from the appropriate distribution, while subsequent
calls simply repeat the random number generated in the first call.
Where am I going wrong?

Thanks in advance for your help.

Sincerely


Thomas Philips

-- 
http://mail.python.org/mailman/listinfo/python-list

Order in which modules are imported

[tkpmep]

I imported two modules (random and matplotlib), and found that the
functions available to me from the random module depended on the order
in which the imports occured. In particular, if I import random first,
none of its functions seem to be available, while if I import it after
matplotlib, I seem to have access to all of its functions. What could
the problem be?

Thomas Philips


>>> import random
>>> from pylab import *
>>> x = random.uniform(0,1)

Traceback (most recent call last):
  File "", line 1, in 
x = random.uniform(0,1)
AttributeError: 'builtin_function_or_method' object has no attribute
'uniform'
>>> help(random)
Help on built-in function random_sample:

random_sample(...)
Return random floats in the half-open interval [0.0, 1.0).

random_sample(size=None) -> random values


In sharp contrast, I get what I expect when I reverse the order of the
imports.
>>> from pylab import *
>>> import random
>>> random.uniform(0,1)
0.75262941795069283
>>> help(random)
Help on module random:

NAME
random - Random variable generators.

FILE
c:\python25\lib\random.py
.
.
.




-- 
http://mail.python.org/mailman/listinfo/python-list

int/long bug in locale?

[tkpmep]

To pretty up some numbers stored as strings, I used locale to format
them with commas. I then found the following error:

>>> import locale

>>>locale.setlocale(locale.LC_ALL, 'English_United States.1252')
'English_United States.1252'

>>> locale.format('%d', float('2244012500.'), grouping = True)

Traceback (most recent call last):
  File "", line 1, in 
locale.format('%d', float('2244012500.'), grouping = True)
  File "C:\Python25\lib\locale.py", line 145, in format
formatted = percent % value
TypeError: int argument required

However, if the number is <= 2**31-1, it works just fine:
>>>locale.format('%d', float('224401250.'), grouping = True)
'224,401,250'

Interestingly, if I first convert the floats to ints, , the function
works just fine, even if the numbers exceed 2**31-1:
>>> locale.format('%d', int(float('2244012500.')), grouping = True)
'2,244,012,500'

Is there an int/long related bug lurking in locale?

Sincerely

Thomas Philips

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Downloading multiple csv files from a website

[tkpmep]

Mike,

Thanks for the pointers. I looked through the ASPN cookbook, but found
a more reliable (and easier to implement) way to get the files I want.
I downloaded GNU Wget from http://users.ugent.be/~bpuype/wget/( the
current version is 1.10.2), and then ran it from Python as follows

import os
rc = os.system('wget --debug --output-document="c:\\downloads\
\russell1000index_cyr.csv" --output-file=log.txt
http://www.russell.com/common/indexes/csvs/russell1000index_cyr.csv')


rc is the return code, and is 0 if the download succeeds. I also tried
the subprocess module

import subprocess
f = subprocess.Popen('wget --debug --output-document="c:\\downloads\
\russell1000index_cyr.csv" --output-file=log.txt
http://www.russell.com/common/indexes/csvs/russell1000index_cyr.csv')

This, too, works just fine. Wget does a lot more error-checking than
the recipe in the Python cookbook, does FTP as well as http, and
supports OpenSSL - its essentially a one-stop solution. In addition, I
can write batch files that do all the downloads without any need for
Python to be installed on the machine.

Thanks again

Thomas Philips

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Downloading multiple csv files from a website

[tkpmep]

Our systems administrator suggested that I try wget, a GNU utility
that is designed to pick up data. It might prove to be the easiest way
to get the data I want, and I am going to try that first.

Thanks again.

Thomas Philips


-- 
http://mail.python.org/mailman/listinfo/python-list

Downloading multiple csv files from a website

[tkpmep]

I'd like to download data from the website
http://www.russell.com/Indexes/performance/daily_values_US.asp. On
this web page, there are links to a number of .csv files, and I'd like
to download all of them automatically each day. The file names are not
visible on the page, but if I click on a link, a csv file opens in
Excel. I've searched this group and looked into urllib, but have not
found functions or code snippets that will allow me to download and
rename each file. Would someone kindly point me to appropriate
libraries/functions and/or code snippets that will get me started?

Thanks in advance

Thomas Philips

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: pyExcelerator bug?

[tkpmep]

John,

I'd be delighted to try xlwt (does it work under Python 2.4 and 2.5?)
I followed the link  to ...svn/xlwt/trunk and found a collection of
files, but no Windows installer. How do I install xlwt?

Thanks in advance

Thomas Philips


-- 
http://mail.python.org/mailman/listinfo/python-list

pyExcelerator bug?

[tkpmep]

My program creates three lists: the first has dates expressed as
strings, the second has floats that are strictly positive, and the
third has floats that are strictly negative. I have no trouble writing
the data in these lists to a .csv file using the csv module using the
following code.

outfile = file(fn + '.csv','wb')
writer  = csv.writer(outfile)
for i in range(len(dateList)):
 writer.writerow([dateList[i], posVals[i], negVals[i]])
outfile.close()

However, when I try to write to an Excel file using pyExcelerator (see
code below), the third list is not always written correctly - my
program sometimes writes positive numbers into the third column of the
spreadsheet. Is this a known bug? if so, is there a workaround? Is
pyExcelerator being developed any longer? My attempts to reach the
developer have gone nowhere.

w  = pyExcelerator.Workbook()
ws = w.add_sheet(fn + p)
for i,d in enumerate(dateList):
ws.write(i+1, 0, dateList[i])
ws.write(i+1, 1, posVals[i])
ws.write(i+1, 2, negVals[i])
w.save(fn+'.xls')

Sincerely

Thomas Philps

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Behavior of mutable class variables

[tkpmep]

Thanks for the insights. I solved the problem as follows: I created a
new class method called cleanUp, which resets NStocks to an empty list
and N1 to 0. Works like a charm - it's the first time I've used a
class method, and I immediately see its utility. Thanks again

class Stock(object):
NStocks = [] #Class variables
N1 = 0

@classmethod
def cleanUp(cls):
Stocks.NStocks = []
Stocks.N1 = 0



def simulation(N, par1, par2, idList, returnHistoryDir):

Stock.cleanUp()
results = ..
print results.



-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Behavior of mutable class variables

[tkpmep]

On May 9, 5:25 pm, [EMAIL PROTECTED] wrote:
> To test some theories, I created a new class variable, an int named
Diez,

Thanks. It is for precisely this reason that I added another class
variable - the immutable int N1. But this too keeps getting
incremented on subsequent calls to simulation( ). I also tried setting
the class variable N1 to 0 at the end of the simulation and also
deleting all the stocks in port (see below). Nothing works - and I
don't understand why.

def simulation(N, par1, par2, idList, returnHistoryDir):
port = []
for i in range(N):
port.append( Stock(idList[i], returnHistoryDir[idList[i]] )

p[0].NStocks = 0
del port[:]
results = ..
print results.



-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Behavior of mutable class variables

[tkpmep]

To test some theories, I created a new class variable, an int named
N1, which is not mutable. So my Stock class now looks as follows:

class Stock(object):
NStocks = [] #Class variables
N1 = 0


def __init__(self, id, returnHistory):
self.id = id
self.retHist = returnHistory

Stock.N1 += 1
for i in range(len(returnHistory)):
if len(Stock.NStocks) <= i   and retHist[i] != '':
Stock.NStocks.append(1)

elif len(Stock.NStocks) <= i and retHist[i] == '':
Stock.NStocks.append(0)

elif retHist[i] != '':
Stock.NStocks[i] +=1

I expect Stock.N1 to reset to zero at each call to simulation(), but
it does not - it keeps incrementing!
I then added a line to simulation( ) that deletes all the stocks at
the end of each simulation (see below). You guessed it - no change!
NStocks and N1 keep increasing! At this point I am thoroughly
mystified. What gives?

def simulation(N, par1, par2, idList, returnHistoryDir):
port = []
for i in range(N):
port.append( Stock(idList[i], returnHistoryDir[idList[i]] )

del port[:]
results = ..
print results.



-- 
http://mail.python.org/mailman/listinfo/python-list

Behavior of mutable class variables

[tkpmep]

I have written a program that runs portfolio simulations with
different parameters and prints the output, but am mystified by the
behavior of a mutable class variable. A simplified version of the
program follows - would you kindly help me understand why it behaves
the way it does.

The function main() reads some data and then repeatedly calls
simulation() with different parameter values. Each time the simulation
runs, it creates a collection of stocks, simulates their behavior and
prints the results.

Here's what I expect to happen each time simulation( ) is called: the
class variable NStocks for the class Stock is initialized to an empty
list, and is then built up by __init__ as stocks are added to the
portfolio. Unfortunately, ths is not what actuallly happens .NStocks
is initialized to an empty list and then built up as I expect on the
first call to simulation( ), but appears to persists between calls to
simulation( ).

Question: Why? Do I not create an entirely new list of stock objects
each time I enter simulation()? I am aware that mutable items can
behave in tricky ways, but am thoroughly mystified by the persistence
of NStocks between calls to simulation()

Sincerely

Thomas Philips

class Stock(object):
NStocks = [] #Class variable, NStocks[i] = number of
valid stocks at time i

def __init__(self, id, returnHistory):
self.id = id
self.retHist = returnHistory

   for i in range(len(returnHistory)):
if len(Stock.NStocks) <= i   and retHist[i] != '':
Stock.NStocks.append(1)

elif len(Stock.NStocks) <= i and retHist[i] == '':
Stock.NStocks.append(0)

elif retHist[i] != '':
Stock.NStocks[i] +=1


def simulation(N, par1, par2, idList, returnHistoryDir):
port = []
for i in range(N):
port.append( Stock(idList[i], returnHistoryDir[idList[i]] )

results = ..
print results.


def main():
   N, idList, returnHistoryDir= readData()
   for par1 in range(10):
for par2 in range(10):
simulation(N, par1, par2, idList, returnHistoryDir)

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Generalized range

[tkpmep]

Thanks - you have covered a fair bit of gorund here - I will modify
myRange taking your suggestions into account. The one suggestion that
I'm going to have to think through is repeatedly incrementing res.

I deliberately did not use this as repeated addition can cause
rounding errors to accumulate, making the loop run a little longer or
shorter than necessary. I thought I would be far less likely to run
into rounding issues with a multiplicative construct - hence my use of
epsilon, and my question about an appropriate value for it

-- 
http://mail.python.org/mailman/listinfo/python-list

Generalized range

[tkpmep]

I need to create ranges that can start and end with real numbers.
Searching this newsgroup brought me to a function that I then modified
as follows:

def myRange(iMin, iMax=None, iStep=1):
"""Extends range to real numbers. Wherever possible, use Python's
range .
   In other cases, make the behavior follow the spirit of Python's
range """
   epsilon = 1.e-8

if iMax == None and iStep == 1:
return range(int(iMin))

elif type(iMin).__name__.lower()  in ('int', 'long') and \
 type(iMax).__name__.lower()  in ('int', 'long') and \
 type(iStep).__name__.lower() in ('int', 'long') and iStep !=
0:
return range( iMin, iMax, iStep)

elif iMin <= iMax and iStep > 0:
return [ iMin+i*iStep for i in range( int(math.ceil((iMax -
iMin - epsilon)/iStep)) )]

elif iMin >= iMax and iStep < 0:
return [ iMin+i*iStep for i in range(-int(math.ceil((iMin -
iMax + epsilon)/iStep)) )]

else:
raise ValueError, 'Cannot construct a range with steps of size
' + str(iStep) + ' between ' + str(iMin) + ' and ' + str(iMax)


The one part of  my implementation that has me a bit queasy (i.e.
works in my current application, but I can see it misbehaving
elsewhere) is the addition/subtraction of a fixed epsilon to ensure
that my rounding goes the right way. A clean implementation would
modify epsilon based on the achievable level of precision given the
particular values of iMax, iMin and iStep. I suspect this requires a
detailed understanding of the implementation of floating point
arithmetic, and would appreciate hearing any thoughts you might have
on gilding this lily.

Sincerely

Thomas Philips

-- 
http://mail.python.org/mailman/listinfo/python-list

Using the Python interpreter

[tkpmep]

Instead of starting IDLE as I normally do, I started the Python
interpreter and tried to run a program. I got a Python prompt (>>>),
and then tried unsuccessfully to run a Python script named Script1.py
that runs perfectly well in IDLE. Here's what I did:

>>>Script1.py
Traceback (most recent call last):
  File "", line 1, in ?
NameError: name Script1 is not defined

>>>python Script1.py
  File "", line 1
   python Script1.py
SyntaxError: invalid syntax

I can load it (and have it execute) by typing

>>>import Script1
0
1
2
3
4

and if I edit it, I can then execute it by reloading it

>>>import Script1
0
1
2
3
4


But this seems contrived - is there no way to repeatedly run Script1
from the interpreter without reloading it?

Thomas Philips

-- 
http://mail.python.org/mailman/listinfo/python-list

Indentifying the LAST occurrence of an item in a list

[tkpmep]

For any list x, x.index(item) returns the index of the FIRST
occurrence of the item in x. Is there a simple way to identify the
LAST occurrence of an item in a list? My solution feels complex -
reverse the list, look for the first occurence of the item in the
reversed list, and then subtract its index from the length of the list
- 1, i.e.

LastOcc = len(x) - 1 - x[::-1].index(item)

Is there a simpler solution?

Thanks in advance

Thomas Philips

-- 
http://mail.python.org/mailman/listinfo/python-list

Finding the insertion point in a list

[tkpmep]

I have an ordered list e.g. x = [0, 100, 200, 1000], and given any
positive integer y, I want to determine its appropriate position in
the list (i.e the point at which I would have to insert it in order to
keep the list sorted. I can clearly do this with a series of if
statements:

if yx[i] for i in range(len(x)) ) - 1

But there  has to be a cleaner way, as the first approach is unwieldy
and does not adapt to changing list lengths, and the second is not
obvious to a casual reader of the code.

My list will typically have 2 to 5 items, so speed is not a huge
issue. I'd appreciate your guidance.

Sincerely

Thomas Philips

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Globbing files by their creation date

[tkpmep]

Thanks a mill - os.path.getctime(f) is what I needed. Unfortunately, my
attempts to turn the integer it returns into a date have failed.

>>> os.path.getctime(fn)#fn was created today, 1/17/2007
1168955503

I tried to convert this to a date object by typing
>>>datetime.date.fromordinal(1168955503)

Traceback (most recent call last):
  File "", line 1, in -toplevel-
datetime.date.fromordinal(1168955503)
ValueError: year is out of range

How can I do the conversion? I'm trying to identify all files that were
created after /MM/DD.

For a quick sanity check, I ran
>>> datetime.date.today().toordinal()
732693

which is orders of magnitude smaller than the number returned by
os.path.getctime(fn).

Thanks in advance for your help

Thomas Philips

-- 
http://mail.python.org/mailman/listinfo/python-list

Globbing files by their creation date

[tkpmep]

I'd like to create a list of all files in a directory that were created
after a certain date. How does one do this? I've used glob.glob to
create a list of all files whose name matches a substring, but I don't
see how I can use it to identify files by their creation date.

Thanks in advance for the assistance.

Thomas Philips

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: list looping error

[tkpmep]

What you really want to write is
for i in x:
for j in i:
print j

The outer loop iterates over the tuples in the list, while the inner
loop iterates over the elements of each tuple. So j (in your example)
is always an integer, and is therefore unsubscriptable, which is
exactly what the error message says.

Thomas Philips

-- 
http://mail.python.org/mailman/listinfo/python-list

SciPy Optimization syntax

[tkpmep]

I'm trying to optimize a function using SciPy's optimize.fmin, but am
clearly getting the syntax wrong, and would be grateful for some
guiidance. First, here's the function

def func(Y,x):
"""Y holds samples of a function sampled at t=-3,-2,-1,0,1,2,3.
Y[3]=0 always.
func returns the absolute value of the maximum NEGATIVE
error from a straight line fit with slope x and intercept 0"""

Y[0] = Y[0] - 3*x
Y[1] = Y[1] - 2*x
Y[2] = Y[2] - x
Y[3] =  0
Y[4] = Y[4] + x
Y[5] = Y[5] + 2*x
Y[6] = Y[6] + 3*x

error = abs(max(min(Y),0)

return 0

I'd now like to minimize this using optimize.fmin. I first defined
>>Y = [0, 0, 0, 0, 1, 2, 3]
>>x = 1

and then typed
>>optimize.fmin(func,  args=(Y,x) )

I expected the function to retun x=0 as the optimal value,  but instead
got the following error messsage:
Traceback (most recent call last):
  File "", line 1, in -toplevel-
optimize.fmin(func,args=(optionPnL,x))
TypeError: fmin() takes at least 2 non-keyword arguments (1 given)

I then tried
>>optimize.fmin(func,  x0 =x, args=(Y,x) )

and got a slightly different error message:
Traceback (most recent call last):
  File "", line 1, in -toplevel-
optimize.fmin(func,x0=x, args=(optionPnL,1))
  File "C:\Python24\lib\site-packages\scipy\optimize\optimize.py", line
176, in fmin
N = len(x0)
TypeError: len() of unsized object

What am I doing wrong, and what's the appropriate fix?

Thanks in advance

Thomas Philips

-- 
http://mail.python.org/mailman/listinfo/python-list

IronPython and scipy/pyExcelerator

[tkpmep]

I'm looking forward to the release IronPython, primarily for its IDE. I
currently use scipy and pyExcelerator to crunch numbers and write them
to Excel: does can these packages be used with IronPython as well?

Thanks in advance

Thomas Philips

-- 
http://mail.python.org/mailman/listinfo/python-list

PyExcelerator

[tkpmep]

I write data to Excel files using PyExcelerator 0.6.3.a and have done
so successfully for small files (10-15 cells). I'm experiencing an
error when writing a big chunk of data (10,000 cells) to Excel. By way
of comparison, the same data writes perfectly well to a csv file using
Python's built in csv module. I run the program in PyScripter, and the
traceback shows the following sequence of calls:

main  (my routine)
writeData(my routine)
save   Line 563
get_biff_data   Line 548
get_biff_data   Line 1357
__row_blocks_rec   Line 1276
get_cells_biff_data  Line 200
get_biff_data   Line 106

SystemError: frexp() result out of range

The line it stops at in get_biff_data is the line that starts with
packed =

def get_biff_data(self):
rk_encoded = 0

packed = struct.pack('http://mail.python.org/mailman/listinfo/python-list

Re: PyExcelerator: How does one close a file?

[tkpmep]

John,

I had spelled PyExcelerator with a capital P, but it worked just fine.
I then checked the directory it was installed in, and found it read
C:\Python24\Lib\site-packages\PyExcelerator. As soon as I changed the
directory name to C:\Python24\Lib\site-packages\pyExcelerator, my
programs stopped working! I then changed PyExcelerator to pyExcelerator
in my programs and they worked alll over again.

After much playing around, I discovered that I had incorrectly indented
the call to writeData, so that it ran only once, after a loop was
exited, and not for every iteration through the loop.

Thanks

Thomas Philips

-- 
http://mail.python.org/mailman/listinfo/python-list

PyExcelerator: How does one close a file?

[tkpmep]

I use PyExcelerator to write data into Excel files, and repeatedly call
the function writeData with different filenames fn from my main
routine. Unfortunately, only one file - the last one - is created. The
problem seems to lie with the fact that I do not close the existing
file before calling the function again, and the existing file is
therefore just saved with a new name. Is there a way to close a
PyExcelerator Excel file so that each call to the function results in a
new file? There is no close() method as far as I can tell.

def writeData(fn, Data):
"""Write the data into an Excel file named fn using PyExcelerator
Data is a list of real numbers"""
w=PyExcelerator.Workbook()
ws = w.add_sheet("Sheet1")
for i in range(len(Data)):
ws.write(i, 0, Data[i])
w.save(fn)

Thanks in advance

Thomas Philips

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Using StopIteration

[tkpmep]

This is just what the doctor ordered. Thanks, as always, everyone!

> By breaking out of the while loop as shown above.

> 
> Peter

-- 
http://mail.python.org/mailman/listinfo/python-list

Using StopIteration

[tkpmep]

I create list of files, open each file in turn, skip past all the blank
lines, and then process the first line that starts with a number (see
code below)

filenames=glob.glob("C:/*.txt")
for fn in filenames:
f =file(fn)
line = " "
while line[0] not in digits:
line = f.next()
ProcessLine(line)

If a file has only blank lines, the while loop terminates with a
StopIteration. How can I just close this file andd skip to the next
file if a StopIteration is raised? I tried the following:

filenames=glob.glob("C:/*.txt")
for fn in filenames:
f =file(fn)
line = " "
while line[0] not in digits:
try:
line = f.next()
except StopIteration:
f.close()
continue

ProcessLine(line)

but got only a ValueError: I/O operation on closed file   for   line =
f.next(). It appears that the continue is taking me back to the top of
the while loop. How can I get back to the top of the for loop?

Thanks in advance 

Thomas Philips

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Partially unpacking a sequence

[tkpmep]

Thank you, everyone, for resolving my question. At one point, while
trying to solve the problem, I typed

>>> y[1,3]
Traceback (most recent call last):
  File "", line 1, in ?
TypeError: list indices must be integers

The error message gave me no clue as to what I was doing wrong (in my
mind, I was just writing out the elements of a range), and I thought
perhaps that my inclusion of a comma was the problem. Perhaps a more
explicit error message would have helped.

Another solution, I suppose, is to use a list comprehension. Store the
indices in a list x. For example, let x = [1,5,4]. Then

a,b,c = [y[i] for i in x]

Thanks

Thomas Philips

-- 
http://mail.python.org/mailman/listinfo/python-list

Partially unpacking a sequence

[tkpmep]

I have a list y
>>>y
['20001201', 'ARRO', '04276410', '18.500', '19.500', '18.500',
'19.500', '224']

from which I want to extract only the 2nd and 4th item by partially
unpacking the list. So I tried
>>>a,b = y[2,4]
Traceback (most recent call last):
  File "", line 1, in ?
TypeError: list indices must be integers

Out of curiosity, I tried
>>>a,b = y[2:4]
>>>a
'04276410'
>>> b
'18.500'

Why does this work (to a point  - it gives me items 2 and 3, not 2 and
4 as I require) and not my first attempt? What is the right syntax to
use when partially upacking a sequence?

Thanks in advance

Thomas Philips

-- 
http://mail.python.org/mailman/listinfo/python-list

Using PyExcelerator

[tkpmep]

I have just installed PyExcelerator, and now want to use it to read
Excel spreadsheets with a variable number of rows and columns and with
multiple sheets. Unfortunately, no documentation seems to accompany
PyExcelerator. Does anyone know of a good source of documentations
and/or examples? The author provides some examples, but these tend to
involve writing data into spreadsheets one cell at a time.

Thanks in advance

Thomas Philips

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Installing PyExcelerator to write Excel files from Python

[tkpmep]

Thanks!!! I had a good laugh at myself after i got it working.

-- 
http://mail.python.org/mailman/listinfo/python-list

Installing PyExcelerator to write Excel files from Python

[tkpmep]

I downloaded PyExcelerator.zip as I need to write some data into Excel
files, and tried unsuccessfully to install it. I unzipped the files
into C:/Python24/Lib/site-packages/PyExcelerator, and in a python
command line window typed

>>>os.chdir("C:/Python24/Lib/site-packages/PyExcelerator")

followed by

>>>python ./setup.py install

exactly as stated in README.txt. However all I get is a SyntaxError.

I then tried various combinations  - with and without enclosing quotes,
with and without ./, and in all cases I get a SyntaxError. Has anyone
installed this program successfully? If so, what's the magic ingredient
for a successful install?

Thanks in advance

Thomas Philips

-- 
http://mail.python.org/mailman/listinfo/python-list

Question about csv writer

[tkpmep]

I expected the following code to work:

f = file(fn,"wb")
writer = csv.writer(f)
for i in range(IMax):
 writer.writerow([dates[i]].append([ReturnHistories[j][i] for j in
range(N)]))

but instead i got the following error message:
Error: sequence expected

However, if i modify the code to read
writer = csv.writer(f)
for i in range(IMax):
x = dates[i]
x.append([ReturnHistories[j][i] for j in range(N)])
 writer.writerow(x)

It works like a charm.

Question: Why does the first form not work?

Thomas Philips

-- 
http://mail.python.org/mailman/listinfo/python-list

Initializing a list of lists

[tkpmep]

I want to create a list of lists, each of which is identical, but which
can be modified independently i.e:

>>>x = [ [0], [0], [0] ]
>>> x[0].append(1)
>>> x
[[0, 1], [0], [0]]

The above construct works if I have only few items, but if I have many,
I'd prefer to write
>>> N =3
>>> x =N*[[0]]
>>> x
[[0], [0], [0]]

If I now try extending the lists indepently, I cannot, as they all
point to the same list object
>>> x[0].append(1)
>>> x
[[0, 1], [0, 1], [0, 1]]

Is there a simple way to create a list of independent lists?

Thanks in advance

Thomas Philips

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Computing correlations with SciPy

[tkpmep]

Tested it and it works like a charm! Thank you very much for fixing
this. Not knowing what an SVN is, I simply copied the code into the
appropriate library files and it works perfectly well.

May I suggest a simple enhancement: modify corrcoef so that if it is
fed two 1 dimensional arrays, it returns a scalar. cov does something
similar for covariances: if you feed it just one vector, it returns a
scalar, and if you feed it two, it returns the covariance matrix i.e:

>>> x = [1, 2, 3, 4, 5]

>>> z = [5, 4, 3, 2, 1]

>>> scipy.cov(x,z)
array([[ 2.5, -2.5],
   [-2.5,  2.5]])

>>> scipy.cov(x)
2.5

I suspect that the majority of users use corrcoef to obtain point
estimates of the covariance of two vectors, and relatively few will
estimate a covariance matrix, as this method tends not to be robust to
the presence of noise and/or errors in the data.

Thomas Philips

-- 
http://mail.python.org/mailman/listinfo/python-list

Computing correlations with SciPy

[tkpmep]

I want to compute the correlation between two sequences X and Y, and
tried using SciPy to do so without success.l Here's what I have, how
can I correct it?

>>> X = [1, 2, 3, 4, 5]
>>> Y = [5, 4, 3, 2, 1]
>>> import scipy
>>> scipy.corrcoef(X,Y)
Traceback (most recent call last):
  File "", line 1, in ?
  File "C:\Python24\Lib\site-packages\numpy\lib\function_base.py", line
671, in corrcoef
d = diag(c)
  File "C:\Python24\Lib\site-packages\numpy\lib\twodim_base.py", line
80, in diag
raise ValueError, "Input must be 1- or 2-d."
ValueError: Input must be 1- or 2-d.
>>> 

Thanks in advance

Thomas Philips

-- 
http://mail.python.org/mailman/listinfo/python-list

Getting started with Scipy/NumPy

[tkpmep]

I installed SciPy and NumPy (0.9.5, because 0.9.6 does not work with
the current version of SciPy), and had some teething troubles. I looked
around for help and observed that the tutorial is dated October 2004,
and is not as thorough as Python's documentation. Is there an
alternative source of information that lists all the functions and
their usage?

I tried using scipy.info to get information on the std function in the
stats libary, and ran into the following problem.


>>> x = [1, 2, 3, 4]
>>> import scipy
>>> scipy.std(x)
1.2909944487358056

>>> scipy.info(std)

Traceback (most recent call last):
  File "", line 1, in -toplevel-
scipy.info(std)
NameError: name 'std' is not defined

>>> scipy.info(stats)

Traceback (most recent call last):
  File "", line 1, in -toplevel-
scipy.info(stats)
NameError: name 'stats' is not defined

>>> scipy.info(stats.std)

Traceback (most recent call last):
  File "", line 1, in -toplevel-
scipy.info(stats.std)
NameError: name 'stats' is not defined



However, If I redo the import as follows I can get help on std:
>>> from scipy import *
>>> info(std)
 std(a, axis=0, dtype=None)

None
>>>

Question: why did it work this time and not on my first attempt?

Thanks in advance

Thomas Philips

-- 
http://mail.python.org/mailman/listinfo/python-list

Module imports

[tkpmep]

I have written a number of functions and stored them in a file named
myFunctions.py. This keeps all my functions under one roof, and if I
want to use or one or more of them in a program, I can start the
program off with

from myFunctions import f1,f2

Some of these functions require various math routines, so they in turn
have an import line

def f1(x,y):
from math import log
k = log(x)

I'd like to import all the math routines just once with a single import
at the top of myFunctions.py
and then rewrite all my functions to use this single import i.e.

import math

def f1(x,y):
k = math.log(x)

However, if I now import one of my functions into a program, i.e.
from myFunctions import f1,f2

f1 and f2 no longer have access the math functions they need. Is there
a way to make python automatically execute all the imports it finds at
the top of myFunctions.py so that all these system functions are
visible to the functions in myFunctions.py without having to import
them piecemeal?

Thomas Philips

-- 
http://mail.python.org/mailman/listinfo/python-list

Robust statistics and optimmization from Python

[tkpmep]

I use Python to generate a huge amount of data in a .csv file which I
then process using Excel. In particular, I use Excel's solver to solve
a number of non-linear equation, and then regress the results of
hundreds of calls to Solver against a set of known values, enabling me
to calibrate my model. This is a pain: i'd much rather perform all the
computations in Python and improve on Excels' regression as well.

Questions:
1. Is there a way to perform (or make a call to) a non-linear
optimization from Python?
2. Do Python packages for robust statistics (robust regression in
particular) exist. If so, which one would you recommend/

Thanks, as always, in advance for the guidance

Thomas Philips

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Reading just a few lines from a text file

[tkpmep]

Right now my code reads as follows:

infile=file(FileName)
for line in reversed(infile.readlines()):  #Search from the bottom up
 if int(line.split()[0]) == MyDate:
   Data= float(line.split()[-1])
   break
infile.close()

I have to read about 10,000 files, each with data. I'm looking to speed
up each individual file open/close cycle.

Thomas Philips

-- 
http://mail.python.org/mailman/listinfo/python-list

Reading just a few lines from a text file

[tkpmep]

I have a text file with many hundreds of lines of data. The data of
interest to me, however, resides at the bottom of the file, in the last
20 lines. Right now, I read the entire file and discard the stuff I
don't need. I'd like to speed up my program by reading only the last 20
lines. How do I do this?

Thomas Philips

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Using properties

[tkpmep]

Thanks for the help. I now understand it better. As Bruno points out, I
really don't need a property in this case, as my attribute is public,
and I will remove it.

Thomas Philips

-- 
http://mail.python.org/mailman/listinfo/python-list

Using properties

[tkpmep]

I have a class with a name attribute, which I want to modify using
property.The following code works just fine:

class Portfolio(object):

def __init__( self, name="Port1"):
self.name=name

def getname(self):
return self._name

def setname(self,newname="Port2"):
self._name=newname

name=property(getname,setname,None,None)

However, it no longer works if I modify getname and setname to

def getname(self):
return self.name

def setname(self,newname="Port2"):
self.name=newname

Why is it so critical to have getname and setname modify _name and not
name? The constructor does not make name a private attribute, so why do
getname and setname have to treat it as such?

Thomas Philips

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Using reverse iteration to clean up a list

[tkpmep]

Thank you all so much for the generous dollop of help: the dictionary
suggestion is particularly helpful. The problem arises as follows: A
software application stores the securities held in a portfolio in a
.csv file, one row per security, with three colulmns.

The first has a security identifier or descriptor (such as a ticker)
the second has a single letter that identifies the type of the
identifier (T for ticker, C for cusip etc.) and the third has the
number of shares. A typical line looks like this:

IBM, T, 500

I need to read in one or more portfolios and aggregate their holdings.
To do so, I read in the portfolios using the csv package, convert each
line to a list and then append it to a list of lists. Eventually the
list of lists contains all the securities, and can then be sorted and
aggregated.

I suppose I could convert it instead to a dictionary, and the most
natural key would be the first two items, i.e. a portfolio containing
just 500 shares of IBM ought to be represented as
{("IBM", "T") : 500 }

How can I translate the data I read in using csv.reader into a
dictionary?

Thomas Philips

-- 
http://mail.python.org/mailman/listinfo/python-list

Using reverse iteration to clean up a list

[tkpmep]

I have list of lists of the following form

L=[['A', 100], ['B', 300], ['A', 400], ['B', -100]]

I want to aggregate these lists, i.e. to reduce L to
L=[['A', 500], ['B', 200]] #500 = 100+400, 200=300-100

Here's how I have done it:
L.sort()
for i in range(len(L),0,-1):
if L[i-1][0]=L[i][0]:
L[i-1][2] += L[i][2]
del L[i]

Is there a simpler way to do this using the new reverse iteration
facility in Python 2.4?

Thomas Philips

-- 
http://mail.python.org/mailman/listinfo/python-list

PythonWin

[tkpmep]

I have a Python program that collects user input using

msg = "Enter the full path and name of the file to be processed: "
answer = raw_input(msg)

If I run it in IDLE, the question is splashed across the execution
window, and if it is long, simply wraps to the next line. Most
importantly, it is intelligible, because I see the entire message. I
enter my answer on the next line, and once again, I can see the entire
path and file name, even if it is longer than long.

However, if I run it in ActivePython's PythonWin, a small message box
pops up, with hardly any space to diplay msg and a smallish space into
which I can type my answer. How can I force PythonWin to get its input
from the execution window or to enlarge its message box sufficiently to
display the entire question?

Thomas Philips

-- 
http://mail.python.org/mailman/listinfo/python-list

PythonWin line spacing

[tkpmep]

I've just downloaded and installed ActivePython and am trying to
customize the PythonWin editor. Its line spacing seems to default to 2,
and consequently, I cannot see many lines of code on a screen. How do I
set the line spacing to 1?

Thomas Philips

-- 
http://mail.python.org/mailman/listinfo/python-list