Re: A tool for find dependencies relationships behind Python projects
On 2月23日, 上午12時32分, Hellmut Weber wrote:
> Hi Victor,
> I would be intereseted to use your tool ;-)
>
> My system is Sabayon-5.1 on Lenovo T61.
> Trying for the first time easy_install I get the following error:
>
>
>
> r...@sylvester ~ # easy_install gluttony
> Searching for gluttony
> Readinghttp://pypi.python.org/simple/gluttony/
> Readinghttp://code.google.com/p/python-gluttony/
> Best match: Gluttony 0.3
>
> Downloadinghttp://pypi.python.org/packages/source/G/Gluttony/Gluttony-0.3.zip#md...
>
> Processing Gluttony-0.3.zip
> Running Gluttony-0.3/setup.py -q bdist_egg --dist-dir
> /tmp/easy_install-uPz7qO/Gluttony-0.3/egg-dist-tmp-CJI_LD
> Traceback (most recent call last):
> File "/usr/bin/easy_install", line 9, in
> load_entry_point('distribute==0.6.8', 'console_scripts',
> 'easy_install')()
> File
> "/usr/lib64/python2.6/site-packages/setuptools/command/easy_install.py",
> line 1708, in main
> with_ei_usage(lambda:
> File
> "/usr/lib64/python2.6/site-packages/setuptools/command/easy_install.py",
> line 1696, in with_ei_usage
> return f()
> File
> "/usr/lib64/python2.6/site-packages/setuptools/command/easy_install.py",
> line 1712, in
> distclass=DistributionWithoutHelpCommands, **kw
> File "/usr/lib64/python2.6/distutils/core.py", line 152, in setup
> dist.run_commands()
> File "/usr/lib64/python2.6/distutils/dist.py", line 975, in run_commands
> self.run_command(cmd)
> File "/usr/lib64/python2.6/distutils/dist.py", line 995, in run_command
> cmd_obj.run()
> File
> "/usr/lib64/python2.6/site-packages/setuptools/command/easy_install.py",
> line 236, in run
> self.easy_install(spec, not self.no_deps)
> File
> "/usr/lib64/python2.6/site-packages/setuptools/command/easy_install.py",
> line 471, in easy_install
> return self.install_item(spec, dist.location, tmpdir, deps)
> File
> "/usr/lib64/python2.6/site-packages/setuptools/command/easy_install.py",
> line 501, in install_item
> dists = self.install_eggs(spec, download, tmpdir)
> File
> "/usr/lib64/python2.6/site-packages/setuptools/command/easy_install.py",
> line 680, in install_eggs
> return self.build_and_install(setup_script, setup_base)
> File
> "/usr/lib64/python2.6/site-packages/setuptools/command/easy_install.py",
> line 957, in build_and_install
> self.run_setup(setup_script, setup_base, args)
> File
> "/usr/lib64/python2.6/site-packages/setuptools/command/easy_install.py",
> line 946, in run_setup
> run_setup(setup_script, args)
> File "/usr/lib64/python2.6/site-packages/setuptools/sandbox.py", line
> 29, in run_setup
> lambda: execfile(
> File "/usr/lib64/python2.6/site-packages/setuptools/sandbox.py", line
> 70, in run
> return func()
> File "/usr/lib64/python2.6/site-packages/setuptools/sandbox.py", line
> 31, in
> {'__file__':setup_script, '__name__':'__main__'}
> File "setup.py", line 9, in
> File "/tmp/easy_install-uPz7qO/Gluttony-0.3/gluttony/__init__.py",
> line 1, in
> #
> File "/tmp/easy_install-uPz7qO/Gluttony-0.3/gluttony/gluttony.py",
> line 13, in
> ImportError: No module named pip.log
> r...@sylvester ~ #
>
>
>
> I emerged app-misc/pip, but that didn't help, the error remains the same
>
> What is missing?
>
> Any help appreciated
>
> Best regards
>
> Hellmut
>
> Am 19.02.2010 17:16, schrieb Victor Lin:
>
>
>
>
>
> > Hi,
>
> > I just wrote a tool for drawing dependencies relationships diagram of
> > Python project on Pypi. Here is the home page of the tool:
>
> >http://code.google.com/p/python-gluttony/
>
> > Some examples:
> > Sprox:
> >http://static.ez2learn.com/gluttony/sprox_dot.png
>
> > TurboGears2:
> >http://static.ez2learn.com/gluttony/tg2_dot.png
>
> > Hope this could be helpful :P
>
> > Regards.
> > Victor Lin.
>
> --
> Dr. Hellmut Weber m...@hellmutweber.de
> Degenfeldstraße 2 tel +49-89-3081172
> D-80803 München-Schwabing mobil +49-172-8450321
> please: No DOCs, no PPTs. why: tinyurl.com/cbgq
Hi,
That is a mistake I made in when I am making the distribute. Thanks
your reporting. I have already fixed the problem and released
Gluttony 0.4, and I tried to install that, it works fine.
Victor Lin.
--
http://mail.python.org/mailman/listinfo/python-list
A tool for find dependencies relationships behind Python projects
Hi, I just wrote a tool for drawing dependencies relationships diagram of Python project on Pypi. Here is the home page of the tool: http://code.google.com/p/python-gluttony/ Some examples: Sprox: http://static.ez2learn.com/gluttony/sprox_dot.png TurboGears2: http://static.ez2learn.com/gluttony/tg2_dot.png Hope this could be helpful :P Regards. Victor Lin. -- http://mail.python.org/mailman/listinfo/python-list
Memory usage problem of twisted server
Hi, I encountered an increasing memory usage problem of my twisted server. I have posted a question on stackoverflow: http://stackoverflow.com/questions/2100192/how-to-find-the-source-of-increasing-memory-usage-of-a-twisted-server I have read the article "Improving Python's Memory Allocator" ( http://evanjones.ca/memoryallocator/ ) and Python Memory Management ( http://evanjones.ca/python-memory.html ). And I now know little about how Python manages memory. I am wondering, is that the increasing memory usage problem of my audio broadcasting caused by the how python manage memory? In my server, there are lots of audio data chunks, they are not all in fixed size. For example, the size of audio stream might looks like this: ... chunk size 822 chunk size 878 chunk size 1690 chunk size 1659 chunk size 1643 chunk size 1952 chunk size 373 chunk size 763 chunk size 1535 chunk size 1665 ... All of those chunks are str object. The size of chunks depend on mp3 encoder. You can see most the size of those chunks is bigger than 256 bytes, so that Python uses malloc to allocate spaces for those chunks. In my mind, I saw those chunks occupy different place in heap, all of them are not in fixed size, so that even when the chunk is freed, it is difficult for malloc to find the right place for the new chunk. As the result, there are so many chunks placed in large range of heap, it is difficult for OS to swap pages out. Then the RSS is always high. Is that my guessing correct? How can I monitor the memory allocation of Python? Is there any available tools for this? Or is that modifying the Python source code and recompiling the only way to monitor allocation of memory? Thanks. Victor Lin. -- http://mail.python.org/mailman/listinfo/python-list
Any c header parser for generate ctypes module?
Hi, I am writing python binding for some c library. It is really a super boring job. Copy... paste... modify.. copy paste...modify I am wondering, I am a programmer, why I can't do this job like a programmer? So I think the best way to write binding for those c libraries, is to write a parser that parse header of c library and generate ctypes python module automatically. My question is, is there any available tools can achieve this? If not, what tool can I use to such a job easily. I need a c parser, is there any C parser written in python? Thanks. Victor Lin. -- http://mail.python.org/mailman/listinfo/python-list
Re: Callback from c thread with ctypes
On 3月8日, 下午10時20分, Christian Heimes wrote: > Victor Lin wrote: > > Hi, > > > I am going to develop a c library binding with ctypes. That c library > > will call callback from worker threads it created. Here comes the > > problem : Will the GIL be acquired before it goes into Python > > function? > > > I got a little try.. > > > DSPPROC = WINFUNCTYPE(None, DWORD, DWORD, c_void_p, DWORD, c_void_p) > > > def test(handle, channel, buffer, length, user): > > print handle, channel, buffer, length, user > > > dsp = BASS_ChannelSetDSP(stream, DSPPROC(test), None, 123) > > > I got "access violation" when I run it... It seems that the ctypes > > did't acquire GIL before it call the python callback. As the document > > says. > > > WINFUNCTYPE will release GIL during the call > > > But it does not mention callback about Python function? How about a > > call from another thread? Could somebody help me? > > You can't call Python code from an arbitrary thread. Before you are > allowed to call a Python function from a thread, you must enable > Python's threading subsystem and register the thread. Python need to > store information (like the exception information) in a thread local > variable (TLS). > > I can't tell you how to register your thread with Python, though. > > Christian I know I have to call PyEval_InitThreads if my module create threads that will contact python stuff, for example, call a python callback function from threads. But however, it is ctypes. I have no idea should I do that for the imported dll? If it is, how? Thanks. -- http://mail.python.org/mailman/listinfo/python-list
Re: Callback from c thread with ctypes
On 3月8日, 下午9時56分, "Diez B. Roggisch" wrote: > Victor Lin schrieb: > > > > > Hi, > > > I am going to develop a c library binding with ctypes. That c library > > will call callback from worker threads it created. Here comes the > > problem : Will the GIL be acquired before it goes into Python > > function? > > > I got a little try.. > > > DSPPROC = WINFUNCTYPE(None, DWORD, DWORD, c_void_p, DWORD, c_void_p) > > > def test(handle, channel, buffer, length, user): > > print handle, channel, buffer, length, user > > > dsp = BASS_ChannelSetDSP(stream, DSPPROC(test), None, 123) > > > I got "access violation" when I run it... It seems that the ctypes > > did't acquire GIL before it call the python callback. As the document > > says. > > > WINFUNCTYPE will release GIL during the call > > > But it does not mention callback about Python function? How about a > > call from another thread? Could somebody help me? > > The releasing takes only place when entering a c-function. A > python-callback acquires the GIL, see callbacks.c in the python source. > > The access violation has nothing to do with that I presume, that's just > a general programming error as it happens with ctypes during development. > > Diez Hi, Thanks your replying, I try it again, found that my program only crash when I call it from python's IDLE. If I click it, namely execute it with python directly, it works fine. Why the program will crash within IDLE? -- http://mail.python.org/mailman/listinfo/python-list
Callback from c thread with ctypes
Hi, I am going to develop a c library binding with ctypes. That c library will call callback from worker threads it created. Here comes the problem : Will the GIL be acquired before it goes into Python function? I got a little try.. DSPPROC = WINFUNCTYPE(None, DWORD, DWORD, c_void_p, DWORD, c_void_p) def test(handle, channel, buffer, length, user): print handle, channel, buffer, length, user dsp = BASS_ChannelSetDSP(stream, DSPPROC(test), None, 123) I got "access violation" when I run it... It seems that the ctypes did't acquire GIL before it call the python callback. As the document says. WINFUNCTYPE will release GIL during the call But it does not mention callback about Python function? How about a call from another thread? Could somebody help me? Thanks. Victor Lin. -- http://mail.python.org/mailman/listinfo/python-list
How to debug deadlock?
Hi, I am developing a multi-threading application, I encounter a deadlock. I use Visual C++ Express 2008 to trace the program. Once the deadlock occurs, I just pause the program and trace. I found that when deadlock occurs, there will be two threads called python from my C++ extension. All of them use Queue in python code, so I guess the deadlock might caused by Queue. But however, once the extension goes into python code, I can't see nothing but asm code and binary from the VC++ debugger. I would like to know are there any way to dump the call stack of python code after I paused the program? And how can I know what lock are there in threads caused the deadlock? Thanks. Victor Lin. -- http://mail.python.org/mailman/listinfo/python-list
Re: How to call python from a foreign language thread (C++)
On 2月4日, 上午2時44分, Philip Semanchuk wrote:
> On Feb 3, 2009, at 12:51 PM, Victor Lin wrote:
>
>
>
> > It seems that my program can't call to Python's function from thread
> > directly, because there is Global Interpreter Lock. The python's GIL
> > is so complex, I have no idea how it works. I'm sorry, what I can do
> > is to ask. My question is. What should I do before and after I call a
> > Python function from threads?
>
> Hi Victor,
> I asked a similar question yesterday and have gotten no response yet
> -- I hope you'll have better luck. I'm writing a C extension that
> wants to implement a callback in a new thread. Here's what I think
> I've learned from reading the threads API doc. Please note that this
> is a classic case of the blind leading the blind! I'm sure some (most?
> all?) of the ideas below are wrong, but I'm hoping that thinking
> through some of this "out loud" will help both of us. Or maybe some
> more knowledgeable person will take pity on/be appalled by my
> ignorance and come to our rescue. =)
>
> Python's infamous GIL doesn't exist when a program is single-threaded.
> Before a new thread is created, the main thread must call
> PyEval_InitThreads() to create the GIL. However, "It is not safe to
> call this function when it is unknown which thread (if any) currently
> has the global interpreter lock." Therefore my extension must do this:
>
> i_have_the_gil = 0;
> if (!PyEval_ThreadsInitialized()) {
> PyEval_InitThreads();
> /* "...when this function initializes the lock, it also acquires
> it." */
> i_have_the_gil = 1;
>
> }
>
> That ensures that the GIL is created.
>
> My extension will be calling from a newly-created C thread. The
> *Python* thread doesn't exist yet; so next I have to create it.
> Therefore --
>
> if (!i_have_the_gil)
> PyEval_AcquireLock();
>
> // Might not actually be the main thread but conceptually
> // it is OK to assume so here.
> main_thread = PyThreadState_Get();
>
> callback_thread = PyThreadState_New(main_thread->interp);
>
> PyThreadState_Swap(callback_thread);
>
> gstate = PyGILState_Ensure();
>
> call_callback_function();
>
> // Now unwind the above
>
> PyGILState_Release(gstate);
>
> PyThreadState_Swap(main_thread);
>
> PyThreadState_Clear(callback_thread);
>
> PyThreadState_Delete(callback_thread);
>
> PyEval_ReleaseLock();
>
> I haven't yet tested this! But hopefully it is the right idea and just
> needs a little fine tuning.
>
> I found this discussion
> useful:http://mail.python.org/pipermail/python-list/2006-November/413088.html
>
> It includes the quote, "The current thread state API doc, as you read
> it from top to bottom now, is in fact totally confusing for anyone who
> didn't develop Python himself"
>
> I like! =)
>
> Cheers
> Philip
Hi Philip,
It does not work. But however, thanks your help. I have tired so many
methods to do. But it
crash...crash..deadlock...deadlock..crash...crash... I have no any
tried success. I am going crazy. Could someone help me, thanks.
--
http://mail.python.org/mailman/listinfo/python-list
How to call python from a foreign language thread (C++)
Hi,
I am developing a program that use DirectShow to grab audio data from
media files. DirectShow use thread to pass audio data to the callback
function in my program, and I let that callback function call another
function in Python.
I use Boost.Python to wrapper my library, the callback function :
class PythonCallback {
private:
object m_Function;
public:
PythonCallback(object obj)
: m_Function(obj)
{}
void operator() (double time, const AudioData &data) {
// Call the callback function in python
m_Function(time, data);
}
};
Here comes the problem, a thread of DirectShow calls my
PythonCallback, namely, call the function in Python. Once it calls, my
program just crash. I found this should be threading problem. Then I
found this document:
http://docs.python.org/c-api/init.html
It seems that my program can't call to Python's function from thread
directly, because there is Global Interpreter Lock. The python's GIL
is so complex, I have no idea how it works. I'm sorry, what I can do
is to ask. My question is. What should I do before and after I call a
Python function from threads?
It may looks like this.
void operator() (double time, const AudioData &data) {
// acquire lock
m_Function(time, data);
// release lock
}
Thanks.
Victor Lin.
--
http://mail.python.org/mailman/listinfo/python-list
Logging library unicode problem
Hi,
I'm writting a application using python standard logging system. I
encounter some problem with unicode message passed to logging library.
I found that unicode message will be messed up by logging handler.
piese of StreamHandler:
try:
self.stream.write(fs % msg)
except UnicodeError:
self.stream.write(fs % msg.encode("UTF-8"))
It just write the message to stream. If there is some unicode error,
it would rewrite msg with utf8 encoding.
I write some code to try:
import sys
print u'中文字測試'
print sys.stdout.encoding
sys.stdout.write(u'中文')
result of that program:
中文字測試
cp950
Traceback (most recent call last):
File "update_stockprice.py", line 92, in
sys.stdout.write(u'銝剜?')
UnicodeEncodeError: 'ascii' codec can't encode characters in position
0-1: ordin
al not in range(128)
It show that
1. print statement encode what it get with stream.encoding?
2. stream.write don't do anything like encoding, just write it
(because it might be binary data?)
So the problem is : the StreamHandler of standard logging library use
stream.write to log message, if there is unicode error, unicode string
will be encode to utf8. This behavior mess my unicode up.
Here I modify the code of StreamHandler:
try:
print >> self.stream, msg
#self.stream.write(fs % msg)
except UnicodeError:
self.stream.write(fs % msg.encode("UTF-8"))
I replace stream.write with print statement, so that it will try to
use stream.encoding to encode msg. Now everything works fine.
My question is :
Could the behavior of StreamHandler be considered as a bug?
If it is, how to report this bug?
Is my solution correct?
Are there any side effect will caused by doing so?
If the code I write is fine, and solve that problem, how to report it
to Python's project?
I think this could be helpful for people who also encountered this
problem.
Thanks.
Victor Lin.
--
http://mail.python.org/mailman/listinfo/python-list
How to redirect operation methods to some sepcific method easily?
Hi,
I'd like to write some class that can help me build reusable formula
easily, some simple code like this.
# -*- coding: utf8 -*-
class OperationResult:
def __init__(self, left, right):
self.dataSource = dataSource
def __add__(self, other):
self.dataSource.stack.append('+')
return self
def __div__(self, other):
self.dataSource.stack.append('/')
return self
class Operand:
def __init__(self, dataSource, name):
self.dataSource = dataSource
self.name = name
def __add__(self, other):
self.dataSource.stack.append(self)
self.dataSource.stack.append(other)
self.dataSource.stack.append('+')
return OperationResult(self.dataSource)
class DataSource:
def __init__(self):
self.stack = []
def __getitem__(self, key):
return Operand(self, key)
def ROE(dataSource):
a = dataSource[u'股東權益總額']
b = dataSource[u'每股淨額']
c = dataSource[u'當季平均市價']
return (a + b) / c
Now I can use ROE formula easily, it will return a object that contain
formula stack like this
[a, b, +, c, /]
And then I can get real data of those items from database or other
place. The formula depends on nothing. It don't care where to get data
and how to calculate it. I can reuse it easily.
Now, here comes the problem : I have to override all the operation
methods, such as __add__ and __mul__.
I know the most stupid way is just to write all of them like this.
def __add__(self, other):
self.leftOperation('add', other)
def __mul__(self, other):
self.leftOperation('mul', other)
But I don't want to do that in this stupid way. I want to use little
code to redirect these operation methods to some specific method with
it's name.
What can I do?
Thanks.
--
http://mail.python.org/mailman/listinfo/python-list
Problem with import
Hi, I got some problem with import. I have some python file that should be executed. Something like unit-test or other small tool. The project is not so small. So I want to separate these files into different packages. Might like this: project/ run_task.py pkg1/ __init__.py run_lettle_task.py module_a.py module_b.py test1.py pkg2/ __init__.py module_c.py module_d.py test2.py Now here comes the problem: How to import these file correctly? For example: # file module_a.py - # This import force who run and import this file most in project/ this directory import pkg2.module_c # file test1.py # oops! If we run this file, # we got a error here, because this file is not in project/ import module_c # do some unittest - As you see, because I use import like this "import pkg2.module_c". This statement force python files which want to import and use this module should be in project. But I don't want to put all the files needed to execute in project. Because there are so many files and tests need to execute. If I put so many different things in project, to separate files to packages would be meaningless. What can I do? Actually, I know I can modify path to make python to search the project directory. Then in test1 or test2 I can import module_c. But here comes another problem: If I move my directory, I have to modify the path again. This is not a library, they are some programs to do specific task. I think they should be executable in every where I move or copy the directory to. Victor Lin. -- http://mail.python.org/mailman/listinfo/python-list
Re: Recursion limit of pickle?
On 2月10日, 上午11時42分, "Gabriel Genellina" <[EMAIL PROTECTED]>
wrote:
> En Sat, 09 Feb 2008 09:49:46 -0200, Victor Lin <[EMAIL PROTECTED]>
> escribi�:
>
> > I encounter a problem with pickle.
> > I download a html from:
>
> >http://www.amazon.com/Magellan-Maestro-4040-Widescreen-Navigator/dp/B...
>
> > and parse it with BeautifulSoup.
> > This page is very huge.
> > When I use pickle to dump it, a RuntimeError: maximum recursion depth
> > exceeded occur.
>
> BeautifulSoup objects usually aren't pickleable, independently of your
> recursion error.
But I pickle and unpickle other soup objects successfully.
Only this object seems too deep to pickle.
>
> py> import pickle
> py> import BeautifulSoup
> py> soup = BeautifulSoup.BeautifulSoup("Hello, world!")
> py> print pickle.dumps(soup)
> Traceback (most recent call last):
> ...
> TypeError: 'NoneType' object is not callable
> py>
>
> Why do you want to pickle it? Store the downloaded page instead, and
> rebuild the BeautifulSoup object later when needed.
>
> --
> Gabriel Genellina
Because parsing html cost a lots of cpu time. So I want to cache soup
object as file. If I have to get same page, I can get it from cache
file, even the parsed soup file. My program's bottleneck is on parsing
html, so if I can parse once and unpickle them later, it could save a
lots of time.
--
http://mail.python.org/mailman/listinfo/python-list
Recursion limit of pickle?
Hi,
I encounter a problem with pickle.
I download a html from:
http://www.amazon.com/Magellan-Maestro-4040-Widescreen-Navigator/dp/B000NMKHW6/ref=sr_1_2?ie=UTF8&s=electronics&qid=1202541889&sr=1-2
and parse it with BeautifulSoup.
This page is very huge.
When I use pickle to dump it, a RuntimeError: maximum recursion depth
exceeded occur.
I think it is cause by this problem at first :
http://bugs.python.org/issue1757062
But and then I do not think so, because I log recursion call of pickle
in file
I found that the recursion limit is exceeded in mid-way to expand
whole the BeautifulSoup object.
Not repeat to call some methods.
This is the code for test.
from BeautifulSoup import *
import pickle as pickle
import urllib
doc = urllib.urlopen('http://www.amazon.com/Magellan-Maestro-4040-
Widescreen-Navigator/dp/B000NMKHW6/ref=sr_1_2?
ie=UTF8&s=electronics&qid=1202541889&sr=1-2')
import sys
sys.setrecursionlimit(4)
soup = BeautifulSoup(doc)
print pickle.dumps(soup)
---
What I want to ask is: Is this cause by the limit of recursion limit
and stack size?
I had tired cPickle at first, and then I try pickle, cPickle just stop
running program without any message.
I think it is also implement with recursion way, and it also over flow
stack when dumping soup.
Are there any version of pickle that implement with no-recursion way?
Thanks.
Victor Lin.
--
http://mail.python.org/mailman/listinfo/python-list
Strange sqlite3 library behavior
Now I am now developing a program that base on sqlite3 in python.
But there is a strange problem.
That is, all data I insert into sqlite database do not goes into file
in disk.
It is really strange
Why do these data just keep in memory and discarded?
All things that really store in file is the table.
If I create a table, it would appears in the sqlite file.
This is the example :
import sqlite3
createSyntax = """CREATE TABLE IF NOT EXISTS category(
id INTEGER NOT NULL PRIMARY KEY
)"""
createSyntax2 = """CREATE TABLE IF NOT EXISTS category2(
id INTEGER NOT NULL PRIMARY KEY
)"""
insertSyntax = """INSERT INTO category VALUES (1234)"""
db = sqlite3.connect('test.db')
cur = db.cursor()
cur.execute(createSyntax)
cur.execute(insertSyntax)
# only by doing so can let inserted data store in file
cur.execute(createSyntax2)
cur.execute('select * from category')
print cur.fetchall()
db.close()
raw_input()
After I tried some way. I found that create table force data goes into
file. Otherwise all data would just keep in memory and discarded with
the program's ending.
Why sqlite3 just keep inserted data in memory? And how to force
inserted data into file?
Thanks.
Victor Lin.
--
http://mail.python.org/mailman/listinfo/python-list
