Re: [Python-Dev] Fwd: Broken link to download (Mac OS X)
> What happened to the big-ass computer farm for Python which was > being put together by someone at (I think) Michigan State? That sounds a lot like Snakebite (www.snakebite.org), which is still... uhhh, a work in progress ;-) We've run into an issue recently that's thwarted progress, but that'll hopefully be resolved in the next couple of weeks. And then... full steam ahead! Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: Broken link to download (Mac OS X)
> > What happened to the big-ass computer farm for Python which was > > being put together by someone at (I think) Michigan State? > > That sounds a lot like Snakebite (www.snakebite.org), which is > still... uhhh, a work in progress ;-) Actually, for those that are interested, here's a copy of the presentation I gave at the Testing in Python session at PyCon a few months ago: http://www.snakebite.org/presentations/snakebite-pycon2010-tip.pptx (Office 2007-2010) http://www.snakebite.org/presentations/snakebite-pycon2010-tip.ppt (Office 97-2003) If anything, it'll shed some light on all the unforeseen issues we've been running into since the project's inception. The presentation is a little out of date -- I spent three months earlier this year on the network and it's definitely in the most respectable state it's been in yet. Coupla' photos for those that are interested: http://snakebite.org/images/IMG_4384.JPG http://snakebite.org/images/IMG_4392.JPG http://snakebite.org/images/IMG_4393.JPG http://snakebite.org/images/IMG_4394.JPG http://snakebite.org/images/IMG_4395.JPG http://snakebite.org/images/IMG_4396.JPG http://snakebite.org/images/IMG_4401.JPG http://snakebite.org/images/IMG_4402.JPG http://snakebite.org/images/IMG_4403.JPG http://snakebite.org/images/IMG_4405.JPG http://snakebite.org/images/IMG_4410.JPG http://snakebite.org/images/IMG_4418.JPG http://snakebite.org/images/IMG_4424.JPG http://snakebite.org/images/IMG_4425.JPG We've got three racks filled to the brim with all sorts of servers: - 4xItanium 2 @ 1.5GHz, 16GB RAM, HP-UX 11iv3 - 4xItanium 2 @ 1.5GHz, 30GB RAM, RHEL 5.3 - 2xUltraSPARC III 900MHz, 8GB, Solaris 10 (file/zfs/nfs server -- 16x146GB 2Gb FC) - 2xUltraSPARC III 1.2GHz, 4GB, Solaris 10 - 2xPA-RISC 875MHz, 8GB, HP-UX 11iv1 - 4 AIX boxes w/ 2x1.5GHz, 8GB, AIX 5.1, 5.2, 5.3 & 6.1 - 10 dedicated VMware x86/64 boxes, ranging from dual core 8GB to 8 core monsters with 64GB - 4x667MHz AlphaServer, 8GB, Tru64 - 4x600MHz SGI Octane 300, IRIX 6.22 - and lots of other stuff. Actually, the only platform we don't have is Mac OS X. Although I've got a contact at Apple that I'll start harassing again once I'm back in East Lansing. Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Python on Windows with CoApp
Howdy folks, Quick e-mail at 34,000ft (aren't wifi-enabled flights great?) to mention a new initiative that's been started by Microsoft called CoApp (Common Opensource Application Publishing Platform). The aim is simple: make open source software rock on Windows ;-) It's probably easiest to think of it as a Microsoft-endorsed-but-community-run open source distribution for Windows, akin to all the various package managers for Linux distributions and ports/packages for the *BSDs. There are specific user and developer experiences we'll be addressing -- like making it easy to install and use open source software, or use it within your own project (open source or not). CoApp will affect Python in one of two ways. Once there's a clear-cut specification for open source projects to follow, Python can either decide to follow it, or not. The same applies to all open source packages, actually. For those that follow it, great! If not, no problem -- the plan is to shallow-fork such projects via launchpad and the CoApp community will take responsibility for getting releases of open source projects into CoApp shape. It's in its infancy at the moment -- it took the chap (Garrett Serack) who's spearheading it at Microsoft about six months to get it all signed off by the lawyers and platform/server VPs. So, for those of you out there who are Windows-inclined, now's a perfect time to get involved to help shape the direction of CoApp going forward. The website/wiki is http://coapp.org/ and the launchpad project site is http://launchpad.net/coapp (which is where the mailing list is hosted). We're actually having a 'CoApp Development Summit' tomorrow and Friday in Seattle (that Microsoft's graciously sponsored). The event will be accessible via Live Meeting for those that are interested: http://coapp.org/Project_Planning/CoApp_Design_and_Development_Summit Regards, Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 7 updated
Does anyone know of a way to teach vim that C sources in a python checkout
should have 4-space indents without changing the defaults for other C files?
I use this in my vimrc:
""
" indentation: use detectindent plugin if possible
""
set autoindent
set smartindent
try
let g:detectindent_preferred_expandtab = 1
let g:detectindent_preferred_tabsize = 8
let g:detectindent_preferred_indent = 4
source $VIMRUNTIME/plugin/detectindent.vim
au BufNewFile,BufRead * .* DetectIndent
catch
set smarttab
set expandtab
set tabstop=8
set shiftwidth=4
set softtabstop=4
set textwidth=80
endtry
*** And this is plugin/detectindent.vim:
" Name: detectindent (global plugin)
" Version: 1.0
" Author:Ciaran McCreesh
" Updates: http://dev.gentoo.org/~ciaranm/vim/
" Purpose: Detect file indent settings
"
" License: You may redistribute this plugin under the same terms
as Vim
"itself.
"
" Usage: :DetectIndent
"
"" to prefer expandtab to noexpandtab when detection is
"" impossible:
":let g:detectindent_preferred_expandtab = 1
"
"" to set a preferred indent level when detection is
"" impossible:
":let g:detectindent_preferred_indent = 4
"
" Requirements: Untested on Vim versions below 6.2
fun! IsCommentStart(line)
" &comments isn't reliable
if &ft == "c" || &ft == "cpp"
return -1 != match(a:line, '/\*')
else
return 0
endif
endfun
fun! IsCommentEnd(line)
if &ft == "c" || &ft == "cpp"
return -1 != match(a:line, '\*/')
else
return 0
endif
endfun
fun! DetectIndent()
let l:has_leading_tabs= 0
let l:has_leading_spaces = 0
let l:shortest_leading_spaces_run = 0
let l:longest_leading_spaces_run = 0
let l:idx_end = line("$")
let l:idx = 1
while l:idx <= l:idx_end
let l:line = getline(l:idx)
" try to skip over comment blocks, they can give really screwy
indent
" settings in c/c++ files especially
if IsCommentStart(l:line)
while l:idx <= l:idx_end && ! IsCommentEnd(l:line)
let l:line = getline(l:idx)
let l:idx = l:idx + 1
endwhile
let l:idx = l:idx + 1
continue
endif
let l:leading_char = strpart(l:line, 0, 1)
if l:leading_char == "\t"
let l:has_leading_tabs = 1
elseif l:leading_char == " "
" only interested if we don't have a run of spaces followed
by a
" tab.
if -1 == match(l:line, '^ \+\t')
let l:has_leading_spaces = 1
let l:spaces = strlen(matchstr(l:line, '^ \+'))
if l:shortest_leading_spaces_run == 0 ||
\ l:spaces < l:shortest_leading_spaces_run
let l:shortest_leading_spaces_run = l:spaces
endif
if l:spaces > l:longest_leading_spaces_run
let l:longest_leading_spaces_run = l:spaces
endif
endif
endif
let l:idx = l:idx + 1
endwhile
if l:has_leading_tabs && ! l:has_leading_spaces
" tabs only, no spaces
set noexpandtab
if exists("g:detectindent_preferred_tabsize")
let &shiftwidth = g:detectindent_preferred_indent
let &tabstop = g:detectindent_preferred_indent
endif
elseif l:has_leading_spaces && ! l:has_leading_tabs
" spaces only, no tabs
set expandtab
let &shiftwidth = l:shortest_leading_spaces_run
elseif l:has_leading_spaces && l:has_leading_tabs
" spaces and tabs
set noexpandtab
let &shiftwidth = l:shortest_leading_spaces_run
" , time to guess how big tabs are
if l:longest_leading_spaces_run < 2
let &tabstop = 2
elseif l:longest_leading_spaces_run < 4
let &tabstop = 4
else
let &tabstop = 8
endif
else
" no spaces, no tabs
if exists("g:detectindent_preferred_tabsize")
let &shiftwidth = g:detectindent_preferred_indent
let &tabstop = g:detectindent_preferred_indent
endif
if exists("g:detectindent_preferred_expandtab")
set expandtab
endif
endif
endfun
command! -nargs=0 DetectIndent call DetectIndent()
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [snakebite] snakebite for GSoC?
On Thu, Mar 19, 2009 at 10:32:03AM -0700, ajaksu wrote: > Does anyone have good ideas for assigning students to snakebite? Is it > too early? Perhaps a little too early, python-dev@ won't know anything about Snakebite yet as I haven't publicly announced it there ;-) Watch this space closer to PyCon. FWIW, though, we're planning for Snakebite to be *very* involved with GSoC/GHOP. > I think the client-side 'Snakebite daemon' and server-side stuff > described at http://tinyurl.com/beyond-buildbot would be great > projects. Indeed. Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Py_ssize_t support for ctypes arrays and pointers
On Fri, Mar 20, 2009 at 08:00:46PM +0100, Thomas Heller wrote: > Since I do not have a machine with so much memory: Does one > of the buildbots allow to run tests for this feature, or > do I have to wait for the snakebite farm? Will you be at PyCon? The wait might not be as bad as you think ;-) Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Python 2.4, VS 2005 & Profile Guided Optmization
Hi,
Has anyone else built Python with Visual Studio 2005 and played around
with Profile Guided Optimization? I had to build Python from source w/
VS 2005 as I had a few .pyd's built with VS 2005 that I wanted to load;
I ended up playing around with Profile Guided Optimization, running
``python.exe pystones.py'' to collect call-graph data after
python.exe/Python24.dll had been instrumented, then recompiling with the
optimizations fed back in.
Results were interesting, an average speedup of around 33% was
noticeable:
ActiveState 2.4.3 python.exe:
C:\Python24>python.exe Lib\test\pystone.py
Pystone(1.1) time for 5 passes = 0.980119 This machine
benchmarks at
51014.2 pystones/second
The python compiled from branches/release24-maint with VS 2005 + profile
guided optimization:
C:\Python24>python.exe Lib\test\pystone.py
Pystone(1.1) time for 5 passes = 0.73261 This machine benchmarks
at
68249.2 pystones/second
Is there any motivation in the Win32 Python dev camp to switch from VC6
to VS 2005?
FWIW, although there were a shed-load of warnings when compiling python
and pythoncore (and a lot more errors when compiling other modules), I
only had to apply one patch to get it working well enough to run
pystone.py. Without this patch, the VC8 CRT aborts at runtime as soon
as an invalid signal is passed to signal(); which is inevitable given
the current code in the initsignal() method:
for (i = 1; i < NSIG; i++) {
void (*t)(int);
t = PyOS_getsig(i);
Regards,
Trent.
--
http://www.onresolve.com
Index: signalmodule.c
===
--- signalmodule.c (revision 47196)
+++ signalmodule.c (working copy)
@@ -280,7 +280,21 @@
{NULL, NULL} /* sentinel */
};
+#define WIN32VS2005HACK
+#ifdef WIN32VS2005HACK
+#include
+#include
+#include
+void dummy_handler(const wchar_t *exp,
+ const wchar_t *fn,
+ const wchar_t *file,
+ unsigned int line,
+ uintptr_t reserved)
+{
+}
+#endif
+
PyDoc_STRVAR(module_doc,
"This module provides mechanisms to use signal handlers in Python.\n\
\n\
@@ -339,6 +353,12 @@
goto finally;
Py_INCREF(IntHandler);
+#ifdef WIN32VS2005HACK
+(void)_set_invalid_parameter_handler(&dummy_handler);
+_CrtSetReportMode(_CRT_ASSERT, 0);
+#endif
+
+
Handlers[0].tripped = 0;
for (i = 1; i < NSIG; i++) {
void (*t)(int);
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Any tips to tell sprinter at PyCon about developing on Windows?
Feb 2, 2008 7:34 PM, Christian Heimes <[EMAIL PROTECTED]> wrote: > > Brett Cannon wrote: > > It would be really cool if you can recruit some experienced Windows > > developers. :] > That's the point in all of this. =) > -Brett I'll be around for the sprints -- didn't really have a plan as to what I'd like to sprint on but if there's some interest in farming Windows developers, I'll raise my hand. Anything in particular you can point myself or others in the Windows camp at such that we're a bit better prepared come sprint time (i.e. open issues)? (Also, I'm looking to acquire a new reasonably well-spec'd Windows box for work. If it's available in time for PyCon, I should be able to set up a couple of virtual 64-bit Vista/Server 2008 images with VS 2008 dev environments that non-Windows developers could use, if that would be desirable.) Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Adding a new Windows x64 buildbot
Hi, I've got a Windows Server 2008 x64 server I'd like to contribute as a buildbot. As per the recommendation on http://wiki.python.org/moin/BuildBot, it sounds like I'm looking for Martin, Anthony or Neal to sort me out with slave credentials. Feel free to drop me a line! Regards, Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-3000] Python 2.6 and 3.0
> (unless a complete working solution is presented in that other technology, > and as long as that other technology still creates MSI files with > free-as-in-beer tools). Just out of interest, what's the reason for enforcing that the installer must be an MSI? Or, rather, if I were to present an alternative .exe installer that ticks all of the above boxes, exceeds the capabilities of the current installer and above all is easier to extend and maintain -- would that be a non-starter because it's not an MSI? Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Fixing buildbot/external(-amd64).bat files on Windows
Howdy, I'm going through the motions of getting my newly added build slave in a half decent state. The external.bat and external-amd64.bat files needed the following in order to build db-4.4.20: Index: external.bat === --- external.bat(revision 61125) +++ external.bat(working copy) @@ -10,7 +10,8 @@ @rem Sleepycat db if not exist db-4.4.20 svn export http://svn.python.org/projects/external/db-4.4.20 if not exist db-4.4.20\build_win32\debug\libdb44sd.lib ( - vcbuild db-4.4.20\build_win32\Berkeley_DB.sln /build Debug /project db_static + devenv /upgrade db-4.4.20\build_win32\Berkeley_DB.sln + devenv db-4.4.20\build_win32\Berkeley_DB.sln /build Debug /project db_static ) @rem OpenSSL (This is against trunk, same thing would apply to py3k I guess, given that we're using %VS90COMNTOOLS%vsvars32.bat there too.) Regards, Trent. -- http://www.onresolve.com external.bat.patch Description: external.bat.patch external-amd64.bat.patch Description: external-amd64.bat.patch ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fixing buildbot/external(-amd64).bat files on Windows
> > I'm going through the motions of getting my newly added build slave > in a half decent state. > > I think the buildbot should have a name different from 'x86 XP'. > (Martin, Neal?) > > Thomas Yeah, I've dropped Martin a note regarding this. The community bots refer to Windows Server 2003 boxes as just that, so perhaps a rename to 'x86 Windows Server 2008' is appropriate. FWIW as it's a 64-bit box, I'm hoping to get a slave set up for 'x64 Windows Server 2008' as well. (As far as I can see, the x64/x86 nature of the slave is dictated by the master, correct? i.e. I can't tweak/clone this myself?) Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fixing buildbot/external(-amd64).bat files on Windows
Christian Heimes: > Trent Nelson wrote: > > - vcbuild db-4.4.20\build_win32\Berkeley_DB.sln /build Debug > /project db_static > > + devenv /upgrade db-4.4.20\build_win32\Berkeley_DB.sln > > + devenv db-4.4.20\build_win32\Berkeley_DB.sln /build Debug > /project db_static > > The upgrade is requires only once. It probably belongs next to the > checkout or svn up and not in the build section. Makes sense. So we're looking at something like: @rem Sleepycat db if not exist db-4.4.20 ( svn export http://svn.python.org/projects/external/db-4.4.20 devenv /upgrade db-4.4.20\build_win32\Berkeley_DB.sln ) if not exist db-4.4.20\build_win32\debug\libdb44sd.lib ( devenv db-4.4.20\build_win32\Berkeley_DB.sln /build Debug ) I'll test this when I get to work and report back. Trent. -- http://www.onresolve.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fixing buildbot/external(-amd64).bat files on Windows
Christian Heimes: > Thomas Heller wrote: > > What's the difference between these two? > > > > vcbuild db-4.4.20\build_win32\Berkeley_DB.sln /build Debug > > > > devenv db-4.4.20\build_win32\Berkeley_DB.sln /build Debug > > Devenv is the name of the VS GUI executable but it can *also* be used as > a CLI to build stuff. devenv doesn't work for Express Edition. > > vcbuild seems to be the preferred CLI app to build a project but it's > limited. I think it doesn't support /upgrade. Hummm. My answer would be more along the lines of "devenv works, vcbuild doesn't" ;-) S:\buildbots\python\trunk.nelson-windows\db-4.4.20\build_win32>vcbuild Berkeley_DB.sln /build Debug /project db_static Microsoft (R) Visual C++ Project Builder - Command Line Version 9.00.21022 Copyright (C) Microsoft Corporation. All rights reserved. vcbuild.exe : warning VCBLD6002: invalid option /build specified. The option was ignored. vcbuild.exe : warning VCBLD6002: invalid option /project specified. The option was ignored. vcbuild.exe : warning VCBLD6002: invalid option db_static specified. The option was ignored. vcbuild.exe : error VCBLD0006: invalid configuration name: DEBUG. Compare this to: S:\buildbots\python\trunk.nelson-windows\db-4.4.20\build_win32>devenv Berkeley_DB.sln /build Debug /project db_static Microsoft (R) Visual Studio Version 9.0.21022.8. Copyright (C) Microsoft Corp. All rights reserved. == Build: 0 succeeded, 0 failed, 1 up-to-date, 0 skipped == I don't know how the existing vcbuild line ever worked, given the following output from vcbuild /?: Usage: vcbuild [options] [project|solution] [config|$ALL] Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Windows buildbot test_bsddb3 problems (was RE: Buildbots for trunk are all red)
Spent some time on my buildbot (x86 2k8 trunk) this morning trying to track down why test_bsddb3 is failing (trunk with db-4.4.20). The first test that fails is this: test01_GetsAndPuts (bsddb.test.test_basics.BasicBTreeWithEnvTestCase) ... ERROR That's slightly misleading though as the test runs fine -- the actual exception is being thrown in test_basics.BasicTestCase.tearDown() when os.remove() is called against the first db file (i.e. '__db.001'): WindowsError: [Error 5] Access is denied: 'c:\users\trent~1.nel\appdata\local\temp\2\db_home2808\__db.001 This isn't surprising, given that the python_d.exe process still seems to have __db.001, __db.002 and __db.003 open at the time os.remove() is called. The aforementioned tearDown() method looks like this: def tearDown(self): self.d.close() if self.env is not None: test_support.rmtree(self.homeDir) self.env.close() ## Make a new DBEnv to remove the env files from the home dir. ## (It can't be done while the env is open, nor after it has been ## closed, so we make a new one to do it.) #e = db.DBEnv() #e.remove(self.homeDir) #os.remove(os.path.join(self.homeDir, self.filename)) else: os.remove(self.filename) If I switch the order of statements such that self.env.close() is called before test_suppot.rmtree(self.homeDir), this particular test and a host of others that were also failing now pass (a runtime abort is no longer raised by the CRT half way into the tests either). (Note that the order was switched to that shown above by Neal in r61052 on Feb 24th, which is when these issues started occurring.) That said, there are still a lot more test failures down the track once this change has been made, either via the access denied WindowsError, or a DBInvalidArgError, e.g.: ERROR: test02_WithSource (bsddb.test.test_recno.SimpleRecnoTestCase) -- Traceback (most recent call last): File "S:\src\svn\svn.python.org\projects\python\trunk\lib\bsddb\test\test_recno.py", line 33, in tearDown test_support.rmtree(self.homeDir) File "S:\src\svn\svn.python.org\projects\python\trunk\lib\test\test_support.py", line 70, in rmtree shutil.rmtree(path) File "S:\src\svn\svn.python.org\projects\python\trunk\lib\shutil.py", line 184, in rmtree onerror(os.remove, fullname, sys.exc_info()) File "S:\src\svn\svn.python.org\projects\python\trunk\lib\shutil.py", line 182, in rmtree os.remove(fullname) WindowsError: [Error 5] Access is denied: 'c:\\users\\trent~1.nel\\appdata\\local\\temp\\2\\db_home4656\\tmp04_knk' == ERROR: test01_1WriterMultiReaders (bsddb.test.test_thread.BTreeConcurrentDataStore) -- Traceback (most recent call last): File "S:\src\svn\svn.python.org\projects\python\trunk\lib\bsddb\test\test_thread.py", line 62, in setUp self.env.open(homeDir, self.envflags | db.DB_CREATE) DBInvalidArgError: (22, 'Invalid argument -- configured environment flags incompatible with existing environment') The DBInvalidArgError exception is only raised after a previous WindowsError is encountered, so I assume it's a side-effect of the tearDown() method not cleaning the environment correctly. It seems this comment in tearDown() is quite pertinent to our situation: ## Make a new DBEnv to remove the env files from the home dir. ## (It can't be done while the env is open, nor after it has been ## closed, so we make a new one to do it.) #e = db.DBEnv() #e.remove(self.homeDir) #os.remove(os.path.join(self.homeDir, self.filename)) Not sure why this is commented out -- quick search of svn history indicates it's been like that for at least the last year and a half. Will have some more time this evening to spend on this, however, work calls at the moment. Regards, Trent. From: [EMAIL PROTECTED] [EMAIL PROTECTED] On Behalf Of Facundo Batista [EMAIL PROTECTED] Sent: 26 February 2008 06:22 To: Thomas Hervé Cc: python-dev@python.org Subject: Re: [Python-Dev] Buildbots for trunk are all red 2008/2/25, Thomas Hervé <[EMAIL PROTECTED]>: > I've worked on that problem during the bug day. I've open a ticket with > a patch at http://bugs.python.org/issue2168. Most of the buildbots are green now!!! Thank you all! This community is as awesome as Python itself, ;) Three remains in red, though: - Alpha Tru64: test_smtplib.py is flaky, and _ssl.c is not compiled correctly. Neil is hunting this, I think. - X86 XP-3: seems to crash after test_bsddb3.py. - X86 XP-4: idem. For this two, how can be tried if the bsddb lib in those windows is correctly installed? Thanks again. -- .Fac
Re: [Python-Dev] Windows buildbot test_bsddb3 problems (was RE: Buildbots for trunk are all red)
> Trent, thanks for working on the buildbot. I fixed the first case you > mentioned in r61233 wrt removing the directory before closing the > file. It would be great if you could submit a patch when you are able > to fix the remaining problems. Nod, found a few more things now that test_bsddb3 isn't causing a CRT abortion. tmpfile() needs to be reworked on Windows, see http://bugs.python.org/issue2232. Going to spend some more time on it this evening. I'm determined to see a flippin' green build/test status for my slave if it kills me :> Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Windows buildbot test_bsddb3 problems (was RE: Buildbots for trunk are all red)
> Trent, thanks for working on the buildbot. I fixed the first case you > mentioned in r61233 wrt removing the directory before closing the > file. It would be great if you could submit a patch when you are able > to fix the remaining problems. % svn diff Index: test_dbshelve.py === --- test_dbshelve.py(revision 61233) +++ test_dbshelve.py(working copy) @@ -267,8 +267,8 @@ def tearDown(self): +self.do_close() test_support.rmtree(self.homeDir) -self.do_close() class EnvBTreeShelveTestCase(BasicEnvShelveTestCase): Index: test_thread.py === --- test_thread.py (revision 61233) +++ test_thread.py (working copy) @@ -73,9 +73,9 @@ self.d.open(self.filename, self.dbtype, self.dbopenflags|db.DB_CREATE) def tearDown(self): -test_support.rmtree(self.homeDir) self.d.close() self.env.close() +test_support.rmtree(self.homeDir) def setEnvOpts(self): pass I'm getting 100% success rate with test_bsddb3 on Windows now with this patch. Yay! Trent. -- http://www.onresolve.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] signal.alarm(3) in trunk test_socketserver.py
r61099 added the following to trunk/Lib/test/test_socketserver.py: if __name__ == "__main__": test_main() + signal.alarm(3) # Shutdown shouldn't take more than 3 seconds. which breaks platforms that don't have signal.alarm, like, say, !unix ;-) Trent. -- http://www.onresolve.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] signal.alarm(3) in trunk test_socketserver.py
> r61099 added the following to trunk/Lib/test/test_socketserver.py: > > if __name__ == "__main__": > test_main() > + signal.alarm(3) # Shutdown shouldn't take more than 3 seconds. > Actually, signal.alarm() was introduced all over the place in that revision. I understand the intent of this commit was to speed up the runtime of this test (something like 28s -> 0.3s was quoted in the commit log). FWIW, runtime of the test with the following patch on Windows is 0.125s: Index: test_socketserver.py === --- test_socketserver.py(revision 61233) +++ test_socketserver.py(working copy) @@ -28,6 +28,9 @@ HAVE_UNIX_SOCKETS = hasattr(socket, "AF_UNIX") HAVE_FORKING = hasattr(os, "fork") and os.name != "os2" +def signal_alarm(n): +if hasattr(signal, 'alarm'): +signal.alarm(n) def receive(sock, n, timeout=20): r, w, x = select.select([sock], [], [], timeout) @@ -99,7 +102,7 @@ """Test all socket servers.""" def setUp(self): -signal.alarm(20) # Kill deadlocks after 20 seconds. +signal_alarm(20) # Kill deadlocks after 20 seconds. self.port_seed = 0 self.test_files = [] @@ -112,7 +115,7 @@ except os.error: pass self.test_files[:] = [] -signal.alarm(0) # Didn't deadlock. +signal_alarm(0) # Didn't deadlock. def pickaddr(self, proto): if proto == socket.AF_INET: @@ -267,4 +270,4 @@ if __name__ == "__main__": test_main() -signal.alarm(3) # Shutdown shouldn't take more than 3 seconds. +signal_alarm(3) # Shutdown shouldn't take more than 3 seconds. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Patch for trunk test_winsound.py (fixes my buildbot)
winsound.Beep fails for me on the 'x86 2k8 trunk' build slave, which is a virtual Windows Server 2008 instance running under Hyper-V. Not surprisingly, there's not a single audio-related device on this system. The attached patch to test_winsound.py incorporates the _have_soundcard() checks to the BeepTest class, which fixes the problem for me. (I've also tested the patch on a Vista system (that does have a soundcard) and everything works as expected.) Trent. test_winsound.py.patch Description: test_winsound.py.patch ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] signal.alarm(3) in trunk test_socketserver.py
> Yep, the alarm is only there to prevent what would be deadlocks from > running forever. Sorry for breaking !unix. Your patch looks fine to > me. Do you want to submit it or shall I? I'm not a committer, so it's all yours. Thanks for the quick turnaround! Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Windows buildbots randomly die with twisted ConnectionLost errors?
I've started to see my build slave dying every so often with a twisted error half way through tests: ... test_htmlparser test_httplib remoteFailed: [Failure instance: Traceback (failure with no frames): twisted.internet.error.ConnectionLost: Connection to the other side was lost in a non-clean fashion. ] Examples: http://www.python.org/dev/buildbot/all/x86%20W2k8%20trunk/builds/46/step-test/0 http://www.python.org/dev/buildbot/all/x86%20W2k8%20trunk/builds/36/step-test/0 I'm not sure if I should read into the fact that it's occurring after networking-oriented tests like test_httplib and test_ftplib. Running rt.bat on the resulting build manually doesn't indicate any errors in these tests. Have other Windows buildbot owners seen this? Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Windows buildbots randomly die with twisted ConnectionLost errors?
Had a chat with some Twisted/buildbot folk and they can confirm they've seen it as well on Windows. They've given me a few things to look into. Out of interest, how are you running your buildbot? Via the command line in an interactive desktop session, as a service, or as a scheduled task, or some other way? From: [EMAIL PROTECTED] [EMAIL PROTECTED] On Behalf Of Thomas Heller [EMAIL PROTECTED] Sent: 05 March 2008 10:03 To: python-dev@python.org Subject: Re: [Python-Dev] Windows buildbots randomly die with twisted ConnectionLost errors? Trent Nelson schrieb: > I've started to see my build slave dying every so often with a > twisted error half way through tests: ... test_htmlparser > test_httplib > > remoteFailed: [Failure instance: Traceback (failure with no frames): > twisted.internet.error.ConnectionLost: Connection to the other side > was lost in a non-clean fashion. ] > > Examples: > http://www.python.org/dev/buildbot/all/x86%20W2k8%20trunk/builds/46/step-test/0 > > http://www.python.org/dev/buildbot/all/x86%20W2k8%20trunk/builds/36/step-test/0 > > > I'm not sure if I should read into the fact that it's occurring after > networking-oriented tests like test_httplib and test_ftplib. Running > rt.bat on the resulting build manually doesn't indicate any errors in > these tests. Have other Windows buildbot owners seen this? > > Trent. I have not observed this behaviour on my buildbots. Have you looked into the twistd.log logfile? Thomas ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/tnelson%40onresolve.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] r61264 - in python/trunk: Lib/test/test_os.py Misc/NEWS
Hurrah, 'x86 W2k8 trunk' has just experienced its first green build and test! Thanks to everyone that committed the various patches I sent out in such a timely fashion. Martin, does this mean I can have a slave set up for x64 now? }:> Trent. > -Original Message- > From: [EMAIL PROTECTED] [mailto:python-checkins- > [EMAIL PROTECTED] On Behalf Of martin.v.loewis > Sent: 06 March 2008 01:55 > To: [EMAIL PROTECTED] > Subject: [Python-checkins] r61264 - in python/trunk: > Lib/test/test_os.py Misc/NEWS > > Author: martin.v.loewis > Date: Thu Mar 6 07:55:22 2008 > New Revision: 61264 > > Modified: >python/trunk/Lib/test/test_os.py >python/trunk/Misc/NEWS > Log: > Patch #2232: os.tmpfile might fail on Windows if the user has no > permission to create files in the root directory. > Will backport to 2.5. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Request for another build slave
Can someone set me up with a build slave for an x86 FreeBSD box (6.2-STABLE, although we'll be migrating to 7.x in a week or so)? Thanks. [Suggestion: perhaps we could set up a [EMAIL PROTECTED] list for discussing buildbot administrative minutiae, rather than polluting python-dev?] Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Windows x64 & bsddb 4.4.20 woes
I've been trying to give the Windows x64 builds a bit of TLC the past few
evenings. I managed to get a successful build with all external modules last
night (Tcl/Tk required about a half a dozen code/configuration changes each in
order to build in a Windows x64 environment with Visual Studio 9, I'll deal
with that in a separate thread or roundup issue).
Unfortunately, though, we're back to more bsddb issues. I got about 15 tests
in without error before test_whichdb ran, which results in the following being
called in dbhash.py:
return bsddb.hashopen(file, flag, mode)
I can trace that call to DBEnv_open() in _bsddb.c:
static PyObject*
DBEnv_open(DBEnvObject* self, PyObject* args)
{
int err, flags=0, mode=0660;
char *db_home;
if (!PyArg_ParseTuple(args, "z|ii:open", &db_home, &flags, &mode))
return NULL;
CHECK_ENV_NOT_CLOSED(self);
MYDB_BEGIN_ALLOW_THREADS;
err = self->db_env->open(self->db_env, db_home, flags, mode);
^
Placing a breakpoint at the line above and stepping in results in Visual Studio
reporting: " A buffer overrun has occurred in python_d.exe which has corrupted
the program's internal state. Press Break to debug the program or Continue to
terminate the program.". FWIW, the exception is being raised as part of the
/GS buffer overflow checks (implemented in gs_result.c, which is provided in my
VS9 installation).
This has been annoyingly awkward to debug. I can't break down that call into
multiple steps in order to try place breakpoints in the db_static module. The
callstack isn't that useful either:
_bsddb_d.pyd!__crt_debugger_hook()
_bsddb_d.pyd!__report_gsfailure(unsigned __int64 StackCookie=2211040)
_bsddb_d.pyd!__GSHandlerCheckCommon(void * EstablisherFrame=0x0021bce0,
...)
_bsddb_d.pyd!__GSHandlerCheck(_EXCEPTION_RECORD *
ExceptionRecord=0x0021bbc0, ...)
ntdll.dll!773ae13d()
[Frames below may be incorrect and/or missing, no symbols loaded for ntdll.dll]
ntdll.dll!773aea57()
ntdll.dll!773b59f8()
_bsddb_d.pyd!__os_strdup() + 0x18 bytes
_bsddb_d.pyd!__os_tmpdir() + 0x281 bytes
You'd think placing breakpoints in db 4.4.20's __os_strdup and __os_tmpdir
methods would do something, but alas, the bufferoverflow exception is raised
before any breakpoints are set. This makes me suspect there's something funky
going on with the entire build and linking of db_static (VS should honour those
breakpoints if the code is being entered, I even added db_static to pcbuild.sln
and rebuilt but no dice). I've noticed that they're not using consistent
compiler flags by default (we use /GS, they use /GS-, we allow function level
linking, they don't -- note that I did change db_static's options to align with
_bsddb's but the bufferoverflow exception is still being thrown).
Greg, Jesús, I'm CC'ing you guys as stfw'ing seems to bring back you two the
most when it comes to bsddb issues. I've still got a list of things to try
with regarding to debugging this x64 issue, but I wanted to reach out now to
see if anyone else had encountered it before. Has bsddb ever been built
successfully on Win64 and passed all tests or am I venturing into new ground?
Martin, you've changed externals/bsddb-4.4.20 with regards to x64 builds
recently -- have you been able to get things working in your x64 environments?
Regards,
Trent.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Windows x64 & bsddb 4.4.20 woes
Hey Greg,
I'm at PyCon indeed, staying through the sprints 'til next Thursday. I'll drop
you a note offline re catching up.
The other query I had was whether or not I should try a later version of
BerkeleyDB -- are we committed to 4.4.20 (or 4.4.x?) for 2.6/3.0 or is it worth
investigating newer versions? Martin/Jesus, any thoughts on this?
Regarding the db_static build and conflicting compile/link options -- I'm going
to bring the db_static source directly into the _bsddb project (for now) which
should make this a lot easier to debug.
Trent.
From: Gregory P. Smith [EMAIL PROTECTED]
Sent: 13 March 2008 22:00
To: Trent Nelson
Cc: python-dev@python.org; Jesus Cea
Subject: Re: Windows x64 & bsddb 4.4.20 woes
I haven't built the bsddb stuff on windows myself in a few years and have never
had access to a windows x64 system so I'm no silver bullet. Making the
BerkeleyDB compile and link options match with those of python is the first
place I'd start. Also you should be able to make a debug build of BerkeleyDB
(though it sounds like you may have tried that already?). Next off in the
debugging i'd take a look at the assembly to see what exactly it was failing to
do.
If you're at PyCon right now we should meet up and try to figure it out (I just
arrived).
On 3/13/08, Trent Nelson <[EMAIL PROTECTED]> wrote:
I've been trying to give the Windows x64 builds a bit of TLC the past few
evenings. I managed to get a successful build with all external modules last
night (Tcl/Tk required about a half a dozen code/configuration changes each in
order to build in a Windows x64 environment with Visual Studio 9, I'll deal
with that in a separate thread or roundup issue).
Unfortunately, though, we're back to more bsddb issues. I got about 15 tests
in without error before test_whichdb ran, which results in the following being
called in dbhash.py:
return bsddb.hashopen(file, flag, mode)
I can trace that call to DBEnv_open() in _bsddb.c:
static PyObject*
DBEnv_open(DBEnvObject* self, PyObject* args)
{
int err, flags=0, mode=0660;
char *db_home;
if (!PyArg_ParseTuple(args, "z|ii:open", &db_home, &flags, &mode))
return NULL;
CHECK_ENV_NOT_CLOSED(self);
MYDB_BEGIN_ALLOW_THREADS;
err = self->db_env->open(self->db_env, db_home, flags, mode);
^
Placing a breakpoint at the line above and stepping in results in Visual Studio
reporting: " A buffer overrun has occurred in python_d.exe which has corrupted
the program's internal state. Press Break to debug the program or Continue to
terminate the program.". FWIW, the exception is being raised as part of the
/GS buffer overflow checks (implemented in gs_result.c, which is provided in my
VS9 installation).
This has been annoyingly awkward to debug. I can't break down that call into
multiple steps in order to try place breakpoints in the db_static module. The
callstack isn't that useful either:
_bsddb_d.pyd!__crt_debugger_hook()
_bsddb_d.pyd!__report_gsfailure(unsigned __int64 StackCookie=2211040)
_bsddb_d.pyd!__GSHandlerCheckCommon(void * EstablisherFrame=0x0021bce0,
...)
_bsddb_d.pyd!__GSHandlerCheck(_EXCEPTION_RECORD *
ExceptionRecord=0x0021bbc0, ...)
ntdll.dll!773ae13d()
[Frames below may be incorrect and/or missing, no symbols loaded for ntdll.dll]
ntdll.dll!773aea57()
ntdll.dll!773b59f8()
_bsddb_d.pyd!__os_strdup() + 0x18 bytes
_bsddb_d.pyd!__os_tmpdir() + 0x281 bytes
You'd think placing breakpoints in db 4.4.20's __os_strdup and __os_tmpdir
methods would do something, but alas, the bufferoverflow exception is raised
before any breakpoints are set. This makes me suspect there's something funky
going on with the entire build and linking of db_static (VS should honour those
breakpoints if the code is being entered, I even added db_static to pcbuild.sln
and rebuilt but no dice). I've noticed that they're not using consistent
compiler flags by default (we use /GS, they use /GS-, we allow function level
linking, they don't -- note that I did change db_static's options to align with
_bsddb's but the bufferoverflow exception is still being thrown).
Greg, Jesús, I'm CC'ing you guys as stfw'ing seems to bring back you two the
most when it comes to bsddb issues. I've still got a list of things to try
with regarding to debugging this x64 issue, but I wanted to reach out now to
see if anyone else had encountered it before. Has bsddb ever been built
successfully on Win64 and passed all tests or am I venturing into new ground?
Martin, you've changed externals/bsddb-4.4.20 with regards to x64 builds
recently -- have you been able to get things working in your x64 environments?
Regards,
Re: [Python-Dev] Windows x64 & bsddb 4.4.20 woes
Ah, and to think I just fixed 4.4.20 ;-)
Removing the dependency on db_static.vcproj and merging the relevant source
code files into _bsddb.vcproj did the trick -- all x64 bsddb-related tests now
pass. The only issue with this approach is that it locks _bsddb.vcproj into
4.4.20. However, considering that this approach (i.e. bringing their source
files into our build instead of linking against a static lib compiled with
wildly incompatible flags) only took me about two minutes to implement and
immediately fixed every bsddb problem I was encoutering, I'm convinced it's the
right way to go. (I can separate the dependencies easily enough.)
Woeful PyCon/hotel connectivity is preventing me from getting to
bugs.python.org at the moment; I'll raise a ticket later to capture this stuff
and we can move the discussion there once I've attached some patches.
Trent.
From: Gregory P. Smith [EMAIL PROTECTED]
Sent: 14 March 2008 00:23
To: Trent Nelson
Cc: python-dev@python.org; Jesus Cea
Subject: Re: Windows x64 & bsddb 4.4.20 woes
On 3/13/08, Trent Nelson <[EMAIL PROTECTED]> wrote:
Hey Greg,
I'm at PyCon indeed, staying through the sprints 'til next Thursday. I'll drop
you a note offline re catching up.
The other query I had was whether or not I should try a later version of
BerkeleyDB -- are we committed to 4.4.20 (or 4.4.x?) for 2.6/3.0 or is it worth
investigating newer versions? Martin/Jesus, any thoughts on this?
Python 2.6/3.0 should be built on Windows using BerkeleyDB 4.5.x for now.
4.6.x is out but has some bugs on some platforms so i don't recommend shipping
our release using it; 4.7.x is in beta and some bugs are being worked on; if
its out and shows no signs of obvious issues before the 2.6/3.0 beta period is
over I recommend we build our binary releases using it. Otherwise 4.5 it will
be. There is no reason to use 4.4.x.
Regarding the db_static build and conflicting compile/link options -- I'm going
to bring the db_static source directly into the _bsddb project (for now) which
should make this a lot easier to debug.
Trent.
From: Gregory P. Smith [EMAIL PROTECTED]
Sent: 13 March 2008 22:00
To: Trent Nelson
Cc: python-dev@python.org; Jesus Cea
Subject: Re: Windows x64 & bsddb 4.4.20 woes
I haven't built the bsddb stuff on windows myself in a few years and have never
had access to a windows x64 system so I'm no silver bullet. Making the
BerkeleyDB compile and link options match with those of python is the first
place I'd start. Also you should be able to make a debug build of BerkeleyDB
(though it sounds like you may have tried that already?). Next off in the
debugging i'd take a look at the assembly to see what exactly it was failing to
do.
If you're at PyCon right now we should meet up and try to figure it out (I just
arrived).
On 3/13/08, Trent Nelson <[EMAIL PROTECTED]> wrote:
I've been trying to give the Windows x64 builds a bit of TLC the past few
evenings. I managed to get a successful build with all external modules last
night (Tcl/Tk required about a half a dozen code/configuration changes each in
order to build in a Windows x64 environment with Visual Studio 9, I'll deal
with that in a separate thread or roundup issue).
Unfortunately, though, we're back to more bsddb issues. I got about 15 tests
in without error before test_whichdb ran, which results in the following being
called in dbhash.py:
return bsddb.hashopen(file, flag, mode)
I can trace that call to DBEnv_open() in _bsddb.c:
static PyObject*
DBEnv_open(DBEnvObject* self, PyObject* args)
{
int err, flags=0, mode=0660;
char *db_home;
if (!PyArg_ParseTuple(args, "z|ii:open", &db_home, &flags, &mode))
return NULL;
CHECK_ENV_NOT_CLOSED(self);
MYDB_BEGIN_ALLOW_THREADS;
err = self->db_env->open(self->db_env, db_home, flags, mode);
^
Placing a breakpoint at the line above and stepping in results in Visual Studio
reporting: " A buffer overrun has occurred in python_d.exe which has corrupted
the program's internal state. Press Break to debug the program or Continue to
terminate the program.". FWIW, the exception is being raised as part of the
/GS buffer overflow checks (implemented in gs_result.c, which is provided in my
VS9 installation).
This has been annoyingly awkward to debug. I can't break down that call into
multiple steps in order to try place breakpoints in the db_static module. The
callstack isn't that useful either:
_bsddb_d.pyd!__crt_debugger_hook()
_bsddb_d.pyd!__report_gsfailure(unsigned __int64 StackCookie=2211040)
_bsddb_d.pyd!__GSHandlerCheckCommon(void * EstablisherFrame=0x0021bce0,
...)
_bsddb_d.pyd!__GSHandlerCheck(_EXCEPTION_RECORD *
ExceptionRecord=0x0
Re: [Python-Dev] Windows x64 & bsddb 4.4.20 woes
> > Removing the dependency on db_static.vcproj and merging the relevant > > source code files into _bsddb.vcproj did the trick -- all x64 > > bsddb-related tests now pass. The only issue with this approach is > > that it locks _bsddb.vcproj into 4.4.20. However, considering that > > this approach (i.e. bringing their source files into our build > > instead of linking against a static lib compiled with wildly > > incompatible flags) only took me about two minutes to implement and > > immediately fixed every bsddb problem I was encoutering, I'm > > convinced it's the right way to go. (I can separate the dependencies > > easily enough.) > > I'm convinced this is the wrong approach. Are you sure you copied > all compiler settings over to the project correctly? What is the > procedure to upgrade such a setup? What is the procedure for people > who want to build with a different version of bsddb? I reviewed all the compiler options used by db_static.vcproj -- the only thing I needed to bring over was -DDIAGNOSTIC for debug builds. Everything else either had no impact and could be safely dropped, or conflicted with compiler options used by the rest of the python build (function level linking, buffer overflow checks, etc). Regarding support for users who want to build with different versions of bsddb; if they want a functional build that passes tests they're going to have to do something similar to the work I've done anyway. As it stands now, the .lib generated by db_static.vcproj for x64 builds just straight out does not work. That can be fixed in two ways: coerce db_static.vcproj into matching our build, or mimicking db_static in a new .vcproj that's contained with our build, inheriting our property sheets. I chose the latter. Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] 3.0 buildbots all red
http://www.python.org/dev/buildbot/3.0/ New sprint idea: getting all (inc. trunk) the buildbots green by Thursday. Anyone interested? Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 2.6 and 3.0 tasks
> > * Replace Windows API calls with wide versions to support unicode > >for file names, environment etc. > > +1. This should be broken into separate tasks for each API. What are we referring to here? Calling the W versions explicitly and using wchar_t for everything, or using the TCHAR/TEXT() approach and keeping the API calls the same, letting the #define UNICODE do the work behind the scenes? Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 3.0 buildbots all red
> > New sprint idea: getting all (inc. trunk) the buildbots green by > Thursday. Anyone interested? > > I think the chance to achieve that is close to zero. Sounds like a challenge if ever I've heard one -- care to wager a beer on it? (Only applies to buildbots that are connected/online.) (FWIW, I've got the x64 Windows build green on my dev server, tcl/tk and bsddb required patching, so did some tests, and so did some C code -- I'm in the process of filtering the efforts back into the tracker.) Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 3.0 buildbots all red
Yeah test_tokenize is weird, I've been looking into it as well. Here's a
sample failure from a Windows buildbot:
File "S:\buildbots\python\3.0.nelson-windows\build\lib\test\test_tokenize.py",
line ?, in test.test_tokenize.__test__.doctests
Failed example:
for testfile in testfiles:
if not roundtrip(open(testfile)): break
else: True
Exception raised:
Traceback (most recent call last):
File "S:\buildbots\python\3.0.nelson-windows\build\lib\doctest.py", line
1227, in __run
compileflags, 1), test.globs)
File "", line 2, in
if not roundtrip(open(testfile)): break
File "", line 3, in
roundtrip
token_list = list(generate_tokens(f.readline))
File "S:\buildbots\python\3.0.nelson-windows\build\lib\tokenize.py", line
264, in generate_tokens
line = readline()
File "S:\buildbots\python\3.0.nelson-windows\build\lib\io.py", line 1467,
in readline
readahead, pending = self._read_chunk()
File "S:\buildbots\python\3.0.nelson-windows\build\lib\io.py", line 1278,
in _read_chunk
pending = self._decoder.decode(readahead, not readahead)
File "S:\buildbots\python\3.0.nelson-windows\build\lib\io.py", line 1081,
in decode
output = self.decoder.decode(input, final=final)
File
"S:\buildbots\python\3.0.nelson-windows\build\lib\encodings\cp1252.py", line
23, in decode
return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 17:
character maps to
The following is at the end of the doctests in test_tokenize:
>>> tempdir = os.path.dirname(f) or os.curdir
>>> testfiles = glob.glob(os.path.join(tempdir, "test*.py"))
>>> if not test_support.is_resource_enabled("compiler"):
... testfiles = random.sample(testfiles, 10)
...
>>> for testfile in testfiles:
... if not roundtrip(open(testfile)): break
... else: True
True
On that first line, 'f' is lib/test/tokenize_tests.txt, so basically, it's
grabbing ten random test*.py files in lib/test and running
untokenize(generate_tokens(f.readline)) on each one. In order to figure out
which file it's dying on, I added the following to test_tokenize.py:
def test_tokenize_all():
import glob
import os
tempdir = os.path.dirname(__file__) or os.curdir
testfiles = glob.glob(os.path.join(tempdir, "test*.py"))
for file in testfiles:
print("processing file: " + file)
print("roundtrip(open(file)): " + roundtrip(open(file)))
This results in different results:
Python 3.0a3+ (py3k, Mar 16 2008, 10:41:45) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from test import test_tokenize
[50808 refs]
>>> test_tokenize.test_tokenize_all()
processing file:
s:\src\svn\svn.python.org\projects\python\branches\py3k\lib\test\testcodec.py
Traceback (most recent call last):
File "", line 1, in
File
"s:\src\svn\svn.python.org\projects\python\branches\py3k\lib\test\test_tokenize.py",
line 565, in test_tokenize_all
print("roundtrip(open(file)): " + roundtrip(open(file)))
File
"s:\src\svn\svn.python.org\projects\python\branches\py3k\lib\test\test_tokenize.py",
line 514, in roundtrip
source = untokenize(generate_tokens(f.readline))
File
"s:\src\svn\svn.python.org\projects\python\branches\py3k\lib\tokenize.py", line
238, in untokenize
return ut.untokenize(iterable)
File
"s:\src\svn\svn.python.org\projects\python\branches\py3k\lib\tokenize.py", line
183, in untokenize
self.add_whitespace(start)
File
"s:\src\svn\svn.python.org\projects\python\branches\py3k\lib\tokenize.py", line
172, in add_whitespace
assert row <= self.prev_row
AssertionError
[52668 refs]
Yay. And to make this even more interesting:
s:\src\svn\svn.python.org\projects\python\branches\py3k\PCbuild>python_d
..\Lib\test\test_tokenize.py
doctest (test.test_tokenize) ... 62 tests with zero failures
[61919 refs]
Oh, and while we're here:
s:\src\svn\svn.python.org\projects\python\branches\py3k\PCbuild>python_d
..\lib\test\regrtest.py -q -uall -rw test_tokenize
**
File
"s:\src\svn\svn.python.org\projects\python\branches\py3k\lib\test\test_tokenize.py",
line ?, in test.test_tokenize.__test__.doc
tests
Failed example:
for testfile in testfiles:
if not roundtrip(open(testfile)): break
else: True
Exception raised:
Traceback (most recent call last):
File
"s:\src\svn\svn.python.org\projects\python\branches\py3k\lib\doctest.py", line
1227, in __run
compileflags, 1), test.globs)
File "", line 2, in
if not roundtrip(open(testfile)): break
File "", line 3, in
roundtrip
token_list = list(generate_tokens(f.readline))
File
"s:\src\svn\svn.python.org\projects\python\branches\py3k\lib\tokenize.py", line
264, in generate_tokens
Re: [Python-Dev] 3.0 buildbots all red
As it turns out, it's not memory related, but has to do with tokenize not supporting coding cookies in files. Mark picked up on this and linked it to an issue already in roundup that was raised way back in 2003: http://bugs.python.org/issue71988. I've just finished patching test_tokenizer.py to better represent this test case -- the current implementation doesn't lend itself very well to being debugged when things go wrong (I think Mark and I both felt like we were on a bit of a wild goose chase). I've fixed that and have a bunch of text files with various utf-8/bom sig permutations that are now used to test tokenizer's compliance with PEP 0263. I'll upload that now then move on to actually patching tokenizer.py. Trent "wishes-there-was-somewhere-to-get-some-food-after-11pm-at-pycon" Nelson. From: [EMAIL PROTECTED] [EMAIL PROTECTED] On Behalf Of ocean [EMAIL PROTECTED] Sent: 17 March 2008 01:34 To: Neal Norwitz; Mark Dickinson Cc: Python Dev Subject: Re: [Python-Dev] 3.0 buildbots all red > Yeah, sounds like a memory issue. Did you try running with valgrind > or purify? I haven't done so for a long time, perhaps never on 3k > branch. It would be a good thing to run a tool soon. Maybe is this related? [Potential overflows due to incorrect usage of PyUnicode_AsString] http://bugs.python.org/issue1950 Thank you. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/tnelson%40onresolve.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 3.0 buildbots all red
> As it turns out, it's not memory related, but has to do with > tokenize not supporting coding cookies in files. > Mark picked up on this and linked it to an issue already > in roundup that was raised way back in 2003: > http://bugs.python.org/issue71988. Oops, left off an 8. That's meant to read http://bugs.python.org/issue719888. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Consistent platform name for 64bit windows (was: distutils.util.get_platform() for Windows)
+1 for avoiding a bikeshed, so +1 to AMD64. From: [EMAIL PROTECTED] [EMAIL PROTECTED] On Behalf Of Christian Heimes [EMAIL PROTECTED] Sent: 18 March 2008 13:54 To: [EMAIL PROTECTED] Cc: [EMAIL PROTECTED]; python-dev@python.org Subject: Re: [Python-Dev] Consistent platform name for 64bit windows (was: distutils.util.get_platform() for Windows) [EMAIL PROTECTED] schrieb: > So, at the risk of painting a bike-shed, I'd like to propose that we adopt > 'AMD64' in distutils (needs a change), platform.py (needs a change to use > sys.getwindowsversion() in preference to pywin32, if possible, anyway), > and the Python banner (which already uses AMD64). +1 for AMD64 If we ever need names for Itanium and i386 compatible arch I propose IA64 and X86. Christian ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/tnelson%40onresolve.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] r61577 - in python/trunk: Include/code.h Include/compile.h Include/parsetok.h Include/pythonrun.h Lib/__future__.py Lib/test/test_print.py Misc/ACKS Misc/NEWS Parser
This change breaks all the trunk buildbots: == ERROR: testCompileLibrary (test.test_compiler.CompilerTest) -- Traceback (most recent call last): File "S:\buildbots\python\trunk.nelson-windows\build\lib\test\test_compiler.py", line 52, in testCompileLibrary compiler.compile(buf, basename, "exec") File "S:\buildbots\python\trunk.nelson-windows\build\lib\compiler\pycodegen.py", line 64, in compile gen.compile() File "S:\buildbots\python\trunk.nelson-windows\build\lib\compiler\pycodegen.py", line 112, in compile gen = ModuleCodeGenerator(tree) File "S:\buildbots\python\trunk.nelson-windows\build\lib\compiler\pycodegen.py", line 1275, in __init__ self.futures = future.find_futures(tree) File "S:\buildbots\python\trunk.nelson-windows\build\lib\compiler\future.py", line 59, in find_futures walk(node, p1) File "S:\buildbots\python\trunk.nelson-windows\build\lib\compiler\visitor.py", line 106, in walk walker.preorder(tree, visitor) File "S:\buildbots\python\trunk.nelson-windows\build\lib\compiler\visitor.py", line 63, in preorder self.dispatch(tree, *args) # XXX *args make sense? File "S:\buildbots\python\trunk.nelson-windows\build\lib\compiler\visitor.py", line 57, in dispatch return meth(node, *args) File "S:\buildbots\python\trunk.nelson-windows\build\lib\compiler\future.py", line 27, in visitModule if not self.check_stmt(s): File "S:\buildbots\python\trunk.nelson-windows\build\lib\compiler\future.py", line 37, in check_stmt "future feature %s is not defined" % name SyntaxError: future feature print_function is not defined From: [EMAIL PROTECTED] [EMAIL PROTECTED] On Behalf Of eric.smith [EMAIL PROTECTED] Sent: 18 March 2008 19:45 To: [EMAIL PROTECTED] Subject: [Python-checkins] r61577 - in python/trunk: Include/code.h Include/compile.h Include/parsetok.h Include/pythonrun.h Lib/__future__.py Lib/test/test_print.py Misc/ACKS Misc/NEWSParser/parser.c Parser/parsetok.c Python/bltinmodule.c Python/future.c Pyth... Author: eric.smith Date: Wed Mar 19 00:45:49 2008 New Revision: 61577 Added: python/trunk/Lib/test/test_print.py Modified: python/trunk/Include/code.h python/trunk/Include/compile.h python/trunk/Include/parsetok.h python/trunk/Include/pythonrun.h python/trunk/Lib/__future__.py python/trunk/Misc/ACKS python/trunk/Misc/NEWS python/trunk/Parser/parser.c python/trunk/Parser/parsetok.c python/trunk/Python/bltinmodule.c python/trunk/Python/future.c python/trunk/Python/pythonrun.c Log: Backport of the print function, using a __future__ import. This work is substantially Anthony Baxter's, from issue 1633807. I just freshened it, made a few minor tweaks, and added the test cases. I also created issue 2412, which is to check for 2to3's behavior with the print function. I also added myself to ACKS. Modified: python/trunk/Include/code.h == --- python/trunk/Include/code.h (original) +++ python/trunk/Include/code.h Wed Mar 19 00:45:49 2008 @@ -48,11 +48,12 @@ #define CO_FUTURE_DIVISION 0x2000 #define CO_FUTURE_ABSOLUTE_IMPORT 0x4000 /* do absolute imports by default */ #define CO_FUTURE_WITH_STATEMENT 0x8000 +#define CO_FUTURE_PRINT_FUNCTION 0x1 /* This should be defined if a future statement modifies the syntax. For example, when a keyword is added. */ -#if 0 +#if 1 #define PY_PARSER_REQUIRES_FUTURE_KEYWORD #endif Modified: python/trunk/Include/compile.h == --- python/trunk/Include/compile.h (original) +++ python/trunk/Include/compile.h Wed Mar 19 00:45:49 2008 @@ -24,6 +24,8 @@ #define FUTURE_DIVISION "division" #define FUTURE_ABSOLUTE_IMPORT "absolute_import" #define FUTURE_WITH_STATEMENT "with_statement" +#define FUTURE_PRINT_FUNCTION "print_function" + struct _mod; /* Declare the existence of this type */ PyAPI_FUNC(PyCodeObject *) PyAST_Compile(struct _mod *, const char *, Modified: python/trunk/Include/parsetok.h == --- python/trunk/Include/parsetok.h (original) +++ python/trunk/Include/parsetok.h Wed Mar 19 00:45:49 2008 @@ -27,6 +27,10 @@ #define PyPARSE_WITH_IS_KEYWORD0x0003 #endif +#define PyPARSE_PRINT_IS_FUNCTION 0x0004 + + + PyAPI_FUNC(node *) PyParser_ParseString(const char *, grammar *, int, perrdetail *); PyAPI_FUNC(node *) PyParser_ParseFile (FILE *, const char *, grammar *, int, Modified: python/trunk/Include/pythonrun.h == --- python/trunk/Include/pythonrun.h(o
Re: [Python-Dev] 3.0 buildbots all red
> > Sounds like a challenge if ever I've heard one -- care to wager a beer on > > it? > > (Only applies to buildbots that are connected/online.) > Make sure you get a screen shot for OnYourDesktop if/when they *do* go > green! Screenshot? I'm going to buy a pack of iron-on transfers and sell t-shirts of it online. "All the buildbots were green momentarily after PyCon 2008... and all I got was this lousy t-shirt." Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] First green Windows x64 buildbots!
We've just experienced our first 2.6 green x64 Windows builds on the build slaves! Well, almost green. Thomas's 'amd64 XP trunk' ran out of disk: 304 tests OK. 1 test failed: test_largefile == ERROR: test_seek (test.test_largefile.TestCase) -- Traceback (most recent call last): File "C:\buildbot\trunk.heller-windows-amd64\build\lib\test\test_largefile.py", line 42, in test_seek f.flush() IOError: [Errno 28] No space left on device Sorry about that Thomas ;-) Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] how to build extensions for Windows?
Having recently sunk a lot of time into the Windows build process, I'd recommend going with Visual C++ Express 2008 rather than MinGW, as this is the official compiler for 2.6/3.0. (You can download a free copy.) FWIW, I've probably been working on the Windows build side of things on and off for the past month or so, and we've only just reached a point where 32bit and 64bit Windows builds are compiling with all extension modules (bsddb, tcl/tk, ssl etc) and passing all tests (most work has gone into the x64 builds though, the 32-bit ones were already green on XP and below for 32bit). Using MinGW/gcc on Windows hasn't seen anywhere near so much attention, so, YMWV. In terms of my Windows-oriented priorities, they are as follows: - Get 3.0 32/64 Windows builds actually compiling successfully and then passing all tests (given that all build slaves for 3.0 are red that's not exactly a quick action). - Move on to the MSI installer improvements for 2.6/3.0, specifically with regards to the VCRT9 runtime and signing of the installer/binaries. - Maybe putting some cycles into Python builds on MinGW. To be honest though, the main motivation for doing that will be to demonstrate that a Python executable compiled with Visual Studio 2008 Professional with Profile Guided Optimisation will outperform a MinGW/gcc build ;-) Trent. From: [EMAIL PROTECTED] [EMAIL PROTECTED] On Behalf Of Bill Janssen [EMAIL PROTECTED] Sent: 19 March 2008 20:02 To: python-dev@python.org Subject: [Python-Dev] how to build extensions for Windows? I've set up a Parallels virtual machine on my Mac, and have succeeded in getting Windows XP running in it! And I've installed MinGW, as well. Now I'd like to learn how to build the SSL module from source on Windows for Python 2.5.2. Is there any documentation on the process of building an extension from scratch that's appropriate for someone who doesn't know much about Windows? I'm looking for step-by-step. What about this? http://www.mingw.org/MinGWiki/index.php/Python%20extensions Bill ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/tnelson%40onresolve.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] trunk buildbot status
Quick update on the status of the trunk buildbots: Failing: [x86 gentoo trunk (Neal Norwitz)] This has been failing at the same point for the past couple of days now: test_sqlite command timed out: 1800 seconds without output, killing pid 15168 process killed by signal 9 program finished with exit code -1 None of the other buildbots seem to be encountering the same problem. Neal, got any idea what's going on with this one? [alpha True64 5.1 trunk (Neal Norwitz)] test_tarfile started failing recently (within the past few days) with CRC checks. See http://www.python.org/dev/buildbot/trunk/alpha%20Tru64%205.1%20trunk/builds/2712/step-test/0. Greg updated the test such that it prints out some more detail about the failure so we're waiting on that at the moment. [hppa Ubuntu trunk (Matthias Klose)] This has been consistently failing in test_socketserver for as long as I can remember: test_socketserver make: *** [buildbottest] Alarm clock program finished with exit code 2 I just updated that test such that it waits 20 seconds instead of 3 seconds at the end of the test if the server hasn't shutdown -- waiting for the test results of this still. [x86 XP trunk (Joseph Armbruster)] This box didn't survive the recent build changes, but I can't figure out why, none of the other Windows boxes encounter this error: The following error has occurred during XML parsing: File: C:\python\buildarea\trunk.armbruster-windows\build\PCbuild\_bsddb.vcproj Line: 179 Column: 1 Error Message: Illegal qualified name character. The file 'C:\python\buildarea\trunk.armbruster-windows\build\PCbuild\_bsddb.vcproj' has failed to load. Can someone check a clean trunk build on a Windows system that *only* has Visual C++ Express 2008? The latest build system updates don't rely on any features of Visual Studio Professional, but the tools use a lot of common files, and perhaps a Service Pack needs to be applied or something. [amd64 XP trunk (Thomas Heller)] Builds fine, all tests pass except for test_largefile, which is failing as there's no more space left on the drive ;-) [x86 XP-4 trunk (David Bolen)] This is currently flagged as having failed test, but I don't think it's finished building since the finalised build updates, so hopefully the BSDDB errors in the last run will be resolved when it finished the latest build. [x86 FreeBSD 2 trunk (Jeroen Ruigrok van der Werven)] This is a FreeBSD 6.3-STABLE box (which switched to curses 5.6 from 5.2) -- there's been an ongoing thread with regards to why curses has started failing, Jeroen can probably provide more info on that. Either way I don't anticipate a quick fix for this particular slave, unfortuantely. Neal/Martin, I'd like to promote the following slaves to the stable list: [g4 osx.4] [x86 W2k8] [AMD64 W2k8] [ppc Debian unstable] [sparc Ubuntu] [sparc Debian] [PPC64 Debian] [S-390 Debian] [x86 XP-3] [amd64 XP] [x86 FreeBSD] [x86 FreeBSD 3] The trunk builds of these slaves have been the most reliable since I've been tracking. Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] trunk buildbot status
I'd recommend cd'ing to your trunk root directory and running Tool\buildbot\build.bat from there -- it'll automatically check out all the dependencies and build via command line with vcbuild (building via Visual Studio usually always Does The Right Thing, command line builds often take a bit more coercing). From: Eric Smith [EMAIL PROTECTED] Sent: 19 March 2008 20:49 To: Trent Nelson Cc: python-dev@python.org; [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED] Subject: Re: [Python-Dev] trunk buildbot status Trent Nelson wrote: > Quick update on the status of the trunk buildbots: > > [x86 XP trunk (Joseph Armbruster)] > This box didn't survive the recent build changes, but I can't figure out why, > none of the other Windows boxes encounter this error: > The following error has occurred during XML parsing: > File: C:\python\buildarea\trunk.armbruster-windows\build\PCbuild\_bsddb.vcproj > Line: 179 > Column: 1 > Error Message: > Illegal qualified name character. > The file > 'C:\python\buildarea\trunk.armbruster-windows\build\PCbuild\_bsddb.vcproj' > has failed to load. > > Can someone check a clean trunk build on a Windows system that *only* has > Visual C++ Express 2008? The latest build system updates don't rely on any > features of Visual Studio Professional, but the tools use a lot of common > files, and perhaps a Service Pack needs to be applied or something. I just built the trunk on a Windows XP x86 box that only has Visual C++ Express 2008 installed. I got a bunch of errors with sqlite, tcl, db-4.4.20, and ssl, but the interpreter built and appears to run ok. But since I don't have bsddb installed, I don't think I'm executing the portion of the build process that you find failing. I don't have time to install bsddb tonight, but I can do that in about 24 hours if you still need me to. Eric. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] trunk buildbot status
Thanks Eric, very useful to know. I guess it's just that particular build slave... From: Eric Smith [EMAIL PROTECTED] Sent: 20 March 2008 02:55 To: Trent Nelson Cc: python-dev@python.org Subject: Re: [Python-Dev] trunk buildbot status Trent Nelson wrote: > I'd recommend cd'ing to your trunk root directory and running > Tool\buildbot\build.bat from there -- it'll automatically check out all the > dependencies and build via command line with vcbuild (building via Visual > Studio usually always Does The Right Thing, command line builds often take a > bit more coercing). Okay, that's extremely helpful. With that (and installing nasmw.exe), a trunk checkout builds correctly and passes all tests (although skipping test_tcl) on my box. As I said, it's XP x86 with 2008 Express Edition only. Let me know if I can provide any other information. Unfortunately I don't have access to this box during the work day (EDT), and I'm leaving for vacation tomorrow (Friday). But I'll help as best I can. Eric. > > > From: Eric Smith [EMAIL PROTECTED] > Sent: 19 March 2008 20:49 > To: Trent Nelson > Cc: python-dev@python.org; [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL > PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED] > Subject: Re: [Python-Dev] trunk buildbot status > > Trent Nelson wrote: >> Quick update on the status of the trunk buildbots: >> >> [x86 XP trunk (Joseph Armbruster)] >> This box didn't survive the recent build changes, but I can't figure out >> why, none of the other Windows boxes encounter this error: >> The following error has occurred during XML parsing: >> File: >> C:\python\buildarea\trunk.armbruster-windows\build\PCbuild\_bsddb.vcproj >> Line: 179 >> Column: 1 >> Error Message: >> Illegal qualified name character. >> The file >> 'C:\python\buildarea\trunk.armbruster-windows\build\PCbuild\_bsddb.vcproj' >> has failed to load. >> >> Can someone check a clean trunk build on a Windows system that *only* has >> Visual C++ Express 2008? The latest build system updates don't rely on any >> features of Visual Studio Professional, but the tools use a lot of common >> files, and perhaps a Service Pack needs to be applied or something. > > I just built the trunk on a Windows XP x86 box that only has Visual C++ > Express 2008 installed. I got a bunch of errors with sqlite, tcl, > db-4.4.20, and ssl, but the interpreter built and appears to run ok. > > But since I don't have bsddb installed, I don't think I'm executing the > portion of the build process that you find failing. > > I don't have time to install bsddb tonight, but I can do that in about > 24 hours if you still need me to. > > Eric. > ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] fixing tests on windows
> def rename_and_remove (filename): > os.rename (filename, filename + ".deleted") > os.remove (filename + ".deleted") > Isn't this still going to run into problems when the rename > fails because the earlier tests remove still left the .deleted > file around due to some other running desktop search service > that now has the .deleted file open? I haven't looked into all the various places the tests write temp files to, but if we could localise everything to a common root directory, i.e. %TEMP%\python-regrtest, we could then attempt to blow this away at the start of regrtest.py before any tests run, and refuse to run if this fails. This would be in combination with the unlinking/renaming approach discussed. This approach seems like it would cover all bases a bit more effectively. Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] No time for svn merge
> > Yes, that's all I meant: make it the committer's job > > to merge or block as appropriate. I just wasn't sure if > > there was some reason that this would be difficult or > > undesirable. > > Ah, yes. It is indeed difficult or undesirable, or was > so in the past: Some committers don't care (didn't care) > at all about 3k. They would have to setup sandboxes, > learn what the nature of changes is, and invest some > regular time into forward-porting. Is this *really* the case still? Who are these rogue committers? ;-) I think holding a developer accountable for merging or blocking to py3k when they commit to trunk is a great idea. Who better to pass judgement on such an activity than the person closest to it? Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Distutils] FW: [issue2513] 64bit cross compilation on windows
> Further, I > assert that there are a greater number of build tools which do not support > cross-compilation, but will build natively on x64 and expect 'PCBuild' > to have libraries they can link with to create an x64 binary. I'm with Martin on this one as well I think. If I understand correctly, you're proposing: PCbuild - may have contents of x86 or x64 depending on the build machine's architecture PCbuild/amd64 - always x64 PCbuild/x86 - always x86 And what we've currently got is: PCbuild/- always x86 PCbuild/amd64 - always x64 I think this is fine; we don't really have a notion of compiling for a native platform, nor is the build machine's architecture factored into the equation. I believe this is a Good Thing. If you want a 32-bit build, use the 'Win32' target platform in VS; if you want a 64-bit build, use 'x64'. Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-3000] the release gods are angry at python
> > In the py3k branch I've assigned the audio resource to the winsound > > tests. Only regrtest.py -uall or -uaudio runs the winsound test. > Reason: > > the test sound was freaking out my poor cat. :/ > > I feel with your cat ;-). > This would not help on the buildbot since it runs 'rt.bat -d -q -uall - > rw'. I feel for the poor NOC engineers at my colo that freak out when some random server in a farm of thousands starts making bizarre sounds. I detest test_winsound. There are so many corner cases you need to account for that makes the test pointless as you end up wrapping everything in except: pass blocks. Does the system have a legacy beep driver? Is it enabled? Is it disabled? Is there a sound card? Is it enabled or disabled? Pah! +1 to removing audio out of -uall, if only for the sake of cats, erroneously red buildbots, and poor ServerCentral NOC engineers. Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] No time for svn merge
Christian Heimes [mailto:[EMAIL PROTECTED]: > Trent Nelson schrieb: > > I think holding a developer accountable for merging or blocking to > py3k when they commit to trunk is a great idea. Who better to pass > judgement on such an activity than the person closest to it? > > Blocking a revision makes my job as The Merger easier. > > I'm not so sure about the merging part. It takes some experience with > the Python 3.0 code base to know the subtle differences in the C API. > Most merges are straight forward for me. If you enforce the forward > merging on every developer it may slow down development. > Each regular merge takes me about 45 minutes of computer time but less > than 15 supervisor time. The computer time is mostly compile and test > time in the background. If everybody merges her own code to 3.0 it > still takes *everybody* about 10 minutes of time and 45 minutes of computer > time. Ah, right, I wasn't thinking about the implication of code affecting the C base for some reason, but that's entirely reasonable. Perhaps each developer should be accountable for either: a) blocking b) merging, if they're able to do so c) if they're unable to merge, replying to the relevant python-checkins@ e-mail indicating that they're unable to handle trunk -> py3k for whatever reason (e.g. not familiar with py3k code base) Other developers could then pitch in and help merge if someone requests it via e-mail. I'd think that would make The Merger's life easier. Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Tools\buildbot\kill_python.c can't be helping our cause
Looking into some of the recent Windows buildbot failures, I see things like
this:
sqlite3 : error PRJ0008 : Could not delete file
'c:\buildbot\trunk.heller-windows-amd64\build\PCbuild\amd64\sqlite3_d.dll'.
build-amd64.bat doesn't go through the kill_python.c hoopla, so I figure the
above error is being caused by the fact that an erroneous/stalled python_d.exe
from a previous run is still open. I was looking at modifying kill_python.c to
accept an 'x64' argument if we want to kill amd64\python_d.exe instead of the
usual 32-bit exe, however, this caught my attention:
if ((strstr(path, "pcbuild\\python_d.exe") != NULL) ||
(strstr(path, "\\build\\python.exe") != NULL)) {
printf("Terminating %s (pid %d)\n", path, pids[i]);
if (!TerminateProcess(hProcess, 1)) {
printf("Termination failed: %d\n", GetLastError());
return 1;
}
return 0;
That'll kill the first python_d.exe instance it finds matching the given path;
given that our buildbots run trunk/release25-maint/py3k in parallel, it seems
as though it wouldn't be hard for us to get into a situation where
kill_python.exe ends up killing the wrong python_d.exe (i.e. trunk checkin,
trunk builds, starts testing, py3k checkin, calls kill_python.exe, kills
trunk's python_d.exe that was in the middle of testing).
That can't be helping our cause, unless I'm missing something... Unless anyone
advises otherwise, I'll start on a patch.
Trent.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tools\buildbot\kill_python.c can't be helping our cause
> > That'll kill the first python_d.exe instance it finds matching the > > given path; given that our buildbots run trunk/release25-maint/py3k > > in parallel > > That's actually not a given: we currently *don't* run multiple builds > simultaneously on the same slave. I thought the slave lock only applied per branch, not per host? > > Unless anyone advises otherwise, I'll start on a patch. > > If you can make it less error-prone, sure, go ahead. Spent a bit of time on it this evening; as it happens, in order to enumerate 64-bit processes, you need to be a 64-bit process yourself. As it's much easier managing 32-bit vs. x64 binaries when they're a .vcproj part of pcbuild.sln, I'm looking into adding kill_python as a .vcproj and configure the solution such that it builds and runs this before any other project. That'll automatically take care of choosing the right version to run depending on whether 'Win32' or 'x64' is selected as the platform. It'll also simplify the verification logic that checks if it's the right python_d.exe -- the path of the .exe needs to match the path of the running kill_python.exe. Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] fixing tests on windows
[Disclaimer: thought dump e-mail, signal to noise ratio may be subpar.] Sounds like you're at least making steps forward in the right direction, despite the activity probably being quite disheartening. Based on what you've said below and the rest of the conversation, here are my thoughts for an approach: 1. For a given python[_d].exe, always use the same test directory, but hash it against the entire python process path such that it's unique only for a given python instance. 2. Make sure every time a test wants a temp file, it gets a unique one in this directory. Sounds like your TESTFN modification would take care of this for those tests using TESTFN; if TESTFN is the preferred approach then any other tests using tempfile or hardcoding file names would then be changed to use this instead. 3. In order to address tests that either call the test_support methods for removing files/directories, or those that call os.(unlink|remove), do what ever needs to be done to make these no-ops on Windows if an error occurs. 4. At the end of the regrtest.py run, create a suspended arbitrary process (i.e. explorer.exe), hijack the main thread context of the process and inject a routine (i.e. overwrite the thread context's instruction pointers) that takes care of removing the temporary directory that was used for testing -- patiently re-trying if any failures occur until all rogue processes also accessing the file(s) stop doing so. Resume the thread before exiting python. Heh. Sounds crazy? It did to me as well, until I came across http://www.catch22.net/tuts/selfdel.asp, which documents the approach. It's not particularly necessary in our case, we could simply spawn another python process at the end of regrtest.py that patiently attempts to remove the test directory we just used when the python process that was executing regrtest.py exits. We could then modify regrtest.py such that it will use the same approach if the hashed test directory already exists at the start of a run and it can't remove it via os.unlink. If we *still* run into issues here on the buildbots, say if regrtest.py blocks on our helper process, which for the life of it can't remove some/all of the test files -- it'd be interesting to write something that grok's all open handles for all processes and attempts to figure out what it is that keeps these files open -- i.e. same thing that procexp.exe does when you search for a handle. Or, keeping it simple, rather than a separate process and hashed test directory based on python process path, just have a common directory, i.e. %TEMP%\python-regrtest, and use an incrementing sequence number for each test run's test directory, i.e. if there are directories 001, 002 and 003 in the temp dir, when regrtest.py starts, it'll try delete all of these -- if it can't (which is what we'd want if another test is already running), it adds 1 to the highest numbered directory it couldn't delete. Guess it all depends on how much effort we want to put into cleaning up our test directory really -- just ensuring tests get a clean area and unique file names each run is the easy part. Trent. From: [EMAIL PROTECTED] [EMAIL PROTECTED] On Behalf Of Tim Golden [EMAIL PROTECTED] Sent: 03 April 2008 09:39 To: Python-Dev Subject: Re: [Python-Dev] fixing tests on windows [re tests which fail because something's holding a file open with SHARE_DELETE] Well I've tried my best, but I can't come up with a solution which guarantees to overcome this obstacle. I set up a fairly aggressive directory watcher which, when it sees a test file being created, takes a SHARE_DELETE handle on it and releases it immediately. (Guessing that this is what the other apps are doing). Running the test suite from HEAD, this generates all manner of access-denied type errors as per the original output. I used tempfile to generate a random TESTFN in the current directory rather than the static @test. And implemented the rename-shim discussed previously, renaming to a different random name, courtesy of mktemp. With those in place, most tests run without error. But I'm still getting errors in the same sort of areas which Steven B originally reported. The one error I can't see a way round is the access denied (which manifests as Permission Error) which is the documented result of attempting to open a file with a pending delete -- ie the delete succeeded but hasn't completed yet. An additional complication is that there are hundreds of instances throughout the tests where the test simply calls os.unlink/os.remove to undo the test file. To have some more robust central deletion I had to go through and update 68 tests. I'll keep trying, but in the current state I'm not convinced the situation's improved enough for me to bother uploading a patch. TJG ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/m
Re: [Python-Dev] Tools\buildbot\kill_python.c can't be helping our cause
Committed new version of kill_python to trunk in r62129. Trent. From: "Martin v. Löwis" [EMAIL PROTECTED] Sent: 02 April 2008 14:39 To: Trent Nelson Cc: python-dev@python.org Subject: Re: [Python-Dev] Tools\buildbot\kill_python.c can't be helping our cause > That'll kill the first python_d.exe instance it finds matching the > given path; given that our buildbots run trunk/release25-maint/py3k > in parallel That's actually not a given: we currently *don't* run multiple builds simultaneously on the same slave. > Unless anyone advises otherwise, I'll start on a patch. If you can make it less error-prone, sure, go ahead. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] socket.SOL_REUSEADDR: different semantics between Windows vs Unix (or why test_asynchat is sometimes dying on Windows)
I started looking into this:
http://www.python.org/dev/buildbot/all/x86%20W2k8%20trunk/builds/289/step-test/0
Pertinent part:
test_asyncore
test_asynchat
command timed out: 1200 seconds without output
SIGKILL failed to kill process
using fake rc=-1
program finished with exit code -1
remoteFailed: [Failure instance: Traceback from remote host -- Traceback (most
recent call last):
Failure: buildbot.slave.commands.TimeoutError: SIGKILL failed to kill process
]
I tried to replicate it on the buildbot in order to debug, which, surprisingly,
I could do consistently by just running rt.bat -q -d -uall test_asynchat. As
the log above indicates, the python process becomes completely and utterly
wedged, to the point that I can't even attach a remote debugger and step into
it.
Digging further, I noticed that if I ran the following code in two different
python consoles, EADDRINUSE was *NOT* being raised by socket.bind():
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind(('127.0.0.1', 54322))
However, take out the setsockopt line, and wallah, the second s.bind() will
raise EADDRINUSE, as expected. This manifests into a really bizarre issue with
test_asynchat in particualr, as subsequent sock.accept() calls on the socket
put python into the uber wedged state (can't even ctrl-c out at the console,
need to kill the process directly).
Have to leave the office and head home so I don't have any more time to look at
it tonight -- just wanted to post here for others to mull over.
Trent.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] fixing tests on windows
I agree with Tim, you can jump through as many hoops as you want (setting directories private, using %TEMP% exclusively, etc), but I doubt anything is going to change the behaviour of things like virus scanners, for example. Tim, let me know if you need help with anything, perhaps we could set up a temporary branch outside of trunk to play around with various approaches to see what works best. This'll ensure we don't adversely affect the main buildbots, but also give us the option to get different Windows boxes to build our test branch on demand. Trent. From: [EMAIL PROTECTED] [EMAIL PROTECTED] On Behalf Of Tim Golden [EMAIL PROTECTED] Sent: 04 April 2008 04:04 Cc: python-dev@python.org Subject: Re: [Python-Dev] fixing tests on windows Terry Reedy wrote: > If the testdir disallows the search indexer, then there should be no need > to disable Windows Search Service. If privatizing the dir kept other > programs out, then likewise. > > | Or were you suggesting that there is some programmatic way for the > | test suite to create directories that disallow the Search Service, > | etc.? > > I suspect, but do not know, that the dialog box effects changes through > user-programmable interfaces. So while I would start with manual changes > to see if that solves the problem, I presume there must be a system call > for changing dir attributes. The problem is, I think, that it isn't just the Indexing Service which generates this issue. TortoiseSVN is well known for doing the same thing, and there could be now and potentially will be in the future other programs. I don't think that hunting down and turning off their interference case by case is a viable solution in the long-term. Although it would obviously be a way forward in the short term, _faute de mieux_. [Tony Nelson] > I'd think that files and directories created in the TEMP > directory would normally not be indexed on any OS, including > MSWindows. But this is just a guess. I'm inclined to think you're right. And a first pass I had at producing a solution simply used tempfile to do everything. Unfortunately that's far more invasive than I was really comfortable with at the time: at the very least, you have to patch several tests which fail if there's an (escaped) backslash in the path. However, it's clear that my attempt to cause the minimum damage isn't enough to clear the problem 100%. So I think the next move is indeed to turn test_support.TESTFN into a function (in some way) which generates a unique tempfile reference, possibly with a context manager to clean up. Or something. The complication is that, while most test simply want a handy file to exist or be written to, and don't really care what happens afterwards, some tests are testing the very mechanism of creating/deleting a file etc. So a wholesale replacement isn't necessarily straightforward. On we go. TJG ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/tnelson%40onresolve.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] r62129 - in python/trunk: PCbuild/debug.vsprops PCbuild/kill_python.c PCbuild/kill_python.vcproj PCbuild/pcbuild.sln PCbuild/pythoncore.vcproj PCbuild/release.vsprops Tools/buildbot/M
> I don't like the part where the solution kills the Python process during > a rebuild. It's too surprising for the user. Hmmm. When you say rebuild, I assume you mean the change I made to the pythoncore project's pre-link step to call kill_python.exe, and not to the post-build step of kill_python that runs itself? Personally, when I'm doing development, if I've got the pcbuild\python_d.exe console open, it's usually to test one liners, I'm not using it to do anything important. If I forget to close it before I kick off a build, it's annoying running into errors at the link stage, I'd certainly prefer the build system to kill off anything that'll inhibit a successful link before actually linking. What do others that do Windows development think? I don't have a problem changing the build behaviour if the approach I've taken is generally disliked. Trent. From: Christian Heimes [EMAIL PROTECTED] Sent: 04 April 2008 09:25 To: [EMAIL PROTECTED]; Trent Nelson Subject: Re: r62129 - in python/trunk: PCbuild/debug.vsprops PCbuild/kill_python.c PCbuild/kill_python.vcproj PCbuild/pcbuild.sln PCbuild/pythoncore.vcproj PCbuild/release.vsprops Tools/buildbot/Makefile Tools/buildbot/build-amd64.bat Tools/buildbot/bui... trent.nelson schrieb: > Author: trent.nelson > Date: Thu Apr 3 20:27:06 2008 > New Revision: 62129 > > Added: >python/trunk/PCbuild/kill_python.c (contents, props changed) >python/trunk/PCbuild/kill_python.vcproj > Removed: >python/trunk/Tools/buildbot/Makefile >python/trunk/Tools/buildbot/kill_python.bat >python/trunk/Tools/buildbot/kill_python.c >python/trunk/Tools/buildbot/kill_python.mak > Modified: >python/trunk/PCbuild/debug.vsprops >python/trunk/PCbuild/pcbuild.sln >python/trunk/PCbuild/pythoncore.vcproj >python/trunk/PCbuild/release.vsprops >python/trunk/Tools/buildbot/build-amd64.bat >python/trunk/Tools/buildbot/build.bat >python/trunk/Tools/buildbot/buildmsi.bat > Log: > Reimplement kill_python. The existing version had a number of flaws, namely, > it didn't work for x64 and it wasn't precise about which python_d.exe it was > killing -- it just killed the first one it came across that happened to have > 'pcbuild\python_d.exe' or 'build\python_d.exe' in it's path. The new version > has been rewritten from the ground up and now lives in PCbuild, instead of > Tools\buildbot, and it has also been incorporated into the Visual Studio > solution (pcbuild.sln) as 'kill_python'. The solution has also been altered > such that kill_python is called where necessary in the build process in order > to prevent any linking errors due to open file locks. In lieu of this, all > of the existing bits and pieces in Tools\buildbot that called out to > kill_python at various points have also be > en removed as they are now obsolete. Tested on both Win32 and x64. I don't like the part where the solution kills the Python process during a rebuild. It's too surprising for the user. Christian ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] socket.SOL_REUSEADDR: different semantics between Windows vs Unix (or why test_asynchat is sometimes dying on Windows)
I've raised issue 2550 to track this problem. I've also provided a patch on
the tracker to test_socket.py that reproduces the issue. Anyone mind if I
commit this to trunk? I'd like to observe if any other platforms exhibit
different behaviour via buildbots. It'll cause all the Windows slaves to fail
on test_socket though. (I can revert it once I've seen how the buildbots
behave until I can come up with an actual patch for Windows that fixes the
issue.)
http://bugs.python.org/issue2550
http://bugs.python.org/file9939/test_socket.py.patch
Trent.
From: [EMAIL PROTECTED] [EMAIL PROTECTED] On Behalf Of Trent Nelson [EMAIL
PROTECTED]
Sent: 03 April 2008 22:40
To: python-dev@python.org
Subject: [Python-Dev] socket.SOL_REUSEADDR: different semantics between Windows
vs Unix (or why test_asynchat is sometimes dying on Windows)
I started looking into this:
http://www.python.org/dev/buildbot/all/x86%20W2k8%20trunk/builds/289/step-test/0
Pertinent part:
test_asyncore
test_asynchat
command timed out: 1200 seconds without output
SIGKILL failed to kill process
using fake rc=-1
program finished with exit code -1
remoteFailed: [Failure instance: Traceback from remote host -- Traceback (most
recent call last):
Failure: buildbot.slave.commands.TimeoutError: SIGKILL failed to kill process
]
I tried to replicate it on the buildbot in order to debug, which, surprisingly,
I could do consistently by just running rt.bat -q -d -uall test_asynchat. As
the log above indicates, the python process becomes completely and utterly
wedged, to the point that I can't even attach a remote debugger and step into
it.
Digging further, I noticed that if I ran the following code in two different
python consoles, EADDRINUSE was *NOT* being raised by socket.bind():
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind(('127.0.0.1', 54322))
However, take out the setsockopt line, and wallah, the second s.bind() will
raise EADDRINUSE, as expected. This manifests into a really bizarre issue with
test_asynchat in particualr, as subsequent sock.accept() calls on the socket
put python into the uber wedged state (can't even ctrl-c out at the console,
need to kill the process directly).
Have to leave the office and head home so I don't have any more time to look at
it tonight -- just wanted to post here for others to mull over.
Trent.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/tnelson%40onresolve.com
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] r62129 - in python/trunk: PCbuild/debug.vsprops PCbuild/kill_python.c PCbuild/kill_python.vcproj PCbuild/pcbuild.sln PCbuild/pythoncore.vcproj PCbuild/release.vsprops Tools/buildbot/M
Ok, I'll change the approach this weekend. Trent. From: "Martin v. Löwis" [EMAIL PROTECTED] Sent: 04 April 2008 19:57 To: Trent Nelson Cc: Christian Heimes; python-dev@python.org Subject: Re: [Python-Dev] r62129 - in python/trunk: PCbuild/debug.vsprops PCbuild/kill_python.c PCbuild/kill_python.vcproj PCbuild/pcbuild.sln PCbuild/pythoncore.vcproj PCbuild/release.vsprops Tools/buildbot/Makefile Tools/buildbot/build-amd64.bat Tools/buildbo... > What do others that do Windows development think? I don't have a > problem changing the build behaviour if the approach I've taken is > generally disliked. I think kill_python should only ever be invoked in the build slaves; it should *not* be part of the regular build. If developers find they can't build because some files are still open, they should kill the processes themselves. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] socket.SOL_REUSEADDR: different semantics between Windows vs Unix (or why test_asynchat is sometimes dying on Windows)
Interesting results! I committed the patch to test_socket.py in r62152. I was expecting all other platforms except for Windows to behave consistently (i.e. pass). That is, given the following: import socket host = '127.0.0.1' sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) sock.bind((host, 0)) port = sock.getsockname()[1] sock.close() del sock sock1 = socket.socket(socket.AF_INET, socket.SOCK_STREAM) sock1.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) sock1.bind((host, port)) sock2 = socket.socket(socket.AF_INET, socket.SOCK_STREAM) sock2.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) sock2.bind((host, port)) the second bind should fail with EADDRINUSE, at least according to the 'SO_REUSEADDR and SO_REUSEPORT Socket Options' section in chapter 7.5 of Stevens' UNIX Network Programming Volume 1 (2nd Ed): "With TCP, we are never able to start multiple servers that bind the same IP address and same port: a completely duplicate binding. That is, we cannot start one server that binds 198.69.10.2 port 80 and start another that also binds 198.69.10.2 port 80, even if we set the SO_REUSEADDR socket option for the second server." The results: both Windows *and* Linux fail the patched test; none of the buildbots for either platform encountered an EADDRINUSE socket.error after the second bind(). FreeBSD, OS X, Solaris and Tru64 pass the test -- EADDRINUSE is raised on the second bind. (Interesting that all the ones that passed have a BSD lineage.) I've just reverted the test in r62156 as planned. The real issue now is that there are tests that are calling test_support.bind_socket() with the assumption that the port returned by this method is 'unbound', when in fact, the current implementation can't guarantee this: def bind_port(sock, host='', preferred_port=54321): for port in [preferred_port, 9907, 10243, 32999, 0]: try: sock.bind((host, port)) if port == 0: port = sock.getsockname()[1] return port except socket.error, (err, msg): if err != errno.EADDRINUSE: raise print >>sys.__stderr__, \ ' WARNING: failed to listen on port %d, trying another' % port This logic is only correct for platforms other than Windows and Linux. I haven't looked into all the networking test cases that rely on bind_port(), but I would think an implementation such as this would be much more reliable than what we've got for returning an unused port: def bind_port(sock, host='127.0.0.1', *args): s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.bind((host, 0)) port = s.getsockname()[1] s.close() del s sock.bind((host, port)) return port Actually, FWIW, I just ran a full regrtest.py against trunk on Win32 with this change in place and all the tests still pass. Thoughts? Trent. ____ From: [EMAIL PROTECTED] [EMAIL PROTECTED] On Behalf Of Trent Nelson [EMAIL PROTECTED] Sent: 04 April 2008 17:07 To: python-dev@python.org Subject: Re: [Python-Dev] socket.SOL_REUSEADDR: different semantics between Windows vs Unix (or why test_asynchat is sometimes dying on Windows) I've raised issue 2550 to track this problem. I've also provided a patch on the tracker to test_socket.py that reproduces the issue. Anyone mind if I commit this to trunk? I'd like to observe if any other platforms exhibit different behaviour via buildbots. It'll cause all the Windows slaves to fail on test_socket though. (I can revert it once I've seen how the buildbots behave until I can come up with an actual patch for Windows that fixes the issue.) http://bugs.python.org/issue2550 http://bugs.python.org/file9939/test_socket.py.patch Trent. From: [EMAIL PROTECTED] [EMAIL PROTECTED] On Behalf Of Trent Nelson [EMAIL PROTECTED] Sent: 03 April 2008 22:40 To: python-dev@python.org Subject: [Python-Dev] socket.SOL_REUSEADDR: different semantics between Windows vs Unix (or why test_asynchat is sometimes dying on Windows) I started looking into this: http://www.python.org/dev/buildbot/all/x86%20W2k8%20trunk/builds/289/step-test/0 Pertinent part: test_asyncore test_asynchat command timed out: 1200 seconds without output SIGKILL failed to kill process using fake rc=-1 program finished with exit code -1 remoteFailed: [Failure instance: Traceback from remote host -- Traceback (most recent call last): Failure: buildbot.slave.commands.TimeoutError: SIGKILL failed to kill process ] I tried to replicate it on the buildbot in order to debug, which, surprisingly, I could do consistently by just running rt.bat -q -d -uall test_asy
Re: [Python-Dev] socket.SOL_REUSEADDR: different semantics between Windows vs Unix (or why test_asynchat is sometimes dying on Windows)
> >"With TCP, we are never able to start multiple servers that bind > > the same IP address and same port: a completely duplicate binding. > > That is, we cannot start one server that binds 198.69.10.2 port 80 > > and start another that also binds 198.69.10.2 port 80, even if we > > set the SO_REUSEADDR socket option for the second server." > Notice that the quoted text explains that you cannot start multiple > servers that etc. Since you didn't call listen on either socket, it's > arguable that you didn't start any servers, so there should be no > surprise regarding the behavior. Try adding listen calls at various > places in the example and you'll see something different happen. I agree in principle, Stevens says nothing about what happens if you *do* try and bind two sockets on two identical host/port addresses. Even so, test_support.bind_port() makes an assumption that bind() will raise EADDRINUSE if the port is not available, which, as has been demonstrated, won't be the case on Windows or Linux. > FWIW, AIUI, SO_REUSEADDR behaves just as described in the above quote > on Linux/BSD/UNIX/etc. On Windows, however, that option actually means > something quite different. It means that the address should be stolen > from any process which happens to be using it at the moment. Probably explains why the python process wedges when this happens on Windows... > There is another option, SO_EXCLUSIVEADDRUSE, only on Windows I think, > which, AIUI, makes it impossible for another process to steal the port > using SO_REUSEADDR. Nod, if SO_EXCLUSIVEADDRUSE is used instead in the code I posted, Windows raises EADDRINUSE on the second bind(). I don't have access to any Linux boxes at the moment, so I can't test what sort of error is raised with the example I posted if listen() and accept() are called on the two sockets bound to identical addresses. Can anyone else shed some light on this? I'd be interested in knowing if the process wedges on Linux as badly as it does on Windows (to the point where it's not respecting ctrl-c or sigkill). Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] r62129 - in python/trunk: PCbuild/debug.vsprops PCbuild/kill_python.c PCbuild/kill_python.vcproj PCbuild/pcbuild.sln PCbuild/pythoncore.vcproj PCbuild/release.vsprops Tools/buildbot/M
Fixed in r62193. From: [EMAIL PROTECTED] [EMAIL PROTECTED] On Behalf Of Trent Nelson [EMAIL PROTECTED] Sent: 04 April 2008 20:41 To: "Martin v. Löwis" Cc: Christian Heimes; python-dev@python.org Subject: Re: [Python-Dev] r62129 - in python/trunk: PCbuild/debug.vsprops PCbuild/kill_python.c PCbuild/kill_python.vcproj PCbuild/pcbuild.sln PCbuild/pythoncore.vcproj PCbuild/release.vsprops Tools/buildbot/Makefile Tools/buildbot/build-amd64.bat Tools/buildbo... Ok, I'll change the approach this weekend. Trent. From: "Martin v. Löwis" [EMAIL PROTECTED] Sent: 04 April 2008 19:57 To: Trent Nelson Cc: Christian Heimes; python-dev@python.org Subject: Re: [Python-Dev] r62129 - in python/trunk: PCbuild/debug.vsprops PCbuild/kill_python.c PCbuild/kill_python.vcproj PCbuild/pcbuild.sln PCbuild/pythoncore.vcproj PCbuild/release.vsprops Tools/buildbot/Makefile Tools/buildbot/build-amd64.bat Tools/buildbo... > What do others that do Windows development think? I don't have a > problem changing the build behaviour if the approach I've taken is > generally disliked. I think kill_python should only ever be invoked in the build slaves; it should *not* be part of the regular build. If developers find they can't build because some files are still open, they should kill the processes themselves. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/tnelson%40onresolve.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] socket.SOL_REUSEADDR: different semantics between Windows vs Unix (or why test_asynchat is sometimes dying on Windows)
I've attached a patch (http://bugs.python.org/file9966/trunk.2550.patch) to issue 2550 that addresses the original problem here: test_support.bind_port() potentially returning ports that have already been bound to. The patch updates the tests that relied on this method, such that they call it with the new calling convention (test_ftplib, test_httplib, test_socket, test_ssl_socket, test_asynchat, test_telnetlib). Any objections to the patch? Would like to commit it sooner rather than later, as it'll fix my buildbots from wedging on test_asynchat at the very least. Trent. From: [EMAIL PROTECTED] [EMAIL PROTECTED] On Behalf Of Trent Nelson [EMAIL PROTECTED] Sent: 05 April 2008 18:22 To: Jean-Paul Calderone; python-dev@python.org Subject: Re: [Python-Dev] socket.SOL_REUSEADDR: different semantics between Windows vs Unix (or why test_asynchat is sometimes dying on Windows) > >"With TCP, we are never able to start multiple servers that bind > > the same IP address and same port: a completely duplicate binding. > > That is, we cannot start one server that binds 198.69.10.2 port 80 > > and start another that also binds 198.69.10.2 port 80, even if we > > set the SO_REUSEADDR socket option for the second server." > Notice that the quoted text explains that you cannot start multiple > servers that etc. Since you didn't call listen on either socket, it's > arguable that you didn't start any servers, so there should be no > surprise regarding the behavior. Try adding listen calls at various > places in the example and you'll see something different happen. I agree in principle, Stevens says nothing about what happens if you *do* try and bind two sockets on two identical host/port addresses. Even so, test_support.bind_port() makes an assumption that bind() will raise EADDRINUSE if the port is not available, which, as has been demonstrated, won't be the case on Windows or Linux. > FWIW, AIUI, SO_REUSEADDR behaves just as described in the above quote > on Linux/BSD/UNIX/etc. On Windows, however, that option actually means > something quite different. It means that the address should be stolen > from any process which happens to be using it at the moment. Probably explains why the python process wedges when this happens on Windows... > There is another option, SO_EXCLUSIVEADDRUSE, only on Windows I think, > which, AIUI, makes it impossible for another process to steal the port > using SO_REUSEADDR. Nod, if SO_EXCLUSIVEADDRUSE is used instead in the code I posted, Windows raises EADDRINUSE on the second bind(). I don't have access to any Linux boxes at the moment, so I can't test what sort of error is raised with the example I posted if listen() and accept() are called on the two sockets bound to identical addresses. Can anyone else shed some light on this? I'd be interested in knowing if the process wedges on Linux as badly as it does on Windows (to the point where it's not respecting ctrl-c or sigkill). Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/tnelson%40onresolve.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Changing all network-oriented tests to facilitate (virtually unlimited) parallel test execution [Issue#: 2550]
I've forwarding my most recent update to issue 2550 here such that the proposed patch (and in general, the approach to network-oriented test cases) can be vetted by a wider audience: http://bugs.python.org/file9980/trunk.2550-2.patch This patch works towards fixing a large proportion of the tests that were written in such a way that often leads to buildbot errors when port conflicts arise. With this patch applied, I can run many instances of the test suite in parallel and none of the network-oriented tests fail (which they do currently if you try and do this). Plenty of discussion (mostly me replying to my own comments) on the subject at: http://bugs.python.org/issue2550. Anyone have any issues with this new approach? I'm particularily interested in whether or not people disagree with the assumption I've drawn regarding never using SO_REUSEADDR in tests for TCP/IP sockets (see below). Trent. From: [EMAIL PROTECTED] [EMAIL PROTECTED] On Behalf Of Trent Nelson [EMAIL PROTECTED] Sent: 08 April 2008 12:49 To: [EMAIL PROTECTED] Subject: [issue2550] SO_REUSEADDR doesn't have the same semantics on Windows as on Unix Trent Nelson <[EMAIL PROTECTED]> added the comment: Invested quite a few cycles on this issue last night. The more time I spent on it, the more I became convinced that every single test working with sockets should be changed in one fell swoop in order to facilitate (virtually unlimited) parallel test execution without fear of port conflicts. I've attached a second patch, trunk.2550-2.patch, which is my progress so far on doing just this. The main changes can be expressed by the following two points: a) do whatever it takes in network-oriented tests to ensure unique ports are obtained (relying on the bind_port() and find_unused_port() methods exposed by test_support) b) never, ever, ever call SO_REUSEADDR on a socket from a test; because we're putting so much effort into obtaining a unique port, this should never be necessary -- in the rare cases that our attempts to obtain a unique port fail, then we absolutely should fail with EADDRINUSE, as the ability to obtain a unique port for the duration of a client/server test is an invariant that we *must* be able to depend upon. If the invariant is broken, fail immediately (don't mask the problem with SO_REUSEADDR). With this patch applied, I can spawn a handful of Python processes and run the entire test suite (without -r, ensuring all tests are run in the same order, which should encourage port conflicts (if there were any)) without any errors*. Doing that now is completely and utterly impossible. [*] Well, almost without error. All the I/O related tests that try and open @test fail. I believe there's still outstanding work to do with this patch with regards to how the intracacies of SO_REUSEADDR and SO_EXCLUSIVEADDRUSE should be handled in the rest of the stdlib. I'm still thinking about the best approach for this. However, the patch as it currently stands is still quite substantial so I wanted to get it out sooner rather than later for review. (I'll forward this to python-dev@ to try and encourage more eyes from people with far more network-fu than I.) Added file: http://bugs.python.org/file9980/trunk.2550-2.patch __ Tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue2550> __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/tnelson%40onresolve.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] [OT] Wingware IDE key for sprinters at PyCon
Anyone happen to have the key handy that Wingware were giving out to sprinters at PyCon? For the life of me, I can't find what I did with that piece of paper. If someone could forward me it off list, that'd be great. Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [OT] Wingware IDE key for sprinters at PyCon
All sorted, thanks. > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] > On Behalf Of Trent Nelson > Sent: 08 April 2008 15:48 > To: python-dev@python.org > Subject: [Python-Dev] [OT] Wingware IDE key for sprinters at PyCon > > Anyone happen to have the key handy that Wingware were giving > out to sprinters at PyCon? For the life of me, I can't find > what I did with that piece of paper. If someone could > forward me it off list, that'd be great. > > Trent. > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/tnelson%40on resolve.com > ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Changing all network-oriented tests to facilitate (virtually unlimited) parallel test execution [Issue#: 2550]
Committed the patch in r62234. Hopefully the work paid off! (He says moments before all the buildbots turn red...) From: Gregory P. Smith [mailto:[EMAIL PROTECTED] Sent: 08 April 2008 20:58 To: Trent Nelson Cc: python-dev@python.org Subject: Re: [Python-Dev] Changing all network-oriented tests to facilitate (virtually unlimited) parallel test execution [Issue#: 2550] On Tue, Apr 8, 2008 at 5:00 AM, Trent Nelson <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote: I've forwarding my most recent update to issue 2550 here such that the proposed patch (and in general, the approach to network-oriented test cases) can be vetted by a wider audience: http://bugs.python.org/file9980/trunk.2550-2.patch This patch works towards fixing a large proportion of the tests that were written in such a way that often leads to buildbot errors when port conflicts arise. With this patch applied, I can run many instances of the test suite in parallel and none of the network-oriented tests fail (which they do currently if you try and do this). Plenty of discussion (mostly me replying to my own comments) on the subject at: http://bugs.python.org/issue2550. Anyone have any issues with this new approach? I'm particularily interested in whether or not people disagree with the assumption I've drawn regarding never using SO_REUSEADDR in tests for TCP/IP sockets (see below). Trent. I think your assumptions are fair. Note that not using SO_REUSEADDR can reserve the port for a few minutes beyond the actual unbinding destruction of the server socket on some OSes but with tens of thousands of ports available and likely not more than 1-200 being needed by a full test suite run and how long the test suite takes to run I think that is acceptable and is not a problem we're likely to ever run into (fix it only iffwe ever do). -gps From: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]> [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>] On Behalf Of Trent Nelson [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>] Sent: 08 April 2008 12:49 To: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]> Subject: [issue2550] SO_REUSEADDR doesn't have the same semantics on Windows as on Unix Trent Nelson <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> added the comment: Invested quite a few cycles on this issue last night. The more time I spent on it, the more I became convinced that every single test working with sockets should be changed in one fell swoop in order to facilitate (virtually unlimited) parallel test execution without fear of port conflicts. I've attached a second patch, trunk.2550-2.patch, which is my progress so far on doing just this. The main changes can be expressed by the following two points: a) do whatever it takes in network-oriented tests to ensure unique ports are obtained (relying on the bind_port() and find_unused_port() methods exposed by test_support) b) never, ever, ever call SO_REUSEADDR on a socket from a test; because we're putting so much effort into obtaining a unique port, this should never be necessary -- in the rare cases that our attempts to obtain a unique port fail, then we absolutely should fail with EADDRINUSE, as the ability to obtain a unique port for the duration of a client/server test is an invariant that we *must* be able to depend upon. If the invariant is broken, fail immediately (don't mask the problem with SO_REUSEADDR). With this patch applied, I can spawn a handful of Python processes and run the entire test suite (without -r, ensuring all tests are run in the same order, which should encourage port conflicts (if there were any)) without any errors*. Doing that now is completely and utterly impossible. [*] Well, almost without error. All the I/O related tests that try and open @test fail. I believe there's still outstanding work to do with this patch with regards to how the intracacies of SO_REUSEADDR and SO_EXCLUSIVEADDRUSE should be handled in the rest of the stdlib. I'm still thinking about the best approach for this. However, the patch as it currently stands is still quite substantial so I wanted to get it out sooner rather than later for review. (I'll forward this to python-dev@ to try and encourage more eyes from people with far more network-fu than I.) Added file: http://bugs.python.org/file9980/trunk.2550-2.patch __ Tracker <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> <http://bugs.python.org/issue2550> __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/tnelson%40onresolve.com ___ Python-Dev mailing list Python-Dev@python.org<mailto:Python-D
Re: [Python-Dev] fixing tests on windows
> -Original Message- > From: Tim Golden [mailto:[EMAIL PROTECTED] > Sent: 04 April 2008 09:30 > To: Trent Nelson > Cc: Python-Dev > Subject: Re: [Python-Dev] fixing tests on windows > Yes. I'm trying desperately hard to stick to a narrow remit > of getting tests to run consistently in the face of messy > file-locking semantics under Windows. I really don't want to > wade into the minor minefield of making all the tests run > with consistent temp file semantics. But I may have to. Hey Tim, any progress on this? I'd like to start working on this towards the weekend... Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Next monthly sprint/bugfix day?
Hi, Is there another online sprint/bugfix day in the pipeline? If not, can there be? ;-) Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] fixing tests on windows
> Thanks for the prod, Trent. In short, yes, I spent some time > on this over the weekend but haven't had time since. I found > myself becoming more and more worried at the amount I had to > change, especially given rumblings on the list concerning not > changing the tests if possible. > I was heartened to see that you'd gone ahead with the -- > almost as invasive -- socket test changes. If you'd like to > take on what I've got so far, or simply to adopt your own > strategy, please feel free. I don't hold any weight to rumblings regarding "not changing the tests if possible". The tests need to change. They were never written with the intent to run in parallel without any resource contention. That's an important goal, now. Please send over any patches you've got, everything will be useful! Now that the dust has (sort of) settled on the network-oriented tests, I'm ready to start attacking the I/O ones. Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Deco
Neal, Martin, We're in the process of decommissioning the box the 'x86 FreeBSD 3' build slave is running on, can you remove it from the list? Our new FreeBSD 7.0 server is up, which we're slowly migrating to, and I'll be able to set a slave up on that probably some time next week once we've moved our production stuff over. Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Global Python Sprint Weekends: May 10th-11th and June 21st-22nd.
Following on from the success of previous sprint/bugfix weekends and sprinting efforts at PyCon 2008, I'd like to propose the next two Global Python Sprint Weekends take place on the following dates: * May 10th-11th (four days after 2.6a3 and 3.0a5 are released) * June 21st-22nd (~week before 2.6b2 and 3.0b2 are released) It seems there are a few of the Python User Groups keen on meeting up in person and sprinting collaboratively, akin to PyCon, which I highly recommend. I'd like to nominate Saturday across the board as the day for PUGs to meet up in person, with Sunday geared more towards an online collaboration day via IRC, where we can take care of all the little things that got in our way of coding on Saturday (like finalising/preparing/reviewing patches, updating tracker and documentation, writing tests ;-). For User Groups that are planning on meeting up to collaborate, please reply to this thread on python-dev@python.org and let every- one know your intentions! As is commonly the case, #python-dev on irc.freenode.net will be the place to be over the course of each sprint weekend; a large proportion of Python developers with commit access will be present, increasing the amount of eyes available to review and apply patches. For those that have an idea on areas they'd like to sprint on and want to look for other developers to rope in (or just to communicate plans in advance), please also feel free to jump on this thread via python-dev@ and indicate your intentions. For those that haven't the foggiest on what to work on, but would like to contribute, the bugs tracker at http://bugs.python.org is the best place to start. Register an account and start searching for issues that you'd be able to lend a hand with. All contributors that submit code patches or documentation updates will typically get listed in Misc/ACKS.txt; come September when the final release of 2.6 and 3.0 come about, you'll be able to point at the tarball or .msi and exclaim loudly ``I helped build that!'', and actually back it up with hard evidence ;-) Bring on the pizza and Red Bull! Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Code signing of Windows .msi/.dll/.exe's for 2.6/3.0
Hi Barry, Friendly poke to see if there's been any progress on acquiring the relevant certificates such that we can code sign binaries on Windows for 2.6/3.0 ;-) Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Code signing of Windows .msi/.dll/.exe's for 2.6/3.0
> * Why the binaries should be signed? Makes the installation process on Windows Vista and Server 2008 a little nicer; instead of getting an "unknown-executable-could- be-a-virus-aa-watchout"-type dialog with a big red flag, you get a less threatening message saying that you're about to run something that's been digitally signed by the Python Software Foundation. (I've come across a few entities (NSA, government bodies, etc), who mandate that all installers/binaries they get must be digitally signed.) > * What is required to sign the binaries? 1. Obtain a code signing certificate from someone. I used VeriSign. You end up with an .spc and a .pvk file. You need to combine them into a single .pfx file via a tool called pvk2pfx.exe: Usage: pvk2pfx -pvk [-pi ] -spc [-pfx [-po ] [-f]] -pvk - input PVK file name. -spc - input SPC file name. -pfx - output PFX file name. -pi- PVK password. -po- PFX password; same as -pi if not given. -f - force overwrite existing PFX file. if -pfx option is not given, an export wizard will pop up. in this case, options -po and -f are ignored. C:\..> pvk2pfx.exe -pvk verisign-privatekey.pvk -pi ** -spc onresolve-verisign.spc -po ** -pfx onresolve-verisign.pfx 3. The resulting .pfx file, onresolve-verisign.pfx in this case, can then be installed as a 'Personal' certificate in Windows, using the Certificate Management facility (CertMgr.exe). When you install it, you provide a name that the certificate can be referred to by apps; in my case I just used 'VeriSign'. This name is used below by the signtool.exe app. 4. Sign the executable, MSI or DLL as follows: C:\..> signtool.exe sign /i "VeriSign" /d "Python 2.6.0" /du http://www.python.org /t http://timestamp.verisign.com/scripts/timstamp.dll Python-2.6.msi Successfully signed and timestamped: Python-2.6.msi > * Which binaries should be signed? Personally, once I figured out the steps above, I hooked the signing process into all my Visual Studio projects as a post-build step, such that I sign all .exe and .dll files. Not really necessary, but eh, it does have the advantage of looking more professional (users can view properties on the .dll, for example, and see that it's been digitally signed by the PSF). Additionally, it prevents any tampering; Windows can detect if it's been altered in any way since it's been signed, and will flat out prevent it from being loaded/run if that's the case. Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] BSDDB3
Hi Jesus, > Martin v. Löwis wrote: > | I think it would be helpful if you could analyze the crashes that > | bsddb caused on Windows. Just go back a few revisions in the > | subversion tree to reproduce the crashes. > > I have no MS Windows machines in my environment :-( I remember those rampant BSDDB crashes on Windows well. I brought this up with Martin at PyCon; I really don't think we can fault BSDDB here -- basically, the tests weren't cleaning up their environment in the right order, so BSDDB was getting passed completely and utterly bogus values. I *think* I managed to persuade Martin that this was indeed our fault, and we can't really hold BSDDB accountable. (My argument being that if a 3rd party app says the behaviour of a method is undefined if you pass it a null pointer, and you pass it a null pointer, and it crashes your program, it's your fault, not theirs.) Once this was addressed, the BSDDB tests ran more or less on Windows 32-bit without error. Windows x64 was another matter though -- I traced the problem down to wildly conflicting compiler and linker flags between our Python build and how we were building BSDDB (or rather how BSDDB builds out of the box on Windows). My solution was to drop our reliance on the Berkeley_DB.sln/db_static.vcproj files completely, and mimic a bsddb44 vcproj in our own pcbuild.sln, which basically meant all the BSDDB source code got built in the exact same fashion as the rest of Python. I also took this approach with sqlite3 and it's worked really well -- there have been no issues with either module since this change. I've also got bsddb45.vcproj and bsddb46.vcproj projects floating around in one of my local branches somewhere. These mimic the corresponding BSDDB projects, with the intent being that when it comes to release time for 2.6 and 3.0, we'd make a decision about which one to ship with, and then set the Python _bsddb module to use that. I should probably pick that up again... Hope this clarifies things... Regards, Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] BSDDB3
> Next week I will (if nothing goes wrong) publish pybsddb > 4.6.4. This release supports distributed transactions and > replication, testsuite is way faster, and rewritten to be > able to launch tests from multiple threads/processes if you > wish, setuptools/pypi support, etc. Great! Once you've settled in with your changes let me know and I'll help with doing the necessary things on the Windows-side to set up the bsddb46.vcproj and switching the build to use that. Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Do we still need BaseAddress in .vcproj files?
I just did this locally to all my .vcproj files for .pyds: - BaseAddress="0x1e00" + RandomizedBaseAddress="2" This was partly out of curiosity, and partly because it was quicker doing that than finding a new unique base address to use for a new module I added to my local tree. Rebuilt everything and ran a full regression test, and everything passed. What am I missing? Do we have parts of Python that rely on finding modules at explicit base addresses? Is BaseAddress a relic from the past? Christian or Martin? Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] socket.try_reuse_address()
Since the recent changes to networking-oriented tests (issue 2550, r62234 and r62237), I think it's safe to say stability of the test suite on all the non-Windows platforms has improved significantly in this area (I don't recall seeing any socket errors in *nix buildbot logs since those commits). However, Windows buildbots are still periodically failing. More specifically, my Windows buildbots are still failing. One thing that's different about my buildbots is that two are being run at the same time for both trunk and py3k -- one doing an x86 build, the other doing x64 build. Since the changes in the aforementioned revisions, the behaviour of my buildbots has definitely improved -- they no longer completely wedge on test_async(chat|core), mainly due to abolishing all use of SO_REUSEADDR as a socket option in any network-oriented tests. However, periodically, they're still dying/failing in a variety of ways -- see relevant log snippets at the end of this e-mail for examples. I attribute this to the fact that SO_REUSEADDR is still set as a socket option in asyncore.py and SocketServer.py. Basically, SO_REUSEADDR should *never* be set on Windows for TCP/IP sockets. Using asyncore.py as an example, here are two ways we could handle this: 1. Hard code the Windows opt-out: --- asyncore.py (revision 62509) +++ asyncore.py (working copy) @@ -267,6 +267,8 @@ def set_reuse_addr(self): # try to re-use a server port if possible +if os.name == 'nt' and self.socket.socket_type != socket.SOCK_DGRAM: +return try: self.socket.setsockopt( socket.SOL_SOCKET, socket.SO_REUSEADDR, 2. Introduce socket.try_reuse_address(): --- asyncore.py (revision 62509) +++ asyncore.py (working copy) @@ -266,15 +266,7 @@ self.add_channel(map) def set_reuse_addr(self): -# try to re-use a server port if possible -try: -self.socket.setsockopt( -socket.SOL_SOCKET, socket.SO_REUSEADDR, -self.socket.getsockopt(socket.SOL_SOCKET, - socket.SO_REUSEADDR) | 1 -) -except socket.error: -pass +self.socket.try_reuse_address() With try_use_address implemented as follows: --- socket.py (revision 62509) +++ socket.py (working copy) @@ -197,6 +197,10 @@ Return a new socket object connected to the same system resource.""" return _socketobject(_sock=self._sock) +def try_reuse_address(self): +if not (os.name == 'nt' and self._sock.type != SOCK_DGRAM): +self._sock.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1) + def makefile(self, mode='r', bufsize=-1): """makefile([mode[, bufsize]]) -> file object I prefer the latter as it's cleaner, easier to document and encapsulates what we're trying to do relatively well. The affected modules would be asyncore.py, SocketServer.py and idlelib/rpc.py. Thoughts? Regards, Trent. test_ftplib remoteFailed: [Failure instance: Traceback (failure with no frames): twisted.internet.error.ConnectionLost: Connection to the other side was lost in a non-clean fashion. ] test_asynchat test test_asynchat failed -- errors occurred; run in verbose mode for details [snip to bottom of log where test_asynchat is re-run] 1 test failed: test_asynchat 33 tests skipped: test__locale test_aepack test_applesingle test_cProfile test_commands test_crypt test_curses test_dbm test_epoll test_fcntl test_fork1 test_gdbm test_grp test_ioctl test_kqueue test_macostools test_mhlib test_nis test_openpty test_ossaudiodev test_pipes test_poll test_posix test_pty test_pwd test_resource test_scriptpackages test_signal test_syslog test_threadsignals test_wait3 test_wait4 test_zipfile64 Those skips are all expected on win32. Re-running failed tests in verbose mode Re-running test 'test_asynchat' in verbose mode test_close_when_done (test.test_asynchat.TestAsynchat) ... ok test_empty_line (test.test_asynchat.TestAsynchat) ... ok test_line_terminator1 (test.test_asynchat.TestAsynchat) ... ok test_line_terminator2 (test.test_asynchat.TestAsynchat) ... ok test_line_terminator3 (test.test_asynchat.TestAsynchat) ... ok test_none_terminator (test.test_asynchat.TestAsynchat) ... ok test_numeric_terminator1 (test.test_asynchat.TestAsynchat) ... ok test_numeric_terminator2 (test.test_asynchat.TestAsynchat) ... ok test_simple_producer (test.test_asynchat.TestAsynchat) ... ok test_string_producer (test.test_asynchat.TestAsynchat) ... ok test_close_when_done (test.test_asynchat.TestAsynchat_WithPoll) ... ok test_empty_line (test.test_asynchat.TestAsynchat_WithPoll) ... ok test_line_terminator1 (test.test_asynchat.TestAsynchat_WithPoll) ... ok test_line_terminator2 (test.test_asynchat.TestAsynchat_WithPoll) ... ok test_line_terminator3 (test.test_asynchat.TestAsynchat_WithPoll) ... ok test_none_terminator (test.test_async
Re: [Python-Dev] socket.try_reuse_address()
> Giampaolo Rodola' wrote:
> > Maybe it would be better considering Windows CE systems too:
> >
> > - if os.name == 'nt'
> > + if os.name in ('nt', 'ce')
> >
> Cygwin? I don't know how Unix-like it is.
Yeah, that's a fair point, it's the behaviour of the underlying Winsock API
we're targeting, so it would apply to Cygwin as well. (And CE and anything
else on Windows.)
Trent.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] socket.try_reuse_address()
> > Giampaolo Rodola' wrote:
> > > Maybe it would be better considering Windows CE systems too:
> > >
> > > - if os.name == 'nt'
> > > + if os.name in ('nt', 'ce')
> > >
> > Cygwin? I don't know how Unix-like it is.
>
> Yeah, that's a fair point, it's the behaviour of the
> underlying Winsock API we're targeting, so it would apply to
> Cygwin as well. (And CE and anything else on Windows.)
including Jython and IronPython -- which all exhibit the same undesirable
behaviour on Windows when SO_REUSEADDR is set against a TCP/IP socket. Updated
patch below. Assuming there are no objections, I'd like to clean this up and
commit over the weekend, once I've updated the various parts of the stdlib
currently using SO_REUSEADDR, as well as affected documentation.
Index: socket.py
===
--- socket.py (revision 62509)
+++ socket.py (working copy)
@@ -143,8 +143,18 @@
'sendall', 'setblocking',
'settimeout', 'gettimeout', 'shutdown')
+# Attempt to determine if we're running on Windows, irrespective of our Python
+# incarnation. We need to know this so that we *don't* set the SO_REUSEADDR
+# against TCP/IP sockets in socket.try_reuse_addr(). Note that IronPython is
+# covered below as it sets os.name to 'nt'.
if os.name == "nt":
_socketmethods = _socketmethods + ('ioctl',)
+_is_windows = True
+elif os.name == 'java':
+from java.lang import System
+_is_windows = 'windows' in System.getProperty('os.name').lower()
+elif os.name == 'posix' and sys.platform == 'cygwin':
+_is_windows = True
if sys.platform == "riscos":
_socketmethods = _socketmethods + ('sleeptaskw',)
@@ -197,6 +207,13 @@
Return a new socket object connected to the same system resource."""
return _socketobject(_sock=self._sock)
+def try_reuse_address(self):
+if not (_is_windows and self._sock.type != SOCK_DGRAM):
+try:
+self._sock.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1)
+except socket.error:
+pass
+
def makefile(self, mode='r', bufsize=-1):
"""makefile([mode[, bufsize]]) -> file object
Trent.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] socket.try_reuse_address()
> if os.name == "nt":
> _socketmethods = _socketmethods + ('ioctl',)
> +_is_windows = True
> +elif os.name == 'java':
> +from java.lang import System
> +_is_windows = 'windows' in System.getProperty('os.name').lower()
> +elif os.name == 'posix' and sys.platform == 'cygwin':
> +_is_windows = True
Oops, that last line should have been:
elif os.name == 'ce' or (os.name == 'posix' and sys.platform == 'cygwin'):
_is_windows = True
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] socket.try_reuse_address()
> This one will not work.
>
> >>> 'windows' in System.getProperty('os.name').lower()
> Traceback (innermost last):
> File "", line 1, in ?
> TypeError: string member test needs char left operand
> >>>
Interesting, which version of Jython were you using?
> You may have to do something like
> System.getProperty('os.name').lower().find('windows').
That didn't work for me. I assume the following works for you:
System.getProperty('os.name').lower().startswith('windows')
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Switching to Visual Studio 2010
On Thu, Jan 26, 2012 at 12:54:31PM -0800, mar...@v.loewis.de wrote: > > Is this considered a new feature that has to be in by the first beta? > > I'm hoping to have it completed much sooner than that so we can get > > mileage on it, but is there a cutoff for changing the compiler? > > At some point, I'll start doing this myself if it hasn't been done by > then, and I would certainly want the build process adjusted (with > all buildbots updated) before beta 1. I... I think I might have already done this, inadvertently. I needed an x64 VS2010 debug build of Subversion/APR*/Python a few weeks ago -- forgetting the fact that we're still on VS2008. By the time I got to building Python, I'd already coerced everything else to use VS2010, so I just bit the bullet and coerced Python to use it too, including updating all the buildbot scripts and relevant externals to use VS2010, too. Things that immediately come to mind as potentially being useful: * Three new buildbot scripts: - build-amd64-vs10.bat - clean-amd64-vs10.bat - external-amd64-vs10.bat * Updates to externals/(tcl|tk)-8.5.9.x so that they both build with VS2010. This was a tad fiddly. I ended up creating makefile.vs10 from win/makefile.vc and encapsulating the changes there, then calling that from the buildbot *-vs10.bat scripts. I had to change win/rules.vc, too. * A few other things I can't remember off the top of my head. So, I guess my question is, is that work useful? Based on Martin's original list, it seems to check a few boxes. Brian, what are your plans? Are you going to continue working in hg.python.org/sandbox/vs2010port then merge everything over when ready? I have some time available to work on this for the next three weeks or so and would like to help out. Regards, Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Switching to Visual Studio 2010
On Sun, Jan 29, 2012 at 12:23:14PM -0800, Trent Nelson wrote: > * Updates to externals/(tcl|tk)-8.5.9.x so that they both build with > VS2010. Before I go updating tcl/tk, any thoughts on bumping our support to the latest revision, 8.5.11? I guess the same question applies to all the externals, actually (zlib, openssl, sqlite, bsddb, etc). In the past we've typically bumped up our support to the latest version prior to beta, then stuck with that for the release's life, right? Semi-related note: is svn.python.org/externals still the primary repo for externals? (I can't see a similarly named hg repo.) Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] IBM P-690 server looking for a home
On 19-Aug-10 10:48 AM, Randall Walls wrote: Greetings, The company I work for has an IBM P-690 server that is in the process of being retired. It is still a viable server, and has seen almost 0 use (it was our failover machine). Unfortunately for us, this machine has little to no resale value, and will probably be junked. I'd rather it go to a good home, and having taken advantage of the work of the python development community for a number of years (we use python extensively in system admin and database work), I saw this as an opportunity to give back a little. So, If anyone is interested in this machine, please let me know. We are looking at perhaps a November time frame for when it will be removed from our remote site. The P690 is no small machine, it is the size of a full rack and has 32 Power4 processors in it and takes (I believe) 2 or 3 phase 220 Volt power. It weighs nearly a ton. We are running AIX5.3 on it, but I believe that the machine is capable of running a PowerPC flavor of Linux as well. This would make a great test machine for python HPC modules or as a community box where developers could test their code against a PowerPC architecture. It has lots of life left and I'd rather see it put to use then thrown away. Snakebite[1]'s always got an eye out for free hardware, but dang, that's one chunky piece of kit. I'll follow up in private. (And yeah, I'm still working on Snakebite, for those that are interested. Turns out hosting three racks of heavy-duty hardware in the corner room of a (graciously donated) science lab takes a bit longer than originally anticipated. Who would have thought.) Regards, Trent "no-news-is-good-news" Nelson. [1]: http://www.snakebite.org/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Snakebite, buildbot and low hanging fruit -- feedback wanted! (Was Re: SSH access against buildbot boxes)
On 07-Nov-10 1:55 AM, Nick Coghlan wrote: On Sun, Nov 7, 2010 at 3:53 AM, Giampaolo Rodolà wrote: In such cases I would find more easy to be able to connect to the machine and test myself rather than create a separate branch, commit, schedule a buildbot run, wait for it to complete and see whether everything is "green". On the other side I perfectly understand how opening up blanket ssh access is not something everyone is comfortable with doing. AFAICR there was someone who was setting up an evironment to solve exactly this problem but I'm not sure whether this is already usable. Dealing with exactly this problem is one of the goals of the Snakebite project. As far as I know, the folks behind that project are still working on it - I've cc'ed Trent Nelson to see if he can provide any additional info on the topic. Thanks for the ping Nick, I might have missed this otherwise. Good timing, too, as Titus and I were just discussing which low hanging fruit/pain points Snakebite should tackle first (now that all the server room stuff has finally been taken care of). Luckily, the problems that we faced 2.5 years ago when I came up with the idea of Snakebite are still just as ever present today ;-) 1. Not having access to buildbots is a pain when something doesn't work right. Doing dummy debug commits against trunk to try and coerce some more information out of a failing platform is painful. Losing a build slave entirely due to a particularly hard crash and requiring the assistance of the owner is also super frustrating. 2. The buildbot web interface for building non-(trunk|2.x|py3k) branches is also crazy unfriendly. Per-activity branches are a great way to isolate development, even with Subversion, but it kinda' blows that you don't *really* get any feedback about how your code behaves on other platforms until you re-integrate your changes back into a mainline branch. (I'm sure none of us have been masochistic enough to manually kick off individual builds for every platform via the buildbot web page after every commit to a non-standard branch.) So, enter Snakebite. We've got three racks filled with way more hardware than I should have ever purchased. Ignoring the overhead of having to set machines up and whatnot, let's just assume that over the next couple of months, if there's a platform we need a stable buildbot for, Snakebite can provide it. (And if we feel like bringing IRIX/MIPS and Tru64/Alphas back as primary platforms, we've got the hardware to do that, too ;-).) Now, the fact that they're all in the one place and under my complete control is a big advantage, as I can start addressing some of the pain points that lead me down this twisted path 2.5 years ago. I'd like to get some feedback from the development community on what they'd prefer. In my mind, I could take one of the following two steps: 1. Set up standard build slaves on all the platforms, but put something in place that allowed committers to ssh/mstsc in to said slaves when things go wrong in order to aid with debugging and/or maintaining general buildbot health (OK'ing modal crash dialogues on Windows, for example). 2. Address the second problem of the buildbot web interface sucking for non-standard branches. I'm thinking along the lines of a hack to buildbot, such that upon creation of new per-activity branches off a mainline, something magically runs in the background and sets up a complete buildbot view at python.snakebite.org/dev/buildbot/, just as if you were looking at a trunk buildbot page. I'm not sure how easy the second point will be when we switch to hg; and I'll admit if there have been any python-dev discussions about buildbot once we're on hg, I've missed them. Of course there's a third option, which is using the infrastructure I've mentioned to address a similarly annoying pain point I haven't thought of -- so feel free to mention anything else you'd like to see first instead of the above two things. Titus, for example, alluded to some nifty way for a committer to push his local hg branch/changes somewhere, such that it would kick off builds on multiple platforms in the same sorta' vein as point 2, but able to leverage cloud resources like Amazon's EC2, not just Snakebite hardware. Look forward to hearing some feedback! Regards, Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Stable buildbots
On 14-Nov-10 3:48 AM, David Bolen wrote: This is a completely separate issue, though probably around just as long, and like the popup problem its frequency changes over time. By "hung" here I'm referring to cases where something must go wrong with a test and/or its cleanup such that a python_d process remains running, usually several of them at the same time. My guess: the "hung" (single-threaded) Python process has called select() without a timeout in order to wait for some data. However, the data never arrives (due to a broken/failed test), and the select() never returns. On Windows, processes seem harder to kill when they get into this state. If I purposely wedge a Windows process via select() via the interactive interpreter, ctrl-c has absolutely no effect (whereas on Unix, ctrl-c will interrupt the select()). As for why kill_python.exe doesn't seem to be able to kill said wedged processes, the MSDN documentation on TerminateProcess[1] states the following: The terminated process cannot exit until all pending I/O has been completed or canceled. (sic) It's not unreasonable to assume a wedged select() constitutes pending I/O, so that's a possible explanation as to why kill_python.exe isn't able to terminate the processes. (Also, kill_python currently assumes TerminateProcess() always works; perhaps this optimism is misplaced. Also note the XXX TODO regarding the fact that we don't kill processes that have loaded our python*.dll, but may not be named python_d.exe. I don't think that's the issue here, though.) On 14-Nov-10 5:32 AM, David Bolen wrote: > "Martin v. Löwis" writes: > >> This is what kill_python.exe is supposed to solve. So I recommend to >> investigate why it fails to kill the hanging Pythons. > > Yeah, I know, and I can't say I disagree in principle - not sure why > Windows doesn't let the kill in that module work (or if there's an > issue actually running it under all conditions). > > At the moment though, I do know that using the sysinternals pskill > utility externally (which is what I currently do interactively) > definitely works so to be honest, That's interesting. (That kill_python.exe doesn't kill the wedged processes, but pskill does.) kill_python is pretty simple, it just calls TerminateProcess() after acquiring a handle with the relevant PROCESS_TERMINATE access right. That being said, that's the recommended way to kill a process -- I doubt pskill would be going about it any differently (although, it is sysinternals... you never know what kind of crazy black magic it's doing behind the scenes). Are you calling pskill with the -t flag? i.e. kill process and all dependents? That might be the ticket, especially if killing the child process that wedged select() is waiting on causes it to return, and thus, makes it killable. Otherwise, if it happens again, can you try kill_python.exe first, then pskill, and confirm if the former fails but the latter succeeds? Trent. [1]: http://msdn.microsoft.com/en-us/library/ms686714(VS.85).aspx ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] REMINDER: Python Sprint Weekend This Weekend!
Just a friendly reminder that this weekend is the Python sprint weekend! Look forward to seeing everyone on #python-dev irc.freenode.net over the course of the weekend! Trent. On 16 Apr, 18:52, Trent Nelson wrote: > >Following on from the success of previous sprint/bugfix weekends and >sprinting efforts at PyCon 2008, I'd like to propose the next two >Global Python Sprint Weekends, taking place on the following dates: > >* May 10th-11th (four days after 2.6a3 and 3.0a5 are released) >* June 21st-22nd (~week before 2.6b2 and 3.0b2 are released) > >It seems there are a few of the Python User Groups keen on meeting >up in person and sprinting collaboratively, akin to PyCon, which I >highly recommend. I'd like to nominate Saturday across the board >as the day for PUGs to meet up in person, with Sunday geared more >towards an online collaboration day via IRC, where we can take care >of all the little things that got in our way of coding on Saturday >(like finalising/preparing/reviewing patches, updating tracker and >documentation, writing tests ;-). > >For User Groups that are planning on meeting up to collaborate, >please reply to this thread on python-dev@python.org and let every- >one know your intentions! > >As is commonly the case, #python-dev on irc.freenode.net will be >the place to be over the course of each sprint weekend; a large >proportion of Python developers with commit access will be present, >increasing the amount of eyes available to review and apply patches. > >For those that have an idea on areas they'd like to sprint on and >want to look for other developers to rope in (or just to communicate >plans in advance), please also feel free to jump on this thread via >python-dev@ and indicate your intentions. > >For those that haven't the foggiest on what to work on, but would >like to contribute, the bugs tracker at http://bugs.python.org is >the best place to start. Register an account and start searching >for issues that you'd be able to lend a hand with. > >All contributors that submit code patches or documentation updates >will typically get listed in Misc/ACKS.txt; come September when the >final release of 2.6 and 3.0 come about, you'll be able to point at >the tarball or .msi and exclaim loudly ``I helped build that!'', >and actually back it up with hard evidence ;-) > >Bring on the pizza and Red Bull! > >Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 370 extras
> Christian, you get this week's Awesome Award. Hah, 'Weekly Awesome Award', we should make that one of our things, like QOTW. Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Importing bsddb 4.6.21; with or without AES encryption?
I was about to commit an initial import of the bsddb 4.6.21 source to the 'external' area in the repo, which I obtained from the following URL: http://www.oracle.com/technology/software/products/berkeley-db/index.html I downloaded the source that includes AES encryption, for no reason other than it was first on the list. I'm now wondering if we should only be importing the 'NC' source that doesn't contain any encryption? Jesus, does pybsddb use any of the Berkeley DB encryption facilities? Would anything break if we built the bsddb module without encryption? (Note that all of this only applies to Windows.) Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importing bsddb 4.6.21; with or without AES encryption?
Gah. I just went and visited the Berkeley DB download site as I was preparing my commit message so I could refer to the exact .tar.gz being imported, only to notice that the latest version is now 4.7.25. Jesus, can we use this version? Trent. From: [EMAIL PROTECTED] [EMAIL PROTECTED] On Behalf Of Jesus Cea [EMAIL PROTECTED] Sent: 19 May 2008 23:46 To: Python DEV Subject: Re: [Python-Dev] Importing bsddb 4.6.21; with or without AES encryption? -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Trent Nelson wrote: | I downloaded the source that includes AES encryption, for no reason | other than it was first on the list. I'm now wondering if we should | only be importing the 'NC' source that doesn't contain any | encryption? Jesus, does pybsddb use any of the Berkeley DB | encryption facilities? Would anything break if we built the | bsddb module without encryption? Yes, pybsddb3 4.6.4 supports cryptography if the underlying Berkeley DB library is crypto enabled. In principle, you can compile BDB without crypto, and pybsddb3 should work, but you would lose ability to open any DB formerly created using page encryption or page checksum. Export laws aside, we better compile with crypto :). Details: http://www.jcea.es/programacion/pybsddb_doc/dbenv.html#set_encrypt <http://www.oracle.com/technology/documentation/berkeley-db/xml/api_c/db_set_flags.html#DB_CHKSUM> - -- Jesus Cea Avion _/_/ _/_/_/_/_/_/ [EMAIL PROTECTED] - http://www.jcea.es/ _/_/_/_/ _/_/_/_/ _/_/ jabber / xmpp:[EMAIL PROTECTED] _/_/_/_/ _/_/_/_/_/ ~ _/_/ _/_/_/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/_/_/ _/_/_/_/ _/_/ "My name is Dump, Core Dump" _/_/_/_/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.8 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iQCVAwUBSDIDWplgi5GaxT1NAQKaagP+Myn6t364B9cVMUtlKjTIX1LBZkkEG7SX pv+hjUZ68r0Suw8SkhXnBbL2Ek3/yFhvNHH8qAZvEtRGsFGmKiNXccn9Ce3oy0me rAYhuuICleAUJ8RO8FZz+sGK3SR4kITrqnGfMnG46u0rXDQy82NSRW2+uWz/D96d U+X/wfV2HgQ= =io+q -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/tnelson%40onresolve.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importing bsddb 4.6.21; with or without AES encryption?
> Trent Nelson wrote: > | Gah. I just went and visited the Berkeley DB download site as > | I was preparing my commit message so I could refer to the > | exact .tar.gz being imported, only to notice that the latest > | version is now 4.7.25. Jesus, can we use this version? > > Err No. > > It is not clear to me that python 2.6/3.0 will be published with BDB 4.6 > or 4.7 support. 4.6 has several known issues, apparently solved in 4.7. I could have sworn I heard a few people mention that "4.5 has issues, but they're solved in 4.6" at PyCon ;-) > I keep a private branch in pybsddb for BDB 4.7 support, waiting for > Oracle publication. Since they already pushed 4.7.25 out (no pre warning > for bindings developers, too bad!), I think I can publish pybsddb 4.7.0 > in a couple of days. That done, I will update python version. > > PS: pybsddb 4.7.0 will support Berkeley DB 4.0 to 4.7. So, the buildbots > don't need to be upgraded.. unless we decide that Python 4.6/3.0 will > have Berkeley DB 4.7. Seems like the amount of work you have to do has doubled now that you've been added as an svn committer, given that you're maintaining multiple branches of code, one for pybsddb, and one for an official Python branch. I was under the impression that pybsddb would be assimilated into trunk/ Lib/bsddb and become the sole pybsddb incarnation. That is, you'd ditch the separate SourceForge pybsddb project and just work directly in the Python tree. I think I remember reading some concerns you had about doing this last week though, surrounding things like wanting to be able to release pybsddb versions more frequently than Python versions are released. Just because you live in trunk/Lib/bsddb shouldn't prevent you from doing this though; in fact, as long as you're sensitive to major release timeframes, I'm sure we'd love trunk to always track the latest Berkeley DB version; if all the buildbots stay green with 4.7 and there are no regressions in functionality, then hey, lets go with 4.7 for 2.6/3.0. It's probably safe to say that you're the one doing the most work with the code base and Berkeley DB in general, which means you'll be in a much better position to advise us as to which version we should be including or ignoring for a given Python release. In general, if we can support the latest release, we will. If Oracle come out with Berkeley DB 4.8 a week after 2.6 is released, that's fine, we'll keep release26-maint chugging along with 4.7, but we can switch trunk over to 4.8 once you're ready. By the time it's ready to cut 2.7, who knows, trunk's bsddb may be supporting Berkeley DB 5.2 at that stage. I would also think that you could cut independent releases (in the sense that they're not linked to any Python release schedule) of 'pybsddb' from the code living in trunk/Lib/bsddb. That way, users of 2.6 could still easy_install (or whatever) the latest pybsddb 4.8.0 to pick up the newest features. Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importing bsddb 4.6.21; with or without AES encryption?
> > In any case, what would be the procedure to update the buildbot > > infraestructure?. > You need to import the sources into the Python subversion repository; > I understand Trent was about to do that. Indeed. Where do we stand with regards to svn commit e-mails being sent for 'external'? DB 4.6 is ~16MB... Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importing bsddb 4.6.21; with or without AES encryption?
> I just committed bsddb3 4.7.0. Please install Berkeley DB >=4.7.25. I'll be working on this over the weekend. Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importing bsddb 4.6.21; with or without AES encryption?
> I just committed bsddb3 4.7.0. Please install Berkeley DB >=4.7.25.
Oh my. Just finished plugging 4.7.25 into the Windows build environment. Upon
running regrtest.py -v -u bsddb test_bsddb test_bsddb3, I get about 300 lines
like this printed to stderr:
DB->key_range: method not permitted before handle's open method
illegal flag specified to DB->remove
Then about 15 lines like this:
DB_ENV->rep_start interface requires an environment configured for the
replication subsystem
DB_ENV->rep_start interface requires an environment configured for the
replication subsystem
DB_ENV->rep_start interface requires an environment configured for the
replication subsystem
DB_ENV->rep_start interface requires an environment configured for the
replication subsystem
Then Python crashes. Quick perusal of stdout reveals this happening on just
about every test:
==
ERROR: test__no_deadlock_first (test.test_bsddb.TestBTree)
--
Traceback (most recent call last):
File "O:[EMAIL PROTECTED]", line 16, in setUp
self.f = self.openmethod[0](self.fname, self.openflag, cachesize=32768)
File "O:[EMAIL PROTECTED]", line 320, in btopen
e = _openDBEnv(cachesize)
File "O:[EMAIL PROTECTED]", line 360, in _openDBEnv
e.open('.', db.DB_PRIVATE | db.DB_CREATE | db.DB_THREAD | db.DB_INIT_LOCK |
db.DB_INIT_MPOOL)
DBInvalidArgError: (22, 'Invalid argument')
The last lines before python_d.exe crashes are as follows:
testCheckElapsedTime (test.test_bsddb3.TimingCheck) ... ok
test_pget (bsddb.test.test_cursor_pget_bug.pget_bugTestCase) ... ERROR
testCheckElapsedTime (test.test_bsddb3.TimingCheck) ... ok
test01_both (bsddb.test.test_dbobj.dbobjTestCase) ... ERROR
test02_dbobj_dict_interface (bsddb.test.test_dbobj.dbobjTestCase) ... ERROR
test03_dbobj_type_before_open (bsddb.test.test_dbobj.dbobjTestCase) ... FAIL
Think I'll call it a night for now.
Trent.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] BSDDB3
Thanks Jesus. I'll import BSDDB 4.6.4 into svn.python.org/projects/external today. Once that's done, I'll create a new bsddb46.vcproj and update the pcbuild.sln to use this project instead of the bsddb44 one currently being used. (Hopefully I'll be able to get that done today as well.) I assume you're just working on trunk at the moment? i.e. should I block or merge the changes to py3k? For everyone else: just a heads up that there will probably be a little bit of buildbot instability whilst we transition BSDDB versions on Windows. Trent. -Original Message- From: Jesus Cea [mailto:[EMAIL PROTECTED] Sent: Wednesday, May 14, 2008 12:50 AM To: Trent Nelson Cc: python-dev@python.org Subject: Re: [Python-Dev] BSDDB3 -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Trent Nelson wrote: |> Next week I will (if nothing goes wrong) publish pybsddb |> 4.6.4. This release supports distributed transactions and |> replication, testsuite is way faster, and rewritten to be |> able to launch tests from multiple threads/processes if you |> wish, setuptools/pypi support, etc. | | Great! Once you've settled in with your changes let me know and I'll help with doing the necessary things on the Windows-side to set up the bsddb46.vcproj and switching the build to use that. Python SVN updated. Let me know if you need anything from me. - -- Jesus Cea Avion _/_/ _/_/_/_/_/_/ [EMAIL PROTECTED] - http://www.jcea.es/ _/_/_/_/ _/_/_/_/ _/_/ jabber / xmpp:[EMAIL PROTECTED] _/_/_/_/ _/_/_/_/_/ ~ _/_/ _/_/_/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/_/_/ _/_/_/_/ _/_/ "My name is Dump, Core Dump" _/_/_/_/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.8 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iQCVAwUBSCopHZlgi5GaxT1NAQKrCQQAj/HTk5oqSbF2PYkZpCw2T7Dd6yEgddlY L+qW1Cde/b3duClEfawy7kf5DkSjlGLVZ9XSS+njAMKszzYK6ZIg9N4GEu5A9TO2 Rg2PiytaPbs88Jo5IIlDjvaVFPPqsOasn7WH1wcawtUKNei8whMReOveZgYXfFFf QphSJ7zspNU= =L6jr -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 370 extras
> Christian, you get this week's Awesome Award. Hah, 'Weekly Awesome Award', we should make that one of our things, like QOTW. Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Committing bsddb 4.6.4, and where can I put testsuite temp files?
Hi Jesus,
Regarding where to place test files, you might want to look at how the current
bsddb test suite in Lib/bsddb/test handles the need to create temporary files
and such. A bit of work has gone into this particular aspect to improve
reliability and robustness on buildbots -- particularily when it comes to
Windows. (I believe the current test suite comes up with a unique directory
based on process ID..)
Trent.
-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jesus Cea
Sent: Tuesday, May 13, 2008 8:10 PM
To: Python DEV
Subject: [Python-Dev] Committing bsddb 4.6.4, and where can I put testsuite
temp files?
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
I just committed pybsddb 4.6.4 to python svn. My next step (after a
successfull buildbot cycle, I hope!) is to commit the new testsuite.
First I need to review any changes there since I maintain pybsddb.
The testsuite creates a lot of files/directories while working. Fine,
since it cleans later, unless some test crashes hard. My testcode
defines a variable to the path I want to use to keep those files/dirs.
Currently it is "/tmp/z-BerkeleyDB", or something like that.
Since my code allows for testcases to be run in multiple threads and/or
processes, where would be a sensible path choice for temporal files?.
Current working directory would be fine, if "somebody" is cleaning after
running all tests.
Ramdisk ("/tmp" in some systems, for example Solaris) would be nice,
since some test uses transactions, and transactions are write-synchronous.
In my system, runnning all tests serially (you can run all of them in
parallel, if you wish) takes about 15-20 seconds. Far faster than
current tests in python svn, and that can be improved even more.
- --
Jesus Cea Avion _/_/ _/_/_/_/_/_/
[EMAIL PROTECTED] - http://www.jcea.es/ _/_/_/_/ _/_/_/_/ _/_/
jabber / xmpp:[EMAIL PROTECTED] _/_/_/_/ _/_/_/_/_/
~ _/_/ _/_/_/_/ _/_/ _/_/
"Things are not so easy" _/_/ _/_/_/_/ _/_/_/_/ _/_/
"My name is Dump, Core Dump" _/_/_/_/_/_/ _/_/ _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.8 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iQCVAwUBSCnngZlgi5GaxT1NAQLhPwP9GMMyNMgz6mfmfHXVOV8bDJGOQRok95uL
14A+K9zXW3/wlp1rhvoPmCHYqvRoCLVkPZ/7pLEQlo1ZksOlHy6BH3MDeDJEjVax
69XlzUUeuqplGbTiMdX8qd0dPi2Jp+Akg7U6ZmBdADhF7D21umU474OalKO2eIoL
ba/wnlMvens=
=1lai
-END PGP SIGNATURE-
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/tnelson%40onresolve.com
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Importing bsddb 4.6.21; with or without AES encryption?
I was about to commit an initial import of the bsddb 4.6.21 source to the 'external' area in the repo, which I obtained from the following URL: http://www.oracle.com/technology/software/products/berkeley-db/index.html I downloaded the source that includes AES encryption, for no reason other than it was first on the list. I'm now wondering if we should only be importing the 'NC' source that doesn't contain any encryption? Jesus, does pybsddb use any of the Berkeley DB encryption facilities? Would anything break if we built the bsddb module without encryption? (Note that all of this only applies to Windows.) Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Obtaining short file path
If you want a short path name, you should use win32api.GetShortPathName().
Attempting to compute it yourself isn’t as straight forward as you think.
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Hartwell Bryan
Sent: 27 May 2008 08:00
To: python-dev@python.org
Subject: [Python-Dev] Obtaining short file path
Hi,
Purpose: obtaining the system (“short”) path from a full path
Background: File dialogs (visual studio) return a full path (e.g. f=“C:\this
path has spaces\thisfilenameislongerthan8char.txt”). If this value is provided
to Python, it will not recongize this as a file. In fact os.path.isfile(f)
doesn’t return false, it crashes. Likewise, when calling executables (from
Python) with files as arguments a short path is required. VB FileSystemObject
has the ShortPath method, while os.path and path (www.jorendorff.com) modules
do not (at least as far as my googling could determine). Why bother creating a
COM interface when you’re just going to pass as shell run-time arguments all
the values the server is better at computing?
System: Python 2.3; Windows XP
Sample Code:
import win32com.client
import time
import os,sys
import os.path
#-
def shortpath(x):
z=''
for y in x.split('\\'):
if len(y.split('.')[0])>8:
if ('.' in y):
z=z+'\\'+y.split('.')[0][:6].upper()+'~1'+'.'+y.split('.')[1]
else:
z=z+'\\'+y[:6].upper()+'~1'
else:
z=z+'\\'+y
return z[1:]
#-
xlApp = win32com.client.Dispatch("Excel.Application")
xlBook = xlApp.ActiveWorkbook
savFile = str(sys.argv[1])
rawFile = str(xlBook.Sheets("Timestamp").TextBox2)
#print os.path.isfile(savFile)
r=shortpath(rawFile)
print r
try:
print os.path.isfile(r)
except:
print 'something rude'
time.sleep(7)
Notes: This code does not account for peer paths or files that share the first
8 characters (and file extension). I’m also aware that this is not the normal
means for submitting a “patch”, but in my job function I don’t see myself
regularly participating in python development (and I’m probably not savvy
enough) so the effort wasn’t worth it. However I still thought others might
benefit from what seems to be (to me) a fundamental path function. Do with it,
or ignore it, as you please.
Cheers,
Bryan Hartwell
This message is intended only for the use of the intended recipients, and it
may be privileged and confidential. If you are not the intended recipient, you
are hereby notified that any review, retransmission, conversion to hard copy,
copying, circulation or other use of this message is strictly prohibited. If
you are not the intended recipient, please notify me immediately by return
e-mail, and delete this message from your system.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Progress on switching Windows build to Berkeley DB 4.7.25...
Hi all, Jesus, apologies that this has taken so long for me to get back to, I've been completely and utterly swamped with client work the past few weeks. However, thanks to a couple of hours spare at Detroit airport yesterday, I was finally able to make some progress on updating the Windows Berkeley DB build to 4.7.25. I've checked in the work I've done so far to branches/tnelson-trunk-bsddb-47-upgrade. One thing I wanted to double check with you is the following change: Modified: python/branches/tnelson-trunk-bsddb-47-upgrade/Lib/bsddb/test/test_replication.py == --- python/branches/tnelson-trunk-bsddb-47-upgrade/Lib/bsddb/test/test_replication.py (original) +++ python/branches/tnelson-trunk-bsddb-47-upgrade/Lib/bsddb/test/test_replication.py Wed Jun 18 06:13:44 2008 @@ -94,7 +94,7 @@ # The timeout is necessary in BDB 4.5, since DB_EVENT_REP_STARTUPDONE # is not generated if the master has no new transactions. # This is solved in BDB 4.6 (#15542). -timeout = time.time()+2 +timeout = time.time()+10 while (time.time()= (4,6) : Basically, when using +2, the test failed every so often when running the entire test_bsddb3 suite. I picked 10 arbitrarily; it improves things, but it's still not 100%, I still encounter the following failure every so often: == ERROR: test01_basic_replication (bsddb.test.test_replication.DBReplicationManager) -- Traceback (most recent call last): File "s:[EMAIL PROTECTED]", line 101, in setUp self.assertTrue(time.time()http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] test_multiprocessing: test_listener_client flakiness
I gave my Windows buildbots a little bit of TLC last night. This little
chestnut in test_multiprocessing.py around line 1346 is causing my buildbots to
wedge more often than not:
def test_listener_client(self):
for family in self.connection.families:
l = self.connection.Listener(family=family)
p = self.Process(target=self._test, args=(l.address,))
p.set_daemon(True)
p.start()
conn = l.accept()
self.assertEqual(conn.recv(), 'hello')
p.join()
l.close()
The wedging will be a result of that accept() call. Not knowing anything about
the module or the test suite, I can only assume that there's a race condition
introduced between when the subprocess attempts to connect to the listener,
versus when the l.accept() call is actually entered. (On the basis that a race
condition would explain why sometimes it wedges and sometimes it doesn't.)
Just FYI, the error in the buildbot log
(http://www.python.org/dev/buildbot/all/x86%20W2k8%20trunk/builds/810/step-test/0)
when this occurs is as follows:
test_multiprocessing
command timed out: 1200 seconds without output
SIGKILL failed to kill process
using fake rc=-1
program finished with exit code -1
remoteFailed: [Failure instance: Traceback from remote host -- Traceback (most
recent call last):
Failure: buildbot.slave.commands.TimeoutError: SIGKILL failed to kill process
]
(The fact it can't be killed cleanly is a bug in Twisted's
signalProcess('KILL') method, which doesn't work against Python processes that
have entered accept() calls on Windows (which present the 'wedged' behaviour
and have to be forcibly killed with OpenProcess/TerminateProcess).)
Trent.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
