[sage-devel] Re: potential cause for #557: cython missing dictionary deallocation

2007-10-28 Thread mabshoff 



On Oct 27, 1:26 am, Robert Bradshaw [EMAIL PROTECTED]
wrote:
 On Oct 25, 2007, at 9:35 PM, mabshoff wrote:

  Carl Witty was also wondering whether your newCython.spkg was ready
  for 2.8.10 or if we should wait.

  I believe so, it compiles all of SAGE and passes all doctests. Stefan
  (the other mainCythonguy) is having trouble getting his code to
  work though.

  ok.


Hi Robert,

 BTW, I just uploaded a new spkg with one more tiny fix (has to do
 with error reporting).


For 2.8.10.alpha1+Craig's fix and python compiled using --without-
pymalloc I get with the default cleanup level 1:

==6569== LEAK SUMMARY:
==6569==definitely lost: 181,030 bytes in 2,926 blocks.
==6569==  possibly lost: 266,925 bytes in 747 blocks.
==6569==still reachable: 29,184,388 bytes in 179,707 blocks.
==6569== suppressed: 0 bytes in 0 blocks.

With cleanup level 3 in general and cleanup level 1 just for
integer.pyx I get

==23818== LEAK SUMMARY:
==23818==definitely lost: 181,117 bytes in 2,927 blocks.
==23818==  possibly lost: 266,838 bytes in 746 blocks.
==23818==still reachable: 29,157,453 bytes in 179,383 blocks.
==23818== suppressed: 0 bytes in 0 blocks.

So my conclusion is that you are on the right way with the cleanup
code. Hopefully once we can compile integer.pyx with cleanup level 3
and not crash the amount of still reachable memory should decrease
tremendously because integer.pyx should just about be referenced in
every extension we have.

 - Robert

Cheers,

Michael


--~--~-~--~~~---~--~~
To post to this group, send email to sage-devel@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at http://groups.google.com/group/sage-devel
URLs: http://sage.scipy.org/sage/ and http://modular.math.washington.edu/sage/
-~--~~~~--~~--~--~---

[sage-devel] Re: potential cause for #557: cython missing dictionary deallocation

2007-10-28 Thread Robert Bradshaw 
On Oct 28, 2007, at 1:29 AM, mabshoff wrote:

 For 2.8.10.alpha1+Craig's fix and python compiled using --without-
 pymalloc I get with the default cleanup level 1:

 ==6569== LEAK SUMMARY:
 ==6569==definitely lost: 181,030 bytes in 2,926 blocks.
 ==6569==  possibly lost: 266,925 bytes in 747 blocks.
 ==6569==still reachable: 29,184,388 bytes in 179,707 blocks.
 ==6569== suppressed: 0 bytes in 0 blocks.

 With cleanup level 3 in general and cleanup level 1 just for
 integer.pyx I get

 ==23818== LEAK SUMMARY:
 ==23818==definitely lost: 181,117 bytes in 2,927 blocks.
 ==23818==  possibly lost: 266,838 bytes in 746 blocks.
 ==23818==still reachable: 29,157,453 bytes in 179,383 blocks.
 ==23818== suppressed: 0 bytes in 0 blocks.

 So my conclusion is that you are on the right way with the cleanup
 code.

What is our goal (i.e. what is a clean python run?) It's a bit better  
but it seems like we've barely made a dent in it--are the  
dictionaries we were looking at decrefing earlier still around? (They  
could be holding onto a lot of stuff.)

 Hopefully once we can compile integer.pyx with cleanup level 3
 and not crash

Try the below patch--this will prevent the double-free from the  
global dummy integer. There might be a better fix, as this will make  
it so that this integer object is never freed.

 the amount of still reachable memory should decrease
 tremendously because integer.pyx should just about be referenced in
 every extension we have.

It doesn't matter how many things reference integer.pyx, only how  
many things are referenced by integer.pyx (which isn't too much, but  
may include such things as element.pyx).

- Robert



--~--~-~--~~~---~--~~
To post to this group, send email to sage-devel@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at http://groups.google.com/group/sage-devel
URLs: http://sage.scipy.org/sage/ and http://modular.math.washington.edu/sage/
-~--~~~~--~~--~--~---



no-collect-integer.patch
Description: Binary data

[sage-devel] Re: potential cause for #557: cython missing dictionary deallocation

2007-10-28 Thread mabshoff 

On Oct 28, 9:48 am, Robert Bradshaw [EMAIL PROTECTED]
wrote:
 On Oct 28, 2007, at 1:29 AM, mabshoff wrote:


Hi Robert,



  For 2.8.10.alpha1+Craig's fix and python compiled using --without-
  pymalloc I get with the default cleanup level 1:

  ==6569== LEAK SUMMARY:
  ==6569==definitely lost: 181,030 bytes in 2,926 blocks.
  ==6569==  possibly lost: 266,925 bytes in 747 blocks.
  ==6569==still reachable: 29,184,388 bytes in 179,707 blocks.
  ==6569== suppressed: 0 bytes in 0 blocks.

  With cleanup level 3 in general and cleanup level 1 just for
  integer.pyx I get

  ==23818== LEAK SUMMARY:
  ==23818==definitely lost: 181,117 bytes in 2,927 blocks.
  ==23818==  possibly lost: 266,838 bytes in 746 blocks.
  ==23818==still reachable: 29,157,453 bytes in 179,383 blocks.
  ==23818== suppressed: 0 bytes in 0 blocks.


With the patch valgrind says:

==3482== LEAK SUMMARY:
==3482==definitely lost: 181,235 bytes in 2,929 blocks.
==3482==  possibly lost: 266,720 bytes in 744 blocks.
==3482==still reachable: 29,157,648 bytes in 179,395 blocks.
==3482== suppressed: 0 bytes in 0 blocks.

So the number are slightly worst.

  So my conclusion is that you are on the right way with the cleanup
  code.

 What is our goal (i.e. what is a clean python run?) It's a bit better
 but it seems like we've barely made a dent in it--are the
 dictionaries we were looking at decrefing earlier still around? (They
 could be holding onto a lot of stuff.)

  Hopefully once we can compile integer.pyx with cleanup level 3
  and not crash

 Try the below patch--this will prevent the double-free from the
 global dummy integer. There might be a better fix, as this will make
 it so that this integer object is never freed.

with the patch everything compiles and runs fine with cleanup level 3.


  the amount of still reachable memory should decrease
  tremendously because integer.pyx should just about be referenced in
  every extension we have.

 It doesn't matter how many things reference integer.pyx, only how
 many things are referenced by integer.pyx (which isn't too much, but
 may include such things as element.pyx).


Ok, but I plan to actually play with this using only a couple small
modules independent of Sage in order to understand the problem better.

The only valgrind complaint I get is the following:

==3482== Invalid free() / delete / delete[]
==3482==at 0x4A1B74A: free (vg_replace_malloc.c:320)
==3482==by 0x43FE9A: dict_dealloc (dictobject.c:847)
==3482==by 0x43FE9A: dict_dealloc (dictobject.c:847)
==3482==by 0x499E5B: _PyImport_Fini (import.c:229)
==3482==by 0x4A5D66: Py_Finalize (pythonrun.c:419)
==3482==by 0x4A58AA: handle_system_exit (pythonrun.c:1616)
==3482==by 0x4A5AA8: PyErr_PrintEx (pythonrun.c:1062)
==3482==by 0x4A62B6: PyRun_SimpleFileExFlags (pythonrun.c:976)
==3482==by 0x412319: Py_Main (main.c:134)
==3482==by 0x4FD94C9: (below main) (in /lib/libc-2.3.6.so)
==3482==  Address 0xb2de9f8 is 32 bytes inside a block of size 80
alloc'd
==3482==at 0x4A1BB35: malloc (vg_replace_malloc.c:207)
==3482==by 0x4B0079: _PyObject_GC_Malloc (gcmodule.c:1321)
==3482==by 0x454C29: PyType_GenericAlloc (typeobject.c:454)
==3482==by 0x975D160:
__pyx_tp_new_4sage_9structure_7element_Element (element.c:14429)
==3482==by 0xAC89AAA: __pyx_tp_new_4sage_5rings_7integer_Integer
(integer.c:14228)
==3482==by 0x458D92: type_call (typeobject.c:422)
==3482==by 0x415542: PyObject_Call (abstract.c:1860)
==3482==by 0x47C480: PyEval_CallObjectWithKeywords (ceval.c:3433)
==3482==by 0xAC85823: initinteger (integer.c:15100)
==3482==by 0x49E17D: _PyImport_LoadDynamicModule (importdl.c:53)
==3482==by 0x49C08D: import_submodule (import.c:2394)
==3482==by 0x49C550: load_next (import.c:2214)

It might be related. I also get a lot of the following:

==3482== 45 bytes in 1 blocks are definitely lost in loss record 525
of 10,255
==3482==at 0x4A1BB35: malloc (vg_replace_malloc.c:207)
==3482==by 0x44D68B: PyString_FromString (stringobject.c:130)
==3482==by 0x13B4DF15: __Pyx_ImportType (real_double_vector.c:
3786)
==3482==by 0x13B52202: initreal_double_vector
(real_double_vector.c:3255)
==3482==by 0x49E17D: _PyImport_LoadDynamicModule (importdl.c:53)
==3482==by 0x49C08D: import_submodule (import.c:2394)
==3482==by 0x49C550: load_next (import.c:2214)
==3482==by 0x49C7AD: import_module_level (import.c:2002)
==3482==by 0x49CBF4: PyImport_ImportModuleLevel (import.c:2066)
==3482==by 0x47BEE8: builtin___import__ (bltinmodule.c:47)
==3482==by 0x415542: PyObject_Call (abstract.c:1860)
==3482==by 0x47C480: PyEval_CallObjectWithKeywords (ceval.c:3433)

 - Robert


Cheers,

Michael

  no-collect-integer.patch
 1KDownload


--~--~-~--~~~---~--~~
To post to this group, send email to sage-devel@googlegroups.com
To unsubscribe from this group, send email

[sage-devel] Re: potential cause for #557: cython missing dictionary deallocation

2007-10-28 Thread Robert Bradshaw 
On Oct 28, 2007, at 2:17 AM, mabshoff wrote:


 On Oct 28, 9:48 am, Robert Bradshaw [EMAIL PROTECTED]
 wrote:
 On Oct 28, 2007, at 1:29 AM, mabshoff wrote:

 Hi Robert,



 For 2.8.10.alpha1+Craig's fix and python compiled using --without-
 pymalloc I get with the default cleanup level 1:

 ==6569== LEAK SUMMARY:
 ==6569==definitely lost: 181,030 bytes in 2,926 blocks.
 ==6569==  possibly lost: 266,925 bytes in 747 blocks.
 ==6569==still reachable: 29,184,388 bytes in 179,707 blocks.
 ==6569== suppressed: 0 bytes in 0 blocks.

 With cleanup level 3 in general and cleanup level 1 just for
 integer.pyx I get

 ==23818== LEAK SUMMARY:
 ==23818==definitely lost: 181,117 bytes in 2,927 blocks.
 ==23818==  possibly lost: 266,838 bytes in 746 blocks.
 ==23818==still reachable: 29,157,453 bytes in 179,383 blocks.
 ==23818== suppressed: 0 bytes in 0 blocks.


 With the patch valgrind says:

 ==3482== LEAK SUMMARY:
 ==3482==definitely lost: 181,235 bytes in 2,929 blocks.
 ==3482==  possibly lost: 266,720 bytes in 744 blocks.
 ==3482==still reachable: 29,157,648 bytes in 179,395 blocks.
 ==3482== suppressed: 0 bytes in 0 blocks.

 So the number are slightly worst.

Hmm... strange. I wonder if the dummy integer was getting decrefed  
somewhere else. Also, how deterministic are these numbers?

 Hopefully once we can compile integer.pyx with cleanup level 3
 and not crash

 Try the below patch--this will prevent the double-free from the
 global dummy integer. There might be a better fix, as this will make
 it so that this integer object is never freed.

 with the patch everything compiles and runs fine with cleanup level 3.

Good, though it looks like not all issues are resolved (as seen below).

 the amount of still reachable memory should decrease
 tremendously because integer.pyx should just about be referenced in
 every extension we have.

 It doesn't matter how many things reference integer.pyx, only how
 many things are referenced by integer.pyx (which isn't too much, but
 may include such things as element.pyx).


 Ok, but I plan to actually play with this using only a couple small
 modules independent of Sage in order to understand the problem better.

That sound like a good idea.

 The only valgrind complaint I get is the following:

 ==3482== Invalid free() / delete / delete[]
 ==3482==at 0x4A1B74A: free (vg_replace_malloc.c:320)
 ==3482==by 0x43FE9A: dict_dealloc (dictobject.c:847)
 ==3482==by 0x43FE9A: dict_dealloc (dictobject.c:847)
 ==3482==by 0x499E5B: _PyImport_Fini (import.c:229)
 ==3482==by 0x4A5D66: Py_Finalize (pythonrun.c:419)
 ==3482==by 0x4A58AA: handle_system_exit (pythonrun.c:1616)
 ==3482==by 0x4A5AA8: PyErr_PrintEx (pythonrun.c:1062)
 ==3482==by 0x4A62B6: PyRun_SimpleFileExFlags (pythonrun.c:976)
 ==3482==by 0x412319: Py_Main (main.c:134)
 ==3482==by 0x4FD94C9: (below main) (in /lib/libc-2.3.6.so)
 ==3482==  Address 0xb2de9f8 is 32 bytes inside a block of size 80
 alloc'd
 ==3482==at 0x4A1BB35: malloc (vg_replace_malloc.c:207)
 ==3482==by 0x4B0079: _PyObject_GC_Malloc (gcmodule.c:1321)
 ==3482==by 0x454C29: PyType_GenericAlloc (typeobject.c:454)
 ==3482==by 0x975D160:
 __pyx_tp_new_4sage_9structure_7element_Element (element.c:14429)
 ==3482==by 0xAC89AAA: __pyx_tp_new_4sage_5rings_7integer_Integer
 (integer.c:14228)
 ==3482==by 0x458D92: type_call (typeobject.c:422)
 ==3482==by 0x415542: PyObject_Call (abstract.c:1860)
 ==3482==by 0x47C480: PyEval_CallObjectWithKeywords (ceval.c:3433)
 ==3482==by 0xAC85823: initinteger (integer.c:15100)
 ==3482==by 0x49E17D: _PyImport_LoadDynamicModule (importdl.c:53)
 ==3482==by 0x49C08D: import_submodule (import.c:2394)
 ==3482==by 0x49C550: load_next (import.c:2214)

This looks like it has something to do with the very first integer  
created, though I'm not totally sure how to read the log.

 It might be related. I also get a lot of the following:

 ==3482== 45 bytes in 1 blocks are definitely lost in loss record 525
 of 10,255
 ==3482==at 0x4A1BB35: malloc (vg_replace_malloc.c:207)
 ==3482==by 0x44D68B: PyString_FromString (stringobject.c:130)
 ==3482==by 0x13B4DF15: __Pyx_ImportType (real_double_vector.c:
 3786)
 ==3482==by 0x13B52202: initreal_double_vector
 (real_double_vector.c:3255)
 ==3482==by 0x49E17D: _PyImport_LoadDynamicModule (importdl.c:53)
 ==3482==by 0x49C08D: import_submodule (import.c:2394)
 ==3482==by 0x49C550: load_next (import.c:2214)
 ==3482==by 0x49C7AD: import_module_level (import.c:2002)
 ==3482==by 0x49CBF4: PyImport_ImportModuleLevel (import.c:2066)
 ==3482==by 0x47BEE8: builtin___import__ (bltinmodule.c:47)
 ==3482==by 0x415542: PyObject_Call (abstract.c:1860)
 ==3482==by 0x47C480: PyEval_CallObjectWithKeywords (ceval.c:3433)

I see what's going wrong here, try applying the attached patch. This  
is probably all over the place.

-

[sage-devel] Re: potential cause for #557: cython missing dictionary deallocation

2007-10-28 Thread mabshoff 



On Oct 28, 11:26 am, mabshoff [EMAIL PROTECTED]
dortmund.de wrote:
 On Oct 28, 10:33 am, Robert Bradshaw [EMAIL PROTECTED]
 wrote:

  On Oct 28, 2007, at 2:17 AM, mabshoff wrote:

 Hi Robert,





   On Oct 28, 9:48 am, Robert Bradshaw [EMAIL PROTECTED]
   wrote:
   On Oct 28, 2007, at 1:29 AM, mabshoff wrote:

   Hi Robert,

   For 2.8.10.alpha1+Craig's fix and python compiled using --without-
   pymalloc I get with the default cleanup level 1:

   ==6569== LEAK SUMMARY:
   ==6569==definitely lost: 181,030 bytes in 2,926 blocks.
   ==6569==  possibly lost: 266,925 bytes in 747 blocks.
   ==6569==still reachable: 29,184,388 bytes in 179,707 blocks.
   ==6569== suppressed: 0 bytes in 0 blocks.

   With cleanup level 3 in general and cleanup level 1 just for
   integer.pyx I get

   ==23818== LEAK SUMMARY:
   ==23818==definitely lost: 181,117 bytes in 2,927 blocks.
   ==23818==  possibly lost: 266,838 bytes in 746 blocks.
   ==23818==still reachable: 29,157,453 bytes in 179,383 blocks.
   ==23818== suppressed: 0 bytes in 0 blocks.

   With the patch valgrind says:

   ==3482== LEAK SUMMARY:
   ==3482==definitely lost: 181,235 bytes in 2,929 blocks.
   ==3482==  possibly lost: 266,720 bytes in 744 blocks.
   ==3482==still reachable: 29,157,648 bytes in 179,395 blocks.
   ==3482== suppressed: 0 bytes in 0 blocks.

   So the number are slightly worst.

  Hmm... strange. I wonder if the dummy integer was getting decrefed
  somewhere else. Also, how deterministic are these numbers?

 The amount of memory leaked can vary a little, but the number of
 blocks is usually constant.



   Hopefully once we can compile integer.pyx with cleanup level 3
   and not crash

   Try the below patch--this will prevent the double-free from the
   global dummy integer. There might be a better fix, as this will make
   it so that this integer object is never freed.

   with the patch everything compiles and runs fine with cleanup level 3.

  Good, though it looks like not all issues are resolved (as seen below).

 Well, it is certainly a step by step process and we are making
 progress, so I am happy.

   the amount of still reachable memory should decrease
   tremendously because integer.pyx should just about be referenced in
   every extension we have.

   It doesn't matter how many things reference integer.pyx, only how
   many things are referenced by integer.pyx (which isn't too much, but
   may include such things as element.pyx).

   Ok, but I plan to actually play with this using only a couple small
   modules independent of Sage in order to understand the problem better.

  That sound like a good idea.

 Yep, all I now need is the time to actually do it.



   The only valgrind complaint I get is the following:

   ==3482== Invalid free() / delete / delete[]
   ==3482==at 0x4A1B74A: free (vg_replace_malloc.c:320)
   ==3482==by 0x43FE9A: dict_dealloc (dictobject.c:847)
   ==3482==by 0x43FE9A: dict_dealloc (dictobject.c:847)
   ==3482==by 0x499E5B: _PyImport_Fini (import.c:229)
   ==3482==by 0x4A5D66: Py_Finalize (pythonrun.c:419)
   ==3482==by 0x4A58AA: handle_system_exit (pythonrun.c:1616)
   ==3482==by 0x4A5AA8: PyErr_PrintEx (pythonrun.c:1062)
   ==3482==by 0x4A62B6: PyRun_SimpleFileExFlags (pythonrun.c:976)
   ==3482==by 0x412319: Py_Main (main.c:134)
   ==3482==by 0x4FD94C9: (below main) (in /lib/libc-2.3.6.so)
   ==3482==  Address 0xb2de9f8 is 32 bytes inside a block of size 80
   alloc'd
   ==3482==at 0x4A1BB35: malloc (vg_replace_malloc.c:207)
   ==3482==by 0x4B0079: _PyObject_GC_Malloc (gcmodule.c:1321)
   ==3482==by 0x454C29: PyType_GenericAlloc (typeobject.c:454)
   ==3482==by 0x975D160:
   __pyx_tp_new_4sage_9structure_7element_Element (element.c:14429)
   ==3482==by 0xAC89AAA: __pyx_tp_new_4sage_5rings_7integer_Integer
   (integer.c:14228)
   ==3482==by 0x458D92: type_call (typeobject.c:422)
   ==3482==by 0x415542: PyObject_Call (abstract.c:1860)
   ==3482==by 0x47C480: PyEval_CallObjectWithKeywords (ceval.c:3433)
   ==3482==by 0xAC85823: initinteger (integer.c:15100)
   ==3482==by 0x49E17D: _PyImport_LoadDynamicModule (importdl.c:53)
   ==3482==by 0x49C08D: import_submodule (import.c:2394)
   ==3482==by 0x49C550: load_next (import.c:2214)

  This looks like it has something to do with the very first integer
  created, though I'm not totally sure how to read the log.

   It might be related. I also get a lot of the following:

   ==3482== 45 bytes in 1 blocks are definitely lost in loss record 525
   of 10,255
   ==3482==at 0x4A1BB35: malloc (vg_replace_malloc.c:207)
   ==3482==by 0x44D68B: PyString_FromString (stringobject.c:130)
   ==3482==by 0x13B4DF15: __Pyx_ImportType (real_double_vector.c:
   3786)
   ==3482==by 0x13B52202: initreal_double_vector
   (real_double_vector.c:3255)
   ==3482==by 0x49E17D: _PyImport_LoadDynamicModule

[sage-devel] Re: potential cause for #557: cython missing dictionary deallocation

2007-10-26 Thread Robert Bradshaw 

On Oct 25, 2007, at 9:35 PM, mabshoff wrote:

 Carl Witty was also wondering whether your new Cython.spkg was ready
 for 2.8.10 or if we should wait.

 I believe so, it compiles all of SAGE and passes all doctests. Stefan
 (the other main Cython guy) is having trouble getting his code to
 work though.


 ok.

BTW, I just uploaded a new spkg with one more tiny fix (has to do  
with error reporting).

- Robert

--~--~-~--~~~---~--~~
To post to this group, send email to sage-devel@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at http://groups.google.com/group/sage-devel
URLs: http://sage.scipy.org/sage/ and http://modular.math.washington.edu/sage/
-~--~~~~--~~--~--~---

[sage-devel] Re: potential cause for #557: cython missing dictionary deallocation

2007-10-25 Thread Robert Bradshaw 

See http://sage.math.washington.edu/home/robertwb/cython/ All tests  
pass with cleanup level 1 (now default). The command line option

--cleanup n

sets the level (0 = n = 3). Things crash on quit (in SAGE, though  
not with simple examples) with level 3. This may or may not fix #557,  
but is a start (but at the very least should make the memcheck logs  
somewhat cleaner). We can play with it from here).

- Robert


On Oct 24, 2007, at 12:56 PM, Robert Bradshaw wrote:


 On Oct 24, 2007, at 12:39 PM, mabshoff wrote:

 On Oct 24, 9:01 pm, Robert Bradshaw [EMAIL PROTECTED]
 wrote:

 Robert,

 I've implemented a function and hook to do this in the new  
 version of
 Cython,

 Cool.

 but I'm not sure how well it will work in practice on the
 entire SAGE library. It will decref local variable and all, but then
 if anything (e.g. other destructors) ever try and use any of the
 module again it will crash with high probability (because the
 expected variables will no longer be set).


 Well, if it crashes upon exit I would consider that a bug and we can
 hopefully fix them.

 There are three levels of cleanup--each more aggressive than the
 previous, but even the most basic level will potentially render any
 of the module-level functions unusable. Hopefully we can find the
 minimum set of things to decref so that the python interpreter can
 take care of the rest.

 If you have an inofficial spkg I would really like to take it for a
 spin.

 I'm hoping to have time to package it all up tonight and make sure
 SAGE still builds fine. If not, at the very least I'll post an
 unofficial package.

 - Robert


 

--~--~-~--~~~---~--~~
To post to this group, send email to sage-devel@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at http://groups.google.com/group/sage-devel
URLs: http://sage.scipy.org/sage/ and http://modular.math.washington.edu/sage/
-~--~~~~--~~--~--~---

[sage-devel] Re: potential cause for #557: cython missing dictionary deallocation

2007-10-25 Thread mabshoff 



On Oct 25, 9:06 am, Robert Bradshaw [EMAIL PROTECTED]
wrote:

Hi Robert,

 Seehttp://sage.math.washington.edu/home/robertwb/cython/All tests
 pass with cleanup level 1 (now default). The command line option

 --cleanup n

 sets the level (0 = n = 3). Things crash on quit (in SAGE, though
 not with simple examples) with level 3. This may or may not fix #557,
 but is a start (but at the very least should make the memcheck logs
 somewhat cleaner). We can play with it from here).

Excellent, I will play with this tonight and report back.


 - Robert


Cheers,

Michael

SNIP


--~--~-~--~~~---~--~~
To post to this group, send email to sage-devel@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at http://groups.google.com/group/sage-devel
URLs: http://sage.scipy.org/sage/ and http://modular.math.washington.edu/sage/
-~--~~~~--~~--~--~---

[sage-devel] Re: potential cause for #557: cython missing dictionary deallocation

2007-10-25 Thread mabshoff 



On Oct 25, 11:25 am, mabshoff [EMAIL PROTECTED]
dortmund.de wrote:
 On Oct 25, 9:06 am, Robert Bradshaw [EMAIL PROTECTED]
 wrote:

 Hi Robert,

  Seehttp://sage.math.washington.edu/home/robertwb/cython/Alltests
  pass with cleanup level 1 (now default). The command line option

  --cleanup n

  sets the level (0 = n = 3). Things crash on quit (in SAGE, though
  not with simple examples) with level 3. This may or may not fix #557,
  but is a start (but at the very least should make the memcheck logs
  somewhat cleaner). We can play with it from here).

 Excellent, I will play with this tonight and report back.



Hmm, with 2.8.9 I get the following error when using

http://sage.math.washington.edu/home/robertwb/cython/cython-0.9.6.8.spkg


Building sage/matrix/misc.c because it depends on sage/matrix/
misc.pyx.
touch sage/matrix/misc.pyx; cython --embed-positions --incref-local-
binop -I/tmp/Work-mabshoff/sage-2.8.9/devel/sage-main -o sage/matrix/
misc.c sage/matrix/misc.pyx

Error converting Pyrex file to C:

...
 0, 0, 0)
cdef mpz_t* L_row
cdef mod_int* A_row
for i from 0 = i  A._nrows:
L_row = L._matrix[i]
A_row = A._matrix[i]
^


/tmp/Work-mabshoff/sage-2.8.9/devel/sage-main/sage/matrix/misc.pyx:
54:25: Cannot assign type 'sage.matrix.matrix_modn_dense.mod_int *' to
'sage.matrix.misc.mod_int *'
sage: Error running cython.
sage: There was an error installing modified sage library code.

I don't think misc.pyx has changed in a while. Any ideas?


  - Robert


Cheers,

Michael

SNIP


--~--~-~--~~~---~--~~
To post to this group, send email to sage-devel@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at http://groups.google.com/group/sage-devel
URLs: http://sage.scipy.org/sage/ and http://modular.math.washington.edu/sage/
-~--~~~~--~~--~--~---

[sage-devel] Re: potential cause for #557: cython missing dictionary deallocation

2007-10-25 Thread Robert Bradshaw 

On Oct 25, 2007, at 10:03 AM, mabshoff wrote:

 On Oct 25, 11:25 am, mabshoff [EMAIL PROTECTED]
 dortmund.de wrote:
 On Oct 25, 9:06 am, Robert Bradshaw [EMAIL PROTECTED]
 wrote:

 Hi Robert,

 Seehttp://sage.math.washington.edu/home/robertwb/cython/Alltests
 pass with cleanup level 1 (now default). The command line option

 --cleanup n

 sets the level (0 = n = 3). Things crash on quit (in SAGE, though
 not with simple examples) with level 3. This may or may not fix  
 #557,
 but is a start (but at the very least should make the memcheck logs
 somewhat cleaner). We can play with it from here).

 Excellent, I will play with this tonight and report back.



 Hmm, with 2.8.9 I get the following error when using

 http://sage.math.washington.edu/home/robertwb/cython/ 
 cython-0.9.6.8.spkg


 Building sage/matrix/misc.c because it depends on sage/matrix/
 misc.pyx.
 touch sage/matrix/misc.pyx; cython --embed-positions --incref-local-
 binop -I/tmp/Work-mabshoff/sage-2.8.9/devel/sage-main -o sage/matrix/
 misc.c sage/matrix/misc.pyx

 Error converting Pyrex file to C:
 
 ...
  0, 0, 0)
 cdef mpz_t* L_row
 cdef mod_int* A_row
 for i from 0 = i  A._nrows:
 L_row = L._matrix[i]
 A_row = A._matrix[i]
 ^
 

 /tmp/Work-mabshoff/sage-2.8.9/devel/sage-main/sage/matrix/misc.pyx:
 54:25: Cannot assign type 'sage.matrix.matrix_modn_dense.mod_int *' to
 'sage.matrix.misc.mod_int *'
 sage: Error running cython.
 sage: There was an error installing modified sage library code.

 I don't think misc.pyx has changed in a while. Any ideas?

Did you apply the accompanying .hg patch to sage-main?


--~--~-~--~~~---~--~~
To post to this group, send email to sage-devel@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at http://groups.google.com/group/sage-devel
URLs: http://sage.scipy.org/sage/ and http://modular.math.washington.edu/sage/
-~--~~~~--~~--~--~---

[sage-devel] Re: potential cause for #557: cython missing dictionary deallocation

2007-10-25 Thread mabshoff 



On Oct 25, 8:04 pm, Robert Bradshaw [EMAIL PROTECTED]
wrote:
 On Oct 25, 2007, at 10:03 AM, mabshoff wrote:



  On Oct 25, 11:25 am, mabshoff [EMAIL PROTECTED]
  dortmund.de wrote:
  On Oct 25, 9:06 am, Robert Bradshaw [EMAIL PROTECTED]
  wrote:

  Hi Robert,

  Seehttp://sage.math.washington.edu/home/robertwb/cython/Alltests
  pass with cleanup level 1 (now default). The command line option

  --cleanup n

  sets the level (0 = n = 3). Things crash on quit (in SAGE, though
  not with simple examples) with level 3. This may or may not fix
  #557,
  but is a start (but at the very least should make the memcheck logs
  somewhat cleaner). We can play with it from here).

  Excellent, I will play with this tonight and report back.

  Hmm, with 2.8.9 I get the following error when using

 http://sage.math.washington.edu/home/robertwb/cython/
  cython-0.9.6.8.spkg

  Building sage/matrix/misc.c because it depends on sage/matrix/
  misc.pyx.
  touch sage/matrix/misc.pyx; cython --embed-positions --incref-local-
  binop -I/tmp/Work-mabshoff/sage-2.8.9/devel/sage-main -o sage/matrix/
  misc.c sage/matrix/misc.pyx

  Error converting Pyrex file to C:
  
  ...
   0, 0, 0)
  cdef mpz_t* L_row
  cdef mod_int* A_row
  for i from 0 = i  A._nrows:
  L_row = L._matrix[i]
  A_row = A._matrix[i]
  ^
  

  /tmp/Work-mabshoff/sage-2.8.9/devel/sage-main/sage/matrix/misc.pyx:
  54:25: Cannot assign type 'sage.matrix.matrix_modn_dense.mod_int *' to
  'sage.matrix.misc.mod_int *'
  sage: Error running cython.
  sage: There was an error installing modified sage library code.

  I don't think misc.pyx has changed in a while. Any ideas?


Hi Robert,

 Did you apply the accompanying .hg patch to sage-main?

nope, but that fixed the problem. It didn't seem obvious, so I ignored
that bundle.

With the old Cython:

==26784== LEAK SUMMARY:
==26784==definitely lost: 0 bytes in 0 blocks.
==26784==  possibly lost: 251,078 bytes in 625 blocks.
==26784==still reachable: 36,000,918 bytes in 20,368 blocks.
==26784== suppressed: 0 bytes in 0 blocks.

With the new Cython and  n=1 I get:

==31741== LEAK SUMMARY:
==31741==definitely lost: 0 bytes in 0 blocks.
==31741==  possibly lost: 253,574 bytes in 633 blocks.
==31741==still reachable: 36,244,988 bytes in 20,359 blocks.
==31741== suppressed: 0 bytes in 0 blocks.

So there has been a slight decrease in the number of blocks that are
still reachable or possibly lost, even thought the amount of memory
increased. I will try with n=2 to see if anything changes. How do I
change the default by the way?

Cheers,

Michael


--~--~-~--~~~---~--~~
To post to this group, send email to sage-devel@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at http://groups.google.com/group/sage-devel
URLs: http://sage.scipy.org/sage/ and http://modular.math.washington.edu/sage/
-~--~~~~--~~--~--~---

[sage-devel] Re: potential cause for #557: cython missing dictionary deallocation

2007-10-25 Thread Robert Bradshaw 

On Oct 25, 2007, at 12:14 PM, mabshoff wrote:

 On Oct 25, 8:04 pm, Robert Bradshaw [EMAIL PROTECTED]
 wrote:
 On Oct 25, 2007, at 10:03 AM, mabshoff wrote:

 On Oct 25, 11:25 am, mabshoff  
 [EMAIL PROTECTED]
 dortmund.de wrote:
 On Oct 25, 9:06 am, Robert Bradshaw [EMAIL PROTECTED]
 Did you apply the accompanying .hg patch to sage-main?

 nope, but that fixed the problem. It didn't seem obvious, so I ignored
 that bundle.

There's often a few tweaks that need to be done to the SAGE codebase  
to get it to work with the new cython...that one there was fixing a  
hack that didn't work anymore.

 With the old Cython:

 ==26784== LEAK SUMMARY:
 ==26784==definitely lost: 0 bytes in 0 blocks.
 ==26784==  possibly lost: 251,078 bytes in 625 blocks.
 ==26784==still reachable: 36,000,918 bytes in 20,368 blocks.
 ==26784== suppressed: 0 bytes in 0 blocks.

 With the new Cython and  n=1 I get:

 ==31741== LEAK SUMMARY:
 ==31741==definitely lost: 0 bytes in 0 blocks.
 ==31741==  possibly lost: 253,574 bytes in 633 blocks.
 ==31741==still reachable: 36,244,988 bytes in 20,359 blocks.
 ==31741== suppressed: 0 bytes in 0 blocks.

 So there has been a slight decrease in the number of blocks that are
 still reachable or possibly lost, even thought the amount of memory
 increased.

More stuff is interned with the new cython (e.g. python int constants  
are pre-computed rather than constructed/destructed on the fly  
throughout the code) as well as other changes. I'm curious how n=0  
vs. n=1 behaves, as well as, of course, n=2+.

 I will try with n=2 to see if anything changes. How do I
 change the default by the way?

Try adding --clean 2 to line 1008 of process_cython_file() in  
setup.py.

- Robert


--~--~-~--~~~---~--~~
To post to this group, send email to sage-devel@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at http://groups.google.com/group/sage-devel
URLs: http://sage.scipy.org/sage/ and http://modular.math.washington.edu/sage/
-~--~~~~--~~--~--~---

[sage-devel] Re: potential cause for #557: cython missing dictionary deallocation

2007-10-25 Thread mabshoff 



On Oct 25, 9:34 pm, Robert Bradshaw [EMAIL PROTECTED]
wrote:
 On Oct 25, 2007, at 12:14 PM, mabshoff wrote:

  On Oct 25, 8:04 pm, Robert Bradshaw [EMAIL PROTECTED]
  wrote:
  On Oct 25, 2007, at 10:03 AM, mabshoff wrote:

  On Oct 25, 11:25 am, mabshoff
  [EMAIL PROTECTED]
  dortmund.de wrote:
  On Oct 25, 9:06 am, Robert Bradshaw [EMAIL PROTECTED]
  Did you apply the accompanying .hg patch to sage-main?

  nope, but that fixed the problem. It didn't seem obvious, so I ignored
  that bundle.

 There's often a few tweaks that need to be done to the SAGE codebase
 to get it to work with the new cython...that one there was fixing a
 hack that didn't work anymore.



  With the old Cython:

  ==26784== LEAK SUMMARY:
  ==26784==definitely lost: 0 bytes in 0 blocks.
  ==26784==  possibly lost: 251,078 bytes in 625 blocks.
  ==26784==still reachable: 36,000,918 bytes in 20,368 blocks.
  ==26784== suppressed: 0 bytes in 0 blocks.

  With the new Cython and  n=1 I get:

  ==31741== LEAK SUMMARY:
  ==31741==definitely lost: 0 bytes in 0 blocks.
  ==31741==  possibly lost: 253,574 bytes in 633 blocks.
  ==31741==still reachable: 36,244,988 bytes in 20,359 blocks.
  ==31741== suppressed: 0 bytes in 0 blocks.

  So there has been a slight decrease in the number of blocks that are
  still reachable or possibly lost, even thought the amount of memory
  increased.

 More stuff is interned with the new cython (e.g. python int constants
 are pre-computed rather than constructed/destructed on the fly
 throughout the code) as well as other changes. I'm curious how n=0
 vs. n=1 behaves, as well as, of course, n=2+.

  I will try with n=2 to see if anything changes. How do I
  change the default by the way?

Hi Robert,

 Try adding --clean 2 to line 1008 of process_cython_file() in
 setup.py.


that doesn't work yet, because the parsing of the command line doesn't
seem to comprehend --clean yet, so I hardcoded the value I wanted for
now.

With n=2 I get the following which might be the cause for the memory
corruption:

==14064== Invalid free() / delete / delete[]
==14064==at 0x4A1B74A: free (vg_replace_malloc.c:320)
==14064==by 0xB98B0D8:
__pyx_f_4sage_5rings_7integer_fast_tp_dealloc (integer.c:13607)
==14064==by 0x4B0022: collect (gcmodule.c:714)
==14064==by 0x4B0373: PyGC_Collect (gcmodule.c:1265)
==14064==by 0x4A612C: Py_Finalize (pythonrun.c:387)
==14064==by 0x4A5C7A: handle_system_exit (pythonrun.c:1616)
==14064==by 0x4A5E78: PyErr_PrintEx (pythonrun.c:1062)
==14064==by 0x4A6686: PyRun_SimpleFileExFlags (pythonrun.c:976)
==14064==by 0x412319: Py_Main (main.c:134)
==14064==by 0x4FD94C9: (below main) (in /lib/libc-2.3.6.so)
==14064==  Address 0x73287d8 is 0 bytes inside a block of size 8
free'd
==14064==at 0x4A1B74A: free (vg_replace_malloc.c:320)
==14064==by 0xB98B0D8:
__pyx_f_4sage_5rings_7integer_fast_tp_dealloc (integer.c:13607)
==14064==by 0xB9847E2: cleanup (integer.c:15198)
==14064==by 0x415522: PyObject_Call (abstract.c:1860)
==14064==by 0x481ACA: PyEval_EvalFrameEx (ceval.c:3844)
==14064==by 0x484F3A: PyEval_EvalCodeEx (ceval.c:2831)
==14064==by 0x4CE527: function_call (funcobject.c:517)
==14064==by 0x415522: PyObject_Call (abstract.c:1860)
==14064==by 0x47C850: PyEval_CallObjectWithKeywords (ceval.c:3433)
==14064==by 0x4A60BC: Py_Finalize (pythonrun.c:1589)
==14064==by 0x4A5C7A: handle_system_exit (pythonrun.c:1616)
==14064==by 0x4A5E78: PyErr_PrintEx (pythonrun.c:1062)


 - Robert

Carl Witty was also wondering whether your new Cython.spkg was ready
for 2.8.10 or if we should wait.

Cheers,

Michael


--~--~-~--~~~---~--~~
To post to this group, send email to sage-devel@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at http://groups.google.com/group/sage-devel
URLs: http://sage.scipy.org/sage/ and http://modular.math.washington.edu/sage/
-~--~~~~--~~--~--~---

[sage-devel] Re: potential cause for #557: cython missing dictionary deallocation

2007-10-25 Thread Robert Bradshaw 

On Oct 25, 2007, at 7:46 PM, mabshoff wrote:

 On Oct 25, 9:34 pm, Robert Bradshaw [EMAIL PROTECTED]
 wrote:
 Hi Robert,

 Try adding --clean 2 to line 1008 of process_cython_file() in
 setup.py.


 that doesn't work yet, because the parsing of the command line doesn't
 seem to comprehend --clean yet, so I hardcoded the value I wanted for
 now.

Sorry, that should have been --cleanup 2, but hardcoding works too.


 With n=2 I get the following which might be the cause for the memory
 corruption:

 ==14064== Invalid free() / delete / delete[]
 ==14064==at 0x4A1B74A: free (vg_replace_malloc.c:320)
 ==14064==by 0xB98B0D8:
 __pyx_f_4sage_5rings_7integer_fast_tp_dealloc (integer.c:13607)
 ==14064==by 0x4B0022: collect (gcmodule.c:714)
 ==14064==by 0x4B0373: PyGC_Collect (gcmodule.c:1265)
 ==14064==by 0x4A612C: Py_Finalize (pythonrun.c:387)
 ==14064==by 0x4A5C7A: handle_system_exit (pythonrun.c:1616)
 ==14064==by 0x4A5E78: PyErr_PrintEx (pythonrun.c:1062)
 ==14064==by 0x4A6686: PyRun_SimpleFileExFlags (pythonrun.c:976)
 ==14064==by 0x412319: Py_Main (main.c:134)
 ==14064==by 0x4FD94C9: (below main) (in /lib/libc-2.3.6.so)
 ==14064==  Address 0x73287d8 is 0 bytes inside a block of size 8
 free'd
 ==14064==at 0x4A1B74A: free (vg_replace_malloc.c:320)
 ==14064==by 0xB98B0D8:
 __pyx_f_4sage_5rings_7integer_fast_tp_dealloc (integer.c:13607)
 ==14064==by 0xB9847E2: cleanup (integer.c:15198)
 ==14064==by 0x415522: PyObject_Call (abstract.c:1860)
 ==14064==by 0x481ACA: PyEval_EvalFrameEx (ceval.c:3844)
 ==14064==by 0x484F3A: PyEval_EvalCodeEx (ceval.c:2831)
 ==14064==by 0x4CE527: function_call (funcobject.c:517)
 ==14064==by 0x415522: PyObject_Call (abstract.c:1860)
 ==14064==by 0x47C850: PyEval_CallObjectWithKeywords (ceval.c:3433)
 ==14064==by 0x4A60BC: Py_Finalize (pythonrun.c:1589)
 ==14064==by 0x4A5C7A: handle_system_exit (pythonrun.c:1616)
 ==14064==by 0x4A5E78: PyErr_PrintEx (pythonrun.c:1062)

This last one might be due to funny stuff we do with the fast alloc/ 
dealloc stuff. Try compiling just the integer file with --cleanup 1  
and see if that fixes it. Perhaps we could incref  
__pyx_v_4sage_5rings_7integer_global_dummy_Integer manually so it's  
never actually collected. Or it could be something more subtle.

 Carl Witty was also wondering whether your new Cython.spkg was ready
 for 2.8.10 or if we should wait.

I believe so, it compiles all of SAGE and passes all doctests. Stefan  
(the other main Cython guy) is having trouble getting his code to  
work though.

- Robert



--~--~-~--~~~---~--~~
To post to this group, send email to sage-devel@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at http://groups.google.com/group/sage-devel
URLs: http://sage.scipy.org/sage/ and http://modular.math.washington.edu/sage/
-~--~~~~--~~--~--~---

[sage-devel] Re: potential cause for #557: cython missing dictionary deallocation

2007-10-25 Thread Robert Bradshaw 

On Oct 25, 2007, at 9:35 PM, mabshoff wrote:

 Yep, for n=2 and n=3 there isn't anything funny going on. Ironically
 valgrind pretty much reports the same amount of still reachable
 memory:

 ==17336== LEAK SUMMARY:
 ==17336==definitely lost: 0 bytes in 0 blocks.
 ==17336==  possibly lost: 253,574 bytes in 633 blocks.
 ==17336==still reachable: 36,239,900 bytes in 20,342 blocks.
 ==17336== suppressed: 0 bytes in 0 blocks.

I was hoping for more than that. Perhaps we could manually decref the  
module's dictionary, but that might be dangerous... Perhaps there's  
other cleanup to do too.

 One more thing: I compiled python --pydebug and I get the following
 crash related to nteger_fast_tp_dealloc, so it seems that something
 very odd is going on there:

Yes, there are a lot of funny things going on in that class for  
speed...in particular, to run in debug mode, you need to uncomment  
the PyObject_INIT line of fast_tp_new() in integer.pyx. You might  
want to talk to Martin Albrecht.

- Robert


 --
 | SAGE Version 2.8.9.alpha0, Release Date: 2007-10-23|
 | Type notebook() for the GUI, and license() for information.|
 --
 sage:
 Exiting SAGE (CPU time 0m0.07s, Wall time 0m2.15s).

 Program received signal SIGSEGV, Segmentation fault.
 [Switching to Thread -1208604992 (LWP 20053)]
 0x0078341b in free () from /lib/libc.so.6
 (gdb) bt
 #0  0x0078341b in free () from /lib/libc.so.6
 #1  0x00ea2f38 in __pyx_f_7integer_fast_tp_dealloc
 (__pyx_v_o=0x91c046c) at sage/rings/integer.c:15544
 #2  0x08083bdf in dict_dealloc (mp=0x91b7334) at Objects/dictobject.c:
 847
 #3  0x08083bdf in dict_dealloc (mp=0x8cf7474) at Objects/dictobject.c:
 847
 #4  0x080d9659 in _PyImport_Fini () at Python/import.c:229
 #5  0x080e4da4 in Py_Finalize () at Python/pythonrun.c:419
 #6  0x080e48ea in handle_system_exit () at Python/pythonrun.c:1616
 #7  0x080e4ae5 in PyErr_PrintEx (set_sys_last_vars=1) at Python/
 pythonrun.c:1062
 #8  0x080e5303 in PyRun_SimpleFileExFlags (fp=0x0,
 filename=0xbfa27cb3 /tmp/Work/sage-2.8.9.rc1/local/bin/sage-gdb-
 pythonstartup, closeit=0, flags=0xbfa268e8)
 at Python/pythonrun.c:976
 #9  0x080571a6 in Py_Main (argc=0, argv=0xbfa269b4) at Modules/main.c:
 134
 #10 0x08056432 in main (argc=Cannot access memory at address 0x1
 ) at ./Modules/python.c:23
 (gdb) quit

 I will try with a clean build to see if I can get to the bottom of
 this.

 Cheers,

 Michael

 - Robert


 

--~--~-~--~~~---~--~~
To post to this group, send email to sage-devel@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at http://groups.google.com/group/sage-devel
URLs: http://sage.scipy.org/sage/ and http://modular.math.washington.edu/sage/
-~--~~~~--~~--~--~---

[sage-devel] Re: potential cause for #557: cython missing dictionary deallocation

2007-10-25 Thread mabshoff 



On Oct 26, 5:18 am, Robert Bradshaw [EMAIL PROTECTED]
wrote:
 On Oct 25, 2007, at 7:46 PM, mabshoff wrote:

  On Oct 25, 9:34 pm, Robert Bradshaw [EMAIL PROTECTED]
  wrote:
  Hi Robert,

  Try adding --clean 2 to line 1008 of process_cython_file() in
  setup.py.

  that doesn't work yet, because the parsing of the command line doesn't
  seem to comprehend --clean yet, so I hardcoded the value I wanted for
  now.

 Sorry, that should have been --cleanup 2, but hardcoding works too.





  With n=2 I get the following which might be the cause for the memory
  corruption:

  ==14064== Invalid free() / delete / delete[]
  ==14064==at 0x4A1B74A: free (vg_replace_malloc.c:320)
  ==14064==by 0xB98B0D8:
  __pyx_f_4sage_5rings_7integer_fast_tp_dealloc (integer.c:13607)
  ==14064==by 0x4B0022: collect (gcmodule.c:714)
  ==14064==by 0x4B0373: PyGC_Collect (gcmodule.c:1265)
  ==14064==by 0x4A612C: Py_Finalize (pythonrun.c:387)
  ==14064==by 0x4A5C7A: handle_system_exit (pythonrun.c:1616)
  ==14064==by 0x4A5E78: PyErr_PrintEx (pythonrun.c:1062)
  ==14064==by 0x4A6686: PyRun_SimpleFileExFlags (pythonrun.c:976)
  ==14064==by 0x412319: Py_Main (main.c:134)
  ==14064==by 0x4FD94C9: (below main) (in /lib/libc-2.3.6.so)
  ==14064==  Address 0x73287d8 is 0 bytes inside a block of size 8
  free'd
  ==14064==at 0x4A1B74A: free (vg_replace_malloc.c:320)
  ==14064==by 0xB98B0D8:
  __pyx_f_4sage_5rings_7integer_fast_tp_dealloc (integer.c:13607)
  ==14064==by 0xB9847E2: cleanup (integer.c:15198)
  ==14064==by 0x415522: PyObject_Call (abstract.c:1860)
  ==14064==by 0x481ACA: PyEval_EvalFrameEx (ceval.c:3844)
  ==14064==by 0x484F3A: PyEval_EvalCodeEx (ceval.c:2831)
  ==14064==by 0x4CE527: function_call (funcobject.c:517)
  ==14064==by 0x415522: PyObject_Call (abstract.c:1860)
  ==14064==by 0x47C850: PyEval_CallObjectWithKeywords (ceval.c:3433)
  ==14064==by 0x4A60BC: Py_Finalize (pythonrun.c:1589)
  ==14064==by 0x4A5C7A: handle_system_exit (pythonrun.c:1616)
  ==14064==by 0x4A5E78: PyErr_PrintEx (pythonrun.c:1062)

Hello Robert,


 This last one might be due to funny stuff we do with the fast alloc/
 dealloc stuff. Try compiling just the integer file with --cleanup 1
 and see if that fixes it.

Yep, for n=2 and n=3 there isn't anything funny going on. Ironically
valgrind pretty much reports the same amount of still reachable
memory:

==17336== LEAK SUMMARY:
==17336==definitely lost: 0 bytes in 0 blocks.
==17336==  possibly lost: 253,574 bytes in 633 blocks.
==17336==still reachable: 36,239,900 bytes in 20,342 blocks.
==17336== suppressed: 0 bytes in 0 blocks.


 Perhaps we could incref
 __pyx_v_4sage_5rings_7integer_global_dummy_Integer manually so it's
 never actually collected. Or it could be something more subtle.

  Carl Witty was also wondering whether your new Cython.spkg was ready
  for 2.8.10 or if we should wait.

 I believe so, it compiles all of SAGE and passes all doctests. Stefan
 (the other main Cython guy) is having trouble getting his code to
 work though.


ok.

One more thing: I compiled python --pydebug and I get the following
crash related to nteger_fast_tp_dealloc, so it seems that something
very odd is going on there:

--
| SAGE Version 2.8.9.alpha0, Release Date: 2007-10-23|
| Type notebook() for the GUI, and license() for information.|
--
sage:
Exiting SAGE (CPU time 0m0.07s, Wall time 0m2.15s).

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread -1208604992 (LWP 20053)]
0x0078341b in free () from /lib/libc.so.6
(gdb) bt
#0  0x0078341b in free () from /lib/libc.so.6
#1  0x00ea2f38 in __pyx_f_7integer_fast_tp_dealloc
(__pyx_v_o=0x91c046c) at sage/rings/integer.c:15544
#2  0x08083bdf in dict_dealloc (mp=0x91b7334) at Objects/dictobject.c:
847
#3  0x08083bdf in dict_dealloc (mp=0x8cf7474) at Objects/dictobject.c:
847
#4  0x080d9659 in _PyImport_Fini () at Python/import.c:229
#5  0x080e4da4 in Py_Finalize () at Python/pythonrun.c:419
#6  0x080e48ea in handle_system_exit () at Python/pythonrun.c:1616
#7  0x080e4ae5 in PyErr_PrintEx (set_sys_last_vars=1) at Python/
pythonrun.c:1062
#8  0x080e5303 in PyRun_SimpleFileExFlags (fp=0x0,
filename=0xbfa27cb3 /tmp/Work/sage-2.8.9.rc1/local/bin/sage-gdb-
pythonstartup, closeit=0, flags=0xbfa268e8)
at Python/pythonrun.c:976
#9  0x080571a6 in Py_Main (argc=0, argv=0xbfa269b4) at Modules/main.c:
134
#10 0x08056432 in main (argc=Cannot access memory at address 0x1
) at ./Modules/python.c:23
(gdb) quit

I will try with a clean build to see if I can get to the bottom of
this.

Cheers,

Michael

 - Robert


--~--~-~--~~~---~--~~
To post to this group, send email to sage-devel@googlegroups.com
To unsubscribe from this group, send email to [EMAIL

[sage-devel] Re: potential cause for #557: cython missing dictionary deallocation

2007-10-24 Thread Robert Bradshaw 

I've implemented a function and hook to do this in the new version of  
Cython, but I'm not sure how well it will work in practice on the  
entire SAGE library. It will decref local variable and all, but then  
if anything (e.g. other destructors) ever try and use any of the  
module again it will crash with high probability (because the  
expected variables will no longer be set).

- Robert


On Oct 17, 2007, at 9:20 PM, mabshoff wrote:

 On Oct 18, 6:16 am, Robert Bradshaw [EMAIL PROTECTED]
 wrote:

 Hi Robert,

 No cleanup function is ever generated, but atexit sounds perfect.

 Good that you seem to agree with my analysis. I ran across it be sheer
 accident while looking at some linker issue that roed asked me about.

 I'm the obvious person to implement this,

 Pretty much 100% what I just claimed in IRC, but cwitty might look at
 it during Bug Day 4.

 I'll see if I can find the time to do so soon.


 Let's hope so.

 - Robert


 Cheers,

 Michael

 On Oct 17, 2007, at 9:04 PM, mabshoff wrote:



 Hello,

 for every module Cython generates runtime support code that is  
 place
 at the end of the converted file. Each one of those has an
 __Pyx_Import function that gets called from

 PyMODINIT_FUNC init$MODULENAME(void)

 static PyObject *__Pyx_Import(PyObject *name, PyObject *from_list) {
 PyObject *__import__ = 0;
 PyObject *empty_list = 0;
 PyObject *module = 0;
 PyObject *global_dict = 0;
 PyObject *empty_dict = 0;
 PyObject *list;
 __import__ = PyObject_GetAttrString(__pyx_b, __import__);
 if (!__import__)
 goto bad;
 if (from_list)
 list = from_list;
 else {
 empty_list = PyList_New(0);
 if (!empty_list)
 goto bad;
 list = empty_list;
 }
 global_dict = PyModule_GetDict(__pyx_m);
 if (!global_dict)
 goto bad;
 empty_dict = PyDict_New();
 if (!empty_dict)
 goto bad;
 module = PyObject_CallFunction(__import__, ,
 name, global_dict, empty_dict, list);
 bad:
 Py_XDECREF(empty_list);
 Py_XDECREF(__import__);
 Py_XDECREF(empty_dict);
 return module;
 }

 In that function PyMODINIT_FUNC init$MODULENAME(void) we also call
 __Pyx_ImportType quite often:

 static PyTypeObject *__Pyx_ImportType(char *module_name, char
 *class_name,
 long size)
 {
 PyObject *py_module_name = 0;
 PyObject *py_class_name = 0;
 PyObject *py_name_list = 0;
 PyObject *py_module = 0;
 PyObject *result = 0;

 py_module_name = PyString_FromString(module_name);
 if (!py_module_name)
 goto bad;
 py_class_name = PyString_FromString(class_name);
 if (!py_class_name)
 goto bad;
 py_name_list = PyList_New(1);
 if (!py_name_list)
 goto bad;
 Py_INCREF(py_class_name);
 if (PyList_SetItem(py_name_list, 0, py_class_name)  0)
 goto bad;
 py_module = __Pyx_Import(py_module_name, py_name_list);
 if (!py_module)
 goto bad;
 result = PyObject_GetAttr(py_module, py_class_name);
 if (!result)
 goto bad;
 if (!PyType_Check(result)) {
 PyErr_Format(PyExc_TypeError,
 %s.%s is not a type object,
 module_name, class_name);
 goto bad;
 }
 if (((PyTypeObject *)result)-tp_basicsize != size) {
 PyErr_Format(PyExc_ValueError,
 %s.%s does not appear to be the correct type object,
 module_name, class_name);
 goto bad;
 }
 goto done;
 bad:
 Py_XDECREF(result);
 result = 0;
 done:
 Py_XDECREF(py_module_name);
 Py_XDECREF(py_class_name);
 Py_XDECREF(py_name_list);
 return (PyTypeObject *)result;
 }

 As one can see certain PyObject have refcounts greater 0 when
 initialization is successful. But I cannot find a cleanup function
 that would decrease the reference count upon exit. Consequently
 python's garbage collector never frees those dictionaries, which in
 most cases doesn't matter because we are exiting python anyway [some
 people claim that one shouldn't care about still reachable memory,
 because cleaning it up the nice way just slows down the  
 termination of
 a process because the system will reap the heap and stack
 automatically]. But it pollutes the memcheck log, which is why I  
 care
 about this.

 The usual way to call those functions for each module would be to
 register an atexit function, see

 http://docs.python.org/lib/module-atexit.html

 So if anybody want to write am autogenerated cleanup function for
 Cython and register it via atexit that person would be very  
 welcome :)

 Cheers,
 Michael


 

--~--~-~--~~~---~--~~
To post to this group, send email to sage-devel@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at http://groups.google.com/group/sage-devel
URLs: http://sage.scipy.org/sage/ and http://modular.math.washington.edu/sage/

[sage-devel] Re: potential cause for #557: cython missing dictionary deallocation

2007-10-24 Thread mabshoff 



On Oct 24, 9:01 pm, Robert Bradshaw [EMAIL PROTECTED]
wrote:

Robert,

 I've implemented a function and hook to do this in the new version of
 Cython,

Cool.

 but I'm not sure how well it will work in practice on the
 entire SAGE library. It will decref local variable and all, but then
 if anything (e.g. other destructors) ever try and use any of the
 module again it will crash with high probability (because the
 expected variables will no longer be set).


Well, if it crashes upon exit I would consider that a bug and we can
hopefully fix them.

If you have an inofficial spkg I would really like to take it for a
spin.

 - Robert

Cheers,

Michael

SNIP


--~--~-~--~~~---~--~~
To post to this group, send email to sage-devel@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at http://groups.google.com/group/sage-devel
URLs: http://sage.scipy.org/sage/ and http://modular.math.washington.edu/sage/
-~--~~~~--~~--~--~---

[sage-devel] Re: potential cause for #557: cython missing dictionary deallocation

2007-10-24 Thread Robert Bradshaw 

On Oct 24, 2007, at 12:39 PM, mabshoff wrote:

 On Oct 24, 9:01 pm, Robert Bradshaw [EMAIL PROTECTED]
 wrote:

 Robert,

 I've implemented a function and hook to do this in the new version of
 Cython,

 Cool.

 but I'm not sure how well it will work in practice on the
 entire SAGE library. It will decref local variable and all, but then
 if anything (e.g. other destructors) ever try and use any of the
 module again it will crash with high probability (because the
 expected variables will no longer be set).


 Well, if it crashes upon exit I would consider that a bug and we can
 hopefully fix them.

There are three levels of cleanup--each more aggressive than the  
previous, but even the most basic level will potentially render any  
of the module-level functions unusable. Hopefully we can find the  
minimum set of things to decref so that the python interpreter can  
take care of the rest.

 If you have an inofficial spkg I would really like to take it for a
 spin.

I'm hoping to have time to package it all up tonight and make sure  
SAGE still builds fine. If not, at the very least I'll post an  
unofficial package.

- Robert


--~--~-~--~~~---~--~~
To post to this group, send email to sage-devel@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at http://groups.google.com/group/sage-devel
URLs: http://sage.scipy.org/sage/ and http://modular.math.washington.edu/sage/
-~--~~~~--~~--~--~---

[sage-devel] Re: potential cause for #557: cython missing dictionary deallocation

2007-10-17 Thread Robert Bradshaw 

No cleanup function is ever generated, but atexit sounds perfect. I'm  
the obvious person to implement this, I'll see if I can find the time  
to do so soon.

- Robert


On Oct 17, 2007, at 9:04 PM, mabshoff wrote:


 Hello,

 for every module Cython generates runtime support code that is place
 at the end of the converted file. Each one of those has an
 __Pyx_Import function that gets called from

 PyMODINIT_FUNC init$MODULENAME(void)

 static PyObject *__Pyx_Import(PyObject *name, PyObject *from_list) {
 PyObject *__import__ = 0;
 PyObject *empty_list = 0;
 PyObject *module = 0;
 PyObject *global_dict = 0;
 PyObject *empty_dict = 0;
 PyObject *list;
 __import__ = PyObject_GetAttrString(__pyx_b, __import__);
 if (!__import__)
 goto bad;
 if (from_list)
 list = from_list;
 else {
 empty_list = PyList_New(0);
 if (!empty_list)
 goto bad;
 list = empty_list;
 }
 global_dict = PyModule_GetDict(__pyx_m);
 if (!global_dict)
 goto bad;
 empty_dict = PyDict_New();
 if (!empty_dict)
 goto bad;
 module = PyObject_CallFunction(__import__, ,
 name, global_dict, empty_dict, list);
 bad:
 Py_XDECREF(empty_list);
 Py_XDECREF(__import__);
 Py_XDECREF(empty_dict);
 return module;
 }

 In that function PyMODINIT_FUNC init$MODULENAME(void) we also call
 __Pyx_ImportType quite often:

 static PyTypeObject *__Pyx_ImportType(char *module_name, char
 *class_name,
 long size)
 {
 PyObject *py_module_name = 0;
 PyObject *py_class_name = 0;
 PyObject *py_name_list = 0;
 PyObject *py_module = 0;
 PyObject *result = 0;

 py_module_name = PyString_FromString(module_name);
 if (!py_module_name)
 goto bad;
 py_class_name = PyString_FromString(class_name);
 if (!py_class_name)
 goto bad;
 py_name_list = PyList_New(1);
 if (!py_name_list)
 goto bad;
 Py_INCREF(py_class_name);
 if (PyList_SetItem(py_name_list, 0, py_class_name)  0)
 goto bad;
 py_module = __Pyx_Import(py_module_name, py_name_list);
 if (!py_module)
 goto bad;
 result = PyObject_GetAttr(py_module, py_class_name);
 if (!result)
 goto bad;
 if (!PyType_Check(result)) {
 PyErr_Format(PyExc_TypeError,
 %s.%s is not a type object,
 module_name, class_name);
 goto bad;
 }
 if (((PyTypeObject *)result)-tp_basicsize != size) {
 PyErr_Format(PyExc_ValueError,
 %s.%s does not appear to be the correct type object,
 module_name, class_name);
 goto bad;
 }
 goto done;
 bad:
 Py_XDECREF(result);
 result = 0;
 done:
 Py_XDECREF(py_module_name);
 Py_XDECREF(py_class_name);
 Py_XDECREF(py_name_list);
 return (PyTypeObject *)result;
 }

 As one can see certain PyObject have refcounts greater 0 when
 initialization is successful. But I cannot find a cleanup function
 that would decrease the reference count upon exit. Consequently
 python's garbage collector never frees those dictionaries, which in
 most cases doesn't matter because we are exiting python anyway [some
 people claim that one shouldn't care about still reachable memory,
 because cleaning it up the nice way just slows down the termination of
 a process because the system will reap the heap and stack
 automatically]. But it pollutes the memcheck log, which is why I care
 about this.

 The usual way to call those functions for each module would be to
 register an atexit function, see

 http://docs.python.org/lib/module-atexit.html

 So if anybody want to write am autogenerated cleanup function for
 Cython and register it via atexit that person would be very welcome :)

 Cheers,
 Michael


 

--~--~-~--~~~---~--~~
To post to this group, send email to sage-devel@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at http://groups.google.com/group/sage-devel
URLs: http://sage.scipy.org/sage/ and http://modular.math.washington.edu/sage/
-~--~~~~--~~--~--~---

[sage-devel] Re: potential cause for #557: cython missing dictionary deallocation

2007-10-17 Thread mabshoff 



On Oct 18, 6:16 am, Robert Bradshaw [EMAIL PROTECTED]
wrote:

Hi Robert,

 No cleanup function is ever generated, but atexit sounds perfect.

Good that you seem to agree with my analysis. I ran across it be sheer
accident while looking at some linker issue that roed asked me about.

 I'm the obvious person to implement this,

Pretty much 100% what I just claimed in IRC, but cwitty might look at
it during Bug Day 4.

 I'll see if I can find the time to do so soon.


Let's hope so.

 - Robert


Cheers,

Michael

 On Oct 17, 2007, at 9:04 PM, mabshoff wrote:



  Hello,

  for every module Cython generates runtime support code that is place
  at the end of the converted file. Each one of those has an
  __Pyx_Import function that gets called from

  PyMODINIT_FUNC init$MODULENAME(void)

  static PyObject *__Pyx_Import(PyObject *name, PyObject *from_list) {
  PyObject *__import__ = 0;
  PyObject *empty_list = 0;
  PyObject *module = 0;
  PyObject *global_dict = 0;
  PyObject *empty_dict = 0;
  PyObject *list;
  __import__ = PyObject_GetAttrString(__pyx_b, __import__);
  if (!__import__)
  goto bad;
  if (from_list)
  list = from_list;
  else {
  empty_list = PyList_New(0);
  if (!empty_list)
  goto bad;
  list = empty_list;
  }
  global_dict = PyModule_GetDict(__pyx_m);
  if (!global_dict)
  goto bad;
  empty_dict = PyDict_New();
  if (!empty_dict)
  goto bad;
  module = PyObject_CallFunction(__import__, ,
  name, global_dict, empty_dict, list);
  bad:
  Py_XDECREF(empty_list);
  Py_XDECREF(__import__);
  Py_XDECREF(empty_dict);
  return module;
  }

  In that function PyMODINIT_FUNC init$MODULENAME(void) we also call
  __Pyx_ImportType quite often:

  static PyTypeObject *__Pyx_ImportType(char *module_name, char
  *class_name,
  long size)
  {
  PyObject *py_module_name = 0;
  PyObject *py_class_name = 0;
  PyObject *py_name_list = 0;
  PyObject *py_module = 0;
  PyObject *result = 0;

  py_module_name = PyString_FromString(module_name);
  if (!py_module_name)
  goto bad;
  py_class_name = PyString_FromString(class_name);
  if (!py_class_name)
  goto bad;
  py_name_list = PyList_New(1);
  if (!py_name_list)
  goto bad;
  Py_INCREF(py_class_name);
  if (PyList_SetItem(py_name_list, 0, py_class_name)  0)
  goto bad;
  py_module = __Pyx_Import(py_module_name, py_name_list);
  if (!py_module)
  goto bad;
  result = PyObject_GetAttr(py_module, py_class_name);
  if (!result)
  goto bad;
  if (!PyType_Check(result)) {
  PyErr_Format(PyExc_TypeError,
  %s.%s is not a type object,
  module_name, class_name);
  goto bad;
  }
  if (((PyTypeObject *)result)-tp_basicsize != size) {
  PyErr_Format(PyExc_ValueError,
  %s.%s does not appear to be the correct type object,
  module_name, class_name);
  goto bad;
  }
  goto done;
  bad:
  Py_XDECREF(result);
  result = 0;
  done:
  Py_XDECREF(py_module_name);
  Py_XDECREF(py_class_name);
  Py_XDECREF(py_name_list);
  return (PyTypeObject *)result;
  }

  As one can see certain PyObject have refcounts greater 0 when
  initialization is successful. But I cannot find a cleanup function
  that would decrease the reference count upon exit. Consequently
  python's garbage collector never frees those dictionaries, which in
  most cases doesn't matter because we are exiting python anyway [some
  people claim that one shouldn't care about still reachable memory,
  because cleaning it up the nice way just slows down the termination of
  a process because the system will reap the heap and stack
  automatically]. But it pollutes the memcheck log, which is why I care
  about this.

  The usual way to call those functions for each module would be to
  register an atexit function, see

 http://docs.python.org/lib/module-atexit.html

  So if anybody want to write am autogenerated cleanup function for
  Cython and register it via atexit that person would be very welcome :)

  Cheers,
  Michael


--~--~-~--~~~---~--~~
To post to this group, send email to sage-devel@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at http://groups.google.com/group/sage-devel
URLs: http://sage.scipy.org/sage/ and http://modular.math.washington.edu/sage/
-~--~~~~--~~--~--~---