Dennis Lee Bieber wrote:
declaimed the following
So maybe we need to redesign the hardware.
Remember the iAPX-432?
http://en.wikipedia.org/wiki/Intel_iAPX_432#Garbage_collection
Not quite what I had in mind. That sounds like a conventional
GC algorithm that happens to be implemente
On 9/8/2010 6:20 PM, Paul Rubin wrote:
What tricks on tricks? Even the fanciest GC's are orders of magnitude
less complicated than any serious database, optimizing compiler, OS
kernel, file system, etc. Real-world software is complicated. Get used
to that fact, and look for ways to manage the
On Thu, 09 Sep 2010 12:41:20 +1200, Lawrence D'Oliveiro wrote:
>> Part of the problem is C itself.
>
> And yet, what are these complicated garbage collectors, that you intend
> relying on to work correctly with all their layers of tricks upon
> tricks, written in? C.
Not necessarily.
Pascal, de
Lawrence D'Oliveiro writes:
> In that case, I can similarly discount your attempts to fix up issues with
> garbage collectors after they’re pointed out, can I not?
Adapting GC algorithms to improve their performance or better use the
capabilities of new hardware is much different than saying the
In message <7xeid9gtuq@ruckus.brouhaha.com>, Paul Rubin wrote:
> Lawrence D'Oliveiro writes:
>
>> That reinforces my point, about how easy it was to check the correctness
>> of the code. In this case one simple fix, like this ...
>> would render the code watertight. See how easy it is?
>
> W
Lawrence D'Oliveiro wrote:
But alone of all of these, garbage collection still remains just as costly
to implement as ever. That should tell you something about how poorly it
matches the characteristics of modern computing hardware.
So maybe we need to redesign the hardware.
Perhaps referenc
Paul Rubin wrote:
Now extrapolate that error rate from 30 lines to a program the size of
Firefox (something like 5 MLOC), and you should see how fraught with
danger that style of programming is.
But you don't write 5 MLOC of code using that programming style.
You use it to write a small core s
Lawrence D'Oliveiro writes:
> But you’ll notice that Free Software comes with no such
> restrictions. In fact, it is contrary to commonly-accepted Free
> Software guidelines to impose any sort of restrictions on areas of use.
Free software comes with an explicit disclaimer of liability (you get
w
John Nagle writes:
> I've argued for an approach in which only synchronized or immutable
> objects can be shared between threads. Then, only synchronized objects
> have refcounts. See
> "http://www.animats.com/papers/languages/pythonconcurrency.html";
I'm going to have to read this carefull
On 9/4/2010 11:51 PM, Paul Rubin wrote:
John Nagle writes:
Unoptimized reference counting, which is what CPython does, isn't
all that great either. The four big bottlenecks in Python are boxed
numbers, attribute lookups, reference count updates, and the GIL.
The performance hit of havin
John Nagle writes:
> Unoptimized reference counting, which is what CPython does, isn't
> all that great either. The four big bottlenecks in Python are boxed
> numbers, attribute lookups, reference count updates, and the GIL.
The performance hit of having to lock the refcounts before update h
On 9/4/2010 6:44 PM, Lawrence D'Oliveiro wrote:
In message<4c82b097$0$1661$742ec...@news.sonic.net>, John Nagle wrote:
Personally, I'd like to have reference counting only, an enforced
prohibition on loops (backpointers must be weak pointers), RAII,
and reliably ordered finalization.
Is
Lawrence D'Oliveiro writes:
>> A minimal naive implementation indeed doubles the memory requirements,
>> but from a Python perspective where every integer takes something like
>> 24 bytes already, even that doesn't seem so terrible.
>
> Doubling 24 is less terrible than doubling 4 or 8?? You’re ki
In message <7x7hj2kyd6@ruckus.brouhaha.com>, Paul Rubin wrote:
> Lawrence D'Oliveiro writes:
>
>> In message <7xmxs2uez1@ruckus.brouhaha.com>, Paul Rubin wrote:
>>
>>> GC's for large systems generally don't free (or even examine) individual
>>> garbage objects. They copy the live objects
In message <4c82b097$0$1661$742ec...@news.sonic.net>, John Nagle wrote:
> Personally, I'd like to have reference counting only, an enforced
> prohibition on loops (backpointers must be weak pointers), RAII,
> and reliably ordered finalization.
Is there a cheap way of checking at runtime for c
In message , MRAB
wrote:
> Lawrence D'Oliveiro writes:
>>
>> Wonder why Sun’s licence explicitly forbade its use in danger-critical
>> areas like nuclear power plants and the like, then?
>
> I thought it was just that if it wasn't explicitly forbidden then
> someone might try to use it and then
On 8/28/2010 5:42 AM, Aahz wrote:
In article<4c78572c$0$28655$c3e8...@news.astraweb.com>,
Steven D'Aprano wrote:
On Fri, 27 Aug 2010 09:16:52 -0700, Aahz wrote:
In article, MRAB
wrote:
An object will be available for garbage collection when nothing refers
to it either directly or indirectl
Lawrence D'Oliveiro writes:
> That reinforces my point, about how easy it was to check the correctness of
> the code. In this case one simple fix, like this ...
> would render the code watertight. See how easy it is?
Well, no, it's irrelevant how easy it is to fix the issue after it's
pointed ou
[gc]
In article <7x7hj2kyd6@ruckus.brouhaha.com>,
Paul Rubin wrote:
>
>A minimal naive implementation indeed doubles the memory requirements,
>but from a Python perspective where every integer takes something like
>24 bytes already, even that doesn't seem so terrible.
Many people still us
Dennis Lee Bieber writes:
> Not to mention having to ensure that one finds ALL the references to
> the object so that they can be updated to the new address! Somehow I
> don't see a C compiler being smart enough to find intermediary pointer
We're not talking about C compilers, which can cas
Lawrence D'Oliveiro writes:
>> GC's for large systems generally don't free (or even examine) individual
>> garbage objects. They copy the live objects to a new contiguous heap
>> without ever touching the garbage, and then they release the old heap.
>
> And suddenly you’ve doubled the memory requ
On 04/09/2010 03:21, Paul Rubin wrote:
Lawrence D'Oliveiro writes:
Java has considerably greater reputation for reliability than C or C++.
Wonder why Sun’s licence explicitly forbade its use in danger-critical
areas like nuclear power plants and the like, then?
Probably because Sun lawyers
Lawrence D'Oliveiro writes:
>> Java has considerably greater reputation for reliability than C or C++.
>
> Wonder why Sun’s licence explicitly forbade its use in danger-critical
> areas like nuclear power plants and the like, then?
Probably because Sun lawyers demanded it. Is there a Sun C or C+
In message <7xr5heufhb@ruckus.brouhaha.com>, Paul Rubin wrote:
> Java has considerably greater reputation for reliability than C or C++.
Wonder why Sun’s licence explicitly forbade its use in danger-critical areas
like nuclear power plants and the like, then?
> Ada is a different story, but
In message <7xmxs2uez1@ruckus.brouhaha.com>, Paul Rubin wrote:
> Lawrence D'Oliveiro writes:
>
>> Whereas garbage collection will happen at some indeterminate time long
>> after the last access to the object, when it very likely will no longer
>> be in the cache, and have to be brought back i
In message <7xiq2que93@ruckus.brouhaha.com>, Paul Rubin wrote:
> Lawrence D'Oliveiro writes:
>>
>>> Refcounting is susceptable to the same pauses for reasons already
>>> discussed.
>>
>> Doesn’t seem to happen in the real world, though.
>
> def f(n):
> from time import time
>
In message <7xvd6sv0n4@ruckus.brouhaha.com>, Paul Rubin wrote:
> Lawrence D'Oliveiro writes:
>>> AddrObj = PyTuple_GetItem(TheBufferInfo, 0);
>>> LenObj = PyTuple_GetItem(TheBufferInfo, 1);
>>>
>>> the first PyTuple_GetItem succeeds and the second one fails.
>>
>> Admittedl
Dennis Lee Bieber writes:
>> GC's for large systems ... copy the live objects to a new contiguous heap
> That sounds suspiciously like the original Macintosh OS, with its
> "handles"... IE, double-indirection.
Nah, a double indirection on every access would be a terrible
performance hit.
On 8/30/2010 12:22 AM, Paul Rubin wrote:
I guess that is how the so-called smart pointers in the Boost C++
template library work. I haven't used them so I don't have personal
experience with how convenient or reliable they are, or what kinds of
constraints they imposed on programming style. I'v
Lawrence D'Oliveiro writes:
>> Refcounting is susceptable to the same pauses for reasons already
>> discussed.
>
> Doesn’t seem to happen in the real world, though.
def f(n):
from time import time
a = [1] * n
t0 = time()
del a
t1 = time()
return
Lawrence D'Oliveiro writes:
> Whereas garbage collection will happen at some indeterminate time long after
> the last access to the object, when it very likely will no longer be in the
> cache, and have to be brought back in just to be freed,
GC's for large systems generally don't free (or eve
Lawrence D'Oliveiro writes:
> And yet Java code, for example, doesn’t have a reputation for greater
> reliability compared to, say code written in Ada or C++, or even C. What is
> the Java runtime written in? C. Why not use Java, if there is no speed
> penalty in doing so?
The Java runtime (su
In message <7xmxs4uzjl@ruckus.brouhaha.com>, Paul Rubin wrote:
> Gregory Ewing writes:
>
>> I'd be disappointed if CPython ditched refcounting and
>> then became unsuitable for real-time games as a result.
>
> Refcounting is susceptable to the same pauses for reasons already
> discussed.
Do
In message <7x39tz42fd@ruckus.brouhaha.com>, Paul Rubin wrote:
> Dennis Lee Bieber writes:
>
>> Heap marking, OTOH, tends to run at indeterminate times, which could
>> have an impact if one needs predictable response timings
>
> Reference counting has the same problem. If you drop the last
In message <7xtymbzixt@ruckus.brouhaha.com>, Paul Rubin wrote:
> It's pretty well established by now that GC doesn't have any significant
> speed penalty compared with manual allocation. It does consume more
> memory, which is acceptable a lot of the time. It certainly leads to
> more reliab
Lawrence D'Oliveiro writes:
> Meanwhile, real-world programmers get on to writing real-world code that is
> productive and efficient on real-world systems.
It's pretty well established by now that GC doesn't have any significant
speed penalty compared with manual allocation. It does consume mor
In message <7xr5hguzzi@ruckus.brouhaha.com>, Paul Rubin wrote:
> JHC (experimental Haskell compiler) recently started using a mixture of
> gc and region inference. It will be interesting to see how that works
> out.
That’s what people have been saying about garbage collection for about half
In article <4c7b279d$0$28650$c3e8...@news.astraweb.com>,
Steven D'Aprano wrote:
>On Sun, 29 Aug 2010 17:52:38 -0700, Paul Rubin wrote:
>>Attribution lost:
>>>
>>> That's a problem with the CPython API, not reference counting. The
>>> problem is that the CPython API is written at too low a level,
Gregory Ewing writes:
>> These days I think the GC pause issue is overrated except for real-time
>> control applications.
>
> Also for games, which are a fairly common application
> these days. Even a few milliseconds can be too long when
> you're trying to achieve smooth animation.
The usual hac
Lawrence D'Oliveiro writes:
>> If it hasn't happened to you yet, you're either burning a bunch of effort
>> that programmers of more automatic systems can put to more productive
>> uses ...
>
> What makes you say that? Avoiding bugs is not a “productive use”?
Avoiding any particular bug through c
Lawrence D'Oliveiro writes:
>> AddrObj = PyTuple_GetItem(TheBufferInfo, 0);
>> LenObj = PyTuple_GetItem(TheBufferInfo, 1);
>>
>> the first PyTuple_GetItem succeeds and the second one fails.
>
> Admittedly, I did take a shortcut here: array.buffer_info returns a tuple of
> two i
In message <7x39twpuxi@ruckus.brouhaha.com>, Paul Rubin wrote:
> Lawrence D'Oliveiro writes:
>
>>> the CPython API means endlessly finding and fixing refcount bugs that
>>> lead to either crashes/security failures, or memory leaks.
>>
>> I don’t see why that should be so. It seems a very simp
In message <7xr5hg3a7s@ruckus.brouhaha.com>, Paul Rubin wrote:
> Actually that code looks suspicious. Suppose in
>
> AddrObj = PyTuple_GetItem(TheBufferInfo, 0);
> LenObj = PyTuple_GetItem(TheBufferInfo, 1);
>
> the first PyTuple_GetItem succeeds and the second one fails.
Paul Rubin wrote:
These days I think the GC pause issue is overrated except for real-time
control applications.
Also for games, which are a fairly common application
these days. Even a few milliseconds can be too long when
you're trying to achieve smooth animation.
I'd be disappointed if CPyt
On Mon, 30 Aug 2010 00:22:17 -0700, Paul Rubin wrote:
> I don't think a C compiler could really manage automatic decrementing
> while still being C. Think especially of the common style of exception
> handling in C using longjmp.
You might very well be right. But that's the problem with C -- it'
Steven D'Aprano writes:
> I'm not saying that ref counting systems can avoid incrementing and
> decrementing the ref counts. That would be silly. But I'm saying that it
> is an accident of implementation that writing C extensions requires you
> to manually manage the counts yourself. That's a s
Lawrence D'Oliveiro writes:
> static void GetBufferInfo
> ( ...
> do /*once*/
> {
> TheBufferInfo = PyObject_CallMethod(FromArray, "buffer_info", "");
> if (TheBufferInfo == 0)
> break;
> AddrObj = PyTuple_GetItem(TheBufferInfo, 0);
> LenObj
Lawrence D'Oliveiro writes:
>> the CPython API means endlessly finding and fixing refcount bugs that lead
>> to either crashes/security failures, or memory leaks.
>
> I don’t see why that should be so. It seems a very simple discipline to
> follow: initialize, allocate, free. Here’s an example sn
In message <7x4oeftuk4@ruckus.brouhaha.com>, Paul Rubin wrote:
> I'd say [reference-counting is] not real gc because 1) it's unsound
> (misses reference cycles), and 2) it requires constant attention from the
> mutator to incr and decr the reference counts. So developing modules for
> the CPy
On Sun, 29 Aug 2010 17:52:38 -0700, Paul Rubin wrote:
>> That's a problem with the CPython API, not reference counting. The
>> problem is that the CPython API is written at too low a level, beneath
>> that at which the garbage collector exists, so naturally you have to
>> manually manage memory.
>
Steven D'Aprano writes:
> I could be wrong, but how can they not be subject to the same performance
> issue? If you have twenty thousand components that all have to be freed,
> they all have to be freed whether you do it when the last reference is
> cleared, or six seconds later when the gc doe
Steven D'Aprano writes:
> You can add cycle detection to a reference count gc, at the cost of more
> complexity.
But then it's not purely a refcount gc. ;)
> If you read the Wikipedia article I linked to, tracing algorithms can
> also be unsound: [describes "conservative" gc]
Yeah, whether t
Hans Mulder writes:
> Parallelizable garbage collectors have performance issues, but they're
> not the same issues as mark&sweep collectors have. Parallelizable GCs
> break up their work in a zillion little pieces and allow the VM to do
> some real work after each piece. They won't free your twe
Steven D'Aprano wrote:
On Sat, 28 Aug 2010 00:33:10 -0700, Paul Rubin wrote:
If you drop the last reference
to a complex structure, it could take quite a long time to free all the
components. By contrast there are provably real-time tracing gc
schemes, including some parallelizeable ones.
I
In article <4c78e7a5$0$28655$c3e8...@news.astraweb.com>,
Steven D'Aprano wrote:
>
>On the other hand, the reason that CPython still has reference counting
>is that the alternatives tried so far are unacceptably for non-threaded
>code.
No, it's *a* reason, the other main reason being that refco
In article <4c78572c$0$28655$c3e8...@news.astraweb.com>,
Steven D'Aprano wrote:
>On Fri, 27 Aug 2010 09:16:52 -0700, Aahz wrote:
>> In article , MRAB
>> wrote:
>>>
>>>An object will be available for garbage collection when nothing refers
>>>to it either directly or indirectly. If it's unreferen
On Fri, 27 Aug 2010 18:06:19 -0700, Paul Rubin wrote:
> Steven D'Aprano writes:
>> I've repeatedly asked, both here and elsewhere, why reference counting
>> isn't "real" garbage collection. Nobody has been able to give me a
>> satisfactory answer. As far as I can tell, it's a bit of
>> pretentiou
On Sat, 28 Aug 2010 00:33:10 -0700, Paul Rubin wrote:
> Dennis Lee Bieber writes:
>> The nice thing about it [reference counting] is that it is sort
>> of deterministic -- one can examine code and determine that an object
>> is collected at some point in the execution...
>> Heap marking
Dennis Lee Bieber writes:
> The nice thing about it [reference counting] is that it is sort
> of deterministic -- one can examine code and determine that an object
> is collected at some point in the execution...
> Heap marking, OTOH, tends to run at indeterminate times, which could
>
Steven D'Aprano writes:
> I've repeatedly asked, both here and elsewhere, why reference counting
> isn't "real" garbage collection. Nobody has been able to give me a
> satisfactory answer. As far as I can tell, it's a bit of pretentiousness
> with no basis in objective fact.
Well, it's a bit o
On Fri, 27 Aug 2010 09:16:52 -0700, Aahz wrote:
> In article , MRAB
> wrote:
>>
>>An object will be available for garbage collection when nothing refers
>>to it either directly or indirectly. If it's unreferenced then it will
>>go away.
>
> This isn't actually garbage collection as most people
On 8/11/2010 1:26 PM, EW wrote:
On Aug 11, 2:52 pm, Paul Rubin wrote:
EW writes:
Well I cared because I thought garbage collection would only happen
when the script ended - the entire script. Since I plan on running
this as a service it'll run for months at a time without ending. So I
thoug
In article ,
MRAB wrote:
>
>An object will be available for garbage collection when nothing refers
>to it either directly or indirectly. If it's unreferenced then it will
>go away.
This isn't actually garbage collection as most people think of it.
Refcounting semantics mean that objects get reap
EW writes:
> Well I can't really explain it but 1 Queue per task for what I'm
> designing just doesn't feel right to me. It feels like it will lack
> future flexibility.
That makes no sense at all. Multiple readers and writers per queue are
the way Python queues are designed to work. The norma
On Aug 11, 2:52 pm, Paul Rubin wrote:
> EW writes:
> > Well I cared because I thought garbage collection would only happen
> > when the script ended - the entire script. Since I plan on running
> > this as a service it'll run for months at a time without ending. So I
> > thought I was going to
Paul Rubin wrote:
EW writes:
Well I cared because I thought garbage collection would only happen
when the script ended - the entire script. Since I plan on running
this as a service it'll run for months at a time without ending. So I
thought I was going to have heaps of Queues hanging out in
EW writes:
> Well I cared because I thought garbage collection would only happen
> when the script ended - the entire script. Since I plan on running
> this as a service it'll run for months at a time without ending. So I
> thought I was going to have heaps of Queues hanging out in memory,
> unr
On Aug 11, 2:16 pm, Paul Rubin wrote:
> EW writes:
> > I thought about doing it that way and I could do it that way but it
> > still seems like there should be a way to clean up Queues on my own.
> > If I did it this way then I guess I'd be relying on garbage collection
> > when the script ended
EW writes:
> I thought about doing it that way and I could do it that way but it
> still seems like there should be a way to clean up Queues on my own.
> If I did it this way then I guess I'd be relying on garbage collection
> when the script ended to clean up the Queues for me.
Oh, I see. As lo
On Aug 11, 1:55 pm, MRAB wrote:
> EW wrote:
>
> [snip]
>
>
>
> > So here the P2 thread has ended and gone away but I still have his
> > Queue lingering.
>
> > So on a thread I can use is_alive() to check status and use join() to
> > clean up but I don't see any analogous functionality for Queues.
EW wrote:
[snip]
So here the P2 thread has ended and gone away but I still have his
Queue lingering.
So on a thread I can use is_alive() to check status and use join() to
clean up but I don't see any analogous functionality for Queues. How
do I kill them? I thought about putting a suicide mess
On Aug 11, 1:18 pm, Paul Rubin wrote:
> EW writes:
> > I also might have different consumer threads do
> > different tasks (for example one might write to a log and one might
> > write to SQL) so that again means I can't plan for a set ratio of
> > consumers to producers So it's unknown.
>
>
EW writes:
> I also might have different consumer threads do
> different tasks (for example one might write to a log and one might
> write to SQL) so that again means I can't plan for a set ratio of
> consumers to producers So it's unknown.
>
> So this means that instead of having 1 Queue tha
On Aug 11, 12:55 pm, EW wrote:
> Hi
>
> I'm writing a multithreaded app that relies on Queues to move data
> between the threads. I'm trying to write my objects in a general way
> so that I can reuse them in the future so I need to write them in such
> a way that I don't know how many producer an
Hi
I'm writing a multithreaded app that relies on Queues to move data
between the threads. I'm trying to write my objects in a general way
so that I can reuse them in the future so I need to write them in such
a way that I don't know how many producer and how many consumer
threads I might need.
75 matches
Mail list logo
