Re: RFC 130 (v4) Transaction-enabled variables for Perl6
/--- On Tue, Sep 05, 2000 at 10:57:30PM -0400, Chaim Frenkel wrote: | "JH" == Jarkko Hietaniemi [EMAIL PROTECTED] writes: | | Now, "all" that needs to be taken care of, is make sure that | the final | assignment from the localized and changed variables to their | outer-scoped counterparts happens in *one step*, i.e. no task | switching | while this is going on. | | JH Well, that can't be done. (I assume that by 'task switching' | you mean | JH here 'thread switching') Or, if it can, it *shouldn't* be | done. | JH Threads run on their own until they yield, they decide to | synchronize | JH and/or join, or they exit. Note the "they decide". \--- It is a very interesting problem, and this maybe out of my profession, but I think we need to try to solve this, because it will worth later! My suggestions: 1, If we use locks for transactions, then we need a deadlock detection, which is (in the simplest case): timeout. It can be implemented very easily. 2, Releasing the lock after the end of transaction has no problem (I think), because we can hold the lock on the variables until we finishes all the COMMITS and TIE_COMMITS, and then we can start releasing the locks. If a task-switching occures in this cycle, then nothing interresting happens: half of the variables are usable by others, half ot that is not. Anyone can try to use that, and may succeed. Next thread-switch to our thread will continue to release the locks until all the locks are released. I still think that we can implement this in a quite simple way IF we assume we already designed our perl interpreter to thread-safe. If you still have objections, share with us! dLux --
Re: RFC 136 (v2) Implementation of hash iterators
In message [EMAIL PROTECTED]
Dan Sugalski [EMAIL PROTECTED] wrote:
We punt. If the programmer wants consistent data in a multithreaded
program, he or she needs to lock the hash down. I'm all up for the
iterators looking at the hash as it exists--if the programmer wants
a snapshot of the hash keys or values, they can flatten them down to
a list or not mess with the hash in ways that'll geek the
iterator. I don't see any real reason that multiple iterators should
behave any differently than the single one we have now in that
respect.
It will certainly make life a lot easier if we're prepared to make
changes like that - the problem is that it will likely break existing
code.
I guess we can translate all uses of keys and values when doing
the p52p6 conversion - so that this:
foreach my $x (keys %y)
{
$y{$x+1} = 1;
}
becomes:
foreach my $x (@temp = keys %y)
{
$y{$x+1} = 1;
}
Of course we'd have to make sure the optimised realised that the
assignment caused the list to be flattened so it didn't try and
elide the 'useless' assignment ;-)
Tom
--
Tom Hughes ([EMAIL PROTECTED])
http://www.compton.nu
Re: RFC 130 (v4) Transaction-enabled variables for Perl6
"JH" == Jarkko Hietaniemi [EMAIL PROTECTED] writes:
I don't think we can do this immediately. Can you come up with the right
API and/or hooks that are needed so that it might be retrofited?
JH I think language magic helping to do the user level data locking is
JH a dead-in-the-water idea. For some very simply problems it may work,
JH but that support need not be implemented by internals.
I'm not advocating language magic. Just what level of support from the
core is needed to be able to do this properly. The inter-thread
communication and (the actual) locking would need to be in the core.
Other callbacks and notifications would be delivered by the core.
I think this is similar to what was done for perldb. Or what might
happen for n-D matrix/tensors/arrays/whatevers. A plugable replacement
module for doing transactions.
Though I don't think you would mind having
sub mycritical : lock { } # critical section here.
or
sub onlyone : method, lock { } # lock the object/class
chaim
--
Chaim FrenkelNonlinear Knowledge, Inc.
[EMAIL PROTECTED] +1-718-236-0183
Re: RFC 178 (v2) Lightweight Threads
"NI" == Nick Ing-Simmons [EMAIL PROTECTED] writes: NI The snag with attempting to automate such things is illustrated by : NI thread Athread B NI $a = $a + $b++; $b = $b + $a++; NI So we need to 'lock' both $a and $b both sides. NI So thread A will attempt to acquire locks on $a,$b (say) NI and (in this case by symetry but perhaps just by bad luck) thread B will NI go for locks on $b,$a - opposite order. They then both get 1st lock NI they wanted and stall waiting for the 2nd. We are in then in NI a "classic" deadly embrace. NI So the 'dragons' that Dan alludes to are those of intuiting the locks NI and the sequence of the locks to acquire, deadlock detection and backoff, ... Agreed. But for a single 'statement', it may be possible to gather all the objects needing a lock and then grabbing them in order (say by address). Also the thread doesn't need to make any changes until all the locks are available so a backoff algorithm may work. This would keep a _single_ statment 'consistent'. But wouldn't do anything for anything more complex. chaim -- Chaim FrenkelNonlinear Knowledge, Inc. [EMAIL PROTECTED] +1-718-236-0183
Re: RFC 178 (v2) Lightweight Threads
"DS" == Dan Sugalski [EMAIL PROTECTED] writes: DS Well, there'll be safe access to individual variables when perl needs to DS access them, but that's about it. DS Some things we can guarantee to be atomic. The auto increment/decrement DS operators can be reasonably guaranteed atomic, for example. But I don't DS think we should go further than "instantaneous access to shared data will DS see consistent internal data structures". This is going to be tricky. A list of atomic guarentees by perl will be needed. $a[++$b]; pop(@a); push(@a, @b); Will these? And given that users will be doing the locking. What do you see for handling deadlock detection and recovery/retry. chaim -- Chaim FrenkelNonlinear Knowledge, Inc. [EMAIL PROTECTED] +1-718-236-0183
Re: RFC 136 (v2) Implementation of hash iterators
"TH" == Tom Hughes [EMAIL PROTECTED] writes:
TH I guess we can translate all uses of keys and values when doing
TH the p52p6 conversion - so that this:
TH foreach my $x (keys %y)
TH {
TH $y{$x+1} = 1;
TH }
TH becomes:
TH foreach my $x (@temp = keys %y)
TH {
TH $y{$x+1} = 1;
TH }
TH Of course we'd have to make sure the optimised realised that the
TH assignment caused the list to be flattened so it didn't try and
TH elide the 'useless' assignment ;-)
Why not
lock(%y);
foreach my $x (keys %y) {
$y{$x+1} = 1;
}
unlock(%y);
Hmm, I just realized, perhaps we can just punt. Any p5 program that
doesn't use Threads can be left alone. Using p5 threads would
then need manual intervention.
chaim
--
Chaim FrenkelNonlinear Knowledge, Inc.
[EMAIL PROTECTED] +1-718-236-0183
Re: RFC 178 (v2) Lightweight Threads
But for a single 'statement', it may be possible to gather all the objects needing a lock and then grabbing them in order (say by address). I still don't buy that. In Perl even simple assignments and increments are not atomic which means that even 'single statements' would require locking and unlocking of a pile of data structures, leaving plenty of room to both inconsistencies and deadlocks. This would keep a _single_ statment 'consistent'. But wouldn't do anything for anything more complex. Why bother, then? Multithreaded programming is hard and for a given program the only person truly knowing how to keep the data consistent and threads not strangling each other is the programmer. Perl shouldn't try to be too helpful and get in the way. Just give user the bare minimum, the basic synchronization primitives, and plenty of advice. -- $jhi++; # http://www.iki.fi/jhi/ # There is this special biologist word we use for 'stable'. # It is 'dead'. -- Jack Cohen
Re: RFC 178 (v2) Lightweight Threads
"JH" == Jarkko Hietaniemi [EMAIL PROTECTED] writes: JH Multithreaded programming is hard and for a given program the only JH person truly knowing how to keep the data consistent and threads not JH strangling each other is the programmer. Perl shouldn't try to be too JH helpful and get in the way. Just give user the bare minimum, the JH basic synchronization primitives, and plenty of advice. my views exactly. most perl programs will not be multithreaded. we can support them with locks, a thread per interpreter paradigm and support other stuff like event and signal delivery and such. but no implied locks. the coder has to do some work and take responsibility. uri -- Uri Guttman - [EMAIL PROTECTED] -- http://www.sysarch.com SYStems ARCHitecture, Software Engineering, Perl, Internet, UNIX Consulting The Perl Books Page --- http://www.sysarch.com/cgi-bin/perl_books The Best Search Engine on the Net -- http://www.northernlight.com
Re: RFC 130 (v4) Transaction-enabled variables for Perl6
/--- On Wed, Sep 06, 2000 at 05:16:03PM -0500, Jarkko Hietaniemi
wrote:
| Okay, I have read it now. Now I'm going to make suggestions :-)
| (Note
| that so far I've been commenting only on the aspects of making
| things
| 'thread-safe', not on the RFC itself. 'Threadsafing' Perl is what
| I've deemed to be "too complex".) (Also note that I'm not a
| database
| guru, so please bear with me, and don't ask me to write the code
| :-)
Implementing threads must be done in a very clever way. It may be
put in a shared library (mutex handling code, locking, etc.), but I
think there are more clevery guys out there who are more competent
in this, and I think it is covered with some other RFCs...
| If I understood correctly, you basically want to extend the
| existing
| tie()-interface to be useful for versioned variables. (That they
| are
| 'transactioned' is just a choice of words.) To *safely* do data
| versioning one needs database-like COMMIT and ROLLBACK features.
Basically yes.
| While I do understand your basic request and I quite like the idea
| of
| having versioned data, I do not see the need for a new variable
| type,
| neither by introducing a new keyword, nor a by a new variable
| attribute. Also, from implementation viewpoint introducing all the
| required thread-safeness into the very core of Perl doesn't sound
| good
| to me. Yes, it would be a very neat thing to have. But making all
| non-transaction-requiring Perl programs to pay the {speed,size}
| price
| for the thread-safing locking machinery doesn't sound good. Then
| again, having such functionality as an external "loadable module"
| doesn't sound too bad. What would be valuable from Perl 6
| RFC/design
| viewpoint would be to take a careful look at the tie() interface
| and
| for Perl 6 design an extendable interface so that methods like
| COMMIT,
| ROLLBACK, et cetera can be easily implemented.
I also don't like the overhead, that's why I made the "simple" mode
default (look at the "use transaction" pragma again...). This means
NO overhead, no locking between threads: this can be used in
single-thread or multi-process environment. Other modes CAN switch
on locking functions, but this is not default! If you implement that
intelligently (separated .so for the thread handling), then it means
minimal overhead (some more callback call, and that's all).
Are you satisfied with this? I think this is a good compromise, and
still powerful :-)
\---
dLux
--
#!/bin/perl -sp0777iX+d*lMLa^*lN%0]dsXx++lMlN/dsM0j]dsj
$/=unpack('H*',$_);$_=`echo 16dio\U$k"SK$/SM$n\EsN0p[lN*1
lK[d2%Sa2/d0$^Ixp"|dc`;s/\W//g;$_=pack('H*',/((..)*)$/)
Re: RFC 136 (v2) Implementation of hash iterators
In message [EMAIL PROTECTED]
Chaim Frenkel [EMAIL PROTECTED] wrote:
Why not
lock(%y);
foreach my $x (keys %y) {
$y{$x+1} = 1;
}
unlock(%y);
Hmm, I just realized, perhaps we can just punt. Any p5 program that
doesn't use Threads can be left alone. Using p5 threads would
then need manual intervention.
I wasn't just talking about the threaded case though - the point
which I was making was that of what happens if a single threaded
program alters a hash in the middle of iterating it.
Currently keys and values are flattened when they are seen so any
change to the hash is not reflected in the resulting list. If we
are iterating instead of flattening then we need to address that
somehow.
Tom
--
Tom Hughes ([EMAIL PROTECTED])
http://www.compton.nu/
...It pays to be obvious, especially if you have a reputation for subtlety.
Re: RFC 130 (v4) Transaction-enabled variables for Perl6
Are you satisfied with this? I think this is a good compromise, and still powerful :-) Me satisfied? Well, kind of. I see the need, I just disagree with the proposed interface and extent. I will not comment on the subject much more because I sense that soon we'll be hip deep in database technology and as I said, I do not know that subject well. What I know is that trying to build thread-safety and atomicity to the Perl core is decidedly non-trivial; all that should be the applications' concern. -- $jhi++; # http://www.iki.fi/jhi/ # There is this special biologist word we use for 'stable'. # It is 'dead'. -- Jack Cohen
Re: RFC 178 (v2) Lightweight Threads
DS Some things we can guarantee to be atomic. This is going to be tricky. A list of atomic guarentees by perl will be needed. From RFC 178 ...we have to decide which operations are [atomic]. As a starting point, we can take all the operators documented in Cperlop.pod and all the functions documented in Cperlfunc.pod as [atomic]. - SWM
Re: RFC 178 (v2) Lightweight Threads
what if i do $i++ and overflow into the float (or bigint) domain? that
is enough work that you would need to have a lock around the ++. so then
all ++ would have implied locks and their baggage. i say no atomic ops
in perl.
From RFC 178
[Atomic] operations typically lock their operands to avoid race
conditions
Perl source C Implementation
$a = $b lock(a.mutex);
lock(b.mutex);
free(a.pData);
a.length = b.length;
a.pData = malloc(a.length);
memcpy(a.pData, b.pData, a.length);
unlock(a.mutex);
unlock(b.mutex);
leave the locking to the coder and keep perl clean.
If we don't provide this level of locking internally, then
async { $a = $b }
is liable to crash the interpreter.
- SWM
Re: RFC 178 (v2) Lightweight Threads
"UG" == Uri Guttman [EMAIL PROTECTED] writes: UG i don't see how you can do atomic ops easily. assuming interpreter UG threads as the model, an interpreter could run in the middle of another UG and corrupt it. most perl ops do too much work for any easy way to make UG them atomic without explicit locks/mutexes. leave the locking to the UG coder and keep perl clean. in fact the whole concept of transactions in UG perl makes me queasy. leave that to the RDBMS and their ilk. If this is true, then give up on threads. Perl will have to do atomic operations, if for no other reason than to keep from core dumping and maintaining sane states. If going from an int to a bigint is not atomic, woe to anyone using threads. If it is atomic, then the ++ has to be atomic, since the actual operation isn't complete until it finishes. Think ++$a(before int, after ++ value is bigint) Some series of points (I can't remember what they are called in C) where operations are consider to have completed will have to be defined, between these points operations will have to be atomic. chaim -- Chaim FrenkelNonlinear Knowledge, Inc. [EMAIL PROTECTED] +1-718-236-0183
Re: RFC 136 (v2) Implementation of hash iterators
"TH" == Tom Hughes [EMAIL PROTECTED] writes:
TH I wasn't just talking about the threaded case though - the point
TH which I was making was that of what happens if a single threaded
TH program alters a hash in the middle of iterating it.
TH Currently keys and values are flattened when they are seen so any
TH change to the hash is not reflected in the resulting list. If we
TH are iterating instead of flattening then we need to address that
TH somehow.
I'd rather not have the expansion performed. Some other mechanism, either
under the covers or perhaps even specified in the language.
The only real issue is if the change effects the iterator order. Changes
to values should be allowed without out any adverse effects.
Changes to the iterator order (inserted/deleted keys, push/pop) can
be either, "don't do that", or queued up until the iterator is done or
past the effected point.
I'm partial to the don't do that approach. It can easily be handled
at the user level.
delete @hash{@delete_these};
@hash{keys %add_these} = values %add_these;
chaim
(Hmm, push(%hash, %add_these))
--
Chaim FrenkelNonlinear Knowledge, Inc.
[EMAIL PROTECTED] +1-718-236-0183
