Re: I could not resist
On Mon, Dec 29, 2003 at 04:44:33PM -0500, Uri Guttman wrote: HJ == Harry Jackson [EMAIL PROTECTED] writes: HJ I was searching on google for HJ core.html parrot HJ http://www.gurney.co.uk/parrots/dandan.html and if dan keeps leading parrot he will soon pluck out his own hairs (there is a name for this human neurotic disease). :) That would be trichotillomania http://psychcentral.com/library/hair_pulling.htm -- Dave
Re: Threads and Events (was Re: threads and shared interpreter data structures)
On Tuesday, December 23, 2003, at 08:40 , Rod Adams wrote: - Most treaded code can be converted to an event loop (+async I/O) without issue. Esp if we have co-routines. - For the other stuff, we'd still have Type 2 threads, which gets the job done. (Just got back from vacation and was reviewing this aging thread...) Not to throw a damper on the events party or anything, but an event-based system with asynchronous I/O isn't capable of saturating an SMP box's processors. This is one of the major reasons for using threads in web servers. It's also a significant reason for using threads in desktop applications. Yes, N interpreters for N processors will get you there, but at the cost of times-N static memory consumption (for JITted code, type registries, vtables, etc.: interpreter overhead), and at the cost of fine-grained, lightweight inter-thread communication between the segregated threads. Further, threading implemented as context switches at block time amounts to a cooperative multithreading environment. Yes, it may provide near-optimal throughput. Despite that, it also has some very bad indeed worst-case latency characteristics. If a worker thread fails to block, the thread which started it will never (or rarely) run and the program will become unresponsive. This makes such a threading model unsuitable for use as in a web application host. One misbehaving HTTP request handler mustn't block other requests. A worker thread mustn't block the UI thread. Sidenote: Shades of System 7: CPU benchmarks on the old Mac OS do run several percentage points faster than on preemptive systems. The preemptive model is clearly superior in the general case, though; its perceived performance under load is by far superior. Also worth noting: parrot will already be paying the preemptive performance penalty on any modern OS. I can only hail core events and asynchronous I/O as great advances in parrot. But they are not a general replacement for preemptive multithreading. Of course, TMTOWTDI and YMMV, but parrot should support both models well, and the above line of thought isn't doing threading justice in my opinion. Gordon Henriksen [EMAIL PROTECTED]
Strangeness with '.sub' in macros
Hi, I have been playing around with 'libpcre' for Parrot m4. For some reason I couldn't compile two regular expressions in the same PIR script. I created a sample C program and that worked like it should. It looks like the error has nothing to do with 'libpcre'. So I boiled down my code to a small test script. When a macro contains a '.sub' call, and that macro is used twice, then I get a 'memory error'. [EMAIL PROTECTED]:~/devel/Parrot/parrot/languages/m4 ../../parrot t/regex/macro_used_twice.imc Speicherzugriffsfehler Could sombody test the attached script on another machine? I'm working here on a Linux laptop: [EMAIL PROTECTED]:~/devel/Parrot/parrot uname -a Linux linux 2.4.20-4GB-athlon #1 Mon Mar 17 17:56:47 UTC 2003 i686 unknown unknown GNU/Linux How can I tell 'parrot' to dump a core file? CU, Bernhard macro_used_twice.imc Description: Binary data
Re: Threads
Leopold Toetsch wrote:
Dan Sugalski wrote:
Leopold Toetsch wrote:
Again, not only strings but all kind of containers using managed
resources suffer from that problem.
All that seems to imply, that we need two locks for shared access: one
for protecting DOD/GC and a second for the PMC. Any container
(including
PerlStrings) would have to aquire the first (interpreter-global) lock,
while a PerlNum only needs the second per PMC mutex. Ugly.
Yeah, it all potentially gets nasty.
[...]
[3] They can't be suspended inmidst of an arbitrary operation (and
pthread doesn't even provide suspending) so they would be sent a
suspend event so that they are in a safe state in the run-loop, or
better in the event-handler opcode, called from there.
(Just got back from vacation and was reviewing this aging thread...)
Yes, yes! Stopping other threads when a rare condition occurs is a great
way to avoid penalizing a threaded program's common case with excessive
synchronization overhead. There are just too many things that can go
wrong otherwise; fine-grained locking is the alternative, and programs
will very, very rapidly find themselves spending 40% of their cycles
acquiring mutexes.
This is definitely the way to go to achieve blistering performance while
retaining dynamic capabilities. The work is already done to ensure that
while true { ; } will receive events in a timely fashion. Broadcasting
parrot events should be an excellent means of implementing inter-thread
synchronization.
Gordon Henriksen
[EMAIL PROTECTED]
nci
Some questions
Please note:
I have been unable to test these patches with make test due to the
problems I mentioned in an earler post. I have managed to get the
postgres lib working again and I am hoping this is the only thing I have
affected with these patches although if someone would like to apply and
try them I would appreciate it.
I am getting errors when trying to load the libpq library. I added a few
lines to the /parrot/build_tools/build_nativecall.pl script to see what
was going on when I got the core dump. I have attached a patch to add
these. There may be some problems with it. I was also thinking that it
would be handy to have the facility to print out variables in the
message, what do people think?
I noticed that there are several signatures missing from call_list.txt
so I added these as well. Patch attached. There where quite a few which
probably means I am barking up the wrong tree.
On side note: I was reading the docs on strings and noticed that we
should always be using STRING but I have noticed a few references to
String while rummaging around. From what I can gather they are one and
the same and String is redundant. Is this the case?
On another side note I noticed a reference to TWEAKS, purely by
chance. For those not in the know
From the summarizer:
TWEAKS
(Takers Wanted -- Effort And Knowledge Sought).
http://xrl.us/o2g
From what I can gather this is a list of tasks compiled by leo that
need to be carried out and help is required. I think it would be a good
idea to have a well kept todo list particularly for tasks that are
fairly easy to complete. I am sure there are people on this list like
myself who do not mind doing the smaller stuff (including documention
;-) in order to get to know the guts. Although some of the stuff leo is
asking for is not that small.
On another side note I noticed the string.pod documentations asks at the
bottom
Should the following functions be mentioned? string_append,
string_from_cstring, string_from_int, string_from_num, string_index,
string_replace, string_set, string_str_index, string_to_cstring,
string_to_int, string_to_num, string_transcode.
Yes they should, it is things like this that would be good to
put on the TWEAKS along with where or how to start[1]. The
strings doc has been one of the handiest I have found to date.
Harry (on a side note) Jackson
1. If anyone wants to take it on and is unsure where to start. Use the
folowing file.
/parrot/include/parrot/string_funcs.h
file to see what each of the functions do and try some of them and
document what you do in strings.pod and submit your results. I will
probably make some entries on it myself if no one else does.
--- build_nativecall.pl Tue Dec 30 15:54:20 2003
+++ build_nativecall.pl.new Tue Dec 30 15:53:09 2003
@@ -143,7 +143,6 @@
* Notes:
* References:
*/
-
#include parrot/parrot.h
#if defined(HAS_JIT) defined(I386) defined(threaded_NCI_is_ok)
@@ -207,8 +206,13 @@
to a function that can call it. */
void *
build_call_func(struct Parrot_Interp *interpreter, PMC *pmc_nci,
-String *signature)
+STRING *signature)
{
+
+STRING *ns;
+STRING *message;
+char *c;
+
#if defined(CAN_BUILD_CALL_FRAMES)
/* This would be a good place to put the code that builds the
frames. Undoubtedly painfully platform-dependent */
@@ -221,7 +225,20 @@
UNUSED(pmc_nci);
if (0 == string_length(signature)) return F2DPTR(pcf_v_v);
$icky_global_bit
-PANIC(Unknown signature type);
+
+
+/*
+ These three lines have been added to aid debugging. I want to be able to
+ see which signature has an unknown type. I am sure someone can come up
+ with a neater way to do this.
+ */
+ns = string_make(interpreter, is an unknown signature type, 30, NULL, 0, NULL);
+message = string_concat(interpreter, signature, ns, 0);
+
+// I think there may be memory issues with this but if we get to here we are
+// aborting.
+c = string_to_cstring(interpreter, message);
+PANIC(c);
return NULL;
#endif
}
--- call_list.txt Tue Dec 30 15:54:20 2003
+++ call_list.txt.new Tue Dec 30 15:53:34 2003
@@ -181,6 +181,18 @@
# Oddball ones for postgres
p ptiLTLLi
+p t
+p ttt
+c p
+t p
+v pp
+t tl4
+t t4
+p pt
+p pi33ipi
+t pii
+p pi
+i pitl
# The following are used by library/pcre.imc
p tiB3P
Re: Strangeness with '.sub' in macros
Bernhard Schmalhofer wrote:
Hi,
Could sombody test the attached script on another machine?
I'm working here on a Linux laptop:
I am geting a seg fault. Its fine when the second call is commented out.
I am not sure if any of the following is any use to you.
The follwoing PASM was part generated from your IMC, I added the bits
between ###. This calls the sub twice I am not sure if it is any use
to you. Please see some output from GDB at the end.
_main:
set P16, P1
newsub P16, 45, _dummy
@pcc_sub_call_2:
set I5, 1
set P0, P16
set I0, 1
set I1, 0
set I2, 0
set I3, 0
savetop
invokecc
restoretop
###
set I5, 2
set I0, 1
set I1, 0
set I2, 0
set I3, 0
savetop
invokecc
restoretop
###
end
_dummy:
print dummy was called with:
print I5
print \n
@pcc_sub_ret_25:
set I0, 1
set I1, 0
set I2, 0
set I3, 0
set I4, 0
invoke P1
[EMAIL PROTECTED] pbin]$ gdb parrot
GNU gdb Red Hat Linux (5.1-1)
Copyright 2001 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type show copying to see the conditions.
There is absolutely no warranty for GDB. Type show warranty for details.
This GDB was configured as i386-redhat-linux...
(gdb) b runops_jit
Breakpoint 1 at 0x8075236: file src/interpreter.c, line 422.
(gdb) run -j -d test.pbc
Starting program: /home/parrot/bin/parrot -j -d test.pbc
[New Thread 1024 (LWP 17090)]
[New Thread 2049 (LWP 17091)]
[New Thread 1026 (LWP 17092)]
Address of base segment is (nil)
Address of base segment is (nil)
*** Parrot VM: JIT core ***
*** Parrot VM: Setting up ARGV array in P5. Current argc: 1 ***
0: test.pbc
[Switching to Thread 1024 (LWP 17090)]
Breakpoint 1, runops_jit (interpreter=0x8214a08, pc=0x400180d0) at
src/interpreter.c:422
422 jit_f jit_code = (jit_f) D2FPTR(init_jit(interpreter, pc));
(gdb) n
423 (jit_code) (interpreter, pc);
(gdb) n
Address of base segment is 0x400180d0
dummy was called with: 1
425 return NULL;
(gdb) n
426 }
(gdb) n
runops_int (interpreter=0x8214a08, offset=0) at src/interpreter.c:633
633 interpreter-lo_var_ptr = old_lo_var_ptr;
(gdb) n
634 if ((interpreter-resume_flag RESUME_RESTART)
(gdb) n
553 while (interpreter-resume_flag RESUME_RESTART) {
(gdb) n
638 }
(gdb) n
runops_ex (interpreter=0x8214a08, offset=0) at src/interpreter.c:652
652 if (interpreter-resume_flag RESUME_ISJ) {
(gdb) n
679 }
(gdb) n
runops (interpreter=0x8214a08, offset=0) at src/interpreter.c:709
709 }
(gdb) n
Parrot_runcode (interpreter=0x8214a08, argc=1, argv=0xb980) at
src/embed.c:514
514 }
(gdb) n
main (argc=1, argv=0xb980) at imcc/main.c:551
551 Parrot_destroy(interpreter);
(gdb) n
552 if (output)
(gdb) n
554 mem_sys_free(interpreter-imc_info);
(gdb) n
555 Parrot_exit(0);
(gdb) n
*** Parrot VM: Dumping GC info ***
Total memory allocated = 139264
DOD runs = 0
Collect runs = 0
Collect memory = 0
Active PMCs = 1024
Active buffers = 6144
Total PMCs = 1024
Total buffers = 6144
Header allocations since last collect = 7
Memory allocations since last collect = 1
Program exited normally.
(gdb)
Harry Jackson
Need some deep non-POSIX thread info (Attention you Windows folks!)
Well... it's time to start digging into threads more seriously, which Leo's been starting to do. At the moment I'm leaning towards all threads in a thread group sharing string and PMC arenas, and memory pools, as it makes life very much easier in some ways, except... The DOD. And the GC, though that'll be less of an issue if we swap in a non-moving collector when we go multithreaded. The biggest issue with sharing pools is dealing with the system stack, as we'll now have multiple stacks, one for each thread. That makes finding the stacks and walking them somewhat problematic. Now, for POSIX threads, which the various Unices and VMS use, we can handle this without much of a problem. There's sufficient information available, and sufficient calls handy, to get a handle on a thread's stack. (Though we may have to do it at thread creation time. Still, it's doable) However... there's the pesky issue of non-POSIX thread implementations, pesky mainly because I have *no* idea what they do, and by the time I get up to speed with Windows threads I think it's safe to say that it'll be too darned late. So, could someone with some windows experience go digging and find out how one would: 1) Find the address of the base of a thread's stack 2) Find out how big a thread's stack is 3) Find out what a thread's current stack pointer is If we have to do some of this at thread creation time (i.e. creating a thread and handing it a chunk of memory X bytes long to use as its stack) that's fine, and if we can't get some of this info (#3 is definitely problematic) well... we can cope. (This is definitely one of those times where I really want to get a handle on the output of the C compiler and its boilerplate. I take this as a bad sign :) -- Dan --it's like this--- Dan Sugalski even samurai [EMAIL PROTECTED] have teddy bears and even teddy bears get drunk
IMCC keyed crasher
IMCC bus errors (at least on OS X) when presented with the construct: set $P0[$I1], Params[$I1] This little test program triggers it for me: .sub _MAIN prototyped .local Array Foo .local Array Bar set Foo[1], Bar[1] .end IMCC also doesn't like the construct: Foo[1] = Bar[1] but that's a separate issue. -- Dan --it's like this--- Dan Sugalski even samurai [EMAIL PROTECTED] have teddy bears and even teddy bears get drunk
Perl Abstract/Concrete Syntax Tree
Hi, I was wondering if there was anything built in Perl (a Module) that will take in a Perl file and parse that into an abstract or concrete syntax tree. I searched around cpan for a bit and couldn't find what I was looking for. If anyone is wondering what I'm talking about there is a nice Java package that Eclipse uses to create a tree for Java compilation that I can point you toward. Thanks for your time. ~~Andrew
Re: Perl Abstract/Concrete Syntax Tree
--- Potozniak, Andrew [EMAIL PROTECTED] wrote: Hi, I was wondering if there was anything built in Perl (a Module) that will take in a Perl file and parse that into an abstract or concrete syntax tree. I searched around cpan for a bit and couldn't find what I was looking for. If anyone is wondering what I'm talking about there is a nice Java package that Eclipse uses to create a tree for Java compilation that I can point you toward. Thanks for your time. ~~Andrew Andrew, I'm a bit confused. Eclipse is simply an IDE, so that doesn't help me understand what you're looking for. As for turning a file into a syntax tree, what do you mean? Files are in many different formats. I can't imagine a single module handling CSV, XML, and binary JPEG files the same way. If you could be a bit more concrete in what you are looking for, we would be better prepared to help. Perhaps if you could send a code snippet of what you would *like* Perl to do (and specify the file format(s) that you want to work with). Cheers, Ovid = Silence is Evilhttp://users.easystreet.com/ovid/philosophy/indexdecency.htm Ovid http://www.perlmonks.org/index.pl?node_id=17000 Web Programming with Perl http://users.easystreet.com/ovid/cgi_course/
Re: Perl Abstract/Concrete Syntax Tree
Andrew Potozniak wrote in perl.qa : I was wondering if there was anything built in Perl (a Module) that will take in a Perl file and parse that into an abstract or concrete syntax tree. I searched around cpan for a bit and couldn't find what I was looking for. If anyone is wondering what I'm talking about there is a nice Java package that Eclipse uses to create a tree for Java compilation that I can point you toward. Thanks for your time. Hm -- as the task of parsing perl involves some complex interplay between compile-time and run-time -- think about the function prototypes, for example -- perl can't be described by a non-ambiguous grammar, and can't be parsed accurately by tools that don't include a full-fledged perl interpreter already. However, during parsing, perl builds internally an abstract syntax tree for its own usage ; the syntax tree is then walked during execution. It's possible to stop the perl interpreter after the compilation of the main program and to have it print the current contents of the parse tree. This is achieved by the O and B::* modules, that come with perl. You may want something not too far from this : $ perl -MO=Concise -e 'print $a + $b' 8 @ leave[1 ref] vKP/REFC -(end) 1 0 enter -2 2 ; nextstate(main 1 -e:1) v -3 7 @ print vK -8 30 pushmark s -4 62 add[t1] sK/2 -7 - 1 ex-rv2sv sK/1 -5 4 $ gvsv(*a) s -5 - 1 ex-rv2sv sK/1 -6 5 $ gvsv(*b) s -6 -e syntax OK See the docs for O, B and B::Concise for more information.
Re: Threading design
At 11:27 AM -0500 12/30/03, Gordon Henriksen wrote: I wish the threading design for parrot would look more toward successful, performant multithreaded systems, I'm going to be really grumpy here, though it's not directed at Gordon. What *I* wish is that people who've not had any experience trying to build threaded interpreters for languages with data as heavyweight as perl's with a POSIXy share everything requirement that guarantee user threading problems won't crash the interpreter would stop pronouncing judgement on threading designs. It's getting really tiresome and I'm going to start getting viciously rude about it. If you *have* experience with this sort of thing, *please* share. Otherwise stop telling me the design sucks--I *know* that already. What I don't have is a better answer, nor the ability to throw out the troublesome requirements. If you want to help, then great. Specifics are a wonderful thing--X worked because of Y and Z, or Q didn't work because of R and/or S. Details are great, generalities are OK if details aren't available for whatever reason. If, on the other hand, you just want to snipe, then you can either have my job (I'm serious--you want it, it's yours) or shut up. Thanks. -- Dan --it's like this--- Dan Sugalski even samurai [EMAIL PROTECTED] have teddy bears and even teddy bears get drunk
Re: Garbage Collection Tasks
On Dec 29, 2003, at 11:48 AM, Dan Sugalski wrote: As I see it, it's really the allocation that is more complicated with a mark-and-sweep collector (since you have to search for a correct-sized free chunk, efficiently)--the collection itself is the easy part. Actually, it seems like this is just the GC_IS_MALLOC case--that already gives us non-moving GC, and lets malloc (the system version or the Lea version) deal with managing the bookkeeping. Allocation's more complex, but so is deallocation. Properly maintaining a free list with adjacent piece coalescing can be non-trivial, and there's the added complication of COW so multiple string headers may be pointing to the same chunk of memory, so freeing up one's not a sufficient reason to return the memory to the free pool. Yep, I think the Lea allocator takes care of the tricky parts of the free-list management, and the GC_IS_MALLOC code is already handling the COW cases. #2 is a bit more interesting and, if we do it right, means that we'll end up with a pluggable garbage collection system and may (possibly) be able to have type 3 threads share object arenas and memory pools, which'd be rather nice. Even if not, it leaves a good window for experimentation in allocation and collection, which isn't bad either. This, combined with the mention of needing to drive this off of a vtable, means being able to determine the GC algorithm at runtime instead of compile-time (i.e., that was part of your point). I'm all for that--it will mean less #if's in the code, and it will make the code a bit clearer. Yep. Care to take a shot at it? I'll add it to my to-do list. I'm trying to finish up a few things which I've left sitting 90% done, so if someone else wants to tackle this before I get to it then go ahead--just post to the list when you start so that we don't duplicate efforts (and I'll do the same, if I get to it and no one else has started). JEff
Re: Threading design
On Dec 30, 2003, at 11:18 AM, Dan Sugalski wrote: At 11:27 AM -0500 12/30/03, Gordon Henriksen wrote: I wish the threading design for parrot would look more toward successful, performant multithreaded systems, I'm going to be really grumpy here, though it's not directed at Gordon. What *I* wish is that people who've not had any experience trying to build threaded interpreters for languages with data as heavyweight as perl's with a POSIXy share everything requirement that guarantee user threading problems won't crash the interpreter would stop pronouncing judgement on threading designs. It's getting really tiresome and I'm going to start getting viciously rude about it. If you *have* experience with this sort of thing, *please* share. Otherwise stop telling me the design sucks--I *know* that already. What I don't have is a better answer, nor the ability to throw out the troublesome requirements. If you want to help, then great. Specifics are a wonderful thing--X worked because of Y and Z, or Q didn't work because of R and/or S. Details are great, generalities are OK if details aren't available for whatever reason. Please understand the people are not specifically criticizing you--a lot of people care about the design and success of Parrot, and it valuable for everyone to hear their opinions, even if they are seeing the problems and not the solutions. It's important (psychologically) for people to be able to feel heard, and also (statistically) to find out what the community feels is important (this slice of the community, at least). Please try not to take it personally--it's just a discussion, and people are talking just as much to each other as they are to you, even if a design decision on your part sparked the discussion. I realize it can be frustrating for you, but everyone really does seem to have the same goal at heart, and we'll end up with a better product in the end if discussion is encouraged. Also, please share specifics that you have about design decisions that others seem to disagree with. With more information on the table, the rationale will be clearer, and that will either help convince people that it really is the best design, or it will allow them to identify specific flaws--places where some bit of data may have been overlooked. And also, if you feel stumped by something, put that out there too--I think it sets a very different tone if a (seemingly questionable) design decision is one which you think is great (or as good as it can get), versus just the best which you've been able to come up with so far. This will help steer the discussion, and make it more productive. I hope you find this feedback useful. JEff
Re: Threading design
DS == Dan Sugalski [EMAIL PROTECTED] writes: DS I'm going to be really grumpy here, though it's not directed at DS Gordon. What *I* wish is that people who've not had any experience DS trying to build threaded interpreters for languages with data as DS heavyweight as perl's with a POSIXy share everything requirement DS that guarantee user threading problems won't crash the interpreter DS would stop pronouncing judgement on threading designs. It's DS getting really tiresome and I'm going to start getting viciously DS rude about it. just don't start to pluck your (or parrot's) feathers. :) lightening things a little i hope, uri -- Uri Guttman -- [EMAIL PROTECTED] http://www.stemsystems.com --Perl Consulting, Stem Development, Systems Architecture, Design and Coding- Search or Offer Perl Jobs http://jobs.perl.org
Re: Problem during make test
On Dec 29, 2003, at 2:12 PM, Harry Jackson wrote:
During
[EMAIL PROTECTED] parrot]$ make test
echo imcc/imcc.y -d -o imcc/imcparser.c
imcc/imcc.y -d -o imcc/imcparser.c
perl -e 'open(A,qq{$_}) or die foreach @ARGV' imcc/imcc.y.flag
imcc/imcparser.c imcc/imcparser.h
perl t/harness --gc-debug --running-make-test -b t/op/*.t t/pmc/*.t
t/native_pbc/*.t
t/op/00ff-dos...
This is as far as it gets. I am assuming since no one else has noticed
this that it is a problem with my set up but I am at a bit of a loss
as to what has happened to cause it.
It gets even stranger. If I do a make clean and make test again it
does not necessarily stop in the same place each time ie.
Here are 3 things to try:
1) When it hangs there, check with 'top' to see if it is using CPU (ie,
is it blocking, or in an infinite loop).
2) Try running one of the tests which blocks, individually. If you can
get it to happen this way, then run it in gdb and see what it's doing.
(Or, attach to an already blocked one from 'make test'--this is
assuming it's parrot that's actually blocking, and not t/harness.)
3) Try building from a clean checkout, and see if that shows the
problem. If not, it's probably something you've changed and don't
realize.
JEff
Threads. Design. Go for it
It's pretty obvious that we've a number of folks who've got Thread Religion. It's also very obvious that there is more than one One True Thread Religion. And it's *definitely* obvious that I'm getting cranky. So. This is everyone's chance. You have what you think is the Right Way to do it? Fine. Time for design details. Your constraints: 1) A POSIX-style everything is shared implicitly mode must be available. It may be slow. 2) A perl 5 iThreads it's not a thread, it's a fork. Well, sorta... mode must be available 3) You must give details of the proposed changes/additions to the design of the engine and underlying code. 4) Keep in mind that the engine has events and asynchronous I/O floating around. These are a given. 5) Be very aware of the existence of SMP machines, and the potential fun that can ensue with a combination of SMP, DOD, and GC. (You may assume that the current copying GC can be replaced with a non-copying version when running with threads) 6) Bytecode can't, under any circumstances, crash the interpreter because of thread programming errors. (Barring implementation problems) I make no promises that a proposal will get anything other than ignored. The more of the current core design a proposal thinks should be replaced, the less likely it is to go anywhere. Handwave on details at your peril. No warrantee is expressed or implied. Slippery when wet. Your mother says you never write, and you should wear a sweater. I may or may not ask questions, make pointed comments and/or observations, or say anything about any proposal. It's possible, and likely, that there are elements of parrot's design that I've insufficiently expressed which may torpedo a proposal, though I'll try to detail those when they come up. This, as they say, is your one and only chance. Go for it, and good luck. Please start a *new* thread, with a uniquely identifying subject line (a proposal name or your name in it is good) so I have at least some hope of keeping it all straight. -- Dan --it's like this--- Dan Sugalski even samurai [EMAIL PROTECTED] have teddy bears and even teddy bears get drunk
This week's summary
The Perl 6 Summary for the week ending 20031228 It's the last Perl 6 Summary of 2003 already. Where did the year go? A large part of my year went down the plughole in the great double disk disaster which saw about 3 years of mail, a few gigabytes of photos and my entire summary archive disappear into the ether. I have backups now. This means I won't be going back through the archive to give you a potted history of Perl 6 and Parrot 2003. Instead, I'll plough on with what happened last week, starting with perl6-internals. More object stuff A few weeks ago, Dan asked for volunteers to implement a nice Parrot OO interface to the Postgres RDBMS to replace the purely procedural version that he wrote before Parrot had objects. Harry Jackson has stepped up to the plate. Go Harry. http://tinyurl.com/2bq9w The trouble with threads... Threads are evil. But they're also handy tools for a lot of work. This dichotomy means that people have opinions about them. And they express them. The rolling threads thread made it to p6i this week. Well, it made it to p6i last week too, but (in the nature of the roving threads thread) it continued this week too. Later in the week Dan laid down the ground rules and the beginnings of the design of Parrot's thread capabilities. Essentially, Parrot will guarantee that 'nothing that bytecode does should be able to corrupt the internals of Parrot'. C extensions are free to muck up everything, but they should try not to. That's about the only guarantee Parrot will make; keeping user level code thread safe is the job of the author of said code. Dan went onto list the various rules about how data is shared between threads. This being a threads thread a few people dragged it off into a discussion of how to design programs with/without threads. Ho hum. Leo Tötsch got stuck into the technical meat of Dan's post, and kicked off a discussion of the impact of threads on the memory management system (in the presence of threads, a copying garbage collector becomes impractical for instance, and things can get ugly when, say, thread 2 needs to grow a shared string that originated with thread 1). http://tinyurl.com/37w6x http://tinyurl.com/2j3xo ParrotIO objects Cory Spencer wanted to know if it would be possible to push a character/byte back onto a ParrotIO stream. Melvin Smith promised that parrot would get both unget and peek but not necessarily until he'd recovered from a bout of flu. http://tinyurl.com/3f8tp Licensing of library wrappers Why does it have to be so hard to give stuff away if you want to stop other people claiming it as their own? Not content with having the thread thread, this week also saw the return of the licensing thread. Dan wants to have any library wrappers like ncurses.pasm that get included with Parrot be as nearly in the public domain as their authors can make them. The few people that commented agreed with him. http://tinyurl.com/yscmh 'Python' running fast on .NET Not content with having a threads thread and a licensing thread, this week also saw a Python's really quick on .NET thread. It seems that Miguel de Icaza has been showing off some benchmarks from 'IronPython', a Python compiler that targets .NET. Dan was dismissive, pointing out that the bits of Python that will 'give .NET serious fits' hadn't been implemented. Joe Wilson wasn't so sure and, later in the thread, implemented a couple of benchmarks that gave some worrying performance numbers (with Perl 5 going substantially faster than Parrot). This led to a certain amount of rewriting of the benchmarks to make them really equivalent and Dan realizing that there's a need for a faster, less flexible base array type. Joe also came up with a recursive Fibonacci sequence function that ran horribly slowly in its IMCC incarnation when using the full Parrot Calling Convention. Again the code generated needed optimizing, but it did point up some issues that will be looked at. Leo Tötsch in particular redid the original benchmark with continuation creation hoisted out of the loop and ended up with an IMCC version that, unoptimized, ran about 33% faster than the Perl code. Work continues on identifying and fixing the bottlenecks these benchmarks threw up. http://tinyurl.com/2ozzz http://tinyurl.com/29w4y http://tinyurl.com/365x2 Future Win32 JIT issues Jonathan Worthington pointed to some documentation from Microsoft which states that, in the future, Win32 will turn on execution protection of data pages by default, which has implications for any JIT system. The fix is to use the appropriate Win32 API call to allocate memory that's marked as okay for execution. Jonathan wondered if this would have any
Re: Problem during make test
Jeff Clites wrote:
Here are 3 things to try:
1) When it hangs there, check with 'top' to see if it is using CPU (ie,
is it blocking, or in an infinite loop).
Already done that and it is eating no cycles.
2) Try running one of the tests which blocks, individually. If you can
get it to happen this way, then run it in gdb and see what it's doing.
(Or, attach to an already blocked one from 'make test'--this is assuming
it's parrot that's actually blocking, and not t/harness.)
When run individually I get the same error. Complete freeze at what
appears to be an arbitrary point.
Running gdb
0x080a7625 in Perl_sv_gets ()
(gdb) n
Single stepping until exit from function Perl_sv_gets,
which has no line number information.
0x0809d254 in Perl_do_readline ()
(gdb) n
Single stepping until exit from function Perl_do_readline,
which has no line number information.
0x08099fd8 in Perl_runops_standard ()
(gdb) n
Single stepping until exit from function Perl_runops_standard,
which has no line number information.
t/op/arithmeticsok 13/18
This is where gdb freezes execution. CTL-C then frees it up to continue
until the next one freezes.
3) Try building from a clean checkout, and see if that shows the
problem. If not, it's probably something you've changed and don't realize.
I have tried something a bit more drastic. Deleted the entire tree and
downloaded it again (sorry about the bandwidth).
I have also tried strace and got the following.
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0x401fe000
read(3, #! perl -w\n\nuse Parrot::Test tes..., 4096) = 4096
brk(0x81e2000) = 0x81e2000
close(3)= 0
munmap(0x401fe000, 4096)= 0
pipe([3, 4])= 0
pipe([5, 6])= 0
fork() = 19676
close(4)= 0
close(6)= 0
read(5, , 4) = 0
close(5)= 0
fcntl64(0x3, 0x3, 0xb5c4, 0)= 0
fstat64(3, {st_mode=S_IFIFO|0600, st_size=0, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0x401fe000
_llseek(3, 0, 0xb410, SEEK_CUR) = -1 ESPIPE (Illegal seek)
fcntl64(0x3, 0x2, 0x1, 0x1d)= 0
read(3, 1..18\n, 4096)= 6
read(3,
Doing a ps ax reveals the following (ignore the test number it keeps
changing)
20802 pts/11 S 0:00 make test
21598 pts/11 S 0:00 perl t/harness --gc-debug --running-make-test
-b t/op/00ff-dos.t t/op/00ff-unix.t t/op/arithmetics.t t/op/basic.t
21610 pts/11 S 0:00 perl -w t/op/arithmetics.t
21620 pts/11 S 0:00 ./parrot --gc-debug -b t/op/arithmetics_4.pasm
21621 pts/11 S 0:00 ./parrot --gc-debug -b t/op/arithmetics_4.pasm
21622 pts/11 S 0:00 ./parrot --gc-debug -b t/op/arithmetics_4.pasm
From all of this I am guessing that something has corrupted a module in
Perl at least that is all I can think of.
Harry Jackson
Re: Problem during make test
On Dec 30, 2003, at 3:11 PM, Harry Jackson wrote: 2) Try running one of the tests which blocks, individually. If you can get it to happen this way, then run it in gdb and see what it's doing. (Or, attach to an already blocked one from 'make test'--this is assuming it's parrot that's actually blocking, and not t/harness.) When run individually I get the same error. Complete freeze at what appears to be an arbitrary point. Running gdb 0x080a7625 in Perl_sv_gets () I meant try running parrot in the debugger--Perl is probably hanging b/c it's waiting for parrot to exit. For instance, see if the following hangs, and if so run it in gdb: ./parrot --gc-debug -b t/op/arithmetics_4.pasm This is where gdb freezes execution. CTL-C then frees it up to continue until the next one freezes. The ctrl-C is probably killing the subprocess (parrot), which does imply that it's parrot that's hanging. 3) Try building from a clean checkout, and see if that shows the problem. If not, it's probably something you've changed and don't realize. I have tried something a bit more drastic. Deleted the entire tree and downloaded it again (sorry about the bandwidth). Yep, that's what I meant. I have also tried strace and got the following. Try this on parrot rather than Perl. Doing a ps ax reveals the following (ignore the test number it keeps changing) 20802 pts/11 S 0:00 make test 21598 pts/11 S 0:00 perl t/harness --gc-debug --running-make-test -b t/op/00ff-dos.t t/op/00ff-unix.t t/op/arithmetics.t t/op/basic.t 21610 pts/11 S 0:00 perl -w t/op/arithmetics.t 21620 pts/11 S 0:00 ./parrot --gc-debug -b t/op/arithmetics_4.pasm 21621 pts/11 S 0:00 ./parrot --gc-debug -b t/op/arithmetics_4.pasm 21622 pts/11 S 0:00 ./parrot --gc-debug -b t/op/arithmetics_4.pasm That does seem odd that it appears to be running the same test 3 times JEff
Re: Need some deep non-POSIX thread info (Attention you Windows folks!)
From: Dan Sugalski [EMAIL PROTECTED] So, could someone with some windows experience go digging and find out how one would: 1) Find the address of the base of a thread's stack 3) Find out what a thread's current stack pointer is I would do 1), 3) this way ... thdl = _beginthreadex(NULL, 0, thread_function, arg, CREATE_SUSPENDED, thraddr); #if defined(_X86_) ctx.ContextFlags = CONTEXT_FULL; #elif defined(_ALPHA_) ctx.ContextFlags = CONTEXT_INTEGER; #else /* _MIPS_ and etc */ #endif GetThreadContext(thdl, ctx); #if defined(_X86_) current_stack_ptr = ctx.Esp; #elif defined(_ALPHA_) current_stack_ptr = ctx.IntSp; #else /* _MIPS_ and etc */ #endif VirtualQuery((LPCVOID)current_stack_ptr, mbi, sizeof(mbi)); stack_allocbase_addr = mbi.AllocationBase; 2) Find out how big a thread's stack is By default, OS reserves 1 MB of memory for a thread's stack. One can specify a different size with a linker option or a STACKSIZE statement in the .DEF file. So it's, sort of, up to us how big a thread stack is. No problem here. Dan 0x4C56
Re: Need some deep non-POSIX thread info (Attention you Windows folks!)
And a note for 3): It's importatnt to create a thread with CREATE_SUSPENDED, and at thread runtime we have to suspend thread while checking out its registers so that to get the true values. SuspendThread(thdl); GetThreadContext(thdl, ctx); ... ResumeThread(thdl); 0x4C56
Re: Problem during make test
Jeff Clites wrote:
On Dec 30, 2003, at 3:11 PM, Harry Jackson wrote:
2) Try running one of the tests which blocks, individually. If you
can get it to happen this way, then run it in gdb and see what it's
doing. (Or, attach to an already blocked one from 'make test'--this
is assuming it's parrot that's actually blocking, and not t/harness.)
When run individually I get the same error. Complete freeze at what
appears to be an arbitrary point.
Running gdb
0x080a7625 in Perl_sv_gets ()
I meant try running parrot in the debugger--Perl is probably hanging b/c
it's waiting for parrot to exit. For instance, see if the following
hangs, and if so run it in gdb:
../parrot --gc-debug -b t/op/arithmetics_4.pasm
[parrot]$ ./parrot --gc-debug -b t/op/arithmetics_4.pasm
4123
4123
[parrot]$ ./parrot --gc-debug -b t/op/arithmetics_3.pasm
3877
3877
[parrot]$ ./parrot --gc-debug -b t/op/arithmetics_2.pasm
0
1234567890
1234567890
0
1234567890
1234567890
[parrot]$ ./parrot --gc-debug -b t/op/arithmetics_1.pasm
0
-1234567890
1234567890
0
-1234567890
1234567890
Looks good to me. I have taken the drastic measure of upgrading to 5.8.2
and... no change. I am still locking up during tests, or should I
say, parrot is still locking up during tests, I seem to be continuing
along fine which is why I am still complaining ;-)
I have also tried strace and got the following.
Try this on parrot rather than Perl.
strace on parrot gets to
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
wait4(-1, [WIFEXITED(s) WEXITSTATUS(s) == 0], 0, NULL) = 8423
--- SIGCHLD (Child exited) ---
sigreturn() = ? (mask now [])
write(1, : blib/lib/libparrot.a\n, 23: blib/lib/libparrot.a
) = 23
rt_sigprocmask(SIG_BLOCK, [HUP INT QUIT TERM XCPU XFSZ], NULL, 8) = 0
vfork() = 8424
--- SIGCHLD (Child exited) ---
sigreturn() = ? (mask now [HUP INT QUIT TERM
XCPU XFSZ])
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
wait4(-1, [WIFEXITED(s) WEXITSTATUS(s) == 0], 0, NULL) = 8424
rt_sigprocmask(SIG_BLOCK, [HUP INT QUIT TERM XCPU XFSZ], NULL, 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
stat64(blib/lib/libparrot.a, {st_mode=S_IFREG|0664, st_size=18102092,
...}) = 0
write(1, cc -o parrot -Wl,-E -g imcc/ma..., 98cc -o parrot -Wl,-E
-g imcc/main.o blib/lib/libparrot.a -lnsl -ldl -lm -lcrypt -lutil -lpthread
) = 98
rt_sigprocmask(SIG_BLOCK, [HUP INT QUIT TERM XCPU XFSZ], NULL, 8) = 0
vfork() = 8425
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
wait4(-1, [WIFEXITED(s) WEXITSTATUS(s) == 0], 0, NULL) = 8425
--- SIGCHLD (Child exited) ---
sigreturn() = ? (mask now [])
rt_sigprocmask(SIG_BLOCK, [HUP INT QUIT TERM XCPU XFSZ], NULL, 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
stat64(parrot, {st_mode=S_IFREG|0775, st_size=4133629, ...}) = 0
stat64(test_prep, 0xbfffddf0) = -1 ENOENT (No such file or
directory)
stat64(testb, 0xbfffddf0) = -1 ENOENT (No such file or
directory)
write(1, /usr/local/bin/perl5.8.2 t/harne...,
106/usr/local/bin/perl5.8.2 t/harness --gc-debug --running-make-test -b
t/op/*.t t/pmc/*.t t/native_pbc/*.t
) = 106
rt_sigprocmask(SIG_BLOCK, [HUP INT QUIT TERM XCPU XFSZ], NULL, 8) = 0
vfork() = 8428
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
t/op/00ff-dos...ok
t/op/00ff-unix..ok
t/op/arithmeticsok 15/18
freeze punk it's the police.
I am now convinced that due to the baffling nature of the problem that
it will be something stupid.
Harry Jackson
Re: [perl #24769] [PATCH] mem_sys_allocate_executable - initial draft
From: Jonathan Worthington [EMAIL PROTECTED]
Here is my attempt at a patch for executable memory allocation, which
makes
+void *
+mem_alloc_executable(size_t size)
+{
+ void *ptr = VirtualAlloc(NULL, size, MEM_COMMIT, PAGE_EXECUTE_READWRITE);
^^^
MEM_RESERVE | MEM_COMMIT
or more preferrable MEM_RESERVE | MEM_TOP_DOWN | MEM_COMMIT
+ if (memInfo.RegionSize = size)
+ {
+ CopyMemory(newBlock, memblock, memInfo.RegionSize);
+ }
+ else
+ {
+ CopyMemory(newBlock, memblock, size);
+ }
memInfo.RegionSize is always either greater or equal size. So just
CopyMemory(newBlock, memblock, size);
would be enough
+void
+mem_free_executable(void *addr)
+{
+ /* We need to decommit, then release the pages. */
+ VirtualFree(addr, 1, MEM_DECOMMIT); No need
+ VirtualFree(addr, 0, MEM_RELEASE);
+}
There is no need to decommit pages if next you release the region.
0x4C56
Re: [perl #24769] [PATCH] mem_sys_allocate_executable - initial draft
- Original Message -
From: Vladimir Lipsky [EMAIL PROTECTED]
To: perl6-internals [EMAIL PROTECTED]
Sent: Wednesday, December 31, 2003 2:42 AM
Subject: Re: [perl #24769] [PATCH] mem_sys_allocate_executable - initial
draft
From: Jonathan Worthington [EMAIL PROTECTED]
Here is my attempt at a patch for executable memory allocation, which
makes
+void *
+mem_alloc_executable(size_t size)
+{
+ void *ptr = VirtualAlloc(NULL, size, MEM_COMMIT,
PAGE_EXECUTE_READWRITE);
^^^
MEM_RESERVE | MEM_COMMIT
or more preferrable MEM_RESERVE | MEM_TOP_DOWN | MEM_COMMIT
Agree.
+ if (memInfo.RegionSize = size)
+ {
+ CopyMemory(newBlock, memblock, memInfo.RegionSize);
+ }
+ else
+ {
+ CopyMemory(newBlock, memblock, size);
+ }
memInfo.RegionSize is always either greater or equal size. So just
CopyMemory(newBlock, memblock, size);
would be enough
I'd disagree here. realloc allows somebody to resize a block of memory.
Here I am checking the lengths of the old block and the new block. If they
are asking to enlarge their block of memory, then if we do CopyMemory with
the size of the new block we will be reading into memory that wasn't a part
of the old block, which is a Bad Thing.
+void
+mem_free_executable(void *addr)
+{
+ /* We need to decommit, then release the pages. */
+ VirtualFree(addr, 1, MEM_DECOMMIT); No need
+ VirtualFree(addr, 0, MEM_RELEASE);
+}
There is no need to decommit pages if next you release the region.
OK.
Provided Leo likes the general gist of my patch, I'll update it with your
suggested changes. Let me know if you still disagree on the second point.
Thanks, and happy new year to you and everyone for tomorrow,
Jonathan
Re: Need some deep non-POSIX thread info (Attention you Windows folks!)
At 5:07 AM +0300 12/31/03, Vladimir Lipsky wrote: From: Dan Sugalski [EMAIL PROTECTED] So, could someone with some windows experience go digging and find out how one would: 1) Find the address of the base of a thread's stack 3) Find out what a thread's current stack pointer is I would do 1), 3) this way ... [Some snippage] Cool, thanks. I'm not sure exactly what we're going to do at the moment, but I'm glad we can do it. Being able to temporarily pause a thread'll be handy--that's something the POSIX thread interface won't let you do and, while I see the point, it's really handy. 2) Find out how big a thread's stack is By default, OS reserves 1 MB of memory for a thread's stack. One can specify a different size with a linker option or a STACKSIZE statement in the .DEF file. So it's, sort of, up to us how big a thread stack is. No problem here. Yow. Unless there's a good reason for it to be that big, I think I'd like to go with a smaller stack by default, something more like 20-30K. (Though if there are OS functions that can chew up great gobs of stack space we can certainly go larger) I appreciate the info, and it'll help get the generic interface for threads nailed down. -- Dan --it's like this--- Dan Sugalski even samurai [EMAIL PROTECTED] have teddy bears and even teddy bears get drunk
pdd16
I might be going mad here and maybe I have been up too long but, does: http://dev.x.perl.org/perl6/pdd/pdd16_native_call.html have two identical Parrot_callback_C and Parrot_callback_D function signatures. Harry Jackson
Re: pdd16
At 2:28 AM + 12/31/03, Harry Jackson wrote: I might be going mad here and maybe I have been up too long but, does: http://dev.x.perl.org/perl6/pdd/pdd16_native_call.html have two identical Parrot_callback_C and Parrot_callback_D function signatures. Possibly, though they're supposed to have their parameters reversed. I'll go fix as soon as I can. -- Dan --it's like this--- Dan Sugalski even samurai [EMAIL PROTECTED] have teddy bears and even teddy bears get drunk
Re: PMC registry
At 07:38 PM 12/28/2003 -0500, Dan Sugalski wrote: At 7:19 PM -0500 12/28/03, Matt Fowles wrote: Dan Sugalski wrote: At 3:27 PM -0500 12/28/03, Matt Fowles wrote: Leopold Toetsch wrote: I'd use a custom hash with the PMC address being the key[1]. /Me thinks, it doesn't help, when a PMC gets registered multiple times - its always the same address - removing it multiple times is fine, the first succeeds, following fail silently, they do nothing. On a side note, couldn't this be used for the explicit root set augmentation version of DOD that was discussed? If you're speaking of what I think you are (my memory sucks) then this is exactly that. If you're not speaking of what I think you are, then no it isn't, but you can probably use it for that. :) I am, but there is a little more. The explicit root set augmentation was suggested as a way to avoid stackwalking. Ah, OK. No, this isn't that. There's far too much overhead involved for it to be reasonable for that purpose. I know people really hate the idea of walking the system stack, but it really isn't evil, and is definitely a much more stable and safe alternative to the gyrations that'd otherwise be needed. Reading my backlog from the holidays and flu. My 2 cents. It isn't guaranteed that disabling stack walking is going to work since there may be other places (even in Parrot internals) where someone is holding onto a PMC pointer while a DOD runs (besides extension code). The stack walking was added to help fix the infant mortality problem, which was way before extension API. -Melvin
Re: Strangeness with '.sub' in macros
At 05:45 PM 12/30/2003 +0100, Bernhard Schmalhofer wrote: Hi, I have been playing around with 'libpcre' for Parrot m4. For some reason I couldn't compile two regular expressions in the same PIR script. I created a sample C program and that worked like it should. It looks like the error has nothing to do with 'libpcre'. So I boiled down my code to a small test script. When a macro contains a '.sub' call, and that macro is used twice, then I get a 'memory error'. [EMAIL PROTECTED]:~/devel/Parrot/parrot/languages/m4 ../../parrot t/regex/macro_used_twice.imc Speicherzugriffsfehler Could sombody test the attached script on another machine? I'm working here on a Linux laptop: Two answers here. 1) We probably have a bug in macro expansion 2) Dan has declared that macros are coming out of IMCC. (Mainly because I told him I don't want them and they make the lexer more complicated than I want it to be.. and.. macros are less useful for compilers than people hand-writing IMC code) I prefer macros to be a part of a preprocessor tool external to the builtin IMC language. If enough people really have strong opinions on macros, now is the time to voice it, because otherwise I'm removing them. -Melvin
