[perl6/specs] 0f5bdf: added NCI
Branch: refs/heads/master Home: https://github.com/perl6/specs Commit: 0f5bdf9d17ae996c2cd0910b5a4242f6722ca38a https://github.com/perl6/specs/commit/0f5bdf9d17ae996c2cd0910b5a4242f6722ca38a Author: Stéphane Payrard cognomi...@gmail.com Date: 2015-03-19 (Thu, 19 Mar 2015) Changed paths: M S99-glossary.pod Log Message: --- added NCI
[perl #62244] [BUG] recent rakudo changes have broken NCI
On Mon Jan 12 07:34:33 2009, donaldhunter wrote: It appears that rakudo changes committed over the weekend have broken NCI. The last known working revision was #35300 for ext/SQLite3/t/test.p6 which is crashing on #35440. You'll need to patch ext/SQLite3/Makefile.PL so that it works, then you can run make test. I can still run PIR code that uses Sqlite3 so I'm pretty sure it's isolated to rakudo. Cheers, Donald. Sorry for the delay; this ticket is now obsolete. See https://github.com/jnthn/zavolaj/, especially the current SQLite3 example here: https://github.com/jnthn/zavolaj/blob/master/examples/sqlite3.p6 Zavolaj is included in the latest version of Rakudo Star (http://rakudo.org/2012/01/28/rakudo-star-2012-01-released/). Cheers. -- Will Coke Coleda
Re: [perl #62244] [BUG] recent rakudo changes have broken NCI
On Mon, Jan 12, 2009 at 07:34:33AM -0800, Donald Hunter wrote: It appears that rakudo changes committed over the weekend have broken NCI. The last known working revision was #35300 for ext/SQLite3/t/test.p6 which is crashing on #35440. You'll need to patch ext/SQLite3/Makefile.PL so that it works, then you can run make test. I can still run PIR code that uses Sqlite3 so I'm pretty sure it's isolated to rakudo. In order to get parameter passing and object attributes to work properly, some places that had previously been performing a simple bind operation now perform a copy or create an ObjectRef PMC. This has the potential to affect NCI in two ways: 1. Until we can completely update method dispatch in Rakudo (or Parrot), sometimes a method will be invoked with 'self' set to an ObjectRef PMC instead of the actual 'self'. I don't yet know of a good way to improve this using Parrot's existing method dispatch. 2. Some assignment/initialization operations may result in a PMC being cloned where previously it was simply referenced. The NCI method might be confused about operating on a clone. In general, Rakudo currently creates ObjectRef PMCs (references) whenever it comes across a PMC that isn't in Rakudo's type hierarchy and that doesn't do the (Parrot) 'scalar' role. If we have some specific code that is failing, we can see about either updating Rakudo to work with that code or providing some workarounds until we get method dispatch straightened out. Pm
[perl #62244] [BUG] recent rakudo changes have broken NCI
# New Ticket Created by Donald Hunter # Please include the string: [perl #62244] # in the subject line of all future correspondence about this issue. # URL: http://rt.perl.org/rt3/Ticket/Display.html?id=62244 It appears that rakudo changes committed over the weekend have broken NCI. The last known working revision was #35300 for ext/SQLite3/t/test.p6 which is crashing on #35440. You'll need to patch ext/SQLite3/Makefile.PL so that it works, then you can run make test. I can still run PIR code that uses Sqlite3 so I'm pretty sure it's isolated to rakudo. Cheers, Donald. Makefile.PL.patch Description: Binary data
Re: [perl #62244] AutoReply: [BUG] recent rakudo changes have broken NCI
grind.pid12075 Description: Binary data
[perl #60338] [BUG] NCI segfaults with null pcc_params_signature
# New Ticket Created by Jeff Horwitz
# Please include the string: [perl #60338]
# in the subject line of all future correspondence about this issue.
# URL: http://rt.perl.org/rt3/Ticket/Display.html?id=60338
changes in r32203 cause mod_parrot to segfault, as the param signature is
an uninitialized string. specifically, the pcc_params_signature NCI
attribute is null. not sure if this corresponds to a specific parrot
test...
creating the signature as a constant string appears to cause this problem,
as removing PObj_constant_FLAG from src/pmc/nci.pmc:82 fixes the issue,
though that's most certainly not a solution.
chromatic thinks we need more COW: My theory is that the JIT STRING
conversion needs a Parrot_make_COW_reference.
tewk has a workaround: short-term comment tools/build/nativecall.pl:308
to disable jitted NCI
backtrace is included, now with more stack smashing!
Program received signal SIGSEGV, Segmentation fault.
0x40798304 in Parrot_init_arg_sig (interp_unused=0x81b7bf8, ctx=0x832fd78,
sig=0x0, ap=0x0, sti=0xb398) at src/inter_call.c:422
422 if (*sig) {
(gdb) bt
#0 0x40798304 in Parrot_init_arg_sig (interp_unused=0x81b7bf8,
ctx=0x832fd78,
sig=0x0, ap=0x0, sti=0xb398) at src/inter_call.c:422
#1 0x40797f6c in Parrot_init_arg_nci (interp=0x81b7bf8, st=0xb36c,
sig=0x0) at src/inter_call.c:276
#2 0x082f6451 in ?? ()
#3 0x081b7bf8 in ?? ()
#4 0xb36c in ?? ()
#5 0x in ?? ()
#6 0x08193458 in ?? ()
#7 0x08310418 in ?? ()
#8 0x0200 in ?? ()
#9 0x40cdd2d8 in ?? ()
#10 0x082dda10 in ?? ()
#11 0x0832fd78 in ?? ()
#12 0x0001 in ?? ()
#13 0x in ?? ()
#14 0x in ?? ()
#15 0x in ?? ()
#16 0x in ?? ()
#17 0x081b7bf8 in ?? ()
#18 0x08193458 in ?? ()
#19 0x0100 in ?? ()
#20 0x407a0bc3 in key_next (interp=0x81b7bf8, key=0x82e1f4c) at
src/key.c:480
#21 0x40938a6e in Parrot_NCI_invoke (interp=0x81b7bf8, pmc=0x82e1f4c,
next=0x40cdd2ec) at nci.pmc:303
#22 0x40717116 in Parrot_invokecc_p (cur_opcode=0x40cdd2e4,
interp=0x81b7bf8)
at core.ops:475
#23 0x407cd83c in runops_slow_core (interp=0x81b7bf8, pc=0x40cdd2e4)
at src/runops_cores.c:222
#24 0x4079edb9 in runops_int (interp=0x81b7bf8, offset=80)
at src/interpreter.c:938
#25 0x4079f65f in runops (interp=0x81b7bf8, offs=80) at
src/inter_run.c:101
#26 0x4079f904 in runops_args (interp=0x81b7bf8, sub=0x82dd7fc,
obj=0x8328fd4,
meth_unused=0x826cc5c, sig=0x40ad50e5 v, ap=0xb594 \2172\b)
at src/inter_run.c:236
#27 0x4079fc96 in Parrot_run_meth_fromc_args (interp=0x81b7bf8,
sub=0x82dd7fc,
obj=0x8328fd4, meth=0x826cc5c, sig=0x40ad50e5 v) at
src/inter_run.c:426
#28 0x409a9042 in initialize_parents (interp=0x81b7bf8, object=0x8328fd4,
all_parents=0x8329108) at class.pmc:344
#29 0x409aaf7c in Parrot_Class_instantiate (interp=0x81b7bf8,
pmc=0x832923c,
init=0x82c8028) at class.pmc:1131
#30 0x40736446 in Parrot_new_p_pc (cur_opcode=0x40c981b0,
interp=0x81b7bf8)
at pmc.ops:130
#31 0x407cd83c in runops_slow_core (interp=0x81b7bf8, pc=0x40c981b0)
at src/runops_cores.c:222
#32 0x4079edb9 in runops_int (interp=0x81b7bf8, offset=2101)
at src/interpreter.c:938
#33 0x4079f65f in runops (interp=0x81b7bf8, offs=2101) at
src/inter_run.c:101
#34 0x4079f904 in runops_args (interp=0x81b7bf8, sub=0x82e17f8,
obj=0x82c8028,
meth_unused=0x0, sig=0x4043b4d5 IP,
ap=0xb7ec
\2242\b-\btz,\b[EMAIL PROTECTED]@[EMAIL PROTECTED][EMAIL PROTECTED]()
at src/inter_run.c:236
#35 0x4079fe56 in Parrot_runops_fromc_arglist_reti (interp=0x81b7bf8,
sub=0x82e17f8, sig=0x4043b4d5 IP,
args=0xb7ec
\2242\b-\btz,\b[EMAIL PROTECTED]@[EMAIL PROTECTED][EMAIL PROTECTED]()
at src/inter_run.c:527
#36 0x4078995c in Parrot_call_sub_ret_int (interp=0x81b7bf8,
sub=0x82e17f8,
signature=0x4043b4d5 IP) at src/extend.c:929
#37 0x4043834b in modparrot_call_sub_IP (interp=0x81b7bf8,
namespace=0x82301d0 PIR, name=0x4043af49
post_read_request_handler,
ret=0xb878, pmc=0x83294a4) at src/parrot_util.c:167
#38 0x40435a17 in modparrot_call_meta_handler (interp=0x81b7bf8,
hll=0x82301d0 PIR, hook=0x4043af49 post_read_request_handler,
ret=0xb878) at src/mod_parrot.c:233
#39 0x40435e91 in modparrot_meta_post_read_request_handler (r=0x83cab60)
at src/mod_parrot.c:382
[perl #58438] [PATCH] nci can't pass NULL string arguments
No objections and no problems, closing ticket.
Re: [perl #58438] [PATCH] nci can't pass NULL string arguments
Hearing no objections, and because I needed it to be able to do tests with mysqlclient, applied in r30790 -- Salu2
[perl #58438] [PATCH] nci can't pass NULL string arguments
# New Ticket Created by NotFound
# Please include the string: [perl #58438]
# in the subject line of all future correspondence about this issue.
# URL: http://rt.perl.org/rt3/Ticket/Display.html?id=58438
I was doing a simple test of NCI calls with the xlib function
XDisplayName, and found that nci can't handle a NULL string argument.
Passing a NULL Parrot String segfaults.
The attached patch passes a NULL char * when a NULL Parrot String is
used. Don't know if this is the desired behaviour, I expect comments.
--
Salu2
Index: tools/build/nativecall.pl
===
--- tools/build/nativecall.pl (revision 30618)
+++ tools/build/nativecall.pl (working copy)
@@ -505,8 +505,8 @@
/t/ do {
push @{$temps_ref}, char *t_$temp_num;;
push @{$extra_preamble_ref},
-t_$temp_num = string_to_cstring(interp, GET_NCI_S($reg_num));;
-push @{$extra_postamble_ref}, string_cstring_free(t_$temp_num);;
+{STRING * s= GET_NCI_S($reg_num); t_$temp_num = s ? string_to_cstring(interp, s) : (char *) NULL;};
+push @{$extra_postamble_ref}, do { if (t_$temp_num) string_cstring_free(t_$temp_num); } while (0);;
return t_$temp_num;
};
/b/ do {
NCI and Calling Conventions (esp. on Windows)
AFAICT, Parrot uses function pointers for NCI. This means NCI uses whatever calling convention the compiler uses by default. Unfortunately, there's More Than One Way To Do It. On Windows, there's the C calling convention (__cdecl), which is usually used by default by the Visual C++ compiler. There's also the standard (__stdcall) and fast (__fastcall) calling convention. I haven't seen any use of fastcall, but stdcall is used by the Win32 API. Using the wrong calling convention most certainly blows the stack. I think we need a way to select the calling convention for a function, similar to, or maybe even part of, the signature. Also, it would be good to have a way to select a calling convention when loading a library, as a calling convention is usually used consistently, and providing defaults for well known libraries. Ron
Re: NCI and Calling Conventions (esp. on Windows)
On Wed, 2008-08-20 at 22:20 +0200, Ron Blaschke wrote: I think we need a way to select the calling convention for a function, similar to, or maybe even part of, the signature. Also, it would be good to have a way to select a calling convention when loading a library, as a calling convention is usually used consistently, and providing defaults for well known libraries. tewk's C99 parser / NCI JIT code should handle part of this (and could be expanded to do more). We may want to just settle on extending that work as needed, rather than trying to shoehorn it into old-style NCI. -'f
Re: [perl #53406] [TODO] Add 'long long' to types supported by NCI
On Sunday 27 April 2008 18:08:08 Mark Glines wrote: In the future, INTVAL will probably be 128 bits for some platforms. I'd really like to see a set of fixed-width types (similar to what p5 has for pack and unpack), so you have the option of native int or exactly 32 bits or whatever you need. HUGEINTVAL is 64 bits on both linux-x86 and linux-amd64, so it seems like a good start. Here's an ugly patch that attempts to add it. Problems with this patch: 1. It probably won't work on win32-x64... I seem to recall someone mentioning HUGEINTVAL is just 32 bits there. 2. The patch adds a HUGEINTVAL to the PObj union, and would probably require additional fixups all over parrot to be able to access the resulting object. The above makes me think we might be better off using a BigInt object for this data. But anyway, here's what I got. How much work is it to go the BigInt route? Just an estimate is fine right now, unless it's so easy that you can make that patch right away. -- c
[perl #53406] [TODO] Add 'long long' to types supported by NCI
# New Ticket Created by Geoffrey Broadwell # Please include the string: [perl #53406] # in the subject line of all future correspondence about this issue. # URL: http://rt.perl.org/rt3/Ticket/Display.html?id=53406 At least MySQL and OpenGL have function prototypes either taking or returning Clong long or Cunsigned long long. There is no way to correctly handle this with Parrot's NCI signature types. Currently both MySQL and OpenGL are just substituting Clong instead, which will work on 64-bit architectures, but fail miserably on 32-bit arches. (I wish we could just insist that INTVAL be 64 bits everywhere, but alas that doesn't make a lot of sense for very memory-constrained environments. Though those do get rarer by the day ) -'f
Re: [perl #53406] [TODO] Add 'long long' to types supported by NCI
On Sat, 26 Apr 2008 19:18:21 -0700
Geoffrey Broadwell (via RT) [EMAIL PROTECTED] wrote:
# New Ticket Created by Geoffrey Broadwell
# Please include the string: [perl #53406]
# in the subject line of all future correspondence about this issue.
# URL: http://rt.perl.org/rt3/Ticket/Display.html?id=53406
(I wish we could just insist that INTVAL be 64 bits everywhere, but
alas that doesn't make a lot of sense for very memory-constrained
environments. Though those do get rarer by the day )
In the future, INTVAL will probably be 128 bits for some platforms.
I'd really like to see a set of fixed-width types (similar to what p5
has for pack and unpack), so you have the option of native int or
exactly 32 bits or whatever you need.
HUGEINTVAL is 64 bits on both linux-x86 and linux-amd64, so it
seems like a good start. Here's an ugly patch that attempts to add it.
Problems with this patch:
1. It probably won't work on win32-x64... I seem to recall someone
mentioning HUGEINTVAL is just 32 bits there.
2. The patch adds a HUGEINTVAL to the PObj union, and would probably
require additional fixups all over parrot to be able to access the
resulting object.
The above makes me think we might be better off using a BigInt object
for this data. But anyway, here's what I got.
Mark
Index: src/call_list.txt
===
--- src/call_list.txt (revision 27216)
+++ src/call_list.txt (working copy)
@@ -13,6 +13,7 @@
# s - short
# i - int
# l - long
+# L - long long
# NUM register stuff
# N - FLOATVAL
@@ -34,6 +35,7 @@
# 2 - pointer to short
# 3 - pointer to int
# 4 - pointer to long
+# 5 - pointer to long long
# void stuff
# v - void
Index: tools/build/nativecall.pl
===
--- tools/build/nativecall.pl (revision 27216)
+++ tools/build/nativecall.pl (working copy)
@@ -43,6 +43,8 @@
3 = int *,
l = long,
4 = long *,
+L = long long,
+5 = long long *,
c = char,
s = short,
2 = short *,
@@ -65,6 +67,8 @@
3 = int *,
l = long,
4 = long *,
+L = long long,
+5 = long long *,
c = char,
s = short,
2 = short *,
@@ -105,6 +109,8 @@
3 = int *,
l = long,
4 = long *,
+L = long long,
+5 = long long *,
c = char,
s = short,
2 = short *,
@@ -126,8 +132,10 @@
i = set_nci_I(interp, st, return_data);,
I = set_nci_I(interp, st, return_data);,
l = set_nci_I(interp, st, return_data);,
+L = set_nci_L(interp, st, return_data);,
s = set_nci_I(interp, st, return_data);,
c = set_nci_I(interp, st, return_data);,
+5 = set_nci_L(interp, st, *return_data);,
4 = set_nci_I(interp, st, *return_data);,
3 = set_nci_I(interp, st, *return_data);,
2 = set_nci_I(interp, st, *return_data);,
@@ -147,10 +155,12 @@
my %func_call_assign = (
p = return_data = ,
i = return_data = ,
+2 = return_data = ,
3 = return_data = ,
-2 = return_data = ,
4 = return_data = ,
+5 = return_data = ,
l = return_data = ,
+L = return_data = ,
c = return_data = ,
s = return_data = ,
f = return_data = ,
@@ -173,6 +183,7 @@
2 = P,
4 = P,
l = I,
+L = L,
c = I,
s = I,
f = N,
@@ -320,6 +331,18 @@
return UVal_int(st-val);
}
+static HUGEINTVAL
+get_nci_L(PARROT_INTERP, ARGMOD(call_state *st), int n)
+{
+if (n = st-src.n) {
+real_exception(interp, NULL, E_ValueError,
+too few arguments passed to NCI function);
+}
+Parrot_fetch_arg_nci(interp, st);
+
+return UVal_hugeint(st-val);
+}
+
static FLOATVAL
get_nci_N(PARROT_INTERP, ARGMOD(call_state *st), int n)
{
@@ -367,6 +390,7 @@
#define GET_NCI_I(n) get_nci_I(interp, st, n)
#define GET_NCI_S(n) get_nci_S(interp, st, n)
#define GET_NCI_N(n) get_nci_N(interp, st, n)
+#define GET_NCI_L(n) get_nci_L(interp, st, n)
#define GET_NCI_P(n) get_nci_P(interp, st, n)
/*
@@ -384,6 +408,17 @@
}
static void
+set_nci_L(PARROT_INTERP, ARGOUT(call_state *st), HUGEINTVAL val)
+{
+Parrot_init_ret_nci(interp, st, L);
+if (st-dest.i st-dest.n) {
+UVal_hugeint(st-val) = val;
+Parrot_convert_arg(interp, st);
+Parrot_store_arg(interp, st);
+}
+}
+
+static void
set_nci_N(PARROT_INTERP, ARGOUT(call_state *st), FLOATVAL val)
{
Parrot_init_ret_nci(interp, st, N);
@@ -454,6 +489,11 @@
push @{$extra_preamble_ref}, t_$temp_num =
(long)GET_NCI_I($reg_num);;
return t_$temp_num;
};
+/L/ do {
+push @{$temps_ref}, long long t_$temp_num;;
+push @{$extra_preamble_ref}, t_$temp_num = (long
long)GET_NCI_I($reg_num);;
+return t_$temp_num;
+};
/I/ do {
push @{$temps_ref}, INTVAL t_$temp_num;;
push @{$extra_preamble_ref}, t_$temp_num = GET_NCI_I($reg_num
Re: [perl #52916] [PATCH] Making NCI test C++ compatible
On Sat, 19 Apr 2008 11:26:13 +0530 Senaka Fernando [EMAIL PROTECTED] wrote: Hi Mark, What does the fix to 't/codingstd/c_indent.t' do? Does it ignore anything inside #ifdef __cplusplus blocks? Yes. This lets us add any g++-specific stuff without throwing off the indentation. Mark
Re: [perl #52916] [PATCH] Making NCI test C++ compatible
Hi Mark,
What does the fix to 't/codingstd/c_indent.t' do? Does it ignore anything
inside #ifdef __cplusplus blocks?
Regards,
Senaka
On Sat, Apr 19, 2008 at 6:38 AM, Mark Glines via RT
[EMAIL PROTECTED] wrote:
On Thu Apr 17 09:03:57 2008, infinoid wrote:
On Tue Apr 15 02:58:18 2008, [EMAIL PROTECTED] wrote:
+extern C
+{
This patch causes a failure in t/codingstd/c_indent.t. Otherwise, I
think your patch is the right fix for this.
Checked in as r27025, with a hack to c_indent.t to ignore __cplusplus
preprocessor clauses.
NCI questions
I've been working on parsing the function prototypes in the GL/GLU/GLUT/GLX headers, in preparation for generating bindings for all of them during the Parrot build. Currently my prototype parser is able to compute NCI signatures for all but 14 out of the 1835 total prototypes found in my system's headers. Those 14 failures break down as follows: 5 Parrot NCI doesn't handle 'long long', only 'long long *' 4 Callback registration functions neither Parrot nor I handle yet 3 GetProcAddress-type functions; not sure how to deal with these 2 SGI-specific (IRIS?) headers required, which I don't have -- 14 TOTAL The second group I'll handle soon, and the fourth group I'll just ignore for now. That leaves the first and third groups, for which I could use some advice: 1. Any suggestions on a sane way to handle 'long long' NCI arguments, when Parrot could be built with either 32 or 64 bit INTVAL? 2. How do you create an NCI Sub (like the dlfunc op does) using a function pointer returned anonymously by a function, instead of by name using the normal dlfunc op? Also, I'm having to squash away signed/unsigned distinctions for non-pointer NCI arguments with integer types, because current Parrot NCI always forces integer types to signed. Any way to lift this limitation? -'f
[perl #52916] [PATCH] Making NCI test C++ compatible
# New Ticket Created by Senaka Fernando
# Please include the string: [perl #52916]
# in the subject line of all future correspondence about this issue.
# URL: http://rt.perl.org/rt3/Ticket/Display.html?id=52916
Index: src/nci_test.c
===
--- src/nci_test.c (revision 26966)
+++ src/nci_test.c (working copy)
@@ -37,6 +37,12 @@
#include stdlib.h
#include parrot/config.h
+#ifdef __cplusplus
+extern C
+{
+#endif
+
+
/* Declarations of structs */
typedef struct Nested {
@@ -1070,6 +1076,10 @@
#endif
+#ifdef __cplusplus
+}
+#endif
+
/*
=back
Hi all,
The provided patch makes the NCI test C++ compatible.
Regards,
Senaka
Portable OpenGL via Parrot - was: Extending Parrot NCI callback functionality Options
Cool - great to hear from you - sorry that it's been a while since I've posted an update! It'd be great to have you participate - you've done quite a lot for POGL. Regarding your work/ideas on abstracting GLUT - definitely want to leverage that - I'll let you know as soon as I'm ready to tackle this. As for automating support for new OpenGL extensions, I wrote a perl script back when I first started this project that parses glext.h to generate glext_procs.h - which helps abstract WGL and GLX implementations. Attempting to fully automate it may be problematic - particularly for APIs involving pointers. Worth investigating, tho. A related task that I've been contemplating is to support OpenGL extensions dynamically (like WGL) - so that a binary build can be distributed for a given distro - rather than having to be built from source. POGL's binary distros for Windows (PPMs) works on any Windows box with an OpenGL driver (standard on NT/XP/Vista) - this doesn't work for other platforms. BTW - since we're talking about doing a better Perl OpenGL implementation - it has always bugged me that the Perl APIs involving pointers have a _c suffix. I get the _s and _p suffixes, but my preference would be to get rid of the _c suffix. Thoughts? Thanks! - Bob - Original Message - From: Geoffrey Broadwell [EMAIL PROTECTED] To: Grafman [EMAIL PROTECTED] Cc: [EMAIL PROTECTED]; [EMAIL PROTECTED] Sent: Wednesday, March 26, 2008 3:44 PM Subject: Re: Extending Parrot NCI callback functionality Options On Wed, 2008-03-26 at 09:31 -0700, Grafman wrote: He lives! Just kidding, I know you've had actual paid work going on. :-) Hi chromatic - as you know, I took over the Perl OpenGL project over a year ago - you had mentioned that I might consider doing a port to Parrot; Geoffrey had suggested the same. FWIW, I still think that's a good path. My time comes in short, widely spaced bursts -- it's much better to have someone who can be doing more steady work running the whole thing. I can then jump in here and there with patches, whenever I am able. I've also been wanting to rework POGL's architecture to make it more object-oriented for server- side use. I was hoping that OpenGL 3 would be out about 6 months ago and give us a clear ARB-approved set of classes to build on, but the ARB has become a black hole of late (and there are persistent rumors that certain very large companies have been actively blockading release of the spec). Thus, probably best to work from the best info we have about their intended design, and expect to be making some changes when OpenGL 3 is finally specced. While OpenGL itself does not use callbacks, it is assumed to be single- threaded (there's a current context state). Well, at least that there is only one thread doing rendering; there may freely be other threads doing non-rendering tasks. It has been suggested that one might fix Parrot's NCI to support generic callbacks. There's a fundamental flaw in going down this path: the fix would essentially make any Parrot module using this generic callback method single-threaded - which would make it unusable on most modern language bindings - thereby invalidating the point of using Parrot in the first place. There is only one Parrot async scheduler thread, correct? Wouldn't it be safe to direct all implicitely single-threaded bindings through the async scheduler thread? Or has it been too long since I read that PDD? The better solution would be to make the underlying C libs (like GLUT) thread-safe by extending their callback interfaces. You can't fix the GLUT callback issue without addressing OpenGL's current context interface. Since I was already planning to make POGL object-oriented, I had intended to do a callback wrapper for GLUT anyway. I had finally been convinced by others this was the right thing to do anyway, since the generic callback mechanism was not likely coming any time soon. I just hadn't had the tuits yet. :-) In my mind, this is the right way to address this problem. If no one has any objections (and I'm not stepping on any toes), I'd like to drive this project - just so that we don't have duplication of effort. Otherwise, I'm open to other suggestions. Please do. Like I said, I'd rather be patching than in charge. FWIW, I had written some proof of concept code that might help you get up to speed. I've also got some code to hide the nastiness needed to handle toolkits (like GLUT) that want to preparse the command line arguments at init time and hide toolkit-specific args from the calling program. Let me know if you want a copy of any of this. Also, I've been thinking about the problem of staying up to date and complete with the current OpenGL version and all extensions. I did some investigating on CPAN and found a couple different ways to handle auto-wrapping C headers, including one I found very interesting that deeply separated the concepts of parsing the C
Extending Parrot NCI callback functionality Options
I'm not even sure *that* will work. To invoke a Sub PMC from C, you need to pass in an Interp as well as the PMC. Unless you know both of those at compile time, I'm not sure how to make the callback mechanism work. ... although I just had an evil idea regarding memcpy and a hash table. Hi chromatic - as you know, I took over the Perl OpenGL project over a year ago - you had mentioned that I might consider doing a port to Parrot; Geoffrey had suggested the same. I've also been wanting to rework POGL's architecture to make it more object-oriented for server- side use. While OpenGL itself does not use callbacks, it is assumed to be single- threaded (there's a current context state). As such, GLUT was designed to assume single-threaded uses - which is why it's callbacks do not have user data. It has been suggested that one might fix Parrot's NCI to support generic callbacks. There's a fundamental flaw in going down this path: the fix would essentially make any Parrot module using this generic callback method single-threaded - which would make it unusable on most modern language bindings - thereby invalidating the point of using Parrot in the first place. The better solution would be to make the underlying C libs (like GLUT) thread-safe by extending their callback interfaces. You can't fix the GLUT callback issue without addressing OpenGL's current context interface. Since I was already planning to make POGL object-oriented, I had intended to do a callback wrapper for GLUT anyway. In my mind, this is the right way to address this problem. If no one has any objections (and I'm not stepping on any toes), I'd like to drive this project - just so that we don't have duplication of effort. Otherwise, I'm open to other suggestions. cheers - grafman
Re: Extending Parrot NCI callback functionality Options
On Wed, 2008-03-26 at 09:31 -0700, Grafman wrote:
He lives! Just kidding, I know you've had actual paid work going
on. :-)
Hi chromatic - as you know, I took over the Perl OpenGL project over a
year ago - you had mentioned that I might consider doing a port to
Parrot; Geoffrey had suggested the same.
FWIW, I still think that's a good path. My time comes in short, widely
spaced bursts -- it's much better to have someone who can be doing more
steady work running the whole thing. I can then jump in here and there
with patches, whenever I am able.
I've also been wanting to
rework POGL's architecture to make it more object-oriented for server-
side use.
I was hoping that OpenGL 3 would be out about 6 months ago and give us a
clear ARB-approved set of classes to build on, but the ARB has become a
black hole of late (and there are persistent rumors that certain very
large companies have been actively blockading release of the spec).
Thus, probably best to work from the best info we have about their
intended design, and expect to be making some changes when OpenGL 3 is
finally specced.
While OpenGL itself does not use callbacks, it is assumed to be single-
threaded (there's a current context state).
Well, at least that there is only one thread doing rendering; there may
freely be other threads doing non-rendering tasks.
It has been suggested that one might fix Parrot's NCI to support
generic callbacks. There's a fundamental flaw in going down this
path: the fix would essentially make any Parrot module using this
generic callback method single-threaded - which would make it
unusable on most modern language bindings - thereby invalidating the
point of using Parrot in the first place.
There is only one Parrot async scheduler thread, correct? Wouldn't it
be safe to direct all implicitely single-threaded bindings through the
async scheduler thread? Or has it been too long since I read that PDD?
The better solution would be to make the underlying C libs (like GLUT)
thread-safe by extending their callback interfaces. You can't fix the
GLUT callback issue without addressing OpenGL's current context
interface. Since I was already planning to make POGL object-oriented,
I had intended to do a callback wrapper for GLUT anyway.
I had finally been convinced by others this was the right thing to do
anyway, since the generic callback mechanism was not likely coming any
time soon. I just hadn't had the tuits yet. :-)
In my mind, this is the right way to address this problem. If no one
has any objections (and I'm not stepping on any toes), I'd like to
drive this project - just so that we don't have duplication of
effort. Otherwise, I'm open to other suggestions.
Please do. Like I said, I'd rather be patching than in charge. FWIW, I
had written some proof of concept code that might help you get up to
speed. I've also got some code to hide the nastiness needed to handle
toolkits (like GLUT) that want to preparse the command line arguments at
init time and hide toolkit-specific args from the calling program. Let
me know if you want a copy of any of this.
Also, I've been thinking about the problem of staying up to date and
complete with the current OpenGL version and all extensions. I did some
investigating on CPAN and found a couple different ways to handle
auto-wrapping C headers, including one I found very interesting that
deeply separated the concepts of parsing the C headers from generating
the wrapper code.
The basic idea is that a dedicated C header parser converts from all the
crazy nasty hacks found in normal C headers to canonicalized data about
structures, enums, defines, and prototypes. It then spits out some
intermediate files that are both vastly easier to parse and easy for
humans to modify. (This latter point is key, as I'll point out in a
minute.) The auto-wrapper then reads one or more intermediate files and
produces wrappers from it.
My immediate thought when I saw this was to make sure that re-running
the C header parser would not mess with human-made changes. This could
probably be most easily done by saying that the header parser generates
one set of files ('gl*.auto' for instance), and humans edit another set
('gl*.delta' or what have you) containing only overrides. These
overrides could mark something to be removed, or wrapped with a slightly
different set of parameters, or to do a typemapping, or even to add new
convenience APIs. We might even support overrides expressed as Perl
code, that do things like convert any single function that expects a
pointer to a big array of floats into multiple versions, one taking a
handle to a raw memory buffer, one taking a PMC number array, another a
HLL list, etc.
In any case, having this sort of two-pass wrapping with a nice powerful
hook in the middle to add human intelligence seems like a good way to
get 90+% of the core and extension APIs handled cleanly. Any remainders
can still be hand coded
Re: Extending Parrot NCI callback functionality Options
As far as I understand OpenGL, it's got one current context per thread, and libGL does all sorts of evil things with threads and thread-local storage to make it all work transparently. An object-oriented OpenGL interface seems like the right way to go, though, for all sorts of other reasons. Arcady On Wed, Mar 26, 2008 at 12:31 PM, Grafman [EMAIL PROTECTED] wrote: I'm not even sure *that* will work. To invoke a Sub PMC from C, you need to pass in an Interp as well as the PMC. Unless you know both of those at compile time, I'm not sure how to make the callback mechanism work. ... although I just had an evil idea regarding memcpy and a hash table. Hi chromatic - as you know, I took over the Perl OpenGL project over a year ago - you had mentioned that I might consider doing a port to Parrot; Geoffrey had suggested the same. I've also been wanting to rework POGL's architecture to make it more object-oriented for server- side use. While OpenGL itself does not use callbacks, it is assumed to be single- threaded (there's a current context state). As such, GLUT was designed to assume single-threaded uses - which is why it's callbacks do not have user data. It has been suggested that one might fix Parrot's NCI to support generic callbacks. There's a fundamental flaw in going down this path: the fix would essentially make any Parrot module using this generic callback method single-threaded - which would make it unusable on most modern language bindings - thereby invalidating the point of using Parrot in the first place. The better solution would be to make the underlying C libs (like GLUT) thread-safe by extending their callback interfaces. You can't fix the GLUT callback issue without addressing OpenGL's current context interface. Since I was already planning to make POGL object-oriented, I had intended to do a callback wrapper for GLUT anyway. In my mind, this is the right way to address this problem. If no one has any objections (and I'm not stepping on any toes), I'd like to drive this project - just so that we don't have duplication of effort. Otherwise, I'm open to other suggestions. cheers - grafman
Re: Portable OpenGL via Parrot - was: Extending Parrot NCI callback functionality Options
On Wed, 2008-03-26 at 16:27 -0700, Bob Free wrote: Cool - great to hear from you - sorry that it's been a while since I've posted an update! Real life can be so darned intrusive ;-) It'd be great to have you participate - you've done quite a lot for POGL. Regarding your work/ideas on abstracting GLUT - definitely want to leverage that - I'll let you know as soon as I'm ready to tackle this. Fair enough. As for automating support for new OpenGL extensions, I wrote a perl script back when I first started this project that parses glext.h to generate glext_procs.h - which helps abstract WGL and GLX implementations. Attempting to fully automate it may be problematic - particularly for APIs involving pointers. Worth investigating, tho. Definitely. Would you mind sending that over? I can poke and prod at it a bit, see what I can manage. A related task that I've been contemplating is to support OpenGL extensions dynamically (like WGL) - so that a binary build can be distributed for a given distro - rather than having to be built from source. POGL's binary distros for Windows (PPMs) works on any Windows box with an OpenGL driver (standard on NT/XP/Vista) - this doesn't work for other platforms. I keep thinking we should just do runtime stub patching everywhere, as with GLee/GLEW/SDL/etc. The nice thing is that this can be done cleanly in Parrot, because setting up a Parrot NCI sub involves doing dlfunc calls anyway [*] -- and Parrot's core handles making the 'dlfunc' op do the right thing on every platform. [*] Unless you manually create a library wrapper in C using Parrot's C API -- which *might* produce very slightly faster code (debateable, in the face of JIT for PIR code), but which seems to be leaving us back in the hellish XS place again. BTW - since we're talking about doing a better Perl OpenGL implementation - it has always bugged me that the Perl APIs involving pointers have a _c suffix. I get the _s and _p suffixes, but my preference would be to get rid of the _c suffix. Oh certainly, those are not only ugly, but won't be mnemonic for all Parrot languages. Since Parrot very deeply has multiple dispatch, we may not even need to have different names at all -- they may just properly dispatch based on arguments. I have no experience with the performance of Parrot's multiple dispatch -- if it proves to be slow, we can revisit the suffices, but definitely choose better ones. Or we can present the evidence to the core Parrot team and ask them to make multiple dispatch faster, which ought to help the performance of several high level languages as a bonus. ;-) -'f
[perl #50360] Redesign Parrot NCI callback functionality
# New Ticket Created by Geoffrey Broadwell # Please include the string: [perl #50360] # in the subject line of all future correspondence about this issue. # URL: http://rt.perl.org/rt3/Ticket/Display.html?id=50360 Parrot's NCI callback subsystem design will not work (and cannot be directly extended) for callbacks that do not carry an opaque user data pointer. Many existing callback-based APIs do not fit this description, including certain GUI toolkits such as GLUT. Allison and others have brainstormed a few possible ideas, but none was fleshed out to the point of being implementable. The following threads on the parrot-porters mailing list covered part of this: http://tinyurl.com/3crzpu http://tinyurl.com/3886qw Also, I've attached some IRC discussion of this (slightly edited as per participant's request). -'f irc.perl.org-#parrot.log Description: Binary data
Re: Extending Parrot NCI callback functionality
chromatic wrote: On Friday 18 January 2008 20:25:16 Allison Randal wrote: It's true that the generalized solution for varying callback signatures doesn't exist yet, but it's easy enough to create your own C callback layer, with a separate C function for each callback you need. (Note, not one-per-signature, but one-per-callback.) The callback functions can invoke a Parrot sub that they lookup by name, register a callback sub with the concurrency scheduler, or simply directly perform the actions needed. I'm not even sure *that* will work. To invoke a Sub PMC from C, you need to pass in an Interp as well as the PMC. Unless you know both of those at compile time, I'm not sure how to make the callback mechanism work. The concurrency scheduler stores the interp, so it's not a problem for invoking the PIR callback Sub. But, the current way of looking up the concurrency scheduler is through the interpreter, so that just pushes the problem back a step. Really, any scheduler will do, since it can pass a message to all the other schedulers. So, the C callback function can just grab the 0th interpreter from the global interp array and send a callback event to that scheduler. The actual PIR Sub for the callback can be registered as a handler in the scheduler, and invoked by the scheduler as a handler for the callback event. Allison
Re: Crazy NCI Callback Solution Idea
chromatic wrote: Here's my crazy idea. When you register a callback into PIR from C, you need to pass a few arguments: the interpreter into which to call, the PMC Sub to call, and the C signature of the C function which gets called from C. I presume that the PIR function will get called through one of the external API calls which need to know the interpreter, the PMC to invoke, and the C signature of the incoming arguments. The callback registry stores this information in a hash (or array) or other registry and... Replace callback registry with concurrency scheduler and we're on the same page. .. looks up a function pointer in a table of precompiled function pointers keyed off of C signatures. These functions call a return a varargs pointer from my arguments function, then call the invoke registered PIR callback function with the varargs and an identifier. (I'll get to where the identifier comes from in a moment.) Instead of returning that function pointer directly, the registry copies it (we have to know its size somehow, but I think we can guess pretty well) and changes one spot which records its identifier. It's an integer, probably unsigned. Relying on a JIT might make this easier, but I think we can use this scheme with a fair degree of ease. We do need to make sure that the copied functions are on executable and writeable memory pages, but otherwise I think we're fine. I have doubts about how stable/cross-platform compatible any solution that involves rewriting parts of a function pointer in memory will be. It's likely to cause problems on exactly the same platforms that have trouble with the JIT solution. Between the two, the JIT solution seems preferable. But, feel free to prototype it. Also, no, I never wrote viruses. Why do you ask? Stack smasher, Allison
Re: Extending Parrot NCI callback functionality
On Jan 16, 2008, at 7:39 PM, Geoffrey Broadwell wrote:
I am starting to implement a GLUT and OpenGL binding for Parrot. GLUT
is extremely callback-oriented.
Unfortunately, none of the GLUT callbacks fall within the current
limitations on Parrot NCI callbacks.
As you've discovered, callbacks are easy enough to implement portably
provided that an opaque pointer-sized value is one of the arguments
-- you register a single C function, which uses information stored in
the opaque structure to dispatch to the real handler -- and it's
unfortunate that GLUT's callbacks don't support this.
So ... how can Parrot support callbacks of the types that GLUT uses?
The trick of putting magical properties on a special user data
parameter
won't work anymore. I've been tanking on this for a while, and
come up
with several possible schemes.
My first idea was that each time a new callback was registered, Parrot
would copy a tiny shim function and then poke the address(es) of the
data it needed to figure out what PIR routine to call directly into
the
copied shim code.
This requires special knowledge per platform.
So my next idea was to have a shim function that did nothing but call
another function, thus putting its own address on the stack. The
other
function would then reach up above its own stack frame and grab that
return address, using *that address* to look up the needed information
in a global callback registry. By making copies of this simple shim
every time a new callback was registered, each copy would get its own
address, and everything would magically work.
This was certainly better than the previous ideas, because it didn't
involve poking anything into the shim code, just being able to copy
it.
But it still requires per-platform support, at least for retrieving
the stack frame pointer.
But I still don't know how to do that in pure portable C. My guess is
that you could declare another function (which need not do
anything) in
some special way so that it is guaranteed to be compiled into the
object
code directly after the magic shim. Subtracting the address of the
shim
function from the dummy function would then tell you exactly how many
bytes long the compiled shim is (including perhaps some padding, which
doesn't matter). This would allow you to copy the shim at will.
I've seen this done (so it 'works') but the underlying assumption
isn't portable.
But this still leaves the problems of:
A. Being able to portably force two functions two be compiled
sequentially in memory. I would hope this is easy, but I don't
recall which set of declarations makes exactly this guarantee.
B. Being able to execute a copy of the shim on architectures that have
non-executable stack and/or heap. I assume this had to be solved
before JIT could work, so I'm guessing this isn't a real problem.
C. Being able to reach up above one's own stack frame and grab the
return address from the shim function's dummy call. Is there a
trick to make this work in pure portable C?
No. What you're trying to do can't be done portably. Period.
However, once you place per-platform implementation on the table, it
becomes simple (in theory, at least). Rather than teach Parrot to
cope with GLUT callbacks, I suggest a scheme for creating 'GLUT-
prime' callbacks which accept the opaque structure pointer you're
familiar with, which Parrot can already deal with.
For example: Here's a callback interface that includes the opaque
handle:
void generic_handler( void* user_data );
struct foo foo_data;
install_generic_handler( generic_handler, foo_data );
Here's one without:
struct bar bar_data;
typedef void (*handler)();
handler bar_handler = new_handler( generic_handler, bar_data );
install_handler( bar_handler );
The new_handler() function creates a thunk that loads the data
argument and jumps to the generic handler. In 68K, the thunk code
might (if it were written out) look like:
asm void thunk_zero_args()
{
MOVEA.L #$11223344,A0; // load immediate handler address into
A0
MOVE.L #$55667788,-(SP); // load immediate data pointer onto
stack
JMP (A0)
}
asm void thunk_one_arg( void* arg1 )
{
MOVEA.L #$11223344,A0; // load immediate handler address into
A0
MOVE.L #$55667788,-(SP); // load immediate data pointer onto
stack
MOVE.L 8(SP),-(SP); // load previous arg 1 onto stack
JMP (A0)
}
asm void thunk_two_args( void* arg1, void* arg2 )
{
MOVEA.L #$11223344,A0; // load immediate handler address into
A0
MOVE.L #$55667788,-(SP); // load immediate data pointer onto
stack
MOVE.L 12(SP),-(SP); // load previous arg 2 onto stack
Re: Extending Parrot NCI callback functionality
Geoffrey Broadwell wrote: On Wed, 2008-01-16 at 22:38 -0700, Paul Seamons wrote: I am starting to implement a GLUT and OpenGL binding for Parrot. [...] I don't often get concentrated time to help out with Parrot and Perl 6, but I have some now. If this is going to be blocked indefinitely, my tuits are likely to evaporate. And dangit, I don't want to miss out on my window of opportunity! It's true that the generalized solution for varying callback signatures doesn't exist yet, but it's easy enough to create your own C callback layer, with a separate C function for each callback you need. (Note, not one-per-signature, but one-per-callback.) The callback functions can invoke a Parrot sub that they lookup by name, register a callback sub with the concurrency scheduler, or simply directly perform the actions needed. Allison
Re: Extending Parrot NCI callback functionality
On Sat, 2008-01-19 at 17:25 +1300, Allison Randal wrote: Geoffrey Broadwell wrote: On Wed, 2008-01-16 at 22:38 -0700, Paul Seamons wrote: I am starting to implement a GLUT and OpenGL binding for Parrot. [...] I don't often get concentrated time to help out with Parrot and Perl 6, but I have some now. If this is going to be blocked indefinitely, my tuits are likely to evaporate. And dangit, I don't want to miss out on my window of opportunity! It's true that the generalized solution for varying callback signatures doesn't exist yet, but it's easy enough to create your own C callback layer, with a separate C function for each callback you need. (Note, not one-per-signature, but one-per-callback.) The callback functions can invoke a Parrot sub that they lookup by name, register a callback sub with the concurrency scheduler, or simply directly perform the actions needed. Barring some better option coming up in the mean time, I will probably end up doing this. At the very least, I'll get more exposure to Parrot's guts this way. Still, I'll also file an RT ticket asking for the general solution under the twice requested rule. I'll reference this thread, Paul's old thread, and our IRC conversation in the ticket. -'f
Re: Extending Parrot NCI callback functionality
On Friday 18 January 2008 20:25:16 Allison Randal wrote: It's true that the generalized solution for varying callback signatures doesn't exist yet, but it's easy enough to create your own C callback layer, with a separate C function for each callback you need. (Note, not one-per-signature, but one-per-callback.) The callback functions can invoke a Parrot sub that they lookup by name, register a callback sub with the concurrency scheduler, or simply directly perform the actions needed. I'm not even sure *that* will work. To invoke a Sub PMC from C, you need to pass in an Interp as well as the PMC. Unless you know both of those at compile time, I'm not sure how to make the callback mechanism work. ... although I just had an evil idea regarding memcpy and a hash table. -- c
Crazy NCI Callback Solution Idea
Here's my crazy idea. When you register a callback into PIR from C, you need to pass a few arguments: the interpreter into which to call, the PMC Sub to call, and the C signature of the C function which gets called from C. I presume that the PIR function will get called through one of the external API calls which need to know the interpreter, the PMC to invoke, and the C signature of the incoming arguments. The callback registry stores this information in a hash (or array) or other registry and... ... looks up a function pointer in a table of precompiled function pointers keyed off of C signatures. These functions call a return a varargs pointer from my arguments function, then call the invoke registered PIR callback function with the varargs and an identifier. (I'll get to where the identifier comes from in a moment.) Instead of returning that function pointer directly, the registry copies it (we have to know its size somehow, but I think we can guess pretty well) and changes one spot which records its identifier. It's an integer, probably unsigned. Relying on a JIT might make this easier, but I think we can use this scheme with a fair degree of ease. We do need to make sure that the copied functions are on executable and writeable memory pages, but otherwise I think we're fine. Also, no, I never wrote viruses. Why do you ask? -- c
Re: Extending Parrot NCI callback functionality
I am starting to implement a GLUT and OpenGL binding for Parrot. GLUT is extremely callback-oriented. You essentially can't do anything beyond open the top-level window until callbacks are functional. You can't even draw in that window -- the rendered image never actually appears, and in fact the window itself is simply transparent with just the chrome showing. I started down this path several months ago. The following is the thread on the topic. http://tinyurl.com/3crzpu In that short thread, Allison Randall responded with: The immediate solution, to get OpenGL working now without waiting for the implementation of the concurrency scheduler and JITed call/callback thunks, is to add a few more callback signatures to the current set of alternatives, and to write a little bit of custom C code for the cases that can't pass dispatch information in a user data argument. It appears that NCI is on the roadmap a couple of months out (or more), but that when it is done it will be done right - for some definition of right. I would've put more work into a temporary solution, but real life has interfered. I'm glad to see other people interested in OpenGL on parrot. When the NCI PDD is up for drafting, it will be good to have a listing of sample libraries and required callback signatures available. I'm excited about the progress Parrot is making. Paul
Re: Extending Parrot NCI callback functionality
On Wed, 2008-01-16 at 22:38 -0700, Paul Seamons wrote: I am starting to implement a GLUT and OpenGL binding for Parrot. I started down this path several months ago. The following is the thread on the topic. http://tinyurl.com/3crzpu OK, read it now. I think I got a little farther, but I'm blocked now just as you were, on the same problem. It appears that NCI is on the roadmap a couple of months out (or more), but that when it is done it will be done right - for some definition of right. I think that Allison hand-waved over exactly the case that you and I tried to address first -- no JIT available, and want to create the shims in portable C instead of hand-coded assembly per platform. I'll try to take a look at the other PDDs she mentioned in a day or two, but from what she wrote in your thread, I didn't get the feeling her solution quite covered the same ground. I would've put more work into a temporary solution, but real life has interfered. Yes, and that's exactly my worry. I don't often get concentrated time to help out with Parrot and Perl 6, but I have some now. If this is going to be blocked indefinitely, my tuits are likely to evaporate. And dangit, I don't want to miss out on my window of opportunity! I'm glad to see other people interested in OpenGL on parrot. When the NCI PDD is up for drafting, it will be good to have a listing of sample libraries and required callback signatures available. I had already made a list of all of the callback signatures needed by GLUT, which I've included below. I'm excited about the progress Parrot is making. Ditto that! I'm hoping that when this blocking issue is dealt with, I'll be able to create a fully functioning OpenGL binding, which is actually for me a bit of yak shaving -- what I really *want* to do is port some of my OpenGL code to Perl 6, but I need a working binding first. :-) -'f *** =head1 Needed Callback Signatures XXX: Currently Parrot does not support the callback signatures needed by GLUT to function properly (at all, really). The following are the missing signatures: =over 4 =item 'v' CglutDisplayFunc, CglutOverlayDisplayFunc, CglutIdleFunc, CglutCloseFunc, CglutMenuDestroyFunc, CglutWMCloseFunc(*) =item 'vi' CglutEntryFunc, CglutTimerFunc, CglutWindowStatusFunc, CglutVisibilityFunc(*), CglutMenuStateFunc(*) =item 'vii' CglutReshapeFunc, CglutMotionFunc, CglutPassiveMotionFunc, CglutSpaceballButtonFunc(!), CglutButtonBoxFunc(!), CglutDialsFunc(!), CglutTabletMotionFunc(!) =item 'vcii' CglutKeyboardFunc, CglutKeyboardUpFunc =item 'viii' CglutSpecialFunc, CglutSpecialUpFunc, CglutMenuStatusFunc, CglutSpaceballMotionFunc(!), CglutSpaceballRotateFunc(!) =item 'v' CglutMouseFunc, CglutMouseWheelFunc, CglutJoystickFunc, CglutTabletButtonFunc(!) =back Special marks are as follows: =over 4 =item (*) Function is deprecated in OpenGLUT in favor of a more modern callback. =item (!) Function is unimplemented in freeglut and OpenGLUT (these all support rarer input devices; in many cases, the input device in question is SGI-specific and most likely no longer manufactured). =back =cut
Extending Parrot NCI callback functionality
Right now, Parrot's support for NCI callbacks (C code calling back into PIR code) is relatively limited. In particular, there are at least the following limitations: 1. The return type must be void (C library does not expect a response). 2. The callback must have exactly two arguments (from both the C and PIR perspectives). 3. One of these arguments must be a user data pointer, provided by the PIR code when it registers the callback with the C library, and handed back by the C code to the PIR code during every callback. 4. The C code must treat the user data pointer as entirely opaque (no modification or assumed structure). 5. The other argument's type can vary, but is much more limited than an arbitrary (normal) NCI argument. I am starting to implement a GLUT and OpenGL binding for Parrot. GLUT is extremely callback-oriented. You essentially can't do anything beyond open the top-level window until callbacks are functional. You can't even draw in that window -- the rendered image never actually appears, and in fact the window itself is simply transparent with just the chrome showing. Unfortunately, none of the GLUT callbacks fall within the current limitations on Parrot NCI callbacks. I was thus beginning to look at improving the Parrot NCI code accordingly, but soon realized I didn't know how to get around some pretty fundamental problems (at least, not in pure C code). It turns out there's a good reason that limitations #3 and #4 exist. What actually gets registered with the external C library is a pointer to a C function in src/inter_cb.c. This function calls a chain of other functions. First, they check that the user data argument has not been obviously corrupted, and that the interpreter state is still sane. Next, they pull the Sub PMC and signature for the PIR function that should be called *from properties of the user data argument*. A little more sanity checking and a switch statement later, and this PIR Sub is called using Parrot_runops_fromc_args_event(). And therein lies the problem. The callback entry point is compiled C code. There could be dozens of callbacks registered by the PIR code, but only *one* set of C functions that handles all of them. In order to have a single compiled Parrot function that brokers all these different callbacks from C to PIR, Parrot cheats. Parrot adds several properties to the user data parameter before handing it off to the C library, and when the C library calls back, Parrot reads these properties to figure out which PIR function should get called. But this cheat doesn't work if the C library's callbacks don't all support a completely opaque user data parameter. And guess what? Not a single one of the GLUT callbacks is of this form. In fact, several of the most important GLUT callbacks have *no parameters at all*, and all of the callback parameters have defined meanings on the C side. So ... how can Parrot support callbacks of the types that GLUT uses? The trick of putting magical properties on a special user data parameter won't work anymore. I've been tanking on this for a while, and come up with several possible schemes. My first idea was that each time a new callback was registered, Parrot would copy a tiny shim function and then poke the address(es) of the data it needed to figure out what PIR routine to call directly into the copied shim code. The shim copy could then be handed off to the C library callback registration. When a callback came, the shim copy would then magically call the right PIR function, since it had all the information it needed hardcoded. A variant of this involved having a global registry of callbacks, so that instead of poking a pile of data into the copied shim, Parrot would simply need to poke in the callback number (one integer). When the callback came in, the shim copy would then use this single integer to look up the info it needed in the global registry. But I don't know how to portably do either kind of compiled code poking in pure C. (Not that I know it can't be done -- I just personally don't know how.) So my next idea was to have a shim function that did nothing but call another function, thus putting its own address on the stack. The other function would then reach up above its own stack frame and grab that return address, using *that address* to look up the needed information in a global callback registry. By making copies of this simple shim every time a new callback was registered, each copy would get its own address, and everything would magically work. This was certainly better than the previous ideas, because it didn't involve poking anything into the shim code, just being able to copy it. But I still don't know how to do that in pure portable C. My guess is that you could declare another function (which need not do anything) in some special way so that it is guaranteed to be compiled into the object code directly after the magic shim. Subtracting the address
Re: A quick NCI question
Sorry, essentially what I'm asking is if i can pass a pir method as a function pointer on a c call. essentially what I'm looking to do is get into the message pump on windows, and I'm trying to keep all of the code within pir. I'm trying to see how many hoops I need to do in order to write a gui setup using coroutines from pir alone. chromatic wrote: On Monday 07 January 2008 09:22:43 robby wrote: With skimming across the past 400 or so messages on this list I'm not exactly sure if this would be the correct list to post this, but I have a quick question regarding NCI's and callbacks. I've read http://www.parrotcode.org/docs/pdd/pdd16_native_call.htm and nci.t and I haven't came up with the answer off hand, so I may be missing something, or it may not be possible in pir. I have a function I'm wanting to call that is long func(long, long, long, callback) and a callback with a sig (long, long, long, long). While doing callbacks, do I *have* to create a wrapper in C in order to use them? In the draft doc noted above it says However, many callback functions share a common signature, and parrot provides some ready-made functions for this purpose that should serve for most of the callback uses. But I think I may be taking it out of context. I'm trying to tap into the win32 procs and don't know if I can keep it all in pir (which is the goal, because I'd rather port bytecode than c code across platforms) Am I reading the docs wrong, or is there no functionality to do what I'm looking at? As I understand it, the discussion of callbacks in the docs is about callbacks implemented in PIR. The callback you describe looks like a C function, in which case you should be able to get a pointer to it and pass it as a pointer. Apologies if I misunderstood your question though. -- c
A quick NCI question
With skimming across the past 400 or so messages on this list I'm not exactly sure if this would be the correct list to post this, but I have a quick question regarding NCI's and callbacks. I've read http://www.parrotcode.org/docs/pdd/pdd16_native_call.htm and nci.t and I haven't came up with the answer off hand, so I may be missing something, or it may not be possible in pir. I have a function I'm wanting to call that is long func(long, long, long, callback) and a callback with a sig (long, long, long, long). While doing callbacks, do I *have* to create a wrapper in C in order to use them? In the draft doc noted above it says However, many callback functions share a common signature, and parrot provides some ready-made functions for this purpose that should serve for most of the callback uses. But I think I may be taking it out of context. I'm trying to tap into the win32 procs and don't know if I can keep it all in pir (which is the goal, because I'd rather port bytecode than c code across platforms) Am I reading the docs wrong, or is there no functionality to do what I'm looking at? thanks
Re: A quick NCI question
On Monday 07 January 2008 09:22:43 robby wrote: With skimming across the past 400 or so messages on this list I'm not exactly sure if this would be the correct list to post this, but I have a quick question regarding NCI's and callbacks. I've read http://www.parrotcode.org/docs/pdd/pdd16_native_call.htm and nci.t and I haven't came up with the answer off hand, so I may be missing something, or it may not be possible in pir. I have a function I'm wanting to call that is long func(long, long, long, callback) and a callback with a sig (long, long, long, long). While doing callbacks, do I *have* to create a wrapper in C in order to use them? In the draft doc noted above it says However, many callback functions share a common signature, and parrot provides some ready-made functions for this purpose that should serve for most of the callback uses. But I think I may be taking it out of context. I'm trying to tap into the win32 procs and don't know if I can keep it all in pir (which is the goal, because I'd rather port bytecode than c code across platforms) Am I reading the docs wrong, or is there no functionality to do what I'm looking at? As I understand it, the discussion of callbacks in the docs is about callbacks implemented in PIR. The callback you describe looks like a C function, in which case you should be able to get a pointer to it and pass it as a pointer. Apologies if I misunderstood your question though. -- c
Re: NCI PMC comments
Jonathan Worthington wrote: Hi, At the top of the NCI PMC, there are these comments: -- Invoking an NCI function changes some registers according to PDD 3. The caller has to preserve registers if needed. -- Am I right in thinking that's no longer true? Yes, PDD 3 no longer uses specific registers for the calling conventions. Allison
Re: NCI PMC comments
These statements are removed in r23981. kjs On Dec 16, 2007 4:16 PM, Allison Randal [EMAIL PROTECTED] wrote: Jonathan Worthington wrote: Hi, At the top of the NCI PMC, there are these comments: -- Invoking an NCI function changes some registers according to PDD 3. The caller has to preserve registers if needed. -- Am I right in thinking that's no longer true? Yes, PDD 3 no longer uses specific registers for the calling conventions. Allison
NCI PMC comments
Hi, At the top of the NCI PMC, there are these comments: -- Invoking an NCI function changes some registers according to PDD 3. The caller has to preserve registers if needed. -- Am I right in thinking that's no longer true? Thanks, Jonathan
Re: [DOCS] NCI
On Nov 26, 7:42 am, [EMAIL PROTECTED] (Will Coleda) wrote: # New Ticket Created by Will Coleda # Please include the string: [perl #47826] # in the subject line of all future correspondence about this issue. # URL:http://rt.perl.org/rt3/Ticket/Display.html?id=47826 The NCI information in docs/books is out of date. per codyl on #parrot, the code doesn't compile. The best link so far seems to be at the perlreview in a pay-for article; Need to update the docs for NCI in the repository to be usable. -- Will Coke Coleda [EMAIL PROTECTED] The example in PDD16 - NCI is up-to-date. kjs
[perl #47826] [DOCS] NCI
# New Ticket Created by Will Coleda # Please include the string: [perl #47826] # in the subject line of all future correspondence about this issue. # URL: http://rt.perl.org/rt3/Ticket/Display.html?id=47826 The NCI information in docs/books is out of date. per codyl on #parrot, the code doesn't compile. The best link so far seems to be at the perlreview in a pay-for article; Need to update the docs for NCI in the repository to be usable. -- Will Coke Coleda [EMAIL PROTECTED]
Generic callback mechanism in NCI
I started to write an OpenGL library and was only a couple of dozen lines into the pir when I remembered the documentation about callbacks in docs/pdds/draft/pdd16_native_call.pod . Currently there are only two signatures supported on callbacks: one with user_data and extern_data and the other with extern_data and user_data (the positions are all that is different). These are nice and general functions and work well because the user_data is/should be opaque to the calling C function, the user_data provides a place to store the PMC, and the user_data is used to lookup the interpreter that should be used to run the sub stored in the PMC. The pdd says that outside the two provided signatures, anybody wanting to implement NCI connections to callback functions will need to do some hand C coding. Hand C coding isn't at all bad, but it would be nice to have a generic mechanism for storing callbacks from C functions that don't provide a slot for opaque user_data. I was on my way to coding a solution with my meager C skills and wanted to make sure that what I was doing was sane (for some definition of the word sane). So my proposal goes something like this: - Get a new unique id from a new op called get_callback_id - Call a variant of the new_callback op, passing in the unique id - The new_callback variant stores the user data under the unique id - The callback references a unique C function that will return that unique id - When the callback is fired, the unique id is used to lookup the user_data - Any external data is parsed as needed - The registered sub is played Ok, sounds hard enough - but it gets worse. Here is how I would implement this generic callback system and what I think would be required. There will probably need to a file similar to src/call_list.txt such as src/callback_list.txt with entries similar to: 10 v v 10 v ii Where the first number is the number of functions to pre-create, and the other two parts are the return type and signature. I'd rather not have the number be hardcoded and go with a jit solution or some other solution for runtime C function generation - but I'm not aware of anything else that would work besides pre-compiled C functions. I think it would be nice if libraries could generate/compile the needed functions independent of anything hardcoded (and that goes for call_list.txt too for that matter). The entires in callback_list.txt would generate functions similar to the following in nci.c # parrot callback functions void pcbf_v_JV_0(void) # depending up signature void pcbf_v_JV_1(void) ... void pcbf_v_JV_9(void) I would then add two more ops (I shudder to say that - I'm not sure if adding ops is a frowned upon thing). Those ops are (I haven't played with it to know if this is the right format): op get_callback_id(out INT, in STR) # the output is a unique INT for the given signature in STR # it would fail if called more times than the 10 listed # in callback_list.txt (unless a jit solution could be implemented) op delete_callback_id(in INT, in STR) # deletes the user_data in the storage structure, freeing it # for later use Currently the following op is defined in src/ops/core.ops: op new_callback(out PMC, invar PMC, invar PMC, in STR) I want to add one more variant of this op op new_callback(out PMC, in INT, invar PMC, invar PNC, in STR) Another piece that is required is that there be a global ResizablePMCArray (actually there may be multiple based on the registered signatures). Again, I can hear everybody shuddering at the mention of a global. I don't know enought about parrot yet to know if it should be a true global variable, or if it should be tied to an interpreter, or if there is somewhere I should register the PMC with, or if there is already a central structure that would take care of functions like this. My questions are: - Does anybody else want a generic callback function mechanism in NCI? - Is this a relatively sane/clean way to do it? - Is there a better way to generate the functions for each signature? - What is the right way to store that global user_data until the callbacks are fired? I don't think I've been clear enough, but I'll post and then answer questions. I think that it would be good to have something that libraries could use without having to drop down to the C level - I just am not sure if this is too much bloat to implement it. Paul Seamons I've been here all along - I'm just really quiet.
Re: Generic callback mechanism in NCI
[EMAIL PROTECTED] wrote: I started to write an OpenGL library and was only a couple of dozen lines into the pir when I remembered the documentation about callbacks in docs/pdds/draft/pdd16_native_call.pod . [...] My questions are: - Does anybody else want a generic callback function mechanism in NCI? - Is this a relatively sane/clean way to do it? - Is there a better way to generate the functions for each signature? - What is the right way to store that global user_data until the callbacks are fired? NCI isn't fully specified yet, but I'll outline some of our current thoughts and likely directions. Discussion welcome. We would like to eliminate the massive list of precompiled thunks for C function calls generated by call_list.txt. The tricky part is that you can't compile a C function at runtime. The best you can do is JIT it, and that depends on having a working JIT for the platform. We might be able to use LLVM's JIT, which would gain us a working JIT on a number of platforms. The JIT solution would continue to use the dlfunc interface, but instead of looking up a precompiled thunking function for the passed in signature, it would JIT one. As with the precompiled thunk, the JITed thunk is incorporated into an NCI sub object, which can be stored in a namespace, or passed around anonymously, and invoked, just like an ordinary sub object. That said, it's unlikely that we'll ever completely eliminate the list of precompiled thunks. Some platforms just won't have a JIT, and we can't afford to cut off NCI for the lack of a JIT. (For one thing, Parrot internals use NCI quite heavily.) But we can make them more manageable. We'll probably end up with something similar to src/ops/*.ops, with multiple files of signatures. The core file would be the absolute minimum required to run Parrot without loading external C libraries. Then we could add a file for each subsystem (MySQL, Postgres, pcre, SDL, Python builtins, and tcl, are a few already mentioned in call_list.txt). A configuration option when compiling Parrot could decide whether to precompile a restricted set, the full set, or add in some additional signatures for another external system. Some signature files for a particular library could be generated by a limited C parser, but it would always need to be checked by a human. Duplicate signatures between files would be merged. (And remember, this is only a fallback for platforms that can't JIT.) So, that's one part of the question. The other part is callbacks. If/when we develop a JIT solution for NCI call thunks, we can use the same technique for generating callback thunks. In the mean time, we'll have to continue with the precompiled callback thunks. Callbacks will use the concurrency scheduler. (There's some info about the concurrency scheduler in the new Events PDD, but more details will be in the Concurrency PDD next month.) For the moment all you need to know is that the concurrency scheduler is a central dispatcher in the Parrot interpreter that handles events, exceptions, async I/O, threads, etc. When you call 'new_callback', you pass it a Parrot sub, a user_data argument, and a signature. At the moment the signature is limited to two alternatives, in the future it will allow all the same signature options as an NCI call. The signature of the Parrot sub should match the signature passed into 'new_callback'. The 'new_callback' op will take these arguments and create a CallbackHandler PMC which stores the user data and the passed in sub, as well as any other information needed to properly invoke the right sub in the right interpreter. It then registers that CallbackHandler with the concurrency scheduler. Registering a callback handler returns a unique concurrency id (CID) for the handler, kind of like a process id. After registering the callback handler, 'new_callback' will look up a precompiled thunk or JIT a thunk with the requested signature. For the JITed thunk it will embed the CID as a constant within the thunk. For precompiled thunks, we're probably going to have to add more information to the CallbackHandler PMC (the signature, possibly a library identifier, etc). When the callback thunk is called from C, it bundles the C arguments into appropriate Parrot arguments, then notifies the concurrency scheduler that it has a callback with a particular CID, or particular set of characteristics (similar to scheduling an event of a particular type). The concurrency scheduler will look through its registered callback handlers to look for a matching handler, and if it finds one, invoke it, passing it the arguments that were passed to the C thunk. Essentially, this uses the concurrency scheduler as your global data store and as the source of unique identifiers. But, it's integrated with a core system. The immediate solution, to get OpenGL working now without waiting for the implementation of the concurrency scheduler
Generic callback mechanism in NCI
I started to write an OpenGL library and was only a couple of dozen lines into the pir when I remembered the documentation about callbacks in docs/pdds/draft/pdd16_native_call.pod . Currently there are only two signatures supported on callbacks: one with user_data and extern_data and the other with extern_data and user_data (the positions are all that is different). These are nice and general functions and work well because the user_data is/should be opaque to the calling C function, the user_data provides a place to store the PMC, and the user_data is used to lookup the interpreter that should be used to run the sub stored in the PMC. The pdd says that outside the two provided signatures, anybody wanting to implement NCI connections to callback functions will need to do some hand C coding. Hand C coding isn't at all bad, but it would be nice to have a generic mechanism for storing callbacks from C functions that don't provide a slot for opaque user_data. I was on my way to coding a solution with my meager C skills and wanted to make sure that what I was doing was sane (for some definition of the word sane). So my proposal goes something like this: - Get a new unique id from a new op called get_callback_id - Call a variant of the new_callback op, passing in the unique id - The new_callback variant stores the user data under the unique id - The callback references a unique C function that will return that unique id - When the callback is fired, the unique id is used to lookup the user_data - Any external data is parsed as needed - The registered sub is played Ok, sounds hard enough - but it gets worse. Here is how I would implement this generic callback system and what I think would be required. There will probably need to a file similar to src/call_list.txt such as src/callback_list.txt with entries similar to: 10 v v 10 v ii Where the first number is the number of functions to pre-create, and the other two parts are the return type and signature. I'd rather not have the number be hardcoded and go with a jit solution or some other solution for runtime C function generation - but I'm not aware of anything else that would work besides pre-compiled C functions. I think it would be nice if libraries could generate/compile the needed functions independent of anything hardcoded (and that goes for call_list.txt too for that matter). The entires in callback_list.txt would generate functions similar to the following in nci.c # parrot callback functions void pcbf_v_JV_0(void) # depending up signature void pcbf_v_JV_1(void) ... void pcbf_v_JV_9(void) I would then add two more ops (I shudder to say that - I'm not sure if adding ops is a frowned upon thing). Those ops are (I haven't played with it to know if this is the right format): op get_callback_id(out INT, in STR) # the output is a unique INT for the given signature in STR # it would fail if called more times than the 10 listed # in callback_list.txt (unless a jit solution could be implemented) op delete_callback_id(in INT, in STR) # deletes the user_data in the storage structure, freeing it # for later use Currently the following op is defined in src/ops/core.ops: op new_callback(out PMC, invar PMC, invar PMC, in STR) I want to add one more variant of this op op new_callback(out PMC, in INT, invar PMC, invar PNC, in STR) Another piece that is required is that there be a global ResizablePMCArray (actually there may be multiple based on the registered signatures). Again, I can hear everybody shuddering at the mention of a global. I don't know enought about parrot yet to know if it should be a true global variable, or if it should be tied to an interpreter, or if there is somewhere I should register the PMC with, or if there is already a central structure that would take care of functions like this. My questions are: - Does anybody else want a generic callback function mechanism in NCI? - Is this a relatively sane/clean way to do it? - Is there a better way to generate the functions for each signature? - What is the right way to store that global user_data until the callbacks are fired? I don't think I've been clear enough, but I'll post and then answer questions. I think that it would be good to have something that libraries could use without having to drop down to the C level - I just am not sure if this is too much bloat to implement it. Paul Seamons I've been here all along - I'm just really quiet.
Re: [svn ci] NCI methods now name-mangled
On Apr 5, 2007, at 5:45 PM, Leopold Toetsch wrote: Am Donnerstag, 5. April 2007 00:39 schrieb Jonathan Worthington: Don't really need a policy to tell me that breaking stuff for languages folks sucks. :-) I try hard to avoid it, but unfortunately stuff slips through the net occasionally. In this case, I wasn't even aware that you could use SELF.meth(...) to call non-vtable methods, and it's done nowhere in core PMCs or I'd have noticed it. This could be also read as a notice to language maintainers: If you are using some features of parrot, then please, please submit a core test for it, if such a test doesn't exist yet. This will help Parrot and You in the future ... Thanks. leo Not only to language maintainers, but anyone using parrot where a feature isn't tested. And it also technically falls on whoever implemented said feature in parrot, for not adding a test. What if we had a repository, ala pugs with it's open commits, solely for people to commit tests. It could help improve bug discovery and test coverage, as well as ambiguity about features in parrot. Then developers could just update it and run it seperately(and check it to make sure nothing malicious gets through which is always a potential of course).
Ease of Committing Tests (was Re: [svn ci] NCI methods now name-mangled)
On Friday 06 April 2007 00:58, Joshua Isom wrote: What if we had a repository, ala pugs with it's open commits, solely for people to commit tests. It could help improve bug discovery and test coverage, as well as ambiguity about features in parrot. Then developers could just update it and run it seperately(and check it to make sure nothing malicious gets through which is always a potential of course). Is that a big convenience over mailing patches to the list or nopasting them for IRC such that people who won't do either of those will happily commit them? Are the specifications good and complete enough that tests can be unambiguously correct? Who will merge these tests into the standard tree? Just looking for more information, -- c
Re: Ease of Committing Tests (was Re: [svn ci] NCI methods now name-mangled)
On Apr 6, 2007, at 11:48 AM, chromatic wrote: On Friday 06 April 2007 00:58, Joshua Isom wrote: What if we had a repository, ala pugs with it's open commits, solely for people to commit tests. It could help improve bug discovery and test coverage, as well as ambiguity about features in parrot. Then developers could just update it and run it seperately(and check it to make sure nothing malicious gets through which is always a potential of course). Is that a big convenience over mailing patches to the list or nopasting them for IRC such that people who won't do either of those will happily commit them? It would add a convenience if we had a web form that just listed what testing method, the input, and the output for example. If it's promoted on the main page or somewhere on the site, then someone wouldn't have the join the list(which they may not want to do for some reason), or they may not have an irc client. I never really used an irc client before I started joining #parrot. I was working with parrot a couple months before I ever joined. Are the specifications good and complete enough that tests can be unambiguously correct? Reading over a pdd isn't the same as writing code based on that pdd. Something the ambiguoity isn't intentional and was just missed. Plus if they're random user tests, we should consider that it might not be accurate. Who will merge these tests into the standard tree? Who merges patches sent to the list into the standard tree? In my experience, it's almost anyone. Just looking for more information, -- c
Re: [svn ci] NCI methods now name-mangled
Am Donnerstag, 5. April 2007 00:39 schrieb Jonathan Worthington: Don't really need a policy to tell me that breaking stuff for languages folks sucks. :-) I try hard to avoid it, but unfortunately stuff slips through the net occasionally. In this case, I wasn't even aware that you could use SELF.meth(...) to call non-vtable methods, and it's done nowhere in core PMCs or I'd have noticed it. This could be also read as a notice to language maintainers: If you are using some features of parrot, then please, please submit a core test for it, if such a test doesn't exist yet. This will help Parrot and You in the future ... Thanks. leo
Re: [svn ci] NCI methods now name-mangled
At 01:33 04/04/2007 +0100, Jonathan Worthington wrote:
Hi,
I was working on starting to move class functionality into v-table
methods, as discussed on list recently. However, I hit an interesting
issue - you cannot have a METHOD or PCCMETHOD with the same name as a
vtable method. We need to be able to do that to implement the interface
specified in PDD15, though.
As of r17967, instead of turning any non-vtable method to:
Parrot_CLASSNAME_METHODNAME
Which could conflict with the vtable method, pmc2c now generates:
Parrot_CLASSNAME_nci_METHODNAME
I had to fix a couple of bits of code that directly called such methods,
but some of them had this is evil and needs fixing comments above them
anyway. (Since calling a method directly like this ignores inheritance,
it's generally the wrong thing to do.) The Complex PMC was the only PMC
that needed changing to get Parrot building again after the change.
Therefore I expect the impact of this change will be small.
Anyway, hope this is agreeable, and please do report any issues.
This new behavior breaks the build of Lua PMC.
in languages/lua/pmc/luastring.pmc
#line 295
PMC* add (PMC* value, PMC* dest) {
MMD_LuaNumber: {
PMC* n = SELF.tonumber();
if (n-vtable-base_type == dynpmc_LuaNumber) {
in languages/lua/pmc/pmc_luastring.h (generated correct)
#line 109
PARROT_DYNEXT_EXPORT extern PMC* Parrot_LuaString_nci_tonumber(Interp *, PMC*);
in languages/lua/pmc/luastring.c (generated NOT CORRECT)
#line 172 luastring.c
PMC*
Parrot_LuaString_add_LuaNumber(Interp *interp, PMC* pmc, PMC* value, PMC* dest)
{
PMC* n = Parrot_LuaString_tonumber(interp, pmc);// need
_nci_ !!!
if (n-vtable-base_type == dynpmc_LuaNumber) {
Regards.
François.
Thanks,
Jonathan
Re: [svn ci] NCI methods now name-mangled
François PERRAD wrote: Anyway, hope this is agreeable, and please do report any issues. This new behavior breaks the build of Lua PMC. For me too, even after a 'make realclean' on Parrot. In the interests of our developing policy on a stable trunk: Jonathan, fix the problem for Lua or revert the change before continuing with further development. Thanks! Allison
Re: [svn ci] NCI methods now name-mangled
Jonathan Worthington wrote: Hi, I was working on starting to move class functionality into v-table methods, as discussed on list recently. However, I hit an interesting issue - you cannot have a METHOD or PCCMETHOD with the same name as a vtable method. We need to be able to do that to implement the interface specified in PDD15, though. As of r17967, instead of turning any non-vtable method to: Parrot_CLASSNAME_METHODNAME Which could conflict with the vtable method, pmc2c now generates: Parrot_CLASSNAME_nci_METHODNAME This is a leftover from the old days when the way to override a vtable method was to define a method with an '__' name, so they never conflicted. I expect this assumption will need to be rooted out in several other places as well, so its worth a thorough review for the PMC PDD. This is a good enough fix in the meantime. (After Lua is working.) Allison
Re: [svn ci] NCI methods now name-mangled
François PERRAD wrote: This new behavior breaks the build of Lua PMC. Argh, sorry. :-( Thanks for the detailed bug report - I know where the problem is, and am working on a fix. Jonathan
Re: [svn ci] NCI methods now name-mangled
Allison Randal wrote: For me too, even after a 'make realclean' on Parrot. In the interests of our developing policy on a stable trunk: Jonathan, fix the problem for Lua or revert the change before continuing with further development. Don't really need a policy to tell me that breaking stuff for languages folks sucks. :-) I try hard to avoid it, but unfortunately stuff slips through the net occasionally. In this case, I wasn't even aware that you could use SELF.meth(...) to call non-vtable methods, and it's done nowhere in core PMCs or I'd have noticed it. Anyway, fixed in r17982. May need a realclean - I had to do one anyway to run the buildtools tests. Thanks, Jonathan
Re: [svn ci] NCI methods now name-mangled
Allison Randal wrote: This is a leftover from the old days when the way to override a vtable method was to define a method with an '__' name, so they never conflicted. Not really - this was a conflict in the names of the methods at a C level. The '__' prefix was a PIR-level thing. In PIR we may have moved away from using name mangling to prevent the conflict, but in C that's probably all we've got at our disposal. Note that name of method at a C level and name of the method for method lookups do not have to be at all related. I expect this assumption will need to be rooted out in several other places as well, so its worth a thorough review for the PMC PDD. Sure, I'm sure there's more than one way to deal with this issue, but for now this works and unblocks stuff. Thanks, Jonathan
Re: [svn ci] NCI methods now name-mangled
Jonathan Worthington wrote: Anyway, fixed in r17982. May need a realclean - I had to do one anyway to run the buildtools tests. Awesome. Works for me even without realclean. Thanks! Allison
[svn ci] NCI methods now name-mangled
Hi, I was working on starting to move class functionality into v-table methods, as discussed on list recently. However, I hit an interesting issue - you cannot have a METHOD or PCCMETHOD with the same name as a vtable method. We need to be able to do that to implement the interface specified in PDD15, though. As of r17967, instead of turning any non-vtable method to: Parrot_CLASSNAME_METHODNAME Which could conflict with the vtable method, pmc2c now generates: Parrot_CLASSNAME_nci_METHODNAME I had to fix a couple of bits of code that directly called such methods, but some of them had this is evil and needs fixing comments above them anyway. (Since calling a method directly like this ignores inheritance, it's generally the wrong thing to do.) The Complex PMC was the only PMC that needed changing to get Parrot building again after the change. Therefore I expect the impact of this change will be small. Anyway, hope this is agreeable, and please do report any issues. Thanks, Jonathan
[PATCH] PDD16 NCI update with simple example
hi,
I've been playing with NCI calls and more fun (embedding a Parrot, that
runs a PIR program, which invokes a C function, that then invokes a PIR
callback function).
As a result, I added a simple example to PDD16. I didnt' put too much
work in it (there are many more places that could be improved), because
I didn't know if updating this doc is desirable at this moment (any big
changes pending?).
If desired, I'd be happy to do more updates. Please let me know.
At least there's full working code to do a simple NCI invocation.
regards,
klaas-jan
Index: docs/pdds/draft/pdd16_native_call.pod
===
--- docs/pdds/draft/pdd16_native_call.pod (revision 17146)
+++ docs/pdds/draft/pdd16_native_call.pod (working copy)
@@ -126,8 +126,48 @@
=head2 Examples
Most of the function parameters are reasonably self-evident. Some, however,
-merit additional explanation. The {{ ??? }}
+merit additional explanation.
+This section describes the simplest example for NCI possible. To every NCI
+invocation, there are two parts: the native function to be invoked, and the
+PIR code to do the invocation.
+
+First the native function, to be written in C.
+On Windows, it is necessary to do a DLL export specification of the NCI function:
+
+ /* foo.c */
+
+ /* specify the function prototype */
+
+ void foo(void);
+
+ /* or on Windows using Microsoft Visual Studio: */
+
+ __declspec(dllexport) void foo(void);
+
+ void foo(void) {
+printf(Hello Parrot!\n);
+ }
+
+Then, after having compiled the file as a shared library, the PIR code looks
+like this:
+
+ .sub main :main
+ .local pmc lib, func
+
+ # load the shared library
+ lib = loadlib hello # no extension, .so or .dll is assumed
+
+ # get a reference to the function from the library just
+ # loaded, called foo, and signature void (and no arguments)
+ func = dlfunc lib, foo, v
+
+ # invoke
+ func()
+
+ .end
+
+
=head2 Callbacks
Some libraries, particularly ones implementing more complex functionality such
@@ -224,9 +264,9 @@
Maintainer: Dan Sugalski
Class: Internals
PDD Number: 16
-Version: 1.1
+Version: 1.2
Status: Developing
-Last Modified: Oct 12, 2004
+Last Modified: Feb 23, 2007
PDD Format: 1
Language: English
@@ -234,6 +274,10 @@
=over 4
+=item version 1.2
+
+Updated with basic example.
+
=item version 1.1
Changed callback section to reflect current status.
@@ -248,6 +292,10 @@
=over 4
+=item version 1.2
+
+Updated with basic example.
+
=item version 1.1
Changed callback section to reflect current status.
Re: [PATCH] PDD16 NCI update with simple example
On Friday 23 February 2007 05:16, Klaas-Jan Stol wrote: As a result, I added a simple example to PDD16. I didnt' put too much work in it (there are many more places that could be improved), because I didn't know if updating this doc is desirable at this moment (any big changes pending?). If desired, I'd be happy to do more updates. Please let me know. At least there's full working code to do a simple NCI invocation. Please do. There are no big changes pending. Thanks, applied as r17164. -- c
[perl #40822] Pg NCI Test Makes Unportable Connection
# New Ticket Created by chromatic
# Please include the string: [perl #40822]
# in the subject line of all future correspondence about this issue.
# URL: http://rt.perl.org/rt3/Ticket/Display.html?id=40822
t/library/pg.t fails pretty miserably on this line:
con = cl.'connectdb'('') # assume table = user is present
I have PostgreSQL installed, but this assumption doesn't work for me. I've
checked in a skip for the remaining tests in this case, as there's no point
in running them. (They cause NCI assertion failures, as the NULL connection
isn't what NCI expects to pass to the Pg functions.)
This test needs another approach to run on multiple platforms successfully.
Setting an environment variable for user name and password might work better.
-- c
Re: NCI and libparrot Question
Am Freitag, 21. Juli 2006 07:53 schrieb chromatic: Should this code work? I think so. This can't work for several reasons: 1) the NCI call signatures don't match the C functions' args: .store_nci_func( 'const_string', 'ppt' ) should be: 'SIt' (return STRING*, interp C-cstring args) .store_nci_func( 'string_to_cstring', 'tpp' ) should be: tIS (return C-string, interp STRING args) (You could use 'pIt' and 'pIp' signatures too, then the STRING* result of the 'const_string' would be wrapped into a ManagedStruct PMC). 2) print does *not* take c-strings, but STRINGs, therefore: hello = string_to_cstring( interpreter, greet_string ) print hello is likely to just segfault. -- c leo
Re: NCI and libparrot Question
On Friday 21 July 2006 00:57, Leopold Toetsch wrote: This can't work for several reasons: 1) the NCI call signatures don't match the C functions' args: .store_nci_func( 'const_string', 'ppt' ) should be: 'SIt' (return STRING*, interp C-cstring args) .store_nci_func( 'string_to_cstring', 'tpp' ) should be: tIS (return C-string, interp STRING args) (You could use 'pIt' and 'pIp' signatures too, then the STRING* result of the 'const_string' would be wrapped into a ManagedStruct PMC). I don't see documentation for the S or I parameters anywhere in the NCI PDD. Is there a better place for them? 2) print does *not* take c-strings, but STRINGs, therefore: hello = string_to_cstring( interpreter, greet_string ) print hello is likely to just segfault. I did oversimplify the code somewhat. Here's what I really want to run. -- c
Re: NCI and libparrot Question
Am Freitag, 21. Juli 2006 20:08 schrieb chromatic: On Friday 21 July 2006 00:57, Leopold Toetsch wrote: (You could use 'pIt' and 'pIp' signatures too, then the STRING* result of the 'const_string' would be wrapped into a ManagedStruct PMC). I don't see documentation for the S or I parameters anywhere in the NCI PDD. Is there a better place for them? src/call_list.txt is probably the best place to look. And a good one to look into occasionally, I've mixed up 'I' and 'J' - sorry. I did oversimplify the code somewhat. Here's what I really want to run. Without futher looking - you have to replace a lot of 'p's (some structure) with 'J' (the Interpreter* interp structure). E.g. .store_nci_func( 'Parrot_readbc', 'ppt' ) = .store_nci_func( 'Parrot_readbc', 'pJt' ) You migh also grep for enter_nci_method in src/pmc/*.c - all METHODs in .pmc are converted to NCI calls in the .c files with appropriate signatures. -- c leo
Re: NCI and libparrot Question
On Friday 21 July 2006 11:43, Leopold Toetsch wrote: Without futher looking - you have to replace a lot of 'p's (some structure) with 'J' (the Interpreter* interp structure). E.g. .store_nci_func( 'Parrot_readbc', 'ppt' ) = .store_nci_func( 'Parrot_readbc', 'pJt' ) I'm not convinced; this works just fine from the Perl 5 and Ruby bindings. Most of that code uses the interpreter returned from Parrot_new(), rather than the current interpreter. (It only uses the current interpreter as the parent argument to Parrot_new()). You migh also grep for enter_nci_method in src/pmc/*.c - all METHODs in .pmc are converted to NCI calls in the .c files with appropriate signatures. That ought to work. -- c
NCI and libparrot Question
Should this code work? I think so. I don't think it's necessarily *good* practice, but I do think it should work. -- c
[perl #39771] [PATCH] missing signature for examples/nci/win32api.pir
# New Ticket Created by Rene Hangstrup Møller # Please include the string: [perl #39771] # in the subject line of all future correspondence about this issue. # URL: http://rt.perl.org/rt3/Ticket/Display.html?id=39771 The win32 example in parrot/examples/nci/win32api.pir fails because the signature llttl is not defined in call_list.txt. This patch add the signature to call_list.txt Modified files: src/call_list.txt Index: call_list.txt === --- call_list.txt (revision 13220) +++ call_list.txt (working copy) @@ -410,3 +410,6 @@ l pttl # unsigned long mysql_real_escape_string(MYSQL *mysql, char *to,const char *from, unsigned long length) v t# void mysql_debug(const char *debug) # --- end mysqlclient library --- + +# win32 +l lttl
[perl #39771] [PATCH] missing signature for examples/nci/win32api.pir
Thanks, appled as r13221
Re: NCI 'v' vs '' in function parameter signatures
On Friday 03 March 2006 02:41, Tim Bunce wrote: Any news on this? Is it okay? Should I send it via parrotbug? It looked good to me. I say check it in and see what the smokes do. -- c
Re: NCI 'v' vs '' in function parameter signatures
Any news on this? Is it okay? Should I send it via parrotbug? Tim. On Tue, Feb 28, 2006 at 03:36:20PM +, Tim Bunce wrote: On Tue, Feb 14, 2006 at 10:04:59PM +0100, Leopold Toetsch wrote: On Feb 14, 2006, at 18:29, Tim Bunce wrote: The runtime dlfunc code will need to be altered to normalize away the trailing v so old code won't break. Should it warn about that? Yes, a warning please. Here's the patch. - removes 'v' argument entries from src/call_list.txt - adds mysqlclient signatures to src/call_list.txt [*] - tweaks docs/pdds/clip/pdd16_native_call.pod to match - adds list of definition files used into generated nci.c - adds sanity checking of return and argument sig chars - adds compile time warning for deprecated 'v' argument - adds optional duplicate signature warning (disabled) - adds runtime warning for deprecated 'v' argument Tim. [*] I'm planning a followup patch that splits src/call_list.txt into multiple files in a subdirectory. The mysqlclient defs would then be in their own file. With the current arrangement no one can safely remove a signature because there's no indication of what that signature exists for. The same ncidef file could then be used for both tools/build/nativecall.pl and tools/util/ncidef2pasm.pl (which I also plan to work on). Any objections or comments? Index: src/call_list.txt === --- src/call_list.txt (revision 11741) +++ src/call_list.txt (working copy) @@ -49,14 +49,12 @@ c# t/pmc/nci.t cp cpi -cv d# t/pmc/nci.t dd dJOd # Parrot builtins I JOS S JOS # ParrotIO.readline -dv IJI # Parrot_is_char_* v JOSP # String.trans v JOS # String.reverse @@ -83,7 +81,6 @@ f# t/pmc/nci.t fff # t/pmc/nci.t fis -fv i ib# t/pmc/nci.t @@ -148,7 +145,6 @@ i it iti -iv i4 i4i i42p @@ -160,7 +156,6 @@ lpi lpii lp33l -lv l33l PJi # Needed for parrot threads @@ -185,10 +180,8 @@ pt ptpp pttt -pv s # t/pmc/nci.t -sv t # t/pmc/nci.t ti @@ -200,7 +193,6 @@ tt ttl4 tt4 -tv v vJiiip# examples/japh/japh11.pasm @@ -215,7 +207,6 @@ vp vpl vpp -vv # These are needed for parrotio.pmc iJP @@ -343,3 +334,77 @@ # Make lua stop panic'ing. P JOI + + +# --- start mysqlclient library --- +# Created from mysql.h using the following manual method: +# Edited copy of mysql.h using vi by doing g/, *$/j (repeat) then g/\* *$/j (repeat) +# to get all functions on one line each. +# Extracted list of api func names from http://dev.mysql.com/doc/refman/4.1/en/c-api-functions.html +# and copied to a temporary file to clean up (mysql_api_names.txt) +# Stripped down to bare names and merged into one line separated by | +# then egrep -w `cat mysql_api_names.txt` mysql.h mysql_api.ncidef +# then edit mysql_api.ncidef in vi: %s/^/ # / +# to create space for nci signatures and to use original definition as a # comment. +# This method isn't ideal, I'm just noting it here in case it helps others. +# Ideally the process should be automated - but there be many dragons along # that path. +# +# long long values (my_ulonglong) aren't handled by nci - spec'd as just long for now +# +#MYSQL_FIELD and MYSQL_RES are structs +#typedef char **MYSQL_ROW; /* return data as array of strings */ +#typedef unsigned int MYSQL_FIELD_OFFSET; /* offset to current field */ +#typedef MYSQL_ROWS *MYSQL_ROW_OFFSET; /* offset to current row */ +# +l p #! my_ulonglong mysql_num_rows(MYSQL_RES *res) +i p # unsigned int mysql_num_fields(MYSQL_RES *res) +c p # my_bool mysql_eof(MYSQL_RES *res) +p pi # MYSQL_FIELD *mysql_fetch_field_direct(MYSQL_RES *res, unsigned int fieldnr) +p p # MYSQL_FIELD * mysql_fetch_fields(MYSQL_RES *res) +p p # MYSQL_ROW_OFFSET mysql_row_tell(MYSQL_RES *res) +i p # MYSQL_FIELD_OFFSET mysql_field_tell(MYSQL_RES *res) +i p # unsigned int mysql_field_count(MYSQL *mysql) +l p #! my_ulonglong mysql_affected_rows(MYSQL *mysql) +l p #! my_ulonglong mysql_insert_id(MYSQL *mysql) +i p # unsigned int mysql_errno(MYSQL *mysql) +t p # const char * mysql_error(MYSQL *mysql) +t p # const char * mysql_info(MYSQL *mysql) +l p # unsigned long mysql_thread_id(MYSQL *mysql) +t p # const char * mysql_character_set_name(MYSQL *mysql) +p p # MYSQL * mysql_init(MYSQL *mysql) +i pt # int mysql_ssl_set(MYSQL *mysql, const char *key, const char *cert, const char *ca, const char *capath, const char *cipher) +c pttt # my_bool mysql_change_user(MYSQL
Re: NCI 'v' vs '' in function parameter signatures
On Tue, Feb 14, 2006 at 10:04:59PM +0100, Leopold Toetsch wrote: On Feb 14, 2006, at 18:29, Tim Bunce wrote: The runtime dlfunc code will need to be altered to normalize away the trailing v so old code won't break. Should it warn about that? Yes, a warning please. Here's the patch. - removes 'v' argument entries from src/call_list.txt - adds mysqlclient signatures to src/call_list.txt [*] - tweaks docs/pdds/clip/pdd16_native_call.pod to match - adds list of definition files used into generated nci.c - adds sanity checking of return and argument sig chars - adds compile time warning for deprecated 'v' argument - adds optional duplicate signature warning (disabled) - adds runtime warning for deprecated 'v' argument Tim. [*] I'm planning a followup patch that splits src/call_list.txt into multiple files in a subdirectory. The mysqlclient defs would then be in their own file. With the current arrangement no one can safely remove a signature because there's no indication of what that signature exists for. The same ncidef file could then be used for both tools/build/nativecall.pl and tools/util/ncidef2pasm.pl (which I also plan to work on). Any objections or comments? Index: src/call_list.txt === --- src/call_list.txt (revision 11741) +++ src/call_list.txt (working copy) @@ -49,14 +49,12 @@ c# t/pmc/nci.t c p c pi -c v d# t/pmc/nci.t d d d JOd # Parrot builtins I JOS S JOS # ParrotIO.readline -d v I JI # Parrot_is_char_* v JOSP # String.trans v JOS # String.reverse @@ -83,7 +81,6 @@ f# t/pmc/nci.t f ff # t/pmc/nci.t f is -f v i i b# t/pmc/nci.t @@ -148,7 +145,6 @@ i i t i ti -i v i 4 i 4i i 42p @@ -160,7 +156,6 @@ l pi l pii l p33l -l v l 33l P Ji # Needed for parrot threads @@ -185,10 +180,8 @@ p t p tpp p ttt -p v s # t/pmc/nci.t -s v t # t/pmc/nci.t t i @@ -200,7 +193,6 @@ t t t tl4 t t4 -t v v v Jiiip# examples/japh/japh11.pasm @@ -215,7 +207,6 @@ v p v pl v pp -v v # These are needed for parrotio.pmc i JP @@ -343,3 +334,77 @@ # Make lua stop panic'ing. P JOI + + +# --- start mysqlclient library --- +# Created from mysql.h using the following manual method: +# Edited copy of mysql.h using vi by doing g/, *$/j (repeat) then g/\* *$/j (repeat) +# to get all functions on one line each. +# Extracted list of api func names from http://dev.mysql.com/doc/refman/4.1/en/c-api-functions.html +# and copied to a temporary file to clean up (mysql_api_names.txt) +# Stripped down to bare names and merged into one line separated by | +# then egrep -w `cat mysql_api_names.txt` mysql.h mysql_api.ncidef +# then edit mysql_api.ncidef in vi: %s/^/ # / +# to create space for nci signatures and to use original definition as a # comment. +# This method isn't ideal, I'm just noting it here in case it helps others. +# Ideally the process should be automated - but there be many dragons along # that path. +# +# long long values (my_ulonglong) aren't handled by nci - spec'd as just long for now +# +# MYSQL_FIELD and MYSQL_RES are structs +# typedef char **MYSQL_ROW; /* return data as array of strings */ +# typedef unsigned int MYSQL_FIELD_OFFSET; /* offset to current field */ +# typedef MYSQL_ROWS *MYSQL_ROW_OFFSET; /* offset to current row */ +# +l p#! my_ulonglong mysql_num_rows(MYSQL_RES *res) +i p# unsigned int mysql_num_fields(MYSQL_RES *res) +c p# my_bool mysql_eof(MYSQL_RES *res) +p pi # MYSQL_FIELD *mysql_fetch_field_direct(MYSQL_RES *res, unsigned int fieldnr) +p p# MYSQL_FIELD * mysql_fetch_fields(MYSQL_RES *res) +p p# MYSQL_ROW_OFFSET mysql_row_tell(MYSQL_RES *res) +i p# MYSQL_FIELD_OFFSET mysql_field_tell(MYSQL_RES *res) +i p# unsigned int mysql_field_count(MYSQL *mysql) +l p#! my_ulonglong mysql_affected_rows(MYSQL *mysql) +l p#! my_ulonglong mysql_insert_id(MYSQL *mysql) +i p# unsigned int mysql_errno(MYSQL *mysql) +t p# const char * mysql_error(MYSQL *mysql) +t p# const char * mysql_info(MYSQL *mysql) +l p# unsigned long mysql_thread_id(MYSQL *mysql) +t p# const char * mysql_character_set_name(MYSQL *mysql) +p p# MYSQL * mysql_init(MYSQL *mysql) +i pt # int mysql_ssl_set(MYSQL *mysql, const char *key, const char *cert, const char *ca, const char *capath, const char *cipher) +c pttt # my_bool mysql_change_user(MYSQL *mysql, const char *user, const char *passwd, const char *db) +p piti # MYSQL * mysql_real_connect(MYSQL *mysql, const char *host, const char *user
Re: NCI 'v' vs '' in function parameter signatures
Tim Bunce wrote:
What's the difference between 'v' and '' for NCI function parameters?
There isn't any, except the extra 'v' char.
I ask because both 'fv' and 'f' are in src/call_list.txt
Yeah.
In fact there are several 'duplicated' signatures:
[ ... ]
I'd say, we should drop all the '?v' variants. The extra 'v' doesn't
cover any information, it's just causing an init call to the argument
passing code.
Those warnings come from a version of tools/build/nativecall.pl I've
modified to 'normalize' the signatures to use 'v' and detect duplicates.
(As a side effect the nci.o file dropped from 354K to 347K.)
Good. But as said, I'd prefer the shorter signature.
Also, what's the protocol for adding signatures to call_list.txt?
I've at least one I want to add ('p itl' for mysql_real_connect)
and may have more soon. Should I just post a patch here?
Yep.
Tim.
Thanks for looking into this,
leo
Re: NCI 'v' vs '' in function parameter signatures
On Tue, Feb 14, 2006 at 02:48:41PM +0100, Leopold Toetsch wrote:
Tim Bunce wrote:
What's the difference between 'v' and '' for NCI function parameters?
There isn't any, except the extra 'v' char.
I ask because both 'fv' and 'f' are in src/call_list.txt
Yeah.
In fact there are several 'duplicated' signatures:
[ ... ]
I'd say, we should drop all the '?v' variants. The extra 'v' doesn't
cover any information, it's just causing an init call to the argument
passing code.
Those warnings come from a version of tools/build/nativecall.pl I've
modified to 'normalize' the signatures to use 'v' and detect duplicates.
(As a side effect the nci.o file dropped from 354K to 347K.)
Good. But as said, I'd prefer the shorter signature.
Also, what's the protocol for adding signatures to call_list.txt?
I've at least one I want to add ('p itl' for mysql_real_connect)
and may have more soon. Should I just post a patch here?
Yep.
Tim.
Thanks for looking into this,
I'll aim to work up a patch this week.
The runtime dlfunc code will need to be altered to normalize away the
trailing v so old code won't break. Should it warn about that?
Tim.
Re: NCI 'v' vs '' in function parameter signatures
On Tuesday 14 February 2006 05:48, Leopold Toetsch wrote: I'd say, we should drop all the '?v' variants. The extra 'v' doesn't cover any information, it's just causing an init call to the argument passing code. To avoid confusion, I suggest requiring that functions returning void always use 'v' and disallowing the signature ''. Allowing people to drop 'v' for the return in some cases seems like it will bite them later when they try to write 'ff' for void foo ( float, float ) and wonder why it doesn't work. -- c
Re: NCI 'v' vs '' in function parameter signatures
On Feb 14, 2006, at 21:45, chromatic wrote: To avoid confusion, I suggest requiring that functions returning void always use 'v' and disallowing the signature ''. Exactly my thoughts too. leo
Re: NCI 'v' vs '' in function parameter signatures
On Feb 14, 2006, at 18:29, Tim Bunce wrote: I'll aim to work up a patch this week. Great, thanks. The runtime dlfunc code will need to be altered to normalize away the trailing v so old code won't break. Should it warn about that? Yes, a warning please. Tim. leo
NCI 'v' vs '' in function parameter signatures
What's the difference between 'v' and '' for NCI function parameters?
Here, for example, is the code for 'fv' and 'f':
static void
pcf_f_v(Interp *interpreter, PMC *self)
{
typedef float (*func_t)(void);
func_t pointer;
struct call_state st;
float return_data;
Parrot_init_arg_nci(interpreter, st, v);
pointer = (func_t)D2FPTR(PMC_struct_val(self));
return_data = (float)(*pointer)();
set_nci_N(interpreter, st, return_data);
}
static void
pcf_f(Interp *interpreter, PMC *self)
{
float (*pointer)(void);
float return_data;
struct call_state st;
pointer = (float (*)(void))D2FPTR(PMC_struct_val(self));
return_data = (float)(*pointer)();
set_nci_N(interpreter, st, return_data);
}
The code is a little different but it's not clear (to me) if there's any
practical difference.
I ask because both 'fv' and 'f' are in src/call_list.txt
In fact there are several 'duplicated' signatures:
Ignored signature 'cv' on line 52 (previously seen on line 49)
Ignored signature 'dv' on line 58 (previously seen on line 54)
Ignored signature 'fv' on line 85 (previously seen on line 82)
Ignored signature 'iv' on line 150 (previously seen on line 87)
Ignored signature 'lv' on line 162 (previously seen on line 155)
Ignored signature 'pv' on line 187 (previously seen on line 170)
Ignored signature 'sv' on line 190 (previously seen on line 189)
Ignored signature 'tv' on line 202 (previously seen on line 192)
Ignored signature 'vv' on line 217 (previously seen on line 204)
Those warnings come from a version of tools/build/nativecall.pl I've
modified to 'normalize' the signatures to use 'v' and detect duplicates.
(As a side effect the nci.o file dropped from 354K to 347K.)
Also, what's the protocol for adding signatures to call_list.txt?
I've at least one I want to add ('p itl' for mysql_real_connect)
and may have more soon. Should I just post a patch here?
Tim.
T and L parameter types for NCI
I just went, after ages, and sync'd up with a current parrot for my work project. Fixing things to work with the changes has been... interesting. The big hang up has been the removal of the T and L parameter types for NCI calls. T was a string pointer array and L was a long array. They're still documented in call_list.txt, there are still references to them in parts of the library, and there are fragments of the code for them in nativecall.pl. Change 9260 did this back in september (yes, it has been ages, I'm just syncing up now). This breaks the postgres.pir interface code -- making calls into postgres now isn't possible, as the interpreter pukes and dies when you try. Are there alternatives? The documentation for this stuff is worse now than when I wrote it originally, and it's not clear what needs to be done to duplicate the functionality of the removed call types. -- Dan --it's like this--- Dan Sugalski even samurai [EMAIL PROTECTED] have teddy bears and even teddy bears get drunk
Re: T and L parameter types for NCI
On Jan 2, 2006, at 19:36, Dan Sugalski wrote:
The big hang up has been the removal of the T and L parameter types
for NCI calls. T was a string pointer array and L was a long array.
[ ... ]
Are there alternatives? The documentation for this stuff is worse now
than when I wrote it originally, and it's not clear what needs to be
done to duplicate the functionality of the removed call types.
Sorry for the documentation mismatch, but you know that's the hardest
part. Anyway using {Un,}ManagedStruct for any item more complex than a
simple type (incl. C-strings) is the way to go. There are by far too
many possible permutations of foo* items to be covered with just one or
two plain signature chars.
This pod:
$ perldoc docs/pmc/struct.pod
should cover most of it.
Some examples are e.g. t/pmc/nci.t: nci_pi - struct with ints
or runtime/parrot/library/libpcre.pir, which is using array of ints too.
The SDL libraries are also using rather complex structures built with
the *Struct interface.
BTW: while looking at runtime/parrot/library/postgres.pir and
runtime/parrot/library/postgres.declarations I discovered some
inconsistencies, especially with these more complex structures, e.g.:
dlfunc $P2, $P1, 'PQexecParams', 'pptiLTLLi'
dlfunc $P2, $P1, 'PQexecPrepared', 'pptit33i'
whike the C declarations are vastly the same, the PIR counterparts are
totally different.
HTH,
leo
A step closer to ffcall NCI
Here's another update to the ffcall POC that I posted a week or two
ago. It's now got a Configure test to work out which backend to use
(and an --ask option to override).
It still needs more work (which is fine as there's a feature freeze),
and some work on memory allocation and stuff. I also need to clarify a
few things; such a when I allocate some PMCs in C whether I need to
mark them so that they aren't swept away by a DOD (I suspect so).
Some known issues:
* Can't clone the NCI object
* compilers/pge/mklib.pir might die during the build; I can't do
debugging on the system that I'm on at the moment, so expect a fix
soon
* The callback tests are skipped as I need to do some work to unify
new_callback between backends
An opinion on the build would be appreciated. Should I have
build_nativecall.pl write out nci.c for each backend (as it is now),
or should there be a nci_native.c, nci_ffcall.c etc. and have the
makefile build the right one?
Cheers,
Nick
p.s. I may have difficultly getting online over the next few days
Index: build_tools/build_nativecall.pl
===
--- build_tools/build_nativecall.pl (revision 9774)
+++ build_tools/build_nativecall.pl (working copy)
@@ -28,10 +28,29 @@
=cut
use strict;
+use Parrot::Config;
# This file will eventually be compiled
open NCI, src/nci.c or die Can't open nci.c!;
+
+my $nci_implementation = $PConfig{nci_impl};
+
+print NCI Implementation: $nci_implementation\n;
+
+if ($nci_implementation eq 'ffcall')
+{
+# Generate
+open NCI_FFCALL, build_tools/build_nativecall_ffcall.in or die;
+
+my @nci_ffcall = NCI_FFCALL;
+
+print NCI @nci_ffcall;
+close;
+exit;
+}
+
+
print_head();
my %ret_type =
@@ -649,6 +668,14 @@
return NULL;
}
+void *clone_call_func(Interp *interpreter, PMC *pmc_nci, void *args)
+{
+}
+
+void release_call_func(void *nci_args)
+{
+}
+
TAIL
}
Index: build_tools/build_nativecall_ffcall.in
===
--- build_tools/build_nativecall_ffcall.in (revision 0)
+++ build_tools/build_nativecall_ffcall.in (revision 0)
@@ -0,0 +1,815 @@
+/* NCI Implementation which uses ffcall */
+
+#include nci.str
+
+#include avcall.h
+#include callback.h
+
+#include parrot/parrot.h
+#include parrot/method_util.h
+#include parrot/oplib/ops.h
+
+#if defined(HAS_JIT) defined(I386)
+# include parrot/exec.h
+# include parrot/jit.h
+/*# define CAN_BUILD_CALL_FRAMES*/
+#endif
+
+/* Structure used for storing arguments and return values */
+
+typedef union UnionArg
+{
+char _char;
+int_int;
+short _short;
+long _long;
+
+float _float;
+double _double;
+
+int*_int_p;
+long *_long_p;
+short *_short_p;
+
+float *_float_p;
+double *_double_p;
+
+char *_string;
+
+void *_pointer;
+} UnionArg;
+
+
+/* The NCI data structure which stores the NCI and parrot signatures
+ as well as arguments and return value */
+
+typedef struct NCIArgs
+{
+char *signature;
+char *signature_parrot;
+
+UnionArg result;
+UnionArg args[10];
+
+} NCIArgs;
+
+
+/* Convenience routines for fetching values */
+
+static INTVAL
+get_nci_I(Interp *interpreter, struct call_state *st, int n)
+{
+assert(n st-src.n);
+Parrot_fetch_arg_nci(interpreter, st);
+
+return UVal_int(st-val);
+}
+
+static FLOATVAL
+get_nci_N(Interp *interpreter, struct call_state *st, int n)
+{
+assert(n st-src.n);
+Parrot_fetch_arg_nci(interpreter, st);
+
+return UVal_num(st-val);
+}
+
+static STRING*
+get_nci_S(Interp *interpreter, struct call_state *st, int n)
+{
+assert(n st-src.n);
+Parrot_fetch_arg_nci(interpreter, st);
+
+return UVal_str(st-val);
+}
+
+static PMC*
+get_nci_P(Interp *interpreter, struct call_state *st, int n)
+{
+/*
+ * exessive args are passed as NULL
+ * used by e.g. MMD infix like __add
+ */
+if (n st-src.n)
+Parrot_fetch_arg_nci(interpreter, st);
+else
+UVal_pmc(st-val) = NULL;
+
+return UVal_pmc(st-val);
+}
+
+#define GET_NCI_I(n) get_nci_I(interpreter, st, n)
+#define GET_NCI_S(n) get_nci_S(interpreter, st, n)
+#define GET_NCI_N(n) get_nci_N(interpreter, st, n)
+#define GET_NCI_P(n) get_nci_P(interpreter, st, n)
+
+
+/* Convenience routines for setting values */
+
+static void
+set_nci_I(Interp *interpreter, struct call_state *st, INTVAL val)
+{
+Parrot_init_ret_nci(interpreter, st, I);
+UVal_int(st-val) = val;
+Parrot_convert_arg(interpreter, st);
+Parrot_store_arg(interpreter, st);
+}
+
+static void
+set_nci_N(Interp *interpreter, struct call_state *st, FLOATVAL val)
+{
+Parrot_init_ret_nci(interpreter, st, N);
+UVal_num(st-val) = val;
+Parrot_convert_arg(interpreter, st);
+Parrot_store_arg(interpreter, st);
+}
+
+static void
+set_nci_S(Interp *interpreter, struct call_state *st, STRING *val
Re: [POC] nci interface using ffcall library
On Oct 31, 2005, at 7:49, John Lenz wrote: On Fri, October 28, 2005 2:22 pm, Nick Glencross said: Guys, As mentioned on the list yesterday I started evaluating ffcall as a way of providing NCI functionality. http://www.haible.de/bruno/packages-ffcall.html I am a SWIG (www.swig.org) developer, and I recently started looking at what it will take to add a parrot module to SWIG. I will post some of the issues I ran into, but I was wondering if there would be any interest for SWIG to become one of the official way to do NCI? SWIG isn't a replacement for NCI. Parrot NCI is fully dynamic and doesn't need any compiler, linker, interface definition file and what not. SWIG has a whole bunch of other benefits which I won't list here. I think, it would need good reasons to include another ffi-alike. But that doesn't preclude SWIG of course. The main issue resolves around the problem that there isn't a standard way to just register a function as callable from parrot (that I can see). Any PMC that provides the 'invoke' vtable is a callable. SWIG can easily generate functions of the form void* _wrap_foo(Interp* interpreter, PMC* pmc, void * next) one for each function being wrapped. Can we inject that function directly into the vtable somehow? Or just create a class that passes on the invoke function? I presume that would be a NCI_SWIG PMC. If this callable is inserted into the vtable or dynamically looked up is an implementation detail. One of the existing SWIG modules (CHICKEN) deals with a continuation passing language, so SWIG can deal with continuation-aware languages. Sounds interesting SWIG does not require any code besides what it generates... think lex or yacc... Well, it's fully static, while NCI is dynamic: $ cat n.pir .sub main :main .local pmc l, f .local string fn .local float v l = loadlib libm loop: (fn, v) = input_func_and_val() f = dlfunc l, fn, dd v = f(v) print v print \n goto loop .end .sub input_func_and_val .local pmc in, out out = getstdout print out.flush() .local string fn, v in = getstdin fn = readline in chopn fn, 1 v = readline in chopn v, 1 .return (fn, v) .end $ ./parrot n.pir sin 2 0.909297 cos 2 -0.416147 John leo
RE: [POC] nci interface using ffcall library
Nick Glencross wrote: As mentioned on the list yesterday I started evaluating ffcall as a way of providing NCI functionality. http://www.haible.de/bruno/packages-ffcall.html I actually really like the current NCI implementation, although it suffers from a large nci.c file and all the usable prototypes need to be specified at compile time in call_list.txt. Over time as more libraries become supported this list will quickly grow. Here's a reference to the last time this thread came up: http://groups.google.com/group/perl.perl6.internals/browse_frm/thread/8b1b5e 7e343ce3c4/ and a shortened URL forwarded version: http://ffcall-perl6-internals.notlong.com Here's the tail end: 8. Dan Sugalski Jan 12 2004, 12:49 pm At 10:13 AM -0600 1/12/04, Garrett Goebel wrote: Tim Bunce wrote: I see Dan says in his blog Yeah, I know, we should use libffi, and we may as a fallback, if we don't just give in and build up the function headers everywhere. I'm not familiar with libffi so this may be a dumb question, but why the apparent reluctance to use it? The reluctance probably doesn't have anything to do with its very liberal licensing terms... Nope. The two issues were portability and the prospect of a library which isn't ours that we'd need to maintain if we were distributing it. I'm fine with someone taking a stab at teaching Configure to probe for it on platforms that don't have the JIT building up function headers already, and teaching nci.c to use it in that case. (As long as the order of preference is JIT building functions, libffi, current hackish NCI scheme is maintained) -- Dan 9. Garrett Goebel Jan 12 2004, 1:48 pm Tim Bunce wrote: Tim Bunce wrote: I see Dan says in his blog Yeah, I know, we should use libffi, and we may as a fallback, if we don't just give in and build up the function headers everywhere. I'm not familiar with libffi so this may be a dumb question, but why the apparent reluctance to use it? In http://www.nntp.perl.org/group/perl.perl6.internals/253 I see Garrett Goebel quotes Bruno Haible saying I could agree to the LGPL license. Perl could then use ffcall as a shared library (linked or via dlopen) And I see http://www.parrotcode.org/docs/faq.pod.html says Code accepted into the core interpreter must fall under the same terms as parrot. Library code (for example the ICU library we're using for Unicode) we link into the interpreter can be covered by other licenses so long as their terms don't prohibit this. So it seems there's no licensing issue preventing parrot using libffi. Is that right? Are there any others? My bad. In my comments on Dan's blog, I confused libffi with ffcall. Both do roughly the same thing... The libffi was originally produced by Cygnus, but is now part of GCC. http://sources.redhat.com/libffi/ http://gcc.gnu.org/cgi-bin/cvsweb.cgi/gcc/libffi/LICENSE ffcall was produced by Bruno Haibel as part of his CLISP package. http://www.haible.de/bruno/packages-ffcall.html ffcall used to be considered more mature/stable, but since libffi was included in GCC the general impression true or not is that libffi is more actively maintained. From mailing lists and csv logs, it looks like both are actively maintained... 10. Tim Bunce Jan 13 2004, 9:49 am And since it seems both are usable for parrot from a licencing perspective we're free to use whichever suits best on a given platform - assuming someone implements the relevant interface code. Tim. -- Garrett Goebel IS Development Specialist ScriptPro Direct: 913.403.5261 5828 Reeds Road Main: 913.384.1008 Mission, KS 66202 Fax: 913.384.2180 www.scriptpro.com garrett at scriptpro dot com
Re: [POC] nci interface using ffcall library
Garrett Goebel wrote: Nick Glencross wrote: As mentioned on the list yesterday I started evaluating ffcall as a way of providing NCI functionality. http://www.haible.de/bruno/packages-ffcall.html I actually really like the current NCI implementation, although it suffers from a large nci.c file and all the usable prototypes need to be specified at compile time in call_list.txt. Over time as more libraries become supported this list will quickly grow. Here's a reference to the last time this thread came up: http://groups.google.com/group/perl.perl6.internals/browse_frm/thread/8b1b5e 7e343ce3c4/ and a shortened URL forwarded version: http://ffcall-perl6-internals.notlong.com Thanks for that Garrett. It's great that you've talked to Bruno in the past about this! Although I had planned to look at both ffcall and libffi, ffcall has so far looked more compelling to me. I started looking for the latest version of libffi, couldn't be sure which it was so I put it on the backburner. Then I looked at ffcall, instantly thought 'perfect' because it provided a really usable API, and then knocked up a quick POC. I certainly agree about what was being said about supporting multiple backends as it protects us against future nasty platforms. However, I'm unclear what the benefits of the favoured option of using JIT code was; that sounds like reinventing a ff library. One thing that would be nice would be extend build_native.pl so that it builds the other supported NCI layers; that way they would all support the same types and not get out of sync with one another. I'd be interested in trying this if no one objects... My word of the week seems to be 'callbacks' because the current implementation doesn't seem to cut it (in my opinion), which is really the driving reason for looking at other alternatives. Cheers, Nick
Re: [POC] nci interface using ffcall library
On Mon, 2005-10-31 at 20:12 +, Nick Glencross wrote: I certainly agree about what was being said about supporting multiple backends as it protects us against future nasty platforms. However, I'm unclear what the benefits of the favoured option of using JIT code was; that sounds like reinventing a ff library. Having a JIT was a goal anyway. It avoids potential license conflicts. It removes one dependency. It potentially reduces executable size and memory use. It's something we can patch and test and customize as we need, especially with regard to our calling conventions. Those might not all be current benefits; I'm just going by memory and not arguing for or against ffcall or libffi. -- c
Re: [POC] nci interface using ffcall library - updated
Nick Glencross wrote:
Guys,
As mentioned on the list yesterday I started evaluating ffcall as a
way of providing NCI functionality.
Ok, here's an updated version with (hopefully) working callbacks -- at
least enough for a POC.
If you tried out my previous version, run 'rm */*ffcall*' before
applying this patch to avoid creating botched files.
I'll show an example of how you might use it...
I started by hacking nci_test.c to include:
typedef int (*cb_C1_func)(int, int, int);
void
nci_cb_C1(cb_C1_func cb, int a, int b, int c) {
int result = (cb)(a, b, c);
printf (Result is %d\n, result);
}
This routine accepts a callback pointer and three arguments, and then
calls the callback with these arguments and prints the value received back.
The parrot harness might look like this:
.sub test :main
.local pmc libnci_test
libnci_test = loadlib libnci_test
.local pmc nci_cb_C1
nci_cb_C1 = dlfunc libnci_test, nci_cb_C1, ipiii
.local pmc user_data
user_data = new .Integer
.local pmc cb
cb = newsub _call_back
.local pmc cb_wrapped
cb_wrapped = new_callback cb, user_data,
nci_cb_C1 (cb_wrapped, 1,2,3)
print Finish!\n
.end
.sub _call_back
.param int a
.param int b
.param int c
print Hello world!\n
print a
print
print b
print
print c
print
print_newline
.return (b)
.end
The script looks similar to before, but certain limitations have been
lifted. A PMC must still be supplied to new_callback (although this
requirement could be lifted), but this does not need to be seen by the
callback (unless you want it to -- just pass it with a normal 'p'
signature) Moreover, the callback signature can be anything, and would
now include the return type which will be returned to C.
To clarify, 'ipiii' refers to the C nci_cb_C1 routine, and the ''
refers to the prototype for _call_back.
Running it gives
Hello world!
1 2 3
Result is 2
Finish!
'1 2 3' were passed from parrot into C and then back out into parrot to
be printed. parrot then returned the middle one which was printed back
in C-land.
Have fun!
Nick
Index: src/nci_ffcall.c
===
--- src/nci_ffcall.c(revision 0)
+++ src/nci_ffcall.c(revision 0)
@@ -0,0 +1,777 @@
+/* *Proof Of Concept* - show how ffcall can provide NCI functionality */
+
+#include avcall.h
+#include callback.h
+
+#include parrot/parrot.h
+#include parrot/method_util.h
+#include parrot/oplib/ops.h
+
+#include nci_ffcall.str
+
+#if defined(HAS_JIT) defined(I386)
+# include parrot/exec.h
+# include parrot/jit.h
+/*# define CAN_BUILD_CALL_FRAMES*/
+#endif
+
+/* Structure used for storing arguments and return values */
+
+typedef union UnionArg
+{
+char _char;
+int_int;
+short _short;
+long _long;
+
+float _float;
+double _double;
+
+int*_int_p;
+long *_long_p;
+short *_short_p;
+
+float *_float_p;
+double *_double_p;
+
+char *_string;
+
+void *_pointer;
+} UnionArg;
+
+
+/* The NCI data structure which stores the NCI and parrot signatures
+ as well as arguments and return value */
+
+typedef struct NCIArgs
+{
+char *signature;
+char *signature_parrot;
+
+UnionArg result;
+UnionArg args[10];
+
+} NCIArgs;
+
+
+/* Convenience routines */
+
+static INTVAL
+get_nci_I(Interp *interpreter, struct call_state *st, int n)
+{
+assert(n st-src.n);
+Parrot_fetch_arg_nci(interpreter, st);
+
+return UVal_int(st-val);
+}
+
+static FLOATVAL
+get_nci_N(Interp *interpreter, struct call_state *st, int n)
+{
+assert(n st-src.n);
+Parrot_fetch_arg_nci(interpreter, st);
+
+return UVal_num(st-val);
+}
+
+static STRING*
+get_nci_S(Interp *interpreter, struct call_state *st, int n)
+{
+assert(n st-src.n);
+Parrot_fetch_arg_nci(interpreter, st);
+
+return UVal_str(st-val);
+}
+
+static PMC*
+get_nci_P(Interp *interpreter, struct call_state *st, int n)
+{
+/*
+ * exessive args are passed as NULL
+ * used by e.g. MMD infix like __add
+ */
+if (n st-src.n)
+Parrot_fetch_arg_nci(interpreter, st);
+else
+UVal_pmc(st-val) = NULL;
+
+return UVal_pmc(st-val);
+}
+
+#define GET_NCI_I(n) get_nci_I(interpreter, st, n)
+#define GET_NCI_S(n) get_nci_S(interpreter, st, n)
+#define GET_NCI_N(n) get_nci_N(interpreter, st, n)
+#define GET_NCI_P(n) get_nci_P(interpreter, st, n)
+
+/*
+ * set return value
+ */
+static void
+set_nci_I(Interp *interpreter, struct call_state *st, INTVAL val)
+{
+Parrot_init_ret_nci(interpreter, st, I);
+UVal_int(st-val) = val;
+Parrot_convert_arg(interpreter, st);
+Parrot_store_arg(interpreter, st);
+}
+
+static void
+set_nci_N(Interp *interpreter, struct call_state *st, FLOATVAL val)
+{
+Parrot_init_ret_nci(interpreter, st, N);
+UVal_num(st-val) = val
Re: [POC] nci interface using ffcall library
On Fri, October 28, 2005 2:22 pm, Nick Glencross said: Guys, As mentioned on the list yesterday I started evaluating ffcall as a way of providing NCI functionality. http://www.haible.de/bruno/packages-ffcall.html I am a SWIG (www.swig.org) developer, and I recently started looking at what it will take to add a parrot module to SWIG. I will post some of the issues I ran into, but I was wondering if there would be any interest for SWIG to become one of the official way to do NCI? SWIG has a whole bunch of other benefits which I won't list here. The main issue resolves around the problem that there isn't a standard way to just register a function as callable from parrot (that I can see). perl5 allowed XS() and then calling the newXS function. SWIG would prefer to use the best representation/match to the internal interpreter... SWIG can easily generate functions of the form void* _wrap_foo(Interp* interpreter, PMC* pmc, void * next) one for each function being wrapped. Can we inject that function directly into the vtable somehow? Or just create a class that passes on the invoke function? One of the existing SWIG modules (CHICKEN) deals with a continuation passing language, so SWIG can deal with continuation-aware languages. Pros: * Makes any prototype possible, not just those in call_list.txt * Probably won't initially make code much smaller, but would over time (stripped nci_ffcall.o library is currently 1/6 size of nci.o) * Callbacks can have any prototype as well, not just a data/user data pair * Callbacks no longer require user data to be managed by the 3rd party library (this is a biggy) * Callbacks will be able to support a return type Cons: * Reliance on a library, although NCI may be an optional feature SWIG does not require any code besides what it generates... think lex or yacc... * Performance? There are a few additional loops and switches, but I can't see this being a real issue * NCI would only be available on platforms which support the library we use; from what I've seen dynamic function calls is not easy to do cross-platform SWIG works on pretty much every platform and every compiler... * Possibly incompatible 3rd party licensing SWIG uses a BSD like license, so no problem at all... can even include SWIG directly in the parrot code... It is something to think about... John
[POC] nci interface using ffcall library
Guys,
As mentioned on the list yesterday I started evaluating ffcall as a way
of providing NCI functionality.
http://www.haible.de/bruno/packages-ffcall.html
I actually really like the current NCI implementation, although it
suffers from a large nci.c file and all the usable prototypes need to be
specified at compile time in call_list.txt. Over time as more libraries
become supported this list will quickly grow.
My main reason for visiting this is that I couldn't write the simple NCI
example that I had in mind because callbacks are too restrictive, and
unlikely to be fixable.
I've now got a working POC for function calls, and although callbacks
don't work yet, they're not far off. All but 11 of the 59 NCI tests
pass, and those that don't work exercise callbacks or reading back
values from types for which I haven't done yet.
For those that want to try it, you'll of course need ffcall installed
and rerun Configure. Being a Proof of Concept, I've hacked the '-l's for
the libraries into the Makefile in the wrong place which means some of
the other tests fail. For now, just try:
perl -Ilib t/pmc/nci.t
To recap the (mostly obvious) Pros Cons.
Pros:
* Makes any prototype possible, not just those in call_list.txt
* Probably won't initially make code much smaller, but would over time
(stripped nci_ffcall.o library is currently 1/6 size of nci.o)
* Callbacks can have any prototype as well, not just a data/user data pair
* Callbacks no longer require user data to be managed by the 3rd party
library (this is a biggy)
* Callbacks will be able to support a return type
Cons:
* Reliance on a library, although NCI may be an optional feature
* Performance? There are a few additional loops and switches, but I
can't see this being a real issue
* NCI would only be available on platforms which support the library
we use; from what I've seen dynamic function calls is not easy to do
cross-platform
* Possibly incompatible 3rd party licensing
To address some of the points in the Cons section, it would be possible
to maintain multiple NCI backends, although more work.
As far as PIR/PASM code, the interface is unchanged, although I would
like to suggest that at some point the newclass signature include the
return type in common with other opcodes, as hopefully we can support this.
Cheers,
Nick
p.s. As you can see, I've been won over by ffcall, but I've only been
using it for a day so there may well be show stoppers to using it.
Index: src/nci_ffcall.c
===
--- src/nci_ffcall.c(revision 0)
+++ src/nci_ffcall.c(revision 0)
@@ -0,0 +1,618 @@
+#include avcall.h
+#include callback.h
+
+#include parrot/parrot.h
+#include parrot/method_util.h
+#include parrot/oplib/ops.h
+
+#include nci_ffcall.str
+
+#if defined(HAS_JIT) defined(I386)
+# include parrot/exec.h
+# include parrot/jit.h
+/*# define CAN_BUILD_CALL_FRAMES*/
+#endif
+
+
+typedef union UnionArg
+{
+char _char;
+int_int;
+short _short;
+long _long;
+
+float _float;
+double _double;
+
+int*_int_p;
+long *_long_p;
+short *_short_p;
+
+float *_float_p;
+double *_double_p;
+
+char *_string;
+
+void *_pointer;
+} UnionArg;
+
+
+typedef struct NCIArgs
+{
+char *signature;
+char *signature_parrot;
+
+UnionArg result;
+UnionArg args[10];
+
+} NCIArgs;
+
+
+static INTVAL
+get_nci_I(Interp *interpreter, struct call_state *st, int n)
+{
+assert(n st-src.n);
+Parrot_fetch_arg_nci(interpreter, st);
+
+return UVal_int(st-val);
+}
+
+static FLOATVAL
+get_nci_N(Interp *interpreter, struct call_state *st, int n)
+{
+assert(n st-src.n);
+Parrot_fetch_arg_nci(interpreter, st);
+
+return UVal_num(st-val);
+}
+
+static STRING*
+get_nci_S(Interp *interpreter, struct call_state *st, int n)
+{
+assert(n st-src.n);
+Parrot_fetch_arg_nci(interpreter, st);
+
+return UVal_str(st-val);
+}
+
+static PMC*
+get_nci_P(Interp *interpreter, struct call_state *st, int n)
+{
+/*
+ * exessive args are passed as NULL
+ * used by e.g. MMD infix like __add
+ */
+if (n st-src.n)
+Parrot_fetch_arg_nci(interpreter, st);
+else
+UVal_pmc(st-val) = NULL;
+
+return UVal_pmc(st-val);
+}
+
+#define GET_NCI_I(n) get_nci_I(interpreter, st, n)
+#define GET_NCI_S(n) get_nci_S(interpreter, st, n)
+#define GET_NCI_N(n) get_nci_N(interpreter, st, n)
+#define GET_NCI_P(n) get_nci_P(interpreter, st, n)
+
+/*
+ * set return value
+ */
+static void
+set_nci_I(Interp *interpreter, struct call_state *st, INTVAL val)
+{
+Parrot_init_ret_nci(interpreter, st, I);
+UVal_int(st-val) = val;
+Parrot_convert_arg(interpreter, st);
+Parrot_store_arg(interpreter, st);
+}
+
+static void
+set_nci_N(Interp *interpreter, struct call_state *st, FLOATVAL val)
+{
+Parrot_init_ret_nci(interpreter, st, N);
+UVal_num(st
Re: [POC] nci interface using ffcall library
On Oct 28, 2005, at 22:22, Nick Glencross wrote: Guys, As mentioned on the list yesterday I started evaluating ffcall as a way of providing NCI functionality. http://www.haible.de/bruno/packages-ffcall.html I've downloaded it and had a short look into sources. The list of supported platforms is quite impressive. OTOH it looks a bit under documented to me. I've now got a working POC for function calls, and although callbacks don't work yet, they're not far off. All but 11 of the 59 NCI tests pass, and those that don't work exercise callbacks or reading back values from types for which I haven't done yet. That's a lot for this early stage. Good work. Pros: Makes ffcall very worthy to include. Cons: * Reliance on a library, although NCI may be an optional feature A library per se isn't a problem. You mentioned licence issues yesterday due to GPL. Keeping a fallback to NCI (with possibly reduced functionality would be ok too probably). * Performance? There are a few additional loops and switches, but I can't see this being a real issue Can be measured. * Possibly incompatible 3rd party licensing Ah, yep - IANALL. As far as PIR/PASM code, the interface is unchanged, although I would like to suggest that at some point the newclass signature include the return type in common with other opcodes, as hopefully we can support this. You are talking about new_callback ? Cheers, Nick p.s. As you can see, I've been won over by ffcall, but I've only been using it for a day so there may well be show stoppers to using it. Great work for one day. leo
Re: [POC] nci interface using ffcall library
Nick Glencross wrote: Guys, As mentioned on the list yesterday I started evaluating ffcall as a way of providing NCI functionality. I seem to be seeing rather strange, and it's probably a misunderstanding on my part. I've pinched the GET_NCI_x(n) macros and accompanying get_nci_x functions from nci.c, but they don't seem to be working consistently. In particular, where I have a signature pp I often seem to be getting the same pointer twice. If the following line is added as line 267 in nci_ffcall.c after applying the patch: printf (Pointer is %p\n, nci_args-args[i]._pointer); and the following script run: .sub test :main .local pmc libnci_test libnci_test = loadlib libnci_test .local pmc nci_cb_C1 nci_cb_C1 = dlfunc libnci_test, nci_cb_C1, vpp .local pmc user_data user_data = new .Integer user_data = -99 .local pmc cb cb = newsub _call_back .local pmc cb_wrapped cb_wrapped = new_callback cb, user_data, iii nci_cb_C1 (user_data, cb_wrapped) .end .sub _call_back print Hello world!\n .end then you get Pointer is 0x84a33d0 Pointer is 0x84a33d0 [probably a segfault!] while these are meant to be different addresses (corresponding to user_data and cb_wrapped). From what I've seen you get the second address twice. I must be doing something wrong as it works in nci.c. Any thoughts would be appreciated as I can't see the wood from the trees anymore, Cheers, Nick p.s. Since posting the patch, I've realised that I probably need another Parrot_init_arg_nci(interpreter, st, nci_args-signature_parrot); before reading back the values, and probably a 'default' in the switch which steps over skipped items. This only enables one extra test to succeed though.
Re: [POC] nci interface using ffcall library
Nick Glencross wrote: Nick Glencross wrote: Guys, As mentioned on the list yesterday I started evaluating ffcall as a way of providing NCI functionality. I seem to be seeing rather strange, and it's probably a misunderstanding on my part. Ok, I see what the problem is. PMC_data(x) is a macro which can expand x out twice and because GET_NCI_P has the side effect of advancing to the next PMC, it starts by returning the second PMC. That will hopefully fix the problems that I've been having with callbacks! Nick
Re: [POC] nci interface using ffcall library
Leopold Toetsch wrote: On Oct 28, 2005, at 22:22, Nick Glencross wrote: Guys, As mentioned on the list yesterday I started evaluating ffcall as a way of providing NCI functionality. http://www.haible.de/bruno/packages-ffcall.html I've downloaded it and had a short look into sources. The list of supported platforms is quite impressive. OTOH it looks a bit under documented to me. The page that I've linked to is a bit bland, but the manual pages for the 4 routines are very thorough. It's being able to create the closures which made this library stand out, as that would allow us to create callbacks with any library. Still haven't looked under the hood to see how on earth that's possible. I've now got a working POC for function calls, and although callbacks don't work yet, they're not far off. All but 11 of the 59 NCI tests pass, and those that don't work exercise callbacks or reading back values from types for which I haven't done yet. That's a lot for this early stage. Good work. I was hoping to have shiny callbacks too, but something bad was happening with the arguments being passed around, but I've got to the bottom of that nasty. As far as PIR/PASM code, the interface is unchanged, although I would like to suggest that at some point the newclass signature include the return type in common with other opcodes, as hopefully we can support this. You are talking about new_callback ? Yes, thought one thing, typed another, Nick
Re: [perl #37512] [PATCH] Adds nci support for 'T' and 'L' prototypes
Leopold Toetsch via RT wrote: On Oct 23, 2005, at 17:08, Nick Glencross (via RT) wrote: Guys, call_list.txt lists 'T' and 'L' as being prototypes for passing arrays to nci functions, but no implementation exists in build_nativecall.pl. This patch provides an implementation, as well as new tests. I don't think that this is a good idea. There are way too many different array types. The code inside *ManagedStruct already deals with arrays of all standard types. See e.g. runtime/parrot/library/libpcre.imc, which works fine without new signatures. There is a second problem with this approach: permutation. All of these signature chars might be combined to create arbitrary function signatures. nci.c is already too big. I tend to agree. :-) Let's reject it. I was looking at callbacks the other evening. Am I right in thinking that only two callback prototypes are supported, or have I missed a trick there as well? Nick
NCI callbacks
Nick Glencross schrieb: I was looking at callbacks the other evening. Am I right in thinking that only two callback prototypes are supported, or have I missed a trick there as well? Yes, I think that hasn't changed since I've looking into interfacing Parrot with libsyck. As far as I remember, the standard workaround was to write some C-code that wraps the relevant shared lib, and provides the convenience functions for Parrot. CU, Bernhard
Re: [perl #37512] [PATCH] Adds nci support for 'T' and 'L' prototypes
On Oct 25, 2005, at 23:32, Nick Glencross wrote: I was looking at callbacks the other evening. Am I right in thinking that only two callback prototypes are supported, or have I missed a trick there as well? That's right. There are 2 callbacks (functions with 2 arguments only), one with the *transparent* (void *) user argument first, and one with it as second arg. Before including one or more 3 arg variants, I'd prefer to have some investigations (and results ;-) WRT possible alternatives like libffi. Nick leo
[perl #37512] [PATCH] Adds nci support for 'T' and 'L' prototypes
# New Ticket Created by Nick Glencross
# Please include the string: [perl #37512]
# in the subject line of all future correspondence about this issue.
# URL: https://rt.perl.org/rt3/Ticket/Display.html?id=37512
Guys,
call_list.txt lists 'T' and 'L' as being prototypes for passing arrays
to nci functions, but no implementation exists in build_nativecall.pl.
This patch provides an implementation, as well as new tests.
Someone might raise the point as to whether the data should be read back
into parrot's array, but this is generally going to be tricky and a job
for another day.
Cheers,
Nick
Index: build_tools/build_nativecall.pl
===
--- build_tools/build_nativecall.pl (revision 9534)
+++ build_tools/build_nativecall.pl (working copy)
@@ -171,6 +171,8 @@
B = S,
v = v,
J = ,
+ T = P,
+ L = P,
);
my $temp_cnt = 0;
@@ -466,6 +468,48 @@
t_$temp_num = GET_NCI_S($reg_num);;
return t_$temp_num;
};
+/T/ do {
+ push @{$temps_ref},
+ PMC* p_$temp_num ; char **m_$temp_num; .
+int i_$temp_num; int s_$temp_num;;
+ push @{$extra_preamble_ref},
+ p_$temp_num = GET_NCI_P($reg_num);\n .
+ s_$temp_num=VTABLE_get_integer (interpreter,p_$temp_num);\n .
+m_$temp_num=(char **) .
+malloc((s_$temp_num+1)*sizeof (char *));\n .
+for(i_$temp_num=0; i_$temp_nums_$temp_num;i_$temp_num++)\n.
+{STRING* string = VTABLE_get_string_keyed_int (interpreter, .
+p_$temp_num, i_$temp_num);\n .
+ m_${temp_num}[i_$temp_num] = .
+string_to_cstring (interpreter, string);}\n .
+m_${temp_num}[s_$temp_num] = NULL;;
+
+ push @{$extra_postamble_ref},
+for(i_$temp_num=0; i_$temp_nums_$temp_num;i_$temp_num++)\n.
+ {string_cstring_free(m_${temp_num}[i_$temp_num]);}\n .
+free (m_${temp_num});;
+
+ return m_$temp_num;
+};
+/L/ do {
+ push @{$temps_ref},
+ PMC* p_$temp_num ; long *m_$temp_num; .
+int i_$temp_num; int s_$temp_num;;
+ push @{$extra_preamble_ref},
+ p_$temp_num = GET_NCI_P($reg_num);\n .
+ s_$temp_num=VTABLE_get_integer (interpreter,p_$temp_num);\n .
+m_$temp_num=(long *) .
+malloc(s_$temp_num*sizeof (long));\n .
+for(i_$temp_num=0; i_$temp_nums_$temp_num;i_$temp_num++)\n.
+{long val = VTABLE_get_integer_keyed_int (interpreter, .
+p_$temp_num, i_$temp_num);\n .
+ m_${temp_num}[i_$temp_num] = val;};
+
+ push @{$extra_postamble_ref},
+free (m_${temp_num});;
+
+ return m_$temp_num;
+};
}
sub print_function {
Index: src/call_list.txt
===
--- src/call_list.txt (revision 9534)
+++ src/call_list.txt (working copy)
@@ -41,9 +41,9 @@
# 0 - insert a NULL (pointer) - doesn't comsume a register
# J - Parrot_Interp param
-# Arrayish things, terminated with NULL/0
+# Arrayish things
# L - Long array
-# T - Array of string pointers (Converted to cstrings)
+# T - Array of string pointers (converted to cstrings) terminated with a NULL
# Callback Things
@@ -322,3 +322,10 @@
# Sub.__get_regs_used
I JOt
+
+# Enables passing of arrays of char * and long
+v iT
+v 3T
+
+v iL
+v 3L
Index: src/nci_test.c
===
--- src/nci_test.c (revision 9534)
+++ src/nci_test.c (working copy)
@@ -89,6 +89,8 @@
void nci_vP(void *);
void nci_vpii(Outer *, int, int);
void nci_vv(void);
+void nci_viT(int argc, char *argv[]);
+void nci_v3L(int* count, long *longs);
/* Declarations for callback tests */
@@ -560,6 +562,33 @@
nci_dlvar_int *= 3;
}
+/*
+
+=head1 Functions which act upon Long and String arrays
+
+=cut
+
+*/
+
+
+void
+nci_viT (int argc, char *argv[]) {
+int i;
+for (i = 0 ; i argc ; i++) {
+printf (%d: %s\n, i, argv[i]);
+}
+}
+
+
+void
+nci_v3L (int* count, long *longs) {
+int i;
+for (i = 0 ; i *count ; i++)
+ printf (%d: %ld\n, i, longs[i]);
+
+if (*count) (*count)--;
+}
+
#ifdef TEST
char l2 = 4;
Index: t/pmc/nci.t
===
--- t/pmc/nci.t (revision 9534)
+++ t/pmc/nci.t (working copy)
@@ -23,7 +23,7 @@
=cut
-use Parrot::Testtests = 59;
+use Parrot::Testtests = 61;
use Parrot::Config;
SKIP: {
@@ -2459,3 +2459,74 @@
CODE
42
OUTPUT
+
+pir_output_is( 'CODE', 'OUTPUT', nci_viT - (int argc, char *argv[]));
+.sub test :main
+.local string library_name
+library_name = 'libnci_test'
+.local pmc libnci_test
+libnci_test = loadlib
Re: [perl #37512] [PATCH] Adds nci support for 'T' and 'L' prototypes
On Oct 23, 2005, at 17:08, Nick Glencross (via RT) wrote: Guys, call_list.txt lists 'T' and 'L' as being prototypes for passing arrays to nci functions, but no implementation exists in build_nativecall.pl. This patch provides an implementation, as well as new tests. I don't think that this is a good idea. There are way too many different array types. The code inside *ManagedStruct already deals with arrays of all standard types. See e.g. runtime/parrot/library/libpcre.imc, which works fine without new signatures. There is a second problem with this approach: permutation. All of these signature chars might be combined to create arbitrary function signatures. nci.c is already too big. leo
[perl #37439] [PATCH] Skip two nci tests on HP-UX
Applied as r9496. Thanks. -J
[perl #37439] [PATCH] Skip two nci tests on HP-UX
# New Ticket Created by Nick Glencross
# Please include the string: [perl #37439]
# in the subject line of all future correspondence about this issue.
# URL: https://rt.perl.org/rt3/Ticket/Display.html?id=37439
I plan to investigate the exact cause in due course, but for now I
suggest that we skip two nci tests which are causing hangs on HP-UX and
as such prevent smoke testing.
Once this patch is in I'll set up automated HP-UX smoke testing next
week. FYI the smoke entry for HP-UX has some failed MANIFEST tests; one
is due to the image having been checked out from subversion on another
system, and then copied to HP-UX system which did not have svn; the
second was caused by me botching disabling the aforementioned nci tests.
On a related note, are the dynclasses tests going to be enabled in smoke
at some point?
Index: t/pmc/nci.t
===
--- t/pmc/nci.t (revision 9481)
+++ t/pmc/nci.t (working copy)
@@ -291,7 +291,11 @@
333
OUTPUT
+SKIP:
+{
+skip (nci_dlvar_int hangs on HP-UX, 1) if $^O eq 'hpux';
+
pir_output_is( 'CODE', 'OUTPUT', nci_v and nci_dlvar_int );
.include datatypes.pasm
@@ -351,6 +355,7 @@
-000
-
OUTPUT
+}
output_is('CODE', 'OUTPUT', nci_dd - PASM);
loadlib P1, libnci_test
@@ -2226,6 +2231,11 @@
OUTPUT
+SKIP:
+{
+
+skip (nci_dlvar_int hangs on HP-UX, 1) if $^O eq 'hpux';
+
pir_output_is( 'CODE', 'OUTPUT', nci_vv and nci_dlvar_int );
.include datatypes.pasm
@@ -2285,6 +2295,7 @@
-119988
-359964
OUTPUT
+}
pir_output_is( 'CODE', 'OUTPUT', dlvar - unknown symbol );
Re: NCI: passing PMCs and using them
Leopold Toetsch wrote:
Klaas-Jan Stol wrote:
hi,
I'm currently trying to check out the NCI. As my Lua compiler only
uses PMCs (and not I/N/S registers), calling C functions will only be
done with PMCs.
I couldn't find info on this matter: suppose I have this string PMC,
how can I access the actual string in it?
so, when calling:
void print_pmc(void *PMC)
{
// how to access the string in P?
}
If it's a stringish PMC, then STRING *s = VTABLE_get_string(INTERP,
p); will do it.
The C code is just in a .c file, it's not like a PMC or something. So, I
don't think I can use INTERP, or VTABLE_get_string() at all. Or should
I link to parrotlib.a or something?
Is there some public API for accessing the fields in a PMC (for standard
PMCs, so accessing the intval for the Integer PMC and its decendents,
floatval for Float PMC, stringval for String PMCs, etc) ?
You might also consider switching to branches/leo-ctx5. The new
calling conventions do auto-conversion betwen PMCs and I/S/N in both
directions for NCI too.
Forgive my ignorance, but I don't really understand. I'm only using
PMCs, no I/S/N values. Why would using these auto-conversions be
interesting?
Btw, I also got this leo-ctx5 branch, compiled it, and run it from the
directory where my lua stuff is, like this:
$ ../leo-ctx5/parrot a.pir
it says:
PackFile_FixupTable_unpack: Unknown fixup type 1953066601!
PackFile_unpack segment 'FIXUP_luaglobals.pir' failed
Segmentation fault
I got this branch by doing this: (was not sure about the command, but I
considered this most probable)
svn checkout http://svn.perl.org/parrot/branches/leo-ctx5
Any idea what's wrong and causing the error?
klaas-jan
Re: NCI: passing PMCs and using them
On Aug 23, 2005, at 14:38, Klaas-Jan Stol wrote:
Leopold Toetsch wrote:
Klaas-Jan Stol wrote:
void print_pmc(void *PMC)
{
// how to access the string in P?
}
If it's a stringish PMC, then STRING *s = VTABLE_get_string(INTERP,
p); will do it.
The C code is just in a .c file, it's not like a PMC or something. So,
I don't think I can use INTERP, or VTABLE_get_string() at all. Or
should I link to parrotlib.a or something?
INTERP is just translated to interpreter (see any classes/.c file).
VTABLE_get_string is available in the whole Parrot tree, it's a macro
define in include/parrot/vtable.h. I don't know in what relation your
plain .c file is to parrot, but you might have a look at
src/extend.c, which also provides interfaces for such stuff.
Is there some public API for accessing the fields in a PMC (for
standard PMCs, so accessing the intval for the Integer PMC and its
decendents, floatval for Float PMC, stringval for String PMCs, etc) ?
Have a look at src/extend.c for the most abstract API.
You might also consider switching to branches/leo-ctx5. The new
calling conventions do auto-conversion betwen PMCs and I/S/N in both
directions for NCI too.
Forgive my ignorance, but I don't really understand. I'm only using
PMCs, no I/S/N values. Why would using these auto-conversions be
interesting?
If you are calling a NCI method, which takes native types (or returns
one) from PIR, you can still pass in PMCs (and will get a PMC as
result, if appropriate).
Btw, I also got this leo-ctx5 branch, compiled it, and run it from the
directory where my lua stuff is, like this:
$ ../leo-ctx5/parrot a.pir
it says:
PackFile_FixupTable_unpack: Unknown fixup type 1953066601!
PackFile_unpack segment 'FIXUP_luaglobals.pir' failed
$ make realclean and be sure to delete your .pbc files too. You can't
share PBCs between trunk and leo-ctx5 - run a.pir in distinct
directories.
klaas-jan
leo
NCI: passing PMCs and using them
hi,
I'm currently trying to check out the NCI. As my Lua compiler only uses
PMCs (and not I/N/S registers), calling C functions will only be done
with PMCs.
I couldn't find info on this matter: suppose I have this string PMC, how
can I access the actual string in it?
so, when calling:
void print_pmc(void *PMC)
{
// how to access the string in P?
}
Thanks,
klaas-jan
