Hey ,
I wonder if there is some documentation on how the freeglut library
is being used statically, and how it could be changed to be used dynamically.
I read this over and would like to say a little bit about this part here
___________________________________________________
2.1.2 Implement PDL objects with C-level method modifiers
Here we address problems of efficiency by building the new PDL
object structure with direct support for method modifiers and
possibly some MOP stuff as well. The idea is that we replace the
static build-time generation of code for threading by roles and
method modifiers to do the same. The pupose of the C level
support is to ensure the needed level of performance is available
to the computations. This support would consist of essentially an
array/chain/list of pointers to C functions. The nice thing about
this approach is that we can start with an implementation that
drops right back to perl for the implementation. Once we've
ironed out a working engine, we can roll things back into
C/assembly/JIT...
___________________________________________________
I wonder if the Ram usage problem "for example with the spherical dynamic
simulation"
is because of the "static library "opengl32... , if this library were dynamic
would
this Ram usage problem be fixed, I could see PDL competing with GPU's , if that
problem and the multi-thread problem were fixed, I have been looking at some
masm32 multi-threading; it seems to show a good demo of how to use
multi-threading in the windows environment, yet it is a low level procedure
and needs to be rewritten in "gas" for the gcc compiler to work with PDL...
freepascal has some demos as well for OpenCL, and OpenGL that are
Multi-Architecture
that might be of good use here too ...
I think to be really fast tho would mean rewriting perl in a "gas" mode, maybe
where
each perl keyword is put into the "gas" assembly, that would redefine modules..
in the assembly sense.. if only to pass that processes to multi-threads or to
the GPU...
that might be overkill, or it might be necessary, as much as I like perl, and
am still learning C;
the miryid's of other languages loom in the midst, where I find Assembly being
the real answer
to the vast amount of problems, plaguing my programming limitations .. yet the
amount or resources
in perl could be better utilized if the architecture of it is not interpreted ,
really compiled..
no not interpreted or compiled more like both at the same time, where the code
can be changed
as it is being run, more like a batch file ...
just some thoughts ...
Cheers
-Mark
#########################################33
----- Forwarded Message -----
From: "[email protected]"
<[email protected]>
To: [email protected]
Sent: Sunday, April 21, 2013 3:33 PM
Subject: PDL-porters Digest, Vol 92, Issue 13
Send PDL-porters mailing list submissions to
[email protected]
To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.jach.hawaii.edu/mailman/listinfo/pdl-porters
or, via email, send a message with subject or body 'help' to
[email protected]
You can reach the person managing the list at
[email protected]
When replying, please edit your Subject line so it is more specific
than "Re: Contents of PDL-porters digest..."
Today's Topics:
1. PDLv3 engine architecture (A Moo-dest Proposal) (Chris Marshall)
----------------------------------------------------------------------
Message: 1
Date: Sun, 21 Apr 2013 18:33:26 -0400
From: Chris Marshall <[email protected]>
To: pdl-porters <[email protected]>
Subject: [Pdl-porters] PDLv3 engine architecture (A Moo-dest Proposal)
Message-ID:
<CAPTtexLwcbdkGkqXiYcnVHP7LSC1yOMKyEES1A1PdfKO=6D=z...@mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"
I've attached the PDF version for readability and the text
only one here for reference. --Chris
Abstract
The version 2.x of the Perl Data Language (PDL) is a Perl5
extension module providing a mature framework for numerical
computation in perl. It supports scientific "array type" objects
consisting of binary buffers of supported C data types together
with arbitrary multi-dimensional indexing and automatic
computational threading across multiple dimensions (implicit
and/or explicit). This makes possible compact and powerful
numeric computation that can be composed from simple statements
of method chains...and it is fast!
However, as functional as the current implementation of the PDL
objects and their thread-enabled code generation is, it is
lacking support for a number of features such as piddles of
arbitrary data types or transparent GPU or multi-core processing
kernel support outside of the high-level pthreads decomposition
of the underlying PDL thread operations. I believe that by
re-architecting PDL at the lowest level (the object and the
threading engine) we can better support new functionality and
features that could be prohibitive within the current v2.x
implementation.
This paper develops the ideas of a new PDL engine that is based
on concepts from meta-object programming (MOP) and appears to
address many of the current limitations with a number of
additional benefits as well. Always good to have in a refactoring
effort!
1 Train of Thought
This section details in a "stream of consciousness" fashion
various thoughts leading to the proposed architecture. Listing
them here in some for context and for cross-reference in case
they go somewhere further interesting of in need of further
documentation or development.
1.1 New engine as a C function chain
This was a hazy thought to have PDL operations taking place via a
chain of functions to replace the current static structure plus
PP code generation. The use of C-level functions (or methods)
would enable sufficient generality to support the existing
capability and allow for powerful extensions. E.g., the method
calls could be replaced by RPC routines allowing for operations
across a general network (cluster computations), by JIT
compilation to support more computational efficiency since more
FLOPs could be used to hide memory or IO traffic, or even runtime
code generation to replace our current compile-time generation.
1.2 Use Meta Object Programming
By using the MOP capabilities of Moose, it would be possible for
the new PDL engine to make use of the context in which is is run
to provide key functionality. Initially this was just a thought
and the enormous amount of dependencies of Moose to provide this
capability and the relatively poor performance made it seem like
the idea was heading in the right direction but not quite a
match.
1.3 pdl2 needs to be installable
Developing the pdl2 shell brought me up the Moose learning curve
(slowly). However, the extremely large number of Moose
dependencies meant that not as many PDL users were actually using
pdl2 with the new functionality. By default, it would fall back
to the original perldl implementation if Moose and Devel::REPL
were not found. This needed to be fixed before the hoped-for
unification and release of a new "version 2" PDL shell. And,
along came Moo and a CPAN developer who started a
re-implementation of Devel::REPL using Moo.
A couple of things came out of the Moose to Moo conversion idea.
One, we don't need all that MOP overhead all the time. Moo, a
pure perl module, would allow pdl2 to be implemented as compact
pure-perl code and would be a viable replacement for the pdl2 +
perldl duo. Second, the new concepts of Roles and Method
Modifiers seemed to map into the previously fuzzy ideas about
implementing the new PDL engine via chains of functions.
1.4 Roles can be applied to classes or instances
This was the missing link to connect the ideas of method
modifiers to a possible implementation relevant to PDL. Method
modifiers allow one to apply routines before, around, and after a
given method so it naturally maps onto the idea of PDL operations
as a chain of methods. The fact that they can be applied to
classes and object instances suggests they could support both
compile time operations/code and runtime (JIT) code operations.
1.5 But it needs to be FAST!!!
The whole MOP idea with method modifiers and roles sounds great
(and feels right!) but whatever implementation we use going
forward must be capable of getting out of the way of raw
performance. It is nice to have high level programming for
numeric computation but it is even better if you really could get
"C-level performance". Our current implementation has a tight, C
kernel of computation inside the thread loops wrappers. Using JIT
code generation or selection it should be possible to push more
of the data access into the computational code so that the PDL
performance really is the C performance.
1.6 Current PDL warts
The current PDL implementation has a number of problems that
should be addressed by any new architecture. Here is a quick list
for reference:
* Inconsistent support for inheritance
* Implement has-a via the $pdl->{PDL} hash key
* pdl_hdr as a special case
* Need consistency in implementation and performance
* Incomplete support for C data types
* No support for arrays of structs (or objects)
* Out of date and inconsistent OO implementation
* Make consistent throughout the PDL core
* Make compatible with Moose /Moo
* Move pdl_hdr hardwired stuff out of engine
* Undocumented thread and pthread architecture
* Even current experts haven't or can't document it
* Inconsistent usages in many places
* Need to unify the computational framework
2 PDLv3 Proposal
The previous section hit a number of the generative ideas along
the way to this architecture proposal. This section starts at the
general ideas of the implementation, then covers ideas for how
the implementation might accomplish various tasks in general, and
then a final section covers actual specifics of the
implementation. At the moment, that will need to be fleshed out
from pdl-porter discussions.
2.1 High-Level Architecture
2.1.1 Start with 'use Moo'
First, we refactor the existing OO code in PDL to use Moose/Moo
with the actual hard dependency being on Moo. Since Moo is small
and perl-only, we could even include a version of that in the
distribution to avoid version skew if that were needed. This
would allow for a fully consistent OO framework for PDLv3 and
would enable the use of Moose and full MOP as needed for the
actual implementations.
2.1.2 Implement PDL objects with C-level method modifiers
Here we address problems of efficiency by building the new PDL
object structure with direct support for method modifiers and
possibly some MOP stuff as well. The idea is that we replace the
static build-time generation of code for threading by roles and
method modifiers to do the same. The pupose of the C level
support is to ensure the needed level of performance is available
to the computations. This support would consist of essentially an
array/chain/list of pointers to C functions. The nice thing about
this approach is that we can start with an implementation that
drops right back to perl for the implementation. Once we've
ironed out a working engine, we can roll things back into
C/assembly/JIT...
2.1.3 PP becomes Roles for PDL threading
The cool thing about roles is that they can be applied to classes
or object instances (e.g., at compile time or at run time). In
addition, since they become part of the resulting class, the
costs of inheritance can be avoided with proper design. From the
top, I think we could start with a first implementation at
compile time which generated the equivalent of what we do now.
Once that is working, we would extend things to runtime supported
code generation which could tremendously speed up the PDL
footprint and probably reduce build time since we could start
with a minimal number of threadloops actually compiled and the
rest supported by conversion wrappers or some such.
2.2 PDLv3 Implementation Ideas
This is a list of various functionality needed or desired and my
idea for how it could be done within the proposed framework. I am
not a MOP or Moo/Moose guru but I hope the gist of these are true
if the specific details are not necessarily correct.
GPU Support via Roles
We could add a role for GPU support which puts
method modifiers around the PDL operations
which check for availability or use of a GPU
computing context. If one were present or
desired, it could transparently move data
if required or handle references to it as
required to track the computation and return a
result. Lazy returns would allow computations
to remain on the GPU until actually required
on the CPU.
PDLv2 Kickstart
The first version of the new implementation
might start as a role applied to the
PDLv2 compute framework. This could allow
back compatability to earlier PDL engine
functionality while enabling a transition to
new features as they come on line.
XS Level Moo[se] Support
The C-level method modifiers seem a
promising idea to enable both performance
and functionality. We should include the
Moose experts in the dialog for the new
implementation as they probably have some
ideas on how this might be best done.
Perl OPtrees to JIT Kernels Ideally, we could
use the underlying perl optrees as generated
by the compiler as the starting point for
threadloop code generation. It would be nice
to have $d=$a+$a*$b nicely generate a mult-add
threadloop kernel rather than loading $a twice
for each element of $d. Again, expert help
would be nice.
2.3 PDLv3 Architecture Specifics
This is pretty much blank for now since I think the above ideas
would need to be tested on a number of trial implementations to
sort out the details. We could start, of course, by refactoring
PDL to use Moo so that we're ready to jump in with the next phase
once that is figured out.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mailman.jach.hawaii.edu/pipermail/pdl-porters/attachments/20130421/3dd8a8e2/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PDLv3_Architecture_Proposed.pdf
Type: application/pdf
Size: 174239 bytes
Desc: not available
URL:
<http://mailman.jach.hawaii.edu/pipermail/pdl-porters/attachments/20130421/3dd8a8e2/attachment.pdf>
------------------------------
_______________________________________________
PDL-porters mailing list
[email protected]
http://mailman.jach.hawaii.edu/mailman/listinfo/pdl-porters
End of PDL-porters Digest, Vol 92, Issue 13
*******************************************_______________________________________________
Perldl mailing list
[email protected]
http://mailman.jach.hawaii.edu/mailman/listinfo/perldl
