Hello Robert,
Am 26.10.11 07:16, schrieb Robert Jacques:
On Tue, 25 Oct 2011 19:35:52 -0400, Jonny Dee <jon...@gmx.net> wrote:
Am 25.10.11 16:41, schrieb Robert Jacques:
On Tue, 25 Oct 2011 09:40:47 -0400, Jonny Dee <jon...@gmx.net> wrote:
[...]
Hi Robert,
Well, before I tell you what I would like to see I'll cite Wikipedia [1]:
"
[...]
- Discover and modify source code constructions (such as code blocks,
classes, methods, protocols, etc.) as a first-class object at runtime.
- Convert a string matching the symbolic name of a class or function
into a reference to or invocation of that class or function.
[...]
"
Here is what I would dream of for arbitrary objects/classes (not
necessarily known at compile-time):
- Query an object for its list of methods together with their
signatures. Select a method, bind some values to its arguments, call it,
and retrieve the return type (if any).
- Query an object for its public fields (at least), and provide a way to
get/set their values.
- Query an object's class for all implemented interfaces and its base
class.
- Query a module for all type definitions and provide a way to
introspect these types in more detail. For instance, it would be really
cool if I could find a class with name "Car" in module "cars", get a
list of all defined constructors, select one, bind values to the
constructor's parameters, and create a corresponding object.
[...]
Implementing such a DI container heavily depends on reflection, because
the DI container component doesn't know anything about the objects to be
created during runtime.
Qt also extends C++ with a reflection mechanism through the help of its
meta object compiler (moc). It analyses the C++ source code, generates
meta class definitions [6,7] and weaves them into your Qt class. Hence,
in Qt, you can query an object for fields, methods, interfaces, etc. and
you can call methods with arbitrary parameters, or you can instantiate a
class using an arbitrary constructor. Consequently, somone implemented a
DI container for C++ which is based on Qt and works more or less the
same way the Spring DI container does. You can build up object trees
simply by specifying such trees in an XML file.
I don't go into why dependency injection is a very powerful feature.
This is Martin Fowler's [3] job ;) But when I program with C++ I miss
such a flexible dependency injection mechanism a lot. And I hope this
will eventually be available for D.
Cheers,
Jonny
[1] http://en.wikipedia.org/wiki/Reflection_%28computer_programming%29
[2] http://en.wikipedia.org/wiki/Dependency_injection
[3] http://martinfowler.com/articles/injection.html
[4]
http://en.wikipedia.org/wiki/Spring_Framework#Inversion_of_Control_container_.28Dependency_injection.29
[5] http://qtioccontainer.sourceforge.net/
[6] http://doc.qt.nokia.com/stable/qmetaobject.html
[7]
http://blogs.msdn.com/b/willy-peter_schaub/archive/2010/06/03/unisa-chatter-reflection-using-qt.aspx
Hi Jonny,
Thank you for your informative (and well cited) post. It has provided me
with a new take on an old design pattern and some enjoyable reading. In
return, let me outline my opinion of reflection in D today, and
tomorrow, as it pertains to your wish list.
Many thanks to you, too, for your very elaborate answer :)
Reflection in D today is very different from the host of VM languages
that have popularized the concept. Being a compiled systems language,
actual runtime self-modification is too virus like to become at a
language level feature. However, given the compilation speed of D,
people have made proof of concept libraries that essentially wrapped the
compiler and dynamically loaded the result. As LDC uses LLVM, which has
a jit backend, I'd expect to see something get into and D 'eval' library
into etc eventually. (phobos uses the BOOST license, which isn't
compatible with LLVM).
I know, that "runtime self-modification" and runtime code generation is
a "dangerous" feature. And there really are rare cases where using such
an approach might justify the risc in using it. Although this feature is
not on my wish list, it might be good for generating dynamic proxies to
arbitrary object instances like they are used by some ORMs. See
Hibernate/NHibernate, for example [1,2]. Another example is
aspect-oriented programming. But while I can't see the exacty reason for
it, such a feature might indeed be a feature which is more appropriate
for VM languages.
Compile-time reflection and generation of code, on the other hand, is
something D does in spades. It fulfills your dream list, although I
think module level reflection might only be available in the github
version. The API design is still in flux and we are actively iterating /
improving it as find new uses cases and bugs. The current plan is to
migrate all the traits functions over to a special 'meta' namespace
(i.e. __traits(allMembers,D) => meta.allMembers(T) ). Good solid
libraries for each of the concepts I listed, (prototype objects,
duck-typing/casting or serialization), have been written using the
compile-time meta-programming features in D. So that's the good.
A absolutely agree! D's compile-time reflection is very good. And it's
hard to top this.
On the other hand, D's runtime capabilities are limited to
object.factory, the under implemented RTTI and library solutions which
manually expose information gathered by D's compile-time mechanisms. And
so far, these tools have been more than enough, from a functionality
point of view. Most of our desire for better runtime reflection stems
from a desire for efficiency, composition, cleanliness of user syntax
and simplification of library code. These are all important issues for
the widespread use of reflection based libraries, but they're not 'I
can't implement X' issues.
You are right, there IS certainly always some way to 'implement X'. But,
as you know, there is always a consideration of the effort you need to
implement X.
As for the future, I believe that the division in D between compile-time
and run-time reflection warrants a serious look at the design of the
run-time half of the system. To that end, I believe that implementing
reflection in a library would be the best way to experiment and iterate
an API. To that end, I have a proposal in the review queue to improve
std.variant which contains dynamic dispatch (i.e. the ability to get/set
public fields and call methods), duck-typeing/casting and
prototype-style objects. Notably, it's missing introspection
capabilities as thats what I'm most unsure about API wise, and simplest
to add. Designing reflection inside a library keeps Walter & Co's
bandwidth free for other bugs/features and provides a very good stress
test of D's meta-programming capabilities. (Both of which I think are
good things)
I've got no problem with an approach which puts runtime reflection
capabiities into a separate library. No matter were you look at, Java,
C#, or Qt, all have a library for reflection purpose. I do not see,
however, how this might be done without compiler support. As already
mentioned, Qt has its own moc compiler, which parses your C++ source
code and generates the necessary infra structure. I'm still a beginner
with respect to D, so I don't know what is really already possible, and
what not. But my current feeling is that a similar approach would also
be needed for D. If the D compiler itself, or another post-compiler
component should generate the meta information for runtime reflection is
another question. I could live with both, although I'd prefer the former
built-in one.
(I am soliciting feedback, if you care to take a look:
https://jshare.johnshopkins.edu/rjacque2/public_html/variant.mht)
Thanks for this link, I'll certainly have a look at it.
And there are many reasons we might want to experiment with D's runtime
reflection API instead of just copying someone. For example, take
Dependency Injection. Using Java-style reflection, DI suffers from the
Reflection injection and Unsafe Reflection security vulnerabilities. To
do DI safely, you have to validate your inputs and D's compile-time
reflection provides a perfect way to implement validated DI. Every time
I hear about some form of injection, be it SQL or JSON or Reflection,
hit the news, makes me think that 5-10 years from now well look back on
the use of unvalidated dynamic code constructs the same way we do about
null terminated arrays today.
Considering security vulnerabilities is of course a very important
matter. However, I think security must be assured a by software's
design. Enforcing it solely by programming language constructs will not
work for all cases. Particularly not, if this programming language
allows direct access to a computer's memory, like C, C++, and D does.
There is no sandbox out-of-the-box, where the compiled program runs in.
So if you have a private field in a class that carries a password, one
has to make sure it's only in memory as long as it is required. And if
it is not required anymore one should clear it out with zeros, for
example. So making a field private is by no means a secure solution. A
hacker will not give up just because the official API declares a field
to be private.
I consider 'unsafe reflection', as you call it, as a tool. Let's compare
it to a knife, for instance. It is really a useful tool, I think nobody
doubts, but at the same time it can be very dangerous. But inspite of
this fact, everybody has not only one at home, I guess. Pointers are a
useful tool, too. But they are also dangerous and still available in D,
which is a good thing, because you can use this tool where needed. And
actually, I don't think runtime reflection must be unsafe. The
reflection mechanism provides type information for an object's
properties, functions, arguments, etc. So validated DI is even possible
with runtime reflection. Let's consider the the XML configuration of
object trees once again. If you want to store a string to an int-field,
for instance, then the DI container can refuse to do this, because it
has access to all information required to enforce correct value types.
One more use case for reflection is data binding with GUI components.
This approach is heavily used in Windows Presentation Foundation library
[3,4]. GUI components can update an object's properties by using
reflection. You don't need to register listeners for this purpose anymore.
That said, I don't think dynamic code constructs will be old-fashioned
in 5-10 years, because you don't have to go an "unvalidated dynamic" way
with runtime reflection.
BTW, as this thread also discusses an opt-in or opt-out implementation
for runtime reflection. I'd prefer an opt-out way, too. Code bloat (I
guess the binaries are meant) is not as bad as it might sound with
today's memory sizes. And if one wants to avoid it for optimization
purpose, one can do it. The use of D as a systems programming language
for embedded systems is, as I've read somewhere, not a first citizen
anyway, because you'll get a lot of code for the garbage collection
mechanism.
Cheers,
Jonny
[1] http://en.wikipedia.org/wiki/Hibernate_%28Java%29
[2] http://en.wikipedia.org/wiki/Nhibernate
[3] http://en.wikipedia.org/wiki/Windows_Presentation_Foundation
[4]
http://blogs.msdn.com/b/wpfsdk/archive/2006/10/19/wpf-basic-data-binding-faq.aspx
