Re: Flow-Design: OOP component programming
On Wednesday, 14 February 2018 at 22:43:21 UTC, Ola Fosheim Grøstad wrote: On Wednesday, 14 February 2018 at 18:43:34 UTC, Mark wrote: Luna [1], a new programming language that was recently mentioned on Reddit, also appears to take this "flow-oriented design" approach. It's purely functional, not OO, and I'm curious to see how it evolves. [1] http://www.luna-lang.org Many flow based languages... https://en.wikipedia.org/wiki/Dataflow https://en.wikipedia.org/wiki/Max_(software) https://en.wikipedia.org/wiki/Reactive_programming does have classes. thanks for the link.
Re: Flow-Design: OOP component programming
On Wednesday, 14 February 2018 at 18:43:34 UTC, Mark wrote: Luna [1], a new programming language that was recently mentioned on Reddit, also appears to take this "flow-oriented design" approach. It's purely functional, not OO, and I'm curious to see how it evolves. [1] http://www.luna-lang.org Many flow based languages... https://en.wikipedia.org/wiki/Dataflow https://en.wikipedia.org/wiki/Max_(software) https://en.wikipedia.org/wiki/Reactive_programming
Re: Flow-Design: OOP component programming
On Wednesday, 14 February 2018 at 19:53:31 UTC, H. S. Teoh wrote: On Wed, Feb 14, 2018 at 06:43:34PM +, Mark via Digitalmars-d wrote: On Wednesday, 14 February 2018 at 09:39:20 UTC, Luís Marques wrote: > It seems that someone once again rediscovered the benefits > of component programming, in the context of OOP, but (as > usual) without the more mathematical and principled approach > of something like ranges and algorithms: > > [...] Luna [1], a new programming language that was recently mentioned on Reddit, also appears to take this "flow-oriented design" approach. It's purely functional, not OO, and I'm curious to see how it evolves. [...] This so-called "flow-oriented design" is hardly a new idea. It has been around since Lisp and the Unix command-line. And of course, D ranges benefit greatly from it. :-) T It's not a new idea but they've added a visual twist to it - you can see the program's data flow graph, manipulate it in various ways, and see how your input changes as it flows through the graph. So that's nice.
Re: Flow-Design: OOP component programming
On Wed, Feb 14, 2018 at 06:43:34PM +, Mark via Digitalmars-d wrote: > On Wednesday, 14 February 2018 at 09:39:20 UTC, Luís Marques wrote: > > It seems that someone once again rediscovered the benefits of > > component programming, in the context of OOP, but (as usual) without > > the more mathematical and principled approach of something like > > ranges and algorithms: > > > > [...] > > Luna [1], a new programming language that was recently mentioned on > Reddit, also appears to take this "flow-oriented design" approach. > It's purely functional, not OO, and I'm curious to see how it evolves. [...] This so-called "flow-oriented design" is hardly a new idea. It has been around since Lisp and the Unix command-line. And of course, D ranges benefit greatly from it. :-) The power of this idiom comes from the way it confers symmetry of API across diverse components. In the traditional situation, if a library X exports methods foo() and goo(), and another library Y exports methods bar() and car(), then you cannot simply plug Y in where X is used, because their APIs are incompatible, and you need to write proxying code to bridge the "API impedance mismatch" between them. For sufficiently complex APIs, this is highly non-trivial, and often leads to a lot of boilerplate like proxy objects, shim code, and other such artifacts. For example, if X calls Y and Y calls Z, and X, Y and Z all have different APIs, then removing Y from between X and Z will require rewriting parts of X so that it will fit with Z's API. And often this may not even be possible without essentially rewriting the entire subsystem. However, under the component programming model, both X and Y would be unified under a single, universal API: input and output. This makes their APIs symmetric: they become interchangeable with each other, from the API point of view. Now you can insert/remove components in the middle of a pipeline without needing to rewrite everything around them, because the symmetry of the API ensures that the new component(s) will simply "just fit", no matter what order you attach them. Of course, the limitation of this approach is that it only applies to problems that are amenable to be modelled as a linear pipeline. For more complex problems that require non-linear algorithms, it will be necessary to use some other kind of API to accomplish what's needed. Though, the linearizable pieces of such algorithms can still benefit from the component / pipeline approach. But even in non-linear problems, if you can boil down the problem to its barest essentials and unify the APIs of the relevant modules that way, then you can still reap the benefits of having an API that's symmetric across different modules. One example of this is user-defined numerical types: in languages that support operator overloading, the expression syntax serves as the unifying "API"; as long as your custom numeric type conforms to this API (i.e., it is symmetric w.r.t expression syntax), you can simply just "plug it in" and it will Just Work(tm), for the most part (complications like floating-point idiosyncrasies aside). Finding this sort of symmetry in API design is a hard problem in general, though. While almost everyone can agree on "input + output" in the pipeline model, a universal API that encapsulates more complex tasks may not be so obvious. And as long as you can't get everyone to agree to it, it will be difficult to interoperate the respective modules. This is why API design is very hard. :-) T -- "Real programmers can write assembly code in any language. :-)" -- Larry Wall
Re: Flow-Design: OOP component programming
On Wednesday, 14 February 2018 at 09:39:20 UTC, Luís Marques wrote: It seems that someone once again rediscovered the benefits of component programming, in the context of OOP, but (as usual) without the more mathematical and principled approach of something like ranges and algorithms: [...] Luna [1], a new programming language that was recently mentioned on Reddit, also appears to take this "flow-oriented design" approach. It's purely functional, not OO, and I'm curious to see how it evolves. [1] http://www.luna-lang.org
Re: Flow-Design: OOP component programming
On Wednesday, 14 February 2018 at 09:39:20 UTC, Luís Marques wrote: It seems that someone once again rediscovered the benefits of component programming, in the context of OOP, but (as usual) without the more mathematical and principled approach of something like ranges and algorithms: <http://geekswithblogs.net/theArchitectsNapkin/archive/2011/03/19/flow-design-cheat-sheet-ndash-part-i-notation.aspx> BTW, I just wrote my DConf proposal. I've been experimenting with a different style of not-quite-OOP in a real project and so far I'm really happy with the results. I've been making use of Jean-Louis' openmethods.d library, as well as other D features and techniques. The result is a quite nice balance of simplicity, expressiveness and performance. I'm really looking forward to telling you all about it :-) "Flow-Orientation is about tackling complexity at its root cause: that´s dependencies." That's an interesting statement (from that article). Seems we are getting closer and closer to modelling programming in accordance with how the brain programs itself. Kinda of makes sense really - nature seemed to work this all out a long, long time ago. Small units. Data just flows in and out. The units don't 'know' each other. Their are no explicit dependencies between the units themselves (hence neuroplasticity). Instead, what's important, are pathways by which they can communicate (concatenate) their input and output, and the subsequent data flows that arise from that collaboration. In nature, increasing mass (complexity) arises from simple components. If only that were so in the world of programming.
Flow-Design: OOP component programming
It seems that someone once again rediscovered the benefits of component programming, in the context of OOP, but (as usual) without the more mathematical and principled approach of something like ranges and algorithms: <http://geekswithblogs.net/theArchitectsNapkin/archive/2011/03/19/flow-design-cheat-sheet-ndash-part-i-notation.aspx> "The purpose of FUs is to process input and produce output. FUs are transformational. However, FUs are not called and do not call other FUs. There is no dependency between FUs. Data just flows into a FU (input) and out of it (output). From where and where to is of no concern to a FU. This way FUs can be concatenated in arbitrary ways: (...) Each FU can accept input from many sources and produce output for many sinks: (...) Connected FUs form a flow with a start and an end. Data is entering a flow at a source, and it´s leaving it through a sink. (...) You could say, Flow-Orientation is about tackling complexity at its root cause: that´s dependencies. “Natural” dependencies are depicted naturally, i.e. implicitly. And whereever possible dependencies are not even created. Functional units don´t know their collaborators within a flow. This is core to Flow-Orientation. That makes for high composability of functional units." BTW, I just wrote my DConf proposal. I've been experimenting with a different style of not-quite-OOP in a real project and so far I'm really happy with the results. I've been making use of Jean-Louis' openmethods.d library, as well as other D features and techniques. The result is a quite nice balance of simplicity, expressiveness and performance. I'm really looking forward to telling you all about it :-)
Suggestion for Walter/Andrei: follow-up to Component Programming in D
I recently reread Walter's 2012 article, as it's _the_ piece I recommend to others who want to know about D from a technical perspective: http://www.drdobbs.com/architecture-and-design/component-programming-in-d/240008321 Given the great success of H.S Teoh's subsequent article with a long example of implementing a calendar, http://wiki.dlang.org/Component_programming_with_ranges https://www.reddit.com/r/programming/comments/1jtzez/component_programming_with_ranges/ to the point where there was even a CppCon talk stepping through Teoh's calendar example using Boost Ranges and the equivalent C++ syntax, https://youtube.com/watch?v=mFUXNMfaciE it would be great to have a follow-up article that expanded on the original article, talked about practical experience over the subsequent years, and emphasized the context and differences from other languages of this concept, some of which Andrei mentioned in the reddit comments at that time: https://www.reddit.com/r/programming/comments/10u6sk/component_programming_in_d/ I think a follow-up article, perhaps published as a post on the dlang blog, would get more attention nowadays and maybe get more people to read the original article, similar to how Voldemort types got much more attention on reddit recently: https://www.reddit.com/r/programming/comments/telhj/voldemort_types_in_d/ https://www.reddit.com/r/programming/comments/523hm3/til_the_d_language_has_a_voldemort_type/ I would like to read such an article, and I think it would get more attention on the language, ie provide some marketing.
Re: Non-pipeline component programming
Sorry for the very late reply; my internet connection was down for 2
weeks and only came back yesterday.
On Thu, Sep 04, 2014 at 08:03:51AM +, David Ledez via Digitalmars-d wrote:
Dear T,
I've read with a great attention your reply regarding ECS and OOP. You
mention that you work on high-dimensional polytope and a data
structure able to support fast topological query with compact storage.
I would be very interested in having a look to your data structure. Do
you have any document about your design and the concepts? Any github?
I'm sorry to disappoint, but I was merely alluding to the fact that
separating the range-based interface from the container allows one to
freely choose the most suitable structure to represent the polytope,
while using a proxy type to expose a range interface to allow generic
algorithms to work with it. My code is currently still very much a work
in progress, and I don't really have very much to show for it yet.
The basic design, however, is along these lines:
- A Polytope interface representing any abstract polytope-like structure
in the program (see below for details);
- A ConcretePolytope class (implementing a Polytope) that stores a face
lattice, represented as a graph, the nodes of which are obtained by
running a convex hull + lattice enumeration algorithm on the source
data.
- The Polytope interface has abstract methods for enumerating
sub-polytopes (as input/forward ranges), which concrete classes like
ConcretePolytope implement; the idea being that a face of a Polytope
is also a Polytope, and while the underlying implementation may be
different (e.g., the range of Polytopes representing the
(n-1)-dimensional facets may have a different representation from,
say, the 1-dimensional edges), they implement the same interface so
generic algorithms can work with them without special-casing. This is,
of course, just plain old OO, except perhaps for the range API part.
The sub-polytope enumeration methods return proxy types that implement
the range API, with Polytope as the element type, so you can use the
usual std.algorithm / std.range tools on them -- .find, .filter,
.chain, .cycle, .takeN, etc.. The proxy types are themselves
interfaces, meaning that you can write a proxy Polytope implementation
that performs some complex query over the lattice structure of
ConcretePolytope, and still present a nice range API over its
elements, and you can pass this to any of the usual range-based
algorithms for searching, filtering, composing, etc., with no code
change required in the algorithms. Again, this is just plain ole OO
(Liskov substitution, in particular), seasoned with some D-flavored
range-styled code. So you could, in theory ('cos I haven't actually
implemented this part yet), construct some complex query over the
polytope, and ask the program to enumerate, say, all 5D faces subject
to some given filter, say they must have a certain number of 3D
subfaces. In code, it could be as simple as:
ConcretePolytope p = ...;
auto query = new PolyQuery(...) // construct complex proxy Polytope
from p
// The query proxy object is still a Polytope
assert(is(typeof(query) : Polytope));
query.faces(5) // get range of faces of dimension 5
.filter!((p) = p.faces(3).count 10) // filter by faces with
more than 10 3D faces
.copy((p) {
// print a list of face ID's in the result
writefln(%s, p.id);
});
Notice how the second block of code can actually apply directly to
ConcretePolytope (since it implements the Polytope interface), or to a
PolyQuery object (which represents some subset of the
ConcretePolytope's surface), or to anything, really, that implements
the Polytope interface -- perhaps a specific face of a
ConcretePolytope. Due to the range API implemented by the .faces
method, you can then process this range downstream using all of the
usual range-based algorithms.
Also note that by abstracting the implementation details of
ConcretePolytope away, it is also possible to implement a different kind
of concrete polytope, say one where the full face lattice isn't computed
beforehand, like I'm doing now, but which is lazily computed depending
on what kind of queries you run on it -- i.e., if you call faces(6),
then it will lazily construct the 6-dimensional faces of the polytope on
the fly as you iterate over the range, and if you iterate over, say, the
5-dimensional faces of a subset of the 6D faces, then the lattice
enumeration algorithm will only be run for those subset of faces, rather
than the entire polytope. As you probably know, the face lattice of a
general polytope can be exponential in the size of its initial
vertex/hyperplane representation, so allowing this kind of lazy
evaluation *without needing to change any downstream code* is a big
advantage.
It is also possible to implement a
Re: Non-pipeline component programming
Dear T, I've read with a great attention your reply regarding ECS and OOP. You mention that you work on high-dimensional polytope and a data structure able to support fast topological query with compact storage. I would be very interested in having a look to your data structure. Do you have any document about your design and the concepts? Any github? This thred is really extremely instructive. I m looking forward to reading your reply. Regards, - David - On Wednesday, 12 February 2014 at 17:38:30 UTC, H. S. Teoh wrote: Sorry for this belated reply, I have been rather busy with other matters. On Tue, Feb 04, 2014 at 04:12:48AM +, digitalmars-d-boun...@puremagic.com wrote: I loved reading Walter's component programming article in Dr. Dobb's [0] in late 2012. I had missed H. S. Teoh's mid 2013 article on calendar textual formatting using the component approach [1], but fortunately Ali brought it to my attention recently, and I also find it absolutely fascinating! Thanks! I think what's most interesting with Teoh's article (and I think that was Ali's point when he mentioned the article to me) is that the calendar example is not as an obvious target for the component approach, or at least that the design and implementation is not as obvious for someone new to that approach. Now, while Teoh's example is much more complex than Walter's, both examples are for cases of pipelined problems (source - filter1 - filter2 - sink). What I have been wondering during the last few days is how much this component programming approach could be applied to scenarios where you would normally have a jumble of objects. For instance, try to picture a game or simulation where spaceships fire at each other, pick up objects, communicate, and so on, or something like that. My instinct would be to code a solution which would be classified as typical OOP code. Would it be possible to come up with a solution that would be more in the spirit of component programming? Or are such solutions only practical/applicable for pipeline-like scenarios? [...] I would say that while it's insightful to apply different paradigms to solve the same problem, one shouldn't make the mistake of shoehorning *everything* into the same approach. This is what Java does with OO, for example, to the detriment of every other paradigm, and frankly, after a while all those singleton classes with static methods just start to smell more and more like ways of working around the OO rather than with it. Having said that, though, the component approach is highly applicable, often in unexpected areas and unexpected ways, esp. when you couple it with D's range-based concept. There are certainly algorithms where it makes more sense to treat your data as a graph rather than a linear sequence of nodes, but it's also true that a good percentage of all code is just variations on linear processing, so pipelined component-style programming would definitely be applicable in many places. And nothing says you can't intermix component-style code with OO, or something else. One key insight is that sometimes you want to separate the object itself from a range over that object -- for example, I work with polytopes (higher-dimensional analogues of polygons and polyhedra), and it's useful to have, say, a range over all vertices, or a range over all edges, but it's also useful to separate these ranges from the polytope itself, which can be stored in a more compact form, or in a form that's more amenable to fast queries, e.g., find all faces that contain vertex X without needing to iterate over every face in the polytope (which you'd end up doing if you use filter() on the range of all faces). The query function can return a range over faces, so that it can be piped into other range-based functions for further processing. Thus, you can have a mix of different paradigms complementing each other. The other underlying theme in my article, which is also one of the key points of the Jackson Structured Programming that I alluded to, is the identification and separation of mismatching structures in order to simplify the code and eliminate code smells caused by ad hoc methods of structure conflict resolution (boolean flags are a common symptom of this malady). This isn't limited to pipelined programs, but applies in general. One could analyze OOP in this way, for example. OO lore says that objects should be cohesive and loosely-coupled -- we could say that cohesiveness means that the data stored in the object has corresponding structures, and loose coupling means that if an object's data has conflicting structures, it's time to consider splitting it into two different objects instead. T
Re: Non-pipeline component programming
On Wednesday, 12 February 2014 at 17:38:30 UTC, H. S. Teoh wrote: I would say that while it's insightful to apply different paradigms to solve the same problem, one shouldn't make the mistake of shoehorning *everything* into the same approach. This is what Java does with OO, for example, to the detriment of every other paradigm, and frankly, after a while all those singleton classes with static methods just start to smell more and more like ways of working around the OO rather than with it. I found myself using singelton classes more and more until I decided it was time to drop a strict OO approach. Having said that, though, the component approach is highly applicable, often in unexpected areas and unexpected ways, esp. when you couple it with D's range-based concept. There are certainly algorithms where it makes more sense to treat your data as a graph rather than a linear sequence of nodes, but it's also true that a good percentage of all code is just variations on linear processing, so pipelined component-style programming would definitely be applicable in many places. And nothing says you can't intermix component-style code with OO, or something else. That's what I've been doing for the last 1 1/2 years. I use classes where it makes _sense_, not as the ruling paradigm, then add structs (components), ranges and templates. The good thing about the freedom D offers is that it encourages you to think about the fundamental logic of your program and use tailor made solutions for a given problem - instead of a one size fits all approach that is bound to lead you down a cul de sac. In a way D has given the power back to the programmer's brain. One key insight is that sometimes you want to separate the object itself from a range over that object -- for example, I work with polytopes (higher-dimensional analogues of polygons and polyhedra), and it's useful to have, say, a range over all vertices, or a range over all edges, but it's also useful to separate these ranges from the polytope itself, which can be stored in a more compact form, or in a form that's more amenable to fast queries, e.g., find all faces that contain vertex X without needing to iterate over every face in the polytope (which you'd end up doing if you use filter() on the range of all faces). The query function can return a range over faces, so that it can be piped into other range-based functions for further processing. Thus, you can have a mix of different paradigms complementing each other. The other underlying theme in my article, which is also one of the key points of the Jackson Structured Programming that I alluded to, is the identification and separation of mismatching structures in order to simplify the code and eliminate code smells caused by ad hoc methods of structure conflict resolution (boolean flags are a common symptom of this malady). This isn't limited to pipelined programs, but applies in general. One could analyze OOP in this way, for example. OO lore says that objects should be cohesive and loosely-coupled -- we could say that cohesiveness means that the data stored in the object has corresponding structures, and loose coupling means that if an object's data has conflicting structures, it's time to consider splitting it into two different objects instead. T
Re: Non-pipeline component programming
I've written a coffeescript Entity Component System once(https://github.com/NikkiKoole/ces), I am planning on using https://github.com/elvisxzhou/artemisd for my D needs and I am hoping it's good (and hoping it does the components in some cache friendly manner). how does your nitro compare?
Re: Non-pipeline component programming
http://t-machine.org/index.php/2014/03/08/data-structures-for-entity-systems-contiguous-memory/ has some very nice insights in the memory issue.
Re: Non-pipeline component programming
On Thursday, 4 September 2014 at 10:12:13 UTC, Chris wrote: On Wednesday, 12 February 2014 at 17:38:30 UTC, H. S. Teoh wrote: I would say that while it's insightful to apply different paradigms to solve the same problem, one shouldn't make the mistake of shoehorning *everything* into the same approach. This is what Java does with OO, for example, to the detriment of every other paradigm, and frankly, after a while all those singleton classes with static methods just start to smell more and more like ways of working around the OO rather than with it. I found myself using singelton classes more and more until I decided it was time to drop a strict OO approach. In Smalltalk and Eiffel, where everything is an object this tends not to be a problem, because they lack modules anyway. So objects with class methods play the role of modules in a way that doesn't feel so strange like in Java and C#. Which given its package support, it would be more natural to associate certain code with the package, not with objects. C# extension methods feel funny because of their requirement to be encapsulated into a (dummy) class, for example. Having said that, though, the component approach is highly applicable, often in unexpected areas and unexpected ways, esp. when you couple it with D's range-based concept. There are certainly algorithms where it makes more sense to treat your data as a graph rather than a linear sequence of nodes, but it's also true that a good percentage of all code is just variations on linear processing, so pipelined component-style programming would definitely be applicable in many places. And nothing says you can't intermix component-style code with OO, or something else. That's what I've been doing for the last 1 1/2 years. I use classes where it makes _sense_, not as the ruling paradigm, then add structs (components), ranges and templates. The good thing about the freedom D offers is that it encourages you to think about the fundamental logic of your program and use tailor made solutions for a given problem - instead of a one size fits all approach that is bound to lead you down a cul de sac. In a way D has given the power back to the programmer's brain. This was actually my first issue when I learned OO in Turbo Pascal 5.5, my first OO language. It was a learning process to understand what should become an object, and what should stay as usual procedural (functions/records/units) code. Regarding component programming in general, this used to be a good book on the subject. Only read the first edition (based in Component Pascal), the second was updated to more modern languages. http://www.amazon.com/Component-Software-Object-Oriented-Programming-Addison-Wesley/dp/032175302X -- Paulo
Re: Non-pipeline component programming
Sorry for this belated reply, I have been rather busy with other matters. On Tue, Feb 04, 2014 at 04:12:48AM +, digitalmars-d-boun...@puremagic.com wrote: I loved reading Walter's component programming article in Dr. Dobb's [0] in late 2012. I had missed H. S. Teoh's mid 2013 article on calendar textual formatting using the component approach [1], but fortunately Ali brought it to my attention recently, and I also find it absolutely fascinating! Thanks! I think what's most interesting with Teoh's article (and I think that was Ali's point when he mentioned the article to me) is that the calendar example is not as an obvious target for the component approach, or at least that the design and implementation is not as obvious for someone new to that approach. Now, while Teoh's example is much more complex than Walter's, both examples are for cases of pipelined problems (source - filter1 - filter2 - sink). What I have been wondering during the last few days is how much this component programming approach could be applied to scenarios where you would normally have a jumble of objects. For instance, try to picture a game or simulation where spaceships fire at each other, pick up objects, communicate, and so on, or something like that. My instinct would be to code a solution which would be classified as typical OOP code. Would it be possible to come up with a solution that would be more in the spirit of component programming? Or are such solutions only practical/applicable for pipeline-like scenarios? [...] I would say that while it's insightful to apply different paradigms to solve the same problem, one shouldn't make the mistake of shoehorning *everything* into the same approach. This is what Java does with OO, for example, to the detriment of every other paradigm, and frankly, after a while all those singleton classes with static methods just start to smell more and more like ways of working around the OO rather than with it. Having said that, though, the component approach is highly applicable, often in unexpected areas and unexpected ways, esp. when you couple it with D's range-based concept. There are certainly algorithms where it makes more sense to treat your data as a graph rather than a linear sequence of nodes, but it's also true that a good percentage of all code is just variations on linear processing, so pipelined component-style programming would definitely be applicable in many places. And nothing says you can't intermix component-style code with OO, or something else. One key insight is that sometimes you want to separate the object itself from a range over that object -- for example, I work with polytopes (higher-dimensional analogues of polygons and polyhedra), and it's useful to have, say, a range over all vertices, or a range over all edges, but it's also useful to separate these ranges from the polytope itself, which can be stored in a more compact form, or in a form that's more amenable to fast queries, e.g., find all faces that contain vertex X without needing to iterate over every face in the polytope (which you'd end up doing if you use filter() on the range of all faces). The query function can return a range over faces, so that it can be piped into other range-based functions for further processing. Thus, you can have a mix of different paradigms complementing each other. The other underlying theme in my article, which is also one of the key points of the Jackson Structured Programming that I alluded to, is the identification and separation of mismatching structures in order to simplify the code and eliminate code smells caused by ad hoc methods of structure conflict resolution (boolean flags are a common symptom of this malady). This isn't limited to pipelined programs, but applies in general. One could analyze OOP in this way, for example. OO lore says that objects should be cohesive and loosely-coupled -- we could say that cohesiveness means that the data stored in the object has corresponding structures, and loose coupling means that if an object's data has conflicting structures, it's time to consider splitting it into two different objects instead. T -- I've been around long enough to have seen an endless parade of magic new techniques du jour, most of which purport to remove the necessity of thought about your programming problem. In the end they wind up contributing one or two pieces to the collective wisdom, and fade away in the rearview mirror. -- Walter Bright
Re: Non-pipeline component programming
On 2/7/2014 12:14 AM, Paul Freund wrote:
You're example almost works. With the EntityComponentManager API it
works like this:
auto shipEntities = ecm.query!SpaceShip();
foreach (i, spaceship; taskPool.parallel(shipEntities))
{
auto shipData = spaceship.getComponent!SpaceShip();
shipData.name = format(Ship %i, i);
}
Now that unittests are written, documentation is on its way and should
be ready in a few days.
Ah, so there's no data-orientation here. It's strictly entity-centric.
Or do you have a way to iterate components independently of entities?
Re: Non-pipeline component programming
On Friday, 7 February 2014 at 08:09:10 UTC, Mike Parker wrote:
On 2/7/2014 12:14 AM, Paul Freund wrote:
You're example almost works. With the EntityComponentManager
API it
works like this:
auto shipEntities = ecm.query!SpaceShip();
foreach (i, spaceship; taskPool.parallel(shipEntities))
{
auto shipData = spaceship.getComponent!SpaceShip();
shipData.name = format(Ship %i, i);
}
Now that unittests are written, documentation is on its way
and should
be ready in a few days.
Ah, so there's no data-orientation here. It's strictly
entity-centric. Or do you have a way to iterate components
independently of entities?
There will be a way to iterate directly over components.
Component fields will automatically be split into an structure of
arrays.
So let's assume we have:
struct Point {int x; int y; int z; }
@Component struct Translation { int a; Point b; Point c; }
Nitro will store Translation like this, so it is even possible to
iterate over parts of components:
Entity[]
int[] for a
int[] for b.x
int[] for b.y
int[] for b.z
int[] for c.x
int[] for c.y
int[] for c.z
Nitros getComponent functions will return Accessor!Translation,
so if you do this, only int[] for b.x is accessed.
foreach(e; ecs.query!Translation)
{
auto trans = e.getComponent!Translation();
trans.b.x = 0;
}
Re: Non-pipeline component programming
On Friday, 7 February 2014 at 09:47:43 UTC, Zoadian wrote: On Friday, 7 February 2014 at 08:09:10 UTC, Mike Parker wrote: Nitro will store Translation like this, so it is even possible to iterate over parts of components: Entity[] int[] for a int[] for b.x int[] for b.y int[] for b.z int[] for c.x int[] for c.y int[] for c.z This looks nice and everything, but won't it slow down access times quite a lot?
Re: Non-pipeline component programming
Am Fri, 07 Feb 2014 09:58:52 + schrieb Francesco Cattoglio francesco.cattog...@gmail.com: On Friday, 7 February 2014 at 09:47:43 UTC, Zoadian wrote: On Friday, 7 February 2014 at 08:09:10 UTC, Mike Parker wrote: Nitro will store Translation like this, so it is even possible to iterate over parts of components: Entity[] int[] for a int[] for b.x int[] for b.y int[] for b.z int[] for c.x int[] for c.y int[] for c.z This looks nice and everything, but won't it slow down access times quite a lot? Yeah, looks like a point is now spread over 3 cache lines. As long as you access memory strictly forwards or backwards, x86 can at least stream the data from RAM in the background. Now if you check for collisions of 2 objects you'll need up to 6 spread out cache lines, where a compact layout would need one cache-line per object (2). The trouble is that RAM is not only much slower to access, but also it seems to take a while for the data from RAM to arrive in the CPU, making it important to know ahead of time where the next data has to be read from. The combined effect can mean a ~100 times slow down. So x86 got a sophisticated prefetcher, that can track several sequential memory reads (e.g. up to 16) either forwards or backwards through memory and loads those locations into CPU caches before they are needed. If you access pattern is seemingly random to the CPU it might in the worst case still try to prefetch, clogging the memory bandwidth and not achieving anything useful :p. -- Marco
Re: Non-pipeline component programming
On Thursday, 6 February 2014 at 05:27:22 UTC, Mike Parker wrote:
A cursory look at Nitro suggests they are using an ECS with
DOP. Given that they're providing ranges for iterating the
data, it will fit into D's range-based component pipelines
(std.algorithm and such).
That seems cool. I can only imagine the possibilities...
auto shipComponents = components.filter!(a =
a.hasComponent!SpaceShip);
foreach (i, spaceship; taskPool.parallel(shipComponents))
{
spaceship.name = format(Ship %i, i);
}
Re: Non-pipeline component programming
On Thursday, 6 February 2014 at 13:47:22 UTC, Meta wrote:
On Thursday, 6 February 2014 at 05:27:22 UTC, Mike Parker wrote:
A cursory look at Nitro suggests they are using an ECS with
DOP. Given that they're providing ranges for iterating the
data, it will fit into D's range-based component pipelines
(std.algorithm and such).
That seems cool. I can only imagine the possibilities...
auto shipComponents = components.filter!(a =
a.hasComponent!SpaceShip);
foreach (i, spaceship; taskPool.parallel(shipComponents))
{
spaceship.name = format(Ship %i, i);
}
You're example almost works. With the EntityComponentManager API
it works like this:
auto shipEntities = ecm.query!SpaceShip();
foreach (i, spaceship; taskPool.parallel(shipEntities))
{
auto shipData = spaceship.getComponent!SpaceShip();
shipData.name = format(Ship %i, i);
}
Now that unittests are written, documentation is on its way and
should be ready in a few days.
Re: Non-pipeline component programming
On Tuesday, 4 February 2014 at 09:49:39 UTC, Zoadian wrote: We're currently working on an Entity Component System for D ( https://github.com/Zoadian/nitro) Here some good explanations: http://t-machine.org/index.php/2007/09/03/entity-systems-are-the-future-of-mmog-development-part-1/ http://www.richardlord.net/blog/what-is-an-entity-framework After carefully reading the material, I think that the Entity System / Entity Component System (ECS) framework is indeed a good design, and provides a good answer for the specific example I was considering (spaceships, etc.). Still, I wonder how related to the (pipelined) component approach described by Walter and Teoh this ECS approach really is. Perhaps more specifically, I wonder if the ECS approach also allows using a lot of the std.algorithms, like was done in the pipelined component approach, and in the calendar example in particular. Also, I'm a bit lost with the Nitro code and how to use it. (Tangentially, two things that I was wondering while reading about the ECS: how important would it be to have non-polymorphic reference types, and why are we paying for the object monitor field when that approach seems to be outdated, especially in D)
Re: Non-pipeline component programming
On Wednesday, 5 February 2014 at 15:29:14 UTC, Luís Marques wrote: Still, I wonder how related to the (pipelined) component approach described by Walter and Teoh this ECS approach really is. Perhaps more specifically, I wonder if the ECS approach also allows using a lot of the std.algorithms, like was done in the pipelined component approach, and in the calendar example in particular. Conceptually, it's not related. The only similarity is the use of the word component. An ECS is concerned with breaking traditional objects into disparate, composable parts. This can be combined with the concept of Data-Oriented Programming (DOP), to improve cache locality when iterating entities, by storing all components of the same type in a contiguous block of memory. Then rather than iterating an array of entities and calling a setFoo method, you can instead more efficiently iterate an array of Foos independent of each entity. The pipelined component concept is about chaining a set of operations to transform a dataset where the output of one transformation is the input to the next. Here, the components are not the data but the transformations, which can be swapped in and out, reordered, or added to and removed from the chain. A cursory look at Nitro suggests they are using an ECS with DOP. Given that they're providing ranges for iterating the data, it will fit into D's range-based component pipelines (std.algorithm and such).
Re: Non-pipeline component programming
We're currently working on an Entity Component System for D ( https://github.com/Zoadian/nitro) Here some good explanations: http://t-machine.org/index.php/2007/09/03/entity-systems-are-the-future-of-mmog-development-part-1/ http://www.richardlord.net/blog/what-is-an-entity-framework On Tuesday, 4 February 2014 at 04:12:49 UTC, Luís Marques wrote: I loved reading Walter's component programming article in Dr. Dobb's [0] in late 2012. I had missed H. S. Teoh's mid 2013 article on calendar textual formatting using the component approach [1], but fortunately Ali brought it to my attention recently, and I also find it absolutely fascinating! I think what's most interesting with Teoh's article (and I think that was Ali's point when he mentioned the article to me) is that the calendar example is not as an obvious target for the component approach, or at least that the design and implementation is not as obvious for someone new to that approach. Now, while Teoh's example is much more complex than Walter's, both examples are for cases of pipelined problems (source - filter1 - filter2 - sink). What I have been wondering during the last few days is how much this component programming approach could be applied to scenarios where you would normally have a jumble of objects. For instance, try to picture a game or simulation where spaceships fire at each other, pick up objects, communicate, and so on, or something like that. My instinct would be to code a solution which would be classified as typical OOP code. Would it be possible to come up with a solution that would be more in the spirit of component programming? Or are such solutions only practical/applicable for pipeline-like scenarios? -- [0] http://www.drdobbs.com/architecture-and-design/component-programming-in-d/240008321 [1] http://wiki.dlang.org/User:Quickfur/Component_programming_with_ranges
Re: Non-pipeline component programming
On Tuesday, 4 February 2014 at 09:49:39 UTC, Zoadian wrote: We're currently working on an Entity Component System for D ( https://github.com/Zoadian/nitro) Here some good explanations: I've seen it too. I like it so far, because of the nice usage of templates, but right now I didn't gave it a try due to zero docs. The questions asked by Luìs probably has more to do with if I'm iterating one entity/component at a time, how can I access two components at one time for my computations? The easy way is just using the cartesian product of your range with itself. http://dlang.org/phobos/std_algorithm.html#cartesianProduct This way you get a range of tuples, so you get pairs made of each entity with all the other ones, and you can process that pair with your next filter function. After that I'm not really sure about what one could do. Another approach could be: pass (a copy of your) RA range to a filter, and use that to compute the effects for each entity. Bang, every entity is now processed.
Re: Non-pipeline component programming
On Tuesday, 4 February 2014 at 10:06:18 UTC, Francesco Cattoglio wrote: On Tuesday, 4 February 2014 at 09:49:39 UTC, Zoadian wrote: We're currently working on an Entity Component System for D ( https://github.com/Zoadian/nitro) Here some good explanations: I've seen it too. I like it so far, because of the nice usage of templates, but right now I didn't gave it a try due to zero docs. We are currently working on the first stable version (and API). The master branch is the last working state but our current development is in the finalize_basics (https://github.com/Zoadian/nitro/tree/finalize_basics) branch, including a work in progress documentation. It should be ready to use and documented in a few days.
Re: Non-pipeline component programming
On Tuesday, 4 February 2014 at 10:24:57 UTC, Paul Freund wrote: We are currently working on the first stable version (and API). The master branch is the last working state but our current development is in the finalize_basics (https://github.com/Zoadian/nitro/tree/finalize_basics) branch, including a work in progress documentation. It should be ready to use and documented in a few days. That's really good news! A friend of mine was trying to get an entity system working with a etc.c.sqlite3 backend, but right now he isn't that much satisfied by it. The main reason for having it on DB was having every component on a separate table, and because we plan on having LOTS of entities (~1 Million) (a DB should be able to handle that). Throw in the mix free saving of state on file and you can see why he made this choice. Taking a better look at it, your store every entity in a separate table, too! If performance will be good, I think we might switch to nitro for our project. Besides, I don't think that sqlite performance is that great for the amount of granularity we need. Pretty sure there are just too many casts going around. (3 bytes integers??? ARE YOU SERIOUS?)
Non-pipeline component programming
I loved reading Walter's component programming article in Dr. Dobb's [0] in late 2012. I had missed H. S. Teoh's mid 2013 article on calendar textual formatting using the component approach [1], but fortunately Ali brought it to my attention recently, and I also find it absolutely fascinating! I think what's most interesting with Teoh's article (and I think that was Ali's point when he mentioned the article to me) is that the calendar example is not as an obvious target for the component approach, or at least that the design and implementation is not as obvious for someone new to that approach. Now, while Teoh's example is much more complex than Walter's, both examples are for cases of pipelined problems (source - filter1 - filter2 - sink). What I have been wondering during the last few days is how much this component programming approach could be applied to scenarios where you would normally have a jumble of objects. For instance, try to picture a game or simulation where spaceships fire at each other, pick up objects, communicate, and so on, or something like that. My instinct would be to code a solution which would be classified as typical OOP code. Would it be possible to come up with a solution that would be more in the spirit of component programming? Or are such solutions only practical/applicable for pipeline-like scenarios? -- [0] http://www.drdobbs.com/architecture-and-design/component-programming-in-d/240008321 [1] http://wiki.dlang.org/User:Quickfur/Component_programming_with_ranges
Re: Component programming
On Mon, Aug 19, 2013 at 01:26:42PM +0200, Chris wrote: On Monday, 12 August 2013 at 12:28:36 UTC, Jason den Dulk wrote: [...] Btw, I got an error message compiling dcal.d with ldmd2 dcal.d(34): Error: no property 'recurrence' for type 'Date' It compiles with dmd and works. What version of ldmd2 are you using? Looks like my code is incompatible with earlier versions of the compiler. :-( I'd like to fix that, if possible. T -- Meat: euphemism for dead animal. -- Flora
Re: Component programming
On Monday, 12 August 2013 at 12:28:36 UTC, Jason den Dulk wrote:
On Wednesday, 31 July 2013 at 10:20:57 UTC, Chris wrote:
This is only losely related to D, but I don't fully understand
the separation of component programming and OOP
What the wikipedia entry is saying, in a roundabout way is:
All objects are components, but not all components are objects.
whereas in pure OOP:
All components are objects.
In an OO framwork, the objects are basically components.
It's the other way around. In OOP frameworks, components are
objects, a small but important distinction. If you relax the
insistance on all components being objects, then OOP becomes a
subset of CP (component programming).
Walter's example
(http://www.drdobbs.com/architecture-and-design/component-programming-in-d/240008321)
void main() {
stdin.byLine(KeepTerminator.yes)// 1
map!(a = a.idup). // 2
array. // 3
sort. // 4
copy( // 5
stdout.lockingTextWriter());// 6
}
This is a design pattern called pipes and filters, or simply,
the pipeline. There appears to be a bit of confusion about
this. Pipelines are a part of CP, but is not the whole of CP.
Pipelines make use of a type of component called a service.
At its simplest, a service is a function, but it could be a
larger construct or even a whole program. Basically a service
takes input, processes it (with a possible side effect) and
gives a response (output).
Often CP is defined as being exclusively about services, while
other definitions include objects and OOP. Functional
programming is exclusively service oriented.
Purists would insist on using either objects or services
exclusively (OOP vs FP), but there is nothing wrong with
working with both.
Back to pipelines. In a pipeline, you have a chain of services
in which the output of one service is given as the input for
the next. In mathematics it is called functional composition.
The pipeline itself is a service in its own right. You can put
together a pipeline of any length as long as the output -
input interfaces are compatible. In Walter's article, he goes
further to make all interfaces the same to make the components
interchangeable, but this is not necessary in general.
Hope this helps to explain a few things.
Regards
Jason
Thanks for the explanation, Jason.
Btw, I got an error message compiling dcal.d with ldmd2
dcal.d(34): Error: no property 'recurrence' for type 'Date'
It compiles with dmd and works.
Re: D component programming is a joke (Was: Re: Component programming)
On 2013-08-04 06:53, Jonathan M Davis wrote: std.datetime has something like that internally for some of what it does (in particular, toSimpleString, which I wouldn't even put in there now if I could go back), but we explicitly didn't make anything like that public, because it's English-specific. The solution for that is to make it possible to plug in support for new languages and make the default English. -- /Jacob Carlborg
Re: Component programming
On Wednesday, 31 July 2013 at 10:20:57 UTC, Chris wrote:
This is only losely related to D, but I don't fully understand
the separation of component programming and OOP
What the wikipedia entry is saying, in a roundabout way is:
All objects are components, but not all components are objects.
whereas in pure OOP:
All components are objects.
In an OO framwork, the objects are basically components.
It's the other way around. In OOP frameworks, components are
objects, a small but important distinction. If you relax the
insistance on all components being objects, then OOP becomes a
subset of CP (component programming).
Walter's example
(http://www.drdobbs.com/architecture-and-design/component-programming-in-d/240008321)
void main() {
stdin.byLine(KeepTerminator.yes)// 1
map!(a = a.idup). // 2
array. // 3
sort. // 4
copy( // 5
stdout.lockingTextWriter());// 6
}
This is a design pattern called pipes and filters, or simply,
the pipeline. There appears to be a bit of confusion about this.
Pipelines are a part of CP, but is not the whole of CP.
Pipelines make use of a type of component called a service. At
its simplest, a service is a function, but it could be a larger
construct or even a whole program. Basically a service takes
input, processes it (with a possible side effect) and gives a
response (output).
Often CP is defined as being exclusively about services, while
other definitions include objects and OOP. Functional programming
is exclusively service oriented.
Purists would insist on using either objects or services
exclusively (OOP vs FP), but there is nothing wrong with working
with both.
Back to pipelines. In a pipeline, you have a chain of services in
which the output of one service is given as the input for the
next. In mathematics it is called functional composition. The
pipeline itself is a service in its own right. You can put
together a pipeline of any length as long as the output - input
interfaces are compatible. In Walter's article, he goes further
to make all interfaces the same to make the components
interchangeable, but this is not necessary in general.
Hope this helps to explain a few things.
Regards
Jason
Re: Component programming with ranges
On Tue, 6 Aug 2013 12:56:48 -0700 Andrei Alexandrescu seewebsiteforem...@erdani.org wrote: Vote up! http://www.reddit.com/r/programming/comments/1jtzez/component_programming_with_ranges/ Great article. Even as an experienced D user, it really gave me a lot to think about.
Re: Component programming with ranges
On 6-8-2013 21:56, Andrei Alexandrescu wrote: Vote up! http://www.reddit.com/r/programming/comments/1jtzez/component_programming_with_ranges/ Andrei If I understand it correctly chunkBy cannot accept a string lambda, due to the template constraint. Therefore the line alias attr = unaryFun!attrFun; seems unnecessary to me. By the way, when I try the code from github, it only prints the first 3 months. DMD32 D Compiler v2.063 (win7 64)
Component programming with ranges
Vote up! http://www.reddit.com/r/programming/comments/1jtzez/component_programming_with_ranges/ Andrei
Re: Component programming with ranges
Andrei Alexandrescu: http://www.reddit.com/r/programming/comments/1jtzez/component_programming_with_ranges/ It's an interesting article, but I wrote some comments. Not even some answers before posting that to Reddit? http://forum.dlang.org/thread/20130802052432.ga...@quickfur.ath.cx#post-ibbicmqochwtirxdbmud:40forum.dlang.org Bye, bearophile
Re: Component programming with ranges
On 08/06/2013 12:56 PM, Andrei Alexandrescu wrote: Vote up! http://www.reddit.com/r/programming/comments/1jtzez/component_programming_with_ranges/ Andrei This article is excellent. Some notes: 1) More recent versions of std.datetime makes it possible to write 1.days instead of dur!days(1) From: .recurrence!((a,n) = a[n-1] + dur!days(1)) To: .recurrence!((a,n) = a[n-1] + 1.days) 2) It is mildly annoying that sometimes it is more convenient to do non-trivial processings in .front rather than .popFront. The problem of doing so is that the user of the code may call .front multiple times assuming that the result is a reference to an already processed data when in fact each call consumes CPU cycles. 3) I haven't compiled the code but it looks like some of the signature constraints can be simplified by taking advantage of isDateRange() more. Ali
Re: D component programming is a joke (Was: Re: Component programming)
H. S. Teoh: It looks like I may have to sort out some issues with compiler bugs before officially posting this article, though, since the code apparently fails to compile with many versions of DMD. :-( If not already present, I suggest you to put a reduced version of the problems you have found (with map or something else) in Bugzilla. (Otherwise I'll try to do it later). Bye, bearophile
Re: D component programming is a joke (Was: Re: Component programming)
On Tue, Aug 06, 2013 at 06:20:23PM +0200, bearophile wrote: H. S. Teoh: It looks like I may have to sort out some issues with compiler bugs before officially posting this article, though, since the code apparently fails to compile with many versions of DMD. :-( If not already present, I suggest you to put a reduced version of the problems you have found (with map or something else) in Bugzilla. (Otherwise I'll try to do it later). [...] Actually, the problem has been fixed in git HEAD, it's just that 2.063 doesn't have the fix. The basic cause is that std.range.chunks in 2.063 and earlier requires slicing, but I need to use it with a forward range that doesn't have slicing. This limitation has been removed in git HEAD (upcoming 2.064). I've already solved the problem in the code by using static if (__VERSION__ 2064L) and providing a bare-bones replacement of std.range.chunks that does support forward ranges. T -- Customer support: the art of getting your clients to pay for your own incompetence.
Re: D component programming is a joke (Was: Re: Component programming)
On Friday, 2 August 2013 at 05:26:05 UTC, H. S. Teoh wrote: On Thu, Aug 01, 2013 at 10:34:24AM -0700, Walter Bright wrote: On 8/1/2013 2:23 AM, John Colvin wrote: On Thursday, 1 August 2013 at 00:47:43 UTC, H. S. Teoh wrote: Add in some code examples and that could make a nice article. Yes, please! Alright, so I decided to prove my point about component programming by actually writing a fully-functional version of the calendar layout program, so that I have a solid piece of evidence that component programming lives up to its promise. :) In addition, I decided that for maximum reusability, I want the output lines available in an input range, with absolutely no binding to writeln whatsoever (except in main() where the range is handed to writeln for output). In retrospect, that was perhaps a bit too ambitious... I ran into a few roadblocks to actually get the code working, so it took me a lot longer than I anticipated to finish the code. However, I *will* say that I'm very proud of the code: already there are a few pieces that, if properly cleaned up and refined, probably deserve inclusion in Phobos. Reusability FTW!! Now, just tell me if you've ever seen a calendar layout program made of straightforward, reusable pieces. I for sure haven't. I tried looking at the C code for the Unix cal program once... It looked frighteningly similar to an IOCCC entry. :-/ My D version, however, built using ranges through and through, has many pieces that are easily reusable. For example, if you wanted to output only a single month instead, you could just call join(\n) on the range of formatted month lines that the full year layout algorithm uses to splice lines from multiple months together -- it's *that* reusable. Anyway. Enough hand-waving in the air. Let the actual code speak for itself: https://github.com/quickfur/dcal/blob/master/dcal.d Now, w.r.t. the roadblocks I alluded to. When I first started working on the code, my goal was to maximize usage of existing Phobos facilities in order to show how many batteries D already comes with. As it turned out, I could only use basic Phobos components; some of the more complex pieces like frontTransversal, which would've been perfect for the bit that splices formatted month lines together, couldn't be used because it wasn't flexible enough to handle the insertion of fillers when some subranges are empty. In the end, I had to code that range by hand, and I can't say I'm that happy with it yet. But at least, it's nothing compared to the hairy complexity of the C version of cal. Another place where I wanted to use existing Phobos components was chunkBy. There's probably a way to do it if you think hard enough about it, but in the end I felt it was simpler to just write the code myself. Might be a failure on my part to recognize how to put existing Phobos ranges in a clever enough way to achieve what I wanted. I did try to do something similar to byWeek(), but somehow it didn't do what I wanted and I decided to just code it by hand instead of investigating further. By far the biggest roadblock I ran into was that after I wrote everything up to (and including) pasteBlocks, my unittests refused to work. Somehow, pasteBlocks kept repeating the first line of the output (the month names, if you look at the unittest) and refused to advance farther. Eventually I traced the problem to Lines.popFront(), which pops each subrange off the range of ranges. The problem is that this only works on some ranges, but not others; if you pass the output of formatMonths() straight to pasteBlocks(), it will NOT work. Why? Because pasteBlocks return a std.algorithm.Map object, which recreates the subrange each time, so Lines.popFront() is only popping a temporary copy of the subrange, not the real thing. I was about to give up and try another approach, when out of the blue I decided to try and see if I could stuff the range returned by formatMonths() into an array, and then pass *that* to pasteBlocks() -- and behold, it worked!! This was a totally unexpected fix, that a newbie probably would never have thought of, so this is a potential trap for newcomers to D who expect components to just be pluggable. In retrospect, it makes sense -- you need to somehow buffer the ranges of formatted month lines *somewhere* in order to be able to splice them together out of their natural depth-first outer/inner range order. But this is not obvious at all from first glance; perhaps it's a sign of a leaky abstraction somewhere. We should probably look into why this is happening and how to fix it. And there should be a way to test for this in pasteBlocks' signature constraint so that future code won't fall into the same trap, but I can't think of one right now. Once this last bit worked, though, everything fell into place quickly. After all unittests were passing, no more bugs were found!! The program can print beautifully laid out
Re: D component programming is a joke (Was: Re: Component programming)
On Mon, Aug 05, 2013 at 02:11:32PM +0200, Dejan Lekic wrote: [...] Good work! I've read the article yesterday. Very educational! Thanks! I did actually make a major revision to the article last night based on some feedback I got; I rewrote much of the first part of it, so if you're interested you might want to re-read it. T -- Caffeine underflow. Brain dumped.
Re: D component programming is a joke (Was: Re: Component programming)
On Sun, Aug 04, 2013 at 05:02:05AM +0200, bearophile wrote:
[...]
A bit improved chunkBy could go in Phobos.
Yeah I'll look into that sometime. It'll definitely be a useful thing to
have, I think.
---
For our purposes, though, we can't just do this in a loop, because
it has to interface with the other components, which do not have a
matching structure to a loop over dates.
It's just because D doesn't yet have a yield designed like C#.
yield for coroutines is a very nice kind of glue.
Don't fibres already have yield()?
---
auto datesInYear(int year) {
return Date(year, 1, 1)
.recurrence!((a,n) = a[n-1] + dur!days(1))
.until!(a = a.year year);
}
===
auto datesInYear(in uint year) pure /*nothrow*/
I think in uint is redundant. Uints don't need to be const.
The Date ctor isn't nothrow, unfortunately.
Also, std.datetime docs explicitly state that years can be negative (in
which case they represent B.C..
in {
assert(year 1900);
Good idea, I should be using contracts. :)
But in this case, there's no good reason to restrict it to 1900,
because std.datetime can handle years past 0 AD.
} body {
return Date(year, 1, 1)
.recurrence!((a, n) = a[n - 1] + dur!days(1))
.until!(d = d.year year);
}
I suggest to align the dots vertically like that. And generally
_all_ variables/arguments that don't need to mutate should be const
or immutable (or enum), unless holes in Phobos or in the type system
or other factors prevent you to do it.
Hmm. OK, but I still think in uint is redundant.
---
return chunkBy!a.month()(dates);
===
return dates.chunkBy!q{ a.month };
I don't really like using q{} here. But the UFCS thing probably looks
better, considering the rest of the code.
---
byWeek() is not so simple. Most of its code is boilerplate code.
Perhaps using a yield it becomes simpler.
I think Phobos needs better primitives for constructing chunked ranges.
:) I was thinking if there's a common core that can be factored out of
chunkBy and byWeek, to make a generic range chunking function that can
handle both.
---
string spaces(size_t n) {
return repeat(' ').take(n).array.to!string;
}
===
string spaces(in size_t n) pure nothrow {
return std.array.replicate( , n);
}
Thanks! I didn't know about std.array.replicate. :)
Currently in programs that import both std.range and std.array you
have to qualify the module for replicate.
In Python this is just:
' ' * n
The closest I ever got to that in D was to define:
string x(string s, size_t n) { ... }
so you could write .x(10) as a visual approximation to *10. But
this is not a good idea for other reasons (x is too confusing a name
and likely to clash with other symbols).
---
auto buf = appender!string();
Perhaps this suffices:
appender!string buf;
Actually, that doesn't compile.
---
string[] days = map!((Date d) = %2d.format(d.day))(r.front)
.array;
(not tested) ==
const days = r.front.map!(d = %2d.format(d.day)).array;
I wanted to make the return type explicit for the reader's benefit.
[...]
If you put the days inside buf, do you really need to turn days into
an array with array()?
The reason is actually so that I can assert on its length. Otherwise
you're right, I could just put it directly into buf.
[...]
---
If not already present this array should go in std.datetime or
core.time:
static immutable string[] monthNames = [
January, February, March, April, May, June,
July, August, September, October, November,
December
];
Well, once we have a proper i18n module...
---
return to!string(spaces(before) ~ name ~ spaces(after));
== (untested)
return text(before.spaces, name, after.spaces);
Or maybe even just (untested):
return before.spaces ~ name ~ after.spaces;
You're right, there's no need to call to!string. I think that was left
over from older code when some of the pieces were char[].
---
auto formatMonth(Range)(Range monthDays)
if (isInputRange!Range is(ElementType!Range == Date))
{
assert(!monthDays.empty);
assert(monthDays.front.day == 1);
return chain(
[ monthTitle(monthDays.front.month) ],
monthDays.byWeek().formatWeek());
}
=== (untested)
auto formatMonth(R)(R monthDays)
if (isInputRange!R is(ElementType!R == Date))
in {
assert(!monthDays.empty);
assert(monthDays.front.day == 1);
} body {
return [monthDays.front.month.monthTitle]
.chain(monthDays.byWeek.formatWeek);
I don't like overusing UFCS when it makes the code harder to read.
}
Generally I suggest to use pre- and post conditions.
Good idea. We should use contracts more in D... in part so that there's
enough code out there to
Re: D component programming is a joke (Was: Re: Component programming)
On Thursday, 1 August 2013 at 22:45:10 UTC, bearophile wrote:
Walter Bright:
But consider that optimizers are built to optimize typical
code patterns. Component programming is fairly non-existent in
C and C++, and is new in D. Hence, optimizers are not set up
to deal with those patterns (yet).
I agree.
GHC also works with a LLVM back-end, so those optimizations are
done in some kind of middle-end.
Probably a silly idea: perhaps we can collect some money, like
1000-2000 dollars, to pay for a 3 day long course for Walter
(total about 15 hours) about such matters.
Who's giving the course and where will it be held?
'Modern Compiler Design' by D. Grune, et al. 'Compiler Design -
Analysis and Transformation' by H. Seidl, et al. discusses some
basic optimizations for functional programs. But I'm pretty sure
Walter is already familiar with these.
Taking an example from Ali's book P' in D:
import std.stdio;
import std.algorithm;
void main()
{
auto values = [ 1, 2, 3, 4, 5 ];
writeln(values
.map!(a = a * 10)
.map!(a = a / 3)
.filter!(a = !(a % 2)));
}
As stated this implies 3 separate traversals of the list (or
array to be specific) which is what a naïve implementation would
do. But all three operations can run on the same traversal. Given
that all three operations are monotonic functions (preserving the
order and cardinality of the set, i.e. 1-1 mapping) they are also
inherently parallelizable (i.e. amenable to auto vectorization or
loop unrolling).
Re: D component programming is a joke (Was: Re: Component programming)
On Saturday, 3 August 2013 at 13:35:56 UTC, Andre Artus wrote:
import std.stdio;
import std.algorithm;
void main()
{
auto values = [ 1, 2, 3, 4, 5 ];
writeln(values
.map!(a = a * 10)
.map!(a = a / 3)
.filter!(a = !(a % 2)));
}
As stated this implies 3 separate traversals of the list (or
array to be specific) which is what a naïve implementation
would do.
In this example, no, as all involved ranges are evaluated lazily.
(I see your general point, though.)
David
Re: D component programming is a joke (Was: Re: Component programming)
On Saturday, 3 August 2013 at 13:46:38 UTC, David Nadlinger wrote:
On Saturday, 3 August 2013 at 13:35:56 UTC, Andre Artus wrote:
import std.stdio;
import std.algorithm;
void main()
{
auto values = [ 1, 2, 3, 4, 5 ];
writeln(values
.map!(a = a * 10)
.map!(a = a / 3)
.filter!(a = !(a % 2)));
}
As stated this implies 3 separate traversals of the list (or
array to be specific) which is what a naïve implementation
would do.
In this example, no, as all involved ranges are evaluated
lazily. (I see your general point, though.)
David
I probably could have worded it better: I did not intend to imply
that D follows the naïve implementation suggested. What I meant
is that, on the face of it, given typical textbook
implementations of map and filter you would be iterating 3
separate times. To be clear I don't know of any serious
implementations of these algorithms that do not address this in
some way.
Re: D component programming is a joke (Was: Re: Component programming)
On Fri, Aug 02, 2013 at 06:07:02PM -0700, Andrei Alexandrescu wrote:
On 2013-08-02 23:27:20 +, Timon Gehr said:
Also, you may want to replace some of the manually implemented
ranges where this makes sense.
Eg, datesInYear can be expressed more to the point as:
auto datesInYear(int year){
return Date(year,1,1).recurrence!((a,n)=a[n-1]+1.dur!days)
.until!(a=a.yearyear);
}
(This closes over year though. The following version uses only
closed lambdas by embedding year in the returned range object:
auto datesInYear(int year){
return Date(year,1,1)
.recurrence!((a,n)=a[n-1]+1.dur!days)
.zip(year.repeat)
.until!(a=a[0].yeara[1]).map!(a=a[0]);
})
Thanks! I replaced the code with the first version above. I decided that
it's OK to close over year; it's a good example of the convenience of D
closures. And I also don't feel like explaining the functional
gymnastics of using zip and map just to avoid a closure. :)
Would be nice to have a couple of these both explicit and also
implemented with the stdlib.
[...]
I felt the article was approaching the long side, so I decided to just
use Timon's simplified code instead of the original explicit
implementation.
Or do you think it's better to have both, for comparison?
T
--
Ignorance is bliss... but only until you suffer the consequences!
Re: D component programming is a joke (Was: Re: Component programming)
On 8/3/2013 6:46 AM, David Nadlinger wrote: In this example, no, as all involved ranges are evaluated lazily. (I see your general point, though.) The rules for ranges do not specify if they are done eagerly, lazily, or in parallel. Meaning, of course, that a library writer could provide all three forms and the user could select which one he wanted.
Re: D component programming is a joke (Was: Re: Component programming)
On Sat, Aug 03, 2013 at 04:12:58AM +0200, Timon Gehr wrote: On 08/03/2013 01:05 AM, H. S. Teoh wrote: Actually, I just pulled git HEAD again, and it's still working fine. Maybe you just need to update your repo? ... I think it pulled in the wrong version of druntime. OK, I've written a simple replacement for 2.063 std.range.chunks inside a static if (__VERSION__ = 2063) block, so you should be able to compile it now. The code has been pushed to github. T -- It is of the new things that men tire --- of fashions and proposals and improvements and change. It is the old things that startle and intoxicate. It is the old things that are young. -- G.K. Chesterton
Re: D component programming is a joke (Was: Re: Component programming)
On Saturday, August 03, 2013 01:27:20 Timon Gehr wrote:
Also, you may want to replace some of the manually implemented ranges
where this makes sense.
Eg, datesInYear can be expressed more to the point as:
auto datesInYear(int year){
return Date(year,1,1).recurrence!((a,n)=a[n-1]+1.dur!days)
.until!(a=a.yearyear);
}
You could also use std.datetime.Interval and do something like
auto datesInYear(int year)
{
auto interval = Interval!Date(Date(year, 1, 1), Date(year + 1, 1, 1));
return interval.fwdRange((a){return a + dur!days(1);});
}
I do think that I need to revisit how ranges work with intervals in
std.datetime, as they're a bit clunky.
- Jonathan M Davis
Re: D component programming is a joke (Was: Re: Component programming)
On Fri, Aug 02, 2013 at 03:02:24PM -0700, H. S. Teoh wrote: On Thu, Aug 01, 2013 at 10:49:00PM -0700, Walter Bright wrote: [...] I think this is awesome, and this + your previous post are sufficient to create a great article! OK, here's a draft of the article: http://wiki.dlang.org/User:Quickfur/Component_programming_with_ranges [...] Alright, the article is done: http://wiki.dlang.org/Component_programming_with_ranges I haven't linked it to the articles page yet, though. It'd be better if somebody else vetted it before it is included there, I think. :) T -- Holy war is an oxymoron. -- Lazarus Long
Re: D component programming is a joke (Was: Re: Component programming)
H. S. Teoh:
OK, here's a draft of the article:
http://wiki.dlang.org/User:Quickfur/Component_programming_with_ranges
Most of the code below is not tested. So my suggestions may
contain bugs or mistakes.
A bit improved chunkBy could go in Phobos.
---
For our purposes, though, we can't just do this in a loop,
because it has to interface with the other components, which do
not have a matching structure to a loop over dates.
It's just because D doesn't yet have a yield designed like C#.
yield for coroutines is a very nice kind of glue.
---
auto datesInYear(int year) {
return Date(year, 1, 1)
.recurrence!((a,n) = a[n-1] + dur!days(1))
.until!(a = a.year year);
}
===
auto datesInYear(in uint year) pure /*nothrow*/
in {
assert(year 1900);
} body {
return Date(year, 1, 1)
.recurrence!((a, n) = a[n - 1] + dur!days(1))
.until!(d = d.year year);
}
I suggest to align the dots vertically like that. And generally
_all_ variables/arguments that don't need to mutate should be
const or immutable (or enum), unless holes in Phobos or in the
type system or other factors prevent you to do it.
---
return chunkBy!a.month()(dates);
===
return dates.chunkBy!q{ a.month };
---
byWeek() is not so simple. Most of its code is boilerplate code.
Perhaps using a yield it becomes simpler.
---
string spaces(size_t n) {
return repeat(' ').take(n).array.to!string;
}
===
string spaces(in size_t n) pure nothrow {
return std.array.replicate( , n);
}
Currently in programs that import both std.range and std.array
you have to qualify the module for replicate.
In Python this is just:
' ' * n
---
auto buf = appender!string();
Perhaps this suffices:
appender!string buf;
---
string[] days = map!((Date d) = %2d.format(d.day))(r.front)
.array;
(not tested) ==
const days = r.front.map!(d = %2d.format(d.day)).array;
Or:
const string[] days = r
.front
.map!(d = %2d.format(d.day))
.array;
---
If you put the days inside buf, do you really need to turn days
into an array with array()?
string[] days = map!((Date d) = %2d.format(d.day))(r.front)
.array;
assert(days.length = 7 - startDay);
days.copy(buf);
Isn't this enough?
auto days = r.front.map!(d = %2d.format(d.day));
---
If not already present this array should go in std.datetime or
core.time:
static immutable string[] monthNames = [
January, February, March, April, May, June,
July, August, September, October, November,
December
];
---
return to!string(spaces(before) ~ name ~ spaces(after));
== (untested)
return text(before.spaces, name, after.spaces);
Or maybe even just (untested):
return before.spaces ~ name ~ after.spaces;
---
auto formatMonth(Range)(Range monthDays)
if (isInputRange!Range is(ElementType!Range == Date))
{
assert(!monthDays.empty);
assert(monthDays.front.day == 1);
return chain(
[ monthTitle(monthDays.front.month) ],
monthDays.byWeek().formatWeek());
}
=== (untested)
auto formatMonth(R)(R monthDays)
if (isInputRange!R is(ElementType!R == Date))
in {
assert(!monthDays.empty);
assert(monthDays.front.day == 1);
} body {
return [monthDays.front.month.monthTitle]
.chain(monthDays.byWeek.formatWeek);
}
Generally I suggest to use pre- and post conditions.
---
return months.map!((month) = month.formatMonth());
=== (untested)
return months.map!formatMonth;
---
.map!((r) =
===
.map!(r =
---
int year = to!int(args[1]);
===
int year = args[1].to!int;
---
On Rosettacode there is a shorter calendar:
http://rosettacode.org/wiki/Calendar#D
If you want we can put, as second D entry, your calendar code
(without unittests) in that page too.
Bye,
bearophile
Re: D component programming is a joke (Was: Re: Component programming)
Bearophile: If not already present this array should go in std.datetime or core.time: static immutable string[] monthNames = [ January, February, March, April, May, June, July, August, September, October, November, December ]; It should probably be picked up from the OS, to support localization.
Re: D component programming is a joke (Was: Re: Component programming)
On Sunday, August 04, 2013 06:20:57 Andre Artus wrote: Bearophile: If not already present this array should go in std.datetime or core.time: static immutable string[] monthNames = [ January, February, March, April, May, June, July, August, September, October, November, December ]; It should probably be picked up from the OS, to support localization. std.datetime has something like that internally for some of what it does (in particular, toSimpleString, which I wouldn't even put in there now if I could go back), but we explicitly didn't make anything like that public, because it's English-specific. - Jonathan M Davis
Re: Component Programming example
On Friday, 2 August 2013 at 17:03:44 UTC, Justin Whear wrote:
On Fri, 02 Aug 2013 18:59:12 +0200, Jonathan A Dunlap wrote:
The example:
http://www.drdobbs.com/architecture-and-design/component-programming-in-
d/240008321?pgno=4
import std.stdio;
import std.array;
import std.algorithm;
void main() {
stdin.byLine(KeepTerminator.yes)// 1 map!(a =
a.idup).
// 2 array.
// 3
sort. // 4 copy(
// 5
stdout.lockingTextWriter());// 6
}
I don't understand what happens to the output. On windows, I
can keep
entering lines but no output gets displayed. Also, can someone
explain a
bit more about lockingTextWriter?
Thanks!
1) The example has a typo; there should be a '.' between the
byLine call
and the array call.
void main() {
stdin.byLine(KeepTerminator.yes) // 1
.map!(a = a.idup) // 2 '.' was missing start
of this line
.array // 3
.sort// 4
.copy( // 5
stdout.lockingTextWriter()); // 6
}
2) The example collects all input before writing anything (so
that it can
sort). Hit your end-of-file character (Ctrl-D) for me to end
the input.
Or direct a file into the process' stdin (not sure how to do
this on
Windows, it's been so long).
On Windows you send EOF by using the Ctrl-Z key sequence.
Re: Component programming
On Wednesday, 31 July 2013 at 10:20:57 UTC, Chris wrote:
This is only losely related to D, but I don't fully understand
the separation of component programming and OOP (cf.
https://en.wikipedia.org/wiki/Component-based_software_engineering#Differences_from_object-oriented_programming).
In an OO framwork, the objects are basically components. See
also
Brad Cox of Stepstone largely defined the modern concept of a
software component.[4] He called them Software ICs and set out
to create an infrastructure and market for these components by
inventing the Objective-C programming language. (see link
above)
Walter's example
(http://www.drdobbs.com/architecture-and-design/component-programming-in-d/240008321)
void main() {
stdin.byLine(KeepTerminator.yes)// 1
map!(a = a.idup). // 2
array. // 3
sort. // 4
copy( // 5
stdout.lockingTextWriter());// 6
}
This is more or less how mature OO programs look like. Ideally
each class (component) does one thing (however small the class
might be) and can be used or called to perform this task. All
other classes or components can live independently. From my
experience this is exactly what Objective-C does. Rather than
subclassing, it uses other classes to get a job done.
A few days ago, there was a discussion about APL on HN [0]. What
we call Component Programming here, looks somewhat like the APL
style to me. Sure, APLers have a single weird symbol for stuff
like sort., but this chaining of powerful modular operations is
what APL seems to be all about.
The APL paradigm is not integrated into modern languages so far.
I am excited that it might make an introduction now. Compare for
example Functional Programming, which is integrated into most
mainstream languages by now. Or Logic Programming, which seems
not worthy enough to get its own syntax, but is available in the
business rules world with libraries and DSLs and its minor
brother Datalog is also still alive.
[0] https://news.ycombinator.com/item?id=6115727
Re: D component programming is a joke (Was: Re: Component programming)
On Thu, 01 Aug 2013 22:24:32 -0700, H. S. Teoh wrote: Now, w.r.t. the roadblocks I alluded to. When I first started working on the code, my goal was to maximize usage of existing Phobos facilities in order to show how many batteries D already comes with. As it turned out, I could only use basic Phobos components; some of the more complex pieces like frontTransversal, which would've been perfect for the bit that splices formatted month lines together, couldn't be used because it wasn't flexible enough to handle the insertion of fillers when some subranges are empty. In the end, I had to code that range by hand, and I can't say I'm that happy with it yet. I recently wrote a range component for my current project that is similar but with a twist. It takes a bunch of ranges, each of which is assumed to be sorted with some predicate, then it walks through them, returning a range of the fronts of each range. The twist is that it has to call a user-supplied `produce` function whenever it encounters a mismatch (e.g. a range's front is greater than the others or a range is empty).
Re: D component programming is a joke (Was: Re: Component programming)
On Fri, Aug 02, 2013 at 04:06:46PM +, Justin Whear wrote: On Thu, 01 Aug 2013 22:24:32 -0700, H. S. Teoh wrote: Now, w.r.t. the roadblocks I alluded to. When I first started working on the code, my goal was to maximize usage of existing Phobos facilities in order to show how many batteries D already comes with. As it turned out, I could only use basic Phobos components; some of the more complex pieces like frontTransversal, which would've been perfect for the bit that splices formatted month lines together, couldn't be used because it wasn't flexible enough to handle the insertion of fillers when some subranges are empty. In the end, I had to code that range by hand, and I can't say I'm that happy with it yet. I recently wrote a range component for my current project that is similar but with a twist. It takes a bunch of ranges, each of which is assumed to be sorted with some predicate, then it walks through them, returning a range of the fronts of each range. The twist is that it has to call a user-supplied `produce` function whenever it encounters a mismatch (e.g. a range's front is greater than the others or a range is empty). It would be nice to collect these custom ranges and see if there's some common functionality that can be added to Phobos. T -- The trouble with TCP jokes is that it's like hearing the same joke over and over.
Re: D component programming is a joke (Was: Re: Component programming)
H. S. Teoh: It would be nice to collect these custom ranges and see if there's some common functionality that can be added to Phobos. chunkBy seems OK for Phobos. Bye, bearophile
Re: D component programming is a joke (Was: Re: Component programming)
On 08/02/2013 07:24 AM, H. S. Teoh wrote: ... Anyway. Enough hand-waving in the air. Let the actual code speak for itself: https://github.com/quickfur/dcal/blob/master/dcal.d ... Which version of the compiler are you using? I get the dreaded forward reference errors with at least DMD 2.060, DMD 2.063 and DMD 2.063.2 and the 2.x build on dpaste. Git head gives me: Error: undefined identifier '_xopCmp' dmd: clone.c:690: FuncDeclaration* StructDeclaration::buildXopCmp(Scope*): Assertion `s' failed. Aborted (core dumped)
Re: D component programming is a joke (Was: Re: Component programming)
On Fri, Aug 02, 2013 at 08:49:30PM +0200, Timon Gehr wrote: On 08/02/2013 07:24 AM, H. S. Teoh wrote: ... Anyway. Enough hand-waving in the air. Let the actual code speak for itself: https://github.com/quickfur/dcal/blob/master/dcal.d ... Which version of the compiler are you using? I'm using git HEAD. I get the dreaded forward reference errors with at least DMD 2.060, DMD 2.063 and DMD 2.063.2 and the 2.x build on dpaste. Can you send me the error messages? I'll see if I can reorder the code to fix them. Git head gives me: Error: undefined identifier '_xopCmp' dmd: clone.c:690: FuncDeclaration* StructDeclaration::buildXopCmp(Scope*): Assertion `s' failed. Aborted (core dumped) That's new. It was working as of yesterday; must've been a new regression in the commits since then? T -- The richest man is not he who has the most, but he who needs the least.
Re: D component programming is a joke (Was: Re: Component programming)
On Thu, Aug 01, 2013 at 10:49:00PM -0700, Walter Bright wrote: On 8/1/2013 10:24 PM, H. S. Teoh wrote: Once this last bit worked, though, everything fell into place quickly. After all unittests were passing, no more bugs were found!! The program can print beautifully laid out calendars with no problems whatsoever. I'm so in love with D right now... If I'd done this exercise in C or C++, I'd be spending the next 2 days debugging before I could present the code for the world to see. D ranges and unittest blocks are t3h k00l. I think this is awesome, and this + your previous post are sufficient to create a great article! OK, here's a draft of the article: http://wiki.dlang.org/User:Quickfur/Component_programming_with_ranges It looks like I may have to sort out some issues with compiler bugs before officially posting this article, though, since the code apparently fails to compile with many versions of DMD. :-( T -- Живёшь только однажды.
Re: D component programming is a joke (Was: Re: Component programming)
On Fri, Aug 02, 2013 at 03:00:01PM -0700, H. S. Teoh wrote: On Fri, Aug 02, 2013 at 08:49:30PM +0200, Timon Gehr wrote: [...] I get the dreaded forward reference errors with at least DMD 2.060, DMD 2.063 and DMD 2.063.2 and the 2.x build on dpaste. Can you send me the error messages? I'll see if I can reorder the code to fix them. I just checked DMD 2.063, and it appears that the error is caused by a limitation in std.range.chunks in 2.063, where it requires slicing and length, but formatYear can only give it an input range. This is kinda sad, since that means I'll have to implement chunks myself on 2.063. :-/ I've no idea why it seems to be somehow conflated with an error from invoking std.conv.to!() to convert from string to int; apparently some kind of compiler bug that obscures the real problem with std.range.chunks. Git head gives me: Error: undefined identifier '_xopCmp' dmd: clone.c:690: FuncDeclaration* StructDeclaration::buildXopCmp(Scope*): Assertion `s' failed. Aborted (core dumped) That's new. It was working as of yesterday; must've been a new regression in the commits since then? [...] Actually, I just pulled git HEAD again, and it's still working fine. Maybe you just need to update your repo? T -- Real programmers can write assembly code in any language. :-) -- Larry Wall
Re: D component programming is a joke (Was: Re: Component programming)
On 8/2/2013 3:02 PM, H. S. Teoh wrote: OK, here's a draft of the article: http://wiki.dlang.org/User:Quickfur/Component_programming_with_ranges It looks like I may have to sort out some issues with compiler bugs before officially posting this article, though, since the code apparently fails to compile with many versions of DMD. :-( Get 'em up on bugzilla! (At least any that fail with HEAD.)
Re: D component programming is a joke (Was: Re: Component programming)
On 08/03/2013 12:02 AM, H. S. Teoh wrote:
On Thu, Aug 01, 2013 at 10:49:00PM -0700, Walter Bright wrote:
On 8/1/2013 10:24 PM, H. S. Teoh wrote:
Once this last bit worked, though, everything fell into place quickly.
After all unittests were passing, no more bugs were found!! The program
can print beautifully laid out calendars with no problems whatsoever.
I'm so in love with D right now... If I'd done this exercise in C or
C++, I'd be spending the next 2 days debugging before I could present
the code for the world to see. D ranges and unittest blocks are t3h
k00l.
I think this is awesome, and this + your previous post are
sufficient to create a great article!
OK, here's a draft of the article:
http://wiki.dlang.org/User:Quickfur/Component_programming_with_ranges
It looks like I may have to sort out some issues with compiler bugs
before officially posting this article, though, since the code
apparently fails to compile with many versions of DMD. :-(
T
Also, you may want to replace some of the manually implemented ranges
where this makes sense.
Eg, datesInYear can be expressed more to the point as:
auto datesInYear(int year){
return Date(year,1,1).recurrence!((a,n)=a[n-1]+1.dur!days)
.until!(a=a.yearyear);
}
(This closes over year though. The following version uses only closed
lambdas by embedding year in the returned range object:
auto datesInYear(int year){
return Date(year,1,1)
.recurrence!((a,n)=a[n-1]+1.dur!days)
.zip(year.repeat)
.until!(a=a[0].yeara[1]).map!(a=a[0]);
})
Re: D component programming is a joke (Was: Re: Component programming)
On 2013-08-02 23:27:20 +, Timon Gehr said:
Also, you may want to replace some of the manually implemented ranges
where this makes sense.
Eg, datesInYear can be expressed more to the point as:
auto datesInYear(int year){
return Date(year,1,1).recurrence!((a,n)=a[n-1]+1.dur!days)
.until!(a=a.yearyear);
}
(This closes over year though. The following version uses only closed
lambdas by embedding year in the returned range object:
auto datesInYear(int year){
return Date(year,1,1)
.recurrence!((a,n)=a[n-1]+1.dur!days)
.zip(year.repeat)
.until!(a=a[0].yeara[1]).map!(a=a[0]);
})
Would be nice to have a couple of these both explicit and also
implemented with the stdlib.
Andrei
Re: D component programming is a joke (Was: Re: Component programming)
On 08/03/2013 01:05 AM, H. S. Teoh wrote: Actually, I just pulled git HEAD again, and it's still working fine. Maybe you just need to update your repo? ... I think it pulled in the wrong version of druntime.
Component Programming example
The example:
http://www.drdobbs.com/architecture-and-design/component-programming-in-d/240008321?pgno=4
import std.stdio;
import std.array;
import std.algorithm;
void main() {
stdin.byLine(KeepTerminator.yes)// 1
map!(a = a.idup). // 2
array. // 3
sort. // 4
copy( // 5
stdout.lockingTextWriter());// 6
}
I don't understand what happens to the output. On windows, I can
keep entering lines but no output gets displayed. Also, can
someone explain a bit more about lockingTextWriter?
Thanks!
Re: Component Programming example
On Fri, 02 Aug 2013 18:59:12 +0200, Jonathan A Dunlap wrote:
The example:
http://www.drdobbs.com/architecture-and-design/component-programming-in-
d/240008321?pgno=4
import std.stdio;
import std.array;
import std.algorithm;
void main() {
stdin.byLine(KeepTerminator.yes)// 1 map!(a = a.idup).
// 2 array. // 3
sort. // 4 copy(
// 5
stdout.lockingTextWriter());// 6
}
I don't understand what happens to the output. On windows, I can keep
entering lines but no output gets displayed. Also, can someone explain a
bit more about lockingTextWriter?
Thanks!
1) The example has a typo; there should be a '.' between the byLine call
and the array call.
2) The example collects all input before writing anything (so that it can
sort). Hit your end-of-file character (Ctrl-D) for me to end the input.
Or direct a file into the process' stdin (not sure how to do this on
Windows, it's been so long).
Re: Component Programming example
On Fri, 02 Aug 2013 18:59:12 +0200, Jonathan A Dunlap wrote: I don't understand what happens to the output. On windows, I can keep entering lines but no output gets displayed. Also, can someone explain a bit more about lockingTextWriter? Thanks! lockingTextWriter wraps stdout (or any other file) with an OutputRange that locks the file while writing. This ensures that thread-shared files (which stdout is) don't accidentally interleave; if you write a line, you'll get the same line unbroken in your output, even if other threads are trying to write to stdout.
Re: Component programming
The one thing that confused me at first when I read Walter's article was that I thought he was talking about the *other* component programming, a method commonly used by game developers to avoid deep class hierarchies. http://gameprogrammingpatterns.com/component.html
Re: D component programming is a joke (Was: Re: Component programming)
On Wednesday, 31 July 2013 at 22:23:54 UTC, bearophile wrote:
Justin Whear:
If anything, component programming is just functional
programming + templates and some nice syntactic sugar.
And a healthy dose of pure awesome.
What D calls component programming is very nice and good, but
in D it's almost a joke.
Currently this code inlines nothing (the allocations, the
difference and the product):
import std.numeric: dotProduct;
int main() {
enum N = 50;
auto a = new int[N];
auto b = new int[N];
auto c = new int[N];
c[] = a[] - b[];
int result = dotProduct(c, c);
return result;
}
If you write it in component-style (using doubles here):
import std.math;
import std.algorithm, std.range;
int main() {
enum N = 50;
alias T = double;
auto a = new T[N];
auto b = new T[N];
return cast(int)zip(a, b)
.map!(p = (p[0] - p[1]) ^^ 2)
.reduce!q{a + b};
}
The situation gets much worse, you see many functions in the
binary, that even LDC2 often not able to inline. The GHC
Haskell compiler turns similar components code in efficient
SIMD asm (that uses packed doubles, like double2), it inlines
everything, merges the loops, produces a small amount of asm
output, and there is no c intermediate array. In GHC
component programming is mature (and Intel is developing an
Haskell compiler that is even more optimizing), while in
D/dmd/Phobos this stuff is just started. GHC has twenty+ years
of head start on this and it shows.
The situation should be improved for D/dmd/Phobos, otherwise
such D component programming remains partially a dream, or a
toy.
Bye,
bearophile
I was honestly thinking whether I should reply to this rant or
not... Obviously I picked the first. - Component programming, as
you probably know yourself already, is not about making
super-fast, super-optimized applications, but about making it
easy both to write the code and to understand the code, as well
as making it easy to combine components (algorithms mostly) and
get the result quickly, where by quickly I think about time I
need to write the code.
If you really want a super-optimized solution you will in most
cases write the piece in question in C. Well, that is at least
what my experience tells me. Luckily, I do business applications
most of the time, so performance is rarely an issue. CONVENIENCE
is! In other words, I shamelessly admit, I only care about the
time I have to spend coding in order to implement something that
is of value to the business.
Re: D component programming is a joke (Was: Re: Component programming)
On Thursday, 1 August 2013 at 00:47:43 UTC, H. S. Teoh wrote: Most non-trivial loops in imperative code have both, which makes them doubly prone to bugs. In the example I gave above, the mismatch between the code structure (a single loop) and the file structure (three sequential sections) often prompts people to add boolean flags, state variables, and the like, in order to resolve the conflict between the two structures. Such ad hoc structure resolutions are a breeding ground for bugs, and often lead to complicated loop conditions, which invite even more bugs. T I agree, and to be honest, loops have given me more than one headache. It's so easy to lose track of what is going on where and why. And if you have ever had the pleasure of adding to or debugging code that handles three or more different issues in one loop, then you will know how mind boggling loops can be. Your example is very good (you should write an article about it) and similar examples occur in web development all the time (creating tables, lists etc). I once wrote an event calendar for a homepage and _partly_ disentagled the loop for simplicity's sake. I say partly because it is still a bit loopy. And I guess this is what component programming is all about, disentangling code. The only difficulty I have is the opposition to OOP. I don't really see how the two concepts are mutually exclusive. OOP can benefit from component programming and vice versa. Component programming is a good choice for web programming where loops abound. I'm tired of the infinite loops (pardon the pun again) in JavaScript and the like. Sure there's gotta be a better way.
Re: D component programming is a joke (Was: Re: Component programming)
On Thursday, 1 August 2013 at 00:47:43 UTC, H. S. Teoh wrote:
On Wed, Jul 31, 2013 at 11:52:35PM +, Justin Whear wrote:
On Thu, 01 Aug 2013 00:23:52 +0200, bearophile wrote:
The situation should be improved for D/dmd/Phobos, otherwise
such D
component programming remains partially a dream, or a toy.
Bye,
bearophile
I disagree with your toy assessment. I've been using this
chaining,
component style for a while now and have really enjoyed the
clarity
it's brought to my code. I hadn't realized how bug-prone
non-trivial
loops tend to be until I started writing this way and avoided
them
entirely.
[...]
One of the more influential courses I took in college was on
Jackson
Structured Programming. It identified two sources of programming
complexity (i.e., where bugs are most likely to occur): (1)
mismatches
between the structure of the program and the structure of the
data
(e.g., you're reading an input file that has a preamble, body,
and
epilogue, but your code has a single loop over lines in the
file); (2)
writing loop invariants (or equivalently, loop conditions).
Most non-trivial loops in imperative code have both, which
makes them
doubly prone to bugs. In the example I gave above, the mismatch
between
the code structure (a single loop) and the file structure (three
sequential sections) often prompts people to add boolean flags,
state
variables, and the like, in order to resolve the conflict
between the
two structures. Such ad hoc structure resolutions are a
breeding ground
for bugs, and often lead to complicated loop conditions, which
invite
even more bugs.
In contrast, if you structure your code according to the
structure of
the input (i.e., one loop for processing the preamble, one loop
for
processing the body, one loop for processing the epilogue), it
becomes
considerably less complex, easier to read (and write!), and far
less bug
prone. Your loop conditions become simpler, and thus easier to
reason
about and leave less room for bugs to hide.
But to be able to process the input in this way requires that
you
encapsulate your input so that it can be processed by 3
different loops.
Once you go down that road, you start to arrive at the concept
of input
ranges... then you abstract away the three loops into three
components,
and behold, component style programming!
In fact, with component style programming, you can also address
another
aspect of (1): when you need to simultaneously process two data
structures whose structures don't match. For example, if you
want to lay
out a yearly calendar using writeln, the month/day cells must
be output
in a radically different order than the logical
foreach(m;1..12) {
foreach(day;1..31) } structure). Writing this code in the
traditional
imperative style produces a mass of spaghettii code: either you
have
bizarre loops with convoluted loop conditions for generating
the dates
in the order you want to print them, or you have to fill out
some kind
of grid structure in a complicated order so that you can
generate the
dates in order.
Using ranges, though, this becomes considerably more tractable:
you can
have an input range of dates in chronological order, two output
ranges
corresponding to chunking by week / month, which feed into a
third
output range that buffers the generated cells and prints them
once
enough has been generated to fill a row of output. By
separating out
these non-corresponding structures into separate components,
you greatly
simplify the code within each component and thus reduce the
number of
bugs (e.g. it's far easier to ensure you never put more than 7
days in a
week, since the weekly output range is all in one place, as
opposed to
sprinkled everywhere across multiple nested loops in the
imperative
style calendar code). The code that glues these components
together is
also separated out and becomes easier to understand and debug:
you
simply read from the input range of dates, write to the two
output
ranges, and check if they are full (this isn't part of the
range API but
added so for this particular example); if the weekly range is
full,
start a new week; if the monthly range is full, start a new
month. Then
the final output range takes care of when to actually produce
output --
you just write stuff to it and don't worry about it in the glue
code.
OK, this isn't really a good example of the linear pipeline
style code
we're talking about, but it does show how using ranges as
components can
untangle very complicated code into simple, tractable parts
that are
readable and easy to debug.
T
Add in some code examples and that could make a nice article.
Re: D component programming is a joke (Was: Re: Component programming)
On 08/01/2013 03:40 AM, bearophile wrote: Take a look at this thread in D.learn: http://forum.dlang.org/thread/mailman.304.1375190212.22075.digitalmars-d-le...@puremagic.com Yea, this was a frustration. :-( It was really nice to be able to write simple, clean, elegant code using D -- it was sad to discover that while this was great for a prototype, the performance gap was far too large to make it a viable long-term solution. Most of the issues seem to centre around GC, so there might be some low-hanging fruit there for performance improvements.
Re: D component programming is a joke (Was: Re: Component programming)
On 7/31/13 6:40 PM, bearophile wrote: According to this article it seems better, but I have no direct experience of it: http://www.leafpetersen.com/leaf/publications/hs2013/haskell-gap.pdf ERROR 404 - PAGE NOT FOUND Andrei
Re: D component programming is a joke (Was: Re: Component programming)
On Thursday, 1 August 2013 at 00:47:43 UTC, H. S. Teoh wrote:
On Wed, Jul 31, 2013 at 11:52:35PM +, Justin Whear wrote:
On Thu, 01 Aug 2013 00:23:52 +0200, bearophile wrote:
The situation should be improved for D/dmd/Phobos, otherwise
such D
component programming remains partially a dream, or a toy.
Bye,
bearophile
I disagree with your toy assessment. I've been using this
chaining,
component style for a while now and have really enjoyed the
clarity
it's brought to my code. I hadn't realized how bug-prone
non-trivial
loops tend to be until I started writing this way and avoided
them
entirely.
[...]
One of the more influential courses I took in college was on
Jackson
Structured Programming. It identified two sources of programming
complexity (i.e., where bugs are most likely to occur): (1)
mismatches
between the structure of the program and the structure of the
data
(e.g., you're reading an input file that has a preamble, body,
and
epilogue, but your code has a single loop over lines in the
file); (2)
writing loop invariants (or equivalently, loop conditions).
Most non-trivial loops in imperative code have both, which
makes them
doubly prone to bugs. In the example I gave above, the mismatch
between
the code structure (a single loop) and the file structure (three
sequential sections) often prompts people to add boolean flags,
state
variables, and the like, in order to resolve the conflict
between the
two structures. Such ad hoc structure resolutions are a
breeding ground
for bugs, and often lead to complicated loop conditions, which
invite
even more bugs.
In contrast, if you structure your code according to the
structure of
the input (i.e., one loop for processing the preamble, one loop
for
processing the body, one loop for processing the epilogue), it
becomes
considerably less complex, easier to read (and write!), and far
less bug
prone. Your loop conditions become simpler, and thus easier to
reason
about and leave less room for bugs to hide.
But to be able to process the input in this way requires that
you
encapsulate your input so that it can be processed by 3
different loops.
Once you go down that road, you start to arrive at the concept
of input
ranges... then you abstract away the three loops into three
components,
and behold, component style programming!
In fact, with component style programming, you can also address
another
aspect of (1): when you need to simultaneously process two data
structures whose structures don't match. For example, if you
want to lay
out a yearly calendar using writeln, the month/day cells must
be output
in a radically different order than the logical
foreach(m;1..12) {
foreach(day;1..31) } structure). Writing this code in the
traditional
imperative style produces a mass of spaghettii code: either you
have
bizarre loops with convoluted loop conditions for generating
the dates
in the order you want to print them, or you have to fill out
some kind
of grid structure in a complicated order so that you can
generate the
dates in order.
Using ranges, though, this becomes considerably more tractable:
you can
have an input range of dates in chronological order, two output
ranges
corresponding to chunking by week / month, which feed into a
third
output range that buffers the generated cells and prints them
once
enough has been generated to fill a row of output. By
separating out
these non-corresponding structures into separate components,
you greatly
simplify the code within each component and thus reduce the
number of
bugs (e.g. it's far easier to ensure you never put more than 7
days in a
week, since the weekly output range is all in one place, as
opposed to
sprinkled everywhere across multiple nested loops in the
imperative
style calendar code). The code that glues these components
together is
also separated out and becomes easier to understand and debug:
you
simply read from the input range of dates, write to the two
output
ranges, and check if they are full (this isn't part of the
range API but
added so for this particular example); if the weekly range is
full,
start a new week; if the monthly range is full, start a new
month. Then
the final output range takes care of when to actually produce
output --
you just write stuff to it and don't worry about it in the glue
code.
OK, this isn't really a good example of the linear pipeline
style code
we're talking about, but it does show how using ranges as
components can
untangle very complicated code into simple, tractable parts
that are
readable and easy to debug.
T
This post deserves to become an article somewhere. D Wiki, some
blog, whatever. All to the point. Respect.
Re: D component programming is a joke (Was: Re: Component programming)
On Thursday, 1 August 2013 at 16:13:55 UTC, Andrei Alexandrescu wrote: On 7/31/13 6:40 PM, bearophile wrote: According to this article it seems better, but I have no direct experience of it: http://www.leafpetersen.com/leaf/publications/hs2013/haskell-gap.pdf ERROR 404 - PAGE NOT FOUND Andrei This one, perhaps? http://www.leafpetersen.com/leaf/publications/ifl2013/haskell-gap.pdf Bye
Re: D component programming is a joke (Was: Re: Component programming)
On 8/1/2013 2:23 AM, John Colvin wrote: On Thursday, 1 August 2013 at 00:47:43 UTC, H. S. Teoh wrote: Add in some code examples and that could make a nice article. Yes, please!
Re: D component programming is a joke (Was: Re: Component programming)
anonymous observer: ERROR 404 - PAGE NOT FOUND Andrei This one, perhaps? http://www.leafpetersen.com/leaf/publications/ifl2013/haskell-gap.pdf Yes, it's the same, thank you. Another comparison (I have not yet read this): http://www.leafpetersen.com/leaf/publications/hs2013/hrc-paper.pdf Bye, bearophile
Re: D component programming is a joke (Was: Re: Component programming)
On Wednesday, 31 July 2013 at 22:23:54 UTC, bearophile wrote: snip Currently this code inlines nothing (the allocations, the difference and the product): snip If you write it in component-style (using doubles here): snip Resident compiler guys, How difficult would it be to make sure stuff like this gets inlined and optimized more thoroughly? I'm very ignorant of compiler internals but it's kind of disheartening that LDC can't inline them well despite being a fairly good optimizing compiler. Is this a frontend issue or a backend issue?
Re: Component programming
On Thursday, 1 August 2013 at 07:23:42 UTC, Meta wrote: The one thing that confused me at first when I read Walter's article was that I thought he was talking about the *other* component programming, a method commonly used by game developers to avoid deep class hierarchies. http://gameprogrammingpatterns.com/component.html Component programing is kind of a crowded term in programming which means a lot of different things to different people. Digital Mars should trademark a new term for it like Ultra Stream Processing™. Steven Schveighoffer may object to the use of the word stream without a read/write interface though :P.
Re: D component programming is a joke (Was: Re: Component programming)
On 8/1/2013 2:35 PM, Brad Anderson wrote: How difficult would it be to make sure stuff like this gets inlined and optimized more thoroughly? I'm very ignorant of compiler internals but it's kind of disheartening that LDC can't inline them well despite being a fairly good optimizing compiler. Is this a frontend issue or a backend issue? I don't know. But consider that optimizers are built to optimize typical code patterns. Component programming is fairly non-existent in C and C++, and is new in D. Hence, optimizers are not set up to deal with those patterns (yet).
Re: Component programming
On Thu, Aug 01, 2013 at 11:40:21PM +0200, Brad Anderson wrote: On Thursday, 1 August 2013 at 07:23:42 UTC, Meta wrote: The one thing that confused me at first when I read Walter's article was that I thought he was talking about the *other* component programming, a method commonly used by game developers to avoid deep class hierarchies. http://gameprogrammingpatterns.com/component.html Component programing is kind of a crowded term in programming which means a lot of different things to different people. Digital Mars should trademark a new term for it like Ultra Stream Processing™. Steven Schveighoffer may object to the use of the word stream without a read/write interface though :P. What about Ultra Range Processing? :) T -- He who does not appreciate the beauty of language is not worthy to bemoan its flaws.
Re: Component programming
On Thursday, 1 August 2013 at 21:55:56 UTC, H. S. Teoh wrote: On Thu, Aug 01, 2013 at 11:40:21PM +0200, Brad Anderson wrote: On Thursday, 1 August 2013 at 07:23:42 UTC, Meta wrote: The one thing that confused me at first when I read Walter's article was that I thought he was talking about the *other* component programming, a method commonly used by game developers to avoid deep class hierarchies. http://gameprogrammingpatterns.com/component.html Component programing is kind of a crowded term in programming which means a lot of different things to different people. Digital Mars should trademark a new term for it like Ultra Stream Processing™. Steven Schveighoffer may object to the use of the word stream without a read/write interface though :P. What about Ultra Range Processing? :) T Range-Flow Processing. Flow referring to the L-R data flow of ranges + std.algorithm + UFCS Even just Data Flow Processing would be ok, then you could say Range-based Data Flow Processing and sound uber-cool :p
Re: Component programming
On Thursday, 1 August 2013 at 22:01:08 UTC, John Colvin wrote: On Thursday, 1 August 2013 at 21:55:56 UTC, H. S. Teoh wrote: On Thu, Aug 01, 2013 at 11:40:21PM +0200, Brad Anderson wrote: On Thursday, 1 August 2013 at 07:23:42 UTC, Meta wrote: The one thing that confused me at first when I read Walter's article was that I thought he was talking about the *other* component programming, a method commonly used by game developers to avoid deep class hierarchies. http://gameprogrammingpatterns.com/component.html Component programing is kind of a crowded term in programming which means a lot of different things to different people. Digital Mars should trademark a new term for it like Ultra Stream Processing™. Steven Schveighoffer may object to the use of the word stream without a read/write interface though :P. What about Ultra Range Processing? :) T Range-Flow Processing. Flow referring to the L-R data flow of ranges + std.algorithm + UFCS Even just Data Flow Processing would be ok, then you could say Range-based Data Flow Processing and sound uber-cool :p Alternatively, substitute Procesing with Programming.
Re: D component programming is a joke (Was: Re: Component programming)
Walter Bright: But consider that optimizers are built to optimize typical code patterns. Component programming is fairly non-existent in C and C++, and is new in D. Hence, optimizers are not set up to deal with those patterns (yet). I agree. GHC also works with a LLVM back-end, so those optimizations are done in some kind of middle-end. Probably a silly idea: perhaps we can collect some money, like 1000-2000 dollars, to pay for a 3 day long course for Walter (total about 15 hours) about such matters. Bye, bearophile
Re: D component programming is a joke (Was: Re: Component programming)
On Thu, Aug 01, 2013 at 10:34:24AM -0700, Walter Bright wrote: On 8/1/2013 2:23 AM, John Colvin wrote: On Thursday, 1 August 2013 at 00:47:43 UTC, H. S. Teoh wrote: Add in some code examples and that could make a nice article. Yes, please! Alright, so I decided to prove my point about component programming by actually writing a fully-functional version of the calendar layout program, so that I have a solid piece of evidence that component programming lives up to its promise. :) In addition, I decided that for maximum reusability, I want the output lines available in an input range, with absolutely no binding to writeln whatsoever (except in main() where the range is handed to writeln for output). In retrospect, that was perhaps a bit too ambitious... I ran into a few roadblocks to actually get the code working, so it took me a lot longer than I anticipated to finish the code. However, I *will* say that I'm very proud of the code: already there are a few pieces that, if properly cleaned up and refined, probably deserve inclusion in Phobos. Reusability FTW!! Now, just tell me if you've ever seen a calendar layout program made of straightforward, reusable pieces. I for sure haven't. I tried looking at the C code for the Unix cal program once... It looked frighteningly similar to an IOCCC entry. :-/ My D version, however, built using ranges through and through, has many pieces that are easily reusable. For example, if you wanted to output only a single month instead, you could just call join(\n) on the range of formatted month lines that the full year layout algorithm uses to splice lines from multiple months together -- it's *that* reusable. Anyway. Enough hand-waving in the air. Let the actual code speak for itself: https://github.com/quickfur/dcal/blob/master/dcal.d Now, w.r.t. the roadblocks I alluded to. When I first started working on the code, my goal was to maximize usage of existing Phobos facilities in order to show how many batteries D already comes with. As it turned out, I could only use basic Phobos components; some of the more complex pieces like frontTransversal, which would've been perfect for the bit that splices formatted month lines together, couldn't be used because it wasn't flexible enough to handle the insertion of fillers when some subranges are empty. In the end, I had to code that range by hand, and I can't say I'm that happy with it yet. But at least, it's nothing compared to the hairy complexity of the C version of cal. Another place where I wanted to use existing Phobos components was chunkBy. There's probably a way to do it if you think hard enough about it, but in the end I felt it was simpler to just write the code myself. Might be a failure on my part to recognize how to put existing Phobos ranges in a clever enough way to achieve what I wanted. I did try to do something similar to byWeek(), but somehow it didn't do what I wanted and I decided to just code it by hand instead of investigating further. By far the biggest roadblock I ran into was that after I wrote everything up to (and including) pasteBlocks, my unittests refused to work. Somehow, pasteBlocks kept repeating the first line of the output (the month names, if you look at the unittest) and refused to advance farther. Eventually I traced the problem to Lines.popFront(), which pops each subrange off the range of ranges. The problem is that this only works on some ranges, but not others; if you pass the output of formatMonths() straight to pasteBlocks(), it will NOT work. Why? Because pasteBlocks return a std.algorithm.Map object, which recreates the subrange each time, so Lines.popFront() is only popping a temporary copy of the subrange, not the real thing. I was about to give up and try another approach, when out of the blue I decided to try and see if I could stuff the range returned by formatMonths() into an array, and then pass *that* to pasteBlocks() -- and behold, it worked!! This was a totally unexpected fix, that a newbie probably would never have thought of, so this is a potential trap for newcomers to D who expect components to just be pluggable. In retrospect, it makes sense -- you need to somehow buffer the ranges of formatted month lines *somewhere* in order to be able to splice them together out of their natural depth-first outer/inner range order. But this is not obvious at all from first glance; perhaps it's a sign of a leaky abstraction somewhere. We should probably look into why this is happening and how to fix it. And there should be a way to test for this in pasteBlocks' signature constraint so that future code won't fall into the same trap, but I can't think of one right now. Once this last bit worked, though, everything fell into place quickly. After all unittests were passing, no more bugs were found!! The program can print beautifully laid out calendars with no problems whatsoever. I'm so in love with D right now... If I'd done this exercise in C or C++, I'd
Re: D component programming is a joke (Was: Re: Component programming)
On 8/1/2013 10:24 PM, H. S. Teoh wrote: Once this last bit worked, though, everything fell into place quickly. After all unittests were passing, no more bugs were found!! The program can print beautifully laid out calendars with no problems whatsoever. I'm so in love with D right now... If I'd done this exercise in C or C++, I'd be spending the next 2 days debugging before I could present the code for the world to see. D ranges and unittest blocks are t3h k00l. I think this is awesome, and this + your previous post are sufficient to create a great article!
Component programming
This is only losely related to D, but I don't fully understand
the separation of component programming and OOP (cf.
https://en.wikipedia.org/wiki/Component-based_software_engineering#Differences_from_object-oriented_programming).
In an OO framwork, the objects are basically components. See also
Brad Cox of Stepstone largely defined the modern concept of a
software component.[4] He called them Software ICs and set out to
create an infrastructure and market for these components by
inventing the Objective-C programming language. (see link above)
Walter's example
(http://www.drdobbs.com/architecture-and-design/component-programming-in-d/240008321)
void main() {
stdin.byLine(KeepTerminator.yes)// 1
map!(a = a.idup). // 2
array. // 3
sort. // 4
copy( // 5
stdout.lockingTextWriter());// 6
}
This is more or less how mature OO programs look like. Ideally
each class (component) does one thing (however small the class
might be) and can be used or called to perform this task. All
other classes or components can live independently. From my
experience this is exactly what Objective-C does. Rather than
subclassing, it uses other classes to get a job done.
Re: Component programming
On Wed, Jul 31, 2013 at 12:20:56PM +0200, Chris wrote:
This is only losely related to D, but I don't fully understand the
separation of component programming and OOP (cf.
https://en.wikipedia.org/wiki/Component-based_software_engineering#Differences_from_object-oriented_programming).
In an OO framwork, the objects are basically components. See also
Brad Cox of Stepstone largely defined the modern concept of a
software component.[4] He called them Software ICs and set out to
create an infrastructure and market for these components by
inventing the Objective-C programming language. (see link above)
Walter's example
(http://www.drdobbs.com/architecture-and-design/component-programming-in-d/240008321)
[...]
Thanks for the link to Walter's article, it was a very insightful read.
I can't say I'm that clear about the difference between component
programming and OO, so I'll decline comment.
One question that the article did raise, though, was what to do when
your algorithms require non-linear interconnectivity between components.
For example, say I have an expression evaluator that takes an object
that represents a math expression, and another object representing a set
of identifier-to-value mappings, and returns the value of the expression
given those mappings:
Value evalExpr(Expr,Ident,Value)(Expr e, Value[Ident] mappings)
{
...
}
In the spirit of component programming, one would conceivably have an
expression parsing component that takes, say, an input stream of
characters and returns an expression object:
Expr parseExpr(InputRange)(InputRange input)
if (is(ElementType!InputRange : dchar))
{
...
}
And conceivably, one would also have a variable assignment parser that
parses an input stream of characters containing user-typed value
assignments, and returns a Value[Ident] hash (obviously, this last bit
can be generalized to any AA-like interface, but let's keep it simple
for now):
Value[Ident] parseBindings(InputRange)(InputRange input) { ... }
So now, my main code would look like this:
void main(string[] args) {
assert(args.length == 3);
parseExpr(args[1])
.evalExpr(parseBindings(args[2]))
.copy(stdout);
}
Which is not as pretty, because of the non-linear dependence on args[1]
and arg[2]. I've a hard time turning this into its own reusable
component, because it requires multiple disparate inputs. It's also easy
to come up with examples with multiple outputs, and, in the general
case, components with n-to-m input/output connectivity. How do we still
maximize reusability in those cases?
T
--
There are two ways to write error-free programs; only the third one works.
Re: Component programming
On Wed, 31 Jul 2013 12:20:56 +0200, Chris wrote:
This is only losely related to D, but I don't fully understand the
separation of component programming and OOP (cf.
https://en.wikipedia.org/wiki/Component-
based_software_engineering#Differences_from_object-oriented_programming).
In an OO framwork, the objects are basically components. See also
Brad Cox of Stepstone largely defined the modern concept of a software
component.[4] He called them Software ICs and set out to create an
infrastructure and market for these components by inventing the
Objective-C programming language. (see link above)
Walter's example
(http://www.drdobbs.com/architecture-and-design/component-programming-
in-d/240008321)
void main() {
stdin.byLine(KeepTerminator.yes)// 1 map!(a = a.idup).
// 2 array. // 3
sort. // 4 copy(
// 5
stdout.lockingTextWriter());// 6
}
This is more or less how mature OO programs look like. Ideally each
class (component) does one thing (however small the class might be) and
can be used or called to perform this task. All other classes or
components can live independently. From my experience this is exactly
what Objective-C does. Rather than subclassing, it uses other classes to
get a job done.
A few things:
1) The functions used in Walter's example are not methods, they are
generic free functions. The interfaces they require are not actual OOP
interfaces, but rather a description of what features the supplied type
must supply.
2) The avoidance of actual objects, interfaces, and methods means that
the costly indirections of OOP are also avoided. The compiler is free to
inline as much of the pipeline as it wishes.
3) Component programming simplifies usage requirements, OOP frameworks
complicate usage requirements (e.g. you must inherit from this class).
If anything, component programming is just functional programming +
templates and some nice syntactic sugar. And a healthy dose of pure
awesome.
D component programming is a joke (Was: Re: Component programming)
Justin Whear:
If anything, component programming is just functional
programming + templates and some nice syntactic sugar.
And a healthy dose of pure awesome.
What D calls component programming is very nice and good, but
in D it's almost a joke.
Currently this code inlines nothing (the allocations, the
difference and the product):
import std.numeric: dotProduct;
int main() {
enum N = 50;
auto a = new int[N];
auto b = new int[N];
auto c = new int[N];
c[] = a[] - b[];
int result = dotProduct(c, c);
return result;
}
If you write it in component-style (using doubles here):
import std.math;
import std.algorithm, std.range;
int main() {
enum N = 50;
alias T = double;
auto a = new T[N];
auto b = new T[N];
return cast(int)zip(a, b)
.map!(p = (p[0] - p[1]) ^^ 2)
.reduce!q{a + b};
}
The situation gets much worse, you see many functions in the
binary, that even LDC2 often not able to inline. The GHC Haskell
compiler turns similar components code in efficient SIMD asm
(that uses packed doubles, like double2), it inlines everything,
merges the loops, produces a small amount of asm output, and
there is no c intermediate array. In GHC component
programming is mature (and Intel is developing an Haskell
compiler that is even more optimizing), while in D/dmd/Phobos
this stuff is just started. GHC has twenty+ years of head start
on this and it shows.
The situation should be improved for D/dmd/Phobos, otherwise such
D component programming remains partially a dream, or a toy.
Bye,
bearophile
Re: Component programming
On Wed, Jul 31, 2013 at 09:16:20PM +, Justin Whear wrote: On Wed, 31 Jul 2013 12:20:56 +0200, Chris wrote: This is only losely related to D, but I don't fully understand the separation of component programming and OOP [...] A few things: 1) The functions used in Walter's example are not methods, they are generic free functions. The interfaces they require are not actual OOP interfaces, but rather a description of what features the supplied type must supply. 2) The avoidance of actual objects, interfaces, and methods means that the costly indirections of OOP are also avoided. The compiler is free to inline as much of the pipeline as it wishes. 3) Component programming simplifies usage requirements, OOP frameworks complicate usage requirements (e.g. you must inherit from this class). If anything, component programming is just functional programming + templates and some nice syntactic sugar. And a healthy dose of pure awesome. Keep in mind, though, that you pay for the avoidance of OO indirections by template bloat. Every combination of types passed to your components will create a new instantiation of that component. In simple cases, this is generally only a handful of copies, so it's not a big deal; but for certain frequently-used components, this can explode to a huge amount of duplicated code. They will all avoid costly OO indirections, sure, but you pay for that with larger code size, which means higher rate of CPU cache misses, larger memory footprint of the code, etc.. This makes me wonder if we can somehow have a happy marriage of the two approaches. Is it possible to have automatic template instantiation factoring, such that in highly-used templates that generate a lot of copies, can the compiler be made smart enough to figure out that automatically adding indirections to the code to reduce the number of instantiations might be better? One case I've come across before is containers. For the sake of genericity, we usually use templates to implement containers like, say, a red-black tree. However, most of the code that deals with RB trees don't really care about what type the data is at all; they implement algorithms that operate on the structure of the RB tree, not on the data. Only a small subset of RB tree methods actually need to know what type the data should be (the methods that create a new node and initialize it with data, return the data from a node, etc.). Yet, in a template implementation of RB trees, every single method must be repeatedly instantiated over and over, for every type you put into the container. Most of these method instantiations may in fact be essentially identical to each other, except perhaps for one or two places where it may use a different node size, or call some comparison function on the data in the nodes. Ideally, the compiler should be able to know when a method of a templated struct/class is transitively independent of the template parameter, and only emit code for that method once. All other instantiations of that method will simply become aliases of that one instantiation. This doesn't cover the case where the call chain may pass through methods that don't care about data types but eventually ends at a method that *does* have to care about data types; but this is solvable by factoring the code so that the type-independent code is separated from the type-dependent code, except for one or two runtime parameters (e.g. size of the data type, or a function pointer to the type-dependent code that must be called at the end of, say, a tree traversal). The compiler may even be able to do this automatically in certain simple cases. T -- If it breaks, you get to keep both pieces. -- Software disclaimer notice
Re: D component programming is a joke (Was: Re: Component programming)
On 7/31/2013 3:23 PM, bearophile wrote:
The situation should be improved for D/dmd/Phobos, otherwise such D component
programming remains partially a dream, or a toy.
Ironically, the component program from the article I wrote:
void main() {
stdin.byLine(KeepTerminator.yes)// 1
map!(a = a.idup). // 2
array. // 3
sort. // 4
copy( // 5
stdout.lockingTextWriter());// 6
}
is 2x faster than the Haskell version:
import Data.List
import qualified Data.ByteString.Lazy.Char8 as L main = L.interact $
L.unlines . sort . L.lines
Re: D component programming is a joke (Was: Re: Component programming)
On 07/31/2013 03:46 PM, Walter Bright wrote: is 2x faster What do you mean exactly? :p Ali
Re: D component programming is a joke (Was: Re: Component programming)
Walter Bright: Ironically, the component program from the article I wrote: ... is 2x faster than the Haskell version: Benchmarking code written in two different languages is tricky, there are so many sources of mistakes, even if you know well both languages. But I accept your timing. And I say that it's good :-) We should aim to be better than the Intel Labs Haskell Research Compiler (HRC) :-) Bye, bearophile
Re: D component programming is a joke (Was: Re: Component programming)
On 7/31/2013 4:17 PM, bearophile wrote: Walter Bright: Ironically, the component program from the article I wrote: ... is 2x faster than the Haskell version: Benchmarking code written in two different languages is tricky, there are so many sources of mistakes, even if you know well both languages. But I accept your timing. And I say that it's good :-) We should aim to be better than the Intel Labs Haskell Research Compiler (HRC) :-) You are right to be skeptical of cross language benchmarks. Some data points you might find interesting: 1. I made no attempt to optimize the D version (other than throwing the appropriate compiler switches). It's meant to be the straightforward naive implementation. 2. I did not write the Haskell version - Bartosz Milewski did. He admits to not being an expert on Haskell, and there may be faster ways to do it. I'll also agree with you that the component programming style is new in D, and probably could benefit a great deal from 20 years of concerted effort :-) I disagree with you that it is a toy, however. Speed is only one measure of utility.
Re: D component programming is a joke (Was: Re: Component programming)
On Thu, 01 Aug 2013 00:23:52 +0200, bearophile wrote: The situation should be improved for D/dmd/Phobos, otherwise such D component programming remains partially a dream, or a toy. Bye, bearophile I disagree with your toy assessment. I've been using this chaining, component style for a while now and have really enjoyed the clarity it's brought to my code. I hadn't realized how bug-prone non-trivial loops tend to be until I started writing this way and avoided them entirely. My policy is to aim for clarity and legibility first and to rewrite for performance only if necessary. Thus far, I don't think I've rewritten anything out of the component programming style, so while probably not optimal, it's been more than good enough.
Re: D component programming is a joke (Was: Re: Component programming)
On 7/31/13 4:17 PM, bearophile wrote: Walter Bright: Ironically, the component program from the article I wrote: ... is 2x faster than the Haskell version: Benchmarking code written in two different languages is tricky, there are so many sources of mistakes, even if you know well both languages. I measured the timings. It was in a discussion between Walter, myself, and a common friend. That friend said Haskell will do a lot better on the same task. I measured by piping 1,000,000 lines of real log data through the tested programs. His first version was: import Data.List main = interact $ unlines . sort . lines This took 51 seconds. The friend got a bit miffed complaining I only measured to ridicule his code (but this is hardly the first time hard numbers offended someone) and went to ask on Haskell fora on how to make the code faster. His second version was: import Data.List import qualified Data.ByteString.Lazy.Char8 as L main = L.interact $ L.unlines . sort . L.lines This version took 7 seconds. A debug version of the D code took 3 seconds. But I accept your timing. This is most gracious considering the crass statement with which you opened. And I say that it's good :-) We should aim to be better than the Intel Labs Haskell Research Compiler (HRC) :-) Is that a lot better than ghc? Andrei
Re: D component programming is a joke (Was: Re: Component programming)
On Wed, Jul 31, 2013 at 11:52:35PM +, Justin Whear wrote:
On Thu, 01 Aug 2013 00:23:52 +0200, bearophile wrote:
The situation should be improved for D/dmd/Phobos, otherwise such D
component programming remains partially a dream, or a toy.
Bye,
bearophile
I disagree with your toy assessment. I've been using this chaining,
component style for a while now and have really enjoyed the clarity
it's brought to my code. I hadn't realized how bug-prone non-trivial
loops tend to be until I started writing this way and avoided them
entirely.
[...]
One of the more influential courses I took in college was on Jackson
Structured Programming. It identified two sources of programming
complexity (i.e., where bugs are most likely to occur): (1) mismatches
between the structure of the program and the structure of the data
(e.g., you're reading an input file that has a preamble, body, and
epilogue, but your code has a single loop over lines in the file); (2)
writing loop invariants (or equivalently, loop conditions).
Most non-trivial loops in imperative code have both, which makes them
doubly prone to bugs. In the example I gave above, the mismatch between
the code structure (a single loop) and the file structure (three
sequential sections) often prompts people to add boolean flags, state
variables, and the like, in order to resolve the conflict between the
two structures. Such ad hoc structure resolutions are a breeding ground
for bugs, and often lead to complicated loop conditions, which invite
even more bugs.
In contrast, if you structure your code according to the structure of
the input (i.e., one loop for processing the preamble, one loop for
processing the body, one loop for processing the epilogue), it becomes
considerably less complex, easier to read (and write!), and far less bug
prone. Your loop conditions become simpler, and thus easier to reason
about and leave less room for bugs to hide.
But to be able to process the input in this way requires that you
encapsulate your input so that it can be processed by 3 different loops.
Once you go down that road, you start to arrive at the concept of input
ranges... then you abstract away the three loops into three components,
and behold, component style programming!
In fact, with component style programming, you can also address another
aspect of (1): when you need to simultaneously process two data
structures whose structures don't match. For example, if you want to lay
out a yearly calendar using writeln, the month/day cells must be output
in a radically different order than the logical foreach(m;1..12) {
foreach(day;1..31) } structure). Writing this code in the traditional
imperative style produces a mass of spaghettii code: either you have
bizarre loops with convoluted loop conditions for generating the dates
in the order you want to print them, or you have to fill out some kind
of grid structure in a complicated order so that you can generate the
dates in order.
Using ranges, though, this becomes considerably more tractable: you can
have an input range of dates in chronological order, two output ranges
corresponding to chunking by week / month, which feed into a third
output range that buffers the generated cells and prints them once
enough has been generated to fill a row of output. By separating out
these non-corresponding structures into separate components, you greatly
simplify the code within each component and thus reduce the number of
bugs (e.g. it's far easier to ensure you never put more than 7 days in a
week, since the weekly output range is all in one place, as opposed to
sprinkled everywhere across multiple nested loops in the imperative
style calendar code). The code that glues these components together is
also separated out and becomes easier to understand and debug: you
simply read from the input range of dates, write to the two output
ranges, and check if they are full (this isn't part of the range API but
added so for this particular example); if the weekly range is full,
start a new week; if the monthly range is full, start a new month. Then
the final output range takes care of when to actually produce output --
you just write stuff to it and don't worry about it in the glue code.
OK, this isn't really a good example of the linear pipeline style code
we're talking about, but it does show how using ranges as components can
untangle very complicated code into simple, tractable parts that are
readable and easy to debug.
T
--
If you compete with slaves, you become a slave. -- Norbert Wiener
