On 02/04/16 01:18, Thomas Gelf wrote:
Hi Henrik,
thanks a lot for your response!
Am 01.04.2016 um 20:21 schrieb Henrik Lindberg:
The C++ implementation is several orders of magnitudes faster than the
ruby implementation. i.e. something silly like tens of thousands of
times faster.
No doubt on this, I believe you without any benchmark.
Although, I got an order or two too carried away with superlatives here.
We will probably end up with 100s' to 1000's times faster.
The Ruby lexing/parsing and validation alone can take minutes on a
complex set up. We have shown earlier though benchmarks that lexing
alone is a bottleneck in any catalog compilation - every optimization
there contributes greatly to the bottom line.
Could you share some details on this? What kind of catalogs are you
talking about? How many resources, parameters, how large are they - and
what makes them so large and slow? Still, no doubt that C++ will be able
to lex the same catalogs in a fraction of the time.
It is pretty much linear to the amount of source text. Though, depending
on which kind of performance we are talking about here, it does
naturally have 0 impact when an environment is cached since everything
is parsed just once.
We have already measured the approach. The benefit on the ruby side is
that the lexing is delegated to a native implementation that reads
binary. A spike was performed with Ruby Marshal, which also compared to
a native MsgPack.
Ok, so basically a linked c-based lexer could give the same performance
boost? Yes, I know, JRuby. But still, could this be true?
A c++ based lexer would indeed be beneficial. The parser is already
using a c++ driver, but it makes judicious call-outs to Ruby. The
construction of the AST is the second bottleneck - on par with lexing.
Thirdly, the decoupling of the lexing/parsing from the runtime makes it
possible to tabulate and compact the AST (as data is serialized). That
is less meaningful if done in the same process since it takes time
although it would reduce the number of objects managed by the ruby
runtime (less memory, lighter job for the GC).
We felt that although doable (we have this as an alternative; link the
lexer/parser/validator into the runtime (MRI or into Puppet
Server/JRuby) - that would give us a number of interesting technical
challenges to solve that we are not sure the right problems to solve.
It may be - we are not sure, but initially we rather spend time making
the c++ based lexer/parser/validator as good as possible (rather than
solving the technical packaging the most complicated way).
The main point here is that we are transitioning to a full
implementation of the puppet catalog compiler to C++ ... The use of XPP
makes this possible.
This is where I started to feel no longer comfortable while reading the
proposal. No caching mechanism that helped my in Puppet comes to my
mind, but I could immediately tell a lot of anecdotes involving severe
Puppet issues breaking whole environments just because of caching issues.
We are happy if we initially only get 5-10% out of this...
And this is where I currently disagree. Very often I invest lots of time
for just 1%. But being able to run without a fragile caching layer could
be worth even 50% as long as I'm able to scale. When someone has to stop
a deployment chain because he needs to troubleshoot a caching layer,
lot's of people are sitting around and cannot work. Ask them whether
they would have preferred to buy more hardwar.
We have to start somewhere and when doing so we want to apply the KISS
principle. The intent is for puppet server to automatically keep the XPP
files in sync. There may be no need for "caching" - it is simply done as
a step in atomic deploy of modified puppet code.
We are hoping for more though.
I hope you're pretty confident on this ;)
yes, but unsure how quickly we get there and if facing a tradeoff of
doing more work on the Ruby side vs. spending the time on the c++ side
to make more parts of compilation work (i.e. integrating "ruby legacy")
- what we actually will decide to do. There is some fairly low hanging
fruit on the Ruby side we could speed up like the AST model itself.
That is not the intent (drop all Ruby things) - our plan is to make a
smooth transition. All "boil the ocean" strategies tends to fail, so
backwards compatibility and gradual change is important to us.
Eeeeeeh... Sorry, this is slightly OT, but by the end it isn't. This
"transition" is the root cause for a proposal with the potential for a
lot of additional trouble. You do not want to "drop all the Ruby
things", but you want to have a smooth transition. Without knowing where
this transition should lead this sounds like a contradiction to me.
So, where will this transit phase lead to? That's IMO the question that
many would love to see answered. Will ruby still be there? So where is
the transition? If it won't, how would it's successor look like? I guess
you know what I mean, please enlighten us!
It is too premature to describe this in detail. Happy to share the ideas
which we plan to pursuit though.
At this point we have decided to try an approach where the c++ compiler
will use an RPC mechanism to talk to co-processors. When doing so it
will use the same serialization technology that is used in XPP
(basically based on the Puppet Type System). (Rationale: linking native
things into the same memory image is complex and creates vulnerabilities).
The actual implementation of types and providers are actually not needed
at compile time - only the meta data (that a resource type exists
basically + a few other details) - today we do not even validate the
attribute values of a resource type at compile time - that takes place
when the catalog is applied. Thus, all we need at compile time is the
meta data for the resource types. For this we have just started to
explore ways to provide this (tools to extract the information from the
implemented types and providers). We will continue with this after XPP.
That pretty much leaves functions written in Ruby, and hiera backends.
As a hiera backend/data provider can be thought of as functions as well,
we believe that the RPC based approach will work fine. This also to be
continued after XPP (as we then have the serialization/deserialization
parts in place in both the c++ and ruby implementations.
In the long run, in general, we want it to be possible to express as
much as possible using the Puppet Language itself, and where that is not
practical, that it is easy to integrate an implementation (written in
c++, ruby, or whatever the logic is best written in for the target).
In a current Puppet ecosystem a C++ parser able to generate an AST from
a .pp file to me still seems far from anything that could completely
replace the current Ruby-based parser in a helpful way very soon. At
least not in a real-world environment with lot's of modules, custom
functions and external data sources, often provided by custom lookup
functions. At least not in a way that would bring any benefit to the
average Puppet user.
The goal is to do this transparently.
Sorry, couldn't follow you. Referring what?
"this" = integrating existing Ruby code that are implementations of
functions and lookup backends. (I think the answer to the question above
outlines the ideas for how we think this will work.
So, to me the former one remains a key question to the performance
benefit we could get from all this. As long as the Ruby runtime is
supported, I do not really see how this could work out. But this is just
a blind guess, please prove me wrong on this. ... But then we should
add something else to the big picture: how should we build custom
extensions and interfaces to custom data in the future? Forking plugins?
That topic is indeed a big topic, and one that will continue as we are
working towards a C++ based environment. The key here is
interoperability where extensions are supported in Ruby, or in a
language it makes sense to implement them in.
Shouldn't those questions be answered first? Aren't external data
lookups, Hiera, Database persistence, plugin-sync, file-shipping and all
the rest still far more expensive than the lexer? I would love to
understand how my I should expect to do my daily work in a world unless
the "smooth transition away from Ruby".
We (puppet labs) are working on many fronts here. XPP is not the only
work going on to speed things up in the overall process. Other teams
should talk about those things.
It's hard to judge the value of a brick without knowing how the expected
building should look like.
Expect to see a lot more about this later in the game.
I'm sure I will. But Eric asked for feedback on XPP right now ;)
:-)
It is anything but academic. What we could do, but are reluctant to do
is to link the C++ parser into the ruby runtime...
I would immediately support that approach!
...it would still need to pass the native/ruby object barrier - which
XPP is handling - if linked into memory it would just be an internal
affair.
Correct. I have no problem with this part of "XPP". Eric presented it as
"something like pyc", being pre-parsed and therefore behaving like a
caching layer. This week I worked for a European national bank, brought
them Puppet Enterprise, deployments are rare and well planned. It would
work for them. In ten days I work for a customer where I see Puppetfile
commits every two minutes, r10k and more, OpenSource Puppet, all
environments changing and moving all the time. Not only wouldn't they
benefit from some "intelligent" caching layer. I bet they would suffer.
Badly.
So: C++ Lexer -> fine. Linked into Ruby -> my preferred variant. Using
an XPP-like interface: also fine. ".pyc"-like precaching: no. This is
what I'm completely against right now, this is where I see no real
advantage. Please postpone this, do not even suggest to store those
files. Let the lexer grow and get mature, then let's re-evaluate whether
polluting our modules (or mirrored structures) with all those files
would make any sense.
But please do forget that the extensibility of a tool is one of the key
features of any OpenSource software. ...breaking them is a no-go.
Backwards compatibility and interop is of the utmost concern. We belive
that breaking things apart and specifying good APIs and providing well
performing communication between the various part of the system is key
from moving away from the now quite complicated and slow monolithic
implementation in Ruby.
Cool! I know I repeat myself, but could you already leak some details
how this Ruby-less "interop" will look like?
Some clues above to what we are thinking above. Cannot promise when we
have something more concrete to talk about - would love to be able to do
so around next Puppet Conf.
* longevity of file formats: ... An AST would per definition be a lot
more fragile. Why should we believe that those cache files would survive
longer?
Because the are well defined as opposed to how things were earlier where
things just happened to be a certain way because of how it was
implemented. Knowing what something means is the foundation that allows
it to be transformed. And when something is "all data" as opposed to
"all messy code", it can be processed by tools.
I would mostly agree, but experience teaches me to not trust such
statements. And your problem is: an AST is not data. It cannot be
represented in a defined structure. And we are in a phase where even
data types are still subject to change, with lot's of new related
features in 4.4. All this would affect an AST, wouldn't it?
The AST is indeed a data structure, not even a very complicated one.
The rate of change has dramatically gone done. We rarely touch the
grammar and the AST itself, and the last couple of changes have been
additions. This is the benefit of the "expression based approach" taken
in the "future parser" - the semantics are not implemented in the
grammar, and they are not implemented as methods/behavior inside the AST
objects.
The operation is described by this function call:
evaluate(validate(parse(lex(source))))
the lex function produces tokens (a data structure; array of tokens)
the parse function produces AST (a tree data structure)
the validate function walks the AST and checks for semantic errors
the evaluate function walks the AST to evaluate its result (and side
effects)
This wouldn't be an issue for the "C++ is our lexer" approach, but it is
obviously essential when XPP will be used as cache files, designed to be
shipped with modules.
The "shipped with modules" is what seems to be what most have concerns
about and where it seems that a "produce all of them at deploy time" is
perceived as far less complex.
As noted in the document - the requirements where thought to be where we
needed to spend more time ensuring that we define a process that works
well. (There will be revisions there :-).
As an example - what makes things expensive in Ruby is creation of many
object and garbage collection. (in lexing, each and every character in
the source needs to be individually processed... When this is done with
a C++ serializer all of the cost is on the serializing side...
Ruby didn't impress me with it's unserialization speed either. So some
cost will still be there in our overall picture. I blindly believe that
the C++ lexer is ways faster. But the only number I'm interested in is
the the difference between "catalog built and shipped by Ruby" and
"catalog built while being lexed with c++, serialized, unserialized with
Ruby and shipped with Clojure". That's the real saving.
Yes, it is naturally the "time to build the catalog" that everyone sees
and measures.
(These secondary effects have not been benchmarked in puppet, but has
proven to be very beneficial in implementations we have used in the past).
Would be interesting. Languages behaving similar have proven to
outperform "better" ones in specific use cases even if wasting a lot
more memory. But honestly, no, it doesn't really interest me. But I'd
love to learn more about what kind of catalogs you where talking about
when you are facing minutes! of lexing time.
Well, not just lexing - did I say that? That was wrong. Compilations
often take minutes though. In most cases the process
validate(parse(lex(source))) shows up at the top of any compilation
profiling. Many of the other bottlenecks are more of algorithmic nature,
and are caused by things you have also found (managing thousands of
small files instead of a large file, etc).
Even lot's of .pp files summing up to a few thousand single resources
shouldn't require more than 10-30 MB of lexing memory (blind guess,
didn't measure) and more than 3 seconds of parsing/validation time in
Ruby. None of the large environments I'm playing with are facing such
issues.
Disclaimer: all "my" large ones are still running 3.x, so no idea
whether 4.x and/or Puppet Server is so much slower - but I don't think
so. And usually when catalogs tend to have tens of thousands of
resources the root cause is quickly identified and easily replaced with
a cheaper approach. Something like "Use a custom function, aggregate on
the master, ship a single file instead of thousands" more than once
helped to bring Puppet runs from lasting more than half an hour down to
10 seconds.
Back to my question: could you let us know what kind of catalogs tend to
require minutes of lexing time?
Basically extrapolated from benchmarks of small/medium catalog
compilation doing non crazy stuff. It assumes though that very long
compilation times are more of the same rather than user "design flaws"
(managing lots of small things vs. larger, poor design of data lookup,
poor algorithms used for data transformation etc.).
These concerns are shared. It is the overall process more than the lower
level technical things that I worry about getting right.
:)
The requirements and exactly how/when/where XPPs gets created and used
will require an extra round or two of thought and debate.
Agreed. C++ Lexer, AST handed over to Ruby, linked or not: go for it.
XPPs on my disk: please not. Not yet. Not unless we have more experience
with the new lexing construct. Not unless we know how to tackle various
potential caching pitfalls in endless customized variants of Puppet
module deployments.
Thanks you Thomas for all of the valuable comment and insights.
Thank you for reading all this, Henrik - and thanks a lot for sharing
your thoughts!
To be continued over beers somewhere...
- henrik
Cheers,
Thomas
--
Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/
--
You received this message because you are subscribed to the Google Groups "Puppet
Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-dev/5701C1A4.10307%40puppetlabs.com.
For more options, visit https://groups.google.com/d/optout.
