Re: Refactoring in a large code base
On Sat, Jan 23, 2016 at 12:48 AM, Marko Rauhamaa wrote: > Chris Angelico : > >> Okay. Start persuading "management" (presumably the PSU) that CPython >> needs to be more modular, with different release cycles for different >> components. Your first step is to figure out the boundaries between >> those components. Get started. > > Gladly, I don't need to do anything about CPython. This particular > strawman was erected by you: > >> CPython is a large and complex program. How do you propose doing it >> "right"? > > It's up to you to take my proposal or ignore it. > > However, I have had my share of windmills to battle; I'm happy to say > I've managed to bring down a few of them! Okay. You have fun releasing nothing that you aren't confident can fit within your definitions (and by the way, you were the one who brought up the three-month rewrite, not me); I'm going to keep on following the principle that "practicality beats purity", and release actual working code. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Refactoring in a large code base
On Sat, Jan 23, 2016 at 12:30 AM, Rustom Mody wrote:
> You just gave a graphic vivid description...
> of the same thing Marko is describing: ;-) viz.
> A full-size language parser is something that you - an experienced developer -
> make a point of avoiding.
It's worth noting that "experienced developer" covers a huge range of
skills. There are quite a few other areas that I do not tinker with
(crypto, CPU-level optimizations, and such), not because they're
impossible to understand, but because *I* have not the skill to
understand and improve them. This does mean they're complicated
(they're beyond the "one weekend of tinkering" barrier that any
serious geek should be able to invest), but I'm sure there are
language nerds out there who are so familiar with the grammar of
that they'll pick up CPython's grammar and make
a change with confidence that it'll do what they expect.
> So then the question comes down to this: Is this the order of nature?
> Or is it man-made disorder?
> Jury's out on that one for lexers/parsers specifically.
Lexers/parsers are as complicated as the grammars they parse. A lexer
for a simple structured text file can be pretty easy to implement; for
instance, JSON is pretty straight-forward, with only a handful of
cases (insignificant whitespace, three keywords, two recursive
structures that start with specific characters ('{' and '['), strings
(which start with '"'), and numbers (which start with a digit or a
hyphen)), so a parser need only look for those few possibilities and
it knows exactly what else to fetch up. I could probably write a JSON
parser in a fairly short space of time, and wouldn't be scared of
digging into the internals of someone else's. It's when the grammar
adds complexities to deal with the real-world issues of full size
programming languages that it becomes hairier. The CPython grammar is
only ~150 lines of fairly readable directives, but the parser that
implements it is ~3500 lines of C code. Pike merges the two into a
YACC file of nearly 5000 lines of highly optimized code (it has
different grammar paths for things a human would consider the same, in
order to produce distinct code). That's where I'm ubercautious.
> For arbitrary code in general, the problem that it may be arbitrarily and
> unboundedly
> complex/complicated is the oldest problem in computer science: the halting
> problem.
>
> IOW anyone who thinks that *arbitrary* complexity can *always* be tamed either
> has a visa to utopia or needs to re-evaluate (or get) a CS degree
Exactly.
ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: Refactoring in a large code base
On Friday, January 22, 2016 at 7:13:49 PM UTC+5:30, Marko Rauhamaa wrote: > Rustom Mody : > > > IOW anyone who thinks that *arbitrary* complexity can *always* be > > tamed either has a visa to utopia or needs to re-evaluate (or get) a > > CS degree > > Not all complexity can be tamed, but what you can't tame you shouldn't > release, either. And how do you propose to legislate that? If you leave it to the wetware (untamed!) boxes atop our shoulders will not two developers have wildly different complexity thresholds? And as soon as you suggest an objective (∴ algorithmic) solution to detecting complexity you have landed with the halting problem (or more precisely Rice theorem) tl;dr The HP is amazingly deceptive and you just got tripped by it -- https://mail.python.org/mailman/listinfo/python-list
Re: Refactoring in a large code base
Chris Angelico : > Okay. Start persuading "management" (presumably the PSU) that CPython > needs to be more modular, with different release cycles for different > components. Your first step is to figure out the boundaries between > those components. Get started. Gladly, I don't need to do anything about CPython. This particular strawman was erected by you: > CPython is a large and complex program. How do you propose doing it > "right"? It's up to you to take my proposal or ignore it. However, I have had my share of windmills to battle; I'm happy to say I've managed to bring down a few of them! Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: Refactoring in a large code base
Rustom Mody : > IOW anyone who thinks that *arbitrary* complexity can *always* be > tamed either has a visa to utopia or needs to re-evaluate (or get) a > CS degree Not all complexity can be tamed, but what you can't tame you shouldn't release, either. Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: Refactoring in a large code base
On Friday, January 22, 2016 at 6:05:02 PM UTC+5:30, Chris Angelico wrote: > On Fri, Jan 22, 2016 at 11:04 PM, Rustom Mody wrote: > > 2. My students trying to work inside the lexer made a mess because the > > extant lexer is a mess. > > I.e. while python(3) *claims* to accept Unicode input, the actual lexer is > > an ASCII lexer special-cased for unicode rather than pre-lexing utf8 to > > unicode > > > > These are just specific examples that I am familiar with > > > Regarding lexers specifically, I have never seen any full-size > language parser that I've wanted to tinker with. They're always highly > optimized pieces of code, dealing with innumerable edge and corner > cases, and exploring them is always like dipping my toe into something > that's either ice-cold water or highly caustic acid, and I can't tell > which. > You just gave a graphic vivid description... of the same thing Marko is describing: ;-) viz. A full-size language parser is something that you - an experienced developer - make a point of avoiding. So then the question comes down to this: Is this the order of nature? Or is it man-made disorder? Jury's out on that one for lexers/parsers specifically. For arbitrary code in general, the problem that it may be arbitrarily and unboundedly complex/complicated is the oldest problem in computer science: the halting problem. IOW anyone who thinks that *arbitrary* complexity can *always* be tamed either has a visa to utopia or needs to re-evaluate (or get) a CS degree -- https://mail.python.org/mailman/listinfo/python-list
Re: Refactoring in a large code base
On Sat, Jan 23, 2016 at 12:00 AM, Marko Rauhamaa wrote: > However, as a matter of rule, older code bases have been bloated till > they can barely be maintained. That's when the management starts to > listen to new ideas. Better late than never. Okay. Start persuading "management" (presumably the PSU) that CPython needs to be more modular, with different release cycles for different components. Your first step is to figure out the boundaries between those components. Get started. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Refactoring in a large code base
Chris Angelico : > Alright. Can you rewrite all of those modules in three months? The point is not to rewrite modules except as a fallback for a hopelessly badly written module. > And then there's the language itself. The cpython/Python directory has > 58 .c files, many of which are closely tied to each other. Putting modularization in place after the spaghetti has been created is very hard. It can be done, though, and good things come out of the effort. > I don't agree with everything Joel says, but seriously, do not waste > your time with a full rewrite - even in theory. And I can say this > from hard experience on both sides. As long as you are happy with your code base, you don't have to change everything just for an abstract principle. However, as a matter of rule, older code bases have been bloated till they can barely be maintained. That's when the management starts to listen to new ideas. Better late than never. > Would it have been better to throw the code away and start over? Again, modularization doesn't entail rewriting -- it simply makes localized rewriting a practical option for desperate situations. > So how can you rewrite *any* large project in three months? Let go of the rewriting already. Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: Refactoring in a large code base
On Fri, Jan 22, 2016 at 11:04 PM, Rustom Mody wrote: > On Friday, January 22, 2016 at 4:49:19 PM UTC+5:30, Chris Angelico wrote: >> On Fri, Jan 22, 2016 at 9:19 PM, Marko Rauhamaa wrote: >> > The knowhow, vision and skill is apparently very rare. On the product >> > management side, we have the famous case of Steve Jobs, who simply told >> > the engineers to go back to the drawing boards when he didn't like the >> > user experience. Most others would have simply surrendered to the >> > mediocre designs and shipped the product. >> > >> > We need similar code sanity management. Developers are given much too >> > much power to mess up the source code. That's why "legacy" is considered >> > a four-letter word among developers. >> >> So what do you do with a huge program? Do you send it back to the >> developers and say "Do this is less lines of code"? >> >> CPython is a large and complex program. How do you propose doing it "right"? > > Put thus 'generistically' this is a rhetorical question and makes Marko look > like > he's making a really foolish point > > Specifically, what little Ive seen under the CPython hood looked distinctly > improvable. egs. > > 1. My suggestion to have the docs re. generator-function vs generator-objects > cleaned up had no takers > 2. My students trying to work inside the lexer made a mess because the extant > lexer is a mess. > I.e. while python(3) *claims* to accept Unicode input, the actual lexer is > an ASCII lexer special-cased for unicode rather than pre-lexing utf8 to > unicode > > These are just specific examples that I am familiar with Yes, there are some parts of CPython that can be improved. That's true of every large project (it's said that every program has at least one bug and could be shortened by at least one instruction, from which it can be deduced that every program can be reduced to a single instruction that doesn't work). Regarding lexers specifically, I have never seen any full-size language parser that I've wanted to tinker with. They're always highly optimized pieces of code, dealing with innumerable edge and corner cases, and exploring them is always like dipping my toe into something that's either ice-cold water or highly caustic acid, and I can't tell which. > Chris' general point still stands, viz take the large and complex program > that is cpython > and clean up these messinesses: You will still have a large and complex > program Right. You could definitely spin off *some* of CPython into a separate project (flip through the standard library - quite a few of those modules, if proposed for stdlib inclusion today, would be denied "better on PyPI"), but my point isn't that it can't be improved, but that there's an irreducible complexity to it that exceeds the "rewrite in a quarter" mark by a huge margin. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Refactoring in a large code base
On Fri, 22 Jan 2016 04:04:44 -0800, Rustom Mody wrote: > These are just specific examples that I am familiar with Chris' general > point still stands, viz take the large and complex program that is > cpython and clean up these messinesses: You will still have a large and > complex program I'm not really sure what the point is we're working on... let me propose these: - unix principle is good: keep things simple, limited in scope. Then leverage that. - there will always be complexity, but if the complexity is modularized, it's controlled. In particular, the complexity of a program should represent the complexity of the problem. I call that "structural complexity". To be avoided, corrected, is "superficial complexity", where the complexity of a system is squished into a single (or reduced number of) planes. Like vomiting a program onto a desk. - "Advice" that the program needs to be refracted is generally not helpful. -- https://mail.python.org/mailman/listinfo/python-list
Re: Refactoring in a large code base
On Fri, Jan 22, 2016 at 10:54 PM, Marko Rauhamaa wrote: > Chris Angelico : > >> On Fri, Jan 22, 2016 at 9:19 PM, Marko Rauhamaa wrote: >> So what do you do with a huge program? > > Modularize. Treat each module as a separate product with its own release > cycle, documentation, apis, ownership etc. > > What is a reasonable size of a module? It is something you would > consider replacing with a new implementation with a moderate effort > (say, in a single quarter). > >> CPython is a large and complex program. How do you propose doing it >> "right"? > > I don't know CPython specifically to give solid recommendations, but I > would imagine the core language engine should be in a repository > separate from the standard library, and most standard library modules > should be in their respective repositories and have their individual > internal release cycles. > > A CPython release would then weave the package together from the > components that were previously (internally) released. Okay. So let's suppose we strip out huge slabs of the standard library and make an absolutely minimal "base library", with an "extended library" that can run on its own separate release cycle. (This has already been discussed; the biggest problems with the idea aren't technical, but logistical - not just for the Python devs but for everyone who has to get approval for software upgrades.) Let's suppose the base library consists of just the modules necessary for a basic invocation: rosuav@sikorsky:~$ python3 -c 'import sys; print(sorted(sys.modules.keys()))' ['__main__', '_codecs', '_collections_abc', '_frozen_importlib', '_frozen_importlib_external', '_imp', '_io', '_signal', '_sitebuiltins', '_stat', '_sysconfigdata', '_thread', '_warnings', '_weakref', '_weakrefset', 'abc', 'builtins', 'codecs', 'encodings', 'encodings.aliases', 'encodings.latin_1', 'encodings.utf_8', 'errno', 'genericpath', 'io', 'marshal', 'os', 'os.path', 'posix', 'posixpath', 'site', 'stat', 'sys', 'sysconfig', 'zipimport'] Alright. Can you rewrite all of those modules in three months? Not even digging into the language itself, just the base library. This is the bare minimum to get a viable Python execution environment going (you might be able to cut it down a bit, but not much), so it can't be modularized into separate projects. And then there's the language itself. The cpython/Python directory has 58 .c files, many of which are closely tied to each other. The cpython/Objects directory has another 39, representing specific object types (bytes, tuple, range, method) that are implemented in C. And cpython/Parser has 17 more just to handle the language parser. Edits often affect multiple files and must be kept in sync. How would you modularize that out? Which part would you spin off as a separate project with its own release cycle? The garbage collector? The string object? The peephole optimizer? The import machinery? Each of these is already too big to rewrite in three months, plus they're fairly tightly linked to all the other modules. All that code represents the accumulation of hundreds of thousands of fixes to prevent tens of millions of bugs (some of which will be visible on bugs.python.org, but most would have been found and prevented during early testing); throwing the code away means throwing all that away. http://www.joelonsoftware.com/articles/fog69.html I don't agree with everything Joel says, but seriously, do not waste your time with a full rewrite - even in theory. And I can say this from hard experience on both sides. I have an active project for a MUD server, which was originally deployed as a byte-oriented service (it took ASCII-compatible bytes from clients and sent those same octets out to other clients). When I decided that the server should work with Unicode text internally (expecting and transmitting UTF-8), I kept on coming across stupid problems where the code had been written with faulty assumptions, and I had to keep on fixing those. Would it have been better to throw the code away and start over? Well, let me tell you, it would certainly have made the Unicode handling a lot easier, so if you're looking at starting your own project, make sure you learn from my hassles and bake in Unicode support from the start! But that would have meant throwing away all the bugfixes for all the bugs that I'd noticed across the years, such as: 1) On login, typing "quit" when prompted for a user name or password would log you out. The "passwd" (change password) command had to also prevent you from *setting* your password to "quit", because that would effectively lock your account against login. 2) Some clients send backspace as 08; others send FF; some send 08 20 08. Cope with them all. 3) If a bug prevents the admin account from working, there needs to be a way to diagnose and fix that code using shell access to the back-end server, without needing the actual admin account. Etcetera, etcetera, etcetera. There's no way to "rewrite but k
Re: Refactoring in a large code base
Rustom Mody : > These are just specific examples that I am familiar with Chris' > general point still stands, viz take the large and complex program > that is cpython and clean up these messinesses: You will still have a > large and complex program No, as long as the ugly parts are compartmentalized, you have a better chance at refactoring them -- or replacing them altogether. Modularization is an obvious, but under-practiced, method of managing complexity. Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: Refactoring in a large code base
On Friday, January 22, 2016 at 4:49:19 PM UTC+5:30, Chris Angelico wrote: > On Fri, Jan 22, 2016 at 9:19 PM, Marko Rauhamaa wrote: > > The knowhow, vision and skill is apparently very rare. On the product > > management side, we have the famous case of Steve Jobs, who simply told > > the engineers to go back to the drawing boards when he didn't like the > > user experience. Most others would have simply surrendered to the > > mediocre designs and shipped the product. > > > > We need similar code sanity management. Developers are given much too > > much power to mess up the source code. That's why "legacy" is considered > > a four-letter word among developers. > > So what do you do with a huge program? Do you send it back to the > developers and say "Do this is less lines of code"? > > CPython is a large and complex program. How do you propose doing it "right"? Put thus 'generistically' this is a rhetorical question and makes Marko look like he's making a really foolish point Specifically, what little Ive seen under the CPython hood looked distinctly improvable. egs. 1. My suggestion to have the docs re. generator-function vs generator-objects cleaned up had no takers 2. My students trying to work inside the lexer made a mess because the extant lexer is a mess. I.e. while python(3) *claims* to accept Unicode input, the actual lexer is an ASCII lexer special-cased for unicode rather than pre-lexing utf8 to unicode These are just specific examples that I am familiar with Chris' general point still stands, viz take the large and complex program that is cpython and clean up these messinesses: You will still have a large and complex program -- https://mail.python.org/mailman/listinfo/python-list
Re: Refactoring in a large code base
Chris Angelico : > On Fri, Jan 22, 2016 at 9:19 PM, Marko Rauhamaa wrote: > So what do you do with a huge program? Modularize. Treat each module as a separate product with its own release cycle, documentation, apis, ownership etc. What is a reasonable size of a module? It is something you would consider replacing with a new implementation with a moderate effort (say, in a single quarter). > CPython is a large and complex program. How do you propose doing it > "right"? I don't know CPython specifically to give solid recommendations, but I would imagine the core language engine should be in a repository separate from the standard library, and most standard library modules should be in their respective repositories and have their individual internal release cycles. A CPython release would then weave the package together from the components that were previously (internally) released. Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: Refactoring in a large code base
On Fri, 22 Jan 2016 12:19:50 +0200, Marko Rauhamaa wrote: > We need similar code sanity management. Developers are given much too > much power to mess up the source code. That's why "legacy" is considered > a four-letter word among developers. When I started in this business, in the mid-70s, there was the prospect of my working under a "programmer-analyst" - there was, then, a whole hierarchy of programmers. I resisted that bitterly and was "lucky" enough to be at the forefront of changes - I would be able to avoid that until the concept was dead. Now, 40 years later ... it seems like a good idea to me ... but more dead than it's ever been and getting deader all the time. Part of the problem is that the whole profession is dead - now the people doing the programming are the application experts, and just programmers are considered at the level that we used to consider "operators" were at :) -- https://mail.python.org/mailman/listinfo/python-list
Re: Refactoring in a large code base
On Fri, Jan 22, 2016 at 9:19 PM, Marko Rauhamaa wrote: > The knowhow, vision and skill is apparently very rare. On the product > management side, we have the famous case of Steve Jobs, who simply told > the engineers to go back to the drawing boards when he didn't like the > user experience. Most others would have simply surrendered to the > mediocre designs and shipped the product. > > We need similar code sanity management. Developers are given much too > much power to mess up the source code. That's why "legacy" is considered > a four-letter word among developers. So what do you do with a huge program? Do you send it back to the developers and say "Do this is less lines of code"? CPython is a large and complex program. How do you propose doing it "right"? ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Refactoring in a large code base
Rustom Mody : > On Friday, January 22, 2016 at 1:59:15 PM UTC+5:30, Marko Rauhamaa wrote: >> I've been there. I think the root problem is to have a code base >> that's so large and complex. > > Bizarre comment... Are you saying large and complex code-bases should > non-exist? Why, yes, I am. >> It *could* be avoided if the engineering director only cared. > > Some problems are trivially solvable... for those who have the knowhow > Some problems are inherently hard but easily detectable as such... > once again for those who have the knowhow And some are literally (and > ironically trivially) unsolvable The knowhow, vision and skill is apparently very rare. On the product management side, we have the famous case of Steve Jobs, who simply told the engineers to go back to the drawing boards when he didn't like the user experience. Most others would have simply surrendered to the mediocre designs and shipped the product. We need similar code sanity management. Developers are given much too much power to mess up the source code. That's why "legacy" is considered a four-letter word among developers. Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: Refactoring in a large code base
On Friday, January 22, 2016 at 1:59:15 PM UTC+5:30, Marko Rauhamaa wrote: > Ben Finney : > > > The author points out there are times when a code base is large and > > complex enough that refactoring puts the programmer in a state of not > > knowing whether they're making progress, because until the whole > > refactoring is complete the errors just cascade and it's hard to tell > > which ones are relevant. > > I've been there. I think the root problem is to have a code base that's > so large and complex. Bizarre comment... Are you saying large and complex code-bases should non-exist? > > It *could* be avoided if the engineering director only cared. Some problems are trivially solvable... for those who have the knowhow Some problems are inherently hard but easily detectable as such... once again for those who have the knowhow And some are literally (and ironically trivially) unsolvable The CS-trinity: 'normal' problems, problems in NP, undecidable problems is a classic example of this. However applying that in real-world practice can be highly non-trivial, requiring from specialized knowledge to intelligence to genius. IOW "engineering director does not care" is likely true but also a gross oversimplification -- https://mail.python.org/mailman/listinfo/python-list
Re: Refactoring in a large code base
Ben Finney : > The author points out there are times when a code base is large and > complex enough that refactoring puts the programmer in a state of not > knowing whether they're making progress, because until the whole > refactoring is complete the errors just cascade and it's hard to tell > which ones are relevant. I've been there. I think the root problem is to have a code base that's so large and complex. It *could* be avoided if the engineering director only cared. Marko -- https://mail.python.org/mailman/listinfo/python-list
Refactoring in a large code base (was: importing: what does "from" do?)
Rustom Mody writes: > You may find this strategy useful: > https://pchiusano.github.io/2015-04-23/unison-update7.html > > particularly the section "Limitations of refactoring via modifying > text files in place, and what to do instead" A direct link to that section is https://pchiusano.github.io/2015-04-23/unison-update7.html#refactoring-lessons>. The author points out there are times when a code base is large and complex enough that refactoring puts the programmer in a state of not knowing whether they're making progress, because until the whole refactoring is complete the errors just cascade and it's hard to tell which ones are relevant. That’s bad for two reasons. Without this feedback, you may be writing code that is making things worse, not better, building further on faulty assumptions. But more than the technical difficulties, working in this state is demoralizing, and it kills focus and productivity. All right, so what do we do instead? Should we just avoid even considering any codebase transformations that are intractable with the “edit and fix errors” approach? No, that’s too conservative. Instead, we just have to *avoid modifying our program in place*. This lets us make absolutely any codebase transformation while keeping the codebase compiling at all times. Here’s a procedure, it’s quite simple […] https://pchiusano.github.io/2015-04-23/unison-update7.html#refactoring-lessons> The technique is not presented as flawless, and interesting trade-offs are discussed. Quite an interesting article. -- \ “Science and religion are incompatible in the same sense that | `\ the serious pursuit of knowledge of reality is incompatible | _o__) with bullshit.” —Paul Z. Myers, 2010-03-14 | Ben Finney -- https://mail.python.org/mailman/listinfo/python-list
