(DJ): > The language assumption I make is that relative occurrence of
> operators is the same across languages.
(ROK): > I don't see any reason to believe that.
> From my little study (examining a program Derek Jones has), we know
> that the relative occurrence of operators isn't the same across
> programs within ONE language, so why should it be the same across
> languages?
If lots of source is measured we can find an average for that
language.
This is true by definition: measure any set of 1 or more things and
you can find an average. But such averages are never interesting without
an indication of how much things vary.
There will obviously be variation across different programs.
How much is that variation
As I have shown, it can be quite large. The major difference between
the program I measured and the programs that DJ measured is that I
measured a program that does lots of number-crunching.
and to what extent will this variation impact subject's
performance in this experiment (to the extent that performance
is correlated with number of occurrences encountered)?
Over reasonably large code bases the variation is surprisingly
small (at least for C and Pascal, for which I have measurements).
OK, let's try another program, and see if it has a similar distribution
of operators. In fact, let's try two.
Program R: the one I reported before.
Program S: 333 k lines, 199 kSLOC.
Program E: 261 k lines, 167 kSLOC
Program R Program S Program E Jones data
35.00% * 37.62% * 43.35% * 17.62% ==
11.48% - 11.47% == 9.72% + 12.46% +
10.58% == 9.46% && 8.78% & 9.59% &
7.84% & 8.67% & 8.34% == 8.73% !=
7.10% + 7.46% - 7.75% - 8.55% &&
4.99% != 5.03% + 4.56% != 7.59% <
4.42% < 4.38% < 4.03% < 6.27% -
4.29% && 4.15% != 2.96% && 5.23% ||
3.11% || 2.75% || 2.38% || 5.19% |
2.89% > 2.54% > 2.15% > 4.32% *
1.75% <= 1.43% | 1.36% >= 3.82% >
1.75% >= 1.30% >= 1.11% | 2.67% <<
1.71% / 1.24% / 0.80% << 2.18% >=
1.32% % 0.99% <= 0.74% % 1.92% /
0.76% | 0.79% % 0.71% <= 1.84% >>
0.58% << 0.25% << 0.62% / 1.34% <=
0.34% >> 0.23% ^ 0.55% >> 0.49% %
0.09% ^ 0.23% >> 0.09% ^ 0.22% ^
I presume that Derek Jones used a program that can tell the difference
between binary and unary operators, not that this can be done reliably
in the presence of unexpanded macros. I used a crude token-counting
program that I happened to have handy, which cannot tell prefix * from
infix *, which is presumably why my counts for * are so very high.
Clearly I need a better program, but I also wanted to get some results
out today, so made do with what I had. The unary operators are common
in C: ++ and -- are around the 5% level each. As someone used to
programming languages in which unary operators can have many different
precedence levels, I think that removing them from consideration actually
made the C operator precedence scheme seem simpler than it really is.
The fact that unary "!" and "-" have the wrong precedence is a fact that
one has to remember, after all.
I happen to have picked one code base to present. I think it is
representative.
Let's not "think". Let's measure.
In Derek Jones's sample (890423 operators), the bitwise operators
|, ^, <<, and >> account for 9.91% of the operators.
In my sample (50296 operators), they account for 2.11% of the operators.
In his sample, another 9.59% are the bitwise "&" operator; even with
the unary address-of operator mixed in, my sample can't match that.
In a 3D computer graphics libary I checked just now (28 k lines,
18 kSLOC) bitwise operators were 1.2% of the total.
To avoid the problem with &, let's look at the ratio
(|,^,<<,>>) / (&&,!!).
It ranges from 0.18 to 0.72, and the Jones sample is the high one.
At the very crudest estimate, Derek Jones's sample does a lot more
bitwise operations than any of the programs I inspected.
It is precisely the addition of the bitwise operators to an otherwise
straightforward (multiplication, addition, comparison, and, or) scheme
that makes C operators complicated, and one of the things that prudent
developers do, in my experience with some of them, is to *actively*
ignore the precedence of the bitwise operators and parenthesise them
fully. Ignorance of the relative precedence of (say) "&" and "=="
need not indicate a developer's failure to learn; it may instead
reflect a developer's realisation that it is dangerous to rely on
and prudent to forget.
In Derek Jones's sample, "%" is the second rarest operator. It's 100
times less frequent than bitwise operations. But in my sample, it's
only (rough guess) 4 times less frequent than bitwise operations. In
one program, "%" was 20% COMMONER than "/", while in his sample it was
4 times rarer.
Consider the ratio of + to -. This is substantial variation, about
Program R : 0.62 a factor of two each way, between two
Program S : 0.67 of the commonest operators in the C
Program E : 1.26 language.
Jones data: 1.99
Each of the programs I mentioned is a large open source project containing
much code in at least one other programming language as well; one of them
has one principal maintainer, while the others have large support teams.
All of them have been actively maintained for many years, mostly by the
same core teams (one each).
While there are some broad similarities between the samples, the
relative abundance of bitwise operations is *importantly* different.
> Let me offer the obvious point that we know that operator usage is
> quite different between C and C++ for the simple reason that C++ has
> overloaded << and >> . In C, << and >> are moderately rare operators
> meaning only bitwise shift. In C++, they are also frequent operators
> meaning input/output.
Overloading is much talked about, but the number of occurrences in
comparison to other operator usage is remarkable small.
I had hoped to confine my counts to C. OK, here goes.
A certain C++ program has 94 k lines, 54 kSLOC.
38.95% *
9.78% &
8.78% <<
8.59% <
7.51% -
7.00% ==
6.71% >
3.56% &&
2.18% !=
1.91% ||
1.83% +
1.03% |
0.50% >=
0.48% <=
0.45% >>
0.38% %
0.21% /
0.12% ^
Notice something? That's right, "<<" up near the top. NOT "remarkable
small". It turns out that 250 out of the 54 kSLOC are operator overloading
declarations (not counting 3 for operator new). Practically all of the
C++ operators that can be overloaded are in this program, including "()"
and "[]". This is a real program, designed to process complex input and
produce complex output.
In large programs language related I/O operations (eg, calls to
printf and << in C++) is very small.
Well, in this program, they are about 8% of the total, if not more.
And if I didn't have unary * and & in there as well, the proportions
would be even higher. It's not even my program; it's from IBM.
See figure 2023.1 of
www.knosof.co.uk/cbook/usefigtab.pdf for C usage measurements.
When it contradicts my own measurements? The point I am making here is
that different programs do different kinds of things; some developers
may have experience working on programs that match your statistics,
other developers may have very different experience working on programs
that match my measurements, and other developers may work on programs
even more different, getting even more different experiences.
You also need the figure that function calls occur on average
1 every 5 statements.
My measurement on my sample is that 1 line in 3 contains a function or
macro call. Much depends on what you count as a statement, and whether
you regard inline functions as functions or as macros. None of this has
anything much to do with operators.
Fortran & Ada have different sets of operator precedences.
Not *much* different. The only major difference is that Ada forbids
mixing "and" and "or" without parentheses, so that you can't tell whether
one has higher precedence than the other or not.
> If it doesn't, then people are doing it wrong. What's OO for if not
> to hide complex code and make it simply reusable?
OO is for many things, many of which never seem to occur to many
users of the language. But we digress. Do we equate complex code
with expressions that contain two or more binary operators? I
would be a bit more sophisticated than that.
Come on! I didn't *equate* complex code with such expressions.
But I do *include* such expressions amongst "complex code", and the
C++ code I've looked at does tend to hide such things inside inline
member functions.
> The way to find out what people believe is to ask them.
I did ask them, but in a round about way.
No, you didn't ask them about their beliefs.
You asked them to DO something, but it is entirely possible that their
beliefs could be perfectly correct and yet their performance could
still be highly inaccurate. Someone could know the i before e rule
"i before e, except before gh or ve" perfectly, and yet fail to apply
it consistently. While I was writing Algol and Pascal, I kept on typing
"being" when I meant "begin". *I* knew what I meant, but my fingers didn't!
When I switched to C and other languages that don't use "begin", for several
years I kept typing "begin" when I meant "begin". Performance mistakes do
*not* necessarily indicate errors in belief.
One thing point, for example, is that programmers can know whether they
have a belief about the relative precedence of two operators or not.
For example, I know that I don't know the relative precedence of "|" and "^".
(I did know yesterday, when I looked it up; today I have forgotten again.
It is better that I shouldn't know.) The paper did not describe any way
for subjects to indicate that they didn't know an answer, and did not
analyse the expected four way (right, wrong, didn't know, didn't answer)
split.
Do you think I would have gotten a straight answer to a request
to list operator precedence?
Yes, why not? It's not as if you were asking "are you now, or have you
ever been, a member of the Communist Party."
as things stand, you don't have an answer at all to
the question of what programmers believe about C precedence.
You haven't even reported on the question of whether people solve
your problem consistently. Given the task of adding parentheses to
x $ y @ z
and
x @ y $ z
do they always put the parentheses around the same operator?
Someone who adopted the rule "always put the parentheses around the
left operator" would get 50% right (not _that_ far from the actual
performance...) without having ANY beliefs about precedence at all.
Without being able to assume that the code seen by subjects is
close to an average measured over some code base it is not possible
to make any claims about experience vs performance.
Right. Given your figure 1, there seems to be no point in trying.
Given the significant amount of experience of the subjects I
feel comfortable making this assumption.
I have provided evidence that three different people could have many years
experience working in C and *still* have encountered quite different
operator profiles. Notably, bitwise operations might have been quite
common, or they might have been much rarer.
What tool did you use for the measurements? grep often gives
inaccurate values.
That's why I didn't use grep, but a C tokeniser.
You can download an early version of the
source of the measurement tools I used at:
www.knosof.co.uk/cbook/srccnt.tgz
Let me know if you have any problems.
Thank you, I am downloading that now.
> (a) Page 5 says that the subjects were told
> "THE TASK CONSISTS OF REMEMBERING THE VALUE OF THREE DIFFERENT
> VARIABLES AND RECALLING THESE VALUES LATER."
Subjects were asked to perform both tasks. The assignment problem
was certainly given greater prominence.
_Much_ greater prominence.
They were certainly asked to answer both kinds of problems. The
title of the experiment could certainly have misled them into thinking
that the () problem had low priority.
Great, we agree.
> We can go further. We are told on page 8 that
> "Experience shows that many developers are competetive and that
> accurately recalling the assignment information, after
parenthesizing
> the expression list, would be seen as the ideal performance to
aim for."
> That being so, the subjects would have had a strong incentive NOT to
> devote much of their processing capacity to accurately performing a
task
> that they had been led to believe was not the task they were being
tested
> on.
This is certainly a possibility and is listed in the validity threats.
Listing something amongst validity threats doesn't make the problem
actually go away. It just says "my results may not mean anything, but
I spotted that before you did." We are not talking about an unlikely
possibility listed for completeness; we are talking about something
that is _very likely_.
> Bear in mind that the subjects were given 200 parenthesis problems to
> solve in 20 minutes PLUS what they thought was the real task.
Subjects
They were given 200 problems because I wanted to impress on them that
they could not possibly finish them all and might as well go at their
'normal' speed. This issue is discussed in part 2. They were told
that they were not expected to answer 200 problems (two people actually
did).
You can *tell* people to work at their normal speed, but *did* they?
Amongst other things, this is not something that developers normally
do, so they don't *have* a "normal speed" for this kind of problem.
When you give people more problems to solve than you expect them to have
time for, we can confidently expect a higher error rate due to the
experimental situation, not to the skill being tested.
Based on other human error data, we'd expect a 5% error rate _without_
the time stress. Does anyone know where to find something about how
error rate depends on time stress?
My priority at the moment is to rerun giving subjects the opportunity
to answer "Don't know, would look it up".
That is an excellent idea that should give some interseting answers.
Answers to the rest of the response tomorrow.
By the way, there is a variant of the Bradley-Terry model discussed in
a paper in Biometrics in 1984 that can cope with the possibility of the
judges having difference pi values.
----------------------------------------------------------------------
PPIG Discuss List (discuss@ppig.org)
Discuss admin: http://limitlessmail.net/mailman/listinfo/discuss
Announce admin: http://limitlessmail.net/mailman/listinfo/announce
PPIG Discuss archive: http://www.mail-archive.com/discuss%40ppig.org/
