I'm not sure you read what I said very carefully { about use case 3, too,
but that use case is admittedly pretty lame :-/ }.
Greg Ewing wrote:
>There are reasons for that. In theory, Pyrex should never
>generate invalid C code, so
I never mentioned compiler errors, but only warnings. Obviously, errors are
more important for the code generator writer, as are file & line numbers in
the generated code. Yet, some warnings come from data flow analysis in gcc.
Yes, you could reinvent such analysis. It sure seems easier to emit #line
and let gcc (or whatever compiler) tell users and try to keep warnings
produced by the generated code to a minimum. They're already very thin.
I've mostly only gotten incompletely filled-in struct initializer warnings.
>if there are any C compilation errors, I usually want to
>see where they came from in the C file so that I can figure
>out what went wrong.
I noted that more powerful users [ system authors obviously included :) ]
might prefer generated lines but are also more empowered to flip the switch
to get whichever they want. Of course, I'd just happy to be able to turn
it on any old way, (and don't really expect to be able to any time soon).
Just trying to motivate the best default and motivate the behavior...
>there's no way of selectively reporting .pyx file positions just for those
Well, don't be selective. :) Just emit a #line for everything and let
the whole feature be turned on or off.
Easy or not to search through my pyx source, it still seems better to have
a language system refer to as high-level source coordinates as possible.
Yacc and other C-generating preprocessors or C-targeting compilers do this.
File, line, and maybe function coordinates are a tenuous but useful thread
connecting the highest level to the lowest in language toolchains.
>There's also the problem that some of the code Pyrex
>generates isn't clearly attributable to any particular
>place in a .pxd or .pyx file.
The same is true of assembly from C or C from Yacc as mentioned above.
A more significant issue than missing values is that with certain kinds
of optimization (e.g. common subexpression elimination) you can get a
one->many mapping of what is generated to the source (since multiple
source addresses are all related to one generated bit). I don't know if
any sorts of transforms like that happen currently in going from Cython
But, consider -- minor imperfections (loose ends and knots in the thread,
if you will) don't seem to degrade the value much in practice (at least
mine). If you have the thread 99.9% of the times you need it, and it often
saves 5X or who knows how much human time in addressing some issue at the
pyx source to be told where something is based on the source you edit vs.
generated code..., well, it seems a pretty clear win. { To NOT be a win,
it would have to be thousands of times harder to deal with things in the
0.1% of the time you don't have the thread. It's not clearly any harder
at all. Maybe just strip it with sed -i -e'/^#file/d' -e'/^#line/d' foo.c
or cython --no-coords or something. }
I'm not saying it's not "work", and other judgements are of course in play
as to why it's not present either historically or on anyone's task list.
Cython's current ability to emit the source lines with <<<<< pointers makes
me strongly suspect that it's not much work. Parsing upstream/pre-processed
directives like #file and #line is likely more work, and more valuable as a
pre-processor toolchain becomes done as per Stefan's ambitious description.
I argue here more about "shoulds" and reasons for things, only because you put
it in those terms. This kind of feature is the sort that language designers
often see less value in because they can read what is generated easily enough.
More external, casual and/or novice language users really do appreciate it.
If the point of Cython is to NOT have to learn and use and deal with all the
guts of the Python C API, then the better integrated the tools are, the better
the user experience.
E.g., I'm pretty sure from using it that Cython --annotate to generate HTML of
what is most "Pythonic" is pretty heuristic and approximate. Yet, it's still
useful and interesting. Yes, *I'm* fine with reading generated code to find
out what I need, but I've been using Python 16 years, seen Great Renamings in
the C API come & go and I'm the sort to disassemble his code to see what the
C compiler is up to. There is a LOT of Python C API noise that can be fore-
boding to less intrepid users.
Higher level/propagated source coords are just one more thing in the vein of
making less necessary to look at the generated code. Source-level debugging
in gdb with a "backtrace" that understood PyObjectCall and the ability to
"print " Cython expressions would be a dream I barely dare utter (and also
likely a mountain of work).....HOWEVER...
Robert Bradshaw wrote:
>> Source coordinate propagation is a good thing. I actually have a
>> little profiling toolkit that would be able to tell me which line
>> numbers of pyx were my performance problems, if only the coordinates
>> were propagated.
>That sounds very cool. I would love to have something like that.
.....if you give me source coordinate propagation, then I can at least give
people I work with (or others) source-level profiling with almost no effort,
at least on Linux.
http://pdos.csail.mit.edu/cb/pct
All you do is make sure -g is on in all your CFLAGS and then you can just
go "PCT_FMT=line% profile python myEntryPoint". Code in modules that Python
dlopen()s is tracked on Linux as it is used. { All you need to do is re-read
/proc/PID/maps whenever an unknown address is found, and record where in
virutal memory the .so is; Wrapping dlopen() with LD_PRELOAD would be a
more direct & efficient, but invasive/tricky approach }.
I believe that if you emit #file and #line correctly that the rest of the
toolchain including things like "addr2line" will just work. I've profiled
C, C++, Java/gcj, Fortran, and even gdc, the GNU D compiler, all with the
gnu toolchain with no trouble and yielding pretty interpretable results.
In the case of Python loading modules from Cython and SciPy/NumPy you can
get pretty interestingly mixed multi-language profiling reports, actually,
with C API things, my own Cython code, and even Fortran lib calls in there.
At the moment, I just get by with function-level (with mangled names (or
not with 'public')) profiling of Cython code, or line-level reports on the
generated C.
That is, again, kind of fine for me, but it would be nicer to be able to
give a little more to others. A line-level report would be complementary
to cython --annotate. The former would tell you which regions get run a lot
while the latter which are "Pythonic". The combination of both "Pythonic"
and used a lot is almost exactly what you should be paying attention to in
terms of missing cdef's/easy speed-up opportunities while not wasting time
optimizing code that doesn't get run a lot.
Why, it wouldn't be too hard to make a source-level HTML annotater for my
profiler and then have a browser where you looked at a sizeable Cython
program in the two mark-ups side-by-side, one for Pythonic-ness and one
for CPU-time-hogging. There's already an annotated disassembly mark-up,
but without HTML. Another possibility would be to try to combine or
weight the scores into an overall "pay attention to me" score.
All the best!
cb
_______________________________________________
Cython-dev mailing list
[email protected]
http://codespeak.net/mailman/listinfo/cython-dev
