Re: [Tutor] bytecode primer, and avoiding a monster download
On 27 May 2013 16:20, Alan Gauld alan.ga...@btinternet.com wrote: To me, the bytecodes are the literal hex values corresponding to the Python assembler statements. Are you sure you need the bytecodes? You can use the standard library to generate the assembler listing from the Python code. Ah good, how do I do that? What's the module? Thanks. Jim ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] bytecode primer, and avoiding a monster download
On 27 May 2013 16:32, Steven D'Aprano st...@pearwood.info wrote: On 28/05/13 06:01, Jim Mooney wrote: Shall we guess what package that is? I love guessing games! Ah, who am I kidding. No I don't. Well, I would hate to keep you guessing ;') It's called decompyle - pip couldn't find it, though, and it requires either Msoft Visual or python(x,y) install, so it's in abeyance. Jim ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] bytecode primer, and avoiding a monster download
On 05/28/2013 07:02 AM, Jim Mooney wrote: On 27 May 2013 16:20, Alan Gauld alan.ga...@btinternet.com wrote: To me, the bytecodes are the literal hex values corresponding to the Python assembler statements. Are you sure you need the bytecodes? You can use the standard library to generate the assembler listing from the Python code. Ah good, how do I do that? What's the module? Thanks. Alan and Devin already gave more specifics, but to repeat, import dis dis.dis(myfunction) will disassemble one function. That's not all that's in the byte-code file, but this is 98% of what you probably want out of it. And you can do it in the debugger with just the standard library. -- DaveA ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] bytecode primer, and avoiding a monster download
On Tue, May 28, 2013 at 7:18 AM, Dave Angel da...@davea.name wrote: dis.dis(myfunction) will disassemble one function. That's not all that's in the byte-code file, but this is 98% of what you probably want out of it. And you can do it in the debugger with just the standard library. The argument for dis.dis() can be a module, class, function or code object. It disassembles all the top-level code objects that it finds, but it doesn't recursively disassemble code objects that are in the co_consts. I'm not sure what Dave means by 'byte-code file'. A .pyc? That's a marshaled code object with a small header that has a magic number and a timestamp. Here's an example of reading the pyc for the dis module itself: import dis import marshal import struct import datetime # magic number for Python 2.7 MAGIC27 = 62211 | (ord('\r') 16) | (ord('\n') 24) pyc = open(dis.__file__, 'rb') # dis.pyc hdr = pyc.read(8) magic, tstamp = struct.unpack('ll', hdr) tstamp = datetime.datetime.fromtimestamp(tstamp) # the rest of the file is the code object code = marshal.load(pyc) magic == MAGIC27 True tstamp datetime.datetime(2013, 1, 2, 12, 45, 58) code.co_consts[0] 'Disassembler of Python byte code into mnemonics.' The code object's co_consts tuple also has the code objects for the defined functions, plus the anonymous functions that build class objects. The latter are subsequently discarded, as is the .pyc code object itself after the module is imported/executed. There's no point in keeping it around. My previous post is a light intro to instantiating code and function objects. I think the arguments are mostly self-explanatory -- except for co_lnotab, co_flags, and closure support (co_cellvars, co_freevars, func_closure). I assembled the bytecode with the help of opcode.opmap, to make it more readable. But to be clear, CPython bytecode is simply a byte string stored in the co_code attribute. Disassembling the bytecode nicely with source line numbers requires co_lnotab. While I did reference the text file that explains co_lnotab, I neglected to provide the following link: http://hg.python.org/cpython/file/687295c6c8f2/Objects/lnotab_notes.txt co_flags indicates various aspects of how the bytecode was compiled (e.g. optimized to use fastlocals). It's inherited from the current context when you use exec or eval. compile() can disable this via the argument dont_inherit. The code for a class body or a function requires a new local namespace (CO_NEWLOCALS). For a function, locals is also optimized (CO_OPTIMIZED) to use the fastlocals array instead of a dict. On the other hand, the code that creates a module is evaluated with locals and globals set to the same namespace, so it won't have the CO_NEWLOCALS flag. Including the metadata that there are no free variables (CO_NOFREE) can make a simple function call more efficient, but only if there are no default arguments or keyword arguments in the call. Refer to fast_function() in Python/ceval.c. http://hg.python.org/cpython/file/ab05e7dd2788/Python/ceval.c#l4060 Here are all of the flags for code objects in 2.7: CO_OPTIMIZED0x01 CO_NEWLOCALS0x02 CO_VARARGS 0x04 CO_VARKEYWORDS 0x08 CO_NESTED 0x10 CO_GENERATOR0x20 CO_NOFREE 0x40 /* __future__ imports */ CO_FUTURE_DIVISION 0x02000 CO_FUTURE_ABSOLUTE_IMPORT 0x04000 CO_FUTURE_WITH_STATEMENT 0x08000 CO_FUTURE_PRINT_FUNCTION 0x1 CO_FUTURE_UNICODE_LITERALS 0x2 For a high-level view on scoping, read the section on the execution model in the language reference: http://docs.python.org/2/reference/executionmodel Subsequently, if you want, we can talk about how this is implemented in the VM, and especially with respect to closures, cellvars, freevars, and the opcodes LOAD_DEREF, STORE_DEREF, and MAKE_CLOSURE. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] bytecode primer, and avoiding a monster download
On 28 May 2013 17:48, eryksun eryk...@gmail.com wrote: On Tue, May 28, 2013 at 7:18 AM, Dave Angel da...@davea.name wrote: dis.dis(myfunction) will disassemble one function. That's not all that's in the byte-code file, but this is 98% of what you probably want out of it. And you can do it in the debugger with just the standard library. The argument for dis.dis() can be a module, class, function or code object. It disassembles all the top-level code objects that it finds, but it doesn't recursively disassemble code objects that are in the co_consts. What do you mean by this? I tried passing a test module into dis.dis and nothing happens: $ python Python 2.7.5 (default, May 15 2013, 22:43:36) [MSC v.1500 32 bit (Intel)] on win32 Type help, copyright, credits or license for more information. import tmp tmp.a 1 import dis dis.dis(tmp) print(dis.dis(tmp)) None This module contains no functions but it does contain lines of code and the interpreter will turn those into bytecode somewhere. I think that's what Dave meant. Oscar ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] bytecode primer, and avoiding a monster download
On Tue, May 28, 2013 at 1:12 PM, Oscar Benjamin oscar.j.benja...@gmail.com wrote: On 28 May 2013 17:48, eryksun eryk...@gmail.com wrote: The argument for dis.dis() can be a module, class, function or code object. It disassembles all the top-level code objects that it finds, but it doesn't recursively disassemble code objects that are in the co_consts. What do you mean by this? I tried passing a test module into dis.dis and nothing happens: Obviously dis() has nothing to do if the module or class namespace has no code objects to disassemble. import tmp tmp.a 1 import dis dis.dis(tmp) print(dis.dis(tmp)) None This module contains no functions but it does contain lines of code and the interpreter will turn those into bytecode somewhere. I think that's what Dave meant. Did you read the rest of my post? I went through an example of loading the cached code object from a .pyc. But we can just compile one instead: code = compile(r''' ... 'module docstring' ... class Test(object): ... 'class docstring' ... def spam(self): ... pass ... ''', 'test', 'exec') dis.dis(code) 2 0 LOAD_CONST 0 ('module docstring') 3 STORE_NAME 0 (__doc__) 3 6 LOAD_CONST 1 ('Test') 9 LOAD_NAME1 (object) 12 BUILD_TUPLE 1 15 LOAD_CONST 2 (code object Test) 18 MAKE_FUNCTION0 21 CALL_FUNCTION0 24 BUILD_CLASS 25 STORE_NAME 2 (Test) 28 LOAD_CONST 3 (None) 31 RETURN_VALUE dis.dis(code.co_consts[2]) 3 0 LOAD_NAME0 (__name__) 3 STORE_NAME 1 (__module__) 4 6 LOAD_CONST 0 ('class docstring') 9 STORE_NAME 2 (__doc__) 5 12 LOAD_CONST 1 (code object spam) 15 MAKE_FUNCTION0 18 STORE_NAME 3 (spam) 21 LOAD_LOCALS 22 RETURN_VALUE Let's exec this in a new module and run dis() on it: testmod = imp.new_module('testmod') exec code in vars(testmod) dis.dis(testmod) Disassembly of Test: Disassembly of spam: 6 0 LOAD_CONST 0 (None) 3 RETURN_VALUE Admittedly it's not very interesting since spam() does nothing. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] bytecode primer, and avoiding a monster download
On 28 May 2013 18:24, eryksun eryk...@gmail.com wrote: On Tue, May 28, 2013 at 1:12 PM, Oscar Benjamin oscar.j.benja...@gmail.com wrote: On 28 May 2013 17:48, eryksun eryk...@gmail.com wrote: The argument for dis.dis() can be a module, class, function or code object. It disassembles all the top-level code objects that it finds, but it doesn't recursively disassemble code objects that are in the co_consts. What do you mean by this? I tried passing a test module into dis.dis and nothing happens: Obviously dis() has nothing to do if the module or class namespace has no code objects to disassemble. Did you read the rest of my post? Yes. Although I'll admit that I didn't fully absorb all of it. :) I went through an example of loading the cached code object from a .pyc. But we can just compile one instead: Okay, got you. So the way to dis an ordinary module is to load the .py file as a string first: dis.dis(compile(open('tmp.py').read(), 'tmp', 'exec')) 2 0 LOAD_CONST 0 (1) 3 STORE_NAME 0 (a) 3 6 LOAD_CONST 1 (2) 9 STORE_NAME 1 (b) 4 12 LOAD_NAME0 (a) 15 LOAD_NAME1 (b) 18 BINARY_ADD 19 STORE_NAME 2 (c) 22 LOAD_CONST 2 (None) 25 RETURN_VALUE So what happens to the code object for the top-level code in the module itself when it is imported in the normal way? Is it just discarded once import is complete? Is it temporarily accessible during import? Oscar ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] bytecode primer, and avoiding a monster download
On Tue, May 28, 2013 at 1:58 PM, Oscar Benjamin oscar.j.benja...@gmail.com wrote: So what happens to the code object for the top-level code in the module itself when it is imported in the normal way? Is it just discarded once import is complete? Is it temporarily accessible during import? Normally, nothing holds a reference to it, so it gets deallocated, as do the anonymous functions used to build classes. However, in CPython the current frame is accessible during import via sys._getframe(). The code object is the f_code attribute: code = compile(r''' ... import sys ... frame = sys._getframe() ... ''', 'test', 'exec') ns = {} exec code in ns dis.dis(ns['frame'].f_code) 2 0 LOAD_CONST 0 (0) 3 LOAD_CONST 1 (None) 6 IMPORT_NAME 0 (sys) 9 STORE_NAME 0 (sys) 3 12 LOAD_NAME0 (sys) 15 LOAD_ATTR1 (_getframe) 18 CALL_FUNCTION0 21 STORE_NAME 2 (frame) 24 LOAD_CONST 1 (None) 27 RETURN_VALUE ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] bytecode primer, and avoiding a monster download
On 28 May 2013 04:18, Dave Angel da...@davea.name wrote: On 05/28/2013 07:02 AM, Jim Mooney wrote: Alan and Devin already gave more specifics, but to repeat, import dis dis.dis(myfunction) will disassemble one function. I think authors miss a didactic opportunity by not using bytecode as a teaching tool now and then, since it's easily explained, at least for basic statements. Some concepts seem hard to get across in English but are obvious when you actually see what they do, step-by-step. Not true for everything for course, but for some things. I guess it has to do with sales. If someone new to any programming opens a Python book to buy and sees all these prettily laid-out Python statements that look almost like natural language, only more ordered, but then sees a page of bytecode, they'd probably drop the book and run ;') Jim ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] bytecode primer, and avoiding a monster download
On 28/05/13 21:15, Jim Mooney wrote: I think authors miss a didactic opportunity by not using bytecode as a teaching tool now and then, since it's easily explained, at least for basic statements. Assuming you mean the assembler statements then it may be true. Looking at Bytecode is just an exercise in pain for first year programming students... or a way to reverse engineer a binary file for which you don't have source :-( I guess it has to do with sales. If someone new to any programming opens a Python book to buy and sees all these prettily laid-out Python statements that look almost like natural language, only more ordered, but then sees a page of bytecode, they'd probably drop the book and run ;') It's not just sales, it's more that the majority of programmers (including professionals) simply have no idea how a computer does things internally. They don't think about programs that way they think of them in terms of the abstract calculation engine that a computer pretends to be. Seeing assembler would not help them at all. But if someone has any appreciation of the computer architecture, how CPU and memory are related then yes, assembler can clarify some things. But, especially in a language like Python, too much awareness of how the language works at the machine level can be counter productive because you start programming for the machine. And that's dangerous because the mechanisms are liable to change in a future version and your code stops doing what you think it does. Occasionally it is necessary but I've been using Python more or less exclusively for the last 12 years and I've only twice had to delve into the assembler (and maybe another 5 or 6 times out of curiosity). -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] bytecode primer, and avoiding a monster download
On 27 May 2013 21:01, Jim Mooney cybervigila...@gmail.com wrote: I was looking at the bytecode doc page as a break from the Lutz book, since I like Assembler-type code due to its total non-ambiguity, but the page doesn't say much. Is there a doc somewhere that corresponds some of the bytecode to Python source? I thought rot_1, 2, 3, and 4 looked useful, but it would take awhile to disassemble random programs to see what source they come from. What do you want these for? I've never needed to know what the interpreter's bytecodes are. Another question. I tried installing a package that back-compiles (in win 7), so I could see things that way, and got the error Unable to find vcvarsall.bat I don't know what you mean by back-compiles but I guess this package contains extension modules in C. To install that from source you will need a C compiler. The project may provide an installer with pre-compiled binaries for Windows (check on their download page). Looking around it said I needed to install Visual Studio (Express, of course ;'). I'm on a very slow connection, right now. Is there any alternative to downloading this monster 600 Mb ISO just to install a package, or am I doomed to getting Visual Studio? You can use MinGW instead of Visual Studio but you will need to apply the fix to cygwinccompiler.py described here: http://stackoverflow.com/questions/6034390/compiling-with-cython-and-mingw-produces-gcc-error-unrecognized-command-line-o Otherwise you could install Python(x, y). This will install Python, MinGW and a bunch of other useful packages in one go: https://code.google.com/p/pythonxy/wiki/Downloads?tm=2 Oscar ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] bytecode primer, and avoiding a monster download
Oscar Benjamin oscar.j.benja...@gmail.com What do you want these for? I've never needed to know what the interpreter's bytecodes are. I programmed A86 Assembler years ago, and just find the bytecode has a comfortable clarity as a learning tool, at least for me. When an author vaguely tried to explain that a Name in Python is not the Data, it confused me at first what he meant. But I could literally see the disjunction in bytecode. I'm sure it's not for everyone - just aging Assembler hackers ;') Thanks for the Python(x,y) tip. That's probably useful for other stuff, too. -- Jim Mooney Atlantis Anyone? http://www.pnas.org/content/early/2013/05/17/1301760110.abstract?sid=8313cec8-3ee0-47a0-9488-5bc33983b081 ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] bytecode primer, and avoiding a monster download
On 27/05/13 21:21, Jim Mooney wrote: Oscar Benjamin oscar.j.benja...@gmail.com What do you want these for? I've never needed to know what the interpreter's bytecodes are. I programmed A86 Assembler years ago, and just find the bytecode has a comfortable clarity as a learning tool, To me, the bytecodes are the literal hex values corresponding to the Python assembler statements. Are you sure you need the bytecodes? You can use the standard library to generate the assembler listing from the Python code. Like Oscar I'm not sure how the bytecodes would help? I'm an ex assembler hacker myself but I've never found a need for the Python bytecodes. Its not like you ever need to translate the binary into assembler (the usual reason for reading bytecodes) since you usually have the source and from that you can get the dis-assembly listing. If you don't have the source it probably means it's a C module and you need to look at the native machine code disassembly not the Python one. Can you give a use case of how you think you could use them? There may be another way to do what you want. -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] bytecode primer, and avoiding a monster download
On Mon, May 27, 2013 at 7:20 PM, Alan Gauld alan.ga...@btinternet.com wrote: Can you give a use case of how you think you could use them? There may be another way to do what you want. I can't speak for Jim, but I have used the dis module in the past to quickly check how python parsed an expression (e.g. not x is y). -- Devin ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] bytecode primer, and avoiding a monster download
On 28/05/13 06:01, Jim Mooney wrote: I was looking at the bytecode doc page as a break from the Lutz book, since I like Assembler-type code due to its total non-ambiguity, but the page doesn't say much. Is there a doc somewhere that corresponds some of the bytecode to Python source? I thought rot_1, 2, 3, and 4 looked useful, but it would take awhile to disassemble random programs to see what source they come from. Python byte-code is not officially documented, and is subject to change. But it should be relatively straight-forward to understand, the names are mostly self-explanatory. The main thing to remember is that the byte-code is reverse-polish stack-based, like Forth, Postscript, and most Hewlett-Packard scientific calculators. http://en.wikipedia.org/wiki/Stack-oriented_programming_language Another question. I tried installing a package that back-compiles (in win 7), so I could see things that way, and got the error Unable to find vcvarsall.bat Shall we guess what package that is? I love guessing games! Ah, who am I kidding. No I don't. -- Steven ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] bytecode primer, and avoiding a monster download
I can't speak for Jim, but I have used the dis module in the past to quickly check how python parsed an expression (e.g. not x is y). Yes, I've done that too. But that's not the bytecodes (at least not what I understand as bytecodes) that's the dis-assembly listing which is usually far more useful. And as you say the dis module provides that without any need for other tools. Alan G.___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] bytecode primer, and avoiding a monster download
On 28/05/2013 00:32, Steven D'Aprano wrote: On 28/05/13 06:01, Jim Mooney wrote: Another question. I tried installing a package that back-compiles (in win 7), so I could see things that way, and got the error Unable to find vcvarsall.bat Shall we guess what package that is? I love guessing games! Ah, who am I kidding. No I don't. Been there, seen it, done it, got the t-shirt http://old.nabble.com/Re%3A-Unable-to-install-PyWin32-with-Python-2.6.2-p25202902.html -- If you're using GoogleCrap™ please read this http://wiki.python.org/moin/GoogleGroupsPython. Mark Lawrence ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] bytecode primer, and avoiding a monster download
On 05/27/2013 07:56 PM, ALAN GAULD wrote: I can't speak for Jim, but I have used the dis module in the past to quickly check how python parsed an expression (e.g. not x is y). Yes, I've done that too. But that's not the bytecodes (at least not what I understand as bytecodes) that's the dis-assembly listing which is usually far more useful. And as you say the dis module provides that without any need for other tools. Certainly dis.dis() doesn't show the entire file, it shows only a single function. And it doesn't show the const-table, or the local-table. (Whatever they're called in byte-code-speak) But within the function it's probably got most all of what you would see in a byte code file, barring the hex values of the bytes themselves. Notice it shows you the offsets, so you know how many bytes each instruction takes. And when the instruction is more than one byte long, it shows the operands, in both decimal form and in mnemonic. When I first got my hands on the (very good) spec for the java byte code file, I wrote a disassembler for it. Seems to me the python one is very analogous to that one. Note, though that I have never tried to even look at the python one, even with a hex editor. So I could be all wet. -- DaveA ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] bytecode primer, and avoiding a monster download
On Mon, May 27, 2013 at 4:01 PM, Jim Mooney cybervigila...@gmail.com wrote: I was looking at the bytecode doc page as a break from the Lutz book, since I like Assembler-type code due to its total non-ambiguity, but the page doesn't say much. Is there a doc somewhere that corresponds some of the bytecode to Python source? I thought rot_1, 2, 3, and 4 looked useful, but it would take awhile to disassemble random programs to see what source they come from. Are you referring to the documentation of the dis module? http://docs.python.org/2/library/dis AFAIK, that's all the documentation there is. Heed the warning in bold that bytecode is an implementation detail of the target virtual machine. For example, Jython creates a .class file containing JVM bytecode, and its code objects don't even have a co_code attribute... I once manually instantiated a CPython 2.x code object to create a very simple function with a default argument. Here was my first and only attempt, FWIW: from opcode import opmap from types import FunctionType, CodeType from inspect import CO_OPTIMIZED, CO_NEWLOCALS, CO_NOFREE OP = type('OP', (), opmap) def foo1(x='spam'): print x foo2 = FunctionType( CodeType(# func_code 1, # co_argcount 1, # co_nlocals 1, # co_stacksize CO_OPTIMIZED | # co_flags CO_NEWLOCALS | CO_NOFREE, str(bytearray([# co_code OP.LOAD_FAST, 0, 0, # stack.push(fastlocals[0]) # x is fastlocals[0] OP.PRINT_ITEM, # print stack.pop() OP.PRINT_NEWLINE, OP.LOAD_CONST, 0, 0, # stack.push(co_consts[0]) # co_consts[0] is None OP.RETURN_VALUE, # return stack.pop() ])), (None,), # co_consts (),# co_names ('x',),# co_varnames __file__, # co_filename 'foo2',# co_name 10,# co_firstlineno '\x00\x01',# co_lnotab, # see: Objects/lnotab_notes.txt (),# co_freevars (),# co_cellvars ), globals(), # func_globals 'foo2', # func_name ('spam',), # func_defaults None,# func_closure ) For example: foo2() spam foo2('eggs') eggs There's an assembler package on PyPI that does this for real, if this sort of thing interests you: https://pypi.python.org/pypi/BytecodeAssembler As to the VM's ROT operations, I'm not sure what you mean by seeing what source they come from. They simply manipulate the top 2, 3, or 4 items on the frame's stack: ROT_TWO() Swaps the two top-most stack items. ROT_THREE() Lifts second and third stack item one position up, moves top down to position three. ROT_FOUR() Lifts second, third and forth stack item one position up, moves top down to position four. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor