Re: [Pharo-project] [Preview] Branch test coverage analysis tool for Pharo 3.0 :)
I just implemented something to this effect for our project at Cadence. It works by - an assembler/disassebler that can convert a method into a sequence of messages, one for each of its bytecodes and back - using the disassembler to obtain labels which occur at the start of each basic block - replacing the first bytecode in every basic block with an illegal bytecode, remembering the pc and its correct bytecode in the method properties - implementing a method on MethodContext which receives the unknowBytecode message sent by the VM when it encounters an illegal bytecode - the method replaces the illegal bytecode with the correct bytecode, removing the entry from properties, and continues The JIT refuses to compile methods that contain illegal bytecodes so this runs at interpreter speed until a method no longer contains illegal bytecodes (has been completely covered). But the illegal bytecode is executed at most once for each basic block, since it is replaced immediately it is executed. The information remaining in properties is that of the basic blocks that have not been executed. In the Squeak/Pharo bytecode set there are three unknown bytecodes, 126, 127 139. I have a Monticello package and tests if you're interested. Yes please. I am interested! Alexandre -- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.
Re: [Pharo-project] [Preview] Branch test coverage analysis tool for Pharo 3.0 :)
Le 06/04/2013 12:00, Clément Bera a écrit : Actually, it is quite different than Jejak. Implementation wise, Jejak used Opal so I guess it relies on bytecode modification to do its analysis where my tool relies on ast annotations. Jejak has allways worked by AST modifications. It's just that, at a point, it recompiled the code with Opal to try to get the same behavior that its original tracer under visualworks had, i.e. the source code modifications are invisible to the user... It also played with the bytecode because this had huge performance impacts under VisualWorks, bringing traces with a very low overhead (sadly, the same trick doesn't work with Pharo :(). This implies some difference in features. It seems that Jejak record all calls, all values and all assignments in a method (which is byte code level feature), whereas I record for each ast node if it was interpreted or not (which is ast level feature). I guess there would be a little difference in the use cases. The current version of Jejak works in the following way : 1- get AST, store original source code 2- search and replace in AST for what you want to trace 3- generate source with traces and compile 4- Wait for it to run... collect trace events on the way. 5- revert to original sources, recompile It's a bit faster than interpreting the AST; it's a bit more invasive as well, but you can change it to record less events if you want. It also works for tracing difficult things such as core Morphic, KeyMapping and Display methods. Thierry 2013/4/6 Frank Shearar frank.shea...@gmail.com mailto:frank.shea...@gmail.com On 6 April 2013 07:47, Stéphane Ducasse stephane.duca...@inria.fr mailto:stephane.duca...@inria.fr wrote: Clement built in a couple of hours the following nice branch analysecoverage tools :) It is dead slow but cool. By branch coverage tool do you mean Jejak? frank -- Clément Béra Mate Virtual Machine Engineer Bâtiment B 40, avenue Halley 59650 /Villeneuve d'Ascq/ -- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 / 83 95
Re: [Pharo-project] [Preview] Branch test coverage analysis tool for Pharo 3.0 :)
What I had in mind, is something like: -=-=-=-=-=-=-=-=-=-= myMethodWithBranch self foo. v ifTrue: [ ... ] ifFalse: [ ... ] -=-=-=-=-=-=-=-=-=-= -=-=-=-=-=-=-=-=-=-= myMethodWithBranch self foo. v ifTrue: [ self checkPoint: 1. ... ] ifFalse: [ self checkPoint: 2. ... ]. self checkPoint: 3. -=-=-=-=-=-=-=-=-=-= All these checkpoint will be inserted by manipulating the bytecode. You can then optimize a bit by removing checkpoints that have been reached. The method #checkPoint: could then use some attributes to the CompiledMethod. This is a common technique when (i) the VM does not give you what you need and (ii) when you do not want to change the VM. This is something I wanted to do for year in Hapao... Alexandre On Apr 7, 2013, at 3:21 AM, Clément Bera bera.clem...@gmail.com wrote: I don't know. The main problem is that the AST-interpreter is not fast enough. The best solution for speed would be to improve it as they do in this paper : http://dl.acm.org/citation.cfm?id=2384587 (their ast-interpreter is faster than a byte code interpreter). Another solution would be to execute some method with the vm and other one with the ast-interpreter, but it is quite some work. 2013/4/7 Alexandre Bergel alexandre.ber...@me.com Would it be possible to annotate the method with some checkpoints within the method? This should be fast enough I guess. Alexandre On Apr 6, 2013, at 10:35 AM, Clément Bera bera.clem...@gmail.com wrote: This is exactly that. The tree coverage tool is a subclass of AST-interpreter, which adding to the interpretation marks the nodes. And then I use this marking to color the source code. And I use as you said RB ast. This is why I implemented it in a few hours, I had already an AST-interpreter, so I've just created a quick subclass and a spec UI. 2013/4/6 Frank Shearar frank.shea...@gmail.com So if you don't mind me being very loose with terminology, you interpret (RB) ASTs, and the act of interpreting marks the nodes in these ASTs, which you can then use to colour source? frank On 6 April 2013 11:00, Clément Bera bera.clem...@gmail.com wrote: Actually, it is quite different than Jejak. Implementation wise, Jejak used Opal so I guess it relies on bytecode modification to do its analysis where my tool relies on ast annotations. This implies some difference in features. It seems that Jejak record all calls, all values and all assignments in a method (which is byte code level feature), whereas I record for each ast node if it was interpreted or not (which is ast level feature). I guess there would be a little difference in the use cases. 2013/4/6 Frank Shearar frank.shea...@gmail.com On 6 April 2013 07:47, Stéphane Ducasse stephane.duca...@inria.fr wrote: Clement built in a couple of hours the following nice branch analysecoverage tools :) It is dead slow but cool. By branch coverage tool do you mean Jejak? frank -- Clément Béra Mate Virtual Machine Engineer Bâtiment B 40, avenue Halley 59650 Villeneuve d'Ascq -- Clément Béra Mate Virtual Machine Engineer Bâtiment B 40, avenue Halley 59650 Villeneuve d'Ascq -- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;. -- Clément Béra Mate Virtual Machine Engineer Bâtiment B 40, avenue Halley 59650 Villeneuve d'Ascq -- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.
Re: [Pharo-project] [Preview] Branch test coverage analysis tool for Pharo 3.0 :)
Well my stuff is much more basic I just wanted to implement a prototype to show what you could do with the AST-interpreter. Basically I overrided 'interpretNode:' of the AST-interpreter by : interpretNode: aNode aNode mark. super interpretNode: aNode And then at the end I have the result expected. 2013/4/8 Alexandre Bergel alexandre.ber...@me.com What I had in mind, is something like: -=-=-=-=-=-=-=-=-=-= myMethodWithBranch self foo. v ifTrue: [ ... ] ifFalse: [ ... ] -=-=-=-=-=-=-=-=-=-= -=-=-=-=-=-=-=-=-=-= myMethodWithBranch self foo. v ifTrue: [ *self checkPoint: 1.* ... ] ifFalse: [ *self checkPoint: 2.* ... ]. *self checkPoint: 3.* -=-=-=-=-=-=-=-=-=-= All these checkpoint will be inserted by manipulating the bytecode. You can then optimize a bit by removing checkpoints that have been reached. The method #checkPoint: could then use some attributes to the CompiledMethod. This is a common technique when (i) the VM does not give you what you need and (ii) when you do not want to change the VM. This is something I wanted to do for year in Hapao... Alexandre On Apr 7, 2013, at 3:21 AM, Clément Bera bera.clem...@gmail.com wrote: I don't know. The main problem is that the AST-interpreter is not fast enough. The best solution for speed would be to improve it as they do in this paper : http://dl.acm.org/citation.cfm?id=2384587 (their ast-interpreter is faster than a byte code interpreter). Another solution would be to execute some method with the vm and other one with the ast-interpreter, but it is quite some work. 2013/4/7 Alexandre Bergel alexandre.ber...@me.com Would it be possible to annotate the method with some checkpoints within the method? This should be fast enough I guess. Alexandre On Apr 6, 2013, at 10:35 AM, Clément Bera bera.clem...@gmail.com wrote: This is exactly that. The tree coverage tool is a subclass of AST-interpreter, which adding to the interpretation marks the nodes. And then I use this marking to color the source code. And I use as you said RB ast. This is why I implemented it in a few hours, I had already an AST-interpreter, so I've just created a quick subclass and a spec UI. 2013/4/6 Frank Shearar frank.shea...@gmail.com So if you don't mind me being very loose with terminology, you interpret (RB) ASTs, and the act of interpreting marks the nodes in these ASTs, which you can then use to colour source? frank On 6 April 2013 11:00, Clément Bera bera.clem...@gmail.com wrote: Actually, it is quite different than Jejak. Implementation wise, Jejak used Opal so I guess it relies on bytecode modification to do its analysis where my tool relies on ast annotations. This implies some difference in features. It seems that Jejak record all calls, all values and all assignments in a method (which is byte code level feature), whereas I record for each ast node if it was interpreted or not (which is ast level feature). I guess there would be a little difference in the use cases. 2013/4/6 Frank Shearar frank.shea...@gmail.com On 6 April 2013 07:47, Stéphane Ducasse stephane.duca...@inria.fr wrote: Clement built in a couple of hours the following nice branch analysecoverage tools :) It is dead slow but cool. By branch coverage tool do you mean Jejak? frank -- Clément Béra Mate Virtual Machine Engineer Bâtiment B 40, avenue Halley 59650 Villeneuve d'Ascq -- Clément Béra Mate Virtual Machine Engineer Bâtiment B 40, avenue Halley 59650 Villeneuve d'Ascq -- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;. -- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;. -- Clément Béra Mate Virtual Machine Engineer Bâtiment B 40, avenue Halley 59650 *Villeneuve d'Ascq*
Re: [Pharo-project] [Preview] Branch test coverage analysis tool for Pharo 3.0 :)
On Mon, Apr 8, 2013 at 9:32 AM, Alexandre Bergel alexandre.ber...@me.comwrote: What I had in mind, is something like: -=-=-=-=-=-=-=-=-=-= myMethodWithBranch self foo. v ifTrue: [ ... ] ifFalse: [ ... ] -=-=-=-=-=-=-=-=-=-= -=-=-=-=-=-=-=-=-=-= myMethodWithBranch self foo. v ifTrue: [ *self checkPoint: 1.* ... ] ifFalse: [ *self checkPoint: 2.* ... ]. *self checkPoint: 3.* -=-=-=-=-=-=-=-=-=-= All these checkpoint will be inserted by manipulating the bytecode. You can then optimize a bit by removing checkpoints that have been reached. The method #checkPoint: could then use some attributes to the CompiledMethod. This is a common technique when (i) the VM does not give you what you need and (ii) when you do not want to change the VM. I just implemented something to this effect for our project at Cadence. It works by - an assembler/disassebler that can convert a method into a sequence of messages, one for each of its bytecodes and back - using the disassembler to obtain labels which occur at the start of each basic block - replacing the first bytecode in every basic block with an illegal bytecode, remembering the pc and its correct bytecode in the method properties - implementing a method on MethodContext which receives the unknowBytecode message sent by the VM when it encounters an illegal bytecode - the method replaces the illegal bytecode with the correct bytecode, removing the entry from properties, and continues The JIT refuses to compile methods that contain illegal bytecodes so this runs at interpreter speed until a method no longer contains illegal bytecodes (has been completely covered). But the illegal bytecode is executed at most once for each basic block, since it is replaced immediately it is executed. The information remaining in properties is that of the basic blocks that have not been executed. In the Squeak/Pharo bytecode set there are three unknown bytecodes, 126, 127 139. I have a Monticello package and tests if you're interested. This is something I wanted to do for year in Hapao... Alexandre On Apr 7, 2013, at 3:21 AM, Clément Bera bera.clem...@gmail.com wrote: I don't know. The main problem is that the AST-interpreter is not fast enough. The best solution for speed would be to improve it as they do in this paper : http://dl.acm.org/citation.cfm?id=2384587 (their ast-interpreter is faster than a byte code interpreter). Another solution would be to execute some method with the vm and other one with the ast-interpreter, but it is quite some work. 2013/4/7 Alexandre Bergel alexandre.ber...@me.com Would it be possible to annotate the method with some checkpoints within the method? This should be fast enough I guess. Alexandre On Apr 6, 2013, at 10:35 AM, Clément Bera bera.clem...@gmail.com wrote: This is exactly that. The tree coverage tool is a subclass of AST-interpreter, which adding to the interpretation marks the nodes. And then I use this marking to color the source code. And I use as you said RB ast. This is why I implemented it in a few hours, I had already an AST-interpreter, so I've just created a quick subclass and a spec UI. 2013/4/6 Frank Shearar frank.shea...@gmail.com So if you don't mind me being very loose with terminology, you interpret (RB) ASTs, and the act of interpreting marks the nodes in these ASTs, which you can then use to colour source? frank On 6 April 2013 11:00, Clément Bera bera.clem...@gmail.com wrote: Actually, it is quite different than Jejak. Implementation wise, Jejak used Opal so I guess it relies on bytecode modification to do its analysis where my tool relies on ast annotations. This implies some difference in features. It seems that Jejak record all calls, all values and all assignments in a method (which is byte code level feature), whereas I record for each ast node if it was interpreted or not (which is ast level feature). I guess there would be a little difference in the use cases. 2013/4/6 Frank Shearar frank.shea...@gmail.com On 6 April 2013 07:47, Stéphane Ducasse stephane.duca...@inria.fr wrote: Clement built in a couple of hours the following nice branch analysecoverage tools :) It is dead slow but cool. By branch coverage tool do you mean Jejak? frank -- Clément Béra Mate Virtual Machine Engineer Bâtiment B 40, avenue Halley 59650 Villeneuve d'Ascq -- Clément Béra Mate Virtual Machine Engineer Bâtiment B 40, avenue Halley 59650 Villeneuve d'Ascq -- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;. -- Clément Béra Mate Virtual Machine Engineer Bâtiment B 40, avenue Halley 59650 Villeneuve d'Ascq -- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu
Re: [Pharo-project] [Preview] Branch test coverage analysis tool for Pharo 3.0 :)
I don't know. The main problem is that the AST-interpreter is not fast enough. The best solution for speed would be to improve it as they do in this paper : http://dl.acm.org/citation.cfm?id=2384587 (their ast-interpreter is faster than a byte code interpreter). Another solution would be to execute some method with the vm and other one with the ast-interpreter, but it is quite some work. 2013/4/7 Alexandre Bergel alexandre.ber...@me.com Would it be possible to annotate the method with some checkpoints within the method? This should be fast enough I guess. Alexandre On Apr 6, 2013, at 10:35 AM, Clément Bera bera.clem...@gmail.com wrote: This is exactly that. The tree coverage tool is a subclass of AST-interpreter, which adding to the interpretation marks the nodes. And then I use this marking to color the source code. And I use as you said RB ast. This is why I implemented it in a few hours, I had already an AST-interpreter, so I've just created a quick subclass and a spec UI. 2013/4/6 Frank Shearar frank.shea...@gmail.com So if you don't mind me being very loose with terminology, you interpret (RB) ASTs, and the act of interpreting marks the nodes in these ASTs, which you can then use to colour source? frank On 6 April 2013 11:00, Clément Bera bera.clem...@gmail.com wrote: Actually, it is quite different than Jejak. Implementation wise, Jejak used Opal so I guess it relies on bytecode modification to do its analysis where my tool relies on ast annotations. This implies some difference in features. It seems that Jejak record all calls, all values and all assignments in a method (which is byte code level feature), whereas I record for each ast node if it was interpreted or not (which is ast level feature). I guess there would be a little difference in the use cases. 2013/4/6 Frank Shearar frank.shea...@gmail.com On 6 April 2013 07:47, Stéphane Ducasse stephane.duca...@inria.fr wrote: Clement built in a couple of hours the following nice branch analysecoverage tools :) It is dead slow but cool. By branch coverage tool do you mean Jejak? frank -- Clément Béra Mate Virtual Machine Engineer Bâtiment B 40, avenue Halley 59650 Villeneuve d'Ascq -- Clément Béra Mate Virtual Machine Engineer Bâtiment B 40, avenue Halley 59650 Villeneuve d'Ascq -- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;. -- Clément Béra Mate Virtual Machine Engineer Bâtiment B 40, avenue Halley 59650 *Villeneuve d'Ascq*
Re: [Pharo-project] [Preview] Branch test coverage analysis tool for Pharo 3.0 :)
cool work! On 2013-04-06, at 08:47, Stéphane Ducasse stephane.duca...@inria.fr wrote: Clement built in a couple of hours the following nice branch analysecoverage tools :) It is dead slow but cool. With the AST navigation Gisela just started to work on we will have a really cool system. Stef image.png
Re: [Pharo-project] [Preview] Branch test coverage analysis tool for Pharo 3.0 :)
On 6 April 2013 07:47, Stéphane Ducasse stephane.duca...@inria.fr wrote: Clement built in a couple of hours the following nice branch analysecoverage tools :) It is dead slow but cool. By branch coverage tool do you mean Jejak? frank
Re: [Pharo-project] [Preview] Branch test coverage analysis tool for Pharo 3.0 :)
Actually, it is quite different than Jejak. Implementation wise, Jejak used Opal so I guess it relies on bytecode modification to do its analysis where my tool relies on ast annotations. This implies some difference in features. It seems that Jejak record all calls, all values and all assignments in a method (which is byte code level feature), whereas I record for each ast node if it was interpreted or not (which is ast level feature). I guess there would be a little difference in the use cases. 2013/4/6 Frank Shearar frank.shea...@gmail.com On 6 April 2013 07:47, Stéphane Ducasse stephane.duca...@inria.fr wrote: Clement built in a couple of hours the following nice branch analysecoverage tools :) It is dead slow but cool. By branch coverage tool do you mean Jejak? frank -- Clément Béra Mate Virtual Machine Engineer Bâtiment B 40, avenue Halley 59650 *Villeneuve d'Ascq*
Re: [Pharo-project] [Preview] Branch test coverage analysis tool for Pharo 3.0 :)
So if you don't mind me being very loose with terminology, you interpret (RB) ASTs, and the act of interpreting marks the nodes in these ASTs, which you can then use to colour source? frank On 6 April 2013 11:00, Clément Bera bera.clem...@gmail.com wrote: Actually, it is quite different than Jejak. Implementation wise, Jejak used Opal so I guess it relies on bytecode modification to do its analysis where my tool relies on ast annotations. This implies some difference in features. It seems that Jejak record all calls, all values and all assignments in a method (which is byte code level feature), whereas I record for each ast node if it was interpreted or not (which is ast level feature). I guess there would be a little difference in the use cases. 2013/4/6 Frank Shearar frank.shea...@gmail.com On 6 April 2013 07:47, Stéphane Ducasse stephane.duca...@inria.fr wrote: Clement built in a couple of hours the following nice branch analysecoverage tools :) It is dead slow but cool. By branch coverage tool do you mean Jejak? frank -- Clément Béra Mate Virtual Machine Engineer Bâtiment B 40, avenue Halley 59650 Villeneuve d'Ascq
Re: [Pharo-project] [Preview] Branch test coverage analysis tool for Pharo 3.0 :)
This is exactly that. The tree coverage tool is a subclass of AST-interpreter, which adding to the interpretation marks the nodes. And then I use this marking to color the source code. And I use as you said RB ast. This is why I implemented it in a few hours, I had already an AST-interpreter, so I've just created a quick subclass and a spec UI. 2013/4/6 Frank Shearar frank.shea...@gmail.com So if you don't mind me being very loose with terminology, you interpret (RB) ASTs, and the act of interpreting marks the nodes in these ASTs, which you can then use to colour source? frank On 6 April 2013 11:00, Clément Bera bera.clem...@gmail.com wrote: Actually, it is quite different than Jejak. Implementation wise, Jejak used Opal so I guess it relies on bytecode modification to do its analysis where my tool relies on ast annotations. This implies some difference in features. It seems that Jejak record all calls, all values and all assignments in a method (which is byte code level feature), whereas I record for each ast node if it was interpreted or not (which is ast level feature). I guess there would be a little difference in the use cases. 2013/4/6 Frank Shearar frank.shea...@gmail.com On 6 April 2013 07:47, Stéphane Ducasse stephane.duca...@inria.fr wrote: Clement built in a couple of hours the following nice branch analysecoverage tools :) It is dead slow but cool. By branch coverage tool do you mean Jejak? frank -- Clément Béra Mate Virtual Machine Engineer Bâtiment B 40, avenue Halley 59650 Villeneuve d'Ascq -- Clément Béra Mate Virtual Machine Engineer Bâtiment B 40, avenue Halley 59650 *Villeneuve d'Ascq*
Re: [Pharo-project] [Preview] Branch test coverage analysis tool for Pharo 3.0 :)
Would it be possible to annotate the method with some checkpoints within the method? This should be fast enough I guess. Alexandre On Apr 6, 2013, at 10:35 AM, Clément Bera bera.clem...@gmail.com wrote: This is exactly that. The tree coverage tool is a subclass of AST-interpreter, which adding to the interpretation marks the nodes. And then I use this marking to color the source code. And I use as you said RB ast. This is why I implemented it in a few hours, I had already an AST-interpreter, so I've just created a quick subclass and a spec UI. 2013/4/6 Frank Shearar frank.shea...@gmail.com So if you don't mind me being very loose with terminology, you interpret (RB) ASTs, and the act of interpreting marks the nodes in these ASTs, which you can then use to colour source? frank On 6 April 2013 11:00, Clément Bera bera.clem...@gmail.com wrote: Actually, it is quite different than Jejak. Implementation wise, Jejak used Opal so I guess it relies on bytecode modification to do its analysis where my tool relies on ast annotations. This implies some difference in features. It seems that Jejak record all calls, all values and all assignments in a method (which is byte code level feature), whereas I record for each ast node if it was interpreted or not (which is ast level feature). I guess there would be a little difference in the use cases. 2013/4/6 Frank Shearar frank.shea...@gmail.com On 6 April 2013 07:47, Stéphane Ducasse stephane.duca...@inria.fr wrote: Clement built in a couple of hours the following nice branch analysecoverage tools :) It is dead slow but cool. By branch coverage tool do you mean Jejak? frank -- Clément Béra Mate Virtual Machine Engineer Bâtiment B 40, avenue Halley 59650 Villeneuve d'Ascq -- Clément Béra Mate Virtual Machine Engineer Bâtiment B 40, avenue Halley 59650 Villeneuve d'Ascq -- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.
