Re: [Pharo-users] OpenGLES2 binding available!

[Thibault Raffaillac]

+1 for your analysis!

> One of the biggest issue of OpenGL is related with vsync, which is an 
> operation that can block all of the Pharo processes.

It does indeed, but is one way to sync with display rate (until we get threaded 
FFI of course then a thread can handle the wait). Currently we have to way to 
know the ideal time to start redraw, so as to minimise the delay between 
processing an event and displaying its feedback. Standard way is redraw with 
fixed independent frequency, which incurs an average feedback delay of (display 
period+redraw period)/2, and a waste of processing time if faster than display 
rate.

VSync in video games has a bad press because people redraw right after it 
returns:
_ Process events
_ Draw scene
_ Swap buffers <- pauses until scene is displayed
_ Loop back

That gives a worse feedback delay, but we could to pause a bit before 
processing stuff:
_ Delay(display period - last redraw duration - redraw jitter)
_ Process events
_ Draw scene
_ Swap buffers
_ Loop back

I find it very hard to find good info on VSync, and low latency event handling 
on PC in general, so please don't pan this idea too hard, this is just for 
experiment fun :)

> A far worse problem with OpenGL is that the current OpenGL context is a
> thread local variable, which do not interact at all with Pharo processes.
> This is the reason of why I added the OSWindowRenderThread class for
> serializing OSWindow and OpenGL animations. SDL2 renderers are usually
> implemented using OpenGL. They are two correct solution for this problem:
> - Having a VM that maps each Pharo Process into an operating system thread,
> - Not using OpenGL at all.
> 
> The others low level graphics API (Vulkan, Direct 3D and Metal) do not rely
> on a global thread local storage variable. These APIs are object oriented,
> so handles are passed in each API call.

Yes, this is a major flaw in OpenGL. A GL context is global to all Pharo 
processes, meaning one Process could draw in an other Process area if preempted 
at the wrong moment. But a simple way to circumvent this is giving all display 
Processes (using OpenGL) the same priority.

> Anyway, implementing a Athens backend from scratch is a really hard problem.

Agreed. The difficulty with a hardware backend for Athens is it comes from 
software world. Some primitives are a nightmare to accelerate, resulting in 
usually incomplete hardware backends 
(https://developer.android.com/guide/topics/graphics/hardware-accel.html#unsupported).
 This is annoying for users because some primitives may turn off acceleration 
entirely, and force you to dig into the internal GL mud.
To me the use of hardware drawing is either explicit or none. If appended to 
Athens it should be very clear what goes to the GPU and what not.

Re: [Pharo-users] OpenGLES2 binding available!

[Alexandre Bergel]

+1

> On Jul 28, 2016, at 3:52 PM, stepharo  wrote:
> 
> Thanks ronie for your analysis (especially the thread local point). 
> Stef
> 
> Le 28/7/16 à 20:36, Ronie Salgado a écrit :
>> Isn't it what Ronie is working on?
>> I am not working with OpenGL or OpenGL ES anymore. I am moving all of my 
>> efforts into Woden 2, the AbstractGPU (an abstraction layer for Vulkan, 
>> Direct3D 12 and Metal), and Dastrel (Data Stream Language) a custom shader 
>> language whose compiler I implemented in Pharo, with a backend for SpirV, 
>> GLSL and C++. I still have to implement the backends for Metal and HLSL.
>> 
>> Thibault is interested more on being able to use OpenGL ES 2.0 directly in 
>> the playground, for teaching purposes without any abstraction layer 
>> underlying. I do not agree with many of his approaches, too low level, and 
>> many of themp do not work well with Pharo VM. One of the biggest issue of 
>> OpenGL is related with vsync, which is an operation that can block all of 
>> the Pharo processes, until we get a VM with the threaded FFI, which will 
>> have in the near future. It will use the UnifiedFFI API, so we will not have 
>> to further changes to take advantage of the threaded ffi in the user code.
>> 
>> A far worse problem with OpenGL is that the current OpenGL context is a 
>> thread local variable, which do not interact at all with Pharo processes. 
>> This is the reason of why I added the OSWindowRenderThread class for 
>> serializing OSWindow and OpenGL animations. SDL2 renderers are usually 
>> implemented using OpenGL. They are two correct solution for this problem:
>> - Having a VM that maps each Pharo Process into an operating system thread, 
>> - Not using OpenGL at all.
>> 
>> The others low level graphics API (Vulkan, Direct 3D and Metal) do not rely 
>> on a global thread local storage variable. These APIs are object oriented, 
>> so handles are passed in each API call.
>> 
>> Currently, I have an Athens backend above Woden 2, which is only supporting 
>> Vulkan for now. Since this backend is for Woden 2, it has some dependencies 
>> in the Woden 2 Math and the Lowcode NativeStructure. This means that the 
>> Woden 2 Athens backend has a strong dependency in the the modified 
>> OpalCompiler that uses the Lowcode extended byte code set, and the the 
>> modified PharoVM that implements these extra bytecode in the just in time 
>> compiler.
>> 
>> Anyway, implementing a Athens backend from scratch is a really hard problem, 
>> because paths can be concave, curve and self intersecting. Drawing a path, 
>> unlike drawing a triangle is a global operation, for which GPUs were not 
>> designed for. The easiest and most robust technique for drawing a path using 
>> the GPU is to flatten the path using De Casteljau's algorithm and then using 
>> the stencil buffer technique for drawing concave polygons that was described 
>> in the OpenGL red book ( 
>> http://www.glprogramming.com/red/chapter14.html#name13 ). The problem of 
>> this technique is that involves many state changes(glStencil*)  and drawing 
>> commands(glDrawElements) per path, which are the most expensive operations 
>> when using OpenGL. The alternative is to perform a triangulation in the CPU 
>> of paths, or to use one of the newer low level APIs such as Vulkan and 
>> Direct 3D 12 which were designed to reduce or eliminate the cost of state 
>> changes and drawing commands.
>> 
>> Path stroking is an even harder problem, which I have not solved yet. The 
>> standard approach is to convert a stroked path into a path that can be 
>> filled. The problem is that bezier offset curves are not bezier curves, so 
>> they have to be approximated. Fortunately, since this operation is backend 
>> agnostic, it can be used and reused by many Athens backends. Since stroking 
>> is not critical for my needs, I do not think I will implement it soon 
>> properly. Currently I am just drawing a 1 pixel thick line, which can look 
>> horrible, but is better than nothing. One algorithm that seems to be simple 
>> to implement is one described in a paper from the Cairo people, which uses 
>> the Minkowksi  Sum between the outline of the path to be stroked, and shape 
>> of the pen that is stroking
>> the path: http://keithp.com/~keithp/talks/cairo2003.pdf
>> 
>> Soon I will merge the Lowcode instructions into the SqueakVM/Cog VM to 
>> integrate these bytecodes into the mainline VM.
>> 
>> I will not publish Woden 2 until the dependencies and Woden 2 itself can be 
>> loaded easily.  Soon I will publish properly the Dastrel shader language and 
>> the Slovim backend. Slovim name comes from Smalltalk Low levelish VIrtual 
>> Machine, an intermediate representation heavily modeled after the one used 
>> in LLVM, implemented in Pharo, some very basic optimizations(basic constant 
>> folding, inlining and some dead code generation), and a code generator 
>> backend for SpirV, GLSL and C++. My biggest issue before publishing 

Re: [Pharo-users] OpenGLES2 binding available!

[stepharo]

Anyway, implementing a Athens backend from scratch is a really hard 
problem, because paths can be concave, curve and self intersecting. 
Drawing a path, unlike drawing a triangle is a global operation, for 
which GPUs were not designed for. The easiest and most robust 
technique for drawing a path using the GPU is to flatten the path 
using De Casteljau's algorithm and then using the stencil buffer 
technique for drawing concave polygons that was described in the 
OpenGL red book ( 
http://www.glprogramming.com/red/chapter14.html#name13 ). The problem 
of this technique is that involves many state changes(glStencil*)  and 
drawing commands(glDrawElements) per path, which are the most 
expensive operations when using OpenGL. The alternative is to perform 
a triangulation in the CPU of paths, or to use one of the newer low 
level APIs such as Vulkan and Direct 3D 12 which were designed to 
reduce or eliminate the cost of state changes and drawing commands.


Path stroking is an even harder problem, which I have not solved yet. 
The standard approach is to convert a stroked path into a path that 
can be filled. The problem is that bezier offset curves are not bezier 
curves, so they have to be approximated. Fortunately, since this 
operation is backend agnostic, it can be used and reused by many 
Athens backends. Since stroking is not critical for my needs, I do not 
think I will implement it soon properly. Currently I am just drawing a 
1 pixel thick line, which can look horrible, but is better than 
nothing. One algorithm that seems to be simple to implement is one 
described in a paper from the Cairo people, which uses the Minkowksi 
 Sum between the outline of the path to be stroked, and shape of the 
pen that is stroking
the path: http://keithp.com/~keithp/talks/cairo2003.pdf 



Thanks because indeed this is a lot of work. Thanks for the reality check.




Soon I will merge the Lowcode instructions into the SqueakVM/Cog VM to 
integrate these bytecodes into the mainline VM.


I will not publish Woden 2 until the dependencies and Woden 2 itself 
can be loaded easily.  Soon I will publish properly the Dastrel shader 
language and the Slovim backend. Slovim name comes from Smalltalk Low 
levelish VIrtual Machine, an intermediate representation heavily 
modeled after the one used in LLVM, implemented in Pharo, some very 
basic optimizations(basic constant folding, inlining and some dead 
code generation), and a code generator backend for SpirV, GLSL and 
C++. My biggest issue before publishing this compiler are:

- Implementing a command line interface.
- Doing an easy to install and use package for Linux, OS X and Windows.
- Documenting the language and the compiler.

Best regards,
Ronie

2016-07-28 19:31 GMT+02:00 Alexandre Bergel >:



I would love to see an athens back-end for it.


Isn't it what Ronie is working on?

Alexandre



Le 19/7/16 à 13:19, Thibault Raffaillac a écrit :

Hi all,

My tiny binding for OpenGLES2 is ready :)
http://smalltalkhub.com/#!/~ThibaultRaffaillac/OpenGLES2/


It takes a different direction than that of NBOpenGL. I found the support 
for all versions of OpenGL overwhelming, both for beginners and to maintain. 
With a limited number of functions I could carefully name all messages.

A demo using either SDL2 or GLFW (faster) is included, supporting VSync and 
retina displays.
Tested only on Mac OSX, patches welcome!

Cheers,
Thibault

Re: [Pharo-users] OpenGLES2 binding available!

[stepharo]

Thanks ronie for your analysis (especially the thread local point).

Stef


Le 28/7/16 à 20:36, Ronie Salgado a écrit :


Isn't it what Ronie is working on?

I am not working with OpenGL or OpenGL ES anymore. I am moving all of 
my efforts into Woden 2, the AbstractGPU (an abstraction layer for 
Vulkan, Direct3D 12 and Metal), and Dastrel (Data Stream Language) a 
custom shader language whose compiler I implemented in Pharo, with a 
backend for SpirV, GLSL and C++. I still have to implement the 
backends for Metal and HLSL.


Thibault is interested more on being able to use OpenGL ES 2.0 
directly in the playground, for teaching purposes without any 
abstraction layer underlying. I do not agree with many of his 
approaches, too low level, and many of themp do not work well with 
Pharo VM. One of the biggest issue of OpenGL is related with vsync, 
which is an operation that can block all of the Pharo processes, until 
we get a VM with the threaded FFI, which will have in the near future. 
It will use the UnifiedFFI API, so we will not have to further changes 
to take advantage of the threaded ffi in the user code.


A far worse problem with OpenGL is that the current OpenGL context is 
a thread local variable, which do not interact at all with Pharo 
processes. This is the reason of why I added the OSWindowRenderThread 
class for serializing OSWindow and OpenGL animations. SDL2 renderers 
are usually implemented using OpenGL. They are two correct solution 
for this problem:
- Having a VM that maps each Pharo Process into an operating system 
thread,

- Not using OpenGL at all.

The others low level graphics API (Vulkan, Direct 3D and Metal) do not 
rely on a global thread local storage variable. These APIs are object 
oriented, so handles are passed in each API call.


Currently, I have an Athens backend above Woden 2, which is only 
supporting Vulkan for now. Since this backend is for Woden 2, it has 
some dependencies in the Woden 2 Math and the Lowcode NativeStructure. 
This means that the Woden 2 Athens backend has a strong dependency in 
the the modified OpalCompiler that uses the Lowcode extended byte code 
set, and the the modified PharoVM that implements these extra bytecode 
in the just in time compiler.


Anyway, implementing a Athens backend from scratch is a really hard 
problem, because paths can be concave, curve and self intersecting. 
Drawing a path, unlike drawing a triangle is a global operation, for 
which GPUs were not designed for. The easiest and most robust 
technique for drawing a path using the GPU is to flatten the path 
using De Casteljau's algorithm and then using the stencil buffer 
technique for drawing concave polygons that was described in the 
OpenGL red book ( 
http://www.glprogramming.com/red/chapter14.html#name13 ). The problem 
of this technique is that involves many state changes(glStencil*)  and 
drawing commands(glDrawElements) per path, which are the most 
expensive operations when using OpenGL. The alternative is to perform 
a triangulation in the CPU of paths, or to use one of the newer low 
level APIs such as Vulkan and Direct 3D 12 which were designed to 
reduce or eliminate the cost of state changes and drawing commands.


Path stroking is an even harder problem, which I have not solved yet. 
The standard approach is to convert a stroked path into a path that 
can be filled. The problem is that bezier offset curves are not bezier 
curves, so they have to be approximated. Fortunately, since this 
operation is backend agnostic, it can be used and reused by many 
Athens backends. Since stroking is not critical for my needs, I do not 
think I will implement it soon properly. Currently I am just drawing a 
1 pixel thick line, which can look horrible, but is better than 
nothing. One algorithm that seems to be simple to implement is one 
described in a paper from the Cairo people, which uses the Minkowksi 
 Sum between the outline of the path to be stroked, and shape of the 
pen that is stroking
the path: http://keithp.com/~keithp/talks/cairo2003.pdf 



Soon I will merge the Lowcode instructions into the SqueakVM/Cog VM to 
integrate these bytecodes into the mainline VM.


I will not publish Woden 2 until the dependencies and Woden 2 itself 
can be loaded easily.  Soon I will publish properly the Dastrel shader 
language and the Slovim backend. Slovim name comes from Smalltalk Low 
levelish VIrtual Machine, an intermediate representation heavily 
modeled after the one used in LLVM, implemented in Pharo, some very 
basic optimizations(basic constant folding, inlining and some dead 
code generation), and a code generator backend for SpirV, GLSL and 
C++. My biggest issue before publishing this compiler are:

- Implementing a command line interface.
- Doing an easy to install and use package for Linux, OS X and Windows.
- Documenting the language and the compiler.

Best regards,
Ronie

2016-07-28 19:31 GMT+02:00 

Re: [Pharo-users] OpenGLES2 binding available!

[Ronie Salgado]

>
> Isn't it what Ronie is working on?

I am not working with OpenGL or OpenGL ES anymore. I am moving all of my
efforts into Woden 2, the AbstractGPU (an abstraction layer for
Vulkan, Direct3D 12 and Metal), and Dastrel (Data Stream Language) a custom
shader language whose compiler I implemented in Pharo, with a backend for
SpirV, GLSL and C++. I still have to implement the backends for Metal and
HLSL.

Thibault is interested more on being able to use OpenGL ES 2.0 directly in
the playground, for teaching purposes without any abstraction layer
underlying. I do not agree with many of his approaches, too low level, and
many of themp do not work well with Pharo VM. One of the biggest issue of
OpenGL is related with vsync, which is an operation that can block all of
the Pharo processes, until we get a VM with the threaded FFI, which will
have in the near future. It will use the UnifiedFFI API, so we will not
have to further changes to take advantage of the threaded ffi in the user
code.

A far worse problem with OpenGL is that the current OpenGL context is a
thread local variable, which do not interact at all with Pharo processes.
This is the reason of why I added the OSWindowRenderThread class for
serializing OSWindow and OpenGL animations. SDL2 renderers are usually
implemented using OpenGL. They are two correct solution for this problem:
- Having a VM that maps each Pharo Process into an operating system thread,
- Not using OpenGL at all.

The others low level graphics API (Vulkan, Direct 3D and Metal) do not rely
on a global thread local storage variable. These APIs are object oriented,
so handles are passed in each API call.

Currently, I have an Athens backend above Woden 2, which is only supporting
Vulkan for now. Since this backend is for Woden 2, it has some dependencies
in the Woden 2 Math and the Lowcode NativeStructure. This means that the
Woden 2 Athens backend has a strong dependency in the the modified
OpalCompiler that uses the Lowcode extended byte code set, and the the
modified PharoVM that implements these extra bytecode in the just in time
compiler.

Anyway, implementing a Athens backend from scratch is a really hard
problem, because paths can be concave, curve and self intersecting. Drawing
a path, unlike drawing a triangle is a global operation, for which GPUs
were not designed for. The easiest and most robust technique for drawing a
path using the GPU is to flatten the path using De Casteljau's algorithm
and then using the stencil buffer technique for drawing concave polygons
that was described in the OpenGL red book (
http://www.glprogramming.com/red/chapter14.html#name13 ). The problem of
this technique is that involves many state changes(glStencil*)  and drawing
commands(glDrawElements) per path, which are the most expensive operations
when using OpenGL. The alternative is to perform a triangulation in the CPU
of paths, or to use one of the newer low level APIs such as Vulkan and
Direct 3D 12 which were designed to reduce or eliminate the cost of state
changes and drawing commands.

Path stroking is an even harder problem, which I have not solved yet. The
standard approach is to convert a stroked path into a path that can be
filled. The problem is that bezier offset curves are not bezier curves, so
they have to be approximated. Fortunately, since this operation is backend
agnostic, it can be used and reused by many Athens backends. Since stroking
is not critical for my needs, I do not think I will implement it soon
properly. Currently I am just drawing a 1 pixel thick line, which can look
horrible, but is better than nothing. One algorithm that seems to be simple
to implement is one described in a paper from the Cairo people, which uses
the Minkowksi  Sum between the outline of the path to be stroked, and shape
of the pen that is stroking
the path: http://keithp.com/~keithp/talks/cairo2003.pdf

Soon I will merge the Lowcode instructions into the SqueakVM/Cog VM to
integrate these bytecodes into the mainline VM.

I will not publish Woden 2 until the dependencies and Woden 2 itself can be
loaded easily.  Soon I will publish properly the Dastrel shader language
and the Slovim backend. Slovim name comes from Smalltalk Low levelish
VIrtual Machine, an intermediate representation heavily modeled after the
one used in LLVM, implemented in Pharo, some very basic optimizations(basic
constant folding, inlining and some dead code generation), and a code
generator backend for SpirV, GLSL and C++. My biggest issue before
publishing this compiler are:
- Implementing a command line interface.
- Doing an easy to install and use package for Linux, OS X and Windows.
- Documenting the language and the compiler.

Best regards,
Ronie

2016-07-28 19:31 GMT+02:00 Alexandre Bergel :

> I would love to see an athens back-end for it.
>
> Isn't it what Ronie is working on?
>
> Alexandre
>
>
> Le 19/7/16 à 13:19, Thibault Raffaillac a écrit :
>
> Hi all,
>
> My tiny binding for 

Re: [Pharo-users] OpenGLES2 binding available!

[Alexandre Bergel]

> I would love to see an athens back-end for it.
> 
Isn't it what Ronie is working on?

Alexandre

> 
> Le 19/7/16 à 13:19, Thibault Raffaillac a écrit :
>> Hi all,
>> 
>> My tiny binding for OpenGLES2 is ready :)
>> http://smalltalkhub.com/#!/~ThibaultRaffaillac/OpenGLES2/ 
>> 
>> 
>> It takes a different direction than that of NBOpenGL. I found the support 
>> for all versions of OpenGL overwhelming, both for beginners and to maintain. 
>> With a limited number of functions I could carefully name all messages.
>> 
>> A demo using either SDL2 or GLFW (faster) is included, supporting VSync and 
>> retina displays.
>> Tested only on Mac OSX, patches welcome!
>> 
>> Cheers,
>> Thibault
>> 
> 

Re: [Pharo-users] OpenGLES2 binding available!

[stepharo]

Hi thibault


I took some times to look at your project. Nice.

Now the GLW example is not working.

I understand that this is a binding. I do not want to have to work that 
low level.


I'm curious to see what people will be able to build on top of it.


Stef


Le 19/7/16 à 13:19, Thibault Raffaillac a écrit :

Hi all,

My tiny binding for OpenGLES2 is ready :)
http://smalltalkhub.com/#!/~ThibaultRaffaillac/OpenGLES2/

It takes a different direction than that of NBOpenGL. I found the support for 
all versions of OpenGL overwhelming, both for beginners and to maintain. With a 
limited number of functions I could carefully name all messages.

A demo using either SDL2 or GLFW (faster) is included, supporting VSync and 
retina displays.
Tested only on Mac OSX, patches welcome!

Cheers,
Thibault

Re: [Pharo-users] OpenGLES2 binding available!

[stepharo]

Thanks for publishing it.

I would love to see an athens back-end for it.


Stef


Le 19/7/16 à 13:19, Thibault Raffaillac a écrit :

Hi all,

My tiny binding for OpenGLES2 is ready :)
http://smalltalkhub.com/#!/~ThibaultRaffaillac/OpenGLES2/

It takes a different direction than that of NBOpenGL. I found the support for 
all versions of OpenGL overwhelming, both for beginners and to maintain. With a 
limited number of functions I could carefully name all messages.

A demo using either SDL2 or GLFW (faster) is included, supporting VSync and 
retina displays.
Tested only on Mac OSX, patches welcome!

Cheers,
Thibault

[Pharo-users] OpenGLES2 binding available!

[Thibault Raffaillac]

Hi all,

My tiny binding for OpenGLES2 is ready :)
http://smalltalkhub.com/#!/~ThibaultRaffaillac/OpenGLES2/

It takes a different direction than that of NBOpenGL. I found the support for 
all versions of OpenGL overwhelming, both for beginners and to maintain. With a 
limited number of functions I could carefully name all messages.

A demo using either SDL2 or GLFW (faster) is included, supporting VSync and 
retina displays.
Tested only on Mac OSX, patches welcome!

Cheers,
Thibault