Re: color lib

[Andrea Fontana via Digitalmars-d]

On Tuesday, 11 October 2016 at 12:14:37 UTC, Manu wrote:

Oh dear... thanks for digging that up.
I didn't know the web had a standard for alpha. Certainly 
0xAARRGGBB
has been used in windows code for as long as I've been 
programming...

but now there's a competing #RRGGBBAA version...
How to resolve this? I guess, go with the web? I should probably
change it to the CSS4 way.


My idea is still to use a template:
colorFromString!"rgba" or colorFromString!"argb" (please notice 
that AFAIK, the second most used way is actually "abgr" rather 
than "argb" - because of byte-order)


And I think it's a good idea to set template argument to some 
default.


MS is not sure about this, anyway.

Read carefully this:
https://msdn.microsoft.com/en-us/library/windows/desktop/ms534427(v=vs.85).aspx

VS:

https://msdn.microsoft.com/en-us/library/ee505694(v=winembedded.60).aspx

VS:

https://msdn.microsoft.com/it-it/library/windows/desktop/dd183449(v=vs.85).aspx


OpenGL use rgba (and abgr?) directX argb:
https://www.opengl.org/wiki/Image_Format
https://www.opengl.org/wiki/Direct3D_Compatibility

Andrea

Re: color lib

[Manu via Digitalmars-d]

On 11 October 2016 at 18:10, Andrea Fontana via Digitalmars-d
 wrote:
> On Monday, 10 October 2016 at 23:26:53 UTC, Manu wrote:
>>
>> I'm not sure why it matters what format the colour you have is...
>> Strings are in the form #RRGGBB, or #AARRGGBB. That is all.
>> It's the standard I've seen used everywhere ever, including the web,
>> which is a pretty good precedent :P
>
>
> If the web is a good precedent (CSS4 specs):
> "The first 6 digits are interpreted identically to the 6-digit notation. The
> last pair of digits, interpreted as a hexadecimal number, specifies the
> alpha channel of the color, where 00 represents a fully transparent color
> and ff represent a fully opaque color."
>
> https://drafts.csswg.org/css-color/#hex-notation
>
> CSS3 doesn't support hex string with alpha but they suggest you to use
> rgba() function. I think argb() doesn't exists instead.
> https://www.w3.org/TR/2011/REC-css3-color-20110607/#rgba-color
>
> Chrome 52 supports it:
> https://googlechrome.github.io/samples/css-alpha-channel/
>
> Android instead:
> https://developer.android.com/reference/android/graphics/Color.html#parseColor(java.lang.String)
> "Parse the color string, and return the corresponding color-int. If the
> string cannot be parsed, throws an IllegalArgumentException exception.
> Supported formats are: #RRGGBB #AARRGGBB [...]"
>
> Please notice that on PNG file format rgba is quite common (also on bmp with
> semi-official apha support)
> PNG: http://www.libpng.org/pub/png/book/chapter08.html

Oh dear... thanks for digging that up.
I didn't know the web had a standard for alpha. Certainly 0xAARRGGBB
has been used in windows code for as long as I've been programming...
but now there's a competing #RRGGBBAA version...
How to resolve this? I guess, go with the web? I should probably
change it to the CSS4 way.

Re: color lib

[Andrea Fontana via Digitalmars-d]

On Monday, 10 October 2016 at 23:26:53 UTC, Manu wrote:
I'm not sure why it matters what format the colour you have 
is...

Strings are in the form #RRGGBB, or #AARRGGBB. That is all.
It's the standard I've seen used everywhere ever, including the 
web,

which is a pretty good precedent :P


If the web is a good precedent (CSS4 specs):
"The first 6 digits are interpreted identically to the 6-digit 
notation. The last pair of digits, interpreted as a hexadecimal 
number, specifies the alpha channel of the color, where 00 
represents a fully transparent color and ff represent a fully 
opaque color."


https://drafts.csswg.org/css-color/#hex-notation

CSS3 doesn't support hex string with alpha but they suggest you 
to use rgba() function. I think argb() doesn't exists instead.

https://www.w3.org/TR/2011/REC-css3-color-20110607/#rgba-color

Chrome 52 supports it:
https://googlechrome.github.io/samples/css-alpha-channel/

Android instead:
https://developer.android.com/reference/android/graphics/Color.html#parseColor(java.lang.String)
"Parse the color string, and return the corresponding color-int. 
If the string cannot be parsed, throws an 
IllegalArgumentException exception. Supported formats are: 
#RRGGBB #AARRGGBB [...]"


Please notice that on PNG file format rgba is quite common (also 
on bmp with semi-official apha support)

PNG: http://www.libpng.org/pub/png/book/chapter08.html

Re: color lib

[Nicholas Wilson via Digitalmars-d]

On Tuesday, 11 October 2016 at 00:10:04 UTC, Nicholas Wilson 
wrote:
What about forwarding the array ops to a foreach of the static 
array?

Like as above but instead of:

ElementType!(R)[N] batch;

have:

static struct Batch
{
ElementType!(R)[N] elements;
auto get() { return elements[];}

Batch opBinary(string op)(Batch rhs) 
if(hasOperator!(ElementType!(R),op))

{
Batch b;
foreach(i; iota(N)) mixin("b.elements[i] = elememts[i]" 
~op~"rhs.elements[i]");

return b;
}
//repeat for opUnary,opOpAssign...
}
Batch batch;

I'll make another forum thread for this.


whoops missed an
alias get this;

Re: color lib

[Nicholas Wilson via Digitalmars-d]

On Sunday, 9 October 2016 at 13:28:05 UTC, Manu wrote:
On 9 October 2016 at 15:34, Ilya Yaroshenko via Digitalmars-d 
 wrote:

On Sunday, 9 October 2016 at 05:21:32 UTC, Manu wrote:


On 9 October 2016 at 14:03, Nicholas Wilson via Digitalmars-d 
 wrote:


[...]



Well the trouble is the lambda that you might give to 'map' 
won't work anymore. Operators don't work on batches, you need 
to use a completely different API, and I think that's 
unfortunate.



Could you please give an example what type of operation should 
be vectorized?


Even operations that don't require shuffling, eg:
  RGBA8[] a, b;
  zip(a, b).map!(e => e[0] + e[1]).copy(output);

Which I've suggested before (and Walter liked the idea), could 
be
sugared up by making use of the languages largely under-used 
array

operation syntax:
  output[] = a[] + b[]; // ie, types have overloaded addition
operators, so this array expression would be lowered to the 
pipeline
expression above. This would be super-useful for HEAPS of 
things!


Even these still need to be done in batches since colour adds 
are saturating operations, and there are SIMD instructions for 
saturating arithmetic, so we basically always have to do colour 
work in SIMD, which means batching, and that basically ruins 
any chance for natural, readable, expressions in your code. I 
want to find a way that we can express these operations 
naturally, without having to always manually handle the 
batching.


If we can get there, then I will say D is a good language for 
stream-data processing.


What about forwarding the array ops to a foreach of the static 
array?

Like as above but instead of:

ElementType!(R)[N] batch;

have:

static struct Batch
{
ElementType!(R)[N] elements;
auto get() { return elements[];}

Batch opBinary(string op)(Batch rhs) 
if(hasOperator!(ElementType!(R),op))

{
Batch b;
foreach(i; iota(N)) mixin("b.elements[i] = elememts[i]" 
~op~"rhs.elements[i]");

return b;
}
//repeat for opUnary,opOpAssign...
}
Batch batch;

I'll make another forum thread for this.

Re: color lib

[Manu via Digitalmars-d]

On 10 October 2016 at 23:41, Andrea Fontana via Digitalmars-d
 wrote:
> On Monday, 10 October 2016 at 13:25:07 UTC, Manu wrote:
>>
>> On 10 October 2016 at 23:00, Andrea Fontana via Digitalmars-d
>>  wrote:
>>>
>>> On Monday, 10 October 2016 at 08:44:49 UTC, Manu wrote:
>>>
>>> From doc:
>>> colorFromString Create a color from a string. May be a hex color in the
>>> standard forms: (#/$)rgb/argb/rrggbb/aarrggbb May also be the name of any
>>> color from the Colors enum.
>>>
>>> It seems it reads just rgb. (+ enum)
>>>
>>> I think that:
>>> colorFromString("red");
>>> colorFromString!"rgb"("#3212ff");
>>> colorFromString!"bgra"("#ff1232dd");
>>>
>>> makes more sense.
>>>
>>> Andrea
>>
>>
>> Why? I see no value in that function being a template... It's not like you
>> can confuse "#FF0080" and "LightGoldenrodYellow". As far as I know, there's
>> no possible ambiguity in colour strings, so why make them separate
>> functions?
>
>
> But it would be useful to create rgb, bgr, argb, bgra, or other color space
> using a string.

Give the preferred format as template arg?

> If a third party library or source gives me code in rgba, I have to
> preprocess it to convert as argb and then pass it to your library.

Sorry, what are we talking about? My lib supports basically every
format or arrangement I've ever encountered... you can work with
practically any data format you can think of.

> Anyway, I don't know if a code with letters a-f can be composed. In that
> case an ambiguity exists.

It must also be exactly 3,4,6,8 letters long, and begin with the
letter '#' or '$' ;)


I'm not sure why it matters what format the colour you have is...
Strings are in the form #RRGGBB, or #AARRGGBB. That is all.
It's the standard I've seen used everywhere ever, including the web,
which is a pretty good precedent :P
If you support swizzled forms of strings, then ambiguity exists.
Better not to allow it.

If you want a BGR from a string, use: colorFromString!BGR8("#FF");
If you want Lab: colorFromString!(Lab!float)("#FF");

Re: color lib

[Andrea Fontana via Digitalmars-d]

On Monday, 10 October 2016 at 13:25:07 UTC, Manu wrote:
On 10 October 2016 at 23:00, Andrea Fontana via Digitalmars-d 
 wrote:

On Monday, 10 October 2016 at 08:44:49 UTC, Manu wrote:

From doc:
colorFromString Create a color from a string. May be a hex 
color in the
standard forms: (#/$)rgb/argb/rrggbb/aarrggbb May also be the 
name of any

color from the Colors enum.

It seems it reads just rgb. (+ enum)

I think that:
colorFromString("red");
colorFromString!"rgb"("#3212ff");
colorFromString!"bgra"("#ff1232dd");

makes more sense.

Andrea


Why? I see no value in that function being a template... It's 
not like you can confuse "#FF0080" and "LightGoldenrodYellow". 
As far as I know, there's no possible ambiguity in colour 
strings, so why make them separate functions?


But it would be useful to create rgb, bgr, argb, bgra, or other 
color space using a string.


If a third party library or source gives me code in rgba, I have 
to preprocess it to convert as argb and then pass it to your 
library.


Anyway, I don't know if a code with letters a-f can be composed. 
In that case an ambiguity exists.

Re: color lib

[Manu via Digitalmars-d]

On 10 October 2016 at 23:00, Andrea Fontana via Digitalmars-d
 wrote:
> On Monday, 10 October 2016 at 08:44:49 UTC, Manu wrote:
>
> From doc:
> colorFromString Create a color from a string. May be a hex color in the
> standard forms: (#/$)rgb/argb/rrggbb/aarrggbb May also be the name of any
> color from the Colors enum.
>
> It seems it reads just rgb. (+ enum)
>
> I think that:
> colorFromString("red");
> colorFromString!"rgb"("#3212ff");
> colorFromString!"bgra"("#ff1232dd");
>
> makes more sense.
>
> Andrea

Why? I see no value in that function being a template... It's not like
you can confuse "#FF0080" and "LightGoldenrodYellow". As far as I
know, there's no possible ambiguity in colour strings, so why make
them separate functions?

Re: color lib

[Andrea Fontana via Digitalmars-d]

On Monday, 10 October 2016 at 08:44:49 UTC, Manu wrote:
On 10 October 2016 at 17:29, Andrea Fontana via Digitalmars-d 
 wrote:

On Thursday, 6 October 2016 at 14:53:52 UTC, Manu wrote:


I've done another pass incorporating prior feedback, mostly 
focusing on documentation.



http://dtest.thecybershadow.net/artifact/website-b6e2e44dd40dd7c70eb45829c02060b99ae3937b-57272ccdf902fa3f0c050d522129f2be/web/library-prerelease/std/experimental/color.html

Can interested parties please give it another once-over and 
add

further comments?
How can I get this to a point where people would like to see 
it in phobos?


Repo: https://github.com/TurkeyMan/color
PR: https://github.com/dlang/phobos/pull/2845



Nice work!

colorFromString should be colorFromRGBString :)


Nar. It parses any form of colour-in-a-string.


From doc:
colorFromString	Create a color from a string. May be a hex color 
in the standard forms: (#/$)rgb/argb/rrggbb/aarrggbb May also be 
the name of any color from the Colors enum.


It seems it reads just rgb. (+ enum)

I think that:
colorFromString("red");
colorFromString!"rgb"("#3212ff");
colorFromString!"bgra"("#ff1232dd");

makes more sense.

Andrea

Re: color lib

[Ethan Watson via Digitalmars-d]

On Monday, 10 October 2016 at 12:10:56 UTC, Jacob Carlborg wrote:

Isn't std.typecons.Flag metaprogramming ;)


Hahaha, oh wow. If ever there was a case for mixins.

Re: color lib

[Jacob Carlborg via Digitalmars-d]

On 2016-10-10 12:39, Ethan Watson wrote:


I'm especially trying to make Binderoo readable as there's so many
programmers that are scared by metaprogramming.


Isn't std.typecons.Flag metaprogramming ;)

--
/Jacob Carlborg

Re: color lib

[Ethan Watson via Digitalmars-d]

On Saturday, 8 October 2016 at 13:06:42 UTC, Manu wrote:

Oh no, you too? >_<


Yeah, I've been going on a readability bender lately, especially 
in public facing code.


My thinking there is that statements in code that don't 
immediately give context are essentially a cipher. Because that's 
exactly what you need to do to understand the code - look 
something up to see what it means. Named parameters and variable 
names that provide the context avoid that to a large degree.


I'm especially trying to make Binderoo readable as there's so 
many programmers that are scared by metaprogramming. My GDCE talk 
spent a lot of time attempting to make it all understandable. 
Making the code descriptive seals the deal. If I can make my code 
more descriptive, and it compiles out just the same but makes the 
compiler do a bit more work... Make the compiler do more work and 
optimise the compiler.


I'm far more lax on not-publicly-facing code (so basically API 
implementations and supporting code that isn't part of a public 
interface). Anything I expect someone other than myself to 
interact with gets the readability treatment. Which, as you know, 
is important because readable code generally isn't efficient code 
- as is evidenced by the vectorisation/buffer processing thread 
going on in here.


It's also interesting how many programmers get vehemently 
defensive when you call out non-descriptive programming practices 
as programming for your own convenience and no one else. I have 
this argument with using i/j/k/foo/bar/etc in loops as well.


Incidentally, have you had a geez over the core API? An 
efficient API

will emerge when we work out how to work batched operations into
ranges...


Been slowly making my way through it. Seems solid enough, but I 
haven't looked through it all thoroughly yet.

Re: color lib

[Manu via Digitalmars-d]

On 10 October 2016 at 17:29, Andrea Fontana via Digitalmars-d
 wrote:
> On Thursday, 6 October 2016 at 14:53:52 UTC, Manu wrote:
>>
>> I've done another pass incorporating prior feedback, mostly focusing on
>> documentation.
>>
>>
>> http://dtest.thecybershadow.net/artifact/website-b6e2e44dd40dd7c70eb45829c02060b99ae3937b-57272ccdf902fa3f0c050d522129f2be/web/library-prerelease/std/experimental/color.html
>>
>> Can interested parties please give it another once-over and add
>> further comments?
>> How can I get this to a point where people would like to see it in phobos?
>>
>> Repo: https://github.com/TurkeyMan/color
>> PR: https://github.com/dlang/phobos/pull/2845
>
>
> Nice work!
>
> colorFromString should be colorFromRGBString :)

Nar. It parses any form of colour-in-a-string.

Re: color lib

[Andrea Fontana via Digitalmars-d]

On Thursday, 6 October 2016 at 14:53:52 UTC, Manu wrote:
I've done another pass incorporating prior feedback, mostly 
focusing on documentation.


http://dtest.thecybershadow.net/artifact/website-b6e2e44dd40dd7c70eb45829c02060b99ae3937b-57272ccdf902fa3f0c050d522129f2be/web/library-prerelease/std/experimental/color.html

Can interested parties please give it another once-over and add
further comments?
How can I get this to a point where people would like to see it 
in phobos?


Repo: https://github.com/TurkeyMan/color
PR: https://github.com/dlang/phobos/pull/2845


Nice work!

colorFromString should be colorFromRGBString :)

Re: color lib

[Manu via Digitalmars-d]

On 9 October 2016 at 23:36, Ilya Yaroshenko via Digitalmars-d
 wrote:
> On Sunday, 9 October 2016 at 13:18:22 UTC, Manu wrote:
>>
>> On 9 October 2016 at 15:34, Ilya Yaroshenko via Digitalmars-d
>>  wrote:
>>>
>>> On Sunday, 9 October 2016 at 05:21:32 UTC, Manu wrote:


 On 9 October 2016 at 14:03, Nicholas Wilson via Digitalmars-d
  wrote:
>
>
> [...]



 Well the trouble is the lambda that you might give to 'map' won't work
 anymore. Operators don't work on batches, you need to use a completely
 different API, and I think that's unfortunate.
>>>
>>>
>>>
>>> Could you please give an example what type of operation should be
>>> vectorized?
>>
>>
>> Let's consider a super simple blend:
>>   dest = src.rgb * src.a + dest.rgb * (1-src.alpha);
>
>
> This code do not need transposition. And can be partially vectorised using
> mir.ndslice.algorithm.

How so?

Sorry, let me write the actual work in full:

  RGBA8[] src, dest; // allocate somehow

  zip(src, dest).map!((e) {
ubyte r = cast(ubyte)clamp(((cast(int)e[0].r * e[0].a * 0x1011) >>
20) + ((cast(int)e[1].r * (255 - e[0].a) * 0x1011) >> 20), 0, 255);
ubyte g = cast(ubyte)clamp(((cast(int)e[0].g * e[0].a * 0x1011) >>
20) + ((cast(int)e[1].g * (255 - e[0].a) * 0x1011) >> 20), 0, 255);
ubyte b = cast(ubyte)clamp(((cast(int)e[0].b * e[0].a * 0x1011) >>
20) + ((cast(int)e[1].b * (255 - e[0].a) * 0x1011) >> 20), 0, 255);
return RGBA8(r, g, b, e[0].a);
  }).copy(dest);

If you can coerce the auto-vectoriser to do the right thing with that,
I'll be shocked.


> To perform full vectorization image should be regrouped in memory channels.
> For example many computer vision algorithms work better with each channel
> regrouped in memory channels.

Yes, but many (most) applications just want to call a function, and
don't want to rearrange their data. Most people aren't researchers
writing high-tech computer vision software ;)
Also, most images in realtime software are textures, which are usually
compressed in some way, and don't generally work well split into
channels; that would be multiple textures (for each plane), and
multiple samples per texel (sample for each plane).
Image processing definitely needs to work without having the working
dataset pre-arranged into planes.


> Relevant issue for this type of optmizations:
> https://github.com/libmir/mir/issues/343
>
> Please comment on this issue and provide a set of functions you would like
> to be vectorised. --Ilya

I'll take a look.

Re: color lib

[Ilya Yaroshenko via Digitalmars-d]

On Sunday, 9 October 2016 at 13:18:22 UTC, Manu wrote:
On 9 October 2016 at 15:34, Ilya Yaroshenko via Digitalmars-d 
 wrote:

On Sunday, 9 October 2016 at 05:21:32 UTC, Manu wrote:


On 9 October 2016 at 14:03, Nicholas Wilson via Digitalmars-d 
 wrote:


[...]



Well the trouble is the lambda that you might give to 'map' 
won't work anymore. Operators don't work on batches, you need 
to use a completely different API, and I think that's 
unfortunate.



Could you please give an example what type of operation should 
be vectorized?


Let's consider a super simple blend:
  dest = src.rgb * src.a + dest.rgb * (1-src.alpha);


This code do not need transposition. And can be partially 
vectorised using mir.ndslice.algorithm.


To perform full vectorization image should be regrouped in memory 
channels. For example many computer vision algorithms work better 
with each channel regrouped in memory channels.


Relevant issue for this type of optmizations: 
https://github.com/libmir/mir/issues/343


Please comment on this issue and provide a set of functions you 
would like to be vectorised. --Ilya

Re: color lib

[Manu via Digitalmars-d]

On 9 October 2016 at 18:39, Nicholas Wilson via Digitalmars-d
 wrote:
> On Sunday, 9 October 2016 at 05:34:06 UTC, Ilya Yaroshenko wrote:
>>
>> On Sunday, 9 October 2016 at 05:21:32 UTC, Manu wrote:
>>>
>>> On 9 October 2016 at 14:03, Nicholas Wilson via Digitalmars-d
>>>  wrote:

 [...]
>>>
>>>
>>> Well the trouble is the lambda that you might give to 'map' won't work
>>> anymore. Operators don't work on batches, you need to use a completely
>>> different API, and I think that's unfortunate.
>>
>>
>> Could you please give an example what type of operation should be
>> vectorized?
>
>
> anything that is able to be. Given that ElementType!InBatchesOfN are a
> static array
> of ElementType!(R), the compiler can* (assuming no branching and anything
> else that impedes vectorisation) combine most expressions into equivalent
> vector instruction.
> This approach might not work so well for colours as is but should work if we
> "transpose" the colour i.e. rgbargbargbargba ->  and then
> transpose it back.
>
> *I know this is the sufficiently intelligent compiler argument

I've been waiting for that compiler for almost 2 decades. I've shipped
17 commercial games in that time while waiting, and I had to resort to
manual code that didn't yet have an opportunity to leverage such a
sufficiently intelligent compilers awesome optimiser.
I've never seen an auto-vectoriser go ANYWHERE NEAR the sort of
intelligence required here. I've seen it start to do some good work
with arrays of floats... that's about where it ends.
Arrays of RGBA_10_10_10_2 require unpacking, and then shuffling. When
you stuff the unpack in the way, that will almost always throw an
auto-vectoriser off the scent. I'm also waiting to see saturation
expressions written in C code promote to saturating SIMD arithmetic.
Basically every operation is saturating when working with colours.

Re: color lib

[Manu via Digitalmars-d]

On 9 October 2016 at 18:25, Nicholas Wilson via Digitalmars-d
 wrote:
> On Sunday, 9 October 2016 at 05:21:32 UTC, Manu wrote:
>>
>> On 9 October 2016 at 14:03, Nicholas Wilson via Digitalmars-d
>>  wrote:
>>>
>>> How far would `r.inBatchesOf!(N)` go in terms of compiler optimisations
>>> (e.g. vectorisation) if N is a power of 2?
>>>
>>> auto inBatchesOf(size_t N,R)(R r) if(N!=0 &!R &&
>>> hasLength!R)
>>> {
>>> struct InBatchesOfN
>>> {
>>> R r;
>>> ElementType!(R)[N] batch;
>>> this(R _r)
>>> {
>>>  assert(_r.length % N ==0);// could have overloads where
>>> undefined elements == ElementType!(R).init
>>>  r = _r;
>>>  foreach( i; 0..N)
>>>  {
>>>   batch[i] = r.front;
>>>   r.popFront;
>>>  }
>>> }
>>>
>>> bool empty() { return r.empty; }
>>> auto front { return batch; }
>>> void popFront()
>>> {
>>>  foreach( i; 0..N)
>>>  {
>>>   batch[i] = r.front;
>>>   r.popFront;
>>>  }
>>> }
>>> }
>>>
>>> return InBatchesOfN(r);
>>> }
>>
>>
>> Well the trouble is the lambda that you might give to 'map' won't work
>> anymore. Operators don't work on batches, you need to use a completely
>> different API, and I think that's unfortunate.
>
>
> How?  All you need is an extra `each` e.g. r.inBatchesOf!(8).each!(a
> =>a[].map!(convertColor!RGBA8))
>
> perhaps define a helper function for it that does each + the explicit slice
> + map, but it certainly doesn't scream completely different API to me.

As you demonstrate; convertColor doesn't accept RGBA8[16], it accepts
a single RGBA8... there's no way the optimiser will be able to
magic-up an efficient inline of convertColor which works with 16
elements at a time, but I could easily write a super-fast version by
hand.

My point about the separate API is, any function that works on a
single element would need a compliment of functions that work on 'n'
elements, where 'n' is some context-specific number of elements that
suits that particular workload.
Now, that's conceivable, and it's even possible to make the magic meta
that calls these functions work out there is a batch overload and call
it if it can, but we're miles away from std.algorithm and common
ranges now.
The other issue is that every such efficient batch version would need
to be hand-written, and that sucks because there are too many
permutations.

Re: color lib

[Manu via Digitalmars-d]

On 9 October 2016 at 15:34, Ilya Yaroshenko via Digitalmars-d
 wrote:
> On Sunday, 9 October 2016 at 05:21:32 UTC, Manu wrote:
>>
>> On 9 October 2016 at 14:03, Nicholas Wilson via Digitalmars-d
>>  wrote:
>>>
>>> [...]
>>
>>
>> Well the trouble is the lambda that you might give to 'map' won't work
>> anymore. Operators don't work on batches, you need to use a completely
>> different API, and I think that's unfortunate.
>
>
> Could you please give an example what type of operation should be
> vectorized?

Even operations that don't require shuffling, eg:
  RGBA8[] a, b;
  zip(a, b).map!(e => e[0] + e[1]).copy(output);

Which I've suggested before (and Walter liked the idea), could be
sugared up by making use of the languages largely under-used array
operation syntax:
  output[] = a[] + b[]; // ie, types have overloaded addition
operators, so this array expression would be lowered to the pipeline
expression above. This would be super-useful for HEAPS of things!

Even these still need to be done in batches since colour adds are
saturating operations, and there are SIMD instructions for saturating
arithmetic, so we basically always have to do colour work in SIMD,
which means batching, and that basically ruins any chance for natural,
readable, expressions in your code. I want to find a way that we can
express these operations naturally, without having to always manually
handle the batching.

If we can get there, then I will say D is a good language for
stream-data processing.

Re: color lib

[Manu via Digitalmars-d]

On 9 October 2016 at 15:34, Ilya Yaroshenko via Digitalmars-d
 wrote:
> On Sunday, 9 October 2016 at 05:21:32 UTC, Manu wrote:
>>
>> On 9 October 2016 at 14:03, Nicholas Wilson via Digitalmars-d
>>  wrote:
>>>
>>> [...]
>>
>>
>> Well the trouble is the lambda that you might give to 'map' won't work
>> anymore. Operators don't work on batches, you need to use a completely
>> different API, and I think that's unfortunate.
>
>
> Could you please give an example what type of operation should be
> vectorized?

Let's consider a super simple blend:
  dest = src.rgb * src.a + dest.rgb * (1-src.alpha);

This is perhaps the most common blend that exists. If this is a
ubyte[4] color, which is the most common format, then to do it
efficiently, runs of 16 colors (4x ubyte[16] vectors), needs to be
rearranged into:
  ubyte[16][3] rgb = [ [RGBRGBRGBRGBRGBR], [GBRGBRGBRGBRGBRG],
[BRGBRGBRGBRGBRGB] ];
  ubyte[16][3] a = [ [AAAaaaAAAaaaAAAa], [aaAAAaaaAAAaaaAA],
[AaaaAAAaaaAAAaaa] ];
You can do this with gather loads, or with a couple of shuffle's after loading.
Then obviously do the work in this configuration.

Or you might expand it to [ [], [],
[] ], etc, depends on the work, and which expansion is
cheaper for the platform (ie, shuffling limitations).

Now, this might not look like much of a win for this blend, but as you
extend the sequence of ops, the win gets much much bigger.
Particularly so if you want to do gamma-correct stuff, which would
usually involve expanding those ubytes into floats, then doing vector
pow's and stuff like that. Either way, you need to iterate the image 4
vectors at a time.

That's the sort of batching I'm talking about. Trouble is, this work
needs to be wrapped into a function that receives inputs in batches,
like:
  RGBA8[16] doBulkBlend(RGBA8[16] buffer) { ... bulk blend code ... }

This sort of thing:
  buffer.map!(e => src.rgb * src.a).copy(output);
Super readable! Would be really nice to express, but I have no idea
how we can make that sort of thing efficient.

You could start writing this sort of thing:
  buffer.chunksOf!16.map!(e => doBulkBlend(e[0..16])).deChunk.copy(output);

Yeah, it's code... it would compile, but I consider that to be
completely obfuscated. You can't look at that and understand anything
much about what it does... so I don't think that's a good goal-post at
all.
If I showed that to a colleague, I don't think they'd be impressed. We
can't reach that point and say D is awesome for data-stream
processing... we need to go a lot further than that.

Anyway, I think this sort of thing is a minimum target. I'd like to
see how this sort of batching would integrate into ndslice nicely,
because it introduces the 'nd' iteration element... there's a heap of
challenges; element alignment, unaligned line-strides, mid-vector
slices, etc.
I can imagine certain filter algorithms that work on 2d slices
('blocks') rather than 1d slices like the example above. What if the
image is rotated or transposed? How can applying a per-pixel operation
on a buffer iterate the memory-linear fashion even though it's working
in batched of elements?

I haven't sat and tried plugging this into ndslice much yet. Haven't
had enough time, and I really wanted to get colour to a point I'm
happy with.

Re: color lib

[Ilya Yaroshenko via Digitalmars-d]

On Sunday, 9 October 2016 at 08:39:57 UTC, Nicholas Wilson wrote:
On Sunday, 9 October 2016 at 05:34:06 UTC, Ilya Yaroshenko 
wrote:

On Sunday, 9 October 2016 at 05:21:32 UTC, Manu wrote:
On 9 October 2016 at 14:03, Nicholas Wilson via Digitalmars-d 
 wrote:

[...]


Well the trouble is the lambda that you might give to 'map' 
won't work anymore. Operators don't work on batches, you need 
to use a completely different API, and I think that's 
unfortunate.


Could you please give an example what type of operation should 
be vectorized?


anything that is able to be. Given that 
ElementType!InBatchesOfN are a static array
of ElementType!(R), the compiler can* (assuming no branching 
and anything else that impedes vectorisation) combine most 
expressions into equivalent vector instruction.
This approach might not work so well for colours as is but 
should work if we "transpose" the colour i.e. rgbargbargbargba 
->  and then transpose it back.


*I know this is the sufficiently intelligent compiler argument


static foreach can help for static arrays. For example, ndslice 
uses static foreach a lot. mir.ndslice.algorithm allows to 
perform vectorized operations. Some conversion algorithms for 
ndslices will be added to Mir or DCV. Please open a pool request 
or fill an issue.

Re: color lib

[Nicholas Wilson via Digitalmars-d]

On Sunday, 9 October 2016 at 08:25:40 UTC, Nicholas Wilson wrote:
How?  All you need is an extra `each` e.g. 
r.inBatchesOf!(8).each!(a =>a[].map!(convertColor!RGBA8))


perhaps define a helper function for it that does each + the 
explicit slice + map, but it certainly doesn't scream 
completely different API to me.


Ha, realised I went full circle. Still might be useful if the 
compiler is able to use the fact that the range is a multiple of 
N (particularly is N is a power of 2).
 Are you able to apply arbitrary attributes to the delegate 
passed (i.e. things like @fastmath)?

Re: color lib

[Nicholas Wilson via Digitalmars-d]

On Sunday, 9 October 2016 at 05:34:06 UTC, Ilya Yaroshenko wrote:

On Sunday, 9 October 2016 at 05:21:32 UTC, Manu wrote:
On 9 October 2016 at 14:03, Nicholas Wilson via Digitalmars-d 
 wrote:

[...]


Well the trouble is the lambda that you might give to 'map' 
won't work anymore. Operators don't work on batches, you need 
to use a completely different API, and I think that's 
unfortunate.


Could you please give an example what type of operation should 
be vectorized?


anything that is able to be. Given that ElementType!InBatchesOfN 
are a static array
of ElementType!(R), the compiler can* (assuming no branching and 
anything else that impedes vectorisation) combine most 
expressions into equivalent vector instruction.
This approach might not work so well for colours as is but should 
work if we "transpose" the colour i.e. rgbargbargbargba -> 
 and then transpose it back.


*I know this is the sufficiently intelligent compiler argument

Re: color lib

[Nicholas Wilson via Digitalmars-d]

On Sunday, 9 October 2016 at 05:21:32 UTC, Manu wrote:
On 9 October 2016 at 14:03, Nicholas Wilson via Digitalmars-d 
 wrote:
How far would `r.inBatchesOf!(N)` go in terms of compiler 
optimisations

(e.g. vectorisation) if N is a power of 2?

auto inBatchesOf(size_t N,R)(R r) if(N!=0 &!R &&  
hasLength!R)

{
struct InBatchesOfN
{
R r;
ElementType!(R)[N] batch;
this(R _r)
{
 assert(_r.length % N ==0);// could have overloads 
where

undefined elements == ElementType!(R).init
 r = _r;
 foreach( i; 0..N)
 {
  batch[i] = r.front;
  r.popFront;
 }
}

bool empty() { return r.empty; }
auto front { return batch; }
void popFront()
{
 foreach( i; 0..N)
 {
  batch[i] = r.front;
  r.popFront;
 }
}
}

return InBatchesOfN(r);
}


Well the trouble is the lambda that you might give to 'map' 
won't work anymore. Operators don't work on batches, you need 
to use a completely different API, and I think that's 
unfortunate.


How?  All you need is an extra `each` e.g. 
r.inBatchesOf!(8).each!(a =>a[].map!(convertColor!RGBA8))


perhaps define a helper function for it that does each + the 
explicit slice + map, but it certainly doesn't scream completely 
different API to me.

Re: color lib

[Ilya Yaroshenko via Digitalmars-d]

On Sunday, 9 October 2016 at 05:21:32 UTC, Manu wrote:
On 9 October 2016 at 14:03, Nicholas Wilson via Digitalmars-d 
 wrote:

[...]


Well the trouble is the lambda that you might give to 'map' 
won't work anymore. Operators don't work on batches, you need 
to use a completely different API, and I think that's 
unfortunate.


Could you please give an example what type of operation should be 
vectorized?

Re: color lib

[Manu via Digitalmars-d]

On 9 October 2016 at 14:03, Nicholas Wilson via Digitalmars-d
 wrote:
> On Saturday, 8 October 2016 at 13:06:42 UTC, Manu wrote:
>>
>> Oh no, you too? >_<
>> Incidentally, have you had a geez over the core API? An efficient API
>> will emerge when we work out how to work batched operations into
>> ranges...
>
>
> How far would `r.inBatchesOf!(N)` go in terms of compiler optimisations
> (e.g. vectorisation) if N is a power of 2?
>
> auto inBatchesOf(size_t N,R)(R r) if(N!=0 &!R &&  hasLength!R)
> {
> struct InBatchesOfN
> {
> R r;
> ElementType!(R)[N] batch;
> this(R _r)
> {
>  assert(_r.length % N ==0);// could have overloads where
> undefined elements == ElementType!(R).init
>  r = _r;
>  foreach( i; 0..N)
>  {
>   batch[i] = r.front;
>   r.popFront;
>  }
> }
>
> bool empty() { return r.empty; }
> auto front { return batch; }
> void popFront()
> {
>  foreach( i; 0..N)
>  {
>   batch[i] = r.front;
>   r.popFront;
>  }
> }
> }
>
> return InBatchesOfN(r);
> }

Well the trouble is the lambda that you might give to 'map' won't work
anymore. Operators don't work on batches, you need to use a completely
different API, and I think that's unfortunate.

Re: color lib

[Nicholas Wilson via Digitalmars-d]

On Saturday, 8 October 2016 at 13:06:42 UTC, Manu wrote:

Oh no, you too? >_<
Incidentally, have you had a geez over the core API? An 
efficient API

will emerge when we work out how to work batched operations into
ranges...


How far would `r.inBatchesOf!(N)` go in terms of compiler 
optimisations (e.g. vectorisation) if N is a power of 2?


auto inBatchesOf(size_t N,R)(R r) if(N!=0 &!R &&  
hasLength!R)

{
struct InBatchesOfN
{
R r;
ElementType!(R)[N] batch;
this(R _r)
{
 assert(_r.length % N ==0);// could have overloads 
where undefined elements == ElementType!(R).init

 r = _r;
 foreach( i; 0..N)
 {
  batch[i] = r.front;
  r.popFront;
 }
}

bool empty() { return r.empty; }
auto front { return batch; }
void popFront()
{
 foreach( i; 0..N)
 {
  batch[i] = r.front;
  r.popFront;
 }
}
}

return InBatchesOfN(r);
}

Re: color lib

[Manu via Digitalmars-d]

On 8 October 2016 at 23:28, Ilya Yaroshenko via Digitalmars-d
 wrote:
> On Saturday, 8 October 2016 at 13:06:42 UTC, Manu wrote:
>>
>> On 7 October 2016 at 18:09, Ethan Watson via Digitalmars-d
>>  wrote:
>>>
>>> On Friday, 7 October 2016 at 01:57:06 UTC, Manu wrote:


 Regarding 'Linear.No'... yeah... I dunno. I've had this argument before.
 I really hate that pattern. If it's required, I'll do it
>>>
>>>
>>>
>>> At least as far as readability goes, explicit parameterisation lets you
>>> understand the invocation at a glance rather than already knowing the actual
>>> name of the parameter or having to go elsewhere in code to see the
>>> prototype. For a library, I'd favor readability.
>>
>>
>> Oh no, you too? >_<
>> Incidentally, have you had a geez over the core API? An efficient API
>> will emerge when we work out how to work batched operations into
>> ranges...
>
>
> I like true/false here :-)

I like this guy ;)

Re: color lib

[Ilya Yaroshenko via Digitalmars-d]

On Saturday, 8 October 2016 at 13:06:42 UTC, Manu wrote:
On 7 October 2016 at 18:09, Ethan Watson via Digitalmars-d 
 wrote:

On Friday, 7 October 2016 at 01:57:06 UTC, Manu wrote:


Regarding 'Linear.No'... yeah... I dunno. I've had this 
argument before. I really hate that pattern. If it's 
required, I'll do it



At least as far as readability goes, explicit parameterisation 
lets you understand the invocation at a glance rather than 
already knowing the actual name of the parameter or having to 
go elsewhere in code to see the prototype. For a library, I'd 
favor readability.


Oh no, you too? >_<
Incidentally, have you had a geez over the core API? An 
efficient API

will emerge when we work out how to work batched operations into
ranges...


I like true/false here :-)

Re: color lib

[Manu via Digitalmars-d]

On 7 October 2016 at 18:09, Ethan Watson via Digitalmars-d
 wrote:
> On Friday, 7 October 2016 at 01:57:06 UTC, Manu wrote:
>>
>> Regarding 'Linear.No'... yeah... I dunno. I've had this argument before.
>> I really hate that pattern. If it's required, I'll do it
>
>
> At least as far as readability goes, explicit parameterisation lets you
> understand the invocation at a glance rather than already knowing the actual
> name of the parameter or having to go elsewhere in code to see the
> prototype. For a library, I'd favor readability.

Oh no, you too? >_<
Incidentally, have you had a geez over the core API? An efficient API
will emerge when we work out how to work batched operations into
ranges...

Re: color lib

[Ethan Watson via Digitalmars-d]

On Friday, 7 October 2016 at 01:57:06 UTC, Manu wrote:
Regarding 'Linear.No'... yeah... I dunno. I've had this 
argument before.

I really hate that pattern. If it's required, I'll do it


At least as far as readability goes, explicit parameterisation 
lets you understand the invocation at a glance rather than 
already knowing the actual name of the parameter or having to go 
elsewhere in code to see the prototype. For a library, I'd favor 
readability.

Re: color lib

[Ilya Yaroshenko via Digitalmars-d]

On Friday, 7 October 2016 at 01:42:08 UTC, Manu wrote:
On 7 October 2016 at 03:03, Ilya Yaroshenko via Digitalmars-d 
 wrote:

On Thursday, 6 October 2016 at 14:53:52 UTC, Manu wrote:

[...]



Could you please make `colorFromString` nothrow @nogc? Or make 
`nothrow @nogc` analog. -- Ilya


I think throwing is the precedented action in that failure 
case...

what would you suggest?
I could have an overload that returns an error or something...?

I wonder how the work towards throwing RC things is going?


No idea. The reason to do not throw Exceptions is to be able to 
use Phobos in betterC mode without DRuntime at all.

Re: color lib

[Manu via Digitalmars-d]

On 7 October 2016 at 12:55, Manu  wrote:
> On 7 October 2016 at 12:38, Manu  wrote:
>> On 7 October 2016 at 12:25, Chris Wright via Digitalmars-d
>>  wrote:
>>> On Fri, 07 Oct 2016 11:42:08 +1000, Manu via Digitalmars-d wrote:
 I think throwing is the precedented action in that failure case...
 what would you suggest?
>>>
>>> In C# 1.0, the standard pattern was to throw on errors. Later, they
>>> revised their preferred mechanism and started offering methods like:
>>>
>>>   static bool TryParse(string str, out DateTime dt) {...}
>>
>> Indeed, but what should WE do?
>
> I've rolled with:
>   bool colorFromString(Color = RGB8)(scope const(char)[] str, out
> Color color) pure nothrow @safe @nogc

More updates and doco tweaks pushed.

Re: color lib

[Manu via Digitalmars-d]

On 7 October 2016 at 12:38, Manu  wrote:
> On 7 October 2016 at 12:25, Chris Wright via Digitalmars-d
>  wrote:
>> On Fri, 07 Oct 2016 11:42:08 +1000, Manu via Digitalmars-d wrote:
>>> I think throwing is the precedented action in that failure case...
>>> what would you suggest?
>>
>> In C# 1.0, the standard pattern was to throw on errors. Later, they
>> revised their preferred mechanism and started offering methods like:
>>
>>   static bool TryParse(string str, out DateTime dt) {...}
>
> Indeed, but what should WE do?

I've rolled with:
  bool colorFromString(Color = RGB8)(scope const(char)[] str, out
Color color) pure nothrow @safe @nogc

Re: color lib

[Manu via Digitalmars-d]

On 7 October 2016 at 12:25, Chris Wright via Digitalmars-d
 wrote:
> On Fri, 07 Oct 2016 11:42:08 +1000, Manu via Digitalmars-d wrote:
>> I think throwing is the precedented action in that failure case...
>> what would you suggest?
>
> In C# 1.0, the standard pattern was to throw on errors. Later, they
> revised their preferred mechanism and started offering methods like:
>
>   static bool TryParse(string str, out DateTime dt) {...}

Indeed, but what should WE do?

Re: color lib

[Manu via Digitalmars-d]

On 7 October 2016 at 11:42, Manu  wrote:
> On 7 October 2016 at 03:03, Ilya Yaroshenko via Digitalmars-d
>  wrote:
>> On Thursday, 6 October 2016 at 14:53:52 UTC, Manu wrote:
>>>
>>> I've done another pass incorporating prior feedback, mostly focusing on
>>> documentation.
>>>
>>>
>>> http://dtest.thecybershadow.net/artifact/website-b6e2e44dd40dd7c70eb45829c02060b99ae3937b-57272ccdf902fa3f0c050d522129f2be/web/library-prerelease/std/experimental/color.html
>>>
>>> Can interested parties please give it another once-over and add
>>> further comments?
>>> How can I get this to a point where people would like to see it in phobos?
>>>
>>> Repo: https://github.com/TurkeyMan/color
>>> PR: https://github.com/dlang/phobos/pull/2845
>>
>>
>> Could you please make `colorFromString` nothrow @nogc? Or make `nothrow
>> @nogc` analog. -- Ilya
>
> I think throwing is the precedented action in that failure case...
> what would you suggest?
> I could have an overload that returns an error or something...?

Problem with overloading is that the function type only differs by
return type... is there a phobos convention for function naming where
a parallel nothrow @nogc version is also supplied?

Re: color lib

[Chris Wright via Digitalmars-d]

On Fri, 07 Oct 2016 11:42:08 +1000, Manu via Digitalmars-d wrote:
> I think throwing is the precedented action in that failure case...
> what would you suggest?

In C# 1.0, the standard pattern was to throw on errors. Later, they 
revised their preferred mechanism and started offering methods like:

  static bool TryParse(string str, out DateTime dt) {...}

Re: color lib

[Manu via Digitalmars-d]

On 7 October 2016 at 04:48, Random D user via Digitalmars-d
 wrote:
> On Thursday, 6 October 2016 at 14:53:52 UTC, Manu wrote:
>>
>> I've done another pass incorporating prior feedback, mostly focusing on
>> documentation.
>
>
> Just a quick minor comment on:
> A8  RGB!("a",ubyte,false,0)  8 bit alpha-only color type.
> -->
> Reads like, "False what ???". Also "What is 0 ???".
> -->
> How about RGB!("a", ubyte, Linear.No, Colorspace.sRGB) or something like
> that,
> since there's going to be a list of these in the docs.
>
> What does colorspace 0 mean actually? (AdobeRGB??? i.e. first from
> colorspace enum according the docs).

It's kind of upsetting that the docs don't show the default arg as the
enum it is...
The struct is defined:
  struct RGB(string components_, ComponentType_, bool linear_ = false,
RGBColorSpace colorSpace_ = RGBColorSpace.sRGB)

I don't know why the docs translate that to be '0'? Why not just omit
the default args from the docs as they are in the code:
  alias A8 = RGB!("a", ubyte);

??

It also seems the docs have rearranged the order of the enum... seriously?
Why wouldn't that retain the order the programmer specified?

Regarding 'Linear.No'... yeah... I dunno. I've had this argument before.
I really hate that pattern. If it's required, I'll do it, otherwise
I'm really not enthusiastic about it personally.
It just seems like pointless bloat in symbol names to me, is if they
weren't already bloated enough.

Re: color lib

[Manu via Digitalmars-d]

On 7 October 2016 at 03:03, Ilya Yaroshenko via Digitalmars-d
 wrote:
> On Thursday, 6 October 2016 at 14:53:52 UTC, Manu wrote:
>>
>> I've done another pass incorporating prior feedback, mostly focusing on
>> documentation.
>>
>>
>> http://dtest.thecybershadow.net/artifact/website-b6e2e44dd40dd7c70eb45829c02060b99ae3937b-57272ccdf902fa3f0c050d522129f2be/web/library-prerelease/std/experimental/color.html
>>
>> Can interested parties please give it another once-over and add
>> further comments?
>> How can I get this to a point where people would like to see it in phobos?
>>
>> Repo: https://github.com/TurkeyMan/color
>> PR: https://github.com/dlang/phobos/pull/2845
>
>
> Could you please make `colorFromString` nothrow @nogc? Or make `nothrow
> @nogc` analog. -- Ilya

I think throwing is the precedented action in that failure case...
what would you suggest?
I could have an overload that returns an error or something...?

I wonder how the work towards throwing RC things is going?

Re: color lib

[Random D user via Digitalmars-d]

On Thursday, 6 October 2016 at 14:53:52 UTC, Manu wrote:
I've done another pass incorporating prior feedback, mostly 
focusing on documentation.


Just a quick minor comment on:
A8  RGB!("a",ubyte,false,0)  8 bit alpha-only color type.
-->
Reads like, "False what ???". Also "What is 0 ???".
-->
How about RGB!("a", ubyte, Linear.No, Colorspace.sRGB) or 
something like that,

since there's going to be a list of these in the docs.

What does colorspace 0 mean actually? (AdobeRGB??? i.e. first 
from colorspace enum according the docs).

Re: color lib

[Ilya Yaroshenko via Digitalmars-d]

On Thursday, 6 October 2016 at 14:53:52 UTC, Manu wrote:
I've done another pass incorporating prior feedback, mostly 
focusing on documentation.


http://dtest.thecybershadow.net/artifact/website-b6e2e44dd40dd7c70eb45829c02060b99ae3937b-57272ccdf902fa3f0c050d522129f2be/web/library-prerelease/std/experimental/color.html

Can interested parties please give it another once-over and add
further comments?
How can I get this to a point where people would like to see it 
in phobos?


Repo: https://github.com/TurkeyMan/color
PR: https://github.com/dlang/phobos/pull/2845


Could you please make `colorFromString` nothrow @nogc? Or make 
`nothrow @nogc` analog. -- Ilya