Re: [pygame] pygame.pixelcopy.array_to_surface(Dest, Src) not working

[Mikhail V]

On 27 December 2016 at 18:51, Ian Mallett  wrote:

>
>
>> Most interesting that 1. method with buffer write seems to be always
>> faster
>> than others, by ca. 5%. Not a big win, but still interesting...
>> And if I try it with FORTRAN order, it becomes 2 times slower!
>>
> ​I'm not sure I fully parse what you're doing here. As long as it's safe,
> copying buffers should be slightly faster since it's 1D--maybe the buffer
> API is smart enough to step in larger chunks that might potentially
> straddle a scanline, and you also have one fewer loop variable. When you
> try it with FORTRAN order, to produce a buffer of the same format would
> require an allocation and then a copy, so that's probably why it's slower.
>
> The NumPy internals
> 
> has salient things to say on this issue.
>

Good article, it explains good what I observe. And that is right, pixelcopy
seems to do much more checking, I think I saw 4 different error messages
when I was trying different inputs.


> So I would still look forward to having methods dealing with C order,
>> just to avoid writing extra transposing and full compliance
>> with default numpy notation.
>>
>> Any comments or opinions about it?
>> It would be good to know first, which of those things
>> people use more often and make some use case examples.
>>
> ​Personally, I would like C order just because it's "expected" in
> graphics*. Under this assumption, I wrote all my code e.g. looping over "y"
> first, using the buffer API for GL interop, etc. This is optimal in the C
> order every graphics programmer would expect, but in FORTRAN order, it's
> *exactly* wrong. I never profiled both options because it's a nearly
> fundamental assumption.
>

Agreed, also one who is already familiar with Numpy, or e.g. OpenCV
is already got used to [y,x] notation, so from many sides,
it would be logical to use C order for array-surface ingest.


>
> I mean, it's not terribly important. Python is not a fast language. One
> writes stuff in Python because your program running 5x/50x slower is a
> non-issue and you want the expressivity. But free perf is free, so it's a
> bit annoying.
>

Yes this is a big dilemma. There is an opinion, one should make
everything in C/+, and most industry-level products are done so.
But seriously, I can hardly see the *code* behind all those brackets,
asterisks, cryptic expressions. So for me it is almost no-go,
but as a grafic-man I am just too sensitive to these things :)

I see there is a good solution to all this: one needs a nice tool,
namely an IDE with refined simple syntax,
which produces C-compilable code. I mean if one
concentrates around this, it is not that hard to make.
Strangely I don't find much exactly related projects, one attempt
probably is SDL Basic. Cython is related, but not exactly.

This idea floats for years in air, hmm, what
could be preventing it, I wonder. I mean I don't even
consider myself a professional programmer, but it
seems, making a toy-prototype of such IDE is
a question of several months of "free evenings" job.
And actually to make such a tool, even Python+Pygame would
be totally adequate, since it does not need to be
so performant as a complex game.


Mikhail

Re: [pygame] pygame.pixelcopy.array_to_surface(Dest, Src) not working

[Ian Mallett]

On Sun, Dec 25, 2016 at 12:53 PM, Mikhail V  wrote:

> ​On Sat, Dec 24, 2016 at 5:12 PM, Mikhail V  wrote:
>>
>>> Probably there is more criterias here that I am not aware of
>>> and objective arguments to prefer "FORTRAN" order, apart
>>> from having more traditional [x,y] notation?
>>>
>> ​The argument I think comes from building/slicing matrices out of
>> (column) vectors. You see this a lot in numerical work. If the row is of
>> pointers, you can build sparse systems that reference underlying vector
>> without doing any copying (you can do this with row data instead, but then
>> you need row vectors, and that would be morally wrong). This is important
>> since building sparse systems can be very slow if you're not careful.
>>
>> I still avoid FORTRAN order because it's not mathy. E.g., the matrix
>> element "a_{0,2}" should be accessed as "a[0][2]". For an objective
>> argument, I'll note that graphics hardware--in particular VGA/VBE hardware,
>> which influenced latter standards, e.g. HDMI--is row-major, top-to-bottom
>> raster order. This has been hugely influential, and is more-or-less
>> expected today by graphics programmers. It explains everything from most
>> windowing systems today having GUI controls at the top and left, to why GL
>> takes padded scanlines as texture input.
>>
>> One way or another, at this point, changing the order in PyGame is
>> probably a bad idea (backwards compatibility and suchlike). At the very
>> least, it would needs to be deferred to a major update with breaking API
>> changes.
>>
>
> So you kind of agree, that surfarray/pixelcopy should better deal with C
> order?
>
​Definitely.
​


> I am curious, if it is worth proposing adding methods which do so.
> I agree, one should not touch the existing API.
>
> Now I have tested the performance one more time, namely
> comparing 3 variants to copy data from array to surface:
> 1. buf = Dest.get_buffer()
> buf.write(Src.tostring(), 0)
> 2. pygame.pixelcopy.array_to_surface(Dest, Src)
> 3. pygame.pixelcopy.array_to_surface(Dest, Src.T)
>
> And it turned out that I was wrong about transpose being expensive.
> Actually transpose itself does not add significant overhead. First time
> I was testing it, I did something wrong.
>
> For method 2. if I define order="FORTRAN" for original array,
> there is no difference in comparison to 3. But if I leave default (C)
> order then the performance degrades with bigger arrays
> (ca. 20% slower by 800x600 8bit array).
> So it is indeed important thing.
>
​Makes sense. For bigger arrays, caching becomes more important in the
copying, and implicit transposes of the order mean you thrash on reading.​


> Most interesting that 1. method with buffer write seems to be always
> faster
> than others, by ca. 5%. Not a big win, but still interesting...
> And if I try it with FORTRAN order, it becomes 2 times slower!
>
​I'm not sure I fully parse what you're doing here. As long as it's safe,
copying buffers should be slightly faster since it's 1D--maybe the buffer
API is smart enough to step in larger chunks that might potentially
straddle a scanline, and you also have one fewer loop variable. When you
try it with FORTRAN order, to produce a buffer of the same format would
require an allocation and then a copy, so that's probably why it's slower.

The NumPy internals

has salient things to say on this issue.

So I would still look forward to having methods dealing with C order,
> just to avoid writing extra transposing and full compliance
> with default numpy notation.
>
> Any comments or opinions about it?
> It would be good to know first, which of those things
> people use more often and make some use case examples.
>
​Personally, I would like C order just because it's "expected" in
graphics*. Under this assumption, I wrote all my code e.g. looping over "y"
first, using the buffer API for GL interop, etc. This is optimal in the C
order every graphics programmer would expect, but in FORTRAN order, it's
*exactly* wrong. I never profiled both options because it's a nearly
fundamental assumption.

I mean, it's not terribly important. Python is not a fast language. One
writes stuff in Python because your program running 5x/50x slower is a
non-issue and you want the expressivity. But free perf is free, so it's a
bit annoying.

*In the interest of fairness, it should be noted that there is an offshoot
of image processing (a subset of graphics) that might disagree. They're
very FORTRAN-y, using langs with 1-based indexing and both array orders.
They also tend to be non-CS/non-math types who work in industry, generating
appalling code.

Mikhail
>
​Ian​

Re: [pygame] A quick 1.9.3 release

[Thomas Kluyver]

On 26 December 2016 at 17:26, René Dudfield  wrote:

> @Lenard, @Thomas, please let me know your pypi usernames? I fixed the
> permissions for Paul... so he should be able to also add users on pypi.
>

My username is 'takowl'.


> I'm not sure why that mirror isn't updated... will look into it. I think
> the 'pygame' organisation on github should be used for the mirror (and
> other pygame things).


Do you control the 'pygame' org? If so, can you add extra admins? I'm
'takluyver' on Github.

I created a new org called 'pygame-org' because we didn't have access to
'pygame', but if we can get access it's better to use the obvious name.

Thanks,
Thomas

Re: [pygame] New pygame.org website

[Thomas Kluyver]

On 26 December 2016 at 18:49, René Dudfield  wrote:

> The download page can be updated with the same admin interface that
> someone managed to post news with. There's currently 6 users able to edit
> it... so I guess one of those did it. I added Paul and yourself now as
> well. See this issue on 'the website needs updates':
> https://bitbucket.org/pygame/pygame/issues/204/ Updating news, and
> community management is definitely a role which is useful, and needs to be
> shared amongst a few people. I'll track the 'make document of admin people'
> work in that issue.
>

Thanks! I don't see any admin link, though? Is there a special URL I need
to go to? My username on the site is takluyver, just in case you've given
admin permissions to another account.


> The open issue on the topic of dynamically generating the downloads page:
> https://bitbucket.org/pygame/pygame/issues/152 This needs a jinja2
> template, and a script which iterates over the 'downloads' repo. I'd
> suggest basing the html on the existing html. Whilst there are better
> layout options, that would be pretty easy to do for now, and is familiar
> for existing pygame downloads page users. There are more details to
> consider of course (like making an actual downloads repo and moving
> existing files in... and meta data for the files).
>
> There's been a couple of efforts to move the wiki into version control.
> There's even a wiki in the bitbucket version control, and scripts to
> convert the html into markup formats that bitbucket supports. (there's an
> issue open on this topic). However, in practice, way less people
> updated it without a gui (Yes, even markdown and rst is a turn off for
> quick edits). Pointing them to the bitbucket interface also meant very few
> edits. I think it was 5-10x less edits. Furthermore even requiring a login
> means 5-10x less edits (100x if you include the deluge of spam). When those
> numbers are multiplied together... it meant a tiny amount of wiki edits
> were happening. Then the bitbucket wiki started getting spammed to hell,
> but we didn't have tools to moderate it. Along with the edits dropping
> close to zero, I moved things back to the website wiki. But now there are
> pretty good web gui tools for markdown. Here's the issue on the wiki
> topic... but it needs a lot more thought on the topics of spam and ease of
> use. https://bitbucket.org/pygame/pygame/issues/153/wiki-website-
> generation-from-wiki-repo
>
> Burn out for the website admins, and website defacement because of spam
> means that a workable solution for spam prevention needs to be in place
> first for any wiki replacement.
>
> Keeping the existing game data for me, and a lot of project authors is
> very important. For me the main purpose of the website should be to help
> people making games have a community of makers. Showing your work is often
> one of the only rewards for making these projects in the first place.
> Seeing people upload their game to a website for the first time, I always
> see a grin on their face.
>

OK, it sounds like we'll need to build on the old site more than creating a
new one, then. What is available on the server (e.g. Python version)? Could
you give other people SSH access to it?

Thanks,
Thomas