Bug#741640: packaging vcmi

[Johannes Schauer]

Hi,

I recently opened this ITP https://bugs.debian.org/741640 because I wanted to
get VCMI into Debian. The game has been suggested for inclusion here
https://wiki.debian.org/Games/Suggested#VCMI and became much more interesting
recently with campaign support.

I'm not a DD so I will need a sponsor for vcmi. It would be nice if I could
join the pkg-games team and maintain vcmi in git on alioth. You can see my
first attempts of packaging it here:

http://mentors.debian.net/package/vcmi

Upstream ships their own ./debian directory because they maintain a ubuntu ppa:
https://launchpad.net/~saven-ivan/+archive/vcmi I solved that problem with a
Files-Excluded directive in debian/copyright.

My packaging is based on theirs but does some cleanup and fixes a couple of
lintian problems. I reported some of the remaining problems (spelling, man
pages and missing key in *.desktop files) to upstream and they will be fixed in
the next upstream version. The current packaging carries a patch to make it use
avconv instead of ffmpeg but upstream will include functionality that can
handle both systems in the next release, so the patch can be dropped then.

I so far failed to solve the lintian hardening warnings. I can't figure out why
the vcmiclient binary does not generate them while other binaries which are
built by cmake in the exact same way do.

The game needs bitmaps, animations, texts, sounds and videos from the original
game (though sounds and videos are optional). There exists a big modding
community for vcmi but the programs that are so far used by artists were
created many years ago, only run on windows (some work under wine) and do not
come with any source code. Therefore, I created some python code which can
unpack the game files into commonly editable formats (png and json) and repack
them into the proprietary formats.

Here is the code and two videos demonstrating the successful unpacking,
modification and repacking of all graphical elements:

https://github.com/josch/lodextract
https://mister-muffin.de/p/kpyL.ogg
https://mister-muffin.de/p/hSF6.ogg

The first video shows all graphics replaced by colored rectangles. Such a
graphics set would probably be free (only the width and height of the original
artwork remain) but unplayable. The second video shows all colors other than
transparency replaced by a solid color. This allows to differentiate between
different objects because their shape remains but this kind of derivative work
is most likely as non-free as the original graphics (though it shows that the
packing code works correctly). The colored rectangle version could be the basis
of a free graphics pack. Creating such a replacement is surely a multi-year
effort and might never happen but it made me more motivated to package vcmi for
Debian if there was a DFSG compatible way to replace the proprietary assets
with free ones. I'm not sure yet where to put the unpacking and repacking
scripts. Maybe they can be integrated upstream (they have the same license).

cheers, josch


--
To UNSUBSCRIBE, email to debian-wnpp-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20140317153554.1302.95390@hoothoot

Bug#741640: packaging vcmi

[Markus Koschany]

Hi josch and welcome,

On 17.03.2014 16:35, Johannes Schauer wrote:
[...]
> I'm not a DD so I will need a sponsor for vcmi. It would be nice if I could
> join the pkg-games team and maintain vcmi in git on alioth. You can see my
> first attempts of packaging it here:

That would be great. You just need to register an account on alioth and
ask for joining the team there.

[...]
> I so far failed to solve the lintian hardening warnings. I can't figure out 
> why
> the vcmiclient binary does not generate them while other binaries which are
> built by cmake in the exact same way do.

I tried to build your package in a cowbuilder environment but the build
is stuck at 7% and seems to require more than 4 GB RAM. However I can
see that hardening flags are passed to the compiler, so perhaps the
lintian warnings might be a false-positive. What did blhc report for
your .build file?


> The game needs bitmaps, animations, texts, sounds and videos from the original
> game (though sounds and videos are optional). There exists a big modding
> community for vcmi but the programs that are so far used by artists were
> created many years ago, only run on windows (some work under wine) and do not
> come with any source code. Therefore, I created some python code which can
> unpack the game files into commonly editable formats (png and json) and repack
> them into the proprietary formats.

At first glance that means vcmi must be uploaded to contrib. We had a
recent discussion about game engines and interpreters and we seemed to
agree that those can be in main as long as there is free game data
available either in Debian or somewhere else. That means if you can
provide enough free content for demonstration purposes, then vcmi might
be even suitable for main.

Anyway it is a good idea to document your efforts to create free content
somewhere and write down what kind of data is missing to achieve this
goal. That makes it easier for other people who want to work on it too.


> Here is the code and two videos demonstrating the successful unpacking,
> modification and repacking of all graphical elements:
> 
> https://github.com/josch/lodextract
> https://mister-muffin.de/p/kpyL.ogg
> https://mister-muffin.de/p/hSF6.ogg

Nice feat and a great idea to provide these scripts. I'd suggest to ask
upstream for including them. As noted above, if you are able to create
enough free content that demonstrates the capabilities of vcmi, it might
be even suitable for main.

Regards,

Markus




signature.asc
Description: OpenPGP digital signature

Bug#741640: packaging vcmi

[Johannes Schauer]

Hi,

Quoting Markus Koschany (2014-03-17 17:25:55)
> That would be great. You just need to register an account on alioth and ask
> for joining the team there.

okay, done :)

> I tried to build your package in a cowbuilder environment but the build is
> stuck at 7% and seems to require more than 4 GB RAM.

And also insane amounts of harddisk space :(

> However I can see that hardening flags are passed to the compiler, so perhaps
> the lintian warnings might be a false-positive. What did blhc report for your
> .build file?

I didnt know about blhc (for some reason no mentioned in
https://wiki.debian.org/Hardening). Handy tool! I attached its output. There
indeed seem to be some -fPIE flags missing for a couple of things. I have to
dive deeper into the cmake build to figure out how to make sure that they get
passed but thanks to blhc I can now better check whether or not things get
built the right way.

> At first glance that means vcmi must be uploaded to contrib. We had a recent
> discussion about game engines and interpreters and we seemed to agree that
> those can be in main as long as there is free game data available either in
> Debian or somewhere else. That means if you can provide enough free content
> for demonstration purposes, then vcmi might be even suitable for main.

My original plan was to replace all graphics by colored rectangles with text
inside. The text would be the name of the object and thus one would know what
object the rectangle represents. The result would look similar to the first
video in my last email which only has the text missing in some objects. This
way, the game would at least be technically playable - even though it would not
be fun because it would look ugly. But at least one would know which rectangle
represented what.  I wanted to do it like that because I'm much more motivated
about having a package in main than in contrib. Though when I approached
upstream about this solution for getting vcmi into Debian main instead of
contrib I met only criticism. In their opinion it is not worth having vcmi in
main if that means that the default free graphics are crappy.  Since I do not
want to offend upstream and respect their opinion, I stopped my efforts of
creating auto-generated rectangle graphics and instead prepared the package for
inclusion in contrib.

cheers, josch
CXXFLAGS missing (-fPIE): cd /home/josch/debpkg/vcmi-0.95/obj-x86_64-linux-gnu/client && /usr/bin/c++   -DM_BIN_DIR=\"/usr/games\" -DM_DATA_DIR=\"/usr/share/vcmi\" -DM_LIB_DIR=\"/usr/lib/x86_64-linux-gnu/vcmi\" -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2  -std=c++0x -Wall -Wextra -Wpointer-arith -Wno-switch -Wno-sign-compare -Wno-unused-parameter -Wuninitialized -Wno-overloaded-virtual  -O2 -g -DNDEBUG -I/home/josch/debpkg/vcmi-0.95 -I/home/josch/debpkg/vcmi-0.95/client -I/home/josch/debpkg/vcmi-0.95/lib -I/usr/include/SDL -include "/home/josch/debpkg/vcmi-0.95/obj-x86_64-linux-gnu/client/cotire/vcmiclient_CXX_prefix.hxx" -Winvalid-pch  -o CMakeFiles/vcmiclient.dir/StdInc.cpp.o -c /home/josch/debpkg/vcmi-0.95/client/StdInc.cpp
CXXFLAGS missing (-fPIE): cd /home/josch/debpkg/vcmi-0.95/obj-x86_64-linux-gnu/client && /usr/bin/c++   -DM_BIN_DIR=\"/usr/games\" -DM_DATA_DIR=\"/usr/share/vcmi\" -DM_LIB_DIR=\"/usr/lib/x86_64-linux-gnu/vcmi\" -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2  -std=c++0x -Wall -Wextra -Wpointer-arith -Wno-switch -Wno-sign-compare -Wno-unused-parameter -Wuninitialized -Wno-overloaded-virtual  -O2 -g -DNDEBUG -I/home/josch/debpkg/vcmi-0.95 -I/home/josch/debpkg/vcmi-0.95/client -I/home/josch/debpkg/vcmi-0.95/lib -I/usr/include/SDL -include "/home/josch/debpkg/vcmi-0.95/obj-x86_64-linux-gnu/client/cotire/vcmiclient_CXX_prefix.hxx" -Winvalid-pch  -o CMakeFiles/vcmiclient.dir/__/CCallback.cpp.o -c /home/josch/debpkg/vcmi-0.95/CCallback.cpp
CXXFLAGS missing (-fPIE): cd /home/josch/debpkg/vcmi-0.95/obj-x86_64-linux-gnu/client && /usr/bin/c++   -DM_BIN_DIR=\"/usr/games\" -DM_DATA_DIR=\"/usr/share/vcmi\" -DM_LIB_DIR=\"/usr/lib/x86_64-linux-gnu/vcmi\" -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2  -std=c++0x -Wall -Wextra -Wpointer-arith -Wno-switch -Wno-sign-compare -Wno-unused-parameter -Wuninitialized -Wno-overloaded-virtual  -O2 -g -DNDEBUG -I/home/josch/debpkg/vcmi-0.95 -I/home/josch/debpkg/vcmi-0.95/client -I/home/josch/debpkg/vcmi-0.95/lib -I/usr/include/SDL -include "/home/josch/debpkg/vcmi-0.95/obj-x86_64-linux-gnu/client/cotire/vcmiclient_CXX_prefix.hxx" -Winvalid-pch  -o CMakeFiles/vcmiclient.dir/battle/CBattleInterface.cpp.o -c /home/josch/debpkg/vcmi-0.95/client/battle/CBattleInterface.cpp
CXXFLAGS missing (-fPIE): cd /home/josch/debpkg/vcmi-0.95/obj-x86_64-linux-gnu/client && /usr/bin/c++   -DM_BIN_DIR=\"/usr/games\" -DM_DATA_DIR=\"/usr/share/vcmi\" -DM_LIB_DIR=\"/usr/lib/x86_64-linux-gnu/vcmi\" -g -O2

Bug#741640: packaging vcmi

[Markus Koschany]

On 17.03.2014 17:48, Johannes Schauer wrote:
[...]
> I didnt know about blhc (for some reason no mentioned in
> https://wiki.debian.org/Hardening). Handy tool! I attached its output. There
> indeed seem to be some -fPIE flags missing for a couple of things. I have to
> dive deeper into the cmake build to figure out how to make sure that they get
> passed but thanks to blhc I can now better check whether or not things get
> built the right way.

You can try to pass CFLAGS, CXXFLAGS and LDFLAGS to CMake by overriding
dh_auto_configure with e.g.

CFLAGS:=$(shell dpkg-buildflags --get CFLAGS)
CXXFLAGS:=$(shell dpkg-buildflags --get CXXFLAGS)
#CFLAGS+=$(CXXFLAGS)

dh_auto_configure -- \
-DCMAKE_C_FLAGS="$(CFLAGS)" \
-DCMAKE_CXX_FLAGS="$(CXXLAGS)"

You can also pass CXXFLAGS to CFLAGS or vice versa. It depends mostly on
upstream's (broken) build system. Sometimes they forget to add variables
for custom CFLAGS or CXXFLAGS to CMakelist.txt. Your other option is
then to patch CMakelist.txt and forward the changes upstream.

[...]

> My original plan was to replace all graphics by colored rectangles with text
> inside. The text would be the name of the object and thus one would know what
> object the rectangle represents. The result would look similar to the first
> video in my last email which only has the text missing in some objects. This
> way, the game would at least be technically playable - even though it would 
> not
> be fun because it would look ugly. But at least one would know which rectangle
> represented what.  I wanted to do it like that because I'm much more motivated
> about having a package in main than in contrib.

Excellent attitude. You don't have to create a complete free game. The
idea is to provide users of the engine with enough material and
information, so that they are able to create more content and improve
the current status. It's a common problem that people develop some sort
of free engine but don't provide free game data as well. However I
believe we are not only here to package games but also to provide tools
for game development. Free data packages are a contribution to these
efforts.

> Though when I approached
> upstream about this solution for getting vcmi into Debian main instead of
> contrib I met only criticism. In their opinion it is not worth having vcmi in
> main if that means that the default free graphics are crappy.  Since I do not
> want to offend upstream and respect their opinion, I stopped my efforts of
> creating auto-generated rectangle graphics and instead prepared the package 
> for
> inclusion in contrib.

It's a simple packaging issue. VCMI is the free engine and Heroes of
Might and Magic 3 is the (non-free) game. The game should depend on the
engine not vice versa. That leaves you with the following options:

If Heroes of Might and Magic is non-free but distributable, you can
prepare a package for non-free which in turn depends on vcmi.

If the game is non-free and non-distributable game-data-packager is your
tool of choice.

You can avoid any criticism when you create a distinct vcmi-data package
that provides enough data for demonstrating the basic gameplay and
perhaps some documentation how people can built upon this work. Then
vcmi-data (main) depends on vcmi (main) and Heroes3 (non-free) also
depends on vcmi (main). If the package description is meaningful enough,
those who want to play Heroes3 have to install the non-free data package
and everyone else can start developing a free replacement.
It would be great if upstream developed some free tools so that others
could create more free content for their engine.

Regards,

Markus














signature.asc
Description: OpenPGP digital signature

Bug#741640: packaging vcmi

[Johannes Schauer]

Hi,

Quoting Markus Koschany (2014-03-17 19:45:04)
> You can try to pass CFLAGS, CXXFLAGS and LDFLAGS to CMake by overriding
> dh_auto_configure with e.g.
> 
> CFLAGS:=$(shell dpkg-buildflags --get CFLAGS)
> CXXFLAGS:=$(shell dpkg-buildflags --get CXXFLAGS)
> #CFLAGS+=$(CXXFLAGS)
> 
> dh_auto_configure -- \
> -DCMAKE_C_FLAGS="$(CFLAGS)" \
> -DCMAKE_CXX_FLAGS="$(CXXLAGS)"
> 
> You can also pass CXXFLAGS to CFLAGS or vice versa. It depends mostly on
> upstream's (broken) build system. Sometimes they forget to add variables
> for custom CFLAGS or CXXFLAGS to CMakelist.txt. Your other option is
> then to patch CMakelist.txt and forward the changes upstream.

after having tried that and other things without avail I tried using

export DEB_BUILD_HARDENING=1

in debian/rules. This enables hardening-wrapper and should thus work no matter
what goes wrong in the build system. Interestingly the problems remain. I tried
running hardening-check manually on the generated binary:

$ hardening-check ./obj-x86_64-linux-gnu/server/vcmiserver
./obj-x86_64-linux-gnu/server/vcmiserver:
 Position Independent Executable: yes
 Stack protected: yes
 Fortify Source functions: no, only unprotected functions found!
 Read-only relocations: yes
 Immediate binding: yes

The output is not only unexpected because hardening-wrapper was used but also
because investigating the build log reveals that there was no call to c++
without the -D_FORTIFY_SOURCE=2 flag passed for example. Still, no functions
are fortified. Weird. Possibly I should ask on the debian mentors list about
this or do you have another idea?

> It's a simple packaging issue. VCMI is the free engine and Heroes of Might
> and Magic 3 is the (non-free) game. The game should depend on the engine not
> vice versa.

This is an interesting way to look at things. I thought since the engine is of
not much use without assets, should the engine not depend on something to
become useful? If it doesnt need to depend on the assets, then there is no
reason for it to be in contrib instead of main.

> That leaves you with the following options:
> 
> If Heroes of Might and Magic is non-free but distributable, you can
> prepare a package for non-free which in turn depends on vcmi.
> 
> If the game is non-free and non-distributable game-data-packager is your
> tool of choice.

Heroes of might and magic 3 is non-free and non-distributable. One good thing
is, that it can be obtain without DRM on gog.com without much hassle.

I had a look at game-data-packager earlier today. Surely I can write up a patch
for it to support building a heroes of might and magic 3 package which vcmi can
then use. So far, upstream provides a bash script called vcmibuilder to do the
downloading and extracting of resources.

> You can avoid any criticism when you create a distinct vcmi-data package that
> provides enough data for demonstrating the basic gameplay and perhaps some
> documentation how people can built upon this work. Then vcmi-data (main)
> depends on vcmi (main) and Heroes3 (non-free) also depends on vcmi (main).

Okay so this is how it would look like when the assets depend on the engine.
Interesting.

Then what I might want to do would be to create a very tiny mod for vcmi which
includes only one town, one hero, one unit and only a handful of map objects.
The graphics for all of this can be poorly drawn as it's only meant for
demonstration purposes.

> If the package description is meaningful enough, those who want to play
> Heroes3 have to install the non-free data package and everyone else can start
> developing a free replacement.

Sounds like a plan!

> It would be great if upstream developed some free tools so that others could
> create more free content for their engine.

Indeed. But they are also very open to patches and improvements, so if somebody
else creates something that integrates well, I'm sure it will be added.

Thanks for your help!

cheers, josch


--
To UNSUBSCRIBE, email to debian-wnpp-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20140318000327.1302.38092@hoothoot

Bug#741640: packaging vcmi

[Johannes Schauer]

Hi again,

Quoting Johannes Schauer (2014-03-18 01:03:27)
> after having tried that and other things without avail I tried using
> 
> export DEB_BUILD_HARDENING=1
> 
> in debian/rules. This enables hardening-wrapper and should thus work no matter
> what goes wrong in the build system. Interestingly the problems remain. I 
> tried
> running hardening-check manually on the generated binary:
> 
> $ hardening-check ./obj-x86_64-linux-gnu/server/vcmiserver
> ./obj-x86_64-linux-gnu/server/vcmiserver:
>  Position Independent Executable: yes
>  Stack protected: yes
>  Fortify Source functions: no, only unprotected functions found!
>  Read-only relocations: yes
>  Immediate binding: yes

turns out that the unprotected functions are probably false positives because
"blhc --all" shows no output at all. This means that all hardening options are
set during compilation. Here the verbose hardening-check output:

 Position Independent Executable: yes
 Stack protected: yes
 Fortify Source functions: no, only unprotected functions found!
unprotected: memset
unprotected: memmove
unprotected: poll
unprotected: memcpy
 Read-only relocations: yes
 Immediate binding: yes

This is now produced without hardening-wrapper but instead by using

export DEB_BUILD_MAINT_OPTIONS=hardening=+all

in debian/rules. Turns out that the build system is actually not broken and
passes all flags on just fine and no hackery with CMAKE_CXX_FLAGS or the like
is needed.  :)

The fixed version has been uploaded to mentors.

Now I need somebody to look over the packaging and a mentor :)

cheers, josch


--
To UNSUBSCRIBE, email to debian-wnpp-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20140318174638.1302.16415@hoothoot

Bug#741640: packaging vcmi

[Markus Koschany]

Hi,

On 18.03.2014 18:46, Johannes Schauer wrote:
[...]
> turns out that the unprotected functions are probably false positives because
> "blhc --all" shows no output at all.

Indeed. I also believe the warning is most likely a false positive.

[...]
> The fixed version has been uploaded to mentors.
> 
> Now I need somebody to look over the packaging and a mentor :)

Good luck in finding a sponsor. You can also add the game to

https://wiki.debian.org/Games/Sponsors/Queue

to increase your chances and filing a request for sponsorship bug report
is also always a good idea. I would review your package but it seems I
have to buy some new hardware first...

Cheers,

Markus



signature.asc
Description: OpenPGP digital signature

Bug#741640: packaging vcmi

[Nils Dagsson Moskopp]

Johannes Schauer  writes:

> The colored rectangle version could be the basis of a free graphics
> pack. Creating such a replacement is surely a multi-year effort

True for a beautiful replacement – so i started drawing an ugly one:

git clone http://daten.dieweltistgarnichtso.net/src/vcmi-data-libre.git

Is CC BY-SA 3.0+ free enough for Debian?

-- 
Nils Dagsson Moskopp // erlehmann



--
To UNSUBSCRIBE, email to debian-wnpp-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/87pplikp9u@dieweltistgarnichtso.net

Bug#741640: packaging vcmi

[Boris Pek]

Hi,

>>  The colored rectangle version could be the basis of a free graphics
>>  pack. Creating such a replacement is surely a multi-year effort
>
> True for a beautiful replacement – so i started drawing an ugly one:
>
> git clone http://daten.dieweltistgarnichtso.net/src/vcmi-data-libre.git
>
> Is CC BY-SA 3.0+ free enough for Debian?

Yes, if source files for these data files are available.

Best wishes,
Boris


-- 
To UNSUBSCRIBE, email to debian-wnpp-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/31395266...@web21g.yandex.ru

Bug#741640: packaging vcmi

[Nils Dagsson Moskopp]

Boris Pek  writes:

>> Is CC BY-SA 3.0+ free enough for Debian?
>
> Yes, if source files for these data files are available.

Source is SVG drawings.

-- 
Nils Dagsson Moskopp // erlehmann



-- 
To UNSUBSCRIBE, email to debian-wnpp-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/874n2tyczz@dieweltistgarnichtso.net