Re: [crossfire] Archetypes collection changes proposal
Hello.
I've committed my (massive) changes following the 1.75 release :)
It is now possible to run the server without "packed" archetypes, faces,
treasures, ... ("assets"). But you still can "pack" everything in single files
for speed or distribution reasons :)
There may be some issues yet, but the code has been running on invidious since
a few weeks, so major bugs are hopefully ironed out.
To correctly compile and install, you must symlink "maps" in the server/lib
directory, like the "arch" one.
I think the first "make install" (or "make") will fail in "lib" on the file
"crossfire.arc", in which case from the server root do
( cd lib && make do-collect )
to generate that file.
Note that "do-collect" has been changed to use the server (and not the Perl
script), and that it isn't (yet) totally efficient as it will run the server 3
times... I will fix that at some point in the future.
If you wish to run against the raw "arch" repository:
- rm crossfire.{arc,tar,trs,face} from the "share" install directory
- maybe remove those files from the "server/lib" directory and replace them
with empty files (through "touch")
- symlink "arch" in the "share" directory, along the "maps" one
Note that you may very well have both crossfire.{arc,...} along the "arch"
symlink, it should work (that's the aim of the no collect mechanism after all
:D so please report bugs if you find any), but it may be slower to start the
server.
The explanation for the slower start is that the server will first process the
"packed" crossfire.{arc,...} files, then the subdirectories. During so, it will
update notably its archetypes definitions, which may take some time (especially
when running in debug).
Of course the disk speed is also an issue, especially for reading all the PNG
files.
The packed "client images" is a tar file, uncompressed. In the future this .tar
may be extended to contain also arches, face, and such, and maybe compressed
too.
Too many technical changes were introduced so I won't list'em all :)
The major point about archetypes, faces, animations, treasures and others
("assets") is that their "get_xxx" function will NEVER return NULL anymore. If
you need to check for existence, there is a "try_find" (or "find") function.
Function documentation is hopefully updated to reflect that.
Internally, the code works by keeping assets' pointer valid all the time and
updating the underlying structure with updated version.
So during initialisation, if an archetype references a not-yet-known treasure
list, this treasure list is internally created empty. When the treasure list
is loaded (from the .trs file), this empty treasure is updated.
After loading all assets, the server will abort if one was referenced but not
defined (ie you ask for a treasure list that is never actually defined), except
for faces which may be left undefined (reason: too many existing faces aren't
defined right now). There's a command-line flag to allow going on, in which
case
undefined items are blank (ie empty treasure list, empty face, and such) - the
server should not crash anyway, but you may have strange behaviours.
I'll of course reply to inquiries on technical questions :)
Best regards
Nicolas
signature.asc
Description: This is a digitally signed message part.
___
crossfire mailing list
crossfire@metalforge.org
http://mailman.metalforge.org/mailman/listinfo/crossfire
Re: [crossfire] Archetypes collection changes proposal
Hi Nicolas, Thank you for working on this. As we've discussed, this code is already running on Invidious and seems to be in working order. I think this is ready to be committed to trunk. Regards, Kevin ___ crossfire mailing list crossfire@metalforge.org http://mailman.metalforge.org/mailman/listinfo/crossfire
Re: [crossfire] Archetypes collection changes proposal
Hello. I think my code works nicely now :) It's been running on invidious for some days, no crash as far as I can tell. What I added as support: - of course, ability to run the server with a simple "arch" directory, without any collect to do :) - ability of the server to read .tar files with .png image files (only png for now because of loading routines using FILE* directly - will be changed ultimately) - ability to create "packed" archetypes, treasures, faces, png, and such things. Again this is done by the server directly. When everything is packed in a single file per asset, loading is quite fast :) I did not add gz support, I'm not sure it's worth it (we're talking about some MB of data only), but once .tar support is complete it shouldn't be too hard to add. What I plan to change/fix: - move "smoothface" stuff from .arc to .face files - change the "image_info" file format (removing the index and using the prefix for fallback) - probably add some extensions to various files, to not rely on the name only (artifacts, messages, attackmess, image_info, formulae notably) - while I'm at it, I'll also remove the "race reset" functionality from the races file, it is not used at all - besides, what's the point since one can simply change the archetype directly?) - also, when "packing" archetypes or treasures or ..., comments are not stored (as opposed to the current collect system). This may be fixed if I'm courageous enough ;) So, should I just commit to trunk? :D Best regards Nicolas signature.asc Description: This is a digitally signed message part. ___ crossfire mailing list crossfire@metalforge.org http://mailman.metalforge.org/mailman/listinfo/crossfire
Re: [crossfire] Archetypes collection changes proposal
Hello. For information, the patch is being tested (and bugs ironed out :)) on invidious. Best regards Nicolas Le lundi 30 novembre 2020, 19:28:29 CET Nicolas Weeger a écrit : > Hello. > > > So, anyone courageous (or crazy? :)) enough to have looked at my patch? :) > > > Best regards > > > Nicolas signature.asc Description: This is a digitally signed message part. ___ crossfire mailing list crossfire@metalforge.org http://mailman.metalforge.org/mailman/listinfo/crossfire
Re: [crossfire] Archetypes collection changes proposal
Hello. So, anyone courageous (or crazy? :)) enough to have looked at my patch? :) Best regards Nicolas Le mardi 24 novembre 2020, 23:00:12 CET Nicolas Weeger a écrit : > Hello. > > > Here's the latest version of my patch :) > > Comments & suggestions welcome of course. > > > > > I'd really like some feedback or ideas on how to integrate it to trunk :D > > > > Best regards > > > Nicolas signature.asc Description: This is a digitally signed message part. ___ crossfire mailing list crossfire@metalforge.org http://mailman.metalforge.org/mailman/listinfo/crossfire
Re: [crossfire] Archetypes collection changes proposal
Hello.
I finally managed to make good progress, so joining a patch to this message for
comments & review & whatever else :)
This is a work in progress, and may crash at any point :)
The following files are now grabbed from individual files instead of
"collected"
files:
- treasures (from .trs files)
- face and animations (from .face files)
- archetypes (from .arc files)
It should be safe to point to the install directory containing the collected
files, if the "arch" directory is linked from there.
Still needs to be done:
- artifacts
- messages
- alchemy formulae
- faceset informations
- face data
- change the location of "smooth face" definition - it's in the .arc but
applied to the face with that name... (this will require changes to Gridarta
if used)
Maybe specific extensions may be introduced for eg artifacts, messages,
formulae, to not use the exact filename.
Collect is done from the base directory, each file is processed, then for each
sub-directory the process is repeated. Names are sorted case-sensitive in both
cases.
With those changes, it is now possible to redefine archetypes, treasures, faces
or animations. It's allowed to have multiple definitions, the latest found will
be used.
Archetypes in particular should correctly update items in inventories of other
archetypes when redefined (thus you could redefine the "event_apply" if you
want, it should be ok and update eg the altar of Valkyrie).
Warning: for multiple tiles monsters, this may crash, especially if you change
the size of monsters...
For artifacts, since it's hard to identify exactly an artifact (no exact
name), I think I'll add a mechanism to "remove" choices & such for types -
thus one could clean part of artifacts before adding one's specific ones.
Same mechanism for formulae probably.
Various comments on the code:
- I did introduce C++ (C++11 I think) for the collect code, and moved arch.c
to arch.cpp
- CRE is broken
- unit tests are broken / invalid / need to be updated
- much cleaning will probably be done, to separate cleanly "collecting" and
"use" parts (thus a collecting class, maintaining eg the list of treasures, in
its file, and functions related to actual treasure generation and thus). I do
intend to use C++ features for this, but not change everything for the sake of
it
Best regards
Nicolasdiff --git a/common/Makefile.am b/common/Makefile.am
index cdd74fd..63da86d 100644
--- a/common/Makefile.am
+++ b/common/Makefile.am
@@ -2,9 +2,10 @@ noinst_LIBRARIES = libcross.a
libcross_a_SOURCES = \
anim.c \
- arch.c\
+ arch.cpp\
arch_types_valid.c \
artifact.c\
+ assets.cpp\
button.c\
cf_random.c \
dialog.c \
diff --git a/common/anim.c b/common/anim.c
index 4cc4935..15c9e3b 100644
--- a/common/anim.c
+++ b/common/anim.c
@@ -24,6 +24,8 @@
#include
#include
+#include "assets.h"
+
static int animations_allocated = 0;
/**
@@ -32,134 +34,139 @@ static int animations_allocated = 0;
void free_all_anim(void) {
int i;
-for (i = 0; i <= num_animations; i++) {
-free_string(animations[i].name);
-free(animations[i].faces);
+for (i = 0; i < num_animations; i++) {
+free_string(animations[i]->name);
+free(animations[i]->faces);
+free(animations[i]);
}
free(animations);
}
/**
- * Loads the lib/animations file.
- * Can be called multiple times without ill effects.
+ * Init animations. Must be called only once, when assets are collecting.
**/
-void init_anim(void) {
-char buf[MAX_BUF];
-FILE *fp;
-int num_frames = 0, i;
-const Face *faces[MAX_ANIMATIONS];
-Animations *alloc_ptr; // Used to temporarily catch the result of realloc, in case of failure.
+static void init_anim(void) {
+animations_allocated = 10;
-animations_allocated = 9;
-num_animations = 0;
/* Make a default. New animations start at one, so if something
* thinks it is animated but hasn't set the animation_id properly,
* it will have a default value that should be pretty obvious.
*/
-animations = malloc(10*sizeof(Animations));
-/* set the name so we don't try to dereferance null.
+animations = calloc(animations_allocated, sizeof(Animations**));
+
+/* set the name so we don't try to dereference null.
* Put # at start so it will be first in alphabetical
* order.
*/
-animations[0].name = add_string("###none");
-animations[0].num_animations = 1;
-animations[0].num = 0;
-animations[0].faces = malloc(sizeof(Face*));
-animations[0].faces[0] = find_face("bug.111", NULL);
-animations[0].facings = 0;
-
-snprintf(buf, sizeof(buf), "%s/animations", settings.datadir);
-if ((fp = fopen(buf, "r")) == NULL) {
-LOG(llevError, "Cannot open animations file %s: %s\n", buf, strerror(errno));
-exit(-1);
+animations[0] = calloc(1, sizeof(Animations));
+animations[0]->name = add_string("###none");
+
Re: [crossfire] Archetypes collection changes proposal
On 10/7/20 3:22 AM, Nicolas Weeger wrote: I've made good progress, with a global skeleton on which to add more files to read (right now I read .face & .png files). Right now, partial collect process takes approx 10s. It'll probably take more at the end, 'cause of more file processing. Is this code somewhere that we can take an early look at? I'll just freely troll (to vent some frustration :)), but I have to do really similar stuff for many structure types (faces, archetypes, treasures, artifacts, and probably more), and frankly, I'm really tempted to "accidentally" switch to C++ just to reduce that duplicated code by using good old std::map<>... I would support code changes that reduce duplication and make code more maintainable. On the other hand, I'm not sure I have the same in-depth understanding (I like to tell myself) of C++ as I do C. If something inevitably breaks, I'm not sure I'd know how to fix it. I've run many C++ programs in a debugger to figure out what's going on, only to find that some constructor or destructor caused a segfault. I have a vague understanding of C++ name mangling, but am not sure I could fix a broken C++ build. Perhaps all that's needed is some C++ education on my part. Regards, Kevin ___ crossfire mailing list crossfire@metalforge.org http://mailman.metalforge.org/mailman/listinfo/crossfire
Re: [crossfire] Archetypes collection changes proposal
Hello. I've made good progress, with a global skeleton on which to add more files to read (right now I read .face & .png files). Right now, partial collect process takes approx 10s. It'll probably take more at the end, 'cause of more file processing. I'll just freely troll (to vent some frustration :)), but I have to do really similar stuff for many structure types (faces, archetypes, treasures, artifacts, and probably more), and frankly, I'm really tempted to "accidentally" switch to C++ just to reduce that duplicated code by using good old std::map<>... Best regards Nicolas signature.asc Description: This is a digitally signed message part. ___ crossfire mailing list crossfire@metalforge.org http://mailman.metalforge.org/mailman/listinfo/crossfire
Re: [crossfire] Archetypes collection changes proposal
Hello. > Do maps have a place here, as well? Didn't think about them, good question. They could probably, but I don't plan to have them handled right now like other assets. > To keep things simple, why keep the collect mechanism? Couldn't we put > the "official" assets into a big ZIP file, and teach the server how to > open ZIP files like ordinary directories? Scan the index, create sorted > case-sensitive depth-first order, etc. Having a .zip is indeed a possibility, but that's not something I plan at start. For the 'collect', I like the idea of having consolidated files that were "validated" for consistency (like all referenced archetypes are defined, things like that). Also, what I like with the current system is that most files are plain text files that are easy to adjust if needed. So .zip/.tar support could be added, but imo shouldn't be mandatory, to keep it easy to tweak. > `make do-collect` on a cold cache on my dated computer, with an SSD, > takes 6.3 sec. I imagine it could be a bit slower for those with > spinning magnetic media. This seems acceptable for a video game, and > particularly for a server that's supposed to be long-running. But, we > should still profile, see what's slow, and try to make this faster. Yes, performance isn't that high on my topic list right now. Also since I plan on adding the collect mechanism back later, we will be able to use single files like right now, so I don't expect any performance issue. > What would your strategy be for a change like this? One interim > strategy, just to keep everything working, could be to invoke the > current collect process at server start via fork/exec, read in output > via loader, then start. Then merge the steps until they become one. Current strategy: - have the server recursively find files in "arch" directory - remove the current collect mechanism - maybe rename some files (add extensions and such) - add new collect mechanism to consolidate archetypes, treasures, and such - format for pictures to be defined - optionally add tar/zip/... support (which could solve the picture format question) Best regards Nicolas signature.asc Description: This is a digitally signed message part. ___ crossfire mailing list crossfire@metalforge.org http://mailman.metalforge.org/mailman/listinfo/crossfire
Re: [crossfire] Archetypes collection changes proposal
> Do maps have a place here, as well? FWIW, I think distributing maps, arches, artifacts, recipes, treasurelists, etc., as a single unit makes sense to me. The original proposal of moving the collect phase into server-start, instead of server-compile, is a good one. A few more seconds on server startup is really not so bad. More optimization would always be nice, but a server that can still start faster than a client means it is more than fast enough already. --DraugTheWhopper ___ crossfire mailing list crossfire@metalforge.org http://mailman.metalforge.org/mailman/listinfo/crossfire
Re: [crossfire] Archetypes collection changes proposal
This idea seems reasonable. The original collection dates back to a time when hard drives were much slower (and amount of memory in systems was less, so likelihood of that data being cached was low) Like Kevin, I'm not sure if there is still need for collected archetypes - the tar file could just be distributed as is now, and have it unpacked as part of the install. At least for the .arc files, in theory, I think a 'cat */*.arc > everything.arc' would more or less work - archetype files can currently have more than one archetype. The issue is the png files - those can not be simply combined - some form of container would be needed, whether that is zip, tar, or custom format currently used. One advantage in the past was that having large files was more efficient if you wanted to compress them, but that isn't an issue now days. ___ crossfire mailing list crossfire@metalforge.org http://mailman.metalforge.org/mailman/listinfo/crossfire
Re: [crossfire] Archetypes collection changes proposal
On 9/16/20 2:38 AM, Nicolas Weeger wrote: > To simplify, I'll call "assets" everything that is in the "arch" repository: > archetypes, pictures, faces, animations, treasures, artifacts, and what ever > else I'm missing ;) Do maps have a place here, as well? > The goal: make it easier to add or replace existing assets for contributors, > simplify assets distribution, and make it easy for server admins to > add/remove > extra or "official" assets. These sound like good goals to me. > My proposal: > - have the server recursively browse "data" directory to find all assets > - browse in a deterministic manner (sorted case-sensitive, depth first) > - allow override of defined assets (thus the need to have a deterministic > browsing pattern) > - remove the requirement for "collect" > - keep the "collect" mechanism to consolidate assets in big files easy for > distribution - have it work for non official assets too To keep things simple, why keep the collect mechanism? Couldn't we put the "official" assets into a big ZIP file, and teach the server how to open ZIP files like ordinary directories? Scan the index, create sorted case-sensitive depth-first order, etc. > One big drawback is that the server startup will probably be slower, and the > code more complex (to handle overriding). > > I think that's an acceptable drawback, but that's subject to discussion ;) `make do-collect` on a cold cache on my dated computer, with an SSD, takes 6.3 sec. I imagine it could be a bit slower for those with spinning magnetic media. This seems acceptable for a video game, and particularly for a server that's supposed to be long-running. But, we should still profile, see what's slow, and try to make this faster. What would your strategy be for a change like this? One interim strategy, just to keep everything working, could be to invoke the current collect process at server start via fork/exec, read in output via loader, then start. Then merge the steps until they become one. Regards, Kevin ___ crossfire mailing list crossfire@metalforge.org http://mailman.metalforge.org/mailman/listinfo/crossfire
Re: [crossfire] Archetypes collection changes proposal
So the simple version of the proposed change is to have the arch directory work like the maps do now instead of like source code that is complied in. (And other details like allowing overrides.) I like the idea. The current system is broken in that I can make arch changes and the Makefiles don't detect any changes, so I have to manually do the collect to pull in the changes. Fixing that would be a pain. As to the start-up time, how much longer will it really take? Running 'make do-collect' just took 1.166s on my laptop. Even 5s wouldn't be a big problem for recovering from a server crash. ___ crossfire mailing list crossfire@metalforge.org http://mailman.metalforge.org/mailman/listinfo/crossfire
[crossfire] Archetypes collection changes proposal
Hello. I'd like to revamp the archetypes / faces / treasures / etc. "collect" mechanism. This proposal is opened to discussion & suggestions & criticism of course :) To simplify, I'll call "assets" everything that is in the "arch" repository: archetypes, pictures, faces, animations, treasures, artifacts, and what ever else I'm missing ;) The goal: make it easier to add or replace existing assets for contributors, simplify assets distribution, and make it easy for server admins to add/remove extra or "official" assets. My proposal: - have the server recursively browse "data" directory to find all assets - browse in a deterministic manner (sorted case-sensitive, depth first) - allow override of defined assets (thus the need to have a deterministic browsing pattern) - remove the requirement for "collect" - keep the "collect" mechanism to consolidate assets in big files easy for distribution - have it work for non official assets too This way, one could: - use collected asset files in the "data" directory => have "official" Crossfire assets - use collected asset files, add a directory "my-assets" with additional or replacement assets => extend the game - have an empty "data" directory and a symlink to "arch" repository => use trunk assets - add as many directories as wanted to alter assets as needed - use "official" collected asset files, and add custom collected asset files - and so on I think that for asset contributors it'll be easier to test changes (no need to "make do-collect && make install"); it'll allow contributors to distribute "asset packs" which can simply be put into data/subdir by server admins; it'll make it easy for people wanting to maintain extra/non official stuff without messing with the "arch" repository (avoiding conflicts with changes and such). One big drawback is that the server startup will probably be slower, and the code more complex (to handle overriding). I think that's an acceptable drawback, but that's subject to discussion ;) Nicolas signature.asc Description: This is a digitally signed message part. ___ crossfire mailing list crossfire@metalforge.org http://mailman.metalforge.org/mailman/listinfo/crossfire
