> Message: 7 > Date: Thu, 1 Dec 2011 12:36:02 -0500 > From: Chad <innocentkil...@gmail.com> > Subject: Re: [Wikitech-l] Proposal for new table image_metadata > To: Wikimedia developers <wikitech-l@lists.wikimedia.org> > Message-ID: > <cadn73rnusx8regdubcesyg8mz1qg5sa49vamb5ed_y-vb-l...@mail.gmail.com> > Content-Type: text/plain; charset=UTF-8 > > On Thu, Dec 1, 2011 at 12:34 PM, William Lee <w...@wikia-inc.com> wrote: > > I'm a developer at Wikia. We have a use case for searching through a file's > > metadata. This task is challenging now, because the field > > Image.img_metadata is a blob. > > > > We propose expanding the metadata field into a new table. We propose the > > name image_metadata. It will have three columns: img_name, attribute > > (varchar) and value (varchar). It can be joined with Image on img_name. > > > > On the application side, LocalFile's load* and decodeRow methods will have > > to be changed to support the new table. > > > > One issue to consider is the file archive. Should we replicate the metadata > > table for file archive? Or serialize the data and store it in a new table > > (something like fa_metadata)? > > > > Please let us know if you see any issues with this plan. We hope that this > > will be useful to the MediaWiki project, and a candidate to merge back. > > > > That was part of bawolff's plan last summer for GSoC when he overhauled > our metadata support. He got a lot of his project done, but never quite got > to this point. Something we'd definitely like to see though! > > -Chad > >
Chad beat me to writing essentially what I was going to say. Basically my project ended up being more about extracting more information, and i didn't really touch what we did with it after we extracted. However, it should be noted that storing the image metadata nicely is a little more complicated then it appears at first glance (and that's mostly my fault due to stuff i added during gsoc ;) Basically there's 4 different types of metadata values we store (in terms of the types of metadata you think of when you think EXIF et al. We stuff other stuff into img_metadata for extra fun) *Normal values - Things like Shutter speed = 1/110 *unordered array - For example we can extract a "tags" field that's an arbitrary list of tags, The subject field (from XMP) is an unordered list, etc *Ordered array - Not used for a whole lot Most prominent example is the XMP author field is supposed to be an ordered list of authors, in order of importance. Honestly, we could just ditch caring about this, and probably nobody would notice. *Language array - XMP and PNG text chunks support a special value where you can specify language alternatives. In essence this looks like an associative array of "lang-code" => "translation of field into that lang", plus a special fallback "x-default" dummy lang code. *Also Contact info and software fields are stored kind of weirdly.... Thus, just storing a table of key/value pairs is kind of problematic - how do you store an "array" value. Additionally you have to consider finding info. You probably want to efficiently be able to search through lang values in a specific language, or for a specific property and not caring for the language. Also consider how big a metadata field can get. Theoretically it's not really limited, well I don't expect it to be huge, > 255 bytes of utf-8 seems a totally reasonable size for a value of a metadata field. Last of all, you have to keep in mind all sorts of stuff is stored in the img_metadata. This includes things like the text layer of Djvu files (although arguably that shouldn't be stored there...) and other handler specific things (OggHandler stores some very complex structures in img_metadata). Of course, we could just keep the img_metadata blob there, and simply stop using it for "exif-like" data, but continue using it for handler specific ugly metadata that's generally invisible to user [probably a good idea. The two types of data are actually quite different]. > One issue to consider is the file archive. Should we replicate the metadata > table for file archive? Or serialize the data and store it in a new table > (something like fa_metadata)? Honestly, I wouldn't worry about that, especially in the beginning. As far as i know, the only place fa_metadata/oi_metadata is used, is that you can request it via api (I suppose it's copied over during file reverts as well). I don't think anyone uses that field on archived images really. (maybe one day bug 26741 will be fixed and this would be less of a concern). Anyhow, I do believe it would be awesome to store this data better. I can definitely think of many uses for being able to efficiently query it. (While I'm on the subject, making lucene index it would also be awesome). Cheers, Bawolff p.s. If its helpful - some of my ideas from last year for making a new metadata table are at http://www.mediawiki.org/wiki/User:Bawolff/metadata_table and the thread http://thread.gmane.org/gmane.science.linguistics.wikipedia.technical/48268 . However, they're probably over-complicated/otherwise not ideal (I was naive back then ;). They also try and be able to encode anything encodable by XMP, which is most definitely a bad idea, since XMP is very complicated... _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
