Just landed from the new year holiday to read this detailed replay, very
good cover Aevar :)
*(in EXR) have you gotten around the 31 character limit*
Yes { :( } that's where the external metadata file will help. but 31 might
not be enough to store the full path ( as I believe you're pointing out )
1st step - create the metadata file in the same location as the EXR.
2nd Step - adding a pointer ID ( as a metadata ) into the EXR/DPX files.
the ID will point into a database entry ( MySQL or Shotgun ) that will
point to the full metadata.
This will cover most angles, making sure the EXR's could always be coupled
with their metadata.
*you can always try out the Shotgun Pipeline Toolkit (Formerly Tank)*
As for storing render metadata in Shotgun good or bad, it's a very
interesting/good point:
- Pro: Metadata entities could be linked to Version entities of the
renders.
- Pro: User can easily find the metadata vie Shotgun web interface.
- Cons: Shotgun API query are slow ( even more so when it's not a local
host )
Currently we do use Shotgun as the main database.
but I'm planning, one day, to migrate data that is less for user-viewing
and more for back-end to out local MySQL which is much faster for query.
*migrating could end up in syncing data between the two and
not necessarily "hard-migrate"
* a single Shotgun query is fast enough ( for an interactive tool or a
process ) and using API3, most times, queries can be bundled into one using
"deep-query". But... more than often, there's the need to "attack" shotgun
with a multi query which I find are slower than a locally hosted database.
happy new years :)
Asi
On Mon, Dec 23, 2013 at 6:05 PM, <[email protected]> wrote:
> * Good answer Asi, got a rant here below, hope it helps ( got too much
> time on my hands over the holidays apparently) *
>
> *Asi wrote:*
> *After some rethinking I changed my approach a little, reason is,EXR have
> a good support for long/custom metadata but others don't have limited
> support, DPX for example, which we use as well.*
>
> Cool, I’m not sure about how awesome exr is when it comes to long/custom
> metadata, it’s better than dpx true, don’t get me wrong, the first time
> .exr was opened to the public I was so excited I learnt to write them out
> command line pixel by pixel, exr rocks beyond believe but the character
> limit can get annoying, have you gotten around the 31 character limit yet (
> By your description I assume so, but sending the note your way so it
> doesn’t come as a surprise if it hits )
>
> *http://www.openexr.com/openexrfilelayout.pdf*<http://www.openexr.com/openexrfilelayout.pdf>
> *http://www.openexr.com/MultiViewOpenEXR.pdf*<http://www.openexr.com/MultiViewOpenEXR.pdf>
> *http://lists.nongnu.org/archive/html/openexr-devel/2006-05/msg00009.html*<http://lists.nongnu.org/archive/html/openexr-devel/2006-05/msg00009.html>
>
> It gets better though:
>
> *http://download.autodesk.com/global/docs/maya2014/en_us/index.html?url=files/Vari_Subfolders_and_names_of_rendered_images.htm,topicNumber=d30e685154*<http://download.autodesk.com/global/docs/maya2014/en_us/index.html?url=files/Vari_Subfolders_and_names_of_rendered_images.htm%2ctopicNumber=d30e685154>
> And with exr2 and all that limits are closer to the sky than ever before.
>
> *Asi wrote:*
> * So instead of figuring out each format metadata support, I rather dump
> the metadata in a file or a Database,*
>
> Good call, it’s what most studios do, less risk regarding a frame that's
> been sitting on the farm for 48 hours and such, however my attitude towards
> that has always been;
> a)
> Why don't those guys hire a render pipeline developer? a 48 hour render
> has not been needed since at least 5 years ago, it’s horribly risky and
> regeneration is near impossible in a tight deadline, trust me I’ve seen
> the biggest frames being shown in cinema these days go through massive
> farms and there is nothing in the entire world of games/film/commercials
> which would justify a pipeline where one falls to the grace of the
> processors and won’t know a thing until days later.
> b)
> If done right, this information belongs inside the actual frame as the
> base location, any database or file system info file should derive it’s
> information from the image header. It’s only sensible to include
> information about things inside things, of course! It’s only because most
> people who attempt this completely mess it up that I feel most decision
> makers rightfully choose the path which has burnt the least and additional
> databases, info files, excel sheets, and so forth will forever exist.
>
> *Asi wrote:*
> * located where the images/movie are*,
> Careful;
> [pro]
> Easy access to the data for everyone.
> [con]
> -You increase the number of files in the project directory, making for
> more convulated browsing, raising more questions to inherently inquisitive
> groups of people who really should rather be focusing on their modelling
> task at hand than be wondering about what the difference between
> this_shot.mb / this_shot.info / this_shot.dep files are.
> -One can actually foolhardily enough script enough files to get
> generated into there so it goes beyond 2000 files, loading times from a
> database environment turns into massive debug sessions to enhance the
> loading process when all that is happening is unix struggling due to sloppy
> coding, check out this link, it’s a really good read, suggests only
> 1020 files per directory which is an absolute most i.m.h.o
> *http://www.frank4dd.com/howto/various/maxfiles-per-dir.htm*<http://www.frank4dd.com/howto/various/maxfiles-per-dir.htm>
> Additional things to back this up below, it’s so crazy I still ran into
> people in 2013 hitting this limit 😊
>
> *http://www.linuxquestions.org/questions/general-10/maximum-number-of-files-in-a-directory-183643/*<http://www.linuxquestions.org/questions/general-10/maximum-number-of-files-in-a-directory-183643/>
>
> *http://stackoverflow.com/questions/466521/how-many-files-in-a-directory-is-too-many*<http://stackoverflow.com/questions/466521/how-many-files-in-a-directory-is-too-many>
>
> *http://serverfault.com/questions/59454/whats-the-maxium-number-of-files-a-unix-folder-can-hold*<http://serverfault.com/questions/59454/whats-the-maxium-number-of-files-a-unix-folder-can-hold>
>
> It’s always good to have that stuff accessible but if it’s not inside
> the image file I’d say keep it away from the general staff ( *tell them
> where it is if they are interested of course but don't complicate their
> workspace with metadata information they don’t need* )
>
> *Asi wrote:*
> * I could have infinite amount of metadata without relying on each format
> ability to store metadata. this is doable in our environment since the
> rendered files won't get moved from the metadata file. *
> It’s a good idea, the reason I’m getting into this in further detail
> is it’s getting to be a bit of an old sport. Works of course but history
> has proven requires massive amounts of human resources to support it.
> Alternates do exist out there, for example since you bring up the
> environment have you considered a context sensitive directory scheme?
> Roughly it goes { setenv this_x[1-nearinfinity] for levels of directory
> depth; that_[x-nearinfinity] for token in naming convention + non
> identifiable elements} , It’s a fun one, means you only ever have 1
> configuration file that you drop in the root of projects and all metadata
> is actually generated into the environment based on where you are looking
> at it from ( hooks into cd on Unix and folder callbacks on Windows, change
> directory; your environment is different, this can include things all
> the way down to artist notes even, all still driven from a single algorithm
> dependant on who is reading it from where, some studios have a rudimentary
> scheme like this going on but I’ve never seen it fully utilized )
>
> Then of course, you can always try out the Shotgun Pipeline Toolkit
> (Formerly Tank) which handles all this at a phenomenal level and since it’s
> so widespread these days you will be working with a scheme that is
> consistent across studios, I do think the top of the line with regards to
> technology we have at hand today is generating metadata into the rendered
> files ( *internally by the renderer as you describe, not a post process,
> vray and Arnold have the ease of access nailed down* ) then push that
> information towards a hosted Shotgun server as a centralized container.
> Must note I have no affiliations with Shotgun development beyond customer
> level but can verify after having used it since it’s first iteration then
> develop towards it as Tank became the pipeline toolkit it just makes
> everything so straight forward. Keep it in mind, Shotgun rocks for what
> you are after but mostly so if you manage to embed your metadata.
> *http://www.shotgunsoftware.com/* <http://www.shotgunsoftware.com/>
>
> *Asi wrote:*
> *I still wish MR would have a built in hook to inject metadata :) but this
> workaround would help improve on other wider issues as well*
>
> hehe, ok, I fold, just promise to be careful, mental ray picks up custom
> attributes based on a naming convention, as easy as it gets
>
> - *Experimental:* if an attribute with name staring with "meta:" prefix
> is used (like attribute string "meta:my_key" "my_value"), the value is
> passed to the image file headers (i.e."my_key" "my_value" pair). This
> is currently only supported for OpenEXR images. Boolean parameters are
> converted to integer ones.
>
>
> *http://docs.autodesk.com/MENTALRAY/2013/ENU/mental-ray-help/files/relnotes/relnotes.html*<http://docs.autodesk.com/MENTALRAY/2013/ENU/mental-ray-help/files/relnotes/relnotes.html>
>
>
> Care to move on to compression? I can show you how to turn single
> render into something which halts a 50T server in a few seconds vs. renders
> without anyone noticing by only toggling a compression setting :)
> exr is awesome to play with, the only format which lets you actually
> just set that other setting and not lose over 30 hours a day of accumulated
> time only spent due to number of files in directories and compression
> schemes, without it the visual effects industry would not have expanded as
> it has done in the last few years, it’s a weird thing to claim I know but
> the efficiency of the format has allowed teams to increase their
> attentiveness to detail and volume in such a manner that the limitless
> environment allows for amazing things to be created, the learning curve
> regarding it though can be a bit tricky as it’s not like other
> formats. But hey! it doesn’t have to, it does it’s thing really well :)
>
> *http://forum.nvidia-arc.com/showthread.php?9434-OpenExr-and-zip-scanline-compressions*<http://forum.nvidia-arc.com/showthread.php?9434-OpenExr-and-zip-scanline-compressions>
>
> Have a nice holiday ahead
> hope your metadata adventures go well
> -aeVar
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Python Programming for Autodesk Maya" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/python_inside_maya/Tggunq6Bo0Q/unsubscribe
> .
> To unsubscribe from this group and all its topics, send an email to
> [email protected].
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/python_inside_maya/52b9014a.aacbc20a.5f76.0242%40mx.google.com
> .
> For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups
"Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/python_inside_maya/CAP-786epnLhMPQdQWsr8HdD5_hO21xFy0M0otvyU2g1nnpq6Hw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.
