You probably can c2nim [ffms2](https://github.com/FFMS/ffms2). This is what avisynth/vapoursynth uses to store a video in a variable like in my script from vapoursynth import core video = core.ffms2.Source(source='Rule6.mkv')
And then it provides facility to get specific frames const FFMS_Frame *FFMS_GetFrame(FFMS_VideoSource *V, int n, FFMS_ErrorInfo *ErrorInfo); const FFMS_Frame *FFMS_GetFrameByTime(FFMS_VideoSource *V, double Time, FFMS_ErrorInfo *ErrorInfo);