Neal, I think this is a great project! I am going to have to dig into FREAK before I can say how the SP will handle its feature set. But since the SP was originally designed to handle topology, you might want to look into the passthrough encoder and using the output of FREAK as a direct input to the SP as long as it is a sparse vector.
Also, right now, the 'pooling' aspect of the TP isn't enabled but I think your project could definitely take advantage of it, so working with a few others to get that back in shape might be a great contribution. A couple questions, what is your timeline and level of comfort with C++ and/or Python? Ian On Mon, Oct 28, 2013 at 9:31 PM, Neal Donnelly <[email protected]> wrote: > Hey everybody, > > I'm a senior at Princeton University looking to apply NuPIC as my senior > thesis. Specifically, I'm intrigued to see how well it works for computer > vision as I've been learning a lot about traditional approaches here at > school. I'd like to run my plan by the community to get some feedback > before I dive in. > > I would like to build a pipeline for video classification. I understand > the online prediction framework isn't designed for classification, but I > want to get a sense of how NuPIC can perform on traditional problems as the > tool to build high level features. > > I would read in each video with OpenCV <http://opencv.org/>, and for each > frame I would extract the keypoints with the > FREAK<http://infoscience.epfl.ch/record/175537/files/2069.pdf>feature > descriptor and the corresponding detector. FREAK appeals to me > because its implemented in OpenCV, its fast, it produces state of the art > results, and it draws direct inspiration from the retina. Each frame would > be turned into a row in a CSV file with a timestamp and a list of features > and their descriptors. > > The CSV file would then be fed into the swarm, and then I would run the > resulting model. I would feed all the training videos to the model once > through to get it to learn certain types of features, then for each > training video feed it through again and take the top level representation > as the representation of the video. Once I have a top level representation > of each video, I'll feed those to a Bayesian classifier and see if I can > train it to recognize the types of videos. My plan is to start with the six > types of human actions in this > dataset<http://www.nada.kth.se/cvap/actions/>from KTH. > > My questions are > > 1) Do I have to translate all my videos into CSVs? How do I separate the > different video clips so that I can feed them all in - how can I put them > all in the same file if they all need a time stamp? > > 2) Do I want to just build one model or do I need a separate one for each > classification class? > > 3) What doesn't make sense about my plan? > > Thanks so much! > Neal Donnelly > > _______________________________________________ > nupic mailing list > [email protected] > http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org > >
_______________________________________________ nupic mailing list [email protected] http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org
