I thought maybe I should post my final conclusions for people who may
run into this issue in the future. Commercial detection as
implemented by Freevo is unreliable. Period. It searches for 'black
frames' it assumes are the beginning and end of commercials. It
cannot account for the fact that there are black frames in scene
transitions, as well as in the gaps between some commercials.
There are other methods for detecting commercials which Freevo does
not support that must also be employed.
On Tue, Mar 17, 2009 at 9:59 AM, Bill Bierman <b...@thebiermans.org> wrote:
> I'm using tvtime, though I don't remember why =)
>
> I've inserted a few debug statements in the commdetectcore.py code
> which parses the mencoder output. Hopefully this will shed some light
> on the situation.
>
> Incidentally, the mencoder vf_blackframes output is hideous. It's
> neither human-friendly nor tokenizer friendly.
>
> On Tue, Mar 17, 2009 at 6:44 AM, Evan Hisey <ehi...@gmail.com> wrote:
>> On Tue, Mar 17, 2009 at 2:39 AM, Bill Bierman <b...@thebiermans.org> wrote:
>>> I'm actually using tvtime, but I have manually ran mplayer using the
>>> EDL files generated by Freevo, and it still doesn't work.
>>>
>>> I assume the problem is with the mencoder execution, or possibly the
>>> parsing of its output, as the EDL file generated does not even seem
>>> sane.
>>>
>>> "3544 2603 0" is the single line in the example I gave in my original
>>> post. Multiple attempts at recording hour-long programs resulted in
>>> similar outputs.. a single line with the first parameter being greater
>>> than the second. The EDL format as I understand it is a start time
>>> followed by a stop time, and is therefore semantically invalid for the
>>> second parameter to be less than the first.
>>>
>>> As for whether the culprit is the mencoder parameters/execution or the
>>> parsing of its output, I haven't got a clue at this point. I'm still
>>> trying to figure out what the mencoder output means exactly, and
>>> trying to understand how commdetectcore.py handles it.
>>>
>> This I can help you with alittle. The process works like this. The
>> recordeserver uses mencoder to record the whole show first, so
>> mencoder output is not actually seen by the commdetect server. The
>> commdect server then runs mplayer over the whole the whole thing
>> looking for 98% black frames and assumes the first one is the start of
>> the commercial and the second one is the end of one. I would agree it
>> looks like you are getting strange index numbers. Are you using tvtime
>> to record the shows or mencoder? I have not had this particular
>> problem yet with commdetect
>>
>> Evan
>>
>> ------------------------------------------------------------------------------
>> Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are
>> powering Web 2.0 with engaging, cross-platform capabilities. Quickly and
>> easily build your RIAs with Flex Builder, the Eclipse(TM)based development
>> software that enables intelligent coding and step-through debugging.
>> Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com
>> _______________________________________________
>> Freevo-users mailing list
>> Freevo-users@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/freevo-users
>>
>
--- commdetectcore.py.ORIG 2008-05-31 06:14:43.000000000 -1000
+++ commdetectcore.py 2009-03-17 15:54:11.000000000 -1000
@@ -57,14 +57,12 @@
nosound='-nosound'
videoFilter='-vf blackframe'
output='-o /dev/null'
- grep='| grep vf_blackframe'
outfile='> /tmp/blackframes.txt'
string=" "+videoCodec+\
" "+nosound+\
" "+videoFilter+\
" "+self.source+\
" "+output+\
- " "+grep+\
" "+outfile
self.cls = [string]
@@ -85,76 +83,74 @@
endSkipTime=0.0
action=0.0
- def grabBlackFrames(self):
+ def genEdl(self):
#Grab all possible blackframes
fileHandle = open('/tmp/blackframes.txt','r')
+
+ frames = 0
+ maxFrames = 1000
+ FrameQueue = []
+ BlackFrames = []
+
for line in fileHandle.readlines():
splitln = line.split("\r")
- line=splitln[-2]+splitln[-1]
- #output with blackframe
- newBlackFrame = self.blackframe()
- splitframe = line.split("vf_blackframe:")
- matchSeconds = re.compile('\d+\.\d+s')
- match = matchSeconds.search(splitframe[0])
- if match:
- seconds=re.sub('s','',match.group())
- newBlackFrame.seconds=float(seconds)
- matchFrame = re.compile('\d+f')
- match = matchFrame.search(splitframe[0])
- if match:
- frame=re.sub('f','',match.group())
- newBlackFrame.frame=float(frame)
- matchTime = re.compile('\d+min')
- match = matchTime.search(splitframe[0])
- if match:
- time=re.sub('min','',match.group())
- newBlackFrame.time=int(time)
- self.blackframes.append(newBlackFrame)
+
+ for ln in splitln:
+ if ln[0:4] == "Pos:":
+ frameSeconds = float(ln[4:10])
+ frameNumber = int(ln[11:18])
+ FrameQueue.append([frameNumber, frameSeconds])
+ elif ln[0:14] == "vf_blackframe:":
+ frameNumber = int(ln[14:20])
+
+ for i in reversed(FrameQueue):
+ if i[0] != frameNumber:
+ continue
+ else:
+ frameSeconds = i[1]
+ break
+
+ BlackFrames.append([frameNumber, frameSeconds])
+
fileHandle.close()
- def findCommercials(self):
- startFrame=None
- startFrameTime=0
- endFrame=None
- for bframe in self.blackframes:
- if (startFrameTime==0):
- #Commerical break
- startFrame=bframe
- startFrameTime=bframe.time
- else:
- if ((bframe.time==startFrameTime)or(bframe.time==(startFrameTime-1))):
- #Same commercial break
- startFrameTime=bframe.time
- endFrame=bframe
- else:
- #New commercial break
- if not endFrame:
- #Sometimes a blackframe is thrown in the beginning
- endFrame=startFrame
- if ((endFrame.seconds-startFrame.seconds)>0):
- newEdl = self.edl()
- newEdl.startSkipTime=startFrame.seconds
- newEdl.endSkipTime=endFrame.seconds
- self.edlList.append(newEdl)
- startFrame=None
- startFrameTime=0
- endFrame=None
- if ((len(self.edlList)==0)and(startFrame and endFrame)):
- #Only one commercial
- newEdl = self.edl()
- newEdl.startSkipTime=startFrame.seconds
- newEdl.endSkipTime=endFrame.seconds
- self.edlList.append(newEdl)
-
- def writeEdl(self):
- if (len(self.edlList)>0):
- outputFile=self.source.split(".")
- output=outputFile[0]+".edl"
- fileHandle = open(output,'w')
- for skipSegment in self.edlList:
- fileHandle.write("%d %d %d\n" % (skipSegment.startSkipTime, \
- skipSegment.endSkipTime, \
- skipSegment.action))
+ fadeStartNum = -1
+ fadeStartTime = 0
+ lastFrameNum = 0
+ lastFrameTime = 0
+ startBlock = 1
+ startBlockTime = -1
+
+ if BlackFrames.length():
+ outputFile = self.source.split(".")
+ output = outputFile[0] + ".edl"
+ fileHandle = open(output, 'w')
+
+ for i in BlackFrames:
+ if fadeStartNum == -1:
+ fadeStartNum = i[0]
+ fadeStartTime = i[1]
+ lastFrameNum = i[0]
+ lastFrameTime = i[1]
+ continue
+
+ if i[0] - lastFrameNum > 5:
+ if startBlock:
+ startBlock = 0
+ startBlockTime = fadeStartTime
+ else:
+ fileHandle.write("%.1f %.1f 0\n" % (startBlockTime, lastFrameTime))
+ startBlock = 1
+
+ fadeStartNum = i[0]
+ fadeStartTime = i[1]
+ lastFrameNum = i[0]
+ lastFrameTime = i[1]
+ continue
+
+ lastFrameNum = i[0]
+ lastFrameTime = i[1]
+
fileHandle.close()
class CommandThread(threading.Thread):
@@ -192,12 +188,8 @@
except:
pass
self.kill_pipe()
- _debug_("Grabbing Blackframes from file")
- self.parent.grabBlackFrames()
- _debug_("Finding Commercials")
- self.parent.findCommercials()
- _debug_("Writing edl file")
- self.parent.writeEdl()
+ _debug_("Parsing mencoder output and generating edl file")
+ self.parent.genEdl()
self.parent.busy = 2
self.finalfunc()
sys.exit(2)
# -*- coding: iso-8859-1 -*-
# -----------------------------------------------------------------------
# Interface to CommDetectServer
# -----------------------------------------------------------------------
#
# Author: Justin Wetherell
# some parts taken or inspired by Freevo's encodingserver
#
# -----------------------------------------------------------------------
# Copyright (C) 2004 den_RDC (RVDM)
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of MER-
# CHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General
# Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation
#
# -----------------------------------------------------------------------
"""
Interface to Commercial Detection Server
"""
#Import statements
import threading
from time import sleep
import sys, os, re, popen2 #, ConfigParser, copy
import config
import kaa.metadata as mmpython
from copy import copy
from string import split, join
class CommDetectJob:
"""Class for creation & configuration of CommDetectJobs. This generates the mencoder commands"""
def __init__(self, source, idnr):
"""Initialize class instance"""
_debug_('commdetectcore.CommDetectJob.__init__(%s)' % (source))
self.source = source
self.name = source
self.idnr = idnr
self.pid = 0
self.busy = 0
"""0==NEW, 1==BUSY 2==DONE"""
self.blackframes=[]
self.edlList=[]
videoCodec='-ovc lavc'
nosound='-nosound'
videoFilter='-vf blackframe'
output='-o /dev/null'
outfile='> /tmp/blackframes.txt'
string=" "+videoCodec+\
" "+nosound+\
" "+videoFilter+\
" "+self.source+\
" "+output+\
" "+outfile
self.cls = [string]
def _run(self, program, arguments, source, flushbuffer, finalfunc):
"""Runs a program; supply program name (string) and arguments (list)"""
command = program
command += arguments
self.thread = CommandThread(self, command, flushbuffer, finalfunc)
self.thread.start()
class blackframe:
frame=0.0
seconds=0.0
time=0.0
class edl:
startSkipTime=0.0
endSkipTime=0.0
action=0.0
def genEdl(self):
#Grab all possible blackframes
fileHandle = open('/tmp/blackframes.txt','r')
frames = 0
maxFrames = 1000
FrameQueue = []
BlackFrames = []
for line in fileHandle.readlines():
splitln = line.split("\r")
for ln in splitln:
if ln[0:4] == "Pos:":
frameSeconds = float(ln[4:10])
frameNumber = int(ln[11:18])
FrameQueue.append([frameNumber, frameSeconds])
elif ln[0:14] == "vf_blackframe:":
frameNumber = int(ln[14:20])
for i in reversed(FrameQueue):
if i[0] != frameNumber:
continue
else:
frameSeconds = i[1]
break
BlackFrames.append([frameNumber, frameSeconds])
fileHandle.close()
fadeStartNum = -1
fadeStartTime = 0
lastFrameNum = 0
lastFrameTime = 0
startBlock = 1
startBlockTime = -1
if BlackFrames.length():
outputFile = self.source.split(".")
output = outputFile[0] + ".edl"
fileHandle = open(output, 'w')
for i in BlackFrames:
if fadeStartNum == -1:
fadeStartNum = i[0]
fadeStartTime = i[1]
lastFrameNum = i[0]
lastFrameTime = i[1]
continue
if i[0] - lastFrameNum > 5:
if startBlock:
startBlock = 0
startBlockTime = fadeStartTime
else:
fileHandle.write("%.1f %.1f 0\n" % (startBlockTime, lastFrameTime))
startBlock = 1
fadeStartNum = i[0]
fadeStartTime = i[1]
lastFrameNum = i[0]
lastFrameTime = i[1]
continue
lastFrameNum = i[0]
lastFrameTime = i[1]
fileHandle.close()
class CommandThread(threading.Thread):
"""Handle threading of external commands"""
def __init__(self, parent, command, flushbuffer, finalfunc):
threading.Thread.__init__(self)
self.parent = parent
self.command = command
self.flushbuffer = 0
self.finalfunc = finalfunc
_debug_('command=\"%s\"' % command)
def run(self):
self.pipe = popen2.Popen4(self.command)
pid = self.pipe.pid
self.parent.pid = copy(pid)
totallines = []
_debug_("Mencoder running at PID %s" % self.pipe.pid)
self.parent.busy = 1
while 1:
if self.flushbuffer:
line = self.pipe.fromchild.read(1000)
else:
line = self.pipe.fromchild.readline()
_debug_(line)
if not line:
break
else:
#don't save up the output if flushbuffer is enabled
if self.flushbuffer != 1:
totallines.append(line)
sleep(0.5)
try:
os.waitpid(pid, os.WNOHANG)
except:
pass
self.kill_pipe()
_debug_("Parsing mencoder output and generating edl file")
self.parent.genEdl()
self.parent.busy = 2
self.finalfunc()
sys.exit(2)
def kill_pipe(self):
"""Kills current process (pipe)"""
try:
os.kill(self.pipe.pid, 9)
os.waitpid(self.pipe.pid, os.WNOHANG)
except:
pass
class CommDetectQueue:
"""Class for generating an commdetect queue"""
def __init__(self):
#we keep a list and a dict because a dict doesn't store an order
self.qlist = []
self.qdict = {}
self.running = False
def addCommDetectJob(self, encjob):
"""Adds an commdetectjob to the queue"""
self.qlist += [encjob]
self.qdict[encjob.idnr] = encjob
def startQueue(self):
"""Start the queue"""
if not self.running:
self.running = True
_debug_("queue started", DINFO)
self._runQueue()
def listJobs(self):
"""Returns a list of queue'ed jobs"""
if self.qdict == {}:
return []
else:
jlist = []
for idnr, job in self.qdict.items():
jlist += [ (idnr, job.name) ]
return jlist
def _runQueue(self, line="", data=""):
"""Executes the jobs in the queue, and gets called after every mencoder run is completed"""
if self.qlist == []:
#empty queue, do nothing
self.running = False
if hasattr(self,"currentjob"):
del self.currentjob
_debug_("queue empty, stopping processing...", DINFO)
return
_debug_("runQueue callback data : %s" % line)
#get the first queued object
self.currentjob = self.qlist[0]
_debug_("PID %s" % self.currentjob.pid)
if self.currentjob.busy == 0:
#NEW
_debug_("Running Mencoder, to write the blackframes to a file")
self.currentjob.busy = True
self.currentjob._run(config.CONF.mencoder, \
self.currentjob.cls[0], \
self.currentjob.source, \
0, \
self._runQueue)
_debug_("Started job %s, PID %s" % (self.currentjob.idnr, self.currentjob.pid))
if self.currentjob.busy == 2:
#DONE
del self.qlist[0]
del self.qdict[self.currentjob.idnr]
self.currentjob.busy = 0
self.running = False
self._runQueue()
------------------------------------------------------------------------------
This SF.net email is sponsored by:
High Quality Requirements in a Collaborative Environment.
Download a free trial of Rational Requirements Composer Now!
http://p.sf.net/sfu/www-ibm-com
_______________________________________________
Freevo-users mailing list
Freevo-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/freevo-users
