Hey Matt, everyone, I debugged the code and managed to get some sensible results. HTM is doing a great job of learning sequences but performing very poorly at generalization. So while it can recognize a sequence it had learned with high accuracy, when it’s fed a test sequence that it’s never seen, its classification accuracy plummets. To be clear, classification here is performed by assigning an HTM region to each class and observing which region outputs the least anomaly score averaged along a test sequence.
I’ve tried tweaking the encoder parameters to quantize the input with a lower resolution in the hope that similar inputs will be better pooled. That didn’t pan out. Also, changing encoder output length or number of columns is causing the HTM to output no predictions at times even with a non-empty active column list. I have little idea why that keeps happening. Any hints as to how to get HTM to better perform here? I’ve included HMM results for comparison. SVM results are all 95+%. Thank you, Nick HTM Results: Data = sequence of directions (8 discrete direction) Note on accuracy: M1/M2 is shown here to represent 2 performance metrics. M1 is average anomaly, M2 is the sum of average anomaly normalized and prediction error normalized. Base training accuracy: 100 % at 2 training passes User Dependent: 56.25%/56.25% User Independent: N/A Mixed: 65.00 %/ 71.25% HMM (22-states) Results: Data = sequence of directions (16 discrete direction) Base training accuracy: 97.5% User Dependent: 76.25 % User Independent: 88.75 % Mixed: 88.75 % > On Dec 11, 2014, at 7:16 PM, Matthew Taylor <[email protected]> wrote: > > Nicholas, can you paste a sample of the input data file? > > --------- > Matt Taylor > OS Community Flag-Bearer > Numenta > > On Thu, Dec 11, 2014 at 7:50 AM, Nicholas Mitri <[email protected] > <mailto:[email protected]>> wrote: > Hey all, > > I’m running into some trouble with using HTM for a gesture recognition > application and would appreciate some help. > First, the data is collected from 17 users performing 5 gestures of each of > 16 different gesture classes using motion sensors. The feature vector for > each sample is a sequence of discretized directions calculated using bezier > control points after curve fitting the gesture trace. > > For a baseline, I fed the data to 16 10-state HMMs for training and again for > testing. The classification accuracy achieved is 95.7%. > > For HTM, I created 16 CLA models using parameters from a medium swarm. I ran > the data through the models for training where each model is trained on only > 1 gesture class. For testing, I fed the same data again with learning turned > off and recorded the anomaly score (averaged across each sequence) for each > model. Classification was done by seeking the model with the minimum anomaly > score. Accuracy turned out to be a puzzling 0.0%!! > > Below is the relevant section of the code. I would appreciate any hints. > Thanks, > Nick > > def run_experiment(): > print "Running experiment..." > > model = [0]*16 > for i in range(0, 16): > model[i] = ModelFactory.create(model_params, logLevel=0) > model[i].enableInference({"predictedField": FIELD_NAME}) > > with open(FILE_PATH, "rb") as f: > csv_reader = csv.reader(f) > data = [] > labels = [] > for row in csv_reader: > r = [int(item) for item in row[:-1]] > data.append(r) > labels.append(int(row[-1])) > > # data_train, data_test, labels_train, labels_test = > cross_validation.train_test_split(data, labels, test_size=0.4, random_state=0) > data_train = data > data_test = data > labels_train = labels > labels_test = labels > > for passes in range(0, TRAINING_PASSES): > sample = 0 > for (ind, row) in enumerate(data_train): > for r in row: > value = int(r) > result = model[labels_train[ind]].run({FIELD_NAME: value, > '_learning': True}) > prediction = result.inferences["multiStepBestPredictions"][1] > anomalyScore = result.inferences["anomalyScore"] > model[labels[ind]].resetSequenceStates() > sample += 1 > print "Processing training sample %i" % sample > if sample == 100: > break > > sample = 0 > labels_predicted = [] > for row in data_test: > anomaly = [0]*16 > for i in range(0, 16): > model[i].resetSequenceStates() > for r in row: > value = int(r) > result = model[i].run({FIELD_NAME: value, '_learning': False}) > prediction = result.inferences["multiStepBestPredictions"][1] > anomalyScore = result.inferences["anomalyScore"] > # print value, prediction, anomalyScore > if value == int(prediction) and anomalyScore == 0: > # print "No prediction made" > anomalyScore = 1 > anomaly[i] += anomalyScore > anomaly[i] /= len(row) > sample += 1 > print "Processing testing sample %i" % sample > labels_predicted.append(np.min(np.array(anomaly))) > print anomaly, row[-1] > if sample == 100: > break > > accuracy = np.sum(np.array(labels_predicted) == > np.array(labels_test))*100.0/len(labels_test) > print "Testing accuracy is %0.2f" % accuracy > >
