When evaluating prediction models, the bottom line method of measuring
performance is to compare the predicted values NuPIC is returning to
the actual values in the data *when they occur*. So if you are
predicting 10 steps ahead, you'll not know how well the predictions
really are until you get to 10 steps in the future. While I understand
how to set predictionSteps in swarming, I am not sure how to use that
results to actually generate a future prediction, e.g. 10 steps ahead.
Take the run.py in your hot gym example, it seems to only do
prediction 1 step ahead and only runs until the actual data runs out.
Can you show me, preferably with an example based on run.py, how to do
predictions 10 steps ahead? I tried to do this:
_METRIC_SPECS = (
MetricSpec(field='price', metric='multiStep',
inferenceElement='multiStepBestPredictions',
params={'errorMetric': 'aae', 'window': 1000, 'steps': 10}),
MetricSpec(field='price', metric='trivial',
inferenceElement='prediction',
params={'errorMetric': 'aae', 'window': 1000, 'steps': 10}),
MetricSpec(field='price', metric='multiStep',
inferenceElement='multiStepBestPredictions',
params={'errorMetric': 'altMAPE', 'window': 1000, 'steps': 10}),
MetricSpec(field='price', metric='trivial',
inferenceElement='prediction',
params={'errorMetric': 'altMAPE', 'window': 1000, 'steps': 10}),
)
But I don’t see any difference than if I set ‘steps’ to 1. Also I am
not sure what this do:
prediction = result.inferences["multiStepBestPredictions"][1]
Thanks.
