Re: NUPIC variable mess v2

Wakan Tanka Sun, 17 May 2015 13:39:07 -0700

On 05/17/2015 08:15 PM, Matthew Taylor wrote:

Strange, running this from any directory works for me:

$NUPIC/scripts/run_swarm.py $NUPIC/examples/swarm/simple/search_def.json  
--maxWorkers=4


Instead of the strace, what is the console output of the command above?

---------
Matt Taylor
OS Community Flag-Bearer
Numenta


On Sat, May 16, 2015 at 1:34 PM, Wakan Tanka <[email protected]> wrote:

On 05/16/2015 09:56 PM, Wakan Tanka wrote:



S installation completes successfully, but I get exactly the same problem:

$NUPIC/scripts/run_swarm.py $NUPIC/examples/swarm/simple/search_def.json
   --maxWorkers=4

Works only from $NUPIC/nupic



Output of following command are in attachment

marek@ubuntu:~/nupic$ strace -ff -o ~/experiments/trace_false
$NUPIC/scripts/run_swarm.py $NUPIC/examples/swarm/simple/search_def.json
--maxWorkers=4

marek@ubuntu:~/nupic/nupic$ strace -ff -o ~/experiments/trace_true
$NUPIC/scripts/run_swarm.py $NUPIC/examples/swarm/simple/search_def.json
--maxWorkers=4

Finally I've made the progress ;) keep on reading. Been following yourvideo predicting sine waves https://www.youtube.com/watch?v=KuFfm3ncEwI

So I've created custom directory and created generate_data.py and alsosearch_def.json for those generated data. Here is full output:



marek@ubuntu:~/experiments/sine-waves$ pwd
/home/marek/experiments/sine-waves
marek@ubuntu:~/experiments/sine-waves$ ls -l
total 128
-rw-rw-r-- 1 marek marek  1608 May 16 00:02 generate_data.py
-rw-rw-r-- 1 marek marek   561 May 16 03:44 search_def.json
-rw-rw-r-- 1 marek marek 39014 May 16 00:03 sine100.csv
-rw-rw-r-- 1 marek marek 38722 May 16 00:02 sine50.csv
-rw-rw-r-- 1 marek marek 39014 May 16 00:18 sine.csv



marek@ubuntu:~/experiments/sine-waves$ cat search_def.json
{
  "includedFields": [
    {
      "fieldName": "sine",
      "fieldType": "float",
      "maxValue": "1.0",
      "minValue": "-1.0"
    }
  ],
  "streamDef": {
    "info": "sine",
    "version": 1,
    "streams": [
      {
        "info": "sine.csv",
        "source": "file:/home/marek/experiments/sine-waves/sine.csv",
        "columns": [
          "*"
        ]
      }
    ]
  },
  "inferenceType": "TemporalAnomaly",
  "inferenceArgs": {
    "predictionSteps": [
      1
    ],
    "predictedField": "sine"
  },
  "swarmSize": "medium"
}









marek@ubuntu:~/experiments/sine-waves$ cat generate_data.py | grep -v '^#'

"""A simple script to generate a CSV with sine data."""

import csv
import math

ROWS = 1000


def run(filename="sine.csv"):
  print "Generating sine data into %s" % filename
  fileHandle = open(filename,"w")
  writer = csv.writer(fileHandle)
  writer.writerow(["angle","sine"])
  writer.writerow(["float","float"])
  writer.writerow(["",""])

  for i in range(ROWS):
    angle = (i * math.pi) / 100.0
    sine_value = math.sin(angle)
    writer.writerow([angle, sine_value])

  fileHandle.close()
  print "Generated %i rows of output data into %s" % (ROWS, filename)



if __name__ == "__main__":
  run()

When I want to run swarm on data.csv using search_def.json I get thementioned error:

marek@ubuntu:~/experiments/sine-waves$ $NUPIC/scripts/run_swarm.py/home/marek/experiments/sine-waves/search_def.json --maxWorkers=6

Traceback (most recent call last):
  File "/home/marek/nupic/scripts/run_swarm.py", line 189, in <module>
    runPermutations(sys.argv[1:])

File "/home/marek/nupic/scripts/run_swarm.py", line 180, inrunPermutations

    fileArgPath, optionsDict, outputLabel, permWorkDir)

File"/home/marek/.local/lib/python2.7/site-packages/nupic-0.3.0.dev0-py2.7-linux-x86_64.egg/nupic/swarming/permutations_runner.py",line 309, in runWithJsonFile

    verbosity=verbosity)

File"/home/marek/.local/lib/python2.7/site-packages/nupic-0.3.0.dev0-py2.7-linux-x86_64.egg/nupic/swarming/permutations_runner.py",line 266, in runWithConfig

    _generateExpFilesFromSwarmDescription(swarmConfig, outDir)

File"/home/marek/.local/lib/python2.7/site-packages/nupic-0.3.0.dev0-py2.7-linux-x86_64.egg/nupic/swarming/permutations_runner.py",line 197, in _generateExpFilesFromSwarmDescription

    "--outDir=%s" % (outDir)])

File"/home/marek/.local/lib/python2.7/site-packages/nupic-0.3.0.dev0-py2.7-linux-x86_64.egg/nupic/frameworks/opf/exp_generator/ExpGenerator.py",line 2027, in expGenerator

    claDescriptionTemplateFile = options.claDescriptionTemplateFile)

File"/home/marek/.local/lib/python2.7/site-packages/nupic-0.3.0.dev0-py2.7-linux-x86_64.egg/nupic/frameworks/opf/exp_generator/ExpGenerator.py",line 191, in _handleDescriptionOption

    claDescriptionTemplateFile = claDescriptionTemplateFile)

File"/home/marek/.local/lib/python2.7/site-packages/nupic-0.3.0.dev0-py2.7-linux-x86_64.egg/nupic/frameworks/opf/exp_generator/ExpGenerator.py",line 1077, in _generateExperiment

    "%s\nOPTION ARG=%s") % (str(e), pprint.pformat(options)))

nupic.frameworks.opf.exp_generator.ExpGenerator._InvalidCommandArgException:JSON arg validation failed for option --description: Failed to validatefield 'includedFields' list schema: Value u'1.0' for field 'maxValue' isnot of type number

OPTION ARG={u'includedFields': [{u'fieldName': u'sine',
                      u'fieldType': u'float',
                      u'maxValue': u'1.0',
                      u'minValue': u'-1.0'}],
 u'inferenceArgs': {u'predictedField': u'sine', u'predictionSteps': [1]},
 u'inferenceType': u'TemporalAnomaly',
 u'streamDef': {u'info': u'sine',
                u'streams': [{u'columns': [u'*'],
                              u'info': u'sine.csv',

u'source':u'file:/home/marek/experiments/sine-waves/sine.csv'}],

                u'version': 1},
 u'swarmSize': u'medium'}






But when I start from scratch in new directory and download files:
generate_data.py
sine_experiment.py
nupic_output.py

from your repo:https://github.com/rhyolight/nupic.examples/tree/master/sine-prediction



marek@ubuntu:~/experiments/sine-waves-git$ ls -al
total 140
drwxrwxr-x 2 marek marek   4096 May 17 21:44 .
drwxrwxr-x 6 marek marek   4096 May 17 19:18 ..
-rw-rw-r-- 1 marek marek   1607 May 17 21:26 generate_data.py
-rw-rw-r-- 1 marek marek   5326 May 17 21:26 nupic_output.py
-rw-rw-r-- 1 marek marek 116399 May 17 21:26 sine.csv
-rwxrwxr-x 1 marek marek   2797 May 17 21:42 sine_experiment.py
marek@ubuntu:~/experiments/sine-waves-git$ pwd
/home/marek/experiments/sine-waves-git

And run

marek@ubuntu:~/experiments/sine-waves-git$ ./sine_experiment.py


Then the swarming starts with no problem and here is output:
http://termbin.com/nutm

So the result from this is: running swarm from command line does notworking (only for example data) but running swarm programmatically does.In youtube video is mentioned that the API has changed, so I supposethis may be related but I'm not sure. I've noticed few misleadings, e.g.inhttps://github.com/rhyolight/nupic.examples/blob/master/sine-prediction/sine_experiment.py


Following is defined bud never used
SWARM_DEF = "search_def.json"

Because this is replaced by
SWARM_CONFIG

Then I tried to test my json so I've replaced SWARM_CONFIG with my jsonand get same error as if I were running script from console :)

marek@ubuntu:~/experiments/sine-waves-git2/nupic.examples/sine-prediction$./sine_experiment.py

  nerating sine data into sine.csv
▽enerated 3000 rows of output data into sine.csv
Traceback (most recent call last):
  File "./sine_experiment.py", line 106, in <module>
    run_sine_experiment()
  File "./sine_experiment.py", line 78, in run_sine_experiment
    model_params = swarm_over_data()
  File "./sine_experiment.py", line 71, in swarm_over_data
    {'maxWorkers': 8, 'overwrite': True})