[web2py] Re: Importing a large file

2010-01-12 Thread Brian M
Jason, glad you've got things working! :) No, you don't want to try showing that many records on screen all at once. Well at least not if you're letting web2py format it automatically, in experimenting with my own good sized datasets I've found that if you code out the table yourself in a view it

Re: [web2py] Re: Importing a large file

2010-01-12 Thread Jason Brower
Cool! That seemed to be importing properly. I have to be careful with displaying that much data in my browser. Actually filled my 2gb of ram! One more small thing to note here is that it doesn't account for old paint names in the database. So running the same file twice dupplicates, what seems t

[web2py] Re: Importing a large file

2010-01-11 Thread Brian M
Jason, put your csv file in the private subfolder of your application and call it "paint.csv" and then add the following code to your controller and it should work. (Well at least it imported the few records you provided, I haven't taken the time to totally check that everything lines up. ~Brian

Re: [web2py] Re: Importing a large file

2010-01-11 Thread Jason Brower
Your requested some more data... "paint_name","base_paint","colorant","amount","colorant","amount","colorant","amount","colorant","amount","colorant","amount" "10 JORASDUN",C,DD1,56,fS1,42,So1,110,AA1,8,, "10 SVFGH",W,To1,180 1002-DG5R,R,Co2,30,la1,4,To1,5 1008-Y24R,A,DD2,88,la1,17,Fo1,

Re: [web2py] Re: Importing a large file

2010-01-11 Thread Jason Brower
Using only what I know I have implemented this... def rebuild_database: import sys data = open("/home/jason/Desktop/colorant_tints.txt") parsed_data = [] for line in data.readlines(): parsed_data.append(line.rsplit(",")) sys.stdout.write(".") for line in parsed_d

Re: [web2py] Re: Importing a large file

2010-01-11 Thread Jason Brower
It seems to give me the error: type error '_csv.reader' object is unsubscriptable Regards, Jason On Sat, 2010-01-09 at 07:32 -0800, Brian M wrote: > paint_formulas is supposed to get the contents of the csv file. (See > http://docs.python.org/library/csv.html) It should be populated by the > lin

[web2py] Re: Importing a large file

2010-01-09 Thread Brian M
paint_formulas is supposed to get the contents of the csv file. (See http://docs.python.org/library/csv.html) It should be populated by the line: paint_formulas = csv.reader(csvfile) where csv file should be the path to your csv file. Actually, I think that line should read like this instead:

Re: [web2py] Re: Importing a large file

2010-01-08 Thread Jason Brower
It is much more understandable. But it seems that the name paint_formulas is not defined. What is supposed to be populated there? Best Regards, Jason On Fri, 2010-01-08 at 20:43 -0800, Brian M wrote: > Jason, > What's the syntax error? > Try this, it's untested but should be verbose enough to ge

[web2py] Re: Importing a large file

2010-01-08 Thread Brian M
Jason, What's the syntax error? Try this, it's untested but should be verbose enough to get you there. If you have trouble perhaps provide a sample file (few dozen records) that we can test against. import csv paint_formulas = csv.reader(csvfile) header = None #variable to store the paints we've

Re: [web2py] Re: Importing a large file

2010-01-08 Thread Jason Brower
I tried the script but couldn't get through the syntax error. Sorry, lots of commands I don't know there. :/ BR, Jason On Fri, 2010-01-08 at 09:17 -0800, Brian M wrote: > I have been working on using web2py to import csv files and find that > in general it works very well. You will need to be car

[web2py] Re: Importing a large file

2010-01-08 Thread Brian M
I have been working on using web2py to import csv files and find that in general it works very well. You will need to be careful with pulling in the colorant and amount fields - if you use the csv.DictReader() to refer to columns in the CSV file by name instead of index you'll find that you only ge

[web2py] Re: Importing a large file

2010-01-08 Thread mdipierro
Something like this? import csv reader = csv.reader(csvfile) header = None paints = {} colorants = {} for line in reader: if not header: header = line else: row = dict([header[i],item) for i,item in enumerate(line)]) name = row['Color Name'] colorant = row['Colorant