Thanks for your interest Dave. Yes the files are big, the last one was 8mb with about 44k records
About the scheduler. i start it manually at the control panel launching the script below ("python scheduler.py" from a pythonanywhere console). After a few days it becomes unresponsive and i have to kill the console and restart it. Also pythonanywhere use to stops their servers 30 minutes once in a month for mantainance. Im using a template they provided. (guess thats to avoid multiple instances) #/usr/bin/env python import os import socket import subprocess import sys filename = os.path.abspath(__file__) # we use this to generate a unique socket name # we use a local socket as a lock. # it can only be bound once, and will be released if the process dies # we want to keep the socket open until the very end of # our script so we use a global variable to avoid going # out of scope and being garbage-collected lock_socket = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM) try: lock_socket.bind('\0' + filename) except socket.error: print("Failed to acquire lock, task must already be running") sys.exit() subprocess.call(["python", "web2py/web2py.py", "-K", "myapp"]) I also have the problem of concurrency, each user should be able to import their owns csv files. This cannot be achieved by the scheduler in a sane way. You have to declare the how many workers you need in advance (i dont know how many users are gonna upload files at the same time) and if i declare like lets say, 3 or 4 workers, im adding a lot of overhead to the db because of the heartbeats and task's outputs. I have just did the same test with those 40k records on a custom thread as the first post, not using dal, but mysql.connector so the thread has its own connection, it took 10 seconds, when the scheduler takes 30+minutes. The question remains, is there anyway to use dal instead mysql.connector? Ive try this inside the thread's __init__ self.db = DAL(cnnstring, ..., migrate=False) self.db.define_table(fields.., migrate=False) but doesnt work, i get the error above. Thanks -- Resources: - http://web2py.com - http://web2py.com/book (Documentation) - http://github.com/web2py/web2py (Source code) - https://code.google.com/p/web2py/issues/list (Report Issues) --- You received this message because you are subscribed to the Google Groups "web2py-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.