Hello, I'm importing data to my app datastore (slave/master) with the bulkloader. I'm on Windows Xp, with Python 2.7.1 and Appengine SDK 1.6.0
appcfg.py upload_data --num_threads=1 --batch_size=1000 --rps_limit=1000 --config_file=bulkloader.yaml --filename=contrats.csv --kind=Contrat --url "https://XXX.appspot.com/remote_api"; Uploading data records. [INFO ] Logging to bulkloader-log-20111122.102427 [INFO ] Throttling transfers: [INFO ] Bandwidth: 250000 bytes/second [INFO ] HTTP connections: 8/second [INFO ] Entities inserted/fetched/modified: 1000/second [INFO ] Batch Size: 1000 [INFO ] Opening database: bulkloader-progress-20111122.102427.sql3 [INFO ] Connecting to XXX.appspot.com/remote_api [INFO ] Starting import; maximum 1000 entities per post .[INFO ] [WorkerThread-0] Backing off due to errors: 1.0 seconds .[INFO ] [WorkerThread-0] Backing off due to errors: 2.0 seconds .[INFO ] [WorkerThread-0] Backing off due to errors: 4.0 seconds .[INFO ] [WorkerThread-0] Backing off due to errors: 8.0 seconds .[INFO ] [WorkerThread-0] Backing off due to errors: 16.0 seconds .[INFO ] [WorkerThread-0] Backing off due to errors: 32.0 seconds .[INFO ] [WorkerThread-0] Backing off due to errors: 64.0 seconds .[INFO ] [WorkerThread-0] Backing off due to errors: 128.0 seconds .[INFO ] [WorkerThread-0] Backing off due to errors: 256.0 seconds .[INFO ] [WorkerThread-0] Backing off due to errors: 512.0 seconds .[INFO ] [WorkerThread-0] Backing off due to errors: 1024.0 seconds .[INFO ] [WorkerThread-0] Backing off due to errors: 2048.0 seconds .[INFO ] [WorkerThread-0] Backing off due to errors: 4096.0 seconds .[INFO ] [WorkerThread-0] Backing off due to errors: 8192.0 seconds .[INFO ] [WorkerThread-0] Backing off due to errors: 16384.0 seconds [INFO ] 14115 entities total, 0 previously transferred [INFO ] 14115 entities (17642966 bytes) transferred in 17328.7 seconds [INFO ] All entities successfully transferred When looking in the log, I have this : [DEBUG 2011-11-22 10:24:31,710 adaptive_thread_pool.py] [WorkerThread-0] WorkerThread: started [DEBUG 2011-11-22 10:24:32,444 bulkloader.py] [Thread-2] ProgressTrackerThread: started [DEBUG 2011-11-22 10:24:32,444 bulkloader.py] [Thread-3] DataSourceThread: started [INFO 2011-11-22 10:24:32,444 bulkloader.py] Starting import; maximum 1000 entities per post [DEBUG 2011-11-22 10:24:33,085 adaptive_thread_pool.py] [WorkerThread-0] Got work item [1-1000] [DEBUG 2011-11-22 10:24:37,944 bulkloader.py] [Thread-3] DataSourceThread: exiting [DEBUG 2011-11-22 10:24:37,960 bulkloader.py] Waiting for worker threads to finish... [DEBUG 2011-11-22 10:25:45,318 bulkloader.py] [WorkerThread-0] [1-1000] Transferred 1000 entities in 71.1 seconds [INFO 2011-11-22 10:25:45,318 adaptive_thread_pool.py] [WorkerThread-0] Backing off due to errors: 1.0 seconds [DEBUG 2011-11-22 10:25:46,318 adaptive_thread_pool.py] [WorkerThread-0] Got work item [1001-2000] [DEBUG 2011-11-22 10:26:39,582 bulkloader.py] [WorkerThread-0] [1001-2000] Transferred 1000 entities in 53.2 seconds [INFO 2011-11-22 10:26:39,582 adaptive_thread_pool.py] [WorkerThread-0] Backing off due to errors: 2.0 seconds [DEBUG 2011-11-22 10:26:41,582 adaptive_thread_pool.py] [WorkerThread-0] Got work item [2001-3000] [DEBUG 2011-11-22 10:27:36,644 bulkloader.py] [WorkerThread-0] [2001-3000] Transferred 1000 entities in 55.0 seconds [INFO 2011-11-22 10:27:36,644 adaptive_thread_pool.py] [WorkerThread-0] Backing off due to errors: 4.0 seconds [DEBUG 2011-11-22 10:27:40,644 adaptive_thread_pool.py] [WorkerThread-0] Got work item [3001-4000] [DEBUG 2011-11-22 10:28:34,846 bulkloader.py] [WorkerThread-0] [3001-4000] Transferred 1000 entities in 54.2 seconds [INFO 2011-11-22 10:28:34,846 adaptive_thread_pool.py] [WorkerThread-0] Backing off due to errors: 8.0 seconds [DEBUG 2011-11-22 10:28:42,846 adaptive_thread_pool.py] [WorkerThread-0] Got work item [4001-5000] [DEBUG 2011-11-22 10:29:36,253 bulkloader.py] [WorkerThread-0] [4001-5000] Transferred 1000 entities in 53.4 seconds [INFO 2011-11-22 10:29:36,253 adaptive_thread_pool.py] [WorkerThread-0] Backing off due to errors: 16.0 seconds [DEBUG 2011-11-22 10:29:52,253 adaptive_thread_pool.py] [WorkerThread-0] Got work item [5001-6000] [DEBUG 2011-11-22 10:30:45,611 bulkloader.py] [WorkerThread-0] [5001-6000] Transferred 1000 entities in 53.3 seconds [INFO 2011-11-22 10:30:45,611 adaptive_thread_pool.py] [WorkerThread-0] Backing off due to errors: 32.0 seconds [DEBUG 2011-11-22 10:31:17,627 adaptive_thread_pool.py] [WorkerThread-0] Got work item [6001-7000] [DEBUG 2011-11-22 10:32:18,844 bulkloader.py] [WorkerThread-0] [6001-7000] Transferred 1000 entities in 61.2 seconds [INFO 2011-11-22 10:32:18,844 adaptive_thread_pool.py] [WorkerThread-0] Backing off due to errors: 64.0 seconds [DEBUG 2011-11-22 10:33:22,891 adaptive_thread_pool.py] [WorkerThread-0] Got work item [7001-8000] [DEBUG 2011-11-22 10:34:21,124 bulkloader.py] [WorkerThread-0] [7001-8000] Transferred 1000 entities in 58.2 seconds [INFO 2011-11-22 10:34:21,124 adaptive_thread_pool.py] [WorkerThread-0] Backing off due to errors: 128.0 seconds [DEBUG 2011-11-22 10:36:30,684 adaptive_thread_pool.py] [WorkerThread-0] Got work item [8001-9000] [DEBUG 2011-11-22 10:38:00,964 bulkloader.py] [WorkerThread-0] [8001-9000] Transferred 1000 entities in 90.2 seconds [INFO 2011-11-22 10:38:00,964 adaptive_thread_pool.py] [WorkerThread-0] Backing off due to errors: 256.0 seconds [DEBUG 2011-11-22 10:42:16,993 adaptive_thread_pool.py] [WorkerThread-0] Got work item [9001-10000] [DEBUG 2011-11-22 10:44:01,756 bulkloader.py] [WorkerThread-0] [9001-10000] Transferred 1000 entities in 104.7 seconds [INFO 2011-11-22 10:44:01,756 adaptive_thread_pool.py] [WorkerThread-0] Backing off due to errors: 512.0 seconds [DEBUG 2011-11-22 10:52:34,328 adaptive_thread_pool.py] [WorkerThread-0] Got work item [10001-11000] [DEBUG 2011-11-22 10:53:32,483 bulkloader.py] [WorkerThread-0] [10001-11000] Transferred 1000 entities in 58.1 seconds [INFO 2011-11-22 10:53:32,483 adaptive_thread_pool.py] [WorkerThread-0] Backing off due to errors: 1024.0 seconds [DEBUG 2011-11-22 11:10:37,174 adaptive_thread_pool.py] [WorkerThread-0] Got work item [11001-12000] [DEBUG 2011-11-22 11:11:34,361 bulkloader.py] [WorkerThread-0] [11001-12000] Transferred 1000 entities in 57.2 seconds [INFO 2011-11-22 11:11:34,361 adaptive_thread_pool.py] [WorkerThread-0] Backing off due to errors: 2048.0 seconds [DEBUG 2011-11-22 11:45:43,578 adaptive_thread_pool.py] [WorkerThread-0] Got work item [12001-13000] [DEBUG 2011-11-22 11:46:42,576 bulkloader.py] [WorkerThread-0] [12001-13000] Transferred 1000 entities in 59.0 seconds [INFO 2011-11-22 11:46:42,576 adaptive_thread_pool.py] [WorkerThread-0] Backing off due to errors: 4096.0 seconds [DEBUG 2011-11-22 12:55:00,586 adaptive_thread_pool.py] [WorkerThread-0] Got work item [13001-14000] [DEBUG 2011-11-22 12:56:00,555 bulkloader.py] [WorkerThread-0] [13001-14000] Transferred 1000 entities in 59.9 seconds [INFO 2011-11-22 12:56:00,555 adaptive_thread_pool.py] [WorkerThread-0] Backing off due to errors: 8192.0 seconds [DEBUG 2011-11-22 15:12:41,778 adaptive_thread_pool.py] [WorkerThread-0] Got work item [14001-14115] [DEBUG 2011-11-22 15:13:15,792 bulkloader.py] [WorkerThread-0] [14001-14115] Transferred 115 entities in 17.1 seconds [INFO 2011-11-22 15:13:15,823 adaptive_thread_pool.py] [WorkerThread-0] Backing off due to errors: 16384.0 seconds [DEBUG 2011-11-22 15:13:15,871 adaptive_thread_pool.py] Waiting for Anonymous_WorkerThread to exit [DEBUG 2011-11-22 15:13:16,089 adaptive_thread_pool.py] [WorkerThread-0] WorkerThread: exiting [DEBUG 2011-11-22 15:13:16,167 bulkloader.py] Waiting for progress_thread to terminate... [DEBUG 2011-11-22 15:13:16,371 bulkloader.py] [Thread-2] ProgressTrackerThread: exiting [DEBUG 2011-11-22 15:13:16,417 bulkloader.py] ... done. [INFO 2011-11-22 15:13:16,434 bulkloader.py] 14115 entities total, 0 previously transferred [INFO 2011-11-22 15:13:16,434 bulkloader.py] 14115 entities (17642966 bytes) transferred in 17328.7 seconds [INFO 2011-11-22 15:13:16,434 bulkloader.py] All entities successfully transferred The import was working ok before, but I can't tell since when the problem started : SDK 1.6.0 or GAE new pricing. Does anyone know how to fix it ? Thanks, Mathieu -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/mNVt5exklFMJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Go to Master Mind Web Earner Blog :- http://adf.ly/624707/www.bignlife.in http://adf.ly/624707/www.bignlife.in Unsubscribe All Group Post :- http://j.gs/624707/unsubscribe http://q.gs/624707/unsubscribe Subscribe All Group Post :- http://j.gs/624707/subscribe http://q.gs/624707/subscribe Make Money E-mail Sending Job :- http://adf.ly/624707/www.bignlife.in http://adf.ly/624707/www.bignlife.in http://adf.ly/624707/www.bignlife.in http://adf.ly/624707/www.bignlife.in http://adf.ly/624707/www.bignlife.in http://adf.ly/624707/www.bignlife.in http://adf.ly/624707/www.bignlife.in http://adf.ly/624707/www.bignlife.in http://adf.ly/624707/www.bignlife.in http://adf.ly/624707/www.bignlife.in http://adf.ly/624707/www.bignlife.in -- You received this message because you are subscribed to the Google Groups "SAP-Jobs-SAP-FAQ" group. To post to this group, send email to sap-jobs-sap-...@googlegroups.com. To unsubscribe from this group, send email to sap-jobs-sap-faq+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sap-jobs-sap-faq?hl=en.
