[google-appengine] Re: Importing data to datastore with bulkloader : error and long back off, but import finish ok

[Brian]

Hi Vincent,

The tools I hacked together are pretty rough, but not difficult.

I used the Python CSV module to read the source data CSV file and rewrite 
it into a tmp folder as multiple CSV part files with no more than 60 
records. Also included the writing of a command file that I can call to 
upload each part.

Good luck!

On Saturday, June 1, 2013 5:04:39 PM UTC-5, Vincent Lee wrote:
>
> brian,
>
> Can I borrow your local tools?
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] Re: Importing data to datastore with bulkloader : error and long back off, but import finish ok

[Vincent Lee]

brian,

Can I borrow your local tools?

On Tuesday, November 6, 2012 2:51:49 PM UTC-5, Brian wrote:
>
> Yes, I ran into the same problem a couple of years ago with the local SDK, 
> and again recently when I first deployed my app to appspot.com. The 
> solution that worked for me was to batch upload my data with no more that 
> 60 records in each batch. (Had to write some local tools just to work 
> around this problem!) Otherwise the delays kept expanding exponentially 
> until hours became days and days became weeks -- it would never finish the 
> upload. Hope this helps!
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] Re: Importing data to datastore with bulkloader : error and long back off, but import finish ok

[Brian]

Yes, I ran into the same problem a couple of years ago with the local SDK, 
and again recently when I first deployed my app to appspot.com. The 
solution that worked for me was to batch upload my data with no more that 
60 records in each batch. (Had to write some local tools just to work 
around this problem!) Otherwise the delays kept expanding exponentially 
until hours became days and days became weeks -- it would never finish the 
upload. Hope this helps!

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/dklg7iwwehAJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Importing data to datastore with bulkloader : error and long back off, but import finish ok

[Adam Serafini]

Same problem - does anyone know how to fix it?

On Tuesday, 22 November 2011 14:35:30 UTC, Mathieu Clavel wrote:
>
> Hello,
>
> I'm importing data to my app datastore (slave/master) with the bulkloader.
> I'm on Windows Xp, with Python 2.7.1 and Appengine SDK 1.6.0
>
> appcfg.py upload_data --num_threads=1 --batch_size=1000 --rps_limit=1000 
> --config_file=bulkloader.yaml --filename=contrats.csv --kind=Contrat --url "
> https://XXX.appspot.com/remote_api";
>
> Uploading data records.
> [INFO] Logging to bulkloader-log-2022.102427
> [INFO] Throttling transfers:
> [INFO] Bandwidth: 25 bytes/second
> [INFO] HTTP connections: 8/second
> [INFO] Entities inserted/fetched/modified: 1000/second
> [INFO] Batch Size: 1000
> [INFO] Opening database: bulkloader-progress-2022.102427.sql3
> [INFO] Connecting to XXX.appspot.com/remote_api
> [INFO] Starting import; maximum 1000 entities per post
> .[INFO] [WorkerThread-0] Backing off due to errors: 1.0 seconds
> .[INFO] [WorkerThread-0] Backing off due to errors: 2.0 seconds
> .[INFO] [WorkerThread-0] Backing off due to errors: 4.0 seconds
> .[INFO] [WorkerThread-0] Backing off due to errors: 8.0 seconds
> .[INFO] [WorkerThread-0] Backing off due to errors: 16.0 seconds
> .[INFO] [WorkerThread-0] Backing off due to errors: 32.0 seconds
> .[INFO] [WorkerThread-0] Backing off due to errors: 64.0 seconds
> .[INFO] [WorkerThread-0] Backing off due to errors: 128.0 seconds
> .[INFO] [WorkerThread-0] Backing off due to errors: 256.0 seconds
> .[INFO] [WorkerThread-0] Backing off due to errors: 512.0 seconds
> .[INFO] [WorkerThread-0] Backing off due to errors: 1024.0 seconds
> .[INFO] [WorkerThread-0] Backing off due to errors: 2048.0 seconds
> .[INFO] [WorkerThread-0] Backing off due to errors: 4096.0 seconds
> .[INFO] [WorkerThread-0] Backing off due to errors: 8192.0 seconds
> .[INFO] [WorkerThread-0] Backing off due to errors: 16384.0 seconds
>
> [INFO] 14115 entities total, 0 previously transferred
> [INFO] 14115 entities (17642966 bytes) transferred in 17328.7 seconds
> [INFO] All entities successfully transferred
>
> When looking in the log, I have this :
> [DEBUG2011-11-22 10:24:31,710 adaptive_thread_pool.py] 
> [WorkerThread-0] WorkerThread: started
> [DEBUG2011-11-22 10:24:32,444 bulkloader.py] [Thread-2] 
> ProgressTrackerThread: started
> [DEBUG2011-11-22 10:24:32,444 bulkloader.py] [Thread-3] 
> DataSourceThread: started
> [INFO 2011-11-22 10:24:32,444 bulkloader.py] Starting import; maximum 
> 1000 entities per post
> [DEBUG2011-11-22 10:24:33,085 adaptive_thread_pool.py] 
> [WorkerThread-0] Got work item [1-1000]
> [DEBUG2011-11-22 10:24:37,944 bulkloader.py] [Thread-3] 
> DataSourceThread: exiting
> [DEBUG2011-11-22 10:24:37,960 bulkloader.py] Waiting for worker 
> threads to finish...
> [DEBUG2011-11-22 10:25:45,318 bulkloader.py] [WorkerThread-0] [1-1000] 
> Transferred 1000 entities in 71.1 seconds
> [INFO 2011-11-22 10:25:45,318 adaptive_thread_pool.py] 
> [WorkerThread-0] Backing off due to errors: 1.0 seconds
> [DEBUG2011-11-22 10:25:46,318 adaptive_thread_pool.py] 
> [WorkerThread-0] Got work item [1001-2000]
> [DEBUG2011-11-22 10:26:39,582 bulkloader.py] [WorkerThread-0] 
> [1001-2000] Transferred 1000 entities in 53.2 seconds
> [INFO 2011-11-22 10:26:39,582 adaptive_thread_pool.py] 
> [WorkerThread-0] Backing off due to errors: 2.0 seconds
> [DEBUG2011-11-22 10:26:41,582 adaptive_thread_pool.py] 
> [WorkerThread-0] Got work item [2001-3000]
> [DEBUG2011-11-22 10:27:36,644 bulkloader.py] [WorkerThread-0] 
> [2001-3000] Transferred 1000 entities in 55.0 seconds
> [INFO 2011-11-22 10:27:36,644 adaptive_thread_pool.py] 
> [WorkerThread-0] Backing off due to errors: 4.0 seconds
> [DEBUG2011-11-22 10:27:40,644 adaptive_thread_pool.py] 
> [WorkerThread-0] Got work item [3001-4000]
> [DEBUG2011-11-22 10:28:34,846 bulkloader.py] [WorkerThread-0] 
> [3001-4000] Transferred 1000 entities in 54.2 seconds
> [INFO 2011-11-22 10:28:34,846 adaptive_thread_pool.py] 
> [WorkerThread-0] Backing off due to errors: 8.0 seconds
> [DEBUG2011-11-22 10:28:42,846 adaptive_thread_pool.py] 
> [WorkerThread-0] Got work item [4001-5000]
> [DEBUG2011-11-22 10:29:36,253 bulkloader.py] [WorkerThread-0] 
> [4001-5000] Transferred 1000 entities in 53.4 seconds
> [INFO 2011-11-22 10:29:36,253 adaptive_thread_pool.py] 
> [WorkerThread-0] Backing off due to errors: 16.0 seconds
> [DEBUG2011-11-22 10:29:52,253 adaptive_thread_pool.py] 
> [WorkerThread-0] Got work item [5001-6000]
> [DEBUG2011-11-22 10:30:45,611 bulkloader.py] [WorkerThread-0] 
> [5001-6000] Transferred 1000 entities in 53.3 seconds
> [INFO 2011-11-22 10:30:45,611 adaptive_thread_pool.py] 
> [WorkerThread-0] Backing off due to errors: 32.0 seconds
> [DEBUG2011-11-22 10:31:17,627 adaptive_thread_pool.py] 
> [