fmcquillan99 commented on issue #16: MADlib image loader: load dataset from disk URL: https://github.com/apache/madlib-site/pull/16#issuecomment-512584393 If number of examples is less than `ROWS_PER_FILE` then no rows get written to the target table, it seems. ``` %sql drop table if exists load_train_1; # Load images from file system iloader.load_dataset_from_disk('/Users/fmcquillan/tmp/cifar-10-png-sample1', 'load_train_1', num_labels='all', append=False, no_temp_files=False) ``` produces ``` Done. MainProcess: Connected to madlib db. Executing: CREATE TABLE load_train_1 (id SERIAL, x REAL[], y TEXT, img_name TEXT) CREATE TABLE Created table load_train_1 in madlib db .DS_Store is not a directory, skipping number of labels = 3 Found 3 image labels in /Users/fmcquillan/tmp/cifar-10-png-sample1 Spawning 5 workers... Initializing PoolWorker-1 [pid 48064] Initializing PoolWorker-2 [pid 48065] PoolWorker-1: Created temporary directory /tmp/madlib_lOZkc8k825 Initializing PoolWorker-3 [pid 48066] PoolWorker-2: Created temporary directory /tmp/madlib_B8aEPagCVQ PoolWorker-3: Created temporary directory /tmp/madlib_dbgZdbFCkG Initializing PoolWorker-4 [pid 48067] Initializing PoolWorker-5 [pid 48068] PoolWorker-4: Created temporary directory /tmp/madlib_opXnfyKRjF PoolWorker-5: Created temporary directory /tmp/madlib_YLxwFljDy1 PoolWorker-4: Connected to madlib db. PoolWorker-2: Connected to madlib db. PoolWorker-1: Connected to madlib db. PoolWorker-5: Connected to madlib db. PoolWorker-3: Connected to madlib db. filenames ['35097_truck.png', '35116_truck.png', '35127_truck.png'] filenames ['35096_bird.png', '35104_bird.png', '35107_bird.png'] index 0 filename 35096_bird.png index 1 filename 35104_bird.png index 2 filename 35107_bird.png filenames ['35100_cat.png', '35101_cat.png', '35102_cat.png'] index 0 filename 35097_truck.png index 0 filename 35100_cat.png index 1 filename 35116_truck.png index 1 filename 35101_cat.png index 2 filename 35127_truck.png index 2 filename 35102_cat.png PoolWorker-5: Removed temporary directory /tmp/madlib_YLxwFljDy1 PoolWorker-3: Removed temporary directory /tmp/madlib_dbgZdbFCkG PoolWorker-4: Removed temporary directory /tmp/madlib_opXnfyKRjF PoolWorker-2: Removed temporary directory /tmp/madlib_B8aEPagCVQ PoolWorker-1: Removed temporary directory /tmp/madlib_lOZkc8k825 Done! Loaded 3 image categories in 0.356377124786s 5 workers terminated. ``` and get empty table ``` madlib=# select count(*) from load_train_1; count ------- 0 (1 row) ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
