fmcquillan99 commented on issue #392: DL: Improve performance for predict URL: https://github.com/apache/madlib/pull/392#issuecomment-494201060 LGTM (0) timings on a 2 segment GP5 cluster on CPU ``` 10K predictions Time: 6077.533 ms 50K predictions Time: 26784.859 ms ``` This is much better than before, so looks OK for now. (1) response ``` DROP TABLE IF EXISTS mnist_test_prediction; SELECT madlib.madlib_keras_predict('model1', -- model 'mnist_test', -- test_table 'id', -- id column 'x', -- independent var 'mnist_test_prediction' -- output table ); SELECT round(count(*)*100/10000.0,2) as test_accuracy_percent from (select mnist_test.y as actual, mnist_test_prediction.estimated_y as estimated from mnist_test_prediction inner join mnist_test on mnist_test.id=mnist_test_prediction.id) q WHERE q.actual=q.estimated; test_accuracy_percent ----------------------- 97.18 (1 row) ``` OK, accuracy as expected (2) prob ``` DROP TABLE IF EXISTS mnist_test_prediction; SELECT madlib.madlib_keras_predict('model1', -- model 'mnist_test', -- test_table 'id', -- id column 'x', -- independent var 'mnist_test_prediction', -- output table 'prob' ); SELECT * FROM mnist_test_prediction LIMIT 10; id | prob_0 | prob_1 | prob_2 | prob_3 | prob_4 | prob_5 | prob_6 | prob_7 | prob_8 | prob_9 ----+---------------+----------------+---------------+---------------+---------------+---------------+----------------+---------------+---------------+---------------- 2 | 9.880677e-07 | 0.0010419372 | 0.9986218 | 6.84927e-05 | 1.2382638e-12 | 0.00024825096 | 6.948901e-07 | 6.1711205e-13 | 1.7804521e-05 | 6.0889234e-14 4 | 0.9999366 | 2.8256621e-09 | 6.6410103e-06 | 8.4593694e-07 | 7.4489925e-08 | 3.2383898e-05 | 2.9831072e-06 | 9.729254e-06 | 5.108415e-08 | 1.07344185e-05 12 | 0.0011051608 | 4.6904443e-08 | 3.491906e-05 | 2.4090486e-05 | 2.4900895e-05 | 0.0010728389 | 0.99681705 | 1.4937936e-07 | 0.00092049537 | 2.8245486e-07 14 | 0.9999608 | 1.6933312e-08 | 3.732735e-06 | 5.4127717e-08 | 9.5467556e-08 | 4.5280945e-06 | 3.560989e-06 | 4.5551647e-06 | 2.301657e-06 | 2.0342342e-05 20 | 2.2352947e-07 | 9.69698e-09 | 1.6347584e-06 | 1.932719e-06 | 0.9992822 | 3.8802187e-05 | 4.0969678e-07 | 3.759048e-05 | 3.6773997e-06 | 0.0006335543 22 | 4.0293417e-05 | 6.249896e-09 | 2.8016773e-06 | 7.212842e-07 | 4.9328148e-05 | 0.0026104918 | 0.9971624 | 1.2448003e-08 | 0.00012794712 | 5.9117415e-06 26 | 0.9999931 | 1.24117915e-14 | 2.2590778e-09 | 1.8620958e-12 | 6.718903e-08 | 2.8648715e-11 | 5.973386e-06 | 4.3913162e-10 | 2.1172746e-09 | 8.0753307e-07 28 | 1.634239e-07 | 3.018619e-10 | 3.811128e-06 | 7.628142e-08 | 0.9992101 | 5.254892e-05 | 1.4238293e-06 | 5.970224e-06 | 3.5786666e-06 | 0.0007222573 34 | 0.033928625 | 1.19020935e-08 | 7.676703e-05 | 1.4824612e-07 | 0.94482905 | 0.0005254827 | 0.020210627 | 0.00023137043 | 7.968004e-05 | 0.00011828837 36 | 2.8703305e-06 | 0.0022232432 | 0.9966936 | 0.0010433431 | 5.737119e-11 | 1.1605422e-05 | 1.04220526e-08 | 3.3659873e-08 | 2.5263162e-05 | 8.308043e-09 (10 rows) ``` OK (3) ask for a GPU but there isn't one avail ``` DROP TABLE IF EXISTS mnist_test_prediction; SELECT madlib.madlib_keras_predict('model1', -- model 'mnist_test', -- test_table 'id', -- id column 'x', -- independent var 'mnist_test_prediction', -- output table NULL, 1 -- GPU's per segment host ); ERROR: plpy.SPIError: plpy.Error: Cannot assign a device for operation Placeholder: node Placeholder (defined at /home/gpadmin/.local/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py:2464) was explicitly assigned to /device:GPU:0 but available devices are [ /job:localhost/replica:0/task:0/device:CPU:0, /job:localhost/replica:0/task:0/device:XLA_CPU:0 ]. Make sure the device specification refers to a valid device. The requested device appears to be a GPU, but CUDA is not enabled. (plpython.c:5038) (seg1 slice1 10.128.0.41:40001 pid=14544) (plpython.c:5038) DETAIL: [[node Placeholder (defined at /home/gpadmin/.local/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py:2464) ]] Caused by op u'Placeholder', defined at: File "<string>", line 1, in <module> File "<string>", line 8, in __plpython_procedure_internal_keras_predict_66207 File "/home/gpadmin/madlib/build/src/ports/greenplum/5/modules/deep_learning/madlib_keras_predict.py", line 165, in internal_keras_predict set_model_weights(model, device_name, model_data, model_shapes) File "/home/gpadmin/madlib/build/src/ports/greenplum/5/modu Traceback (most recent call last): PL/Python function "internal_keras_predict", line 7, in <module> return madlib_keras_predict.internal_keras_predict(**globals()) PL/Python function "internal_keras_predict", line 210, in internal_keras_predict PL/Python function "internal_keras_predict" CONTEXT: Traceback (most recent call last): PL/Python function "madlib_keras_predict", line 28, in <module> gpus_per_host) PL/Python function "madlib_keras_predict", line 112, in predict PL/Python function "madlib_keras_predict" ``` OK
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
