fmcquillan99 edited a comment on issue #451: DL: Update evaluate and predict for multi model outputs URL: https://github.com/apache/madlib/pull/451#issuecomment-549566751 LGTM Retesting after last commits (4) (5) (6) (7) (8) (9) (10) below seems OK: (1) review after recent commit needed to re-run to get new results: ``` DROP TABLE IF EXISTS iris_multi_model, iris_multi_model_summary, iris_multi_model_info; SELECT madlib.madlib_keras_fit_multiple_model('iris_train_packed', -- source_table 'iris_multi_model', -- model_output_table 'mst_table', -- model_selection_table 10, -- num_iterations 0, -- use gpus 'iris_test_packed' -- validation dataset ); SELECT * FROM iris_multi_model_info ORDER BY training_metrics_final DESC; -[ RECORD 1 ]------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------- -------------------- mst_key | 2 model_id | 1 compile_params | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['accuracy'] fit_params | batch_size=8,epochs=1 model_type | madlib_keras model_size | 0.7900390625 metrics_elapsed_time | {0.16501784324646,0.154597043991089,0.158796072006226,0.171799898147583,0.168801069259644,0.160946846008301,0.168446063995361,0.249520063400269,0.723811149597168, 0.157901048660278} metrics_type | {accuracy} training_metrics_final | 0.975000023842 training_loss_final | 0.184436321259 training_metrics | {0.666666686534882,0.708333313465118,0.975000023841858,0.975000023841858,0.458333343267441,0.666666686534882,0.666666686534882,0.716666638851166,0.816666662693024 ,0.975000023841858} training_loss | {0.654977560043335,0.473680555820465,0.407350271940231,0.344571173191071,1.01371514797211,0.536512732505798,0.542378485202789,0.427120715379715,0.301178097724915, 0.184436321258545} validation_metrics_final | 0.966666638851 validation_loss_final | 0.229953154922 validation_metrics | {0.666666686534882,0.633333325386047,0.933333337306976,0.933333337306976,0.333333343267441,0.666666686534882,0.666666686534882,0.666666686534882,0.800000011920929 ,0.966666638851166} validation_loss | {0.632323026657104,0.544658362865448,0.475189208984375,0.403949052095413,1.15320909023285,0.571010947227478,0.57290381193161,0.472318321466446,0.329961955547333,0 .229953154921532} -[ RECORD 2 ]------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------- -------------------- mst_key | 6 model_id | 1 compile_params | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['accuracy'] fit_params | batch_size=8,epochs=1 model_type | madlib_keras model_size | 0.7900390625 metrics_elapsed_time | {0.151521921157837,0.170912981033325,0.162657022476196,0.154775142669678,0.152937889099121,0.15539813041687,0.167464017868042,0.155567169189453,0.182074785232544, 0.164788961410522} metrics_type | {accuracy} training_metrics_final | 0.933333337307 training_loss_final | 0.653343558311 training_metrics | {0.00833333376795053,0.133333340287209,0.358333319425583,0.491666674613953,0.75,0.958333313465118,0.958333313465118,0.949999988079071,0.908333361148834,0.93333333 7306976} training_loss | {1.24945497512817,1.10760903358459,1.04750990867615,0.997703850269318,0.913850665092468,0.828234732151031,0.770057380199432,0.728686928749084,0.687141954898834,0. 653343558311462} validation_metrics_final | 0.833333313465 validation_loss_final | 0.668307900429 validation_metrics | {0,0.100000001490116,0.433333337306976,0.566666662693024,0.766666650772095,0.899999976158142,0.899999976158142,0.866666674613953,0.800000011920929,0.8333333134651 18} validation_loss | {1.30171895027161,1.11200094223022,1.03035044670105,1.00190925598145,0.922983944416046,0.828430116176605,0.771750569343567,0.736835598945618,0.698546707630157,0.6 68307900428772} -[ RECORD 3 ]------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------- -------------------- mst_key | 8 model_id | 2 compile_params | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['accuracy'] fit_params | batch_size=8,epochs=1 model_type | madlib_keras model_size | 1.2197265625 metrics_elapsed_time | {0.178833961486816,0.178119897842407,0.193637132644653,0.183725118637085,0.181838989257812,0.190093994140625,0.190335988998413,0.183039903640747,0.189705848693848 ,0.196530103683472} metrics_type | {accuracy} training_metrics_final | 0.925000011921 training_loss_final | 0.191128313541 training_metrics | {0.675000011920929,0.699999988079071,0.708333313465118,0.850000023841858,0.691666662693024,0.966666638851166,0.949999988079071,0.858333349227905,0.766666650772095 ,0.925000011920929} training_loss | {1.00050187110901,0.477404177188873,0.449770420789719,0.367935240268707,0.715284705162048,0.233321651816368,0.206968143582344,0.247707515954971,0.513520300388336, 0.191128313541412} validation_metrics_final | 0.899999976158 validation_loss_final | 0.238029867411 validation_metrics | {0.566666662693024,0.666666686534882,0.666666686534882,0.833333313465118,0.566666662693024,0.966666638851166,0.933333337306976,0.866666674613953,0.699999988079071 ,0.899999976158142} validation_loss | {1.10490369796753,0.515014708042145,0.473093926906586,0.429382801055908,0.964671075344086,0.282983720302582,0.234358683228493,0.304766476154327,0.506977200508118, 0.23802986741066} -[ RECORD 4 ]------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------- -------------------- mst_key | 9 model_id | 2 compile_params | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['accuracy'] fit_params | batch_size=4,epochs=1 model_type | madlib_keras model_size | 1.2197265625 metrics_elapsed_time | {0.179304122924805,0.177167892456055,0.197530031204224,0.186438083648682,0.183886051177979,0.179242849349976,0.261205911636353,0.180387020111084,0.190798997879028 ,0.199844121932983} metrics_type | {accuracy} training_metrics_final | 0.925000011921 training_loss_final | 0.129962354898 training_metrics | {0.666666686534882,0.875,0.975000023841858,0.758333325386047,0.966666638851166,0.800000011920929,0.966666638851166,0.983333349227905,0.966666638851166,0.925000011 920929} training_loss | {0.631272912025452,0.311742216348648,0.201621904969215,0.389256983995438,0.119620576500893,0.486419379711151,0.105227634310722,0.0892100557684898,0.13771826028823 9,0.129962354898453} validation_metrics_final | 1 validation_loss_final | 0.099272236228 validation_metrics | {0.666666686534882,0.833333313465118,0.966666638851166,0.733333349227905,0.899999976158142,0.733333349227905,0.899999976158142,0.966666638851166,0.866666674613953 ,1} validation_loss | {0.655599772930145,0.343398422002792,0.23225425183773,0.460039615631104,0.179262787103653,0.432839661836624,0.164745837450027,0.110677324235439,0.236034005880356, 0.0992722362279892} -[ RECORD 5 ]------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------- -------------------- mst_key | 12 model_id | 2 compile_params | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['accuracy'] fit_params | batch_size=8,epochs=1 model_type | madlib_keras model_size | 1.2197265625 metrics_elapsed_time | {0.179224967956543,0.178079128265381,0.202226161956787,0.190601825714111,0.179889917373657,0.182010173797607,0.223275899887085,0.186384916305542,0.1926109790802,0 .205245018005371} metrics_type | {accuracy} training_metrics_final | 0.916666686535 training_loss_final | 0.557382702827 training_metrics | {0.400000005960464,0.466666668653488,0.641666650772095,0.758333325386047,0.850000023841858,0.875,0.866666674613953,0.908333361148834,0.908333361148834,0.916666686 534882} training_loss | {1.03677904605865,0.961223006248474,0.898576974868774,0.84269106388092,0.784159600734711,0.72914981842041,0.679587423801422,0.637798070907593,0.592370092868805,0. 557382702827454} validation_metrics_final | 0.933333337307 validation_loss_final | 0.633718490601 validation_metrics | {0.233333334326744,0.333333343267441,0.433333337306976,0.666666686534882,0.800000011920929,0.933333337306976,1,0.933333337306976,0.966666638851166,0.9333333373069 76} validation_loss | {1.07957482337952,1.01587510108948,0.959396243095398,0.91132265329361,0.857007622718811,0.808456540107727,0.762722790241241,0.718301713466644,0.671158373355865,0. 633718490600586} -[ RECORD 6 ]------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------- -------------------- mst_key | 10 model_id | 2 compile_params | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['accuracy'] fit_params | batch_size=8,epochs=1 model_type | madlib_keras model_size | 1.2197265625 metrics_elapsed_time | {0.185405015945435,0.240031957626343,0.179027080535889,0.188817024230957,0.186041116714478,0.189532041549683,0.191230058670044,0.189625978469849,0.199972867965698 ,0.184904098510742} metrics_type | {accuracy} training_metrics_final | 0.899999976158 training_loss_final | 0.202067151666 training_metrics | {0.691666662693024,0.675000011920929,0.691666662693024,0.875,0.899999976158142,0.983333349227905,0.983333349227905,0.908333361148834,0.933333337306976,0.899999976 158142} training_loss | {0.936435461044312,0.705836236476898,0.493957877159119,0.422138810157776,0.360373467206955,0.24227599799633,0.1996029317379,0.195797547698021,0.160537555813789,0. 202067151665688} validation_metrics_final | 1 validation_loss_final | 0.204844340682 validation_metrics | {0.566666662693024,0.566666662693024,0.566666662693024,0.833333313465118,1,0.966666638851166,0.933333337306976,1,1,1} validation_loss | {0.968072831630707,0.788067102432251,0.579671025276184,0.491580307483673,0.424855649471283,0.284231871366501,0.230658054351807,0.215303242206573,0.172062397003174 ,0.20484434068203} -[ RECORD 7 ]------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------- -------------------- mst_key | 4 model_id | 1 compile_params | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['accuracy'] fit_params | batch_size=8,epochs=1 model_type | madlib_keras model_size | 0.7900390625 metrics_elapsed_time | {0.158287048339844,0.155457973480225,0.576164960861206,0.162032127380371,0.15852689743042,0.156464099884033,0.161139965057373,0.160786867141724,0.167949914932251, 0.656886100769043} metrics_type | {accuracy} training_metrics_final | 0.899999976158 training_loss_final | 0.169732123613 training_metrics | {0.666666686534882,0.666666686534882,0.816666662693024,0.991666674613953,0.741666674613953,0.899999976158142,0.925000011920929,0.741666674613953,0.983333349227905 ,0.899999976158142} training_loss | {0.76077276468277,0.5220667719841,0.420222669839859,0.327254116535187,0.341780602931976,0.252596199512482,0.206999227404594,0.435290902853012,0.150588378310204,0. 169732123613358} validation_metrics_final | 1 validation_loss_final | 0.175337061286 validation_metrics | {0.666666686534882,0.666666686534882,0.666666686534882,0.933333337306976,0.666666686534882,0.833333313465118,1,0.633333325386047,0.966666638851166,1} validation_loss | {0.767970025539398,0.556378245353699,0.499317765235901,0.392887800931931,0.395217329263687,0.307017415761948,0.249646931886673,0.518399477005005,0.179019212722778 ,0.175337061285973} -[ RECORD 8 ]------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------- -------------------- mst_key | 5 model_id | 1 compile_params | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['accuracy'] fit_params | batch_size=4,epochs=1 model_type | madlib_keras model_size | 0.7900390625 metrics_elapsed_time | {0.152910947799683,0.177506923675537,0.163512945175171,0.153976917266846,0.15569806098938,0.155301809310913,0.166845083236694,0.15528392791748,0.162101984024048,0 .198511123657227} metrics_type | {accuracy} training_metrics_final | 0.875 training_loss_final | 0.530299961567 training_metrics | {0.333333343267441,0.191666662693024,0.508333325386047,0.791666686534882,0.883333325386047,0.816666662693024,0.875,0.783333361148834,0.75,0.875} training_loss | {1.42261958122253,1.11098456382751,0.942022323608398,0.839245975017548,0.778925061225891,0.719573795795441,0.665915071964264,0.617532074451447,0.570941388607025,0 .530299961566925} validation_metrics_final | 0.866666674614 validation_loss_final | 0.546338915825 validation_metrics | {0.333333343267441,0.16666667163372,0.400000005960464,0.833333313465118,0.899999976158142,0.833333313465118,0.899999976158142,0.833333313465118,0.699999988079071, 0.866666674613953} validation_loss | {1.58205306529999,1.19787907600403,0.984605967998505,0.874651849269867,0.803973138332367,0.732114672660828,0.681202590465546,0.62757009267807,0.578481078147888,0. 54633891582489} -[ RECORD 9 ]------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------- -------------------- mst_key | 3 model_id | 1 compile_params | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['accuracy'] fit_params | batch_size=4,epochs=1 model_type | madlib_keras model_size | 0.7900390625 metrics_elapsed_time | {0.155992031097412,0.153978109359741,0.572959899902344,0.162055015563965,0.155320167541504,0.156923055648804,0.163105964660645,0.161538124084473,0.168941020965576 ,0.654891967773438} metrics_type | {accuracy} training_metrics_final | 0.824999988079 training_loss_final | 0.347870081663 training_metrics | {0.633333325386047,0.766666650772095,0.666666686534882,0.666666686534882,0.691666662693024,0.691666662693024,0.908333361148834,0.899999976158142,0.675000011920929 ,0.824999988079071} training_loss | {0.815440833568573,0.520094633102417,0.449681550264359,0.457792103290558,0.437296599149704,0.43956533074379,0.400028377771378,0.407060354948044,0.410939812660217, 0.347870081663132} validation_metrics_final | 0.800000011921 validation_loss_final | 0.418093115091 validation_metrics | {0.600000023841858,0.666666686534882,0.666666686534882,0.666666686534882,0.566666662693024,0.566666662693024,0.833333313465118,0.866666674613953,0.666666686534882 ,0.800000011920929} validation_loss | {0.782467901706696,0.585534811019897,0.514401435852051,0.50047355890274,0.542211294174194,0.564250826835632,0.486649245023727,0.488315939903259,0.454440474510193, 0.418093115091324} -[ RECORD 10 ]-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------- -------------------- mst_key | 1 model_id | 1 compile_params | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['accuracy'] fit_params | batch_size=4,epochs=1 model_type | madlib_keras model_size | 0.7900390625 metrics_elapsed_time | {0.154391050338745,0.155292987823486,0.166854858398438,0.157708883285522,0.15703010559082,0.15372896194458,0.161499977111816,0.15468692779541,0.163357973098755,0. 166243076324463} metrics_type | {accuracy} training_metrics_final | 0.716666638851 training_loss_final | 0.498635411263 training_metrics | {0.666666686534882,0.666666686534882,0.816666662693024,0.991666674613953,0.824999988079071,0.933333337306976,0.824999988079071,0.975000023841858,0.816666662693024 ,0.716666638851166} training_loss | {0.597955822944641,0.43520125746727,0.371508300304413,0.0662390738725662,0.332939654588699,0.406542897224426,0.290035516023636,0.127334654331207,0.290014386177063 ,0.498635411262512} validation_metrics_final | 0.699999988079 validation_loss_final | 0.547126591206 validation_metrics | {0.666666686534882,0.666666686534882,0.800000011920929,0.933333337306976,0.800000011920929,1,0.766666650772095,0.933333337306976,0.833333313465118,0.6999999880790 71} validation_loss | {0.594180822372437,0.464261531829834,0.383160322904587,0.0937399193644524,0.414926111698151,0.368867307901382,0.376850306987762,0.15509657561779,0.317675232887268 ,0.547126591205597} -[ RECORD 11 ]-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------- -------------------- mst_key | 11 model_id | 2 compile_params | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['accuracy'] fit_params | batch_size=4,epochs=1 model_type | madlib_keras model_size | 1.2197265625 metrics_elapsed_time | {0.189064979553223,0.181553840637207,0.179675817489624,0.196763038635254,0.186076879501343,0.191986083984375,0.194756984710693,0.19181489944458,0.202284812927246, 0.188621997833252} metrics_type | {accuracy} training_metrics_final | 0.708333313465 training_loss_final | 0.781156182289 training_metrics | {0.333333343267441,0.333333343267441,0.333333343267441,0.308333337306976,0.333333343267441,0.333333343267441,0.474999994039536,0.758333325386047,0.75,0.7083333134 65118} training_loss | {1.1848406791687,1.14267539978027,1.11269533634186,1.09739363193512,1.09032690525055,1.08066284656525,1.06443786621094,0.940568923950195,0.859182178974152,0.78115 6182289124} validation_metrics_final | 0.666666686535 validation_loss_final | 0.827600717545 validation_metrics | {0.333333343267441,0.333333343267441,0.333333343267441,0.300000011920929,0.333333343267441,0.333333343267441,0.566666662693024,0.766666650772095,0.866666674613953 ,0.666666686534882} validation_loss | {1.15087306499481,1.12277579307556,1.10153806209564,1.09295105934143,1.08225655555725,1.07222008705139,1.05354011058807,0.967542886734009,0.901465713977814,0.8276 00717544556} -[ RECORD 12 ]-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------- -------------------- mst_key | 7 model_id | 2 compile_params | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['accuracy'] fit_params | batch_size=4,epochs=1 model_type | madlib_keras model_size | 1.2197265625 metrics_elapsed_time | {0.178417921066284,0.60815691947937,0.202095985412598,0.178900003433228,0.18206000328064,0.181649923324585,0.191691875457764,0.182677984237671,0.183932065963745,0 .189748048782349} metrics_type | {accuracy} training_metrics_final | 0.358333319426 training_loss_final | 1.10466384888 training_metrics | {0.358333319425583,0.358333319425583,0.358333319425583,0.308333337306976,0.358333319425583,0.358333319425583,0.358333319425583,0.358333319425583,0.333333343267441 ,0.358333319425583} training_loss | {1.10114789009094,1.10259783267975,1.09921455383301,1.1069358587265,1.09716868400574,1.10427033901215,1.0967606306076,1.10034203529358,1.09822237491608,1.10466384 887695} validation_metrics_final | 0.233333334327 validation_loss_final | 1.14768576622 validation_metrics | {0.233333334326744,0.233333334326744,0.233333334326744,0.433333337306976,0.233333334326744,0.233333334326744,0.233333334326744,0.233333334326744,0.333333343267441 ,0.233333334326744} validation_loss | {1.14770805835724,1.11313581466675,1.09988021850586,1.10149800777435,1.10710287094116,1.10837662220001,1.11643397808075,1.14481270313263,1.10837042331696,1.147685 76622009} ``` (4) review after recent commit predict on training data ``` DROP TABLE IF EXISTS iris_predict; SELECT madlib.madlib_keras_predict('iris_multi_model', -- model 'iris_train', -- train_table 'id', -- id column 'attributes', -- independent var 'iris_predict', -- output table 'response', -- prediction type NULL, -- use gpus 2 -- mst_key to use ); SELECT COUNT(*) FROM iris_predict JOIN iris_train USING (id) WHERE iris_predict.estimated_class_text != iris_train.class_text; -[ RECORD 1 ] count | 3 ``` error % ``` SELECT round(count(*)*100/(150*0.8),2) as train_accuracy_percent from (select iris_train.class_text as actual, iris_predict.estimated_class_text as estimated from iris_predict inner join iris_train on iris_train.id=iris_predict.id) q WHERE q.actual=q.estimated; train_accuracy_percent ------------------------ 97.50 (1 row) ``` This now matches ``` training_metrics_final | 0.975000023842 ``` from the training output table. OK (5) review after recent commit predict on test data ``` DROP TABLE IF EXISTS iris_predict; SELECT madlib.madlib_keras_predict('iris_multi_model', -- model 'iris_test', -- test_table 'id', -- id column 'attributes', -- independent var 'iris_predict', -- output table 'response', -- prediction type NULL, -- use gpus 2 -- mst_key to use ); SELECT COUNT(*) FROM iris_predict JOIN iris_test USING (id) WHERE iris_predict.estimated_class_text != iris_test.class_text; count ------- 1 (1 row) ``` as a % ``` SELECT round(count(*)*100/(150*0.2),2) as test_accuracy_percent from (select iris_test.class_text as actual, iris_predict.estimated_class_text as estimated from iris_predict inner join iris_test on iris_test.id=iris_predict.id) q WHERE q.actual=q.estimated; test_accuracy_percent ----------------------- 96.67 (1 row) ``` which matches: ``` validation_metrics_final | 0.966666638851 ``` from the training output table. OK (7) review after last commit pick wrong model ``` DROP TABLE IF EXISTS iris_validate; SELECT madlib.madlib_keras_evaluate('iris_multi_model', -- model 'iris_test_packed', -- test table 'iris_validate', -- output table NULL, -- use gpus 999 -- mst_key to use ); ERROR: plpy.Error: madlib_keras_evaluate: mst_key 999 does not exist in the info table (plpython.c:5038) CONTEXT: Traceback (most recent call last): PL/Python function "madlib_keras_evaluate", line 21, in <module> return madlib_keras.evaluate(**globals()) PL/Python function "madlib_keras_evaluate", line 575, in evaluate PL/Python function "madlib_keras_evaluate", line 244, in create_summary_view PL/Python function "madlib_keras_evaluate" ``` which is a better message OK (8) review after last commit input table does not exist ``` DROP TABLE IF EXISTS iris_validate; SELECT madlib.madlib_keras_evaluate('iris_multi_model999', -- model 'iris_test_packed', -- test table 'iris_validate', -- output table NULL, -- use gpus 2 -- mst_key to use ); ERROR: plpy.Error: madlib_keras_evaluate: Missing summary and/or info tables for iris_multi_model999 (plpython.c:5038) CONTEXT: Traceback (most recent call last): PL/Python function "madlib_keras_evaluate", line 21, in <module> return madlib_keras.evaluate(**globals()) PL/Python function "madlib_keras_evaluate", line 575, in evaluate PL/Python function "madlib_keras_evaluate", line 237, in create_summary_view PL/Python function "madlib_keras_evaluate" ``` which is a better message OK (9) review after last commit: spot check on errors looks OK (10) review after last commit: ``` SELECT * FROM iris_multi_model_summary; -[ RECORD 1 ]-------+--------------------------------------------- source_table | iris_train_packed validation_table | iris_test_packed dependent_varname | class_text independent_varname | attributes model_arch_table | model_arch_library num_iterations | 10 start_training_time | 2019-11-04 21:46:55.109008 end_training_time | 2019-11-04 21:49:41.586433 madlib_version | 1.17-dev num_classes | 3 class_values | {Iris-setosa,Iris-versicolor,Iris-virginica} dependent_vartype | character varying normalizing_const | 1 ``` OK
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
