fmcquillan99 edited a comment on issue #451: DL: Update evaluate and predict for multi model outputs URL: https://github.com/apache/madlib/pull/451#issuecomment-547525859 There are issues with (4) (5) (6) (7) (8) below that need to be looked at: (1) - iris data - 2 MLP models (1 and 2 hidden layers) ``` SELECT * FROM mst_table ORDER BY mst_key; mst_key | model_id | compile_params | fit_params ---------+----------+---------------------------------------------------------------------------------+----------------------- 1 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['accuracy'] | batch_size=4,epochs=1 2 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['accuracy'] | batch_size=8,epochs=1 3 | 1 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['accuracy'] | batch_size=4,epochs=1 4 | 1 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['accuracy'] | batch_size=8,epochs=1 5 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['accuracy'] | batch_size=4,epochs=1 6 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['accuracy'] | batch_size=8,epochs=1 7 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['accuracy'] | batch_size=4,epochs=1 8 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['accuracy'] | batch_size=8,epochs=1 9 | 2 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['accuracy'] | batch_size=4,epochs=1 10 | 2 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['accuracy'] | batch_size=8,epochs=1 11 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['accuracy'] | batch_size=4,epochs=1 12 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['accuracy'] | batch_size=8,epochs=1 (12 rows) ``` Train ``` DROP TABLE IF EXISTS iris_multi_model, iris_multi_model_summary, iris_multi_model_info; SELECT madlib.madlib_keras_fit_multiple_model('iris_train_packed', -- source_table 'iris_multi_model', -- model_output_table 'mst_table', -- model_selection_table 10, -- num_iterations 0 -- use gpus ); SELECT * FROM iris_multi_model_info ORDER BY training_metrics_final DESC; -[ RECORD 1 ]------------+------------------------------------------------------------------------------------------------------------------------------------------------ ----------------------------------------- mst_key | 3 model_id | 1 compile_params | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['accuracy'] fit_params | batch_size=4,epochs=1 model_type | madlib_keras model_size | 0.7900390625 metrics_elapsed_time | {0.15962290763855,0.171411037445068,0.154904842376709,0.151599884033203,0.160181999206543,0.152440071105957,0.223697900772095,0.156054973602295 ,0.202955007553101,0.152594089508057} metrics_type | {accuracy} training_metrics_final | 0.975000023842 training_loss_final | 0.0928164944053 training_metrics | {0.675000011920929,0.875,0.725000023841858,0.975000023841858,0.966666638851166,0.983333349227905,0.975000023841858,0.975000023841858,0.88333332 5386047,0.975000023841858} training_loss | {0.520763278007507,0.379715353250504,0.406634271144867,0.182054653763771,0.14105860888958,0.108805403113365,0.109423443675041,0.101339735090733 ,0.215542033314705,0.0928164944052696} validation_metrics_final | 0.933333337307 validation_loss_final | 0.196297511458 validation_metrics | {0.733333349227905,0.899999976158142,0.766666650772095,0.933333337306976,0.899999976158142,0.966666638851166,0.933333337306976,0.93333333730697 6,0.800000011920929,0.933333337306976} validation_loss | {0.487521052360535,0.348948538303375,0.474558025598526,0.232045486569405,0.22162064909935,0.155132472515106,0.202082633972168,0.196883022785187 ,0.382103145122528,0.196297511458397} -[ RECORD 2 ]------------+------------------------------------------------------------------------------------------------------------------------------------------------ ----------------------------------------- mst_key | 10 model_id | 2 compile_params | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['accuracy'] fit_params | batch_size=8,epochs=1 model_type | madlib_keras model_size | 1.2197265625 metrics_elapsed_time | {0.183917045593262,0.373061180114746,0.182140111923218,0.177560091018677,0.192739009857178,0.177842140197754,0.201126098632812,0.17956304550170 9,0.437918901443481,0.184372901916504} metrics_type | {accuracy} training_metrics_final | 0.966666638851 training_loss_final | 0.182971566916 training_metrics | {0.658333361148834,0.658333361148834,0.858333349227905,0.691666662693024,0.983333349227905,0.958333313465118,0.800000011920929,0.91666668653488 2,0.958333313465118,0.966666638851166} training_loss | {0.839326024055481,0.552537620067596,0.427368670701981,0.395185142755508,0.339115768671036,0.288732022047043,0.310250341892242,0.31521281599998 5,0.203457966446877,0.182971566915512} validation_metrics_final | 0.933333337307 validation_loss_final | 0.21536397934 validation_metrics | {0.699999988079071,0.699999988079071,0.766666650772095,0.733333349227905,0.899999976158142,0.866666674613953,0.766666650772095,0.89999997615814 2,0.833333313465118,0.933333337306976} validation_loss | {0.841938138008118,0.521961748600006,0.390463948249817,0.383590310811996,0.336574584245682,0.328944146633148,0.382724314928055,0.32695025205612 2,0.299405694007874,0.2153639793396} -[ RECORD 3 ]------------+------------------------------------------------------------------------------------------------------------------------------------------------ ----------------------------------------- mst_key | 4 model_id | 1 compile_params | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['accuracy'] fit_params | batch_size=8,epochs=1 model_type | madlib_keras model_size | 0.7900390625 metrics_elapsed_time | {0.158810138702393,0.171705007553101,0.157279968261719,0.149960994720459,0.161519050598145,0.151033163070679,0.167169094085693,0.15194702148437 5,0.181770086288452,0.160482168197632} metrics_type | {accuracy} training_metrics_final | 0.933333337307 training_loss_final | 0.109170868993 training_metrics | {0.658333361148834,0.933333337306976,0.983333349227905,0.933333337306976,0.933333337306976,0.983333349227905,0.983333349227905,0.98333334922790 5,0.991666674613953,0.933333337306976} training_loss | {0.703937888145447,0.404870539903641,0.223930105566978,0.183146089315414,0.137469902634621,0.0971243679523468,0.0983915776014328,0.095256045460 701,0.0844478830695152,0.109170868992805} validation_metrics_final | 0.933333337307 validation_loss_final | 0.140566334128 validation_metrics | {0.699999988079071,0.899999976158142,0.899999976158142,0.899999976158142,0.933333337306976,0.899999976158142,0.899999976158142,0.89999997615814 2,0.966666638851166,0.933333337306976} validation_loss | {0.648390173912048,0.386223673820496,0.268760353326797,0.211181059479713,0.171986550092697,0.192580834031105,0.217913091182709,0.21975873410701 8,0.143908932805061,0.14056633412838} -[ RECORD 4 ]------------+------------------------------------------------------------------------------------------------------------------------------------------------ ----------------------------------------- mst_key | 9 model_id | 2 compile_params | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['accuracy'] fit_params | batch_size=4,epochs=1 model_type | madlib_keras model_size | 1.2197265625 metrics_elapsed_time | {0.184342861175537,0.186679124832153,0.178445100784302,0.198610067367554,0.179869174957275,0.176558017730713,0.191009998321533,0.18026208877563 5,0.191850900650024,0.180068969726562} metrics_type | {accuracy} training_metrics_final | 0.933333337307 training_loss_final | 0.132100492716 training_metrics | {0.658333361148834,0.966666638851166,0.916666686534882,0.899999976158142,0.708333313465118,0.966666638851166,0.975000023841858,0.875,0.97500002 3841858,0.933333337306976} training_loss | {0.645366847515106,0.232219696044922,0.187613621354103,0.240386798977852,0.561961591243744,0.117926396429539,0.0994494929909706,0.2343605309724 81,0.0802617892622948,0.132100492715836} validation_metrics_final | 0.899999976158 validation_loss_final | 0.159418046474 validation_metrics | {0.699999988079071,0.866666674613953,0.833333313465118,0.800000011920929,0.733333349227905,0.933333337306976,0.933333337306976,0.86666667461395 3,0.966666638851166,0.899999976158142} validation_loss | {0.614455163478851,0.268589198589325,0.27994567155838,0.367912858724594,0.662781298160553,0.164628267288208,0.192523822188377,0.292809307575226 ,0.132961437106133,0.159418046474457} -[ RECORD 5 ]------------+------------------------------------------------------------------------------------------------------------------------------------------------ ----------------------------------------- mst_key | 5 model_id | 1 compile_params | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['accuracy'] fit_params | batch_size=4,epochs=1 model_type | madlib_keras model_size | 0.7900390625 metrics_elapsed_time | {0.177564144134521,0.156223058700562,0.154070854187012,0.161600112915039,0.150278091430664,0.164193868637085,0.155445098876953,0.17075300216674 8,0.155435085296631,0.153830051422119} metrics_type | {accuracy} training_metrics_final | 0.899999976158 training_loss_final | 0.44444206357 training_metrics | {0.708333313465118,0.658333361148834,0.658333361148834,0.658333361148834,0.675000011920929,0.708333313465118,0.708333313465118,0.75833332538604 7,0.766666650772095,0.899999976158142} training_loss | {0.771447777748108,0.682944893836975,0.630482733249664,0.594800889492035,0.564319252967834,0.533434450626373,0.50351870059967,0.481856644153595 ,0.46294954419136,0.444442063570023} validation_metrics_final | 0.833333313465 validation_loss_final | 0.424488693476 validation_metrics | {0.766666650772095,0.699999988079071,0.699999988079071,0.699999988079071,0.733333349227905,0.733333349227905,0.733333349227905,0.73333334922790 5,0.733333349227905,0.833333313465118} validation_loss | {0.759826004505157,0.662849307060242,0.604207634925842,0.569091618061066,0.536879122257233,0.504582524299622,0.477831065654755,0.45702105760574 3,0.440479785203934,0.424488693475723} -[ RECORD 6 ]------------+------------------------------------------------------------------------------------------------------------------------------------------------ ----------------------------------------- mst_key | 2 model_id | 1 compile_params | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['accuracy'] fit_params | batch_size=8,epochs=1 model_type | madlib_keras model_size | 0.7900390625 metrics_elapsed_time | {0.16278600692749,0.154292821884155,0.161294937133789,0.153645992279053,0.164122104644775,0.152410984039307,0.396697998046875,0.154489994049072 ,0.15544605255127,0.164608001708984} metrics_type | {accuracy} training_metrics_final | 0.783333361149 training_loss_final | 0.442915290594 training_metrics | {0.349999994039536,0.516666650772095,0.975000023841858,0.791666686534882,0.883333325386047,0.949999988079071,0.966666638851166,0.91666668653488 2,0.966666638851166,0.783333361148834} training_loss | {1.17399632930756,0.733614087104797,0.312711358070374,0.494298249483109,0.256883263587952,0.230874612927437,0.164221689105034,0.264884918928146 ,0.129683092236519,0.442915290594101} validation_metrics_final | 0.766666650772 validation_loss_final | 0.658641517162 validation_metrics | {0.300000011920929,0.433333337306976,0.933333337306976,0.866666674613953,0.899999976158142,0.966666638851166,1,0.866666674613953,0.899999976158 142,0.766666650772095} validation_loss | {1.40960335731506,0.879667580127716,0.289381295442581,0.503752708435059,0.241700142621994,0.224781185388565,0.176762118935585,0.270557969808578 ,0.220981314778328,0.658641517162323} -[ RECORD 7 ]------------+------------------------------------------------------------------------------------------------------------------------------------------------ ----------------------------------------- mst_key | 7 model_id | 2 compile_params | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['accuracy'] fit_params | batch_size=4,epochs=1 model_type | madlib_keras model_size | 1.2197265625 metrics_elapsed_time | {0.375895977020264,0.179503917694092,0.181375026702881,0.18864917755127,0.181995868682861,0.197070121765137,0.178755044937134,0.419157028198242 ,0.185692071914673,0.178522109985352} metrics_type | {accuracy} training_metrics_final | 0.75 training_loss_final | 0.37944367528 training_metrics | {0.658333361148834,0.658333361148834,0.958333313465118,0.658333361148834,0.616666674613953,0.416666656732559,0.875,0.966666638851166,0.98333334 9227905,0.75} training_loss | {0.564021706581116,0.656923830509186,0.410611629486084,0.657700955867767,0.783278226852417,0.938268542289734,0.240705713629723,0.19149164855480 2,0.188495144248009,0.379443675279617} validation_metrics_final | 0.733333349228 validation_loss_final | 0.440304368734 validation_metrics | {0.699999988079071,0.699999988079071,0.933333337306976,0.699999988079071,0.633333325386047,0.366666674613953,0.766666650772095,0.96666663885116 6,0.899999976158142,0.733333349227905} validation_loss | {0.518877744674683,0.581225216388702,0.369613200426102,0.575586378574371,0.760259926319122,1.11823546886444,0.337272256612778,0.201485335826874 ,0.231084078550339,0.44030436873436} -[ RECORD 8 ]------------+------------------------------------------------------------------------------------------------------------------------------------------------ ----------------------------------------- mst_key | 11 model_id | 2 compile_params | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['accuracy'] fit_params | batch_size=4,epochs=1 model_type | madlib_keras model_size | 1.2197265625 metrics_elapsed_time | {0.185693979263306,0.176743030548096,0.208944082260132,0.177680969238281,0.188605070114136,0.179240942001343,0.469206094741821,0.18075394630432 1,0.182642936706543,0.18809700012207} metrics_type | {accuracy} training_metrics_final | 0.716666638851 training_loss_final | 0.49047088623 training_metrics | {0.341666668653488,0.341666668653488,0.366666674613953,0.883333325386047,0.783333361148834,0.658333361148834,0.658333361148834,0.66666668653488 2,0.866666674613953,0.716666638851166} training_loss | {1.10631430149078,1.00225436687469,0.952353298664093,0.879558265209198,0.816462278366089,0.759344756603241,0.688487946987152,0.61039125919342,0 .543096899986267,0.490470886230469} validation_metrics_final | 0.766666650772 validation_loss_final | 0.471391111612 validation_metrics | {0.300000011920929,0.300000011920929,0.333333343267441,0.766666650772095,0.766666650772095,0.699999988079071,0.733333349227905,0.73333334922790 5,0.766666650772095,0.766666650772095} validation_loss | {1.18413507938385,1.03528475761414,0.984834551811218,0.890941798686981,0.828823924064636,0.763210475444794,0.687682032585144,0.596106350421906, 0.527397036552429,0.47139111161232} -[ RECORD 9 ]------------+------------------------------------------------------------------------------------------------------------------------------------------------ ----------------------------------------- mst_key | 1 model_id | 1 compile_params | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['accuracy'] fit_params | batch_size=4,epochs=1 model_type | madlib_keras model_size | 0.7900390625 metrics_elapsed_time | {0.155688047409058,0.162626981735229,0.157097101211548,0.175328969955444,0.151535987854004,0.382455110549927,0.155317068099976,0.15313291549682 6,0.161896944046021,0.154218196868896} metrics_type | {accuracy} training_metrics_final | 0.658333361149 training_loss_final | 0.506188511848 training_metrics | {0.658333361148834,0.583333313465118,0.658333361148834,0.808333337306976,0.583333313465118,0.983333349227905,0.975000023841858,0.85833334922790 5,0.666666686534882,0.658333361148834} training_loss | {0.634293258190155,0.62472277879715,0.491173326969147,0.325925976037979,0.834470629692078,0.270386427640915,0.137209206819534,0.237895086407661 ,0.488089859485626,0.50618851184845} validation_metrics_final | 0.699999988079 validation_loss_final | 0.437518566847 validation_metrics | {0.699999988079071,0.633333325386047,0.699999988079071,0.766666650772095,0.633333325386047,0.933333337306976,0.933333337306976,0.89999997615814 2,0.699999988079071,0.699999988079071} validation_loss | {0.570496320724487,0.620569884777069,0.43601530790329,0.389920294284821,1.09760916233063,0.271371573209763,0.166262924671173,0.227906361222267, 0.429942190647125,0.437518566846848} -[ RECORD 10 ]-----------+------------------------------------------------------------------------------------------------------------------------------------------------ ----------------------------------------- mst_key | 6 model_id | 1 compile_params | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['accuracy'] fit_params | batch_size=8,epochs=1 model_type | madlib_keras model_size | 0.7900390625 metrics_elapsed_time | {0.175873041152954,0.155245065689087,0.154464960098267,0.171594858169556,0.157553195953369,0.162687063217163,0.155516862869263,0.16730904579162 6,0.156662940979004,0.396471977233887} metrics_type | {accuracy} training_metrics_final | 0.508333325386 training_loss_final | 0.915417551994 training_metrics | {0.341666668653488,0.341666668653488,0.341666668653488,0.341666668653488,0.308333337306976,0.324999988079071,0.341666668653488,0.34166666865348 8,0.358333319425583,0.508333325386047} training_loss | {2.13231420516968,1.79814553260803,1.51453709602356,1.26933741569519,1.15679371356964,1.07199251651764,1.01928377151489,0.978273630142212,0.942 150175571442,0.915417551994324} validation_metrics_final | 0.600000023842 validation_loss_final | 0.93969720602 validation_metrics | {0.300000011920929,0.300000011920929,0.300000011920929,0.300000011920929,0.333333343267441,0.333333343267441,0.300000011920929,0.30000001192092 9,0.333333343267441,0.600000023841858} validation_loss | {2.51236271858215,2.09125304222107,1.73099553585052,1.41019690036774,1.26361191272736,1.14860534667969,1.07588887214661,1.01970684528351,0.9719 77055072784,0.939697206020355} -[ RECORD 11 ]-----------+------------------------------------------------------------------------------------------------------------------------------------------------ ----------------------------------------- mst_key | 12 model_id | 2 compile_params | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['accuracy'] fit_params | batch_size=8,epochs=1 model_type | madlib_keras model_size | 1.2197265625 metrics_elapsed_time | {0.184770822525024,0.191632032394409,0.178493022918701,0.391031980514526,0.181709051132202,0.17969799041748,0.191845893859863,0.179744958877563 ,0.199568033218384,0.178317070007324} metrics_type | {accuracy} training_metrics_final | 0.341666668653 training_loss_final | 1.09104967117 training_metrics | {0.341666668653488,0.341666668653488,0.341666668653488,0.341666668653488,0.341666668653488,0.341666668653488,0.341666668653488,0.34166666865348 8,0.341666668653488,0.341666668653488} training_loss | {3.08191418647766,2.32425045967102,1.79031324386597,1.4521062374115,1.26406502723694,1.19002187252045,1.14052081108093,1.11407339572906,1.10518 69392395,1.0910496711731} validation_metrics_final | 0.300000011921 validation_loss_final | 1.09330034256 validation_metrics | {0.300000011920929,0.300000011920929,0.300000011920929,0.300000011920929,0.300000011920929,0.300000011920929,0.300000011920929,0.30000001192092 9,0.300000011920929,0.300000011920929} validation_loss | {3.12394428253174,2.34826755523682,1.7957478761673,1.44716620445251,1.25600838661194,1.18111801147461,1.13339793682098,1.11054337024689,1.10561 001300812,1.09330034255981} -[ RECORD 12 ]-----------+------------------------------------------------------------------------------------------------------------------------------------------------ ----------------------------------------- mst_key | 8 model_id | 2 compile_params | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['accuracy'] fit_params | batch_size=8,epochs=1 model_type | madlib_keras model_size | 1.2197265625 metrics_elapsed_time | {0.270622968673706,0.18524694442749,0.178076028823853,0.200053930282593,0.178044080734253,0.198669195175171,0.180491924285889,0.182524919509888 ,0.186138868331909,0.175627946853638} metrics_type | {accuracy} training_metrics_final | 0.341666668653 training_loss_final | 1.09889221191 training_metrics | {0.341666668653488,0.341666668653488,0.341666668653488,0.341666668653488,0.341666668653488,0.341666668653488,0.341666668653488,0.34166666865348 8,0.316666662693024,0.341666668653488} training_loss | {1.09904658794403,1.10012602806091,1.10846817493439,1.09847903251648,1.10446727275848,1.10296499729156,1.097984790802,1.10039520263672,1.103934 88407135,1.09889221191406} validation_metrics_final | 0.300000011921 validation_loss_final | 1.10708355904 validation_metrics | {0.300000011920929,0.300000011920929,0.300000011920929,0.300000011920929,0.300000011920929,0.300000011920929,0.300000011920929,0.30000001192092 9,0.400000005960464,0.300000011920929} validation_loss | {1.10612213611603,1.11198484897614,1.12145566940308,1.10108768939972,1.09653890132904,1.12743330001831,1.10469746589661,1.11833906173706,1.0917 9925918579,1.10708355903625} ``` seems OK (2) evaluate on test data ``` DROP TABLE IF EXISTS iris_validate; SELECT madlib.madlib_keras_evaluate('iris_multi_model', -- model 'iris_test_packed', -- test table 'iris_validate', -- output table NULL, -- use gpus 9 -- mst_key to use ); SELECT * FROM iris_validate; loss | metric | metrics_type -------------------+-------------------+-------------- 0.159418046474457 | 0.899999976158142 | {accuracy} (1 row) ``` seems OK, matches final acc and loss for model 9 on validation data during training (3) evaluate on training data ``` DROP TABLE IF EXISTS iris_validate; SELECT madlib.madlib_keras_evaluate('iris_multi_model', -- model 'iris_train_packed', -- test table 'iris_validate', -- output table NULL, -- use gpus 9 -- mst_key to use ); SELECT * FROM iris_validate; loss | metric | metrics_type -------------------+-------------------+-------------- 0.132100492715836 | 0.933333337306976 | {accuracy} (1 row) ``` seems OK, matches final acc and loss for model 9 on training data during training (4) predict on training data ``` DROP TABLE IF EXISTS iris_predict; SELECT madlib.madlib_keras_predict('iris_multi_model', -- model 'iris_train', -- train_table 'id', -- id column 'attributes', -- independent var 'iris_predict', -- output table 'response', -- prediction type NULL, -- use gpus 9 -- mst_key to use ); SELECT COUNT(*) FROM iris_predict JOIN iris_train USING (id) WHERE iris_predict.estimated_class_text != iris_train.class_text; count ------- 34 ``` error % ``` SELECT round(count(*)*100/(150*0.8),2) as train_accuracy_percent from (select iris_train.class_text as actual, iris_predict.estimated_class_text as estimated from iris_predict inner join iris_train on iris_train.id=iris_predict.id) q WHERE q.actual=q.estimated; train_accuracy_percent ------------------------ 71.67 ``` Why is this not 0.933333337306976 ??? (5) predict on test data ``` DROP TABLE IF EXISTS iris_predict; SELECT madlib.madlib_keras_predict('iris_multi_model', -- model 'iris_test', -- test_table 'id', -- id column 'attributes', -- independent var 'iris_predict', -- output table 'response', -- prediction type NULL, -- use gpus 9 -- mst_key to use ); SELECT COUNT(*) FROM iris_predict JOIN iris_test USING (id) WHERE iris_predict.estimated_class_text != iris_test.class_text; count ------- 7 (1 row) ``` as a % ``` SELECT round(count(*)*100/(150*0.2),2) as test_accuracy_percent from (select iris_test.class_text as actual, iris_predict.estimated_class_text as estimated from iris_predict inner join iris_test on iris_test.id=iris_predict.id) q WHERE q.actual=q.estimated; test_accuracy_percent ----------------------- 76.67 (1 row) ``` Why is this not 0.899999976158142 ??? (6) predict is giving warning errors which should be suppressed ``` WARNING: column "attributes_shape" does not exist CONTEXT: PL/Python function "madlib_keras_predict" madlib_keras_predict ---------------------- ``` Please check other functions in this module to ensure the right level of console reporting is set. (7) pick wrong model ``` DROP TABLE IF EXISTS iris_validate; SELECT madlib.madlib_keras_evaluate('iris_multi_model', -- model 'iris_test_packed', -- test table 'iris_validate', -- output table NULL, -- use gpus 999 -- mst_key to use ); ERROR: plpy.Error: madlib_keras_evaluate error: Input table '__madlib_temp_tmp_view_summary39503791_1572367130_16846764__' is empty! (plpython.c:5038) CONTEXT: Traceback (most recent call last): PL/Python function "madlib_keras_evaluate", line 21, in <module> return madlib_keras.evaluate(**globals()) PL/Python function "madlib_keras_evaluate", line 577, in evaluate PL/Python function "madlib_keras_evaluate", line 628, in validate_evaluate PL/Python function "madlib_keras_evaluate", line 59, in validate_predict_evaluate_tables PL/Python function "madlib_keras_evaluate", line 218, in _validate_model_summary_tbl PL/Python function "madlib_keras_evaluate", line 678, in input_tbl_valid PL/Python function "madlib_keras_evaluate" ``` This error is misleading - should say that 999 is not a valid mst_key or model or whatever. (8) input table does not exist ``` DROP TABLE IF EXISTS iris_validate; SELECT madlib.madlib_keras_evaluate('iris_multi_model999', -- model 'iris_test_packed', -- test table 'iris_validate', -- output table NULL, -- use gpus 9 -- mst_key to use ); ERROR: NameError: global name 'module_name' is not defined (plpython.c:5038) CONTEXT: Traceback (most recent call last): PL/Python function "madlib_keras_evaluate", line 21, in <module> return madlib_keras.evaluate(**globals()) PL/Python function "madlib_keras_evaluate", line 575, in evaluate PL/Python function "madlib_keras_evaluate", line 237, in create_summary_view PL/Python function "madlib_keras_evaluate" ``` This error is misleading - should say source table `iris_multi_model999` does not exist (9) please do a general check that we are trapping input errors in the usual way for fit, predict and evaluate. I did not test them all.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
