[ https://issues.apache.org/jira/browse/SYSTEMML-1238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15868495#comment-15868495 ]
Imran Younus edited comment on SYSTEMML-1238 at 2/15/17 8:12 PM: ----------------------------------------------------------------- output of explain from {{LinearRegCG.dml}}: {code} 17/02/15 12:10:52 INFO api.DMLScript: EXPLAIN (RUNTIME): # Memory Budget local/remote = 3823MB/?MB/?MB/?MB # Degree of Parallelism (vcores) local/remote = 48/? PROGRAM ( size CP/SP = 385/2 ) --MAIN PROGRAM ----GENERIC (lines 86-110) [recompile=false] ------CP createvar pREADX /user/iyounus/data/diabetes_X_train.txt false MATRIX csv 422 1 -1 -1 -1 copy false , true 0.0 ------CP createvar pREADy /user/iyounus/data/diabetes_y_train.txt false MATRIX csv 422 1 -1 -1 -1 copy false , true 0.0 ------CP print BEGIN LINEAR REGRESSION SCRIPT.SCALAR.STRING.true _Var29.SCALAR.STRING ------CP print Reading X and Y....SCALAR.STRING.true _Var30.SCALAR.STRING ------CP createvar _mVar31 scratch_space//_p15212_10.168.31.80//_t0/temp1 true MATRIX binaryblock 422 1 1000 1000 422 copy ------CP rand 422 1 1000 1000 1.0 1.0 1.0 -1 uniform 1.0 48 _mVar31.MATRIX.DOUBLE ------CP createvar _mVar32 scratch_space//_p15212_10.168.31.80//_t0/temp2 true MATRIX binaryblock 422 1 1000 1000 -1 copy ------SPARK csvrblk pREADX.MATRIX.DOUBLE _mVar32.MATRIX.DOUBLE 1000 1000 false , true 0.0 ------CP createvar _mVar33 scratch_space//_p15212_10.168.31.80//_t0/temp3 true MATRIX binaryblock 422 1 1000 1000 -1 copy ------SPARK csvrblk pREADy.MATRIX.DOUBLE _mVar33.MATRIX.DOUBLE 1000 1000 false , true 0.0 ------CP assignvar beta.txt.SCALAR.STRING.true fileB.SCALAR.STRING ------CP assignvar .SCALAR.STRING.true fileO.SCALAR.STRING ------CP assignvar .SCALAR.STRING.true fileLog.SCALAR.STRING ------CP assignvar text.SCALAR.STRING.true fmtB.SCALAR.STRING ------CP assignvar 1.SCALAR.INT.true intercept_status.SCALAR.INT ------CP assignvar 1.0E-6.SCALAR.DOUBLE.true tolerance.SCALAR.DOUBLE ------CP assignvar 0.SCALAR.INT.true max_iteration.SCALAR.INT ------CP assignvar 1.0E-6.SCALAR.DOUBLE.true regularization.SCALAR.DOUBLE ------CP assignvar 422.SCALAR.INT.true n.SCALAR.INT ------CP assignvar 1.SCALAR.INT.true m.SCALAR.INT ------CP assignvar 1.SCALAR.INT.true m_ext.SCALAR.INT ------CP rmvar _Var29 ------CP rmvar _Var30 ------CP cpvar _mVar31 ones_n ------CP cpvar _mVar32 X ------CP cpvar _mVar33 y ------CP rmvar _mVar31 ------CP rmvar _mVar32 ------CP rmvar _mVar33 ----GENERIC (lines 113-114) [recompile=false] ------CP createvar _mVar34 scratch_space//_p15212_10.168.31.80//_t0/temp4 true MATRIX binaryblock 422 2 1000 1000 -1 copy ------CP append X.MATRIX.DOUBLE ones_n.MATRIX.DOUBLE 1.SCALAR.INT.true _mVar34.MATRIX.DOUBLE true ------CP rmvar X ------CP assignvar 2.SCALAR.INT.true m_ext.SCALAR.INT ------CP cpvar _mVar34 X ------CP rmvar _mVar34 ------CP rmvar ones_n ----GENERIC (lines 117-117) [recompile=false] ------CP createvar _mVar35 scratch_space//_p15212_10.168.31.80//_t0/temp5 true MATRIX binaryblock 2 1 1000 1000 -1 copy ------CP rand 2 1 1000 1000 1.0 1.0 1.0 -1 uniform 1.0 48 _mVar35.MATRIX.DOUBLE ------CP cpvar _mVar35 scale_lambda ------CP rmvar _mVar35 ----GENERIC (lines 120-120) [recompile=false] ------CP createvar _mVar36 scratch_space//_p15212_10.168.31.80//_t0/temp6 true MATRIX binaryblock 2 1 1000 1000 -1 copy ------CP leftIndex scale_lambda.MATRIX.DOUBLE 0.SCALAR.INT.true m_ext.SCALAR.INT.false m_ext.SCALAR.INT.false 1.SCALAR.INT.true 1.SCALAR.INT.true _mVar36.MATRIX.DOUBLE ------CP rmvar scale_lambda ------CP cpvar _mVar36 scale_lambda ------CP rmvar _mVar36 ----GENERIC (lines 133-134) [recompile=false] ------CP createvar _mVar37 scratch_space//_p15212_10.168.31.80//_t0/temp7 true MATRIX binaryblock 2 1 1000 1000 -1 copy ------CP rand 2 1 1000 1000 1.0 1.0 1.0 -1 uniform 1.0 48 _mVar37.MATRIX.DOUBLE ------CP createvar _mVar38 scratch_space//_p15212_10.168.31.80//_t0/temp8 true MATRIX binaryblock 2 1 1000 1000 0 copy ------CP rand 2 1 1000 1000 0.0 0.0 1.0 -1 uniform 1.0 48 _mVar38.MATRIX.DOUBLE ------CP cpvar _mVar37 scale_X ------CP cpvar _mVar38 shift_X ------CP rmvar _mVar37 ------CP rmvar _mVar38 ----GENERIC (lines 149-150) [recompile=false] ------CP createvar _mVar39 scratch_space//_p15212_10.168.31.80//_t0/temp9 true MATRIX binaryblock 2 1 1000 1000 -1 copy ------CP * scale_lambda.MATRIX.DOUBLE 1.0E-6.SCALAR.DOUBLE.true _mVar39.MATRIX.DOUBLE ------CP createvar _mVar40 scratch_space//_p15212_10.168.31.80//_t0/temp10 true MATRIX binaryblock 2 1 1000 1000 0 copy ------CP rand 2 1 1000 1000 0.0 0.0 1.0 -1 uniform 1.0 48 _mVar40.MATRIX.DOUBLE ------CP cpvar _mVar39 lambda ------CP cpvar _mVar40 beta_unscaled ------CP rmvar _mVar39 ------CP rmvar _mVar40 ------CP rmvar regularization ------CP rmvar scale_lambda ----GENERIC (lines 153-153) [recompile=false] ------CP assignvar m_ext.SCALAR.INT.false max_iteration.SCALAR.INT ----GENERIC (lines 155-160) [recompile=false] ------CP print Running the CG algorithm....SCALAR.STRING.true _Var41.SCALAR.STRING ------CP createvar _mVar42 scratch_space//_p15212_10.168.31.80//_t0/temp11 true MATRIX binaryblock 1 422 1000 1000 -1 copy ------CP r' y.MATRIX.DOUBLE _mVar42.MATRIX.DOUBLE 48 ------CP createvar _mVar43 scratch_space//_p15212_10.168.31.80//_t0/temp12 true MATRIX binaryblock 1 2 1000 1000 -1 copy ------CP ba+* _mVar42.MATRIX.DOUBLE X.MATRIX.DOUBLE _mVar43.MATRIX.DOUBLE 48 ------CP rmvar _mVar42 ------CP createvar _mVar44 scratch_space//_p15212_10.168.31.80//_t0/temp13 true MATRIX binaryblock 2 1 1000 1000 -1 copy ------CP r' _mVar43.MATRIX.DOUBLE _mVar44.MATRIX.DOUBLE 48 ------CP rmvar _mVar43 ------CP createvar _mVar45 scratch_space//_p15212_10.168.31.80//_t0/temp14 true MATRIX binaryblock 2 1 1000 1000 -1 copy ------CP - 0.SCALAR.INT.true _mVar44.MATRIX.DOUBLE _mVar45.MATRIX.DOUBLE ------CP rmvar _mVar44 ------CP assignvar 0.SCALAR.INT.true i.SCALAR.INT ------CP rmvar _Var41 ------CP cpvar _mVar45 r ------CP rmvar _mVar45 ----GENERIC (lines 166-172) [recompile=false] ------CP createvar _mVar46 scratch_space//_p15212_10.168.31.80//_t0/temp15 true MATRIX binaryblock 2 1 1000 1000 -1 copy ------CP - 0.SCALAR.INT.true r.MATRIX.DOUBLE _mVar46.MATRIX.DOUBLE ------CP createvar _mVar47 scratch_space//_p15212_10.168.31.80//_t0/temp16 true MATRIX binaryblock 1 1 1000 1000 -1 copy ------CP tsmm r.MATRIX.DOUBLE _mVar47.MATRIX.DOUBLE LEFT 48 ------CP castdts _mVar47.MATRIX.DOUBLE.false _Var48.SCALAR.DOUBLE ------CP rmvar _mVar47 ------CP * _Var48.SCALAR.DOUBLE.false 1.0E-12.SCALAR.DOUBLE.true _Var49.SCALAR.DOUBLE ------CP sqrt _Var48.SCALAR.DOUBLE.false _Var50.SCALAR.DOUBLE ------CP + ||r|| initial value = .SCALAR.STRING.true _Var50.SCALAR.DOUBLE.false _Var51.SCALAR.STRING ------CP sqrt _Var49.SCALAR.DOUBLE.false _Var52.SCALAR.DOUBLE ------CP + CG_RESIDUAL_NORM,0,.SCALAR.STRING.true _Var50.SCALAR.DOUBLE.false _Var53.SCALAR.STRING ------CP rmvar _Var50 ------CP + _Var51.SCALAR.STRING.false , target value = .SCALAR.STRING.true _Var54.SCALAR.STRING ------CP rmvar _Var51 ------CP append _Var53.SCALAR.STRING.false CG_RESIDUAL_RATIO,0,1.0.SCALAR.STRING.true -1.SCALAR.INT.true _Var55.SCALAR.STRING true ------CP rmvar _Var53 ------CP + _Var54.SCALAR.STRING.false _Var52.SCALAR.DOUBLE.false _Var56.SCALAR.STRING ------CP rmvar _Var54 ------CP rmvar _Var52 ------CP print _Var56.SCALAR.STRING.false _Var57.SCALAR.STRING ------CP rmvar _Var56 ------CP assignvar _Var48.SCALAR.DOUBLE.false norm_r2.SCALAR.DOUBLE ------CP assignvar _Var48.SCALAR.DOUBLE.false norm_r2_initial.SCALAR.DOUBLE ------CP assignvar _Var49.SCALAR.DOUBLE.false norm_r2_target.SCALAR.DOUBLE ------CP assignvar _Var55.SCALAR.STRING.false log_str.SCALAR.STRING ------CP cpvar _mVar46 p ------CP rmvar _Var48 ------CP rmvar _Var49 ------CP rmvar _Var55 ------CP rmvar _Var57 ------CP rmvar _mVar46 ------CP rmvar tolerance ----GENERIC (lines 174-200) [recompile=false] ----WHILE (lines 174-200) ------CP < i.SCALAR.INT.false max_iteration.SCALAR.INT.false _Var58.SCALAR.BOOLEAN ------CP > norm_r2.SCALAR.DOUBLE.false norm_r2_target.SCALAR.DOUBLE.false _Var59.SCALAR.BOOLEAN ------CP && _Var58.SCALAR.BOOLEAN.false _Var59.SCALAR.BOOLEAN.false _Var60.SCALAR.BOOLEAN ------CP rmvar _Var58 ------CP rmvar _Var59 ------CP rmvar _Var60 ------GENERIC (lines 180-180) [recompile=false] --------CP cpvar p ssX_p ------GENERIC (lines 183-183) [recompile=false] --------CP createvar _mVar61 scratch_space//_p15212_10.168.31.80//_t0/temp17 true MATRIX binaryblock 2 1 1000 1000 -1 copy --------CP mmchain X.MATRIX.DOUBLE ssX_p.MATRIX.DOUBLE _mVar61.MATRIX.DOUBLE XtXv 48 --------CP cpvar _mVar61 q --------CP rmvar _mVar61 --------CP rmvar ssX_p ------GENERIC (lines 189-199) [recompile=false] --------CP createvar _mVar62 scratch_space//_p15212_10.168.31.80//_t0/temp18 true MATRIX binaryblock 1 2 1000 1000 -1 copy --------CP r' p.MATRIX.DOUBLE _mVar62.MATRIX.DOUBLE 48 --------CP createvar _mVar63 scratch_space//_p15212_10.168.31.80//_t0/temp19 true MATRIX binaryblock 2 1 1000 1000 -1 copy --------CP * lambda.MATRIX.DOUBLE p.MATRIX.DOUBLE _mVar63.MATRIX.DOUBLE --------CP + i.SCALAR.INT.false 1.SCALAR.INT.true _Var64.SCALAR.INT --------CP createvar _mVar65 scratch_space//_p15212_10.168.31.80//_t0/temp20 true MATRIX binaryblock 2 1 1000 1000 -1 copy --------CP + q.MATRIX.DOUBLE _mVar63.MATRIX.DOUBLE _mVar65.MATRIX.DOUBLE --------CP rmvar _mVar63 --------CP + Iteration .SCALAR.STRING.true _Var64.SCALAR.INT.false _Var66.SCALAR.STRING --------CP + CG_RESIDUAL_NORM,.SCALAR.STRING.true _Var64.SCALAR.INT.false _Var67.SCALAR.STRING --------CP + CG_RESIDUAL_RATIO,.SCALAR.STRING.true _Var64.SCALAR.INT.false _Var68.SCALAR.STRING --------CP createvar _mVar69 scratch_space//_p15212_10.168.31.80//_t0/temp21 true MATRIX binaryblock 1 1 1000 1000 -1 copy --------CP ba+* _mVar62.MATRIX.DOUBLE _mVar65.MATRIX.DOUBLE _mVar69.MATRIX.DOUBLE 48 --------CP rmvar _mVar62 --------CP + _Var66.SCALAR.STRING.false : ||r|| / ||r init|| = .SCALAR.STRING.true _Var70.SCALAR.STRING --------CP rmvar _Var66 --------CP + _Var67.SCALAR.STRING.false ,.SCALAR.STRING.true _Var71.SCALAR.STRING --------CP rmvar _Var67 --------CP + _Var68.SCALAR.STRING.false ,.SCALAR.STRING.true _Var72.SCALAR.STRING --------CP rmvar _Var68 --------CP castdts _mVar69.MATRIX.DOUBLE.false _Var73.SCALAR.DOUBLE --------CP rmvar _mVar69 --------CP / norm_r2.SCALAR.DOUBLE.false _Var73.SCALAR.DOUBLE.false _Var74.SCALAR.DOUBLE --------CP rmvar _Var73 --------CP createvar _mVar75 scratch_space//_p15212_10.168.31.80//_t0/temp22 true MATRIX binaryblock 2 1 1000 1000 -1 copy --------CP +* beta_unscaled.MATRIX.DOUBLE _Var74.SCALAR.DOUBLE.false p.MATRIX.DOUBLE _mVar75.MATRIX.DOUBLE --------CP createvar _mVar76 scratch_space//_p15212_10.168.31.80//_t0/temp23 true MATRIX binaryblock 2 1 1000 1000 -1 copy --------CP +* r.MATRIX.DOUBLE _Var74.SCALAR.DOUBLE.false _mVar65.MATRIX.DOUBLE _mVar76.MATRIX.DOUBLE --------CP rmvar _Var74 --------CP rmvar _mVar65 --------CP createvar _mVar77 scratch_space//_p15212_10.168.31.80//_t0/temp24 true MATRIX binaryblock 1 1 1000 1000 -1 copy --------CP tsmm _mVar76.MATRIX.DOUBLE _mVar77.MATRIX.DOUBLE LEFT 48 --------CP createvar _mVar78 scratch_space//_p15212_10.168.31.80//_t0/temp25 true MATRIX binaryblock 2 1 1000 1000 -1 copy --------CP - 0.SCALAR.INT.true _mVar76.MATRIX.DOUBLE _mVar78.MATRIX.DOUBLE --------CP castdts _mVar77.MATRIX.DOUBLE.false _Var79.SCALAR.DOUBLE --------CP rmvar _mVar77 --------CP / _Var79.SCALAR.DOUBLE.false norm_r2.SCALAR.DOUBLE.false _Var80.SCALAR.DOUBLE --------CP / _Var79.SCALAR.DOUBLE.false norm_r2_initial.SCALAR.DOUBLE.false _Var81.SCALAR.DOUBLE --------CP sqrt _Var79.SCALAR.DOUBLE.false _Var82.SCALAR.DOUBLE --------CP createvar _mVar83 scratch_space//_p15212_10.168.31.80//_t0/temp26 true MATRIX binaryblock 2 1 1000 1000 -1 copy --------CP +* _mVar78.MATRIX.DOUBLE _Var80.SCALAR.DOUBLE.false p.MATRIX.DOUBLE _mVar83.MATRIX.DOUBLE --------CP rmvar _mVar78 --------CP rmvar _Var80 --------CP sqrt _Var81.SCALAR.DOUBLE.false _Var84.SCALAR.DOUBLE --------CP rmvar _Var81 --------CP + _Var71.SCALAR.STRING.false _Var82.SCALAR.DOUBLE.false _Var85.SCALAR.STRING --------CP rmvar _Var71 --------CP rmvar _Var82 --------CP + _Var70.SCALAR.STRING.false _Var84.SCALAR.DOUBLE.false _Var86.SCALAR.STRING --------CP rmvar _Var70 --------CP append log_str.SCALAR.STRING.false _Var85.SCALAR.STRING.false -1.SCALAR.INT.true _Var87.SCALAR.STRING true --------CP rmvar _Var85 --------CP + _Var72.SCALAR.STRING.false _Var84.SCALAR.DOUBLE.false _Var88.SCALAR.STRING --------CP rmvar _Var72 --------CP rmvar _Var84 --------CP print _Var86.SCALAR.STRING.false _Var89.SCALAR.STRING --------CP rmvar _Var86 --------CP append _Var87.SCALAR.STRING.false _Var88.SCALAR.STRING.false -1.SCALAR.INT.true _Var90.SCALAR.STRING true --------CP rmvar _Var87 --------CP rmvar _Var88 --------CP rmvar p --------CP rmvar r --------CP rmvar beta_unscaled --------CP assignvar _Var64.SCALAR.INT.false i.SCALAR.INT --------CP assignvar _Var79.SCALAR.DOUBLE.false norm_r2.SCALAR.DOUBLE --------CP assignvar _Var90.SCALAR.STRING.false log_str.SCALAR.STRING --------CP rmvar _Var64 --------CP cpvar _mVar75 beta_unscaled --------CP cpvar _mVar76 r --------CP rmvar _Var79 --------CP cpvar _mVar83 p --------CP rmvar _Var89 --------CP rmvar _Var90 --------CP rmvar _mVar75 --------CP rmvar _mVar76 --------CP rmvar _mVar83 --------CP rmvar q ----IF (lines 202-204) ------CP >= i.SCALAR.INT.false max_iteration.SCALAR.INT.false _Var91.SCALAR.BOOLEAN ------CP rmvar _Var91 ------GENERIC (lines 203-203) [recompile=false] --------CP print Warning: the maximum number of iterations has been reached..SCALAR.STRING.true _Var92.SCALAR.STRING --------CP rmvar _Var92 ----GENERIC (lines 205-205) [recompile=false] ------CP print The CG algorithm is done..SCALAR.STRING.true _Var93.SCALAR.STRING ------CP rmvar _Var93 ----GENERIC (lines 212-212) [recompile=false] ------CP cpvar beta_unscaled beta ----GENERIC (lines 215-226) [recompile=false] ------CP print Computing the statistics....SCALAR.STRING.true _Var94.SCALAR.STRING ------CP uak+ y.MATRIX.DOUBLE _Var95.SCALAR.DOUBLE 48 ------CP createvar _mVar96 scratch_space//_p15212_10.168.31.80//_t0/temp27 true MATRIX binaryblock 1 1 1000 1000 -1 copy ------CP tsmm y.MATRIX.DOUBLE _mVar96.MATRIX.DOUBLE LEFT 48 ------CP createvar _mVar97 scratch_space//_p15212_10.168.31.80//_t0/temp28 true MATRIX binaryblock 422 1 1000 1000 -1 copy ------CP ba+* X.MATRIX.DOUBLE beta.MATRIX.DOUBLE _mVar97.MATRIX.DOUBLE 48 ------CP / _Var95.SCALAR.DOUBLE.false 422.SCALAR.INT.true _Var98.SCALAR.DOUBLE ------CP rmvar _Var95 ------CP castdts _mVar96.MATRIX.DOUBLE.false _Var99.SCALAR.DOUBLE ------CP rmvar _mVar96 ------CP createvar _mVar100 scratch_space//_p15212_10.168.31.80//_t0/temp29 true MATRIX binaryblock 422 1 1000 1000 -1 copy ------CP - y.MATRIX.DOUBLE _mVar97.MATRIX.DOUBLE _mVar100.MATRIX.DOUBLE ------CP rmvar _mVar97 ------CP ^ _Var98.SCALAR.DOUBLE.false 2.SCALAR.INT.true _Var101.SCALAR.DOUBLE ------CP uak+ _mVar100.MATRIX.DOUBLE _Var102.SCALAR.DOUBLE 48 ------CP createvar _mVar103 scratch_space//_p15212_10.168.31.80//_t0/temp30 true MATRIX binaryblock 1 1 1000 1000 -1 copy ------CP tsmm _mVar100.MATRIX.DOUBLE _mVar103.MATRIX.DOUBLE LEFT 48 ------CP rmvar _mVar100 ------CP * 422.SCALAR.INT.true _Var101.SCALAR.DOUBLE.false _Var104.SCALAR.DOUBLE ------CP rmvar _Var101 ------CP / _Var102.SCALAR.DOUBLE.false 422.SCALAR.INT.true _Var105.SCALAR.DOUBLE ------CP rmvar _Var102 ------CP castdts _mVar103.MATRIX.DOUBLE.false _Var106.SCALAR.DOUBLE ------CP rmvar _mVar103 ------CP - _Var99.SCALAR.DOUBLE.false _Var104.SCALAR.DOUBLE.false _Var107.SCALAR.DOUBLE ------CP rmvar _Var104 ------CP ^ _Var105.SCALAR.DOUBLE.false 2.SCALAR.INT.true _Var108.SCALAR.DOUBLE ------CP / _Var107.SCALAR.DOUBLE.false 421.SCALAR.INT.true _Var109.SCALAR.DOUBLE ------CP * 422.SCALAR.INT.true _Var108.SCALAR.DOUBLE.false _Var110.SCALAR.DOUBLE ------CP rmvar _Var108 ------CP / _Var106.SCALAR.DOUBLE.false _Var107.SCALAR.DOUBLE.false _Var111.SCALAR.DOUBLE ------CP - _Var106.SCALAR.DOUBLE.false _Var110.SCALAR.DOUBLE.false _Var112.SCALAR.DOUBLE ------CP rmvar _Var110 ------CP - 1.SCALAR.INT.true _Var111.SCALAR.DOUBLE.false _Var113.SCALAR.DOUBLE ------CP rmvar _Var111 ------CP assignvar _Var98.SCALAR.DOUBLE.false avg_tot.SCALAR.DOUBLE ------CP assignvar _Var99.SCALAR.DOUBLE.false ss_tot.SCALAR.DOUBLE ------CP assignvar _Var105.SCALAR.DOUBLE.false avg_res.SCALAR.DOUBLE ------CP assignvar _Var106.SCALAR.DOUBLE.false ss_res.SCALAR.DOUBLE ------CP assignvar _Var107.SCALAR.DOUBLE.false ss_avg_tot.SCALAR.DOUBLE ------CP assignvar _Var109.SCALAR.DOUBLE.false var_tot.SCALAR.DOUBLE ------CP assignvar _Var112.SCALAR.DOUBLE.false ss_avg_res.SCALAR.DOUBLE ------CP assignvar _Var113.SCALAR.DOUBLE.false plain_R2.SCALAR.DOUBLE ------CP rmvar _Var94 ------CP rmvar _Var98 ------CP rmvar _Var99 ------CP rmvar _Var105 ------CP rmvar _Var106 ------CP rmvar _Var107 ------CP rmvar _Var109 ------CP rmvar _Var112 ------CP rmvar _Var113 ------CP rmvar X ------CP rmvar y ----IF (lines 227-233) ------CP > 422.SCALAR.INT.true m_ext.SCALAR.INT.false _Var114.SCALAR.BOOLEAN ------CP rmvar _Var114 ------GENERIC (lines 228-229) [recompile=false] --------CP - 422.SCALAR.INT.true m_ext.SCALAR.INT.false _Var115.SCALAR.INT --------CP / ss_avg_tot.SCALAR.DOUBLE.false 421.SCALAR.INT.true _Var116.SCALAR.DOUBLE --------CP / ss_res.SCALAR.DOUBLE.false _Var115.SCALAR.INT.false _Var117.SCALAR.DOUBLE --------CP rmvar _Var115 --------CP / _Var117.SCALAR.DOUBLE.false _Var116.SCALAR.DOUBLE.false _Var118.SCALAR.DOUBLE --------CP rmvar _Var116 --------CP - 1.SCALAR.INT.true _Var118.SCALAR.DOUBLE.false _Var119.SCALAR.DOUBLE --------CP rmvar _Var118 --------CP assignvar _Var117.SCALAR.DOUBLE.false dispersion.SCALAR.DOUBLE --------CP assignvar _Var119.SCALAR.DOUBLE.false adjusted_R2.SCALAR.DOUBLE --------CP rmvar _Var117 --------CP rmvar _Var119 --------CP rmvar m_ext ----ELSE ------GENERIC (lines 231-232) [recompile=false] --------CP assignvar NaN.SCALAR.DOUBLE.true dispersion.SCALAR.DOUBLE --------CP assignvar NaN.SCALAR.DOUBLE.true adjusted_R2.SCALAR.DOUBLE ----GENERIC (lines 235-236) [recompile=false] ------CP / ss_avg_res.SCALAR.DOUBLE.false ss_avg_tot.SCALAR.DOUBLE.false _Var120.SCALAR.DOUBLE ------CP - 1.SCALAR.INT.true _Var120.SCALAR.DOUBLE.false _Var121.SCALAR.DOUBLE ------CP rmvar _Var120 ------CP assignvar 420.SCALAR.INT.true deg_freedom.SCALAR.INT ------CP assignvar _Var121.SCALAR.DOUBLE.false plain_R2_nobias.SCALAR.DOUBLE ------CP rmvar _Var121 ----IF (lines 237-244) ------CP > deg_freedom.SCALAR.INT.false 0.SCALAR.INT.true _Var122.SCALAR.BOOLEAN ------CP rmvar _Var122 ------GENERIC (lines 238-239) [recompile=false] --------CP / ss_avg_res.SCALAR.DOUBLE.false deg_freedom.SCALAR.INT.false _Var123.SCALAR.DOUBLE --------CP / ss_avg_tot.SCALAR.DOUBLE.false 421.SCALAR.INT.true _Var124.SCALAR.DOUBLE --------CP / _Var123.SCALAR.DOUBLE.false _Var124.SCALAR.DOUBLE.false _Var125.SCALAR.DOUBLE --------CP rmvar _Var124 --------CP - 1.SCALAR.INT.true _Var125.SCALAR.DOUBLE.false _Var126.SCALAR.DOUBLE --------CP rmvar _Var125 --------CP assignvar _Var123.SCALAR.DOUBLE.false var_res.SCALAR.DOUBLE --------CP assignvar _Var126.SCALAR.DOUBLE.false adjusted_R2_nobias.SCALAR.DOUBLE --------CP rmvar _Var123 --------CP rmvar _Var126 --------CP rmvar ss_avg_res --------CP rmvar ss_avg_tot --------CP rmvar deg_freedom ----ELSE ------GENERIC (lines 241-243) [recompile=false] --------CP print Warning: zero or negative number of degrees of freedom..SCALAR.STRING.true _Var127.SCALAR.STRING --------CP assignvar NaN.SCALAR.DOUBLE.true var_res.SCALAR.DOUBLE --------CP assignvar NaN.SCALAR.DOUBLE.true adjusted_R2_nobias.SCALAR.DOUBLE --------CP rmvar _Var127 ----GENERIC (lines 246-246) [recompile=false] ------CP / ss_res.SCALAR.DOUBLE.false ss_tot.SCALAR.DOUBLE.false _Var128.SCALAR.DOUBLE ------CP - 1.SCALAR.INT.true _Var128.SCALAR.DOUBLE.false _Var129.SCALAR.DOUBLE ------CP rmvar _Var128 ------CP assignvar _Var129.SCALAR.DOUBLE.false plain_R2_vs_0.SCALAR.DOUBLE ------CP rmvar _Var129 ----GENERIC (lines 248-248) [recompile=false] ------CP / ss_res.SCALAR.DOUBLE.false 421.SCALAR.INT.true _Var130.SCALAR.DOUBLE ------CP / ss_tot.SCALAR.DOUBLE.false 422.SCALAR.INT.true _Var131.SCALAR.DOUBLE ------CP / _Var130.SCALAR.DOUBLE.false _Var131.SCALAR.DOUBLE.false _Var132.SCALAR.DOUBLE ------CP rmvar _Var130 ------CP rmvar _Var131 ------CP - 1.SCALAR.INT.true _Var132.SCALAR.DOUBLE.false _Var133.SCALAR.DOUBLE ------CP rmvar _Var132 ------CP assignvar _Var133.SCALAR.DOUBLE.false adjusted_R2_vs_0.SCALAR.DOUBLE ------CP rmvar _Var133 ------CP rmvar ss_tot ------CP rmvar m ------CP rmvar n ------CP rmvar ss_res ----GENERIC (lines 253-263) [recompile=false] ------CP toString target=beta _Var134.SCALAR.STRING ------CP + AVG_TOT_Y,.SCALAR.STRING.true avg_tot.SCALAR.DOUBLE.false _Var135.SCALAR.STRING ------CP sqrt var_tot.SCALAR.DOUBLE.false _Var136.SCALAR.DOUBLE ------CP + AVG_RES_Y,.SCALAR.STRING.true avg_res.SCALAR.DOUBLE.false _Var137.SCALAR.STRING ------CP sqrt var_res.SCALAR.DOUBLE.false _Var138.SCALAR.DOUBLE ------CP + DISPERSION,.SCALAR.STRING.true dispersion.SCALAR.DOUBLE.false _Var139.SCALAR.STRING ------CP + PLAIN_R2,.SCALAR.STRING.true plain_R2.SCALAR.DOUBLE.false _Var140.SCALAR.STRING ------CP + ADJUSTED_R2,.SCALAR.STRING.true adjusted_R2.SCALAR.DOUBLE.false _Var141.SCALAR.STRING ------CP + PLAIN_R2_NOBIAS,.SCALAR.STRING.true plain_R2_nobias.SCALAR.DOUBLE.false _Var142.SCALAR.STRING ------CP + ADJUSTED_R2_NOBIAS,.SCALAR.STRING.true adjusted_R2_nobias.SCALAR.DOUBLE.false _Var143.SCALAR.STRING ------CP print _Var134.SCALAR.STRING.false _Var144.SCALAR.STRING ------CP rmvar _Var134 ------CP + STDEV_TOT_Y,.SCALAR.STRING.true _Var136.SCALAR.DOUBLE.false _Var145.SCALAR.STRING ------CP rmvar _Var136 ------CP + STDEV_RES_Y,.SCALAR.STRING.true _Var138.SCALAR.DOUBLE.false _Var146.SCALAR.STRING ------CP rmvar _Var138 ------CP append _Var135.SCALAR.STRING.false _Var145.SCALAR.STRING.false -1.SCALAR.INT.true _Var147.SCALAR.STRING true ------CP rmvar _Var135 ------CP rmvar _Var145 ------CP append _Var147.SCALAR.STRING.false _Var137.SCALAR.STRING.false -1.SCALAR.INT.true _Var148.SCALAR.STRING true ------CP rmvar _Var147 ------CP rmvar _Var137 ------CP append _Var148.SCALAR.STRING.false _Var146.SCALAR.STRING.false -1.SCALAR.INT.true _Var149.SCALAR.STRING true ------CP rmvar _Var148 ------CP rmvar _Var146 ------CP append _Var149.SCALAR.STRING.false _Var139.SCALAR.STRING.false -1.SCALAR.INT.true _Var150.SCALAR.STRING true ------CP rmvar _Var149 ------CP rmvar _Var139 ------CP append _Var150.SCALAR.STRING.false _Var140.SCALAR.STRING.false -1.SCALAR.INT.true _Var151.SCALAR.STRING true ------CP rmvar _Var150 ------CP rmvar _Var140 ------CP append _Var151.SCALAR.STRING.false _Var141.SCALAR.STRING.false -1.SCALAR.INT.true _Var152.SCALAR.STRING true ------CP rmvar _Var151 ------CP rmvar _Var141 ------CP append _Var152.SCALAR.STRING.false _Var142.SCALAR.STRING.false -1.SCALAR.INT.true _Var153.SCALAR.STRING true ------CP rmvar _Var152 ------CP rmvar _Var142 ------CP append _Var153.SCALAR.STRING.false _Var143.SCALAR.STRING.false -1.SCALAR.INT.true _Var154.SCALAR.STRING true ------CP rmvar _Var153 ------CP rmvar _Var143 ------CP assignvar _Var154.SCALAR.STRING.false str.SCALAR.STRING ------CP rmvar _Var144 ------CP rmvar _Var154 ------CP rmvar avg_res ------CP rmvar adjusted_R2 ------CP rmvar plain_R2_nobias ------CP rmvar var_res ------CP rmvar plain_R2 ------CP rmvar var_tot ------CP rmvar adjusted_R2_nobias ------CP rmvar avg_tot ------CP rmvar dispersion ----GENERIC (lines 272-272) [recompile=false] ------CP print str.SCALAR.STRING.false _Var155.SCALAR.STRING ------CP rmvar _Var155 ------CP rmvar str ----GENERIC (lines 276-276) [recompile=false] ------CP print Writing the output matrix....SCALAR.STRING.true _Var156.SCALAR.STRING ------CP rmvar _Var156 ----GENERIC (lines 281-281) [recompile=false] ------CP cpvar beta beta_out ------CP rmvar beta ----GENERIC (lines 283-283) [recompile=false] ------CP write beta_out.MATRIX.DOUBLE beta.txt.SCALAR.STRING.true textcell.SCALAR.STRING.true .SCALAR.STRING.true ------CP rmvar fileB ------CP rmvar beta_out ------CP rmvar fmtB ----GENERIC (lines 288-288) [recompile=false] ------CP print END LINEAR REGRESSION SCRIPT.SCALAR.STRING.true _Var157.SCALAR.STRING ------CP rmvar _Var157 {code} was (Author: iyounus): output of explain from {[LinearRegCG.dml}}: {code} 17/02/15 12:10:52 INFO api.DMLScript: EXPLAIN (RUNTIME): # Memory Budget local/remote = 3823MB/?MB/?MB/?MB # Degree of Parallelism (vcores) local/remote = 48/? PROGRAM ( size CP/SP = 385/2 ) --MAIN PROGRAM ----GENERIC (lines 86-110) [recompile=false] ------CP createvar pREADX /user/iyounus/data/diabetes_X_train.txt false MATRIX csv 422 1 -1 -1 -1 copy false , true 0.0 ------CP createvar pREADy /user/iyounus/data/diabetes_y_train.txt false MATRIX csv 422 1 -1 -1 -1 copy false , true 0.0 ------CP print BEGIN LINEAR REGRESSION SCRIPT.SCALAR.STRING.true _Var29.SCALAR.STRING ------CP print Reading X and Y....SCALAR.STRING.true _Var30.SCALAR.STRING ------CP createvar _mVar31 scratch_space//_p15212_10.168.31.80//_t0/temp1 true MATRIX binaryblock 422 1 1000 1000 422 copy ------CP rand 422 1 1000 1000 1.0 1.0 1.0 -1 uniform 1.0 48 _mVar31.MATRIX.DOUBLE ------CP createvar _mVar32 scratch_space//_p15212_10.168.31.80//_t0/temp2 true MATRIX binaryblock 422 1 1000 1000 -1 copy ------SPARK csvrblk pREADX.MATRIX.DOUBLE _mVar32.MATRIX.DOUBLE 1000 1000 false , true 0.0 ------CP createvar _mVar33 scratch_space//_p15212_10.168.31.80//_t0/temp3 true MATRIX binaryblock 422 1 1000 1000 -1 copy ------SPARK csvrblk pREADy.MATRIX.DOUBLE _mVar33.MATRIX.DOUBLE 1000 1000 false , true 0.0 ------CP assignvar beta.txt.SCALAR.STRING.true fileB.SCALAR.STRING ------CP assignvar .SCALAR.STRING.true fileO.SCALAR.STRING ------CP assignvar .SCALAR.STRING.true fileLog.SCALAR.STRING ------CP assignvar text.SCALAR.STRING.true fmtB.SCALAR.STRING ------CP assignvar 1.SCALAR.INT.true intercept_status.SCALAR.INT ------CP assignvar 1.0E-6.SCALAR.DOUBLE.true tolerance.SCALAR.DOUBLE ------CP assignvar 0.SCALAR.INT.true max_iteration.SCALAR.INT ------CP assignvar 1.0E-6.SCALAR.DOUBLE.true regularization.SCALAR.DOUBLE ------CP assignvar 422.SCALAR.INT.true n.SCALAR.INT ------CP assignvar 1.SCALAR.INT.true m.SCALAR.INT ------CP assignvar 1.SCALAR.INT.true m_ext.SCALAR.INT ------CP rmvar _Var29 ------CP rmvar _Var30 ------CP cpvar _mVar31 ones_n ------CP cpvar _mVar32 X ------CP cpvar _mVar33 y ------CP rmvar _mVar31 ------CP rmvar _mVar32 ------CP rmvar _mVar33 ----GENERIC (lines 113-114) [recompile=false] ------CP createvar _mVar34 scratch_space//_p15212_10.168.31.80//_t0/temp4 true MATRIX binaryblock 422 2 1000 1000 -1 copy ------CP append X.MATRIX.DOUBLE ones_n.MATRIX.DOUBLE 1.SCALAR.INT.true _mVar34.MATRIX.DOUBLE true ------CP rmvar X ------CP assignvar 2.SCALAR.INT.true m_ext.SCALAR.INT ------CP cpvar _mVar34 X ------CP rmvar _mVar34 ------CP rmvar ones_n ----GENERIC (lines 117-117) [recompile=false] ------CP createvar _mVar35 scratch_space//_p15212_10.168.31.80//_t0/temp5 true MATRIX binaryblock 2 1 1000 1000 -1 copy ------CP rand 2 1 1000 1000 1.0 1.0 1.0 -1 uniform 1.0 48 _mVar35.MATRIX.DOUBLE ------CP cpvar _mVar35 scale_lambda ------CP rmvar _mVar35 ----GENERIC (lines 120-120) [recompile=false] ------CP createvar _mVar36 scratch_space//_p15212_10.168.31.80//_t0/temp6 true MATRIX binaryblock 2 1 1000 1000 -1 copy ------CP leftIndex scale_lambda.MATRIX.DOUBLE 0.SCALAR.INT.true m_ext.SCALAR.INT.false m_ext.SCALAR.INT.false 1.SCALAR.INT.true 1.SCALAR.INT.true _mVar36.MATRIX.DOUBLE ------CP rmvar scale_lambda ------CP cpvar _mVar36 scale_lambda ------CP rmvar _mVar36 ----GENERIC (lines 133-134) [recompile=false] ------CP createvar _mVar37 scratch_space//_p15212_10.168.31.80//_t0/temp7 true MATRIX binaryblock 2 1 1000 1000 -1 copy ------CP rand 2 1 1000 1000 1.0 1.0 1.0 -1 uniform 1.0 48 _mVar37.MATRIX.DOUBLE ------CP createvar _mVar38 scratch_space//_p15212_10.168.31.80//_t0/temp8 true MATRIX binaryblock 2 1 1000 1000 0 copy ------CP rand 2 1 1000 1000 0.0 0.0 1.0 -1 uniform 1.0 48 _mVar38.MATRIX.DOUBLE ------CP cpvar _mVar37 scale_X ------CP cpvar _mVar38 shift_X ------CP rmvar _mVar37 ------CP rmvar _mVar38 ----GENERIC (lines 149-150) [recompile=false] ------CP createvar _mVar39 scratch_space//_p15212_10.168.31.80//_t0/temp9 true MATRIX binaryblock 2 1 1000 1000 -1 copy ------CP * scale_lambda.MATRIX.DOUBLE 1.0E-6.SCALAR.DOUBLE.true _mVar39.MATRIX.DOUBLE ------CP createvar _mVar40 scratch_space//_p15212_10.168.31.80//_t0/temp10 true MATRIX binaryblock 2 1 1000 1000 0 copy ------CP rand 2 1 1000 1000 0.0 0.0 1.0 -1 uniform 1.0 48 _mVar40.MATRIX.DOUBLE ------CP cpvar _mVar39 lambda ------CP cpvar _mVar40 beta_unscaled ------CP rmvar _mVar39 ------CP rmvar _mVar40 ------CP rmvar regularization ------CP rmvar scale_lambda ----GENERIC (lines 153-153) [recompile=false] ------CP assignvar m_ext.SCALAR.INT.false max_iteration.SCALAR.INT ----GENERIC (lines 155-160) [recompile=false] ------CP print Running the CG algorithm....SCALAR.STRING.true _Var41.SCALAR.STRING ------CP createvar _mVar42 scratch_space//_p15212_10.168.31.80//_t0/temp11 true MATRIX binaryblock 1 422 1000 1000 -1 copy ------CP r' y.MATRIX.DOUBLE _mVar42.MATRIX.DOUBLE 48 ------CP createvar _mVar43 scratch_space//_p15212_10.168.31.80//_t0/temp12 true MATRIX binaryblock 1 2 1000 1000 -1 copy ------CP ba+* _mVar42.MATRIX.DOUBLE X.MATRIX.DOUBLE _mVar43.MATRIX.DOUBLE 48 ------CP rmvar _mVar42 ------CP createvar _mVar44 scratch_space//_p15212_10.168.31.80//_t0/temp13 true MATRIX binaryblock 2 1 1000 1000 -1 copy ------CP r' _mVar43.MATRIX.DOUBLE _mVar44.MATRIX.DOUBLE 48 ------CP rmvar _mVar43 ------CP createvar _mVar45 scratch_space//_p15212_10.168.31.80//_t0/temp14 true MATRIX binaryblock 2 1 1000 1000 -1 copy ------CP - 0.SCALAR.INT.true _mVar44.MATRIX.DOUBLE _mVar45.MATRIX.DOUBLE ------CP rmvar _mVar44 ------CP assignvar 0.SCALAR.INT.true i.SCALAR.INT ------CP rmvar _Var41 ------CP cpvar _mVar45 r ------CP rmvar _mVar45 ----GENERIC (lines 166-172) [recompile=false] ------CP createvar _mVar46 scratch_space//_p15212_10.168.31.80//_t0/temp15 true MATRIX binaryblock 2 1 1000 1000 -1 copy ------CP - 0.SCALAR.INT.true r.MATRIX.DOUBLE _mVar46.MATRIX.DOUBLE ------CP createvar _mVar47 scratch_space//_p15212_10.168.31.80//_t0/temp16 true MATRIX binaryblock 1 1 1000 1000 -1 copy ------CP tsmm r.MATRIX.DOUBLE _mVar47.MATRIX.DOUBLE LEFT 48 ------CP castdts _mVar47.MATRIX.DOUBLE.false _Var48.SCALAR.DOUBLE ------CP rmvar _mVar47 ------CP * _Var48.SCALAR.DOUBLE.false 1.0E-12.SCALAR.DOUBLE.true _Var49.SCALAR.DOUBLE ------CP sqrt _Var48.SCALAR.DOUBLE.false _Var50.SCALAR.DOUBLE ------CP + ||r|| initial value = .SCALAR.STRING.true _Var50.SCALAR.DOUBLE.false _Var51.SCALAR.STRING ------CP sqrt _Var49.SCALAR.DOUBLE.false _Var52.SCALAR.DOUBLE ------CP + CG_RESIDUAL_NORM,0,.SCALAR.STRING.true _Var50.SCALAR.DOUBLE.false _Var53.SCALAR.STRING ------CP rmvar _Var50 ------CP + _Var51.SCALAR.STRING.false , target value = .SCALAR.STRING.true _Var54.SCALAR.STRING ------CP rmvar _Var51 ------CP append _Var53.SCALAR.STRING.false CG_RESIDUAL_RATIO,0,1.0.SCALAR.STRING.true -1.SCALAR.INT.true _Var55.SCALAR.STRING true ------CP rmvar _Var53 ------CP + _Var54.SCALAR.STRING.false _Var52.SCALAR.DOUBLE.false _Var56.SCALAR.STRING ------CP rmvar _Var54 ------CP rmvar _Var52 ------CP print _Var56.SCALAR.STRING.false _Var57.SCALAR.STRING ------CP rmvar _Var56 ------CP assignvar _Var48.SCALAR.DOUBLE.false norm_r2.SCALAR.DOUBLE ------CP assignvar _Var48.SCALAR.DOUBLE.false norm_r2_initial.SCALAR.DOUBLE ------CP assignvar _Var49.SCALAR.DOUBLE.false norm_r2_target.SCALAR.DOUBLE ------CP assignvar _Var55.SCALAR.STRING.false log_str.SCALAR.STRING ------CP cpvar _mVar46 p ------CP rmvar _Var48 ------CP rmvar _Var49 ------CP rmvar _Var55 ------CP rmvar _Var57 ------CP rmvar _mVar46 ------CP rmvar tolerance ----GENERIC (lines 174-200) [recompile=false] ----WHILE (lines 174-200) ------CP < i.SCALAR.INT.false max_iteration.SCALAR.INT.false _Var58.SCALAR.BOOLEAN ------CP > norm_r2.SCALAR.DOUBLE.false norm_r2_target.SCALAR.DOUBLE.false _Var59.SCALAR.BOOLEAN ------CP && _Var58.SCALAR.BOOLEAN.false _Var59.SCALAR.BOOLEAN.false _Var60.SCALAR.BOOLEAN ------CP rmvar _Var58 ------CP rmvar _Var59 ------CP rmvar _Var60 ------GENERIC (lines 180-180) [recompile=false] --------CP cpvar p ssX_p ------GENERIC (lines 183-183) [recompile=false] --------CP createvar _mVar61 scratch_space//_p15212_10.168.31.80//_t0/temp17 true MATRIX binaryblock 2 1 1000 1000 -1 copy --------CP mmchain X.MATRIX.DOUBLE ssX_p.MATRIX.DOUBLE _mVar61.MATRIX.DOUBLE XtXv 48 --------CP cpvar _mVar61 q --------CP rmvar _mVar61 --------CP rmvar ssX_p ------GENERIC (lines 189-199) [recompile=false] --------CP createvar _mVar62 scratch_space//_p15212_10.168.31.80//_t0/temp18 true MATRIX binaryblock 1 2 1000 1000 -1 copy --------CP r' p.MATRIX.DOUBLE _mVar62.MATRIX.DOUBLE 48 --------CP createvar _mVar63 scratch_space//_p15212_10.168.31.80//_t0/temp19 true MATRIX binaryblock 2 1 1000 1000 -1 copy --------CP * lambda.MATRIX.DOUBLE p.MATRIX.DOUBLE _mVar63.MATRIX.DOUBLE --------CP + i.SCALAR.INT.false 1.SCALAR.INT.true _Var64.SCALAR.INT --------CP createvar _mVar65 scratch_space//_p15212_10.168.31.80//_t0/temp20 true MATRIX binaryblock 2 1 1000 1000 -1 copy --------CP + q.MATRIX.DOUBLE _mVar63.MATRIX.DOUBLE _mVar65.MATRIX.DOUBLE --------CP rmvar _mVar63 --------CP + Iteration .SCALAR.STRING.true _Var64.SCALAR.INT.false _Var66.SCALAR.STRING --------CP + CG_RESIDUAL_NORM,.SCALAR.STRING.true _Var64.SCALAR.INT.false _Var67.SCALAR.STRING --------CP + CG_RESIDUAL_RATIO,.SCALAR.STRING.true _Var64.SCALAR.INT.false _Var68.SCALAR.STRING --------CP createvar _mVar69 scratch_space//_p15212_10.168.31.80//_t0/temp21 true MATRIX binaryblock 1 1 1000 1000 -1 copy --------CP ba+* _mVar62.MATRIX.DOUBLE _mVar65.MATRIX.DOUBLE _mVar69.MATRIX.DOUBLE 48 --------CP rmvar _mVar62 --------CP + _Var66.SCALAR.STRING.false : ||r|| / ||r init|| = .SCALAR.STRING.true _Var70.SCALAR.STRING --------CP rmvar _Var66 --------CP + _Var67.SCALAR.STRING.false ,.SCALAR.STRING.true _Var71.SCALAR.STRING --------CP rmvar _Var67 --------CP + _Var68.SCALAR.STRING.false ,.SCALAR.STRING.true _Var72.SCALAR.STRING --------CP rmvar _Var68 --------CP castdts _mVar69.MATRIX.DOUBLE.false _Var73.SCALAR.DOUBLE --------CP rmvar _mVar69 --------CP / norm_r2.SCALAR.DOUBLE.false _Var73.SCALAR.DOUBLE.false _Var74.SCALAR.DOUBLE --------CP rmvar _Var73 --------CP createvar _mVar75 scratch_space//_p15212_10.168.31.80//_t0/temp22 true MATRIX binaryblock 2 1 1000 1000 -1 copy --------CP +* beta_unscaled.MATRIX.DOUBLE _Var74.SCALAR.DOUBLE.false p.MATRIX.DOUBLE _mVar75.MATRIX.DOUBLE --------CP createvar _mVar76 scratch_space//_p15212_10.168.31.80//_t0/temp23 true MATRIX binaryblock 2 1 1000 1000 -1 copy --------CP +* r.MATRIX.DOUBLE _Var74.SCALAR.DOUBLE.false _mVar65.MATRIX.DOUBLE _mVar76.MATRIX.DOUBLE --------CP rmvar _Var74 --------CP rmvar _mVar65 --------CP createvar _mVar77 scratch_space//_p15212_10.168.31.80//_t0/temp24 true MATRIX binaryblock 1 1 1000 1000 -1 copy --------CP tsmm _mVar76.MATRIX.DOUBLE _mVar77.MATRIX.DOUBLE LEFT 48 --------CP createvar _mVar78 scratch_space//_p15212_10.168.31.80//_t0/temp25 true MATRIX binaryblock 2 1 1000 1000 -1 copy --------CP - 0.SCALAR.INT.true _mVar76.MATRIX.DOUBLE _mVar78.MATRIX.DOUBLE --------CP castdts _mVar77.MATRIX.DOUBLE.false _Var79.SCALAR.DOUBLE --------CP rmvar _mVar77 --------CP / _Var79.SCALAR.DOUBLE.false norm_r2.SCALAR.DOUBLE.false _Var80.SCALAR.DOUBLE --------CP / _Var79.SCALAR.DOUBLE.false norm_r2_initial.SCALAR.DOUBLE.false _Var81.SCALAR.DOUBLE --------CP sqrt _Var79.SCALAR.DOUBLE.false _Var82.SCALAR.DOUBLE --------CP createvar _mVar83 scratch_space//_p15212_10.168.31.80//_t0/temp26 true MATRIX binaryblock 2 1 1000 1000 -1 copy --------CP +* _mVar78.MATRIX.DOUBLE _Var80.SCALAR.DOUBLE.false p.MATRIX.DOUBLE _mVar83.MATRIX.DOUBLE --------CP rmvar _mVar78 --------CP rmvar _Var80 --------CP sqrt _Var81.SCALAR.DOUBLE.false _Var84.SCALAR.DOUBLE --------CP rmvar _Var81 --------CP + _Var71.SCALAR.STRING.false _Var82.SCALAR.DOUBLE.false _Var85.SCALAR.STRING --------CP rmvar _Var71 --------CP rmvar _Var82 --------CP + _Var70.SCALAR.STRING.false _Var84.SCALAR.DOUBLE.false _Var86.SCALAR.STRING --------CP rmvar _Var70 --------CP append log_str.SCALAR.STRING.false _Var85.SCALAR.STRING.false -1.SCALAR.INT.true _Var87.SCALAR.STRING true --------CP rmvar _Var85 --------CP + _Var72.SCALAR.STRING.false _Var84.SCALAR.DOUBLE.false _Var88.SCALAR.STRING --------CP rmvar _Var72 --------CP rmvar _Var84 --------CP print _Var86.SCALAR.STRING.false _Var89.SCALAR.STRING --------CP rmvar _Var86 --------CP append _Var87.SCALAR.STRING.false _Var88.SCALAR.STRING.false -1.SCALAR.INT.true _Var90.SCALAR.STRING true --------CP rmvar _Var87 --------CP rmvar _Var88 --------CP rmvar p --------CP rmvar r --------CP rmvar beta_unscaled --------CP assignvar _Var64.SCALAR.INT.false i.SCALAR.INT --------CP assignvar _Var79.SCALAR.DOUBLE.false norm_r2.SCALAR.DOUBLE --------CP assignvar _Var90.SCALAR.STRING.false log_str.SCALAR.STRING --------CP rmvar _Var64 --------CP cpvar _mVar75 beta_unscaled --------CP cpvar _mVar76 r --------CP rmvar _Var79 --------CP cpvar _mVar83 p --------CP rmvar _Var89 --------CP rmvar _Var90 --------CP rmvar _mVar75 --------CP rmvar _mVar76 --------CP rmvar _mVar83 --------CP rmvar q ----IF (lines 202-204) ------CP >= i.SCALAR.INT.false max_iteration.SCALAR.INT.false _Var91.SCALAR.BOOLEAN ------CP rmvar _Var91 ------GENERIC (lines 203-203) [recompile=false] --------CP print Warning: the maximum number of iterations has been reached..SCALAR.STRING.true _Var92.SCALAR.STRING --------CP rmvar _Var92 ----GENERIC (lines 205-205) [recompile=false] ------CP print The CG algorithm is done..SCALAR.STRING.true _Var93.SCALAR.STRING ------CP rmvar _Var93 ----GENERIC (lines 212-212) [recompile=false] ------CP cpvar beta_unscaled beta ----GENERIC (lines 215-226) [recompile=false] ------CP print Computing the statistics....SCALAR.STRING.true _Var94.SCALAR.STRING ------CP uak+ y.MATRIX.DOUBLE _Var95.SCALAR.DOUBLE 48 ------CP createvar _mVar96 scratch_space//_p15212_10.168.31.80//_t0/temp27 true MATRIX binaryblock 1 1 1000 1000 -1 copy ------CP tsmm y.MATRIX.DOUBLE _mVar96.MATRIX.DOUBLE LEFT 48 ------CP createvar _mVar97 scratch_space//_p15212_10.168.31.80//_t0/temp28 true MATRIX binaryblock 422 1 1000 1000 -1 copy ------CP ba+* X.MATRIX.DOUBLE beta.MATRIX.DOUBLE _mVar97.MATRIX.DOUBLE 48 ------CP / _Var95.SCALAR.DOUBLE.false 422.SCALAR.INT.true _Var98.SCALAR.DOUBLE ------CP rmvar _Var95 ------CP castdts _mVar96.MATRIX.DOUBLE.false _Var99.SCALAR.DOUBLE ------CP rmvar _mVar96 ------CP createvar _mVar100 scratch_space//_p15212_10.168.31.80//_t0/temp29 true MATRIX binaryblock 422 1 1000 1000 -1 copy ------CP - y.MATRIX.DOUBLE _mVar97.MATRIX.DOUBLE _mVar100.MATRIX.DOUBLE ------CP rmvar _mVar97 ------CP ^ _Var98.SCALAR.DOUBLE.false 2.SCALAR.INT.true _Var101.SCALAR.DOUBLE ------CP uak+ _mVar100.MATRIX.DOUBLE _Var102.SCALAR.DOUBLE 48 ------CP createvar _mVar103 scratch_space//_p15212_10.168.31.80//_t0/temp30 true MATRIX binaryblock 1 1 1000 1000 -1 copy ------CP tsmm _mVar100.MATRIX.DOUBLE _mVar103.MATRIX.DOUBLE LEFT 48 ------CP rmvar _mVar100 ------CP * 422.SCALAR.INT.true _Var101.SCALAR.DOUBLE.false _Var104.SCALAR.DOUBLE ------CP rmvar _Var101 ------CP / _Var102.SCALAR.DOUBLE.false 422.SCALAR.INT.true _Var105.SCALAR.DOUBLE ------CP rmvar _Var102 ------CP castdts _mVar103.MATRIX.DOUBLE.false _Var106.SCALAR.DOUBLE ------CP rmvar _mVar103 ------CP - _Var99.SCALAR.DOUBLE.false _Var104.SCALAR.DOUBLE.false _Var107.SCALAR.DOUBLE ------CP rmvar _Var104 ------CP ^ _Var105.SCALAR.DOUBLE.false 2.SCALAR.INT.true _Var108.SCALAR.DOUBLE ------CP / _Var107.SCALAR.DOUBLE.false 421.SCALAR.INT.true _Var109.SCALAR.DOUBLE ------CP * 422.SCALAR.INT.true _Var108.SCALAR.DOUBLE.false _Var110.SCALAR.DOUBLE ------CP rmvar _Var108 ------CP / _Var106.SCALAR.DOUBLE.false _Var107.SCALAR.DOUBLE.false _Var111.SCALAR.DOUBLE ------CP - _Var106.SCALAR.DOUBLE.false _Var110.SCALAR.DOUBLE.false _Var112.SCALAR.DOUBLE ------CP rmvar _Var110 ------CP - 1.SCALAR.INT.true _Var111.SCALAR.DOUBLE.false _Var113.SCALAR.DOUBLE ------CP rmvar _Var111 ------CP assignvar _Var98.SCALAR.DOUBLE.false avg_tot.SCALAR.DOUBLE ------CP assignvar _Var99.SCALAR.DOUBLE.false ss_tot.SCALAR.DOUBLE ------CP assignvar _Var105.SCALAR.DOUBLE.false avg_res.SCALAR.DOUBLE ------CP assignvar _Var106.SCALAR.DOUBLE.false ss_res.SCALAR.DOUBLE ------CP assignvar _Var107.SCALAR.DOUBLE.false ss_avg_tot.SCALAR.DOUBLE ------CP assignvar _Var109.SCALAR.DOUBLE.false var_tot.SCALAR.DOUBLE ------CP assignvar _Var112.SCALAR.DOUBLE.false ss_avg_res.SCALAR.DOUBLE ------CP assignvar _Var113.SCALAR.DOUBLE.false plain_R2.SCALAR.DOUBLE ------CP rmvar _Var94 ------CP rmvar _Var98 ------CP rmvar _Var99 ------CP rmvar _Var105 ------CP rmvar _Var106 ------CP rmvar _Var107 ------CP rmvar _Var109 ------CP rmvar _Var112 ------CP rmvar _Var113 ------CP rmvar X ------CP rmvar y ----IF (lines 227-233) ------CP > 422.SCALAR.INT.true m_ext.SCALAR.INT.false _Var114.SCALAR.BOOLEAN ------CP rmvar _Var114 ------GENERIC (lines 228-229) [recompile=false] --------CP - 422.SCALAR.INT.true m_ext.SCALAR.INT.false _Var115.SCALAR.INT --------CP / ss_avg_tot.SCALAR.DOUBLE.false 421.SCALAR.INT.true _Var116.SCALAR.DOUBLE --------CP / ss_res.SCALAR.DOUBLE.false _Var115.SCALAR.INT.false _Var117.SCALAR.DOUBLE --------CP rmvar _Var115 --------CP / _Var117.SCALAR.DOUBLE.false _Var116.SCALAR.DOUBLE.false _Var118.SCALAR.DOUBLE --------CP rmvar _Var116 --------CP - 1.SCALAR.INT.true _Var118.SCALAR.DOUBLE.false _Var119.SCALAR.DOUBLE --------CP rmvar _Var118 --------CP assignvar _Var117.SCALAR.DOUBLE.false dispersion.SCALAR.DOUBLE --------CP assignvar _Var119.SCALAR.DOUBLE.false adjusted_R2.SCALAR.DOUBLE --------CP rmvar _Var117 --------CP rmvar _Var119 --------CP rmvar m_ext ----ELSE ------GENERIC (lines 231-232) [recompile=false] --------CP assignvar NaN.SCALAR.DOUBLE.true dispersion.SCALAR.DOUBLE --------CP assignvar NaN.SCALAR.DOUBLE.true adjusted_R2.SCALAR.DOUBLE ----GENERIC (lines 235-236) [recompile=false] ------CP / ss_avg_res.SCALAR.DOUBLE.false ss_avg_tot.SCALAR.DOUBLE.false _Var120.SCALAR.DOUBLE ------CP - 1.SCALAR.INT.true _Var120.SCALAR.DOUBLE.false _Var121.SCALAR.DOUBLE ------CP rmvar _Var120 ------CP assignvar 420.SCALAR.INT.true deg_freedom.SCALAR.INT ------CP assignvar _Var121.SCALAR.DOUBLE.false plain_R2_nobias.SCALAR.DOUBLE ------CP rmvar _Var121 ----IF (lines 237-244) ------CP > deg_freedom.SCALAR.INT.false 0.SCALAR.INT.true _Var122.SCALAR.BOOLEAN ------CP rmvar _Var122 ------GENERIC (lines 238-239) [recompile=false] --------CP / ss_avg_res.SCALAR.DOUBLE.false deg_freedom.SCALAR.INT.false _Var123.SCALAR.DOUBLE --------CP / ss_avg_tot.SCALAR.DOUBLE.false 421.SCALAR.INT.true _Var124.SCALAR.DOUBLE --------CP / _Var123.SCALAR.DOUBLE.false _Var124.SCALAR.DOUBLE.false _Var125.SCALAR.DOUBLE --------CP rmvar _Var124 --------CP - 1.SCALAR.INT.true _Var125.SCALAR.DOUBLE.false _Var126.SCALAR.DOUBLE --------CP rmvar _Var125 --------CP assignvar _Var123.SCALAR.DOUBLE.false var_res.SCALAR.DOUBLE --------CP assignvar _Var126.SCALAR.DOUBLE.false adjusted_R2_nobias.SCALAR.DOUBLE --------CP rmvar _Var123 --------CP rmvar _Var126 --------CP rmvar ss_avg_res --------CP rmvar ss_avg_tot --------CP rmvar deg_freedom ----ELSE ------GENERIC (lines 241-243) [recompile=false] --------CP print Warning: zero or negative number of degrees of freedom..SCALAR.STRING.true _Var127.SCALAR.STRING --------CP assignvar NaN.SCALAR.DOUBLE.true var_res.SCALAR.DOUBLE --------CP assignvar NaN.SCALAR.DOUBLE.true adjusted_R2_nobias.SCALAR.DOUBLE --------CP rmvar _Var127 ----GENERIC (lines 246-246) [recompile=false] ------CP / ss_res.SCALAR.DOUBLE.false ss_tot.SCALAR.DOUBLE.false _Var128.SCALAR.DOUBLE ------CP - 1.SCALAR.INT.true _Var128.SCALAR.DOUBLE.false _Var129.SCALAR.DOUBLE ------CP rmvar _Var128 ------CP assignvar _Var129.SCALAR.DOUBLE.false plain_R2_vs_0.SCALAR.DOUBLE ------CP rmvar _Var129 ----GENERIC (lines 248-248) [recompile=false] ------CP / ss_res.SCALAR.DOUBLE.false 421.SCALAR.INT.true _Var130.SCALAR.DOUBLE ------CP / ss_tot.SCALAR.DOUBLE.false 422.SCALAR.INT.true _Var131.SCALAR.DOUBLE ------CP / _Var130.SCALAR.DOUBLE.false _Var131.SCALAR.DOUBLE.false _Var132.SCALAR.DOUBLE ------CP rmvar _Var130 ------CP rmvar _Var131 ------CP - 1.SCALAR.INT.true _Var132.SCALAR.DOUBLE.false _Var133.SCALAR.DOUBLE ------CP rmvar _Var132 ------CP assignvar _Var133.SCALAR.DOUBLE.false adjusted_R2_vs_0.SCALAR.DOUBLE ------CP rmvar _Var133 ------CP rmvar ss_tot ------CP rmvar m ------CP rmvar n ------CP rmvar ss_res ----GENERIC (lines 253-263) [recompile=false] ------CP toString target=beta _Var134.SCALAR.STRING ------CP + AVG_TOT_Y,.SCALAR.STRING.true avg_tot.SCALAR.DOUBLE.false _Var135.SCALAR.STRING ------CP sqrt var_tot.SCALAR.DOUBLE.false _Var136.SCALAR.DOUBLE ------CP + AVG_RES_Y,.SCALAR.STRING.true avg_res.SCALAR.DOUBLE.false _Var137.SCALAR.STRING ------CP sqrt var_res.SCALAR.DOUBLE.false _Var138.SCALAR.DOUBLE ------CP + DISPERSION,.SCALAR.STRING.true dispersion.SCALAR.DOUBLE.false _Var139.SCALAR.STRING ------CP + PLAIN_R2,.SCALAR.STRING.true plain_R2.SCALAR.DOUBLE.false _Var140.SCALAR.STRING ------CP + ADJUSTED_R2,.SCALAR.STRING.true adjusted_R2.SCALAR.DOUBLE.false _Var141.SCALAR.STRING ------CP + PLAIN_R2_NOBIAS,.SCALAR.STRING.true plain_R2_nobias.SCALAR.DOUBLE.false _Var142.SCALAR.STRING ------CP + ADJUSTED_R2_NOBIAS,.SCALAR.STRING.true adjusted_R2_nobias.SCALAR.DOUBLE.false _Var143.SCALAR.STRING ------CP print _Var134.SCALAR.STRING.false _Var144.SCALAR.STRING ------CP rmvar _Var134 ------CP + STDEV_TOT_Y,.SCALAR.STRING.true _Var136.SCALAR.DOUBLE.false _Var145.SCALAR.STRING ------CP rmvar _Var136 ------CP + STDEV_RES_Y,.SCALAR.STRING.true _Var138.SCALAR.DOUBLE.false _Var146.SCALAR.STRING ------CP rmvar _Var138 ------CP append _Var135.SCALAR.STRING.false _Var145.SCALAR.STRING.false -1.SCALAR.INT.true _Var147.SCALAR.STRING true ------CP rmvar _Var135 ------CP rmvar _Var145 ------CP append _Var147.SCALAR.STRING.false _Var137.SCALAR.STRING.false -1.SCALAR.INT.true _Var148.SCALAR.STRING true ------CP rmvar _Var147 ------CP rmvar _Var137 ------CP append _Var148.SCALAR.STRING.false _Var146.SCALAR.STRING.false -1.SCALAR.INT.true _Var149.SCALAR.STRING true ------CP rmvar _Var148 ------CP rmvar _Var146 ------CP append _Var149.SCALAR.STRING.false _Var139.SCALAR.STRING.false -1.SCALAR.INT.true _Var150.SCALAR.STRING true ------CP rmvar _Var149 ------CP rmvar _Var139 ------CP append _Var150.SCALAR.STRING.false _Var140.SCALAR.STRING.false -1.SCALAR.INT.true _Var151.SCALAR.STRING true ------CP rmvar _Var150 ------CP rmvar _Var140 ------CP append _Var151.SCALAR.STRING.false _Var141.SCALAR.STRING.false -1.SCALAR.INT.true _Var152.SCALAR.STRING true ------CP rmvar _Var151 ------CP rmvar _Var141 ------CP append _Var152.SCALAR.STRING.false _Var142.SCALAR.STRING.false -1.SCALAR.INT.true _Var153.SCALAR.STRING true ------CP rmvar _Var152 ------CP rmvar _Var142 ------CP append _Var153.SCALAR.STRING.false _Var143.SCALAR.STRING.false -1.SCALAR.INT.true _Var154.SCALAR.STRING true ------CP rmvar _Var153 ------CP rmvar _Var143 ------CP assignvar _Var154.SCALAR.STRING.false str.SCALAR.STRING ------CP rmvar _Var144 ------CP rmvar _Var154 ------CP rmvar avg_res ------CP rmvar adjusted_R2 ------CP rmvar plain_R2_nobias ------CP rmvar var_res ------CP rmvar plain_R2 ------CP rmvar var_tot ------CP rmvar adjusted_R2_nobias ------CP rmvar avg_tot ------CP rmvar dispersion ----GENERIC (lines 272-272) [recompile=false] ------CP print str.SCALAR.STRING.false _Var155.SCALAR.STRING ------CP rmvar _Var155 ------CP rmvar str ----GENERIC (lines 276-276) [recompile=false] ------CP print Writing the output matrix....SCALAR.STRING.true _Var156.SCALAR.STRING ------CP rmvar _Var156 ----GENERIC (lines 281-281) [recompile=false] ------CP cpvar beta beta_out ------CP rmvar beta ----GENERIC (lines 283-283) [recompile=false] ------CP write beta_out.MATRIX.DOUBLE beta.txt.SCALAR.STRING.true textcell.SCALAR.STRING.true .SCALAR.STRING.true ------CP rmvar fileB ------CP rmvar beta_out ------CP rmvar fmtB ----GENERIC (lines 288-288) [recompile=false] ------CP print END LINEAR REGRESSION SCRIPT.SCALAR.STRING.true _Var157.SCALAR.STRING ------CP rmvar _Var157 {code} > Python test failing for LinearRegCG > ----------------------------------- > > Key: SYSTEMML-1238 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1238 > Project: SystemML > Issue Type: Bug > Components: Algorithms, APIs > Affects Versions: SystemML 0.13 > Reporter: Imran Younus > Assignee: Niketan Pansare > Attachments: python_LinearReg_test_spark.1.6.log, > python_LinearReg_test_spark.2.1.log > > > [~deron] discovered that the one of the python test ({{test_mllearn_df.py}}) > with spark 2.1.0 was failing because the test score from linear regression > was very low ({{~ 0.24}}). I did a some investigation and it turns out the > the model parameters computed by the dml script are incorrect. In > systemml.12, the values of betas from linear regression model are > {{\[152.919, 938.237\]}}. This is what we expect from normal equation. (I > also tested this with sklearn). But the values of betas from systemml.13 > (with spark 2.1.0) come out to be {{\[153.146, 458.489\]}}. These are not > correct and therefore the test score is much lower than expected. The data > going into DML script is correct. I printed out the valued of {{X}} and {{Y}} > in dml and I didn't see any issue there. > Attached are the log files for two different tests (systemml0.12 and 0.13) > with explain flag. -- This message was sent by Atlassian JIRA (v6.3.15#6346)