That looks like it did the trick - thanks! On Thu, Apr 25, 2019 at 9:23 PM jonghough via Programming < [email protected]> wrote:
> Hi Devon. > Did you run the init.ijs script. If you run that initially, everything > should be setup, and you should have no problems. > On Friday, April 26, 2019, 9:54:55 AM GMT+9, Devon McCormick < > [email protected]> wrote: > > Hi - so I tried running the code at " > https://github.com/jonghough/jlearn/blob/master/adv/conv2d.ijs"; but get > numerous value errors. > > Is there another package somewhere that I need to load? > > Thanks, > > Devon > > On Fri, Apr 19, 2019 at 10:57 AM Raul Miller <[email protected]> > wrote: > > > That's the same thing as a dot product on ravels, unless the ranks of > > your arguments are ever different. > > > > Thanks, > > > > -- > > Raul > > > > On Thu, Apr 18, 2019 at 8:13 PM jonghough via Programming > > <[email protected]> wrote: > > > > > > The convolution kernel function is just a straight up elementwise > > multiply and then sum all, it is not a dot product or matrix product. > > > Nice illustration is found here: > https://mlnotebook.github.io/post/CNN1/ > > > > > > so +/@:,@:* works. I don't know if there is a faster way to do it. > > > > > > Thanks, > > > Jon On Friday, April 19, 2019, 5:54:24 AM GMT+9, Raul Miller < > > [email protected]> wrote: > > > > > > They're also not equivalent. > > > > > > For example: > > > > > > (i.2 3 4) +/@:,@:* i.2 3 > > > 970 > > > (i.2 3 4) +/ .* i.2 3 > > > |length error > > > > > > I haven't studied the possibilities of this code base enough to know > > > how relevant this might be, but if you're working with rank 4 arrays, > > > this kind of thing might matter. > > > > > > On the other hand, if the arrays handled by +/@:,@:* are the same > > > shape, then +/ .*&, might be what you want. (Then again... any change > > > introduced on "performance" grounds should get at least enough testing > > > to show that there's a current machine where that change provides > > > significant benefit for plausible data.) > > > > > > Thanks, > > > > > > -- > > > Raul > > > > > > On Thu, Apr 18, 2019 at 4:16 PM Henry Rich <[email protected]> > wrote: > > > > > > > > FYI: +/@:*"1 and +/ . * are two ways of doing dot-products fast. > > > > +/@:,@:* is not as fast. > > > > > > > > Henry Rich > > > > > > > > On 4/18/2019 10:38 AM, jonghough via Programming wrote: > > > > > > > > > > Regarding the test network I sent in the previous email, it will > not > > work. This one should: > > > > > > > > > > NB. > > ========================================================================= > > > > > > > > > > > > > > > NB. 3 classes > > > > > NB. horizontal lines (A), vertical lines (B), diagonal lines (C). > > > > > NB. each class is a 3 channel matrix 3x8x8 > > > > > > > > > > > > > > > A1=: 3 8 8 $ 1 1 1 1 1 1 1 1, 0 0 0 0 0 0 0 0, 0 0 0 0 0 0 0 0, 1 1 > > 1 1 1 1 1 1, 0 0 0 0 0 0 0 0, 1 1 1 1 1 1 1 1, 1 1 1 1 1 1 1 1, 0 0 0 0 > 0 0 > > 0 0, 0 0 0 0 0 0 0 0 > > > > > > > > > > A2=: 3 8 8 $ 1 1 1 1 1 1 1 1, 0 0 0 0 0 0 0 0, 1 1 1 1 1 1 1 1, 1 1 > > 1 1 1 1 1 1, 0 0 0 0 0 0 0 0 > > > > > > > > > > A3=: 3 8 8 $ 1 1 1 1 1 1 1 1, 0 0 0 0 0 0 0 0, 0 0 0 0 0 0 0 0 > > > > > > > > > > A4=: 3 8 8 $ 1 1 1 1 1 1 1 1, 0 0 0 0 0 0 0 0, 0 0 0 0 0 0 0 0, 1 1 > > 1 1 1 1 1 1, 1 1 1 1 1 1 1 1, 0 0 0 0 0 0 0 0 > > > > > > > > > > A5=: 2 |. A4 > > > > > > > > > > > > > > > > > > > > B1=: |:"2 A1 > > > > > > > > > > B2=: |:"2 A2 > > > > > > > > > > B3=: |:"2 A3 > > > > > > > > > > B4=: |:"2 A4 > > > > > > > > > > B5=: |:"2 A5 > > > > > > > > > > > > > > > > > > > > C1=: 3 8 8 $ 1 0 0 0 0 0 0 1, 0 1 0 0 0 0 1 0, 0 0 1 0 0 1 0 0, 0 0 > > 0 1 1 0 0 0, 0 0 0 1 1 0 0 0, 0 0 1 0 0 1 0 0, 0 1 0 0 0 0 1 0, 1 0 0 0 > 0 0 > > 0 1 > > > > > > > > > > C2=: 3 8 8 $ 1 0 0 0 0 0 0 0, 0 1 0 0 0 0 0 0, 0 0 1 0 0 0 0 0, 0 0 > > 0 1 0 0 0 0, 0 0 0 0 1 0 0 0, 0 0 0 0 0 1 0 0, 0 0 0 0 0 0 1 0, 0 0 0 0 > 0 0 > > 0 1 > > > > > > > > > > C3=: 3 8 8 $ 1 0 1 0 1 0 0 0, 0 1 0 1 0 1 0 0, 0 0 1 0 1 0 1 0, 0 0 > > 0 1 0 1 0 1, 1 0 0 0 1 0 1 0, 0 1 0 0 0 1 0 1, 1 0 1 0 0 0 1 0, 0 1 0 1 > 0 0 > > 0 1 > > > > > > > > > > C4=: |."1 C3 > > > > > > > > > > C5=: 3 8 8 $ 1 1 1 1 0 0 0 0, 0 0 1 1 1 1 0 0, 0 0 0 0 1 1 1 1, 1 1 > > 0 0 0 0 1 1, 1 1 1 1 0 0 0 0, 0 0 1 1 1 1 0 0, 0 0 0 0 1 1 1 1, 1 1 0 0 > 0 0 > > 1 1 > > > > > > > > > > > > > > > > > > > > A=: 5 3 8 8 $, A1, A2, A3, A4, A5 > > > > > > > > > > B=: 5 3 8 8 $, B1, B2, B3, B4, B5 > > > > > > > > > > C=: 5 3 8 8 $, C1, C2, C3, C4, C5 > > > > > > > > > > INPUT=: A,B,C > > > > > > > > > > OUTPUT=: 15 3 $ 1 0 0, 1 0 0, 1 0 0, 1 0 0, 1 0 0, 0 1 0, 0 1 0, 0 > 1 > > 0, 0 1 0, 0 1 0, 0 0 1, 0 0 1, 0 0 1, 0 0 1, 0 0 1 > > > > > > > > > > > > > > > > > > > > pipe=: (10;10;'softmax';1;'l2';0.0001) conew 'NNPipeline' > > > > > > > > > > c1=: ((10 3 4 4);2;'relu';'adam';0.01;0) conew 'Conv2D' > > > > > > > > > > b1=: (0; 1 ;0.0001;10;0.01) conew 'BatchNorm2D' > > > > > > > > > > a1=: 'relu' conew 'Activation' > > > > > > > > > > > > > > > > > > > > c2=: ((12 10 2 2); 1;'relu';'adam';0.01;0) conew 'Conv2D' > > > > > > > > > > b2=: (0; 1 ;0.0001;5;0.01) conew 'BatchNorm2D' > > > > > > > > > > a2=: 'relu' conew 'Activation' > > > > > > > > > > p1=: 2 conew 'PoolLayer' > > > > > > > > > > > > > > > > > > > > fl=: 3 conew 'FlattenLayer' > > > > > > > > > > fc=: (12;3;'softmax';'adam';0.01) conew 'SimpleLayer' > > > > > > > > > > b3=: (0; 1 ;0.0001;2;0.01) conew 'BatchNorm' > > > > > > > > > > a3=: 'softmax' conew 'Activation' > > > > > > > > > > > > > > > > > > > > addLayer__pipe c1 > > > > > > > > > > addLayer__pipe p1 > > > > > > > > > > NB.addLayer__pipe b1 > > > > > > > > > > addLayer__pipe a1 > > > > > > > > > > addLayer__pipe c2 > > > > > > > > > > NB.addLayer__pipe b2 > > > > > > > > > > addLayer__pipe a2 > > > > > > > > > > addLayer__pipe fl > > > > > > > > > > addLayer__pipe fc > > > > > > > > > > NB.addLayer__pipe b3 > > > > > > > > > > addLayer__pipe a3 > > > > > > > > > > require 'plot viewmat' > > > > > NB. check the input images (per channel) > > > > > NB. viewmat"2 A1 > > > > > NB. viewmat"2 B1 > > > > > NB. viewmat"2 C1 > > > > > > > > > > > > > > > OUTPUT fit__pipe INPUT NB. <--- should get 100%ish accuracy after > > only a few iterations. > > > > > NB. > > ========================================================================= > > > > > > > > > > > > > > > Running the above doesn't prove much, as there is no training / > > testing set split. It is just to see *if* the training will push the > > networks parameters in the correct direction. Getting accurate > predictions > > on all the A,B,C images will at least show that the network is not doing > > anything completely wrong. It is also just useful as a playground to see > if > > different ideas work. > > > > > > > > > > You can test the accuracy with > > > > > OUTPUT -:"1 1 (=>./)"1 >{: predict__pipe INPUT > > > > > On Thursday, April 18, 2019, 11:36:35 AM GMT+9, Brian Schott < > > [email protected]> wrote: > > > > > > > > > > I have renamed this message because the topic has changed, but > > considered > > > > > moving it to jchat as well. However I settled on jprogramming > > because there > > > > > are definitely some j programming issues to discuss. > > > > > > > > > > Jon, > > > > > > > > > > Your script code is beautifully commented and very valuable, imho. > > The lack > > > > > of an example has slowed down my study of the script, but now I > have > > some > > > > > questions and comments. > > > > > > > > > > I gather from your comments that the word tensor is used to > > designate a 4 > > > > > dimensional array. That's new to me, but it is very logical. > > > > > > > > > > Your definition convFunc=: +/@:,@:* works very well. However, for > > some > > > > > reason I wish I could think of a way to defined convFunc in terms > of > > X=: > > > > > dot=: +/ . * . > > > > > > > > > > The main insight I have gained from your code is that (x u;.+_3 y) > > can be > > > > > used with x of shape 2 n where n>2 (and not just 2 2). This is > great > > > > > information. And that you built the convFunc directly into cf is > > also very > > > > > enlightening. > > > > > > > > > > I have created a couple of examples of the use of your function > `cf` > > to > > > > > better understand how it works. [The data is borrowed from the fine > > example > > > > > at http://cs231n.github.io/convolutional-networks/#conv . Beware > > that the > > > > > dynamic example seen at the link changes everytime the page is > > refreshed, > > > > > so you will not see the exact data I present, but the shapes of the > > data > > > > > are constant.] > > > > > > > > > > Notice that in my first experiments both `filter` and the RHA of > > cf"3 are > > > > > arrays and not tensors. Consequently(?) the result is an array, > not a > > > > > tensor, either. > > > > > > > > > > i=: _7]\".;._2 (0 : 0) > > > > > 0 0 0 0 0 0 0 > > > > > 0 0 0 1 2 2 0 > > > > > 0 0 0 2 1 0 0 > > > > > 0 0 0 1 2 2 0 > > > > > 0 0 0 0 2 0 0 > > > > > 0 0 0 2 2 2 0 > > > > > 0 0 0 0 0 0 0 > > > > > 0 0 0 0 0 0 0 > > > > > 0 2 1 2 2 2 0 > > > > > 0 0 1 0 2 0 0 > > > > > 0 1 1 1 1 1 0 > > > > > 0 2 0 0 0 2 0 > > > > > 0 0 0 2 2 2 0 > > > > > 0 0 0 0 0 0 0 > > > > > 0 0 0 0 0 0 0 > > > > > 0 0 0 1 2 1 0 > > > > > 0 1 1 0 0 0 0 > > > > > 0 2 1 2 0 2 0 > > > > > 0 1 0 0 2 2 0 > > > > > 0 1 0 1 2 2 0 > > > > > 0 0 0 0 0 0 0 > > > > > ) > > > > > > > > > > k =: _3]\".;._2(0 :0) > > > > > 1 0 0 > > > > > 1 _1 0 > > > > > _1 _1 1 > > > > > 0 _1 1 > > > > > 0 0 1 > > > > > 0 _1 1 > > > > > 1 0 1 > > > > > 0 _1 0 > > > > > 0 _1 0 > > > > > ) > > > > > > > > > > $i NB. 3 7 7 > > > > > $k NB. 3 3 3 > > > > > > > > > > filter =: k > > > > > convFunc=: +/@:,@:* > > > > > > > > > > cf=: 4 : '|:"2 |: +/ x filter&(convFunc"3 3);._3 y' > > > > > (1 2 2,:3 3 3) cf"3 i NB. 3 3$1 1 _2 _2 3 _7 _3 1 0 > > > > > > > > > > My next example makes both the `filter` and the RHA into tensors. > And > > > > > notice the shape of the result shows it is a tensor, also. > > > > > > > > > > filter2 =: filter,:_1+filter > > > > > cf2=: 4 : '|:"2 |: +/ x filter2&(convFunc"3 3);._3 y' > > > > > $ (1 2 2,:3 3 3) cf2"3 i,:5+i NB. 2 2 3 3 > > > > > > > > > > Much of my effort regarding CNN has been studying the literature > that > > > > > discusses efficient ways of computing these convolutions by > > translating the > > > > > filters and the image data into flattened (and somewhat sparse} > > forms that > > > > > can be restated in matrix formats. These matrices accomplish the > > > > > convolution and deconvolution as *efficient* matrix products. Your > > > > > demonstration of the way that j's ;._3 can be so effective > > challenges the > > > > > need for such efficiencies. > > > > > > > > > > On the other hand, I could use some help understanding how the 1 0 > 2 > > 3 |: > > > > > transpose you apply to `filter` is effective in the backpropogation > > stage. > > > > > Part of my confusion is that I would have thought the transpose > > would have > > > > > been 0 1 3 2 |:, instead. Can you say more about that? > > > > > > > > > > I have yet to try to understand your verbs `forward` and > `backward`, > > but I > > > > > look forward to doing so. > > > > > > > > > > I could not find definitions for the following functions and wonder > > if you > > > > > can say more about them, please? > > > > > > > > > > bmt_jLearnUtil_ > > > > > setSolver > > > > > > > > > > I noticed that your definitions of relu and derivRelu were more > > complicated > > > > > than mine, so I attempted to test yours out against mine as > follows. > > > > > > > > > > > > > > > > > > > > relu =: 0&>. > > > > > derivRelu =: 0&< > > > > > (relu -: 0:`[@.>&0) i: 4 > > > > > 1 > > > > > (derivRelu -: 0:`1:@.>&0) i: 4 > > > > > 1 > > > > > > > > > > > > > > > > > > > > > > > > > On Sun, Apr 14, 2019 at 8:31 AM jonghough via Programming < > > > > > [email protected]> wrote: > > > > > > > > > >> I had a go writing conv nets in J. > > > > >> See > > > > >> https://github.com/jonghough/jlearn/blob/master/adv/conv2d.ijs > > > > >> > > > > >> This uses ;.3 to do the convolutions. Using a version of this , > > with a > > > > >> couple of fixes/, I managed to get 88% accuracy on the cifar-10 > > imageset. > > > > >> Took several days to run, as my algorithms are not optimized in > any > > way, > > > > >> and no gpu was used. > > > > >> If you look at the references in the above link, you may get some > > ideas. > > > > >> > > > > >> the convolution verb is defined as: > > > > >> cf=: 4 : 0 > > > > >> |:"2 |: +/ x filter&(convFunc"3 3);._3 y > > > > >> ) > > > > >> > > > > >> Note that since the input is an batch of images, each 3-d (width, > > height, > > > > >> channels), we are actually doing the whole forward pass over a 4d > > array, > > > > >> and outputting another 4d array of different shape, depending on > > output > > > > >> channels, filter width, and filter height. > > > > >> > > > > >> Thanks, > > > > >> Jon > > > > >> > > > > > Thank you, > > > > > > > > > > > > > > > > > --- > > > > This email has been checked for viruses by AVG. > > > > https://www.avg.com > > > > > > > > > ---------------------------------------------------------------------- > > > > For information about J forums see > http://www.jsoftware.com/forums.htm > > > ---------------------------------------------------------------------- > > > For information about J forums see http://www.jsoftware.com/forums.htm > > > ---------------------------------------------------------------------- > > > For information about J forums see http://www.jsoftware.com/forums.htm > > ---------------------------------------------------------------------- > > For information about J forums see http://www.jsoftware.com/forums.htm > > > > -- > > Devon McCormick, CFA > > Quantitative Consultant > ---------------------------------------------------------------------- > For information about J forums see http://www.jsoftware.com/forums.htm > > ---------------------------------------------------------------------- > For information about J forums see http://www.jsoftware.com/forums.htm > -- Devon McCormick, CFA Quantitative Consultant ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
