True. In any case, it was a useful exercise for me. Thanks. On Thu, Jan 12, 2012 at 5:42 PM, Ric Sherlock <[email protected]> wrote:
> There may be a case for special handling in 'tables/dsv' when the > string delimiter is declared as empty. > > On Fri, Jan 13, 2012 at 11:03 AM, Tom Szczesny <[email protected]> wrote: > > Yes, they are now both the same. > > ( tblcsv is about 3 times faster on large files ). > > > > On Thu, Jan 12, 2012 at 4:41 PM, Ric Sherlock <[email protected]> wrote: > > > >> I suspect the reason that the minus sign didn't convert is that the > >> column wasn't converted to numeric because the column header is part > >> of the file. If the header doesn't convert to numeric successfully the > >> column will fail the condition that the whole column must be numeric > >> for it to be converted. > >> > >> If that is indeed the problem then the following should work: > >> > >> a=: ({. , makenumcol@}.) fixcsv (1!:1<'sm.csv') -.'"/' > >> > >> On Fri, Jan 13, 2012 at 10:16 AM, Tom Szczesny <[email protected]> > wrote: > >> > No. They are not the same. For a very small sample file the results > >> are: > >> > > >> > a=: makenumcol fixcsv (1!:1<'sm.csv') -.'"/' > >> > b=:'ssdndssnnnnnnnnnnnnndnns' tblcsv 1!:1 <'sm.csv' > >> > a-:b > >> > 0 > >> > (<a:;7){a > >> > +--++---+++----+----+----+ > >> > |qu||444|||2000|6761|-100| > >> > +--++---+++----+----+----+ > >> > (<a:;7){b > >> > +--++---+++----+----+----+ > >> > |qu||444|||2000|6761|_100| > >> > +--++---+++----+----+----+ > >> > > >> > The minus sign did not get converted properly in the table/csv case. > >> > > >> > > >> > On Thu, Jan 12, 2012 at 3:46 PM, Ric Sherlock <[email protected]> > wrote: > >> > > >> >> Out of interest does the following give you the same result? > >> >> load 'tables/csv' > >> >> makenumcol fixcsv csvdata -.'"/' > >> >> > >> >> Where csvdata is the string you are feeding to tblcsv. > >> >> > >> >> On Fri, Jan 13, 2012 at 9:09 AM, Tom Szczesny <[email protected]> > wrote: > >> >> > wow > >> >> > > >> >> > On Thu, Jan 12, 2012 at 2:38 PM, Ric Sherlock <[email protected]> > >> wrote: > >> >> > > >> >> >> ( (x e. 'dn')#i.$x ) <--> ( I. x e. 'dn' ) > >> >> >> > >> >> >> You can lose the for loop and the two transposes by amending all > the > >> >> >> columns at once. > >> >> >> > >> >> >> It is better/safer to use dyadic ". for converting literals to > >> numbers. > >> >> >> > >> >> >> So this should work (untested): > >> >> >> > >> >> >> tblcsv =: 4 : 0 > >> >> >> t=.([: < ;._1 ',',]) ;._2 y-.'"/' > >> >> >> data=. }.t > >> >> >> idx=. I. x e. 'dn' > >> >> >> data=. (_99&". &.> idx{"1 data) (<a:;idx)} data > >> >> >> ({.t),data > >> >> >> ) > >> >> >> > >> >> >> Because in-place amendment is not supported for the boxed > datatype, > >> >> >> you shouldn't lose anything by dropping the ( data=. aa i}data ) > so > >> >> >> you could just do: > >> >> >> > >> >> >> tblcsv =: 4 : 0 > >> >> >> t=.([: < ;._1 ',',]) ;._2 y-.'"/' > >> >> >> idx=. I. x e. 'dn' > >> >> >> ({.t), (_99&". &.> idx{"1 }.t) (<a:;idx)} }.t > >> >> >> ) > >> >> >> > >> >> >> > >> >> >> On Fri, Jan 13, 2012 at 6:25 AM, Tom Szczesny <[email protected]> > >> wrote: > >> >> >> > Sorry for all the "chatter". I think I'm done now... > >> >> >> > Version #4: > >> >> >> > > >> >> >> > tblcsv =: 4 : 0 > >> >> >> > t=.([: < ;._1 ',',]) ;._2 y-.'"/' > >> >> >> > data=. |:}.t > >> >> >> > for_k. (x e. 'dn')#i.$x do. > >> >> >> > data=. (><&.>".'0',"1>k{data) k}data > >> >> >> > end. > >> >> >> > ({.t),|:data > >> >> >> > ) > >> >> >> > > >> >> >> > > >> >> >> > On Thu, Jan 12, 2012 at 12:05 PM, Tom Szczesny < > [email protected]> > >> >> wrote: > >> >> >> > > >> >> >> >> Replacing the - with _ is not necessary. > >> >> >> >> Version #2 only works because of a typo in the 3rd line =, > >> >> >> instead > >> >> >> >> of =. > >> >> >> >> Version #3: > >> >> >> >> > >> >> >> >> tblcsv =: 4 : 0 > >> >> >> >> r=. y-.'"/' > >> >> >> >> r=.([: < ;._1 ',',]) ;._2 r > >> >> >> >> ttl=. {.r > >> >> >> >> dat=. |:}.r > >> >> >> >> for_j. (x e. 'dn')#i.$x do. > >> >> >> >> dat=. (><&.>".'0',"1>j{dat) j}dat > >> >> >> >> end. > >> >> >> >> ttl,|:dat > >> >> >> >> ) > >> >> >> >> > >> >> >> >> > >> >> >> >> On Thu, Jan 12, 2012 at 11:41 AM, Tom Szczesny < > [email protected]> > >> >> >> wrote: > >> >> >> >> > >> >> >> >>> The version I sent earlier appeared to work, but actually does > >> not. > >> >> >> This > >> >> >> >>> version does: > >> >> >> >>> > >> >> >> >>> tblcsv =: 4 : 0 > >> >> >> >>> r=. y-.'"/' > >> >> >> >>> r=, '_' ((r='-')#i.$r) } r > >> >> >> >>> r=.([: < ;._1 ',',]) ;._2 r > >> >> >> >>> ttl=. {.r > >> >> >> >>> dat=. |:}.r > >> >> >> >>> for_k. (x e. 'dn')#i.$x do. > >> >> >> >>> dat=. (><&.>".'0',"1>k{dat) k}dat > >> >> >> >>> end. > >> >> >> >>> ttl,|:dat > >> >> >> >>> ) > >> >> >> >>> > >> >> >> >>> Thanks again to everyone for all the help. > >> >> >> >>> > >> >> >> >>> On Thu, Jan 12, 2012 at 8:59 AM, Tom Szczesny < > [email protected]> > >> >> >> wrote: > >> >> >> >>> > >> >> >> >>>> Thanks much. > >> >> >> >>>> Since I create the csv files, I was able to take some > shortcuts: > >> >> >> >>>> 1) All " are spurious (no commas in columns containing > >> >> comments) > >> >> >> >>>> 2) - only occurs in negative numbers > >> >> >> >>>> 3) / only occus in dates 2004/12/15 (remove the / and > >> don't > >> >> >> need > >> >> >> >>>> getdate at all) > >> >> >> >>>> > >> >> >> >>>> This definition is working fine: > >> >> >> >>>> > >> >> >> >>>> tblcsv =: 4 : 0 > >> >> >> >>>> r=. y-.'"/' > >> >> >> >>>> r=, '_' ((r='-')#i.$r) } r > >> >> >> >>>> r=.([: < ;._1 ',',]) ;._2 r > >> >> >> >>>> ttl=. {.r > >> >> >> >>>> dat=. |:}.r > >> >> >> >>>> for_i. (x e. 'dn')#i.$x do. > >> >> >> >>>> dat=. (<"".>i{ dat) i} dat > >> >> >> >>>> end. > >> >> >> >>>> ttl,|:dat > >> >> >> >>>> ) > >> >> >> >>>> > >> >> >> >>>> For large files, the biggest time user might be the need to > do > >> >> >> transpose > >> >> >> >>>> twice. > >> >> >> >>>> tx=:'ssdndssnnnnnnnnnnnnndnns' tblcsv 1!:1 <'t.csv' > >> >> >> >>>> > >> >> >> >>>> On Thu, Jan 12, 2012 at 6:16 AM, R.E. Boss < > [email protected]> > >> >> >> wrote: > >> >> >> >>>> > >> >> >> >>>>> If you have a lot of dates, say >100k, getdate will be > rather > >> >> slow. > >> >> >> >>>>> But since the nub of these dates probably will contain <10k > >> dates, > >> >> >> you > >> >> >> >>>>> can > >> >> >> >>>>> use nubindex > http://www.jsoftware.com/help/dictionary/d221.htm > >> >> >> >>>>> For that the following adverb will do (tested long time ago) > >> >> >> >>>>> > >> >> >> >>>>> nbind=:1 : '](i.!.0~ { u @:]) ~.' > >> >> >> >>>>> > >> >> >> >>>>> > >> >> >> >>>>> R.E. Boss > >> >> >> >>>>> > >> >> >> >>>>> > >> >> >> >>>>> > -----Oorspronkelijk bericht----- > >> >> >> >>>>> > Van: [email protected] [mailto:general- > >> >> >> >>>>> > [email protected]] Namens R.E. Boss > >> >> >> >>>>> > Verzonden: woensdag 11 januari 2012 21:48 > >> >> >> >>>>> > Aan: 'General forum' > >> >> >> >>>>> > Onderwerp: Re: [Jgeneral] Data from csv files > >> >> >> >>>>> > > >> >> >> >>>>> > require 'dates' > >> >> >> >>>>> > > >> >> >> >>>>> > 100 #. getdate'2012/01/12' > >> >> >> >>>>> > 20120112 > >> >> >> >>>>> > > >> >> >> >>>>> > I would transpose (|:) the matrix and then work a row at a > >> time, > >> >> >> >>>>> depending > >> >> >> >>>>> > on the control vector. > >> >> >> >>>>> > Recently I learned techniques to assign different verbs to > >> >> >> different > >> >> >> >>>>> items. > >> >> >> >>>>> > > >> >> >> >>>>> > > >> >> >> >>>>> > R.E. Boss > >> >> >> >>>>> > > >> >> >> >>>>> > > >> >> >> >>>>> > > -----Oorspronkelijk bericht----- > >> >> >> >>>>> > > Van: [email protected] [mailto:general- > >> >> >> >>>>> > > [email protected]] Namens Tom Szczesny > >> >> >> >>>>> > > Verzonden: woensdag 11 januari 2012 21:13 > >> >> >> >>>>> > > Aan: General forum > >> >> >> >>>>> > > Onderwerp: Re: [Jgeneral] Data from csv files > >> >> >> >>>>> > > > >> >> >> >>>>> > > Thanks, that is very nice to know, but ... > >> >> >> >>>>> > > > >> >> >> >>>>> > > Since the csv files I need were created by me, I also > know > >> >> that > >> >> >> >>>>> > > - the only occurrences of " are spuriously added. > >> >> >> >>>>> > > - the only occurrences of - are in the > representation > >> of > >> >> >> >>>>> negative > >> >> >> >>>>> > > numbers, so I can define > >> >> >> >>>>> > > > >> >> >> >>>>> > > tblcsv=: 3 : 0 > >> >> >> >>>>> > > r=: (-.y='"')#y > >> >> >> >>>>> > > r=: '_' ((r='-')#i.$r) } r > >> >> >> >>>>> > > ([: < ;._1 ',',]) ;._2 r > >> >> >> >>>>> > > ) > >> >> >> >>>>> > > > >> >> >> >>>>> > > Next, I plan to figure out how to convert the columns > with > >> >> >> character > >> >> >> >>>>> > > strings representing numbers into actual numbers, > >> >> >> >>>>> > > and the columns with character strings representing > dates ( > >> >> >> >>>>> 2012/01/12 ) > >> >> >> >>>>> > > into numbers representing dates ( 20120112 ), > >> >> >> >>>>> > > where tblcsv becomes dyadic with a control vector like > >> >> >> 'SDSSNDNSNS' > >> >> >> >>>>> as > >> >> >> >>>>> > the > >> >> >> >>>>> > > left argument > >> >> >> >>>>> > > indicating which columns are strings, dates & numbers. > >> >> >> >>>>> > > > >> >> >> >>>>> > > On Wed, Jan 11, 2012 at 2:27 PM, Ric Sherlock < > >> >> [email protected] > >> >> >> > > >> >> >> >>>>> wrote: > >> >> >> >>>>> > > > >> >> >> >>>>> > > > Note that you don't need to define tblcsv explicitly: > >> >> >> >>>>> > > > tblcsv=: ([: <;._1 ','&,);._2 > >> >> >> >>>>> > > > or > >> >> >> >>>>> > > > tblcsv=: ([: <;._1 ',' , ]);._2 > >> >> >> >>>>> > > > > >> >> >> >>>>> > > > On Thu, Jan 12, 2012 at 6:29 AM, Tom Szczesny < > >> >> >> [email protected]> > >> >> >> >>>>> > > wrote: > >> >> >> >>>>> > > > > tested ...... works . ..... thanks! > >> >> >> >>>>> > > > > > >> >> >> >>>>> > > > > On Wed, Jan 11, 2012 at 12:23 PM, R.E. Boss < > >> >> >> [email protected] > >> >> >> >>>>> > > >> >> >> >>>>> > wrote: > >> >> >> >>>>> > > > > > >> >> >> >>>>> > > > >> tblcsv =: 3 : 0 > >> >> >> >>>>> > > > >> ([: <;._1 ',',]) ;._2 y > >> >> >> >>>>> > > > >> ) > >> >> >> >>>>> > > > >> (untested) > >> >> >> >>>>> > > > >> > >> >> >> >>>>> > > > >> R.E. Boss > >> >> >> >>>>> > > > >> > >> >> >> >>>>> > > > >> > >> >> >> >>>>> > > > >> > -----Oorspronkelijk bericht----- > >> >> >> >>>>> > > > >> > Van: [email protected] [mailto: > general- > >> >> >> >>>>> > > > >> > [email protected]] Namens Tom Szczesny > >> >> >> >>>>> > > > >> > Verzonden: woensdag 11 januari 2012 17:55 > >> >> >> >>>>> > > > >> > Aan: General forum > >> >> >> >>>>> > > > >> > Onderwerp: Re: [Jgeneral] Data from csv files > >> >> >> >>>>> > > > >> > > >> >> >> >>>>> > > > >> > Given > >> >> >> >>>>> > > > >> > > >> >> >> >>>>> > > > >> > cutc =: 3 : 0 > >> >> >> >>>>> > > > >> > < ;._1 ',',y > >> >> >> >>>>> > > > >> > ) > >> >> >> >>>>> > > > >> > > >> >> >> >>>>> > > > >> > tblcsv =: 3 : 0 > >> >> >> >>>>> > > > >> > cutc ;._2 y > >> >> >> >>>>> > > > >> > ) > >> >> >> >>>>> > > > >> > > >> >> >> >>>>> > > > >> > tblcsv 1!:1 <'test.csv' > >> >> >> >>>>> > > > >> > > >> >> >> >>>>> > > > >> > I assumed that I would be able to represent this > as > >> a > >> >> >> single > >> >> >> >>>>> > > > definition, > >> >> >> >>>>> > > > >> > such as > >> >> >> >>>>> > > > >> > > >> >> >> >>>>> > > > >> > tblcsv =: 3 : 0 > >> >> >> >>>>> > > > >> > ( <;._1 ',',) ;._2 y > >> >> >> >>>>> > > > >> > ) > >> >> >> >>>>> > > > >> > > >> >> >> >>>>> > > > >> > or > >> >> >> >>>>> > > > >> > > >> >> >> >>>>> > > > >> > tblcsv =: 3 : 0 > >> >> >> >>>>> > > > >> > (( <;._1',',)&) ;._2 y > >> >> >> >>>>> > > > >> > ) > >> >> >> >>>>> > > > >> > > >> >> >> >>>>> > > > >> > I've tried several other vaiations, and all > result > >> in > >> >> >> 'syntax > >> >> >> >>>>> > error'. > >> >> >> >>>>> > > > Am > >> >> >> >>>>> > > > >> I > >> >> >> >>>>> > > > >> > missing something, or does the verb applied to > each > >> >> 'cut' > >> >> >> >>>>> interval > >> >> >> >>>>> > > > need > >> >> >> >>>>> > > > >> to > >> >> >> >>>>> > > > >> > be defined separately? > >> >> >> >>>>> > > > >> > > >> >> >> >>>>> > > > >> > On Mon, Jan 9, 2012 at 5:00 PM, Arthur Anger < > >> >> >> [email protected]> > >> >> >> >>>>> > > wrote: > >> >> >> >>>>> > > > >> > > >> >> >> >>>>> > > > >> > > I often use Rank to select sub-arrays from an > >> array: > >> >> >> >>>>> > > > >> > > <"0 i. 7 > >> >> >> >>>>> > > > >> > > --Art > >> >> >> >>>>> > > > >> > > ------------------ > >> >> >> >>>>> > > > >> > > Quoting [email protected]: > >> >> >> >>>>> > > > >> > > . . . > >> >> >> >>>>> > > > >> > > > Message: 2 > >> >> >> >>>>> > > > >> > > > Date: Mon, 9 Jan 2012 11:46:24 -0500 > >> >> >> >>>>> > > > >> > > > From: Tom Szczesny <[email protected]> > >> >> >> >>>>> > > > >> > > > Subject: Re: [Jgeneral] Data from csv files > >> >> >> >>>>> > > > >> > > > To: General forum <[email protected]> > >> >> >> >>>>> > > > >> > > > Message-ID: > >> >> >> >>>>> > > > >> > > > > <CABn7SNYFw2gyAPKcjx1DMLEru97NMTst6zoGx= > >> >> >> >>>>> > > > >> > > [email protected]> > >> >> >> >>>>> > > > >> > > > Content-Type: text/plain; charset=ISO-8859-1 > >> >> >> >>>>> > > > >> > > > > >> >> >> >>>>> > > > >> > > > As mentioned in the dictionary entry for cut: > >> >> >> >>>>> > > > >> > > > the phrase u;._2 y applies the verb u > to > >> each > >> >> >> >>>>> interval > >> >> >> >>>>> > > > created > >> >> >> >>>>> > > > >> by > >> >> >> >>>>> > > > >> > > cut, > >> >> >> >>>>> > > > >> > > > where the fret is the last item, and marks > the > >> >> ends of > >> >> >> >>>>> the > >> >> >> >>>>> > > > intervals. > >> >> >> >>>>> > > > >> > > > > >> >> >> >>>>> > > > >> > > > What is the notation for applying the "each" > >> >> concept > >> >> >> to a > >> >> >> >>>>> verb > >> >> >> >>>>> > > > >> > > independent > >> >> >> >>>>> > > > >> > > > of cut ? > >> >> >> >>>>> > > > >> > > > For example, > >> >> >> >>>>> > > > >> > > > <i.7 > >> >> >> >>>>> > > > >> > > > +-------------+ > >> >> >> >>>>> > > > >> > > > |0 1 2 3 4 5 6| > >> >> >> >>>>> > > > >> > > > +-------------+ > >> >> >> >>>>> > > > >> > > > > >> >> >> >>>>> > > > >> > > > How do you express > >> >> >> >>>>> > > > >> > > > < each i.7 > >> >> >> >>>>> > > > >> > > > and get 7 individually boxed items? > >> >> >> >>>>> > > > >> > > > (I could not find an entry for "each" in the > >> Index, > >> >> >> nor > >> >> >> >>>>> in > >> >> >> >>>>> the > >> >> >> >>>>> > > > >> > > Vocabulary.) > >> >> >> >>>>> > > > >> > > . . . > >> >> >> >>>>> > > > >> > > > End of General Digest, Vol 76, Issue 8 > >> >> >> >>>>> > > > >> > > > ************************************** > >> >> >> >>>>> > > > >> > > > >> >> >> >>>>> > > > >> > > > >> >> >> >>>>> > > > > >> >> >> >>>>> > >> >> >> > >> ---------------------------------------------------------------------- > >> >> >> >>>>> > > > >> > > For information about J forums see > >> >> >> >>>>> > > > >> > http://www.jsoftware.com/forums.htm > >> >> >> >>>>> > > > >> > > > >> >> >> >>>>> > > > >> > > >> >> >> >>>>> > > >> >> >> > >> ---------------------------------------------------------------------- > >> >> >> >>>>> > > > >> > For information about J forums see > >> >> >> >>>>> > > > http://www.jsoftware.com/forums.htm > >> >> >> >>>>> > > > >> > >> >> >> >>>>> > > > >> > >> >> >> >>>>> > > >> >> >> > >> ---------------------------------------------------------------------- > >> >> >> >>>>> > > > >> For information about J forums see > >> >> >> >>>>> > > http://www.jsoftware.com/forums.htm > >> >> >> >>>>> > > > >> > >> >> >> >>>>> > > > > > >> >> >> >>>>> > >> >> >> > >> ---------------------------------------------------------------------- > >> >> >> >>>>> > > > > For information about J forums see > >> >> >> >>>>> > > http://www.jsoftware.com/forums.htm > >> >> >> >>>>> > > > > >> >> >> >>>>> > >> >> >> > >> ---------------------------------------------------------------------- > >> >> >> >>>>> > > > For information about J forums see > >> >> >> >>>>> > > http://www.jsoftware.com/forums.htm > >> >> >> >>>>> > > > > >> >> >> >>>>> > > > >> >> >> >>>>> > >> >> >> > >> ---------------------------------------------------------------------- > >> >> >> >>>>> > > For information about J forums see > >> >> >> >>>>> > http://www.jsoftware.com/forums.htm > >> >> >> >>>>> > > >> >> >> >>>>> > > >> >> >> > >> ---------------------------------------------------------------------- > >> >> >> >>>>> > For information about J forums see > >> >> >> >>>>> http://www.jsoftware.com/forums.htm > >> >> >> >>>>> > >> >> >> >>>>> > >> >> >> > >> ---------------------------------------------------------------------- > >> >> >> >>>>> For information about J forums see > >> >> >> http://www.jsoftware.com/forums.htm > >> >> >> >>>>> > >> >> >> >>>> > >> >> >> >>>> > >> >> >> >>> > >> >> >> >> > >> >> >> > > >> ---------------------------------------------------------------------- > >> >> >> > For information about J forums see > >> >> http://www.jsoftware.com/forums.htm > >> >> >> > >> ---------------------------------------------------------------------- > >> >> >> For information about J forums see > >> http://www.jsoftware.com/forums.htm > >> >> >> > >> >> > > ---------------------------------------------------------------------- > >> >> > For information about J forums see > >> http://www.jsoftware.com/forums.htm > >> >> > ---------------------------------------------------------------------- > >> >> For information about J forums see > http://www.jsoftware.com/forums.htm > >> >> > >> > ---------------------------------------------------------------------- > >> > For information about J forums see > http://www.jsoftware.com/forums.htm > >> ---------------------------------------------------------------------- > >> For information about J forums see http://www.jsoftware.com/forums.htm > >> > > ---------------------------------------------------------------------- > > For information about J forums see http://www.jsoftware.com/forums.htm > ---------------------------------------------------------------------- > For information about J forums see http://www.jsoftware.com/forums.htm > ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
