Re: Has Anyone Got A Directory "Walker" Available
Never mind… I found this in a toolbox. Sent by some ago, by a LiveCode deva For what it's worth : # Filters the strings "." and ".." from a list function filterDots pList local tList put pList into tList filter tList without "." filter tList without ".." return tList end filterDots # Returns a filtered list of files in the current directory function filteredFiles return filterDots(the files) end filteredFiles # Returns a filtered list of folders in the current directory function filteredFolders return filterDots(the folders) end filteredFolders # Returns a list of files in the current directory including # each file's full path. function filteredFilesWithPaths local tFiles, tFilesWithPaths put filteredFiles() into tFiles repeat for each line tFile in tFiles put the directory & slash & tFile & return after \ tFilesWithPaths end repeat delete the last char of tFilesWithPaths return tFilesWithPaths end filteredFilesWithPaths # Returns a list of files in a given folder, using recursion to # include files in subfolders if desired. function listFiles pFolder, pRecurse local tTotalFiles, tCurrentFiles, tFolders set the directory to pFolder put filteredFiles() into tCurrentFiles if not pRecurse then return tCurrentFiles if tCurrentFiles is not empty then put tCurrentFiles & return after tTotalFiles end if put filteredFolders() into tFolders repeat for each line tFolder in tFolders put listFiles((pFolder & slash & tFolder), pRecurse) into \ tCurrentFiles if tCurrentFiles is not empty then put tCurrentFiles & return after tTotalFiles end if end repeat delete the last char of tTotalFiles return tTotalFiles end listFiles # Returns a list of files with their containing folders, using # recursion # if desired to descend into sub folders. function listFilesWithFolders pFolder, pRecurse local tTotalFiles, tCurrentFiles, tFolders set the directory to pFolder put filteredFiles() into tCurrentFiles if not pRecurse then return pFolder & return & "--" & return \ & tCurrentFiles if tCurrentFiles is not empty then put pFolder & return & "--" & return after tTotalFiles put tCurrentFiles & return & return after tTotalFiles end if put filteredFolders() into tFolders repeat for each line tFolder in tFolders put listFilesWithFolders((pFolder & slash & tFolder), \ pRecurse) into tCurrentFiles if tCurrentFiles is not empty then put tCurrentFiles & \ return after tTotalFiles end repeat delete the last char of tTotalFiles return tTotalFiles end listFilesWithFolders # Returns a list of files with the full paths function listFilesWithPaths pFolder, pRecurse local tTotalFiles, tCurrentFiles, tFolders set the directory to pFolder put filteredFilesWithPaths() into tCurrentFiles if not pRecurse then return tCurrentFiles if tCurrentFiles is not empty then put tCurrentFiles & \ return after tTotalFiles put filteredFolders() into tFolders repeat for each line tFolder in tFolders put listFilesWithPaths((pFolder & slash & tFolder), \ pRecurse) into tCurrentFiles if tCurrentFiles is not empty then put tCurrentFiles & \ return after tTotalFiles end repeat delete the last char of tTotalFiles return tTotalFiles end listFilesWithPaths # Returns a sorted list of the files in pFolder. Again using # recursion if # required. function listSortedFiles pFolder, pRecurse local tFiles put listFiles(pFolder, pRecurse) into tFiles sort lines of tFiles by listSortedFilesSortKey(each) return tFiles end listSortedFiles # Used by the listSortedFiles() function. Returns a sort key # from each file's name to # allow the sorting to be fully customized. function listSortedFilesSortKey pFile # Use this value for a normal text sort return pFile end listSortedFilesSortKey on doSomething pFile put pFile & return after msg end doSomething --This will achieve similar results to listing the files except that it will be slower because text has to be drawn to the screen at each iteration. # Use this template function to perform an action on each file # in a folder on doForEachFile pFolder, pRecurse set the directory to pFolder repeat for each line tFile in filteredFiles() doSomething (pFolder & slash & tFile) end repeat if pRecurse then repeat for each line tFolder in filteredFolders() doForEachFile (pFolder & slash & tFolder), pRecurse end repeat end if end doForEachFile I have directory available this days of Ken Ray, in the on system of set the default folder and get the the files, registered that path. descend recursively to the level… etc. Do anyone has a new one handy they could share using the folder and file function? BR ___ use-live
Re: Has Anyone Got A Directory "Walker" Available
> On 19 Apr 2018, at 11:20, Sannyasin Brahmanathaswami via use-livecode > mailto:use-livecode@lists.runrev.com>> wrote: > > Never mind… > > I found this in a toolbox. Sent by some ago, by a LiveCode deva That looks very much like the code supplied under the “Resources” menu, from Tutorials>General>Files and Folders. I’ve used it on a few projects n the past and it has worked pretty well for me. Paul ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Has Anyone Got A Directory "Walker" Available
You may recall the dreaded recursion errors that have cropped up in discussions of directory walkers. They happen not because anyone has directory trees > 40,000 folders deep, but because a permissions restriction can prevent going into a subdirectory, causing the current directory to be traversed again and again until the recursion error is thrown. The way to avoid this is to add an error check when setting the directory, e.g.: set the directory to pFolder if the result is not empty then -- skip or report as needed end if -- Richard Gaskin Fourth World Systems Sannyasin Brahmanathaswami wrote: > I found this in a toolbox. Sent by some ago, by a LiveCode deva > > For what it's worth : > > # Filters the strings "." and ".." from a list > function filterDots pList >local tList > >put pList into tList >filter tList without "." >filter tList without ".." > >return tList > end filterDots > # Returns a filtered list of files in the current directory > function filteredFiles >return filterDots(the files) > end filteredFiles > > # Returns a filtered list of folders in the current directory > function filteredFolders >return filterDots(the folders) > end filteredFolders > > # Returns a list of files in the current directory including > # each file's full path. > function filteredFilesWithPaths >local tFiles, tFilesWithPaths > >put filteredFiles() into tFiles >repeat for each line tFile in tFiles > put the directory & slash & tFile & return after \ > tFilesWithPaths >end repeat >delete the last char of tFilesWithPaths > >return tFilesWithPaths > end filteredFilesWithPaths > > > # Returns a list of files in a given folder, using recursion to > # include files in subfolders if desired. > function listFiles pFolder, pRecurse >local tTotalFiles, tCurrentFiles, tFolders > > set the directory to pFolder >put filteredFiles() into tCurrentFiles >if not pRecurse then return tCurrentFiles >if tCurrentFiles is not empty then > put tCurrentFiles & return after tTotalFiles >end if >put filteredFolders() into tFolders > > repeat for each line tFolder in tFolders > put listFiles((pFolder & slash & tFolder), pRecurse) into \ > tCurrentFiles > if tCurrentFiles is not empty then > put tCurrentFiles & return after tTotalFiles > end if >end repeat > delete the last char of tTotalFiles > >return tTotalFiles > end listFiles > > # Returns a list of files with their containing folders, using > # recursion > # if desired to descend into sub folders. > function listFilesWithFolders pFolder, pRecurse >local tTotalFiles, tCurrentFiles, tFolders > > set the directory to pFolder >put filteredFiles() into tCurrentFiles >if not pRecurse then return pFolder & return & "--" & return \ >& tCurrentFiles >if tCurrentFiles is not empty then > put pFolder & return & "--" & return after tTotalFiles > put tCurrentFiles & return & return after tTotalFiles >end if > put filteredFolders() into tFolders >repeat for each line tFolder in tFolders > put listFilesWithFolders((pFolder & slash & tFolder), \ > pRecurse) into tCurrentFiles > if tCurrentFiles is not empty then put tCurrentFiles & \ > return after tTotalFiles > end repeat >delete the last char of tTotalFiles > >return tTotalFiles > end listFilesWithFolders > > # Returns a list of files with the full paths > function listFilesWithPaths pFolder, pRecurse >local tTotalFiles, tCurrentFiles, tFolders > >set the directory to pFolder >put filteredFilesWithPaths() into tCurrentFiles >if not pRecurse then return tCurrentFiles > if tCurrentFiles is not empty then put tCurrentFiles & \ >return after tTotalFiles >put filteredFolders() into tFolders >repeat for each line tFolder in tFolders > put listFilesWithPaths((pFolder & slash & tFolder), \ > pRecurse) into tCurrentFiles > if tCurrentFiles is not empty then put tCurrentFiles & \ > return after tTotalFiles > end repeat >delete the last char of tTotalFiles > >return tTotalFiles > end listFilesWithPaths > > # Returns a sorted list of the files in pFolder. Again using > # recursion if > # required. > function listSortedFiles pFolder, pRecurse >local tFiles > >put listFiles(pFolder, pRecurse) into tFiles >sort lines of tFiles by listSortedFilesSortKey(each) > >return tFiles > end listSortedFiles > > # Used by the listSortedFiles() function. Returns a sort key > # from each file's name to > # allow the sorting to be fully customized. > function listSortedFilesSortKey pFile ># Use this value for a normal text sort >return pFile > end listSortedFilesSortKey > > > on doSomething pFile >put pFile & return after msg > end doSomething > > --This will achieve similar results to listing the files except that > it will be slower because text has to be drawn to the sc
Re: Has Anyone Got A Directory "Walker" Available
Now that we have files(pFolder), folder(pFolder) (and files(pFolder, "detailed"), folders(pFolder, "detailed")), directory walking code can be improved somewhat by not having to set the current folder. Actually, I haven't checked what happens when you get files(pRestrictedFolder) On Fri, Apr 20, 2018 at 7:49 PM Richard Gaskin via use-livecode < use-livecode@lists.runrev.com> wrote: > You may recall the dreaded recursion errors that have cropped up in > discussions of directory walkers. They happen not because anyone has > directory trees > 40,000 folders deep, but because a permissions > restriction can prevent going into a subdirectory, causing the current > directory to be traversed again and again until the recursion error is > thrown. > > The way to avoid this is to add an error check when setting the > directory, e.g.: > > set the directory to pFolder > if the result is not empty then > -- skip or report as needed > end if > > > -- > Richard Gaskin > Fourth World Systems > > > Sannyasin Brahmanathaswami wrote: > > > I found this in a toolbox. Sent by some ago, by a LiveCode deva > > > > For what it's worth : > > > > # Filters the strings "." and ".." from a list > > function filterDots pList > >local tList > > > >put pList into tList > >filter tList without "." > >filter tList without ".." > > > >return tList > > end filterDots > > # Returns a filtered list of files in the current directory > > function filteredFiles > >return filterDots(the files) > > end filteredFiles > > > > # Returns a filtered list of folders in the current directory > > function filteredFolders > >return filterDots(the folders) > > end filteredFolders > > > > # Returns a list of files in the current directory including > > # each file's full path. > > function filteredFilesWithPaths > >local tFiles, tFilesWithPaths > > > >put filteredFiles() into tFiles > >repeat for each line tFile in tFiles > > put the directory & slash & tFile & return after \ > > tFilesWithPaths > >end repeat > >delete the last char of tFilesWithPaths > > > >return tFilesWithPaths > > end filteredFilesWithPaths > > > > > > # Returns a list of files in a given folder, using recursion to > > # include files in subfolders if desired. > > function listFiles pFolder, pRecurse > >local tTotalFiles, tCurrentFiles, tFolders > > > > set the directory to pFolder > >put filteredFiles() into tCurrentFiles > >if not pRecurse then return tCurrentFiles > >if tCurrentFiles is not empty then > > put tCurrentFiles & return after tTotalFiles > >end if > >put filteredFolders() into tFolders > > > > repeat for each line tFolder in tFolders > > put listFiles((pFolder & slash & tFolder), pRecurse) into \ > > tCurrentFiles > > if tCurrentFiles is not empty then > > put tCurrentFiles & return after tTotalFiles > > end if > >end repeat > > delete the last char of tTotalFiles > > > >return tTotalFiles > > end listFiles > > > > # Returns a list of files with their containing folders, using > > # recursion > > # if desired to descend into sub folders. > > function listFilesWithFolders pFolder, pRecurse > >local tTotalFiles, tCurrentFiles, tFolders > > > > set the directory to pFolder > >put filteredFiles() into tCurrentFiles > >if not pRecurse then return pFolder & return & "--" & return \ > >& tCurrentFiles > >if tCurrentFiles is not empty then > > put pFolder & return & "--" & return after tTotalFiles > > put tCurrentFiles & return & return after tTotalFiles > >end if > > put filteredFolders() into tFolders > >repeat for each line tFolder in tFolders > > put listFilesWithFolders((pFolder & slash & tFolder), \ > > pRecurse) into tCurrentFiles > > if tCurrentFiles is not empty then put tCurrentFiles & \ > > return after tTotalFiles > > end repeat > >delete the last char of tTotalFiles > > > >return tTotalFiles > > end listFilesWithFolders > > > > # Returns a list of files with the full paths > > function listFilesWithPaths pFolder, pRecurse > >local tTotalFiles, tCurrentFiles, tFolders > > > >set the directory to pFolder > >put filteredFilesWithPaths() into tCurrentFiles > >if not pRecurse then return tCurrentFiles > > if tCurrentFiles is not empty then put tCurrentFiles & \ > >return after tTotalFiles > >put filteredFolders() into tFolders > >repeat for each line tFolder in tFolders > > put listFilesWithPaths((pFolder & slash & tFolder), \ > > pRecurse) into tCurrentFiles > > if tCurrentFiles is not empty then put tCurrentFiles & \ > > return after tTotalFiles > > end repeat > >delete the last char of tTotalFiles > > > >return tTotalFiles > > end listFilesWithPaths > > > > # Returns a sorted list of
Re: Has Anyone Got A Directory "Walker" Available
Ah, it will throw an error - so best to use try / catch around the function calls. On Sat, Apr 21, 2018 at 9:13 AM Ali Lloyd wrote: > Now that we have files(pFolder), folder(pFolder) (and files(pFolder, > "detailed"), folders(pFolder, "detailed")), directory walking code can be > improved somewhat by not having to set the current folder. > > Actually, I haven't checked what happens when you get > files(pRestrictedFolder) > > > On Fri, Apr 20, 2018 at 7:49 PM Richard Gaskin via use-livecode < > use-livecode@lists.runrev.com> wrote: > >> You may recall the dreaded recursion errors that have cropped up in >> discussions of directory walkers. They happen not because anyone has >> directory trees > 40,000 folders deep, but because a permissions >> restriction can prevent going into a subdirectory, causing the current >> directory to be traversed again and again until the recursion error is >> thrown. >> >> The way to avoid this is to add an error check when setting the >> directory, e.g.: >> >> set the directory to pFolder >> if the result is not empty then >> -- skip or report as needed >> end if >> >> >> -- >> Richard Gaskin >> Fourth World Systems >> >> >> Sannyasin Brahmanathaswami wrote: >> >> > I found this in a toolbox. Sent by some ago, by a LiveCode deva >> > >> > For what it's worth : >> > >> > # Filters the strings "." and ".." from a list >> > function filterDots pList >> >local tList >> > >> >put pList into tList >> >filter tList without "." >> >filter tList without ".." >> > >> >return tList >> > end filterDots >> > # Returns a filtered list of files in the current directory >> > function filteredFiles >> >return filterDots(the files) >> > end filteredFiles >> > >> > # Returns a filtered list of folders in the current directory >> > function filteredFolders >> >return filterDots(the folders) >> > end filteredFolders >> > >> > # Returns a list of files in the current directory including >> > # each file's full path. >> > function filteredFilesWithPaths >> >local tFiles, tFilesWithPaths >> > >> >put filteredFiles() into tFiles >> >repeat for each line tFile in tFiles >> > put the directory & slash & tFile & return after \ >> > tFilesWithPaths >> >end repeat >> >delete the last char of tFilesWithPaths >> > >> >return tFilesWithPaths >> > end filteredFilesWithPaths >> > >> > >> > # Returns a list of files in a given folder, using recursion to >> > # include files in subfolders if desired. >> > function listFiles pFolder, pRecurse >> >local tTotalFiles, tCurrentFiles, tFolders >> > >> > set the directory to pFolder >> >put filteredFiles() into tCurrentFiles >> >if not pRecurse then return tCurrentFiles >> >if tCurrentFiles is not empty then >> > put tCurrentFiles & return after tTotalFiles >> >end if >> >put filteredFolders() into tFolders >> > >> > repeat for each line tFolder in tFolders >> > put listFiles((pFolder & slash & tFolder), pRecurse) into \ >> > tCurrentFiles >> > if tCurrentFiles is not empty then >> > put tCurrentFiles & return after tTotalFiles >> > end if >> >end repeat >> > delete the last char of tTotalFiles >> > >> >return tTotalFiles >> > end listFiles >> > >> > # Returns a list of files with their containing folders, using >> > # recursion >> > # if desired to descend into sub folders. >> > function listFilesWithFolders pFolder, pRecurse >> >local tTotalFiles, tCurrentFiles, tFolders >> > >> > set the directory to pFolder >> >put filteredFiles() into tCurrentFiles >> >if not pRecurse then return pFolder & return & "--" & return \ >> >& tCurrentFiles >> >if tCurrentFiles is not empty then >> > put pFolder & return & "--" & return after tTotalFiles >> > put tCurrentFiles & return & return after tTotalFiles >> >end if >> > put filteredFolders() into tFolders >> >repeat for each line tFolder in tFolders >> > put listFilesWithFolders((pFolder & slash & tFolder), \ >> > pRecurse) into tCurrentFiles >> > if tCurrentFiles is not empty then put tCurrentFiles & \ >> > return after tTotalFiles >> > end repeat >> >delete the last char of tTotalFiles >> > >> >return tTotalFiles >> > end listFilesWithFolders >> > >> > # Returns a list of files with the full paths >> > function listFilesWithPaths pFolder, pRecurse >> >local tTotalFiles, tCurrentFiles, tFolders >> > >> >set the directory to pFolder >> >put filteredFilesWithPaths() into tCurrentFiles >> >if not pRecurse then return tCurrentFiles >> > if tCurrentFiles is not empty then put tCurrentFiles & \ >> >return after tTotalFiles >> >put filteredFolders() into tFolders >> >repeat for each line tFolder in tFolders >> > put listFilesWithPaths((pFolder & slash & tFolder), \ >> > pRecurse
Re: Has Anyone Got A Directory "Walker" Available
FWIW I don't use recursion for directory walking. My typical routine (typing directly into email, may contain errors) is something along the following lines: -- assumes tRootFolder is the path to start reviewing put return into tFoldersToDo repeat until tFoldersToDo = empty -- process another folder put line 1 of tFoldersToDo into tSubFolder delete line 1 of tFoldersToDo set the defaultFolder to (tRootFolder & tSubFolder) -- note sub-folders repeat for each line f in the folders if char 1 of f = "." then next repeat put tSubFolder & "/" & f & return after tFoldersToDo end repeat -- process files repeat for each line f in the files if char 1 of f = "." then next repeat -- do something with f -- or store the path, i.e. tSubFolder & "/" & f end repeat end repeat HTH, Ben On 21/04/2018 09:15, Ali Lloyd via use-livecode wrote: Ah, it will throw an error - so best to use try / catch around the function calls. On Sat, Apr 21, 2018 at 9:13 AM Ali Lloyd wrote: Now that we have files(pFolder), folder(pFolder) (and files(pFolder, "detailed"), folders(pFolder, "detailed")), directory walking code can be improved somewhat by not having to set the current folder. Actually, I haven't checked what happens when you get files(pRestrictedFolder) On Fri, Apr 20, 2018 at 7:49 PM Richard Gaskin via use-livecode < use-livecode@lists.runrev.com> wrote: You may recall the dreaded recursion errors that have cropped up in discussions of directory walkers. They happen not because anyone has directory trees > 40,000 folders deep, but because a permissions restriction can prevent going into a subdirectory, causing the current directory to be traversed again and again until the recursion error is thrown. The way to avoid this is to add an error check when setting the directory, e.g.: set the directory to pFolder if the result is not empty then -- skip or report as needed end if -- Richard Gaskin Fourth World Systems Sannyasin Brahmanathaswami wrote: > I found this in a toolbox. Sent by some ago, by a LiveCode deva > > For what it's worth : > > # Filters the strings "." and ".." from a list > function filterDots pList >local tList > >put pList into tList >filter tList without "." >filter tList without ".." > >return tList > end filterDots > # Returns a filtered list of files in the current directory > function filteredFiles >return filterDots(the files) > end filteredFiles > > # Returns a filtered list of folders in the current directory > function filteredFolders >return filterDots(the folders) > end filteredFolders > > # Returns a list of files in the current directory including > # each file's full path. > function filteredFilesWithPaths >local tFiles, tFilesWithPaths > >put filteredFiles() into tFiles >repeat for each line tFile in tFiles > put the directory & slash & tFile & return after \ > tFilesWithPaths >end repeat >delete the last char of tFilesWithPaths > >return tFilesWithPaths > end filteredFilesWithPaths > > > # Returns a list of files in a given folder, using recursion to > # include files in subfolders if desired. > function listFiles pFolder, pRecurse >local tTotalFiles, tCurrentFiles, tFolders > > set the directory to pFolder >put filteredFiles() into tCurrentFiles >if not pRecurse then return tCurrentFiles >if tCurrentFiles is not empty then > put tCurrentFiles & return after tTotalFiles >end if >put filteredFolders() into tFolders > > repeat for each line tFolder in tFolders > put listFiles((pFolder & slash & tFolder), pRecurse) into \ > tCurrentFiles > if tCurrentFiles is not empty then > put tCurrentFiles & return after tTotalFiles > end if >end repeat > delete the last char of tTotalFiles > >return tTotalFiles > end listFiles > > # Returns a list of files with their containing folders, using > # recursion > # if desired to descend into sub folders. > function listFilesWithFolders pFolder, pRecurse >local tTotalFiles, tCurrentFiles, tFolders > > set the directory to pFolder >put filteredFiles() into tCurrentFiles >if not pRecurse then return pFolder & return & "--" & return \ >& tCurrentFiles >if tCurrentFiles is not empty then > put pFolder & return & "--" & return after tTotalFiles > put tCurrentFiles & return & return after tTotalFiles >end if > put filteredFolders() into tFolders >repeat for each line tFolder in
Re: Has Anyone Got A Directory "Walker" Available
Thanks Ben - that's pretty similar to what I use. Avoiding recursion is (almost) always a good idea, if it can be done without jumping through hoops; for a file walker I timed it (a couple of years ago) at somewhere between 20 and 35% faster. And it's so much easier to add a good progress indicator callback in the non-recursive method. But I would really urge you to add the safety check that Richard recommended against protected-directories - otherwise you'll find yourself in an infinite loop. -- Alex. P.S. the other addition I have is a couple of parameters for "ignore-dot-folders and ignore-dot-files". I rarely want to see, for instance, .git folders, so in my library I default to ignoring directories beginning with a dot. On 03/05/2018 20:48, Ben Rubinstein via use-livecode wrote: FWIW I don't use recursion for directory walking. My typical routine (typing directly into email, may contain errors) is something along the following lines: -- assumes tRootFolder is the path to start reviewing put return into tFoldersToDo repeat until tFoldersToDo = empty -- process another folder put line 1 of tFoldersToDo into tSubFolder delete line 1 of tFoldersToDo set the defaultFolder to (tRootFolder & tSubFolder) -- note sub-folders repeat for each line f in the folders if char 1 of f = "." then next repeat put tSubFolder & "/" & f & return after tFoldersToDo end repeat -- process files repeat for each line f in the files if char 1 of f = "." then next repeat -- do something with f -- or store the path, i.e. tSubFolder & "/" & f end repeat end repeat HTH, Ben On 21/04/2018 09:15, Ali Lloyd via use-livecode wrote: Ah, it will throw an error - so best to use try / catch around the function calls. On Sat, Apr 21, 2018 at 9:13 AM Ali Lloyd wrote: Now that we have files(pFolder), folder(pFolder) (and files(pFolder, "detailed"), folders(pFolder, "detailed")), directory walking code can be improved somewhat by not having to set the current folder. Actually, I haven't checked what happens when you get files(pRestrictedFolder) On Fri, Apr 20, 2018 at 7:49 PM Richard Gaskin via use-livecode < use-livecode@lists.runrev.com> wrote: You may recall the dreaded recursion errors that have cropped up in discussions of directory walkers. They happen not because anyone has directory trees > 40,000 folders deep, but because a permissions restriction can prevent going into a subdirectory, causing the current directory to be traversed again and again until the recursion error is thrown. The way to avoid this is to add an error check when setting the directory, e.g.: set the directory to pFolder if the result is not empty then -- skip or report as needed end if -- Richard Gaskin Fourth World Systems Sannyasin Brahmanathaswami wrote: > I found this in a toolbox. Sent by some ago, by a LiveCode deva > > For what it's worth : > > # Filters the strings "." and ".." from a list > function filterDots pList > local tList > > put pList into tList > filter tList without "." > filter tList without ".." > > return tList > end filterDots > # Returns a filtered list of files in the current directory > function filteredFiles > return filterDots(the files) > end filteredFiles > > # Returns a filtered list of folders in the current directory > function filteredFolders > return filterDots(the folders) > end filteredFolders > > # Returns a list of files in the current directory including > # each file's full path. > function filteredFilesWithPaths > local tFiles, tFilesWithPaths > > put filteredFiles() into tFiles > repeat for each line tFile in tFiles > put the directory & slash & tFile & return after \ > tFilesWithPaths > end repeat > delete the last char of tFilesWithPaths > > return tFilesWithPaths > end filteredFilesWithPaths > > > # Returns a list of files in a given folder, using recursion to > # include files in subfolders if desired. > function listFiles pFolder, pRecurse > local tTotalFiles, tCurrentFiles, tFolders > > set the directory to pFolder > put filteredFiles() into tCurrentFiles > if not pRecurse then return tCurrentFiles > if tCurrentFiles is not empty then > put tCurrentFiles & return after tTotalFiles > end if > put filteredFolders() into tFolders > > repeat for each line tFolder in tFolders > put listFiles((pFolder & slash & tFolder), pRecurse) into \ > tCurrentFiles > if tCurrentFiles is not empty then > put tCurrentFiles & return after tTotalFiles > end if > end repeat > delete the last char of tTotalFiles > > return tTotalFiles > end listFiles > > # Returns a
Re: Has Anyone Got A Directory "Walker" Available
Alex Tweedly wrote: > Avoiding recursion is (almost) always a good idea, if it can be > done without jumping through hoops; for a file walker I timed it > (a couple of years ago) at somewhere between 20 and 35% faster. How does recursion impair performance so significantly? -- Richard Gaskin Fourth World Systems Software Design and Development for the Desktop, Mobile, and the Web ambassa...@fourthworld.comhttp://www.FourthWorld.com ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Has Anyone Got A Directory "Walker" Available
On 05/05/2018 01:29, Richard Gaskin via use-livecode wrote: How does recursion impair performance so significantly? In general, there's significant work involved in a function or handler call and return - you need to establish a new context for locals, copy or calculate, parameters, etc. My claim that recursion is slow relative to a serial- or loop-based version is language-independent (though there might be specific exceptions like LISP :-) Compiler writers spend considerable effort minimizing the overhead of function calls - but still recursion is common, and indeed recognizing "tail-recursion" and optimizing it away is worthwhile. I don't know enough (i.e. I know nothing) about LC's engine, so don't know specifically what might be involved for LC. But here's a minimal test case (code below) 1. 65 : serial 2. 288 : many function calls 3. 471 : same number of calls as 2, more paramters 4. 273 : same number of calls as 2, but recursive Note : result for 2 and 4 are the same - caused by the number of calls, not the fact that it's recursion. Note : 3 (same as 2 but with extra parameters) is slower. This does point the way towards possible improvements in recursive solutions (and specifically in the code I used in my earlier recursive directory walker function). I'll try those out and see if they make a difference. Anyway - here's the code that gave the above results: on mouseup local t1, t2 constant K = 1000 local x constant KX = 100 put the millisecs into t1 repeat K times put KX into x repeat if x = 0 then exit repeat subtract 1 from x end repeat end repeat put the millisecs into t2 put t2 - t1 & CR after msg put the millisecs into t1 repeat K times put KX into x repeat if x = 0 then exit repeat put myfunc(x) into x end repeat end repeat put the millisecs into t2 put t2 - t1 & CR after msg put the millisecs into t1 repeat K times put KX into x repeat if x = 0 then exit repeat put manyParameters(x, 1, 2, "333", "444") into x end repeat end repeat put the millisecs into t2 put t2 - t1 & CR after msg put the millisecs into t1 repeat K times put KX into x put recursive(x) into x end repeat put the millisecs into t2 put t2 - t1 & CR after msg end mouseup function myfunc p return p-1 end myfunc function manyParameters p, p1, p2, p3, p4 return p-1 end manyParameters function recursive p if p=0 then return 0 return recursive(p-1) end recursive -- Alex. ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Has Anyone Got A Directory "Walker" Available
Ah, that makes sense. It's not so much the recursion per se, but the more general advantage of inline processing vs handler calls. In the test case below you know how many levels deep you're going, which seems a good fit for inline loops. But how do we handle cases like directory traversal, where we don't know in advance how deep we'll need to go? It's at least comforting to see that while the overhead of handler calls is significant, it's mostly in relative comparison: If I read that correctly, there are 1000 iterations of 100 operations, for a total of 100k operations. If my coffee is working, that means an overhead of just 0.00208 ms per operation when called in a separate handler, no? (thinking 273ms total for separate handler - 65ms for inline, per 100k operations). With directory traversal, I'd guess the overhead of physically touching each inode would outweigh that manyfold. Still useful to keep in mind: when inline code can do the job clearly, it will be more efficient. -- Richard Gaskin Fourth World Systems Alex Tweedly wrote: On 05/05/2018 01:29, Richard Gaskin via use-livecode wrote: How does recursion impair performance so significantly? In general, there's significant work involved in a function or handler call and return - you need to establish a new context for locals, copy or calculate, parameters, etc. My claim that recursion is slow relative to a serial- or loop-based version is language-independent (though there might be specific exceptions like LISP :-) Compiler writers spend considerable effort minimizing the overhead of function calls - but still recursion is common, and indeed recognizing "tail-recursion" and optimizing it away is worthwhile. I don't know enough (i.e. I know nothing) about LC's engine, so don't know specifically what might be involved for LC. But here's a minimal test case (code below) 1. 65 : serial 2. 288 : many function calls 3. 471 : same number of calls as 2, more paramters 4. 273 : same number of calls as 2, but recursive Note : result for 2 and 4 are the same - caused by the number of calls, not the fact that it's recursion. Note : 3 (same as 2 but with extra parameters) is slower. This does point the way towards possible improvements in recursive solutions (and specifically in the code I used in my earlier recursive directory walker function). I'll try those out and see if they make a difference. Anyway - here's the code that gave the above results: on mouseup local t1, t2 constant K = 1000 local x constant KX = 100 put the millisecs into t1 repeat K times put KX into x repeat if x = 0 then exit repeat subtract 1 from x end repeat end repeat put the millisecs into t2 put t2 - t1 & CR after msg put the millisecs into t1 repeat K times put KX into x repeat if x = 0 then exit repeat put myfunc(x) into x end repeat end repeat put the millisecs into t2 put t2 - t1 & CR after msg put the millisecs into t1 repeat K times put KX into x repeat if x = 0 then exit repeat put manyParameters(x, 1, 2, "333", "444") into x end repeat end repeat put the millisecs into t2 put t2 - t1 & CR after msg put the millisecs into t1 repeat K times put KX into x put recursive(x) into x end repeat put the millisecs into t2 put t2 - t1 & CR after msg end mouseup function myfunc p return p-1 end myfunc function manyParameters p, p1, p2, p3, p4 return p-1 end manyParameters function recursive p if p=0 then return 0 return recursive(p-1) end recursive -- Alex. ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Has Anyone Got A Directory \"Walker\" Available
Alex, a combination of "private" and referenced variables (@) improves the speed of the calls somewhat. Kind regards Bernd -- on mouseup local t1, t2 constant K = 1000 local x constant KX = 100 put the millisecs into t1 repeat K times put KX into x repeat if x = 0 then exit repeat subtract 1 from x end repeat end repeat put the millisecs into t2 put t2 - t1 & CR after msg put the millisecs into t1 repeat K times put KX into x repeat if x = 0 then exit repeat put myfunc(x) into x end repeat end repeat put the millisecs into t2 put t2 - t1 & CR after msg put the millisecs into t1 local tA, tB, tC, tD put 1 into tA put 2 into tB put "333" into tC put "444" into tD repeat K times put KX into x repeat if x = 0 then exit repeat put manyParameters(x, tA, tB, tC, tD) into x end repeat end repeat put the millisecs into t2 put t2 - t1 & CR after msg put the millisecs into t1 repeat K times put KX into x recursive x put the result into x end repeat put the millisecs into t2 put t2 - t1 & CR after msg end mouseup private function myfunc @p return p-1 end myfunc private function manyParameters @p, @p1, @p2, @p3, @p4 return p-1 end manyParameters private command recursive @p if p=0 then return 0 subtract 1 from p recursive p end recursive -- ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Has Anyone Got A Directory \"Walker\" Available
Bernd Niggemann wrote: > a combination of "private" and referenced variables (@) improves the > speed of the calls somewhat. Which version of LC did you test with? I was under the impression that since LC switched to copy-on-write for all arguments we should no longer need to use "@" for performance, only for for logic. -- Richard Gaskin Fourth World Systems Software Design and Development for the Desktop, Mobile, and the Web ambassa...@fourthworld.comhttp://www.FourthWorld.com ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Has Anyone Got A Directory \"Walker\" Available
I wonder if shelling out to DIR on Windows and LS on Linux/Mac wouldn’t be the fastest option… Cheers, Malte ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Has Anyone Got A Directory \\\"Walker\\\" Available
>Which version of LC did you test with? >I was under the impression that since LC switched to copy-on-write for all >>arguments we should no longer need to use "@" for performance, only for for >logic. Richard, I tested using LC 9 GM, what kind of results do you get? Kind regards Bernd ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Has Anyone Got A Directory \"Walker\" Available
Malte Pfaff-Brill wrote: > I wonder if shelling out to DIR on Windows and LS on Linux/Mac > wouldn’t be the fastest option… Maybe, but there's overhead in setting up the shell session. Even if it's measurably faster, I would be surprised if it were noticeably faster. Anything that depends on touching the disk with each call is likely to be far more I/O-bound than CPU-bound. -- Richard Gaskin Fourth World Systems Software Design and Development for the Desktop, Mobile, and the Web ambassa...@fourthworld.comhttp://www.FourthWorld.com ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Has Anyone Got A Directory \\\"Walker\\\" Available
Bernd wrote: > Richard wrote: >> Which version of LC did you test with? >> I was under the impression that since LC switched to copy-on-write >> for all >arguments we should no longer need to use "@" for >> performance, only for for logic. > > > I tested using LC 9 GM, what kind of results do you get? Well, so much for copy-on-write. As written: 31 84 109 81 After removing "@": 33 95 177 93 Looks like I'll go back to the old habit of adding otherwise-unnecessary "@"s in time-sensitive handlers, being as careful as I used to have to be not to modify anything unless that's what I truly want. Did copy-on-write get changed in v9, or is the scope of its effects just more limited than I had understood it to be? -- Richard Gaskin Fourth World Systems Software Design and Development for the Desktop, Mobile, and the Web ambassa...@fourthworld.comhttp://www.FourthWorld.com ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Has Anyone Got A Directory \"Walker\" Available
there's actually a directory walker search one can use in a shell command on mac that is quite speedy... let me report back. sqb -- Stephen Barncard - Sebastopol Ca. USA - mixstream.org On Sun, May 6, 2018 at 2:29 PM, Richard Gaskin via use-livecode < use-livecode@lists.runrev.com> wrote: > Malte Pfaff-Brill wrote: > > > I wonder if shelling out to DIR on Windows and LS on Linux/Mac > > wouldn’t be the fastest option… > > Maybe, but there's overhead in setting up the shell session. > > Even if it's measurably faster, I would be surprised if it were noticeably > faster. > > Anything that depends on touching the disk with each call is likely to be > far more I/O-bound than CPU-bound. > > -- > Richard Gaskin > Fourth World Systems > Software Design and Development for the Desktop, Mobile, and the Web > > ambassa...@fourthworld.comhttp://www.FourthWorld.com > > ___ > use-livecode mailing list > use-livecode@lists.runrev.com > Please visit this url to subscribe, unsubscribe and manage your > subscription preferences: > http://lists.runrev.com/mailman/listinfo/use-livecode > ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Has Anyone Got A Directory "Walker" Available
On 06/05/2018 20:12, Richard Gaskin via use-livecode wrote: Ah, that makes sense. It's not so much the recursion per se, but the more general advantage of inline processing vs handler calls. Exactly. In the test case below you know how many levels deep you're going, which seems a good fit for inline loops. But how do we handle cases like directory traversal, where we don't know in advance how deep we'll need to go? In this test case I do know how many levels - but the code makes no use of that knowledge, it simply exits when we get down to 0. For an example of how to handle directory recursion, see Ben's earlier post in this thread. Or I've included my current version below - I return a 2-level array of folder / file / data ... rather than a file-per-line. It's at least comforting to see that while the overhead of handler calls is significant, it's mostly in relative comparison: If I read that correctly, there are 1000 iterations of 100 operations, for a total of 100k operations. If my coffee is working, that means an overhead of just 0.00208 ms per operation when called in a separate handler, no? (thinking 273ms total for separate handler - 65ms for inline, per 100k operations). Yep - though it gets higher with more parameters. My recursive 'walker' needed 4 parameters, so the overhead was higher. Note that if I was writing a recursive tree-walker now, I'd make it a private function, so I could keep the parameter checks only at the top (i.e. public) level, and recurse without repeating them. I might even move most of the parameters into script-local variables, since I know the code is interrupt-safe. With directory traversal, I'd guess the overhead of physically touching each inode would outweigh that manyfold. Yeah - though it's particularly hard to test adequately. When I was trying to benchmark it, I found that to get consistent results, I had to run the test 2 or 3 times and ignore the first, highly variable, run. Which basically means that I was seeing results with the disk-cache fully engaged, and therefore likely to underestimate the importance of the disk operations in any real context. Still useful to keep in mind: when inline code can do the job clearly, it will be more efficient. Below is my current non-recursive walker. NB - it goes to some trouble to allow you to pass in a relative path name, and maintain that in the result. That is the opposite of what is commonly done (i.e. using absolute path/file names); I did this to make it easier to compare multiple trees. If you want the traditional result, simply do put the defaultfolder into temp -- gives the absolute path getArrayOfFiles temp, tMyArray but if you want "my" twist, you'd do getArrayOfFiles ".", tMyArray -- Alex. # Produce an array of folders + files with the "full" relative paths command getArrayOfFiles pFolder, @pA, pIgnoreDotFolders, pIgnoreDotFiles local tFiles, tFolders local tThisFolder if pIgnoreDotFolders is empty then put TRUE into pIgnoreDotFolders end if if pIgnoreDotFiles is empty then put TRUE into pIgnoreDotFiles end if put the defaultfolder into tThisFolder local tAbsFolder, tFoldersLeft, tSub set the defaultfolder to pFolder put the defaultfolder into tAbsFolder -- changes relative into absolute put CR into tFoldersLeft repeat until tFoldersLeft is empty put line 1 of tFoldersLeft into tSub delete line 1 of tFoldersLeft set the defaultFolder to (tAbsFolder & tSub) if the result is not empty then -- skip or report as needed next repeat end if try put folders() into tFolders end try if pIgnoreDotFolders then filter tFolders without ".*" else filter tFolders without "." filter tFolders without ".." end if repeat for each line L in tFolders put tSub & "/" & L & CR after tFoldersLeft end repeat try put the detailed files into tFiles end try if pIgnoreDotFiles then filter tFiles without ".*" end if repeat for each line L in tFiles put item 2 to -1 of L into pA[pFolder & tSub][item 1 of L] end repeat end repeat end getArrayOfFiles ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Has Anyone Got A Directory \"Walker\" Available
I doubt if it would work out faster. A quick test of time ls -lR > null gives 166 ms vs. the 179 I get from the non-recursive tree walker - so by the time you gather that output, and re-format back to something more usable, I think you'd probably come out the same or slower. Alex. On 06/05/2018 21:46, Malte Pfaff-Brill via use-livecode wrote: I wonder if shelling out to DIR on Windows and LS on Linux/Mac wouldn’t be the fastest option… Cheers, Malte ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Has Anyone Got A Directory \\\"Walker\\\" Available
On 05/06/2018 02:42 PM, Richard Gaskin via use-livecode wrote: Did copy-on-write get changed in v9, or is the scope of its effects just more limited than I had understood it to be? I'm still at the point of not trusting copy-on-write yet, but I think you're misinterpreting your results. If I'm understanding the way in which LC has implemented copy-on-write, then whether or not you use references in parameters, the "return p-1" code will have to make a copy on the stack to return a value - you can't just return a reference. So all you're changing by removing the "@" is one level of copying. Of course, I may well be not quite understanding what's been implemented, in which case I welcome any correction. -- Mark Wieder ahsoftw...@gmail.com ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Has Anyone Got A Directory \\\"Walker\\\" Available
Mark Wieder wrote: > On 05/06/2018 02:42 PM, Richard Gaskin via use-livecode wrote: > >> Did copy-on-write get changed in v9, or is the scope of its effects >> just more limited than I had understood it to be? > > I'm still at the point of not trusting copy-on-write yet, but I think > you're misinterpreting your results. If I'm understanding the way in > which LC has implemented copy-on-write, then whether or not you use > references in parameters, the "return p-1" code will have to make a > copy on the stack to return a value - you can't just return a > reference. So all you're changing by removing the "@" is one level of > copying. That makes sense. Thanks. -- Richard Gaskin Fourth World Systems Software Design and Development for the Desktop, Mobile, and the Web ambassa...@fourthworld.comhttp://www.FourthWorld.com ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Has Anyone Got A Directory \\\"Walker\\\" Available
I think we may be in the territory of pre-mature optimization. There is going to be a bit more overhead creating a copy-on-write object than a pass-by-reference object, but the real gains (over the old way) are going to be seen with objects that are “large”. If you run a profiler and need to improve a section of code, then this would be something to have in the toolbox. It probably isn’t a good practice to use everywhere since then you have to be careful about changing the value of a parameter. It would be interesting to see that same piece of code run before the change. When did copy on write get introduced? On Mon, May 7, 2018 at 2:20 PM Richard Gaskin via use-livecode < use-livecode@lists.runrev.com> wrote: > Mark Wieder wrote: > > > On 05/06/2018 02:42 PM, Richard Gaskin via use-livecode wrote: > > > >> Did copy-on-write get changed in v9, or is the scope of its effects > >> just more limited than I had understood it to be? > > > > I'm still at the point of not trusting copy-on-write yet, but I think > > you're misinterpreting your results. If I'm understanding the way in > > which LC has implemented copy-on-write, then whether or not you use > > references in parameters, the "return p-1" code will have to make a > > copy on the stack to return a value - you can't just return a > > reference. So all you're changing by removing the "@" is one level of > > copying. > > That makes sense. Thanks. > > -- > Richard Gaskin > Fourth World Systems > Software Design and Development for the Desktop, Mobile, and the Web > > ambassa...@fourthworld.comhttp://www.FourthWorld.com > > ___ > use-livecode mailing list > use-livecode@lists.runrev.com > Please visit this url to subscribe, unsubscribe and manage your > subscription preferences: > http://lists.runrev.com/mailman/listinfo/use-livecode > ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode