Re: [Factor-talk] A bug ?

[Björn Lindqvist]

2015-10-01 8:47 GMT+02:00 HP Wei :
> Thanks for suggesting to look at the source of (directory-entries)
>
> I see that the iterator over a directory is the word: with-unix-directory
> and (directory-entries) uses produce to collect the entries into a sequence.
>
> I did not find a word in sequences that is similar to produce but does a
> ‘reduce’ action
> — sot that I could simply replace ‘produce’ in the definition of
> (directory-entries).

You can do it like this:

USING: accessors alien.strings classes.struct combinators
continuations io.backend io.directories.unix io.files.info kernel math
sequences unix.ffi ;
FROM: io.directories.unix.linux => next-dirent ;
IN: examples.sequences

DEFER: directory-size

: entry-size-file ( name -- size )
file-info size>> ;

: entry-size-dir ( name -- size )
dup { "." ".." } member? [ drop 0 ] [
normalize-path directory-size
] if ;

: entry-size ( dirent* -- size )
[ d_name>> alien>native-string ] [ d_type>> ] bi {
{ DT_REG [ entry-size-file ] }
{ DT_DIR [ entry-size-dir ] }
[ 2drop 0 ]
} case ;

: dirent-size ( unix-dir dirent -- size/f )
next-dirent [ entry-size ] [ drop f ] if ;

: (directory-size) ( unix-dir dirent -- total )
2dup dirent-size [ -rot (directory-size) + ] [ 2drop 0 ] if* ;

: directory-size ( path -- total )
[
[ dirent  (directory-size) ] with-unix-directory
] [ 2drop 0 ] recover ;

The code gets a little tricky because you didn't want to ever load a
full directory listing into memory.


-- 
mvh/best regards Björn Lindqvist

--
___
Factor-talk mailing list
Factor-talk@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/factor-talk

Re: [Factor-talk] A bug ?

[HP Wei]

Thanks for suggesting to look at the source of (directory-entries)

I see that the iterator over a directory is the word: with-unix-directory
and (directory-entries) uses produce to collect the entries into a sequence.

I did not find a word in sequences that is similar to produce but does a 
‘reduce’ action
— sot that I could simply replace ‘produce’ in the definition of 
(directory-entries).

So, my next thought will be to come up with a word — each-entry which emits an 
dirent
successively till the end.
Then below code can tally up the total file size more efficiently (memory-wise).

path [ 0 [ quot ] each-entry ] with-unix-directory

—hp


> On Sep 30, 2015, at 5:05 PM, John Benediktsson  wrote:
> 
> I mentioned before that it's not too hard to make an iterative using dirent, 
> especially if you just call it directly yourself. You can see how it works by 
> doing:
> 
> IN: scratchpad \ (directory-entries) see
> 
> Nothing technical prevents it, only that right now the iteration is hidden 
> behind that word where it produces a list of entries. 
> 
> In normal use, where a single directory doesn't have that many entries, this 
> is not a performance issue.  But, like anything with software, if you have a 
> different use case we can adapt the code to it. 
> 
> 
> On Sep 30, 2015, at 1:59 PM, HP wei  > wrote:
> 
>> I see.  That is how factor distinguishes stat and lstat :)  Thanks.
>> 
>> Now I can try out the process on a folder with many subfolders and 
>> with millions of files.  
>> [ I wish in factor, there is a facility to make an iterator type of object 
>> out of dirent. ]
>> 
>> --HP
>> 
>> 
>> On Wed, Sep 30, 2015 at 4:47 PM, Doug Coleman > > wrote:
>> You can do link-info instead.
>> 
>> 
>> On Wed, Sep 30, 2015, 13:42 HP wei > > wrote:
>> While trying out the word each-file,  I bumped into presumably
>> a bug in 
>> 
>> file-info ( path -- info )
>> 
>> Under linux,
>> if the path is a softlink (symbolic link),
>> 
>> path file-info symbolic-link?
>> 
>> gives 'f'    and this is wrong.
>> 
>> I looked at the implementation of file-info
>> and saw that it calls file-status which in turn calls
>> stat-func.
>> The latter calls __xstat64   this seems to be related to stat function.
>> 
>> Under linux, there are two functions that returns the stat-structure.
>> One is stat, the other lstat.
>> If a path is a symbolic link,
>> It is lstat that will return info about the link.
>> 
>> ---
>> 
>> As a result of this 'bug',  the following (as suggested by John the other 
>> day)
>> could not do what is intended (to get the size of a folder).
>> 
>> 0 a_path_to_folder t [ file-info dup symbolic-link? [ drop ] [ size>> + ] if 
>>  ] each-file 
>> 
>> 
>> --HP Wei
>> 
>> --
>> ___
>> Factor-talk mailing list
>> Factor-talk@lists.sourceforge.net 
>> https://lists.sourceforge.net/lists/listinfo/factor-talk 
>> 
>> 
>> --
>> 
>> ___
>> Factor-talk mailing list
>> Factor-talk@lists.sourceforge.net 
>> https://lists.sourceforge.net/lists/listinfo/factor-talk 
>> 
>> 
>> 
>> --
>> ___
>> Factor-talk mailing list
>> Factor-talk@lists.sourceforge.net 
>> https://lists.sourceforge.net/lists/listinfo/factor-talk 
>> 
> --
> ___
> Factor-talk mailing list
> Factor-talk@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/factor-talk

--
___
Factor-talk mailing list
Factor-talk@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/factor-talk

Re: [Factor-talk] A bug ?

[John Benediktsson]

I mentioned before that it's not too hard to make an iterative using dirent, 
especially if you just call it directly yourself. You can see how it works by 
doing:

IN: scratchpad \ (directory-entries) see

Nothing technical prevents it, only that right now the iteration is hidden 
behind that word where it produces a list of entries. 

In normal use, where a single directory doesn't have that many entries, this is 
not a performance issue.  But, like anything with software, if you have a 
different use case we can adapt the code to it. 


> On Sep 30, 2015, at 1:59 PM, HP wei  wrote:
> 
> I see.  That is how factor distinguishes stat and lstat :)  Thanks.
> 
> Now I can try out the process on a folder with many subfolders and 
> with millions of files.  
> [ I wish in factor, there is a facility to make an iterator type of object 
> out of dirent. ]
> 
> --HP
> 
> 
>> On Wed, Sep 30, 2015 at 4:47 PM, Doug Coleman  wrote:
>> You can do link-info instead.
>> 
>> 
>>> On Wed, Sep 30, 2015, 13:42 HP wei  wrote:
>>> While trying out the word each-file,  I bumped into presumably
>>> a bug in 
>>> 
>>> file-info ( path -- info )
>>> 
>>> Under linux,
>>> if the path is a softlink (symbolic link),
>>> 
>>> path file-info symbolic-link?
>>> 
>>> gives 'f'    and this is wrong.
>>> 
>>> I looked at the implementation of file-info
>>> and saw that it calls file-status which in turn calls
>>> stat-func.
>>> The latter calls __xstat64   this seems to be related to stat function.
>>> 
>>> Under linux, there are two functions that returns the stat-structure.
>>> One is stat, the other lstat.
>>> If a path is a symbolic link,
>>> It is lstat that will return info about the link.
>>> 
>>> ---
>>> 
>>> As a result of this 'bug',  the following (as suggested by John the other 
>>> day)
>>> could not do what is intended (to get the size of a folder).
>>> 
>>> 0 a_path_to_folder t [ file-info dup symbolic-link? [ drop ] [ size>> + ] 
>>> if  ] each-file 
>>> 
>>> 
>>> --HP Wei
>>> 
>>> --
>>> ___
>>> Factor-talk mailing list
>>> Factor-talk@lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/factor-talk
>> 
>> --
>> 
>> ___
>> Factor-talk mailing list
>> Factor-talk@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/factor-talk
> 
> --
> ___
> Factor-talk mailing list
> Factor-talk@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/factor-talk
--
___
Factor-talk mailing list
Factor-talk@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/factor-talk

Re: [Factor-talk] A bug ?

[HP wei]

I see.  That is how factor distinguishes stat and lstat :)  Thanks.

Now I can try out the process on a folder with many subfolders and
with millions of files.
[ I wish in factor, there is a facility to make an iterator type of object
out of dirent. ]

--HP


On Wed, Sep 30, 2015 at 4:47 PM, Doug Coleman 
wrote:

> You can do link-info instead.
>
> On Wed, Sep 30, 2015, 13:42 HP wei  wrote:
>
>> While trying out the word each-file,  I bumped into presumably
>> a bug in
>>
>> file-info ( path -- info )
>>
>> Under linux,
>> if the path is a softlink (symbolic link),
>>
>> path file-info symbolic-link?
>>
>> gives 'f'    and this is wrong.
>>
>> I looked at the implementation of file-info
>> and saw that it calls file-status which in turn calls
>> stat-func.
>> The latter calls __xstat64   this seems to be related to stat
>> function.
>>
>> Under linux, there are two functions that returns the stat-structure.
>> One is stat, the other lstat.
>> If a path is a symbolic link,
>> It is lstat that will return info about the link.
>>
>> ---
>>
>> As a result of this 'bug',  the following (as suggested by John the other
>> day)
>> could not do what is intended (to get the size of a folder).
>>
>> 0 a_path_to_folder t [ file-info dup symbolic-link? [ drop ] [ size>> + ]
>> if  ] each-file
>>
>>
>> --HP Wei
>>
>>
>> --
>> ___
>> Factor-talk mailing list
>> Factor-talk@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/factor-talk
>>
>
>
> --
>
> ___
> Factor-talk mailing list
> Factor-talk@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/factor-talk
>
>
--
___
Factor-talk mailing list
Factor-talk@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/factor-talk

Re: [Factor-talk] A bug ?

[Doug Coleman]

You can do link-info instead.

On Wed, Sep 30, 2015, 13:42 HP wei  wrote:

> While trying out the word each-file,  I bumped into presumably
> a bug in
>
> file-info ( path -- info )
>
> Under linux,
> if the path is a softlink (symbolic link),
>
> path file-info symbolic-link?
>
> gives 'f'    and this is wrong.
>
> I looked at the implementation of file-info
> and saw that it calls file-status which in turn calls
> stat-func.
> The latter calls __xstat64   this seems to be related to stat function.
>
> Under linux, there are two functions that returns the stat-structure.
> One is stat, the other lstat.
> If a path is a symbolic link,
> It is lstat that will return info about the link.
>
> ---
>
> As a result of this 'bug',  the following (as suggested by John the other
> day)
> could not do what is intended (to get the size of a folder).
>
> 0 a_path_to_folder t [ file-info dup symbolic-link? [ drop ] [ size>> + ]
> if  ] each-file
>
>
> --HP Wei
>
>
> --
> ___
> Factor-talk mailing list
> Factor-talk@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/factor-talk
>
--
___
Factor-talk mailing list
Factor-talk@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/factor-talk

[Factor-talk] A bug ?

[HP wei]

While trying out the word each-file,  I bumped into presumably
a bug in

file-info ( path -- info )

Under linux,
if the path is a softlink (symbolic link),

path file-info symbolic-link?

gives 'f'    and this is wrong.

I looked at the implementation of file-info
and saw that it calls file-status which in turn calls
stat-func.
The latter calls __xstat64   this seems to be related to stat function.

Under linux, there are two functions that returns the stat-structure.
One is stat, the other lstat.
If a path is a symbolic link,
It is lstat that will return info about the link.

---

As a result of this 'bug',  the following (as suggested by John the other
day)
could not do what is intended (to get the size of a folder).

0 a_path_to_folder t [ file-info dup symbolic-link? [ drop ] [ size>> + ]
if  ] each-file


--HP Wei
--
___
Factor-talk mailing list
Factor-talk@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/factor-talk