Re: [dev] Re: coreutils / moreutils - DC a directory counter
On 07/17/2013 09:02 PM, Calvin Morrison wrote: So it seems a good deal of that time is ls Wait, sbase ls doesn't seem to implement -f. Are you sorting the directory entries?
Re: [dev] Re: coreutils / moreutils - DC a directory counter
On 17 July 2013 16:58, Christian Neukirchen wrote: > Calvin Morrison writes: > >> On 17 July 2013 16:32, Christian Neukirchen wrote: >>> Calvin Morrison writes: >>> Hi guys, I came up with a utility[0] that i think could be useful, and I sent it to the moreutils page, but maybe it might fit better here. All it does is give a count of files in a directory. I was sick of ls | wc -l being so damned slow on large directories, so I thought a more direct solution would be better. calvin@ecoli:~/big_folder> time ls file2v1dir/ | wc -l 687560 real0m7.798s user0m7.317s sys 0m0.700s calvin@ecoli:~/big_folder> time ~/bin/dc file2v1dir/ 687560 real0m0.138s user0m0.057s sys 0m0.081s What do you think? Calvin >>> >>> What's the bottle neck here? >> >> Looking up the filenames and reading them, printing them to standard >> out and then wc parsing for all the \n characters. >> >>> (Or is your dc only faster because the directory index is in cache now...) >> >> No that's not why: >> >> calvin@ecoli:~/big_folder> ls 2v1 | wc -l >> 687560 >> >> real 0m7.678s >> user 0m7.313s >> sys 0m0.579s >> >> calvin@ecoli:~/big_folder> time dc 2v1 >> 687560 >> >> real 0m0.138s >> user 0m0.055s >> sys 0m0.082s >> >> calvin@ecoli:~/big_folder> time ls 2v1 | wc -l >> 687560 >> >> real 0m7.672s >> user 0m7.310s >> sys 0m0.580s > > How fast is find 2v1 -printf x | wc -c ? > > -- > Christian Neukirchenhttp://chneukirchen.org > > time find 2v1 -printf x | wc -c 687561 real 0m0.531s user 0m0.264s sys 0m0.271s time ls 2v1 > /dev/null real 0m7.642s user 0m7.265s sys 0m0.375s So it seems a good deal of that time is ls
[dev] Re: coreutils / moreutils - DC a directory counter
Calvin Morrison writes: > On 17 July 2013 16:32, Christian Neukirchen wrote: >> Calvin Morrison writes: >> >>> Hi guys, >>> >>> I came up with a utility[0] that i think could be useful, and I sent >>> it to the moreutils page, but maybe it might fit better here. All it >>> does is give a count of files in a directory. >>> >>> I was sick of ls | wc -l being so damned slow on large directories, so >>> I thought a more direct solution would be better. >>> >>> calvin@ecoli:~/big_folder> time ls file2v1dir/ | wc -l >>> 687560 >>> >>> real0m7.798s >>> user0m7.317s >>> sys 0m0.700s >>> >>> calvin@ecoli:~/big_folder> time ~/bin/dc file2v1dir/ >>> 687560 >>> >>> real0m0.138s >>> user0m0.057s >>> sys 0m0.081s >>> >>> What do you think? >>> Calvin >> >> What's the bottle neck here? > > Looking up the filenames and reading them, printing them to standard > out and then wc parsing for all the \n characters. > >> (Or is your dc only faster because the directory index is in cache now...) > > No that's not why: > > calvin@ecoli:~/big_folder> ls 2v1 | wc -l > 687560 > > real 0m7.678s > user 0m7.313s > sys 0m0.579s > > calvin@ecoli:~/big_folder> time dc 2v1 > 687560 > > real 0m0.138s > user 0m0.055s > sys 0m0.082s > > calvin@ecoli:~/big_folder> time ls 2v1 | wc -l > 687560 > > real 0m7.672s > user 0m7.310s > sys 0m0.580s How fast is find 2v1 -printf x | wc -c ? -- Christian Neukirchenhttp://chneukirchen.org
Re: [dev] Re: coreutils / moreutils - DC a directory counter
On 17 July 2013 16:32, Christian Neukirchen wrote: > Calvin Morrison writes: > >> Hi guys, >> >> I came up with a utility[0] that i think could be useful, and I sent >> it to the moreutils page, but maybe it might fit better here. All it >> does is give a count of files in a directory. >> >> I was sick of ls | wc -l being so damned slow on large directories, so >> I thought a more direct solution would be better. >> >> calvin@ecoli:~/big_folder> time ls file2v1dir/ | wc -l >> 687560 >> >> real0m7.798s >> user0m7.317s >> sys 0m0.700s >> >> calvin@ecoli:~/big_folder> time ~/bin/dc file2v1dir/ >> 687560 >> >> real0m0.138s >> user0m0.057s >> sys 0m0.081s >> >> What do you think? >> Calvin > > What's the bottle neck here? Looking up the filenames and reading them, printing them to standard out and then wc parsing for all the \n characters. > (Or is your dc only faster because the directory index is in cache now...) No that's not why: calvin@ecoli:~/big_folder> ls 2v1 | wc -l 687560 real 0m7.678s user 0m7.313s sys 0m0.579s calvin@ecoli:~/big_folder> time dc 2v1 687560 real 0m0.138s user 0m0.055s sys 0m0.082s calvin@ecoli:~/big_folder> time ls 2v1 | wc -l 687560 real 0m7.672s user 0m7.310s sys 0m0.580s > -- > Christian Neukirchenhttp://chneukirchen.org > >
[dev] Re: coreutils / moreutils - DC a directory counter
Calvin Morrison writes: > Hi guys, > > I came up with a utility[0] that i think could be useful, and I sent > it to the moreutils page, but maybe it might fit better here. All it > does is give a count of files in a directory. > > I was sick of ls | wc -l being so damned slow on large directories, so > I thought a more direct solution would be better. > > calvin@ecoli:~/big_folder> time ls file2v1dir/ | wc -l > 687560 > > real0m7.798s > user0m7.317s > sys 0m0.700s > > calvin@ecoli:~/big_folder> time ~/bin/dc file2v1dir/ > 687560 > > real0m0.138s > user0m0.057s > sys 0m0.081s > > What do you think? > Calvin What's the bottle neck here? (Or is your dc only faster because the directory index is in cache now...) -- Christian Neukirchenhttp://chneukirchen.org
Re: [dev] Truecolor ST
On Wed, Jul 17, 2013 at 08:44:28PM +0200, pancake wrote:
> I've been doing some ansi tests and found that only xterm (not urxvt, vte,
> iterm2,cmd.exe) supports truecolor ansi escape codes.
>
> It is documented in wikipedia, but afaik, only radare2 uses it (when
> enabled).
>
> Do you think that it should be implemented in st or we should stand
> supporting the most common scenarios?
>
> True RGB (24bit color) is constructed like this:
>
> \x1b[38;5;R;G;Bm
>
> Where 38 is fg (48 is bg)
> And RGB are decimal (1-3chars) from 0-255
Could you explain where did you find this information?, because
I only can see:
\x1b[38;5;color;m
where color is an index between 0 and 255. This is the function
implementeb by st:
case 38:
if(i + 2 < l && attr[i + 1] == 5) {
i += 2;
if(BETWEEN(attr[i], 0, 255)) {
term.c.attr.fg = attr[i];
} else {
fprintf(stderr,
"erresc: bad fgcolor %d\n",
attr[i]);
}
} else {
fprintf(stderr,
"erresc(38): gfx attr %d unknown\n",
attr[i]);
}
break;
Best regards,
--
Roberto E. Vargas Caballero
k...@shike2.com
http://www.shike2.com
Re: [dev] coreutils / moreutils - DC a directory counter
Quoth Chris Down: > On 17 July 2013 19:43, Calvin Morrison wrote: > > Could we focus on the merit of the utility? > > I cannot imagine any period in time where this would have been useful > for me over a simple `set -- * && echo "$#"', but whatever floats your > boat. Dependencies are much more costly than a small amount of time > saved (and dirs with >100k files are very niche). I basically agree, I don't deal with directories with enough files that ls|wc -l takes a noticable amount of time, and I doubt many people do. Still I could see it being useful sometimes, though. Chris, I've never used set like that, I shall have to look it up properly soon - thanks for sharing :)
Re: [dev] coreutils / moreutils - DC a directory counter
On 07/17/2013 06:13 PM, Calvin Morrison wrote: I rely heavily on unioned directories (go plan9!) so mine tend to get large when working with many large datasets. Does anyone use large directories often? Unices avoid large directories to scalability problems like the one you're trying to solve. Many programs and scripts are infrequently tested on large directories, in part because they're rare. Even the GNU coreutils test suite does not test against huge directories unless explicitly allowed to use tens of thousands of inodes. Which is at least once for every release candidate on every supported system. [OT] IIRC, an Ext4 explicitly does not care about ridiculously huge directories because common functions and syscalls will get mildly surprisingly slow anyway. But I haven't tested anything myself. If you can make such an interface that performance problems can be implemented under hood, that would be terrific. But a dedicated utility for such a specific function that it's hardly going to be used in scripts has no place in moreutils. It would be nice to keep around somewhere, though. Package it with other file tree related utilities.
Re: [dev] Truecolor ST
Hey, On Wed, Jul 17, 2013 at 08:44:28PM +0200, pancake wrote: > I've been doing some ansi tests and found that only xterm (not urxvt, vte, > iterm2,cmd.exe) supports truecolor ansi escape codes. > > It is documented in wikipedia, but afaik, only radare2 uses it (when > enabled). Do you have the patch already in the queue? If yes, please sent it here. > Do you think that it should be implemented in st or we should stand > supporting the most common scenarios? uhm. I don't use color. Would a libcolor make sense? Greetings --netbeisser
[dev] Truecolor ST
I've been doing some ansi tests and found that only xterm (not urxvt, vte, iterm2,cmd.exe) supports truecolor ansi escape codes. It is documented in wikipedia, but afaik, only radare2 uses it (when enabled). Do you think that it should be implemented in st or we should stand supporting the most common scenarios? True RGB (24bit color) is constructed like this: \x1b[38;5;R;G;Bm Where 38 is fg (48 is bg) And RGB are decimal (1-3chars) from 0-255
Re: [dev] coreutils / moreutils - DC a directory counter
On 17 July 2013 14:07, Roberto E. Vargas Caballero wrote: >> > Could we focus on the merit of the utility? >> >> I cannot imagine any period in time where this would have been useful >> for me over a simple `set -- * && echo "$#"', but whatever floats your >> boat. Dependencies are much more costly than a small amount of time >> saved (and dirs with >100k files are very niche). > > I agrre that this tool is useful only in corner cases, when the > number of files in the directory is huge, and usually the number > is very small, so it is not enough general for a coreutils package. Good point, maybe large directories are a bit of a corner case. I'm not sure. I rely heavily on unioned directories (go plan9!) so mine tend to get large when working with many large datasets. Does anyone use large directories often?
Re: [dev] coreutils / moreutils - DC a directory counter
On 17 July 2013 13:58, Chris Down wrote: > On 17 July 2013 19:43, Calvin Morrison wrote: >> The name still this has nothing to do with the utility of the >> statement. Please focus the conversation on that. > > If you are going to release things to mailing lists (especially this > one), you are going to have to stop acting so personally offended that > people bring up problems that are outside of the domain you were > expecting. >That your program shares the name as a standardised utility > is a perfectly legitimate concern; acting like a small child when > people try to help you out is not conducive to good relations with > others. Sure, but continuing to beat a dead horse is not conductive to 'good relations' either. > This is a mailing list, this isn't a "talk about what I want > to talk about" list. You are right, but does that mean this is a 'Chris Down Moderates this list with an iron fist list', anyone is free to reply to the emails, I was simply redirecting the discussion after a point had been agreed upon > There is very little that irritates me more than people who reject > feedback after explicitly asking for it. I actually asked for the feedback, accepted it, and then was ready for more! >> Could we focus on the merit of the utility? > > I cannot imagine any period in time where this would have been useful > for me over a simple `set -- * && echo "$#"', but whatever floats your > boat. Dependencies are much more costly than a small amount of time > saved (and dirs with >100k files are very niche). This set command is simple, but still takes a long time, because bash spends a long time doing the globbing of the * Calvin
Re: [dev] coreutils / moreutils - DC a directory counter
> > Could we focus on the merit of the utility? > > I cannot imagine any period in time where this would have been useful > for me over a simple `set -- * && echo "$#"', but whatever floats your > boat. Dependencies are much more costly than a small amount of time > saved (and dirs with >100k files are very niche). I agrre that this tool is useful only in corner cases, when the number of files in the directory is huge, and usually the number is very small, so it is not enough general for a coreutils package. -- Roberto E. Vargas Caballero k...@shike2.com http://www.shike2.com
Re: [dev] coreutils / moreutils - DC a directory counter
On 17 July 2013 19:43, Calvin Morrison wrote: > The name still this has nothing to do with the utility of the > statement. Please focus the conversation on that. If you are going to release things to mailing lists (especially this one), you are going to have to stop acting so personally offended that people bring up problems that are outside of the domain you were expecting. That your program shares the name as a standardised utility is a perfectly legitimate concern; acting like a small child when people try to help you out is not conducive to good relations with others. This is a mailing list, this isn't a "talk about what I want to talk about" list. There is very little that irritates me more than people who reject feedback after explicitly asking for it. > Could we focus on the merit of the utility? I cannot imagine any period in time where this would have been useful for me over a simple `set -- * && echo "$#"', but whatever floats your boat. Dependencies are much more costly than a small amount of time saved (and dirs with >100k files are very niche).
Re: [dev] coreutils / moreutils - DC a directory counter
Yesterday. On Jul 17, 2013 1:39 PM, "Calvin Morrison" wrote: > I know there is a naming conflict, what does that have to do with the > usage of the program? > > What was the last time you used the reverse polish notation calculator > that precedes the invention of C? > > Thank you, > > Calvin > > On 17 July 2013 13:36, wrote: > > dc - desk calculator > > > > http://man.cat-v.org/plan_9/1/dc > > http://man.cat-v.org/unix-1st/1/dc > > > > > >
Re: [dev] coreutils / moreutils - DC a directory counter
Thank you for your input Evan, I've decided that renaming the file would probably only take one command, and so I think another name might be possible to use. Could we focus on the merit of the utility? Thank you, Calvin On 17 July 2013 13:42, Evan Gates wrote: > Most of the time I have an HP-97 sitting on my desk. But when I don't, I use > dc. >
Re: [dev] coreutils / moreutils - DC a directory counter
Fine, I retract my statement. The name still this has nothing to do with the utility of the statement. Please focus the conversation on that. Calvin On 17 July 2013 13:40, Chris Down wrote: > On 17 July 2013 19:38, Calvin Morrison wrote: >> What was the last time you used the reverse polish notation calculator >> that precedes the invention of C? > > Are you serious? I use dc all the time. Just because you don't use it, > don't assume others don't. I find dc to be my calculator of choice. >
Re: [dev] coreutils / moreutils - DC a directory counter
Most of the time I have an HP-97 sitting on my desk. But when I don't, I use dc.
Re: [dev] coreutils / moreutils - DC a directory counter
On 17 July 2013 19:38, Calvin Morrison wrote: > What was the last time you used the reverse polish notation calculator > that precedes the invention of C? Are you serious? I use dc all the time. Just because you don't use it, don't assume others don't. I find dc to be my calculator of choice.
Re: [dev] coreutils / moreutils - DC a directory counter
I know there is a naming conflict, what does that have to do with the usage of the program? What was the last time you used the reverse polish notation calculator that precedes the invention of C? Thank you, Calvin On 17 July 2013 13:36, wrote: > dc - desk calculator > > http://man.cat-v.org/plan_9/1/dc > http://man.cat-v.org/unix-1st/1/dc > >
Re: [dev] coreutils / moreutils - DC a directory counter
dc - desk calculator http://man.cat-v.org/plan_9/1/dc http://man.cat-v.org/unix-1st/1/dc
[dev] coreutils / moreutils - DC a directory counter
Hi guys, I came up with a utility[0] that i think could be useful, and I sent it to the moreutils page, but maybe it might fit better here. All it does is give a count of files in a directory. I was sick of ls | wc -l being so damned slow on large directories, so I thought a more direct solution would be better. calvin@ecoli:~/big_folder> time ls file2v1dir/ | wc -l 687560 real0m7.798s user0m7.317s sys 0m0.700s calvin@ecoli:~/big_folder> time ~/bin/dc file2v1dir/ 687560 real0m0.138s user0m0.057s sys 0m0.081s What do you think? Calvin [0] https://github.com/mutantturkey/dc
Re: [dev] textwin
Well, there's gobject.io_add_watch, if you're ok with something that
sucks somewhat. Feel free to rewrite this in Tcl/Tk. Please, join #suckless.
import gtk,pygtk
import subprocess
import gobject
class CommandTextView(gtk.TextView):
def __init__(self,command):
super(CommandTextView,self).__init__()
self.command = command
def run(self):
proc = subprocess.Popen(self.command,shell=True,stdout=subprocess.PIPE)
gobject.io_add_watch(proc.stdout, gobject.IO_IN & IO_OUT, self.io)
def io(self, fd, condition):
if condition == gobject.IO_IN:
char = fd.read(1)
buf = self.get_buffer()
buf.insert_at_cursor(char)
return True
elif condition == gobject.IO_OUT:
fd.write(â¦
else:
return False
def write(self, fd, condition):
def test():
win=gtk.Window()
win.set_size_request(300,300)
win.connect('delete-event',lambda w,e : gtk.main_quit())
ctv=CommandTextView("ls -l")
win.add(ctv)
win.show_all()
ctv.run()
gtk.main()
if __name__=='__main__': test()
[dev] Re: [sbase] [patch] Adding tar v2
Nick writes: > On Wed, Jul 17, 2013 at 04:50:03PM +, Bjartur Thorlacius wrote: >> If you're just interacting with >> a shell, you should be using a simple I/O text window, with or >> without autocompletion. > > I would very much like this to exist, using non-monospaced fonts. It > wouldn't be hard to knock something up, but I've never got around to > it. Is there such a project already? There is 9term. A modern standalone of it would be very cool to have. -- Christian Neukirchenhttp://chneukirchen.org
Re: [dev] [sbase] [patch] Adding tar v2
On Wed, Jul 17, 2013 at 04:50:03PM +, Bjartur Thorlacius wrote: > If you're just interacting with > a shell, you should be using a simple I/O text window, with or > without autocompletion. I would very much like this to exist, using non-monospaced fonts. It wouldn't be hard to knock something up, but I've never got around to it. Is there such a project already?
Re: [dev] [sbase] [patch] Adding tar v2
On 07/17/2013 01:52 PM, Markus Wichmann wrote: I do partially. That is, I usually list the archive before unpacking, but I don't visually scan each and every entry, because, for one, I use st, so no scrollback buffer (I refuse to run a terminal multiplexer in an environment, were it is never going to see more than one terminal), You can paginate without multiplexing your terminal. Also, using a fixed-width character grid makes sense only for running TUI programs that directly manipulate said grid. If you're just interacting with a shell, you should be using a simple I/O text window, with or without autocompletion.
Re: [dev] [sbase] [patch] Adding tar v2
On 2013-07-17, at 15:52, Markus Wichmann wrote: > Speaking of which, is anyone up for some suckless binutils? Rob Landley has some vaporware here: http://landley.net/qcc/ -Truls
Re: [dev] [sbase] [patch] Adding tar v2
On Tue, Jul 16, 2013 at 08:58:49AM +0100, Nick wrote: > Quoth Chris Down: > > On 14 July 2013 20:42, Nick wrote: > > > I'd be inclined to check for and filter out leading .. and / > > > characters, to avoid tarballs doing unexpectedly evil things. > > > > I think all security onus for stuff like that should be on the user -- > > they can still do unexpectedly evil things either way (even stripping > > .. and /). It should be the user's responsibility to verify what will > > happen when a tarball is extracted using -t. > > What other evil things can tar creators do? > Create a tar that contains itself? > Going back to the workflow question, then, who here always checks > the list of all files in an archive to check that there's nothing > with a suspicious path? I know I don't, because I can trust gnu tar > to check for me, and that's a Good Thing. I do partially. That is, I usually list the archive before unpacking, but I don't visually scan each and every entry, because, for one, I use st, so no scrollback buffer (I refuse to run a terminal multiplexer in an environment, were it is never going to see more than one terminal), and the other is laziness. (I am going to assume that the tarball I regretfully had to download from the FSF's main FTP site actually contains what it says on the tin. Speaking of which, is anyone up for some suckless binutils?) Ciao, Markus
