Re: WITHOUT_GNU_[COMPAT|SUPPORT]
Em 11-10-2012 19:09, Ian Lepore escreveu: I want to build grep without the gnu regex library. The makefile for usr.bin/grep contains .if !defined(WITHOUT_GNU_COMPAT) And man src.conf documents WITHOUT_GNU_SUPPORT but doesn't mention WITHOUT_GNU_COMPAT. Is this a typo in the makefile, or an ommision from the src.conf manpage? That time when I added the WITHOUT_GNU_COMPAT knob I didn't make it global, just used it for testing grep. I didn't think it was of any use for users and I wasn't aware of the existence of WITHOUT_GNU_SUPPORT. If it seems useful, I can change grep to use this global flag instead of the custom knob and it will just be built without the gnu regex library if the knob is set. Gabor ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
[HEADSUP] Upcoming GNU sort removal
Hi, it has been more than 3 months ago that BSD sort became default in HEAD and no serious complaints have been raised against it since then so I plan to permanently remove GNU sort from head in the next days. If you have any objection, please raise it now. Gabor ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [HEADSUP] Upcoming GNU sort removal
Em 04-10-2012 16:50, Dennis Glatting escreveu: On Thu, 2012-10-04 at 12:53 +0200, Gabor Kovesdan wrote: Hi, it has been more than 3 months ago that BSD sort became default in HEAD and no serious complaints have been raised against it since then so I plan to permanently remove GNU sort from head in the next days. If you have any objection, please raise it now. Initially I had problems with multi TB files (--unique, five to ten files) but I haven't had to do that in two(?) months. I will be getting back to that project in a month or so. It challanges a system's resources. :) And did it go much better with base GNU sort? It's quite an extreme case... :) Multi GB is also rare not speaking about multi TB... Gabor ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
[HEADS-UP] BSD sort is the default sort in -CURRENT
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi Folks, as I announced before, the default sort in -CURRENT has been changed to BSD sort. Since the import, the reported minor bugs have been fixed and BSD sort has passed the portbuild test. If you encounter any problems or incompatibility with the old GNU version, please report. Gabor -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk/qooAACgkQkC3QTyNzprELpgCgo1hBQzT/Ns2pnWe3kjn06oIU ddgAn2NyYkRY7vvPK84ZQFkPG9afL0ib =iAEI -END PGP SIGNATURE- ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [HEADS-UP] BSD sort is the default sort in -CURRENT
On 2012.06.27. 10:34, Doug Barton wrote: Great, can you post the results somewhere? I understand what you're saying below that there are situations where worse performance may need explanation, but it would be helpful if we had the data to look at. If something is buggy than it is not comparable in terms of performance because usually those bugs are the exact problem of the better performance because of the negligence of some checks or aspects that were not well considered. And the project cannot grow if we lose users due to gratuitous differences in core utilities. There are more Linux users than FreeBSD users and they all use GNU sort. A great percent of FreeBSD users also manages Linux systems at work so they may also be using new GNU sort features. So in short, what people more usually expect is the behavior of the latest GNU version and POSIX-conformance, not an obsolete, buggy behavior. Losing users because something works better does not seem to be a real threat. But if this is a problem then we should stop fixing bugs and adding features at all since it may discourage someone from using FreeBSD. In the case of grep, there were a fairly large number of people who agreed that a BSD grep with orders of magnitude worse performance than the previous version was not something we, as a project, were willing to stomach. Sufficiently such that the default was switched back. These are relevant critics. But remember that it lived together with GNU grep so the switch back didn't have a great impact. It was slow but at least it worked. Recently the build is so regularly broken that it makes much more harm. With BSD sort, it is the same case, if there are problems that are hard to solve, we will switch back easily. Gabor ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [HEADS-UP] BSD sort is the default sort in -CURRENT
On 2012.06.27. 8:11, O. Hartmann wrote: ... so, can I delete the entry WITH_BSD_SORT=yes in /etc/src.conf then? Yes. BSD sort will still be the default. And if you want default GNU sort, you can add WITH_GNU_SORT=yes. Gabor ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: CURRENT: buildworld fails
On 2012.05.29. 5:44, 山谷崇史 wrote: I had same problem, but I resolved it. Maybe, your sort (/usr/bin/sort) is broken. cd /usr/src make update cd usr.bin/sort make obj depend all install Then, sort is changed. make cleandir cleandir make buildworld Could you please elaborate it a bit? From your mail it is not clear where did things break and what kind of error did you see and how you actually solved it. Thanks, Gabor ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [HEADS-UP] BSD sort coming to -CURRENT
On 2012.05.08. 17:51, Gabor Kovesdan wrote: Hi Folks, Oleg Moskalenko has been working very hard on BSD sort and by now we think it is compatible with the base version (and has even more features, the ideas mostly taken from the latest GNU sort) and it is efficient. I just updated the textproc/bsdsort port to the latest version so that people can test it and I plan to commit it to HEAD in some days, for now installed as bsdsort, leaving GNU sort the default version. If someone has any objection, please raise it now. Future plans are to switch the logic and make BSD sort default once it has undergone enough testing and finally drop GNU sort permanently if no problems appear. Note that the strcoll() and wcscoll() implementations of FreeBSD are incomplete so neither GNU sort nor the new BSD sort work 100% correctly with multi-byte locales but once the underlying functions are implemented, BSD sort will just work fine. As I announced before, today finally I've committed BSD sort to -CURRENT. By default, it will be bsdsort and GNU sort will be installed as sort, like before. If you can, please opt in for the default BSD sort by setting WITH_BSD_SORT, which will cause that BSD sort is installed as sort and GNU sort will be gnusort. If no bugs appear and portmgr can run an exp-run for default BSD sort, I'll switch the logic to a default BSD sort. Gabor ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [HEADS-UP] BSD sort coming to -CURRENT
On 2012.05.09. 1:21, Oleg Moskalenko wrote: Michael, The situation is actually more complex than I described in my email, but the good news is that we already have a fix that supports all older syntax forms. I tested it with ispell ports and it builds just fine. We will update the bsdsort port soon. I've just committed the update. Thanks for the quick fix Oleg and thanks Michael for the bug report. Gabor ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [HEADS-UP] BSD sort coming to -CURRENT
On 2012.05.09. 13:01, Hartmann, O. wrote: On 05/09/12 12:44, Gabor Kovesdan wrote: On 2012.05.09. 1:21, Oleg Moskalenko wrote: Michael, The situation is actually more complex than I described in my email, but the good news is that we already have a fix that supports all older syntax forms. I tested it with ispell ports and it builds just fine. We will update the bsdsort port soon. I've just committed the update. Thanks for the quick fix Oleg and thanks Michael for the bug report. Gabor Will those, who are with FBSD CURRENT, have a special knob set in /etc/src.conf to make the new stuff the default? Or will it just flow in and I do not have to take care of the change? In the first phase, BSD sort will be installed as bsdsort and GNU sort as sort by default and you will have a WITH_BSD_SORT knob, which changes to BSD sort installed as sort and GNU sort as gnusort. In second phase, this will be the default option and you will be able to set WITHOUT_BSD_SORT to restore GNU sort as default and in third phase GNU sort will go away and there will be no knob. This is the currently planned migration scenario. Gabor ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
[HEADS-UP] BSD sort coming to -CURRENT
Hi Folks, Oleg Moskalenko has been working very hard on BSD sort and by now we think it is compatible with the base version (and has even more features, the ideas mostly taken from the latest GNU sort) and it is efficient. I just updated the textproc/bsdsort port to the latest version so that people can test it and I plan to commit it to HEAD in some days, for now installed as bsdsort, leaving GNU sort the default version. If someone has any objection, please raise it now. Future plans are to switch the logic and make BSD sort default once it has undergone enough testing and finally drop GNU sort permanently if no problems appear. Note that the strcoll() and wcscoll() implementations of FreeBSD are incomplete so neither GNU sort nor the new BSD sort work 100% correctly with multi-byte locales but once the underlying functions are implemented, BSD sort will just work fine. Gabor ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: fast bcopy...
Em 03-05-2012 12:28, Steven Atreju escreveu: Yes, of course. Though i was kinda, even shocked, once i've seen this first: http://marc.info/?l=dragonfly-commitsm=132241713812022w=2 I also experimented a bit with some trivial libc functions when testing a change for memcpy (still in queue, will send it out for review once I have some more detailed benchamrks) and I also noticed that sometimes the trivial C version performed just like the assembly code. It is definitely something that needs some cleanup and I'm interested in working on it but cannot afford too much time at the moment. Gabor ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: CFT: new BSD-licensed sort available
On 2012.03.14. 19:01, Mark Felder wrote: Would it be appropriate to perhaps have a port option to OVERWRITE_BASE and then people could just install that port, build world and kernel... build a ton of ports. See if anything that might possibly use it breaks? Yes, I'm working on the update and it will have that option. Thanks for your comment. Gabor ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: CFT: new BSD-licensed sort available
On 2012.03.14. 22:10, Adrian Chadd wrote: So you could intall gnusort, bsdsort, and then some config file would determine which was used. 'sort' would then be a symlink to said magic program, that'd look at its argv[0], look at the contents of that file, and exec() the right one. I prefer simplicity. And GNU sort should go as soon as BSD sort is good enough to replace it. If you check the wiki, we have set a goal for 10.X, which is the GPL-free base system. I think it is possible and I hope we can achieve it. Gabor ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
CFT: new BSD-licensed sort available
Hi Folks, some time ago I started writing a BSDL sort variant from scratch since the OpenBSD version did not support multibyte locales and was hard to modify. The development was a bit stalled but recently, Oleg Moskalenko oleg.moskale...@citrix.com showed interest in continuing this version and he has made a very good job on this BSD sort variant. Now it is compatible with the base version of GNU sort but the performance in most cases (string sort and -n) is quite behind GNU sort (although with -g it is about *4 times* faster). Oleg is still working on optimizing the code and the long-term plan is to drop GNU sort once this variant is good enough to replace it. For now, it is only available in Ports Collection as textproc/bsdsort but if there is no objection or any serious bug report I plan to add it to base installed as bsdsort, being GNU sort still the default sort until it proves that we can safely drop GNU sort. If you are interested in this sort utility, could you please try the port and report us any issue that you experience? Thanks in advance, Gabor ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: WITH_ICONV/WITH_BSD_GREP: Defaults?
On 2012.01.05. 21:04, O. Hartmann wrote: In FreeBSD 9 and 10, src.conf could be populated with those two knobs WITH_ICONV and WITH_BSD_GREP For some testing purposes, I switched them both to enabled, so I could test ports and software against these. I didn't realize any serious issue with WITH_BSD_GREP, but I read, some time ago, that bsdggrep does have some speed issues. What is the actual status of bsdgrep in FreeBSD? Yes, BSD grep has significant performance problems. It is quite a complex issue because GNU grep uses some shortcuts to speed up pattern matching. I don't want to put that in BSD grep because that does not seem correct to me. Such code belongs to the regex library. Not just logically but practically, as well, so that other dependencies can also benefit from a faster pattern matching. However, it cannot be done with the POSIX-compliant API so it requires some extensions. From the last GSoC, there is an ongoing work on this but currently, I'm having exams at the university. I hope I can finish it after my exams. Another story seems to be with WITH_ICONV. I didn't realize problems until I had harsh faults compiling port lang/gcc46 which tend to fail in a Makefile when WITH_ICONV is enabled. But at that point the issue the first time occured, I hadn't deinstalled port converters/libiconv so I can not say whether those issues came due to confusions. But it seems to tend to confusions having remnants of WITH_iconv installation when disabling the build by deleting or commenting out WITH_ICONV. What is the status of WITH_ICONV? The ports seem still to rely on the port libiconv when being built. BSD iconv works quite well in some cases but there are some problems on my TODO list, so I preferred to leave it off by default until they are solved. Gabor ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: bsdgrep-20110912: -F/fgrep enbug
On 2011.09.19. 3:40, poyop...@puripuri.plala.or.jp wrote: Hi, I found another issue, this time in bsdgrep-20110912 in port. Thanks for using BSD grep and reporting bugs! I've checked and confirmed these issues. In my development branch I have already committed a partial fix. It will be fixed shortly in the development version and in ports and I'll also try to get it fixed in 9.0 before it is released. Gabor ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [bsdgrep] --exclude-dir doesn't work
Em 2011.08.09. 14:43, Test Rat escreveu: It seems fnmatch(3) args were accidentally swapped. Try $ bsdgrep -Fr --exclude-dir '*.svn*' grep_ usr.bin/grep | bsdgrep -c svn 72 Thanks! I'll commit this, too. Gabor ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: issues with bsdgrep and lang/go
Em 2011.08.10. 5:02, Jason Hellenthal escreveu: So has the maintainer of bsdgrep updated it yet so these things do not happen ?, if not (and apparently not) I don't understand why you are explicitly defaulting your gnugrep to a broken bsdgrep. This was a known problem a few months back and I for one have not seen updates on that yet. I don't remember of this bug but I may have missed it. Could you tell me where did you see it? I'm very thankful to Alexander for using the buggy BSD grep because he helped fixing a bug. Criticizing is the easiest option but probably not the best. Gabor Kovesdan ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: issues with bsdgrep and lang/go
Em 2011.08.10. 22:09, Alexander Best escreveu: well i'd like to help the author of bsdgrep to improve it. testing it and then going back to gnu grep, because bsdgrep still has bugs isn't going to help much. by using it i'd like to trip over these kind of bugs and report them. but you're right...the author hasn't really paid a lot of attention to bsdgrep, lately.:( i guess that's a problem when you have an open os, where everybody can do as much or as little as it pleases him.;) I do! In this year's Summer of Code I'm working on the regex code. A good and efficient regex support is important for a fast and clean BSD grep. The ultimate goal is to get rid of both GNU grep and libgnuregex. I'm sorry if I temporarily neglected some bug reports. Gabor ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: issues with bsdgrep and lang/go
Em 2011.08.09. 13:27, Test Rat escreveu: Try below workaround. It also makes empty GREP_COLOR behave like gnugrep(1). Thanks a lot, you were so quick that I haven't had a chance to look at it yet. I'll commit the patch soon. Also, thanks to Alexander for using BSD grep and reporting bugs. Gabor Kovesdan ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
[CFT] patch to replace the regex code
Hi Folks, you may know that in the Summer of Code programme I'm working on replacing the old regex code with TRE, which is a BSD-licensed implementation. It supports wide characters, is POSIX-compliant and has a good performance compared to most of the open source implementations. Actually, I got mixed results. With sed, in the cases that I tested, the performance was more or less the same and in some few cases, TRE finished in half of the time. On the other hand, with grep sometimes it was significantly slower than the current regex code but grep has always been a complicated case and it has its own regex code, it was just used for testing here. I'm still working on some optimizations but apart from grep, the current performance may already be satisfying for normal cases. This is one thing that I would ask the interested testers to focus on: whether habitual scripts you execute finish later or sooner. I've also checked the POSIX-compliance and I found some cases when TRE is more permissive than the current implementation but that should not be a problem. The patch that I provide know probably can have a cleanup in the contrib area but it's just an early patch purely for testing purposes, so please avoid nitpicking for now and only report performance and/or functional problems. There's a code slush now so there's plenty of time to arrange this if it proves ready to go to 10-CURRENT. Thanks for all of you, who take the effort to give it a try. The patch is here: http://kovesdan.org/patches/tre-20110724.diff Regards, Gabor Kovesdan ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [CFT] patch to replace the regex code
Forgive me if I'm patronizing, but is there any surprise that a POSIX NFA implementation is slower than grep's DFA? Oh, of course an NFA implementation will always be slightly slower but the memory footprint will also be smaller, which is a big advantage for embedded systems. But in this case, the basis of the comparison was not the GNU DFA implementation but the old libc-regex. At the moment, BSD grep in the base system uses the GNU regex engine but I built a local version for testing purposes that used libc-regex. Gabor ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: HEADSUP: BSD iconv coming to the base system with default off
Hi Alexander, for whatever historic reason I had WITH_ICONV in /etc/make.conf on my desktop, so I got to be your guinea pig without conscious consent for that role on my part. I did hit several issues so far: thanks for your valuable comments about iconv and I'm sorry that you had to try it out without explicit willingness. I didn't count with such special cases like this one. Unfortunately, for a short period I've run out again of free time but I'll work out the solution for the problems you described. I'd like to comment on only one specific one now, the modular architecture. As you might know this implementation comes from the Citrus Project and the original authors designed it in such a way for flexibility and easy extensibility. It is modular at various levels of abstraction. Maybe this modularity isn't really required in real life and reducing the overhead from module handling the performance may improve. But so far I haven't investigated on such changes because I wanted to concentrate on the functional part first, but in the future, I'll definitely consider changes on the modular architecture. Gabor ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
HEADSUP: BSD iconv coming to the base system with default off
Hi Folks, I've got some time again to keep working on iconv. The last time I posted a patch, there were problems with some ports but apart from this it proved to be quite mature, so I decided to commit it to the base system but the default setting will be disabled. It can be enabled by setting the WITH_ICONV knob but whoever does it will take into account that it may break some ports from being built. However, this change will help me with future work and will also help interested people in getting involved in the testing. The rather big changeset is coming soon. Regards, Gabor Kovesdan ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: BSD grep performance
Em 2010.08.19. 8:41, Doug Barton escreveu: The performance is now almost comparable to GNU grep. I think you're using a very liberal definition of comparable. Ok, comparable for the casual cases but not generally comparable. I think with this, BSD grep may remain default if no other serious issues come up. I'm not going to re-state my opinion here except to say it hasn't changed. Even if the performance were not an issue I think the bugs mentioned below combined with your 4-day absence should also have been considerations. However, in regards to this particular case I think it's pretty obvious that I'm either alone, or in a very non-vocal group; so c'est la vie. I think the 4-day absence was not such a big deal given that we are talking about -current, I just wanted to let you know that I would not be responsive because of absence not ignorance. Other people also felt in the same way; it's fine to allow a short unstable period in -current to get things arranged. However, from the standpoint of committer relations I think that first stating that you would change the default, then not doing so before all of the outstanding issues were resolved is not what I would consider a good model for others to follow. I've just changed it back. No, I didn't lie to you, I just wanted to reevaluate if the remaining difference in ther performance was acceptable. We improved it a lot since you let us know about this problem and there is possibility that further improvements are to happen soon. I thank you that you helped the process with valuable feedback but I think it could have been done in a less noisy way and arrogant way. You called the attention to the performance problem, which is fine and necessary but I'd have reacted in the same way if you hadn't CC'd core@ in the very first mail and hadn't included the Official request words in the subject. But I know that there are people, who feel them so important that they have to always call the attention in the noisy way (ego in the Buddhist meaning) and I also have some personal problems that I feel bothered by such type of behaviour. This is a sign for me that I have to practice more acceptance and patience in the life. Gabor ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Official request: Please make GNU grep the default
Em 2010.08.19. 23:42, b. f. escreveu: Gabor: One more thing to look into, in addition to the context problems, ndisgen breakage, and problems on certain file systems: At r211506, 'grep -wq' does not seem to work properly (in the very least, it is not the same as with GNU grep), and has broken the 'check-categories' target (and hence builds) of all ports with 'lisp' in CATEGORIES. Thanks, I added to my TODO list. P.S. I hope that you have recovered from your influenza, and are feeling better now. Oh, thanks, I'm fine now but I'm moving soon to another country, so will be busy for some time. But I've changed back the default to GNU grep, so now the fixes aren't so urgent. Gabor ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: why GNU grep is fast
Em 2010.08.21. 4:31, Mike Haertel escreveu: Hi Gabor, I am the original author of GNU grep. I am also a FreeBSD user, although I live on -stable (and older) and rarely pay attention to -current. However, while searching the -current mailing list for an unrelated reason, I stumbled across some flamage regarding BSD grep vs GNU grep performance. You may have noticed that discussion too... Anyway, just FYI, here's a quick summary of where GNU grep gets its speed. Hopefully you can carry these ideas over to BSD grep. Thanks, Mike, I'll check how I could adopt these ideas into BSD grep. I think your kind help was such a nice gesture and the open souce world should be based on such cooperation instead of meaningless competition. Gabor ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: why GNU grep is fast
Later on, he summarizes some of the existing implementations, including comments about the Plan 9 implementation and his own RE2, both of which efficiently handle international text (which seems to be a major concern of Gabor's). I believe Gabor is considering TRE for a good replacement regex library. Yes. Oniguruma is slow, Google RE2 only supports Perl and fgrep syntax but not standard regex and Plan 9 implementation iirc only supports fgrep syntax and Unicode but not wchar_t in general. The key comment in Mike's GNU grep notes is the one about not breaking into lines. That's simply double-scanning the input; instead, run the matcher over blocks of text and, when it finds a match, work backwards from the match to find the appropriate line beginning. This is efficient because most lines don't match. I do like the idea. So do I. BTW, the fastgrep portion of bsdgrep is my fault/contribution to do a faster search bypassing the regex library. :) It certainly was not written with any encodings in mind; it was purely ASCII. As I have not kept up with it, I do not know if anyone improved it or not. It has been made wchar-compliant. Gabor ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
What to learn from the BSD grep case [Was: why GNU grep is fast]
Hi all, there are some consequences that we can see from the grep case. Here I'd like to add a summary, which raises some questions. All comments are welcome. 1, When grep entered -CURRENT and bugs were found I immediately got kind bug reports and sharp criticism, as well. According to my understanding, -CURRENT is for development and it's fine to expose new pieces of work there but now I'm in doubt about that because of complaining people. On the other hand, an earlier version of BSD grep has been in the ports tree for a very long time and users reported some problems, which have been fixed but still, there is a lot of bugs there which haven't been reported that time. If users don't volunteer to test new pieces of code on a volunteer basis, somehow we have to make them test it, so I think committing BSD grep to -CURRENT was a good decision in the first round. 2, This issue also brought up some bottlenecks and potential optimization points (like memchr() and mmap), which other softwre may benefit from. This is another reason to let such pieces of work in. But unfortunately, this means that noone profiled another utilities because these bottlenecks remained undiscovered. Neither did I. It's a lesson that we have to learn from this particular case. 3, Because of point 2, we need more content to developers-handbook to help development with such ideas and best practices. It has been also raised on another list that our end-user documentation isn't that shiny and cool that it used to be and actually, developers-handbook has never been finished to be more or less complete. If someone looks at it, it looks like a sketch, not a book. I'll see if I can write a section on profiling. 4, We really need a good regex library. From the comments, it seems there's no such in the open source world. GNU libregex isn't efficient because GNU grep uses those workarounds that Mike kindly pointed out. Oniguruma was extremely slow when I checked it. PCRE supports Perl-style syntax with a POSIX-like API but not POSIX regex. Google RE2 is the same with additional egrep syntax but doesn't have support for standard POSIX regexes. Plan 9 regex only supports egrep syntax. It seems that TRE is the best choice. It is BSD-licensed, supports wchar and POSIX(ish) regexes and it is quite fast. I don't know the theoretical background of regex engines but I'm wondering if it's possible top provide an alternative API with byte-counted buffers and use the heuristical speedup with fixed string matching. As Mike pointed out the POSIX API is quite limiting because it works on NUL-terminated strings and not on byte-counted buffers, so we couldn't just do it with a POSIX-conformant library but it would be nice if we could implement it in such a library with an alternative interface. Gabor ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: bsdgrep does not work with tail -f | grep combination
Em 2010.08.18. 7:42, poyop...@puripuri.plala.or.jp escreveu: Hi Gabor and others, As Gabor committed r211364, bsdgrep now works nicely with tail -f. http://svn.freebsd.org/changeset/base/211364 Thank you very much. Acknowledgements also go to you and other users. Without quality feedback I may not have found myself all reported bugs. Gabor ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Official request: Please make GNU grep the default
Em 2010.08.13. 10:43, Doug Barton escreveu: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Gabor, I hope at this point it goes without saying that I have a lot of respect for the work you've done on BSD grep, and I've already told you that I think you're very courageous for taking the project on. I've been testing and evaluating it for some time now, and I think I've given it a fair trial. You've done a fairly good job of responding to bug reports, and I understand that the exposure BSD grep has received as the default in HEAD has been very valuable in exposing additional areas that need work. However, with all that in mind I am officially asking you to please change the default in HEAD to GNU grep. (Note, I am _not_ asking you to remove BSD grep from the tree, just to change the default.) My reason is simple, performance. [...] I've just committed a patch with the kind help of Dimitry Andric, which gives BSD grep a huge performance boost. The performance is now almost comparable to GNU grep. I think with this, BSD grep may remain default if no other serious issues come up. Please report if you notice something weird. I know about some minor issues, which aren't fixed yet. I'll be out for 4 days as of tomorrow but when I come back I'll take care of these: - Infinite loop when reading directory on ZFS/NFS filesystem - Problems with context grepping When reply, please remove core@ from CC, let's not bother them with this, I just wanted to let them know that I'm not neglecting this issue but if still demanded for a good reason, I'll switch back to default GNU grep. Gabor ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Removal of ICC (intel compiler) bits from mk
Em 2010.08.18. 19:37, Rui Paulo escreveu: On 18 Aug 2010, at 18:18, Garrett Cooper wrote: On Wed, Aug 18, 2010 at 9:23 AM, Rui Paulorpa...@gmail.com wrote: Hi, I've been chatting with the ICC ex-users and they seem to be ok with the removal of the ICC bits from share/mk and other places. The reason is that it doesn't work and no one has volunteered to fix it for many years. This seems to indicate that the interest in ICC is low. If there's anyone against this, speak now or forever be silent. :-) Later versions of icc are more gcc compliant aren't they? If so, wouldn't this also be a non-issue to remove the bits, or are there still some incompatibilities between gcc and icc that are worth noting? I really don't know how compatible is the latest icc because no one ever updated the ports version. This is actually a hint that no one really uses this anymore. IIRC, apart from the low interest, the problem was that because of ICC's license using ICC to test this mk stuff requires a commercial license because somehow it is considered a derivative work. It has also prevented us from providing better support. In 2006, I wanted to do some progress as part of my SoC project because that time there was more interest. Alexander (CC'd) may comment on this. I think he has a license for FreeBSD work but he is not allowed to give it out to a third party. Gabor ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Official request: Please make GNU grep the default
Em 2010.08.15. 21:49, Dimitry Andric escreveu: GNU grep Elapsed time: 57 seconds BSD grep (original) Elapsed time: 820 seconds (~14.4x slower than GNU grep) BSD grep (quickfixed) Elapsed time: 115 seconds (~2.0x slower than GNU grep) It proves that getting rid of the fgetc's is certainly worthwhile, and I have attached a more complete patch that: - Replaces the horrendously inefficient grep_fgetln() with mostly the same implementation as the libc fgetln() function. - Uses plain file descriptors instead of struct FILE, since the buffering is done manually anyway, and it makes it easier to support gzip and bzip2. - Let the bzip2 reader just read the file as plain data, when the initial magic number doesn't match, mimicking the behaviour of GNU grep. There is probably more room for optimization, but let's see if this survives a bunch of tests first. :) Thanks Dmitry, I've also started to work on a similar solution but you were extremely fast. :) I'm checking your patch just now and will tell my experiences if I see any regression. Gabor ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Official request: Please make GNU grep the default
Em 2010.08.16. 16:11, Dag-Erling Smørgrav escreveu: Sean C. Farleys...@freebsd.org writes: Dag-Erling Smørgravd...@des.no writes: Did you actually test your patch? It makes absolutely no measurable difference. Yes, I saw a reduction, I didn't... I also saw a reduction by 8-30% depending on the particular case. Gabor ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Official request: Please make GNU grep the default
Em 2010.08.15. 21:07, Dag-Erling Smørgrav escreveu: Ignore the first two lines (that's the profiling code itself). Note that the top five lines are all in stdio, and nothing else even shows up on the radar. I only included enough output to show where the regexp code ranks; the complete output is attached. I hate to suggest reinventing the wheel, but IMHO, this is clearly a case where it would pay to use hand-rolled buffered input routines instead of stdio. Thank you very much for the valuable tests, I've already started to refactor this part but it will take some time and still it doesn't garantize that the performance will be the same as GNU's with this change. It may need more investigation, so I'm also already testing my patch to change the default and we'll see how the performance improvement progresses. Gabor ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Official request: Please make GNU grep the default
Em 2010.08.13. 10:52, Roman Divacky escreveu: what about optimizing BSD grep instead? [... picking one mail from the many that suggest this ...] The problem is that optimization is not that trivial. I think the bottleneck is the regex library because: 1, BSD grep is so simple. There may be optimization opportunities and they may help but not that much. But if someone can check the code and make some suggestions, of course, I'll track those down and try to get more of it. 2, GNU grep uses internal optimizations to get that performance. I think it's a wrong approach because the regex library itself should be optimized instead to keep BSD grep clean and simple and to provide the same efficiency for all utilities that are linked to the regex library. Our libc-regex is definitely need to be replaced at some point in the future but that's a more complex item. See the following references: http://wiki.freebsd.org/BSDgrep http://wiki.freebsd.org/Regex If you can make suggestions to make BSD grep faster without touching the regex library please do it and if we can get a performance that is acceptable, we can reconsider leaving it the default if nobody objects. I'll check Sean's suggestions and make some measures how much does that help. Gabor ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Official request: Please make GNU grep the default
Em 2010.08.14. 17:53, Andrey Chernov escreveu: On Fri, Aug 13, 2010 at 01:43:16AM -0700, Doug Barton wrote: Gabor, I hope at this point it goes without saying that I have a lot of respect I am Nth on this. Although I do a lot of l10n work in the beautefull less bureocracy FreeBSD time and very appreciate what Gabor did, the problem is in other area: BSD grep simple not ready for wide testers circle and needs to be polished further. I talk not about it speed alone but about all its bugs revealed right after import. Let few ethusiast who have spare time for experiments run at at now. What we need is rock stable grep, and changing regex library for speed don't add anything here. From my point of view importing of the latest GNU grep instead would have more benefits. And we need to pend BSD grep with all its can of bugs for a while, until faster regex will be imported (disadvantages must be balanced with somewhat sweet). It is not my final verdict and we'll can return to it after all serious bugs will be eliminated. Right now we'll have f.e. numerous report agaist wrong greeping dirs which are unacceptable for any development system including even non-production one like -current. Yes, I'm sorry for my slow reaction, I got a flu some time ago and that prevented me from fixing the bugs earlier. I have several fixes in my working copy, which are being discussed with my mentor. Probably, today or tomorrow they will be committed. Gabor ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Official request: Please make GNU grep the default
Em 2010.08.13. 10:43, Doug Barton escreveu: My reason is simple, performance. While doing some portmaster work recently I was regression testing some changes I made to the --index* options and noticed that things were dramatically slower than the last time I tested those features. Thinking that I had made a programming mistake I dug into my code, and while the regexps that I was using could be tuned for slightly better performance the problem was not in my code. I then installed textproc/gnugrep to compare, and the differences were very dramatic using a highly pessimized test case (finding a match on the last line of INDEX). The script I used to test is at http://people.freebsd.org/~dougb/grep-time-trial.sh.txt and a typical result was: GNU grep Elapsed time: 2 seconds BSD grep Elapsed time: 47 seconds Ok, I'll take care of this soon, and make GNU grep default, again with a knob to build BSD grep. I agree with you that we cannot allow such a big performance drawback but I my measures only showed significant differences for very big searches and I didn't imagine that it could add up to such a big diference. I'm sorry for the bad decision I took making it default. Gabor ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Official request: Please make GNU grep the default
Em 2010.08.13. 13:09, Matthias Andree escreveu: Gabor Kovesdan wrote on 2010-08-13: Em 2010.08.13. 10:43, Doug Barton escreveu: My reason is simple, performance. While doing some portmaster work recently I was regression testing some changes I made to the --index* options and noticed that things were dramatically slower than the last time I tested those features. Thinking that I had made a programming mistake I dug into my code, and while the regexps that I was using could be tuned for slightly better performance the problem was not in my code. I then installed textproc/gnugrep to compare, and the differences were very dramatic using a highly pessimized test case (finding a match on the last line of INDEX). The script I used to test is at http://people.freebsd.org/~dougb/grep-time-trial.sh.txt and a typical result was: GNU grep Elapsed time: 2 seconds BSD grep Elapsed time: 47 seconds Ok, I'll take care of this soon, and make GNU grep default, again with a knob to build BSD grep. I agree with you that we cannot allow such a big performance drawback but I my measures only showed significant differences for very big searches and I didn't imagine that it could add up to such a big diference. I'm sorry for the bad decision I took making it default. Without knowing any of the details (I am not using 9-CURRENT), Gabor, I suggest that you check the documentation around Google's RE2 library (which is in C++); there are quite a few bits of information relating to (including worst-case) performance of regexp matchers, both directly in the re2 documentation, as well as indirect through links and references. Might be worth a read, together with profiling Doug's test case if he could tell you how to reproduce those. Thanks, Matthias. I haven't looked deeply at this but iirc it uses Perl-syntax. We need an efficient, wchar-aware, POSIX(ish) regex library with a good license and atm only TRE conforms to these criteria. Besides, we need GNU-style regex support, which will have to be added to TRE before we can replace our libc-regex. Gabor ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Official request: Please make GNU grep the default
Em 2010.08.13. 13:33, Anonymous escreveu: Doug Bartondo...@freebsd.org writes: [...] My reason is simple, performance. While doing some portmaster work recently I was regression testing some changes I made to the --index* options and noticed that things were dramatically slower than the last time I tested those features. Thinking that I had made a programming mistake I dug into my code, and while the regexps that I was using could be tuned for slightly better performance the problem was not in my code. I then installed textproc/gnugrep to compare, and the differences were very dramatic using a highly pessimized test case (finding a match on the last line of INDEX). The script I used to test is at http://people.freebsd.org/~dougb/grep-time-trial.sh.txt and a typical result was: GNU grep Elapsed time: 2 seconds BSD grep Elapsed time: 47 seconds Why not allow people to use grep(1) from ports in portmaster, e.g. by not overriding user-specified PATH? It would be a working solution but having seen the performance issue, it may also cause troubles elsewhere. Gabor ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [bsdgrep] -w option matches part of words (Was: Apache 2.2 port and missing modules on current.)
Em 2010.08.10. 19:45, Anonymous escreveu:
Seems like APACHE_MODULES is incorrectly populated.
$ make -V APACHE_MODULES BATCH= GREP=${LOCALBASE-/usr/local}/bin/grep |
fgrep cache
...cache disk_cache file_cache...
$ make -V APACHE_MODULES BATCH= | fgrep cache
...disk_cache file_cache...
I guess the failing line is below in bsd.apache.mk
${ECHO_CMD} ${WITHOUT_MODULES} | ${GREP} -wq $${module} 2 /dev/null || \
It can be reduced to
$ echo mem_cache | grep --color -w cache
I'm sorry for this issue, it didn't come up in the exp-run because it
didn't make the port actually fail. I have a fix in my queue, which I
sent to my mentor and I'll commit it soon.
Gabor
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: read() on dirs - grep
Em 2010.08.09. 21:34, Gennady Proskurin escreveu: I see misbehaviour of new bsd grep in freebsd on tmpfs when grepping dirs. For example (on tmpfs /tmp): mkdir /tmp/qwe grep something /tmp/qwe (grep hangs) Thank you for the report, I'm already aware of the issue and preparing a fix for it. Gabor ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: bsdgrep does not work with tail -f | grep combination
Em 2010.08.04. 20:06, Xin LI escreveu: I'm able to reproduce the GNU behavior on 9.0-CURRENT which is IMO right. I think we need to break at the line end to provide better interactivity (the current code seems to do it (buffer is not full !eof), while what we wanted is (buffer is not full !eof !eol). The attached patch should fix this but I have not yet thoroughly tested it due to job work. I think the patch may break binary detection. That buffer is not a general buffer but filled in only once with the first n bytes of the file to check if the file is binary. If you stop after the first line, only the first line will be used for binary checking. I'll look at this problem soon. Gabor ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: bsdgrep does not work with tail -f | grep combination
Em 2010.08.03. 19:25, poyop...@puripuri.plala.or.jp escreveu: Hi, It seems bsdgrep does not work when piped from tail -f. I'm running r210728. term0$ jot 10 /tmp/1 term0$ tail -f /tmp/1 | grep 0 [no output] otherterm$ jot 10 /tmp/1 [no output to term0] = with GNU grep: term0$ tail -f /tmp/1 | gnugrep 0 10 otherterm$ jot 10 /tmp/1 [on term0] 10 10 I've checked on 8.0 and GNU grep doesn't output anything either for me. If you use tail -f, you will enter more lines and end it with EOF, won't you? And then BSD grep will process the input and print out matches. I don't think it's bad behaviour in itself but if you can explain why you think it's bad I'm willing to change it. Thanks for your comment, Gabor ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: BSD grep fixes
Em 2010.07.27. 5:59, b. f. escreveu: Other important differences between bsdgrep and GNU grep: The --include option in bsdgrep does not have the same effect as the corresponding option in GNU grep -- in GNU grep, that option causes _only_ those files matching the file inclusion pattern to be searched. To obtain the same behavior in bsdgrep, one must issue something like --exclude '*' --include pattern. The effect of multiple overlapping --include and --exclude options is different in the two greps. It would be wise to add comments about precedence of such options in the bsdgrep manpage, as is done, for example, in bsdtar(1). Thanks for bringing this up, I'll take a look and try to implement the same behaviour. And I will also document the behaviour. Gabor ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: BSD grep fixes
Em 2010.07.28. 17:26, b. f. escreveu: I don't think that the current behavior of bsdgrep is necessarily bad -- in fact it seems to me to be simple and intuitive: nothing is excluded or included implicitly, and (I think) the last match wins, unlike in gnu grep. I mention it only because it is different, and you may want to consider how closely you want it to mimic the behavior of gnu grep for the sake of compatibility. If bsdgrep already passed a ports exp-run, and build/installworld now work without any problems, then it may be simpler to just retain the current behavior. Either way, though, the effect of overlapping --(in|ex)clude(-dir) options should be explicitly mentioned in the manpage. Thank you, by the way, for your efforts to provide us with good BSD-licensed tools. I hope that you can find the time and energy to continue with the iconv, regex, and other text-processing tool improvements. Ok, thanks, then I'll just document it. Actually, I also think this behaviour is more intuitive, that's why I implemented it in this way. As for iconv, it needs a portbuild run and in first phase ports will still use GNU libiconv but BSD iconv will make its way into base if it passes the exp-run. Gabor ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: BSD grep fixes
Em 2010.07.28. 17:48, Dag-Erling Smørgrav escreveu: Gabor Kovesdanga...@freebsd.org writes: b. f.bf1...@gmail.com writes: I don't think that the current behavior of bsdgrep is necessarily bad -- in fact it seems to me to be simple and intuitive: nothing is excluded or included implicitly, and (I think) the last match wins, unlike in gnu grep. Ok, thanks, then I'll just document it. Uh, no. GNU grep's behavior is consistent with just about everything else that has --include / --exclude options, e.g. tar and rsync. Please change BSD grep's behavior to match GNU grep's. Ok, thanks for mentioning this. I already have a patch to implement the same behaviour, I'll discuss it with my mentor and fix soon. Gabor ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: BSD grep fixes
`-l' option is not supposed to show the matching line at all, only filename. Thanks. Fix is ready and waiting for my mentor's approval. http://kovesdan.org/patches/grep-lL.diff Gabor ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [bsdgrep] outputs color sequences even when stdout is not tty
I think it should behave like ls(1) and gnu grep(1) and strip color sequences if stdout is not associated with terminal. Thanks for letting me know about this. The fix is being worked on just like for the -q/-l bug. Gabor ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [bsdgrep] grep -ql does not supress output
Em 2010.07.24. 6:19, Doug Barton escreveu: There are several places in portmaster where I use '[e]grep -ql foo' to signal existence of something without having to deal with the output of grep. In oldgrep this worked as advertised. In bsdgrep it doesn't. Furthermore, looking at the code it doesn't seem like it's a trivial fix since you seem to be conflating the idea of -q with don't output the matching lines instead of don't output anything which is what the old one did: case 'l': Lflag = false; lflag = qflag = true; break; Also, looking at the code it's not clear to me that the -q option has its previous behavior of halting processing for that file on the first match, but I've only given it a quick look. So, request number 1, fix it so that bsdgrep -ql doesn't output anything, and make sure that -q actually halts processing on the first match. Of course both this and the color issue will be fixed. Request number 2, think about whether or not introducing this as the default was the right course of action. I held my tongue on this when you committed it, but in the past when such things have been added they start life as an option, allowing those who choose to do so to regression test them. Once they've had a shakeout period THEN the switch is flipped to make them the default. I'm not at the point yet where I'm ready to ask for you to change this, but between the color thing and this issue, that ball has started to roll, in my mind at least. We'll see what happens with more testing. This change was thoroughly tested on pointyhat and by several interested people even by you. Actually, the compatibility for non-standard GNU regexes were added when you requested it after trying out with portmaster. Why didn't you tell me earlier about this bug then, as well? Gabor ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
periodic script in base system to run csup
Hi folks, Steven Kreuzer wrote a periodic script to run csup updates with periodic daily. I've reviewed it and added support for multiple supfiles. I'd like to commit this to head. http://kovesdan.org/files/600.csup Is there any objection? Gabor ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: periodic script in base system to run csup
Em 2010.07.16. 16:15, Jilles Tjoelker escreveu: Hmm, /usr/src/Makefile has an 'update' target that can run csup and other updating tools. Perhaps it should call that instead? It is not just for updating your src tree but your ports tree, doc tree or eventually your local CVS repo. That's why you can specify multiple supfiles. Gabor ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: periodic script in base system to run csup
Em 2010.07.16. 16:23, Alex Kozlov escreveu: On Fri, Jul 16, 2010 at 03:58:33PM +0200, Gabor Kovesdan wrote: Thousands pc simultaneously try to access cvsup servers? Sound like a ddos to me. Yes, this was the only concern and that's why I started this discussion. Btw, if You have time, can You please check conf/124641? Thanks. Ok, looks interesting, will take a look. Gabor ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [CFT] BSDL iconv in base system
Em 2010.07.06. 17:54, Anonymous escreveu: BTW, I think there is regression in iconv(1). It wasn't there in iconv_base_integrate2.diff. (gdb) r Starting program: /usr/bin/iconv During symbol reading, DW_AT_name missing from DW_TAG_base_type. During symbol reading, cannot get low and high bounds for subprogram DIE at 11006. During symbol reading, DW_AT_name missing from DW_TAG_base_type. Program received signal SIGSEGV, Segmentation fault. 0x00401281 in main (argc= During symbol reading, incomplete CFI data; unspecified registers (e.g., rax) at 0x401180. 0, argv=0x7ffefb98) at /a/blah/usr.bin/iconv/iconv.c:198 198 if ((strcmp(opt_f, ) == 0) (strcmp(opt_t, ) == 0)) (gdb) bt f #0 0x00401281 in main (argc=0, argv=0x7ffefb98) at /a/blah/usr.bin/iconv/iconv.c:198 fp =value optimized out opt_f = 0x0 opt_t = 0x0 ch =value optimized out i =value optimized out opt_c = false opt_s = false (gdb) p *argv $1 = 0x0 It works if I specify both `-t' and `-f'. And crashes when none specified or only one of them. Thanks, I've fixed this and the mtree problem, as well. I hope this one now works properly on amd64: http://kovesdan.org/patches/iconv-20100708.diff Gabor ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [CFT] BSDL iconv in base system
Em 2010.07.04. 17:58, Anonymous escreveu: Do you create /usr/lib32/i18n directory before installing into it? Oh, I'm sorry I just tested cross-building but not normal building on amd64. This patch seems to fix the issue, I've added the necessary directories to mtree: http://kovesdan.org/patches/iconv_64bit_fix.diff Gabor ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [CFT] BSDL iconv in base system
Em 2010.06.17. 23:21, Anonymous escreveu: If cross-compiling doesn't work, how did you build the former one that gave you that error? Here is my guess libiconv_modules compiles fine but installs both normal and lib32 objdir into /usr/lib when lib32 should use /usr/lib32. mkcsmapper/mkesdb are failing to install because they're treated as build-tools for host system and never compiled for target system. However, they're not included in lib32 target and so are not built for i386 arch during normal buildworld on amd64 host where host = target. Here's the new patch, which is supposed to fix the following issues: - Fixed build on amd64 and fixed cross-compiling - Fixed hang when linked to libthr - Fixed iconv() prototype as per POSIX - More GNU compatibility: or char means the current local encoding in use http://kovesdan.org/patches/iconv_current.diff I hope some of you can give it a try. Gabor ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [CFT] BSDL iconv in base system
Em 2010.06.17. 23:21, Anonymous escreveu: Gabor Kovesdanga...@freebsd.org writes: [...] $ make installworld TARGET=i386 DESTDIR=/b/bbb ... === usr.bin/mkcsmapper (install) install -s -o root -g wheel -m 555 mkcsmapper /b/bbb/usr/bin strip: /b/bbb/usr/bin/mkcsmapper: File format not recognized install: wait: No such file or directory *** Error code 70 If cross-compiling doesn't work, how did you build the former one that gave you that error? Here is my guess libiconv_modules compiles fine but installs both normal and lib32 objdir into /usr/lib when lib32 should use /usr/lib32. mkcsmapper/mkesdb are failing to install because they're treated as build-tools for host system and never compiled for target system. However, they're not included in lib32 target and so are not built for i386 arch during normal buildworld on amd64 host where host = target. Just a short update on this... The library part is fixed but I couldn't find out a fix for mk* tools yet. Gabor ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [CFT] BSDL iconv in base system
Does it respect lib32?
I don't know to much about the ia32 compatibility layer but I used
conventional FreeBSD Makefiles to build the stuff. I'll try to figure
out what's going wrong but unfortunately I don't have an amd64 test
machine to build.
$ iconv -f ascii
iconv: iconv_open(UTF-8, ascii): Invalid argument
/usr/lib/i18n/libiconv_std.so.4: unsupported file layoutzsh: exit 1
iconv -f ascii
$ file /usr/lib/i18n/libiconv_std.so.4
/usr/lib/i18n/libiconv_std.so.4: ELF 32-bit LSB shared object, Intel 80386,
version 1 (FreeBSD), dynamically linked, stripped
$ uname -vm
FreeBSD 9.0-CURRENT #0 r209191=1463b20-dirty: Tue Jun 15 04:59:40 UTC 2010
h...@raphael.local:/a/objdir/a/dirty_build/sys/PHOENIX amd64
BTW, I tried to crosscompile i386 and got error on installing
usr.bin/{mkesdb,mkcsmapper}.
$ make installworld TARGET=i386 DESTDIR=/b/bbb
...
=== usr.bin/mkcsmapper (install)
install -s -o root -g wheel -m 555 mkcsmapper /b/bbb/usr/bin
strip: /b/bbb/usr/bin/mkcsmapper: File format not recognized
install: wait: No such file or directory
*** Error code 70
If cross-compiling doesn't work, how did you build the former one that
gave you that error?
--
Gabor Kovesdan
FreeBSD Volunteer
EMAIL: ga...@freebsd.org .:|:. ga...@kovesdan.org
WEB: http://people.FreeBSD.org/~gabor .:|:. http://kovesdan.org
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [CFT] BSDL iconv in base system
ok. after installing world i get this when bootingt up: Starting powerd. Starting musicpd. path: invalid filesystem charset: ISO-8859-15 /usr/lib/i18n/libiconv_std.so.4: unsupported file layout/usr/lib/i18n/libiconv_std.so.4: unsupported file layout/usr/lib/i18n/libiconv_std.so.4: unsupported file layout/usr/lib/i18n/libiconv_std.so.4: unsupported file layout/usr/lib/i18n/libiconv_std.so.4: unsupported file layout pid 1078 (mpd), uid 1001: exited on signal 6 Abort trap /etc/rc: WARNING: failed to start musicpd Starting mpdscribble. GLib: Cannot convert message: Conversion from character set 'UTF-8' to 'ASCII' is not supported /usr/lib/i18n/libiconv_std.so.4: unsupported file layout/usr/lib/i18n/libiconv_std.so.4: unsupported file layout/usr/lib/i18n/libiconv_std.so.4: unsupported file layout/usr/lib/i18n/libiconv_std.so.4: unsupported file layoutoption parsing failed: Conversion from character set 'ASCII' to 'UTF-8' is not supported /etc/rc: WARNING: failed to start mpdscribble Seems like the very same error that was already reported.Was it just a normal build? I never got such a result but will look into the Makefiles to find out how ia32 compatibility works. -- Gabor Kovesdan FreeBSD Volunteer EMAIL: ga...@freebsd.org .:|:. ga...@kovesdan.org WEB: http://people.FreeBSD.org/~gabor .:|:. http://kovesdan.org ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [CFT] BSDL iconv in base system
El 2010. 06. 17. 23:21, Anonymous escribió: Gabor Kovesdanga...@freebsd.org writes: [...] $ make installworld TARGET=i386 DESTDIR=/b/bbb ... === usr.bin/mkcsmapper (install) install -s -o root -g wheel -m 555 mkcsmapper /b/bbb/usr/bin strip: /b/bbb/usr/bin/mkcsmapper: File format not recognized install: wait: No such file or directory *** Error code 70 If cross-compiling doesn't work, how did you build the former one that gave you that error? Here is my guess libiconv_modules compiles fine but installs both normal and lib32 objdir into /usr/lib when lib32 should use /usr/lib32. Oh, this seems like a relevant guess. I'll have to handle it in the Makefile then by checking if we are cross-compiling and I'll also have to add an #ifdef or something in the libc part to select the module path conditionally. mkcsmapper/mkesdb are failing to install because they're treated as build-tools for host system and never compiled for target system. However, they're not included in lib32 target and so are not built for i386 arch during normal buildworld on amd64 host where host = target. Yes, another good catch. Thanks a lot for your comments. -- Gabor Kovesdan FreeBSD Volunteer EMAIL: ga...@freebsd.org .:|:. ga...@kovesdan.org WEB: http://people.FreeBSD.org/~gabor .:|:. http://kovesdan.org ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [CFT] BSDL iconv in base system
iconv(3) prototype doesn't conform to POSIX.1-2008. Is it a well-considered decision? No, it was just like that in the Citrus version and I didn't notice the const qualifier. Fixed in my working copy, will be available soon with some minor modifications. Thanks for reporting this. -- Gabor Kovesdan FreeBSD Volunteer EMAIL: ga...@freebsd.org .:|:. ga...@kovesdan.org WEB: http://people.FreeBSD.org/~gabor .:|:. http://kovesdan.org ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [CFT] BSDL iconv in base system
Are there any plans to resurrect/finish multibyte collation support GSoC'2008 project: http://wiki.freebsd.org/KonradJankowski/Collation Yes, my queue is just so long that I haven't got there yet. I'm in SoC 2010 again with a different project and there's still BSD grep from SoC 2008. I'm also fixing the last nits of that. And there are also personal things, like a one-year internship in Portugal, which is going to start in September. But I hope once I'll find time or this. And are you aware of any plans on adding utf8-aware regex? I think NetBSD has already imported one: http://blog.netbsd.org/tnf/entry/efficient_wide_character_regular_expressions Yes, again but same issues. :) Besides, we need/should add a more relaxed regex support to TRE before we can adopt it. GNU regex allows things like [a|], which is not standard, so we should support them to maintain compatibility. This will be important for ports. This is also the reason why BSD grep is linked to GNU regex instead of libc-regex. -- Gabor Kovesdan FreeBSD Volunteer EMAIL: ga...@freebsd.org .:|:. ga...@kovesdan.org WEB: http://people.FreeBSD.org/~gabor .:|:. http://kovesdan.org ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [CFT] BSDL iconv in base system
/usr/src/lib/libc/iconv/citrus_none.c: In function '_citrus_NONE_stdenc_cstomb': /usr/src/lib/libc/iconv/citrus_none.c:114: warning: unused variable 'ret' *** Error code 1 I'm sorry, this one slipped in. Today I've checked the sources with clang and fixed some minor warnings, including this one. The new patch is here: http://www.kovesdan.org/patches/iconv_base_integrate2.diff And a gzipped version: http://www.kovesdan.org/patches/iconv_base_integrate2.diff.gz -- Gabor Kovesdan FreeBSD Volunteer EMAIL: ga...@freebsd.org .:|:. ga...@kovesdan.org WEB: http://people.FreeBSD.org/~gabor .:|:. http://kovesdan.org ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [CFT] BSDL iconv in base system
cc1: warnings being treated as errors /usr/src/lib/libiconv_modules/BIG5/../../libc/iconv/citrus_stdenc_template.h: In function '_citrus_BIG5_stdenc_cstomb': /usr/src/lib/libiconv_modules/BIG5/../../libc/iconv/citrus_stdenc_template.h:138: warning: 'ret' may be used uninitialized in this function Eeh. I'm sorry, I'll look at it tomorrow. For now you can use WERROR= and CWARNFLAGS= to test, actually it was what I used that's why I couldn't catch this. I had the same problem with some code that was not mine so I just set it. I don't know why I'm having that, tohugh, I also have that with a clean unpatched src tree and I'm not using any CFLAGS tweak or such. If you haven't done make clean yet, you can resume the build with: make buildworld -DNO_CLEAN WERROR= CWARNFLAGS= -- Gabor Kovesdan FreeBSD Volunteer EMAIL: ga...@freebsd.org .:|:. ga...@kovesdan.org WEB: http://people.FreeBSD.org/~gabor .:|:. http://kovesdan.org ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
[CFT] BSDL iconv in base system
Hello Folks, during the last summer, Google generously founded my Summer of Code project, which was providing a BSD-licensed iconv implementation for FreeBSD. I'm proud to announce that the work has been completed and a patch is available to add it to the base system. The results of this work are: - The Citrus implementation has been ported from NetBSD. - Some utilities have been added. There is a conversion table generator, which can compare conversion tables to reference data generated by GNU libiconv. This helps ensuring conversion compatibility. - UTF-16 surrogate support and some endianness issues have been fixed. - The rather chaotic Makefiles to build metadata have been refactored and cleaned up, now it is easy to read and it is also easier to add support for new encodings. - A bunch of new encodings and encoding aliases have been added. - Support for 1-2, 1-3 and 1-4 mappings, which is needed for transliterating with flying accents as GNU does, like u. - Lots of warnings have been fixed, the major part of the code is now WARNS=6 clean. - New section 1 and section 5 manual pages have been added. - Some GNU-specific calls have been implemented: iconvlist(), iconvctl(), iconv_canonicalize(), iconv_open_into() - Support for GNU's //IGNORE suffix has been added. - The - argument for stdin is now recognized in iconv(1) as per POSIX. - The Big5 conversion module has been fixed. - The iconv.h header files is supposed to be compatible with the GNU version, i.e. sources should build with base iconv.h and GNU libiconv. I've just did a very quick test and it seems ports can safely link to GNU libiconv, there's no conflict. - Various cleanups and style(9) fixes. - A bachelor thesis written in Hungarian language: http://www.kovesdan.org/files/bsc_iconv.pdf The rather big patch (42,5M) is available here: http://www.kovesdan.org/patches/iconv_base_integrate.diff Any comments, suggestions or bugreports are very welcome. -- Gabor Kovesdan FreeBSD Volunteer EMAIL:ga...@freebsd.org .:|:.ga...@kovesdan.org WEB:http://people.FreeBSD.org/~gabor .:|:.http://kovesdan.org ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: compiling FreeBSD to test a netbook
On Fri, 26 Feb 2010 16:21:06 -0800 Xin LI wrote: I've asked him to try a 9-CURRENT kernel but the Atom based netbook would take some while to compile the code =-) It's very easy/quick to (cross) compile world at a beafy machine and install the system to an USB stick. I also have an Atom-based netbook and it takes quite long but not that long. I usually compile kernel/world directly there. One battery charge (which make it last~4 hours) is enough for me to do a complete upgrade. -- Gabor Kovesdan FreeBSD Volunteer EMAIL: ga...@freebsd.org .:|:. ga...@kovesdan.org WEB: http://people.FreeBSD.org/~gabor .:|:. http://kovesdan.org ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
