Re: UFS Crash and directories now missing
On Thu, May 10, 2012 at 9:29 PM, Edward M eam1edw...@gmail.com wrote: On 05/10/2012 03:45 PM, Alejandro Imass wrote: Regarding Nemeth's I am undecided between the 4th (Unix Linux) or the 3rd. Please advise. i purchased the third edition because I took a look in the 4th the table of contents and it appears anything FreeBSD related was remove and it only focuses on: Solaris Linux( red hat ubuntu) and AIX. However third edition mentions BSDs Yep, agreed. 3rd edition it is. Thanks, -- Alejandro Imass ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
On Wed, May 9, 2012 at 9:48 AM, Polytropon free...@edvax.de wrote: On Wed, 9 May 2012 09:30:37 -0400, Alejandro Imass wrote: On Wed, May 9, 2012 at 8:53 AM, Erich Dollansky er...@alogreentechnologies.com wrote: For your recommendation above, what are the advantages or differences of slicing the disk versus partitioning on a single slice? it could be a misunderstanding. What is a partition? What is a slice. I have to look always into the handbook. Anyway, as long the OS see different units which have to be mounted independent of each other, it all does not matter what is what. I meant in Unix terms of course. Slice is slice (partition in other OS) and partition a thru h The question is if it has any advantage of using a slice to mount the basejail in RO as opposed to doing the same thing on a partition. The answer is: It it not possible. :-) You cannot mount a slice. Given the BSD terminology: A slice _has_ to contain partitions. You cannot format a slice, you can only format partitions. A formatted partition carries a UFS file system. (However, it's possible to omit the slice, and partition the whole disk instead, this is called dedicated mode). A third method is formatting the whole disk (the 'c' device), in that case the 'c' is omitted. The _only_ time you can mount a slice is when it is used in its common meaning, being a DOS primary partition; in this case, a FAT or NTFS file system will be placed directly into a slice, as those do not support any (BSD-style) partitioning. /dev/ad0 - the disk /dev/ad0s1 - 1st slice /dev/ad0s1a - 1st partition on 1st slice THIS is something you can mount. -or- /dev/ad0a - 1st partition on disk (dedicated) THIS can also be mounted. -or- /dev/ad0 - the whole disk (equals /dev/ad0c) Even THIS can be mounted. In case I'm misunderstanding your question, could you alter the expression? Thanks. The question was more advantages of a single slice + single partition versus a slice and multiple partitions, for mounting the EzJail basejail in RO mode. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
On Thu, May 10, 2012 at 06:29:05PM -0700, Edward M wrote: On 05/10/2012 03:45 PM, Alejandro Imass wrote: Regarding Nemeth's I am undecided between the 4th (Unix Linux) or the 3rd. Please advise. i purchased the third edition because I took a look in the 4th the table of contents and it appears anything FreeBSD related was remove and it only focuses on: Solaris Linux( red hat ubuntu) and AIX. However third edition mentions BSDs From the index of my copy of the third edition, I see these entries: 4.4BSD 2 . . . BSD (Berkeley UNIX) 2 . . . FreeBSD 4 From the index of my copy of the fourth edition, I see these entries: BSD Printing 1054-1065 see also printing architecture 1054-1055 configuration 1059-1065 lpc command 1057-1059 lpd daemon 1056 lpq command 1056-1057 lpr command 1056 lprm command 1057 printcap file 1059-1065 PRINTER environment variable 1054 BSD UNIX 8, 12, 1268-1273 . . . FreeBSD 8 . . . NetBSD 8 . . . OpenBSD 8 Page 8 of the fourth edition mentions various BSD Unix systems in the section Friction Between UNIX and Linux. Page 12's mention of BSD Unix in the fourth edition appears to correspond to page 3's un-indexed mention of FreeBSD in the third edition (specifically FreeBSD 3.4), in reference to the example Unix OSes they chose to use when discussing various OSes, though FreeBSD is not mentioned specifically in the fourth edition on that page and BSD Unix is largely referred to in a historical context. This appears to be a legitimate case of BSD Unix being phased out of part of the text as a relevant OS, but it is not a section that actually says anything of specific technical value. Pages 1268-1273 in the fourth edition correspond to the bulk of the section A Brief History of System Administration in the back of the book. The third edition's equivalent is the end of page 2 and a little over half of page 3, The Sordid History of UNIX. The fourth edition's index mentions jail, chroot which, when investigated in the text, has nothing at all to do with FreeBSD jails; it's just about chroot. The third edition also contains information about chroot, but does not mention it under the J section of the index. It looks to me like the fourth edition probably presents quite a bit more historical information particular to BSD Unix systems than the third edition, judging by the index. In the table of contents, I see that the third edition has a section set aside for BSD printing, despite lack of mention in the index. It looks like the table of contents section for BSD and AIX printing in the fourth edition (the first edition to include coverage of AIX, apparently) goes into a fair bit more detail about what's in the equivalent section. It looks to me, at a glance, like the fourth edition probably kept all of the BSD Unix related stuff from the third, probably updated slightly but not expanded outside of historical information. While a failure to expand technical information on BSD Unix systems would result in a reduction of the percentage of the book that covers BSD Unix technical matters, given the growth in size between third and fourth editions, the quantity of technical information about BSD Unix systems does not appear to have shrunk at all, from what I've seen. Of course, I might easily have overlooked something. Is there something else I should try to find in the index or table of contents that would be in the third edition but not the fourth? Can you give me some examples of the sorts of things you'd expect to find in the table of contents that is lacking in the fourth edition but present in the third? -- Chad Perrin [ original content licensed OWL: http://owl.apotheon.org ] ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
On 05/11/2012 10:47 AM, Chad Perrin wrote: Is there something else I should try to find in the index or table of contents that would be in the third edition but not the fourth? Can you give me some examples of the sorts of things you'd expect to find in the table of contents that is lacking in the fourth edition but present in the third? Hi,:-) So far I think I found a few that may make a difference. According to the table of contents in the 4th edition in the chapter called Booting and shuting down it only shows entries for: red hat, HP-UX, AIX, SUSE,Ubuntu. However in the third edition, show entries for FreeBSD's Booting and shuting down process. And another example is in the 4th edition the chapter called Adding new users, only mentions how to add users for: SUSE, Redhat Solaris HP-UX and AIX. However in the 3rd edition, explains how to add users on FreeBSD and how FreeBSD's master.passwd file, login.conf. work,etc The third edition's chapter called Drivers and the kernel shows how to build a freebsd kernel, create a BSD config file, tuning the freebsd kernel, add freebsd device drivers,etc. I was not able to find those entries in the the 4th editions Drivers and the kernel. chapter.the 3rd editions TCP/IP chapter shows network config for freebsd. However in the table of contents of the 4th edition does not. I'm searching for a website that contains the 3rd edition table of contents so one can compare between the two editions for better judgement. unfortunate, those were a few examples i have time to point out. I think may make a great difference. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
On 05/11/2012 12:11 PM, Edward M wrote: So far I think I found a few that may make a difference. According to the table of contents in the 4th edition in the chapter called Booting and shuting down it only shows entries for: red hat, HP-UX, AIX, SUSE,Ubuntu. However in the third edition, show entries for FreeBSD's Booting and shuting down process. And another example is in the 4th edition the chapter called Adding new users, only mentions how to add users for: SUSE, Redhat Solaris HP-UX and AIX. However in the 3rd edition, explains how to add users on FreeBSD and how FreeBSD's master.passwd file, login.conf. work,etc The third edition's chapter called Drivers and the kernel shows how to build a freebsd kernel, create a BSD config file, tuning the freebsd kernel, add freebsd device drivers,etc. I was not able to find those entries in the the 4th editions Drivers and the kernel. chapter.the 3rd editions TCP/IP chapter shows network config for freebsd. However in the table of contents of the 4th edition does not. I'm searching for a website that contains the 3rd edition table of contents so one can compare between the two editions for better judgement. unfortunate, those were a few examples i have time to point out. I think may make a great difference. I apologized, if my email came out looking strange with chopped up/ uneven sentences,etc. I have to check into that:-( ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
On Fri, May 11, 2012 at 12:11:48PM -0700, Edward M wrote: So far I think I found a few that may make a difference. According to the table of contents in the 4th edition in the chapter called Booting and shuting down it only shows entries for: red hat, HP-UX, AIX, SUSE,Ubuntu. However in the third edition, show entries for FreeBSD's Booting and shuting down process. And another example is in the 4th edition the chapter called Adding new users, only mentions how to add users for: SUSE, Redhat Solaris HP-UX and AIX. However in the 3rd edition, explains how to add users on FreeBSD and how FreeBSD's master.passwd file, login.conf. work,etc The third edition's chapter called Drivers and the kernel shows how to build a freebsd kernel, create a BSD config file, tuning the freebsd kernel, add freebsd device drivers,etc. I was not able to find those entries in the the 4th editions Drivers and the kernel. chapter.the 3rd editions TCP/IP chapter shows network config for freebsd. However in the table of contents of the 4th edition does not. I'm searching for a website that contains the 3rd edition table of contents so one can compare between the two editions for better judgement. unfortunate, those were a few examples i have time to point out. I think may make a great difference. Okay, thanks. You've provided a pretty good representative selection, I think. I guess there are two problems: the third edition index is woefully incomplete, and the fourth edition text has for some reason basically traded FreeBSD for AIX -- which makes little sense to me. I appreciate the time you put into this. -- Chad Perrin [ original content licensed OWL: http://owl.apotheon.org ] ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
On 05/11/2012 05:18 PM, Chad Perrin wrote: I appreciate the time you put into this. It was no problem at all:-) had fun comparing. Now that I'm free and have more time I went over the 3rd edition table of contents and found a few instances that mentions FreeBSD. In chapter Adding a Disk describes the FFS, shows a freebsd fstab example file and teaches how to add a disk in FreeBSD,etc. I Continued glancing at the contents and it appears the rest of the book is pretty much on subjects that apply to all UNIX OS. the fourth edition text has for some reason basically traded FreeBSD for AIX -- which makes little sense to me. I found a site that it kinda shows that this is was happened, AIX replaced FreeBSD:-( mid way through the site shows the 4th edition only focuses on redhat, opensuse, rhel, solaris, HPUX and IBM AIX. http://www.admin.com/ I ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
On Tue, May 1, 2012 at 1:58 AM, Robert Bonomi bon...@mail.r-bonomi.com wrote: [...] Reading _both_ of McKusick's Design of .. books, and the 'Unix System Admininstration Handbook', by Nemeth, et al. is a good _start_. I just bought the FreeBSD one only unless there is a reason I should read the older 4.4BSD ? Regarding Nemeth's I am undecided between the 4th (Unix Linux) or the 3rd. Please advise. Thanks, -- Alejandro Imass Having a bunch of the books from O'Reilley Assoc. (http://www.ora.com), especially for 'standard' tools that you need to get the most out of, is also highly recommended. Disclaimer: I know a lot of the authors of those books, persoally. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
On 05/10/2012 03:45 PM, Alejandro Imass wrote: Regarding Nemeth's I am undecided between the 4th (Unix Linux) or the 3rd. Please advise. i purchased the third edition because I took a look in the 4th the table of contents and it appears anything FreeBSD related was remove and it only focuses on: Solaris Linux( red hat ubuntu) and AIX. However third edition mentions BSDs table of contents of 4th edition. http://www.amazon.com/Linux-System-Administration-Handbook-Edition/dp/0131480057/ref=sr_1_1?s=booksie=UTF8qid=1336698969; sr=1-1#reader_0131480057 ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
On Thu, May 3, 2012 at 1:14 PM, Alejandro Imass a...@p2ee.org wrote: On Thu, May 3, 2012 at 9:35 AM, Robert Bonomi bon...@mail.r-bonomi.com wrote: [...] One comment: for 'defensive' purposes it would be useful to break ad6 up into two slices, putting 'basejail' in it's own slice. Then, for production use, that slice can be mounted RO, and with the 'system immutable' flag set on everything in that filesystem. Yes. From one of your posts that became somewhat clear to me: Having all the jails on a single 150GB slice seems like a bad idea. For your recommendation above, what are the advantages or differences of slicing the disk versus partitioning on a single slice? Thanks, -- Alejandro Imass ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
Hi, On Wednesday 09 May 2012 18:57:06 Alejandro Imass wrote: On Thu, May 3, 2012 at 1:14 PM, Alejandro Imass a...@p2ee.org wrote: On Thu, May 3, 2012 at 9:35 AM, Robert Bonomi bon...@mail.r-bonomi.com wrote: [...] One comment: for 'defensive' purposes it would be useful to break ad6 up into two slices, putting 'basejail' in it's own slice. Then, for production use, that slice can be mounted RO, and with the 'system immutable' flag set on everything in that filesystem. Yes. From one of your posts that became somewhat clear to me: Having all the jails on a single 150GB slice seems like a bad idea. For your recommendation above, what are the advantages or differences of slicing the disk versus partitioning on a single slice? it could be a misunderstanding. What is a partition? What is a slice. I have to look always into the handbook. Anyway, as long the OS see different units which have to be mounted independent of each other, it all does not matter what is what. Erich ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
On Wed, May 9, 2012 at 8:53 AM, Erich Dollansky er...@alogreentechnologies.com wrote: Hi, On Wednesday 09 May 2012 18:57:06 Alejandro Imass wrote: On Thu, May 3, 2012 at 1:14 PM, Alejandro Imass a...@p2ee.org wrote: On Thu, May 3, 2012 at 9:35 AM, Robert Bonomi bon...@mail.r-bonomi.com wrote: [...] One comment: for 'defensive' purposes it would be useful to break ad6 up into two slices, putting 'basejail' in it's own slice. Then, for production use, that slice can be mounted RO, and with the 'system immutable' flag set on everything in that filesystem. Yes. From one of your posts that became somewhat clear to me: Having all the jails on a single 150GB slice seems like a bad idea. For your recommendation above, what are the advantages or differences of slicing the disk versus partitioning on a single slice? it could be a misunderstanding. What is a partition? What is a slice. I have to look always into the handbook. Anyway, as long the OS see different units which have to be mounted independent of each other, it all does not matter what is what. I meant in Unix terms of course. Slice is slice (partition in other OS) and partition a thru h The question is if it has any advantage of using a slice to mount the basejail in RO as opposed to doing the same thing on a partition. Thanks, -- Alejandro Imass ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
On Wed, 9 May 2012 09:30:37 -0400, Alejandro Imass wrote: On Wed, May 9, 2012 at 8:53 AM, Erich Dollansky er...@alogreentechnologies.com wrote: For your recommendation above, what are the advantages or differences of slicing the disk versus partitioning on a single slice? it could be a misunderstanding. What is a partition? What is a slice. I have to look always into the handbook. Anyway, as long the OS see different units which have to be mounted independent of each other, it all does not matter what is what. I meant in Unix terms of course. Slice is slice (partition in other OS) and partition a thru h The question is if it has any advantage of using a slice to mount the basejail in RO as opposed to doing the same thing on a partition. The answer is: It it not possible. :-) You cannot mount a slice. Given the BSD terminology: A slice _has_ to contain partitions. You cannot format a slice, you can only format partitions. A formatted partition carries a UFS file system. (However, it's possible to omit the slice, and partition the whole disk instead, this is called dedicated mode). A third method is formatting the whole disk (the 'c' device), in that case the 'c' is omitted. The _only_ time you can mount a slice is when it is used in its common meaning, being a DOS primary partition; in this case, a FAT or NTFS file system will be placed directly into a slice, as those do not support any (BSD-style) partitioning. /dev/ad0- the disk /dev/ad0s1 - 1st slice /dev/ad0s1a - 1st partition on 1st slice THIS is something you can mount. -or- /dev/ad0a - 1st partition on disk (dedicated) THIS can also be mounted. -or- /dev/ad0- the whole disk (equals /dev/ad0c) Even THIS can be mounted. In case I'm misunderstanding your question, could you alter the expression? -- Polytropon Magdeburg, Germany Happy FreeBSD user since 4.0 Andra moi ennepe, Mousa, ... ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
Hi, On Wednesday 09 May 2012 20:30:37 Alejandro Imass wrote: On Wed, May 9, 2012 at 8:53 AM, Erich Dollansky er...@alogreentechnologies.com wrote: Hi, On Wednesday 09 May 2012 18:57:06 Alejandro Imass wrote: On Thu, May 3, 2012 at 1:14 PM, Alejandro Imass a...@p2ee.org wrote: On Thu, May 3, 2012 at 9:35 AM, Robert Bonomi bon...@mail.r-bonomi.com wrote: [...] One comment: for 'defensive' purposes it would be useful to break ad6 up into two slices, putting 'basejail' in it's own slice. Then, for production use, that slice can be mounted RO, and with the 'system immutable' flag set on everything in that filesystem. Yes. From one of your posts that became somewhat clear to me: Having all the jails on a single 150GB slice seems like a bad idea. For your recommendation above, what are the advantages or differences of slicing the disk versus partitioning on a single slice? it could be a misunderstanding. What is a partition? What is a slice. I have to look always into the handbook. Anyway, as long the OS see different units which have to be mounted independent of each other, it all does not matter what is what. I meant in Unix terms of course. Slice is slice (partition in other OS) and partition a thru h The question is if it has any advantage of using a slice to mount the basejail in RO as opposed to doing the same thing on a partition. I do not think so. As long it is mounted as a separated unit, FreeBSD has to keep everything separated. Erich ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
On Mon, Apr 30, 2012 at 6:42 PM, Jerome Herman jher...@dichotomia.fr wrote: [...] I must admit that Robert Bonomi tone was highly insulting for this list, and though I completely condemn the form of his post, I cannot say I disagree with the content. I disagree with both the form and the content and I will tell you why later... I do appreciate however the time you and everyone else (including Robert Bonomi's), have taken to answer and post such lengthy insights. I believe everyone's opinion is important and should be respected. There are quite a lot of things that are wrong with Alejandro Imass' post and analysis. The fist thing is that he did not give is setup in one go. It took quite a while to figure what happened, what system he was using and how he was using it. At first he had to hard reboot an unresponsive system, then at reboot he would have lost all of his jail. Then it appeared that all the jails where inside another jail and that the unresponsiveness came from MySQL. Then we learn that all his daemons are inside jails. Then we learn that ftp-proxy is not. Then we learned that jail are not handled manually but through EZJail. Then we are told that the problem with MySQL is known and comes from a client using TigerCRM with a too much data. There are litterally dozens of little pieces of important knowledge all over the thread. And you have to read it all to make sure you have the global view. Not really a good start. It is OK to forget to mention a thing or two, discarding what you think is irrelevant to the problem at hand, but it is not OK to force people who are trying to help you to read 50+ posts to learn about the basics of your installation. Granted. Nevertheless, the EzJail part (which I admit was a very important piece of information) was left out my first and second post was in fact established in the third post, so it was quite early in the thread. I think that it is not hard to put yourself in my shoes, and understand that in a moment of crisis, your first priority is NOT articulating the most complete and technical bug report you can. On the contrary, it's a cry for help from your peer users to see if you can gain some insight on solving the problem as quickly as possible. What is even more irritating is the fact that Alejandro Imass ignores pretty much anything that would leads toward a human mistake. Most posts implying a possible bad use of jails/nullfs/ezjail are ignored or answered by a simple I have done everything by the book. Now from my experience someone with 6 servers, each containing multiple jails will not do everything by the book every time. It might be that Alejandro is exceptional, but it is more likely Well, we do run everything by the book, precisely to avoid problems. We find one recipe that works and stick to it like religion. I have only used EzJail commands and **normal** use of EzJail. I am not expected to know _extactly_ how it works, I trust that to the experts in each field. As a user I am only expected to RTFM, and use it accordingly. Again let me remind everyone here, this list is precisely for that: FreeBSD ***GENERAL QUESTIONS***. It is NOT a technical list. When you and Robert Bonobi and everyone elese here subscribed to this particular list, it should have been pretty clear: - General lists: The following are general lists which anyone is free (and encouraged) to join: - freebsd-questions: User questions and technical support - About freebsd-questions English (USA) :This is the mailing list for questions about FreeBSD. You should not send how to questions to the technical lists unless you consider the question to be pretty technical. So I am entitled to post general questions and provide information as I see it fit, or if an expert on the list may ask for more. When I posted the first few posts, that's all the information I had, if you thought you needed more information, then you should have said so, but instead your personal guess is a priori judgment call, which I found almost as insulting as all of Bonobi's posts and I simply ignored you. In retrospective, and after re-reading you first post and this one, I can understand that having left EzJail out in the first post was a key piece of information that would have probably caused you to answer very differently, so I can somewhat justify your initial post, but to me at that moment, you should have already known I was using EzJail. that at least one if not more of these jails were not made by the book. Nothing to blame anyone in here, we all get tired/bored/overconfident sometime - but refusing to admit the very possibility of a human mistake won't help at all in finding a solution. Reading the thread I realized that my suggestion that he might have over-used ln had been discarded as stupid, but the information came a lot later in answer to another post. Of Yes, I must apologize for having ignored your post, but I found your a priori *assumption* of
Re: UFS Crash and directories now missing
Alejandro Imass a...@p2ee.org wrote: [ megasnip ] Things to investigate : - When was the last time this box was rebooted normally ? Did it went fine ? After I moved the jails to the right place I archived the jails with ezjail-admin and rebooted the server several times, and everything worked as expected. Rephrasing -- when was the last time _before_the_problem_was_discovered_ that the machine was re-booted? Were the jails created at this time ? No. Most of these jails have been operational for over a year on this server without any incidents. Clarifying the question -- were the jails created at the time of the last _prior_ reboot? i.e., had the machine been re-booted successfully _after_ the jails were installed, or was this the _first_ such reboot? It appears you misunderstood the 'at this time' reference -- it did ot mean 'at the time of the incident', but 'at the time of the last prior reboot'. If English is not your primary language, it is an understandable misread. As I told you earlier, this server has been running for over a year and we have rebooted many times. I don't believe you ever mentioed that particular point (multiple successful reboots after istallation) before. Repeating a prior question, _how_long_ before the problem showed up was the most recent re-boot? (Doesn't have to be exact -- an 'order of magnitude' estimate [a day, a week, a month, several months] is sufficient.) If there are such problems they exist by using the EzJail commands and I find this unlikely. What you 'find unlikely' is irrelevant. The entire situation is 'unlikely', yet it happened. So one -has- to look at unlikely things. wry grin here is the mount output is that's of any help: [ first disk, and 'fdescfs', and 'procfs' references removed, for clarity ] /dev/ad6s1.journal on /usr/jails (ufs, asynchronous, local, gjournal) /usr/jails/basejail on /usr/jails/yabarana-php53/basejail (nullfs, local, read-only) /usr/jails/basejail on /usr/jails/yabarana-php52/basejail (nullfs, local, read-only) /usr/jails/basejail on /usr/jails/yabarana-cat58/basejail (nullfs, local, read-only) /usr/jails/basejail on /usr/jails/testbed/basejail (nullfs, local, read-only) /usr/jails/basejail on /usr/jails/pyugmao/basejail (nullfs, local, read-only) /usr/jails/basejail on /usr/jails/php53base/basejail (nullfs, local, read-only) /usr/jails/basejail on /usr/jails/php52base/basejail (nullfs, local, read-only) /usr/jails/basejail on /usr/jails/mcs-cat58/basejail (nullfs, local, read-only) /usr/jails/basejail on /usr/jails/http-proxy/basejail (nullfs, local, read-only) /usr/jails/basejail on /usr/jails/corcaribe-php53/basejail (nullfs, local, read-only) /usr/jails/basejail on /usr/jails/cm-website/basejail (nullfs, local, read-only) /usr/jails/basejail on /usr/jails/cm-idvida/basejail (nullfs, local, read-only) /usr/jails/basejail on /usr/jails/cat58base/basejail (nullfs, local, read-only) /usr/jails/basejail on /usr/jails/cmm-php52-1/basejail (nullfs, local, read-only) Yes, that is a good start at useful detail. It is, presumably, _after_ the problem, and _after_ you had restored things to their proper places. Is it safe to assume that you do -not- have such a 'mount' output from some time 'before' the problem? ( There's no rational reason why you -would- have such, but _if_ it existed, and there were any differences between 'then' and 'now', it could be very informative.) Aother critical piece of information is what diretories -- by full path name -- disappeared from 'where they were', and where -- by full path name, again -- did you find them, and _with_what_names_? If everything was moved from the same source point to the same destination, it's not necessary to itemize each one, but the details of _one_ 'typicaal' migration is needed. It is also significant if there was 'anything else' in the 'where they belonged' directory that was -not- moved. *OR* if there was anything else (something other than the '/' of a jail) there, that was _also_ moved. Narrative descriptions, as previously provided, and while clear to someone familiar with the machcine in question, are not sufficiently precise to allow an 'outsider' to follow the events without 'logically' replicating the setup, and then guessing at the meaning of any shorthands employed. One comment: for 'defensive' purposes it would be useful to break ad6 up into two slices, putting 'basejail' in it's own slice. Then, for production use, that slice can be mounted RO, and with the 'system immutable' flag set on everything in that filesystem. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
Alejandro Imass ait at p2ee.org writes: ... devfs on /usr/jails/cmm-php52-1/dev (devfs, local, multilabel) ... /usr/jails/basejail on /usr/jails/cmm-php52-1/basejail (nullfs, local, read-only) fdescfs on /usr/jails/cmm-php52-1/dev/fd (fdescfs) procfs on /usr/jails/cmm-php52-1/proc (procfs, local) There is one thing that looks like an anomaly. For each jail, should the master template basejail be mounted into it first, followed by /dev and anything else in there ? /usr/jails/basejail on /usr/jails/cmm-php52-1/basejail (nullfs, local, read-only) devfs on /usr/jails/cmm-php52-1/dev (devfs, local, multilabel) fdescfs on /usr/jails/cmm-php52-1/dev/fd (fdescfs) procfs on /usr/jails/cmm-php52-1/proc (procfs, local) Does it matter ? jb ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
On Thu, May 3, 2012 at 9:35 AM, Robert Bonomi bon...@mail.r-bonomi.com wrote: Alejandro Imass a...@p2ee.org wrote: [ megasnip ] Things to investigate : - When was the last time this box was rebooted normally ? Did it went fine ? After I moved the jails to the right place I archived the jails with ezjail-admin and rebooted the server several times, and everything worked as expected. Rephrasing -- when was the last time _before_the_problem_was_discovered_ that the machine was re-booted? The jails moved Friday 27th so the last reboot before that was Apr 4 and before Feb 29 Feb 29 10:18:46 nune reboot: rebooted by aimass Apr 4 19:45:03 nune reboot: rebooted by aimass Apr 27 19:47:06 nune reboot: rebooted by aimass Apr 28 02:03:57 nune reboot: rebooted by aimass Were the jails created at this time ? No. Most of these jails have been operational for over a year on this server without any incidents. Clarifying the question -- were the jails created at the time of the last _prior_ reboot? i.e., had the machine been re-booted successfully _after_ the jails were installed, or was this the _first_ such reboot? No not at all. Most of these jails were created last year, but here is the detail. cmm_php52_1 is the problematic jail with the MySQL, you will see a recent date in the config file because I recently added some cpuset as a band-aid to limit the jail's ability to bring down the whole system, leaving at least a couple of CPUs free to be able to ssh and shut it down. There is however a new jail corcaribe_php53 and was the reason we rebboted the server on Apr 4th, just to make sure that eveything would boot OK after reboot. -rw-r--r-- 1 root wheel 917 Feb 16 2011 cat58base -rw-r--r-- 1 root wheel 917 Apr 29 2011 cm_idvida -rw-r--r-- 1 root wheel 937 Apr 3 2011 cm_website -rw-r--r-- 1 root wheel 960 May 2 09:48 cmm_php52_1 -rw-r--r-- 1 root wheel 1037 Apr 4 20:00 corcaribe_php53 -rw-r--r-- 1 root wheel 950 Feb 16 2011 http_proxy -rw-r--r-- 1 root wheel 917 Aug 3 2011 mcs_cat58 -rw-r--r-- 1 root wheel 917 Feb 10 2011 php52base -rw-r--r-- 1 root wheel 917 Feb 12 2011 php53base -rw-r--r-- 1 root wheel 877 Dec 27 20:33 pyugmao -rw-r--r-- 1 root wheel 877 Mar 21 22:03 testbed -rw-r--r-- 1 root wheel 1017 May 13 2011 yabarana_cat58 -rw-r--r-- 1 root wheel 1017 Feb 13 2011 yabarana_php52 -rw-r--r-- 1 root wheel 1017 Feb 13 2011 yabarana_php53 It appears you misunderstood the 'at this time' reference -- it did ot mean 'at the time of the incident', but 'at the time of the last prior reboot'. If English is not your primary language, it is an understandable misread. As I told you earlier, this server has been running for over a year and we have rebooted many times. I don't believe you ever mentioed that particular point (multiple successful reboots after istallation) before. Repeating a prior question, _how_long_ before the problem showed up was the most recent re-boot? (Doesn't have to be exact -- an 'order of magnitude' estimate [a day, a week, a month, several months] is sufficient.) Apr 4th If there are such problems they exist by using the EzJail commands and I find this unlikely. What you 'find unlikely' is irrelevant. The entire situation is 'unlikely', yet it happened. So one -has- to look at unlikely things. wry grin funny here is the mount output is that's of any help: [ first disk, and 'fdescfs', and 'procfs' references removed, for clarity ] /dev/ad6s1.journal on /usr/jails (ufs, asynchronous, local, gjournal) /usr/jails/basejail on /usr/jails/yabarana-php53/basejail (nullfs, [...] Yes, that is a good start at useful detail. It is, presumably, _after_ the problem, and _after_ you had restored things to their proper places. Yes. Is it safe to assume that you do -not- have such a 'mount' output from some time 'before' the problem? ( There's no rational reason why you -would- have such, but _if_ it existed, and there were any differences between 'then' and 'now', it could be very informative.) No, but from what I remember it's mostly very similar. I can pull off similar mount statement from other server(s) where we run similar set-ups and that have never failed if needed. Aother critical piece of information is what diretories -- by full path name -- disappeared from 'where they were', and where -- by full path name, again -- did you find them, and _with_what_names_? If everything was moved from the same source point to the same destination, it's not necessary to itemize each one, but the details of _one_ 'typicaal' migration is needed. It is also significant if there was 'anything else' in the 'where they belonged' directory that was -not- moved. *OR* if there was anything else (something other than the '/' of a jail) there, that was _also_ moved. I took a screen shot because I somehow suspected no one would
Re: UFS Crash and directories now missing
On Thu, May 3, 2012 at 12:05 PM, jb jb.1234a...@gmail.com wrote: Alejandro Imass ait at p2ee.org writes: ... devfs on /usr/jails/cmm-php52-1/dev (devfs, local, multilabel) ... /usr/jails/basejail on /usr/jails/cmm-php52-1/basejail (nullfs, local, read-only) fdescfs on /usr/jails/cmm-php52-1/dev/fd (fdescfs) procfs on /usr/jails/cmm-php52-1/proc (procfs, local) There is one thing that looks like an anomaly. For each jail, should the master template basejail be mounted into it first, followed by /dev and anything else in there ? /usr/jails/basejail on /usr/jails/cmm-php52-1/basejail (nullfs, local, read-only) devfs on /usr/jails/cmm-php52-1/dev (devfs, local, multilabel) fdescfs on /usr/jails/cmm-php52-1/dev/fd (fdescfs) procfs on /usr/jails/cmm-php52-1/proc (procfs, local) Does it matter ? I have no idea, but cmm-php52-1 is in fact the problematic jail with the MySQL problem. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
Alejandro Imass ait at p2ee.org writes: ... I have no idea, but cmm-php52-1 is in fact the problematic jail with the MySQL problem. Could you please include displays of 1. your troubled machine's $ cat /etc/fstab Note: you already showed us 'mount' output. 2. your other trouble-free server's $ cat /etc/fstab $ mount jb ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
On Thu, May 3, 2012 at 1:40 PM, jb jb.1234a...@gmail.com wrote: Alejandro Imass ait at p2ee.org writes: ... I have no idea, but cmm-php52-1 is in fact the problematic jail with the MySQL problem. Could you please include displays of 1. your troubled machine's $ cat /etc/fstab Note: you already showed us 'mount' output. 2. your other trouble-free server's $ cat /etc/fstab $ mount The fstab was in a previous mail but here it is again... From the troubled server: # DeviceMountpoint FStype Options DumpPass# /dev/ad4s1b noneswapsw 0 0 /dev/ad4s1a / ufs rw 1 1 /dev/ad4s1d /tmpufs rw 2 2 /dev/ad4s1f /usrufs rw 2 2 /dev/ad4s1e /varufs rw 2 2 /dev/ad6s1.journal /usr/jails ufs rw,async2 2 /dev/cd0/cdrom cd9660 ro,noauto 0 0 From a good server (single disk machine): /dev/ad4s1b noneswapsw 0 0 /dev/ad4s1a / ufs rw 1 1 /dev/ad4s1d /tmpufs rw 2 2 /dev/ad4s1f.journal /usrufs rw,async 2 2 /dev/ad4s1e /varufs rw 2 2 /dev/ad4s1a on / (ufs, local) devfs on /dev (devfs, local, multilabel) /dev/ad4s1d on /tmp (ufs, local, soft-updates) /dev/ad4s1f.journal on /usr (ufs, asynchronous, local, gjournal) /dev/ad4s1e on /var (ufs, local, soft-updates) /usr/jails/basejail on /usr/jails/httpProxy/basejail (nullfs, local, read-only) devfs on /usr/jails/httpProxy/dev (devfs, local, multilabel) fdescfs on /usr/jails/httpProxy/dev/fd (fdescfs) procfs on /usr/jails/httpProxy/proc (procfs, local) /usr/jails/basejail on /usr/jails/cat58base/basejail (nullfs, local, read-only) devfs on /usr/jails/cat58base/dev (devfs, local, multilabel) fdescfs on /usr/jails/cat58base/dev/fd (fdescfs) procfs on /usr/jails/cat58base/proc (procfs, local) /usr/jails/basejail on /usr/jails/watwkyTesting/basejail (nullfs, local, read-only) devfs on /usr/jails/watwkyTesting/dev (devfs, local, multilabel) fdescfs on /usr/jails/watwkyTesting/dev/fd (fdescfs) procfs on /usr/jails/watwkyTesting/proc (procfs, local) /usr/jails/basejail on /usr/jails/mta1/basejail (nullfs, local, read-only) devfs on /usr/jails/mta1/dev (devfs, local, multilabel) fdescfs on /usr/jails/mta1/dev/fd (fdescfs) procfs on /usr/jails/mta1/proc (procfs, local) /usr/jails/basejail on /usr/jails/migdev/basejail (nullfs, local, read-only) devfs on /usr/jails/migdev/dev (devfs, local, multilabel) fdescfs on /usr/jails/migdev/dev/fd (fdescfs) procfs on /usr/jails/migdev/proc (procfs, local) jb ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
Alejandro Imass ait at p2ee.org writes: ... OK. There is this anomaly in your troubled jail. Because I have not been familiar with ezjail so I am learning it now as we go. I will suggest to you some steps by intuition, and you have to judge for both of us how to do it and what's appropriate. Feel free to give feedback anytime. I understand that this troubled jail can be stopped any time without any problems to your clients. Just in case do a backup of it with ezjail admin entry. Use ezjail admin entries by the way (no 'mv', 'cp', etc). I look at an ezjail tutorial right now and see the following: ezjail Default File Locations /usr/jails/ : Default location to store base jail system template. /usr/jails/flavours/ : Customization for each jail can be done via flavours. For e.g. adding default /etc/resolv.conf file or updating existing /etc/make.conf can be done here. /usr/jails/basejail/ : Base jail will be exported and mounted as read only for each jail. This will save disk space. /usr/local/etc/rc.d/ezjail.sh : Stop / Start / Restart jails script. /usr/local/etc/ezjail.conf : Configuration file for ezjail script. contains settings that control the operation of the ezjail rc script. It is also read by the ezjail-admin utility to figure out where it should perform its actions. /usr/local/etc/ezjail/ : All your jail configuration files are stored here. Now, your task is to figure out where your mounts are configured for that jail, those that are present in the 'mount' output. After that you would stop that jail and delete it (assuming no problem to your customer service) to get rid of those lines in mount output. The reconfigure that jail properly anew (do not use backup for this !) and create it again, check the 'mount' output to see proper sequence of mounts. Then restart the jail and see how it goes. Give us feedback as you go so we can follow you. jb ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
Eitan Adler li...@eitanadler.com wrote: On 30 April 2012 07:36, Robert Bonomi bon...@mail.r-bonomi.com wrote: A competennt, not stupid, sysadmin would know these things. And not 'remove all doubt' (in the words of Abraham Lincoln), by raising such nonsense questions. A competent sysadmin would ask questions when they don't know the answer bringing up possibilities they thought about. A stupid sysadmin would yell at someone asking a question claiming they should have known the answer. An informed critic would have recognized that the 'lack of knowledge' issue, and the 'nonsense questions' were two -entirely- different matters. grin One who lacks knowledge of system fundamentals and asks questions _about_ _the_fundammentals_ that they do not understand is not subject to criticizm -- they are educatable. Those who make grossly false-to-fact assumptions about the behavior of those fundamentals, and extrapolate wildly from those erroneous assumptions cannot be engaged in rational conversation -without- hauling them back to the initial erroneous assumptions, and correcting those errors. And, when that is done, it invaliates everything extrapolated from the false premise. Those who continue to extrapolate wildly in such manner cannot be helped. It was also established that the OP's descriptions were woefully incomplete and unreliable. A second disk was involved. 'dangerously dedicated' or otherwise? partitioning? slices? label type? There is indirect indication 'everything of interest' was on a single slice, but that is only an inference. There's no indication of where _in_the_filesystem_ on the slice that the jails '/' directories were located, or by what names they were known to the system outside the jail. The 'pattern' of the names, and placement in the hierarchy _is_ likely of some significance. As is (a) ownership, (b) permissions, and (c) 'flags', of (1) the original 'containing' directory, (b) the external view of the jail '/' directories in that directory, and (c) 'where they ended up'. It is likely that that 'external view' (pre- problem) of the jail '/'s does not exist -- unless one had historical data from before the problem. Everything was running in jails. Except for things that weren't. For any constructive analysis of what happened, one needed to capture *all* the bits in the directory (itself) where the jails ended up -- a directory 'listing', e.g. 'ls' (regardless of options), is not sufficient -- and the same for the directory where they 'should have been', plus a copy of the slice's complete inode table -- i.e., from _all_ the cylinder groups. Then one would examine the 'last modified' timestamp on the directory where the jails were found, and -then- the timestamps on the jail directories themselves. Among other things, this data allows one to establish whether or not the jail directories were ever _really_ where one thought they were, or whether they just 'appeared' to be there, e.g. due to nullfs, or a 'link'. And an 'initial estimate' of -when- it may have happened. (if 'malice' is involved, or certain kinds of backup/restore activities, the timestamps _may_ not be accurate, but they are a 'best available' guess.) Capturing -all- the data from the 'where they were' directory, allows one to examine the 'deleted' entries -- where one _should_ find entries for the jails, and 'last accessed' timestamps which put a lower bound on when the 'move' occured. When the 'apparently impossible' happens, it is *VERY*OFTEN* the case that 'reality' is *NOT* what someone 'knows' it is. No matter how 'obvious' it is, one has to =verify=. It is also _FAR_ 'easier to believe' that (especially) a nullfs mount (or, less likely, a hard link) disappeared, than directories actually got moved. The move may well have happened, but one must 'positively' eliminate the 'more plausible' alternatives first. Things that would 'give the appearance' of what was reported, but from -very- different causations. Of course, to capture this kind of information, one have to know what's where in the filesystem metadata, and have means to capture it _without_ changing any of that data. And _that_ means that you have to have a fair understanding of the mechanics of how the filesystem works. Which rapidly leads into gory details of how the O/S does disk I/O, and the various performance optimizations (and trade-offs) employed. Reading _both_ of McKusick's Design of .. books, and the 'Unix System Admininstration Handbook', by Nemeth, et al. is a good _start_. Having a bunch of the books from O'Reilley Assoc. (http://www.ora.com), especially for 'standard' tools that you need to get the most out of, is also highly recommended. Disclaimer: I know a lot of the authors of those books, persoally. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any
Re: UFS Crash and directories now missing
On 04/30/2012 10:58 PM, Robert Bonomi wrote: Reading_both_ of McKusick's Design of .. books, and the 'Unix System Admininstration Handbook', by Nemeth, et al. is a good_start_. Having a bunch of the books from O'Reilley Assoc. (http://www.ora.com), especially for 'standard' tools that you need to get the most out of, is also highly recommended. After realising I lack ton of knowledge, especially how the internals work. I'm using this advice:-) . ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
On 05/01/2012 06:43 AM, Polytropon wrote: Except buying (good) books, you can also search for articles on the web. For example, A Fast File System for UNIX by M. K. McKusick is very interesting (at least it was for me when I lost all my important data). Some fs-related articles here: http://www.mckusick.com/articles.html They help you to understand how things work or what maybe makes them stop working.:-) Also the documentation of tools like TSK (ports/sleuthkit), ex TCT, is very helpful in understanding all the low-level details that_really_ matter when you_need_ to get your hands dirty in order to perform a forensic analysis or to recover important data. Sadly, that documentation has moved from local storage in/usr/local/share/doc/sleuthkit/ (where I've seen it the last time) to some on-line place or Wiki, something_I_ consider a bad idea especially in worst case considerations (i. e. no internet connection); the only content in README.txt, The docs that used to live in this directory now exist on the wiki: http://wiki.sleuthkit.org/ doesn't make it any better, sorry. Thanks for the help...I will definitely check McKusick site and the docs I'm self learning UNIX/programming. so I need all the info and help I can get.:-) ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
On Tue, May 01, 2012 at 12:58:10AM -0500, Robert Bonomi wrote: Reading _both_ of McKusick's Design of .. books, and the 'Unix System Admininstration Handbook', by Nemeth, et al. is a good _start_. Both? I'm aware of at least three (FreeBSD, 4.3BSD, and 4.4BSD) that are probably within the realm of what you're talking about (learning about the workings of a BSD Unix system), all of which seem a little redundant -- just different editions of the same book, from the look of it. What do you mean by both of McKusick's books? I think there's an answer book for at least one of those, too. Do you perhaps mean the main book and the answer book? Do you mean to include the general-purpose open source book as one of the books (Open Sources: Voices from the Open Source Revolution)? Having a bunch of the books from O'Reilley Assoc. (http://www.ora.com), especially for 'standard' tools that you need to get the most out of, is also highly recommended. Disclaimer: I know a lot of the authors of those books, persoally. If you have a decent ebook reader, I recommend just getting on the O'Reilly mailing list for its periodic announcements of ebook discount deals and picking up an occasional good book from those deals. It's easy to get far more excellent books than you have time to read that way, for really good prices. In fact, O'Reilly has a 50% off deal for a few ebooks about C programming right now: http://shop.oreilly.com/category/deals/c-programming.do O'Reilly's ebook deals are about the only way I've found to get good technical books from a major publisher in digital formats at a reasonable price, considering most of the publishing world still thinks it's okay to charge more for ebooks than for hardcopy books for some asinine reason. O'Reilly is, in fact, pretty far ahead of competitors on its handling of ebooks. For instance, if you have a hardcopy O'Reilly book, you can register it by ISBN with O'Reilly, then get an ebook copy of it for about five bucks. By contrast, The Pragmatic Bookshelf (which produces very high quality books as well) at *best* gives you the opportunity to get a hardcopy book plus a PDF book at the same time for about 150% of the cover price of the hardcopy alone, *only* if you buy them together from the Pragmatic website itself, and if you only have the ebook or the hardcopy book you have no way to get a discount on the other; you have to pay full price. Pragmatic does offer ebooks at slightly lower price than hardcopy, which is at least better than the standard industry practice for science fiction, but it's a ridiculous price for a bundle of bits in a digital file. O'Reilly offers some kind of discount on hardcopies for people who have the ebooks, too, I think. I'm not sure -- I've never taken advantage of that discount, because I only started collecting ebook copies of O'Reilly books after getting an e-ink reader, which I find every bit as good for many (though not all) reading purposes as a physical dead tree format book. Your mileage may vary, I suppose. -- Chad Perrin [ original content licensed OWL: http://owl.apotheon.org ] ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
Hi, On Tuesday 01 May 2012 20:43:43 Polytropon wrote: On Tue, 01 May 2012 00:37:51 -0700, Edward M wrote: On 04/30/2012 10:58 PM, Robert Bonomi wrote: Reading_both_ of McKusick's Design of .. books, and the 'Unix System Admininstration Handbook', by Nemeth, et al. is a good_start_. Having a bunch of the books from O'Reilley Assoc. (http://www.ora.com), especially for 'standard' tools that you need to get the most out of, is also highly recommended. After realising I lack ton of knowledge, especially how the internals work. I'm using this advice:-) . Except buying (good) books, you can also search for articles on the web. For example, A Fast File System for UNIX by M. K. McKusick is very interesting (at least it was for me when I lost all my important data). you wanted to say 'real man do not need a backup'? Some fs-related articles here: http://www.mckusick.com/articles.html This is one advantage of systems like FreeBSD. If the need arises, you can do it yourself. The docs that used to live in this directory now exist on the wiki: http://wiki.sleuthkit.org/ It must be a disease. Erich ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
Alejandro Imass a...@p2ee.org wrote: On Sun, Apr 29, 2012 at 11:49 PM, Erich Dollansky wrote: On Monday 30 April 2012 02:02:41 jb wrote: Alejandro Imass ait at p2ee.org writes: ... Back to theory on how the http-proxy jail 'swallowed' all the other jails including the basejail. A theory that contains assumptions which are, unfortunately, unsupported by any factual evidence. Just like _every_other_ theory you have advanced to date. FACT: It is a virtual certainty that something operating -outside- any jail environment is what did the deed. Available evidence to date is that you 'fixate' on a particular _remote_ possibility -- *without* knowledge of what it would take for that scenario to come to pass -- making a sh*tload of 'assumptions' along the way (many of which are contrary to reality), and offer that as 'the explanation' for events. Given that EzJail uses a single basejail and links/mounts stuff in the child jails it would seem plausible (regression?) that somehow any jail could access other jails' files, Demonstrating, yet again, that you do not understand how jails work. :(( or that _maybe_ in an event of crash the nullsfs mounts confuse the system somehow when fsck restores or the journal is recovered. Demonstrating, yet again, that you do not understand what nullfs is, how it works, or that it is totally -irrelevant- to fsck and/or journaling. Hint: nullfs is merely a 'path translation' mechanism -- it affects _only_ 'file open' syscalls. fsck doesn't _touch_ nullfs. Hint; journaling is an add-on to the UFS filesystem. nullfs doesn't know what journaling is. Journal recovery doesn't _touch_ a nullfs. A competennt, not stupid, sysadmin would know these things. And not 'remove all doubt' (in the words of Abraham Lincoln), by raising such nonsense questions. Whatever the cause, it actually happened and I have already ruled out just about anything. It doesn't seem to have been an attack, it surely wasn't me, and EzJail author agrees it was not the EzJail scripts. So maybe nullfs and journaling, or crash + nullfs + journaling, could cause something like this to happen? Postulating the right combination of _unrelated_ failures, virtually *anything* can happen. cf. Nasal Monnkeys. It has already been demonstrated how the (im-)probability of such an event relates to the age of the universe. Maybe journal has some confusion on restoring the nullfs view of the directories or something after bad crash like this one?? Short answer: No chance. Again, if you had any understandinng of how UFS, and nullfs for that matter, works -- not to mention how disk I/O works inside the kernel, you wouldn't be embarassing yourself by your _continued_ raising of what are, to put it charitiably, such 'patently ridiculous' questions. You can engage in all the 'unfounded speculation' you want to, but you are simply -not- going to determine what happened. IF there was a systemic fault, you have already destroyed the forensic evidence trail that _might_ have allowed a qualified expert to run it down, *if* you could afford to have such an analysis done. (middle five figures is a starting point for such an analysis.) Absent _multiple_ reports of like events, *WITH* enough detail in the reports to have a reasonable chance of identifying a 'pattern' of events leading to the failure, *OR* the existance of a -reliable-, =repeadable=, method of inducing the failure, this simply isn't going to go anywere. Absent any of those things, it is a 'freak' event, *PROBABLY* (read 'virtually certain') caused by human error (despite your claim of the 'impossibility' of that factor) in some form. If you insist on 'knowing' what happened in any future instance of single putatively 'abnormal' events, you will need to change to a MIL-SPEC 'B2' (or higher) rated O/S, with active mandatory access controls, 'security labels' with multi-level, non-hierarchical, security enabled, audit logging of -every- system call, etc. This also requires a staff position of 'security officer', which is _separate_and_distinct_ from 'system administrtor'. I strongly suspect that you cannot afford the required hardware and software for this type of 'solution'. The 'underlying cause' almost certainly falls into the class known as PEBKAC. (The current admin has demonstrated an inability to accurately report the state of his system -- that at least one thing he previously asserted to be true was _not_, in fact, the case. It is *HIGHLY*LIKELY* that _that_ 'exception' to the claimed state is =not= the only such violation on that system.) That there was an action where there was a difference between 'that which was intended', and 'what it really did'. Such things are almost -impossible- for the perpetrator of the action to identify -- they 'know' what they did, and read the act as 'doing what they meant it to do',
Re: UFS Crash and directories now missing
Hi, On Monday 30 April 2012 18:36:08 Robert Bonomi wrote: Alejandro Imass a...@p2ee.org wrote: That simply *ISN'T* going to happen -- not without a -lot- more evidence than any individual can provide from a single =unrepeadable= incident. ok, I am not the original poster but let me tell me of an experience here I have had. I reported also something extremely strange. Of course, the comments I have gotten have been the same as here. But what happened then? I do not know why but somebody found a race condition in the affected system. There is no fix available yet. With other words, no matter how strange things are, I encourage people to report it. Not with the real hope to get a solution at the spot. But with the chance that somebody who knows the code well and has some strange feelings takes a look. I also encourage my clients to do the same for our products and services. Erich ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
On Mon, Apr 30, 2012 at 7:36 AM, Robert Bonomi bon...@mail.r-bonomi.com wrote: Alejandro Imass a...@p2ee.org wrote: On Sun, Apr 29, 2012 at 11:49 PM, Erich Dollansky wrote: On Monday 30 April 2012 02:02:41 jb wrote: Alejandro Imass ait at p2ee.org writes: ... [...] A competennt, not stupid, sysadmin would know these things. And not 'remove all doubt' (in the words of Abraham Lincoln), by raising such nonsense questions. Whatever the cause, it actually happened and I have already ruled out just about anything. It doesn't seem to have been an attack, it surely wasn't me, and EzJail author agrees it was not the EzJail scripts. So maybe nullfs and journaling, or crash + nullfs + journaling, could cause something like this to happen? Postulating the right combination of _unrelated_ failures, virtually *anything* can happen. cf. Nasal Monnkeys. OK, I tried to be patient with you and tried to keep my composure and nettiquete against your insistence to insult me and by doing so, damaging the good spirited nature of this mailing list, FreeBSD and Open Source in general. And sorry beforehand to my fellow co-listers, and other nice people here, that I have to do this publicly but there is a limit and I am sure many of you have been just waiting for this to happen. I mean, I have had a couple of altercations here and there with a few smart asses, but I have *NEVER* seen such an obnoxious little shit in the more than 14 years I have been participating in ANY mailing list. This used to be one of the most enjoyable and helpful lists and it is people like you who draw people away from sharing and trying to help one another. What is your problem man? Who do you think you are? Who gives you the right to go patronizing and insulting people, and by the way using these ridiculous quotes, like some stupid little jerk, relying on other people's wisdom quotes instead of your own words. Instead of being frontal, you are probably frustrated with your own little techy life that you have to take out your frustration on other people. I find you intoxicating to this list and to this community, no matter how smart you are, if half the stuff you say is even accurate or true. You don't contribute anything except to the degradation of the FreeBSD ambiance and to drive people away, and from sharing. You don't have the right to do that. I honestly love FreeBSD and this community and I am not going to let you ruin that for me or anyone else here. Why don't you take the time to read your posts and see that you propose nothing, so why even bother to participate? What are you trying to prove? If you were so smart as *you believe* you are, you would be helping instead of trying to prove something by your condescending attitude. The very fact that you need to use this attitude is proof of your insecurities, and your need to bully other people, but not me, Sir. I have been in this too long. I am surely not going to take this shit from you man so if you don't have anything positive to say, just shut up and let other people help each other, without being scared of being insulted or patronized. I am surely not afraid of you and I am sick and tired of your attitude, so if no one else here has the balls to tell you off, I will. This is the kind of shit that drives people away and refrains people from participating and sharing experiences and knowledge, and I'm not going to let you do that, to me or any one else here. This is not *your list* nor do you have any special right here, you are just like everybody else, just not very helpful or fun. This attitude will get you nowhere but deeper into your creepy little world. -- Alejandro Imass ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
On 04/30/2012 08:38 AM, Alejandro Imass wrote: just not very helpful or fun. This attitude will get He is helping,you need to learn how UFS, jails, nullfs, journaling, disk I/O and other stuff work. I have been following this thread and i must admit I also need to learn more on those subjects.:-) ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
On Mon, Apr 30, 2012 at 8:22 AM, Erich Dollansky erichfreebsdl...@ovitrap.com wrote: Hi, On Monday 30 April 2012 18:36:08 Robert Bonomi wrote: Alejandro Imass a...@p2ee.org wrote: That simply *ISN'T* going to happen -- not without a -lot- more evidence than any individual can provide from a single =unrepeadable= incident. ok, I am not the original poster but let me tell me of an experience here I have had. I reported also something extremely strange. Of course, the comments I have gotten have been the same as here. But what happened then? I do not know why but somebody found a race condition in the affected system. There is no fix available yet. With other words, no matter how strange things are, I encourage people to report it. Not with the real hope to get a solution at the spot. But with the chance that somebody who knows the code well and has some strange feelings takes a look. I also encourage my clients to do the same for our products and services. Thanks Erich for pionting this out. This is the FreeBSD USERS LIST, not the elite exchange. I I was posting this on some expert list like the kernel list or some other more technical list I could understand the attitude, but this is the user's list. We are NOT required to know the details, just share experiences and try to help one another, not put other people down for trying to solve our issues as users. What is really frustrating is that it actually happened and I try to do everything by the book. I don't do any fancy or strange things, so something caused these directories to be moved through NORMAL use of the system, regardless if some people believe it or not, I could care less. It happened, period, and if someone wants to help fine, if not they should just shut up. Thanks again for pointing this out. We are the users, we are the people that keep this project alive and share the good. -- Alejandro Imass ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
On Mon, Apr 30, 2012 at 1:03 PM, Edward M eam1edw...@gmail.com wrote: On 04/30/2012 08:38 AM, Alejandro Imass wrote: just not very helpful or fun. This attitude will get He is helping,you need to learn how UFS, jails, nullfs, journaling, disk I/O and other stuff work. I have been following this thread and i must admit I also need to learn more on those subjects.:-) Oh, please! He's not helping anyone. He's just being an obnoxious prick that thinks that by pointing out a lot of technical blabber and some cheap philosophical posé, he's going to gain some sort of place amongst his peers, and you are just trying to do the same by sucking-up, siding with him and seconding an simply unacceptable attitude in a community of real peers. If you truly know your stuff you don't have to go putting people down and patronizing them to show off. It is only when you go over the top, trying to prove something that your are actually just showing your insecurities and just plain ignorance. Why don't you google and read my posts over the years when I help other people in things they don't know, and tell me if it's remotely close, or if I patronize people. I might go tell someone to RTFM but I would never go and try to put someone down just to show off that I know a lot. Furthermore, this is a user's list, not a deeply technical one. I don't have to read the fsck source code to use FreeBSD or participate on this list. If you are indeed an expert you try to help other people, or at least give the other person the benefit of the doubt. If you have really followed the thread, all I have done is try to find some explanation for a strange behavior of the system under normal use. It hung, and some directories were moved, period. I have posted some ideas to share with other people expecting some insight and maybe similar experience from other users, which there probably are many, but many times afraid to speak up and avoid getting insulted. I don't take that crap from anyone and much less in a community that I have come to love and respect. And it's all about that: RESPECT and you can either learn it the easy way or the hard way, but I will tech respect one way or another. -- Alejandro Imass ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
On 30 April 2012 07:36, Robert Bonomi bon...@mail.r-bonomi.com wrote: A competennt, not stupid, sysadmin would know these things. And not 'remove all doubt' (in the words of Abraham Lincoln), by raising such nonsense questions. A competent sysadmin would ask questions when they don't know the answer bringing up possibilities they thought about. A stupid sysadmin would yell at someone asking a question claiming they should have known the answer. -- Eitan Adler ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
On Mon, Apr 30, 2012 at 1:23 PM, Eitan Adler li...@eitanadler.com wrote: On 30 April 2012 07:36, Robert Bonomi bon...@mail.r-bonomi.com wrote: A competennt, not stupid, sysadmin would know these things. And not 'remove all doubt' (in the words of Abraham Lincoln), by raising such nonsense questions. A competent sysadmin would ask questions when they don't know the answer bringing up possibilities they thought about. A stupid sysadmin would yell at someone asking a question claiming they should have known the answer. Thank you Eitan! I am admittedly limited in the use of the English language and many times frustrated not to be able to redact such beautifully and to the point. -- Alejandro Imass ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
On 04/30/2012 10:22 AM, Alejandro Imass wrote: Oh, please! He's not helping anyone. He's just being an obnoxious prick that thinks that by pointing out a lot of technical blabber and some cheap philosophical posé I guess i was going according to the fact that i have followed his suggestions on a problem i was having and i was able to find the cause and solved the problem. :-) ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
Alejandro Imass ait at p2ee.org writes: ... If you have really followed the thread, all I have done is try to find some explanation for a strange behavior of the system under normal use. It hung, and some directories were moved, period. I have posted some ideas to share with other people expecting some insight and maybe similar experience from other users, which there probably are many, but many times afraid to speak up and avoid getting insulted. ... Well, I try to help only, so I hope I do not get insulted ... I could become vicious :-) Here it is. You said you have your jail env on a separate disk. I looked at problem reports for nullfs and there are quite few. Hierarchical Jails NOTES You said you have your jail env on a separate disk. I looked at problem reports for nullfs and there are quite few. http://www.freebsd.org/cgi/query-pr-summary.cgi?category=severity=priority=cl ass=state=sort=nonetext=nullfsresponsible=multitext=originator=release= As a matter of fact I just mounted a nullfs but was not able to unmount it (device busy) - a Google search shows it as a problem reported for many many years. Nullfs does not seem to be stable. Anyway, I found one PR http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/147420 that is about troubles with jails, nullfs, UFS, and NFS. Synopsis: [ufs] [panic] ufs_dirbad, nullfs, jail panic (corrupt inode) Take a look at this paragraphs: ... After two more failures, I now found the offending inode ... ... As one point, I found the inode in a directory which usually is mounted for an (ez-) jail via nullfs. This proves that problems with jails, nullfs, and fs corruption are possible. So, they can not be excluded up front in your case too because nullfs is just a simple path translation. jb ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
Quoth Eitan Adler on Monday, 30 April 2012: On 30 April 2012 07:36, Robert Bonomi bon...@mail.r-bonomi.com wrote: A competennt, not stupid, sysadmin would know these things. And not 'remove all doubt' (in the words of Abraham Lincoln), by raising such nonsense questions. A competent sysadmin would ask questions when they don't know the answer bringing up possibilities they thought about. A stupid sysadmin would yell at someone asking a question claiming they should have known the answer. -- Eitan Adler ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org Best response in this thread so far. -- .O. | Sterling (Chip) Camden | http://camdensoftware.com ..O | sterl...@camdensoftware.com | http://chipsquips.com OOO | 2048R/D6DBAF91 | http://chipstips.com pgpPEnYYipOaV.pgp Description: PGP signature
Re: UFS Crash and directories now missing
On Mon, Apr 30, 2012 at 1:57 PM, jb jb.1234a...@gmail.com wrote: Alejandro Imass ait at p2ee.org writes: ... If you have really followed the thread, all I have done is try to find some explanation for a strange behavior of the system under normal use. It hung, and some directories were moved, period. I have posted some ideas to share with other people expecting some insight and maybe similar experience from other users, which there probably are many, but many times afraid to speak up and avoid getting insulted. ... I looked at problem reports for nullfs and there are quite few. Hierarchical Jails NOTES You said you have your jail env on a separate disk. Yes. I looked at problem reports for nullfs and there are quite few. http://www.freebsd.org/cgi/query-pr-summary.cgi?category=severity=priority=cl ass=state=sort=nonetext=nullfsresponsible=multitext=originator=release= As a matter of fact I just mounted a nullfs but was not able to unmount it (device busy) - a Google search shows it as a problem reported for many many years. Nullfs does not seem to be stable. Dirk Engling guessed that somehow nullfs was involved. Anyway, I found one PR http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/147420 that is about troubles with jails, nullfs, UFS, and NFS. Synopsis: [ufs] [panic] ufs_dirbad, nullfs, jail panic (corrupt inode) Take a look at this paragraphs: ... After two more failures, I now found the offending inode ... ... As one point, I found the inode in a directory which usually is mounted for an (ez-) jail via nullfs. This proves that problems with jails, nullfs, and fs corruption are possible. So, they can not be excluded up front in your case too because nullfs is just a simple path translation. Up until yesterday (and Dirk's answer) I didn't look for specific references to nullfs, and today I was busy getting vicious myself ;) Thanks for pointing a plausible cause. What I have done so far is limit the offending jail to a specific cpuset and I wanted to add another disk to avoid contention with other jails. MySQL not only consumes the whole CPUs but also limits the whole drive, while it's doing some crazy full scan query on a very large database. I don't have any control of the code or the MySQL myself and the client said it's known problem with VTiger CRM and the way it implements some reports that I guess were not designed for the amount of data they are handling. I have already recommended they move to a dedicated server altogether because their system simply outgrew what we sold them. I really appreciate the time you dedicated to search for a possible explanation and at the very least it helps in taking some immediate steps to avoid it from happening again. Hopefully, someone with deep knowledge will find the root cause and a long-term fix. What is true, that if it happened to me, it can happen to anyone, so maybe your findings will help someone pin-point the problem and fix it. Thanks, -- Alejandro ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
Alejandro Imass ait at p2ee.org writes: ... Thanks for pointing a plausible cause. What I have done so far is limit the offending jail to a specific cpuset and I wanted to add another disk to avoid contention with other jails. MySQL not only consumes the whole CPUs but also limits the whole drive, while it's doing some crazy full scan query on a very large database. ... I would strongly suggest that you file a PR for nullfs, with included reference to that PR#147420 I mentioned. There is enough of circumstancial evidence. I think there is a chance that a jail panic (corrupted fs or nullfs path translation) may be the cause of MySQL going bananas. Anyway, the more similar reports they receive the better a chance they will discover any problems, if any. It would be good if you read these sections: JAIL(8) Setting up the Host Environment - a potential for mixing up traffic of same services in host env and jail env. Jails and File Systems - multiple jails sharing the same file system can influence each other. Hierarchical Jails - jail names reflect the hierarchy, but jids on the other hand exist in a single space, and each jail must have a unique jid. BUGS NOTES JEXEC(8) BUGS (ref to JAIL(8), Hierarchical Jails) jb ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
On Mon, 30 Apr 2012 13:23:40 -0400, Eitan Adler wrote: On 30 April 2012 07:36, Robert Bonomi bon...@mail.r-bonomi.com wrote: A competennt, not stupid, sysadmin would know these things. And not 'remove all doubt' (in the words of Abraham Lincoln), by raising such nonsense questions. A competent sysadmin would ask questions when they don't know the answer bringing up possibilities they thought about. A stupid sysadmin would yell at someone asking a question claiming they should have known the answer. I know I don't add anything substantial by the following statement, but allow me to post it anyway in addition to your statement: There is no problem in mentioning thoughts, possibilities and options. It's also not a problem to admit a lack of knowledge in certain fields (e. g. how UFS, journaling, nullfs and fsck do interact with each other). Things start to be problematic when conclusions are made out of untrue assumptions or expectations. It must be a system error, as I don't see a human error here. The problem is: don't see != doesn't exist, and of course != can't be proven. Such kinds of conclusion often lead into wrong directions. Of course it's hard to narrow down possibilities. A test bed with limited variables is neccessary to have. Also the proper tools and procedures of testing are important. That's the ONLY way to be sure - by eliminating one possibility after the other. What's being found in the end - and even if it's regarded unprobable from the beginning - must be the reason. Robert mentioned important things to consider. If you (unintendedly) destroy evidence for a forensic analysis of what happened (whatever it may be), you'll have a hard time finding out _what_ happened - except you can get it to happen again. In case of security breaches this is something you _don't_ want to risk IN PUBLIC just to be able to observe it. At this point, one could argue politeness vs. importance of arguments. From what I've seen on other lists, Robert's statements are still polite and full of things you can take as a start to what to additionally learn. You should concentrate on that essence. If you take the time to do your homework, you'll be better prepared _if_ such thing should ever happen again. Finding out _what_ has happened is very hard (which I admit), and maybe it's even impossible. You would have needed a more verbose auditing facility to find out what program (user) caused a mv-like syscall. Command logs can be altered, logged syscalls... yes, it's not impossible, but magnitudes _harder_ to remove trails. By the way, I can understand the frustration when something impossible happened and you never can _really_ say what it was, hoping it would not happen again. I've experienced such kinds of trouble myself. (That's why I'm on this list.) -- Polytropon Magdeburg, Germany Happy FreeBSD user since 4.0 Andra moi ennepe, Mousa, ... ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
On 30/04/2012 19:23, Eitan Adler wrote: On 30 April 2012 07:36, Robert Bonomibon...@mail.r-bonomi.com wrote: A competennt, not stupid, sysadmin would know these things. And not 'remove all doubt' (in the words of Abraham Lincoln), by raising such nonsense questions. A competent sysadmin would ask questions when they don't know the answer bringing up possibilities they thought about. A stupid sysadmin would yell at someone asking a question claiming they should have known the answer. I must admit that Robert Bonomi tone was highly insulting for this list, and though I completely condemn the form of his post, I cannot say I disagree with the content. There are quite a lot of things that are wrong with Alejandro Imass' post and analysis. The fist thing is that he did not give is setup in one go. It took quite a while to figure what happened, what system he was using and how he was using it. At first he had to hard reboot an unresponsive system, then at reboot he would have lost all of his jail. Then it appeared that all the jails where inside another jail and that the unresponsiveness came from MySQL. Then we learn that all his daemons are inside jails. Then we learn that ftp-proxy is not. Then we learned that jail are not handled manually but through EZJail. Then we are told that the problem with MySQL is known and comes from a client using TigerCRM with a too much data. There are litterally dozens of little pieces of important knowledge all over the thread. And you have to read it all to make sure you have the global view. Not really a good start. It is OK to forget to mention a thing or two, discarding what you think is irrelevant to the problem at hand, but it is not OK to force people who are trying to help you to read 50+ posts to learn about the basics of your installation. What is even more irritating is the fact that Alejandro Imass ignores pretty much anything that would leads toward a human mistake. Most posts implying a possible bad use of jails/nullfs/ezjail are ignored or answered by a simple I have done everything by the book. Now from my experience someone with 6 servers, each containing multiple jails will not do everything by the book every time. It might be that Alejandro is exceptional, but it is more likely that at least one if not more of these jails were not made by the book. Nothing to blame anyone in here, we all get tired/bored/overconfident sometime - but refusing to admit the very possibility of a human mistake won't help at all in finding a solution. Reading the thread I realized that my suggestion that he might have over-used ln had been discarded as stupid, but the information came a lot later in answer to another post. Of course in the mean time I learned that he was using ezjail, which, if I had known earlier, would have made me wonder if he had not overused nullfs or ln. He furthermore discarded the possibility saying that he did not think that ezjail was using links, just nullfs. Well too bad ezjail is massively using links, at least for basejail, and sometime for port trees or perl setup depending which guide you are using as your reference. During the thread he pretty much bashed anyone who tried to tell him that no amount of jail/ezjail/nullfs/journal screw up could have resulted in the entire content of the jails being moved into another completely unrelated directory node. If one jail had moved it would already have been extraordinary, with a probability of it happening so cleanly that fsck would find nothing already magnitude of order above the chances of winning the national lottery. But all of them ? Not a chance. He finally admitted that he had very little knowledge about UFS and fsck, but still managed to do it in a quite offensive way. That was basically the point were I decided to stop to try to help him. I think others felt the same. This problem is quite interesting in itself, and I think a lot of the most talented people on this list would have been on it but were repelled by the attitude. On the other hand Alejandro Imass pretty much jumped on anything that would be a third party interaction. From someone hacking into his box to a potential nullfs bug that might result in a PR. Now the thing is that EZJail make use of the system immutable flag quite a lot for its config file, resulting in quite a lot of file being impossible to delete or move unless the box is running at kern_secure_level 0. This renders the whole jails moved on their own theory even more improbable. After so much ranting, I would feel bad not to try to help a little : Here are the facts : - In a jail, MySQL was grabbing all the CPU and making the box non responsive. This is due to TigerCRM making requests to a too huge database. - The jail was working - Unless all the data were in memory at this time (unprobable), it means that access path/nullfs/EZJail were OK at this time. - After a force reboot
Re: UFS Crash and directories now missing
Hi, On Monday 30 April 2012 22:38:13 Alejandro Imass wrote: On Mon, Apr 30, 2012 at 7:36 AM, Robert Bonomi bon...@mail.r-bonomi.com wrote: A competennt, not stupid, sysadmin would know these things. And not 'remove all doubt' (in the words of Abraham Lincoln), by raising such nonsense questions. Postulating the right combination of _unrelated_ failures, virtually *anything* can happen. cf. Nasal Monnkeys. the more than 14 years I have been participating in ANY mailing list. oh, you have not been in many then. Or you missed the fine prints there. Just ignore it. Erich ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
On Sun, 29 Apr 2012 00:26:50 -0700, per...@pluto.rain.com wrote: Alejandro Imass a...@p2ee.org wrote: 3) the directories were moved at reboot by journal recovery, fsck or something else I think it's *extremely* unlikely that fsck was involved, because it just doesn't do things like that. The point is: fsck moving directories looks different. In case inodes get de-connected (their reference entries on level n-1 are gone, their data on level n is still present), fsck will access the lost+found/ directory in the corresponding partition's root directory (or create it, if not present) and write _new_ directory entries with the inode as their name, because that's the only naming information possible (as the original names on n-1 aren't accessible anymore). So those directories will have names like #177628676/ and they _can_ contain subtrees full of data, _including_ names from levels n+1 and onward. Files also are named #4767667892 and their names can _maybe_ identified from their content (the file command is helpful, and if they are textfiles containing a CVS or other revision control system data tag, it's possible to find out what they've been in their previous life). However, as it has been explained, fsck will _not_ do so unless being _allowed explicitely_ to do that kind of MODIFICATION to the file system. Flags like -yf can do that, but they are _not_ the default. This is due to the fact that _any_ critical modification of file systems requires the _responsible administrator_ to give permission. -- Polytropon Magdeburg, Germany Happy FreeBSD user since 4.0 Andra moi ennepe, Mousa, ... ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
On Sun, Apr 29, 2012 at 4:37 AM, Polytropon free...@edvax.de wrote: On Sun, 29 Apr 2012 00:26:50 -0700, per...@pluto.rain.com wrote: Alejandro Imass a...@p2ee.org wrote: 3) the directories were moved at reboot by journal recovery, fsck or something else I think it's *extremely* unlikely that fsck was involved, because it just doesn't do things like that. The point is: fsck moving directories looks different. In case inodes get de-connected (their reference entries on level n-1 are gone, their data on level n is still present), fsck will access the lost+found/ directory in the corresponding partition's root directory (or create it, if not present) and write _new_ directory entries with the inode as their name, because that's the only naming information possible (as the original names on n-1 aren't accessible anymore). So those directories will have names like #177628676/ and they _can_ contain subtrees full of data, _including_ names from levels n+1 and onward. Files also are named #4767667892 and their names can _maybe_ identified from their content (the file command is helpful, and if they are textfiles containing a CVS or other revision control system data tag, it's possible to find out what they've been in their previous life). However, as it has been explained, fsck will _not_ do so unless being _allowed explicitely_ to do that kind of MODIFICATION to the file system. Flags like -yf can do that, but they are _not_ the default. This is due to the fact that _any_ critical modification of file systems requires the _responsible administrator_ to give permission. OK, so fsck couldn't have done this. Besides fsck reported the fs as clean so I have to conclude as others have commented that it must have been a mv I've been looking at the logs very carefully and trying to make sense of this. There is a possibility that it could have been an attack because we enabled ftp.proxy so that some clients could upload stuff to their jails using ftp. So I was initially wrong in my assessment because on this particular server we are running a service outside of jails and it's this ftp.proxy that was suppose to be a temporary solution but I guess we never got around to fixing this. The ftp.proxy is started via inetd like so: ftpstream tcp nowait nobody /usr/local/sbin/ftp.proxy ftp.proxy -e And there was a log of a couple of ftp connections the same day this happened, the ONLY 3 messages before the reboot at about 6 pm and they were NOT from any of our customers. Here are the log entries: Apr 27 05:54:37 nune ftp.proxy[2726]: connected to client: host-46-50-183-5.bbcustomer.zsttk.net, interface= 207.158.52.74:21 Apr 27 05:54:37 nune ftp.proxy[2726]: info: monitor mode: off, ccp: unset Apr 27 05:54:38 nune ftp.proxy[2726]: -ERR: missing hostname Apr 27 18:55:42 nune syslogd: kernel boot file is /boot/kernel/kernel OK. So let's suppose ftp.proxy is the culprit is there any way the could have done the mv by cracking ftp and ftp.proxy ?? I have of course disabled the ftp and I am now thinking that another possibility or combination by also using the ftp proxy on the http-proxy jail, that is, the jail that swallowed the other jails. The http-proxy jails was also running apache ftp proxy. So the question now becomes: could a break in ftp.proxy coupled with Apache ftp proxy have caused the http-proxy jails to have swallowed all the other jails into it's configuration directory?? -- Alejandro Imass ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
Alejandro Imass ait at p2ee.org writes: ... And there was a log of a couple of ftp connections the same day this happened, the ONLY 3 messages before the reboot at about 6 pm and they were NOT from any of our customers. Here are the log entries: Apr 27 05:54:37 nune ftp.proxy[2726]: connected to client: host-46-50-183-5.bbcustomer.zsttk.net, interface= 207.158.52.74:21 Apr 27 05:54:37 nune ftp.proxy[2726]: info: monitor mode: off, ccp: unset Apr 27 05:54:38 nune ftp.proxy[2726]: -ERR: missing hostname Apr 27 18:55:42 nune syslogd: kernel boot file is /boot/kernel/kernel ... What you should do right now is to get some recent general or security cd/dvd with chkrootkit and rkhunter and run them from that external read-only media. I would also suggest that you look over config files of all packages involved. jb ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
On Sun, Apr 29, 2012 at 1:15 PM, jb jb.1234a...@gmail.com wrote: Alejandro Imass ait at p2ee.org writes: ... And there was a log of a couple of ftp connections the same day this happened, the ONLY 3 messages before the reboot at about 6 pm and they were NOT from any of our customers. Here are the log entries: Apr 27 05:54:37 nune ftp.proxy[2726]: connected to client: host-46-50-183-5.bbcustomer.zsttk.net, interface= 207.158.52.74:21 Apr 27 05:54:37 nune ftp.proxy[2726]: info: monitor mode: off, ccp: unset Apr 27 05:54:38 nune ftp.proxy[2726]: -ERR: missing hostname Apr 27 18:55:42 nune syslogd: kernel boot file is /boot/kernel/kernel ... What you should do right now is to get some recent general or security cd/dvd with chkrootkit and rkhunter and run them from that external read-only media. I would also suggest that you look over config files of all packages involved. jb Thanks! Will do, but I don't know of any FreeBSD and/or derived distros for security. Or can I use any Linux security distro? I remember reading about some trouble of Linux chkrootkit on FBSD Thanks, -- Alejandro ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
Alejandro Imass ait at p2ee.org writes: ... What you should do right now is to get some recent general or security cd/dvd with chkrootkit and rkhunter and run them from that external read-only media. I would also suggest that you look over config files of all packages involved. jb Thanks! Will do, but I don't know of any FreeBSD and/or derived distros for security. Or can I use any Linux security distro? I remember reading about some trouble of Linux chkrootkit on FBSD It looks like you have only one choice with prebuilt rkhunter package only: http://www.freebsd.org/releases/9.0R/announce.html dvd1 This contains everything necessary to install the base FreeBSD operating system, a collection of pre-built packages aimed at getting a graphical workstation up and running. It also supports booting into a livefs based rescue mode. This should be all you need if you can burn and use DVD-sized media. ftp://ftp.freebsd.org/pub/FreeBSD/ports/packages/security/ rkhunter-1.3.8_1.tbz04/18/1218:56:00 With regard to verification of config files - you said you got backups (those pre-incident would be best) and you have the incident-time files, so do a diff on dirs (in particular /etc and /usr/local/etc) jb ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
Hi, On Monday 30 April 2012 02:02:41 jb wrote: Alejandro Imass ait at p2ee.org writes: ... What you should do right now is to get some recent general or security cd/dvd with chkrootkit and rkhunter and run them from that external read-only media. I would also suggest that you look over config files of all packages involved. jb Thanks! Will do, but I don't know of any FreeBSD and/or derived distros for security. Or can I use any Linux security distro? I remember reading about some trouble of Linux chkrootkit on FBSD It looks like you have only one choice with prebuilt rkhunter package only: http://www.freebsd.org/releases/9.0R/announce.html dvd1 This contains everything necessary to install the base FreeBSD operating system, a collection of pre-built packages aimed at getting a graphical workstation up and running. It also supports booting into a livefs based rescue mode. This should be all you need if you can burn and use DVD-sized media. ftp://ftp.freebsd.org/pub/FreeBSD/ports/packages/security/ rkhunter-1.3.8_1.tbz 04/18/1218:56:00 With regard to verification of config files - you said you got backups (those pre-incident would be best) and you have the incident-time files, so do a diff on dirs (in particular /etc and /usr/local/etc) I would burn the backup of these files to an optical disk, start the system and do a diff as the first step. The system can be started from an USB drive (take the 9.0 installation image) or DVD. Of course, rkhunter can be started in the second step. Erich ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
On Sun, Apr 29, 2012 at 11:49 PM, Erich Dollansky er...@alogreentechnologies.com wrote: Hi, On Monday 30 April 2012 02:02:41 jb wrote: Alejandro Imass ait at p2ee.org writes: ... What you should do right now is to get some recent general or security cd/dvd with chkrootkit and rkhunter and run them from that external read-only media. I would also suggest that you look over config files of all packages involved. jb Thanks! Will do, but I don't know of any FreeBSD and/or derived distros for security. Or can I use any Linux security distro? I remember reading about some trouble of Linux chkrootkit on FBSD It looks like you have only one choice with prebuilt rkhunter package only: http://www.freebsd.org/releases/9.0R/announce.html dvd1 This contains everything necessary to install the base FreeBSD operating system, a collection of pre-built packages aimed at getting a graphical workstation up and running. It also supports booting into a livefs based rescue mode. This should be all you need if you can burn and use DVD-sized media. ftp://ftp.freebsd.org/pub/FreeBSD/ports/packages/security/ rkhunter-1.3.8_1.tbz 04/18/12 18:56:00 With regard to verification of config files - you said you got backups (those pre-incident would be best) and you have the incident-time files, so do a diff on dirs (in particular /etc and /usr/local/etc) I would burn the backup of these files to an optical disk, start the system and do a diff as the first step. The system can be started from an USB drive (take the 9.0 installation image) or DVD. Of course, rkhunter can be started in the second step. ran both, found nothing Back to theory on how the http-proxy jail 'swallowed' all the other jails including the basejail. I noticed that jail had a not so old bug in 2010 FBSD 8.0 which quote The jail(8) utility does not change the current working directory while imprisoning. The current working directory can be accessed by its descendants. /quote Reference: http://security.freebsd.org/advisories/FreeBSD-SA-10:04.jail.asc Given that EzJail uses a single basejail and links/mounts stuff in the child jails it would seem plausible (regression?) that somehow any jail could access other jails' files, or that _maybe_ in an event of crash the nullsfs mounts confuse the system somehow when fsck restores or the journal is recovered. Whatever the cause, it actually happened and I have already ruled out just about anything. It doesn't seem to have been an attack, it surely wasn't me, and EzJail author agrees it was not the EzJail scripts. So maybe nullfs and journaling, or crash + nullfs + journaling, could cause something like this to happen? Maybe journal has some confusion on restoring the nullfs view of the directories or something after bad crash like this one?? ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
On Sat, Apr 28, 2012 at 1:43 AM, Wojciech Puchar woj...@wojtek.tensor.gdynia.pl wrote: All the jails wound up in the /usr/local/etc/apache22 of the only surviving jail which is the http proxy to all the other jails. Right before the server crashed I noticed MySQL at 100% o several CPUs and the server was on it's knees, so I'm wondering was this an attack? is it possible that Apache or MySQL moved the files?? I mean the jails are there, I'm even backing them up right now but how did these directories move here? Anybody has ANY logical explanation??? 99% - someone did moved them. 1% - hardware problem possibly memory. without this there is no way for directory to be accidentally moved I somewhat agree, but it wasn't a person. I am the only administrator, the only one with root access. The jails were effectively moved to the /usr/local/etc/apache22 of the single that survived at the top level. I'm thinking something between mount, EzJail, the journal and the way MySQL created a great deal of head contention, so something must have gotten corrupted at the directory level like you state, but the strange part is no _data_ corruption as such, because I was able to physically archive the jails, move them to the correct directory and archived them all with ezjail-admin to a different disk. I was thinking of formatting the jails drive, but after all this disk activity and no errors, and everything booted up correctly, I am not so sure now that it's needed it. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
I somewhat agree, but it wasn't a person. I am the only administrator, the only one with root access. The jails were effectively moved to the /usr/local/etc/apache22 of the single that survived at the top level. I'm thinking something between mount, EzJail, the journal and the way MySQL created a great deal of head contention, so something must have gotten corrupted at the directory level like you state, but the strange part is no _data_ corruption as such, because I was able to physically archive the jails, move them to the correct directory and no matter what you do FreeBSD DOES NOT ramdomly move directories. if you are sure you didn't move it yourself then it must be machine hardware problem but still unlikely. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
On Sat, Apr 28, 2012 at 3:22 AM, Wojciech Puchar woj...@wojtek.tensor.gdynia.pl wrote: I somewhat agree, but it wasn't a person. I am the only administrator, the only one with root access. The jails were effectively moved to the /usr/local/etc/apache22 of the single that survived at the top level. I'm thinking something between mount, EzJail, the journal and the way MySQL created a great deal of head contention, so something must have gotten corrupted at the directory level like you state, but the strange part is no _data_ corruption as such, because I was able to physically archive the jails, move them to the correct directory and no matter what you do FreeBSD DOES NOT ramdomly move directories. if you are sure you didn't move it yourself then it must be machine hardware problem but still unlikely. After a little more research, ___it it NOT unlikely at all___ that under high distress and a hard boot, UFS could have somehow corrupted the directory structure, whilst maintaining the data intact. From what I've learned so far, UFS is actually divided into 2 layers: one that controls the directory structure and metadata and a lower layer containing the data, so the directories being screwed up and the data intact it is actually quite possible. What I'm trying to do is figure out is how it happened, and try prevent it from happening again, so instead of dismissing it as impossibility, I think we all should spend a little time figuring out how these things can happen and determine how it can be prevented or reduced. Should you find your neighbor's beard catch fire, it's wise to soak one's own -- Alejandro ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
Alejandro Imass aim...@yabarana.com wrote: On Sat, Apr 28, 2012 at 3:22 AM, Wojciech Puchar woj...@wojtek.tensor.gdynia.pl wrote: I somewhat agree, but it wasn't a person. I am the only administrator, the only one with root access. The jails were effectively moved to the /usr/local/etc/apache22 of the single that survived at the top level. I'm thinking something between mount, EzJail, the journal and the way MySQL created a great deal of head contention, so something must have gotten corrupted at the directory level like you state, but the strange part is no _data_ corruption as such, because I was able to physically archive the jails, move them to the correct directory and no matter what you do FreeBSD DOES NOT ramdomly move directories. if you are sure you didn't move it yourself then it must be machine hardware problem but still unlikely. After a little more research, ___it it NOT unlikely at all___ that under high distress and a hard boot, UFS could have somehow corrupted the directory structure, whilst maintaining the data intact. This is techically accurate, *BUT* the specifics of the quote corruption unquote in the case under discussion make it *EXTREMELY* unlikely that this is what happened. 99.99+++% of all UFS filesystem corruption' issues are the result of a system crash _between_ the time cached 'meta-data' is updated in memory and that data is flushed to disk (a deferred write). The second most common (and vanishingly rare) failure mode is a powerfail _as_ a sector of disk is being written -- resulting in 'garbage data' being written to disk. The next possibility is 'cosmic rays'. If running on 'cheap' hardware (i.e., without 'ECC' memory), this can cause a *SINGLE-BIT* error in data being output. The fact that the 'corrupted' filesystem passed fsck -without- any reported errors shows that everything in the filesystem meta-data was consistent Given *that*, there are precisely *TWO* ways that the 'results' that have been reported could have happened. 1) Something did a mv(2) of the various jail directories 'from' their original location to the 'apache' diretory. This involves simply *copying* the diretory entry from the jail's 'parent directory' to the apache directory, and then marking the entry in the original parent as 'unused'. Nothing other than the directory whre the jail 'used to live', and the directory 'where it was found' are touched. This occured _through_ the system 'mv' function, so all the normal 'housekeeping' was done properly. 2) it was -not- done though mv(2) -- but that requires that a whole *series* of corruptions of the filesystem, _ALL_ of which had to occur in 'exactly' the right way. They are: 1) The -size- (filesystem metadata) of the orignal parent directory had to be changed to reflect the smaller size. 2) the 'indirect block' info for the original parent directory had to be changed to reflect the absense of the block(s) that are no longer part of that file. 3) the _size_ of the Apache directory had to be increased to reflect the additional block(s) that are now part o that directory. 4) the 'indirect block' info for the apache directory has to be changed to reflect the presense of the new block(s) that are now part of that file. This requires multiple -hundreds- of bits 'in error', in a minimum of FOUR separate disk locations. A -single- failure simply *CANNOT* cause all of this. The probability of a random single-bit error in a gigabyte of RAM is on the order of one such occurance in six months. The odds of having multiple *simultaneous* errors is the probability of a single-bit error raised to the power of the number of bits in error. e.g. the probability of a simultaneous 10-bit radom error is roughly 1 in 30 million years. The odds of it being a -specific- ten bits out of that gigabyte is preposterously small. The odds of the required specific _multiple-hundreds_ of bits in error occuringis (conservatively) 1 in (30 million years)**50 * ((2**30)!) / ((2^9)!) The first factor, alone, is over 7.1E373 years. I think it is safe to conclude that the probabilities -greatly- favor alternative #1. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
On Sat, Apr 28, 2012 at 11:39 AM, Robert Bonomi bon...@mail.r-bonomi.com wrote: Alejandro Imass aim...@yabarana.com wrote: On Sat, Apr 28, 2012 at 3:22 AM, Wojciech Puchar woj...@wojtek.tensor.gdynia.pl wrote: I somewhat agree, but it wasn't a person. I am the only administrator, the only one with root access. The jails were effectively moved to the /usr/local/etc/apache22 of the single that survived at the top level. I'm thinking something between mount, EzJail, the journal and the way MySQL created a great deal of head contention, so something must have gotten corrupted at the directory level like you state, but the strange part is no _data_ corruption as such, because I was able to physically archive the jails, move them to the correct directory and no matter what you do FreeBSD DOES NOT ramdomly move directories. if you are sure you didn't move it yourself then it must be machine hardware problem but still unlikely. After a little more research, ___it it NOT unlikely at all___ that under high distress and a hard boot, UFS could have somehow corrupted the directory structure, whilst maintaining the data intact. This is techically accurate, *BUT* the specifics of the quote corruption unquote in the case under discussion make it *EXTREMELY* unlikely that this is what happened. 99.99+++% of all UFS filesystem corruption' issues are the result of a system crash _between_ the time cached 'meta-data' is updated in memory and that data is flushed to disk (a deferred write). The second most common (and vanishingly rare) failure mode is a powerfail _as_ a sector of disk is being written -- resulting in 'garbage data' being written to disk. The next possibility is 'cosmic rays'. If running on 'cheap' hardware (i.e., without 'ECC' memory), this can cause a *SINGLE-BIT* error in data being output. The fact that the 'corrupted' filesystem passed fsck -without- any reported errors shows that everything in the filesystem meta-data was consistent Given *that*, there are precisely *TWO* ways that the 'results' that have been reported could have happened. 1) Something did a mv(2) of the various jail directories 'from' their original location to the 'apache' diretory. This involves simply *copying* the diretory entry from the jail's 'parent directory' to the apache directory, and then marking the entry in the original parent as 'unused'. Nothing other than the directory whre the jail 'used to live', and the directory 'where it was found' are touched. This occured _through_ the system 'mv' function, so all the normal 'housekeeping' was done properly. 2) it was -not- done though mv(2) -- but that requires that a whole *series* of corruptions of the filesystem, _ALL_ of which had to occur in 'exactly' the right way. They are: [...] I think it is safe to conclude that the probabilities -greatly- favor alternative #1. OK. So after your comments and further research I concur with you on the mv but if it wasn't a human, then this might be exposing a serious security flaw in the jail system or the way EzJail implements it. The whole point of using jails is to protect things like this from happening. Given that the only jail that survived was the front-end Apache Web server/reverse proxy, then it is also safe to suspect the apache (or other) process running on it was able to perform a mv of the rest of the jails to it's own /usr/local/etc/apache22 directory. Is there no possibility is that after the system crash, the journal recocery process and/or fsck could have moved this directories ? Thanks, -- Alejandro ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
On Sat, Apr 28, 2012 at 12:36 PM, Alejandro Imass aim...@yabarana.com wrote: On Sat, Apr 28, 2012 at 11:39 AM, Robert Bonomi bon...@mail.r-bonomi.com wrote: Alejandro Imass aim...@yabarana.com wrote: On Sat, Apr 28, 2012 at 3:22 AM, Wojciech Puchar woj...@wojtek.tensor.gdynia.pl wrote: I somewhat agree, but it wasn't a person. I am the only administrator, the only one with root access. The jails were effectively moved to the /usr/local/etc/apache22 of the single that survived at the top level. I'm thinking something between mount, EzJail, the journal and the way MySQL created a great deal of head contention, so something must have gotten corrupted at the directory level like you state, but the strange part is no _data_ corruption as such, because I was able to physically archive the jails, move them to the correct directory and no matter what you do FreeBSD DOES NOT ramdomly move directories. if you are sure you didn't move it yourself then it must be machine hardware problem but still unlikely. After a little more research, ___it it NOT unlikely at all___ that under high distress and a hard boot, UFS could have somehow corrupted the directory structure, whilst maintaining the data intact. This is techically accurate, *BUT* the specifics of the quote corruption unquote in the case under discussion make it *EXTREMELY* unlikely that this is what happened. 99.99+++% of all UFS filesystem corruption' issues are the result of a system crash _between_ the time cached 'meta-data' is updated in memory and that data is flushed to disk (a deferred write). The second most common (and vanishingly rare) failure mode is a powerfail _as_ a sector of disk is being written -- resulting in 'garbage data' being written to disk. The next possibility is 'cosmic rays'. If running on 'cheap' hardware (i.e., without 'ECC' memory), this can cause a *SINGLE-BIT* error in data being output. The fact that the 'corrupted' filesystem passed fsck -without- any reported errors shows that everything in the filesystem meta-data was consistent Given *that*, there are precisely *TWO* ways that the 'results' that have been reported could have happened. 1) Something did a mv(2) of the various jail directories 'from' their original location to the 'apache' diretory. This involves simply *copying* the diretory entry from the jail's 'parent directory' to the apache directory, and then marking the entry in the original parent as 'unused'. Nothing other than the directory whre the jail 'used to live', and the directory 'where it was found' are touched. This occured _through_ the system 'mv' function, so all the normal 'housekeeping' was done properly. 2) it was -not- done though mv(2) -- but that requires that a whole *series* of corruptions of the filesystem, _ALL_ of which had to occur in 'exactly' the right way. They are: [...] I think it is safe to conclude that the probabilities -greatly- favor alternative #1. OK. So after your comments and further research I concur with you on the mv but if it wasn't a human, then this might be exposing a serious security flaw in the jail system or the way EzJail implements it. The whole point of using jails is to protect things like this from happening. Given that the only jail that survived was the front-end Apache Web server/reverse proxy, then it is also safe to suspect the apache (or other) process running on it was able to perform a mv of the rest of the jails to it's own /usr/local/etc/apache22 directory. Is there no possibility is that after the system crash, the journal recocery process and/or fsck could have moved this directories ? Also note that even the EzJail basejail was moved also, so it could be a security hole in the way nullfs is used or in nullfs itself. but the curious thing is that the basejail is supposed to be mounted read-only so how did that get moved to the http-proxy jail?? That is why I suspect it could have been something in the boot process like the journal recovery, fsck or something else with that kind of privilege and when the EzJail filesystems were unmounted. -- Alejandro ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
Alejandro Imass aim...@yabarana.com wrote: On Sat, Apr 28, 2012 at 11:39 AM, Robert Bonomi bon...@mail.r-bonomi.com wrote: Alejandro Imass aim...@yabarana.com wrote: After a little more research, ___it it NOT unlikely at all___ that under high distress and a hard boot, UFS could have somehow corrupted the directory structure, whilst maintaining the data intact. This is techically accurate, *BUT* the specifics of the quote corruption unquote in the case under discussion make it *EXTREMELY* unlikely that this is what happened. 99.99+++% of all UFS filesystem corruption' issues are the result of a system crash _between_ the time cached 'meta-data' is updated in memory and that data is flushed to disk (a deferred write). The second most common (and vanishingly rare) failure mode is a powerfail _as_ a sector of disk is being written -- resulting in 'garbage data' being written to disk. The next possibility is 'cosmic rays'. If running on 'cheap' hardware (i.e., without 'ECC' memory), this can cause a *SINGLE-BIT* error in data being output. The fact that the 'corrupted' filesystem passed fsck -without- any reported errors shows that everything in the filesystem meta-data was consistent [...] I think it is safe to conclude that the probabilities -greatly- favor alternative #1. OK. So after your comments and further research I concur with you on the mv but if it wasn't a human, then this might be exposing a serious security flaw in the jail system or the way EzJail implements it. BOGON ALERT!!! Jails only prevent stuff -inside- the jail from affecting stuff outside the jail. They do *NOT* prevent stuff 'oustide the jail' from affecting stuff INSIDE the jail. For any fool-proof system, there exists a *sufficiently*determined* fool capable of breaking it. The whole point of using jails is to protect things like this from happening. FALSE TO FACT. Given that the only jail that survived was the front-end Apache Web server/reverse proxy, then it is also safe to suspect the apache (or other) process running on it was able to perform a mv of the rest of the jails to it's own /usr/local/etc/apache22 directory. FALSE TO FACT. Is there no possibility is that after the system crash, the journal recocery process and/or fsck could have moved this directories ? Anything is 'possible' -- c.f. 'nasal monkeys'. HOWEVER, if, for example, you would bother to examine the source code for fsck you would discover that it doesn't do -anything- 'significant' without ASKING FIRST. You reported it didn't find any problems -- not even anay of the 'petty' ones it will correct w/o asking -if- the '-p' option is specified. Journal revovery _could_, 'theoretically' have done it -- *IF* something else did the 'mv' just before the crash, and that operation was journaled, but not yet committed to disk at the time of the crash. However, on a standard UFS filesystem, filesystem metadata updates are written 'synchronously', which should eliminate _that_ wild, unfounded, speculaction. You sir, don't know what you don't know, and much of what you think you know is incorrect. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
On Sat, Apr 28, 2012 at 1:31 PM, Robert Bonomi bon...@mail.r-bonomi.com wrote: Alejandro Imass aim...@yabarana.com wrote: On Sat, Apr 28, 2012 at 11:39 AM, Robert Bonomi bon...@mail.r-bonomi.com wrote: Alejandro Imass aim...@yabarana.com wrote: After a little more research, ___it it NOT unlikely at all___ that under high distress and a hard boot, UFS could have somehow corrupted the directory structure, whilst maintaining the data intact. This is techically accurate, *BUT* the specifics of the quote corruption unquote in the case under discussion make it *EXTREMELY* unlikely that this is what happened. 99.99+++% of all UFS filesystem corruption' issues are the result of a system crash _between_ the time cached 'meta-data' is updated in memory and that data is flushed to disk (a deferred write). The second most common (and vanishingly rare) failure mode is a powerfail _as_ a sector of disk is being written -- resulting in 'garbage data' being written to disk. The next possibility is 'cosmic rays'. If running on 'cheap' hardware (i.e., without 'ECC' memory), this can cause a *SINGLE-BIT* error in data being output. The fact that the 'corrupted' filesystem passed fsck -without- any reported errors shows that everything in the filesystem meta-data was consistent [...] I think it is safe to conclude that the probabilities -greatly- favor alternative #1. OK. So after your comments and further research I concur with you on the mv but if it wasn't a human, then this might be exposing a serious security flaw in the jail system or the way EzJail implements it. BOGON ALERT!!! I admit my ignorance on how the filesystem works but I don't think your condescending remarks add a lot of value. The issue here is this actually happened and there is a flaw somewhere other than the stupid administrator did it. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
On Sat, 28 Apr 2012 13:52:02 -0400, Alejandro Imass wrote: On Sat, Apr 28, 2012 at 1:31 PM, Robert Bonomi bon...@mail.r-bonomi.com wrote: Alejandro Imass aim...@yabarana.com wrote: On Sat, Apr 28, 2012 at 11:39 AM, Robert Bonomi bon...@mail.r-bonomi.com wrote: Alejandro Imass aim...@yabarana.com wrote: After a little more research, ___it it NOT unlikely at all___ that under high distress and a hard boot, UFS could have somehow corrupted the directory structure, whilst maintaining the data intact. This is techically accurate, *BUT* the specifics of the quote corruption unquote in the case under discussion make it *EXTREMELY* unlikely that this is what happened. 99.99+++% of all UFS filesystem corruption' issues are the result of a system crash _between_ the time cached 'meta-data' is updated in memory and that data is flushed to disk (a deferred write). The second most common (and vanishingly rare) failure mode is a powerfail _as_ a sector of disk is being written -- resulting in 'garbage data' being written to disk. The next possibility is 'cosmic rays'. If running on 'cheap' hardware (i.e., without 'ECC' memory), this can cause a *SINGLE-BIT* error in data being output. The fact that the 'corrupted' filesystem passed fsck -without- any reported errors shows that everything in the filesystem meta-data was consistent [...] I think it is safe to conclude that the probabilities -greatly- favor alternative #1. OK. So after your comments and further research I concur with you on the mv but if it wasn't a human, then this might be exposing a serious security flaw in the jail system or the way EzJail implements it. BOGON ALERT!!! I admit my ignorance on how the filesystem works but I don't think your condescending remarks add a lot of value. The issue here is this actually happened and there is a flaw somewhere other than the stupid administrator did it. If you search the archives of this list, you'll find my _first_ post to that list: I've had a similar problem, df shows data must be there after crash (panic - reboot - fsck trouble), but files aren't there (even _not_ in lost+found). It's quite possible that in _exceptional_ moments this can happen. The fsck program is intended to repair the most typical file system faults, but nothing complicated will be done without interaction: Altering data on disk will _always_ involve the responsible (!) admin to check if it is really intended to do so. There can be many reasons. I've never found out what was the reason for the trouble I've had. Some years ago, I found a make failing because /uss/src/blah... something not found, and a quick memtest revealed the secret: defective RAM module that caused a bit error, and the difference between r and s is just one bit. Replaced the module - everything worked. Mean soldering rays from outer space. :-) You'll find many useful forensic tools in the ports collection that might help locate lost data (quotes intended as long as the data is still on the disk). The more complex your setting is (e. g. striped disks, or ZFS), this can be nearly impossible. Plain old UFS can sometimes be your saviour (but BACKUP should be your real friend). -- Polytropon Magdeburg, Germany Happy FreeBSD user since 4.0 Andra moi ennepe, Mousa, ... ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
On Sat, Apr 28, 2012 at 2:01 PM, Polytropon free...@edvax.de wrote: On Sat, 28 Apr 2012 13:52:02 -0400, Alejandro Imass wrote: On Sat, Apr 28, 2012 at 1:31 PM, Robert Bonomi bon...@mail.r-bonomi.com wrote: Alejandro Imass aim...@yabarana.com wrote: On Sat, Apr 28, 2012 at 11:39 AM, Robert Bonomi bon...@mail.r-bonomi.com wrote: Alejandro Imass aim...@yabarana.com wrote: After a little more research, ___it it NOT unlikely at all___ that under high distress and a hard boot, UFS could have somehow corrupted the directory structure, whilst maintaining the data intact. This is techically accurate, *BUT* the specifics of the quote corruption unquote in the case under discussion make it *EXTREMELY* unlikely that this is what happened. 99.99+++% of all UFS filesystem corruption' issues are the result of a system crash _between_ the time cached 'meta-data' is updated in memory and that data is flushed to disk (a deferred write). The second most common (and vanishingly rare) failure mode is a powerfail _as_ a sector of disk is being written -- resulting in 'garbage data' being written to disk. The next possibility is 'cosmic rays'. If running on 'cheap' hardware (i.e., without 'ECC' memory), this can cause a *SINGLE-BIT* error in data being output. The fact that the 'corrupted' filesystem passed fsck -without- any reported errors shows that everything in the filesystem meta-data was consistent [...] I think it is safe to conclude that the probabilities -greatly- favor alternative #1. OK. So after your comments and further research I concur with you on the mv but if it wasn't a human, then this might be exposing a serious security flaw in the jail system or the way EzJail implements it. BOGON ALERT!!! I admit my ignorance on how the filesystem works but I don't think your condescending remarks add a lot of value. The issue here is this actually happened and there is a flaw somewhere other than the stupid administrator did it. If you search the archives of this list, you'll find my _first_ post to that list: I've had a similar problem, df shows data must be there after crash (panic - reboot - fsck trouble), but files aren't there (even _not_ in lost+found). It's quite possible that in _exceptional_ moments this can happen. The fsck program is intended to repair the most typical file system faults, but nothing complicated will be done without interaction: Altering data on disk will _always_ involve the responsible (!) admin to check if it is really intended to do so. There can be many reasons. I've never found out what was the [...] that might help locate lost data (quotes intended as long as the data is still on the disk). The more complex your setting is (e. g. striped disks, or ZFS), this can be nearly impossible. Plain old UFS can sometimes be your saviour (but BACKUP should be your real friend). Thanks for your reply. I can't figure out how there was no data loss and yet the directories moved just like that. We have nightly backups and it's one of the features we love about EzJail and it's archive feature. The base system sits on another disk entirely and it's pristine, we don't install anything except the basic system on the system disk and the other disk is exclusively divided in jails, so the possibility of an outside process doing the mv is unlikely. Everything point to that something or someone executed a mv but how was this done? or if there is a potential problem and could happen again. And contrary to other comments here, and my admitted ignorance, I believe there are actually 3 possibilities: 1) something inside a jail was able to move the other jails into itself 2) something outside the jails moved the jails 3) the directories were moved at reboot by journal recovery, fsck or something else That is what worries me, is that it wasn't just some random bit or cosmic ray, but the potential of happening again. I am not so sure that it is *impossible* that a jail could affect other jails with EzJail. -- Alejandro ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
On 28/04/2012 19:52, Alejandro Imass wrote: On Sat, Apr 28, 2012 at 1:31 PM, Robert Bonomibon...@mail.r-bonomi.com wrote: Alejandro Imassaim...@yabarana.com wrote: On Sat, Apr 28, 2012 at 11:39 AM, Robert Bonomi bon...@mail.r-bonomi.com wrote: Alejandro Imassaim...@yabarana.com wrote: After a little more research, ___it it NOT unlikely at all___ that under high distress and a hard boot, UFS could have somehow corrupted the directory structure, whilst maintaining the data intact. This is techically accurate, *BUT* the specifics of the quote corruption unquote in the case under discussion make it *EXTREMELY* unlikely that this is what happened. 99.99+++% of all UFS filesystem corruption' issues are the result of a system crash _between_ the time cached 'meta-data' is updated in memory and that data is flushed to disk (a deferred write). The second most common (and vanishingly rare) failure mode is a powerfail _as_ a sector of disk is being written -- resulting in 'garbage data' being written to disk. The next possibility is 'cosmic rays'. If running on 'cheap' hardware (i.e., without 'ECC' memory), this can cause a *SINGLE-BIT* error in data being output. The fact that the 'corrupted' filesystem passed fsck -without- any reported errors shows that everything in the filesystem meta-data was consistent [...] I think it is safe to conclude that the probabilities -greatly- favor alternative #1. OK. So after your comments and further research I concur with you on the mv but if it wasn't a human, then this might be exposing a serious security flaw in the jail system or the way EzJail implements it. BOGON ALERT!!! I admit my ignorance on how the filesystem works but I don't think your condescending remarks add a lot of value. The issue here is this actually happened and there is a flaw somewhere other than the stupid administrator did it. Ok, Not wanting to take any side in what could end up in personal attacks and nasty things being said about any poster genitors but : - Jails are very widely used, in fact it is probably one of the most used functionnality of FreeBSD. Far beyond ZFS, MAC or any of the other nice thingies FreeBSD has. - Jails are very often misused. Though not overly complex, creating a proper jail and upgrading it can sometime be a bit tricky. - Though not entirely devoid of bug and perfect, FreeBSD 8.2 is probably the best thing there is out there when it comes to system stability. It might be lacking some little nooks and cranies when it comes to perfect compliance with obscure standards, it might not behave as expected in some very few situation, but these are extremely rare. FreeBSD 8.2 is very widely used and this is one of the first time I heard of such a problem in jails. Nothing even remotely rings a bell. Take all these information into account and put yourself in our shoes. When reading your problem description, most of us will be inclined to think that you did something wrong. My personnal guess would be that you probably abused ln a bit too much when creating the jails (total shot in the dark here, but it could explain what happened). I don't see how journaling could impact your jails in anyway except if your jails were all extremely new when the crash happened or that the I/O was such that FreeBSD could never sync and commit journal from the time you created your jails to the time where the system crashed. Extremely unlikely. So my question is : where all the jail created properly ? Did you cpdup each and every one of them or were you lazy at some point ? Are all the jails properly declared in rc.conf ? My guess would be that the first jail was created in the right way, but that others were created using cp and ln, resulting in unexpected behaviour in the end. If I am right then the surviving jail would be either the first or the last you created. Jerome Herman ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
On 04/28/2012 11:16 AM, Alejandro Imass wrote: That is what worries me, is that it wasn't just some random bit or cosmic ray, but the potential of happening again. I am not so sure that it is*impossible* that a jail could affect other jails with EzJail. Sorry I'm late to the party. How about contacting EZjail and explaining what has happen:-) it may be a bug? http://erdgeist.org/arts/software/ezjail/#Author ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
Hi, On Saturday 28 April 2012 20:15:25 Alejandro Imass wrote: On Sat, Apr 28, 2012 at 3:22 AM, Wojciech Puchar woj...@wojtek.tensor.gdynia.pl wrote: I somewhat agree, but it wasn't a person. I am the only administrator, the only one with root access. The jails were effectively moved to the /usr/local/etc/apache22 of the single that survived at the top level. I'm thinking something between mount, EzJail, the journal and the way MySQL created a great deal of head contention, so something must have gotten corrupted at the directory level like you state, but the strange part is no _data_ corruption as such, because I was able to physically archive the jails, move them to the correct directory and no matter what you do FreeBSD DOES NOT ramdomly move directories. if you are sure you didn't move it yourself then it must be machine hardware problem but still unlikely. After a little more research, ___it it NOT unlikely at all___ that under high distress and a hard boot, UFS could have somehow corrupted the directory structure, whilst maintaining the data intact. From what I've learned so far, UFS is actually divided into 2 layers: one that controls the directory structure and metadata and a lower layer containing the data, so the directories being screwed up and the data intact it is actually quite possible. What I'm trying to do is figure out is how it happened, and try prevent it from happening again, so instead of dismissing it as impossibility, I think we all should spend a little time figuring out how these things can happen and determine how it can be prevented or reduced. somebody mentioned the links. Did you use links in the jails to access the data? If then the directories of the jails got screwed, the links are gone but the original data is still there. The damaged directory might got fixed during the first reboot after the crash and you never noticed the fix. Erich ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
Alejandro Imass a...@p2ee.org wrote: 3) the directories were moved at reboot by journal recovery, fsck or something else I think it's *extremely* unlikely that fsck was involved, because it just doesn't do things like that. It might move an orphaned directory (or file) to lost+found, but nowhere else. That's in addition to the fact that, as someone already mentioned, it asks before doing anything. I don't know enough about the details of journal recovery to comment on it as a suspect. That is what worries me, is that it wasn't just some random bit or cosmic ray, but the potential of happening again ... Any chance that your base system -- rather than one of the jails -- has somehow been cracked; maybe even that the cracker precipitated the crash? It might be wise to restore the whole system from backup, the base from a moderately old one since it doesn't change anyway, rather than trying to recover. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
On Sat, Apr 28, 2012 at 5:03 PM, Erich Dollansky er...@alogreentechnologies.com wrote: Hi, On Saturday 28 April 2012 20:15:25 Alejandro Imass wrote: On Sat, Apr 28, 2012 at 3:22 AM, Wojciech Puchar woj...@wojtek.tensor.gdynia.pl wrote: I somewhat agree, but it wasn't a person. I am the only administrator, the only one with root access. The jails were effectively moved to the /usr/local/etc/apache22 of the single that survived at the top level. I'm thinking something between mount, EzJail, the journal and the way MySQL created a great deal of head contention, so something must have gotten corrupted at the directory level like you state, but the strange part is no _data_ corruption as such, because I was able to physically archive the jails, move them to the correct directory and no matter what you do FreeBSD DOES NOT ramdomly move directories. if you are sure you didn't move it yourself then it must be machine hardware problem but still unlikely. After a little more research, ___it it NOT unlikely at all___ that under high distress and a hard boot, UFS could have somehow corrupted the directory structure, whilst maintaining the data intact. From what I've learned so far, UFS is actually divided into 2 layers: one that controls the directory structure and metadata and a lower layer containing the data, so the directories being screwed up and the data intact it is actually quite possible. What I'm trying to do is figure out is how it happened, and try prevent it from happening again, so instead of dismissing it as impossibility, I think we all should spend a little time figuring out how these things can happen and determine how it can be prevented or reduced. somebody mentioned the links. Did you use links in the jails to access the data? If then the directories of the jails got screwed, the links are gone but the original data is still there. The damaged directory might got fixed during the first reboot after the crash and you never noticed the fix. Hi Erich, thanks for your reply. I don't know what links you are referring to, but please point me in that direction. I initially suspected that it could have been the journal recovery and/or fsck but as you can see, a couple of people have said this is impossible, but have to admit my ignorance on some specifics of the UFS filesystem, yet out of logic seems like the most plausible explanation. I've been running FBSD since 6.2 and jails since then as well. Today I run 6 public servers in 8.2 with between 15 to 20 jails each and we switched to ezjail last year and use strictly by the book. I do use flavours though, and I may archive and re-create jails with a specific archive but always using ezjail-admin. Since all our servers are 8.2 and all updated the same, I may port jails from one server to the other using the ezjail archive method, but nothing as stupid as someone was suggesting that I was using cp or soft links. I've never had any problems except in _this particular server_ where I have client that has a problem with MySQL and under some conditions it drains the whole server. I suspected corruption of the fs because of all the contention generated by MySQL to the point where it simply hung and had to hard-reboot. I doubt it's hardware because these are relatively new servers Xeon X3370, 8GB RAM, 2 x 150GB 10,000rpm Velociraptor disks. We have the pristine OS in one disk and jails in the other. Nothing runs outside of jails, not even the MTA which runs postfix inside one of the jails. This is the first crash when anything like this has happened in over 6 years running FBSD, and I am surprised as anyone here because of the weirdness of the jail directories moving like that. We had backups of the previous night, but I didn't even use them. The data was all there, intact, just moved inside the only surviving jail, which happens to be the http reverse proxy of all the other jails. If you have any leads as to how this can happen other than cosmic rays I would greatly appreciate it. Thanks! -- Alejandro Erich ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
On Sun, Apr 29, 2012 at 3:26 AM, per...@pluto.rain.com wrote: Alejandro Imass a...@p2ee.org wrote: [...] Any chance that your base system -- rather than one of the jails -- has somehow been cracked; maybe even that the cracker precipitated the crash? It might be wise to restore the whole system from backup, the base from a moderately old one since it doesn't change anyway, rather than trying to recover. There is always that possibility but I strive to keep these servers updated, I block most ap, nigeria and russia ip blocks using updated Wizcrafts' lists, run fail2ban and other security measures. We have a policy of only one password and there are no users or services in the base system other than mine. As I said in another mail I run 6 servers and been runing FBSD for almost 7 years and this is the first time I've seen this happen. -- Alejandro ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
Hi, On Sunday 29 April 2012 08:58:17 Alejandro Imass wrote: On Sat, Apr 28, 2012 at 5:03 PM, Erich Dollansky er...@alogreentechnologies.com wrote: On Saturday 28 April 2012 20:15:25 Alejandro Imass wrote: On Sat, Apr 28, 2012 at 3:22 AM, Wojciech Puchar woj...@wojtek.tensor.gdynia.pl wrote: I somewhat agree, but it wasn't a person. I am the only administrator, the only one with root access. The jails were effectively moved to the /usr/local/etc/apache22 of the single that survived at the top level. I'm thinking something between mount, EzJail, the journal and the way MySQL created a great deal of head contention, so something must have gotten corrupted at the directory level like you state, but the strange part is no _data_ corruption as such, because I was able to physically archive the jails, move them to the correct directory and no matter what you do FreeBSD DOES NOT ramdomly move directories. if you are sure you didn't move it yourself then it must be machine hardware problem but still unlikely. After a little more research, ___it it NOT unlikely at all___ that under high distress and a hard boot, UFS could have somehow corrupted the directory structure, whilst maintaining the data intact. From what I've learned so far, UFS is actually divided into 2 layers: one that controls the directory structure and metadata and a lower layer containing the data, so the directories being screwed up and the data intact it is actually quite possible. What I'm trying to do is figure out is how it happened, and try prevent it from happening again, so instead of dismissing it as impossibility, I think we all should spend a little time figuring out how these things can happen and determine how it can be prevented or reduced. somebody mentioned the links. Did you use links in the jails to access the data? If then the directories of the jails got screwed, the links are gone but the original data is still there. The damaged directory might got fixed during the first reboot after the crash and you never noticed the fix. Hi Erich, thanks for your reply. I don't know what links you are referring to, but please point me in man link They are practical in jails when things are read only. Mark everything read-only and nothing should go wrong. that direction. I initially suspected that it could have been the journal recovery and/or fsck but as you can see, a couple of people I only installed journals on a new machine without any experience there. UFS does normally the job for me. have said this is impossible, but have to admit my ignorance on some specifics of the UFS filesystem, yet out of logic seems like the most plausible explanation. This is not a good reasoning. I have had clients using my own software for years before it crashed with an error which was in there since the start. I've been running FBSD since 6.2 and jails since then as well. Today I run 6 public servers in 8.2 with between 15 to 20 jails each and we switched to ezjail last year and use strictly by the book. I do use flavours though, and I may archive and re-create jails with a specific archive but always using ezjail-admin. Since all our servers are 8.2 and all updated the same, I may port jails from one server to the other using the ezjail archive method, but nothing as stupid as someone was suggesting that I was using cp or soft links. I never used ezjail in real life. I've never had any problems except in _this particular server_ where I have client that has a problem with MySQL and under some conditions it drains the whole server. I suspected corruption of the fs because of all the contention generated by MySQL to the point where it simply hung and had to hard-reboot. I doubt it's hardware because these are relatively new servers Xeon X3370, 8GB RAM, 2 x 150GB 10,000rpm Velociraptor disks. We have the pristine OS in one disk and jails in the other. Nothing runs outside of jails, not even the MTA which runs postfix inside one of the jails. This is the first crash when anything like this has happened in over 6 years running FBSD, and I am surprised as anyone here because of the weirdness of the jail directories moving like that. We had backups of the previous night, but I didn't even use them. The data was all there, intact, just moved inside the only surviving jail, which happens to be the http reverse proxy of all the other jails. If you have any leads as to how this can happen other than cosmic rays I would greatly appreciate it. Check if their are links there after you remade the system. I have also no other idea then. Erich Thanks! -- Alejandro Erich ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
On Sat, Apr 28, 2012 at 10:20 PM, Erich Dollansky erichfreebsdl...@ovitrap.com wrote: Hi, On Sunday 29 April 2012 08:58:17 Alejandro Imass wrote: On Sat, Apr 28, 2012 at 5:03 PM, Erich Dollansky er...@alogreentechnologies.com wrote: [...] Hi Erich, thanks for your reply. I don't know what links you are referring to, but please point me in man link They are practical in jails when things are read only. Mark everything read-only and nothing should go wrong. I though you were referring to something else entirely. No, I don't use soft or hard links with jails, unless EzJail uses them but I doubt it, I think everything like that in EzJail is done by mounting via nullfs. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
UFS Crash and directories now missing
Hi folks, We had a server crash and required a hard reboot. The system is on one disk and another disc mounts /usr/jails and everything runs in jails, pristine base system, and the base system is working perfectly. The second volume, the one with the jails mounted but every jail directory disappeared except one. df still shows the data being used so I'm guessing it's a logical error in the directory structure or something. I unmounted the drive and ran fsck and reported no problems. df shows the data being use so where is the data?? This is FreeBSD 8.2 updated, patched etc. The volume was UFS + Journal Any help is GREATLY appreciated! Thanks! -- Alejandro Imass ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
On Fri, Apr 27, 2012 at 7:52 PM, Alejandro Imass aim...@yabarana.com wrote: Hi folks, We had a server crash and required a hard reboot. The system is on one disk and another disc mounts /usr/jails and everything runs in jails, pristine base system, and the base system is working perfectly. The second volume, the one with the jails mounted but every jail directory disappeared except one. df still shows the data being used so I'm guessing it's a logical error in the directory structure or something. I unmounted the drive and ran fsck and reported no problems. df shows the data being use so where is the data?? OK, so here is an update, maybe someone has some clue here All the jails wound up in the /usr/local/etc/apache22 of the only surviving jail which is the http proxy to all the other jails. Right before the server crashed I noticed MySQL at 100% o several CPUs and the server was on it's knees, so I'm wondering was this an attack? is it possible that Apache or MySQL moved the files?? I mean the jails are there, I'm even backing them up right now but how did these directories move here? Anybody has ANY logical explanation??? Thanks, -- Alejandro Imass This is FreeBSD 8.2 updated, patched etc. The volume was UFS + Journal Any help is GREATLY appreciated! Thanks! -- Alejandro Imass ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
Hi, On Saturday 28 April 2012 09:33:47 Alejandro Imass wrote: On Fri, Apr 27, 2012 at 7:52 PM, Alejandro Imass aim...@yabarana.com wrote: We had a server crash and required a hard reboot. The system is on one disk and another disc mounts /usr/jails and everything runs in jails, pristine base system, and the base system is working perfectly. The second volume, the one with the jails mounted but every jail directory disappeared except one. df still shows the data being used so I'm guessing it's a logical error in the directory structure or something. I unmounted the drive and ran fsck and reported no problems. df shows the data being use so where is the data?? what is du saying? OK, so here is an update, maybe someone has some clue here All the jails wound up in the /usr/local/etc/apache22 of the only surviving jail which is the http proxy to all the other jails. You want to say that all the data you were looking for have been moved to this directory? Right before the server crashed I noticed MySQL at 100% o several CPUs and the server was on it's knees, so I'm wondering was this an attack? is it possible that Apache or MySQL moved the files?? I mean the jails are there, I'm even backing them up right now but how did these directories move here? Anybody has ANY logical explanation??? Journaling is new to me. Could this be the cause? Erich ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
On Fri, Apr 27, 2012 at 11:00 PM, Erich Dollansky er...@alogreentechnologies.com wrote: Hi, On Saturday 28 April 2012 09:33:47 Alejandro Imass wrote: On Fri, Apr 27, 2012 at 7:52 PM, Alejandro Imass aim...@yabarana.com wrote: We had a server crash and required a hard reboot. The system is on one disk and another disc mounts /usr/jails and everything runs in jails, pristine base system, and the base system is working perfectly. The second volume, the one with the jails mounted but every jail directory disappeared except one. df still shows the data being used so I'm guessing it's a logical error in the directory structure or something. I unmounted the drive and ran fsck and reported no problems. df shows the data being use so where is the data?? what is du saying? OK, so here is an update, maybe someone has some clue here All the jails wound up in the /usr/local/etc/apache22 of the only surviving jail which is the http proxy to all the other jails. You want to say that all the data you were looking for have been moved to this directory? EXACTLY THAT. In fact the data is intact and I have already backed-up everything to another disk. Right before the server crashed I noticed MySQL at 100% o several CPUs and the server was on it's knees, so I'm wondering was this an attack? is it possible that Apache or MySQL moved the files?? I mean the jails are there, I'm even backing them up right now but how did these directories move here? Anybody has ANY logical explanation??? Journaling is new to me. Could this be the cause? Maybe so, I have no idea. Maybe it's because EzJail mount volumes with each jail or some other wild explanation. I honestly have never seen this before. I am just glad that UFS was nice enough to keep my data somewhere at least, and after my bad experiences with ZFS I can now say with a lot more certainty that UFS rocks. I mean something got screwed up but the data was not lost. Hope someone can shed some light here.. -- Alejandro Erich ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
something. I unmounted the drive and ran fsck and reported no problems. df shows the data being use so where is the data?? your data is here as df shown usage and fsck see no errors. most probably root directory of that volume got corrupted and subdirs were found and put in lost+found ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UFS Crash and directories now missing
All the jails wound up in the /usr/local/etc/apache22 of the only surviving jail which is the http proxy to all the other jails. Right before the server crashed I noticed MySQL at 100% o several CPUs and the server was on it's knees, so I'm wondering was this an attack? is it possible that Apache or MySQL moved the files?? I mean the jails are there, I'm even backing them up right now but how did these directories move here? Anybody has ANY logical explanation??? 99% - someone did moved them. 1% - hardware problem possibly memory. without this there is no way for directory to be accidentally moved ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
