[Gluster-devel] Performance Translators' Stability and Usefulness

2009-07-03 Thread Gordan Bobic
Just reading through the wiki on this and a few things are unclear, so I'm hoping someone can clarify. 1) readahead - Is there any point in using this on systems where the interconnect <= 1Gb/s? The wiki implies there is no point in this, but doesn't quite state it explicitly. - Is there an

Re: [Gluster-devel] Performance Translators' Stability and Usefulness

2009-07-03 Thread Shehjar Tikoo
Gordan Bobic wrote: Just reading through the wiki on this and a few things are unclear, so I'm hoping someone can clarify. 1) readahead - Is there any point in using this on systems where the interconnect <= 1Gb/s? The wiki implies there is no point in this, but doesn't quite state it explic

Re: [Gluster-devel] Performance Translators' Stability and Usefulness

2009-07-04 Thread Geoff Kassel
Hi Shehjar, I feel I should comment on part of your reply to Gordan's email. > > Finally - which translators are deemed stable (no know issues - > > memory leaks/bloat, crashes, corruption, etc.)? > > We can definitely vouch for a higher degree of stability of the > releases. Otherwise, I dont

Re: [Gluster-devel] Performance Translators' Stability and Usefulness

2009-07-04 Thread Gordan Bobic
Shehjar Tikoo wrote: We can definitely vouch for a higher degree of stability of the releases. Otherwise, I dont think there is any performance translator we can call completely stable/mature because of the roadmap we have for constantly upgrading algorithms, functionality, etc. I'm only talkin

Re: [Gluster-devel] Performance Translators' Stability and Usefulness

2009-07-04 Thread Gordan Bobic
Geoff Kassel wrote: Finally - which translators are deemed stable (no know issues - memory leaks/bloat, crashes, corruption, etc.)? >> We can definitely vouch for a higher degree of stability of the releases. Otherwise, I dont think there is any performance translator we can call completely st

Re: [Gluster-devel] Performance Translators' Stability and Usefulness

2009-07-04 Thread Geoff Kassel
Hi Gordan, > > When will the Gluster team be able to deliver a stable, mature, and > > reliable version of GlusterFS? > > While I can relate to that sentiment to some extent, I think you're a > bit overly harsh. Stability has, in my experience, improved quite a lot > recently. I'm sorry if I come

Re: [Gluster-devel] Performance Translators' Stability and Usefulness

2009-07-04 Thread Gordan Bobic
Geoff Kassel wrote: (If it wasn't for that migrating to another solution would cause considerable, business-destroying downtime for my client base, I would have done so quite some time ago.) There is an argument somewhere in there about deploying things that aren't production ready at time o

Re: [Gluster-devel] Performance Translators' Stability and Usefulness

2009-07-04 Thread Geoff Kassel
Hi Gordan, > There is an argument somewhere in there about deploying things that > aren't production ready at time of deployment. But that's a different > story. GlusterFS was represented as being stable, with its 1.x version number and website spiel. Before the original deployment, I tested ext

Re: [Gluster-devel] Performance Translators' Stability and Usefulness

2009-07-04 Thread Gordan Bobic
Geoff Kassel wrote: Sounds like a lot of effort and micro-downtime compared to a migration to something else. Have you explored other options like PeerFS, GFS and SeznamFS? Or NFS exports with failover rather than Gluster clients, with Gluster only server-to-server? These options are not produ

Re: [Gluster-devel] Performance Translators' Stability and Usefulness

2009-07-04 Thread Geoff Kassel
Hi Gordan, > What is production unready (more than Gluster) about PeerFS or SeznamFS? Well, I'm mostly going by your email comparing these of a few months ago. Your needs are not that dissimilar to mine. I see on the project page for SeznamFS now that there's apparently support for SeznamFS to

Re: [Gluster-devel] Performance Translators' Stability and Usefulness

2009-07-05 Thread Shehjar Tikoo
Thanks Geoff. It is always good to get an external opinion on where we stand. Geoff Kassel wrote: Hi Shehjar, I feel I should comment on part of your reply to Gordan's email. Finally - which translators are deemed stable (no know issues - memory leaks/bloat, crashes, corruption, etc.)? We ca

Re: [Gluster-devel] Performance Translators' Stability and Usefulness

2009-07-05 Thread Geoff Kassel
Hi Shehjar, > I think the QA folks have done some really good work in stabilizing > GlusterFS over the last year or so. The result is there to see in the > 2.0.X releases. You call allowing two critical data corruption bugs through to the final public release in two releases in a row 'good work'

Re: [Gluster-devel] Performance Translators' Stability and Usefulness

2009-07-05 Thread Gordan Bobic
Shehjar Tikoo wrote: Finally - which translators are deemed stable (no know issues - memory leaks/bloat, crashes, corruption, etc.)? We can definitely vouch for a higher degree of stability of the releases. Otherwise, I dont think there is any performance translator we can call completely stab

Re: [Gluster-devel] Performance Translators' Stability and Usefulness

2009-07-05 Thread Gordan Bobic
Geoff Kassel wrote: I've posted logs. I've posted configurations. I've tried to patch it (and a number of other issues which would no doubt affect others) myself. I've had my patches ignored because your team didn't like the presence of any comments in the code. That is actually a very impor

Re: [Gluster-devel] Performance Translators' Stability and Usefulness

2009-07-05 Thread Filipe Maia
I strongly agree with both Gordan and Geoff when it comes to stability. I've been trying to get a simple unify setup (now dht) working to replace my existing nfs file servers and i'm yet to achieve the necessary stability. This kind of feature has long been in GlusterFS and should be stable enough

Re: [Gluster-devel] Performance Translators' Stability and Usefulness

2009-07-05 Thread Geoff Kassel
Hi Gordan, > >> Why wasn't this prioritised after such a disasterous bug? > > > > It could've been for any number of reasons ranging from problems with > > reproducing it, limited functionality for managing bug reports in > > Savannah to even the general constraints of being a commercial > > open-

Re: [Gluster-devel] Performance Translators' Stability and Usefulness

2009-07-06 Thread Geoff Kassel
Hi Filipe, I think if the GlusterFS devs actually implement the QA process they've (allegedly) had since September last year, that would go a long way to improving stability and reliability across the whole application. If the QA process as described on the Wiki was actually implemented, w

Re: [Gluster-devel] Performance Translators' Stability and Usefulness

2009-07-06 Thread Michael Cassaniti
If anyone has kept up with Samba development, they reworked their code and how they write their code for the 4.x series. This was because the development team felt the code was somewhat unstructured and not as transparent. They also now have an IDL compiler available. If the Samba team are anythin

Re: [Gluster-devel] Performance Translators' Stability and Usefulness

2009-07-06 Thread Anand Babu Periasamy
Gordon, Geoff, Fillipe, We are sorry!. We admit we had a rough and difficult past. Here are the reasons, why it was difficult for us: * Limited staff and QA environment. * GlusterFS is a programmable file system. It supported many OS distros, applications, hardware and storage architecture. It

Re: [Gluster-devel] Performance Translators' Stability and Usefulness

2009-07-06 Thread Mickey Mazarick
I have to start with an admonishment for both Gordon and Geoff. Come on guys! A dev thread is not the place to rant at developers, but to look for solutions. Please keep the complaining to a minimum and look constructively for solutions to the problems. I too have had similar pains doing gluste

Re: [Gluster-devel] Performance Translators' Stability and Usefulness

2009-07-06 Thread Vikas Gorur
- "Gordan Bobic" wrote: Gordan, I'd like to clarify the situation with the "mount --bind" issue you're facing. When GlusterFS is started, it does two things in the following order: 1) daemonize itself (by calling daemon(3)). 2) Initialize the translator graph (and thus FUSE and mount). T

Re: [Gluster-devel] Performance Translators' Stability and Usefulness

2009-07-06 Thread Geoff Kassel
Hi Mickey, What you seem to have missed is that we've tried giving constructive feedback, and have offered feasible solutions. In my case, over the span of years. (Check the archives.) It's been ignored. These are not outlandish requests either - requests like the use of regression testin

Re: [Gluster-devel] Performance Translators' Stability and Usefulness

2009-07-06 Thread Geoff Kassel
Hi Anand, Thank you for your explanation. I appreciate the circumstances you're in - I'm in a not-too-dissimilar environment myself. If you don't mind taking some more advice - do you mind taking down your current QA process document? It does not seem to be an accurate representation of y

Re: [Gluster-devel] Performance Translators' Stability and Usefulness

2009-07-06 Thread Gordan Bobic
Vikas Gorur wrote: I'd like to clarify the situation with the "mount --bind" issue you're facing. When GlusterFS is started, it does two things in the following order: 1) daemonize itself (by calling daemon(3)). 2) Initialize the translator graph (and thus FUSE and mount). This introduces a r

Re: [Gluster-devel] Performance Translators' Stability and Usefulness

2009-07-06 Thread Alpha Electronics
I recommended GlusterFS to my client without reservation, but got pissed off because bugs was found from time to time and wasted too much time to trace the source of problem - and also Gluster team is hiding the problem. For an actual example, check this link: http://www.gluster.org/docs/index.php

Re: [Gluster-devel] Performance Translators' Stability and Usefulness

2009-07-06 Thread Anand Avati
>   I've also gone one better than just advice - I've given up significant > portions of my limited spare time to audit and patch a not-insignificant > portion of the GlusterFS code, in order to deal with the stability issues I > and others were encountering. My patches were ignored, on the grounds

Re: [Gluster-devel] Performance Translators' Stability and Usefulness

2009-07-07 Thread Geoff Kassel
Hmm, that's an interesting point about the autoscaling, John. Thanks for mentioning this. Would the GlusterFS devs care to comment on this? Geoff. On Tue, 7 Jul 2009, Alpha Electronics wrote: > I recommended GlusterFS to my client without reservation, but got pissed > off because bugs was found

Re: [Gluster-devel] Performance Translators' Stability and Usefulness

2009-07-07 Thread Geoff Kassel
Hi Anand, If you look back through the list archives, no one other than me replied to the original QA thread where I first posted my patches. Nor to the Savannah patch tracker thread where I also posted my patches. (Interesting how those trackers have been disabled now...) It took me pres

Re: [Gluster-devel] Performance Translators' Stability and Usefulness

2009-07-07 Thread Jacques Mattheij
hello Anand, Geoff & others, This pretty much parallels my interaction with the team about a year ago, lots of really good intentions but no actual follow up. We agreed that an automated test suite was a must and that a whole bunch of other things would have to be done to get glusterfs out of th

Re: [Gluster-devel] Performance Translators' Stability and Usefulness - Regression test outline

2009-07-07 Thread Mickey Mazarick
What kind of requirements does everyone see as necessary for a regression test system? Ultimately the best testing system would use the tracing translator and be able to run tests and generate traces for any problems that occurs, giving us something very concrete to provide the developers. That'

Re: [Gluster-devel] Performance Translators' Stability and Usefulness - Regression test outline

2009-07-07 Thread Geoff Kassel
Hi Mickey, Just so that we're all on the same page here - a regression test suite at its most basic just has to include test cases (i.e. a set of inputs) that can trigger a previously known fault in the code if that fault is present. (i.e it can see if the code has 'regressed' into a conditio

Re: [Gluster-devel] Performance Translators' Stability and Usefulness - Regression test outline

2009-07-07 Thread Jacques Mattheij
hello again, One of the tools we discussed was a test harnass that you could wrap around a gluster module or a stack of modules to put them through their paces. Like that you can test a lot of different combinations of modules and their effects on each other in case one module triggers a bug in a

Re: [Gluster-devel] Performance Translators' Stability and Usefulness - Regression test outline

2009-07-07 Thread Mickey Mazarick
Geoff, Thanks I am well versed in unit testing but probably disagree on level of use in a development cycle. Instead of writing a long email back about testing theory, nondeterministic problems, highly connected dependent systems blah blah I'll just say that most of the problems that have plag

Re: [Gluster-devel] Performance Translators' Stability and Usefulness - Regression test outline

2009-07-07 Thread Gordan Bobic
On 07/07/2009 19:38, Mickey Mazarick wrote: Since I'm running my setup as a storage farm it just doesn't matter to me if there's a memory leak of if a server daemon crashes, I have cron jobs that restart it and I barely take notice. Ouch, ouch, ouch. That sounds like a monumental bodge. If som

Re: [Gluster-devel] Performance Translators' Stability and Usefulness - Regression test outline

2009-07-07 Thread Gordan Bobic
On 07/07/2009 18:24, Jacques Mattheij wrote: Another thing we talked about was a simple logging module which you can place anywhere in the stack that will log the information going up and down without actually doing anything to aid debugging. (but this is not actually an automated test tool, jus

Re: [Gluster-devel] Performance Translators' Stability and Usefulness - Regression test outline

2009-07-07 Thread Anand Avati
>   I hope my extremely long winded rant here :) has explained adequately what > I feel GlusterFS needs to have in a regression testing system. > > Geoff. Geoff, and others, we hear you. Loud and clear. We acknowledge that the project could have fared better if the QA processes were already in pla

Re: [Gluster-devel] Performance Translators' Stability and Usefulness - Regression test outline

2009-07-07 Thread Mickey Mazarick
Oh don't get me wrong I'm not ANTI-unit testing at all. I actually haven't had a server daemon crash since 1.4. I'm still leaving my cron jobs in. Try as I might I just don't care very much about the 1.5 gigs of memory the server process is using. My point isn't to open up a who pragmatic vs p

Re: [Gluster-devel] Performance Translators' Stability and Usefulness - Regression test outline

2009-07-07 Thread Gordan Bobic
Right - now that we're all agreed, I think we should knock this thread on the head, stop criticizing and start heading toward where we want this project to be by providing whatever contribution we can (code patches, putting together reproducible test cases for error conditions, etc.). Gordan

Re: [Gluster-devel] Performance Translators' Stability and Usefulness - Regression test outline

2009-07-07 Thread Erick Tryzelaar
On Tue, Jul 7, 2009 at 12:10 PM, Anand Avati wrote: > > And in the last month, as > Shehjar has mentioned earlier in this thread, a unit test and > regression test framework has indeed been under development, and we > intend to have the framework and test cases published in a couple of > months for

Re: [Gluster-devel] Performance Translators' Stability and Usefulness - Regression test outline

2009-07-07 Thread Geoff Kassel
Hi Mickey, > Thanks I am well versed in unit testing but probably disagree on level > of use in a development cycle. Instead of writing a long email back > about testing theory, nondeterministic problems, highly connected > dependent systems blah blah Sorry, I was just trying to make sure we were

Re: [Gluster-devel] Performance Translators' Stability and Usefulness - Regression test outline

2009-07-07 Thread Geoff Kassel
Hi Jacques, The test harness sounds good to me. Sounds pretty much spot on what's needed to do integration testing. Geoff. On Wed, 8 Jul 2009, Jacques Mattheij wrote: > hello again, > > One of the tools we discussed was a test harnass that you could wrap > around a gluster module or a stack o

Re: [Gluster-devel] Performance Translators' Stability and Usefulness - Regression test outline

2009-07-07 Thread Geoff Kassel
Hi Gordan, > Ouch, ouch, ouch. That sounds like a monumental bodge. If somebody > working for me implemented that kind of a "solution" for a frequently > occuring problem in a production environment, they'd be finding > themselves looking for a new job pretty quickly. Most likely along with > the

Re: [Gluster-devel] Performance Translators' Stability and Usefulness - Regression test outline

2009-07-07 Thread Mickey Mazarick
Wow, you really hit my biggest fear, the one thing I try to test for... data corruption. That's what I wake up afraid of at night... I'm doing a simplified version of the first set of testing you mentioned but nothing as detailed. Really creating a random file and doing an md5 check on it, bu

Re: [Gluster-devel] Performance Translators' Stability and Usefulness - Regression test outline

2009-07-07 Thread Geoff Kassel
Hi Anand, Thanks for the explanation. I'm sorry, you'll have to endure my criticism for just a little while longer, though. I've seen your stable/unstable branch tactic before with 1.3 and 1.4/2.x and it didn't work then. Any stability gained in 1.3 just seemed to be lost again in 1.

Re: [Gluster-devel] Performance Translators' Stability and Usefulness - Regression test outline

2009-07-07 Thread Geoff Kassel
Hi Mickey, > Wow, you really hit my biggest fear, the one thing I try to test for... > data corruption. Yep, it's mine too. Hence all this ranting and raving of mine :) > I'm doing a simplified version of the first set of testing you mentioned Sounds good :) > I would add a few but I haven't h