[jira] [Closed] (COUCHDB-602) Cannot use NFS for storage
[ https://issues.apache.org/jira/browse/COUCHDB-602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Randall Leeds closed COUCHDB-602. - Resolution: Not A Problem > Cannot use NFS for storage > -- > > Key: COUCHDB-602 > URL: https://issues.apache.org/jira/browse/COUCHDB-602 > Project: CouchDB > Issue Type: Bug > Components: Test Suite >Affects Versions: 0.10 >Reporter: Nils Breunese > Attachments: error.gz > > > Our hosting provider claims they cannot upgrade our CouchDB installation from > version 0.9.1 to 0.10 as the test suite for 0.10 is failing. > This is a translation from their report: > > CouchDB can't handle having its data directory on NFS. > It seems CouchDB tries to do this: > 1 mkdir test > 2 start process X with fd under test/foo > 3 do the work > 4 rm test/foo > 5 rmdir test > 6 kill process X > When test/ is on NFS: > In step 4 an open file is being removed. NFS doesn't really support this > and the workaround is that the client implementation does >mv file .nfsXXX > However, the 'rmdir test' in step 5 then cannot succeed anymore, since > the directory is not empty. This eventually causes CouchDB to crash. > In earlier versions of CouchDB the open/rm/rmdir/kill order may have > been different, for instance open/rm/kill/rmdir. > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-602) Cannot use NFS for storage
[ https://issues.apache.org/jira/browse/COUCHDB-602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13251646#comment-13251646 ] Paul Joseph Davis commented on COUCHDB-602: --- Cool. Though a quick note that we still don't recommend running over NFS. Users doing such need to be aware of the issues involved. > Cannot use NFS for storage > -- > > Key: COUCHDB-602 > URL: https://issues.apache.org/jira/browse/COUCHDB-602 > Project: CouchDB > Issue Type: Bug > Components: Test Suite >Affects Versions: 0.10 >Reporter: Nils Breunese > Attachments: error.gz > > > Our hosting provider claims they cannot upgrade our CouchDB installation from > version 0.9.1 to 0.10 as the test suite for 0.10 is failing. > This is a translation from their report: > > CouchDB can't handle having its data directory on NFS. > It seems CouchDB tries to do this: > 1 mkdir test > 2 start process X with fd under test/foo > 3 do the work > 4 rm test/foo > 5 rmdir test > 6 kill process X > When test/ is on NFS: > In step 4 an open file is being removed. NFS doesn't really support this > and the workaround is that the client implementation does >mv file .nfsXXX > However, the 'rmdir test' in step 5 then cannot succeed anymore, since > the directory is not empty. This eventually causes CouchDB to crash. > In earlier versions of CouchDB the open/rm/rmdir/kill order may have > been different, for instance open/rm/kill/rmdir. > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: fractal tree in place of b-trees?
Rudi, Improving the view structures for the btree's is definitely a place that we should start to look at for performance gains. The new view indexing system should hopefully allow people to more easily play with ideas here. As to fractal trees I'm not quite certain. Could definitely be an interesting thing to play with but I'm not entirely convinced by that presentation because it appears to hide some rather important information about CPU and RAM usage during their tests. And the fact that Fusion IO doesn't appear to significantly impact Innodb is a bit suspect to me. Still, an interesting idea to look into. On Wed, Apr 11, 2012 at 9:31 AM, Rudi Benkovic wrote: > Hi, > > Here's a TokuDB presentation on fractal trees: > > http://assets.en.oreilly.com/1/event/36/How%20TokuDB%20Fractal%20Tree%20Databases%20Work%20Presentation.pdf > > Any ideas if this could be used in CouchDB for view indexes? I know > that CouchDB's main storage is an append-only B-Tree, but perhaps this > could be used for high performance/volume views? Just starting a > discussion, not much else. :) > > -- > Best regards, > Rudi mailto:ru...@whiletrue.com >
fractal tree in place of b-trees?
Hi, Here's a TokuDB presentation on fractal trees: http://assets.en.oreilly.com/1/event/36/How%20TokuDB%20Fractal%20Tree%20Databases%20Work%20Presentation.pdf Any ideas if this could be used in CouchDB for view indexes? I know that CouchDB's main storage is an append-only B-Tree, but perhaps this could be used for high performance/volume views? Just starting a discussion, not much else. :) -- Best regards, Rudi mailto:ru...@whiletrue.com
Re: (perceived) barriers to entry
On Tue, Apr 10, 2012 at 8:46 PM, Tim McNamara wrote: > After chatting with Noah S on Twitter, I offered to jot up some > thoughts on things that my reduce or eliminate perceived barriers to > entry to work on CouchDB. > > Here are a few things that I've been able to think of. In the course > of researching this mail though, I've actually answered many of my own > questions. > > "serious business" > A database seems like the kind of project that only extremely talented > people would touch. People depend on the code working. Getting started > would require quite a bit of confidence. Am I good enough? > @antirez of Redis fame did a really great post on data durability a couple weeks ago. If you read and understand that you're briefed enough to start diving into the code. Beyond those basics the advanced details are more about learning the details of how certain types of hardware reacts to those POSIX API's. Ie, stuff that @antirez alludes to but doesn't cover (non-volatile write caches being a fun one). http://antirez.com/post/redis-persistence-demystified.html > > "Erlang, wtf" > This is a barrier that I've been facing for a while. I'm actually in > the process of learning Erlang, trying to expand my horizons from > Python. Still, a new language makes it harder to have the required > confidence. > Heh, I still say "Erlang, wtf?" after using it for a few years. But I think that's true of any language I've used for any extended period of time. As much as I love Python it's still got its warts (outer variables not captured by closures?). In the end I would wager that a big part of the Erlang stumbling block is that it's also functional and that is a stumbling block for some. When I started with it I was like "oh, its one of those academic things the CS people dreamed up," and didn't get into it right away. But once you get into it you realize its not as hard as you made it out to be. > > "I still don't understand rereduce" > I'm personally not 100% clear on how queries work. This is even after > using the db for a while. I don't want to look like a stupid idiot in > front of great developers. Therefore, there's a high risk of offering > suggestions and getting told to "RTFM" > For this I would say come ask in IRC or on the dev@ list. There are also some older resources that still do a decent job of describing the internals. But I would say no one expects any new comer to jump in and tackle some of the core modules without first learning how they're used in other places in the code. http://horicky.blogspot.com/2008/10/couchdb-implementation.html > > "Where are the easy bugs?" > [solved] > > > "wow, big code base" > Is there any documentation on how to project is laid out? Stepping > into a new project is always a little daunting. > You're joking, right? :D CouchDB is something like 15K lines (although in a weird language and not as clean as it could be) but compared to other similar databases that's absolutely tiny. That said, there's not much excuse for us to not have documentation on code layout and the internal and external APIs and so forth. Hopefully we'll be doing some work in the next few months to try and address this issue. > > "Apache project?" > As someone outside of the ASF, I don't really know what contributing > on an Apache project means. Don't worry about this. As you start contributing its the same as any other open source project since the beginning of open source. Look at the code, make a change, submit it back. Other than that, I would say if you're interested in contributing to pick one section of the code base and start reading and trying to understand. Do some hacking and see if you can break it or change its behavior in fun ways. My personal quest into the internals started at the HTTP layer, worked into the view engine, and then finally to the actual core. For me that was just what I found interesting at the time. I could see another approach to start at the level of couch_file and work outwards if that seems more approchable. HTH and always feel free to ask questions, Paul Davis
[jira] [Commented] (COUCHDB-602) Cannot use NFS for storage
[
https://issues.apache.org/jira/browse/COUCHDB-602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13251498#comment-13251498
]
d commented on COUCHDB-602:
---
Although this bug was opened for 0.10 and the current version as i write this
is 1.2.0, I thought it may be interesting for others that since at least 1.1.1.
all tests pass successfully. I have the datafiles on NFS (NetApp toaster),
mount options:
{noformat}
rw,rsize=32768,wsize=32768,nfsvers=3,hard,tcp,nointr,lock,noatime,timeo=700,retrans=3
{noformat}
Ubuntu 10.04/12.04 LTS 64bit Server with backported couchdb
> Cannot use NFS for storage
> --
>
> Key: COUCHDB-602
> URL: https://issues.apache.org/jira/browse/COUCHDB-602
> Project: CouchDB
> Issue Type: Bug
> Components: Test Suite
>Affects Versions: 0.10
>Reporter: Nils Breunese
> Attachments: error.gz
>
>
> Our hosting provider claims they cannot upgrade our CouchDB installation from
> version 0.9.1 to 0.10 as the test suite for 0.10 is failing.
> This is a translation from their report:
>
> CouchDB can't handle having its data directory on NFS.
> It seems CouchDB tries to do this:
> 1 mkdir test
> 2 start process X with fd under test/foo
> 3 do the work
> 4 rm test/foo
> 5 rmdir test
> 6 kill process X
> When test/ is on NFS:
> In step 4 an open file is being removed. NFS doesn't really support this
> and the workaround is that the client implementation does
>mv file .nfsXXX
> However, the 'rmdir test' in step 5 then cannot succeed anymore, since
> the directory is not empty. This eventually causes CouchDB to crash.
> In earlier versions of CouchDB the open/rm/rmdir/kill order may have
> been different, for instance open/rm/kill/rmdir.
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: (perceived) barriers to entry
Nice resource, kowsik. A suggestion on documenting rereduce would be to describe it in terms of standard SQL operations. Before looking at kowsik's link, I also had only a vague idea of what rereduce was doing. Looking at the link, it appears clear that rereduce can be used to do aggregate SQL operations like SUM() and COUNT(), so it might make sense to include more SQL examples paired with corollary CouchDB rereduce functions, similar to this: http://guide.couchdb.org/draft/cookbook.html#aggregate In general, I think the CookBook for SQL Jockeys approach will be useful to a number of experienced developers who are moving from SQL to CouchDB. Explain things in terms of technology they're already familiar with, where possible. Kurt Milam On Wed, Apr 11, 2012 at 3:26 AM, kowsik wrote: > On Tue, Apr 10, 2012 at 5:46 PM, Tim McNamara > wrote: > > [snip] > > > "I still don't understand rereduce" > > I'm personally not 100% clear on how queries work. This is even after > > using the db for a while. I don't want to look like a stupid idiot in > > front of great developers. Therefore, there's a high risk of offering > > suggestions and getting told to "RTFM" > > Try this: > http://blog.mudynamics.com/wp-content/uploads/2009/04/icouch.html > > K. > --- > http://blitz.io > @k0ws1k >
Re: (perceived) barriers to entry
Hi Tim, I am not a developer and I don't think I will be too soon as my free time doesn't allow me that. But I've been helping the users to get their products based on CouchDB done and I noticed 4 types of problems a new developer can attempt to solve: 1. installation problems on different platforms; 2. compatibility problems with different external software; 3. undesired (but understandable) differences in functionality (e.g., push vs. pull replications); 4. new cool staff. If you have time, you may want to start with point 1 and, in time, to reach point 4. In time you reach point 4, the confidence is built up and no need to ask yourself if you are good or not for this project because you are already involved in it. Now, my thoughts about the problems you raised. "serious business" Once one thinks that way, then it's clear he/she needs this task just for adding it at CV. If you do it because you really would like to help, I am sure you will get help with your ideas if you don't manage by yourself. Does everything works in Linux? I recall the time when wireless drivers under Linux were really bad. Implement your ideas at the best of your knowledge and if they work for you, there are many chances to work for everybody (after all, Erlang is supposed to work in the same way on whatever platform). "Erlang, wtf" One way to learn it is reading and trying to understand the code. One step further is to add your own modules. This requires only time and willingness. Not to mention that there are parts in this project which have nothing to do with Erlang. "I still don't understand rereduce" Good point. There may be many points you have no idea how they work or what they actually implement. But nobody asks you to develop CouchDB from scratch. Just focus on what you know at the beginning and leave what you don't know for later. "Where are the easy bugs?" [solved] Really? The easy bugs which are solved are those reported here. Just look at the users list and see how many easy bugs are reported there almost every day and they are not reported in the dev list. That's because they are easy bugs which can suffer a workaround rather than the intervention from the devs (e.g., tests compatibility in different browsers or incompatible tests in the configure script which, in the end, attracted the devs attention, but there are still problems there). You can start by fixing those bugs. "wow, big code base" I think there are few devs here who can claim they know the full code and I haven't seen any message here saying "RTFM" for those who don't know everything. As far as I noticed here, there is a tendency of collecting the knowledge and not to show off with what one knows. Why should it be different for the newcomers? If you have time to study all the code, then it would be great. Nevertheless, in my 2 cents opinion, it's useless to understand the full code from the beginning because you are not going to work on optimization of the code at that point, correct? "Apache project?" Well, it's not like you sign a contract with ASF. I am helping in maintaining the documentation on wiki when it's needed, but I have no idea how it is to work for ASF. It's nothing else but a contribution to a freeware community-based project. Create your own git repository, report your modules and if they proved to be useful, your modules will be added into the CouchDB repository (that I noticed while following this list, but I might be wrong). In time you will be given the committer rights. This is general for all the community-based projects and, therefore, it has nothing to do with ASF at this stage. It may be, like in most of such projects, that from time to time ASF will reward your efforts, but, mainly, people here are doing it without that thought (I think). As I said, even if I wanted to become a dev, I am not able to take that step because I have no time. But if I started, I would follow what I wrote here. I hope these thoughts will help removing the "(perceived) barriers to entry". CGS On Wed, Apr 11, 2012 at 2:46 AM, Tim McNamara wrote: > After chatting with Noah S on Twitter, I offered to jot up some > thoughts on things that my reduce or eliminate perceived barriers to > entry to work on CouchDB. > > Here are a few things that I've been able to think of. In the course > of researching this mail though, I've actually answered many of my own > questions. > > "serious business" > A database seems like the kind of project that only extremely talented > people would touch. People depend on the code working. Getting started > would require quite a bit of confidence. Am I good enough? > > > "Erlang, wtf" > This is a barrier that I've been facing for a while. I'm actually in > the process of learning Erlang, trying to expand my horizons from > Python. Still, a new language makes it harder to have the required > confidence. > > > "I still don't understand rereduce" > I'm personally not 100% clear on how queries work. This is even after
