Because it's a simple feature that I need and I don't want an overkill solution that I do not control its thread count, its memory consumption and cpu usage. It is just a simple component inside my highly performance server that needs to keep a single list of redo log in the disk if it takes to much memory which will be very light and won't impact the server performance. This occurs when the backup server it replicates to is temporarily down and the redo log starts to accumulate. There are many consideration which I can't really explain, but I do not wish to use any embedded or remote third party overkill product for this very simplified capabilities, like I've described above, unlimited single list of elements that can only be added at the end of the list, removed from the first and iterated over the list in read only mode.
Any other generic product addresses a much more complex scenario and probably will perform worse because of that. I already implemented it and it works fine, but I am not sure about the multi reader over a file regarding the visibility issue and I was hoping that someone knows what kind of guarantee is there for the visibility of multiple RandomAccessFile inside the same process over the same file. (only one in rw mode and the rest in r mode) Thanks On Dec 25, 12:41 pm, Viktor Klang <viktor.kl...@gmail.com> wrote: > Why a file mate? > Why not Redis, Cassandra, MongoDB, SQLite whatnot? > > > > On Fri, Dec 25, 2009 at 10:20 AM, Eitan <eitan...@gmail.com> wrote: > > What about what I said above, using multiple FileChannel/ > > RandomAccessFile to read from the same file (not writing, there's a > > single writer multi reader mutual exclusion protection over these > > FileChannel/RandomAccessFile access), all I am worried about is the > > visibility that updates the writer will do will be visible to the > > readers once done, FileChannel guarantee this and I suspect it must > > rely on this kind of guarantee from the RandomAccessFile that creates > > it though it is not documented, But according to the simple analysis I > > of the RandomAccessFile getChannel method I did, it seems that it does > > need to guarantee it in order for FileChannel to guarantee it. > > > Since my implementation here is just an "unlimited" single list of > > objects that starts in the memory and once there's no room continue in > > the disk, I went for this solution and not using an over kill embedded > > data base. I only need to support appending objects at the end of the > > list, removing from the start and iterating over the list in read only > > mode. (Either from the start to the end or from the end to the start). > > > Regarding the caching you suggested, I created caching mechanism and > > there are a few layers of abstraction over this file, but, I am > > talking here about the lowest level of the abstraction which access > > the file and that might occur concurrently and I want to remove the > > contention there as well. (for instance 10 threads currently iterating > > over the list all at a different location) besides the whole issue > > here is to move the data from the memory to the disk because I am out > > of memory space, so caching the 10g file is not an option, I only > > cache the end and the start of the list since they are accesses most > > frequently. > > The time spending ratio here is not the problem, there can be tens of > > threads concurrently iterating over the list and I want them to be > > able to iterate concurrently and not wait for each other. According to > > amdal's law even if the time spent on reading is just 10%, I can only > > accelerate this process by at most 10 so 100 threads iterating will > > work no faster than 10. > > > Basically if I get the visibility guarantee then my multi > > RandomAccessFile in read mode over the file is good enough solution, > > let the disk and the OS do the caching of accessing the same file > > sections and keep my program more concurrent. > > > Thanks > > > On Dec 24, 11:45 pm, Christian Catchpole <christ...@catchpole.net> > > wrote: > > > Is not the simplest solution to allow each thread to get a reference > > > to the *single* RandomAccessFile instance and simply synchronize on it > > > while reading (not forgetting to reset the offset into the file). > > > It's not going to be super performant, but how is it ever going to be > > > with multiple threads thrashing on one file? If anything it might > > > reduce thrashing as you are controlling the points of file > > > contention. What is the ratio of time spent reading the file to time > > > spent operating on the data? > > > > If all these threads will in-fact thrash on the file it might only be > > > solved with: > > > > 1. Better scheduling than "free for all" threads. > > > 2. a large OS level disk cache > > > 3. an in memory disk cache > > > 4. some kind of abstract layer over the file which does caching and > > > returns you "model objects" rather than being seen as a byte stream. > > > > On Dec 25, 1:38 am, Eitan <eitan...@gmail.com> wrote: > > > > > Looking at FileChannel it has the following part in its documentation: > > > > > The view of a file provided by an instance of this class is guaranteed > > > > to be consistent with other views of the same file provided by other > > > > instances in the same program. The view provided by an instance of > > > > this > > > > class may or may not, however, be consistent with the views seen by > > > > other > > > > concurrently-running programs due to caching performed by the > > > > underlying > > > > operating system and delays induced by network-filesystem protocols. > > > > This > > > > is true regardless of the language in which these other programs are > > > > written, and whether they are running on the same machine or on some > > > > other > > > > machine. The exact nature of any such inconsistencies are system- > > > > dependent > > > > and are therefore unspecified. > > > > > This looks exactly like the guarantee I need, now I've looked into > > > > RandomAccessFile.getChannel() implementation and it holds a single > > > > member of FileChannel which is initialized on demand and the next > > > > calls will return the same instance. > > > > > So the only way I see to create multiple FileChannel instances over > > > > the same file is to open multiple RandomAccessFile instances over that > > > > file and use getChannel() on each of these instances. Since > > > > FileChannel does provide this guarantee I tend to believe it relies > > > > upon a guarantee that RandomAccessFile provide, otherwise it would > > > > need some mechanism to identify that different FileChannel instances > > > > over different RandomAccessFile instances are actually over the same > > > > file in order to provide the specified guarantee. > > > > > What do you think? > > > > > On Dec 21, 9:33 am, Ben Schulz <ya...@gmx.net> wrote: > > > > > > My guess is that Java IO takes advantage of any OS optimizations > > there > > > > > are, on another OS there may be a per-file-pointer cache or > > something. > > > > > That would obviously invalidate your assumptions. I suggest you look > > > > > into using asynchronous I/O which will get rid of any > > multiple-reader- > > > > > bottlenecks. This should also rule out third parties tampering with > > > > > the file (since you already hold the write lock). > > > > > > With kind regards > > > > > Ben > > > > > > On 21 Dez., 07:50, Eitan <eitan...@gmail.com> wrote: > > > > > > > I am trying to avoid locking the access to file for multiple > > readers > > > > > > can be multi concurrent readers and it will be a bottleneck. > > > > > > I've created a multi threaded test for it and it seems to work > > fine, > > > > > > but I am not still not confident about it. I was hoping to find > > some > > > > > > documentation about it which specifies the behavior. > > > > > > > On Dec 20, 11:40 pm, Christian Catchpole <christ...@catchpole.net> > > > > > > wrote: > > > > > > > > Can you even get a second lock on a file that way? Sounds > > dangerous. > > > > > > > I would just create a some kind of service that each Thread can > > access > > > > > > > that will synchronise disk access down to one RandomAccessFile. > > > > > > > > On Dec 21, 2:48 am, Eitan <eitan...@gmail.com> wrote: > > > > > > > > > I want to use a file in a single process with single writer or > > multi > > > > > > > > readers mutual exclusion, for that purpose I use number of > > > > > > > > RandomAccessFile instances over the same file which are open in > > read > > > > > > > > mode and are used concurrently. I also have a single > > RandomAccessFile > > > > > > > > open in "rw" mode which updates the file (not when it is being > > > > > > > > actively read). > > > > > > > > I do not close the Random Access File handles at any point. > > > > > > > > > I wanted to know if the data which is being written to the file > > with > > > > > > > > the RandomAccessFile in rw mode will always be visible to the > > readers > > > > > > > > after the the write is complete? > > > > > > > > (I am not using rws mode for performance issues, only for these > > > > > > > > readers which are in the same process). > > > > > > > > > Thanks > > > > > > > > Eitan > > > -- > > > You received this message because you are subscribed to the Google Groups > > "The Java Posse" group. > > To post to this group, send email to javapo...@googlegroups.com. > > To unsubscribe from this group, send email to > > javaposse+unsubscr...@googlegroups.com<javaposse%2bunsubscr...@googlegroups.com> > > . > > For more options, visit this group at > >http://groups.google.com/group/javaposse?hl=en. > > -- > Viktor Klang > | "A complex system that works is invariably > | found to have evolved from a simple system > | that worked." - John Gall > > Blog: klangism.blogspot.com > Twttr: twitter.com/viktorklang > Code: github.com/viktorklang -- You received this message because you are subscribed to the Google Groups "The Java Posse" group. To post to this group, send email to javapo...@googlegroups.com. To unsubscribe from this group, send email to javaposse+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/javaposse?hl=en.
