On Sun, Oct 7, 2012 at 7:49 PM, Nagaev Boris <[email protected]> wrote:
> Hello!
>
> I've been thinking about it.
>
> I am using Postgres and it seems to be bottleneck. But using flat
> files is likely to be slower and more danderous of data loss. I think,
> the most fast and safe method is to use Database, but copy objects to
> RAM. If you have enough memory to copy the entire database to it. When
> something is changed, it is added to queue and saved to Database
> asynchronously (some data may be lost).
>
Data loss is one issue but i want to know if it can helpful in speed of the
application and as of now i am not worried about advanced SQL queries.
Is there any way better to go with flatfiles and db because db can only be
helpful in your data size is of medium range. for very small data and very
"Big data" I believe flat files will be better.
Also storing all records in a single file creates problem if the table size
very big because the server need to load the whole file before processing
and this will create "bottlenecks"
another method is to use flat files and use a indexing technique to make
the retreival faster. there are classing indexing methods which can create
index on very large corpses say a wiki article of size 23 GB
>
> I think, this can be implemented as a SqlConnection sub-class. It
> would work as a front-end of any other backend (postgres, firebird,
> sqlite3 or mysql). Retriving of objects will be very fast, but I do
> not know what to do with advanced SQL queries (e.g., WHERE and ORDER
> BY). Maybe, they also can be implemented in memory, using in-memory
> indices.
>
> Another approach is to use sqlite3 database located in memory or in
> tmpfs. The file is copied to disk from time to time.
>
> On Sun, Oct 7, 2012 at 5:44 PM, Mohammed Rashad
> <[email protected]> wrote:
> > All,
> >
> > Due to large data used in a crowd source mapping project. I had decided
> to
> > completely eliminate the use of database and use Flat files for storage,
> > retrieval and query.
> >
> > I thought of storing each record in db as individual files. so this will
> > help in the retrieval speed and no search is needed in the entire db or a
> > file.
> >
> > but if a table have more than 10000's of records and users accessing
> > (same/different) records from different places will result in N number of
> > File I/O
> >
> > Will this be a bottleneck in the application. consider each file of size
> <=
> > 15KB.?
> >
> > The main reason to eliminate db is because of performance bottleneck in
> > database I/O.
> >
> > So moving to new model will help in anyway as the number of users and
> data
> > will be much more than expected?
> >
> >
> > --
> > Regards,
> > Rashad
> >
> >
> >
> ------------------------------------------------------------------------------
> > Don't let slow site performance ruin your business. Deploy New Relic APM
> > Deploy New Relic app performance management and know exactly
> > what is happening inside your Ruby, Python, PHP, Java, and .NET app
> > Try New Relic at no cost today and get our sweet Data Nerd shirt too!
> > http://p.sf.net/sfu/newrelic-dev2dev
> > _______________________________________________
> > witty-interest mailing list
> > [email protected]
> > https://lists.sourceforge.net/lists/listinfo/witty-interest
> >
>
>
> ------------------------------------------------------------------------------
> Don't let slow site performance ruin your business. Deploy New Relic APM
> Deploy New Relic app performance management and know exactly
> what is happening inside your Ruby, Python, PHP, Java, and .NET app
> Try New Relic at no cost today and get our sweet Data Nerd shirt too!
> http://p.sf.net/sfu/newrelic-dev2dev
> _______________________________________________
> witty-interest mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/witty-interest
>
--
Regards,
Rashad
------------------------------------------------------------------------------
Don't let slow site performance ruin your business. Deploy New Relic APM
Deploy New Relic app performance management and know exactly
what is happening inside your Ruby, Python, PHP, Java, and .NET app
Try New Relic at no cost today and get our sweet Data Nerd shirt too!
http://p.sf.net/sfu/newrelic-dev2dev
_______________________________________________
witty-interest mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/witty-interest
