Edward J. Yoon wrote:
>> Is access to *one* of the 20 million different SQLite files getting
>> progressively slower? How big is that specific SQLite file? Is that
>> the one that is "huge"? I use SQLite over an NAS (at times), and never
>> experience any noticeable slowdown. Is access to his NAS itself slow,
>> perhaps not just via SQLite but just over the regular filesystem?
>>
>
> Each NAS_000 ~ N storages have approximately 300,000 files, the
> average size of file is few MB (not over GB). The broker servers (with
> SQLite library) are on the NAS and The front-end web servers (more
> than 200 servers) communicate with living broker servers after request
> location from location addressing system. There are high frequency
> read/write/delete operations.
>
> The number of files/storages/clients keep increasing little by little.
>
How is the NAS attached to the network? Is it a USB attached drive to a
server?
Many network bandwidth issues aren't caused by the DB it's self but
rather the network architecture. A simple rule of thumb is that the
bandwidth is only as large as the largest bandwidth of an
appliance/component that the data travels through. Some NAS appliances
are notoriously slow due to the restrictions on bandwidth within that
appliance. And you won't notice it until you get to a specific point,
which maybe the number of concurrent users or the amount of date
transfered or a combination of both.
If you have the NAS attached to a server you could setup a routine that
would copy the desired db file to the server (or another server on the
network) while the user is using that db file. That would give you more
of a distributed file system architecture than just trying to serve
everything off of the NAS and would in turn take pressure off of the NAS
appliance.
I would be curious to know at what point this became an issue? Other
factors would be the type/make/model of the NAS and how it's setup on
the network? Are you using hubs or switches? Have you had a traffic
monitor on the network? Is this network MS based or Unix, do you have a
PDC, and how is the NAS authenticating users? Are the connections mapped
drives? What I would suggest doing is stepping back to the point of the
performance degradation and work from there. It maybe that you have just
reached the outer limits of that particular NAS appliance.
Personally, I do not use NAS for DB's that have concurrent users using
the DB's but rather use NAS for archiving and occasional user storage of
non-current data for the reasons I stated above. If NAS is the only
solution I have, I'll plan to expand the NAS with another appliance when
I notice any degradation of performance occurring and then do load
balancing to balance the load between the appliances.
From what I have gathered here, I think I'd be safe to assume that this
isn't a SQLite issue but rather a Network/NAS issue.
Hope this helps,
Carl
Carl Lindgren
C. R. Lindgren Consulting / Business on the Desktop
> /Edward
>
> On Wed, Jan 7, 2009 at 9:57 PM, P Kishor <punk.k...@gmail.com> wrote:
>
>> On 1/7/09, Thomas Briggs <t...@briggs.cx> wrote:
>>
>>>I actually thought the original question was perfectly clear. I
>>> thought the proposed solution (included in the original post) was
>>> perfectly logical too. So what's all the fuss?
>>>
>> The confusion, at least for me, arose from the following sentence in the OP
>> --
>>
>> "I'm using SQLite, all data (very huge and 20 million files) "
>>
>> and the response to request for clarification of the above.
>>
>> - we know he is using SQLite
>>
>> - we know "it" is all data (although, I am not sure what else could
>> SQLite be used for other than "data")
>>
>> - we know "it" is very huge
>>
>> - we know there are 20 million *files* involved
>>
>> No matter how I put together the above four pieces of information, I
>> can't grok it.
>>
>> Is access to *one* of the 20 million different SQLite files getting
>> progressively slower? How big is that specific SQLite file? Is that
>> the one that is "huge"? I use SQLite over an NAS (at times), and never
>> experience any noticeable slowdown. Is access to his NAS itself slow,
>> perhaps not just via SQLite but just over the regular filesystem?
>>
>> So there... no fuss, just a desire to understand better what exactly
>> is the problem.
>>
>>
>>> On Wed, Jan 7, 2009 at 7:28 AM, P Kishor <punk.k...@gmail.com> wrote:
>>> > On 1/6/09, Edward J. Yoon <edwardy...@apache.org> wrote:
>>> >> Thanks,
>>> >>
&
