That sounds reasonable. You mention users but not events. How will the
event files be handled ? Also as we keep adding files, we need to ensure
that the ratios of files in dirs is maintained.
Shanti
Akara Sucharitakul wrote:
100 is kind of low. That will mean 10,000 directories for 1 million
loaded users. I don't like it when we see 10,000 entries on ls. Gets
unwieldy again.
I'd limit to 1,000. Seems reasonable for directories to handle. But
the limit here is really a million users with two level directories.
If we increase the stored:concurrent user ratio to 100:1, we'll be
stuck at 10,000 concurrent users. I think Olio has the potential to
scale much higher.
Coming down, I really think we should do a 3 layer directory (or 2
layer subdirectory). With a limit of 1000 files/users per directory
you can actually go up to 10^9 stored users and 10^7 concurrent users
- and even if we at some point decide the user ratio to be 1000:1, we
are still up to 10^6 concurrent users. That's probably as big a rig as
anybody will want to test.
In summary, I'd go for 2 layer subdirectory limiting each directory to
have no more than 1000 entries.
-Akara
Shanti Subramanyam wrote:
Currently, the filestore is a single flat layer with all files both
pre-loaded and those that are generated during a run going into the
same directory. As the number of users are scaled, this directory is
unwieldy - I'm sure some of you may have tried to do a 'ls
/filestore' and then cursed yourself for doing it !
How about using one directory for every 100 users or so ? Something
similar for events ? Not sure how we'll deal with the added events
and files during a run though.
Thoughts ?
Shanti
