Re: [openstack-dev] Can swift leverage os page buffer?

[Clay Gerrard]

On Wed, Oct 11, 2017 at 3:46 PM, Jialin Liu  wrote:

> Hi,
> I'm new to openstack swift, I'm a HPC user, by several days of exploration
> of swift, I have some naive questions:
> 1. Can swift, e.g., PUT, leverage OS' page buffer?;
>

Sure, but perhaps to a limited degree depending on what you're expecting?
We try pretty hard to fsync everything before we return success:

https://github.com/openstack/swift/blob/master/swift/obj/diskfile.py#L1517

^ and just below that is _finalize_put


> Is there a Linux kernel module for swift?
>

No.


> What are the existing optimizations for caching the write in order to gain
> better bandwidth on spin disks?
>

... um ... that's a pretty broad question, swift has been around for quite
a while, it would require some research.  You can read through the diskfile
modules; most of the bits that touch disk at the object layer are in
there.  There's a number of tunables that effect how Swift will treat the
filesystem - and the low-level specifics of what they do are in the code.
Any results of rigorous classifications you perform and want to publish are
always interesting and welcome in the community.  There's been a few
interesting things published over the years - but there's no been no
community effort to collect them into a single home that I'm aware of;
you'd have to track them down - I recall Seagate did some interesting
analysis a while back.  RedHat's performance group is always doing stuff,
Intel did some stuff.  Current community efforts are focused on further
refining erasure code storage - which have a positive impact on medium and
large object uploads both by reducing the total number of backend bytes to
store (compared to replicated object) and also by fanning that object data
+ parity out to larger numbers of spindles on the backend.  On the other
end of the spectrum OVH is leading an effort to further improve performance
for lots of small files.

2. Does swift support asynchronous PUT/Get?
>
>
I don't know what that means, so I'll say no.  I might hazard a guess it
has something to do with a PUT only storing some reduced redundancy unsafe
staged data and then doing something to improve durability after it already
promised the client their write was "safe" - which is not something Swift
does.


>
> Also please let me know if the dev list is good for me to ask this kind of
> questions.
>

The ML is nice in that people can give more detailed responses and the
archives tend to be a bit more searchable - the trade off is a longer
latency on responses.  You can also jump on IRC - swift folks hang out in
#openstack-swift on freenode.

Best Regards,

Clay
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] Can swift leverage os page buffer?

[Jialin Liu]

Hi,
I'm new to openstack swift, I'm a HPC user, by several days of exploration
of swift, I have some naive questions:
1. Can swift, e.g., PUT, leverage OS' page buffer?; Is there a Linux kernel
module for swift? What are the existing optimizations for caching the write
in order to gain better bandwidth on spin disks?
2. Does swift support asynchronous PUT/Get?


Also please let me know if the dev list is good for me to ask this kind of
questions.

Best,
Jialin
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev