Re: [Openstack] [SWIFT] PUTs and GETs getting slower

[John Dickinson]

There are a few things you can look at.

First, the slowdowns you are seeing may be because of the increased numbers of 
files on disk. This causes the background processes to do more work, which can 
cause some issues for the server processes.

Second (and perhaps more likely), how is your data arranged? Are you putting 
every object into the same container? Swift is optimized for access across the 
entire namespace, so spreading your data across many containers can help 
improve write performance. Since you are running with account and proxy servers 
on the same spinning media as the object servers, as you get your cluster more 
full, write performance will degrade significantly with many objects in a 
single container. With your setup, you should probably have a max of about a 
million objects in a single container. If you separate your account and 
container servers to use SSDs (and perhaps separate machines), you could do 
100x this number effectively. Note that this issue doesn't affect read 
performance. Object eead performance should stay high no matter the cardinality 
of objects in a single container.


--John



On Aug 5, 2013, at 4:09 AM, Jonathan Lu  wrote:

> Hi,
> I suffer the same issue that our swift system gets slower and slower. I 
> tried almost all the guide in the document but it does little help.
> 
> Maybe swift is not suitable to the circumstance of putting and getting 
> many little files. Maybe is it more suitable to the large object such as the 
> image of OS? I am not quite sure.
> 
> Best Regards
> Jonathan Lu
> 
> On 2013/8/5 18:35, Hua ZZ Zhang wrote:
>> Here's the deployment guide which can give you options for tuning container 
>> replicator and auditor, such as concurrency.
>> http://docs.openstack.org/developer/swift/deployment_guide.html 
>> hope this can help you.
>> 
>> Best Regards, 
>> 
>> 
>> Edward Zhang(张华)
>> Advisory Software Engineer
>> Software Standards & Open Source Software
>> Emerging Technology Institute(ETI)
>> IBM China Software Development Lab
>> e-mail: zhu...@cn.ibm.com
>> 
>> 
>> 
>> 
>> 
>> Klaus Schürmann ---08/05/2013 06:13:43 PM---Klaus 
>> Schürmann 
>> 
>> Klaus Schürmann  
>> 08/05/2013 06:14 PM
>> 
>> 
>> To
>> 
>> Hua ZZ Zhang/China/IBM@IBMCN,
>> 
>> cc
>> 
>> "openstack@lists.openstack.org" 
>> 
>> Subject
>> 
>> AW: [Openstack] [SWIFT] PUTs and GETs getting slower
>> 
>> Here is my container-server.conf:
>>  
>> [DEFAULT]
>> log_name = container-server
>> log_facility = LOG_LOCAL6
>> log_level = INFO
>> log_address = /dev/log
>> bind_ip = 10.12.1.1
>> workers = 8
>> log_statsd_host = 10.11.2.1
>> log_statsd_port = 8125
>> log_statsd_default_sample_rate = 0.25
>> log_statsd_metric_prefix = Storage01
>>  
>> [pipeline:main]
>> pipeline = recon container-server
>>  
>> [filter:recon]
>> use = egg:swift#recon
>> recon_cache_path = /var/cache/swift
>>  
>> [app:container-server]
>> use = egg:swift#container
>>  
>> [container-replicator]
>> concurrency = 2
>>  
>> [container-updater]
>> concurrency = 1
>>  
>> [container-auditor]
>>  
>> [container-sync]
>>  
>>  
>> 
>> For the second question, Swift doesn't use  updatedb.mlocate in my mind. it 
>> should be safe to disable it.
>> Based on the TOP output, it seems that the CPU workload of container 
>> replicator and auditor are too high.
>> Perhaps you need to tune the config file to decrease their workload. paste 
>> your config file of
>> container servers.
>> 
>> Best Regards, 
>> Edward Zhang(张华)
>> Advisory Software Engineer
>> Software Standards & Open Source Software
>> Emerging Technology Institute(ETI)
>> IBM China Software Development Lab
>> e-mail: zhu...@cn.ibm.com
>> 
>> 
>> 
>> 
>> Klaus Schürmann ---08/05/2013 05:43:08 PM---Klaus 
>> Schürmann 
>> Klaus Schürmann   
>> 08/05/2013 05:26 PM
>> 
>>  
>> To
>> 
>> "openstack@lists.openstack.org" ,
>>  
>> cc
>> 
>>  
>> Subject
>> 
>> [Openstack] [SWIFT] PUTs and GETs getting slower
>>  
>> 
>> 
>> Hi,
>> 
>> my Swift cluster is working for three weeks now as a mail storage and the 
>> performance is getting slower and slower.
>> 
>> This is my hardware configuration:
>> 
>> - 1 hardware-loadbalancer
>> - 2 proxy-server DELL R420 16 GB RAM 2 Quadcore CPU
>> - 5 proxy-server DELL R720xd 16 GB RAM 2 Quadcore CPU with 3x 2TB HDD SATA 
>> 6GB/s
>> 
>> The cluster contains:
>> 
>> 279.000 containers
>> 8.170.000 objects = 984 GByte
>> 
>> I have about 3.000.000 GETs per day and 1.500.000 PUTs.
>> 
>> The load is raising up to 12 now:
>> 
>> top - 11:06:39 up 114 days, 23:10,  1 user,  load average: 10.88, 11.26, 
>> 11.22
>> Tasks: 175 total,   3 running, 172 sleeping,   0 stopped,   0 zombie
>> %Cpu(s): 15.6 us,  9.8 sy,  0.0 ni, 69.3 id,  4.5 wa,  0.0 hi,  0.8 si,  0.0 
>> st
>> KiB Mem:  16386408 total, 14829360 used,  1557048 free,30796 buffers
>> KiB Swap: 17575932 total,   132604 used, 17443328 free,  5572360 cache

Re: [Openstack] can all objects be stored under a single account?

[John Dickinson]

The auth system and how you organize the data are separate. You can certainly 
store all objects in one account (although I'd recommend you spread the objects 
across many containers). You could also not use any auth at all (by removing 
tempauth or keystone from the pipeline in the proxy server config), although 
only you know if that's a good idea or not.

--John




On Aug 22, 2013, at 6:44 PM, pangj  wrote:

> Hi,
> 
> Given the case that we just want to store objects in Swift, but don't want 
> the auth process (all the storage devices are in intranet).
> 
> Swift has a tempauth filter. Can I store all objects under the system/root 
> account by tempauth? Or is there any better way?
> 
> Thanks.
> 
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [swift] account and container with separated servers

[John Dickinson]

It depends on your cluster, but there are good reasons for doing so. Since 
Swift's background processes are constantly scanning the drives to check for 
errors, processes like the object-auditor and object-replicator can clear out 
the system's page cache simply by scanning the local filesystem's inodes. If 
you have account and container servers deployed on separate machines from your 
object servers, requests to an account or container will be more likely to have 
the data in the local page cache. This can raise the overall object write 
throughput of your cluster.


--John




On Aug 27, 2013, at 6:01 AM, Swift Ops  wrote:

> Hi,
> 
> Is it good to setup accounts and containers with separated servers,
> other than deploying them together with object servers?
> 
> Thanks.
> 
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [swift] Issue of cache eating up most of memory

[John Dickinson]

This seems to be a popular question this morning... Take a look at 
http://lists.openstack.org/pipermail/openstack/2013-August/000849.html and 
https://answers.launchpad.net/swift/+question/234691 as well.

The page cache usage means that the system has to go to disk to get the data 
when it is requested. This is slow, especially when eg a single container  is 
being used frequently. The slowness of spinning drives, especially when they 
are busy with lots of requests, can lead to timeouts in the system. Beyond the 
pithy "buy more memory" answer, there are a couple of things you can do. First, 
you can deploy account and container servers on high-IOPS devices (eg SSDs). 
Second, you can deploy account and container servers on separate physical 
machines than the object servers. Larger clusters (either measured by capacity 
or by throughput needs) generally end up employing both of these strategies to 
ensure a highly performant Swift cluster.

--John



On Aug 27, 2013, at 7:15 AM, Hua ZZ Zhang  wrote:

> Hi Stackers,
> 
> We have memory issue when running COSbench tool against Swift. The memory was 
> consumed rapidly by Swift.
> I carefully checked the memory info and found most of the memory was consumed 
> by page cache and inode cache. It will
> fail the server to accept any request.  I know it can be released by using 
> the command of echo 3 > /proc/sys/drop_caches. 
> But does anybody have better solution to address this issue? 
> 
> -Zhang Hua (Edward)
> 
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] Which functions are invoked in swift source code

[John Dickinson]

swift-init looks for a "--once" argument. If it's present, then the once kwarg 
will be set in the call to swift.common.daemon.run_daemon(). This in turn calls 
the appropriate run_once or run_forever.

https://github.com/openstack/swift/blob/master/swift/common/daemon.py#L55


--John


On Oct 7, 2013, at 9:36 PM, pragya jain  wrote:

> hello all,
> 
> when we give the command 
> $ sudo swift-init accoount-auditor start
> 
>  __init__ functions defined in the class AccountAuditor in 
> swift/account/auditor.py is invoked.
>   
> But, I don't understand
> 1. How does the control go to run_forever() or run_once() function defined in 
> the same class?
> 2. Who will decide whether to run run_forever() function or run_once() 
> function?
> 
> Please help me in this regard
> --
> 
> Thanks 
> 
> Pragya jain
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [SWIFT] managing the ring.gz files?

[John Dickinson]

I've heard of a few ways:

1) use existing puppet or chef or fabric or salt or dsh scripts to treat them 
like normal config files

2) use rsync to sync the ring files every so often (eg via cron)

3) host a web server on the admin box where you made the scripts and wget or 
curl them in the swift cluster

4) as you mentioned, store them in a VCS and fetch/pull/checkout/whatever on 
the storage nodes

5) deploy the builder files and generate the rings locally (be sure to set the 
same seed!)

Basically, the problem is moving a file from one place to many places. There 
are a lot of ways to do this.


--John




On Nov 18, 2013, at 2:48 PM, Stephen Wood  wrote:

> Hello.
> 
> I am wondering how others manage the {account|container|object}.ring.gz files 
> for large deployments. Specifically, how do you propagate these files once a 
> change is made?
> 
> I am manually copying them out to every node after a change, but that does 
> not seem very practical to me. I could store them in some scm system, but I 
> have a feeling someone out there has already given this some thought and can 
> give me advice.
> 
> -- 
> Stephen Wood
> www.heystephenwood.com
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] Swift deployment in production

[John Dickinson]

Swift has a modular design, and this allows you the flexibility to deploy to 
match your needs.

Common deployment patterns are (1) proxy, account, container, object all 
running on the same hardware (2) proxy on one SKU and account+container+object 
on another SKU (3) proxy on one SKU, account+container on one SKU, and object 
on one SKU

I know of production clusters in each of those styles. You would choose one 
based on your use case, scale, budget, and integration concerns. Specifically 
to the last point, you need to think about load balancing, SSL termination, 
identity integration, monitoring and altering, log aggregation, network 
topologies, etc. There's a ton of factors to consider, but the fundamental 
design of Swift let's you compose the pieces of the storage engine into a 
storage system that works very well for you.

--John





On Nov 24, 2013, at 8:56 PM, Pravar Jawalekar  wrote:

> Dear Stackers!!
> 
> I have one question related to 'how swift is actually deployed in production 
> environment',
> How object/container/auth servers made communicate to actual underlying 
> storage infrastructure?
> I mean do deployment strategy of swift imposes any such restrictions that 
> services for container, object etc. should be running on node where actual 
> storage is present?
> 
> I am sorry if my questions is confusing.
> But please throw some light in this area - although i have gone through 
> deployement documents found on open stack sites but they did not seem to give 
> real picture of swift deployment in real world.
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [Swift] Changing local logging destinations?

[John Dickinson]

While a Swift-All-In-One certainly isn't something your should run in 
production, the SAIO document does have some guidance on how to configure 
rsyslogd to split out log messages.

http://docs.openstack.org/developer/swift/development_saio.html#optional-setting-up-rsyslog-for-individual-logging

If you are interested in seeing some configs for syslog-ng (IMO it has a nicer 
config format), then find me on IRC (I'm notmyname) in #openstack-swift and I 
can share what I have.

--John



On Nov 27, 2013, at 3:04 PM, Stephen Wood  wrote:

> Can someone give me advice on how to redirect logs from /var/log/syslog to 
> /var/log/swift/{object|account|container}.logs?
> 
> -- 
> Stephen Wood
> www.heystephenwood.com
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [SWIFT] Clarification for partition number

[John Dickinson]

The guys over at Rackspace put together a nice little ring part power 
calculator.

http://rackerlabs.github.io/swift-ppc/

--John


On Dec 4, 2013, at 12:17 PM, Stephen Wood  wrote:

> I'm wondering if I have the partition number correct for the best performance 
> for our setup (we have 23 hosts with 12 4TB disks each).
> 
> From my understanding, partitions is N in 2^N = x with x being 100 * number 
> of disks in cluster. So in the case above that would be around 15?
> 
> -- 
> Stephen Wood
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [SWIFT] Why is sharding across containers such a big deal?

[John Dickinson]

correct. a single container is replicated like other data in the system 
(typically 3x). This means that a single container is on only 3 spindles, and 
an nymber of concurrent writes to objects in that container will attempt to 
update the container listing (with graceful failure handling). This means that 
under significant concurrency, the concurrent object write speed is limited by 
the time it takes to update one of those container replicas.

There are two easy "fixes" for this: (1) shard your data across containers and 
(2) use faster, dedicated drives for the containers (eg SSDs).

The hard fix for this is to implement container sharding within swift, but this 
is a hard problem to solve (although nobody would be opposed to a successful 
solution).

--John





On Dec 4, 2013, at 3:01 PM, Stephen Wood  wrote:

> Can someone explain to me (or point me to some good literature) about why 
> sharding across containers is such a big deal in terms of performance? Is it 
> that a single container is typically localized across a small number of 
> shards?
> 
> -- 
> Stephen Wood
> www.heystephenwood.com
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [Swift] Object-Expirer not expiring objects

[John Dickinson]

What version of Swift are you using? You can find it with `python -c 'import 
swift; print swift.__version__'`.

If you have a version later than 1.10.0 (ie the Havana release), you may be 
running into https://bugs.launchpad.net/swift/+bug/1257330 (for which we have a 
patch in progress at https://review.openstack.org/#/c/59811/).

--John




On Dec 4, 2013, at 9:52 PM, Matt Harlum  wrote:

> Hi,
> 
> I’m currently trying to get the Object-expiry working, it shows that there 
> are objects to delete, but for some reason it just doesn’t do it
> 
> When object-expirer runs, it shows that there are objects to delete but 
> doesn’t do so. I have been trying to get this working for months without any 
> luck, am I missing something?
> 
> Config:
> 
> [DEFAULT]
> swift_dir = /etc/swift
> user = swift
> log_facility = LOG_LOCAL0
> log_level = DEBUG
> 
> [object-expirer]
> auto_create_account_prefix = .
> 
> [pipeline:main]
> pipeline = recon catch_errors cache proxy-server
> 
> [app:proxy-server]
> use = egg:swift#proxy
> 
> [filter:recon]
> use = egg:swift#recon
> recon_cache_path = /var/cache/swift
> 
> [filter:cache]
> use = egg:swift#memcache
> memcache_servers = 127.0.0.1:11211
> 
> [filter:catch_errors]
> use = egg:swift#catch_errors
> 
> 
> swift-object-expirer /etc/swift/object-expirer.conf -v -o
> object-expirer Run begin
> swift - - 05/Dec/2013/05/49/14 HEAD /v1/.expiring_objects HTTP/1.0 204 - 
> Swift%20Object%20Expirer - - - - txb51bcf9ecc694987b3c40d363519ab00 - 0.0029 -
> object-expirer Pass beginning; 2 possible containers; 2 possible objects
> swift - - 05/Dec/2013/05/49/14 GET 
> /v1/.expiring_objects%3Fformat%3Djson%26marker%3D%26end_marker%3D HTTP/1.0 
> 200 - Swift%20Object%20Expirer - - 96 - tx4d7393a8cfd64cfb9186ce028d4a6dd2 - 
> 0.0021 -
> swift - - 05/Dec/2013/05/49/14 GET 
> /v1/.expiring_objects/1377129600%3Fformat%3Djson%26marker%3D%26end_marker%3D 
> HTTP/1.0 404 - Swift%20Object%20Expirer - - 52 - 
> txc6b0e2737d874afa9aff1227cce0f972 - 0.0055 -
> swift - - 05/Dec/2013/05/49/14 DELETE /v1/.expiring_objects/1377129600 
> HTTP/1.0 404 - Swift%20Object%20Expirer - - 52 - 
> txb3aa5fa3c54b4b478634fec4ddc6845f - 0.0038 -
> object-expirer Run end (txn: txb3aa5fa3c54b4b478634fec4ddc6845f)
> object-expirer Pass completed in 0s; 0 objects expired
> 
> 
> Thanks,
> —Matt
> 
> 
> 
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] can I access the data stored on swift through a web browser

[John Dickinson]

Yes, you can absolutely do this.

As mentioned in another reply, Swift's API is built on HTTP, so it works 
natively with web browsers.

There are a couple of ways to read data stored in Swift via a web browser.

1) For public content:
If you've marked a container as public (ie X-Container-Read set to ".r*"), then 
you can access any object in that container directly without worrying about 
auth tokens. eg `curl -i http://swift.example.com/v1/AUTH_test/images/cat.jpg`

2) For non-public content:
You can generate a tempurl to make a signed URL that has limited-time access to 
read a particular object. See 
http://docs.openstack.org/developer/swift/misc.html#module-swift.common.middleware.tempurl
 for more info.



There are also a couple of different ways to use a web browser (or app running 
in a browser) to write content into Swift.

1) Using a tempurl, you can give time-limted access to write content into a 
cluster, as mentioned above

2) Submit to Swift using an HTML form:
You can send a form post to a Swift cluster as documented at 
http://docs.openstack.org/developer/swift/misc.html#module-swift.common.middleware.formpost


In all cases, tying a Swift cluster to a web application may require you make 
use of the CORS standard to get around browser-based security sandboxes. Swift 
fully supports the CORS standard, as documented at 
http://docs.openstack.org/api/openstack-object-storage/1.0/content/assigning-cors-headers-to-requests.html.
 To read about the CORS spec see http://www.w3.org/TR/cors/.

These features make Swift a perfect pair with applications that demand support 
for high-concurrency access to massive storage pools (like web and mobile apps).

Hope that info helps!


--John






On Dec 13, 2013, at 11:54 AM, pragya jain  wrote:

> hello,
> I installed swift on my machine. 
> I store some data on it using swift command line. 
> Now, I want to know: 
> can I access the data stored on swift through a web browser?
> --
> 
> Pragya Jain
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] Q: Switft OwnCloud

[John Dickinson]

I don't have any personal experience with ownCloud, but I've heard of others 
using it. I will absolutely vouch for Swift as a good solution for file 
sharing, especially as the aggregate size of your content grows.

--John




On Dec 16, 2013, at 9:14 PM, Frans Thamura  wrote:

> Hi all
> 
> 
> I am thinking to use swift on own cloud, is it possible ?
> 
> 
> or is there alternative to use swift as file sharing in office and on internet
> 
> 
> F
> 
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] Object storage only - Swift or Ceph ?

[John Dickinson]

You should try both and choose the one that best meets your needs. Chmouel at 
eNovance wrote a great summary of the two systems at 
http://techs.enovance.com/6427/ceph-and-swift-why-we-are-not-fighting.

--John



On Dec 23, 2013, at 9:24 AM, Cedric Lemarchand  wrote:

> Hello,
> 
> I am evaluating an object storage solution based on OpenStack (only object 
> storage, no needs for Nova/Cinder).
> The common way would be the use of Swift (which I have already played a bit 
> with, the concept of the "ring" is simply awesome ...).
> The other way is Ceph, it seams to be the "new challenger", in the way that 
> it can handle file/block/object, but seams to be not ready yet for production 
> use, and not very well documented (yet).
> 
> So the question is what would be the best technology to start with now ? I 
> mean does Ceph could replace Swift in a near future (eg IceHouse or the next 
> one) ?
> 
> Any advices are welcome, and will be greatly appreciated.
> 
> Cédric
> 
> 
> 
> 
> 
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] international standard for object storage architecture

[John Dickinson]

I think you'll find that object storage is relatively new enough in the storage 
world and there isn't an international standard (or at least one that is widely 
used).

I have seen three things, primarily.

1) Amazon S3: huge because the AWS ecosystem is huge.
2) CDMI: s standard promoted by SNIA that isn't widely adopted.
3) OpenStack Swift: part of OpenStack and being added to other storage systems 
too.

Swift has community plugins to support the S3 API 
(https://github.com/fujita/swift3) and the CDMI API 
(https://github.com/osaddon/cdmi).

I'm obviously biased here, but I want to see the Swift API used any time an 
application is speaking to an object storage system. And I think we've made 
great progress towards this goal in the last few years. By using Swift, you 
have support for some of the other APIs out there and you get native support 
for the Swift API itself.

--John

 
On Jan 1, 2014, at 12:39 AM, pragya jain  wrote:

> hello,
> 
> i am searching for an international standard for object storage architecture 
> on cloud.
> Can somebody help me?
> 
> Pragya Jain
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [Swift] Object use cases

[John Dickinson]

You are asking for practical examples, so here are some links to get started:

https://blog.wikimedia.org/2012/02/09/scaling-media-storage-at-wikimedia-with-swift/

http://www.openstack.org/summit/openstack-summit-hong-kong-2013/session-videos/presentation/case-study-concur-delivering-a-saas-application-with-swift

http://www.mirantis.com/blog/cisco-webex-mirantis-openstack-private-cloud-production/

https://cloud.sdsc.edu/hp/index.php

http://www.encoding.com/help/sub_category/category/Rackspace_Cloud_Files

http://www.prweb.com/releases/2013/10/prweb10893350.htm

http://www.buildcloudstorage.com/2013/12/3-reasons-customers-will-store-exabytes.html

http://engineering.spilgames.com/openstack-swift-lots-small-files/

http://blog.adityapatawari.com/2014/01/using-openstack-swift-as-owncloud.html

https://github.com/wecreatepixels/django-storage-swift

So almost all of the above are using private cloud storage with Swift (ie 
rather than a public cloud provider). You can also look into how Rackspace, HP, 
Softlayer(IBM), NTT, Internap, Memset, KT, Cloudwatt, and other service 
providers around the world use Swift. Their respective customer bases is large 
and has a million different uses.

--John



On Jan 20, 2014, at 5:56 PM, Mark Brown  wrote:

> Hello,
> I wanted to get an idea about what kind of use cases Object is being used 
> for, and had some questions around that.
> 
> 1.  What kind of use cases is object used for in the public cloud? What kinds 
> of data are enterprises storing most commonly?
> 
> 2. Is anyone deploying Swift within the data center for a private cloud 
> object solution, and if yes, what are the use cases for that as compared to 
> say, using a NAS solution within the data center? I am looking for concrete 
> information on actual use cases in terms of kinds of applications and the 
> kind of data stored.
> 
> 3. Are companies building applications with object as the back end in mind 
> rather than file which is the more common case? Any examples?
> 
> Thanks in advance!
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] Verifying Swift - X-Auth problem

[John Dickinson]

In the output you pasted, you don't have any successful response. I'd suggest 
looking at the tempauth stanza in the proxy server conf to make sure the users 
are set up correctly.

--John



On Feb 7, 2014, at 4:55 PM, Adam Lawson  wrote:

> To help with troubleshooting, here is what I've executed thus far on my proxy 
> node...
> Obvious problem/symptom = inability to verify a new Swift install from 
> scratch due to 401 Unauthorized.
>   • 1x proxy node
>   • 5x storage nodes
> I'll continue working this but anyone have any thoughts? See email to 
> -operators list for further history.
> 
> Thanks!
> Adam
> 
> Below is a bash history/output of what is happening right now:
> login as: c5201274
> c5201274@10.173.0.66's password:
> Welcome to Ubuntu 12.04.3 LTS (GNU/Linux 3.2.0-55-generic x86_64)
>  * Documentation:  https://help.ubuntu.com/
> Last login: Thu Feb  6 21:05:32 2014 from 10.7.106.110
>  Powered by Monsoon (Version 2.2.465)  Platform: ubuntu 12.04
>  Hostname : mo-ad1469a10.mo.sap.corp   Name: node0p
>  Organization : c5201274   Project : swift_poc
>  Url  : https://monsoon.mo.sap.corp/instances/mo-ad1469a10
> c5201274@mo-ad1469a10:~$ sudo su
> root@mo-ad1469a10:/home/c5201274# . credrc.sh
> root@mo-ad1469a10:/home/c5201274# swift-init proxy start
> proxy-server running (5502 - /etc/swift/proxy-server.conf)
> proxy-server already started...
> root@mo-ad1469a10:/home/c5201274# curl -k -v -H 'X-Storage-User: test:tester' 
> -H 'X-Storage-Pass: testing' https://$PROXY_LOCAL_NET_IP:8080/auth/v1.0
> * About to connect() to 10.173.0.66 port 8080 (#0)
> *   Trying 10.173.0.66... connected
> * successfully set certificate verify locations:
> *   CAfile: none
>   CApath: /etc/ssl/certs
> * SSLv3, TLS handshake, Client hello (1):
> * SSLv3, TLS handshake, Server hello (2):
> * SSLv3, TLS handshake, CERT (11):
> * SSLv3, TLS handshake, Server finished (14):
> * SSLv3, TLS handshake, Client key exchange (16):
> * SSLv3, TLS change cipher, Client hello (1):
> * SSLv3, TLS handshake, Finished (20):
> * SSLv3, TLS change cipher, Client hello (1):
> * SSLv3, TLS handshake, Finished (20):
> * SSL connection using AES256-SHA
> * Server certificate:
> *subject: C=AU; ST=Some-State; O=Internet Widgits Pty Ltd
> *start date: 2014-01-29 00:34:55 GMT
> *expire date: 2014-02-28 00:34:55 GMT
> * SSL: unable to obtain common name from peer certificate
> > GET /auth/v1.0 HTTP/1.1
> > User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 
> > zlib/1.2.3.4libidn/1.23 librtmp/2.3
> > Host: 10.173.0.66:8080
> > Accept: */*
> > X-Storage-User: test:tester
> > X-Storage-Pass: testing
> >
> < HTTP/1.1 401 Unauthorized
> < Content-Length: 131
> < Content-Type: text/html; charset=UTF-8
> < Date: Fri, 07 Feb 2014 18:20:13 GMT
> <
> * Connection #0 to host 10.173.0.66 left intact
> * Closing connection #0
> * SSLv3, TLS alert, Client hello (1):
> UnauthorizedThis server could not verify that you are 
> authorized to access the document you 
> requested.root@mo-ad1469a10:/home/c5201274#
> root@mo-ad1469a10:/home/c5201274# curl -k -v -H 'X-Auth-Token: 
> AUTH_tkf85b7788c36143ac99e5a5b42d95d628' 
> https://$PROXY_LOCAL_NET_IP:8080/v1/AUTH_system
> * About to connect() to 10.173.0.66 port 8080 (#0)
> *   Trying 10.173.0.66... connected
> * successfully set certificate verify locations:
> *   CAfile: none
>   CApath: /etc/ssl/certs
> * SSLv3, TLS handshake, Client hello (1):
> * SSLv3, TLS handshake, Server hello (2):
> * SSLv3, TLS handshake, CERT (11):
> * SSLv3, TLS handshake, Server finished (14):
> * SSLv3, TLS handshake, Client key exchange (16):
> * SSLv3, TLS change cipher, Client hello (1):
> * SSLv3, TLS handshake, Finished (20):
> * SSLv3, TLS change cipher, Client hello (1):
> * SSLv3, TLS handshake, Finished (20):
> * SSL connection using AES256-SHA
> * Server certificate:
> *subject: C=AU; ST=Some-State; O=Internet Widgits Pty Ltd
> *start date: 2014-01-29 00:34:55 GMT
> *expire date: 2014-02-28 00:34:55 GMT
> * SSL: unable to obtain common name from peer certificate
> > GET /v1/AUTH_system HTTP/1.1
> > User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 
> > zlib/1.2.3.4libidn/1.23 librtmp/2.3
> > Host: 10.173.0.66:8080
> > Accept: */*
> > X-Auth-Token: AUTH_tkf85b7788c36143ac99e5a5b42d95d628
> >
> < HTTP/1.1 401 Unauthorized
> < Content-Length: 131
> < Content-Type: text/html; charset=UTF-8
> < Date: Fri, 07 Feb 2014 18:21:22 GMT
> <
> * Connection #0 to host 10.173.0.66 left intact
> * Closing connection #0
> * SSLv3, TLS alert, Client hello (1):
> UnauthorizedThis server could not verify that you are 
> authorized to access the document you 
> requested.root@mo-ad1469a10:/home/c5201274# ^C
> root@mo-ad1469a10:/home/c5201274# ^C
> root@mo-ad1469a10:/home/c5201274# curl -k -v -H 'X-Auth-Token: 
> AUTH_tkf85b7788c36143ac99e5a5b42d95d628' 
> https://$PROXY_LOCAL_NET_IP:8080/v1

Re: [Openstack] [SWIFT] Container server timeouts

[John Dickinson]

Moving your container servers to SSDs is the fastest way to get better 
performance out of them.

Although there have been many discussions about how to do internal container 
sharding, I'm not aware of any current ongoing work there (although I'd love to 
see someone work on it).

--John




On Feb 10, 2014, at 12:32 PM, Stephen Wood  wrote:

> We've been having issues with our container servers timing out and simply 
> returning a 404. The container servers themselves are on busy disks that 
> aren't particularly overloaded, but are definitely under load.
> 
> The messages look like the following:
> Feb 10 20:22:19 proxy01 swift ERROR with Container server 
> storage01:6001/slot-11 re: Trying to GET /v1/AUTH_swift/very_busy_container: 
> Timeout (10s) (txn: tx6b921c72e5524589839fe-0052f934e7
> 
> I realize this container has too many keys, but unfortunately I cannot shard 
> it for now because we need to keep parity with S3 (which does internal bucket 
> sharding).
> 
> What are the best ways to tackle busy container servers timing out? Should I 
> increase the node timeout, or begin creating separate container servers that 
> are running on SSDs? Is there any serious drawback to increase the node 
> timeout?
> 
> Also, has there been any update on internal container sharding? This feature 
> would make S3 migrations much easier.
> 
> Any advice would be very helpful. Thank you.
> 
> -- 
> Stephen Wood
> www.heystephenwood.com
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [SWIFT] A way to tell how much data is used on container servers?

[John Dickinson]

Look on your existing drives in your cluster and see how much space the 
"account" and "container" directories are using (eg `du -sh 
/srv/node/*/{accounts,containers}`). Sum that across all the servers in your 
cluster, and you've got it.

As a general rule of thumb (ie HUGE ASSUMPTIONS MADE HERE), account and 
container data is generally below 1% of the space used for objects. At least 
that's a starting point you can use before you have real data. As always, base 
decisions on data over recommendations.

--John




On Feb 17, 2014, at 7:41 PM, Stephen Wood  wrote:

> I'm preparing to migrate our container servers to SSD hosts for better 
> performance, but I'm unsure on how to calculate the space required on these 
> new hosts. Is there a way to approximate how much data the container servers 
> are using?
> 
> -- 
> Stephen Wood
> www.heystephenwood.com
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [Swift] Swift versions and release schedule

[John Dickinson]

Swift absolutely and without a doubt participates in OpenStack releases. 
OpenStack releases on a six-month cadence, and Swift participates in this 
integrated release (as do all other OpenStack projects).

In between the OpenStack integrated releases, Swift differs from some other 
OpenStack projects by not doing milestone releases. Instead, Swift releases 
stable, production-ready versions. All Swift releases use a semantic versioning 
scheme. Currently we're on 1.12.0 and our next release will be 1.13. We have a 
release of Swift near the end of the six-month OpenStack cycle that is included 
in the integrated release. For example, OpenStack Havana included Swift 1.10.

I hope this clears things up.

--John



On Feb 20, 2014, at 10:48 AM, Shrinand Javadekar  
wrote:

> Hi,
> 
> I know that Swift releases do not necessarily coincide with major
> Openstack releases (like Grizzly, Havana, etc.).
> 
> 1) Does Swift have a fixed release cadence?
> 
> 2) Also, I see that the current releases are all versioned as 1.XY.
> Does this mean that these are minor releases and some major Swift
> release will happen sometime soon when the version # will be bumped to
> 2.0?
> 
> We develop software that uses Swift as backend storage. I need to
> figure out how many Swift versions we need to keep running tests
> against. And whether we need to do something special for major/minor
> Swift releases.
> 
> Thanks in advance.
> -Shri
> 
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [Swift] Capturing performance penalities Physical versus VM

[John Dickinson]

There are 2 reasons you don't want to run Swift on VMs:

1) (minor reason, depending on the use case) performance. Virtualized IO is 
going to have worse performance for storage than bare metal. In some cases this 
may not matter, and in others you may use Super Awesome Virt (tm) that doesn't 
have any IO penalty. But in general, there will be some performance penalty. 
It's up to you to see if it's worth it or not.

2) (major reason) failure domains. Swift is designed to decouple the data from 
the storage media. This is done by abstracting away the storage volumes (ie 
drives) and presenting it to the application as a unified namespace. Swift does 
data placement and failure handling based on the failure domains in the 
underlying hardware. So if you have 3 replicas of the data and 5 servers, Swift 
will ensure that none of the three replicas are on the same server. This gives 
you high availability and means that hardware failure can be seamlessly 
handled. If you hide the failure domains from Swift by running on virtual 
servers, you will be at higher risk for a loss of availability (at best) and 
loss of data (at worse) because your data may in fact be stored on the same 
physical hardware underlying the virtual machines.[1] This is the main reason 
you don't want to run Swift on top of other systems (hardware or software) that 
hide the failure domains. Swift is designed to work around failures (and does a 
really good job). Let it do it's job.

[1] And since VMs are designed to be moved around, starting with "we'll just 
make sure our VMs are on different physical servers" doesn't seem like a 
long-term tenable plan.

--John






On Feb 21, 2014, at 1:47 PM, Adam Lawson  wrote:

> I have a large client who wants to execute a global deployment of Swift in 
> Production but do it on VM's, shared storage and RAID.
> 
> Read on after you stop laughing. Serenity now!
> 
> Okay, their splendid idea is to start with VM's in Production, add physical 
> hardware later then retire the VM's. This direction is completely against my 
> recommendations but we're still talking and the direction isn't final so 
> that's a good thing. I think I just need some relevant benchmarks to help 
> communicate why deploying on VM's isn't a Production-worthy strategy when 
> we're talking about Swift.
> 
> So, are there benchmarks already published somewhere that demonstrate how/why 
> a virtual infrastructure is such a bad idea with Swift? We all know Swift is 
> designed to be a storage provider - not a storage consumer and that running 
> Swift with shared storage with RAID is utterly foolish. But my recommendation 
> is being overridden by folks who feel the bad performance isn't really all 
> that bad and because the VM's are going to be temporary. We all know 
> temporary = permanent when you're talking about Production, . I don't have 
> physical hardware to execute benchmarks and prove my point (well not yet 
> anyway).
> 
> Thoughts/suggestions?
> 
> 
> Adam Lawson
> AQORN, Inc.
> 427 North Tatnall Street
> Ste. 58461
> Wilmington, Delaware 19801-2230
> Toll-free: (888) 406-7620
> 
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] Openstack Swift load small files

[John Dickinson]

Yep, you're right. Doing a HEAD request before every PUT gets expensive, 
especially for small files.

But don't despair! There's some good news.

First, realize that swiftclient is written for a pretty general use case. If 
you have more knowledge about how your system works, then you can write 
something more efficient. python-swiftclient has a CLI, yes, but it also 
provides a python module so that you can write your own client as needed. For 
example, you can write your own script that doesn't do HEAD requests before a 
PUT.

Second, this specific issue is being addressed in patch 
https://review.openstack.org/#/c/81646/ which adds If-None-Match support to PUT 
requests in Swift. This will allow a PUT without a HEAD and also avoid data 
transfer if the object already exists. I hope this patch lands today and is 
included in Icehouse.

--John





On Apr 2, 2014, at 7:52 AM, Roman Kravets  wrote:

> Hello.
> 
> I have trouble with python-swiftclient during upload to match small
> file to Openstack Swift.
> 
> It client make HEAD request for each file during upload before PUT. It
> make very intencive read load to hard drive and as result all cluster
> show very small upload speed.
> 
> I see that for each file it make 6 HEAD request (3 recplicas * 2).
> 
> When I use cloudfuse this client make only PUT and as result hard
> drive load only to write data. It is make less IOps to hard drive.
> 
> What need make HEAD request for each files? How can I reduce the read
> request for reduce load to hard drive?
> 
> Thank you!
> 
> --
> Best regards,
> Roman Kravets
> 
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [Swift] Restricted replication?

[John Dickinson]

That's not supported today, but it's one of the exact use cases being 
implemented with storage policies.

https://swiftstack.com/blog/2014/01/27/openstack-swift-storage-policies/

http://lists.openstack.org/pipermail/openstack-dev/2014-March/030937.html

--John



On Apr 21, 2014, at 1:20 PM, Adam Lawson  wrote:

> I haven't found this answer yet:
> 
> Is there a setting for a single multi-region Swift cluster to force some 
> regions to keep all data replicated within that one region and nowhere else 
> versus other regions where the data can be replicated throughout the entire 
> cluster?
> 
> 
> Adam Lawson
> AQORN, Inc.
> 427 North Tatnall Street
> Ste. 58461
> Wilmington, Delaware 19801-2230
> Toll-free: (844) 4-AQORN-NOW
> Direct: +1 (302) 268-6914
> 
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [openstack][SWIFT] I want to talk about costs !! (against amazon S3)

[John Dickinson]

Alejandro,

We blogged about this recently at 
https://swiftstack.com/blog/2014/04/24/about-those-public-cloud-price-drops/. 
I've asked around the office here at SwiftStack to see if we can put together 
some more public cost details (specifically in response to you question), so I 
hope we can share some more info soon.

--John



On Apr 30, 2014, at 5:11 AM, Alejandro Comisario 
 wrote:

> Hi guys.
> After amazon dropped its prices among other things, on S3 its seems that i 
> need in my compañy to justify/analyze if swift keeps being the most cost 
> convenient solution (we have it in house)
> 
> When we decided going with swift one year ago, one of the main constraints 
> was "keeping the data in our datacenter" so we did'nt do the excercice of 
> costs, but today since "seems" that this constraint might be no more i need 
> to go through this excercise (of course i want to stay with swift)
> 
> So, im reaching the community to ask experiences of people that went through 
> the same scenario, things to take into account and recent results.
> 
> I bumped with a great aticle from Amar Kapadia at: 
> http://www.buildcloudstorage.com/2012/01/can-openstack-swift-hit-amazon-s3-like.html
>  and since i belive is a great place to start, the calculations an variables 
> might be outdated.
> 
> help is more than apreciated!!
> 
> 
> ​best.
> alejandrito​
> 
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [openstack][SWIFT] I want to talk about costs !! (against amazon S3)

[John Dickinson]

On Apr 30, 2014, at 11:59 AM, Dimitri Maziuk  wrote:

> I'd say an interesting question is how many users want storage that only
> lets you put, get, and delete a file. A private cloud storage is
> trivially re-exportable as a filesystem, how easy is that with
> commercial offerings?

1) "storage that only lets you put, get, and delete a file"

I'm curious what you mean by this. Depending on your view, Swift certainly does 
a lot more than this. It also makes sure those files are stored correctly and 
protected against hardware failure (a non-trivial problem). And from the client 
API side, there are many other features (versioned writes, CORS, manifests, 
signed URLs, ACLs, etc etc etc).

But I suppose from a different perspective, Swift "just" reads and writes files.


2) "trivially re-exportable as a filesystem"

There's nothing trivial about exporting an eventually consistent, stateless, 
distributed object storage system as a filesystem. There are some current 
projects that offer a bridge, but they often have serious performance problems 
(as a result of layering protocols with completely different semantics).


--John





signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] WebDAV support in Swift

[John Dickinson]

Off the top of my head, I don't know any middleware that supports this. If you 
do find or write one, I'd love for you to add it to 
http://docs.openstack.org/developer/swift/associated_projects.html.

I've long thought that WebDAV support would be cool[1]. Not that I've seen a 
lot of people demanding it, but it just sounds neat. On the one hand, it should 
be pretty simple since you "just" need to implement the PROPINFO and PROPFIND 
verbs (IIRC). On the other, it could get really tricky when you try to add auth 
to it. To start with, I'd suggest getting it working with no auth, then figure 
out what your WebDAV clients are doing with auth.


[1] as in, I blogged about it nearly four years ago 
http://programmerthoughts.com/openstack/features-i-would-like-to-see-in-swift/

--John



On May 22, 2014, at 6:49 AM, Tarak Mukherjee  wrote:

> Any link will help which is having multiple container support ..
> --Best Rgds,
> Tarak Mukherjee
> @tmukharj
> 
> On Monday 05 May 2014 09:50 PM, Tarak Mukherjee wrote:
>> Hello Team,
>> 
>> We would like to know from the experts that , is there any middleware where 
>> we can enable "WebDAV" support for Swift .
>> --Best Rgds,
>> Tarak Mukherjee
>> @tmukharj
>> 
> 
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] ownCloud and openstack

[John Dickinson]

That's fantastic news! Hearing more companies and applications supporting Swift 
is great for the community, and it gives us as contributors yet another use 
case to reference.

I work with Hugo, so I'll peek over his shoulder to check it out. Meanwhile, if 
there's anything I can do to help answer questions or get involved in the 
community (from contributing to getting issues addressed), please let me know.

--John


> On Jul 3, 2014, at 6:01 PM, Frank Karlitschek  wrote:
> 
> Hi everyone,
> 
> 
> Tom Fifield recommended that I send a message to this list.
> 
> As some of you might know we at ownCloud just added Swift support as primary 
> storage to ownCloud. This will be released as part of our upcoming release 7 
> in about 3 weeks.
> 
> The code can be seen here.
> https://github.com/owncloud/core/pull/8383
> 
> 
> ownCloud is a fully open source solution that can be used a self hosted 
> Dropbox or Google Drive alternative. The new Swift backend is an important 
> step for us to mask ownCloud more scalable, easier to deploy in bigger setups 
> and integrate it better into OpenStack.
> 
> Tom suggested to ask here for testers who are willing to try ownCloud 7 
> together with Swift.
> We just announced the RC1 a few minutes ago:
> http://mailman.owncloud.org/pipermail/announcements/2014-July/49.html
> 
> 
> So it would be great if you guys could give it a test and give some feedback.
> 
> 
> Thanks a lot
> 
> 
> Cheers
> Frank
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [Swift] Running out of ports or fds?

[John Dickinson]

As Pete mentioned, since Swift can use a lot of sockets and fds when the system 
is under load. Take a look at 
http://docs.openstack.org/developer/swift/deployment_guide.html#general-system-tuning
 for some sysctl settings that can help. Also, note that if you start Swift as 
root (it can drop permissions), it will set the system limits for file 
descriptors. You may need to use ulimit to increase the number of fds available.

--John





On Jul 11, 2014, at 11:18 AM, Shrinand Javadekar  
wrote:

> Thanks for your inputs Edward and Pete. I'll set  sysctl 
> net.ipv4.tcp_tw_reuse.
> 
> On Fri, Jul 11, 2014 at 8:05 AM, Pete Zaitcev  wrote:
>> On Tue, 8 Jul 2014 16:26:10 -0700
>> Shrinand Javadekar  wrote:
>> 
>>> I see that these servers do not use a persistent http connection
>>> between them. So every blob get/put/delete request will create a new
>>> connection, use it and tear it down. In a highly concurrent
>>> environment with thousands of such operations happening per second,
>>> there could be two problems:
>> 
>> It's a well-known problem in Swift. Operators with proxies driving
>> sufficient traffic for it to manifest set sysctl net.ipv4.tcp_tw_reuse.
>> 
>> There were attempts to reuse connections, but they floundered upon
>> the complexities of actually implementing a connection cache.
>> Keep in mind that you still have to allow simultaneous connections
>> to the same node for concurrency. It snowballs quickly.
>> 
>> -- Pete
> 
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [Swift] API changes in v2 vs v1

[John Dickinson]

Semantic versioning is bigger than just API changes. In fact, since we 
separately version the API, we can actually be quite explicit about what's in a 
particular release. In general, we do semantic versioning in Swift for a few 
reasons, but the main one is that it's been requested and confirmed by 
deployers as what they want at almost every OpenStack summit.


With the Swift 2.0 release, we did not change the Swift API version. The API is 
still "v1" (as it has been since OpenStack began). This API stability is 
something we work very hard to keep. Although we have added a few things to the 
API, we have not done anything that would prevent clients written to the Swift 
API 4+ years ago from working against master today. As Anne mentioned, the 
storage policies feature in Swift 2.0 has only added new headers to the API. 
That is, new clients can take advantage of the new features, but old clients 
can still speak the same protocol and still work (and even take advantage of 
storage policies in a limited way).

There are two main reasons we bumped the major version of Swift in this recent 
release from 1 to 2. First, and most importantly, if a deployer upgrades to 
this release and implements multiple storage policies, the deployer will not be 
able to downgrade and still access all of the data stored across all of the 
policies. Although it's still possible to upgrade to 2.0 and then roll the 
upgrade back before defining additional storage policies (and like always 
possible to upgrade with no downtime), we wanted to explicitly communicate the 
upgrade risk with a major version bump.

The second major reason we bumped the major version number in this release is 
because storage policies are such a big deal in the life of the project. This 
is a watershed feature in Swift, and the code changes required to implement it 
significantly impacted a large part of the code base. And from the user 
perspective, storage policies allow an enormous amount of flexibility in how 
the data is stored.

So to sum up, the Swift 2.0 release is a big deal in the life of the project. 
The major version number bump in the version number is intended to communicate 
that to deployers. The API is largely unaffected and existing clients still 
work; therefore the API version has been kept at v1.

Hope this helps clear things up.

--John






On Jul 22, 2014, at 10:05 AM, Shrinand Javadekar  
wrote:

> This is confusing. So does this mean semantic versioning applies to
> the API's and not the Swift product versions? If so how are the
> product versions decided? And given a product version, how does one
> know about what API version is in use? The /info endpoint does not
> mention the API version.
> 
> I had asked about versioning sometime ago [1] and I thought the
> release version and API version are identical.
> 
> -Shri
> 
> [1] http://www.gossamer-threads.com/lists/openstack/dev/36042
> 
> 
> On Tue, Jul 22, 2014 at 5:37 AM, Anne Gentle  wrote:
>> Hi Shri,
>> The Object Storage API change is just to add a header called
>> 'X-Storage-Policy: ' where name is set in the swift configuration
>> files by the cloud provider.
>> 
>> See http://docs.openstack.org/developer/swift/overview_policies.html
>> 
>> We'll add it to the API docs next, no version change needed for the API
>> itself.
>> Anne
>> 
>> 
>> On Tue, Jul 22, 2014 at 12:46 AM, Shrinand Javadekar
>>  wrote:
>>> 
>>> Hi,
>>> 
>>> Swift has been following the semantic versioning scheme. The fact that
>>> the product version changed from v1.x to v2.0 should suggest that the
>>> Swift APIs changed in this release.
>>> 
>>> I see that storage policies has been the biggest change in this
>>> release. Has that impacted the APIs?
>>> 
>>> Is there a document that lists exactly the API difference between v1.x and
>>> v2.0?
>>> 
>>> Thanks in advance.
>>> -Shri
>>> 
>>> ___
>>> Mailing list:
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>> Post to : openstack@lists.openstack.org
>>> Unsubscribe :
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>> 
>> 
> 
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [Swift] API changes in v2 vs v1

[John Dickinson]

Adam, that sounds you're talking about the API versions. So a user is consuming 
eg Keystone v2, Nova v3, and Swift v1 (not to mention all the other projects). 
Are you asking for a singular "OpenStack API"? Or maybe a common API version 
number across projects (similar to how the requirements are managed)?


On Jul 23, 2014, at 11:40 AM, Adam Lawson  wrote:

> This makes perfect sense to me John. But consistency across the project would 
> be a nice-to-have for those who design and communicate the platform to 
> decision makers who don't understand why they're signing off v1 of this, v2 
> of that, v1 of this other one... It portrays a sense of program incongruity 
> (perceived or otherwise) which adds to existing concerns about platform 
> complexity. I spend more time explaining why than otherwise.
> 
> Logical versioning vs "Nice to have". ; )
> 
> 
> Adam Lawson
> AQORN, Inc.
> 427 North Tatnall Street
> Ste. 58461
> Wilmington, Delaware 19801-2230
> Toll-free: (844) 4-AQORN-NOW ext. 702
> Int'l: +1-302-268-6914 ext. 702
> Cell: +1-916-990-1226
> 
> 
> 
> On Tue, Jul 22, 2014 at 10:46 AM, Shrinand Javadekar 
>  wrote:
> > Hope this helps clear things up.
> 
> This does. Thanks for the detailed explanation.
> 
> -Shri
> 
> >
> > --John
> >
> >
> >
> >
> >
> >
> > On Jul 22, 2014, at 10:05 AM, Shrinand Javadekar  
> > wrote:
> >
> >> This is confusing. So does this mean semantic versioning applies to
> >> the API's and not the Swift product versions? If so how are the
> >> product versions decided? And given a product version, how does one
> >> know about what API version is in use? The /info endpoint does not
> >> mention the API version.
> >>
> >> I had asked about versioning sometime ago [1] and I thought the
> >> release version and API version are identical.
> >>
> >> -Shri
> >>
> >> [1] http://www.gossamer-threads.com/lists/openstack/dev/36042
> >>
> >>
> >> On Tue, Jul 22, 2014 at 5:37 AM, Anne Gentle  wrote:
> >>> Hi Shri,
> >>> The Object Storage API change is just to add a header called
> >>> 'X-Storage-Policy: ' where name is set in the swift configuration
> >>> files by the cloud provider.
> >>>
> >>> See http://docs.openstack.org/developer/swift/overview_policies.html
> >>>
> >>> We'll add it to the API docs next, no version change needed for the API
> >>> itself.
> >>> Anne
> >>>
> >>>
> >>> On Tue, Jul 22, 2014 at 12:46 AM, Shrinand Javadekar
> >>>  wrote:
> 
>  Hi,
> 
>  Swift has been following the semantic versioning scheme. The fact that
>  the product version changed from v1.x to v2.0 should suggest that the
>  Swift APIs changed in this release.
> 
>  I see that storage policies has been the biggest change in this
>  release. Has that impacted the APIs?
> 
>  Is there a document that lists exactly the API difference between v1.x 
>  and
>  v2.0?
> 
>  Thanks in advance.
>  -Shri
> 
>  ___
>  Mailing list:
>  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>  Post to : openstack@lists.openstack.org
>  Unsubscribe :
>  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> >>>
> >>>
> >>
> >> ___
> >> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> >> Post to : openstack@lists.openstack.org
> >> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> >
> 
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> 



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [Swift] API changes in v2 vs v1

[John Dickinson]

Although there is an integrated release of OpenStack twice a year, that release 
is the aggregate of different projects and different code repositories that are 
managed by different subsets of the overall developer community. And there are 
new projects added all the time. I don't think it's a technical problem to 
figure out how to do a unified API--I think it's a social problem.

But that being said, being able to isolate different parts of the overall 
cumulative "OpenStack API" and version them independently does offer some 
advantages. Swift is and has always been at v1. This means that you know that 
if you have Swift clients written years ago, they still work. Keystone has a v2 
in wide deployment and is working on a v3. This has allowed them to iterate and 
improve independently of the other thousands of developers in the ecosystem. 
Likewise for all the rest of the projects in OpenStack, and this is very 
important, I think, to the overall ability for us, as an open source community, 
to make forward progress. In general, no one project can hold back or is held 
back by another group in the community. 

Some good news is that there is an openstack-sdk effort underway. My 
understanding is that it will provide a unified CLI and SDK for all openstack 
projects. Perhaps this will give your customers the comfort they need for a 
"single API" perspective.

--John




On Jul 23, 2014, at 8:13 PM, Adam Lawson  wrote:

> Well, please don't mistake my question as a personal resistance to how things 
> are currently done. I am however curious if standardizing things across the 
> ecosystem is possible so the perceived message is consistent. Versioning of 
> the programs themselves is different since folks seem to be picking up that 
> different programs are at different stages of development. But API versions 
> are causing a lot of discussion around "why are we only using v2 in Swift but 
> v3 in Keystone? For instance.
> 
> The versioning doesn't bother ME one bit personally. But clients talking of 
> adopting Openstack are seeing, from my perspective, different API versions as 
> though certain API's are inferior to others. One client recently asked "why 
> can't we use v3 with all of the API's?" I explained and they had an "aha" 
> moment, but that dialog just happens over and over again.
> 
> 
> Adam Lawson
> AQORN, Inc.
> 427 North Tatnall Street
> Ste. 58461
> Wilmington, Delaware 19801-2230
> Toll-free: (844) 4-AQORN-NOW ext. 101
> International: +1 302-387-4660
> Direct: +1 916-246-2072
> 
> 
> 
> On Wed, Jul 23, 2014 at 11:52 AM, John Dickinson  wrote:
> Adam, that sounds you're talking about the API versions. So a user is 
> consuming eg Keystone v2, Nova v3, and Swift v1 (not to mention all the other 
> projects). Are you asking for a singular "OpenStack API"? Or maybe a common 
> API version number across projects (similar to how the requirements are 
> managed)?
> 
> 
> On Jul 23, 2014, at 11:40 AM, Adam Lawson  wrote:
> 
> > This makes perfect sense to me John. But consistency across the project 
> > would be a nice-to-have for those who design and communicate the platform 
> > to decision makers who don't understand why they're signing off v1 of this, 
> > v2 of that, v1 of this other one... It portrays a sense of program 
> > incongruity (perceived or otherwise) which adds to existing concerns about 
> > platform complexity. I spend more time explaining why than otherwise.
> >
> > Logical versioning vs "Nice to have". ; )
> >
> >
> > Adam Lawson
> > AQORN, Inc.
> > 427 North Tatnall Street
> > Ste. 58461
> > Wilmington, Delaware 19801-2230
> > Toll-free: (844) 4-AQORN-NOW ext. 702
> > Int'l: +1-302-268-6914 ext. 702
> > Cell: +1-916-990-1226
> >
> >
> >
> > On Tue, Jul 22, 2014 at 10:46 AM, Shrinand Javadekar 
> >  wrote:
> > > Hope this helps clear things up.
> >
> > This does. Thanks for the detailed explanation.
> >
> > -Shri
> >
> > >
> > > --John
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Jul 22, 2014, at 10:05 AM, Shrinand Javadekar 
> > >  wrote:
> > >
> > >> This is confusing. So does this mean semantic versioning applies to
> > >> the API's and not the Swift product versions? If so how are the
> > >> product versions decided? And given a product version, how does one
> > >> know about what API version is in use? The /info endpoint does not
> > >> mention the API version.
> > >>
> > >> I h

Re: [Openstack] [Swift] Geo replication vs Storage policies

[John Dickinson]

On Jul 30, 2014, at 10:57 AM, Samuel Merritt  wrote:

> On 7/30/14, 10:18 AM, Shrinand Javadekar wrote:
>> Hi,
>> 
>> Swift v1 allowed for geo replication using read and write affinity
>> rules. Now, Swift v2 allows setting storage policies (which can affect
>> replication) per container. I wanted to know if/how these two
>> intersect. Some of the following are straight-forward questions, just
>> wanted to get a confirmation.
>> 
>> 1. If I have Swift v2.0 installation, can I enable geo replication
>> without having a separate storage policy?
> 
> Yes.
> 
>> 2. How does a user enable both geo replication and have storage policies?
> 
> Enable geo-replication with the read_affinity and write_affinity settings as 
> usual, then define some storage policies.
> 
>> 3. Are there plans of consolidating geo replication and storage policies?
> 
> No; they're orthogonal.

But geo-clusters and storage policies go very well together, since polices 
allows you to have some data geographically distributed and other data in 
specific different locations.

--John






signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [Swift] Object invalidation after overwrite

[John Dickinson]

You are describing one of the ways that Swift does eventual consistency. In the 
scenario you describe, it is indeed possible to get the older version of the 
object on a read. There is no whole-cluster invalidation of an object. Swift's 
behavior here gives you high availability of your data even when you have 
failures in your cluster. Your proposed scenario can happen if a server fails 
after the first write and is restored after the second write, but before the 
read.

However, there are a few things you can do on the client side. One option is to 
keep track of the etag of the data you've sent. That way you can verify that 
you're getting back what you expected to get back. Another option would be to 
use a read request with the If-[Un]Modified-Since header (see 
http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html).

--John





On Aug 8, 2014, at 1:18 PM, Shrinand Javadekar  wrote:

> Hi,
> 
> I have a question regarding the way object overwrites work in the
> absence of versioning. I couldn't find this info in the documentation.
> 
> Consider the case when I have an object O already present in the Swift
> cluster. There are N replicas of this object. When a new PUT request
> that overwrites object O is returned, Swift guarantees that the new
> object, say O' got written successfully to N/2 + 1 object servers. But
> there could be some replicas that still have the older object O.
> 
> Does Swift have a way of invalidating the older object O on all object 
> servers?
> 
> If there is a GET request for object O, immediately after the
> overwrite, does Swift guarantee that the older object O cannot be
> returned?
> 
> Thanks in advance.
> -Shri
> 
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] OpenStack sponsorship for OPW - Dec-Mar 2015 internship

[John Dickinson]

I honestly have no idea how this works (the actual logistics), but speaking 
from the Swift perspective within OpenStack, I'd love to help out in any way I 
can.


--John





On Aug 8, 2014, at 7:41 PM, Mahati C  wrote:

> Hi,
> 
> I'm interested in participating in OPW (Outreach Program for Women) 2014 for 
> Dec 2014 -Mar 2015 internship. Would OpenStack be a  participating 
> organization this time around too? (I see that it was for May-Aug: 
> https://wiki.gnome.org/OutreachProgramForWomen/2014/MayAugust#Participating_Organizations).
> 
> I'm currently trying to understand Swift program of OpenStack and interested 
> in exploring that. I wanted to know how relevant is Swift for this round of 
> OPW. Is Swift looking to be a part of it? 
> 
> How relevant would it be for OPW if i continue to explore and learn Swift?
> 
> Thank you in advance.
> 
> -Mahati
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] SWIFT - max file/object limits and tombstone question

[John Dickinson]

inline

On Aug 11, 2014, at 10:38 AM, Brent Troge  wrote:

> 
> By default the maximum object size is 5G. Outside of increased replication 
> times, would there be any impacts if I increase that value to 10G? The reason 
> is I have to store upto 10G media files.  Using the large file manifest just 
> isnt going to work given Smooth Streaming or Adboe HDS delivery.
> 

I'd like to understand more about why a large object manifest (especially the 
static large objects) won't work.

But, to answer your question, increasing the max object size can affect the 
balance of drive fullness. Your drive fullness is directly related to the ratio 
of the max object size to the size of the drives. Swift places approximately 
equal numbers of objects on each drive (normalized to the drive weight). This 
works fine with with large numbers of objects because the hashing splays the 
object across all drives evenly. So, if you increase the max object size, 
everything will still work. But you'll need to keep a more careful eye on your 
capacity planning.



> For Apple HLS delivery the media files are stored with a .ts extension. Would 
> that cause any conflicts with tombstone files? Would Swfit mistakenly  mark 
> these media files as candiates for deletion? What would happen to a HLS .ts 
> file once it needs to be marked as a tombstone file?

The logical object name (eg 
"AUTH_foo/bar_container/some/object/name/here/that/is/long.ts") isn't part of 
the name used on the local media storage (ie the file on the hard drive). There 
is zero issue with using a ".ts" suffix for your object names.


--John





signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] OpenStack sponsorship for OPW - Dec-Mar 2015 internship

[John Dickinson]

Mahati,

I chatted with Anita (anteaya) in IRC this afternoon about the OPW. I'd be 
happy to continue working with you and either be a mentor for Swift in 
OpenStack or help find one.

--John






On Aug 8, 2014, at 8:13 PM, John Dickinson  wrote:

> I honestly have no idea how this works (the actual logistics), but speaking 
> from the Swift perspective within OpenStack, I'd love to help out in any way 
> I can.
> 
> 
> --John
> 
> 
> 
> 
> 
> On Aug 8, 2014, at 7:41 PM, Mahati C  wrote:
> 
>> Hi,
>> 
>> I'm interested in participating in OPW (Outreach Program for Women) 2014 for 
>> Dec 2014 -Mar 2015 internship. Would OpenStack be a  participating 
>> organization this time around too? (I see that it was for May-Aug: 
>> https://wiki.gnome.org/OutreachProgramForWomen/2014/MayAugust#Participating_Organizations).
>> 
>> I'm currently trying to understand Swift program of OpenStack and interested 
>> in exploring that. I wanted to know how relevant is Swift for this round of 
>> OPW. Is Swift looking to be a part of it? 
>> 
>> How relevant would it be for OPW if i continue to explore and learn Swift?
>> 
>> Thank you in advance.
>> 
>> -Mahati
>> ___
>> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>> Post to : openstack@lists.openstack.org
>> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> 
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] Swift And Erasure Code Storage

[John Dickinson]

1) Swift doesn't yet currently support erasure codes. It's something we're 
actively working on now.

2) It depends. Swift will allow you to choose your EC algorithm and the 
parameters for it, so that will be what determines how much raw storage space 
is needed for a given amount of usable storage.

--John





On Aug 13, 2014, at 2:41 PM, Brent Troge  wrote:

> 
> With a 100% 'Erasure Code' policy how much extra storage is needed to satisfy 
> a 1PB usable cluster ?
> 
>  
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [Swift] IP address of Swift nodes : need help

[John Dickinson]

If the IP for a storage node changes, you'll need to update the rings where 
that server's drives are. You can update the IP with the `swift-ring-builder 
set_info ...` command and then use "write_ring" to serialize it. Doing this 
will not cause any data movement in the cluster. Removing the server and 
re-adding it to the ring will cause data movement.

So, no, it's not strictly necessary to use static IPs. You'll be saving 
yourself some management overhead if you do, though.

--John




On Aug 18, 2014, at 11:42 AM, Jyoti Ranjan  wrote:

> Hi,
> 
> We are deploying Swift clusters where we do have IP addresses assigned to 
> Swift nodes via DHCP. Curious to know, what will happen if IP address of a 
> give node change that node is made part of Swift cluster. For e.g. let us 
> assume that Swift object node got IP 192.168.10.2 and later it changes to 
> 192.168.10.9 because of DHCP. Will running Swift cluster get affected?
> 
> In other words, is it necessary to use static IP for Swift nodes?
> 
> 
> Regards,
> Jyoti Ranjan
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [Swift] IP address of Swift nodes : need help

[John Dickinson]

Yes, that's true. If Swift can't find the primary location for data on a write, 
it will put the data on a handoff node. Later, when the primary is available 
(or in this case, the IP is updated in the ring), the data will be moved back. 
To be explicit, this is new data put during the window between the change in IP 
and the ring update.

This window is another reason dynamic IPs are much harder to use in a Swift 
cluster.

--John





On Aug 18, 2014, at 12:10 PM, Luse, Paul E  wrote:

> Don't you also have the issue of having those nodes unavailable during the 
> time when their IP changes and someone gets around to updating and 
> re-propagating those rings right?  Depending on the size of that window and 
> how much traffic is going on at the time you may very well end up with a lot 
> of extra data movement as incoming data makes it to handoffs (assuming their  
> addresses didn't change) and then back again after the rings reflect the new 
> addresses of the primary(ies).  
> 
> -Paul
> 
> -Original Message-
> From: John Dickinson [mailto:m...@not.mn] 
> Sent: Monday, August 18, 2014 11:54 AM
> To: Jyoti Ranjan
> Cc: openstack@lists.openstack.org
> Subject: Re: [Openstack] [Swift] IP address of Swift nodes : need help
> 
> If the IP for a storage node changes, you'll need to update the rings where 
> that server's drives are. You can update the IP with the `swift-ring-builder 
> set_info ...` command and then use "write_ring" to serialize it. Doing this 
> will not cause any data movement in the cluster. Removing the server and 
> re-adding it to the ring will cause data movement.
> 
> So, no, it's not strictly necessary to use static IPs. You'll be saving 
> yourself some management overhead if you do, though.
> 
> --John
> 
> 
> 
> 
> On Aug 18, 2014, at 11:42 AM, Jyoti Ranjan  wrote:
> 
>> Hi,
>> 
>> We are deploying Swift clusters where we do have IP addresses assigned to 
>> Swift nodes via DHCP. Curious to know, what will happen if IP address of a 
>> give node change that node is made part of Swift cluster. For e.g. let us 
>> assume that Swift object node got IP 192.168.10.2 and later it changes to 
>> 192.168.10.9 because of DHCP. Will running Swift cluster get affected?
>> 
>> In other words, is it necessary to use static IP for Swift nodes?
>> 
>> 
>> Regards,
>> Jyoti Ranjan
>> ___
>> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>> Post to : openstack@lists.openstack.org
>> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> 



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] SWIFT AND RING QUESTION

[John Dickinson]

https://swiftstack.com/blog/2012/11/21/how-the-ring-works-in-openstack-swift/ 
is soemthing that should be able to give you a pretty complete overview of how 
the ring works in Swift and how data placement works.

Let me know if you have more questions after you watch that video.

--John






On Aug 19, 2014, at 5:34 PM, Brent Troge  wrote:

> 
> Excuse this question and for lack of basic understanding. I dropped from 
> school at 8th grade, so everything is basically self taught. Here goes.
> 
> I am trying to figure out where each offset/partition is placed on the ring.
> 
> 
> So If I have 50 drives with a weight of 100 each I come up with the below 
> part power
> 
> part power = log2(50 * 100) = 13
> 
> Using that I then come up with the amount of partitions.
> 
> partitions = 2^13 =  8192
> 
> Now here is where my ignorance comes into play. How do I use these datapoints 
> to determine where each offset is on the ring?
> 
> I then guess that for each offset they will have a fixed range of values that 
> map to that partition.
> 
> So for example, for offset 1, all object URL md5 hashes that have a decimal 
> value of 0 through 100 will go here(i just made up the range 0 through 100, i 
> have no idea what the range would be with respect to my given part-power, 
> drive, etc).   
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] SWIFT AND RING QUESTION

[John Dickinson]

M, Brent Troge  wrote:

> Yeah I have watched that multiple times over the weekend, and has helped very 
> much.
> 
> So with respect to my example numbers, I am guessing that each partition will 
> land on every '41538374868278621028243970633760768' of the md5 space.
> 
> 2^(128 - 13)
> 
> or 
> 
> 2^(128)/8192
> 
> Thanks! 
> 
> 
> 
> On Tue, Aug 19, 2014 at 8:00 PM, John Dickinson  wrote:
> https://swiftstack.com/blog/2012/11/21/how-the-ring-works-in-openstack-swift/ 
> is soemthing that should be able to give you a pretty complete overview of 
> how the ring works in Swift and how data placement works.
> 
> Let me know if you have more questions after you watch that video.
> 
> --John
> 
> 
> 
> 
> 
> 
> On Aug 19, 2014, at 5:34 PM, Brent Troge  wrote:
> 
> >
> > Excuse this question and for lack of basic understanding. I dropped from 
> > school at 8th grade, so everything is basically self taught. Here goes.
> >
> > I am trying to figure out where each offset/partition is placed on the ring.
> >
> >
> > So If I have 50 drives with a weight of 100 each I come up with the below 
> > part power
> >
> > part power = log2(50 * 100) = 13
> >
> > Using that I then come up with the amount of partitions.
> >
> > partitions = 2^13 =  8192
> >
> > Now here is where my ignorance comes into play. How do I use these 
> > datapoints to determine where each offset is on the ring?
> >
> > I then guess that for each offset they will have a fixed range of values 
> > that map to that partition.
> >
> > So for example, for offset 1, all object URL md5 hashes that have a decimal 
> > value of 0 through 100 will go here(i just made up the range 0 through 100, 
> > i have no idea what the range would be with respect to my given part-power, 
> > drive, etc).
> > ___
> > Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> > Post to : openstack@lists.openstack.org
> > Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> 
> 



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] Swift Ring Maintenance

[John Dickinson]

You've actually identified the issues involved. Here's a writeup on how you can 
do it, and the general best-practice for capacity management in Swift:

https://swiftstack.com/blog/2012/04/09/swift-capacity-management/


--John




On Aug 22, 2014, at 11:50 AM, Lillie Ross-CDSR11 
 wrote:

> All,
> 
> I want to reconfigure a number of disks in my Swift storage cluster to reside 
> in different zones, and I’m unsure of the best way to accomplish this.
> 
> One way would be to set the drive weights to 0 and wait for data to migrate 
> off the drives, then remove the drive from their current zone and re-add the 
> drive to the new zone, rebalance and push the new ring files out to the 
> cluster.
> 
> Or I could simply remove the drives, re-add the drives to their new zones, 
> rebalance and push out the updated ring files.
> 
> Is one approach better than the other, or is there a better way than I’ve 
> outlined above? Since any approach would be performed over a weekend, I’m not 
> concerned about the effects of cluster performance as partitions are shuffled 
> around.
> 
> Thoughts and inputs are welcome.
> 
> Thanks,
> Ross
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] Swift questions

[John Dickinson]

Just to clarify the first answer. Everything Paul said is correct: with 3 
replicas, Swift requires that 2 (a quorum) are successful before a success can 
be returned to the client. But, assuming there isn't any current issue in the 
cluster (ie it's healthy), then all replicas will be written before success is 
returned to the client. Swift doesn't spool the object for later storage. The 
data is written to disk (for every replica) as it is streamed to the cluster. 
The quorum check simply allows for better availability if a drive fails during 
the write. In most cases, a 2xx response to a PUT means that the data is fully 
replicated and durably flushed to disk for every replica.

Background replication is only done to handle failures (eg a drive failing). 
When a client gets a success response (2xx code) from Swift from a write 
request, the client can assume that Swift has durably stored the data and is 
responsible for keeping it. If the client gets a 5xx response, the client 
should retry the request because Swift is not offering any guarantees about the 
durability of the data stored.

--John




On Aug 27, 2014, at 7:43 AM, Luse, Paul E  wrote:

> Marcus-
> 
> Not sure how much nitty gritty detail you care to know as some of these 
> answers will get into code specifics which you're better off exploring on 
> your own so my explanation isn't potentially dated.  At a high level though, 
> the proxy looks up the nodes that are responsible for the storing of an 
> object and its container via the rings.  It passes that info to the storage 
> nodes when it does the PUT request so when the storage node goes to update 
> the container it's been told "and here are the nodes to send the container 
> update to".  It will send the updates to all of them.  Similarly, once the 
> container server has updated its database it goes and updates the appropriate 
> account databases.
> 
> Make sense?
> 
> Thx
> Paul
> 
> -Original Message-
> From: Marcus White [mailto:roastedseawee...@gmail.com] 
> Sent: Wednesday, August 27, 2014 7:04 AM
> To: Luse, Paul E
> Cc: openstack
> Subject: Re: [Openstack] Swift questions
> 
> Thanks Paul:)
> 
> For the container part, you mentioned that node(meaning object
> server?) contacts the container server. Since you can have multiple container 
> servers, how does the object server know which container server to contact? 
> How and where the container gets updated is a bit confusing. With container 
> rings and account rings being separate and in the proxy part, I am not sure I 
> understand how that path works.
> 
> MW
> 
> On Wed, Aug 27, 2014 at 6:15 PM, Luse, Paul E  wrote:
>> Hi Marcus,
>> 
>> See answers below.  Feel free to ask follow-ups, others may have more to add 
>> as well.
>> 
>> Thx
>> Paul
>> 
>> -Original Message-
>> From: Marcus White [mailto:roastedseawee...@gmail.com]
>> Sent: Wednesday, August 27, 2014 5:04 AM
>> To: openstack
>> Subject: [Openstack] Swift questions
>> 
>> Hello,
>> Some questions on new and old features of Swift. Any help would be
>> great:) Some are very basic, sorry!
>> 
>> 1. Does Swift write two copies and then return back to the client in the 3 
>> replica case, with third in the background?
>> 
>> PL>  Depends on the number of replicas, the formula for what we call a 
>> quorum is n/2 + 1 which is the number of success responses we get from the 
>> back end storage nodes before telling the client that all is good.  So, yes, 
>> with 3 replicas you need 2 good responses before returning OK.
>> 
>> 2. This again is a stupid question, but eventually consistent for an object 
>> is a bit confusing, unless it is updated. If it is created, it is either 
>> there or not and you cannot update the data within the object. Maybe a POST 
>> can change the metadata? Or the container listing shows its there but the 
>> actual object never got there? Those are the only cases I can think of.
>> 
>> PL> No, it's a good question because its asked a lot.  The most common 
>> scenario that we talk about for eventually consistent is the consistency 
>> between the existence of an object and its presence in the container listing 
>> so your thinking is pretty close.  When an object PUT is complete on a 
>> storage node (fully committed to disk), that node will then send a message 
>> to the appropriate container server to update the listing.  It will attempt 
>> to do this synchronously but if it can't, the update may be delayed w/o any 
>> indication to the client.  This is by design and means that it's possible to 
>> get a successful PUT, be able to GET the object w/o any problem however it 
>> may not yet show up in the container listing.  There are other scenarios 
>> that demonstrate the eventually consistent nature of Swift, this is just a 
>> common and easy to explain one.
>> 
>> 3. Once an object has been written, when and how is the container 
>> listing, number of bytes, account listing (if new container created) 
>> etc updated? I

Re: [Openstack] Swift questions

[John Dickinson]

Yup, this is a totally possible failure scenario, and Swift will merge the data 
(using last-write-wins for overwrites) automatically when the partition is 
restored. But you'll still have full durability on writes, even with a 
partitioned global cluster.

--John



On Aug 27, 2014, at 10:49 AM, Marcus White  wrote:

> Yup, thanks for the great explanation:)
> 
> Another question, though related: If there are three regions, and two
> get "split", there is now a partition. Both the split ones can talk to
> the third, but not to each other.
> 
> A PUT comes into one region, and it gets written to the local site.
> Container information presumably gets updated here, including byte
> count.
> 
> Same thing happens on another site, where a PUT comes in and the
> container information is updated with the byte count.
> When the sites get back together, do the container servers make sure
> its all correct in the end?
> 
> If this is not a possible scenario, is there any case where the
> container metadata can be different between two zones or regions
> because of a partition, and independent PUTs can happen, and the data
> has to be merged? Is that all done by the respective servers(container
> or account?)
> 
> I will be looking at the sources soon.
> 
> Thanks again
> MW.
> 
> 
> 
> On Wed, Aug 27, 2014 at 8:13 PM, Luse, Paul E  wrote:
>> Marcus-
>> 
>> Not sure how much nitty gritty detail you care to know as some of these 
>> answers will get into code specifics which you're better off exploring on 
>> your own so my explanation isn't potentially dated.  At a high level though, 
>> the proxy looks up the nodes that are responsible for the storing of an 
>> object and its container via the rings.  It passes that info to the storage 
>> nodes when it does the PUT request so when the storage node goes to update 
>> the container it's been told "and here are the nodes to send the container 
>> update to".  It will send the updates to all of them.  Similarly, once the 
>> container server has updated its database it goes and updates the 
>> appropriate account databases.
>> 
>> Make sense?
>> 
>> Thx
>> Paul
>> 
>> -Original Message-
>> From: Marcus White [mailto:roastedseawee...@gmail.com]
>> Sent: Wednesday, August 27, 2014 7:04 AM
>> To: Luse, Paul E
>> Cc: openstack
>> Subject: Re: [Openstack] Swift questions
>> 
>> Thanks Paul:)
>> 
>> For the container part, you mentioned that node(meaning object
>> server?) contacts the container server. Since you can have multiple 
>> container servers, how does the object server know which container server to 
>> contact? How and where the container gets updated is a bit confusing. With 
>> container rings and account rings being separate and in the proxy part, I am 
>> not sure I understand how that path works.
>> 
>> MW
>> 
>> On Wed, Aug 27, 2014 at 6:15 PM, Luse, Paul E  wrote:
>>> Hi Marcus,
>>> 
>>> See answers below.  Feel free to ask follow-ups, others may have more to 
>>> add as well.
>>> 
>>> Thx
>>> Paul
>>> 
>>> -Original Message-
>>> From: Marcus White [mailto:roastedseawee...@gmail.com]
>>> Sent: Wednesday, August 27, 2014 5:04 AM
>>> To: openstack
>>> Subject: [Openstack] Swift questions
>>> 
>>> Hello,
>>> Some questions on new and old features of Swift. Any help would be
>>> great:) Some are very basic, sorry!
>>> 
>>> 1. Does Swift write two copies and then return back to the client in the 3 
>>> replica case, with third in the background?
>>> 
>>> PL>  Depends on the number of replicas, the formula for what we call a 
>>> quorum is n/2 + 1 which is the number of success responses we get from the 
>>> back end storage nodes before telling the client that all is good.  So, 
>>> yes, with 3 replicas you need 2 good responses before returning OK.
>>> 
>>> 2. This again is a stupid question, but eventually consistent for an object 
>>> is a bit confusing, unless it is updated. If it is created, it is either 
>>> there or not and you cannot update the data within the object. Maybe a POST 
>>> can change the metadata? Or the container listing shows its there but the 
>>> actual object never got there? Those are the only cases I can think of.
>>> 
>>> PL> No, it's a good question because its asked a lot.  The most common 
>>> scenario that we talk about for eventually consistent is the consistency 
>>> between the existence of an object and its presence in the container 
>>> listing so your thinking is pretty close.  When an object PUT is complete 
>>> on a storage node (fully committed to disk), that node will then send a 
>>> message to the appropriate container server to update the listing.  It will 
>>> attempt to do this synchronously but if it can't, the update may be delayed 
>>> w/o any indication to the client.  This is by design and means that it's 
>>> possible to get a successful PUT, be able to GET the object w/o any problem 
>>> however it may not yet show up in the container listing.  There are other 
>>> scenarios that demonstra

Re: [Openstack] Swift questions

[John Dickinson]

On Aug 29, 2014, at 2:43 AM, Marcus White  wrote:

> Thanks John:)
> 
> Some more additional basic questions..
> 
> 1. For the multi site case, does Swift present a single end point, or
> is it two separate regions with two separate IP addresses, and the
> client can talk to either to get the data? I think it is the latter?

Every proxy server in the cluster will be able to handle requests. There isn't 
a single "control node" that brokers everything. Often times, all of the proxy 
servers are put behind a load balancer or geo-dns or something to present a 
single domain for the cluster.


> 
> 2. Are keystone databases synced across sites? I didnt know of an
> option in keystone to do that, hence asking..If we need to
> authenticate against keystone on different sites to access the same
> object, it has to be that way?

That is also my understanding, but I'm not an expert on Keystone options or 
deployment patterns.

> 
> TIA,
> 
> MW
> 
> 
> On Wed, Aug 27, 2014 at 11:37 PM, John Dickinson  wrote:
>> Yup, this is a totally possible failure scenario, and Swift will merge the 
>> data (using last-write-wins for overwrites) automatically when the partition 
>> is restored. But you'll still have full durability on writes, even with a 
>> partitioned global cluster.
>> 
>> --John
>> 
>> 
>> 
>> On Aug 27, 2014, at 10:49 AM, Marcus White  
>> wrote:
>> 
>>> Yup, thanks for the great explanation:)
>>> 
>>> Another question, though related: If there are three regions, and two
>>> get "split", there is now a partition. Both the split ones can talk to
>>> the third, but not to each other.
>>> 
>>> A PUT comes into one region, and it gets written to the local site.
>>> Container information presumably gets updated here, including byte
>>> count.
>>> 
>>> Same thing happens on another site, where a PUT comes in and the
>>> container information is updated with the byte count.
>>> When the sites get back together, do the container servers make sure
>>> its all correct in the end?
>>> 
>>> If this is not a possible scenario, is there any case where the
>>> container metadata can be different between two zones or regions
>>> because of a partition, and independent PUTs can happen, and the data
>>> has to be merged? Is that all done by the respective servers(container
>>> or account?)
>>> 
>>> I will be looking at the sources soon.
>>> 
>>> Thanks again
>>> MW.
>>> 
>>> 
>>> 
>>> On Wed, Aug 27, 2014 at 8:13 PM, Luse, Paul E  wrote:
>>>> Marcus-
>>>> 
>>>> Not sure how much nitty gritty detail you care to know as some of these 
>>>> answers will get into code specifics which you're better off exploring on 
>>>> your own so my explanation isn't potentially dated.  At a high level 
>>>> though, the proxy looks up the nodes that are responsible for the storing 
>>>> of an object and its container via the rings.  It passes that info to the 
>>>> storage nodes when it does the PUT request so when the storage node goes 
>>>> to update the container it's been told "and here are the nodes to send the 
>>>> container update to".  It will send the updates to all of them.  
>>>> Similarly, once the container server has updated its database it goes and 
>>>> updates the appropriate account databases.
>>>> 
>>>> Make sense?
>>>> 
>>>> Thx
>>>> Paul
>>>> 
>>>> -Original Message-
>>>> From: Marcus White [mailto:roastedseawee...@gmail.com]
>>>> Sent: Wednesday, August 27, 2014 7:04 AM
>>>> To: Luse, Paul E
>>>> Cc: openstack
>>>> Subject: Re: [Openstack] Swift questions
>>>> 
>>>> Thanks Paul:)
>>>> 
>>>> For the container part, you mentioned that node(meaning object
>>>> server?) contacts the container server. Since you can have multiple 
>>>> container servers, how does the object server know which container server 
>>>> to contact? How and where the container gets updated is a bit confusing. 
>>>> With container rings and account rings being separate and in the proxy 
>>>> part, I am not sure I understand how that path works.
>>>> 
>>>> MW
>>>> 
>>>> On Wed, Aug 27, 2014 at 6:15 PM, Luse, Paul E  
>>>> wrote:
>>>>&g

Re: [Openstack] Swift questions

[John Dickinson]

On Aug 31, 2014, at 5:25 AM, Marcus White  wrote:

> One last follow up question, hopefully!
> 
> I agree, each proxy server can handle any requests..I wanted to see
> what the typical deployment is for the multi geo case.
> 
> For multi region global clusters, for example, each region would have
> a separate IP address? Behind each regions IP address you can have
> multiple proxy servers in each site, and a customer can connect to any
> site?

I've seen both. Really it depends on the goals of the global cluster and the 
existing infrastructure available. If the goal is a DR story, then often DNS is 
pointed at one region, and when the "R" of DR is needed, the DNS entry is 
updated to point to the other region. Other times, if the goal is to allow for 
lower-latency reads and writes, then then all regions are used for access all 
the time. The client either relies on some sort of geo-dns/anycast system to 
get to the closest endpoint, or the client chooses the preferred region from a 
list of available ones.

Yes, I'd normally expect each region to have a separate public IP address with 
multiple proxies load balanced behind it.

> 
> In a single site case, keystone would give you one end point, which is
> the external load balancer Ip. For the multi site case, what happens
> with keystone? Does it provide multiple end points?
> 
> Another question is, does the keystone server itself have a different
> external IP for each site?

My understanding is that Keystone doesn't really work across wide geographic 
areas (I'd be happy to be wrong here--maybe a Keystone dev can jump in). I 
think the limitation is around the backing store Keystone uses to manage tokens 
and the service catalog.

There are a few things to consider when using a central auth system with a 
globally distributed Swift cluster. First, note that if the auth token 
(identity info) isn't cached in the local region, Swift will need to ask the 
Keystone service for the information. Swift will cache it if possible, but know 
that the auth lookup will add latency (noticeable latency if the auth system is 
across an ocean). Second, make sure memcache and Swift's cache middleware are 
configured properly. For example, you don't want a global memcache pool to 
result in 50% of the internal "fetch from cache" requests to go across the WAN. 
You'd instead want a separate memcache pool for each region.

As a side note, a centralized auth system actually makes the caching strategies 
easier, since there's less complexity to deal with when trying to get and cache 
and validate auth credentials and tokens.


> 
> Just confused a bit on the deployment basics.

Good luck! and feel free to ask more as needed.

> 
> MW.
> 
> 
> On Fri, Aug 29, 2014 at 7:51 PM, John Dickinson  wrote:
>> 
>> On Aug 29, 2014, at 2:43 AM, Marcus White  wrote:
>> 
>>> Thanks John:)
>>> 
>>> Some more additional basic questions..
>>> 
>>> 1. For the multi site case, does Swift present a single end point, or
>>> is it two separate regions with two separate IP addresses, and the
>>> client can talk to either to get the data? I think it is the latter?
>> 
>> Every proxy server in the cluster will be able to handle requests. There 
>> isn't a single "control node" that brokers everything. Often times, all of 
>> the proxy servers are put behind a load balancer or geo-dns or something to 
>> present a single domain for the cluster.
>> 
>> 
>>> 
>>> 2. Are keystone databases synced across sites? I didnt know of an
>>> option in keystone to do that, hence asking..If we need to
>>> authenticate against keystone on different sites to access the same
>>> object, it has to be that way?
>> 
>> That is also my understanding, but I'm not an expert on Keystone options or 
>> deployment patterns.
>> 
>>> 
>>> TIA,
>>> 
>>> MW
>>> 
>>> 
>>> On Wed, Aug 27, 2014 at 11:37 PM, John Dickinson  wrote:
>>>> Yup, this is a totally possible failure scenario, and Swift will merge the 
>>>> data (using last-write-wins for overwrites) automatically when the 
>>>> partition is restored. But you'll still have full durability on writes, 
>>>> even with a partitioned global cluster.
>>>> 
>>>> --John
>>>> 
>>>> 
>>>> 
>>>> On Aug 27, 2014, at 10:49 AM, Marcus White  
>>>> wrote:
>>>> 
>>>>> Yup, thanks for the great explanation:)
>>>>> 
>>>>> Another question, though related: If there are three regions, and two
>

Re: [Openstack] SWIFT - Write Quorum

[John Dickinson]

I'm not sure what the question is.

If you are looking to have a successful response after it's written twice in a 
cluster with 4 replicas, no. Swift's quorum calculation is (replicas DIV 2 + 
1). This means that for 4 replicas, you have a quorum size of 3. What I would 
suggest you look in to is the write_affinity setting so that you can do a 
full-quorum (at least) write to the local region and then asynchronously 
replicate to the other region. See 
http://docs.openstack.org/developer/swift/admin_guide.html#geographically-distributed-clusters
 and 
https://swiftstack.com/blog/2012/09/16/globally-distributed-openstack-swift-cluster/.

If you are looking to ensure that there is at least one replica in each region, 
then yes. The quorum size of three (see above) will ensure that, without any 
write_affinity settings, you'll have at least one replica in each region and 
two in another before the client gets a 2xx success response code to the PUT 
request.

--John



On Sep 9, 2014, at 6:59 AM, Brent Troge  wrote:

> 
> 
> If I configure Swift to use 4 replicas across two regions(two replicas per 
> region), is it possible to only list a newly ingested object if it has 
> written at least twice? The goal is to only list a new object only if it has 
> a presence in each region.
> 
> west coast
> region 1 - zone 1
> region 1 - zone 2
> 
> east coast
> region 2  - zone 1( 3?)
> region 2 -  zone 2( 4?) 
> 
> Thanks!
> 
> 
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] SWIFT - Write Quorum

[John Dickinson]

Well listings is slightly different than being able to read the data if you 
already know the object name.

Assuming no failures in the system, the container listings will be up to date 
before the end-user gets a 2xx response. That is, create a new object, then 
immediately do a listing, and you'll see the object in that listing.

However, we can pretty safely assume that there will be failures at some point 
(especially if it's a larger cluster). In that case, the end-user will still 
get a 2xx response because the data was durably flushed to disk, but the 
container listing update may fall back to an asynchronous update. In that case, 
you may or may not get the object in a container listing until container 
replication has completed. But you'll still be able to read the object directly 
if you know the name.

--John


On Sep 9, 2014, at 9:15 AM, Brent Troge  wrote:

> Thanks..  My understanding is that an object will not be listed within a 
> container until it completes replication throughout the cluster. Thats what I 
> meant by 'listing'
> 
> But you corrected that mis-understanding.  Such that, with respect to my 3 
> write requirement, an object will be available  in each region once I receive 
> a 200 upon file ingest.
> 
> 
> 
> 
> 
> On Tue, Sep 9, 2014 at 10:02 AM, John Dickinson  wrote:
> I'm not sure what the question is.
> 
> If you are looking to have a successful response after it's written twice in 
> a cluster with 4 replicas, no. Swift's quorum calculation is (replicas DIV 2 
> + 1). This means that for 4 replicas, you have a quorum size of 3. What I 
> would suggest you look in to is the write_affinity setting so that you can do 
> a full-quorum (at least) write to the local region and then asynchronously 
> replicate to the other region. See 
> http://docs.openstack.org/developer/swift/admin_guide.html#geographically-distributed-clusters
>  and 
> https://swiftstack.com/blog/2012/09/16/globally-distributed-openstack-swift-cluster/.
> 
> If you are looking to ensure that there is at least one replica in each 
> region, then yes. The quorum size of three (see above) will ensure that, 
> without any write_affinity settings, you'll have at least one replica in each 
> region and two in another before the client gets a 2xx success response code 
> to the PUT request.
> 
> --John
> 
> 
> 
> On Sep 9, 2014, at 6:59 AM, Brent Troge  wrote:
> 
> >
> >
> > If I configure Swift to use 4 replicas across two regions(two replicas per 
> > region), is it possible to only list a newly ingested object if it has 
> > written at least twice? The goal is to only list a new object only if it 
> > has a presence in each region.
> >
> > west coast
> > region 1 - zone 1
> > region 1 - zone 2
> >
> > east coast
> > region 2  - zone 1( 3?)
> > region 2 -  zone 2( 4?)
> >
> > Thanks!
> >
> >
> > ___
> > Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> > Post to : openstack@lists.openstack.org
> > Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> 
> 



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] SWIFT AND ROOT DIRECTORY

[John Dickinson]

Actually, it's even easier than that. Swift provides, out of the box, a 
crossdomain middleware so that you can return the appropriate domain-wide 
policy for your content.

See 
http://docs.openstack.org/developer/swift/middleware.html#module-swift.common.middleware.crossdomain


--John




On Sep 10, 2014, at 7:34 AM, Brent Troge  wrote:

> 
> When calling swift hosted media files through flash or silverlight, the 
> player makes a call to the root directory for clientaccesspolicy.xml or 
> crossdomain.xml. 
> 
> For example, if my flash player calls:
> 
> http://192.168.1.1:8080/v1/AUTH_xxx/media/index.m3u8 
> 
> The player will then look for:
> 
> http://192.168.1.1:8080/crossdomain.xml
> 
> I assume I would have to host the proxy server via apache httpd, and make the 
> needed adjustments.
> 
> Thoughts?
> 
> Thanks!
>  
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] OpenStack Regions

[John Dickinson]

> On Nov 27, 2014, at 1:31 AM, Chris  wrote:
> 
> Hello
> 
> I have a question regarding regions. We have the initial “RegionOne” in our 
> current setup and want to and a second one “RegionTwo”.
> It seems like the shared services between different regions are Keystone and 
> Horizon. The separate services for each region includes Nova, Neutron etc. 
> Please correct me here if I’m wrong.

Not wrong, but I'd like to add another example that's slightly different. You 
can deploy one Swift cluster that spans a wide geography. You might have sets 
of proxy servers that handle the API requests in each region. You can put them 
behind one domain and use some geo-dns routing to get to the "local" Swift 
region. I know several deployers who are doing that in production today. eg 
swift.example.com routes you to the closest set of proxy servers.

One other thing you could do is also expose the individual sets of proxy 
servers in the Swift cluster explicitly. eg you could also have 
east.swift.example.com and west.swift.example.com and those would always go to 
a specific Swift region.

Mostly it's a question of what you want to expose to the user. One endpoint 
that goes to the closest? Explicit endpoints for each region? Both?

In all the examples above, it would be one Swift cluster with data durably 
stored across all regions. Not multiple clusters.

For some background, check out 
https://swiftstack.com/blog/2012/09/16/globally-distributed-openstack-swift-cluster/


--John





> 
> What about the database, does the services from RegionTwo use the initial 
> database from RegionOne or does RegionTwo also needs a new database?
> 
> Cheers
> Chris
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] Create initial rings swift

[John Dickinson]

Great questions, but there isn't a short answer. Let me do my best to give you 
a concise answer.


> On Dec 10, 2014, at 11:00 PM, dhanesh1212121212  wrote:
> 
> Hi All,
> 
> Below details is Mentioned in openstack document to configure object storage.
> 
> For simplicity, this guide uses one region and
> zone with 2^10 (1024) maximum partitions, 3 replicas of each object, and 1 
> hour minimum
> time between moving a partition more than once. For Object Storage, a 
> partition indicates
> a directory on a storage device rather than a conventional partition table.

The partition isn't really a directory. Well, it happens to have an on-disk 
layout where partitions are represented as directories, but conceptually that 
doesn't matter. A partition is one part of the overall keyspace of the hash 
function used by the ring. Specifically, a partition is a certain number of 
prefix bits of the output of md5. The number of prefix bits used is called the 
part_power--in your example, it's 10.

When a request comes to Swift, the URI is hashed, resulting in a long number. 
The prefix of that number is the partition. So, for example, if you hashed a 
URI and it started with the bits 011011000101010010111011... and you have a 
part power of 4, then the partition is 0110, or 6 in decimal. Swift keeps a 
lookup table, called "the ring", that maps partitions to storage volumes (read: 
drives). So in this case, Swift will find right drives for partition 6 in the 
lookup table and then send the data to the right servers that have those drives.

Keeping that lookup table right (so that replicas are distributed evenly and 
across failure domains) is what happens when you rebalance a ring.


(This is a lot of detail, but it's important to answer the questions below.)


> 
> 
> Please Clarify my doubts.
> 
> 1. Does it mean we can create only 1024 folders (directory) if we select 
> 2^10. is it possible to expand the partition later?

The part_power can not be changed after the ring is created! [1] Reason being, 
if the part power changes, the all the data currently in cluster would have to 
be rehashed and moved. At best, that would cause a massive internal network 
load. At worst, rehashing all the objects would simply take an enormous amount 
of time and you'd essentially be left with a large period of downtime.

[1] There have been a few conversations over the years about how to allow 
changing the part power, but it's not been solved yet.

> 
> 
> 
> 2. started swift configuration with three replicas. is it possible add more 
> replicas later?

Yes, absolutely! With the `swift-ring-builder` command, you can change the 
number of replicas. Below is the help/usage message for that command:

swift-ring-builder  set_replicas 
Changes the replica count to the given .  may
be a floating-point value, in which case some partitions will have
floor() replicas and some will have ceiling()
in the correct proportions.

A rebalance is needed to make the change take effect.



> 
> 
> 3. what is the meaning of "moving a partition more than once in one hour".

Swift is optimized for very high availability. If you were to add a lot of 
capacity to your cluster all at once, you may end up with a lot of data needing 
to move around. However, because Swift is a distributed system, you might have 
different versions of the ring deployed in the cluster at the same time. Also, 
if all replicas were assigned to a new location but the data hadn't been moved 
yet, any requests for that data would fail.

To solve this, Swift will only move one replica for a partition in a given 
rebalance operation. This means that your existing data is still available 
(even if that third replica's data hasn't moved yet). The amount of time you 
must wait between rebalances is called the min_part_hours. It should be set to 
something bigger than how long it takes for your cluster to run a complete 
replication pass (ie replication completed on all nodes).

You can adjust the min_part_hours at any time. It's a safety valve for you and 
your end-users, so use it instead of circumventing it.




I hope the answers help. There's a ton of detail about how the rings work in 
Swift. Here's some links for further reading/watching:

http://docs.openstack.org/developer/swift/overview_ring.html
https://swiftstack.com/blog/2012/11/21/how-the-ring-works-in-openstack-swift/
https://swiftstack.com/blog/2012/09/13/how-openstack-swift-handles-hardware-failures/
http://mirror.int.linux.conf.au/linux.conf.au/2013/mp4/Playing_with_OpenStack_Swift.mp4



--John






> 
> 
> 
> Regards,
> Dhanesh M.
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack



signature.asc
Description: Message signed with OpenPGP using GPGMail
___

Re: [Openstack] (Juno) Multiple rings Swift

[John Dickinson]

Sounds like you're looking for a global cluster. You don't need multiple rings 
for this. Swift can support this. When you add a new device to a ring, you add 
it in a different region, and Swift takes care of it for you.

Here's some more information:

http://docs.openstack.org/developer/swift/admin_guide.html#geographically-distributed-clusters
https://www.youtube.com/watch?v=mcaTwhP_rPE
https://www.youtube.com/watch?v=LpmBRqevuVU

https://swiftstack.com/blog/2013/07/02/swift-1-9-0-release/
https://swiftstack.com/blog/2012/09/16/globally-distributed-openstack-swift-cluster/
https://www.swiftstack.com/docs/admin/cluster_management/regions.html



--John





> On Dec 15, 2014, at 10:15 AM, Amit Anand  wrote:
> 
> Hi all,
> 
> I was wondering if anyone knew of any good blog posts and videos that could 
> show/explain what I am trying to do. I have Juno setup and it working great 
> (thanks to everyone on heres help)! Now, I would like to add two more Object 
> Store nodes, but as a separate "datacenter" as it were and replicate between 
> my existing ring and the new one. Now, Im not sure exactly what to do for the 
> account/container/object rings and how to get them to replicate (so if one 
> goes down the other will still continue to serve data). I am also going to 
> try and add another storage node just as a backup for existing data. Would 
> anyone have any good links to send me I would appreciate it!
> 
> Thanks!
> Amit Anand
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] (Juno) Multiple rings Swift

[John Dickinson]

That's normal. See the "...or none can be due to min_part_hours". Swift is 
refusing to move more data until the stuff likely currently in flight has 
settled. See https://swiftstack.com/blog/2012/04/09/swift-capacity-management/

--John






> On Dec 16, 2014, at 9:09 AM, Amit Anand  wrote:
> 
> Hi John thank you!
> 
> So I went ahead and added two more storage nodes to the existing rings 
> (object, account, container) and tried to rebalance on the controller I got 
> this:
> 
> [root@controller swift]# swift-ring-builder object.builder rebalance
> Reassigned 1024 (100.00%) partitions. Balance is now 38.80.
> ---
> NOTE: Balance of 38.80 indicates you should push this
>   ring, wait at least 1 hours, and rebalance/repush.
> ---
> 
> 
> For all three. So while waiting, I went ahead and added the *.gz files and 
> swift.conf to the new nodes and started the Object Storage Services on the 
> both the new storage nodes Now I am seeing this after I try to rebalance 
> after waiting about an hour:
> 
> [root@controller swift]# swift-ring-builder object.builder rebalance
> No partitions could be reassigned.
> Either none need to be or none can be due to min_part_hours [1].
> 
> Devices 4,5,6,7 are the new ones I added in region 2.
> 
> 
> [root@controller swift]#  swift-ring-builder object.builder
> object.builder, build version 9
> 1024 partitions, 3.00 replicas, 2 regions, 2 zones, 8 devices, 38.80 
> balance
> The minimum number of hours before a partition can be reassigned is 1
> Devices:id  region  zone  ip address  port  replication ip  
> replication port  name weight partitions balance meta
>  0   1 1   10.7.5.51  6000   10.7.5.51
>   6000  sda3 100.00501   30.47
>  1   1 1   10.7.5.51  6000   10.7.5.51
>   6000  sda4 100.00533   38.80
>  2   1 1   10.7.5.52  6000   10.7.5.52
>   6000  sda3 100.00512   33.33
>  3   1 1   10.7.5.52  6000   10.7.5.52
>   6000  sda4 100.00502   30.73
>  4   2 1   10.7.5.53  6000   10.7.5.53
>   6000  sda3 100.00256  -33.33
>  5   2 1   10.7.5.53  6000   10.7.5.53
>   6000  sda4 100.00256  -33.33
>  6   2 1   10.7.5.54  6000   10.7.5.54
>   6000  sda3 100.00256  -33.33
>  7   2 1   10.7.5.54  6000   10.7.5.54
>   6000  sda4 100.00256  -33.33
> 
> 
> 
> All three have -33.33 (container, object, account)  for their balance. Is 
> this normal or did do something incorrect? It doesnt seem to be replicating 
> the data to the new nodes (or at least it looks like it stopped?)  but I am 
> not sure. Would appreciate any insight. Thanks!
> 
> Amit
> 
> 
> 
> 
> On Mon, Dec 15, 2014 at 1:49 PM, John Dickinson  wrote:
> Sounds like you're looking for a global cluster. You don't need multiple 
> rings for this. Swift can support this. When you add a new device to a ring, 
> you add it in a different region, and Swift takes care of it for you.
> 
> Here's some more information:
> 
> http://docs.openstack.org/developer/swift/admin_guide.html#geographically-distributed-clusters
> https://www.youtube.com/watch?v=mcaTwhP_rPE
> https://www.youtube.com/watch?v=LpmBRqevuVU
> 
> https://swiftstack.com/blog/2013/07/02/swift-1-9-0-release/
> https://swiftstack.com/blog/2012/09/16/globally-distributed-openstack-swift-cluster/
> https://www.swiftstack.com/docs/admin/cluster_management/regions.html
> 
> 
> 
> --John
> 
> 
> 
> 
> 
> > On Dec 15, 2014, at 10:15 AM, Amit Anand  wrote:
> >
> > Hi all,
> >
> > I was wondering if anyone knew of any good blog posts and videos that could 
> > show/explain what I am trying to do. I have Juno setup and it working great 
> > (thanks to everyone on heres help)! Now, I would like to add two more 
> > Object Store nodes, but as a separate "datacenter" as it were and replicate 
> > between my existing ring and the new one. Now, Im not sure exactly what to 
> > do for the account/container/object rings and how to get them to replicate 
> > (so if one goes down the other will still continue to serve data). I am 
> > also going to try and add another storage node just as a backup for 
> > existing data. Would 

Re: [Openstack] (Juno) Multiple rings Swift

[John Dickinson]

Assuming your regions are pretty close to the same size, that's exactly what 
you'll get with 3 replicas across 2 regions. Some data will have 2 replicas in 
region 1 and one in region 2. Other data will have 1 in region 1 and 2 in 
region 2.

--John




> On Dec 16, 2014, at 9:39 AM, Amit Anand  wrote:
> 
> Ok cool Ill wait it out see what happens. So now I have another stupid 
> question - after all is said and done, how many copies of my data will I 
> have?! What I am aiming for is something like 2 regions and 3 replicas ie, 2 
> copies of the data in region one and one copy in region 2.
> 
> On Tue, Dec 16, 2014 at 12:35 PM, John Dickinson  wrote:
> That's normal. See the "...or none can be due to min_part_hours". Swift is 
> refusing to move more data until the stuff likely currently in flight has 
> settled. See https://swiftstack.com/blog/2012/04/09/swift-capacity-management/
> 
> --John
> 
> 
> 
> 
> 
> 
> > On Dec 16, 2014, at 9:09 AM, Amit Anand  wrote:
> >
> > Hi John thank you!
> >
> > So I went ahead and added two more storage nodes to the existing rings 
> > (object, account, container) and tried to rebalance on the controller I got 
> > this:
> >
> > [root@controller swift]# swift-ring-builder object.builder rebalance
> > Reassigned 1024 (100.00%) partitions. Balance is now 38.80.
> > ---
> > NOTE: Balance of 38.80 indicates you should push this
> >   ring, wait at least 1 hours, and rebalance/repush.
> > ---
> >
> >
> > For all three. So while waiting, I went ahead and added the *.gz files and 
> > swift.conf to the new nodes and started the Object Storage Services on the 
> > both the new storage nodes Now I am seeing this after I try to 
> > rebalance after waiting about an hour:
> >
> > [root@controller swift]# swift-ring-builder object.builder rebalance
> > No partitions could be reassigned.
> > Either none need to be or none can be due to min_part_hours [1].
> >
> > Devices 4,5,6,7 are the new ones I added in region 2.
> >
> >
> > [root@controller swift]#  swift-ring-builder object.builder
> > object.builder, build version 9
> > 1024 partitions, 3.00 replicas, 2 regions, 2 zones, 8 devices, 38.80 
> > balance
> > The minimum number of hours before a partition can be reassigned is 1
> > Devices:id  region  zone  ip address  port  replication ip  
> > replication port  name weight partitions balance meta
> >  0   1 1   10.7.5.51  6000   10.7.5.51  
> > 6000  sda3 100.00501   30.47
> >  1   1 1   10.7.5.51  6000   10.7.5.51  
> > 6000  sda4 100.00533   38.80
> >  2   1 1   10.7.5.52  6000   10.7.5.52  
> > 6000  sda3 100.00512   33.33
> >  3   1 1   10.7.5.52  6000   10.7.5.52  
> > 6000  sda4 100.00502   30.73
> >  4   2 1   10.7.5.53  6000   10.7.5.53  
> > 6000  sda3 100.00256  -33.33
> >  5   2 1   10.7.5.53  6000   10.7.5.53  
> > 6000  sda4 100.00256  -33.33
> >  6   2 1   10.7.5.54  6000   10.7.5.54  
> > 6000  sda3 100.00256  -33.33
> >  7   2 1   10.7.5.54  6000   10.7.5.54  
> > 6000  sda4 100.00    256  -33.33
> >
> >
> >
> > All three have -33.33 (container, object, account)  for their balance. Is 
> > this normal or did do something incorrect? It doesnt seem to be replicating 
> > the data to the new nodes (or at least it looks like it stopped?)  but I am 
> > not sure. Would appreciate any insight. Thanks!
> >
> > Amit
> >
> >
> >
> >
> > On Mon, Dec 15, 2014 at 1:49 PM, John Dickinson  wrote:
> > Sounds like you're looking for a global cluster. You don't need multiple 
> > rings for this. Swift can support this. When you add a new device to a 
> > ring, you add it in a different region, and Swift takes care of it for you.
> >
> > Here's some more information:
> >
> > http://docs.openstack.org/developer/swift/admin_guide.html#geographically-distributed-clusters
> > https://www.youtube.com/watch?v=mcaTwhP_rPE
> > https://www.youtube.com/watch?v=LpmBRqevuVU
> >

Re: [Openstack] (Juno) Multiple rings Swift

[John Dickinson]

> On Dec 16, 2014, at 12:06 PM, Amit Anand  wrote:
> 
> Thanks again John. Out of curiosity, is it possible to see what is where? 
> Lets say I have uploaded a video and wanted to see where the three copies 
> like out of the 4 nodes I have?

You can find this with the swift-get-nodes tool. It does the right lookup and 
shows where a thing of that name would be placed in Swift. For example on my 
SAIO dev environment:

$ swift-get-nodes /etc/swift/object.ring.gz AUTH_foo/bar/baz

Account AUTH_foo
Container   bar
Object  baz


Partition   29
Hash075ad946c11d8c21e2b97a08b5da8c48

Server:Port Device  127.0.0.1:6010 d1
Server:Port Device  127.0.0.1:6030 d3
Server:Port Device  127.0.0.1:6020 d2
Server:Port Device  127.0.0.1:6040 d4


curl -I -XHEAD "http://127.0.0.1:6010/d1/29/AUTH_foo/bar/baz";
curl -I -XHEAD "http://127.0.0.1:6030/d3/29/AUTH_foo/bar/baz";
curl -I -XHEAD "http://127.0.0.1:6020/d2/29/AUTH_foo/bar/baz";
curl -I -XHEAD "http://127.0.0.1:6040/d4/29/AUTH_foo/bar/baz";


Use your own device location of servers:
such as "export DEVICE=/srv/node"
ssh 127.0.0.1 "ls -lah 
${DEVICE:-/srv/node*}/d1/objects/29/c48/075ad946c11d8c21e2b97a08b5da8c48"
ssh 127.0.0.1 "ls -lah 
${DEVICE:-/srv/node*}/d3/objects/29/c48/075ad946c11d8c21e2b97a08b5da8c48"
ssh 127.0.0.1 "ls -lah 
${DEVICE:-/srv/node*}/d2/objects/29/c48/075ad946c11d8c21e2b97a08b5da8c48"
ssh 127.0.0.1 "ls -lah 
${DEVICE:-/srv/node*}/d4/objects/29/c48/075ad946c11d8c21e2b97a08b5da8c48"

note: `/srv/node*` is used as default value of `devices`, the real value is set 
in the config file on each storage node.




> 
> And you guys dont have a free version of Swiftstack available per chance do 
> you :-)


Yes, we do. https://swiftstack.com/customer/signup/


> 
> 
> 
> 
> On Tue, Dec 16, 2014 at 12:48 PM, John Dickinson  wrote:
> Assuming your regions are pretty close to the same size, that's exactly what 
> you'll get with 3 replicas across 2 regions. Some data will have 2 replicas 
> in region 1 and one in region 2. Other data will have 1 in region 1 and 2 in 
> region 2.
> 
> --John
> 
> 
> 
> 
> > On Dec 16, 2014, at 9:39 AM, Amit Anand  wrote:
> >
> > Ok cool Ill wait it out see what happens. So now I have another stupid 
> > question - after all is said and done, how many copies of my data will I 
> > have?! What I am aiming for is something like 2 regions and 3 replicas ie, 
> > 2 copies of the data in region one and one copy in region 2.
> >
> > On Tue, Dec 16, 2014 at 12:35 PM, John Dickinson  wrote:
> > That's normal. See the "...or none can be due to min_part_hours". Swift is 
> > refusing to move more data until the stuff likely currently in flight has 
> > settled. See 
> > https://swiftstack.com/blog/2012/04/09/swift-capacity-management/
> >
> > --John
> >
> >
> >
> >
> >
> >
> > > On Dec 16, 2014, at 9:09 AM, Amit Anand  wrote:
> > >
> > > Hi John thank you!
> > >
> > > So I went ahead and added two more storage nodes to the existing rings 
> > > (object, account, container) and tried to rebalance on the controller I 
> > > got this:
> > >
> > > [root@controller swift]# swift-ring-builder object.builder rebalance
> > > Reassigned 1024 (100.00%) partitions. Balance is now 38.80.
> > > ---
> > > NOTE: Balance of 38.80 indicates you should push this
> > >   ring, wait at least 1 hours, and rebalance/repush.
> > > ---
> > >
> > >
> > > For all three. So while waiting, I went ahead and added the *.gz files 
> > > and swift.conf to the new nodes and started the Object Storage Services 
> > > on the both the new storage nodes Now I am seeing this after I try to 
> > > rebalance after waiting about an hour:
> > >
> > > [root@controller swift]# swift-ring-builder object.builder rebalance
> > > No partitions could be reassigned.
> > > Either none need to be or none can be due to min_part_hours [1].
> > >
> > > Devices 4,5,6,7 are the new ones I added in region 2.
> > >
> > >
> > > [root@controller swift]#  swift-ring-builder object.builder
> > > object.builder, build version 9
> > > 1024 partitions, 3.00 replicas, 2 regions, 2 zones, 8 devices, 38.80 
> > > balance
> > > The minimum number of hours before a partition can be reassigned i

Re: [Openstack] Feature wish list somewhere? 2 Devs looking to contribute

[John Dickinson]

Michael,

In Swift, I've created https://wiki.openstack.org/wiki/Swift/ideas to document 
some ideas that are relatively low-impact, isolated from major ongoing work, 
and still pretty nice to have. And although I have created it yet, I'm working 
on some other tools to have a much better, holistic, answer for your question 
for Swift.

We'd love to have you join in and participate in the Swift project. We've got 
some big stuff going on (erasure codes, encryption, efficiency improvements) 
and a lot of smaller things that would continue to make Swift better. Please 
join us in #openstack-swift if you've got an itch to scratch. We'd love to have 
you.


--John




> On Jan 22, 2015, at 7:56 PM, Michael Gale  wrote:
> 
> Hey,
> 
> Does anyone know where I could find a feature wishlist? Ideally a list of 
> outstanding features or functionality that is wanted. But is currently 
> missing from OpenStack and not being worked on or has been started but needs 
> a lot of dev help?
> 
> I am looking for a way that 2 full time python devs could contribute to the 
> project.
> 
> Thanks
> Michael
> 
> 
> 
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] Failure and Membership detection in Swift

[John Dickinson]

There's a ton of info available at https://swiftstack.com/openstack-swift/. 
Specifically, take a look at 
https://swiftstack.com/openstack-swift/architecture/ for how Swift solves some 
of these issues.

You might also find the info at http://swift.openstack.org useful, especially 
http://docs.openstack.org/developer/swift/admin_guide.html.

For more info on failure handling, see 
https://www.youtube.com/watch?v=mKZ7kDDPSIU where I went through several 
failure conditions and show how Swift recovers from them. There's also a much 
older blog post at 
https://swiftstack.com/blog/2012/09/13/how-openstack-swift-handles-hardware-failures/.

While it might not be the academic paper you're looking for, this should give a 
good groundwork of info. If you have other questions, feel free to ask.

--John


> On Jan 30, 2015, at 12:56 PM, Behrooz Shafiee  wrote:
> 
> Hello All,
> 
>  Unfortunately, I could not find any academic paper which describes Swift in 
> academic terms, most of documents are user manual and general overview. I 
> need to know some details about Swift which I could not find in the public 
> documents.
> 1) How does membership detection works in Swift (discovering newly added 
> nodes)?
> Digging this link, I realized unlike many other systems (Dynamo, 
> Cassandra,...) Swift does not use a gossip based protocol. And it seems that 
> it is being handled by the concept of Rings;however, what is a Ringbuilder? 
> is it a central service or a distributed protocol? what if it fails? How 
> rings info and its changes are shared among different nodes?
> 
> 2)How failure detection works in Swift?
> Going through OpenStack Swift book by Joe Arnold, I realized a timeout-based 
> mechanism by health-monitor is in charge of finding failures. Is that correct?
> 
> 
> Thanks in advance,
> --
> Behrooz
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] Failure and Membership detection in Swift

[John Dickinson]

To give you some more specific answers to your questions...


Nodes are "discovered" by being added to the ring. Simply put, this means an 
operator or external (to swift) management system is responsible for adding the 
device into. It is not automatically done inside of Swift (note that it could 
be done by some provisioning system)

The ring has two parts: the ring file and the builder file. The ring file is 
what's distributed to nodes in the cluster, and it is created from the builder 
file. Rings are updated out-of-band to the Swift processes. That is, there 
isn't an API command to modify rings. Rings are updated and deployed to the 
nodes.

Some of the failure detection is done by timeouts. There are timeouts on 
connecting to another node and timeouts on data transmitted between nodes. 
There are other failure detection scenarios like looking at system messages 
(swift-drive-audit), detecting full or failed drives with mount checks and 
fallocate calls, and checksum calculations for detecting bit rot.


--John






> On Jan 30, 2015, at 1:21 PM, John Dickinson  wrote:
> 
> There's a ton of info available at https://swiftstack.com/openstack-swift/. 
> Specifically, take a look at 
> https://swiftstack.com/openstack-swift/architecture/ for how Swift solves 
> some of these issues.
> 
> You might also find the info at http://swift.openstack.org useful, especially 
> http://docs.openstack.org/developer/swift/admin_guide.html.
> 
> For more info on failure handling, see 
> https://www.youtube.com/watch?v=mKZ7kDDPSIU where I went through several 
> failure conditions and show how Swift recovers from them. There's also a much 
> older blog post at 
> https://swiftstack.com/blog/2012/09/13/how-openstack-swift-handles-hardware-failures/.
> 
> While it might not be the academic paper you're looking for, this should give 
> a good groundwork of info. If you have other questions, feel free to ask.
> 
> --John
> 
> 
>> On Jan 30, 2015, at 12:56 PM, Behrooz Shafiee  wrote:
>> 
>> Hello All,
>> 
>> Unfortunately, I could not find any academic paper which describes Swift in 
>> academic terms, most of documents are user manual and general overview. I 
>> need to know some details about Swift which I could not find in the public 
>> documents.
>> 1) How does membership detection works in Swift (discovering newly added 
>> nodes)?
>> Digging this link, I realized unlike many other systems (Dynamo, 
>> Cassandra,...) Swift does not use a gossip based protocol. And it seems that 
>> it is being handled by the concept of Rings;however, what is a Ringbuilder? 
>> is it a central service or a distributed protocol? what if it fails? How 
>> rings info and its changes are shared among different nodes?
>> 
>> 2)How failure detection works in Swift?
>> Going through OpenStack Swift book by Joe Arnold, I realized a timeout-based 
>> mechanism by health-monitor is in charge of finding failures. Is that 
>> correct?
>> 
>> 
>> Thanks in advance,
>> --
>> Behrooz
>> ___
>> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>> Post to : openstack@lists.openstack.org
>> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> 
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] Help regarding Swift Tese cases

[John Dickinson]

You can test Swift functionality with Swift's included functional tests.

In the Swift source tree, you can run the .functests script 
(https://github.com/openstack/swift/blob/master/.functests). This will look for 
/etc/swift/test.conf (sample at 
https://github.com/openstack/swift/blob/master/test/sample.conf). Set up the 
auth credentials for any remote Swift cluster in the [func_test] section, and 
you'll be able to test its functionality.


--John




> On Feb 8, 2015, at 9:22 PM, saurabh suman <90.su...@gmail.com> wrote:
> 
> Hi All,
> 
> I am planning to implement cloud storage using openstack swift. Now to test 
> the swift functionality,as far as I know, there is only one test case related 
> to  swift test_swift_basic_ops.py present at  the path 
> openstack/tempest/tree/master/tempest/scenario .
> 
> I was wondering if there are other test cases related to swift functionality 
> present.
> 
> Please let me know where can I find those test cases. In case if only a 
> single test case is available ,can anyone guide me through writing a new test 
> case for testing swift.
> 
> 
> Regards,
> Saurav Suman 
> 
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] Help regarding Swift Tese cases

[John Dickinson]

Sounds like what you're looking for are Swift's probe tests. These take a 
cluster of a known configuration and test that the cluster components work 
together as expected. eg PUT some data, delete one replica, run replication, 
and ensure that the missing replica is restored.

The auditors and replicators are also tests via Swift's unit tests.


--John



> On Feb 18, 2015, at 3:05 AM, Chmouel Boudjnah  wrote:
> 
> saurabh suman <90.su...@gmail.com> writes:
> 
>> Futher I don't see any test cases testing the functionality of Auditors and
>> Replicators. Can anyone guide me if they are present. In case
>> they are not, how is there fucntionality tested.
> 
> The functional tests of Swift are "black box" type of testing, AFAIK
> there is indeed no "white box" testing that would test the auditors or
> replicators directly.
> 
> Chmouel
> 
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [Swift] Using unevenly sized disks in a cluster

[John Dickinson]

Well, it's complicated ;-)

Start with http://docs.openstack.org/developer/swift/overview_ring.html


Think about your scenario. if you have differently sized failure domains, then 
there are 2 options: limit capacity to the smallest failure domain or have some 
capacity overweighted (ie more than one replica there). Both of these are 
reasonable and have different tradeoffs.

If you deploy your Swift cluster so that you have evenly sized failure domains 
(regions, zones, servers), then things will "just work" exactly as you expect 
it to. If you don't have even failure domains, then you'll have to figure how 
much you want to accept overweighted failure domains (more than the expected 
number of replicas in a failure domain).

There is a "knob" in Swift you can turn to configure this choice: overload. See 
the docs linked above for info on it. Also, The swift-ring-builder also now 
includes a "dispersion" command so you can see if your cluster is set up to 
have overweighted failure domains.


--John





> On Feb 26, 2015, at 12:40 PM, Shrinand Javadekar  
> wrote:
> 
> Hi,
> 
> I have a question about using unevenly sized disks in a Swift cluster
> configured to do 2x replication.
> 
> Let's say I start with two disks of 10GB each and configure the rings
> so that both the disks have the same weight (say 10). In this case,
> one replica of each object will be on each device.
> 
> At a later point, I increase the capacity of the cluster by adding one
> 500GB disk. I also assign it a weight 500.
> 
> This means that my 500GB disk should have 50x more data than the 10GB
> ones. Swift will move some replicas of existing objects onto this
> 500GB disk.
> 
> How does Swift place data such that it the "as unique as possible"
> policy is maintained. Beyond a point, won't it have to place both
> replicas on the same device (the 500GB disk)?
> 
> If yes, to overcome the above problem, does Swift recommend always
> adding a pair of evenly sized disks (for 2x replication)?
> 
> Thanks in advance.
> -Shri
> 
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] Is openstack swift impacted by CVE-2015-2808 (RC4 vulnerability) ?

[John Dickinson]

Swift does not use RC4

--John



> On Apr 22, 2015, at 11:41 PM, Mohammed, Allauddin  
> wrote:
> 
> Hi All,
>Is openstack swift impacted by CVE-2015-2808 (RC4 vulnerability)?
> 
> https://access.redhat.com/security/cve/CVE-2015-2808
> 
> Any ideas?
> 
> Regards,
> Allauddin
> 
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [Swift] Directory structure during data placement

[John Dickinson]

> On Apr 24, 2015, at 12:42 AM, Shrinand Javadekar  
> wrote:
> 
> Hi,
> 
> I observe that while placing data, the object server creates a
> directory structure:
> 
> /srv/node/r0/objects//<3 byte hash suffix>//.data.
> 
> Is there a reason for the  directory to be created? Couldn't
> this just have been
> /srv/node/r0/objects//<3 byte hash suffix>/.data?

Let's explore that idea. First, the general concept is sound. But let's explore 
the implications.

Suppose we did away with the hash dir and just had has.data. Then, each hash 
suffix directory will end up with an enormous amount of directories in it. This 
itself can cause issues in file systems. In fact, this is exactly why we have 
the hash suffix directory: to prevent the cardinality of the partition 
directory from becoming so large. So just doing away with the hash directory 
could cause some problems for the system as more and more objects get added 
(doing a listdir on a directory with a lot of files in it is _extremely_ slow).

But there's more than just splaying. .data files are not the only thing that 
can be stored about an object. There are also .meta files (which are used for 
"fast-post" but not enabled by default). These files store metadata, as you 
would assume. There are also .ts files (for "tombstone") that identify when an 
object has been deleted. And in the new erasure code storage policy, there is a 
new .durable file that marks when the system has an object that is durable in 
the cluster.

Also, each of these files are named according to a timestamp. So if we did away 
with the hash directory and instead put it all in the hash suffix, then we'd 
have to name files like ..data or something so that we can get 
ordering. Aside from the listdir issues mentioned above, that sort of filtering 
would also be expensive to sort and group all the files so that concurrent 
operations can be resolved. Therefore we've put all the things (ie files) 
associated with an object into its own directory.

Of course, the cost for the deeper directory structure is that there are more 
inodes and dentries in the filesystem.

All that being said, I think that the combined features of DiskFiles and 
Storage Policies should allow for some interesting experimentation of the 
on-disk layout. I'm sure there are optimizations, especially if you have any 
foreknowledge of the kind of data being stored. For example, small files could 
take advantage of some sort of Haystack-style slab storage. These ideas are 
simply that right now--ideas not implementations. But I'd love to see some R&D 
in these areas.


Hope this helps explain why the on-disk layout is the way it is.


--John





> 
> I am seeing a situation where after writing a few hundred Gigs worth
> of data, where each object is 256K, XFS metadata performance is
> deteriorating. Having less number of directories might help in that
> case.
> 
> -Shri
> 
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [Swift] tmp directory causing Swift slowdown

[John Dickinson]

Great, thanks. Sounds like a pretty interesting performance improvement.

--John


> On Apr 30, 2015, at 11:27 AM, Shrinand Javadekar  
> wrote:
> 
> I was able to make the code change to create the tmp directory in the
> 3-byte hash directory and fix the unit tests to get this to work. I
> will file a bug to get a discussion started on this, in case there are
> people not following this thread.
> 
> On Wed, Apr 29, 2015 at 4:08 PM, Shrinand Javadekar
>  wrote:
>> Hi,
>> 
>> I have been investigating a pretty serious Swift performance problem
>> for a while now. I have a single node Swift instance with 16 cores,
>> 64GB memory and 8 MDs of 3TB each. I only write 256KB objects into
>> this Swift instance with high concurrency; 256 parallel object PUTs.
>> Also, I was sharding the objects equally across 32 containers.
>> 
>> On a completely clean system, we were getting ~375 object puts per
>> second. But this kept on reducing pretty quickly and by the time we
>> had 600GB of data in Swift, the throughput was ~100 objects per
>> second.
>> 
>> We used sysdig to get a trace of what's happening in the system and
>> found that the open system calls were taking way longer; several 100s
>> of milliseconds, sometimes even 1 second.
>> 
>> Investigating this further revealed a problem in the way Swift writes
>> the objects on XFS. Swift's object server creates a temp directory
>> under the mount point /srv/node/r0. It create an file under this temp
>> directory first (say /srv/node/r0/tmp/tmpASDF) and eventually renames
>> this file to its final destination.
>> 
>> rename /srv/node/r0/tmp/tmpASDF ->
>> /srv/node/r0/objects/312/eef/deadbeef/33453453454323424.data.
>> 
>> XFS creates an inode in the same allocation group as it parent. So,
>> when the temp file tmpASDF is created, it goes in the same allocation
>> group of "tmp". When the rename happens, only the filesystem metadata
>> gets modified. The allocation groups of the inodes don't change.
>> 
>> Since all object PUTs start off in the tmp directory, all inodes get
>> created in the same allocation group. The B-tree used for keeping
>> track of these inodes in the allocation group grows bigger and bigger
>> as more files are written and parsing this tree for existence checks
>> or for creating new inodes becomes more and more expensive.
>> 
>> See this discussion [1] I had on the XFS mailing list where this issue
>> was brought to light. And this other slightly old thread where the
>> problem was identical [2].
>> 
>> I validated this theory by periodically deleting the temp directory. I
>> observed that the objects per second was not reducing at the same rate
>> as earlier. Staring at ~375 obj/s, after 600GB data in Swift, I was
>> getting ~340 obj/s.
>> 
>> Now, how do we fix this?
>> 
>> One option would be to make the temp directory somewhere deeper in the
>> filesystem rather than immediately under the mount point. E.g. create
>> one temp directory under each of the 3-byte hash directories. And use
>> the temp directory corresponding to the object's hash.
>> 
>> But, it's unclear what other repercussions will this have? Will the
>> replicator start replicating this temp directory?
>> 
>> Another option is to actually delete the tmp directory periodically.
>> Problem is that we don't know when. And whenever we decide to do it,
>> the temp directory may have some file in it making it impossible to
>> delete the directory.
>> 
>> Any other options?
>> 
>> Thanks in advance.
>> -Shri
>> 
>> [1] http://www.spinics.net/lists/xfs/msg32868.html
>> [2] 
>> http://xfs.9218.n7.nabble.com/Performance-degradation-over-time-td28514.html
> 
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] Swift Atomic Operations

[John Dickinson]

Yes. Write operations are atomically written to disk. Simultaneous reads and 
writes will not interfere with one another.

--John





> On Jun 7, 2015, at 5:56 AM, Binan AL Halabi  wrote:
> 
> Hi all
> 
> Does Openstack object store provide atomic operations ?
> If multiple parties update an object, will swift guarantees one access at a 
> time  for data integrity?
> 
> Thank you.
> 
> 
> 
> Den söndag, 7 juni 2015 14:49 skrev Binan AL Halabi :
> 
> 
> Hi all
> 
> Does Openstack object store provide atomic operations ?
> If multiple parties update an object, will swift guarantees one access at a 
> time ?
> 
> Thank you.
> 
> 
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [Swift][Cinder] Installing Swift breaks Cinder ipv6

[John Dickinson]

The python-dnspython package is only used in Swift for the cname_lookup 
middleware. If you are not using that middleware, you can safely not use 
python-dnspython.

--john



> On Jun 30, 2015, at 9:54 PM, Mark Kirkwood  
> wrote:
> 
> On 01/07/15 13:10, Mark Kirkwood wrote:
>> I ran into this a few days ago. The context is we are running a node
>> that performs several api functions and it is going to be a swift proxy
>> too (at a later date when usage ramps up many of these functions may
>> move to individual machines, but for now we'd like to keep them together).
>> 
>> The machines are running Ubuntu Trusty, and we are installing Icehouse
>> packages.
>> 
>> So currently the cinder api service runs and listens on a wide ipv6
>> range (::). After I install the python-swift package the cinder api
>> refuses to start with:
>> 
>> 2015-07-01 00:50:25.327 12159 CRITICAL cinder [-] gaierror: [Errno -9]
>> Address family for hostname not supported
>> 2015-07-01 00:50:25.327 12159 TRACE cinder Traceback (most recent call
>> last):
>> 2015-07-01 00:50:25.327 12159 TRACE cinder   File "/usr/bin/cinder-api",
>> line 57, in 
>> 2015-07-01 00:50:25.327 12159 TRACE cinder server =
>> service.WSGIService('osapi_volume')
>> 2015-07-01 00:50:25.327 12159 TRACE cinder   File
>> "/usr/lib/python2.7/dist-packages/cinder/service.py", line 301, in __init__
>> 2015-07-01 00:50:25.327 12159 TRACE cinder port=self.port)
>> 2015-07-01 00:50:25.327 12159 TRACE cinder   File
>> "/usr/lib/python2.7/dist-packages/cinder/wsgi.py", line 118, in __init__
>> 2015-07-01 00:50:25.327 12159 TRACE cinder backlog=backlog)
>> 2015-07-01 00:50:25.327 12159 TRACE cinder   File
>> "/usr/lib/python2.7/dist-packages/cinder/wsgi.py", line 174, in _get_socket
>> 2015-07-01 00:50:25.327 12159 TRACE cinder family=family)
>> 2015-07-01 00:50:25.327 12159 TRACE cinder   File
>> "/usr/lib/python2.7/dist-packages/eventlet/convenience.py", line 38, in
>> listen
>> 2015-07-01 00:50:25.327 12159 TRACE cinder sock.bind(addr)
>> 2015-07-01 00:50:25.327 12159 TRACE cinder   File
>> "/usr/lib/python2.7/socket.py", line 224, in meth
>> 2015-07-01 00:50:25.327 12159 TRACE cinder return
>> getattr(self._sock,name)(*args)
>> 2015-07-01 00:50:25.327 12159 TRACE cinder gaierror: [Errno -9] Address
>> family for hostname not supported
>> 
>> The culprit appears to be the python-dnspython package that is a dep for
>> python-swift.
>> 
>> Now it appears that there is some monkey patching going on in the
>> various wsgi eventlet modules to do ipv6 - somehow python-dnspython is
>> changing the way that works - any thoughts welcome, as it would be cool
>> to have things working again!
>> 
> 
> After reading up a bit on this whole monkey patching thing, I see that adding:
> 
> EVENTLET_NO_GREENDNS=yes
> 
> to the cinder user's env seems to get it working again. However this will 
> mean that there is *no* patching...which might not be the best answer (still 
> hoping for better suggestions).
> 
> Regards
> 
> Mark
> 
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] SWIFT - File System Kernel Module ?

[John Dickinson]

No. There are no plans for Swift to implement a file system.

--John



> On Jul 16, 2015, at 10:44 AM, Brent Troge  wrote:
> 
> Does the swift project plan to have a file system kernel module in the same 
> manner as ceph's file system?
> 
> 
> 
> 
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] SWIFT - File System Kernel Module ?

[John Dickinson]

(disclosure: I am employed by SwiftStack)

That filesystem gateway from SwiftStack provides a CIFS/NFS interface to data 
stored in Swift so that older, existing applications that cannot be rewritten 
can still take advantage of many of Swift's benefits. However, it is not part 
of the OpenStack Swift project. It is part of SwiftStack's proprietary product 
(which you can try at https://swiftstack.com/try-it-now/).


--John


> On Jul 16, 2015, at 11:31 AM, Martinx - ジェームズ  
> wrote:
> 
> What about this:
> 
> https://swiftstack.com/wp-content/uploads/2015/02/20140710_Datasheet_Filesystem_Gateway.pdf
>  ?
> 
> I never tried it myself...
> 
> On 16 July 2015 at 15:09, John Dickinson  wrote:
> No. There are no plans for Swift to implement a file system.
> 
> --John
> 
> 
> 
> > On Jul 16, 2015, at 10:44 AM, Brent Troge  wrote:
> >
> > Does the swift project plan to have a file system kernel module in the same 
> > manner as ceph's file system?
> >
> >
> >
> >
> > ___
> > Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> > Post to : openstack@lists.openstack.org
> > Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> 
> 
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> 
> 



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] Python API for Dynamic Large Objects in Swift

[John Dickinson]

You can also use swiftclient.service module. It is the wrapper on top of the 
low-level swiftclient.client that is actually what the CLI tool uses for some 
of its higher-order functions (like splitting larger local files and making 
manifest objects).

You'd have something like:

results = swiftclient.service.upload('target_container', 
['path/to/local/file',], options={'segment_size': 1048576000})


--John





> On Jul 28, 2015, at 9:16 PM, Mark Kirkwood  
> wrote:
> 
> On 25/07/15 00:28, Vincenzo Pii wrote:
>> To upload a Dynamic Large Object with the Swift CLI one can just do
>> 
>> swift upload newcont -S 1048576000 large_object
>> 
>> but, is there any equivalent in the swift python APIs (swiftclient module
>> from https://github.com/openstack/python-swiftclient)?
>> 
>> The content_length parameter of put_object will just truncate the content...
>> 
> 
> The general approach is outlined here (Swift API section):
> 
> https://www.mirantis.com/blog/large-objects-in-cloud-storages/
> 
> 
> I have attached a rather simple sample program that uses the idea.
> 
> regards
> 
> Mark
> 
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] Query regarding contribution for monitoring of Nova and Swift

[John Dickinson]

I'm the Project Technical Lead for Swift, and I'd be happy to look over a 
summary of your work about monitoring Swift. Feel free to email me directly or 
find me in #openstack-swift on IRC (I'm notmyname).

--John




On 8 Sep 2015, at 3:20, pragya jain wrote:

>  Hello all
> Me and my colleague, aparna are carrying out research in the area of cloud 
> computing under Department of CS, University f Delhi.  We would like to 
> contribute our research work regarding monitoring of Nova and Swift. We would 
> appreciate if we can find the appropriate link with whom we can connect to 
> know if our work is relevant for contribution. -RegardsPragya 
> JainDepartment of Computer ScienceUniversity of DelhiDelhi, 
> India___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


signature.asc
Description: OpenPGP digital signature
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] Swift storage policies by account

[John Dickinson]

Currently, no.

However, we've talked about (and IIRC there's been a little work on) a sort of 
"defaulter" functionality that would (among other things) allow for setting a 
per-account default storage policy. It's something that I think would be a 
great addition to Swift.

If you want to talk about it further or help out with getting it implemented, 
stop by @openstack-swift on freenode, and ask! I'm notmyname, and I'd be happy 
to help out.

--John




On 3 Dec 2015, at 8:32, Carlos Rodrigues wrote:

> Hi,
>
> I have multiple nodes of Swift and create some storage policies based on
> storage types and zones of nodes.
>
> I would like to know if it's possible to have one storage policy for
> each account?
> Or, for instance, if it's possible to define one default storage policy
> for one account?
>
> Regards,
>
> -- 
> Carlos Rodrigues
>
> Engenheiro de Software Sénior
>
> Eurotux Informática, S.A. | www.eurotux.com
> (t) +351 253 680 300 (m) +351 911 926 110
>
>
>
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


signature.asc
Description: OpenPGP digital signature
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] openstack swift

[John Dickinson]

Th short answer is that no you cannot specify a storage node for a given 
object. Data placement is done with the hash ring, so placement is based on the 
hash of the object name and the current state of the drives in the cluster.

--John



On 22 Feb 2016, at 4:06, pgh pgh wrote:

> Hello All,
>
> I'm a beginner with openstack but I'm mainly interested in Swift...
> I was wonder if in addition to use the Swift modified consistent hashing
> ring, is possible to manually specify the storage node, if so, how does
> Swift to maintain load balancing ?
>
>
> *Thanks and best regards,*
>
>
> *Pgh*
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


signature.asc
Description: OpenPGP digital signature
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [openstack][swift] Question about Swift consistency and concurrent requests

[John Dickinson]

inline

On 22 Feb 2016, at 5:06, Kévin Bernard-Allies wrote:

> Hello,
>
> I've some questions about the handling of concurrent requests by Swift.
> Any help would be great:)
>
>
> If I understand correctly, a PUT request on swift-proxy will be transmitted 
> to at least half the storage nodes plus one, before returning the response.

Correct. And because I've heard it wrongly many time before, I'll note that 
these requests happen concurrently (it's NOT store-and-forward).

> A subsequent GET request should have an updated version (because it also make 
> the GET request to several storage nodes ?), but the listing is not 
> immediately up-to-date.

Correct that that listing might not be immediately up-to-date. The subsequent 
GET should have an updated version because the placement locations are 
relatively stable.

Read-your-writes consistency is as dependent on hardware and tuning as much as 
the software itself. That being said, with Swift' default config options and 
"normal" hardware choices, Swift offers very very good out-of-the-box 
read-your-write consistency. However, Swift' does not provide 
read-your-overwrites consistency. That is, a read after an overwrite of 
existing data (i.e. same object name) may provide the older version of the data 
or the newer version of the data. Normally, this will only happen as a result 
of internal cluster hardware failure or capacity adjustment.

A very simple (and common) way to avoid any reads of older versions of objects 
in Swift is to use unique names to write data.

Note that in no case will the application get corrupted data (part of v1 and 
part of v2). In all cases, the client will get back the newer or older version.

>
> Now, I wonder what happens when two PUT requests with 'If-Match' headers are 
> executed at the same time.
> The 'If' header is active, and one of the two requests is rejected ? Or the 
> header is ignored, and the request with the last timestamp wins ?

Note that Swift only supports "If-Match: *" for PUTs (i.e. only PUT if some 
data already exists). You cannot specify "only PUT if this particular content 
exists".

However, the final question is a simple one. Yes, Swift resolves concurrent 
writes with a last-timestamp-wins strategy. The timestamp is determined by the 
Swift proxy server handling the request, and it is created when the request 
starts. Note that this implies the need to keep Swift proxy servers in sync, eg 
with something like NTP.

At some point (in any system) "at the same time" will get below the resolution 
of the system. So what happens when you have two writes to Swift that happen so 
close to each other that they are within the time resolution of the system? At 
this point, the resolution is not defined. One of them will probably become the 
only version stored, but there's no deterministic way to know which one will.

Swift's timestamp resolution is decimillisecond (5 decimal places to the right 
of the decimal). Why? Because "floating point". See 
https://gist.github.com/notmyname/3fffa234f602f8b8e616.

>
>
> I search a way to ensure I can't lose a version of the files without noticing 
> it.

There are a couple of strategies. Operationally, make sure you keep proxies in 
sync and monitor that. From the application perspective, you can use unique 
names for writes.


--John




> I'll take any idea.
>
>
> Thanks all!
> -- 
> Kévin Bernard-Allies
>
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


signature.asc
Description: OpenPGP digital signature
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] swift ringbuilder and disk size/capacity relationship

[John Dickinson]

On 3 Mar 2016, at 13:32, Mark Kirkwood wrote:

> On 03/03/16 03:57, Peter Brouwer wrote:
>> Hello
>>
>> I am trying to find some information on the relationship of the ring
>> (builder) structure of swift and the size of disks used, i.e. how is
>> capacity of a disk managed.
>> I've seen many docs/blogs explaining the ring and partitions but I have
>> not seen any info on how available space per disk is managed, i.e. the
>> relationship between disk size and allocating ring partitions.
>>
>> Do all disks in a ring need to be of the same capacity? If not how is
>> swift handling the amount of free space?
>>
>> Any pointers to documentation on this?
>>
>
> You specify a weight for each device that is added to the ring, this
> lets you cope with different sized disks e.g:
>
> Suppose sdb is 1TB and sdc 500GB
>
> $ swift-ring-builder object.builder add r1z1-192.168.122.21:6000/sdb 1
> $ swift-ring-builder object.builder add r1z1-192.168.122.21:6000/sdc 0.5


One easy way to manage ring weights is to use the number of GB on the drive. So 
a 1TB drive is 1000, a 500Gb drive is 500, and an 8TB drive is 8000. Makes 
thinking about it easier, at least to me.

The actual weight doesn't matter by itself. It only matters compared to all the 
other weights in the ring.


--John




>
> Cheers
>
> Mark
>
>
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


signature.asc
Description: OpenPGP digital signature
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [swift] Can replication and proxy-to-storage communication be encrypted?

[John Dickinson]

Your code review is correct.

There's some ideas on how to make things more secure that I expect to be 
tackled relatively soon, but for now it's all HTTP.

In single-site deployments, the internal Swift network (i.e. proxy to storage 
and storage to storage) should be on a private network). And and site-to-site 
connectivity for a multi-site deployment should be over a VPN or similar.

--John



On 30 Mar 2016, at 18:50, Mark Kirkwood wrote:

> Hi,
>
> I'm looking at configuring a multi region cluster, and am thinking about
> what type of encryption is needed for inter region traffic, and where
> this needs to be done (e.g VPN or swift encrypting its own communication).
>
> My quick scan of the code seems[1] to point to internal communication
> being http only - but I'm asking in case I've missed something!
>
> regards
>
> Mark
>
> [1]
> Examining files in swift/obj,proxy,common it looks like proxy-to-storage
> (and storage-to-storage) communication is always unencrypted (i.e
> common/bufferedhttp:http_connect is called without ssl set).
>
> Also looking at swift/obj/ssync_sender.py it seems to me that
> replication is not encrypted either.
>
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


signature.asc
Description: OpenPGP digital signature
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] there seems to be a conflict of IP port 6000 for swift

[John Dickinson]

The "bind_port" setting in Swift's config files is required to be explicitly 
set. It doesn't matter what the number is. If you have a port conflict with 
another service, feel free to change it.

As to why port 6000 was used initially... The truth is probably lost to the 
mists of time. However, I know that "we're listening for X11 on the storage 
boxes of our object storage system" was never a concern :-)

There's an open patch to change the docs to use a different port, but for now, 
feel free to use another port number which is more convenient for you.


--John




On 29 Apr 2016, at 4:51, Peter Brouwer wrote:

> On 29/04/2016 10:12, Peter Brouwer wrote:
>> Hello All,
>>
>>
>> When looking at the ports used by the system I noticed port 6000 is already 
>> in use before starting the swift services.
>>
>> In the swift object-server.conf file the default port setting is 6000 to, so 
>> yes when starting the swift object server there is an error message in the 
>> corresponding log file and the object-service is put in a maintenance state.
>>
>> How do I find out which other unix process is using this 6000 port, it seems 
>> that 6001 is in the range (6000-6007)  commonly used by x11. Why did the 
>> swift openstack folks pick the same port range for their storage services 
>> processes?
> To followup on the last q, port 6000 is used by X11 if you enable tcp 
> listening option ( part of startx or in gdm config file)
> This is considered a non secure situation and not recommended, but it leaves 
> the risk of swift using a same port as x11 if an admin chooses to use the tcp 
> listen option for remote X11 connections (vnc)
>
>
>>
>>
>>
>
> -- 
> Regards,
>
> Peter Brouwer, Principal Software Engineer,
> Oracle Application Integration Engineering.
> Phone:  +44 1506 672767, Mobile +44 7720 598 226
> E-Mail: peter.brou...@oracle.com
>
>
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


signature.asc
Description: OpenPGP digital signature
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] there seems to be a conflict of IP port 6000 for swift

[John Dickinson]

...and we just landed a patch to use the 6200 range for examples.

https://review.openstack.org/#/c/274840/


--John




On 29 Apr 2016, at 9:26, John Dickinson wrote:

> The "bind_port" setting in Swift's config files is required to be explicitly 
> set. It doesn't matter what the number is. If you have a port conflict with 
> another service, feel free to change it.
>
> As to why port 6000 was used initially... The truth is probably lost to the 
> mists of time. However, I know that "we're listening for X11 on the storage 
> boxes of our object storage system" was never a concern :-)
>
> There's an open patch to change the docs to use a different port, but for 
> now, feel free to use another port number which is more convenient for you.
>
>
> --John
>
>
>
>
> On 29 Apr 2016, at 4:51, Peter Brouwer wrote:
>
>> On 29/04/2016 10:12, Peter Brouwer wrote:
>>> Hello All,
>>>
>>>
>>> When looking at the ports used by the system I noticed port 6000 is already 
>>> in use before starting the swift services.
>>>
>>> In the swift object-server.conf file the default port setting is 6000 to, 
>>> so yes when starting the swift object server there is an error message in 
>>> the corresponding log file and the object-service is put in a maintenance 
>>> state.
>>>
>>> How do I find out which other unix process is using this 6000 port, it 
>>> seems that 6001 is in the range (6000-6007)  commonly used by x11. Why did 
>>> the swift openstack folks pick the same port range for their storage 
>>> services processes?
>> To followup on the last q, port 6000 is used by X11 if you enable tcp 
>> listening option ( part of startx or in gdm config file)
>> This is considered a non secure situation and not recommended, but it leaves 
>> the risk of swift using a same port as x11 if an admin chooses to use the 
>> tcp listen option for remote X11 connections (vnc)
>>
>>
>>>
>>>
>>>
>>
>> -- 
>> Regards,
>>
>> Peter Brouwer, Principal Software Engineer,
>> Oracle Application Integration Engineering.
>> Phone:  +44 1506 672767, Mobile +44 7720 598 226
>> E-Mail: peter.brou...@oracle.com
>>
>>
>> ___
>> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>> Post to : openstack@lists.openstack.org
>> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


signature.asc
Description: OpenPGP digital signature
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [Swift] Warning when running Swift on PyPy

[John Dickinson]

That's an interesting error, and a little different than what I had seen in my 
tests. I'd be really interested in your experience with patching eventlet 
(directly or monkey-patching from swift) to add those methods.


--John






On 5 May 2016, at 13:01, Shrinand Javadekar wrote:

> Hi,
>
> Based on the "Doubling Performance in Swift with No Code Changes" talk
> at the Openstack Summit, I decided to give running Swift on PyPy a
> shot. I configured a VM with Swift (largely based on the steps
> mentioned in [1], although I did have to change a few things). Swift
> was up and running. The /info endpoint showed the correct output.
>
> However, when I tried to do a simple HEAD request to get all the
> containers, I got the following warning. I did export PYPY_GC_MIN=15M
> before starting the swift services.
>
> May  5 00:35:59 shri-swift-tests swift-proxy-server: STDOUT:
> /home/blah/pypy/lib-python/2.7/SocketServer.py:717:
> RefCountingWarning: 'GreenSocket' object has no _reuse/_drop
> methods#012{{#012You make use (or a library you are using makes
> use) of the internal#012classes '_socketobject' and '_fileobject'
> in socket.py, initializing#012them with custom objects.  On PyPy,
> these custom objects need two#012extra methods, _reuse() and
> _drop(), that maintain an explicit#012reference counter.  When
> _drop() has been called as many times as#012_reuse(), then the
> object should be freed.#012#012Without these methods, you get the
> warning here.  This is to#012prevent the following situation: if
> your (or the library's) code#012relies on reference counting for
> prompt closing, then on PyPy, the#012__del__ method will be called
> later than on CPython.  You can#012easily end up in a situation
> where you open and close a lot of#012(high-level) '_socketobject'
> or '_fileobject', but the (low-level)#012custom objects will
> accumulate before their __del__ are called.#012You quickly risk
> running out of file descriptors, for example.#012}}#012
> self.rfile.close() (txn: txae85f55c9d3d4f26b96a5-00572a956f)
>
> Any ideas how to get over this? What are the consequences of this?
>
> Thanks in advance.
> -Shri
>
> [1] https://01.org/lp/documentation/pypy-enabling-guide-openstack-swift
>
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


signature.asc
Description: OpenPGP digital signature
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [Swift] Unexplained 404s

[John Dickinson]

On 20 May 2016, at 10:27, Shrinand Javadekar wrote:

> Hi,
>
> I am troubleshooting a test setup where Swift returned a 201 for
> objects that were put in it but later when I tried to read it, I got
> back 404s.
>
> The system has been under load. I see lots of connection errors,
> lock-timeouts, etc. However, I am not sure if ever Swift should be
> returning a 404.
>
> I tried simulating some of these on a different setup and always got
> the expected response (which wasn't a 404).
>
> - Stopped memcached and did a blob get. This returned a 401 Unauthorized 
> error.
>
> - Stopped the object-server and did a blob get. This returned a 503
> internal server error.
>
> - Stopped the container-server. This didn't have any effect. The
> container-server is not looked during every GET.
> - Stopped the account-server. Same result as container-server.
>
> Any ideas on when Swift might return a 404 even though the object was
> successfully written?
>
> Thanks in advance.
> -Shri

Use `swift-get-nodes` to see where Swift thinks the object is in the cluster, 
then check those drives for the data. If the data is actually on disk, then the 
issue is either mis-deployed ring files or possibly the timeouts you mentioned.

If the data isn't on disk, then check your logs for every mention of the object 
you're looking for to see if it was deleted. Also check the other drives in the 
cluster for the object to see if it's there (capacity adjustments or failures 
can result in data being put in other places in the cluster).

--John




signature.asc
Description: OpenPGP digital signature
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [swift] Storage node failure modes

[John Dickinson]

A few years ago, I gave this talk at LCA which covers a lot of these details.

https://www.youtube.com/watch?v=_sUvfGKhaMo&list=PLIr7I80Leee5NpoYTd9ffNvWq0pG18CN3&index=9

--John




On 27 Jun 2016, at 17:36, Mark Kirkwood wrote:

> Hi,
>
> I'm in the process of documenting failure modes (for ops documentation etc). 
> Now I think I understand the intent:
>
> - swift tries to ensure you always have the number of configured replicas
>
> In the case of missing or unmounted devices I'm seeing the expected behaviour 
> i.e:
>
> - new object creation results in the configured number of replicas (some 
> stored on handoff nodes)
> - existing objects replicated on handoff to produce the correct replica number
>
> In the case of a node (or a region) I'm *not* seeing analogous behaviour for 
> *existing* objects, i.e I am a replica down after shutting down on of my 
> nodes and waiting a while.
>
> I am testing using swift 2.7.on a small cluster of vms (4 nodes, 4 devices, 2 
> regions) - now it may be that my setup is just too trivial (or maybe I 
> haven't waited long enough for swift decide my storage node is really down). 
> Any thoughts? I'd like to understand precisely what is supposed to happen 
> when a node (and also an entire region) is unavailable.
>
> Cheers
>
> Mark
>
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


signature.asc
Description: OpenPGP digital signature
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [Swift] Is it possible to change a replication factor?

[John Dickinson]

Yes!

http://docs.openstack.org/developer/swift/overview_ring.html#fractional-replicas

Set the replica count to 2.1 then 2.2 then 2.3 etc until you get to 3. That 
will allow you to gradually change the replica count over time to prevent the 
cluster from immediately trying to create a new replica of 100% of your data.

--John




On 30 Jun 2016, at 9:53, Alexandr Porunov wrote:

> Is it possible to change a replication factor from 2 to 3 or vice versa
> from 3 to 2 in a running cluster?
>
> Best regards,
> Alexandr
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


signature.asc
Description: OpenPGP digital signature
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] Swift new starter questions (PoC)

[John Dickinson]

Great to hear that you're looking at Swift. Answers inline...


--John



On 8 Aug 2016, at 7:56, Chris wrote:

> Hello,
>
> We are currently playing with swift and try to find out if it would be useful 
> for our needs. We would use the latest mitaka release. It would be a 
> multi-region deployment in around 4-5 different countries/DCs with a total 
> size of 5 TB and millions of small files. Worst case RTT between two 
> locations would be 400ms. I came across some questions I want to ask.
>
> 1) As far as I understood Swift returns a 200 OK if the majority of replicas 
> are written. Is it possible to change this ratio?

For replicas, "quorum" is defined as ((R DIV 2) + 1) when R is odd and (R DIV 
2) when R is even, where "R" is the replica count. For example, for 3 replicas, 
the quorum is 2. For 4 replicas, the quorum is 2.

It's not possible to change the quorum size, but it is possible to change the 
replica count.

> 2) Does the write affinity now includes containers and accounts? Saw a 
> presentation from the Symantic guys on the Openstack summit where this topic 
> came up. And the acces time spikes if this information are transferred from 
> the WAN.

This sounds accurate and is the current state of the system. It's also one of 
the known ways the community would like to improve the global cluster story.

> 2a) If not does it make sense to do some kind of warm up and get the 
> container and account information saved in memcache for faster access? 
> Anybody experience at this approach?

That sounds like a fantastic idea.

Note that it's a very good idea to have a separate memcache pool for each 
region instead of one pool across all regions.

> 3) Lets say we have 5 regions and a replication factor of 3. When we use 
> write affinity 12 copies would be written in a local cluster and then in the 
> background replicated to the other regions? Isn't there a better way and 
> avoid unnecessary local writes if they would be transferred away anyway?

With 5 regions and 3 replicas, on write the data would be written to 3 
locations in the region that received the request. Later (asynchronously) the 
replicas would be written to the correct "primary" locations. With 5 regions 
and 3 replicas, it could mean that the "local" region will not have any of the 
primary locations.

> 4) What will happen if the WAN connection to a DC fails is there an option to 
> not create a another replication in other DCs just because the WAN is of fore 
> some minutes?

You are describing Swift's normal behavior. A WAN (or even LAN) failure will 
not trigger automatic replication (a drive failing, however, will trigger it).

Basically, Swift treats the inability to connect to another server as an 
availability issue instead of a durability issue. Replication is only 
automatically triggered for durability issues.

> 5) In a Swift multi region deployment is it possible to have local Keystone 
> services in every region/DC or does all authentication need to come from the 
> same keytone machine?

I think this would be possible, but I'm not sure if it's advisable.

Because any Swift endpoint in the cluster can handle any request to the 
cluster, this means that you'd be responsible for "locking" a client to a 
particular endpoint. That is, you don't want UserA in Region1 to try to do 
requests to Region2, because Region2 will have different auth set up and 
therefore the request would likely fail.

For simplicity, I'd recommend starting with one global auth endpoint and local 
memcache pools in each Swift region. I'll leave it up to you to determine if 
you want one global VIP for every region or if you want to expose different 
endpoints for each region in the cluster.

FWIW, I don't think either way is "right". In my opinion, it sounds easier to 
manage local caching instead of managing the client locking to a region.

>
> Any help and ideas appreciated!
>
> Cheers
> Chris
>
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


signature.asc
Description: OpenPGP digital signature
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [Swift] Is it possible to collect metrics from Swift without Statsd?

[John Dickinson]

Any way we could get the stats you need upstream in swift (emitted via statsd) 
and then use something like 
https://github.com/danslimmon/statsd-opentsdb-backend to get it in to opentsdb?

--John




On 9 Aug 2016, at 8:59, Alexandr Porunov wrote:

> Hello,
>
> I need to collect different metrics from swift servers and store those
> metrics into OpenTSDB.
> I found that there is way to collect metrics with statsd and graphite but
> it isn't suitable for me because I have to store metrics into OpenTSDB.
> Is there any way to store metrics from openstack swift into OpenTSDB?
> I haven't found cli commands to get metrics out from Openstack swift.
>
> Sincerely,
> Alexandr
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


signature.asc
Description: OpenPGP digital signature
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [OpenStack] [Swift] Is it possible to create chunked upload?

[John Dickinson]

There's 2 ways you can do this:

1) Use chunked transfer encoding
2) Use Swift's large object manifest objects


For the first, it's the standard HTTP semantics. You can send chunks of data to 
Swift to be stored as one object without knowing the full object size up front. 
Note that the per-object size limits still apply here (default is 5GB).

For the second, see 
http://docs.openstack.org/developer/swift/overview_large_objects.html, 
specifically "Static Large Objects" on that page. This will allow you to upload 
discrete objects and then stitch them together into a single logical object of 
arbitrary size.

--John





On 23 Aug 2016, at 10:13, Alexandr Porunov wrote:

> Hello,
>
> My server accepts files in chunks (4 Kbytes each chunk. File's size can be
> till 8 GB). Is it possible somehow store those chunks in Swift like a
> single file? Does somebody know any solution to solve this problem?
>
> Sincerely,
> Alexandr
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


signature.asc
Description: OpenPGP digital signature
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] Swift object-server Invalid Unexpected file Timestamp value in filename

[John Dickinson]

xprofile is a module that can be used to profile the running code in a swift 
proxy. I would not recommend running it in a production environment. It's 
mostly for a developer use case.

--John



On 25 Aug 2016, at 4:28, Chris wrote:

> Hello Hamza,
>
>
>
> I don’t even have the “filter-xprofile” option in my config what is it for?
>
>
>
> [DEFAULT]
>
> devices = /srv/node
>
> bind_ip = xxx
>
> bind_port = 6000
>
> mount_check = true
>
> user = swift
>
> log_name = object-server
>
> log_facility = LOG_LOCAL2
>
> log_level = INFO
>
> log_address = /dev/log
>
> workers = auto
>
>
>
> [pipeline:main]
>
> pipeline = recon object-server
>
>
>
> [filter:recon]
>
> use = egg:swift#recon
>
> recon_cache_path = /var/cache/swift
>
>
>
> [app:object-server]
>
> use = egg:swift#object
>
> set log_name = object-server
>
> set log_facility = LOG_LOCAL2
>
> set log_level = INFO
>
> set log_requests = true
>
> set log_address = /dev/log
>
>
>
> [object-replicator]
>
> concurrency = 24
>
>
>
> [object-updater]
>
> concurrency = 24
>
>
>
> [object-auditor]
>
>
>
>
>
> Cheers
>
> Chris
>
>
>
>
>
> From: Hamza Nissoul [mailto:mon240...@gmail.com]
> Sent: Wednesday, August 24, 2016 21:32
> To: Chris 
> Subject: Re: [Openstack] Swift object-server Invalid Unexpected file 
> Timestamp value in filename
>
>
>
> Could you share the contents of the config file ? and Could you please check 
> object server configuration file for below options.
>
> dump_interval = ??
> dump_timestamp = false
>
>
>
> 2016-08-24 7:31 GMT+01:00 Chris   >:
>
> Hello,
>
>
>
> I get a lot of this ones at the object servers:
>
>
>
> Aug 24 13:28:26 xxx object-server: Unexpected file 
> /srv/node/device1/objects/9540/45d/25440c02d1c5d4cc2eb8f4eba8f5745d/.1472018782.70494.data.bdYCC2:
>  Invalid Timestamp value in filename '.1472018782.70494.data.bdYCC2'
>
> Aug 24 13:28:26 xxx object-server: Unexpected file 
> /srv/node/device1/objects/4941/786/134d60506dda595a8d02019189094786/.1472019111.97852.data.49A9WI:
>  Invalid Timestamp value in filename '.1472019111.97852.data.49A9WI'
>
> Aug 24 13:28:26 xxx object-server: Unexpected file 
> /srv/node/device1/objects/25350/5c5/63063ab3de6e53da1fe6e207dfbc05c5/.1472018640.94260.data.3FbPyt:
>  Invalid Timestamp value in filename '.1472018640.94260.data.3FbPyt'
>
> Aug 24 13:28:29 xxx object-server: Unexpected file 
> /srv/node/device4/objects/45568/e0d/b2008b422421fa9e5bb1ea3849b49e0d/.1472019798.18900.data.qCd8iX:
>  Invalid Timestamp value in filename '.1472019798.18900.data.qCd8iX'
>
> Aug 24 13:28:29 xxx object-server: Unexpected file 
> /srv/node/device5/objects/30625/cb0/77a1e5f2748f7c6259e94c47d6184cb0/.1472017301.52242.data.sUNpjE:
>  Invalid Timestamp value in filename '.1472017301.52242.data.sUNpjE'
>
> Aug 24 13:28:29 xxx object-server: Unexpected file 
> /srv/node/device5/objects/30625/2bf/77a109863bcfa7e7bc4f3f2b723a52bf/.1472017690.34360.data.jtNsuS:
>  Invalid Timestamp value in filename '.1472017690.34360.data.jtNsuS'
>
> Aug 24 13:28:29 xxx object-server: Unexpected file 
> /srv/node/device5/objects/61332/7d1/ef94cbb322203a60dd38d646a94a67d1/.1472017622.99231.data.zY5rYF:
>  Invalid Timestamp value in filename '.1472017622.99231.data.zY5rYF'
>
> Aug 24 13:28:30 xxx object-server: Unexpected file 
> /srv/node/device5/objects/45166/6b3/b06e7529f65e48ffedb44e84b89ec6b3/.1472017678.58695.data.ugborJ:
>  Invalid Timestamp value in filename '.1472017678.58695.data.ugborJ'
>
> Aug 24 13:28:30 xxx object-server: Unexpected file 
> /srv/node/device5/objects/9035/215/234b21c973131d78b88c0ab56b072215/.1472017956.04018.data.SONASH:
>  Invalid Timestamp value in filename '.1472017956.04018.data.SONASH'
>
> Aug 24 13:28:30 xxx object-server: Unexpected file 
> /srv/node/device5/objects/47953/8ea/bb512db10b898183f55e9698674828ea/.1472017820.44503.data.QBYCYU:
>  Invalid Timestamp value in filename '.1472017820.44503.data.QBYCYU'
>
> Aug 24 13:28:30 xxx object-server: Unexpected file 
> /srv/node/device5/objects/47953/8ea/bb512db10b898183f55e9698674828ea/.1472017820.44503.data.rP7WEO:
>  Invalid Timestamp value in filename '.1472017820.44503.data.rP7WEO'
>
>
>
> Any help appreciated!
>
>
>
> Cheers
>
> Chris
>
>
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org 
> 
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>
>
>
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


signature.asc
Description: OpenPGP digital signature
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : ht

Re: [Openstack] [OpenStack] [Swift] Segments count minimization.

[John Dickinson]

On 13 Sep 2016, at 15:47, Michael Yoon wrote:

> Hello Alex and Clay,
>
> Pardon my jumping in here, we do something similar and have a similar use
> case (we chose 1MB segments for mobile, and 10MB segments for desktop).
> One question I have in comparing copy/bulk-delete vs keeping the manifest
> around is what is the overhead of the manifest within swift for retrieval
> (both DLO and SLO use case)?  My understanding is that in the DLO case, the
> listing is kept in the container, and that in the SLO case the manifest is
> a file in swift itself.  In both cases an extra write/read is necessary to
> store/lookup the manifest, and in the case of SLO it will in the
> containerdb ring (often SSD), and in the case of DLO it would be in the
> object ring (HDD).  Is my understanding correct?

You're mostly right. The DLO manifest becomes a container listing request with 
a given prefix. That makes DLO reads very expensive and prone to issues if the 
container hasn't been updated yet. SLO manifests are explicit and stored as an 
object in the system, without the container server being involved in any way in 
the read path (the same as "normal" objects).

So yes, there's the extra lookup that will delay time-to-first-byte for *LOs, 
but it's a lot more expensive for DLOs than for SLOs. In almost every single 
case, users should use SLOs.

There is a "speed limit" internally to Swift for large object segments. Swift 
can limit the number of segments per second that are returned to the client.[1] 
This protects the cluster from users who put thousands of 1-byte objects into a 
single manifest and then DOS the cluster. The default values are tuned to fully 
utilize a 1gbps network link with 100MB segments, IIRC.

In general, if you're using *LOs, you've go large objects, so the 
time-to-first-byte is a very small percentage of the overall transfer time.

Hope that info help.

[1] 
https://github.com/openstack/swift/blob/master/etc/proxy-server.conf-sample#L669-L678



--John




>
> Thanks,
>
> Michael Yoon
>
> Chief Technology Officer
>
> m.718.791.8670
>
> michael.y...@mimedia.com
>
> MiMedia, Inc.
>
> 32 Court St. Suite 1800
>
> Brooklyn, NY 11201
>
> Sign up! First 10 GB are on us. 
>
> On Tue, Sep 13, 2016 at 5:18 PM, Alexandr Porunov <
> alexandr.poru...@gmail.com> wrote:
>
>> Thank you Clay!
>>
>> I use ffmpeg with remote input and local output for first convertion. Here
>> is a simplified algorithm:
>> 1. Convert with h264 in the highest quality (till 4K) and save it in a
>> transformer server.
>> 2. Convert into lower quality locally. And get additional data (thumbnails
>> and so on).
>> 3. Upload all converted videos and additional data into Swift (as you
>> suggested with sub-segments and segments).
>>
>> Compressed videos then are available to the client throug the transmuxing
>> servers.
>>
>> Best regards,
>> Alexandr
>>
>> On Tue, Sep 13, 2016 at 8:55 PM, Clay Gerrard 
>> wrote:
>>
>>>
>>> On Tue, Sep 13, 2016 at 10:37 AM, Alexandr Porunov <
>>> alexandr.poru...@gmail.com> wrote:
>>>

 Correct me if I am wrong. Algorithm is the next:
 1. Upload 1MB sub-segments (up to 500 sub-segments per segment). After
 they will be uploaded I will use "copy" request to create one 500MB
 segment. After that I will delete useless 1MB segments.
 2. Upload up to 240 segments with 500MB and create a manifest for them.


>>> Is it correct? Will this algorithm be suitable for this situation?


>>> That sounds correct - I think the COPY/bulk-delete is optional - you
>>> could also just have the top level SLO point to the sub-segment SLO's
>>> manifest directly and leave all of the 1MB chunks stored in Swift as is -
>>> could potentially help smooth out the evenness of diskfullness overall.  I
>>> think 1MB is big enough to smooth out any connection overhead - but you
>>> could test with COPY consolidation of the backend subsegments to objects
>>> too.
>>>
>>> Whatever works.
>>>
>>> Regardless it'll still take awhile to download a 120GB object.
>>>
>>> Do you stream the uncompressed videos directly into the encoder as you
>>> upload the compressed format - or stage to disk and re-upload?  What about
>>> the playback of compressed videos - mp4 should http-pseudo stream directly
>>> to an html5 browser just fine!?
>>>
>>> Cool use case!  Good luck!
>>>
>>> -Clay
>>>
>>>
>>>
>>
>> ___
>> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/
>> openstack
>> Post to : openstack@lists.openstack.org
>> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/
>> openstack
>>
>>
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


signature.asc
Description: OpenPGP digital signature
__

Re: [Openstack] [OpenStack] [Swift] Can we use tempauth with version 2 or 3 of identity?

[John Dickinson]

Tempauth is really only for internal use and removing dependencies when 
testing. It does not support v2 or v3 auth APIs. I'd suggest using keystone 
instead.

--John


> On Sep 19, 2016, at 8:46 AM, Alexandr Porunov  
> wrote:
> 
> Hello,
> 
> I am using tempauth for authentication. Here what I use:
> curl -v -H 'X-Auth-User: admin:admin' -H 'X-Auth-Key: admin' 
> http://192.168.0.58:8080/auth/v1.0/
> 
> But I need to use v2 or v3 because I use a Java Client (jclouds) which can 
> use only v2 or v3 authentication. I tried:
> 
> curl -H "Content-Type: application/json" -X POST -d '{"auth" : 
> {"passwordCredentials" : {"username" : "admin","password" : 
> "admin"},"tenantName" : "admin"}}' http://192.168.0.58:8080/auth/v2.0/
> 
> unfortunately it doesn't work.
> 
> Does somebody have any ideas?
> 
> Sincerely,
> Alexandr
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [Swift] Where can I get the libshss library ?

[John Dickinson]

Unfortunately, no (as far as i know). It's a proprietary library used by NTT.

However, if you update to the latest version of liberasurecode, that warning 
message is suppressed.

--John



On 12 Oct 2016, at 19:58, Yu Watanabe wrote:

> Hello.
>
> I would like to ask question related to swift object storage node.
>
> I have set up swift object storage node following the installation guide
> and now finalizing the installation using the guide.
>
> http://docs.openstack.org/mitaka/install-guide-rdo/swift-finalize-installation.html
>
> However, I get error when I start up the services.
>
> systemctl status openstack-swift-account.service
> openstack-swift-account-auditor.service
> openstack-swift-account-reaper.service
> openstack-swift-account-replicator.service
> ● openstack-swift-account.service - OpenStack Object Storage (swift) -
> Account Server
>Loaded: loaded (/usr/lib/systemd/system/openstack-swift-account.service;
> enabled; vendor preset: disabled)
>Active: failed (Result: exit-code) since Wed 2016-10-12 19:44:45 JST;
> 15h ago
>   Process: 27793 ExecStart=/usr/bin/swift-account-server
> /etc/swift/account-server.conf (code=exited, status=1/FAILURE)
>  Main PID: 27793 (code=exited, status=1/FAILURE)
>
> Oct 12 19:44:44 opstack-objstorage1 systemd[1]: Started OpenStack Object
> Storage (swift) - Account Server.
> Oct 12 19:44:44 opstack-objstorage1 systemd[1]: Starting OpenStack Object
> Storage (swift) - Account Server...
> Oct 12 19:44:45 opstack-objstorage1 liberasurecode[27793]:
> liberasurecode_instance_create: dynamic linking error libshss.so.1: cannot
> open shared ...rectory
> Oct 12 19:44:45 opstack-objstorage1 swift-account-server[27793]: Error:
> [swift-hash]: both swift_hash_path_suffix and swift_hash_path_prefix are
> m...ft.conf
> Oct 12 19:44:45 opstack-objstorage1 systemd[1]:
> openstack-swift-account.service: main process exited, code=exited,
> status=1/FAILURE
> Oct 12 19:44:45 opstack-objstorage1 systemd[1]: Unit
> openstack-swift-account.service entered failed state.
> Oct 12 19:44:45 opstack-objstorage1 systemd[1]:
> openstack-swift-account.service failed.
>
> I have already installed libJerasure.so.2 and libisal.so.2 from source code
> but still not able to install libshss.so.1.
>
> Below is information about my environment.
>
> [root@opstack-objstorage1 system 2016.10.13 11:34:26]# cat
> /etc/redhat-release
> CentOS Linux release 7.2.1511 (Core)
> [root@opstack-objstorage1 system 2016.10.13 11:33:53]# rpm -qa | grep swift
> openstack-swift-container-2.7.0-2.el7.noarch
> openstack-swift-2.7.0-2.el7.noarch
> openstack-swift-object-2.7.0-2.el7.noarch
> openstack-swift-account-2.7.0-2.el7.noarch
>
> Is there a web site where I can get the source code for libshss?
>
> Thanks.
>
> -- 
> Yu Watanabe
> 渡辺 裕
>
> LinkedIn : jp.linkedin.com/in/yuwatanabe1


> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


signature.asc
Description: OpenPGP digital signature
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [Swift] Increase partition power?

[John Dickinson]

That method comes from 
https://github.com/openstack/swift/commit/3ff94cb785867382ff6c37cb256d1b0f5381abaa.
 The commit message should give some context. Basically, it was added as a way 
to update existing clusters that needed an emergency fix.

What you're likely interested in is https://review.openstack.org/#/c/337297/

The proposed patch is the more user-friendly way to do things, and the included 
doc updates describe the process from he end-user perspective.


--John






On 3 Nov 2016, at 15:32, Mark Kirkwood wrote:

> Hi,
>
> I stumbled across increase_partition_power() in builder.py - how close is 
> that to being useable? Also is it intended that increasing the partition 
> power will be a one-step operation (i.e existing object etc will not require 
> operator intervention to rename paths etc)?
>
> Cheers
>
> Mark
>
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


signature.asc
Description: OpenPGP digital signature
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [Swift] Rsync temp files produce error like log chatter

[John Dickinson]

On 28 Nov 2016, at 19:49, Mark Kirkwood wrote:

> I'm seeing quite o lot of this sort of thing in the object server log:
>
> Nov 29 12:59:34 cat-wgtn-ostor001 object-server: Unexpected file 
> /srv/node/obj01/objects/1485492/33d/b555a56c0d8e5cc4c146bbe08788d33d/.1479625672.77617.data.xefqO8:
>  Invalid Timestamp value in filename u'.1479625672.77617.data.xefqO8'
>
> It seems to me that this type of unexpected file is perhaps not worth logging 
> unless it is very old (i.e left by a failed rsync). Right now these look like 
> errors (and they probably are not - as I note our older ones are being 
> cleaned up).
>
>
> What so you think?


You're absolutely right.

And I have good news for you. Patch https://review.openstack.org/#/c/391658/ 
fixed this for you, and it's available as part of Swift 2.11.0 (released a 
couple of weeks ago).

--John




>
>
> regards
>
>
> Mark
>
>
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


signature.asc
Description: OpenPGP digital signature
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] {Swift] On demand replication

[John Dickinson]

I'd suggest monitoring overall replications status with a combination of log 
monitoring and swift-dispersion-report. If you find something that is 
under-replicated, you can run the replicator process and give it a list of 
partitions to prioritize. 
http://docs.openstack.org/developer/swift/admin_guide.html#dispersion-report 
I'd recommend running dispersion report with 100% coverage.

However, your question implies that when an object is created it is not durable 
in the system. That's not the case. Swift will not return with a success unless 
the data has been durably persisted in the cluster (a quorum of writes). Quorum 
is determined per storage policy; in a replicated policy, quorum is half + 1 
for odd numbers of replicas and half for even numbers of replicas. This means 
that when a client gets a 201 response to an object PUT request, that object 
has been stored at least 2 times in a 3-replica storage policy (and it's 
extraordinarily likely it was stored a full 3 times).

Let me know if you want more details about this.

--John




On 4 Dec 2016, at 14:56, Mark Kirkwood wrote:

> Suppose you have a newly created object, and want to ensure it has replicated 
> (i.e this object is 'special' in some way). If the replication process is 
> taking a while, is it sensible to 'encourage' Swift replication via direct 
> HTTP calls using the 'replicate' method (or is this dangerous)?
>
>
> E.g:
>
> markir@obj4:/srv/node/vdb/objects/5$ ls 55e
> ls: cannot access '55e': No such file or directory
>
> markir@obj1:~$ curl -v -X REPLICATE "http://obj4:6000/vdb/5/55e";
>
> markir@obj4:/srv/node/vdb/objects/5$ sleep 30;ls 55e
> 5c38bfdd63f01a8e56260105fc68555e
>
>
> regards
>
>
> Mark
>
>
>
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


signature.asc
Description: OpenPGP digital signature
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] {Swift] On demand replication

[John Dickinson]

On 5 Dec 2016, at 12:39, Mark Kirkwood wrote:

> Thanks John - increasing the partition coverage is a great idea (I hadn't 
> considered doing that).
>
>
> Now with respect to the lack of durability implication - I should have said 
> we are using a 2 region topology with (region) affinity set, so altho the 
> initial object placement will be generally durable, in the case where someone 
> was (perhaps overly) concerned about it living in both regions we were 
> looking at how to 'encourage' that to happen (ahead of the steady process 
> that the replicator is making).
>

In a multi-region setup, I really like read-affinity, and I'm not a huge fan of 
write-affinity. Read-affinity gives you a lot of what you're looking for: you 
get responses from the "closest" servers that have the data, lowering 
time-to-first-byte on reads. Write-affinity intentionally forces Swift to place 
data, durably, in handoff locations, thus creating replication work that 
happens later. Obviously, this means that if your inbound traffic to one region 
is higher than the cross-region background replication, your cluster will never 
"catch up" on that deferred work. In other words, you're intentionally 
expanding and exposing an eventual consistency window to clients that they are 
able to detect on reads. Some of this can be hidden by also having read 
affinity that matches the write affinity setting, but it's not possible to hide 
all of the behavior changes that the end-user will see.

Write-affinity can be great if you've got a bursty workload (i.e. periods of 
lower activity when the cluster catches up) or if the use case doesn't have 
much in the way of reading the data immediately after writing it. So I'm not 
totally against it. I just don't think in most cases the benefits are worth the 
cost, and often times it can lead to confusion.

However, all that being said, you're hinting at something that would be a 
pretty cool feature: policy-specific affinity. That's a great idea! Now it's 
just a matter of prioritization...


--John





>
> regards
>
>
> Mark
>
>
> On 06/12/16 05:41, John Dickinson wrote:
>> I'd suggest monitoring overall replications status with a combination of log 
>> monitoring and swift-dispersion-report. If you find something that is 
>> under-replicated, you can run the replicator process and give it a list of 
>> partitions to prioritize. 
>> http://docs.openstack.org/developer/swift/admin_guide.html#dispersion-report 
>> I'd recommend running dispersion report with 100% coverage.
>>
>> However, your question implies that when an object is created it is not 
>> durable in the system. That's not the case. Swift will not return with a 
>> success unless the data has been durably persisted in the cluster (a quorum 
>> of writes). Quorum is determined per storage policy; in a replicated policy, 
>> quorum is half + 1 for odd numbers of replicas and half for even numbers of 
>> replicas. This means that when a client gets a 201 response to an object PUT 
>> request, that object has been stored at least 2 times in a 3-replica storage 
>> policy (and it's extraordinarily likely it was stored a full 3 times).
>>
>> Let me know if you want more details about this.
>>
>> --John
>>
>>
>>
>>
>> On 4 Dec 2016, at 14:56, Mark Kirkwood wrote:
>>
>>> Suppose you have a newly created object, and want to ensure it has 
>>> replicated (i.e this object is 'special' in some way). If the replication 
>>> process is taking a while, is it sensible to 'encourage' Swift replication 
>>> via direct HTTP calls using the 'replicate' method (or is this dangerous)?
>>>
>>>
>>> E.g:
>>>
>>> markir@obj4:/srv/node/vdb/objects/5$ ls 55e
>>> ls: cannot access '55e': No such file or directory
>>>
>>> markir@obj1:~$ curl -v -X REPLICATE "http://obj4:6000/vdb/5/55e";
>>>
>>> markir@obj4:/srv/node/vdb/objects/5$ sleep 30;ls 55e
>>> 5c38bfdd63f01a8e56260105fc68555e
>>>
>>>
>>> regards
>>>
>>>
>>> Mark
>>>
>>>
>>>
>>> ___
>>> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>> Post to : openstack@lists.openstack.org
>>> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


signature.asc
Description: OpenPGP digital signature
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] {Swift] On demand replication

[John Dickinson]

On 5 Dec 2016, at 12:56, John Dickinson wrote:

> On 5 Dec 2016, at 12:39, Mark Kirkwood wrote:
>
>> Thanks John - increasing the partition coverage is a great idea (I hadn't 
>> considered doing that).
>>
>>
>> Now with respect to the lack of durability implication - I should have said 
>> we are using a 2 region topology with (region) affinity set, so altho the 
>> initial object placement will be generally durable, in the case where 
>> someone was (perhaps overly) concerned about it living in both regions we 
>> were looking at how to 'encourage' that to happen (ahead of the steady 
>> process that the replicator is making).
>>
>
> In a multi-region setup, I really like read-affinity, and I'm not a huge fan 
> of write-affinity. Read-affinity gives you a lot of what you're looking for: 
> you get responses from the "closest" servers that have the data, lowering 
> time-to-first-byte on reads. Write-affinity intentionally forces Swift to 
> place data, durably, in handoff locations, thus creating replication work 
> that happens later. Obviously, this means that if your inbound traffic to one 
> region is higher than the cross-region background replication, your cluster 
> will never "catch up" on that deferred work. In other words, you're 
> intentionally expanding and exposing an eventual consistency window to 
> clients that they are able to detect on reads. Some of this can be hidden by 
> also having read affinity that matches the write affinity setting, but it's 
> not possible to hide all of the behavior changes that the end-user will see.
>
> Write-affinity can be great if you've got a bursty workload (i.e. periods of 
> lower activity when the cluster catches up) or if the use case doesn't have 
> much in the way of reading the data immediately after writing it. So I'm not 
> totally against it. I just don't think in most cases the benefits are worth 
> the cost, and often times it can lead to confusion.
>
> However, all that being said, you're hinting at something that would be a 
> pretty cool feature: policy-specific affinity. That's a great idea! Now it's 
> just a matter of prioritization...


Actually, it has been written. Just needs to be reviewed/landed.

https://review.openstack.org/#/c/382766/


>
>
> --John
>
>
>
>
>
>>
>> regards
>>
>>
>> Mark
>>
>>
>> On 06/12/16 05:41, John Dickinson wrote:
>>> I'd suggest monitoring overall replications status with a combination of 
>>> log monitoring and swift-dispersion-report. If you find something that is 
>>> under-replicated, you can run the replicator process and give it a list of 
>>> partitions to prioritize. 
>>> http://docs.openstack.org/developer/swift/admin_guide.html#dispersion-report
>>>  I'd recommend running dispersion report with 100% coverage.
>>>
>>> However, your question implies that when an object is created it is not 
>>> durable in the system. That's not the case. Swift will not return with a 
>>> success unless the data has been durably persisted in the cluster (a quorum 
>>> of writes). Quorum is determined per storage policy; in a replicated 
>>> policy, quorum is half + 1 for odd numbers of replicas and half for even 
>>> numbers of replicas. This means that when a client gets a 201 response to 
>>> an object PUT request, that object has been stored at least 2 times in a 
>>> 3-replica storage policy (and it's extraordinarily likely it was stored a 
>>> full 3 times).
>>>
>>> Let me know if you want more details about this.
>>>
>>> --John
>>>
>>>
>>>
>>>
>>> On 4 Dec 2016, at 14:56, Mark Kirkwood wrote:
>>>
>>>> Suppose you have a newly created object, and want to ensure it has 
>>>> replicated (i.e this object is 'special' in some way). If the replication 
>>>> process is taking a while, is it sensible to 'encourage' Swift replication 
>>>> via direct HTTP calls using the 'replicate' method (or is this dangerous)?
>>>>
>>>>
>>>> E.g:
>>>>
>>>> markir@obj4:/srv/node/vdb/objects/5$ ls 55e
>>>> ls: cannot access '55e': No such file or directory
>>>>
>>>> markir@obj1:~$ curl -v -X REPLICATE "http://obj4:6000/vdb/5/55e";
>>>>
>>>> markir@obj4:/srv/node/vdb/objects/5$ sleep 30;ls 55e
>>>> 5c38bfdd63f01a8e56260105fc68555e
>>>>
>

Re: [Openstack] [Swift] Object replication...

[John Dickinson]

On 13 Dec 2016, at 0:21, Shyam Prasad N wrote:

> Hi,
>
> I have an openstack swift cluster with 2 nodes, and a replication count of
> 2.
> So, theoretically, during a PUT request, both replicas are updated
> synchronously. Only then the request will return a success. Please correct
> me if I'm wrong on this.
>
> I have a script that periodically does a PUT to a small object with some
> random data, and then immediately GETs the object. On some occasions, I'm
> getting older data during the GET.
>
> Is my expectation above correct? Or is there some other setting needed to
> make the replication synchronous?


This is an interesting case of both Swift and your expectations being correct. 
But wait! How can that be when they seem to be at odds? Therein lies the fun* 
of Distributed Systems. Yay.

(*actually, not that much fun)

Ok, here's how it works. I'm assuming you have more than one hard drive on each 
of your two servers. When Swift gets the PUT request, the proxy will determine 
where the object data is supposed to be in the cluster. It does this via 
hashing and ring lookups (this is deterministic, but the details of that 
process aren't important here). The proxy will look for  places 
to put the data. In your case, this is 2. Because of the way the ring works, it 
will look for one drive on each of your two servers first. It will not put the 
data on two drives on one server. So in the Happy Path, the client makes a PUT 
request, the proxy sends the data to both replicas, and after both have been 
fsync'd, the client gets a 201 Created response. [1]

This is well and good, and the greatest part about it is that Swift can 
guarantee read-your-creates. That is, when you create a new object, you are 
immediately able to read it. However, what you describe is slightly different. 
You're overwriting an existing object, and sometimes you're getting back the 
older version of the object on a subsequent read. This is normal and expected. 
Read on for why.

The above process is the Happy Path for when there are no failures in the 
system. A failure could be a hardware failure, but it could also be some part 
of the system being overloaded. Spinning drives have very real physical limits 
to the amount of data they can read and write per unit time. An overloaded hard 
drive can cause a read or write request to time out, thus becoming a "failure" 
in the cluster.

So when you overwrite an object in Swift, the exact same process happens: the 
proxy finds the right locations, sends the data to all those locations, and 
returns a success if a quorum successfully fsync'd the data to disk.

However, what happens if there's a failure?

When the proxy determines the correct location for the object, it chooses what 
we call "primary" nodes. These are the canonical locations where the data is 
supposed to be right now. All the other drives in the cluster are called 
"handoff" nodes. For a given object, some nodes ( of them, to be 
exact) are primary nodes, and all the rest in the cluster are handoffs. For 
another object, a different set of nodes will be primary, and all the rest in 
the cluster are handoffs. This is the same regardless of how many replicas 
you're using or how many drives you have in the cluster.

So when there's a failure in the cluster and a write request comes in, what 
happens? Again, the proxy finds the primary nodes for the object and it tries 
to connect to them. However, if one (or more) can't be connected to, then the 
proxy will start trying to connect to handoff nodes. After the proxy gets 
 successful connections, it sends the data to those storage 
nodes, the data is fsync'd, and the client gets a successful response code 
(assuming at least a quorum were able to be fsync'd). Note that in your case 
with two replicas, if the primary nodes were extra busy (e.g. serving other 
requests) or actually failing (drives do that, pretty often, in fact), then the 
proxy will choose a handoff location to write the data. This means that even 
when the cluster has issues, your writes are still completely durably 
written.[2]

The read request path is very similar: primary nodes are chosen, one is 
selected at random, if the data is there, it's returned. If the data isn't 
there, the next primary is chosen, etc etc.

Ok, we're finally able to get down to answering your question.

Let's assume you have a busy drive in the cluster. You (over)write your object, 
the proxy looks up the primary nodes, sees that one is busy (i.e. gets a 
timeout), chooses a handoff location, writes the data, and you get a 201 
response. Since this is an overwrite, you've got an old version of the object 
on one primary, a new version on another primary, and a new version on a 
handoff node. Now you do the immediate GET. The proxy finds the primary nodes, 
randomly chooses one, and oh no! it chose the one with the old data. Since 
there's data there, that version of the object gets returned to the client, and 
you see the o

Re: [Openstack] [Swift] : Internal Working of PUT Command

[John Dickinson]

On 4 Jan 2017, at 20:44, Sameer Kulkarni wrote:

> Hi All,
> I was eager to know the data-flow of an object on Openstack Swift PUT
> command. I know a highlevel overview from various posts, which I am not
> sure of.
>
>- Client Initiates PUT command specifying the object path on local
>storage to that on swift cloud.
>- The Object is transferred to proxy-server over HTTP request.
>- If (REPLICAS = 3), then three primary nodes are found out using Ring
>Algorithm.
>- Then the object is transferred to these three primary nodes parallely
>from proxy server.
>- Then after majority successful ACK (two here), the client is sent back
>ACK.
>
> I will be happy if someone can confirm the above sequence of steps is
> correct.

Yep, that's a pretty good summary.

> My follow questions are
>
>- What happens when there is NO ACK from the 3rd node?

The client will still get a success (assuming the other two completed just 
fine).

As you said, the data is sent to all replicas at the same time. After an object 
server has fsync()'d the data to disk, it can return success to the proxy. The 
proxy collects these responses and determines what the proper response to the 
client is. Let's suppose that a three-replica system has two normal storage 
nodes and one that is very slow. The proxy sends chunks of the object to each 
server concurrently (default of 64k bytes) as the data is read from the client. 
Each of these has a timeout, so a failing object server may error out. In this 
case, if there are still enough active nodes to give a quorum, then they will 
continue to be sent data and the failing node will no longer get the data. This 
handles the case of "what happens when hardware fails during a write request". 
However, what if the third is just slow, but never quite times out? In that 
case, once the proxy gets a successful response from a quorum of replicas, but 
the third one hasn't responded yet, we know that the client will get a 
successful response, no matter what--we already have a quorum of success. The 
proxy starts a new (much shorter) timer to give the third replica a chance to 
finish, and when the timer fires or the third replica responds (whichever is 
first), the client gets the response.



>- How is rsync used, when there is node failure?

And the above gets us to the "so what about that third node?

The object replication process is running in the background. When it detects 
that a replica is missing from where it should be, it triggers rsync to copy 
the data over. (note, this explanation is greatly simplified)

So, in this way, when a third node fails during a write, the replication 
process will ensure that the correct data is in the correct place, but this is 
not done as part of the client-facing data path.

>
>
> Thank you
> Sameer Kulkarni


> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


signature.asc
Description: OpenPGP digital signature
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [Swift] Dispersion populate - run how frequently?

[John Dickinson]

If you're running 100% coverage with dispersion report, then running it once 
per day seems reasonable.

If you're running something smaller, like 1-10%, then doing it once per hour 
might be reasonable.

The point is, make it automatic and integrate it into your normal ops metrics. 
Track it over time and adjust as needed.

--John




On 12 Jan 2017, at 18:06, Mark Kirkwood wrote:

> We are running a 2 region Swift cluster with write affinity.
>
> I've just managed to deceive myself with the Dispersion report :-( . The last 
> run of the populate was Early Dec, and the corresponding report happily shows 
> 100%. All good - seemingly.
>
> However probing the actual distribution of a number of objects created later 
> in Dec I see they have 3 copies in region 1 and 0 in regions 2. Hmm... I'm 
> guessing that if we ran the population more frequently this would have been 
> highlighted sooner. This begs the question - how frequently is it sensible to 
> run the populate?
>
> regards
>
> Mark
>
>
>
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


signature.asc
Description: OpenPGP digital signature
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [Swift] : Ring Algorithm

[John Dickinson]

The ring is the beating heart of Swift. It's what defines cluster membership 
and ensures disperse and balanced placement. Therefore, understanding the ring 
means you understand a whole lot of how Swift works.

Good luck on the journey!

There's been quite a few words written and spoken about the ring over the 
years, and, in fact, we've made significant improvements as well. Although the 
basic concept hasn't changed, we've gotten better at doing the right thing at 
the right time. That is to say, if you find stuff online about how the ring 
works, even if it's old info, it's likely still accurate in general.

Here's some further things to look at:

https://www.openstack.org/videos/video/forget-everything-you-knew-about-swift-rings-heres-everything-you-need-to-know-about-swift-rings

https://www.swiftstack.com/blog/2012/11/21/how-the-ring-works-in-openstack-swift/

http://docs.openstack.org/developer/swift/ring_background.html

http://docs.openstack.org/developer/swift/overview_ring.html

http://www.florentflament.com/blog/openstack-swift-ring-made-understandable.html

https://www.openstack.org/blog/2012/02/1997/

Hope these help. Please ask if you have more questions, or drop by 
#openstack-swift.

--John





On 9 Feb 2017, at 23:19, Mark Kirkwood wrote:

> Hi Sameer,
>
> A good question - the docs do mention a 'far apart as possible' distribution 
> algorithm. If you study the code (in swift/common/ring*.py) there is a 
> discussion of tiers (zone and region) which relate to this concept. Now I've 
> not studied this in enough detail myself (yet) to give you a good/fast 
> summary (maybe some others can) but this is the area to look at if you want 
> do understand how it works!
>
>
> Cheers
>
>
> Mark
>
>
> On 10/02/17 19:34, Sameer Kulkarni wrote:
>> Hi All,
>>
>> I was trying to understand the Ring Algorithm for Openstack Swift. I have 
>> went through official docs. I found that it uses Consistent Hashing Ring[1].
>>
>> But the problem is, It doesn't mention about the features of Swift like 
>> Given 3 regions, Each file should go to different regions etc.
>>
>> If I have 8 nodes with equal weights, then a couple of nodes are always 
>> empty even after inserting 20 files [with each of 3 replicas]. I feel that 
>> this doesn't happen in Swift Algorithm, But only in Consistent Hashing Ring.
>> *
>> *
>> *So, what other things Swift Ring Algorithm have apart from Consistent 
>> Hashing Ring?*
>>
>> Thank you
>>
>> Sameer Kulkarni
>>
>
>
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


signature.asc
Description: OpenPGP digital signature
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] Replication logic and use of partitions

[John Dickinson]

On 13 Feb 2017, at 17:16, Sai Vishwas wrote:

> Hi,
>
>  I was reading about the replicator process that runs at regular
> intervals. I would like to know if my understanding of it is correct .

Yep, you've got a good understanding.

>
>  This is my understanding :  Since it is partitions that are
> replicated, the replicator process uses the hash of the partition's
> contents to check whether the replicas are all up to date, i.e all changes
> have been updated on all three replica partitions. In case, an operation
> has failed on one node, then the partition on that node will have an older
> 'last updated timestamp' and also its hash will differ from that of the
> partitions on which the operation succeeded. Once, the replicator sees that
> the 2 partitions have a different hash value, it realizes that the one with
> older timestamp has to be updated and hence checks as to which all files
> have to be updated and then rsyncs only those particular files.
>
>Is this right, or does it rsync the entire partition?

Both, sortof. Swift will start an rsync of the partition suffix (up to 4096 of 
these per partition), but because of the magic of rsync, only the changed files 
will be transferred.

>
>Apart from generating this hash value for easy change detection, and
> assigning weights to nodes, for what other purposes does the concept of
> 'swift partitions' come into help ?

The partitions are primarily used for data balance across all of the available 
storage devices and distribution across different failure domains.

Read more at http://docs.openstack.org/developer/swift/overview_ring.html

>
>
> Thank you. Have a great day.
>
> Regards,
> Sai Vishwas
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


signature.asc
Description: OpenPGP digital signature
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [Swift] : Work-flow

[John Dickinson]

Great question. There's not a simple yes/no answer, so let me go into some 
detail about the different ways storage nodes can be selected to handle a 
request.

There are a few config settings in the proxy server config that can affect how 
nodes are selected for reads. Instead of describing these directly (or pasting 
the docs), let me describe it from an implementation perspective.

When the proxy server gets a read request for an object, the proxy looks up in 
the ring the storage nodes (object servers) that may know something about that 
object. The proxy server builds two lists[1]. The first is for "primary" nodes. 
These are the drives where the data is supposed to be. For a replicated storage 
policy with three replicas, the primary nodes will be a list of three items[2]. 
For an 8+4 erasure coded storage policy, it will be the list of 12 nodes where 
the EC fragments are supposed to be. The second list the proxy makes is the 
list of "handoff" nodes. These are alternative places an object (or fragment) 
may be found if they aren't found on a primary node.

Once the proxy has the list of primary nodes, there are a few ways it can 
iterate over that list. The `sorting_method` config option determines this. The 
default `shuffle` value means that the list of fragments is randomly sorted. 
When a proxy server makes a connection to a storage node, it tracks how long it 
took to create the connection. The `timing` value of `sorting_method` will sort 
the list by these saved connection timings. The idea is that a busy server will 
take longer to respond to connection requests, and will then get moved lower in 
the sorted list. The `affinity` value will cause the list of nodes to be sorted 
according to the rules set in the `read_affinity` config option. This allows a 
proxy server to specifically prioritize connections that are local (same DC) 
and de-prioritize remote connections. The `read_affinity` setting is fantastic 
when Swift is deployed with more than one region (i.e. global clusters).

Once the list of primary nodes is sorted, the the proxy will start contacting 
each in turn until the request can be satisfied. With erasure codes, at least 
 fragments need to be contacted (e.g. 8 in the example 
above), so the sorting method value doesn't do much to change performance. For 
replicas, though, only one node is needed to satisfy a read request. The naive 
way to go through the list is: contact the first one, get response, if error, 
repeat with the next node in the list.

However, just walking through the list can be really slow. Swift has another 
config option, `concurrency_timeout` (default to 0.5 seconds) that is a delay 
before the next request is attempted. Basically, it's a balance between network 
connections created and end-user latency. Let's say the `node_timeout` is set 
to 5 seconds. If a server can accept a connection with no problem but a disk is 
slow, this means that the proxy might start a read request but wait five 
seconds before timing out and moving on. Worst case, this could result in a 10 
second delay before the last primary node is even asked if it has the data 
(first two time out after 5 seconds each). The `concurrency_timeout` means that 
the proxy will only wait 500ms before starting that next connection in the 
primary node list. Whichever node responds first will be the one that is used 
to send data to the client, and the rest are closed and cleaned up.

That's an overview of how the proxy chooses which nodes to talk to when 
handling a read request. There's a few different options that can be tuned 
depending on your particular deployment, but the default values (shuffle, 500ms 
concurrency timeout) are really good for most cases.

As a final note, there's also a `write_affinity` setting for the write data 
path. This works very similar to the `read_affinity` setting, but I'm not a big 
fan of it. It seems to cause more problems that not. It causes the proxy server 
to mix in some local handoff nodes into the primary node list on the write. 
This means that all writes in a global cluster will be satisfied in the local 
DC, but it doesn't mean the WAN traversal work goes away. Swift's background 
consistency process will move the data to the right place, but this is more 
expensive than putting it in the right place to start with. I strongly 
recommend that you do not use `write_affinity` in your global Swift clusters.


[1] technically, a list and a lazy-eval'd iterator
[2] I sometimes switch between "node" and "drive" and "server". Each element in 
these lists has (IP, port, mount point) for finding a particular drive.


Hope this info helps you understand more about how Swift works and how you can 
best tune it for your use.


--John





On 28 Apr 2017, at 2:46, Sameer Kulkarni wrote:

> Hi All,
>
> I had a doubt regarding the work-flow of Swift.
>
> *'For read operation, we need to read from one of the three replicas. We
> are aware that geographical origin of 

Re: [Openstack] Unable to upload a file into OpenStack Swift because of "[Errno 13] Permission denied"

[John Dickinson]

On 24 May 2017, at 5:47, Christian Baun wrote:

> Hello all,
>
> I tried again to install Swift only ontop of a Raspberry Pi 3 with

So, first off, I think that's really cool. I tried something like this a while 
back, too. https://github.com/notmyname/swift_on_pi

However, you can see that since it was last updated in early 2013, it's not 
been something I've spend a lot of time maintaining ;-)

> Rapbian and I followed this time the instructions "SAIO - Swift All In
> One"
>
> https://docs.openstack.org/developer/swift/development_saio.html
>
> The installation went well and the service is running, but I still
> cannot upload files into the service. The error message is still this
> one on client side:
>
> Object PUT failed:
> http://10.0.0.253:8080/v1/AUTH_test/s3perf-testbucket/testfiles/s3perf-testfile1.txt
> 503 Service Unavailable  [first 60 chars of response]
> Service UnavailableThe server is currently
>
> and this one inside the file storage1.error:
>
> May 24 13:15:15 raspberrypi object-server: ERROR __call__ error with PUT
> /sdb1/484/AUTH_test/s3perf-testbucket/testfiles/s3perf-testfile1.txt :
> #012Traceback (most recent call last):#012  File
> "/home/pi/swift/swift/obj/server.py", line 1105, in __call__#012res =
> getattr(self, req.method)(req)#012  File
> "/home/pi/swift/swift/common/utils.py", line 1626, in _timing_stats#012
> resp = func(ctrl, *args, **kwargs)#012  File
> "/home/pi/swift/swift/obj/server.py", line 814, in PUT#012
> writer.put(metadata)#012  File "/home/pi/swift/swift/obj/diskfile.py",
> line 2561, in put#012super(DiskFileWriter, self)._put(metadata,
> True)#012  File "/home/pi/swift/swift/obj/diskfile.py", line 1566, in
>  _put#012tpool_reraise(self._finalize_put, metadata, target_path,
> cleanup)#012  File "/home/pi/swift/swift/common/utils.py", line 3536, in
> tpool_reraise#012raise resp#012IOError: [Errno 13] Permission denied
> (txn: txdfe3c7f704be4af8817b3-0059256b43)
>
> I tried the installation instructions inside virtual machine with
> Ubuntu 14.04 and there is no problem. The service runs and I can
> upload files without any error.
>
> Has anybody here an idea how to locate the root cause of this issue
> and how to fix it?

You've done some good first steps. To state the obvious, there's some 
permissions problems somewhere. Probably something to do with the way the drive 
is mounted. If you're running it as the "swift" user, make sure the mounted 
drive is also owned by the swift user.

>
> Best Regards
>Christian
>
> -- Forwarded message --
> From: Christian Baun 
> Date: 2017-05-10 20:03 GMT+02:00
> Subject: Unable to upload a file into OpenStack Swift because of
> "[Errno 13] Permission denied"
> To: openstack@lists.openstack.org
>
>
> Hello,
>
> Creating and erasing buckets (containers) in my single node Swift only
> installation works well, but any attempt to upload a file causes just
> an error message.
>
> $ tail -f /var/log/swift/storage1.error
> ...
> May  9 14:03:10 raspberrypi object-server: ERROR __call__ error with
> PUT /d1/237/AUTH_test/s3perf/s3perf-testfile1.txt : #012Traceback
> (most recent call last):#012  File
> "/home/pi/swift/swift/obj/server.py", line 1105, in __call__#012
> res = getattr(self, req.method)(req)#012  File
> "/home/pi/swift/swift/common/utils.py", line 1626, in
> _timing_stats#012resp = func(ctrl, *args, **kwargs)#012  File
> "/home/pi/swift/swift/obj/server.py", line 814, in PUT#012
> writer.put(metadata)#012  File "/home/pi/swift/swift/obj/diskfile.py",
> line 2561, in put#012super(DiskFileWriter, self)._put(metadata,
> True)#012  File "/home/pi/swift/swift/obj/diskfile.py", line 1566, in
> _put#012tpool_reraise(self._finalize_put, metadata, target_path,
> cleanup)#012  File "/home/pi/swift/swift/common/utils.py", line 3536,
> in tpool_reraise#012raise resp#012IOError: [Errno 13] Permission
> denied (txn: txb986cc22c54f426c8289b-005911affd)
>
> I have started Swift as user pi and assigned these folders to this user:
>
> $ sudo chown pi:pi /etc/swift
> $ sudo chown -R pi:pi /mnt/sdb1/*
> $ sudo chown -R pi:pi /var/cache/swift
> $ sudo chown -R pi:pi /var/run/swift
>
> sdb1 is a loopback device with the XFS file system.
>
> I deployed Swift this way:
> https://github.com/christianbaun/s3perf/wiki/Swift-on-a-Raspberry-Pi-3-with-Raspbian-%28Debian-Jessie-8.0%29
>
> I wonder why creating buckets (containers) works well, but the upload
> of a file fails because of "Permission denied".
>
> $ sudo swift-ring-builder /etc/swift/account.builder
> /etc/swift/account.builder, build version 2
> 256 partitions, 1.00 replicas, 1 regions, 1 zones, 1 devices, 0.00
> balance, 0.00 dispersion
> The minimum number of hours before a partition can be reassigned is 0
> (0:00:00 remaining)
> The overload factor is 0.00% (0.00)
> Ring file /etc/swift/account.ring.gz is up-to-date
> Devices:   id region zone ip address:port replication ip:port  name
> weight partitions balance flags met