Re: [Openstack] Download file from swift extremely slow

[Christian Schwede]

Hello Steve,

On 09.01.2017 23:27, Steve Martinelli wrote:
> Thanks for the hint Clay, we merged
> https://review.openstack.org/#/c/417892/ - expect this issue to be
> fixed in OpenStackClient 3.7.0.

unfortunately the fix didn't help - the issue is still there. I left a
comment in the bugreport, will continue to investigate.

-- Christian


> 
> On Fri, Jan 6, 2017 at 2:03 PM, Clay Gerrard  > wrote:
> 
> Is this *really* the default chunk_size?
> 
> 
> http://docs.python-requests.org/en/master/api/#requests.Response.iter_content
> 
> 
> 
> Because, that'd be like a *lot* of read calls for a large object ;)
> 
> 
> https://github.com/openstack/python-openstackclient/blob/master/openstackclient/api/object_store_v1.py#L383
> 
> 
> 
> Maybe try like, idk, chunk_size=None or 64 * 2 ** 10?
> 
> -Clay
> 
> On Fri, Jan 6, 2017 at 9:39 AM, Rick Jones  > wrote:
> 
> On 01/05/2017 11:52 PM, don...@ahope.com.cn
>  wrote:
> 
> *Also, i can put/get files via dashboard/swift-CLI very
> quickly.*
> *So it is strange why 'openstack object save' so slowly...*
> 
> 
> Well, two additional ways to compare might be to run both the
> openstack and swift CLI versions under an strace and compare the
> system calls they are making:
> 
> strace -v -f -ttt -o  cli command ...
> 
> where -v tells strace to be verbose, the -ttt tells it to
> timestamp each system call with seconds.microseconds since the
> epoch, the -f tells it to follow forks and the -o option gives a
> filename into which the trace should go.
> 
> The second comparison would be to take a packet trace for each
> using tcpdump.  So, on the client something like:
> 
> tcpdump -s 96 -w .pcap -i  "port  port> and host "
> 
> The -s says to capture no more than 96 bytes per packet, the -w
> puts the capture to the named file, the -i selects the network
> interface on the client, and then the last bit is a filter
> expression to select only those packets which are swift and
> to/from the proxy.
> 
> That file can be post-processed in a number of ways, one of
> which is:
> 
> tcpdump -r .pcap -n -ttt > .cooked
> 
> where -r selects the file from which to read captured packets,
> the -n says to disable looking up hostnames for IP addresses,
> and the -ttt says to print the time delta for each packet
> compared to the one before.  I happen to follow a convention of
> calling the resulting output a ".cooked" file - as in it is a
> cooked version of a raw (binary) capture.
> 
> In both cases, you would be looking for large gaps in time -
> particularly the openstack cli traces.
> 
> happy benchmarking,
> 
> rick jones
> 
> 
> 
> ___
> Mailing list:
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> 
> Post to : openstack@lists.openstack.org
> 
> Unsubscribe :
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> 
> 
> 
> 
> ___
> Mailing list:
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> 
> Post to : openstack@lists.openstack.org
> 
> Unsubscribe :
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> 
> 
> 
> 
> 
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> 


___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] Ring rebuild, multiple copies of ringbuilder file, wasRe: swift ringbuilder and disk size/capacity relationship

[Christian Schwede]

On 26.04.16 17:31, Mark Kirkwood wrote:
> Yeah, a builtin method to distribute rings would be nice.

Well, you need some more information - for example the proxy nodes, that
are not included in the rings, but need a copy too. So we would need to
provide some additional data, and IMO then we quickly start to reinvent
the wheel to distribute a few files.

> The puppet module (puppet-swift) does do this. I'm guessing most
> people are going to use some configuration management tool (Pupper,
> Ansible etc) to manage their deployment (especially if installing the
> rest of Openstack too).

Indeed. Ansible provides a very simple way to copy ring files to all
your proxy/storage nodes even without writing any playbook at all.
Just create an inventory file listing all your proxy & storage nodes, a
directory "rings" containing ONLY the ring files (*.ring.gz and maybe
*.builder as well), and then use something like this:

ansible -a "src=rings/ dest=/etc/swift" -m copy -i inventory all

That takes care of distributing your files to all nodes (you might need
a few more options for authentication, but that's basically it).

-- Christian

-- 
Christian Schwede
Principal Software Engineer
Production Chain Tool Team
_

Red Hat GmbH
Technopark II, Haus C, Werner-von-Siemens-Ring 11-15, 85630 Grasbrunn,
Handelsregister: Amtsgericht Muenchen HRB 153243
Geschaeftsfuehrer: Mark Hegarty, Charlie Peters, Michael Cunningham,
Charles Cachera

___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [Swift] Syncing ring but not builder files?

[Christian Schwede]

Hi Mark,

On 09.06.15 10:30, Mark Kirkwood wrote:
> I note that the documentation (e.g
> http://docs.openstack.org/juno/install-guide/install/apt/content/swift-initial-rings-distribute.html)
> suggests copying the ring files:
> 
> - account.ring.gz
> - container.ring.gz
> - object.ring.gz
> 
> to the various other nodes.
> 
> Now this means that the [account|container|object].builder files only
> exist on the original/"master" proxy. I'm wondering if these should be
> copied around too (maybe to all additional proxies), as it seems like
> you need these to make changes to the existing ring (e.g
> replacing/moving that failed original proxy...).

it doesn't matter where you keep your .builder files - but keep them
somewhere safe, otherwise you might get into troubles if you want to
change your ring later and lost them. So yes, copying them to the other
proxis is an option. Just make sure you use a single set of .builder
files that you're changing.

If you lost your .builder files you can still recover with the command

swift-ring-builder  write_builder [min_part_hours]

However, replication times might be much higher in the beginning, and if
you're unlucky some objects are not even accessible for a while. So keep
a copy of the .builder files, and use this command only as a last resort.

-- Christian


___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] Slowness of Swift

[Christian Schwede]

Hello,

On 08.05.15 12:15, Tarak Mukherjee wrote:
> We have an use-case where we upload large object (2GB - 4GB) via form
> post. What we are observing is , it's taking 2 - 3hrs to get the
> upload completed .
> 
> Our Setup is consist of Haproxy => Swift-Proxy => Storage Node .

any chance to test an upload without the haproxy, just uploading to the
swift proxy itself? Also, is there a HTTP proxy somewhere involved, that
the client is maybe using?

In the past I saw some real slow uploads too, but that was always
because either a http proxy was involved or some other network devices
tried to cache data (loadbalancers, proxies, ...).

-- Christian

___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] Savana/Swift large object copy error

[Christian Schwede]

Hello Ross,

On 05.05.15 21:54, Ross Lillie wrote:
> My understanding is that Swift should automagically split files greater
> that 5G into multiple segments grouped under a metafile but this appears
> to not be working. This was working under the Havana release (Ubuntu)
> using the Swift File System jar file downloaded from the Marantis web
> site.  All current testing is based up the Juno release and when
> performing a distcp using the openstack-hadoop jar file shipped as part
> of the latest hadoop distros.

I don't know the client you're using, but Swift itself (on the server
side) never splits data into segments on its own, and never did so.
Currently it's up to the client to ensure data is broken down into
segments of max. 5GB size.

There were some ideas in the past to implement this feature, however
this raised different problems. Have a look at the history of large
objects:

http://docs.openstack.org/developer/swift/overview_large_objects.html#history

I would assume that there was a change on the client side implementation
that changed the behavior for you?

Best Regards,

Christian

___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [Swift] tmp directory causing Swift slowdown

[Christian Schwede]

On 01.05.15 20:33, Samuel Merritt wrote:
> On 5/1/15 7:55 AM, Uwe Sauter wrote:
>>
>>
>> Am 01.05.2015 um 02:21 schrieb Samuel Merritt:
>>>
>>> It seems like 1430268763.41931.data would be in the same allocation
>>> group as
>>> objects/757/a94/bd77129a1cae9e32381776e322efca94, and
>>> bd77129a1cae9e32381776e322efca94 would be in the same allocation
>>> group as objects/757/a94, and so on. Thus, everything would be in the
>>> same allocation group as the root directory.
>>>
>>> This can't be the case, or else there'd be no point to allocation
>>> groups. What am I missing here?
>>
>>
>> Hi,
>>
>> I think what you're missing is, that inodes stay in the allocation
>> group where they first were created. So moving a file
>> around in the filesystem changes the path but not the allocation
>> group. So first creating a temporary file and then
>> moving it into the hash folder leaves the file associated with the
>> temp folder's allocation group, thus the allocation
>> group grows bigger and bigger and searching the allocation group takes
>> more and more time.
> 
> That doesn't really answer the question, though. We have this message
>  which says that "...the
> locality of a new inode is determined by the
> parent inode, and so if all new inodes are created in the same
> directory, then they are all created in the same AG."
> 
> Let's say we start out with a freshly-formatted disk, so there's only
> one inode, and it's for the root directory.
> 
> Then, Swift goes and starts making its directory structure on disk, and
> calls mkdir('objects'). Since a new inode is created in the same AG as
> its parent, the inode for '/objects' is in the same AG as the inode for
> '/'.
> 
> Swift makes another dir: mkdir('objects/757')
> 
> The inode for '/objects/757' is in the same AG as its parent '/objects',
> which is the same as the AG for '/'.
> 
> Keep going a while, and you get
> 
> /
> /objects
> /objects/757
> /objects/757/a94
> /objects/757/a94/bd77129a1cae9e32381776e322efca94
> /objects/757/a94/bd77129a1cae9e32381776e322efca94/1430268763.41931.data
> /tmp
> 
> and they're all in the same AG.
> 
> Now, the XFS developers are not stupid, so what I typed up there can't
> possibly be true, or else every inode on a filesystem would be in the
> same AG.
> 
> So, my question is this: what, of the things I typed above, is false?
> Equivalently, how is an inode created in a *different* AG than its parent?

Hmm, reading the docs at

http://xfs.org/docs/xfsdocs-xml-dev/XFS_Filesystem_Structure/tmp/en-US/html/AG_Free_Space_Management.html

I would assume a different AG is selected if there is one AG with more
free space.

But, and that might be one of the problems here: it seems there is a
default of only 4 allocation groups, at least that's what I see on
various disks executing a xfs_info.

In fact after looking into the sources of mkfs.xfs I found this default
for disks with sizes up to 4TB:

http://oss.sgi.com/cgi-bin/gitweb.cgi?p=xfs/cmds/xfsprogs.git;a=blob;f=mkfs/xfs_mkfs.c;h=5084d755;hb=HEAD#l688

Might be a good idea to do some benchmarking with different AG numbers?

-- Christian

___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] Trying compiling the OpenStack Swift

[Christian Schwede]

Hello,

actually there is a small gap in the docs you stumbled upon :(
The Swift developers are currently working on integrating Erasure
Coding, and thus a new dependency was added. The dependency itself
requires a library that is not available on your system.

Please head over to

http://www.bytebucket.org/tsg-/liberasurecode/downloads

and download & install that library. Afterwards the installation of
Swift should succeed.

I'll submit a separate patch for Swift.

Best Regards

Christian

On 30.03.15 13:40, Leonid Rozenblyum wrote:
> Hello everybody!
> 
> 
> I'm trying to go through the
> http://docs.openstack.org/developer/swift/development_saio.html#loopback-section
> to create a development Swift installation.
> My OS on the VM is CentOS 7
> 
> It fails on step:
> 
> Build a development installation of swift:
> 
> cd $HOME/swift; sudo python setup.py develop; cd -
> 
> The error is :
> Writing /tmp/easy_install-4S0U5S/PyECLib-1.0.6/setup.cfg
> Running PyECLib-1.0.6/setup.py -q bdist_egg --dist-dir
> /tmp/easy_install-4S0U5S/
> 
>   PyECLib-1.0.6/egg-dist-tmp-j3Cljs
> src/c/pyeclib_c/pyeclib_c.c:32:25: fatal error: erasurecode.h: No such
> file or d
> 
> irectory
>  #include 
>  ^
> compilation terminated.
> error: Setup script exited with error: command 'gcc' failed with exit status 1
> 
> 
> Is it something wrong with my configuration or with the souce code of
> Swift itself?
> 
> Thanks for any help
> 
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> 


___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [Swift] Browser-based Health Visualization

[Christian Schwede]

Hello Adam,

Swift is able to send metrics to a statsd server, and thus you can
create your own metric dashboard using graphite-web. I would add
collectd for some basic server metrics (like cpu, memory, load, network
traffic) as well. It requires a little bit of work, but it gives you a
very comprehensive metric overview.

Metrics sent by Swift:

http://docs.openstack.org/developer/swift/admin_guide.html#reporting-metrics-to-statsd

Required software (graphite-web, collectd, statsd or python-bucky):

http://graphite.readthedocs.org/en/latest/

https://collectd.org/

https://github.com/etsy/statsd

Instead of statsd you could also use python-bucky:

https://pypi.python.org/pypi/bucky

Hope that helps!

Christian

On 27.01.15 04:38, Adam Lawson wrote:
> Hey everyone,
> 
> I'm researching for web-based visualization that displays OpenStack
> Swift and/or node status, cluster health etc in some manner. Being able
> to run a command like adding/removing devices would be cool but a little
> more than I need. Does such a thing currently exist? I know about
> SwiftStack but I'm wondering if there are other efforts that have
> produced a way to visualize Swift telemetry.
> 
> Has anyone run across such a thing?
> */
> Adam Lawson/*
> 
> AQORN, Inc.
> 427 North Tatnall Street
> Ste. 58461
> Wilmington, Delaware 19801-2230
> Toll-free: (844) 4-AQORN-NOW ext. 101
> International: +1 302-387-4660
> Direct: +1 916-246-2072
> 
> 
> 
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> 


___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] (Juno) Multiple rings Swift

[Christian Schwede]

On 23.12.14 11:05, dhanesh1212121212 wrote:
> Is it possible to expand the partition of the above set up.
> 
> Scenario : Disk sda1 is 100% utilized.

Do you mean to extend a partition on a disk or the Swift region/zone?

You could extend the partition on the disk if you're using something
like LVM; however most setups I know about simply use a single large
partition per disk without any volume manager.

If you want to add more capacity to the region/zone/cluster, you simply
add more disks to the ring later, for example:

swift-ring-builder object-1.builder add r1z5-192.168.1.1:6000/sdb1_ 3000

However, make sure the weights in your regions/zones are equally
distributed to ensure replicas are equally distributed as well.

Please not that you should avoid to run a Swift cluster close to 100%
capacity.

Christian



> On Tue, Dec 23, 2014 at 1:43 PM, Christian Schwede
> mailto:christian.schw...@enovance.com>>
> wrote:
> 
> On 22.12.14 21:00, Amit Anand wrote:
> > Thanks Christian I think thats exactly what I needed - I will try and
> > make 2 new rings and see how that works. You wouldnt per chance know
> > how I would limit the replicas, ie 2 in the Paris region and 1 at HQ?
> > I only want one replica to goto the HQ region... ... Also, if you
> > notice i my command for the name i gave hq-ip# - do I have to use the
> > naming convention r#z#-ip when I add?
> 
> Yes, the naming convention tells Swift in which region/zone the device
> is located. For example:
> 
> swift-ring-builder object-1.builder add r1z5-192.168.1.1:6000/sda1_ 3000
> 
> r1: Region 1
> z5: Zone 5
> 3000: weight of the device
> 
> Here is a small example with two regions, the first with two replicas,
> the second region with one. The important thing is to assign 2/3 of the
> total cluster weight to one region, and 1/3 to the other.
> 
> ---
> 
> swift-ring-builder object-1.builder create 14 3 1
> 
> swift-ring-builder object-1.builder add r1z1-192.168.1.1:6000/sda1_ 3000
> swift-ring-builder object-1.builder add r1z2-192.168.1.2:6000/sda1_ 3000
> 
> swift-ring-builder object-1.builder add r2z1-192.168.2.1:6000/sda1_ 3000
> 
> swift-ring-builder object-1.builder rebalance
> 
> swift-ring-builder object-1.builder list_parts r1
> swift-ring-builder object-1.builder list_parts r2
> 
> ---
> 
> This example is a perfect match (ie all partitions store two replicas in
> region 1 and one replica in region 2); however due to roundings during
> the calculation it might happen that there are a few partitions that
> have two replicas in the smaller region. You can see the partition and
> replica count using the "list_parts" command; you might need to tweak
> your weight numbers a little bit depending on your setup.
> 
> Christian
> 
> ___
> Mailing list:
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> <mailto:openstack@lists.openstack.org>
> Unsubscribe :
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> 
> 


___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] (Juno) Multiple rings Swift

[Christian Schwede]

On 22.12.14 21:00, Amit Anand wrote:
> Thanks Christian I think thats exactly what I needed - I will try and
> make 2 new rings and see how that works. You wouldnt per chance know
> how I would limit the replicas, ie 2 in the Paris region and 1 at HQ?
> I only want one replica to goto the HQ region... ... Also, if you
> notice i my command for the name i gave hq-ip# - do I have to use the
> naming convention r#z#-ip when I add?

Yes, the naming convention tells Swift in which region/zone the device
is located. For example:

swift-ring-builder object-1.builder add r1z5-192.168.1.1:6000/sda1_ 3000

r1: Region 1
z5: Zone 5
3000: weight of the device

Here is a small example with two regions, the first with two replicas,
the second region with one. The important thing is to assign 2/3 of the
total cluster weight to one region, and 1/3 to the other.

---

swift-ring-builder object-1.builder create 14 3 1

swift-ring-builder object-1.builder add r1z1-192.168.1.1:6000/sda1_ 3000
swift-ring-builder object-1.builder add r1z2-192.168.1.2:6000/sda1_ 3000

swift-ring-builder object-1.builder add r2z1-192.168.2.1:6000/sda1_ 3000

swift-ring-builder object-1.builder rebalance

swift-ring-builder object-1.builder list_parts r1
swift-ring-builder object-1.builder list_parts r2

---

This example is a perfect match (ie all partitions store two replicas in
region 1 and one replica in region 2); however due to roundings during
the calculation it might happen that there are a few partitions that
have two replicas in the smaller region. You can see the partition and
replica count using the "list_parts" command; you might need to tweak
your weight numbers a little bit depending on your setup.

Christian

___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [swift] [ceph-backend]

[Christian Schwede]

On 23.12.14 03:08, Mark Kirkwood wrote:
> I've been taking a look at this
> (https://github.com/stackforge/swift-ceph-backend and forks etc). Looks
> good.

> 1/ Async updates
> 
> There's a comment in rados_server.py about not handling these. What
> exactly is the issue? (I note in Juno we need to add a policy arg, but
> they seem to work AFAICS).

@Babu, can you shed some light on this?

https://github.com/stackforge/swift-ceph-backend/blob/master/swift_ceph_backend/rados_server.py#L95

> 2/ Replication
> 
> Doing some testing to 2 regions and 2 zones (each region with its own
> ceph cluster), I can get the ceph data out of sync between the regions
> if I stop one region and make changes in the other (then restart). This
> same test works fine for standard swift object backend. Are we in need
> of some more code in the replication side of things?

Yes, if you want to use multiple Ceph backends (for example in different
regions) you need much more code to handle the replication between these
sites.

For now the backend only supports one local Ceph cluster.

Christian

___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] (Juno) Multiple rings Swift

[Christian Schwede]

On 22.12.14 19:11, Amit Anand wrote:
> Now I want to try and do the following below:
> 
> http://techs.enovance.com/7094/openstack-swift-2-0-introducing-storage-policies
> 
> So pretty much have a Paris and Montreal center as described - I want to
> add a third datacenter to each as replication. So it the blog it says
> create new rings - there is where I am getting confused - may someone
> help and let me know what are the commands to create a whole new ring?? 

Author of the post here :)

Use the following command to create another ring:

swift-ring-builder /etc/swift/object-1.builder create 15 3 1

Storage Policies are addressed by an index number (see the blog post,
for example storage-policy:0), and these are related to the ring files:

storage-policy:0 -> /etc/swift/object-0.ring.gz

storage-policy:1 -> /etc/swift/object-1.ring.gz

and so on.

Policy 0 is the default one, and if there is no ring file named
"/etc/swift/object-0.ring.gz" Swift tries to use
"/etc/swift/object.ring.gz".

Let me know if this helps!

Christian

___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [swift] openstack swift PUT object question

[Christian Schwede]

> On Fri, Oct 10, 2014 at 7:44 AM, dboyzhong  > wrote:
> 
> Hi. when i put a object(1K) to swift cluster, the performance is so 
> bad, QPS only 16. and i found the function: "_finalize_put" in file 
> Diskfile.py cost too much time . mainly caused by the fsync() inside 
> _finalize_put operation, the fsync() will cost 30ms when write 1KB 
> data to disk . as a result the PUT performance is bad.

Using a single thread, multiple, threads, spinning disks, SSDs? This
highly depends on your configuration. And 30ms for a single fsync() is a
lot of time, is the disk used heavily? Or is this the actual time for
the whole HTTP request?

> how can tuning the performance ? and why call fsync() function?

The fsync() call ensures that all data is written to disk before the
clients gets a response, stating that everything is ok. Doing this (and
doing this on at least the quorum of the configured replicas) a client
can be sure that the data is actually stored multiple times on disk.
Otherwise there might be a failure before the data gets written to the
disk, and in a worst case scenario that data is lost.

Christian

___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] swift shows double port

[Christian Schwede]

Hi Remo,

most likely the exception message itself is wrong (the shown double
port), but it got fixed a while ago:

https://github.com/openstack/python-swiftclient/commit/2e1a5b7e28211b0b2e2f7a2f961d7fcbd7e8abe3

Can you access the proxy server logs to see what's going on there?

Christian



Am 30.07.14 21:32, schrieb Remo Mattei:
> 
> Hi all, 
> i have a server which was build with foreman and it gives me a double
> port see below:
> 
> swiftclient.exceptions.ClientException: Object PUT failed: 
> http://20.19.19.19:8080:8080/v1/AUTH_f3a3a9be2dbf4ac3b3b846c4ec68ac89/container2/obj
> 
> 
> and it returns a 503 of course . 
> 
> 
> 
> Thanks, 
> 
> Remo 
> 
> 
> 
> 
> 
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> 


___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] openstack swift django browser

[Christian Schwede]

Hi,

Am 03.04.14 13:30, schrieb Ankit Deshwal:
> Hello everyone
> 
>  sir i am able to upload or download through swift CLI. 
> 
> 
> After changing that file  i am able to access newly created account
> through browser.
> 
> But why i have to put . admin after every user..???

The .admin is simply a group that grants this user account ownership.
Account ownership itself allows to create/delete/list containers (a
little bit simplified, but that's the basic idea).

> [filter:tempauth]
> use = egg:swift#tempauth
> user_admin_admin = admin .admin .reseller_admin
> user_test_tester = testing .admin
> user_test2_tester2 = testing2 .admin
> user_cit_cit= cit .admin
> 
> 
> Here cit:cit is my new user
> 
> How can admin control all users in Swift Browser??

You can't, at least not with tempauth. Managing users using tempauth
authentication requires write access to the proxy server, and from a
security point of view any public accessible webserver should not be
allowed to write directly to swift proxy configuration files.

If you want to add/delete/modify accounts you have to use an
authentication system with an API, for example Keystone or swauth. But
there is no support for Keystone in swiftbrowser (yet?), and for swauth
it's not yet complete.

Christian

___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] Problem accessing newly created Swift Account on Django Browser.

[Christian Schwede]

Hi,

are you abled to up-/download with the new user using the CLI?

The pasted commands are not sufficient to create a user in Swift if
you're using tempauth, keystoneauth or swauth.

Assuming you're using tempauth you have to add the accounts/users to
/etc/swift/proxy-server.conf.

http://docs.openstack.org/developer/swift/middleware.html#module-swift.common.middleware.tempauth

Christian



Am 01.04.14 08:48, schrieb Ankit Deshwal:
> Hey Everyone,
> I have certain issues in accessing my newly created account on the
> Django swift browser although the account works perfectly with the Swift
> CLI.
> 
> OpenstackPaste URL:
> http://paste.openstack.org/show/74717/
> 
> Thanks a lot
> 
> 
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> 


___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] Django-Swift-Browser

[Christian Schwede]

Hi,

yes, you can use swiftbrowser to access Swift using tempauth. I use a
local Swift all in one installation for testing as well.

Christian

Am 31.03.14 13:24, schrieb Ashish Dobhal:
> Hey everyone
> I have the following queries regarding openstack swift.Please help.
> 
> 1.Can I use the swift browser if I am using temp auth for authentication
> instead of keystone.
> 
> 2.
> 
> 
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> 


___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] Pip Error In Swift Installation (Swift All in One tutorial for a single node)

[Christian Schwede]

I think you hit the bug reported in
https://bugs.launchpad.net/swift/+bug/1217288

A suggested workaround is to do a

sudo pip install -r requirements.txt

right before you run "python setup.py".

Which distribution and release are you using?



Am 28.03.14 09:56, schrieb Ashish Dobhal:
> Hello sir;
> After making the sudo pip install --upgrade I got the following error.
> 
> File "*/usr/lib/python2.7/distutils/cmd.py", line 59, in __init__
> raise TypeError, "dist must be a Distribution instance"
> TypeError: dist must be a Distribution instance*.
> 
> in the following step 
> *sudo python setup.py develop*
> 
> 
> Note:I did the pip update before my swift installation.Should I do the
> update just before the *sudo pip install -r
> swift/test-requirements.txt*_  command.*
> **
> 
> 
> On Wed, Mar 26, 2014 at 12:22 PM, Ashish Dobhal
> mailto:dobhalashish...@gmail.com>> wrote:
> 
> Thanks Sir,
> i also want to know when to make the update.Before the swift
> installation or before  running sudo pip install -r
> swift/test-requirements.txt*_  command.
> 
> 
> On Wed, Mar 26, 2014 at 12:47 PM, Christian Schwede
>  <mailto:christian.schw...@enovance.com>> wrote:
> 
> Hello,
> 
> looks like your pip version is to old. Try updating it with:
> 
> sudo pip install -U pip
> 
> This worked for me in the past.
> 
> Christian
> 
> Am 25.03.14 19:33, schrieb Ashish Dobhal:
> > Dear Sir/Maam;
> > I am following the swift all in one tutorial.
> > I got he following error while running
> > _*sudo pip install -r swift/test-requirements.txt*_  cmd_*
> >
> > *_
> > _*ERROR
> >
> > *_
> > *pkg_resources.VersionConflict: (pip 1.0
> > (/usr/lib/python2.7/dist-packages), Requirement.parse('pip>=1.4'))
> >
> > 
> > Command python setup.py egg_info failed with error code 1
> > Storing complete log in /home/yashika/.pip/pip.log*
> > Please Reply.
> > Thank You.
> >
> >
> > ___
> > Mailing list:
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> > Post to : openstack@lists.openstack.org
> <mailto:openstack@lists.openstack.org>
> > Unsubscribe :
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> >
> 
> 
> 


___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] Pip Error In Swift Installation (Swift All in One tutorial for a single node)

[Christian Schwede]

Hello,

looks like your pip version is to old. Try updating it with:

sudo pip install -U pip

This worked for me in the past.

Christian

Am 25.03.14 19:33, schrieb Ashish Dobhal:
> Dear Sir/Maam;
> I am following the swift all in one tutorial.
> I got he following error while running
> _*sudo pip install -r swift/test-requirements.txt*_  cmd_*
> 
> *_
> _*ERROR
> 
> *_
> *pkg_resources.VersionConflict: (pip 1.0
> (/usr/lib/python2.7/dist-packages), Requirement.parse('pip>=1.4'))
> 
> 
> Command python setup.py egg_info failed with error code 1
> Storing complete log in /home/yashika/.pip/pip.log*
> Please Reply.
> Thank You.
> 
> 
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> 


___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] Fwd: Swift Functional Tests Errors

[Christian Schwede]

Hello,

could you share the error output and your /etc/swift/proxy-server.conf?
You can use paste.openstack.org for sharing.

Christian

Am 25.03.14 17:50, schrieb Ashish Dobhal:
> 
> 
> -- Forwarded message -- From: *Ashish Dobhal*
> mailto:dobhalashish...@gmail.com>> Date:
> Tue, Mar 25, 2014 at 9:52 PM Subject: Swift Functional Tests Errors 
> To: openstack@lists.openstack.org
> 
> 
> 
> Dear Sir/Maam 1.I am getting a lot of errors in the *_functional 
> test(_*$HOME/swift/.functests*_)_* of my swift deployment.How can I 
> rectify them.I am following the Swift All In One 
> Tutorial.(http://docs.openstack.org/developer/swift/development_saio.html)
>
>  2.I am also getting the following *_error while creating a
> container._*
> 
> rock@rohan-Inspiron-5521:/$ swift --os-auth-token 
> AUTH_tkd43461ec44004438818da6e61a55f2ee \--os-storage-url 
> http://127.0.0.1/v1/AUTH_test \post cont Container PUT failed:
> http://127.0.0.1/v1/AUTH_test/cont 404 Not Found [first 60 chars of
> response] Mk�0
> 
> ��� ZO�aQ Please Reply. Thank you
> 
> 
> 
> ___ Mailing list:
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to
> : openstack@lists.openstack.org Unsubscribe :
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> 


___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [Swift] Optimizing single node Swift instance

[Christian Schwede]

Am 19.03.14 23:40, schrieb Shrinand Javadekar:
> 1. Changing the partition power of the object server rings. I was
> running with a partition power of 8. I ran some experiments changing
> this to 4 and then even 0. Turned out that this actually hurt
> performance. I confirmed that the number of directories created in the
> objects directory was 2 ^ partition power. I guess with reduced number
> of directories (in the case of 0, there was just one directory) there
> was more contention at the xfs file-system causing the system to give
> lower throughput.
> 
> I tested with partition powers of 0, 4, 7, 8, 16. 7 because the swift
> ring calculator suggested so [1]. I got the best throughput at 8
> rings.

You might also have a look at
https://swiftstack.com/blog/2012/05/14/part-power-performance/ and
https://answers.launchpad.net/swift/+question/211929

Be careful when selecting a low partition power - you can't change it
later on and if you expect a significant growth of your cluster you
might run into some trouble.
Increasing the partition power later on requires a new ring, and that in
turn requires copying a lot of data (either to a new cluster [1] or
internally to other storage nodes [2]) and should only be done as a last
resort

Christian

[1] https://github.com/stackforge/swiftsync
[2] https://github.com/enovance/swift-ring-tool

___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] problem in log in django swiftbrowser

[Christian Schwede]

Hello Pragya,

please collect the static files in step 5 from the Readme:

> python manage.py collectstatic

If you didn't changed ALLOWED_HOSTS and run swiftbrowser with Debug =
False (default) you also need to run the development server with the
--insecure option (step 6 in the Readme).

> python manage.py runserver --insecure

Christian


Am 03.03.14 09:57, schrieb pragya jain:
> hello all,
> 
> I had installed django swiftbrowser using the link
> https://github.com/cschwede/django-swiftbrowser
> when I login in swifbrowser, I can't login and get the following response.
> 
> root@pragya-VirtualBox:~/django-swiftbrowser/myproj# python manage.py
> runserver
> Validating models...
> 
> 0 errors found
> March 03, 2014 - 09:50:16
> Django version 1.6.2, using settings 'myproj.settings'
> Starting development server at http://127.0.0.1:8000/
> Quit the server with CONTROL-C.
> [03/Mar/2014 09:50:57] "GET / HTTP/1.1" 302 0
> [03/Mar/2014 09:50:57] "GET /login/ HTTP/1.1" 200 3799
> [03/Mar/2014 09:50:58] "GET /static/bootstrap/css/bootstrap.min.css
> HTTP/1.1" 404 11
> [03/Mar/2014 09:50:58] "GET
> /static/bootstrap/css/bootstrap-responsive.min.css HTTP/1.1" 404 11
> [03/Mar/2014 09:50:58] "GET /static/bootstrap/js/bootstrap.min.js
> HTTP/1.1" 404 11
> [03/Mar/2014 09:50:58] "GET /static/jquery.min.js HTTP/1.1" 404 11
> [03/Mar/2014 09:50:58] "GET /static/bootstrap/js/bootstrap.min.js
> HTTP/1.1" 404 11
> [03/Mar/2014 09:51:36] "POST /login/ HTTP/1.1" 500 0
> 
> please help to resolve the problem
> 
> pragya Jain
> 
> 
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> 


___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] problem in installing django-swiftbrowser

[Christian Schwede]

Hello Pragya,

> 5. root@pragya-VirtualBox:~/Django-1.6.2# python runtests.py
>
> and get the following error:
>
> ERROR 1:
> python: can't open file 'runtests.py': [Errno 2] No such file or directory

you're still in the Django-1.6.2 directory, you need to change to the
directoy django-swiftbrowser to run the tests for Swiftbrowser.

Christian (django-swiftbrowser author)


___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] problem in installing django-swiftbrowser

[Christian Schwede]

Hello,

Am 19.02.14 06:48, schrieb pragya jain:
> 6. root@pragya-VirtualBox:~/django-swiftbrowser/setuptools-2.2# sudo
> python ./setup.py install
> 
> 7. root@pragya-VirtualBox:~/django-swiftbrowser# python runtests.py
> 
> and I got the following error :
> 
> Traceback (most recent call last): File "runtests.py", line 2, in
>  from django.conf import settings ImportError: No module
> named django.conf

you also need to execute "sudo python ./setup.py install" inside
~/django-swiftbrowser, you only installed setuptools but not
swiftbrowser itself and thus the required dependencies (Django in this
case) are missing.

Christian

___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] Q: Switft OwnCloud

[Christian Schwede]

Am 17.12.13 06:14, schrieb Frans Thamura:

> or is there alternative to use swift as file sharing in office and on internet

There is also an alternative webclient, you can find it here:

https://github.com/cschwede/django-swiftbrowser

Christian


___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] user interface for using swift

[Christian Schwede]

Hi,

you might also be interested in

https://github.com/cschwede/django-swiftbrowser

Please note that this is not an official OpenStack project.

Christian

Am 05.12.13 12:43, schrieb pragya jain:
> thanks for reply sir,
>
> I had installed swift using link
> http://docs.openstack.org/developer/swift/development_saio.html
>
> Now, what should I do to use horizon for GUI with swift?
>
> Please help
>
>
>
> On Thursday, 5 December 2013 3:04 PM, Hai Quan Chen
>  wrote:
>
> There is GUI in Horizon: http:///project/containers
>
>
> Best Regards!
>
> Chen Hai Quan (???) Tivoli TEL: 86-10-82454165
> Mailto:chenh...@cn.ibm.com 
>
>
>
>
> From:pragya jain  To:Joe Hakim
> Rahme , "openstack@lists.openstack.org"
>  Date:2013/12/05 17:25
> Subject:Re: [Openstack] user interface for using swift
> -
>
>
>
> hello joe,
>
> I have CLI. But I want some GUI to perform all operations , eg.,
> authentication, PUT/GET/DELETE/HEADetc. on account/container/object
>
> please help
>
>
> On Thursday, 5 December 2013 2:12 PM, Joe Hakim Rahme
>  wrote: Hello Pragya,
>
> I'm not totally sure what you mean by user interface, but if you are
> looking for a CLI client, you should check out python-swiftclient[1]
> easily available on PyPI.
>
> --- Joe H. Rahme
>
> [1]: _https://pypi.python.org/pypi/python-swiftclient/1.0_
> ___ Mailing list:
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to
> : openstack@lists.openstack.org Unsubscribe :
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>
>
>
>
>
> ___ Mailing list:
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to
> : openstack@lists.openstack.org Unsubscribe :
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack