Re: [ceph-users] Shadow Files

2015-05-11 Thread Yehuda Sadeh-Weinraub
It's the wip-rgw-orphans branch. - Original Message - > From: "Daniel Hoffman" > To: "Yehuda Sadeh-Weinraub" > Cc: "Ben" , "David Zafman" , > "ceph-users" > Sent: Monday, May 11, 2015 4:30:11 PM > Subject: Re: [

Re: [ceph-users] Shadow Files

2015-05-11 Thread Daniel Hoffman
*To: *"Yehuda Sadeh-Weinraub" > *Cc: *"Ben" , "ceph-users" > *Sent: *Sunday, May 10, 2015 5:03:22 PM > *Subject: *Re: [ceph-users] Shadow Files > > Any updates on when this is going to be released? > > Daniel > > On Wed, May 6, 2015 at 3:51 AM

Re: [ceph-users] Shadow Files

2015-05-11 Thread Yehuda Sadeh-Weinraub
- Original Message - > From: "Daniel Hoffman" > To: "Yehuda Sadeh-Weinraub" > Cc: "Ben" , "ceph-users" > Sent: Sunday, May 10, 2015 5:03:22 PM > Subject: Re: [ceph-users] Shadow Files > Any updates on when this is going t

Re: [ceph-users] Shadow Files

2015-05-10 Thread Daniel Hoffman
maybe see if using a different api would be better compatible > with older backends. > > Yehuda > > - Original Message - > > From: "Anthony Alba" > > To: "Yehuda Sadeh-Weinraub" > > Cc: "Ben" , "ceph-users" >

Re: [ceph-users] Shadow Files

2015-05-05 Thread Yehuda Sadeh-Weinraub
rom: "Anthony Alba" > To: "Yehuda Sadeh-Weinraub" > Cc: "Ben" , "ceph-users" > Sent: Tuesday, May 5, 2015 10:14:38 AM > Subject: Re: [ceph-users] Shadow Files > > Unfortunately it immediately aborted (running against a 0.80.9 Ceph). > Doe

Re: [ceph-users] Shadow Files

2015-05-05 Thread Anthony Alba
5, 2015 at 10:41 PM, Yehuda Sadeh-Weinraub wrote: > Can you try creating the .log pool? > > Yehda > > - Original Message - >> From: "Anthony Alba" >> To: "Yehuda Sadeh-Weinraub" >> Cc: "Ben" , "ceph-users" >> Sent:

Re: [ceph-users] Shadow Files

2015-05-05 Thread Yehuda Sadeh-Weinraub
Can you try creating the .log pool? Yehda - Original Message - > From: "Anthony Alba" > To: "Yehuda Sadeh-Weinraub" > Cc: "Ben" , "ceph-users" > Sent: Tuesday, May 5, 2015 3:37:15 AM > Subject: Re: [ceph-users] Shadow Files &

Re: [ceph-users] Shadow Files

2015-05-05 Thread Anthony Alba
fically 1, 2). >> >> In order to clean up a job's data: >> >> $ radosgw-admin orphans finish --job-id= >> >> Note that the jobs run in the radosgw-admin process context, it does not >> schedule a job on the radosgw process. >> >> Please let me know o

Re: [ceph-users] Shadow Files

2015-05-05 Thread Anthony Alba
sh --job-id= > > Note that the jobs run in the radosgw-admin process context, it does not > schedule a job on the radosgw process. > > Please let me know of any issue you find. > > Thanks, > Yehuda > > - Original Message - >> From: "Ben Hine

Re: [ceph-users] Shadow Files

2015-05-04 Thread Yehuda Sadeh-Weinraub
le a job on the radosgw process. Please let me know of any issue you find. Thanks, Yehuda - Original Message - > From: "Ben Hines" > To: "Ben" > Cc: "Yehuda Sadeh-Weinraub" , "ceph-users" > > Sent: Thursday, April 30, 2015 3:00:16 PM

Re: [ceph-users] Shadow Files

2015-04-30 Thread Ben Hines
nup tool. >> >> Yehuda >> >> - Original Message - >> >>> From: "Ben" >>> To: "Yehuda Sadeh-Weinraub" >>> Cc: "ceph-users" >>> Sent: Sunday, April 26, 2015 11:02:23 PM >>> Subject: Re: [ceph-u

Re: [ceph-users] Shadow Files

2015-04-27 Thread Ben
nd testing the fix, and there's more work to be done on the cleanup tool. Yehuda - Original Message - From: "Ben" To: "Yehuda Sadeh-Weinraub" Cc: "ceph-users" Sent: Sunday, April 26, 2015 11:02:23 PM Subject: Re: [ceph-users] Shadow Files Are the

Re: [ceph-users] Shadow Files

2015-04-27 Thread Yehuda Sadeh-Weinraub
quot; > Sent: Sunday, April 26, 2015 11:02:23 PM > Subject: Re: [ceph-users] Shadow Files > > Are these fixes going to make it into the repository versions of ceph, > or will we be required to compile and install manually? > > On 2015-04-26 02:29, Yehuda Sadeh-Weinraub wrote: > &g

Re: [ceph-users] Shadow Files

2015-04-26 Thread Ben
"Ben" To: "Ben Hines" , "Yehuda Sadeh-Weinraub" Cc: "ceph-users" Sent: Friday, April 24, 2015 5:14:11 PM Subject: Re: [ceph-users] Shadow Files Definitely need something to help clear out these old shadow files. I'm sure our cluster has around 10

Re: [ceph-users] Shadow Files

2015-04-25 Thread Yehuda Sadeh-Weinraub
Yeah, that's definitely something that we'd address soon. Yehuda - Original Message - > From: "Ben" > To: "Ben Hines" , "Yehuda Sadeh-Weinraub" > > Cc: "ceph-users" > Sent: Friday, April 24, 2015 5:14:11 PM > Subj

Re: [ceph-users] Shadow Files

2015-04-24 Thread Ben
ailto:yeh...@redhat.com>> > Cc: "ceph-users" mailto:ceph-us...@ceph.com>> > Sent: Friday, April 24, 2015 3:06:02 PM > Subject: Re: [ceph-users] Shadow Files > > We were firefly, then we upgraded to giant, now we are on hammer. > > What

Re: [ceph-users] Shadow Files

2015-04-24 Thread Ben Hines
95 > http://tracker.ceph.com/issues/11447 > > - Original Message - > > From: "Ben Jackson" > > To: "Yehuda Sadeh-Weinraub" > > Cc: "ceph-users" > > Sent: Friday, April 24, 2015 3:06:02 PM > > Subject: Re: [ceph-users] Shadow

Re: [ceph-users] Shadow Files

2015-04-24 Thread Ben Jackson
Original Message - > > From: "Ben Jackson" > > To: "Yehuda Sadeh-Weinraub" > > Cc: "ceph-users" > > Sent: Friday, April 24, 2015 3:06:02 PM > > Subject: Re: [ceph-users] Shadow Files > > > > We were firefly, then we upg

Re: [ceph-users] Shadow Files

2015-04-24 Thread Yehuda Sadeh-Weinraub
These ones: http://tracker.ceph.com/issues/10295 http://tracker.ceph.com/issues/11447 - Original Message - > From: "Ben Jackson" > To: "Yehuda Sadeh-Weinraub" > Cc: "ceph-users" > Sent: Friday, April 24, 2015 3:06:02 PM > Subject: Re: [

Re: [ceph-users] Shadow Files

2015-04-24 Thread Ben Jackson
gt; > Yehuda > > - Original Message - > > From: "Ben" > > To: "ceph-users" > > Cc: "Yehuda Sadeh-Weinraub" > > Sent: Thursday, April 23, 2015 7:42:06 PM > > Subject: [ceph-users] Shadow Files > > > > We

Re: [ceph-users] Shadow Files

2015-04-24 Thread Yehuda Sadeh-Weinraub
rsday, April 23, 2015 7:42:06 PM > Subject: [ceph-users] Shadow Files > > We are still experiencing a problem with out gateway not properly > clearing out shadow files. > > I have done numerous tests where I have: > -Uploaded a file of 1.5GB in size using s3browser applicati

[ceph-users] Shadow Files

2015-04-23 Thread Ben
We are still experiencing a problem with out gateway not properly clearing out shadow files. I have done numerous tests where I have: -Uploaded a file of 1.5GB in size using s3browser application -Done an object stat on the file to get its prefix -Done rados ls -p .rgw.buckets | grep to count t

Re: [ceph-users] Shadow files

2015-03-18 Thread Yehuda Sadeh-Weinraub
- Original Message - > From: "Abhishek L" > To: "Yehuda Sadeh-Weinraub" > Cc: "Ben" , "ceph-users" > Sent: Wednesday, March 18, 2015 10:54:37 AM > Subject: Re: [ceph-users] Shadow files > > > Yehuda Sadeh-Weinraub wr

Re: [ceph-users] Shadow files

2015-03-18 Thread Abhishek L
Yehuda Sadeh-Weinraub writes: >> Is there a quick way to see which shadow files are safe to delete >> easily? > > There's no easy process. If you know that a lot of the removed data is on > buckets that shouldn't exist anymore then you could start by trying to > identify that. You could do that

Re: [ceph-users] Shadow files

2015-03-18 Thread Yehuda Sadeh-Weinraub
- Original Message - > From: "Ben" > To: "Yehuda Sadeh-Weinraub" > Cc: "Craig Lewis" , "ceph-users" > > Sent: Tuesday, March 17, 2015 7:28:28 PM > Subject: Re: [ceph-users] Shadow files > > None of this helps with tr

Re: [ceph-users] Shadow files

2015-03-17 Thread Ben
y, March 16, 2015 3:38:42 PM Subject: Re: [ceph-users] Shadow files Thats the thing. The peaks and troughs are in USERS BUCKETS only. The actual cluster usage does not go up and down, it just goes up up up. I would expect to see peaks and troughs much the same as the user buckets peaks and trough

Re: [ceph-users] Shadow files

2015-03-17 Thread Yehuda Sadeh-Weinraub
- Original Message - > From: "Ben" > To: "Craig Lewis" > Cc: "Yehuda Sadeh-Weinraub" , "ceph-users" > > Sent: Monday, March 16, 2015 3:38:42 PM > Subject: Re: [ceph-users] Shadow files > > Thats the thing. The peaks a

Re: [ceph-users] Shadow files

2015-03-16 Thread Ben
- Original Message - From: "Ben" To: ceph-us...@ceph.com Sent: Wednesday, March 11, 2015 8:46:25 PM Subject: Re: [ceph-users] Shadow files Anyone got any info on this? Is it safe to delete shadow files? It depends. Shadow files are badly named objects that represent part of the obje

Re: [ceph-users] Shadow files

2015-03-16 Thread Gregory Farnum
hs, it is just a rising >> peak. >> But when looking at users data usage, we can see peaks and troughs as you >> would expect as data is deleted and added. >> >> Our ceph version 0.80.9 >> >> Please ideas? >> >> On 2015-03-13 02:25, Yehuda Sade

Re: [ceph-users] Shadow files

2015-03-16 Thread Craig Lewis
gt;> >>> From: "Ben" >>> To: ceph-us...@ceph.com >>> Sent: Wednesday, March 11, 2015 8:46:25 PM >>> Subject: Re: [ceph-users] Shadow files >>> >>> Anyone got any info on this? >>> >>> Is it safe to del

Re: [ceph-users] Shadow files

2015-03-15 Thread Ben
: "Ben" To: ceph-us...@ceph.com Sent: Wednesday, March 11, 2015 8:46:25 PM Subject: Re: [ceph-users] Shadow files Anyone got any info on this? Is it safe to delete shadow files? It depends. Shadow files are badly named objects that represent part of the objects data. They are only safe

Re: [ceph-users] Shadow files

2015-03-12 Thread Ben
. On 2015-03-13 02:25, Yehuda Sadeh-Weinraub wrote: - Original Message - From: "Ben" To: ceph-us...@ceph.com Sent: Wednesday, March 11, 2015 8:46:25 PM Subject: Re: [ceph-users] Shadow files Anyone got any info on this? Is it safe to delete shadow files? It depends. Shadow

Re: [ceph-users] Shadow files

2015-03-12 Thread ghislain.chevalier
ehuda Sadeh-Weinraub , ceph-us...@ceph.com Objet : Re: [ceph-users] Shadow files Hello Ben, I’m facing with the same issue - #10295<http://tracker.ceph.com/issues/10295> and I remove the object directly from that rados successfully. But is very important map all object before do that. I recomme

Re: [ceph-users] Shadow files

2015-03-12 Thread Italo Santos
-Weinraub wrote: > > > - Original Message - > > From: "Ben" mailto:b@benjackson.email)> > > To: ceph-us...@ceph.com (mailto:ceph-us...@ceph.com) > > Sent: Wednesday, March 11, 2015 8:46:25 PM > > Subject: Re: [ceph-users] Shadow files > >

Re: [ceph-users] Shadow files

2015-03-12 Thread Yehuda Sadeh-Weinraub
- Original Message - > From: "Ben" > To: ceph-us...@ceph.com > Sent: Wednesday, March 11, 2015 8:46:25 PM > Subject: Re: [ceph-users] Shadow files > > Anyone got any info on this? > > Is it safe to delete shadow files? It depends. Shadow files are

Re: [ceph-users] Shadow files

2015-03-11 Thread Ben
Anyone got any info on this? Is it safe to delete shadow files? On 2015-03-11 10:03, Ben wrote: We have a large number of shadow files in our cluster that aren't being deleted automatically as data is deleted. Is it safe to delete these files? Is there something we need to be aware of when del

[ceph-users] Shadow files

2015-03-10 Thread Ben
We have a large number of shadow files in our cluster that aren't being deleted automatically as data is deleted. Is it safe to delete these files? Is there something we need to be aware of when deleting them? Is there a script that we can run that will delete these safely? Is there something w