[ceph-users] Re: rbd export-diff/import-diff hangs

[Tony Liu]

It's export-diff from an in-use image, both from-snapshot and to-snapshot exist.
The same from-snapshot exists in import image, which is the to-snapshot from 
last diff.
export/import is used for local backup, rbd-mirroring is used for remote backup.
Looking for options to get more info to troubleshoot.


Thanks!
Tony

From: Eugen Block 
Sent: August 27, 2023 11:53 AM
To: ceph-users@ceph.io
Subject: [ceph-users] Re: rbd export-diff/import-diff hangs

You mean the image is in use while you’re exporting? Have thought
about creating snapshots and exporting those? Or set up rbd mirroring?

Zitat von Tony Liu :

> To update, hanging happens when updating local image, not remote, networking
> is not a concern here. Any advices how to look into it?
>
> Thanks!
> Tony
> 
> From: Tony Liu 
> Sent: August 26, 2023 10:43 PM
> To: d...@ceph.io; ceph-users@ceph.io
> Subject: [ceph-users] rbd export-diff/import-diff hangs
>
> Hi,
>
> I'm using rbd import and export to copy image from one cluster to another.
> Also using import-diff and export-diff to update image in remote cluster.
> For example, "rbd --cluster local export-diff ... | rbd --cluster
> remote import-diff ...".
> Sometimes, the whole command is stuck. I can't tell it's stuck on
> which end of the pipe.
> I did some search, [1] seems the same issue and [2] is also related.
>
> Wonder if there is any way to identify where it's stuck and get more
> debugging info.
> Given [2], I'd suspect the import-diff is stuck, cause rbd client is
> importing to the
> remote cluster. Networking latency could be involved here? Ping
> latency is 7~8 ms.
>
> Any comments is appreciated!
>
> [1] https://bugs.launchpad.net/cinder/+bug/2031897
> [2] https://stackoverflow.com/questions/69858763/ceph-rbd-import-hangs
>
> Thanks!
> Tony
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: rbd export with export-format 2 exports all snapshots?

[Tony Liu]

Thank you Alex for confirmation!

Tony

From: Alex Gorbachev 
Sent: August 27, 2023 05:29 PM
To: Tony Liu
Cc: d...@ceph.io; ceph-users@ceph.io
Subject: Re: [ceph-users] rbd export with export-format 2 exports all snapshots?

Tony,

From what I recall having worked with snapshots a while ago, you would want 
export-diff to achieve a differential export.  "export" will always go for a 
full image.

--
Alex Gorbachev
https://alextelescope.blogspot.com



On Sun, Aug 27, 2023 at 8:03 PM Tony Liu 
mailto:tonyliu0...@hotmail.com>> wrote:
Hi,

Say, source image has snapshot s1, s2 and s3.

I expect "export" behaves the same as "deep cp", when specify a snapshot,
with "--export-format 2", only the specified snapshot and all snapshots
earlier than that will be exported.

What I see is that, no matter which snapshot I specify, "export" with
"--export-format 2" always exports the whole image with all snapshots.
Is this expected?

Could anyone help to clarify?


Thanks!
Tony
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to 
ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: rbd export with export-format 2 exports all snapshots?

[Alex Gorbachev]

Tony,

From what I recall having worked with snapshots a while ago, you would want
export-diff to achieve a differential export.  "export" will always go for
a full image.

--
Alex Gorbachev
https://alextelescope.blogspot.com



On Sun, Aug 27, 2023 at 8:03 PM Tony Liu  wrote:

> Hi,
>
> Say, source image has snapshot s1, s2 and s3.
>
> I expect "export" behaves the same as "deep cp", when specify a snapshot,
> with "--export-format 2", only the specified snapshot and all snapshots
> earlier than that will be exported.
>
> What I see is that, no matter which snapshot I specify, "export" with
> "--export-format 2" always exports the whole image with all snapshots.
> Is this expected?
>
> Could anyone help to clarify?
>
>
> Thanks!
> Tony
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] rbd export with export-format 2 exports all snapshots?

[Tony Liu]

Hi,

Say, source image has snapshot s1, s2 and s3.

I expect "export" behaves the same as "deep cp", when specify a snapshot,
with "--export-format 2", only the specified snapshot and all snapshots
earlier than that will be exported.

What I see is that, no matter which snapshot I specify, "export" with
"--export-format 2" always exports the whole image with all snapshots.
Is this expected?

Could anyone help to clarify?


Thanks!
Tony
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: What does 'removed_snaps_queue' [d5~3] means?

[Work Ceph]

I see, thanks for the reply!

BTW, while the snapshots are not removed yet, should we be able to delete
the image that had the snapshots being deleted?

We noticed the following for the images that had snapshots deleted, but not
actually removed from the system:
```
This means the image is still open or the client using it crashed. Try
again after closing/unmapping it or waiting 30s for the crashed client to
timeout.
```

After executing the "rbd rm", we receive that message, and the image still
displays with the "rbd ls" command. However, the "rbd info" returns a
message saying that the image does not exist. Is that a known
issue/situation?

On Sat, Aug 26, 2023 at 5:24 AM Eugen Block  wrote:

> Hi,
>
> that specifies a range of (to be) removed snapshots. Do you have rbd
> mirroring configured or some scripted snapshot creation/deletion?
> Snapshot deletion is an asynchronous operation, so they are added to
> the queue and deleted at some point. Does the status/range change?
> Which exact Octopus version are you running? I have two test clusters
> (latest Octopus) with rbd mirroring and when I set that up I expected
> to see something similar, in earlier Ceph versions that was visible in
> the pool ls detail output. Anyway, I wouldn't worry about it as long
> as the queue doesn't grow and the snaps are removed eventually. You
> should see the snaptrimming in the 'ceph -s' output as well, the PGs
> have a respective state (active+snaptrim or active+snaptrim_wait). I
> write this from memory, so the PG state might differ a bit.
> You just need to be aware of the impacts of many snapshots for many
> images, I'm still investigating a customer issue, some of the results
> I posted in this list [1].
>
> Regards,
> Eugen
>
> [1]
>
> https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/ZEMGKBLMEREBZB7SWOLDA6QZX3S7FLL3/#YAHVTTES6YU5IXZJ2UNXKURXSHM5HDEX
>
> Zitat von Work Ceph :
>
> > Hello guys,
> > We are facing/seeing an unexpected mark in one of our pools. Do you guys
> > know what does "removed_snaps_queue" it mean? We see some notation such
> as
> > "d5~3" after this tag. What does it mean? We tried to look into the docs,
> > but could not find anything meaningful.
> >
> > We are running Ceph Octopus on top of Ubuntu 18.04.
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: rbd export-diff/import-diff hangs

[Eugen Block]

You mean the image is in use while you’re exporting? Have thought  
about creating snapshots and exporting those? Or set up rbd mirroring?


Zitat von Tony Liu :


To update, hanging happens when updating local image, not remote, networking
is not a concern here. Any advices how to look into it?

Thanks!
Tony

From: Tony Liu 
Sent: August 26, 2023 10:43 PM
To: d...@ceph.io; ceph-users@ceph.io
Subject: [ceph-users] rbd export-diff/import-diff hangs

Hi,

I'm using rbd import and export to copy image from one cluster to another.
Also using import-diff and export-diff to update image in remote cluster.
For example, "rbd --cluster local export-diff ... | rbd --cluster  
remote import-diff ...".
Sometimes, the whole command is stuck. I can't tell it's stuck on  
which end of the pipe.

I did some search, [1] seems the same issue and [2] is also related.

Wonder if there is any way to identify where it's stuck and get more  
debugging info.
Given [2], I'd suspect the import-diff is stuck, cause rbd client is  
importing to the
remote cluster. Networking latency could be involved here? Ping  
latency is 7~8 ms.


Any comments is appreciated!

[1] https://bugs.launchpad.net/cinder/+bug/2031897
[2] https://stackoverflow.com/questions/69858763/ceph-rbd-import-hangs

Thanks!
Tony
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: rbd export-diff/import-diff hangs

[Tony Liu]

To update, hanging happens when updating local image, not remote, networking
is not a concern here. Any advices how to look into it?

Thanks!
Tony

From: Tony Liu 
Sent: August 26, 2023 10:43 PM
To: d...@ceph.io; ceph-users@ceph.io
Subject: [ceph-users] rbd export-diff/import-diff hangs

Hi,

I'm using rbd import and export to copy image from one cluster to another.
Also using import-diff and export-diff to update image in remote cluster.
For example, "rbd --cluster local export-diff ... | rbd --cluster remote 
import-diff ...".
Sometimes, the whole command is stuck. I can't tell it's stuck on which end of 
the pipe.
I did some search, [1] seems the same issue and [2] is also related.

Wonder if there is any way to identify where it's stuck and get more debugging 
info.
Given [2], I'd suspect the import-diff is stuck, cause rbd client is importing 
to the
remote cluster. Networking latency could be involved here? Ping latency is 7~8 
ms.

Any comments is appreciated!

[1] https://bugs.launchpad.net/cinder/+bug/2031897
[2] https://stackoverflow.com/questions/69858763/ceph-rbd-import-hangs

Thanks!
Tony
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io