[Bug 1540407] Comment bridged from LTC Bugzilla

[bugproxy]

--- Comment From thorsten.di...@de.ibm.com 2016-04-01 04:52 EDT---
fix successfully verified, 300 cycles under z/VM and in LPAR.
kernel 4.4.0-16, multipath version 0.5.0+git1.656f8865-5ubuntu1.
Closed.

-- 
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to multipath-tools in Ubuntu.
https://bugs.launchpad.net/bugs/1540407

Title:
  multipathd drops paths of a temporarily lost device

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1540407/+subscriptions

-- 
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs

[Bug 1540407] Comment bridged from LTC Bugzilla

[bugproxy]

--- Comment From thorsten.di...@de.ibm.com 2016-03-21 13:50 EDT---
Hi Ryan, hi Dimitri,

Update from LP 1551952:
> St?phane Graber (stgraber) wrote on 2016-03-18:   #7
>
> Unfortunate that this only gets in so late in the cycle, but I can certainly
> see the benefit of updating this before we have to support it for 5 years.
>
> Testing seems reasonable. debdiff is unreviewable so we can only rely on the
> testing...
>
> Assuming that you'll take care of any yet unknown regression from this, you
> may go ahead and upload the merge.
> Changed in multipath-tools (Ubuntu):
> status:   New ? Triaged

...let's roll.

-- 
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to multipath-tools in Ubuntu.
https://bugs.launchpad.net/bugs/1540407

Title:
  multipathd drops paths of a temporarily lost device

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1540407/+subscriptions

-- 
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs

[Bug 1540407] Comment bridged from LTC Bugzilla

[bugproxy]

--- Comment From heinz-werner_se...@de.ibm.com 2016-03-16 06:43 EDT---
Canonical: This bug is currently in stage "Triaged". Please provide the info , 
when it will be in /release/main. Many thanks in advance

-- 
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to multipath-tools in Ubuntu.
https://bugs.launchpad.net/bugs/1540407

Title:
  multipathd drops paths of a temporarily lost device

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1540407/+subscriptions

-- 
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs

[Bug 1540407] Comment bridged from LTC Bugzilla

[bugproxy]

--- Comment From thorsten.di...@de.ibm.com 2016-03-11 05:05 EDT---
(In reply to comment #27)
> Here's my initial test with the merged multipath-tools test.
> I had the FCP devices enabled with the 0.5.0+git-1ubuntu2 package
> installed from the merges ppa  and rebooted the system.
>
> After booting, I confirmed the paths were up, then used a vmcp command to
> disconnect the devices.
> I then queried mulitpath over a number of minutes to ensure the path remains
> (but shows faulty).
> After 15 minutues or so I contacted an admin to renable the FCP devices and
> observed the paths
> becoming restored in multipath

Ryan,
congrats, well done. :-) I did an extended test with this multipathd/kpartx on 
a z/VM guest and in LPAR overnight and it ran very well. Even after 300 
"off/on" cycles all paths were "active ready running" at the end.
So, please go ahead with this. I will continue testing this when it is in 
xenial/main, since some of my other tests rely on this.

I will close this defect when this fix is in xenial/main.

-- 
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to multipath-tools in Ubuntu.
https://bugs.launchpad.net/bugs/1540407

Title:
  multipathd drops paths of a temporarily lost device

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1540407/+subscriptions

-- 
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs

[Bug 1540407] Comment bridged from LTC Bugzilla

[bugproxy]

--- Comment From thorsten.di...@de.ibm.com 2016-03-08 09:40 EDT---
Ryan,  what's your decision regarding this defect? Since I have to repeat many 
tests with that fixed multipath version, I'm really looking forward to get it 
soon.

-- 
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to multipath-tools in Ubuntu.
https://bugs.launchpad.net/bugs/1540407

Title:
  multipathd drops paths of a temporarily lost device

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1540407/+subscriptions

-- 
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs

[Bug 1540407] Comment bridged from LTC Bugzilla

[bugproxy]

--- Comment From thorsten.di...@de.ibm.com 2016-03-02 02:36 EDT---
Ryan,

I appreaciate to read that, that's the best approach.

-- 
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to multipath-tools in Ubuntu.
https://bugs.launchpad.net/bugs/1540407

Title:
  multipathd drops paths of a temporarily lost device

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1540407/+subscriptions

-- 
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs

Re: [Bug 1540407] Comment bridged from LTC Bugzilla

[Ryan Harper]

On Tue, Mar 1, 2016 at 3:00 AM, bugproxy  wrote:

> --- Comment From thorsten.di...@de.ibm.com 2016-03-01 03:58 EDT---
> (In reply to comment #19)
> > As already promised, I will continue some long term tests (cable pull and
> > other path loosing scenarios) on a 4.4.0-8 system with
> > multipath-tools_0.5.0+git1.656f8865-4 and kpartx_0.5.0+git1.656f8865-4.
> > Results should be available by tomorrow.
>
> Results show, that this combination does not harm, instead it solves the
> subject problem, even after 100 cycles.
>
> Ryan, but it's up to you what you are going to integrate to fix this
> bug.
>

Thorsten,

Thanks for the thorough testing of the scenarios.  I'm currently examining
the changes
introduced into the newer Debian package and looking at which changes we'd
need
to fix this bug and also understand if a merge of the package makes more
sense.


Ryan


>
> --
> You received this bug notification because you are a member of Canonical
> Server Team, which is a bug assignee.
> https://bugs.launchpad.net/bugs/1540407
>
> Title:
>   multipathd drops paths of a temporarily lost device
>
> To manage notifications about this bug go to:
>
> https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1540407/+subscriptions
>

-- 
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to multipath-tools in Ubuntu.
https://bugs.launchpad.net/bugs/1540407

Title:
  multipathd drops paths of a temporarily lost device

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1540407/+subscriptions

-- 
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs

[Bug 1540407] Comment bridged from LTC Bugzilla

[bugproxy]

--- Comment From thorsten.di...@de.ibm.com 2016-03-01 03:58 EDT---
(In reply to comment #19)
> As already promised, I will continue some long term tests (cable pull and
> other path loosing scenarios) on a 4.4.0-8 system with
> multipath-tools_0.5.0+git1.656f8865-4 and kpartx_0.5.0+git1.656f8865-4.
> Results should be available by tomorrow.

Results show, that this combination does not harm, instead it solves the
subject problem, even after 100 cycles.

Ryan, but it's up to you what you are going to integrate to fix this
bug.

-- 
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to multipath-tools in Ubuntu.
https://bugs.launchpad.net/bugs/1540407

Title:
  multipathd drops paths of a temporarily lost device

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1540407/+subscriptions

-- 
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs

[Bug 1540407] Comment bridged from LTC Bugzilla

[bugproxy]

--- Comment From thorsten.di...@de.ibm.com 2016-02-29 08:50 EDT---
Regarding the above strange behaviour in #5:
On a 4.4.0-8 system freshly installed (not updated from 4.4.0-7) with 
libzfcphbaapi0 installed the paths are recoverd and re-added as expected, as 
already described in case #2. And again, with case #1 the problem is 
reproducable.

What is this libzfcphbaapi0 package doing undercover? Normally it has no
interatice with multipath daemon. Does it change anything in the
multipath configuration?

Adding the libzfcphbaapi0 package to prevent symptoms of this bug can
not be the solution and it might be just by chance, that the symptom
vanished. I talked to zfcp development, and they confirmed that.

As already promised, I will continue some long term tests (cable pull
and other path loosing scenarios) on a 4.4.0-8 system with multipath-
tools_0.5.0+git1.656f8865-4 and kpartx_0.5.0+git1.656f8865-4. Results
should be available by tomorrow.

-- 
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to multipath-tools in Ubuntu.
https://bugs.launchpad.net/bugs/1540407

Title:
  multipathd drops paths of a temporarily lost device

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1540407/+subscriptions

-- 
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs

[Bug 1540407] Comment bridged from LTC Bugzilla

[bugproxy]

--- Comment From thorsten.di...@de.ibm.com 2016-02-29 06:27 EDT---
(In reply to comment #16)
> > Could you attempt to use the Debian package from Sid on top of the Ubuntu
> system?  This will help rule out kernel related issues vs. userspace package
> bugs.
>
> You can download from here
>
> http://ftp.us.debian.org/debian/pool/main/m/multipath-tools/multipath-
> tools_0.5.0+git1.656f8865-4_s390x.deb
>
> And then force install with
>
> sudo dpkg --force all --install
> multipath-tools_0.5.0+git1.656f8865-4_s390x.deb
>
> If this is successful, then I can prepare a package with some updated fixes
> from upstream; there are a number of changes to path related discovery in
> upstream multipath that aren't yet included in the Ubuntu version.

Hi Ryan,

I followed your proposal, installed 
multipath-tools_0.5.0+git1.656f8865-4_s390x.deb as described, and it worked 
fine and as expected, i.e. described problem does not occur.
On another system I installed both 
multipath-tools_0.5.0+git1.656f8865-4_s390x.deb and 
kpartx_0.5.0+git1.656f8865-4_s390x.deb - with the same result.
I did a detach/attach of the zfcp device and found, that it apperas for some 
seconds in status active faulty running, then it changed to failed faulty 
running and remained in that state until reattachment of the device. Then it 
changed back to active ready running.

Hi Christian,

in addition I did the following tests:

1. On a freshly installed xenial (installer version 427, kernel 4.4.0-8) with 
the already mentioned multipath.conf I found, that no zfcphbaapi stuff is 
installed per default.
kernel 4.4.0-8
multipath-tools 0.5.0-7ubuntu15 (was 7ubuntu14 before) and kpartx 
0.5.0-7ubuntu15
I did a detach/attach of the zfcp device and found, that it apperas for some 
seconds in status active faulty offline, then it changed to failed faulty 
offline and vanished and did not reappear upon reattachment of the device. I 
had the zfcp LUNs configure manually (via sysfs)

2. Then I installed zfcp-hbaapi-utils_2.1.1-0ubuntu1_s390x.deb and
libzfcphbaapi0_2.1.1-0ubuntu1_s390x.deb on top. With these two
additional packages I could not reproduce this problem.

3. I installed a system with kernel 4.4.0-7 and multipath-tools
0.5.0-7ubuntu14 + the above mentioned zfcphbaapi stuff. Problem was
reproducable.

4. I updated that system to multipath-tools 0.5.0-7ubuntu15. Problem was
still reproducable.

5. I updated that system to kernel 4.4.0-8. It should be now similar to
case #2. But problem was still reproducable. strange.

6. I switched several times back and forth between multipath-tools
0.5.0-7ubuntu15 and multipath-tools_0.5.0+git1.656f8865-4_s390x.deb.
With git1 it worked always, with 7ubuntu15 never. (although it did on a
freshly installed system, see #2)

So, for now we have two ways to solve/circumvent this problem:
a) use an updated multipath-tools package (from debian sid)
b) use an updated multipath-tools/kpartx packages from ubuntu (>= version 
0.5.0-7ubuntu15) AND zfcp-hbaapi-utils_2.1.1-0ubuntu1_s390x.deb and 
libzfcphbaapi0_2.1.1-0ubuntu1_s390x.deb on top. But that didn't help always 
(see case #5 above)

I found different values for path_selector in multipath-tools versions of 
ubuntu and debian:
ubuntu15: round-robin 0
debian-git: service-time 0
Then I set this in multipath.conf to the (usual) default "round-robin 0"
And with debian-git version the problem was still solved. So this setting 
should not matter.

Ryan,
although obviously only fixes from multipath-tools_0.5.0+git1.656f8865-4 are 
required to fix this problem, please consider and analyze, if it is better to 
get the complete set of patches in both multipath-tools AND kpartx from debian 
upstream (sid) instead of cherry picking some fixes.
The debian sid combination of multipath-tools AND kpartx does obviously not 
harm and is already 6 weeks old.

Anyway, I will do some other test with this combination.

-- 
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to multipath-tools in Ubuntu.
https://bugs.launchpad.net/bugs/1540407

Title:
  multipathd drops paths of a temporarily lost device

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1540407/+subscriptions

-- 
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs

[Bug 1540407] Comment bridged from LTC Bugzilla

[bugproxy]

--- Comment From thorsten.di...@de.ibm.com 2016-02-25 06:05 EDT---
Hi Ryan,

on the most recent repro I already had version: 0.5.0-7ubuntu14 - same symptom.
With
# echo "offline" > /sys/class/block/sdb/device/state
I could not reproduce this symptom on z Systems.

Approved repro is as follows:
z Systems LPAR: chchp -c 0 ;sleep 30;chchp -c 1 
z Systems z/VM guest: vmcp det ;sleep 30;vmcp att  "*"

-- 
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to multipath-tools in Ubuntu.
https://bugs.launchpad.net/bugs/1540407

Title:
  multipathd drops paths of a temporarily lost device

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1540407/+subscriptions

-- 
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs

[Bug 1540407] Comment bridged from LTC Bugzilla

[bugproxy]

--- Comment From thorsten.di...@de.ibm.com 2016-02-24 11:11 EDT---
Problem persists with Ubuntu kernel 4.4.0-7.

-- 
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to multipath-tools in Ubuntu.
https://bugs.launchpad.net/bugs/1540407

Title:
  multipathd drops paths of a temporarily lost device

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1540407/+subscriptions

-- 
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs