[jira] [Comment Edited] (SSHD-966) Deadlock on disconnection at the end of key-exchange
[ https://issues.apache.org/jira/browse/SSHD-966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17519426#comment-17519426 ] Thomas Wolf edited comment on SSHD-966 at 4/8/22 8:49 PM: -- {quote}Why can't we make AbstractSession.writePacket() block until the KEX state is DONE instead of queueing up the packet?{quote} To answer my own question: with the current architecture we can't do that because {{writePacket()}} may be called while the session lock and decoder lock are held. If we'd block then, no other request, not even KEX requests, could be handled. was (Author: wolft): {quote}Why can't we make AbstractSession.writePacket() block until the KEX state is DONE instead of queueing up the packet?{quote} To answer my own question: with the current architecture we can't do that because {{writePacket()}} may be called while the session lock is held. If we'd block then, no other request, not even KEX requests, could be handled. > Deadlock on disconnection at the end of key-exchange > > > Key: SSHD-966 > URL: https://issues.apache.org/jira/browse/SSHD-966 > Project: MINA SSHD > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Francois Ferrand >Assignee: Lyor Goldstein >Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > > We are using git-repo to download projects from Gerrit server, using SSH. > Gerrit is in version 2.16.16. which uses SSHD 2.0.0 and Mina 2.0.17 with NIO2 > backend. > One particularity of this setup is that git-repo creates a single control > master channel, and then downloads *lots* of Git repositories (500 > repositories, some of them relatively large), with some degree of > parallelism. This takes a long time, lots of data, and the multiplexed > connections are handled by gerrit in multiple threads. > In some cases, we experience a deadlock when an error happens at the end of > the key exchange, while sending pending packets: > {noformat} > Warning, the following threads are deadlocked : SSH git-upload-pack /project1 > (myuser), sshd-SshServer[df5f657]-nio2-thread-3 > "SSH git-upload-pack /project1 (myuser)" prio=1 BLOCKED > > org.apache.sshd.common.session.helpers.AbstractSession.writePacket(AbstractSession.java:1107) > > org.apache.sshd.common.channel.AbstractChannel.writePacket(AbstractChannel.java:798) > > org.apache.sshd.common.channel.ChannelOutputStream.flush(ChannelOutputStream.java:227) > > org.apache.sshd.common.channel.ChannelOutputStream.write(ChannelOutputStream.java:127) > > org.eclipse.jgit.transport.UploadPack$ResponseBufferedOutputStream.write(UploadPack.java:2183) > > org.eclipse.jgit.transport.SideBandOutputStream.writeBuffer(SideBandOutputStream.java:174) > > org.eclipse.jgit.transport.SideBandOutputStream.write(SideBandOutputStream.java:153) > > org.eclipse.jgit.internal.storage.pack.PackOutputStream.write(PackOutputStream.java:132) > > org.eclipse.jgit.internal.storage.file.PackFile.copyAsIs2(PackFile.java:614) > > org.eclipse.jgit.internal.storage.file.PackFile.copyAsIs(PackFile.java:433) > > org.eclipse.jgit.internal.storage.file.WindowCursor.copyObjectAsIs(WindowCursor.java:221) > > org.eclipse.jgit.internal.storage.pack.PackWriter.writeObjectImpl(PackWriter.java:1644) > > org.eclipse.jgit.internal.storage.pack.PackWriter.writeObject(PackWriter.java:1621) > > org.eclipse.jgit.internal.storage.pack.PackOutputStream.writeObject(PackOutputStream.java:171) > > org.eclipse.jgit.internal.storage.file.WindowCursor.writeObjects(WindowCursor.java:229) > > org.eclipse.jgit.internal.storage.pack.PackWriter.writeObjects(PackWriter.java:1609) > > org.eclipse.jgit.internal.storage.pack.PackWriter.writeObjects(PackWriter.java:1597) > > org.eclipse.jgit.internal.storage.pack.PackWriter.writePack(PackWriter.java:1154) > org.eclipse.jgit.transport.UploadPack.sendPack(UploadPack.java:2133) > org.eclipse.jgit.transport.UploadPack.sendPack(UploadPack.java:1947) > org.eclipse.jgit.transport.UploadPack.service(UploadPack.java:971) > org.eclipse.jgit.transport.UploadPack.upload(UploadPack.java:776) > com.google.gerrit.sshd.commands.Upload.runImpl(Upload.java:77) > > com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:98) > > com.google.gerrit.sshd.AbstractGitCommand.access$000(AbstractGitCommand.java:31) > > com.google.gerrit.sshd.AbstractGitCommand$1.run(AbstractGitCommand.java:63) > com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:467) > > com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:83) > java.util.concurrent.Executors$
[jira] [Comment Edited] (SSHD-966) Deadlock on disconnection at the end of key-exchange
[ https://issues.apache.org/jira/browse/SSHD-966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17055227#comment-17055227 ] Lyor Goldstein edited comment on SSHD-966 at 3/9/20, 5:22 PM: -- Sorry, I spoke too soon - the code has a bug inasmuch as if there is a lot of traffic the KEX state might never be marked as {{DONE}} (or marked as such very late). Back to the drawing board... was (Author: lgoldstein): Sorry, I spoke too soon - the code has a bug inasmuch as if there is a lot of traffic the KEX state might never be marked as {{DONE}} (or marked as such very late). > Deadlock on disconnection at the end of key-exchange > > > Key: SSHD-966 > URL: https://issues.apache.org/jira/browse/SSHD-966 > Project: MINA SSHD > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Francois Ferrand >Assignee: Lyor Goldstein >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > We are using git-repo to download projects from Gerrit server, using SSH. > Gerrit is in version 2.16.16. which uses SSHD 2.0.0 and Mina 2.0.17 with NIO2 > backend. > One particularity of this setup is that git-repo creates a single control > master channel, and then downloads *lots* of Git repositories (500 > repositories, some of them relatively large), with some degree of > parallelism. This takes a long time, lots of data, and the multiplexed > connections are handled by gerrit in multiple threads. > In some cases, we experience a deadlock when an error happens at the end of > the key exchange, while sending pending packets: > {noformat} > Warning, the following threads are deadlocked : SSH git-upload-pack /project1 > (myuser), sshd-SshServer[df5f657]-nio2-thread-3 > "SSH git-upload-pack /project1 (myuser)" prio=1 BLOCKED > > org.apache.sshd.common.session.helpers.AbstractSession.writePacket(AbstractSession.java:1107) > > org.apache.sshd.common.channel.AbstractChannel.writePacket(AbstractChannel.java:798) > > org.apache.sshd.common.channel.ChannelOutputStream.flush(ChannelOutputStream.java:227) > > org.apache.sshd.common.channel.ChannelOutputStream.write(ChannelOutputStream.java:127) > > org.eclipse.jgit.transport.UploadPack$ResponseBufferedOutputStream.write(UploadPack.java:2183) > > org.eclipse.jgit.transport.SideBandOutputStream.writeBuffer(SideBandOutputStream.java:174) > > org.eclipse.jgit.transport.SideBandOutputStream.write(SideBandOutputStream.java:153) > > org.eclipse.jgit.internal.storage.pack.PackOutputStream.write(PackOutputStream.java:132) > > org.eclipse.jgit.internal.storage.file.PackFile.copyAsIs2(PackFile.java:614) > > org.eclipse.jgit.internal.storage.file.PackFile.copyAsIs(PackFile.java:433) > > org.eclipse.jgit.internal.storage.file.WindowCursor.copyObjectAsIs(WindowCursor.java:221) > > org.eclipse.jgit.internal.storage.pack.PackWriter.writeObjectImpl(PackWriter.java:1644) > > org.eclipse.jgit.internal.storage.pack.PackWriter.writeObject(PackWriter.java:1621) > > org.eclipse.jgit.internal.storage.pack.PackOutputStream.writeObject(PackOutputStream.java:171) > > org.eclipse.jgit.internal.storage.file.WindowCursor.writeObjects(WindowCursor.java:229) > > org.eclipse.jgit.internal.storage.pack.PackWriter.writeObjects(PackWriter.java:1609) > > org.eclipse.jgit.internal.storage.pack.PackWriter.writeObjects(PackWriter.java:1597) > > org.eclipse.jgit.internal.storage.pack.PackWriter.writePack(PackWriter.java:1154) > org.eclipse.jgit.transport.UploadPack.sendPack(UploadPack.java:2133) > org.eclipse.jgit.transport.UploadPack.sendPack(UploadPack.java:1947) > org.eclipse.jgit.transport.UploadPack.service(UploadPack.java:971) > org.eclipse.jgit.transport.UploadPack.upload(UploadPack.java:776) > com.google.gerrit.sshd.commands.Upload.runImpl(Upload.java:77) > > com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:98) > > com.google.gerrit.sshd.AbstractGitCommand.access$000(AbstractGitCommand.java:31) > > com.google.gerrit.sshd.AbstractGitCommand$1.run(AbstractGitCommand.java:63) > com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:467) > > com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:83) > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > java.util.concurrent.FutureTask.run(FutureTask.java:266) > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) >
[jira] [Comment Edited] (SSHD-966) Deadlock on disconnection at the end of key-exchange
[ https://issues.apache.org/jira/browse/SSHD-966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17054317#comment-17054317 ] David Ostrovsky edited comment on SSHD-966 at 3/8/20, 7:12 AM: --- FTR: SSHD Upgrade to 2.4 version is here: [1] and here: [2]. [~Typz] Are you aware of this thread: [3], with this solution suggestion: [4]? {noformat} To cut a long story short after much trial, we fixed the issue by add sshd.rekeyBytesLimit = 0 sshd.rekeyTimeLimit = 0 to the gerrit config. {noformat} This was added by Gustaf in: [4]. [1] https://gerrit-review.googlesource.com/c/gerrit/+/254658 [2] https://gerrit-review.googlesource.com/c/gerrit/+/258152 [3] https://groups.google.com/d/topic/repo-discuss/wx75Ox5_v7o/discussion [4] https://gerrit.googlesource.com/gerrit/+/6f721702f07249c4babdd6d32d87765005081dcc was (Author: davido2): FTR: SSHD Upgrade to 2.4 version is here: [1] and here: [2]. [~Typz] Are you aware of this thread: [3], with this solution suggestion: [3]? {noformat} To cut a long story short after much trial, we fixed the issue by add sshd.rekeyBytesLimit = 0 sshd.rekeyTimeLimit = 0 to the gerrit config. {noformat} This was added by Gustaf in: [4]. [1] https://gerrit-review.googlesource.com/c/gerrit/+/254658 [2] https://gerrit-review.googlesource.com/c/gerrit/+/258152 [3] https://groups.google.com/d/topic/repo-discuss/wx75Ox5_v7o/discussion [4] https://gerrit.googlesource.com/gerrit/+/6f721702f07249c4babdd6d32d87765005081dcc > Deadlock on disconnection at the end of key-exchange > > > Key: SSHD-966 > URL: https://issues.apache.org/jira/browse/SSHD-966 > Project: MINA SSHD > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Francois Ferrand >Assignee: Lyor Goldstein >Priority: Major > > We are using git-repo to download projects from Gerrit server, using SSH. > Gerrit is in version 2.16.16. which uses SSHD 2.0.0 and Mina 2.0.17 with NIO2 > backend. > One particularity of this setup is that git-repo creates a single control > master channel, and then downloads *lots* of Git repositories (500 > repositories, some of them relatively large), with some degree of > parallelism. This takes a long time, lots of data, and the multiplexed > connections are handled by gerrit in multiple threads. > In some cases, we experience a deadlock when an error happens at the end of > the key exchange, while sending pending packets: > {noformat} > Warning, the following threads are deadlocked : SSH git-upload-pack /project1 > (myuser), sshd-SshServer[df5f657]-nio2-thread-3 > "SSH git-upload-pack /project1 (myuser)" prio=1 BLOCKED > > org.apache.sshd.common.session.helpers.AbstractSession.writePacket(AbstractSession.java:1107) > > org.apache.sshd.common.channel.AbstractChannel.writePacket(AbstractChannel.java:798) > > org.apache.sshd.common.channel.ChannelOutputStream.flush(ChannelOutputStream.java:227) > > org.apache.sshd.common.channel.ChannelOutputStream.write(ChannelOutputStream.java:127) > > org.eclipse.jgit.transport.UploadPack$ResponseBufferedOutputStream.write(UploadPack.java:2183) > > org.eclipse.jgit.transport.SideBandOutputStream.writeBuffer(SideBandOutputStream.java:174) > > org.eclipse.jgit.transport.SideBandOutputStream.write(SideBandOutputStream.java:153) > > org.eclipse.jgit.internal.storage.pack.PackOutputStream.write(PackOutputStream.java:132) > > org.eclipse.jgit.internal.storage.file.PackFile.copyAsIs2(PackFile.java:614) > > org.eclipse.jgit.internal.storage.file.PackFile.copyAsIs(PackFile.java:433) > > org.eclipse.jgit.internal.storage.file.WindowCursor.copyObjectAsIs(WindowCursor.java:221) > > org.eclipse.jgit.internal.storage.pack.PackWriter.writeObjectImpl(PackWriter.java:1644) > > org.eclipse.jgit.internal.storage.pack.PackWriter.writeObject(PackWriter.java:1621) > > org.eclipse.jgit.internal.storage.pack.PackOutputStream.writeObject(PackOutputStream.java:171) > > org.eclipse.jgit.internal.storage.file.WindowCursor.writeObjects(WindowCursor.java:229) > > org.eclipse.jgit.internal.storage.pack.PackWriter.writeObjects(PackWriter.java:1609) > > org.eclipse.jgit.internal.storage.pack.PackWriter.writeObjects(PackWriter.java:1597) > > org.eclipse.jgit.internal.storage.pack.PackWriter.writePack(PackWriter.java:1154) > org.eclipse.jgit.transport.UploadPack.sendPack(UploadPack.java:2133) > org.eclipse.jgit.transport.UploadPack.sendPack(UploadPack.java:1947) > org.eclipse.jgit.transport.UploadPack.service(UploadPack.java:971) > org.eclipse.jgit.transport.UploadPack.upload(UploadPack.java:776) > com.google.gerrit.sshd.commands.Upload.runImpl(Upload.java