[jira] [Commented] (SSHD-1179) Getting OOM after upgrading to 2.6.0
[
https://issues.apache.org/jira/browse/SSHD-1179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17359858#comment-17359858
]
Subramaniajeeva commented on SSHD-1179:
---
[~lgoldstein] [~gnodet] Can you please help with this issue?
> Getting OOM after upgrading to 2.6.0
>
>
> Key: SSHD-1179
> URL: https://issues.apache.org/jira/browse/SSHD-1179
> Project: MINA SSHD
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Subramaniajeeva
>Priority: Critical
>
> I upgraded mina version from 2.3.0 to 2.6.0 2 months back. Since then I could
> see the number of JVM threads keep on increasing(till ~12500), leading to OOM
> in production systems.
> Here is the stack trace with 2100+ similar threads:
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park({color:#80}Native Method{color})
> - parking to wait for *<0x00079f7630c8>* (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at
> java.util.concurrent.locks.LockSupport.park({color:#80}LockSupport.java:175{color})
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await({color:#80}AbstractQueuedSynchronizer.java:2039{color})
> at
> java.util.concurrent.LinkedBlockingQueue.take({color:#80}LinkedBlockingQueue.java:442{color})
> at
> org.apache.sshd.sftp.server.SftpSubsystem.run({color:#80}SftpSubsystem.java:267{color})
> at
> java.util.concurrent.Executors$RunnableAdapter.call({color:#80}Executors.java:511{color})
> at
> java.util.concurrent.FutureTask.run({color:#80}FutureTask.java:266{color})
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker({color:#80}ThreadPoolExecutor.java:1149{color})
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run({color:#80}ThreadPoolExecutor.java:624{color})
> at java.lang.Thread.run({color:#80}Thread.java:748{color})
>
> I tried to debug and I'm not able to isolate the root cause. Is it identified
> and fixed already in 2.7.0?
> [~lgoldstein] Any help would be appreciated.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: dev-unsubscr...@mina.apache.org
For additional commands, e-mail: dev-h...@mina.apache.org
[jira] [Created] (SSHD-1181) SFTP Get downloads empty file from servers which supports EOF indication after data
Pavel Pohner created SSHD-1181:
--
Summary: SFTP Get downloads empty file from servers which supports
EOF indication after data
Key: SSHD-1181
URL: https://issues.apache.org/jira/browse/SSHD-1181
Project: MINA SSHD
Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Pavel Pohner
So, apparently there's a bug in Mina implementation when downloading the file,
which is smaller than Mina's buffer (meaning it gets read in one iteration),
from servers, which send EOF indicator at the end of data
(https://datatracker.ietf.org/doc/html/draft-ietf-secsh-filexfer-13#[section-9.3|https://datatracker.ietf.org/doc/html/draft-ietf-secsh-filexfer-13#section-9.3]).
The bug seems to be in setting up EOF indicator in
{noformat}
org.apache.sshd.sftp.client.impl.SftpInputStreamAsync#pollBuffer{noformat}
prematurely (one iteration sooner), trace with me this example situation -
download of the small file (14 B):
At some point we arrive to mentioned function _pollBuffer()_, note that
_this.pendingReads_ is set to 2. In first call of _pollBuffer()_ we go through
this code on receiving SSH_FXP_DATA
{code:java}
if (type == SftpConstants.SSH_FXP_DATA) {
int dlen = buf.getInt();
int rpos = buf.rpos();
buf.rpos(rpos + dlen);
Boolean b = SftpHelper.getEndOfFileIndicatorValue(buf, client.getVersion());
if ((b != null) && b.booleanValue()) {
eofIndicator = true;
}{code}
Here, consider this situation, we're downloading file with 14B, remote server
adds EOF indicator to the end of the data, so it makes 15B, let's add some
necessary overhead (size etc.) which is, according to my experience, 13B, so we
receive 15 + 13 = 28B into initial buffer.
Now, if we populate these variables, _dlen_ = 14B (file size), _rpos_ = 13
(overhead), and we set _buf.rpos_ to 27 (14 + 13), but _wpos_ is at the moment
equal to 28B, as that's what we received. (We're not taking EOF into account
here)
Then the _SftpHelper.getEndOfFileIndicatorValue_ is called, which in it's
implementation looks like this:
{code:java}
public static Boolean getEndOfFileIndicatorValue(Buffer buffer, int version) {
return (version < SftpConstants.SFTP_V6) || (buffer.available() < 1) ? null
: buffer.getBoolean();
}
{code}
Pay attention to the _buffer.available()_ function as it's implemented like
this:
{code:java}
public int available() {
return wpos - rpos;
}
{code}
therefore returning 1 (28 - 27), condition is then resolved in
_SftpHelper.getEndOfFileIndicatorValue_ as a true and true is also returned to
the _pollBuffer_ and _eofIndicator_ is set to true.
If you'd keep tracing further, you'd find out that the buffer is then never
read and returned from _SftpInputStreamAsync_ class, just because
_eofIndicator_ was set sooner than it should have been.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: dev-unsubscr...@mina.apache.org
For additional commands, e-mail: dev-h...@mina.apache.org
[jira] [Commented] (SSHD-1179) Getting OOM after upgrading to 2.6.0
[
https://issues.apache.org/jira/browse/SSHD-1179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17359905#comment-17359905
]
Guillaume Nodet commented on SSHD-1179:
---
The most probable reason is that the {{SftpSubsystem}} are not closed correctly.
> Getting OOM after upgrading to 2.6.0
>
>
> Key: SSHD-1179
> URL: https://issues.apache.org/jira/browse/SSHD-1179
> Project: MINA SSHD
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Subramaniajeeva
>Priority: Critical
>
> I upgraded mina version from 2.3.0 to 2.6.0 2 months back. Since then I could
> see the number of JVM threads keep on increasing(till ~12500), leading to OOM
> in production systems.
> Here is the stack trace with 2100+ similar threads:
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park({color:#80}Native Method{color})
> - parking to wait for *<0x00079f7630c8>* (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at
> java.util.concurrent.locks.LockSupport.park({color:#80}LockSupport.java:175{color})
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await({color:#80}AbstractQueuedSynchronizer.java:2039{color})
> at
> java.util.concurrent.LinkedBlockingQueue.take({color:#80}LinkedBlockingQueue.java:442{color})
> at
> org.apache.sshd.sftp.server.SftpSubsystem.run({color:#80}SftpSubsystem.java:267{color})
> at
> java.util.concurrent.Executors$RunnableAdapter.call({color:#80}Executors.java:511{color})
> at
> java.util.concurrent.FutureTask.run({color:#80}FutureTask.java:266{color})
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker({color:#80}ThreadPoolExecutor.java:1149{color})
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run({color:#80}ThreadPoolExecutor.java:624{color})
> at java.lang.Thread.run({color:#80}Thread.java:748{color})
>
> I tried to debug and I'm not able to isolate the root cause. Is it identified
> and fixed already in 2.7.0?
> [~lgoldstein] Any help would be appreciated.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: dev-unsubscr...@mina.apache.org
For additional commands, e-mail: dev-h...@mina.apache.org
Migrate sshd-core 0.14.0 to 2.6.0
Hello Dev team, Currently, we have a system is using sshd-core 0.14.0. We are plan to migrate to 2.6.0. However, I saw that since v1.0.0 the structure is changed a lot. I tried to find the release note to check what is changed and how can we adapt to v1.0.0 but I could not. Could you give me the link so I can take a look and analyze how can we adapt to v1.0.0 as first step before adapt to v2.6.0 -- *Thank you and best regards.* *Ngo Ngoc Tuan.*
[jira] [Comment Edited] (SSHD-1179) Getting OOM after upgrading to 2.6.0
[
https://issues.apache.org/jira/browse/SSHD-1179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17359905#comment-17359905
]
Guillaume Nodet edited comment on SSHD-1179 at 6/9/21, 11:28 AM:
-
The most probable reason is that the {{SftpSubsystem}} are not closed
correctly. Those are closed automatically when the server channel is closed,
which happens when the client closes the channel or the connection.
was (Author: gnt):
The most probable reason is that the {{SftpSubsystem}} are not closed correctly.
> Getting OOM after upgrading to 2.6.0
>
>
> Key: SSHD-1179
> URL: https://issues.apache.org/jira/browse/SSHD-1179
> Project: MINA SSHD
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Subramaniajeeva
>Priority: Critical
>
> I upgraded mina version from 2.3.0 to 2.6.0 2 months back. Since then I could
> see the number of JVM threads keep on increasing(till ~12500), leading to OOM
> in production systems.
> Here is the stack trace with 2100+ similar threads:
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park({color:#80}Native Method{color})
> - parking to wait for *<0x00079f7630c8>* (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at
> java.util.concurrent.locks.LockSupport.park({color:#80}LockSupport.java:175{color})
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await({color:#80}AbstractQueuedSynchronizer.java:2039{color})
> at
> java.util.concurrent.LinkedBlockingQueue.take({color:#80}LinkedBlockingQueue.java:442{color})
> at
> org.apache.sshd.sftp.server.SftpSubsystem.run({color:#80}SftpSubsystem.java:267{color})
> at
> java.util.concurrent.Executors$RunnableAdapter.call({color:#80}Executors.java:511{color})
> at
> java.util.concurrent.FutureTask.run({color:#80}FutureTask.java:266{color})
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker({color:#80}ThreadPoolExecutor.java:1149{color})
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run({color:#80}ThreadPoolExecutor.java:624{color})
> at java.lang.Thread.run({color:#80}Thread.java:748{color})
>
> I tried to debug and I'm not able to isolate the root cause. Is it identified
> and fixed already in 2.7.0?
> [~lgoldstein] Any help would be appreciated.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: dev-unsubscr...@mina.apache.org
For additional commands, e-mail: dev-h...@mina.apache.org
[GitHub] [mina-sshd] alex-sherwin commented on pull request #198: SSHD-1166 - Support creating signed OpenSSH Certificates
alex-sherwin commented on pull request #198: URL: https://github.com/apache/mina-sshd/pull/198#issuecomment-858040806 This PR should be ready for re-review, all the suggestions have been implemented -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: dev-unsubscr...@mina.apache.org For additional commands, e-mail: dev-h...@mina.apache.org
[jira] [Work logged] (SSHD-1166) Support generating OpenSSH client certificates
[ https://issues.apache.org/jira/browse/SSHD-1166?focusedWorklogId=609351&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-609351 ] ASF GitHub Bot logged work on SSHD-1166: Author: ASF GitHub Bot Created on: 09/Jun/21 19:40 Start Date: 09/Jun/21 19:40 Worklog Time Spent: 10m Work Description: alex-sherwin commented on pull request #198: URL: https://github.com/apache/mina-sshd/pull/198#issuecomment-858040806 This PR should be ready for re-review, all the suggestions have been implemented -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 609351) Time Spent: 4h 20m (was: 4h 10m) > Support generating OpenSSH client certificates > -- > > Key: SSHD-1166 > URL: https://issues.apache.org/jira/browse/SSHD-1166 > Project: MINA SSHD > Issue Type: New Feature >Reporter: Alex Sherwin >Priority: Major > Time Spent: 4h 20m > Remaining Estimate: 0h > > Support generating OpenSSH client certificates > The OpenSSH certificate spec is defined here: > https://cvsweb.openbsd.org/src/usr.bin/ssh/PROTOCOL.certkeys?annotate=HEAD > MINA already supports using OpenSSH client certs for publickey auth via > SSHD-1161 > This ticket is to support creating/generating OpenSSH client certificates -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@mina.apache.org For additional commands, e-mail: dev-h...@mina.apache.org
[jira] [Assigned] (SSHD-1181) SFTP Get downloads empty file from servers which supports EOF indication after data
[
https://issues.apache.org/jira/browse/SSHD-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guillaume Nodet reassigned SSHD-1181:
-
Assignee: Guillaume Nodet
> SFTP Get downloads empty file from servers which supports EOF indication
> after data
> ---
>
> Key: SSHD-1181
> URL: https://issues.apache.org/jira/browse/SSHD-1181
> Project: MINA SSHD
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Pavel Pohner
>Assignee: Guillaume Nodet
>Priority: Major
> Labels: SFTP, mina, sshd
>
> So, apparently there's a bug in Mina implementation when downloading the
> file, which is smaller than Mina's buffer (meaning it gets read in one
> iteration), from servers, which send EOF indicator at the end of data
> (https://datatracker.ietf.org/doc/html/draft-ietf-secsh-filexfer-13#[section-9.3|https://datatracker.ietf.org/doc/html/draft-ietf-secsh-filexfer-13#section-9.3]).
> The bug seems to be in setting up EOF indicator in
> {noformat}
> org.apache.sshd.sftp.client.impl.SftpInputStreamAsync#pollBuffer{noformat}
> prematurely (one iteration sooner), trace with me this example situation -
> download of the small file (14 B):
> At some point we arrive to mentioned function _pollBuffer()_, note that
> _this.pendingReads_ is set to 2. In first call of _pollBuffer()_ we go
> through this code on receiving SSH_FXP_DATA
> {code:java}
> if (type == SftpConstants.SSH_FXP_DATA) {
> int dlen = buf.getInt();
> int rpos = buf.rpos();
> buf.rpos(rpos + dlen);
> Boolean b = SftpHelper.getEndOfFileIndicatorValue(buf,
> client.getVersion());
> if ((b != null) && b.booleanValue()) {
> eofIndicator = true;
> }{code}
> Here, consider this situation, we're downloading file with 14B, remote server
> adds EOF indicator to the end of the data, so it makes 15B, let's add some
> necessary overhead (size etc.) which is, according to my experience, 13B, so
> we receive 15 + 13 = 28B into initial buffer.
> Now, if we populate these variables, _dlen_ = 14B (file size), _rpos_ = 13
> (overhead), and we set _buf.rpos_ to 27 (14 + 13), but _wpos_ is at the
> moment equal to 28B, as that's what we received. (We're not taking EOF into
> account here)
> Then the _SftpHelper.getEndOfFileIndicatorValue_ is called, which in it's
> implementation looks like this:
> {code:java}
> public static Boolean getEndOfFileIndicatorValue(Buffer buffer, int version) {
> return (version < SftpConstants.SFTP_V6) || (buffer.available() < 1) ?
> null : buffer.getBoolean();
> }
> {code}
> Pay attention to the _buffer.available()_ function as it's implemented like
> this:
> {code:java}
> public int available() {
> return wpos - rpos;
> }
> {code}
> therefore returning 1 (28 - 27), condition is then resolved in
> _SftpHelper.getEndOfFileIndicatorValue_ as a true and true is also returned
> to the _pollBuffer_ and _eofIndicator_ is set to true.
> If you'd keep tracing further, you'd find out that the buffer is then never
> read and returned from _SftpInputStreamAsync_ class, just because
> _eofIndicator_ was set sooner than it should have been.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: dev-unsubscr...@mina.apache.org
For additional commands, e-mail: dev-h...@mina.apache.org
[jira] [Commented] (SSHD-1181) SFTP Get downloads empty file from servers which supports EOF indication after data
[
https://issues.apache.org/jira/browse/SSHD-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17360319#comment-17360319
]
Guillaume Nodet commented on SSHD-1181:
---
I've been able to reproduce the problem. The first thing was to change the
server side sftp code to send this EOF indicator when it makes sense. I should
be able to commit a fix with a test tomorrow.
> SFTP Get downloads empty file from servers which supports EOF indication
> after data
> ---
>
> Key: SSHD-1181
> URL: https://issues.apache.org/jira/browse/SSHD-1181
> Project: MINA SSHD
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Pavel Pohner
>Priority: Major
> Labels: SFTP, mina, sshd
>
> So, apparently there's a bug in Mina implementation when downloading the
> file, which is smaller than Mina's buffer (meaning it gets read in one
> iteration), from servers, which send EOF indicator at the end of data
> (https://datatracker.ietf.org/doc/html/draft-ietf-secsh-filexfer-13#[section-9.3|https://datatracker.ietf.org/doc/html/draft-ietf-secsh-filexfer-13#section-9.3]).
> The bug seems to be in setting up EOF indicator in
> {noformat}
> org.apache.sshd.sftp.client.impl.SftpInputStreamAsync#pollBuffer{noformat}
> prematurely (one iteration sooner), trace with me this example situation -
> download of the small file (14 B):
> At some point we arrive to mentioned function _pollBuffer()_, note that
> _this.pendingReads_ is set to 2. In first call of _pollBuffer()_ we go
> through this code on receiving SSH_FXP_DATA
> {code:java}
> if (type == SftpConstants.SSH_FXP_DATA) {
> int dlen = buf.getInt();
> int rpos = buf.rpos();
> buf.rpos(rpos + dlen);
> Boolean b = SftpHelper.getEndOfFileIndicatorValue(buf,
> client.getVersion());
> if ((b != null) && b.booleanValue()) {
> eofIndicator = true;
> }{code}
> Here, consider this situation, we're downloading file with 14B, remote server
> adds EOF indicator to the end of the data, so it makes 15B, let's add some
> necessary overhead (size etc.) which is, according to my experience, 13B, so
> we receive 15 + 13 = 28B into initial buffer.
> Now, if we populate these variables, _dlen_ = 14B (file size), _rpos_ = 13
> (overhead), and we set _buf.rpos_ to 27 (14 + 13), but _wpos_ is at the
> moment equal to 28B, as that's what we received. (We're not taking EOF into
> account here)
> Then the _SftpHelper.getEndOfFileIndicatorValue_ is called, which in it's
> implementation looks like this:
> {code:java}
> public static Boolean getEndOfFileIndicatorValue(Buffer buffer, int version) {
> return (version < SftpConstants.SFTP_V6) || (buffer.available() < 1) ?
> null : buffer.getBoolean();
> }
> {code}
> Pay attention to the _buffer.available()_ function as it's implemented like
> this:
> {code:java}
> public int available() {
> return wpos - rpos;
> }
> {code}
> therefore returning 1 (28 - 27), condition is then resolved in
> _SftpHelper.getEndOfFileIndicatorValue_ as a true and true is also returned
> to the _pollBuffer_ and _eofIndicator_ is set to true.
> If you'd keep tracing further, you'd find out that the buffer is then never
> read and returned from _SftpInputStreamAsync_ class, just because
> _eofIndicator_ was set sooner than it should have been.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: dev-unsubscr...@mina.apache.org
For additional commands, e-mail: dev-h...@mina.apache.org
