Il 11/03/20 13:43, Nicola Murino ha scritto:
Hi all,
I want to share the performances analysis I recently did for SFTPGo,
the fully featured and highly configurable SFTP server written in Go
(https://github.com/drakkan/sftpgo).
When I decided to write an SFTP server I evaluated the available
libraries and I did some quick performance tests too.
Golang's SSH stack and pkg/sftp were able to easily saturate a Gigabit
connection and this seemed enough. I noticed that OpenSSH was faster,
but I didn't investigate further.
So I chose Golang for SFTPGo.
The project is growing fast and one of the users also noticed that
SFTPGo has lower performance than OpenSSH. He opened an issue
providing some stats when using a 40Gb Ethernet card.
I did some more profiling and discovered that the main bottlenecks
are, unsurprisingly, the cipher used and the message authentication.
So we can have a huge performance boost using a fast cipher with
implicit messages authentication, for example aes128-...@openssh.com,
however this cipher is not widely supported.
The most used cipher, and the one used in the user's tests is AES-CTR,
and the Golang implementations seems quite slow.
He noticed that an unmerged patch is available for Golang, greatly
improving AES-CTR performance:
https://go-review.googlesource.com/c/go/+/51670
I applied this patch and, while performance improved, the AES-CTR SFTP
transfers were still slower than the AES-GCM ones. The bottleneck is
now the MAC computation.
The tested hardware supports Intel SHA extensions but Golang's SHA256
implementation only uses the AVX2 extension.
Again, I was lucky: I can simply use minio/sha256-simd as a drop-in
replacement for Golang's SHA256 implementation:
https://github.com/drakkan/crypto/commit/17ef3d252b1c9d6124fa17889033e11eaa5c9ddb
The performance improved again, but OpenSSH was still faster.
To my great surprise I noticed that my very simple SCP implementation
(https://github.com/drakkan/sftpgo/blob/master/sftpd/scp.go) was now
as fast as OpenSSH!
So this time I have to look at pkg/sftp: I found some extraneous
copies/allocations in critical paths and I sent some pull requests
that are now merged in the master branch. Still, SFTP transfers were
slower than OpenSSH ones.
Compared to my SCP implementation the main difference is that pkg/sftp
allocates a new slice for each SFTP packet, while my SCP
implementation allocates a slice once and then reuses this slice.
Basically for each SFTP packet pkg/sftp does something like this:
|
data :=make([]byte,size)
|
So I wrote a proof of concept allocator that tries to avoid all these
extra allocations reusing the previously allocated slices:
https://github.com/drakkan/sftp/blob/master/allocator.go
And bingo! Now SFTPGo performance is very close to OpenSSH! You can
find the full benchmark results here:
https://github.com/drakkan/sftpgo/blob/master/docs/performance.md
Conclusion: I see several complaints about Go performance, especially
compared to Rust, but, at least in my use case, Go can be as fast as a
C project (such as OpenSSH). But some special attention is required
and thus this improved performance is not by default available to all
the users.
Now some questions:
1) for the pkg/sftp allocator I'm discussing with pkg/sftp maintainers
to find a way to get it included. Do you have smarter ideas to avoid
these allocations?
2) There is a patch available for AES-CTR in Golang (since 2017): I
wonder why it is not yet merged?
3) To improve SHA computation performance, I cannot find anything
usable in Golang itself. Is there any plan to have support for Intel
SHA Extensions and AVX512 directly in Golang anytime soon?
Thank you for this great programming language, it makes it really
simple to add new features to SFTPGo!
regards,
Nicola
Hi all,
a quick update on this.
SFTPGo master branch now includes the following performance improvements:
- allocator for pkg/sftp
- a crypto branch in which minio/sha256-simd replaces Go sha256 and some
other non-performance related changes
(https://github.com/drakkan/crypto/tree/sftpgo)
so only the AES-CTR patch for Go has to be manually applied.
Here you can find the updated performance doc:
https://github.com/drakkan/sftpgo/blob/master/docs/performance.md
regards,
Nicola
--
You received this message because you are subscribed to the Google
Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to golang-nuts+unsubscr...@googlegroups.com
<mailto:golang-nuts+unsubscr...@googlegroups.com>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/7cfdd60d-7415-4c2f-bd7d-9d0372461a3c%40googlegroups.com
<https://groups.google.com/d/msgid/golang-nuts/7cfdd60d-7415-4c2f-bd7d-9d0372461a3c%40googlegroups.com?utm_medium=email&utm_source=footer>.
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/9688f05f-e33d-3e88-706a-2ebb7266ce49%40gmail.com.
