Re: [go-nuts] [ANN] go-scp: a scp client library in go
2016-09-21 8:31 GMT+09:00 Paul Borman :
> For cancellation you run the actual operation in a goroutine and return
> results on a channel. You can then select on the Deadline from the context
> and the returned result. If the deadline hits, you just shut down the SSH
> session. Does that make sense?
Yes, thank you for your explanation!
I'll try to implement it in my spare time.
Thanks
Hiroaki Nakamura
>
> -Paul
>
> On Tue, Sep 20, 2016 at 3:36 PM, Hiroaki Nakamura
> wrote:
>>
>> 2016-09-21 0:59 GMT+09:00 Paul Borman :
>> > I would suggest just putting your package out there and see how it is
>> > received. At some point, if it becomes the defacto package, moving it
>> > might
>> > make sense.
>>
>> OK, I understood.
>>
>> >
>> > I actually wrote an internal SCP package that works with a ssh
>> > multiplexer
>> > built on top of crypto/ssh. For various reasons, I am not in a position
>> > to
>> > OpenSource the package.
>>
>> I hope your SCP package will be released as an OpenSource.
>>
>> >
>> > It provides a NewSCP that returns an SCP object. You provide NewSCP
>> > with a
>> > configuration structure that specifies optional default timestamps,
>> > mode, if
>> > -p should be used, and if Start or Shell should be used to make the
>> > session
>> > (some SSH servers do not support exec mode (Start)).
>> >
>> > The SCP type only has 3 methods (and one is actually a convenience
>> > wrapper):
>> >
>> > func (*SCP) Send(ctx context.Context, dst string, srcs ...string) error
>> >
>> > Send sends the srcs to dst. If srcs has a length greater than 1, or any
>> > of
>> > the srcs is a Directory, dst must reference a directory on the remote
>> > host.
>> > Send returns any errors encountered. If an error is returned, it may
>> > contain
>> > multiple errors. User Errors(err) to retrieve the list of errors.
>> >
>> > func (*SCP) Receive(ctx context.Context, src string, f func(*Incoming)
>> > error) error
>> >
>> > Receive requests the directory or file src from the remote host. For
>> > each
>> > directory and file in src, f is called with a populated Incoming
>> > structure.
>> > If IsDir is not set then f must either populate W or return an error. In
>> > any
>> > event, if f returns an error then that file or directory's transfer is
>> > stopped. If the returned error is wrapped by Fatal{}, the entire SCP
>> > session
>> > is shutdown.
>> >
>> > type Incoming struct {
>> > Pathstring // Name of file on remote host
>> > Modeos.FileMode // Mode of the file on the remote host
>> > MTime time.Time // Last modified time from the remote host
>> > ATime time.Time // Last access time from the remote host
>> > Length int64 // Length of incoming file
>> > IsDir bool// Set if receiving a directory
>> > WErrerror // Error encountered during write, if any
>> > W io.Writer // Destination for read data
>> > NoClose bool// Do not close W when finished.
>> > }
>> >
>> > Incoming is where to send information about a file or directory. When an
>> > Incoming is provided to ReceiveFile, W must be set and IsDir must not be
>> > set. If the set W also implements io.Closer, W.Close is called after the
>> > fimal write to W, unless NoClose is set. NoClose is normally only set
>> > when
>> > using a single io.Writer for all data (such as os.Stdout).
>> >
>> > The convenience function is:
>> >
>> > func (*SCP) ReceiveFile(ctx context.Context, in *Incoming, src string)
>> > error
>> >
>> > ReceiveFile requests the remote host send the file named src.
>> > ReceiveFile
>> > writes the received contents to in.W. Path, Mode, MTime and ATime are
>> > set
>> > prior to the first write to in.W. MTime and ATime may be the zero value
>> > for
>> > time.Time if not provided by the remote host.
>> >
>> > If there is an error writing to in.W, in.WErr is set to the error.
>> > in.WErr
>> > is only valid after ReceiveFile returns. If there is an error receiving
>> > the
>> > file from the remote host, it is returned.
>> >
>> > You could easily imagine a wrapper that called Receive and provided an
>> > internal function that did whatever needed to be done, or even a stock
>> > function that can be passed to Receive.
>> >
>> > I hope this is helpful.
>>
>> Thanks for the spec about your SCP package.
>> I think it is more general than mine.
>>
>> However most of my cases are copy a single file or a directory,
>> so I think I'd keep my APIs for the moment.
>>
>> I added NewSCP and SCP
>>
>> https://github.com/hnakamur/go-scp/commit/b43795fc10ef52b02a76653b643fa5aabb7b85a7
>>
>> One thing I would like to add is the cancellation support using
>> context.Context.
>> However ssh.Run does not take a context.Context as an argument,
>> so I don't know how to achieve that.
>> https://godoc.org/golang.org/x/crypto/ssh#Session.Run
>>
>> Could you tell me how you implement the cancellation?
>>
>> Thanks,
>> Hiroaki Nakamura
>>
>> >
>> > -Paul
>> >
>> > On Tue, Sep 20, 2016 at 6:47 AM, Hir
Re: [go-nuts] [ANN] go-scp: a scp client library in go
For cancellation you run the actual operation in a goroutine and return
results on a channel. You can then select on the Deadline from the context
and the returned result. If the deadline hits, you just shut down the SSH
session. Does that make sense?
-Paul
On Tue, Sep 20, 2016 at 3:36 PM, Hiroaki Nakamura
wrote:
> 2016-09-21 0:59 GMT+09:00 Paul Borman :
> > I would suggest just putting your package out there and see how it is
> > received. At some point, if it becomes the defacto package, moving it
> might
> > make sense.
>
> OK, I understood.
>
> >
> > I actually wrote an internal SCP package that works with a ssh
> multiplexer
> > built on top of crypto/ssh. For various reasons, I am not in a position
> to
> > OpenSource the package.
>
> I hope your SCP package will be released as an OpenSource.
>
> >
> > It provides a NewSCP that returns an SCP object. You provide NewSCP
> with a
> > configuration structure that specifies optional default timestamps,
> mode, if
> > -p should be used, and if Start or Shell should be used to make the
> session
> > (some SSH servers do not support exec mode (Start)).
> >
> > The SCP type only has 3 methods (and one is actually a convenience
> wrapper):
> >
> > func (*SCP) Send(ctx context.Context, dst string, srcs ...string) error
> >
> > Send sends the srcs to dst. If srcs has a length greater than 1, or any
> of
> > the srcs is a Directory, dst must reference a directory on the remote
> host.
> > Send returns any errors encountered. If an error is returned, it may
> contain
> > multiple errors. User Errors(err) to retrieve the list of errors.
> >
> > func (*SCP) Receive(ctx context.Context, src string, f func(*Incoming)
> > error) error
> >
> > Receive requests the directory or file src from the remote host. For each
> > directory and file in src, f is called with a populated Incoming
> structure.
> > If IsDir is not set then f must either populate W or return an error. In
> any
> > event, if f returns an error then that file or directory's transfer is
> > stopped. If the returned error is wrapped by Fatal{}, the entire SCP
> session
> > is shutdown.
> >
> > type Incoming struct {
> > Pathstring // Name of file on remote host
> > Modeos.FileMode // Mode of the file on the remote host
> > MTime time.Time // Last modified time from the remote host
> > ATime time.Time // Last access time from the remote host
> > Length int64 // Length of incoming file
> > IsDir bool// Set if receiving a directory
> > WErrerror // Error encountered during write, if any
> > W io.Writer // Destination for read data
> > NoClose bool// Do not close W when finished.
> > }
> >
> > Incoming is where to send information about a file or directory. When an
> > Incoming is provided to ReceiveFile, W must be set and IsDir must not be
> > set. If the set W also implements io.Closer, W.Close is called after the
> > fimal write to W, unless NoClose is set. NoClose is normally only set
> when
> > using a single io.Writer for all data (such as os.Stdout).
> >
> > The convenience function is:
> >
> > func (*SCP) ReceiveFile(ctx context.Context, in *Incoming, src string)
> error
> >
> > ReceiveFile requests the remote host send the file named src. ReceiveFile
> > writes the received contents to in.W. Path, Mode, MTime and ATime are set
> > prior to the first write to in.W. MTime and ATime may be the zero value
> for
> > time.Time if not provided by the remote host.
> >
> > If there is an error writing to in.W, in.WErr is set to the error.
> in.WErr
> > is only valid after ReceiveFile returns. If there is an error receiving
> the
> > file from the remote host, it is returned.
> >
> > You could easily imagine a wrapper that called Receive and provided an
> > internal function that did whatever needed to be done, or even a stock
> > function that can be passed to Receive.
> >
> > I hope this is helpful.
>
> Thanks for the spec about your SCP package.
> I think it is more general than mine.
>
> However most of my cases are copy a single file or a directory,
> so I think I'd keep my APIs for the moment.
>
> I added NewSCP and SCP
> https://github.com/hnakamur/go-scp/commit/b43795fc10ef52b02a76653b643fa5
> aabb7b85a7
>
> One thing I would like to add is the cancellation support using
> context.Context.
> However ssh.Run does not take a context.Context as an argument,
> so I don't know how to achieve that.
> https://godoc.org/golang.org/x/crypto/ssh#Session.Run
>
> Could you tell me how you implement the cancellation?
>
> Thanks,
> Hiroaki Nakamura
>
> >
> > -Paul
> >
> > On Tue, Sep 20, 2016 at 6:47 AM, Hiroaki Nakamura
> > wrote:
> >>
> >> Hi Paul,
> >> Thanks for your feedback!
> >>
> >> 2016-09-20 0:02 GMT+09:00 Paul Borman :
> >> > Adding an scp package is a nice addition.
> >>
> >> I agree.
> >> Should I send a pull request to https://github.com/golang/crypto?
> >> If yes, what package? golang.org/x/crypto/scp or
> >> golang.org
Re: [go-nuts] [ANN] go-scp: a scp client library in go
2016-09-21 0:59 GMT+09:00 Paul Borman :
> I would suggest just putting your package out there and see how it is
> received. At some point, if it becomes the defacto package, moving it might
> make sense.
OK, I understood.
>
> I actually wrote an internal SCP package that works with a ssh multiplexer
> built on top of crypto/ssh. For various reasons, I am not in a position to
> OpenSource the package.
I hope your SCP package will be released as an OpenSource.
>
> It provides a NewSCP that returns an SCP object. You provide NewSCP with a
> configuration structure that specifies optional default timestamps, mode, if
> -p should be used, and if Start or Shell should be used to make the session
> (some SSH servers do not support exec mode (Start)).
>
> The SCP type only has 3 methods (and one is actually a convenience wrapper):
>
> func (*SCP) Send(ctx context.Context, dst string, srcs ...string) error
>
> Send sends the srcs to dst. If srcs has a length greater than 1, or any of
> the srcs is a Directory, dst must reference a directory on the remote host.
> Send returns any errors encountered. If an error is returned, it may contain
> multiple errors. User Errors(err) to retrieve the list of errors.
>
> func (*SCP) Receive(ctx context.Context, src string, f func(*Incoming)
> error) error
>
> Receive requests the directory or file src from the remote host. For each
> directory and file in src, f is called with a populated Incoming structure.
> If IsDir is not set then f must either populate W or return an error. In any
> event, if f returns an error then that file or directory's transfer is
> stopped. If the returned error is wrapped by Fatal{}, the entire SCP session
> is shutdown.
>
> type Incoming struct {
> Pathstring // Name of file on remote host
> Modeos.FileMode // Mode of the file on the remote host
> MTime time.Time // Last modified time from the remote host
> ATime time.Time // Last access time from the remote host
> Length int64 // Length of incoming file
> IsDir bool// Set if receiving a directory
> WErrerror // Error encountered during write, if any
> W io.Writer // Destination for read data
> NoClose bool// Do not close W when finished.
> }
>
> Incoming is where to send information about a file or directory. When an
> Incoming is provided to ReceiveFile, W must be set and IsDir must not be
> set. If the set W also implements io.Closer, W.Close is called after the
> fimal write to W, unless NoClose is set. NoClose is normally only set when
> using a single io.Writer for all data (such as os.Stdout).
>
> The convenience function is:
>
> func (*SCP) ReceiveFile(ctx context.Context, in *Incoming, src string) error
>
> ReceiveFile requests the remote host send the file named src. ReceiveFile
> writes the received contents to in.W. Path, Mode, MTime and ATime are set
> prior to the first write to in.W. MTime and ATime may be the zero value for
> time.Time if not provided by the remote host.
>
> If there is an error writing to in.W, in.WErr is set to the error. in.WErr
> is only valid after ReceiveFile returns. If there is an error receiving the
> file from the remote host, it is returned.
>
> You could easily imagine a wrapper that called Receive and provided an
> internal function that did whatever needed to be done, or even a stock
> function that can be passed to Receive.
>
> I hope this is helpful.
Thanks for the spec about your SCP package.
I think it is more general than mine.
However most of my cases are copy a single file or a directory,
so I think I'd keep my APIs for the moment.
I added NewSCP and SCP
https://github.com/hnakamur/go-scp/commit/b43795fc10ef52b02a76653b643fa5aabb7b85a7
One thing I would like to add is the cancellation support using context.Context.
However ssh.Run does not take a context.Context as an argument,
so I don't know how to achieve that.
https://godoc.org/golang.org/x/crypto/ssh#Session.Run
Could you tell me how you implement the cancellation?
Thanks,
Hiroaki Nakamura
>
> -Paul
>
> On Tue, Sep 20, 2016 at 6:47 AM, Hiroaki Nakamura
> wrote:
>>
>> Hi Paul,
>> Thanks for your feedback!
>>
>> 2016-09-20 0:02 GMT+09:00 Paul Borman :
>> > Adding an scp package is a nice addition.
>>
>> I agree.
>> Should I send a pull request to https://github.com/golang/crypto?
>> If yes, what package? golang.org/x/crypto/scp or
>> golang.org/x/crypto/ssh/scp?
>>
>> > You might want to consider simple names like:
>> >
>> > Send - Sends from []byte to file on remote host
>> > SendDir - Send files in dir to a remote host
>> > SendFile - Sends the contents of a file to the remote host
>> > Fetch - Fetches the contents of a file on remote host into memory
>> > FetchFile - Fetches a file from remote host into file on local host
>> > FetchDir - Fetches the files in a directory from the remote host
>> >
>> > These would translate in code to names like scp.SendFile, which is
>> > pretty
>> > descriptive all by itself
Re: [go-nuts] [ANN] go-scp: a scp client library in go
I would suggest just putting your package out there and see how it is
received. At some point, if it becomes the defacto package, moving it
might make sense.
I actually wrote an internal SCP package that works with a ssh multiplexer
built on top of crypto/ssh. For various reasons, I am not in a position to
OpenSource the package.
It provides a NewSCP that returns an SCP object. You provide NewSCP with a
configuration structure that specifies optional default timestamps, mode,
if -p should be used, and if Start or Shell should be used to make the
session (some SSH servers do not support exec mode (Start)).
The SCP type only has 3 methods (and one is actually a convenience wrapper):
func (*SCP) *Send*(ctx context.Context, dst string, srcs ...string) error
*Send sends the srcs to dst. If srcs has a length greater than 1, or any of
the srcs is a Directory, dst must reference a directory on the remote host.
Send returns any errors encountered. If an error is returned, it may
contain multiple errors. User Errors(err) to retrieve the list of errors.*
func (*SCP) *Receive*(ctx context.Context, src string, f func(*Incoming)
error) error
*Receive requests the directory or file src from the remote host. For each
directory and file in src, f is called with a populated Incoming structure.
If IsDir is not set then f must either populate W or return an error. In
any event, if f returns an error then that file or directory's transfer is
stopped. If the returned error is wrapped by Fatal{}, the entire SCP
session is shutdown.*
type *Incoming* struct {
Pathstring // Name of file on remote host
Modeos.FileMode // Mode of the file on the remote host
MTime time.Time // Last modified time from the remote host
ATime time.Time // Last access time from the remote host
Length int64 // Length of incoming file
IsDir bool// Set if receiving a directory
WErrerror // Error encountered during write, if any
W io.Writer // Destination for read data
NoClose bool// Do not close W when finished.
}
*Incoming is where to send information about a file or directory. When an
Incoming is provided to ReceiveFile, W must be set and IsDir must not be
set. If the set W also implements io.Closer, W.Close is called after the
fimal write to W, unless NoClose is set. NoClose is normally only set when
using a single io.Writer for all data (such as os.Stdout).*
The convenience function is:
func (*SCP) *ReceiveFile*(ctx context.Context, in *Incoming, src string)
error
*ReceiveFile requests the remote host send the file named src. ReceiveFile
writes the received contents to in.W. Path, Mode, MTime and ATime are set
prior to the first write to in.W. MTime and ATime may be the zero value for
time.Time if not provided by the remote host.*
*If there is an error writing to in.W, in.WErr is set to the error. in.WErr
is only valid after ReceiveFile returns. If there is an error receiving the
file from the remote host, it is returned.*
You could easily imagine a wrapper that called Receive and provided an
internal function that did whatever needed to be done, or even a stock
function that can be passed to Receive.
I hope this is helpful.
-Paul
On Tue, Sep 20, 2016 at 6:47 AM, Hiroaki Nakamura
wrote:
> Hi Paul,
> Thanks for your feedback!
>
> 2016-09-20 0:02 GMT+09:00 Paul Borman :
> > Adding an scp package is a nice addition.
>
> I agree.
> Should I send a pull request to https://github.com/golang/crypto?
> If yes, what package? golang.org/x/crypto/scp or
> golang.org/x/crypto/ssh/scp?
>
> > You might want to consider simple names like:
> >
> > Send - Sends from []byte to file on remote host
> > SendDir - Send files in dir to a remote host
> > SendFile - Sends the contents of a file to the remote host
> > Fetch - Fetches the contents of a file on remote host into memory
> > FetchFile - Fetches a file from remote host into file on local host
> > FetchDir - Fetches the files in a directory from the remote host
> >
> > These would translate in code to names like scp.SendFile, which is pretty
> > descriptive all by itself.
>
> Thanks for simple and descriptive function names.
> I renamed functions.
> https://github.com/hnakamur/go-scp/commit/8cd6d9e5ab17187556e5efc3a666d2
> 9b4b561c78
>
> >
> > For the directory copy, it might be better to have a function return the
> > io.Writer to write the file to, rather than force the files into a
> > directory. This would make it easy to keep the contents in memory,
> change
> > file names, or whatever.
>
> Yes, I agree it might be better not to force the files in a directory.
> However I don't think having a function return the io.Writer will do
> since we need to read a reply for each scp protocol header or body.
>
> I had read the article below and the openssh source code and
> implemented my scp package https://github.com/hnakamur/go-scp.
>
> "How the SCP protocol works (Jan Pechanec's weblog)"
> https://blogs.oracle.com/janp/entry/how_the
Re: [go-nuts] [ANN] go-scp: a scp client library in go
Hi Paul, Thanks for your feedback! 2016-09-20 0:02 GMT+09:00 Paul Borman : > Adding an scp package is a nice addition. I agree. Should I send a pull request to https://github.com/golang/crypto? If yes, what package? golang.org/x/crypto/scp or golang.org/x/crypto/ssh/scp? > You might want to consider simple names like: > > Send - Sends from []byte to file on remote host > SendDir - Send files in dir to a remote host > SendFile - Sends the contents of a file to the remote host > Fetch - Fetches the contents of a file on remote host into memory > FetchFile - Fetches a file from remote host into file on local host > FetchDir - Fetches the files in a directory from the remote host > > These would translate in code to names like scp.SendFile, which is pretty > descriptive all by itself. Thanks for simple and descriptive function names. I renamed functions. https://github.com/hnakamur/go-scp/commit/8cd6d9e5ab17187556e5efc3a666d29b4b561c78 > > For the directory copy, it might be better to have a function return the > io.Writer to write the file to, rather than force the files into a > directory. This would make it easy to keep the contents in memory, change > file names, or whatever. Yes, I agree it might be better not to force the files in a directory. However I don't think having a function return the io.Writer will do since we need to read a reply for each scp protocol header or body. I had read the article below and the openssh source code and implemented my scp package https://github.com/hnakamur/go-scp. "How the SCP protocol works (Jan Pechanec's weblog)" https://blogs.oracle.com/janp/entry/how_the_scp_protocol_works I built two structs sourceProtocol and sinkProtocol https://github.com/hnakamur/go-scp/blob/master/protocol.go I had thought to export these structs or make interfaces for that. However the implementation of two functions SendDir and FetchDir which using these structs become somewhat complex. https://github.com/hnakamur/go-scp/blob/8cd6d9e5ab17187556e5efc3a666d29b4b561c78/source.go#L81-L175 https://github.com/hnakamur/go-scp/blob/8cd6d9e5ab17187556e5efc3a666d29b4b561c78/sink.go#L111-L223 So I was not confident about exporting sourceProtocol and sinkProtocol, and I did not export them at the time. If we can define structs, functions or interfaces which are easy to use, I'm glad to export them. Do you have an idea about such structs, functions or interfaces? Thanks! Hiroaki Nakamura > > On Fri, Sep 16, 2016 at 9:41 AM, Hiroaki Nakamura > wrote: >> >> Hi all, >> >> I noticed the golang.org/x/crypto/ssh package exists, but the scp >> package does not. >> So I wrote a scp client library in go. >> https://github.com/hnakamur/go-scp >> >> I also wrote a sshd server just usable for testing go-scp. >> https://github.com/hnakamur/go-sshd >> >> Right now, go-scp only exports high level functions which are supposed >> to be easy to use. >> https://godoc.org/github.com/hnakamur/go-scp >> >> However I wonder if there APIs can be improved. For example, >> better function names and better arguments. >> >> Could you tell me what you think? >> Thanks! >> >> Hiroaki Nakamura >> >> -- >> You received this message because you are subscribed to the Google Groups >> "golang-nuts" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to golang-nuts+unsubscr...@googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. > > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [go-nuts] [ANN] go-scp: a scp client library in go
Adding an scp package is a nice addition. You might want to consider simple names like: Send - Sends from []byte to file on remote host SendDir - Send files in dir to a remote host SendFile - Sends the contents of a file to the remote host Fetch - Fetches the contents of a file on remote host into memory FetchFile - Fetches a file from remote host into file on local host FetchDir - Fetches the files in a directory from the remote host These would translate in code to names like scp.SendFile, which is pretty descriptive all by itself. For the directory copy, it might be better to have a function return the io.Writer to write the file to, rather than force the files into a directory. This would make it easy to keep the contents in memory, change file names, or whatever. On Fri, Sep 16, 2016 at 9:41 AM, Hiroaki Nakamura wrote: > Hi all, > > I noticed the golang.org/x/crypto/ssh package exists, but the scp > package does not. > So I wrote a scp client library in go. > https://github.com/hnakamur/go-scp > > I also wrote a sshd server just usable for testing go-scp. > https://github.com/hnakamur/go-sshd > > Right now, go-scp only exports high level functions which are supposed > to be easy to use. > https://godoc.org/github.com/hnakamur/go-scp > > However I wonder if there APIs can be improved. For example, > better function names and better arguments. > > Could you tell me what you think? > Thanks! > > Hiroaki Nakamura > > -- > You received this message because you are subscribed to the Google Groups > "golang-nuts" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to golang-nuts+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[go-nuts] [ANN] go-scp: a scp client library in go
Hi all, I noticed the golang.org/x/crypto/ssh package exists, but the scp package does not. So I wrote a scp client library in go. https://github.com/hnakamur/go-scp I also wrote a sshd server just usable for testing go-scp. https://github.com/hnakamur/go-sshd Right now, go-scp only exports high level functions which are supposed to be easy to use. https://godoc.org/github.com/hnakamur/go-scp However I wonder if there APIs can be improved. For example, better function names and better arguments. Could you tell me what you think? Thanks! Hiroaki Nakamura -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
